title: MPI_Gather Vector of Objects
body: <p>I am writing a mpich program for parallel sorting. I need to use the mpi_gather interface, but it doesn't support passing vector of objects. So I use <strong>boost_serialization</strong>.</p>
  
  <h2>Implementation</h2>
  
  <p>I use <strong>boost_serialization</strong> to serialize the vector. </p>
  
  <pre><code>std::string serial_str;
  boost::iostreams::back_insert_device&lt;std::string&gt; inserter(serial_str);
  boost::iostreams::stream&lt;boost::iostreams::back_insert_device&lt;std::string&gt;&gt; s(inserter);
  boost::archive::binary_oarchive send_ar(s);
  //samples is the vector&lt;object&gt;
  send_ar &lt;&lt; samples;
  s.flush();
  int len = serial_str.size();
  </code></pre>
  
  <p>Then, I use <strong>mpi_gather</strong> to send all the <strong>serial_str</strong> to root process(<strong>data_recv</strong>).  </p>
  
  <pre><code>char *data_recv = NULL;
  if(myid == 0){
  data_recv = (char*)malloc(sizeof(char) * (len_all+1));
      data_recv[len_all] = '\0';
  }
  MPI_Gather((void*)serial_str.data(), len, MPI_BYTE, data_recv, len, MPI_BYTE, 0, MPI_COMM_WORLD);
  </code></pre>
  
  <p>Finally, I <strong>deserialize the data in data_recv</strong>.</p>
  
  <pre><code>boost::iostreams::basic_array_source&lt;char&gt; device(data_recv,len_all);
      boost::iostreams::stream&lt;boost::iostreams::basic_array_source&lt;char&gt;&gt; s(device);
      boost::archive::binary_iarchive recv_ar(s);
  
  std::vector&lt;mdata&gt; recv_vec;    
      recv_ar &gt;&gt; recv_vec;
  </code></pre>
  
  <p>My implementation is based on <a href="https://stackoverflow.com/questions/31014044/how-to-send-a-set-object-in-mpi-send">How to send a set object in MPI_Send</a></p>
  
  <h2>Problem</h2>
  
  <p>I can't deserialize the data in data_recv correctly. I printed the data_recv, then I found the data in <strong>data_recv</strong> is incorrectly formatted after mpi_gather. <strong>The second archive covered the first</strong>.(marked in bold)</p>
  
  <blockquote>
    <p>serialization::archive                                                                                        </p>
    
    <p>XylvXe-M X 00000000000000000000000000002595  DDDDFFFFCCCCBBBB111133332222DDDD333388888888FFFF2222<strong>serialization::archive</strong>000000000000000000023D0  EEEE7777EEEE44447777BBBB8888AAAA0000AAAAAAAAFFFF1111           </p>
    
    <p>XylvXe-M X 00000000000000000000000000002595  DDDDFFFFCCCCBBBB111133332222DDDD333388888888FFFF2222O@f&amp;!O,t.b X 000000000000000000000000000023D0  EEEE7777EEEE44447777BBBB8888AAAA0000AAAAAAAAFFFF1111</p>
  </blockquote>
  
  <p>The correct format should be:(no overlap so I can deserialize)</p>
  
  <blockquote>
    <p>serialization::archive                                                                                        </p>
    
    <p>XylvXe-M X 00000000000000000000000000002595  DDDDFFFFCCCCBBBB111133332222DDDD333388888888FFFF2222O@f&amp;!O,t.b X 000000000000000000000000000023D0  EEEE7777EEEE44447777BBBB8888AAAA0000AAAAAAAAFFFF1111</p>
    
    <p>serialization::archive                                                                                        </p>
    
    <p>XylvXe-M X 00000000000000000000000000002595  DDDDFFFFCCCCBBBB111133332222DDDD333388888888FFFF2222O@f&amp;!O,t.b X 000000000000000000000000000023D0  EEEE7777EEEE44447777BBBB8888AAAA0000AAAAAAAAFFFF1111</p>
  </blockquote>
  
  <h2>Question</h2>
  
  <p>Why did this happen? Is it because the mpi_gather isn't compatible with c++ object?
  If someone could help me out, it would solve my big problem.
  Thank you!</p>
  
  <h2>code</h2>
  
  <pre><code>//processor rank, and total number of processors
  int myid, world_size;
  
  //for timing used by root processor
  double startwtime = 0.0, endwtime;
  
  //init MPI World
  MPI_Init(&amp;argc, &amp;argv);
  MPI_Comm_size(MPI_COMM_WORLD, &amp;world_size);
  MPI_Comm_rank(MPI_COMM_WORLD, &amp;myid);
  //get the processor name
  char processor_name[MPI_MAX_PROCESSOR_NAME];
  int name_len;
  MPI_Get_processor_name(processor_name,&amp;name_len);
  
  //read local data
  std::vector&lt;mdata&gt; mdatas; 
  string data_path = "/home/jiang/mpi_data";
  readAsciiData(data_path, mdatas);
  cout &lt;&lt;"rank: "&lt;&lt;myid &lt;&lt;" mdata_vector.size(): "&lt;&lt;mdatas.size()&lt;&lt;endl;
  
  //local sort according ASCII order 
  std::sort(mdatas.begin(), mdatas.end());
  
  //regular sample
  std::vector&lt;mdata&gt; samples;
  for(int i=0; i&lt;mdatas.size(); i=i+mdatas.size()/world_size){
  samples.push_back(mdatas[i]);
  }
  
  //gather the regular samples
  //passing data in byte stream by using boost serialization
  
  std::string serial_str;
  boost::iostreams::back_insert_device&lt;std::string&gt; inserter(serial_str);
  boost::iostreams::stream&lt;boost::iostreams::back_insert_device&lt;std::string&gt;&gt; s(inserter);
  boost::archive::binary_oarchive send_ar(s);
  
  send_ar &lt;&lt; samples;
  s.flush();
  int len = serial_str.size();
  //int len = s.str().size();
  int *pivot_lens = NULL;
  if(myid == 0){
  pivot_lens = (int*)malloc(sizeof(int) * world_size);
  }
  
  cout  &lt;&lt;serial_str &lt;&lt;endl;
  //first, gathering the lens and calculate the sum
  cout &lt;&lt; "rank " &lt;&lt; myid &lt;&lt; " on "&lt;&lt; processor_name &lt;&lt; " is sending len: "&lt;&lt; len &lt;&lt; endl;
  MPI_Gather(&amp;len, 1, MPI_INT, pivot_lens, 1, MPI_INT, 0, MPI_COMM_WORLD);
  //calculate the sum of lens
  int len_all = 0;
  if(myid == 0){
  for(int i=0;i&lt;world_size;i++){
         len_all = len_all + pivot_lens[i];
         //cout &lt;&lt; pivot_lens[i] &lt;&lt; endl;     
  }
      cout &lt;&lt; "len_all:" &lt;&lt; len_all &lt;&lt; endl;
      free(pivot_lens);   
  }
  //then, gathering string of bytes from all the processes
  char *data_recv = NULL;
  if(myid == 0){
  data_recv = (char*)malloc(sizeof(char) * (len_all+1));
      data_recv[len_all] = '\0';
  
  }
  
  
  MPI_Gather((void*)serial_str.data(), len, MPI_BYTE, data_recv, len, MPI_BYTE, 0, MPI_COMM_WORLD);
  
  // cout &lt;&lt; serial_str &lt;&lt;endl;
  if(myid == 0){
  //deconstructe from byte of string to vector&lt;mdata&gt;
  boost::iostreams::basic_array_source&lt;char&gt; device(data_recv,len_all);
      boost::iostreams::stream&lt;boost::iostreams::basic_array_source&lt;char&gt;&gt; s(device);
      boost::archive::binary_iarchive recv_ar(s);
  
  std::vector&lt;mdata&gt; recv_vec;    
      recv_ar &gt;&gt; recv_vec;
  int count =0;
  for(int i=0;i&lt;len_all;i++){
      cout&lt;&lt;data_recv[i];
      count ++;
  }
  cout &lt;&lt;endl &lt;&lt;count ;
  cout &lt;&lt;endl;
  //cout &lt;&lt; "rank " &lt;&lt; myid &lt;&lt; " gets the samples: " &lt;&lt; recv_vec.size()&lt;&lt;endl; 
  iterateForTest(myid, recv_vec);
  
  free(data_recv);
  }
  
  MPI_Finalize();
  return 0;
  </code></pre>