title: 3d reconstruction from 2 images without info about the camera
body: <p>I'm new in this field and I'm trying to model a simple scene in 3d out of 2d images and I dont have any info about cameras. I know that there are <a href="https://stackoverflow.com/questions/6744094/cvreprojectimageto3d-3d-modelling-from-2d-images-issue/9013104#9013104">3 options</a>:</p>
  
  <ul>
  <li><p>I have two images and I know the model of my camera (intrisics) that I loaded from a XML for instance <code>loadXMLFromFile()</code> => <code>stereoRectify()</code> => <code>reprojectImageTo3D()</code></p></li>
  <li><p>I don't have them but I can calibrate my camera => <code>stereoCalibrate()</code> => <code>stereoRectify()</code> => <code>reprojectImageTo3D()</code></p></li>
  <li><p>I can't calibrate the camera (it is my case, because I don't have the camera that has taken the 2 images, then I need to find pair keypoints on both images with SURF, SIFT for instance (I can use any blob detector actually), then compute descriptors of these keypoints, then match keypoints from image right and image left according to their descriptors, and then find the fundamental matrix from them. The processing is much harder and would be like this:</p>
  
  <ol>
  <li>detect keypoints (SURF, SIFT) => </li>
  <li>extract descriptors (SURF,SIFT) => </li>
  <li>compare and match descriptors (BruteForce, Flann based approaches) => </li>
  <li>find fundamental mat (<code>findFundamentalMat()</code>) from these pairs => </li>
  <li><code>stereoRectifyUncalibrated()</code> => </li>
  <li><code>reprojectImageTo3D()</code></li>
  </ol></li>
  </ul>
  
  <p>I'm using the last approach and my questions are:</p>
  
  <p>1) Is it right?</p>
  
  <p>2) if it's ok, I have a doubt about the last step <code>stereoRectifyUncalibrated()</code> => <code>reprojectImageTo3D()</code>. The signature of <code>reprojectImageTo3D()</code> function is:</p>
  
  <pre><code>void reprojectImageTo3D(InputArray disparity, OutputArray _3dImage, InputArray Q, bool handleMissingValues=false, int depth=-1 )
  
  cv::reprojectImageTo3D(imgDisparity8U, xyz, Q, true) (in my code)
  </code></pre>
  
  <p>Parameters: </p>
  
  <ul>
  <li><code>disparity</code> – Input single-channel 8-bit unsigned, 16-bit signed, 32-bit signed or 32-bit floating-point disparity image.</li>
  <li><code>_3dImage</code> – Output 3-channel floating-point image of the same size as <code>disparity</code>. Each element of <code>_3dImage(x,y)</code> contains 3D coordinates of the point <code>(x,y)</code> computed from the disparity map.</li>
  <li><code>Q</code> – 4x4 perspective transformation matrix that can be obtained with <code>stereoRectify()</code>.</li>
  <li><code>handleMissingValues</code> – Indicates, whether the function should handle missing values (i.e. points where the disparity was not computed). If <code>handleMissingValues=true</code>, then pixels with the minimal disparity that corresponds to the outliers (see <code>StereoBM::operator()</code>) are transformed to 3D points with a very large Z value (currently set to 10000).</li>
  <li><code>ddepth</code> – The optional output array depth. If it is -1, the output image will have <code>CV_32F</code> depth. <code>ddepth</code> can also be set to <code>CV_16S</code>, <code>CV_32S</code> or `CV_32F'.</li>
  </ul>
  
  <p>How can I get the <code>Q</code> matrix? Is possible to obtain the <code>Q</code> matrix with <code>F</code>, <code>H1</code> and <code>H2</code> or in another way?</p>
  
  <p>3) Is there another way for obtain the xyz coordinates without calibrating the cameras?</p>
  
  <p>My code is:</p>
  
  <pre><code>#include &lt;opencv2/core/core.hpp&gt;
  #include &lt;opencv2/calib3d/calib3d.hpp&gt;
  #include &lt;opencv2/imgproc/imgproc.hpp&gt;
  #include &lt;opencv2/highgui/highgui.hpp&gt;
  #include &lt;opencv2/contrib/contrib.hpp&gt;
  #include &lt;opencv2/features2d/features2d.hpp&gt;
  #include &lt;stdio.h&gt;
  #include &lt;iostream&gt;
  #include &lt;vector&gt;
  #include &lt;conio.h&gt;
  #include &lt;opencv/cv.h&gt;
  #include &lt;opencv/cxcore.h&gt;
  #include &lt;opencv/cvaux.h&gt;
  
  
  using namespace cv;
  using namespace std;
  
  int main(int argc, char *argv[]){
  
      // Read the images
      Mat imgLeft = imread( argv[1], CV_LOAD_IMAGE_GRAYSCALE );
      Mat imgRight = imread( argv[2], CV_LOAD_IMAGE_GRAYSCALE );
  
      // check
      if (!imgLeft.data || !imgRight.data)
              return 0;
  
      // 1] find pair keypoints on both images (SURF, SIFT):::::::::::::::::::::::::::::
  
      // vector of keypoints
      std::vector&lt;cv::KeyPoint&gt; keypointsLeft;
      std::vector&lt;cv::KeyPoint&gt; keypointsRight;
  
      // Construct the SURF feature detector object
      cv::SiftFeatureDetector sift(
              0.01, // feature threshold
              10); // threshold to reduce
                  // sensitivity to lines
                  // Detect the SURF features
  
      // Detection of the SIFT features
      sift.detect(imgLeft,keypointsLeft);
      sift.detect(imgRight,keypointsRight);
  
      std::cout &lt;&lt; "Number of SURF points (1): " &lt;&lt; keypointsLeft.size() &lt;&lt; std::endl;
      std::cout &lt;&lt; "Number of SURF points (2): " &lt;&lt; keypointsRight.size() &lt;&lt; std::endl;
  
      // 2] compute descriptors of these keypoints (SURF,SIFT) ::::::::::::::::::::::::::
  
      // Construction of the SURF descriptor extractor
      cv::SurfDescriptorExtractor surfDesc;
  
      // Extraction of the SURF descriptors
      cv::Mat descriptorsLeft, descriptorsRight;
      surfDesc.compute(imgLeft,keypointsLeft,descriptorsLeft);
      surfDesc.compute(imgRight,keypointsRight,descriptorsRight);
  
      std::cout &lt;&lt; "descriptor matrix size: " &lt;&lt; descriptorsLeft.rows &lt;&lt; " by " &lt;&lt; descriptorsLeft.cols &lt;&lt; std::endl;
  
      // 3] matching keypoints from image right and image left according to their descriptors (BruteForce, Flann based approaches)
  
      // Construction of the matcher
      cv::BruteForceMatcher&lt;cv::L2&lt;float&gt; &gt; matcher;
  
      // Match the two image descriptors
      std::vector&lt;cv::DMatch&gt; matches;
      matcher.match(descriptorsLeft,descriptorsRight, matches);
  
      std::cout &lt;&lt; "Number of matched points: " &lt;&lt; matches.size() &lt;&lt; std::endl;
  
  
      // 4] find the fundamental mat ::::::::::::::::::::::::::::::::::::::::::::::::::::
  
      // Convert 1 vector of keypoints into
      // 2 vectors of Point2f for compute F matrix
      // with cv::findFundamentalMat() function
      std::vector&lt;int&gt; pointIndexesLeft;
      std::vector&lt;int&gt; pointIndexesRight;
      for (std::vector&lt;cv::DMatch&gt;::const_iterator it= matches.begin(); it!= matches.end(); ++it) {
  
           // Get the indexes of the selected matched keypoints
           pointIndexesLeft.push_back(it-&gt;queryIdx);
           pointIndexesRight.push_back(it-&gt;trainIdx);
      }
  
      // Convert keypoints into Point2f
      std::vector&lt;cv::Point2f&gt; selPointsLeft, selPointsRight;
      cv::KeyPoint::convert(keypointsLeft,selPointsLeft,pointIndexesLeft);
      cv::KeyPoint::convert(keypointsRight,selPointsRight,pointIndexesRight);
  
      /* check by drawing the points
      std::vector&lt;cv::Point2f&gt;::const_iterator it= selPointsLeft.begin();
      while (it!=selPointsLeft.end()) {
  
              // draw a circle at each corner location
              cv::circle(imgLeft,*it,3,cv::Scalar(255,255,255),2);
              ++it;
      }
  
      it= selPointsRight.begin();
      while (it!=selPointsRight.end()) {
  
              // draw a circle at each corner location
              cv::circle(imgRight,*it,3,cv::Scalar(255,255,255),2);
              ++it;
      } */
  
      // Compute F matrix from n&gt;=8 matches
      cv::Mat fundemental= cv::findFundamentalMat(
              cv::Mat(selPointsLeft), // points in first image
              cv::Mat(selPointsRight), // points in second image
              CV_FM_RANSAC);       // 8-point method
  
      std::cout &lt;&lt; "F-Matrix size= " &lt;&lt; fundemental.rows &lt;&lt; "," &lt;&lt; fundemental.cols &lt;&lt; std::endl;
  
      /* draw the left points corresponding epipolar lines in right image
      std::vector&lt;cv::Vec3f&gt; linesLeft;
      cv::computeCorrespondEpilines(
              cv::Mat(selPointsLeft), // image points
              1,                      // in image 1 (can also be 2)
              fundemental,            // F matrix
              linesLeft);             // vector of epipolar lines
  
      // for all epipolar lines
      for (vector&lt;cv::Vec3f&gt;::const_iterator it= linesLeft.begin(); it!=linesLeft.end(); ++it) {
  
          // draw the epipolar line between first and last column
          cv::line(imgRight,cv::Point(0,-(*it)[2]/(*it)[1]),cv::Point(imgRight.cols,-((*it)[2]+(*it)[0]*imgRight.cols)/(*it)[1]),cv::Scalar(255,255,255));
      }
  
      // draw the left points corresponding epipolar lines in left image
      std::vector&lt;cv::Vec3f&gt; linesRight;
      cv::computeCorrespondEpilines(cv::Mat(selPointsRight),2,fundemental,linesRight);
      for (vector&lt;cv::Vec3f&gt;::const_iterator it= linesRight.begin(); it!=linesRight.end(); ++it) {
  
          // draw the epipolar line between first and last column
          cv::line(imgLeft,cv::Point(0,-(*it)[2]/(*it)[1]), cv::Point(imgLeft.cols,-((*it)[2]+(*it)[0]*imgLeft.cols)/(*it)[1]), cv::Scalar(255,255,255));
      }
  
      // Display the images with points and epipolar lines
      cv::namedWindow("Right Image Epilines");
      cv::imshow("Right Image Epilines",imgRight);
      cv::namedWindow("Left Image Epilines");
      cv::imshow("Left Image Epilines",imgLeft);
      */
  
      // 5] stereoRectifyUncalibrated()::::::::::::::::::::::::::::::::::::::::::::::::::
  
      //H1, H2 – The output rectification homography matrices for the first and for the second images.
      cv::Mat H1(4,4, imgRight.type());
      cv::Mat H2(4,4, imgRight.type());
      cv::stereoRectifyUncalibrated(selPointsRight, selPointsLeft, fundemental, imgRight.size(), H1, H2);
  
  
      // create the image in which we will save our disparities
      Mat imgDisparity16S = Mat( imgLeft.rows, imgLeft.cols, CV_16S );
      Mat imgDisparity8U = Mat( imgLeft.rows, imgLeft.cols, CV_8UC1 );
  
      // Call the constructor for StereoBM
      int ndisparities = 16*5;      // &lt; Range of disparity &gt;
      int SADWindowSize = 5;        // &lt; Size of the block window &gt; Must be odd. Is the 
                                    // size of averaging window used to match pixel  
                                    // blocks(larger values mean better robustness to
                                    // noise, but yield blurry disparity maps)
  
      StereoBM sbm( StereoBM::BASIC_PRESET,
          ndisparities,
          SADWindowSize );
  
      // Calculate the disparity image
      sbm( imgLeft, imgRight, imgDisparity16S, CV_16S );
  
      // Check its extreme values
      double minVal; double maxVal;
  
      minMaxLoc( imgDisparity16S, &amp;minVal, &amp;maxVal );
  
      printf("Min disp: %f Max value: %f \n", minVal, maxVal);
  
      // Display it as a CV_8UC1 image
      imgDisparity16S.convertTo( imgDisparity8U, CV_8UC1, 255/(maxVal - minVal));
  
      namedWindow( "windowDisparity", CV_WINDOW_NORMAL );
      imshow( "windowDisparity", imgDisparity8U );
  
  
      // 6] reprojectImageTo3D() :::::::::::::::::::::::::::::::::::::::::::::::::::::
  
      //Mat xyz;
      //cv::reprojectImageTo3D(imgDisparity8U, xyz, Q, true);
  
      //How can I get the Q matrix? Is possibile to obtain the Q matrix with 
      //F, H1 and H2 or in another way?
      //Is there another way for obtain the xyz coordinates?
  
      cv::waitKey();
      return 0;
  }
  </code></pre>