title: Super-resolution of license plate in video sequence
body: <p>This is a follow up question to my question here: <a href="https://stackoverflow.com/questions/21715908/local-enhancing-of-license-plate-in-video-sequence">Local enhancing of license plate in video sequence</a></p>
  
  <p>I implemented a basic super-resolution technique suggested by the answer, but I am having trouble obtaining better resolution results. I trimmed the video sequence to 40 frames as seen below:<img src="https://i.stack.imgur.com/vS8Sf.gif" alt="enter image description here"></p>
  
  <p>and using the code below, which takes the first 5 frames and performs Super-resolution, then the super-resolved frame is repeatedly updated with the processing of the rest of the frames in the sequence:</p>
  
  <pre><code>public void Process(Image&lt;Gray, byte&gt; frame)
      {
          SetRegionOfInterest(frame);
  
          var resizedFrame = ResizeFrame(frame);
  
          InputFrames.Add(resizedFrame);
          if(InputFrames.Count &gt; 5)
          {
              InputFrames.RemoveAt(0);
              PerformSuperResolution();
          }
      }
  
      public void PerformSuperResolution()
      {
          // WindowSize = 5
          var referenceFrame = InputFrames[WindowSize-1].Convert&lt;Gray, byte&gt;();
          var featuresToTrack = referenceFrame.GoodFeaturesToTrack(100, 0.1, 5, 10);
  
          referenceFrame.FindCornerSubPix(featuresToTrack, new Size(WindowSize,WindowSize), new Size(-1,-1), new MCvTermCriteria(20, 0.03d));
  
          var resultFrame = InputFrames[WindowSize-1].Convert&lt;Gray, double&gt;();
          for(var frameCounter = 0; frameCounter &lt; WindowSize-1; frameCounter++)
          {
              // Get shift between frames
              var shiftResult = GetShiftResult(InputFrames[frameCounter],referenceFrame, featuresToTrack);
  
              // Warp to correct shift
              var warpMatrix = new Matrix&lt;double&gt;(new[,] { {1, 0, -shiftResult.ShiftX}, {0, 1, -shiftResult.ShiftY}, {0, 0, 1}});
  
              var warpedFrame = InputFrames[frameCounter].WarpPerspective(warpMatrix, 
                  INTER.CV_INTER_NN, 
                  WARP.CV_WARP_DEFAULT, 
                  new Gray(0)); 
  
              resultFrame.RunningAvg(warpedFrame.Convert&lt;Gray,double&gt;(), 1, resultFrame.Convert&lt;Gray, byte&gt;());
          }
          SuperResolutionFrame = resultFrame.Convert&lt;Gray, byte&gt;();
      }
  
      public ShiftResult GetShiftResult(Image&lt;Gray, byte&gt; inputFrame, Image&lt;Gray, byte&gt; referenceFrame, PointF[][] ActualFeature)
      {
          var result = new ShiftResult();
  
          PointF[] NextFeature;
          Byte[] Status;
          float[] TrackError;
  
          // optical flow
          OpticalFlow.PyrLK(referenceFrame, inputFrame, ActualFeature[0], 
              new Size(WindowSize, WindowSize), 5, new MCvTermCriteria(20, 0.1d), 
              out NextFeature, out Status, out TrackError);
  
          //get displacements
          float[] XdisplacementVectors = new float[NextFeature.Length];
          float[] YdisplacementVectors = new float[NextFeature.Length];
          for(int i = 0; i &lt; NextFeature.Length; i++)
          {
              XdisplacementVectors[i] = NextFeature[i].X - ActualFeature[0][i].X;
              YdisplacementVectors[i] = NextFeature[i].Y - ActualFeature[0][i].Y;
          }
  
          // gets average of displacements (disregards outliers)
          result.ShiftX = getAVG(XdisplacementVectors);
          result.ShiftY = getAVG(YdisplacementVectors);
  
          return result;
      }
  </code></pre>
  
  <p>where the program calls <code>Process(frame)</code> on an array of input video frames (totaling 40) as such:</p>
  
  <pre><code>for(int i  = 0; i &lt; image_array.Count; i++)
  {
      Res.Process(image_array[i]);
  }
  </code></pre>
  
  <p>and I obtained this resulting image:
  <img src="https://i.stack.imgur.com/ipeVj.png" alt="enter image description here"></p>
  
  <p>As you can see there are only small improvements in the overall resolution, and not much improvement to the readability of the license plate. I've tried experimenting with the various parameters, but the parameters used above seem to be the best I can do. </p>
  
  <p>I am open to any suggestions for improvement (need not be limited to the use of C#/Emgu CV as above), and even different approaches in implementation, if possible within the realm of mobile applications.</p>