title: How to optimize frame grabbing from video stream in OpenCV?
body: <p>I ran into a problem problem of low frame capture efficiency in OpenCV.</p>
  
  <ol>
  <li><p>Hardware &amp; Software.</p>
  
  <ul>
  <li>Raspberry Pi 3 (1,2  GHz quad-core ARM) with HDMI Display</li>
  <li>IP camera: LAN connected,  RTSP, H264 codec, 1280x720 resolution, 20 fps, 1 GOP, 2500 kB/s VBR bitrate (parameters can be changed).</li>
  <li>OS Raspbian Stretch</li>
  <li>Python 3.5</li>
  <li>OpenCV 4.1</li>
  <li>Gstreamer 1.0</li>
  </ul></li>
  <li><p>Task. </p></li>
  </ol>
  
  <p><strong>Get videostream from IP camera, recognize images and display resulting video (with marks and messages).</strong></p>
  
  <p>Important features: real-time processing, HD resolution (1280x720), high frame rate (>20 fps), continuous operation for several hours.</p>
  
  <ol start="3">
  <li>My solution.</li>
  </ol>
  
  <p>General algorithm: source video stream -> decoding and frame grabbing -> work with frames in OpenCV -> assembling the processed frames into a video stream -> display video using a Raspberry Pi GPU</p>
  
  <p>OpenCV output/display method - <strong>imshow</strong> - does not work well even at low-resolution video. The only library that allows to use a Raspberry Pi GPU to decode and display video is a Gstreamer.</p>
  
  <p>I compiled Gstreamer modules (gstreamer1.0-plugins-bad, gstreamer1.0-omx) with OMX support and tested it:</p>
  
  <pre><code>gst-launch-1.0 rtspsrc location='rtsp://web_camera_ip' latency=400 ! queue ! rtph264depay ! h264parse ! omxh264dec ! glimagesink
  </code></pre>
  
  <p>It works great, <strong>CPU usage is about 9%</strong>.</p>
  
  <p>Next I compiled OpenCV with Gstreamer, NEON, VFPV3 support.</p>
  
  <p>I use the following code for testing:</p>
  
  <pre><code>import cv2
  import numpy as np
  
  src='rtsp://web_camera_ip'
  stream_in = cv2.VideoCapture(src)
  
  pipeline_out = "appsrc ! videoconvert ! video/x-raw, framerate=20/1, format=RGBA ! glimagesink sync=false"
  fourcc = cv2.VideoWriter_fourcc(*'H264')
  
  stream_out = cv2.VideoWriter(pipeline_out, cv2.CAP_GSTREAMER, fourcc, 20.0, (1280,720))
  while True:
      ret, frame = stream_out.read()
      if ret:
        stream_out.write(frame)
        cv2.waitKey(1)
  </code></pre>
  
  <p>It also worked, but not so well as Gstreamer itself. <strong>CPU usage is about 50%</strong>, <strong>without stream_out.write(frame) - 35%</strong>. At <strong>frame rate above 15</strong>, there are lags and delays.</p>
  
  <ol start="4">
  <li>How I tried to solve the problem.</li>
  </ol>
  
  <p>4.1. Use Gstreamer to decode video stream:</p>
  
  <pre><code>pipline_in='rtspsrc location=rtsp://web_camera_ip latency=400 ! queue ! rtph264depay ! h264parse ! omxh264dec ! videoconvert ! appsink'
  stream_in = cv2.VideoCapture(pipline_in)
  </code></pre>
  
  <p>It even worsened the situation - the <strong>CPU load increased by several percent</strong>, the delay has become more.</p>
  
  <p>4.2. I also tried to optimize the library using <a href="https://www.pyimagesearch.com/2015/12/21/increasing-webcam-fps-with-python-and-opencv/" rel="nofollow noreferrer">method</a> from PyImageSearch.com - threading using WebcamVideoStream from imutils library.</p>
  
  <pre><code>from threading import Thread
  import cv2
  import numpy as np
  import imutils
  
  src='rtsp://web_camera_ip'
  stream_in = WebcamVideoStream(src).start()
  pipeline_out = "appsrc ! videoconvert ! video/x-raw, framerate=20/1, format=RGBA ! glimagesink sync=false"
  fourcc = cv2.VideoWriter_fourcc(*'H264')
  
  stream_out = cv2.VideoWriter(pipeline_out, cv2.CAP_GSTREAMER, fourcc, 20.0, (1280,720))
  while True:
      frame = stream_in.read()
      out.write(frame)
      cv2.waitKey(1)
  </code></pre>
  
  <p><strong>CPU usage has increased to 70%</strong>, the quality of the output video stream has not changed.</p>
  
  <p>4.3 Сhanging the following parameters does not help: whaitKey(1-50),  videostream bitrate (1000-5000 kB/s), videostream GOP (1-20).</p>
  
  <ol start="5">
  <li>Questions.</li>
  </ol>
  
  <p>As I understand, VideoCaputre/Videowritter methods has a very low efficiency. Maybe it's not noticeable on PC, but it is critical for Raspberry Pi 3. </p>
  
  <ul>
  <li>Is it possible to increase the performance of the VideoCaputre
  (Videowritter)? </li>
  <li>Is there an alternative way to capture frames from
  video to OpenCV? </li>
  </ul>
  
  <p>Thanks in advance for answers!</p>
  
  <p><strong>UPDATE 1</strong></p>
  
  <p>I think I know what the problem is, but I don't know how to solve it.</p>
  
  <ol>
  <li>Refinement about CPU usage when working with VideoCapture and VideoCapture+Gstreamer. VideoCapture(src)+VideoWriter(gstreamer_piplene_out) - 50-60%, VideoCapture(gstreamer_pipline_in) +VideoWriter(gstreamer_piplene_out) - 40-50%.</li>
  <li><strong>Сolor formats</strong> that different parts of my program work with. H264 video stream - <strong>YUV</strong>, OpenCV - <strong>BGR</strong>, OMX layer output - <strong>RGBA</strong>. OpenCV can work only with frames in BGR color format. OMX layer output when trying to launch the collected video in a different color format, displays a black screen.</li>
  <li><strong>Сolor format conversion</strong> in the Gstremaer pipline is carried out using the <strong>videoconvert</strong>. In some cases the method can work automatically (without specifying parameters), it is also possible to specify the color format forcibly. And I do not know how it works in the "pure" VideoCapture(src).</li>
  </ol>
  
  <p>The main problem is that <strong><em>videoconvert</em> does not support GPU</strong> - the main CPU load is due to the color format conversion!</p>
  
  <p>I tested this assumption using the "pure" Gstreamer, adding the videoconvert:</p>
  
  <pre><code>gst-launch-1.0 rtspsrc location='web_camera_ip' latency=400 ! queue ! rtph264depay ! h264parse ! omxh264dec ! videoconvert ! video/x-raw, format=BGR ! glimagesink sync=false
  </code></pre>
  
  <p>Black display, <strong>CPU load is 25%</strong>.</p>
  
  <p>Check this pipline:</p>
  
  <pre><code>gst-launch-1.0 rtspsrc location='web_camera_ip' latency=400 ! queue ! rtph264depay ! h264parse ! omxh264dec ! videoconvert ! video/x-raw, format=RGBA ! glimagesink sync=false
  </code></pre>
  
  <p>Video is displayed, <strong>CPU load is 5%</strong>. I also assume that the omxh264dec converts the color format YUV to RGBA using GPU (after omxh264dec, videoconver does not load the CPU).</p>
  
  <ol start="4">
  <li>I don't know how to use GPU for color format conversion in VideoCapture/Gstreamer on Raspberry.</li>
  </ol>
  
  <p>In <a href="http://%20https://www.raspberrypi.org/forums/viewtopic.php?t=186123#p1175975" rel="nofollow noreferrer">this</a> thread 6by9, Rapberry engineer and graphics programming specialist, writes that "The IL video_encode component supports OMX_COLOR_Format24bitBGR888 which I seem to recall maps to OpenCV's RGB".</p>
  
  <p>Are there any ideas?</p>