title: Extract video frames on Android with ExtractMpegFramesTest and running a computationally intensive function instead of saving as images
body: <p>On Android, I want to extract all frames of a video and run an <a href="http://webcache.googleusercontent.com/search?q=cache:u15a1NaQIB8J:boofcv.org/javadoc/boofcv/factory/tracker/FactoryTrackerObjectQuad.html+&amp;cd=2&amp;hl=de&amp;ct=clnk&amp;gl=de" rel="nofollow noreferrer">object tracking function</a> from boofcv on each frame. Therefore, I am using the <a href="https://bigflake.com/mediacodec/#ExtractMpegFramesTest" rel="nofollow noreferrer">ExtractMpegFramesTest</a> example and slightly adjusted it to run the tracking on each frame instead of saving the frames as png. I.e. instead of calling outputSurface.saveFrame() I am calling outputSurface.processFrame(), which I implemented as follows:</p>
  <pre><code>public void processFrame() {
      mPixelBuf.rewind();
      GLES20.glReadPixels(0, 0, mWidth, mHeight, GLES20.GL_RGBA, GLES20.GL_UNSIGNED_BYTE,
                      mPixelBuf);
  
      Bitmap bmp = Bitmap.createBitmap(mWidth, mHeight, Bitmap.Config.ARGB_8888);
      mPixelBuf.rewind();
      bmp.copyPixelsFromBuffer(mPixelBuf);
      GrayU8 img = ConvertBitmap.bitmapToGray(bmp, (GrayU8)null, null);
      Log.d(TAG, &quot;Running tracker...&quot;);
      tracker.process(img, location);
      barPath.add(getCenterOf(location));
      Log.d(TAG, &quot;Finished running tracker.&quot;);
              bmp.recycle();
  }
  </code></pre>
  <p>As soon as I comment out the tracker.process(img, location) line, the code works flawlessly. As soon as I include the tracking, I receive no error meassage but also nothing happens. The app gets stuck at tracking the first image, the logcat output looks like this:</p>
  <pre><code>2021-10-19 13:01:05.778 32449-32449/org.boofcv.android D/ExtractMpegFrames: Initialized
  2021-10-19 13:01:05.780 32449-32449/org.boofcv.android I/Choreographer: Skipped 30 frames!  The application may be doing too much work on its main thread.
  2021-10-19 13:01:05.789 32449-32449/org.boofcv.android W/Looper: Slow Looper main: doFrame is 514ms late because of 3 msg, msg 2 took 510ms (seq=242 running=153ms runnable=2ms late=18ms h=android.view.ViewRootImpl$ViewRootHandler c=android.view.View$PerformClick)
  2021-10-19 13:01:05.793 32449-32726/org.boofcv.android D/ExtractMpegFrames: Extractor selected track 0 (video/avc): {track-id=1, file-format=video/mp4, level=8192, mime=video/avc, frame-count=624, profile=8, language=, color-standard=1, display-width=1920, csd-1=java.nio.HeapByteBuffer[pos=0 lim=9 cap=9], color-transfer=3, durationUs=10395844, display-height=1080, width=1920, color-range=2, rotation-degrees=90, max-input-size=1555201, frame-rate=60, height=1080, csd-0=java.nio.HeapByteBuffer[pos=0 lim=23 cap=23]}
  2021-10-19 13:01:05.794 32449-32726/org.boofcv.android D/ExtractMpegFrames: Video size is 1920x1080
  2021-10-19 13:01:05.824 32449-32726/org.boofcv.android D/ExtractMpegFrames: textureID=1
  2021-10-19 13:01:05.830 32449-32728/org.boofcv.android I/OMXClient: IOmx service obtained
  2021-10-19 13:01:05.888 32449-32727/org.boofcv.android D/SurfaceUtils: connecting to surface 0x7864f16010, reason connectToSurface
  2021-10-19 13:01:05.888 32449-32727/org.boofcv.android I/MediaCodec: [OMX.qcom.video.decoder.avc] setting surface generation to 33227777
  2021-10-19 13:01:05.888 32449-32727/org.boofcv.android D/SurfaceUtils: disconnecting from surface 0x7864f16010, reason connectToSurface(reconnect)
  2021-10-19 13:01:05.888 32449-32727/org.boofcv.android D/SurfaceUtils: connecting to surface 0x7864f16010, reason connectToSurface(reconnect)
  2021-10-19 13:01:05.890 32449-32728/org.boofcv.android I/ExtendedACodec: setupVideoDecoder()
  2021-10-19 13:01:05.892 32449-32728/org.boofcv.android I/ExtendedACodec: Decoder will be in frame by frame mode
  2021-10-19 13:01:05.929 32449-32728/org.boofcv.android D/SurfaceUtils: set up nativeWindow 0x7864f16010 for 1920x1080, color 0x7fa30c06, rotation 90, usage 0x20002900
  2021-10-19 13:01:05.939 32449-32728/org.boofcv.android W/Gralloc3: allocator 3.x is not supported
  2021-10-19 13:01:05.939 32449-32726/org.boofcv.android D/ExtractMpegFrames: loop
  2021-10-19 13:01:05.946 32449-32726/org.boofcv.android D/ExtractMpegFrames: submitted frame 0 to dec, size=148096
  2021-10-19 13:01:05.958 32449-32726/org.boofcv.android D/ExtractMpegFrames: no output from decoder available
  2021-10-19 13:01:05.958 32449-32726/org.boofcv.android D/ExtractMpegFrames: loop
  2021-10-19 13:01:05.959 32449-32726/org.boofcv.android D/ExtractMpegFrames: submitted frame 1 to dec, size=14112
  2021-10-19 13:01:05.970 32449-32726/org.boofcv.android D/ExtractMpegFrames: no output from decoder available
  2021-10-19 13:01:05.970 32449-32726/org.boofcv.android D/ExtractMpegFrames: loop
  2021-10-19 13:01:05.972 32449-32726/org.boofcv.android D/ExtractMpegFrames: submitted frame 2 to dec, size=17632
  2021-10-19 13:01:05.974 32449-32726/org.boofcv.android D/ExtractMpegFrames: decoder output format changed: {crop-right=1919, color-format=2141391878, slice-height=1088, mime=video/raw, hdr-static-info=java.nio.HeapByteBuffer[pos=0 lim=25 cap=25], stride=1920, color-standard=2, color-transfer=3, crop-bottom=1079, crop-left=0, width=1920, color-range=1, crop-top=0, height=1088}
  2021-10-19 13:01:05.974 32449-32726/org.boofcv.android D/ExtractMpegFrames: loop
  2021-10-19 13:01:05.975 32449-32726/org.boofcv.android D/ExtractMpegFrames: submitted frame 3 to dec, size=30208
  2021-10-19 13:01:05.976 32449-32726/org.boofcv.android D/ExtractMpegFrames: surface decoder given buffer 16 (size=8)
  2021-10-19 13:01:05.976 32449-32726/org.boofcv.android D/ExtractMpegFrames: awaiting decode of frame 0
  2021-10-19 13:01:05.978 32449-32449/org.boofcv.android D/ExtractMpegFrames: new frame available
  2021-10-19 13:01:05.992 32449-32726/org.boofcv.android D/ExtractMpegFrames: Running tracker...
  2021-10-19 13:01:06.003 32449-32727/org.boofcv.android D/SurfaceUtils: disconnecting from surface 0x7864f16010, reason disconnectFromSurface
  </code></pre>
  <p>The tracker.process() functions itself works flawlessly, e.g. when using MediaMetadataRetriever to extract the frames (however MediaMetadataRetriever is too slow, that's why I am using ExtractMpegFrames). On my device the tracker.process() function itself usually takes about 30-40 ms per frame. Just extracting the frames with ExtractMpegFrames as bitmap without further processing takes about 3-4 ms per frame.</p>
  <p>So I guess the problem might have to do with threading? I would be thankful for every help.</p>
  <h3>Edit</h3>
  <p>On my UI Thread I am calling <code>new ExtractMpegFrames().run()</code> while the ExtractMpegFrames class looks like this:</p>
  <pre><code>public class ExtractMpegFrames {
  
      public void run() throws Throwable {
          this.initialize();
          ExtractMpegFramesWrapper.execute(this);
      }
      private static class ExtractMpegFramesWrapper implements Runnable {
          private Throwable mThrowable;
          private ExtractMpegFrames mExtractor;
  
          private ExtractMpegFramesWrapper(ExtractMpegFrames test) {
              mExtractor = test;
          }
  
          @Override
          public void run() {
              try {
                  mExtractor.extractMpegFrames();
              } catch (Throwable th) {
                  mThrowable = th;
              }
          }
  
          public static void execute(ExtractMpegFrames obj) throws Throwable {
              ExtractMpegFramesWrapper wrapper = new ExtractMpegFramesWrapper(obj);
              Thread th = new Thread(wrapper, &quot;main&quot;);
              th.start();
              //th.join();
              if (wrapper.mThrowable != null) {
                  throw wrapper.mThrowable;
              }
          }
      }
      ... 
      // The rest (except for the processFrame() function above) 
      // looks the same as in ExtractMpegFramesTest 
      // See here: 
      // https://bigflake.com/mediacodec/#ExtractMpegFramesTest
  }
  </code></pre>
  <h3>Edit 2</h3>
  <p>I still did not resolve this issue. I tried to replace the boofcv tracker with an opencv one (<a href="https://javadoc.io/static/org.bytedeco/opencv/4.0.1-1.5/org/opencv/tracking/TrackerMOSSE.html" rel="nofollow noreferrer">org.opencv.tracking.TrackerMOSSE</a>). However, this leads to a deadlock as well (same logcat output as above). Lastly, I tried to replace the tracking function by <a href="https://docs.opencv.org/3.4/de/da9/tutorial_template_matching.html" rel="nofollow noreferrer">this opencv template matching algorithm</a>. This works and does not lead to a deadlock, however it is way too slow (takes about one second per frame on my device).</p>