title: ffmpeg video to opengl texture
body: <p>I'm trying to render frames grabbed and converted from a video using ffmpeg to an OpenGL texture to be put on a quad. I've pretty much exhausted google and not found an answer, well I've found answers but none of them seem to have worked.</p>
  
  <p>Basically, I am using <code>avcodec_decode_video2()</code> to decode the frame and then <code>sws_scale()</code> to convert the frame to RGB and then <code>glTexSubImage2D()</code> to create an openGL texture from it but can't seem to get anything to work.</p>
  
  <p>I've made sure the "destination" AVFrame has power of 2 dimensions in the SWS Context setup. Here is my code:</p>
  
  <pre><code>SwsContext *img_convert_ctx = sws_getContext(pCodecCtx-&gt;width,
                  pCodecCtx-&gt;height, pCodecCtx-&gt;pix_fmt, 512,
                  256, PIX_FMT_RGB24, SWS_BICUBIC, NULL,
                  NULL, NULL);
  
  //While still frames to read
  while(av_read_frame(pFormatCtx, &amp;packet)&gt;=0) {
      glClear(GL_COLOR_BUFFER_BIT);
  
      //If the packet is from the video stream
      if(packet.stream_index == videoStream) {
          //Decode the video
          avcodec_decode_video2(pCodecCtx, pFrame, &amp;frameFinished, &amp;packet);
  
          //If we got a frame then convert it and put it into RGB buffer
          if(frameFinished) {
              printf("frame finished: %i\n", number);
              sws_scale(img_convert_ctx, pFrame-&gt;data, pFrame-&gt;linesize, 0, pCodecCtx-&gt;height, pFrameRGB-&gt;data, pFrameRGB-&gt;linesize);
  
              glBindTexture(GL_TEXTURE_2D, texture);
              //gluBuild2DMipmaps(GL_TEXTURE_2D, 3, pCodecCtx-&gt;width, pCodecCtx-&gt;height, GL_RGB, GL_UNSIGNED_INT, pFrameRGB-&gt;data);
              glTexSubImage2D(GL_TEXTURE_2D, 0, 0,0, 512, 256, GL_RGB, GL_UNSIGNED_BYTE, pFrameRGB-&gt;data[0]);
              SaveFrame(pFrameRGB, pCodecCtx-&gt;width, pCodecCtx-&gt;height, number);
              number++;
          }
      }
  
      glColor3f(1,1,1);
      glBindTexture(GL_TEXTURE_2D, texture);
      glBegin(GL_QUADS);
          glTexCoord2f(0,1);
          glVertex3f(0,0,0);
  
          glTexCoord2f(1,1);
          glVertex3f(pCodecCtx-&gt;width,0,0);
  
          glTexCoord2f(1,0);
          glVertex3f(pCodecCtx-&gt;width, pCodecCtx-&gt;height,0);
  
          glTexCoord2f(0,0);
          glVertex3f(0,pCodecCtx-&gt;height,0);
  
      glEnd();
  </code></pre>
  
  <p>As you can see in that code, I am also saving the frames to .ppm files just to make sure they are actually rendering, which they are.</p>
  
  <p>The file being used is a .wmv at 854x480, could this be the problem? The fact I'm just telling it to go 512x256?</p>
  
  <p>P.S. I've looked at this <a href="https://stackoverflow.com/questions/5548674/how-to-convert-avframe-to-texture-used-by-glteximage2d">Stack Overflow question</a> but it didn't help.</p>
  
  <p>Also, I have <code>glEnable(GL_TEXTURE_2D)</code> as well and have tested it by just loading in a normal bmp.</p>
  
  <p><strong>EDIT</strong></p>
  
  <p>I'm getting an image on the screen now but it is a garbled mess, I'm guessing something to do with changing things to a power of 2 (in the decode, <code>swscontext</code> and <code>gluBuild2DMipmaps</code> as shown in my code). I'm usually nearly exactly the same code as shown above, only I've changed <code>glTexSubImage2D</code> to <code>gluBuild2DMipmaps</code> and changed the types to <code>GL_RGBA</code>.</p>
  
  <p>Here is what the frame looks like:</p>
  
  <p><img src="https://i.stack.imgur.com/tOg7V.png" alt="Ffmpeg as OpenGL Texture garbled"></p>
  
  <p><strong>EDIT AGAIN</strong></p>
  
  <p>Just realised I haven't showed the code for how pFrameRGB is set up:</p>
  
  <pre><code>//Allocate video frame for 24bit RGB that we convert to.
  AVFrame *pFrameRGB;
  pFrameRGB = avcodec_alloc_frame();
  
  if(pFrameRGB == NULL) {
      return -1;
  }
  
  //Allocate memory for the raw data we get when converting.
  uint8_t *buffer;
  int numBytes;
  numBytes = avpicture_get_size(PIX_FMT_RGB24, pCodecCtx-&gt;width, pCodecCtx-&gt;height);
  buffer = (uint8_t *) av_malloc(numBytes*sizeof(uint8_t));
  
  //Associate frame with our buffer
  avpicture_fill((AVPicture *) pFrameRGB, buffer, PIX_FMT_RGB24,
      pCodecCtx-&gt;width, pCodecCtx-&gt;height);
  </code></pre>
  
  <p>Now that I ahve changed the <code>PixelFormat</code> in <code>avgpicture_get_size</code> to <code>PIX_FMT_RGB24</code>, I've done that in <code>SwsContext</code> as well and changed <code>GluBuild2DMipmaps</code> to <code>GL_RGB</code> and I get a slightly better image but it looks like I'm still missing lines and it's still a bit stretched:</p>
  
  <p><img src="https://i.stack.imgur.com/nS6Co.png" alt="Ffmpeg Garbled OpenGL Texture 2"></p>
  
  <p><strong>Another Edit</strong></p>
  
  <p>After following Macke's advice and passing the actual resolution to OpenGL I get the frames nearly proper but still a bit skewed and in black and white, also it's only getting 6fps now rather than 110fps:</p>
  
  <p><img src="https://i.stack.imgur.com/SJeGO.png" alt="enter image description here"></p>
  
  <p>P.S.</p>
  
  <p>I've got a function to save the frames to image after <code>sws_scale()</code> and they are coming out fine as colour and everything so something in OGL is making it B&amp;W.</p>
  
  <p><strong>LAST EDIT</strong></p>
  
  <p>Working! Okay I have it working now, basically I am not padding out the texture to a power of 2 and just using the resolution the video is.</p>
  
  <p>I got the texture showing up properly with a lucky guess at the correct glPixelStorei() </p>
  
  <pre><code>glPixelStorei(GL_UNPACK_ALIGNMENT, 2);
  </code></pre>
  
  <p>Also, if anyone else has the <code>subimage()</code> showing blank problem like me, you have to fill the texture at least once with <code>glTexImage2D()</code> and so I use it once in the loop and then use <code>glTexSubImage2D()</code> after that.</p>
  
  <p>Thanks Macke and datenwolf for all your help.</p>