title: Adding timestamps to HEVC with ffmpeg library, mismatch in timestamps when run with Gstreamer pipeline
body: <p>We are encoding sample frames using NvEncoder with HEVC. Since HEVC frames do not have any timestamps, in order to perform seek operation on the video, we wrote a remuxer in C++ that creates timestamps for the frames in an orderly fashion and writes the encoded frames in a video container (mp4,mov). The output video in mp4 container looks fine when played with ffplay, and timestamps seems correct when checked with ffprobe. However, when we try to play the video in Gstreamer pipeline, 2nd and 3rd frames seem to have the exactly same timestamp. So when the video is played, 3rd frame is skipped and 2nd frame is shown twice. We cannot tolerate any frame loss, so we need to solve this problem, which we think is due to an incompatibility between ffmpeg and gstreamer regarding frame timestamps. I can also provide the source-code of our remuxer and example outputs if that would help.</p>
  <p>I used the following Gstreamer pipeline to play the mp4:</p>
  <pre><code>gst-launch-1.0 filesrc location=5_fps.mp4 ! qtdemux name=demux demux.video_0 ! queue! decodebin ! videoconvert ! videoscale ! autovideosink 
  </code></pre>
  <p>Following  command  also gives the same mismatching frame timestamps:</p>
  <pre><code>ffmpeg -i 5_fps.bin -vcodec copy -acodec copy 5_fps.mp4
  </code></pre>
  <p>Many thanks!</p>
  <p>Edit: I am adding the part of remuxer where each frame from the input stream is read and timestamps are added.</p>
  <pre><code>int frame_no=-1; //starting with -1 gives the same ffprobe results as command line ffmpeg container conversion, starting with 0 again causes the same timestamp problem    
  while (1) {
  AVStream *in_stream, *out_stream;
  
  _status = av_read_frame(_ifmt_ctx, &amp;_pkt);
  if (_status &lt; 0) break;
  
  in_stream  = _ifmt_ctx-&gt;streams[_pkt.stream_index];
  if (_pkt.stream_index &gt;= _stream_mapping_size ||
      _stream_mapping[_pkt.stream_index] &lt; 0) {
    av_packet_unref(&amp;_pkt);
    continue;
  }
  double inputFPS=av_q2d(in_stream-&gt;r_frame_rate);
  
  double outputFPS=av_q2d(in_stream-&gt;r_frame_rate);
  
  _pkt.stream_index = _stream_mapping[_pkt.stream_index];
  out_stream = _ofmt_ctx-&gt;streams[_pkt.stream_index];
  _pkt.pts=frame_no*in_stream-&gt;time_base.den/inputFPS;
  _pkt.dts=_pkt.pts;
  _pkt.duration = in_stream-&gt;time_base.den/inputFPS; 
  _pkt.pos = -1;
  std::cout&lt;&lt;&quot;rescaled pts: &quot;&lt;&lt;_pkt.pts&lt;&lt;&quot; dts: &quot;&lt;&lt;_pkt.dts&lt;&lt;&quot; frame no: &quot;&lt;&lt; frame_no&lt;&lt;std::endl;
  
  std::cout&lt;&lt;&quot;input time_base den: &quot;&lt;&lt;in_stream-&gt;time_base.den&lt;&lt;&quot; output time_base den: &quot;&lt;&lt;out_stream-&gt;time_base.den&lt;&lt;std::endl;
  
  frame_no++;
  _status = av_interleaved_write_frame(_ofmt_ctx, &amp;_pkt);
  if (_status &lt; 0) {
    cout&lt;&lt;&quot;Error muxing packet\n&quot;;
    break;
  }
  av_packet_unref(&amp;_pkt);}
  </code></pre>
  <p>I first tried this method where each frame timestamp (pts and dts) is incremented by packet duration. At first I thought this method would not work since B-frames are decoded in different order, so I first tried videos with no b-frames. However, when I tried with videos with b-frames, it again worked. I thought decoded frames would be in a different order, however that was not the case. The only issue is that only the second and third frames appear to have the same timestamps in Gstream (not in Ffmpeg), other than these two frames remaining video plays just fine. Overall, I am also confused that b-frames do not cause any frame order problem.</p>
  <p>Example <a href="https://drive.google.com/file/d/1RpgouYmin6IIcQf8WL1QdSnYv7LKRCYu/view?usp=sharing" rel="nofollow noreferrer">encoded input</a>, example <a href="https://drive.google.com/file/d/1irclGFjSnuFlKOhX6PSpdqY79-6rfBXR/view?usp=sharing" rel="nofollow noreferrer">output video</a> if you want to examine the frames. (I don't know if it's okay to share files over google drive, please correct me if there is a better way to share, or not.)</p>