Transcoding h264_rkmpp -> h264_rkmpp results in jerky motion (and errors about duplicate frames)

[oryjkov]

Transcoding an h264 video 50fps 1080p from a Canon camera on an rk3588 board.

The resulting videos has jerky motion, it feels like some frames are getting reordered, perhaps around 2 times per second. It is really nauseating to watch this.

The problem does not happen when using a sw decoder (-c:v h264).

I also tried h264 and h265 output, no difference.

Command used:

ffmpeg -hide_banner -y  -hwaccel rkmpp     -hwaccel_output_format drm_prime     -i in.mp4     -map_metadata 0     -c:a libopus -b:a 64k     -strict -2    -c:v hevc_rkmpp     -rc_mode 2     -qp_init 22     -tag:v hvc1 out.mp4

The final summary shows:

frame= 1074 fps= 77 q=-0.0 Lsize=   37349kB time=00:00:21.46 bitrate=14257.5kbits/s dup=90 drop=0 speed=1.54x

Note the dup=90. When using the software decoder, there is no dup.

With more verbose logging the following is printed periodically

[hevc_rkmpp @ 0xaaaad1b6d680] Received a packet
[hevc_rkmpp @ 0xaaaad1b6d680] Wrote 4177920 bytes to encoder
[hevc_rkmpp @ 0xaaaad1b6d680] Received a packet
[hevc_rkmpp @ 0xaaaad1b6d680] Wrote 4177920 bytes to encoder
[hevc_rkmpp @ 0xaaaad1b6d680] Received a packet
[vost#0:0/hevc_rkmpp @ 0xaaaad1b6d1d0] *** 2 dup!

Would really appreciate any help. Running sw decoder is not really an option due to power constraints.
Thanks!

ffmpeg built like this:

# Build and install MPP
WORKDIR /tmp/mpp
RUN git clone -b jellyfin-mpp --depth=1 https://github.com/nyanmisaka/mpp.git rkmpp
WORKDIR /tmp/mpp/build
RUN cmake -DCMAKE_INSTALL_PREFIX=/usr/local -DCMAKE_BUILD_TYPE=Release -DBUILD_SHARED_LIBS=ON -DBUILD_TEST=OFF ../rkmpp
RUN make -j$(nproc) install

# Build and install RGA
WORKDIR /tmp/rkrga
RUN git clone -b jellyfin-rga --depth=1 https://github.com/nyanmisaka/rk-mirrors.git src
WORKDIR /tmp/rkrga/build
RUN meson setup ../src --prefix=/usr/local --libdir=lib --buildtype=release --default-library=shared -Dcpp_args=-fpermissive -Dlibdrm=false -Dlibrga_demo=false
RUN ninja install

# Build and install FFmpeg
WORKDIR /tmp/ffmpeg
RUN git clone --depth=1 https://github.com/nyanmisaka/ffmpeg-rockchip.git src
WORKDIR /tmp/ffmpeg/build
RUN ../src/configure --prefix=/usr/local --enable-gpl --enable-version3 --enable-libdrm --enable-rkmpp --enable-rkrga --enable-libopus
RUN make -j$(nproc) install

[nyanmisaka]

It may be an interlaced video, please provide a sample to reproduce the problem.

[oryjkov]

No its not interlaced, but see for yourself:

https://drive.google.com/file/d/1OAxn8haTBIR-M4PlYhvvCo1ck-BUSufS/view?usp=sharing

[oryjkov]

That's my output on the above sample:

# ffmpeg -hide_banner -y  -hwaccel rkmpp     -hwaccel_output_format drm_prime     -i /logs/raw_videos/sample.mp4    -map_metadata 0     -c:a libopus -b:a 64k     -strict -2    -c:v hevc_rkmpp     -rc_mode 2     -qp_init 22     -tag:v hvc1 /logs/raw_videos/out.mp4
[mov,mp4,m4a,3gp,3g2,mj2 @ 0xaaaac99c30d0] st: 0 edit list: 1 Missing key frame while searching for timestamp: 1000
[mov,mp4,m4a,3gp,3g2,mj2 @ 0xaaaac99c30d0] st: 0 edit list 1 Cannot find an index entry before timestamp: 1000.
Input #0, mov,mp4,m4a,3gp,3g2,mj2, from '/logs/raw_videos/sample.mp4':
  Metadata:
    major_brand     : mp42
    minor_version   : 1
    compatible_brands: mp42avc1CAEP
    make            :
    make-eng        :
    model           :
    model-eng       :
    creation_time   : 2000-01-01T00:00:21.000000Z
  Duration: 00:00:03.88, start: 0.000000, bitrate: 30767 kb/s
  Stream #0:0[0x1](eng): Video: h264 (High) (avc1 / 0x31637661), yuvj420p(pc, bt709, progressive), 1920x1080, 30218 kb/s, 25 fps, 25 tbr, 25k tbn (default)
    Metadata:
      creation_time   : 2000-01-01T00:00:21.000000Z
      vendor_id       : [0][0][0][0]
  Stream #0:1[0x2](eng): Audio: aac (LC) (mp4a / 0x6134706D), 48000 Hz, stereo, fltp, 253 kb/s (default)
    Metadata:
      creation_time   : 2000-01-01T00:00:21.000000Z
      vendor_id       : [0][0][0][0]
Stream mapping:
  Stream #0:0 -> #0:0 (h264 (h264_rkmpp) -> hevc (hevc_rkmpp))
  Stream #0:1 -> #0:1 (aac (native) -> opus (libopus))
Press [q] to stop, [?] for help
Output #0, mp4, to '/logs/raw_videos/out.mp4':
  Metadata:
    major_brand     : mp42
    minor_version   : 1
    compatible_brands: mp42avc1CAEP
    make            :
    make-eng        :
    model           :
    model-eng       :
    creation_time   : 2000-01-01T00:00:21.000000Z
    encoder         : Lavf60.16.100
  Stream #0:0(eng): Video: hevc (Main) (hvc1 / 0x31637668), drm_prime(pc, bt709, progressive), 1920x1080, q=2-31, 2000 kb/s, 25 fps, 12800 tbn (default)
    Metadata:
      creation_time   : 2000-01-01T00:00:21.000000Z
      vendor_id       : [0][0][0][0]
      encoder         : Lavc60.31.102 hevc_rkmpp
  Stream #0:1(eng): Audio: opus (Opus / 0x7375704F), 48000 Hz, stereo, flt, 64 kb/s (default)
    Metadata:
      creation_time   : 2000-01-01T00:00:21.000000Z
      vendor_id       : [0][0][0][0]
      encoder         : Lavc60.31.102 libopus
[out#0/mp4 @ 0xaaaac99d4d40] video:917kB audio:27kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: 0.361994%
frame=   97 fps=0.0 q=-0.0 Lsize=     947kB time=00:00:03.85 bitrate=2012.8kbits/s dup=16 drop=0 speed=8.08x

[oryjkov]

@nyanmisaka is this info sufficient and can you reproduce the problem?

Let me know if you need anything more.

[nyanmisaka]

Sorry for the late reply. I missed this message.

I can reproduce the problem now. I think it is an issue caused by MPP runtime.

[nyanmisaka]

Reported here: rockchip-linux/mpp#872