rfdetr-seg: Triton kernel for post-processing (Jetson Orin NX 8GB)

[aseembits93]

What does this PR do?

It optimizes RF-DETR instance-segmentation RLE post-processing:

inference_models.models.rfdetr.common.post_process_instance_segmentation_results_to_rle_masks

The hot path generates compact COCO RLE records from RF-DETR mask logits with Triton instead of materializing full-size dense upsampled masks and compressing them back to RLE. The implementation selects query/class metadata on GPU, supports up to 4 passing classes per query, generates sparse RLE runs directly from interpolation metadata, and copies only compact metadata/run records back.

The torch reference path remains available when Triton fast-path preconditions are not met.

Type of Change

• Other: Performance improvement

Testing

• I have tested this change locally
• I have added/updated tests or benchmark coverage for this change

Test details:

Everything ran on a Jetson Orin NX 8GB

• Performance gains on TensorRT video input. Run twice; table below uses the warmed run so first Triton compile does not dominate the measurement.

Reference command on main:

env -u INFERENCE_MODELS_RFDETR_TRITON_POSTPROC_ENABLED=false \
  ENABLE_AUTO_CUDA_GRAPHS_FOR_TRT_BACKEND=false \
  INFERENCE_MODELS_RFDETR_TRITON_PREPROC_ENABLED=false \
  RFDETR_PIPELINE_DEPTH=1 \
  PYTHONPATH=<repo>:<repo>/inference_models \
  python development/stream_interface/rfdetr_nano_seg_trt_workflow.py \
    --video_reference <video> \
    --backend trt

Candidate command on opt-python-postproc:

env -u INFERENCE_MODELS_RFDETR_TRITON_POSTPROC_ENABLED=true \
  ENABLE_AUTO_CUDA_GRAPHS_FOR_TRT_BACKEND=false \
  INFERENCE_MODELS_RFDETR_TRITON_PREPROC_ENABLED=false \
  RFDETR_PIPELINE_DEPTH=1 \
  PYTHONPATH=<repo>:<repo>/inference_models \
  python development/stream_interface/rfdetr_nano_seg_trt_workflow.py \
    --video_reference <video> \
    --backend trt

vehicles_312px.mp4 (538 frames, src 312x176):

	fps	elapsed	ms/frame
main reference	35.19	15.29 s	28.42
Triton RLE postproc	41.44	12.98 s	24.13
Delta	+17.8%	-2.31 s	-4.29 ms

vehicles_720p.mp4 (538 frames, src 1280x720):

	fps	elapsed	ms/frame
main reference	18.18	29.59 s	55.00
Triton RLE postproc	21.02	25.59 s	47.57
Delta	+15.6%	-4.00 s	-7.43 ms

vehicles_1080p.mp4 (538 frames, src 1920x1080):

	fps	elapsed	ms/frame
main reference	10.83	49.69 s	92.36
Triton RLE postproc	12.64	42.56 s	79.11
Delta	+16.7%	-7.13 s	-13.25 ms

• Correctness on 1000 same-shape COCO val2017 images vs main flags-off

env PARITY_MODEL_PATH=<repo>/rfdetr-seg-nano-orin-trt-package \
  PYTHONPATH=<repo>:<repo>/inference_models \
  python development/stream_interface/rfdetr_coco_same_shape_parity.py \
    --base-ref main \
    --candidate-ref opt-python-postproc \
    --coco-dir <repo>/coco/val2017 \
    --height 480 \
    --width 640 \
    --image-count 1000 \
    --warmup-frames 10

	main flags-off	opt-python-postproc
Images	1000	1000
Detections	5959	5959
Matched same-class IoU > 0.5	5959 (100.00%)	—
Count-mismatch images	0	—
Class-id disagreements	0	—
Mean / min box IoU	0.999992 / 0.981132	—
Mean / max |Δscore|	0.000e+00 / 0.000e+00	—
Mean / min mask IoU	0.999497 / 0.000000	—
Byte-identical RLEs	5954 / 5959	—

The non-byte-identical RLEs decode to semantically matching masks; the minimum mask IoU is caused by empty-mask IoU bookkeeping.

• Postprocess microbench replay from captured e2e inputs

env INFERENCE_MODELS_RFDETR_TRITON_POSTPROC_ENABLED=true \
  PYTHONPATH=<repo>:<repo>/inference_models \
  python development/stream_interface/rfdetr_rle_postprocess_microbenchmark.py \
    --mode replay \
    --cases-dir <captured-cases-dir> \
    --repeats 3 \
    --warmup-repeats 1 \
    --device auto

Each row uses 100 captured calls and 300 timed replays.

captured case set	mean	p50	p90	p99	correctness
vehicles_312px	1.931 ms	1.875 ms	2.217 ms	2.426 ms	matched captured outputs
vehicles_720p	2.173 ms	2.144 ms	2.473 ms	2.528 ms	matched captured outputs
vehicles_1080p	2.410 ms	2.376 ms	2.819 ms	2.904 ms	matched captured outputs

The captured replay cases can be regenerated with the same script by using --mode capture-and-replay --video_reference <video> --cases-dir <new-cases-dir>.

• Unit tests and compile checks

python -m py_compile \
  development/stream_interface/rfdetr_nano_seg_trt_workflow.py \
  development/stream_interface/rfdetr_rle_postprocess_microbenchmark.py \
  development/stream_interface/rfdetr_coco_same_shape_parity.py \
  inference_models/inference_models/models/rfdetr/triton_postprocess.py

PYTHONPATH=<repo>:<repo>/inference_models \
  python -m pytest -q inference_models/tests/unit_tests/models/rfdetr/test_triton_postprocess.py

PYTHONPATH=<repo>:<repo>/inference_models \
  python -m pytest -q inference_models/tests/unit_tests/models/common/test_rle_utils.py

Results: 22 passed and 36 passed.

Checklist

• My code follows the style guidelines of this project
• I have performed a self-review of my own code
• My changes generate no new runtime errors in the tested paths above
• I have updated the documentation accordingly (if applicable)

Additional Context

This PR is intentionally scoped to Triton RLE post-processing. The next PR adds Triton preprocessing on top of this branch, and the top PR adds CUDA graph / depth-2 pipeline integration plus CPU response-path optimizations.