Add option to use CUDA graphs with TRT for RF-DETR Object Detection in inference_models

[mkaic]

What does this PR do?

Adds a use_cuda_graph flag to RFDetrForObjectDetectionTRT.forward which enables capturing the CUDA graph and replaying it, unlocking a nice ~10% FPS speedup (observed range was anything from 7% to 12% depending on how saturated the GPU is) on rfdetr-nano.

After the CUDA graph is captured in execute_trt_engine, it and other state related to it is packaged into a TRTCudaGraphState dataclass which is returned back up to the RFDetrForObjectDetectionTRT instance, where it is cached. This lets subsequent forward passes simply replay the graph instead of recapturing it.

Because this PR changes the return signature of infer_with_trt_engine, there are a lot of files with 1-or-2 lines of changes where I'm just trying to prevent unpacking errors.

Type of Change

• New feature (non-breaking change that adds functionality)

Testing

• I have tested this change locally
• I have added/updated tests for this change

Test details:
I added an integration test at inference_models/tests/integration_tests/models/test_rfdetr_predictions_trt.py

I also added a profiling script at inference_models/development/profiling/profile_rfdetr_trt_cudagraphs.py benchmarks the speed of RFDetrForObjectDetectionTRT both with and without CUDA graph. Here are some example results (some variance was observed here which depended on GPU cooldown):

==================================================
RF-DETR Nano Object Detection
10,000 iterations at 384x384 (testing on random noise) on an L4 GPU
Forward pass FPS (no CUDA graphs): 574.9
Forward pass FPS (CUDA graphs):    640.0
Speedup: 1.11x
==================================================

I intended to also write a similar script for the rfdetr-seg-* models, but as far as I can tell, the seg models don't have inference_models packages yet, other than rfdetr-seg-preview, and that only has torch and onnx packages. I did have Claude make me a script that downloads the onnx package, uses one of the development compilation convenience functions to compile it to TRT, and then runs my benchmark... but that script is an utter mess (to the point that it's not even included in the PR), and I'm getting the impression that support for TRT as a backend for RF-DETR-Seg might be a work in progress in inference_models? The results from the script nonetheless seem to confirm a speedup for seg models too:

==================================================
RF-DETR Seg Preview (compiled from ONNX to TRT fp16)
10,000 iterations at 384x384 (testing on random noise) on an L4 GPU
Forward pass FPS (no CUDA graphs): 278.6
Forward pass FPS (CUDA graphs):    299.0
Speedup: 1.07x
==================================================

EDIT 2026-02-11: Added a VRAM-profiling script inference_models/development/profiling/profile_cudagraph_vram.py to test what CPU and GPU memory usage look like when caching multiple shapes. Below are three plots of VRAM and system RAM usage in different scenarios.

1. Looping through batchsizes 1..16 in ascending order with cache capacity set to 16.

2. Looping through batchsizes 1..16 in random order with cache capacity set to 16.

3. Looping through batchsizes 1..16 in random order with cache capacity set to 8, causing rolling cache evictions.

I also added a test to check that cache eviction works the way I would expect it to, in test_yolov8_object_detection_predictions_trt.py.

Checklist

• My code follows the style guidelines of this project
• I have performed a self-review of my own code
• I have commented my code where necessary, particularly in hard-to-understand areas
• My changes generate no new warnings or errors
• I have updated the documentation accordingly (if applicable)

Additional Context

N/A

[mkaic]

I have added LRU caching with (shape, dtype, device) keys for CUDA graph execution, and added a test for this caching in test_yolov8_object_detection_predictions_trt.py. While I was at it, I also added a test to make sure outputs are equivalent with use_cuda_graph=True and use_cuda_graph=False for rfdetr-seg-nano since previously I only included a test for the object detection RF-DETR.

I have run GPU trt_extra integration tests through GitHub actions on this branch and can confirm that they pass.