Supported kernel driver versions?

[gnull]

Hi!

I'm trying to get rkllama running on different kernel drivers for rknpu. I got it working with RKNPU, but not with the mainline rocket. Is there a way to know which of these it's supposed to work with?

Ivan

[flappyjet]

Hi, have you got an answer?
I have not a rk3588 device now, but I know the way user's program (tensorflow/pytorch) to NPU driver.
That is below:
NPU hardware <===kernel module<===runtime<===(glue layer)<=== user's program

provider	kernel	runtime	delegate lib(glue layer)
rockchip(vendor)	rknpu.ko	librknnrt.so, librkllmrt.so	librknn_delegate.so
linux	rocket.ko	libgallium.so or sth else	teflon.so

And you have to select ONE provider.

[flappyjet]

Could you or anyone else do me a favour if you still have a rk3588 device? Call for Testing: NPU detection Script Thanks very much.

[gnull]

No, I still haven't gotten an answer. Thanks for the breakdown.

Is rkllama compatible with only one line of your table or both? I got it working with RKNPU once, but it somehow stopped working recently.

Will take a look at your detection script. Thanks!

[gnull]

I think my question is somewhat answered by looking at the rknn-llm repo.

1. Rkllama README says “rkllm-runtime: V 1.2.3”
2. The rknn-llm repo under tag 1.2.3 contains a tarball with npu driver version 0.9.8. I suppose this means that rkllama will work with RKNPU driver version 0.9.8.

[flappyjet]

No, I still haven't gotten an answer. Thanks for the breakdown.

Is rkllama compatible with only one line of your table or both? I got it working with RKNPU once, but it somehow stopped working recently.

Will take a look at your detection script. Thanks!

Yes, just one line. I means RKNPU(vendor) for now. But in future, for more devices, I think it's better to be changed.

[flappyjet]

I think my question is somewhat answered by looking at the rknn-llm repo.

1. Rkllama README says “rkllm-runtime: V 1.2.3”
2. The rknn-llm repo under tag 1.2.3 contains a tarball with npu driver version 0.9.8. I suppose this means that rkllama will work with RKNPU driver version 0.9.8.

Now, there is a tight coupling between the vendor's user-space runtime and the kernel driver.

Gemini says "While librknnrt.so is used for general computer vision models (YOLO, ResNet, etc.), librkllmrt.so is part of the RKLLM software stack. Both librknnrt.so and librkllmrt.so talk to the same rknpu kernel driver. However, because LLMs are extremely demanding, they often require a very recent version of the kernel driver (typically v0.9.8 or higher). Older drivers that worked fine for simple image recognition will often fail or crash when librkllmrt.so tries to use them for LLM inference."