cuda - fix grad3d flat templates for gen

[jeremylt]

Purpose:

Fix my oops. That was totally copypasta

Closes: #1991

LLM/GenAI Disclosure:

None, pure human stupidity here

By submitting this PR, the author certifies to its contents as described by the Developer's Certificate of Origin.
Please follow the Contributing Guidelines for all PRs.

[zatkins-dev]

Does HIP also need this fix?

[jeremylt]

oddly no, I did that one correctly

[zatkins-dev]

huh, fair enough. Did you make sure it isn't still falling back? or should I check

[jeremylt]

I didn't double check

[zatkins-dev]

Seems to compile okay now. I'm now getting:

CUDA_ERROR_LAUNCH_OUT_OF_RESOURCES: max_threads_per_block 256 on block size (27,1,18), shared_size 8352, num_regs 250

[jeremylt]

Yea, I'm not surprised. Flattening should only work for really low order

[zatkins-dev]

Sounds good. Also, is there a reason we fall back straight to ref? In this case, fallback to shared would be faster

[jeremylt]

Honestly, probably an oversight

[jeremylt]

Oh, you know what I bet it is? Actually, should be using shared bases, just ref operator evaluation

[zatkins-dev]

Isn't that what the shared operator application does anyhow?

[jeremylt]

Yea. There is no 'shared operator' so I directly fell back to the compatible backend that implements the target functionality so it was clear (to me) where that operator application/assembly logic actually came from

[zatkins-dev]

Gotcha. We should switch the fallback to shared then?

[jeremylt]

I personally wouldn't because it is the operator implementation that is falling back and ref is what provides that fallback operator implementation, but I don't feel strongly

[zatkins-dev]

Hmm, will it still use the shared bases with the ref operator fallback?

[zatkins-dev]

yes it will

[zatkins-dev]

Okay, I think this is good! I'm going to have a follow up fixing block sizes for the non tensor gen, we can make it work for more cases