Rotation/translation API questions

[clarkbab]

Hi there, thanks for developing this tool!

I'm just a little confused by the rotation/translation values in the introduction. Specifically this part in "Rendering multiple DRRs at once":

rot = torch.tensor([[0.0, 0.0, 0.0], [0.0, 0.0, torch.pi]], device=device)
xyz = torch.tensor([[0.0, 850.0, 0.0], [0.0, 850.0, 0.0]], device=device)
img = drr(rot, xyz, parameterization="euler_angles", convention="ZXY")

1. From the commands it looks like we'd expect a 180 degree flip around the y-axis (rot=[0.0, 0.0, torch.pi], convention="ZXY"). However, the image has been flipped around LR (x-axis)? I thought maybe this is defined in camera coords (z=principal axis, x/y aligned with detector), but that would also be a flip around the x-axis.
2. With the translation param, I expected this to be in camera coords (instead of world coords where translations would need to be rotated also), and the underlying code looks like this is the case with the extrinsic matrix being calculated [R | Rt]. But shouldn't we be translating along the z-axis (camera's principle axis) then?
3. Does convention="ZXY" apply to the rotation only?

I guess I'm just trying to understand what these x/y/z translations/rotations represent and in which coordinate systems they're expressed? Forgive me if this is already documented :)

Thanks,
Brett

[eigenvivek]

Hi @clarkbab , this tutorial might be helpful for understanding which axes we're rotating around: https://vivekg.dev/DiffDRR/tutorials/geometry.html

1. With zero rotation, Z is inferior-superior (world-up), Y is anterior-posterior, and X is medial-lateral.
2. I specifically use the inverse extrinsic matrix to model where the camera is in world coordinates. This makes it easier for me to construct the rays that are used for rendering.
3. Yup convention only applies to the rotation when constructing R!

[eigenvivek]

Also if you want to use more traditional intrinsic/extrinsic matrices (e.g., from Hartley & Zissermann), you may find nanodrr useful: https://github.com/eigenvivek/nanodrr

I recently reimplemented all the core functionality of DiffDRR in this package, but used more conventional camera geometry because I wanted to play around with epipolar geometry. nanodrr still uses inverse intrinsic/extrinsic matrices under the hood for ray construction, but you can just invert standard matrices and plug them in.

[clarkbab]

Thanks for the quick reply! I'll check out nanodrr, it might be exactly what I'm after!

[clarkbab]

Sorry, I haven't had a chance to migrate to nanodrr as all the code I've written is for DiffDRR.

I had a further question: is it possible to set the centre of rotation to something other than the image centre? I'm generating DRRs from a planning CT and trying to replicate the positions of x-ray images taken during treatment. The treatment isocentre is known and different from the CT isocentre. I'm adjusting the CT affine so that the isocentre is at the treatment isocentre, but DiffDRR seems to be just setting the isocentre to the voxel centre of the image for every affine passed. "DRR subject canon." shows the affine within diffdrr after the "canonicalize" call.

Is it intentional that the centre of rotation is always the voxel centre of the image? Or is there some way to set it to another point?

When adjusting affine so treatment iso is at world iso:

volume = tio.ScalarImage(tensor=volume, affine=affine)
subject = read(volume)
>
DRR volume: [[   1.171875      0.            0.         -296.0122118 ]
 [   0.            1.171875      0.         -300.46636909]
 [   0.            0.            1.         -168.01905823]
 [   0.            0.            0.            1.        ]]
DRR density: [[   1.171875      0.            0.         -296.0122118 ]
 [   0.            1.171875      0.         -300.46636909]
 [   0.            0.            1.         -168.01905823]
 [   0.            0.            0.            1.        ]]
DRR subject: [[   1.171875      0.            0.         -296.0122118 ]
 [   0.            1.171875      0.         -300.46636909]
 [   0.            0.            1.         -168.01905823]
 [   0.            0.            0.            1.        ]]
DRR subject canon. [[   1.171875     0.           0.        -299.4140625]
 [   0.           1.171875     0.        -299.4140625]
 [   0.           0.           1.        -175.       ]
 [   0.           0.           0.           1.       ]]

Without adjustment:

volume = tio.ScalarImage(tensor=volume, affine=affine)
subject = read(volume)
>
[[   1.171875     0.           0.        -299.4140625]
 [   0.           1.171875     0.        -483.9140625]
 [   0.           0.           1.        -505.       ]
 [   0.           0.           0.           1.       ]]
DRR volume: [[   1.171875     0.           0.        -299.4140625]
 [   0.           1.171875     0.        -483.9140625]
 [   0.           0.           1.        -505.       ]
 [   0.           0.           0.           1.       ]]
DRR density: [[   1.171875     0.           0.        -299.4140625]
 [   0.           1.171875     0.        -483.9140625]
 [   0.           0.           1.        -505.       ]
 [   0.           0.           0.           1.       ]]
DRR subject: [[   1.171875     0.           0.        -299.4140625]
 [   0.           1.171875     0.        -483.9140625]
 [   0.           0.           1.        -505.       ]
 [   0.           0.           0.           1.       ]]
DRR subject canon. [[   1.171875     0.           0.        -299.4140625]
 [   0.           1.171875     0.        -299.4140625]
 [   0.           0.           1.        -175.       ]
 [   0.           0.           0.           1.       ]]

Thanks,
Brett

[clarkbab]

Update: I found the "center_volume" param and set it to False :) Thanks!

[eigenvivek]

Yup that's the one! To answer your previous question, this is the default in DiffDRR because it makes a lot of optimization on SE(3) much better conditioned. However, because of situations like this, I decided it's weird to center volumes at the origin by default, so that's disabled in nanodrr. :)

Instead, we just figured out how to precondition optimization based on the volume's isocenter so the optimization works just as well.

[clarkbab]

Last question: Does nanodrr support passing masks with multiple channels, i.e. more than one structure? It doesn't look this is currently supported in DiffDRR as it takes a labelmap of shape (1, X, Y, Z) but squeezes it to (X, Y, Z).

[eigenvivek]

Both packages support multistructure rendering, but expect the labelmap to use distinct values for different structures. Here's an example of a mask that we could render.

mask.nii.gz

[clarkbab]

Oh I see, but the single-channel approach doesn't handle overlapping labels - e.g. overlapping GTV and lung masks. Any plans to convert to multi-channel in future?

[eigenvivek]

Ah I understand - cool use case! You're right that we don't natively support multichannel label maps, but it's feasible to render them in nanodrr.

For example, if labels is a list of filepaths of (potentially overlapping) binary labelmaps:

import torch
from jaxtyping import Float
from nanodrr.data import Subject
from nanodrr.drr import DRR

drr: DRR = ...
extrinsic_inv: Float[torch.Tensor, "B 4 4"] = ...
imagepath = ...      # Your image
labelpaths = [...]   # Your list of binary labelmaps

subjects = [Subject.from_filepath(imagepath, labelpath) for labelpath in labels]
img = [drr(subject, extrinsic_inv) for subject in subjects]
img = torch.cat(img, dim=1)

This does unfortunately copy the image tensor C times for each of the binary labels you have... if this turns out to be an insufficient solution for your needs, feel free to open a new issue in nanodrr and we can brainstorm how to best implement this feature.

[clarkbab]

Thanks! I'm planning on using many surrogate structures (3D segmentation masks from total segmentation) so running this in a loop fashion might be slow, but I'll try for now and see how I go!