apple/ml-sharp is very good at generating a 3DGS scene from a single input image. It would be great to be able to pass an existing pointcloud as seed/initialization of the 3Dgs training, the same way one can do it for training a 3dgs scene with eg postshot (with sparse or dense colmap ptc). The relocalization within the scene of the input 2D photo could either be user-provided (extrinsics + intrinsics in the scene coordinate-system, eg as a colmap standard/format for params), or computed on-the-go doing a rough registration of the intermediate metric depth of the 2D image.
The goal is to align/register the input photo generated 3DGS with the existing pointcloud geometry. This would help a lot generating the 3DGS pointcloud for scenes not often found in the training set, like very natural landscapes with varying depth, which currently ml-sharp is pretty bad at estimating.
Proposed Feature
Add support for providing an input seed point cloud (e.g., .ply format) that ml-sharp can use to:
- Initialize the 3DGS positions based on existing geometry
- Constrain/align the predicted Gaussians to match the coordinate system of the seed
- Optionally: perform pose estimation to localize the input photo relative to the seed point cloud
Happy to provide more details about my specific use case or help test any experimental implementations!
apple/ml-sharpis very good at generating a 3DGS scene from a single input image. It would be great to be able to pass an existing pointcloud as seed/initialization of the 3Dgs training, the same way one can do it for training a 3dgs scene with eg postshot (with sparse or dense colmap ptc). The relocalization within the scene of the input 2D photo could either be user-provided (extrinsics + intrinsics in the scene coordinate-system, eg as a colmap standard/format for params), or computed on-the-go doing a rough registration of the intermediate metric depth of the 2D image.The goal is to align/register the input photo generated 3DGS with the existing pointcloud geometry. This would help a lot generating the 3DGS pointcloud for scenes not often found in the training set, like very natural landscapes with varying depth, which currently ml-sharp is pretty bad at estimating.
Proposed Feature
Add support for providing an input seed point cloud (e.g.,
.plyformat) that ml-sharp can use to:Happy to provide more details about my specific use case or help test any experimental implementations!