[Feature] Automated API Documentation Pipeline for Qumat & QDP

[400Ping]

Summary

Implement an automated API documentation pipeline that generates and publishes API reference documentation from the Python (Qumat, QDP) and Rust (qdp-core) codebases, integrated into the project's Docusaurus website and CI.
Current state:

• QuMat API is maintained by hand and can drift from code.
• QDP API has a placeholder "TODO: Add API reference documentation for QDP".
• Rust (qdp-core) has extensive doc comments but no published rustdoc in the website.
  Goals:

1. Generate API reference from source for Python (Qumat).
2. Integrate generated docs into the existing Docusaurus site.
3. Automate the pipeline in CI so doc builds run on changes.
4. Define conventions (docstrings, public API) for future contributors.
   Deliverables:

• Python API doc pipeline for qumat and QDP.
• QuMat API reference either generated or explicitly linked.
• Rust (qdp-core) rustdoc built and linked from the website.
• CI job(s) that build Python API docs and rustdoc and fail on errors.
• Short contribution guide on docstring style and how to update API docs.

[ryankert01]

Nice, we need this feature.

[Vanessamae23]

Hi everyone! I drafted a rough sketch to address this (https://issues.apache.org/jira/browse/GSOC-313) although i am not sure if i am getting it right

This script currently handles the Python API generation using pdoc and rustdoc for the Rust with a automated CI and a bot to deploy latest updates if there is, just a very drafted version here : Vanessamae23#1

With samples here,
https://vanessamae23.github.io/mahout/api/rust/
https://vanessamae23.github.io/mahout/api/python

To run and see how it looks like in the web version: run start the docusaurus

I’d love to get some feedback on this approach and i am interested to take part in the gsoc but if it does not work or someone else has a better approach, sounds great too.. learnt along the way as well

[viiccwen]

Hi, @400Ping .

I’m interested in this GSoC proposal and would like to get involved.

I’ve contributed 15+ PRs to Apache Mahout and have been working closely with the QDP and QuMat codebases (Python, Rust, and CUDA...), so I’m fairly familiar with the current API surfaces, documentation gaps, and the existing website/CI setup.

I’ve started thinking about possible approaches (e.g., Python API docs via Sphinx or pdoc, rustdoc for qdp-core, and integrating or linking them from the Docusaurus site), and I’d really appreciate mentors’ thoughts on scope, tooling choices, and how best to integrate this with the current website.

Happy to discuss further or start with a small PoC if that’s helpful.

Thanks! 🙌

[400Ping]

Hi everyone! I drafted a rough sketch to address this (https://issues.apache.org/jira/browse/GSOC-313) although i am not sure if i am getting it right

This script currently handles the Python API generation using pdoc and rustdoc for the Rust with a automated CI and a bot to deploy latest updates if there is, just a very drafted version here : Vanessamae23#1

With samples here,
https://vanessamae23.github.io/mahout/api/rust/
https://vanessamae23.github.io/mahout/api/python

To run and see how it looks like in the web version: run start the docusaurus

I’d love to get some feedback on this approach and i am interested to take part in the gsoc but if it does not work or someone else has a better approach, sounds great too.. learnt along the way as well

Email me your proposal first and show me the pros and cons of different approaches. Also it will be best if you can show me a video of your PoC for this and write a step by step doc guide for your PoC. Also let's have a chat.

[400Ping]

Hi, @400Ping .

I’m interested in this GSoC proposal and would like to get involved.

I’ve contributed 15+ PRs to Apache Mahout and have been working closely with the QDP and QuMat codebases (Python, Rust, and CUDA...), so I’m fairly familiar with the current API surfaces, documentation gaps, and the existing website/CI setup.

I’ve started thinking about possible approaches (e.g., Python API docs via Sphinx or pdoc, rustdoc for qdp-core, and integrating or linking them from the Docusaurus site), and I’d really appreciate mentors’ thoughts on scope, tooling choices, and how best to integrate this with the current website.

Happy to discuss further or start with a small PoC if that’s helpful.

Thanks! 🙌

Sounds great, email me(jiekaichang@apache.org) and let's have a chat. I will love to have a proposal and a PoC for what you going to do.

[ryankert01]

@Vanessamae23 Sweet, please introduce yourself with the community on the dev mailing list! Thanks!