Add Claude Code Skills for Isaac Lab#4927
Conversation
xteo
requested review from
Mayankm96,
hhansen-bdai,
jtigue-bdai,
kellyguo11 and
ooctipus
as code owners
github-actions
bot
added
documentation
Greptile SummaryThis PR adds Claude Code agent skills/documentation for Isaac Lab and introduces two sets of aarch64/ARM64 compatibility fixes: (1) a Key issues found:
Confidence Score: 2/5
Important Files Changed
Flowchart%%{init: {'theme': 'neutral'}}%%
flowchart TD
A[render called - rgb_array mode] --> B{_rgb_annotator exists?}
B -- No --> C{sim.has_gui?}
C -- Yes --> D[Use cfg.viewer.cam_prim_path]
C -- No --> E[_setup_video_camera - create /World/VideoCamera USD prim]
E --> D
D --> F[Create render_product and attach RGB annotator]
F --> G
B -- Yes --> G[sim.forward - sync physics to Fabric]
G --> H[omni.kit.app.get_app update - advance Kit render loop]
H --> I[_rgb_annotator.get_data - return RGB frame]
subgraph aarch64 warp dtype fix
J[arr.view dtype called] -- RuntimeError on aarch64 --> K[_view_as_structured fallback - wp.array with ptr and new_shape]
J -- Success on x86_64 --> L[Typed warp array returned]
K --> L
end
|
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
source/isaaclab_physx/isaaclab_physx/assets/articulation/articulation_data.py
Outdated
Show resolved
Hide resolved
xteo
changed the title
- CLAUDE.md: Project guide with architecture overview, key commands, and conventions - .claude/skills/isaaclab/: Comprehensive skill set covering: - Setup: fresh install, verification, troubleshooting - Architecture: overview, patterns comparison, sensors/actuators - Environments: builder, MDP catalog, templates - Training: guide, hyperparameters, RL-Games/RSL-RL/SB3 references - Tutorials: cartpole, code walkthrough, experiments - Dev Journey report and results Tested on DGX Spark (GB10) and DGX Station (GB300).
AntoineRichard
left a comment
AntoineRichard
left a comment
There was a problem hiding this comment.
Thanks a lot for the PR, the skill would be really nice! My main concern is around the maintenance of this and keeping it relevant as we keep on developing. I think the skill should really much more heavily on the doc and internal files rather than its own hardcoded things. There are already a bunch of outdated things, and it will just keep on adding. The failures reported with the dtype is just a warp version mismatch and they should be removed everywhere. Maybe we need to improve our doc so that the agent can easily skim through it.
I would not completely ignore the things that are currently in AGENTS.md. Some of it is nice like the doc on the Changelog, etc... Could you look into integrating your CLAUDE.md into AGENTS.md? Also make sure CLAUDE.md is referencing AGENTS.md. This is so that it stays general.
There was a problem hiding this comment.
This should probably integrate the new physical backend and rendering backends / visualizers.
There was a problem hiding this comment.
Emphasis on how base classes in isaaclab (core) define abstract objects and the concrete implementations are then done in isaaclab_physx, isaaclab_newton, and isaaclab_ov.
There was a problem hiding this comment.
agreed, IsaacSim is not really part of the stack anymore and should be removed
| ## For More Detail | ||
|
|
||
| - [patterns-comparison.md](patterns-comparison.md) - Manager-based vs Direct deep-dive | ||
| - [sensors-actuators.md](sensors-actuators.md) - Cameras, IMU, actuator types, visualizers |
There was a problem hiding this comment.
With the above changes might be good to have a more thorough backend.md or something like that, that goes in details about these.
| ## Camera Sensors | ||
|
|
||
| **Important**: Camera/sensor config classes can only be deeply imported after `AppLauncher` | ||
| starts the Kit runtime. Top-level `import isaaclab.sensors` will fail otherwise. | ||
|
|
||
| ### Two Camera Types | ||
|
|
||
| | Type | Config | Use Case | | ||
| |------|--------|----------| | ||
| | **TiledCamera** | `TiledCameraCfg` | GPU-accelerated batched rendering. Recommended for RL training with many envs. | | ||
| | **Camera** | `CameraCfg` | Standard camera with per-camera rendering. Use for single-env or high-quality capture. | | ||
|
|
||
| `TiledCameraCfg` inherits from `CameraCfg` (same API) but uses tiled rendering for | ||
| GPU-parallel image generation across environments. Always prefer `TiledCameraCfg` for | ||
| RL training with visual observations. | ||
|
|
||
| ### TiledCamera Example (GPU-accelerated, recommended) | ||
| ```python | ||
| from isaaclab.sensors import TiledCameraCfg | ||
|
|
||
| camera = TiledCameraCfg( | ||
| prim_path="{ENV_REGEX_NS}/Camera", | ||
| data_types=["rgb", "depth"], | ||
| spawn=sim_utils.PinholeCameraCfg( | ||
| focal_length=24.0, | ||
| clipping_range=(0.1, 20.0), | ||
| ), | ||
| width=100, height=100, | ||
| ) | ||
| ``` |
There was a problem hiding this comment.
This is outdated for Lab 3.0. We now have cameras that do not depend on Kit.
| ### Available Data Types | ||
| - `rgb` - Color image (H, W, 3) | ||
| - `rgba` - Color + alpha (H, W, 4) | ||
| - `depth` / `distance_to_image_plane` - Depth map (H, W, 1) | ||
| - `distance_to_camera` - Ray distance to optical center | ||
| - `albedo` - Fast diffuse-albedo only (optimized, no lighting) | ||
| - `simple_shading_constant_diffuse` - Constant diffuse shading | ||
| - `simple_shading_diffuse_mdl` - MDL diffuse shading | ||
| - `simple_shading_full_mdl` - Full MDL shading | ||
| - `normals` - Surface normals (H, W, 3) | ||
| - `motion_vectors` - Optical flow | ||
| - `semantic_segmentation` - Semantic labels | ||
| - `instance_segmentation_fast` / `instance_id_segmentation_fast` - Instance IDs |
There was a problem hiding this comment.
This might not be true for all cameras in Lab 3.0
| ## Other Sensors | ||
|
|
||
| | Sensor | Config | Data | | ||
| |--------|--------|------| | ||
| | Contact | `ContactSensorCfg` | Contact forces, normals | | ||
| | IMU | `ImuCfg` | Acceleration, angular velocity | | ||
| | Frame Transformer | `FrameTransformerCfg` | Relative body transforms | |
There was a problem hiding this comment.
Here it might be good to have some details about how they handle different physics backend.
| > **Known Bug (develop branch)**: `Isaac-Cartpole-Direct-v0` crashes with a warp | ||
| > dtype error (`RuntimeError: Cannot cast dtypes of unequal byte size`) in | ||
| > `write_root_pose_to_sim()`. This affects ALL RL frameworks on the develop branch | ||
| > (Isaac Sim 6.0). Skip this experiment until the bug is fixed, or use the `main` | ||
| > branch where it may work. |
There was a problem hiding this comment.
Not a bug with the correct warp version.
There was a problem hiding this comment.
This and the comment that come with it are heavily targeted towards cartpole. Will it help agents with other environments? If the doc is lacking let's improve it and link it here.
| ## Validation Results | ||
|
|
||
| ### Test Environment | ||
|
|
||
| | Component | Value | | ||
| |-----------|-------| | ||
| | GPU | NVIDIA GB10 (aarch64, CUDA capability 12.1) | | ||
| | Driver | 580.126.09 | | ||
| | CUDA | 13.0 | | ||
| | OS | Ubuntu 24.04, Linux 6.17.0-1008-nvidia | | ||
| | RAM | 119 GB | | ||
| | Python | 3.12.12 (Kit Python) | | ||
| | PyTorch | 2.9.0+cu130 | | ||
| | Isaac Sim | 6.0.0-rc.13 (source build) | | ||
| | Isaac Lab | 4.0.0 (repo v2.3.2, develop branch) | | ||
|
|
||
| ### Phase 1: Environment Checking (5/5 PASS) | ||
|
|
||
| | Test | Result | Notes | | ||
| |------|--------|-------| | ||
| | GPU/CUDA detection | PASS | nvidia-smi, nvcc both work | | ||
| | Kit Python + PyTorch | PASS | torch.cuda.is_available() = True | | ||
| | All 4 RL framework imports | PASS | SB3 2.7.1, SKRL 1.4.3, RSL-RL 3.1.2, RL-Games 1.6.1 | | ||
| | All 9 Isaac Lab packages | PASS | isaaclab 4.0.0, all sub-packages installed | | ||
| | AppLauncher import | PASS | Omniverse Kit runtime boots correctly | | ||
|
|
||
| ### Phase 2: Installation Verification (5/5 PASS, 1 SKIP) | ||
|
|
||
| | Test | Result | Notes | | ||
| |------|--------|-------| | ||
| | `_isaac_sim` symlink | PASS | Points to IsaacSim build | | ||
| | `python.sh` + `setup_python_env.sh` | PASS | PYTHONPATH/LD_LIBRARY_PATH configured | | ||
| | `isaaclab.sh --help` | PASS | All CLI options documented | | ||
| | TensorBoard in Kit Python | PASS | v2.20.0, NOT on system PATH (documented) | | ||
| | Template creation (`--new`) | SKIP | Requires interactive TTY (InquirerPy) | | ||
|
|
||
| ### Phase 3: Training Readiness (3/3 PASS) | ||
|
|
||
| | Test | Result | Notes | | ||
| |------|--------|-------| | ||
| | All 4 `train.py --help` | PASS | Correct argument parsing | | ||
| | Agent config files | PASS | All YAML/Python configs match skill docs | | ||
| | Loss function locations | PASS | Verified via `inspect.getsourcefile()` | | ||
|
|
||
| ### Phase 4: CartPole Training (8/9 PASS, 1 KNOWN BUG) | ||
|
|
||
| | Test | Result | Time | Final Reward | | ||
| |------|--------|------|-------------| | ||
| | RSL-RL train (4096 envs) | **PASS** | 37s | 4.91 | | ||
| | SKRL train (4096 envs) | **PASS** | 55s | ~4.9 | | ||
| | RL-Games train (4096 envs) | **PASS** | 61s | 4.89 | | ||
| | SB3 train (64 envs) | **PASS** | 103s | 4.70 | | ||
| | TensorBoard reads all 4 logs | **PASS** | - | - | | ||
| | Checkpoints saved (all 4) | **PASS** | - | - | | ||
| | RSL-RL play.py (inference) | **PASS** | runs OK | - | | ||
| | Direct CartPole (RSL-RL) | **FAIL** | crash | warp dtype bug | |
There was a problem hiding this comment.
I'm a noob with this, but is this really needed? What can agents leverage from this. The fixed version numbers (the one from the spark use to develop this) might be more confusing than anything.
| > **Known Bug**: `Isaac-Cartpole-Direct-v0` crashes with `RuntimeError: Cannot cast | ||
| > dtypes of unequal byte size` in warp on the develop branch (Isaac Sim 6.0). This is | ||
| > an upstream issue, not a skill defect. The skill documents this with a warning and | ||
| > redirects users to `Isaac-Cartpole-v0` (manager-based). |
There was a problem hiding this comment.
Not a bug with the correct warp version. Consider removing. I don't know why your install defaulted to warp 1.10 or warp 1.11rc1
There was a problem hiding this comment.
It's not all bad in the current AGENTS.md. Also do not use claude.md directly. Use agents.md instead. Not everyone is using claude so let's keep this general.
### Pre-commit (lint/format hooks)
**CRITICAL: Always run pre-commit hooks BEFORE committing, not after.**
Proper workflow:
1. Make your code changes
2. Run `./isaaclab.sh -f` to check ALL files
3. If pre-commit modifies any files (e.g., formatting), review the changes
4. Stage the modified files with `git add`
5. Run `./isaaclab.sh -f` again to ensure all checks pass
6. Only then create your commit with `git commit`
# Run pre-commit checks on all files
./isaaclab.sh -f
**Common mistake to avoid:**
- Don't commit first and then run pre-commit (requires amending commits)
- Do run pre-commit before committing (clean workflow)
## Changelog
- **Update `CHANGELOG.rst` for every change** targeting the source directory. Each extension has its own changelog at `source/<package>/docs/CHANGELOG.rst` (e.g. `source/isaaclab/docs/CHANGELOG.rst`, `source/isaaclab_physx/docs/CHANGELOG.rst`).
- **Always create a new version heading.** Never add entries to an existing version — they are released and immutable. Bump the patch version (e.g. `1.5.0` → `1.5.1`) and use today's date.
- **Bump `config/extension.toml` to match.** When creating a new changelog version, update the `version` field in `source/<package>/config/extension.toml` to the same version string.
- **Determine which changelog(s) to update** by looking at which `source/<package>/` directories your changes touch. A single PR may require entries in multiple changelogs.
- Use **past tense** matching the section header: "Added X", "Fixed Y", "Changed Z".
- Place entries under the correct category: `Added`, `Changed`, `Deprecated`, `Removed`, or `Fixed`.
- Avoid internal implementation details users wouldn't understand.
- **For `Deprecated`, `Changed`, and `Removed` entries, include migration guidance.**
- Example: "Deprecated `Articulation.A` in favor of `Articulation.B`."
- Use Sphinx cross-reference roles for class/method/module names.
### RST formatting reference
X.Y.Z (YYYY-MM-DD)
~~~~~~~~~~~~~~~~~~
Added
^^^^^
* Added :class:`~package.ClassName` to support feature X.
Fixed
^^^^^
* Fixed edge case in :meth:`~package.ClassName.method` where input was
not validated, causing ``AttributeError`` at runtime.
Key formatting rules:
- Version heading: underline with `~` (tildes), must be at least as long as the heading text.
- Category heading: underline with `^` (carets).
- Entries: `* ` prefix, continuation lines indented by 2 spaces.
- Blank line between the last entry and the next version heading.
## Commit and Pull Request Guidelines
Follow conventional commit message practices.
- **Use feature branches**: All development work should be on branches named `<username>/feature-desc` (e.g., `jdoe/docs-versioning`). Do not commit directly to `main`.
- Keep commits focused and atomic—one logical change per commit.
- Reference related issues in commit messages when applicable.
- **When iterating on PR feedback**, prefer adding new commits over amending existing ones. This avoids force-pushing and lets the reviewer easily verify each change request was addressed.
- **Do not include AI attribution or co-authorship lines** (e.g., "Co-Authored-By: Claude...") in commit messages. Commits should represent human contributions without explicit AI attribution.
- **Commit message format**:
- Separate subject from body with a blank line
- Subject: imperative mood, capitalized, ~50 chars, no trailing period
- Write as a command: "Fix bug" not "Fixed bug" or "Fixes bug"
- Test: "If applied, this commit will _[your subject]_"
- Body: wrap at 72 chars, explain _what_ and _why_ (not _how_—the diff shows that)
This is nice in particular the changelog part so it know how to update things.
There was a problem hiding this comment.
I would remove this entirely. It's not needed, and I don't think we should keep it.
There was a problem hiding this comment.
I would remove this completely. It's not needed.
pascal-roth
left a comment
pascal-roth
left a comment
There was a problem hiding this comment.
I am in huge favor of introducing Claude skills into our codebase to help with debugging and installation, so thanks for that.
However, adding all the explain setup (which I assume claude generated for itself), seems like an overkill to me. I use Claude skills and link them to our docs, which works very well for me, and such a workflow should be able to detect similar issues / help in a similar way as in the case addressed here! Could we try to do it without the extra docs / expand our current docs if something is missing?
There was a problem hiding this comment.
agreed, IsaacSim is not really part of the stack anymore and should be removed
| --- | ||
| name: isaaclab | ||
| description: | | ||
| Isaac Lab robotics RL framework. Use for: environment setup/troubleshooting, |
There was a problem hiding this comment.
| Isaac Lab robotics RL framework. Use for: environment setup/troubleshooting, | |
| Isaac Lab robotics learning framework. Use for: environment setup/troubleshooting, |
Review Response — Skills Slimmed, Docs Referenced, Lab 3.0 UpdatedThanks @AntoineRichard and @pascal-roth for the thorough review. Fully agree with the philosophy: reference docs, don't duplicate. Here's what changed: CLAUDE.md → AGENTS.md integration
Files removed
Files replaced with doc pointers
Files heavily trimmed
Files updated for Lab 3.0
New file
Other fixes
Net result: 326 lines added, 1,042 removed. Skills now lean heavily on official docs and real code examples rather than hardcoded content. |
|
Forgot to push? :D I'm wondering if troubleshooting.md could be in the doc directly. @pascal-roth ? |
Co-authored-by: Pascal Roth <57946385+pascal-roth@users.noreply.github.com> Signed-off-by: xTeo <xteo@xteo.com>
Co-authored-by: Pascal Roth <57946385+pascal-roth@users.noreply.github.com> Signed-off-by: xTeo <xteo@xteo.com>
Co-authored-by: Pascal Roth <57946385+pascal-roth@users.noreply.github.com> Signed-off-by: xTeo <xteo@xteo.com>
Co-authored-by: Pascal Roth <57946385+pascal-roth@users.noreply.github.com> Signed-off-by: xTeo <xteo@xteo.com>
Respond to reviewer feedback from AntoineRichard and pascal-roth: - Make CLAUDE.md a thin pointer to AGENTS.md (avoid duplication) - Remove mdp-catalog.md (maintenance burden, will get outdated) - Replace fresh-install.md, verification.md, templates.md, and training/guide.md with concise pointers to official docs and real code examples - Heavily trim troubleshooting.md: remove warp dtype issues (version mismatch, not real bug), remove extension cache section, remove outdated Kit dependency claims, simplify TensorBoard instructions - Update architecture/overview.md for Lab 3.0: remove Isaac Sim as stack dependency, add multi-backend factory architecture - Update sensors-actuators.md: fix quaternion order to xyzw, add Viser visualizer, note Kit-independent cameras, add backend-aware sensor section, remove viewport controls - Add new backends.md covering physics backends (PhysX/Newton), renderers (Isaac RTX/Newton Warp/OVRTX), and all 4 visualizers - Update SKILL.md index to reflect file changes - Fix env count guidance to 256+ (no upper limit) - Add JAX pre-allocation note (XLA_PYTHON_CLIENT_PREALLOCATE) - Note lazy imports have improved import traversal issues Core philosophy: reference docs, don't duplicate. Keep only agent-specific guidance that won't get outdated.
AntoineRichard
left a comment
AntoineRichard
left a comment
There was a problem hiding this comment.
I think this is still not leaning enough on the doc. I'm fine with files referencing the doc, but I'd like to avoid having things that redefine documentation level information there.
There was a problem hiding this comment.
There is a PR for this here. I'd rather reference the doc directly. #4975
There was a problem hiding this comment.
If the doc is not good enough on that let's just improve it.
There was a problem hiding this comment.
Can these be moved to the doc? https://isaac-sim.github.io/IsaacLab/develop/source/refs/troubleshooting.html
There was a problem hiding this comment.
This should point to the doc. If the doc, is not good enough improve the doc.
There was a problem hiding this comment.
Same here this should point to the doc.
There was a problem hiding this comment.
I would rather have something that points to the doc directly.
3 participants
Summary
Adds Claude Code agent skills and a project guide (CLAUDE.md) for Isaac Lab. No changes to source code, apps, or docs — skills only.
Files Changed
CLAUDE.md— Project guide with architecture overview, key commands, and conventions.claude/skills/isaaclab/— Comprehensive skill set:What are Claude Code Skills?
Claude Code skills (
.claude/skills/) are markdown files that provide domain-specific knowledge to AI coding agents. They help Claude Code (and similar agents) understand Isaac Lab's architecture, conventions, and workflows — enabling more accurate code generation and debugging assistance.Testing
Skills developed and validated on:
Notes
.claude/skills/convention