Skip to content

Eye tracking#91

Merged
jayvdb merged 1 commit into
mainfrom
eye
Jul 20, 2026
Merged

Eye tracking#91
jayvdb merged 1 commit into
mainfrom
eye

Conversation

@jayvdb

@jayvdb jayvdb commented Jul 20, 2026

Copy link
Copy Markdown
Member

Summary by CodeRabbit

  • New Features

    • Added a browser-based eye-detection module using the webcam and MediaPipe FaceLandmarker.
    • Eye-detection results now include face and left/right eye bounding boxes.
    • Face-detection results and visualizations now include five landmark points.
    • Added bundled model assets and packaging for the new eye-detection workflow.
  • Bug Fixes

    • Improved model asset downloading and checksum verification.
    • Improved authentication handling for dependency audits.
  • Tests

    • Added coverage for eye detection, landmarks, model loading, and browser workflow behavior.

@coderabbitai

coderabbitai Bot commented Jul 20, 2026

Copy link
Copy Markdown

Review Change Stack

📝 Walkthrough

Walkthrough

Adds the pyeye1 MediaPipe eye-detection workflow, packages its model asset, integrates Python and browser execution, extends face detection with landmarks, and updates build, coverage, runner, npm, and maintenance configuration.

Changes

Eye workflow integration

Layer / File(s) Summary
Runtime assets and workspace wiring
.mise/*, config/*, data/model-modules/*, libs/edge-toolkit/*, pyproject.toml, services/ws-modules/pyeye1/pyproject.toml
Registers the MediaPipe dependency, eye model asset, workspace package, npm resolution, cache publishing, and runtime metadata.
Eye detection pipeline
services/ws-modules/pyeye1/pyeye1/*, services/ws-modules/pyeye1/tests/*
Adds normalized landmark conversion, eye and face boxes, event/status serialization, runtime controls, validation, and unit tests.
Browser eye workflow
services/ws-modules/pyeye1/pkg/et_ws_pyeye1.js
Loads Pyodide and FaceLandmarker, captures webcam frames, sends websocket events, renders detections, and performs cleanup.
Face landmark output
services/ws-modules/face-detection/*, services/ws-modules/pyface1/*
Adds RetinaFace landmark decoding, labeled detection payloads, canvas rendering, and tests.
Build, coverage, and runner integration
.mise/config.python.toml, .mise/config.coverage.toml, config/coverage.toml, services/ws-web-runner/tests/modules.rs
Adds pyeye1 testing, wheel building, dependency prefetching, combined coverage paths, and hardware-dependent runner coverage.

Maintenance and audit configuration

Layer / File(s) Summary
Audit task and annotations
.mise/config.toml, Cargo.toml
Adds fallback GitHub token authentication for cargo unmaintained and revises ignored-crate annotations.

Estimated code review effort: 4 (Complex) | ~60 minutes

Suggested reviewers: pierre-tenedero

Sequence Diagram(s)

sequenceDiagram
  participant Browser
  participant WsClient
  participant FaceLandmarker
  participant Pyodide

  Browser->>WsClient: Connect
  Browser->>FaceLandmarker: Detect webcam frame
  FaceLandmarker-->>Browser: Return face landmarks
  Browser->>Pyodide: Run eye post-processing
  Pyodide->>WsClient: Send inference event
  Pyodide-->>Browser: Render eye boxes
Loading
🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Docstring Coverage ✅ Passed Docstring coverage is 100.00% which is sufficient. The required threshold is 80.00%.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
Title check ✅ Passed The title is concise and clearly related to the eye-detection/eye-module changes in this PR.
✨ Finishing Touches
📝 Generate docstrings
  • Create stacked PR
  • Commit on current branch
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch eye

Comment @coderabbitai help to get the list of available commands.

@deepsource-io

deepsource-io Bot commented Jul 20, 2026

Copy link
Copy Markdown

DeepSource Code Review

We reviewed changes in a919903...20e722f on this pull request. Below is the summary for the review, and you can see the individual issues we found as inline review comments.

See full review on DeepSource ↗

Important

Some issues found as part of this review are outside of the diff in this pull request and aren't shown in the inline review comments due to GitHub's API limitations. You can see those issues on the DeepSource dashboard.

PR Report Card

Overall Grade  

Focus Area: Reliability
Security  

Reliability  

Complexity  

Hygiene  

Coverage  

Code Review Summary

Analyzer Status Updated (UTC) Details
C# Jul 20, 2026 7:12a.m. Review ↗
C & C++ Jul 20, 2026 7:12a.m. Review ↗
Docker Jul 20, 2026 7:12a.m. Review ↗
Java Jul 20, 2026 7:12a.m. Review ↗
JavaScript Jul 20, 2026 7:12a.m. Review ↗
Python Jul 20, 2026 7:12a.m. Review ↗
Rust Jul 20, 2026 7:12a.m. Review ↗
Secrets Jul 20, 2026 7:12a.m. Review ↗
Code coverage Jul 20, 2026 7:36a.m. Review ↗

Code Coverage Summary

Language Line Coverage (New Code) Line Coverage (Overall)
Aggregate
55.9%
56.4%
[▲ up 0.2% from main]
Python
64.7%
66.2%
[▲ up 12% from main]
Rust
37.1%
56%
[▼ down 0.2% from main]

➟ Additional coverage metrics may have been reported. See full coverage report ↗


Important

AI Review is run only on demand for your team. We're only showing results of static analysis review right now. To trigger AI Review, comment @deepsourcebot review on this thread.

let cfg;
let runtime = null;

export default async function init() {

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Parsing error: 'import' and 'export' may appear only with 'sourceType: module'


Found non-compliant syntax. Confirm that there are no syntax errors before committing your code to a version control system.

@codacy-production

Copy link
Copy Markdown

Not up to standards ⛔

🔴 Issues 29 high · 28 medium · 8 minor

Alerts:
⚠ 65 issues (≤ 0 issues of at least minor severity)

Results:
65 new issues

Category Results
Compatibility 23 medium
12 high
BestPractice 2 medium
Documentation 7 minor
ErrorProne 17 high
Complexity 1 medium
1 minor
Performance 2 medium

View in Codacy

🟢 Metrics 127 complexity · 6 duplication

Metric Results
Complexity 127
Duplication 6

View in Codacy

AI Reviewer: first review requested successfully. AI can make mistakes. Always validate suggestions.

Run reviewer

TIP This summary will be updated as you push new changes.

@codecov

codecov Bot commented Jul 20, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 60.00000% with 78 lines in your changes missing coverage. Please review.
✅ All tests successful. No failed tests found.

Files with missing lines Patch % Lines
services/ws-modules/pyeye1/pyeye1/eye_detection.py 59.82% 47 Missing ⚠️
services/ws-modules/face-detection/src/lib.rs 25.00% 30 Missing ⚠️
libs/edge-toolkit/src/config.rs 87.50% 1 Missing ⚠️

📢 Thoughts on this report? Let us know!

@jayvdb
jayvdb marked this pull request as ready for review July 20, 2026 07:41

@codacy-production codacy-production Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull Request Overview

The pull request is currently not up to standards due to high-complexity code and broad exception handling. While the functional requirements for eye detection and RetinaFace landmark decoding appear fulfilled across the Python and Rust modules, the et_ws_pyeye1.js file is flagged as a high-risk file: it contains a 'God Function' with excessive complexity and no test coverage, making it difficult to maintain and verify.

Furthermore, there is significant logic duplication between the new pyeye1 and the existing pyface1 JavaScript adapters. These issues, combined with the lack of a PR description, suggest that refactoring and documentation are required before merging.

About this PR

  • The PR includes significant changes to maintenance comments in Cargo.toml which are unrelated to the primary feature of eye detection. Consider moving these to a separate maintenance task.
  • The PR description is empty. Please provide context for the extensive changes across the eye detection and face detection modules to assist with future maintenance.

Test suggestions

  • Verify eye bounding box calculation from normalized landmarks in pyeye1
  • Verify RetinaFace landmark decoding and variance application in Rust face-detection
  • Verify RetinaFace landmark decoding and labelling in Python pyface1
  • Verify resolution of scoped npm package directories in the edge-toolkit library
  • Verify pyeye1 module availability in the web runner environment

TIP Improve review quality by adding custom instructions
TIP How was this review? Give us feedback

set_status(status_text(results))
render(results_json(results))
send_event(client_event_json(event_payload(results, capture["width"], capture["height"])))
except Exception as exc:

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔴 HIGH RISK

Catching a broad Exception can hide unexpected bugs. Specify the exact exceptions you expect to handle from the JSON parsing or the infer() callback. Replace the broad except Exception as exc: on line 81 with specific exceptions such as json.JSONDecodeError or KeyError.

let cfg;
let runtime = null;

export default async function init() {

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 MEDIUM RISK

This file shares a high amount of duplicate logic with pyface1/pkg/et_ws_pyface1.js. Common functions like init, cleanup, and log are identical. Propose moving these to a shared utility module to simplify maintenance and ensure consistent behavior across Pyodide adapters.

export const is_running = () => runtime !== null;
export const start = () => run();

export async function run() {

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 MEDIUM RISK

The run method is handling too many responsibilities (connection, camera, model loading, and loop management). This function exceeds complexity thresholds and currently lacks test coverage. Refactor this into distinct lifecycle stages (e.g., initConnection, setupMediaDevices, loadFaceLandmarker) to improve readability and isolation.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 3

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In @.mise/config.toml:
- Line 1406: Remove the duplicated TOML table headers, retaining only one
[tasks.fetch-eye1-rclone] declaration in .mise/config.toml and one
[asset."face_landmarker.task"] declaration in config/upstream-cache/data.toml;
preserve the settings under each table.

In `@services/ws-modules/face-detection/src/lib.rs`:
- Around line 556-568: Add a justified #[expect(clippy::single_call_fn, reason =
"...")] attribute to the private decode_retinaface_landmarks helper, explaining
that it remains separate for testability. Keep the helper’s implementation and
production caller unchanged.

In `@services/ws-web-runner/tests/modules.rs`:
- Line 153: Update the test setup around the pyeye1 case in the relevant test
function so a missing Python mise environment fails explicitly instead of
returning successfully. Ensure the pyeye1 case always executes with its required
environment and is not silently skipped or excluded from coverage.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: b15ee848-d4c5-41be-a80d-85339cff4ae5

📥 Commits

Reviewing files that changed from the base of the PR and between a919903 and 20e722f.

⛔ Files ignored due to path filters (1)
  • uv.lock is excluded by !**/*.lock
📒 Files selected for processing (27)
  • .mise/config.coverage.toml
  • .mise/config.js.toml
  • .mise/config.maint.toml
  • .mise/config.python.toml
  • .mise/config.toml
  • CLAUDE.md
  • Cargo.toml
  • config/coverage.toml
  • config/pyrefly.toml
  • config/upstream-cache/data.toml
  • data/model-modules/.gitignore
  • data/model-modules/model-eye1/pkg/package.json
  • libs/edge-toolkit/src/config.rs
  • libs/edge-toolkit/tests/npm_mod.rs
  • pyproject.toml
  • services/ws-modules/face-detection/src/lib.rs
  • services/ws-modules/face-detection/src/test_face_detection.rs
  • services/ws-modules/pyeye1/pkg/.gitignore
  • services/ws-modules/pyeye1/pkg/et_ws_pyeye1.js
  • services/ws-modules/pyeye1/pyeye1/__init__.py
  • services/ws-modules/pyeye1/pyeye1/eye_detection.py
  • services/ws-modules/pyeye1/pyproject.toml
  • services/ws-modules/pyeye1/tests/test_eye_detection.py
  • services/ws-modules/pyface1/pkg/et_ws_pyface1.js
  • services/ws-modules/pyface1/pyface1/face_detection.py
  • services/ws-modules/pyface1/tests/test_face_detection.py
  • services/ws-web-runner/tests/modules.rs

Comment thread .mise/config.toml
"""
shell = "bash -euo pipefail -c"

[tasks.fetch-eye1-rclone]

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🩺 Stability & Availability | 🟠 Major | ⚡ Quick win

Remove duplicated TOML table headers. Each repeated standard table declaration makes its file invalid TOML and blocks the associated mise or upstream-cache workflow.

  • .mise/config.toml#L1406-L1406: retain only one [tasks.fetch-eye1-rclone] header.
  • config/upstream-cache/data.toml#L30-L30: retain only one [asset."face_landmarker.task"] header.
📍 Affects 2 files
  • .mise/config.toml#L1406-L1406 (this comment)
  • config/upstream-cache/data.toml#L30-L30
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In @.mise/config.toml at line 1406, Remove the duplicated TOML table headers,
retaining only one [tasks.fetch-eye1-rclone] declaration in .mise/config.toml
and one [asset."face_landmarker.task"] declaration in
config/upstream-cache/data.toml; preserve the settings under each table.

Comment on lines +556 to +568
// Decode the five RetinaFace landmark points from their offsets and prior box. Each point is a plain centre
// offset (no width/height exponent), so decode is simpler than the box: the normalised coordinate is the
// prior centre plus the offset scaled by the first variance and the prior size.
fn decode_retinaface_landmarks(landm: &[f32], prior: [f64; 4]) -> [[f64; 2]; 5] {
let mut points = [[0.0_f64; 2]; 5];
for (point, slot) in points.iter_mut().enumerate() {
*slot = [
prior[0] + f64::from(landm[point * 2]) * RETINAFACE_VARIANCES[0] * prior[2],
prior[1] + f64::from(landm[point * 2 + 1]) * RETINAFACE_VARIANCES[0] * prior[3],
];
}
points
}

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

📐 Maintainability & Code Quality | 🟡 Minor | ⚡ Quick win

Annotate the intentional single-call helper.

decode_retinaface_landmarks has one production caller at Line 483. Keep it testable, but add a justified #[expect(clippy::single_call_fn, reason = "...")] instead of leaving the lint unacknowledged. As per coding guidelines, intentional single-call private functions must use a justified #[expect(...)].

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@services/ws-modules/face-detection/src/lib.rs` around lines 556 - 568, Add a
justified #[expect(clippy::single_call_fn, reason = "...")] attribute to the
private decode_retinaface_landmarks helper, explaining that it remains separate
for testability. Keep the helper’s implementation and production caller
unchanged.

Source: Coding guidelines

#[case::speech_recognition("et-ws-speech-recognition", Language::Rust)]
#[case::video1("et-ws-video1", Language::Rust)]
#[case::pyface1("et-ws-pyface1", Language::Python)]
#[case::pyeye1("et-ws-pyeye1", Language::Python)]

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🎯 Functional Correctness | 🟠 Major | ⚡ Quick win

Do not add pyeye1 to a silently skipped test.

When the Python mise environment is absent, this new case returns successfully at lines 155-160 without running or collecting coverage. Make the missing environment fail loudly, or ensure this test always runs with its required environment.

As per coding guidelines, "Do not skip, ignore, platform-disable, feature-disable, or early-return from tests without explicit user approval."

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@services/ws-web-runner/tests/modules.rs` at line 153, Update the test setup
around the pyeye1 case in the relevant test function so a missing Python mise
environment fails explicitly instead of returning successfully. Ensure the
pyeye1 case always executes with its required environment and is not silently
skipped or excluded from coverage.

Source: Coding guidelines

@jayvdb jayvdb changed the title Eye detection Eye tracking Jul 20, 2026
@jayvdb
jayvdb requested a review from pierre-tenedero July 20, 2026 07:56
@jayvdb
jayvdb merged commit 15af375 into main Jul 20, 2026
43 of 48 checks passed
@jayvdb
jayvdb deleted the eye branch July 20, 2026 08:01
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants