Touch classifier - #137
Open
danielwallner wants to merge 8 commits into
Open
Conversation
Route both gesture recognition and force-click conversion through a single stateful TouchClassifier so a resting/moving thumb or a fragmenting palm no longer inflates the finger count. Classification combines size/shape evidence with differential-motion evidence (a contact that stays still or moves against the digit cluster while the others drag), a palm-satellite rule keyed off a heel-sized contact, and sticky per-touch roles. The recognizer forgives an extra ambiguous/stationary contact during an active drag (thumb landing to click) and confirms four-finger cancellation over a few frames so a single classification flicker cannot drop a drag. Force-click conversion and the drag centroid now use the classified digit count, not the raw contact count. Thresholds live in a per-user TouchClassifierCalibration (defaults measured from real trackpad readings) that can be derived from labelled recording sessions and persisted to disk. Includes the session/recording model used for calibration and debug capture, plus unit tests and replay fixtures recorded on real hardware (palm rest, resting thumb, three-finger drag with thumb click) asserting MiddleDrag never falsely starts or cancels.
Advanced menu item opening a window that draws each active contact as an ellipse (sized by major/minor axis, coloured by classified role) with motion trails, the digit-cluster centroid, live raw/digit/incidental/uncertain counts, and recognizer/event state. Supports labelled session recording with JSON and CSV export, used to capture real trackpad data for tuning and calibration.
Guided wizard (Advanced > Palm Rejection) that walks through natural hand postures, records a few seconds per step, validates each sample, and derives per-user classifier thresholds that persist to disk and load at launch. Adds "Ignore Incidental Contacts" to toggle the classifier-based filtering and "Reset Touch Calibration" to revert to defaults.
Two opt-in Advanced toggles for compatibility with certain apps: "Preserve Modifier Keys During Drag" forwards the live keyboard modifier state on the synthetic middle-button events instead of always reporting none, and "Allow Left Click During Drag" passes a real left click (e.g. a thumb press) through to the app while a middle-drag is active instead of suppressing it. Both default off to preserve existing behavior.
…ssifier The shared touch classifier now handles thumb/palm rejection by contact shape/size and motion, making the two crude pre-filters redundant: the fixed bottom-of-trackpad exclusion zone and the zTotal large-contact filter. Removes their config fields, preferences, menu items and actions, and the classifier's passesLegacyFilters pre-pass, along with the tests that covered them. Require Modifier Key is kept — it gates gesture activation rather than rejecting palms.
Lets repeated three-finger swipes accumulate into one continuous middle-drag - e.g. rotating a CAD view further than a single swipe's travel allows - instead of each lift ending the drag outright. Only active when allowLeftClickDuringDrag is on and the real left mouse button stays physically held across the lift. The synthetic middle button is kept down in MouseEventGenerator, but isActivelyDragging still drops like a normal end so the rest of the system (cursor movement, other clicks) stays fully responsive during the pause between swipes; only the mouse generator's internal button state is preserved. A 4-finger Mission Control gesture still cancels unconditionally. The stuck-drag watchdog keeps running throughout with an extended timeout while paused, so a legitimate multi-second gap isn't force-released, but a truly stuck hold still self-heals.
Revoking (or fully removing) MiddleDrag's Accessibility entry while its event tap is alive left the tap registered but broken: a session tap created with .defaultTap (active filtering) that loses trust keeps swallowing every mouse/keyboard event system-wide until the process is killed, rather than failing cleanly (see Apple Developer Forums thread 735204, and deskflow#9562 for the same symptom in another tool). The previous code also blindly re-enabled the tap on any disable reason, including tapDisabledByUserInput - the signal macOS sends specifically when the user revokes trust - actively fighting that action. Toggling the permission off reliably delivers tapDisabledByUserInput, so that path now tears the tap down for real (CFMachPortInvalidate + run loop source removal) instead of just skipping the re-enable. Fully removing the entry (vs. unchecking it) turned out not to deliver that callback at all, and left CGEvent.tapIsEnabled on the existing tap reporting true even though it had gone dead - so a second, independent 1-second timer now probes by actually creating (and discarding) a fresh tap, which macOS reliably refuses when trust is gone, regardless of what the existing tap or AXIsProcessTrusted() report. Also fixes a related gap where MultitouchManager.stop() never told the menu bar to refresh, leaving the "enabled" icon stale after a silent revocation until some unrelated trigger (device connect, polling timeout, manual toggle) happened to run.
![sentry[bot]](https://avatars.githubusercontent.com/in/12637?s=60&v=4){kind=small}
isRealLeftButtonDown and isSustainingDragForLeftClick were plain vars read and written from both processEvent (event tap callback, main thread) and the GestureRecognizerDelegate callbacks (gestureQueue) with no synchronization - a real cross-thread race, unlike currentFingerCount and lastForceClickTime which already use this lock pattern in the same file. processEvent's left-mouse-up handler and gestureRecognizerDidBeginDragging both do a check-then-act on isSustainingDragForLeftClick, so without synchronization both could observe true concurrently and double-handle the same sustained hold - one ending the drag while the other simultaneously tries to resume it. Added claimSustainingDragForLeftClick() to make that check-and-clear atomic under one lock, not just each property access individually.
Comment on lines
1307
to
+1317
| self?.isActivelyDragging = true | ||
| } | ||
|
|
||
| if claimSustainingDragForLeftClick() { | ||
| // The middle button was never released after the last three-finger | ||
| // lift (see gestureRecognizerDidEndDragging) - resume in place rather | ||
| // than pressing it down again, which would look like a fresh drag to | ||
| // the target app and cancel the in-progress CAD rotation. | ||
| mouseGenerator.resumeDragKeepingButtonHeld() | ||
| return | ||
| } |
Contributor
There was a problem hiding this comment.
Bug: A race condition when resuming a drag causes left-clicks to be incorrectly suppressed because isActivelyDragging is set asynchronously, creating a window where the flag is not yet true.
Severity: HIGH
Suggested Fix
To eliminate the race condition, update the isActivelyDragging state synchronously using DispatchQueue.main.sync instead of async. This ensures the flag is set before any other code can check its value, preventing left-clicks from being incorrectly suppressed when a drag is resumed.
Prompt for AI Agent
Review the code at the location below. A potential bug has been identified by an AI
agent. Verify if this is a real issue. If it is, propose a fix; if not, explain why it's
not valid.
Location: MiddleDrag/Managers/MultitouchManager.swift#L1307-L1317
Potential issue: In the `gestureRecognizerDidBeginDragging` method, the
`isActivelyDragging` flag is set to `true` within a `DispatchQueue.main.async` block.
However, subsequent logic, such as `resumeDragKeepingButtonHeld`, executes synchronously
on a separate `gestureQueue` before the async block completes. This creates a race
condition. If a left-click event is processed on the main thread during this brief
window, it will read `isActivelyDragging` as `false`. As a result, the pass-through
check for the click will fail, causing it to be incorrectly suppressed instead of being
passed to the target application, which breaks the intended workflow for the 'allow left
click during drag' feature.
Author
|
I have pushed a fix for the data race and also created a new branch (touch-classifier-rebased) with the commit that introduced this fixed. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
These commits mainly address the shortcoming that MiddleDrag did not support resting your thumb on the touchpad.
This made thumb + two finger scroll trigger a middle drag.
I find that using the left thumb for clicks makes the right hand a lot more relaxed and separating force (click) from movement also increases the precision.
Without this fix MiddleDrag is unusable for anyone that is using the thumb to click.
An additional benefit is that this also makes it possible to support middle drag + left click that e.g. FreeCAD uses to orbit.
With these changes it is possible to pan with three fingers and then click with the thumb to add a left click to rotate/orbit.
There is also a touch debug window and calibration in these commits.
Do note that I am not a Swift programmer and this was mostly vibe coded.
I have tested this on both an M4 Air and a 2013 MacBook Pro.
With these fixes FreeCAD is fully useable with the touchpad. (With Allow Left Click During Drag enabled.)
Type of Change
Related Issues
Testing Performed
Checklist
Code Coverage
Screenshots / Recordings
Additional Notes