Add windows.malware.apihooks plugin for Windows API hook detection

[Mihir-Choudhary]

Summary

This PR adds a new Windows malware-analysis plugin, windows.malware.apihooks, that detects several API tampering techniques from memory:

• inline hooks
• semantic patch bypasses
• IAT hooks
• EAT hooks
• SSDT hooks

The plugin is designed as a combined triage workflow rather than a single-mechanism detector. It also includes scoring and false-positive suppression aimed at reducing benign Windows helper and forwarder noise in default output.

Highlights

• Inline hook detection with direct, indirect, register-based, RIP-relative, and push/ret patterns
• Semantic patch detection for short AMSI and ETW-style bypass stubs
• IAT hook detection using dereferenced live thunk values
• EAT hook detection via export RVA validation
• SSDT owner checks
• Confidence scoring with benign suppressions
• VAD-backed target evidence to distinguish mapped images, private executable regions, and missing or unbacked targets
• Buffered inline scoring to avoid traversal-order-dependent confidence
• --quick mode to skip inline disassembly for faster triage
• PID, PPID, Process, and ParentProcess in output

Tests

Added and updated tests include:

• focused detector and unit tests
• FP-reduction regressions
• large explicit corpus coverage

Verification run:

py -3 -m pytest -p no:cacheprovider --volatility vol.py test\plugins\windows\test_apihooks.py test\plugins\windows\test_apihooks_fp_reduction.py test\plugins\windows\test_apihooks_corpus.py
580 passed

Notes

• Output is buffered before rendering so the CLI does not print headers and partial rows before analysis completes.
• This improves UX but means --low on very noisy images can use more memory.

Related Work

Official Volatility 3 already includes adjacent functionality, but it is split across separate plugins. windows.iat provides visibility into imported API relationships, windows.etwpatch targets ETW patching, and draft PR #1818 (inlinehooks plugin & change to avpatch) explores inline-hook and patch-oriented detection.

The remaining gap is a unified API-hook triage workflow for Windows memory analysis. The windows.malware.apihooks plugin addresses that gap by bringing together inline-hook detection, semantic patch detection, IAT analysis, EAT analysis, SSDT analysis, confidence scoring, VAD-backed target evidence, and cross-process correlation in a single plugin.

[Mihir-Choudhary]

Hi, any views on this?