Skip to content

Prof-GP/Volatility_Parsers_SOF-ELK

BranchesTags

Repository files navigation

Volatility to SOF-ELK Parsers

Memory forensics parsers for ingesting Volatility3 output into SOF-ELK using the Filebeat → Logstash → Elasticsearch architecture.

Overview

This toolkit provides Filebeat configuration, Logstash filters, and Python preprocessing scripts to parse and index Volatility3 memory forensics output into Elasticsearch via SOF-ELK.

Supported Volatility3 Plugins (6 Total)

Tier 1: Process Enumeration (3 plugins)

  • windows.pslist - Lists active processes by walking the process list
  • windows.pstree - Shows process hierarchy in a tree structure
  • windows.psscan - Scans physical memory for process structures (detects hidden processes)

Tier 2: Network & Execution Analysis (3 plugins)

  • windows.netscan - Network connections (IPv4/IPv6) with GeoIP enrichment
  • windows.cmdline - Command line arguments with attack pattern detection
  • windows.netstat - Network statistics (netstat-style enumeration)

Features

Core Capabilities

  • ✅ Filebeat integration for SOF-ELK-compatible ingestion
  • ✅ Logstash filters for automated parsing and enrichment
  • ✅ Python preprocessing for data enrichment and transformation
  • ✅ NDJSON conversion for Logstash compatibility
  • ✅ Batch processing for entire memory dump analysis

Process Analysis

  • ✅ Hidden process detection (pslist vs psscan comparison)
  • ✅ Process classification (system, service, user, browser)
  • ✅ Suspicious indicator tagging (orphaned, no threads, known tools)
  • ✅ Hierarchical tree flattening for better indexing

Network Analysis

  • ✅ Connection classification (listening, established, external)
  • ✅ GeoIP enrichment for foreign IP addresses
  • ✅ Suspicious port detection (backdoor ports, high ports)
  • ✅ Service type identification (HTTP, HTTPS, RDP, SMB)

Execution Analysis

  • ✅ PowerShell attack detection (encoded commands, download cradles)
  • ✅ LOLBin identification (certutil, bitsadmin, mshta)
  • ✅ Lateral movement detection (wmic, psexec)
  • ✅ Credential dumping detection (lsass, ntds.dit)
  • ✅ Base64 pattern matching
  • ✅ Command line length analysis

Data Management

  • ✅ Case metadata tracking
  • ✅ Daily index rotation in Elasticsearch
  • ✅ Multi-case support

Architecture

┌─────────────────────────────────────────────────────────────┐
│                  SOF-ELK Data Flow                          │
└─────────────────────────────────────────────────────────────┘

Volatility3 → Python Parsers → NDJSON Files
                                     ↓
                            /logstash/volatility/
                                     ↓
                   Filebeat (monitors directories)
                                     ↓
                   Logstash:5044 (Beats input)
                                     ↓
                    Filters (enrichment)
                                     ↓
                   Elasticsearch (indices)
                                     ↓
                    Kibana (visualization)

Repository Structure

volatility-to-sof-elk/                     # This repository
├── lib/
│   ├── filebeat_inputs/
│   │   └── volatility.yml               # → Deploy to: /usr/local/sof-elk/lib/filebeat_inputs/
│   └── README.md                         # Library documentation
├── logstash/
│   └── conf.d/
│       ├── 1500-preprocess-volatility.conf  # → Deploy to: /usr/local/sof-elk/configfiles/
│       ├── 6000-volatility-input.conf       # Documentation only (no Logstash input needed)
│       ├── 6001-volatility-pslist.conf      # → Deploy to: /usr/local/sof-elk/configfiles/
│       ├── 6002-volatility-pstree.conf      # → Deploy to: /usr/local/sof-elk/configfiles/
│       ├── 6003-volatility-psscan.conf      # → Deploy to: /usr/local/sof-elk/configfiles/
│       ├── 6010-volatility-netscan.conf     # → Deploy to: /usr/local/sof-elk/configfiles/
│       ├── 6011-volatility-cmdline.conf     # → Deploy to: /usr/local/sof-elk/configfiles/
│       └── 6012-volatility-netstat.conf     # → Deploy to: /usr/local/sof-elk/configfiles/
├── python_parsers/
│   ├── volatility_to_elk.py             # Python enrichment script (v2.0 - 6 plugins)
│   ├── convert_to_ndjson.py             # Simple JSON → NDJSON converter
│   └── batch_process.py                 # Batch processor with enrichment
├── examples/                             # Example NDJSON outputs (not for deployment)
├── docs/
│   ├── SOF-ELK_INSTALLATION.md          # Detailed installation guide
│   └── QUICKSTART.md                    # Quick reference guide
├── DEPLOYMENT_GUIDE.md                  # ⭐ Complete step-by-step deployment guide
├── SOF-ELK_INTEGRATION.md               # Integration architecture guide
├── ARCHITECTURE_CORRECTION.md           # Filebeat architecture explanation
├── PLUGIN_RECOMMENDATIONS.md            # Analysis of all 19 Volatility plugins
├── TIER2_COMPLETE.md                    # Tier 2 implementation summary
├── CHEAT_SHEET.md                       # Command reference
├── SUMMARY.md                           # Project summary
├── .gitignore                           # Git ignore rules
├── LICENSE                              # MIT License
└── README.md                            # This file

Deployment Target Paths (SOF-ELK VM)

Source File Target Path on SOF-ELK
lib/filebeat_inputs/volatility.yml /usr/local/sof-elk/lib/filebeat_inputs/volatility.yml
logstash/conf.d/1500-preprocess-volatility.conf /usr/local/sof-elk/configfiles/1500-preprocess-volatility.conf
logstash/conf.d/6001-volatility-pslist.conf /usr/local/sof-elk/configfiles/6001-volatility-pslist.conf
logstash/conf.d/6002-volatility-pstree.conf /usr/local/sof-elk/configfiles/6002-volatility-pstree.conf
logstash/conf.d/6003-volatility-psscan.conf /usr/local/sof-elk/configfiles/6003-volatility-psscan.conf
logstash/conf.d/6010-volatility-netscan.conf /usr/local/sof-elk/configfiles/6010-volatility-netscan.conf
logstash/conf.d/6011-volatility-cmdline.conf /usr/local/sof-elk/configfiles/6011-volatility-cmdline.conf
logstash/conf.d/6012-volatility-netstat.conf /usr/local/sof-elk/configfiles/6012-volatility-netstat.conf

Note: Logstash configs require symlinks from /etc/logstash/conf.d/ → /usr/local/sof-elk/configfiles/

Data File Paths (SOF-ELK VM)

NDJSON files should be placed in:

/logstash/volatility/
├── pslist/*.json or *.ndjson
├── pstree/*.json or *.ndjson
├── psscan/*.json or *.ndjson
├── netscan/*.json or *.ndjson
├── cmdline/*.json or *.ndjson
└── netstat/*.json or *.ndjson

Quick Start

Prerequisites

1. Set Up SOF-ELK

⭐ For complete step-by-step instructions, see DEPLOYMENT_GUIDE.md

On the SOF-ELK VM:

# 1. Create data directories
sudo mkdir -p /logstash/volatility/{pslist,pstree,psscan,netscan,cmdline,netstat}
sudo chown -R root:root /logstash/volatility/
sudo chmod -R 755 /logstash/volatility/

# 2. Copy and install Logstash configurations
scp logstash/conf.d/15*.conf logstash/conf.d/60*.conf user@sof-elk:/tmp/
sudo cp /tmp/1500-preprocess-volatility.conf /usr/local/sof-elk/configfiles/
sudo cp /tmp/60*-volatility-*.conf /usr/local/sof-elk/configfiles/

# 3. Create symlinks
cd /etc/logstash/conf.d/
sudo ln -s /usr/local/sof-elk/configfiles/1500-preprocess-volatility.conf .
sudo ln -s /usr/local/sof-elk/configfiles/6001-volatility-pslist.conf .
sudo ln -s /usr/local/sof-elk/configfiles/6002-volatility-pstree.conf .
sudo ln -s /usr/local/sof-elk/configfiles/6003-volatility-psscan.conf .
sudo ln -s /usr/local/sof-elk/configfiles/6010-volatility-netscan.conf .
sudo ln -s /usr/local/sof-elk/configfiles/6011-volatility-cmdline.conf .
sudo ln -s /usr/local/sof-elk/configfiles/6012-volatility-netstat.conf .

# 4. Install Filebeat input
scp lib/filebeat_inputs/volatility.yml user@sof-elk:/tmp/
sudo mv /tmp/volatility.yml /usr/local/sof-elk/lib/filebeat_inputs/
sudo chown root:root /usr/local/sof-elk/lib/filebeat_inputs/volatility.yml
sudo chmod 644 /usr/local/sof-elk/lib/filebeat_inputs/volatility.yml

# 5. Restart services
sudo systemctl restart filebeat
sudo systemctl restart logstash

See DEPLOYMENT_GUIDE.md for detailed deployment instructions.

2. Run Volatility3

Extract all plugin data from your memory dump:

vol3 -f memory.dmp windows.pslist --output-format json > pslist.json
vol3 -f memory.dmp windows.pstree --output-format json > pstree.json
vol3 -f memory.dmp windows.psscan --output-format json > psscan.json
vol3 -f memory.dmp windows.netscan --output-format json > netscan.json
vol3 -f memory.dmp windows.cmdline --output-format json > cmdline.json
vol3 -f memory.dmp windows.netstat --output-format json > netstat.json

3. Process and Enrich Data

cd python_parsers

# Batch process all plugins at once
python batch_process.py \
    --input-dir /path/to/volatility/output \
    --output-dir /logstash/volatility \
    --case "Investigation-2025-001" \
    --image "memory.dmp"

Output:

Processing pslist: windows_pslist_PsList.json
  → 171 processes → windows_pslist_PsList_enriched.ndjson
Processing netscan: windows_netscan_NetScan.json
  → 164 connections → windows_netscan_NetScan_enriched.ndjson
Processing cmdline: windows_cmdline_CmdLine.json
  → 171 command lines → windows_cmdline_CmdLine_enriched.ndjson
...

=== Processing Complete ===
Output directory: /logstash/volatility

4. Transfer to SOF-ELK (if not running on SOF-ELK)

# Transfer NDJSON files to SOF-ELK
scp -r /path/to/output/* user@sof-elk:/logstash/volatility/

5. Verify Ingestion

# On SOF-ELK: Check Filebeat is harvesting
sudo tail -f /var/log/filebeat/filebeat.log | grep volatility

# Check Elasticsearch indices
curl -s localhost:9200/_cat/indices/volatility-* | sort

# Query data
curl -s localhost:9200/volatility-netscan-*/_count

6. Explore in Kibana

http://<sof-elk-ip>:5601

1. Create index pattern: volatility-*
2. Go to Discover
3. Search for suspicious indicators

Installation

For complete installation instructions, see SOF-ELK_INSTALLATION.md.

Key points:

  • SOF-ELK uses Filebeat (not Logstash file input) to monitor directories
  • Files must be at least 1,024 bytes (Filebeat 9+ requirement)
  • Default directory: /logstash/volatility/ (SOF-ELK standard)
  • Filebeat sends to Logstash on port 5044 (Beats input)

Elasticsearch Indices

Data is indexed into daily-rotated indices:

Index Pattern Plugin Typical Docs/Day
volatility-pslist-YYYY.MM.DD windows.pslist 150-200
volatility-pstree-YYYY.MM.DD windows.pstree 150-200
volatility-psscan-YYYY.MM.DD windows.psscan 180-220
volatility-netscan-YYYY.MM.DD windows.netscan 100-300
volatility-cmdline-YYYY.MM.DD windows.cmdline 150-200
volatility-netstat-YYYY.MM.DD windows.netstat 50-100

Usage Examples

Detect PowerShell Attacks

# Encoded PowerShell commands
suspicious_indicators:"powershell_encoded"

# Download cradles
cmdline_analysis.has_download:true AND process_type:"powershell"

# Execution policy bypass
suspicious_indicators:"execution_policy_bypass"

Find External Connections

# External connections
connection_scope:"external"

# Suspicious ports
tags:"suspicious_port"

# System process making external connections
suspicious_indicators:"system_process_external_connection"

Identify LOLBin Abuse

# Any LOLBin usage
suspicious_indicators:lolbin*

# Specific tools
suspicious_indicators:"lolbin_certutil" OR suspicious_indicators:"lolbin_bitsadmin"

Detect Lateral Movement

# WMI lateral movement
suspicious_indicators:"wmi_lateral_movement"

# PSExec usage
process_name:"psexec*" OR suspicious_indicators:"lateral_movement"

Find Hidden Processes

# Processes found by psscan but not pslist
_index:volatility-psscan-* AND NOT _index:volatility-pslist-*

Correlate Process + Network + Command Line

# Suspicious PowerShell with external connections
process_name:"powershell.exe" AND
connection_scope:"external" AND
suspicious_indicators:"powershell_encoded"

Field Reference

Process Fields

Field Type Description
process_id integer Process ID (PID)
parent_process_id integer Parent process ID (PPID)
process_name string Image file name
process_path string Full executable path
process_create_time date Process creation timestamp
process_status string "running" or "exited"
process_classification string Process type (system, service, user, browser)
suspicious_indicators array List of suspicious indicators

Network Fields

Field Type Description
local_address ip Local IP address
local_port integer Local port number
foreign_address ip Remote IP address
foreign_port integer Remote port number
network_protocol string Protocol (TCPv4, UDPv6, etc.)
connection_state string Connection state (LISTENING, ESTABLISHED)
connection_scope string "internal" or "external"
service_type string Service classification (http, https, rdp, smb)
foreign_geo.* object GeoIP data (city, country, location)

Command Line Fields

Field Type Description
command_line string Full command line
cmdline_length integer Length of command line
process_type string Process type (powershell, cmd, script_host)
cmdline_analysis.has_powershell boolean Contains PowerShell
cmdline_analysis.has_encoded boolean Contains encoded commands
cmdline_analysis.has_download boolean Contains download patterns
cmdline_analysis.has_base64 boolean Contains Base64

Suspicious Indicators

Process Indicators

  • orphaned_process - PPID 0 (not System process)
  • terminated_process - Exited process still in memory
  • no_threads - Zero threads (possible process hollowing)
  • known_tool - Known security/hacking tool

Network Indicators

  • high_port - Port > 49152
  • suspicious_port - Known malware port (4444, 31337, etc.)
  • system_process_external_connection - System process with external connection

Command Line Indicators

  • powershell_encoded - Encoded PowerShell command
  • powershell_download - PowerShell download cradle
  • execution_policy_bypass - Execution policy bypass
  • hidden_window - WindowStyle Hidden
  • attack_tool - Known attack tool (mimikatz, psexec)
  • lolbin_* - LOLBin abuse (certutil, bitsadmin, mshta)
  • lateral_movement - Lateral movement indicators
  • wmi_lateral_movement - WMI-based lateral movement
  • credential_dumping - Credential dumping patterns
  • base64_pattern - Contains Base64 data
  • long_cmdline - Unusually long command line (>500 chars)

Advanced Analysis

Cross-Plugin Correlation

Find processes making external connections with suspicious command lines:

_index:volatility-cmdline-* AND suspicious_indicators:*
| join process_id
| _index:volatility-netscan-* AND connection_scope:"external"

Detect C2 beaconing:

connection_scope:"external" AND
foreign_port:(4444 OR 8443 OR 443) AND
process_name:("powershell.exe" OR "rundll32.exe" OR "regsvr32.exe")

Identify credential dumping with network exfiltration:

suspicious_indicators:"credential_dumping" AND
_index:volatility-netscan-* AND
connection_scope:"external"

Detection Use Cases

  1. C2 Beaconing - External connections from suspicious processes
  2. LOLBin Abuse - Living-off-the-land binary usage
  3. PowerShell Attacks - Encoded commands, download cradles, execution policy bypass
  4. Credential Dumping - LSASS access, ntds.dit extraction
  5. Lateral Movement - WMI, PSExec, remote execution
  6. Process Injection - Hidden processes, process hollowing
  7. Persistence - Unusual parent-child relationships
  8. Data Exfiltration - Large external data transfers

Troubleshooting

Filebeat Not Harvesting Files

Problem: Files in /logstash/volatility/ but not ingested

Solutions:

# Check file size (must be ≥1024 bytes)
ls -lh /logstash/volatility/netscan/

# Check permissions
sudo chown filebeat:filebeat /logstash/volatility/*/*.ndjson

# Check Filebeat logs
sudo tail -f /var/log/filebeat/filebeat.log

# Restart Filebeat
sudo systemctl restart filebeat

No Elasticsearch Indices

Problem: Filebeat harvesting but no indices created

Solutions:

# Check Logstash is running
sudo systemctl status logstash

# Check Logstash logs
sudo tail -100 /var/log/logstash/logstash-plain.log

# Test Logstash config
sudo -u logstash /usr/share/logstash/bin/logstash \
  --config.test_and_exit \
  --path.config=/etc/logstash/conf.d/

NDJSON Parsing Errors

Problem: Events ingested but fields missing

Solutions:

# Validate NDJSON format (each line = valid JSON)
head -1 /logstash/volatility/netscan/test.ndjson | jq .

# Check for parsing errors
sudo grep "_jsonparsefailure" /var/log/logstash/logstash-plain.log

Performance Tuning

For large memory dumps (>16GB):

# Increase Logstash heap
sudo nano /etc/logstash/jvm.options
# Change: -Xms1g → -Xms4g
# Change: -Xmx1g → -Xmx4g

# Batch file transfers (one plugin at a time)
scp netscan/*.ndjson sof-elk:/logstash/volatility/netscan/
# Wait for ingestion...
scp cmdline/*.ndjson sof-elk:/logstash/volatility/cmdline/

Documentation

Roadmap

Implemented (v2.0)

  • ✅ Tier 1: Process enumeration (pslist, pstree, psscan)
  • ✅ Tier 2: Network & execution (netscan, cmdline, netstat)
  • ✅ Filebeat integration for SOF-ELK compatibility
  • ✅ 50+ suspicious pattern detections
  • ✅ GeoIP enrichment

Planned (v3.0)

  • 🔲 Tier 3: Malware detection (malfind, svcscan)
  • 🔲 Tier 4: DLL analysis (dlllist, modules - filtered)
  • 🔲 Kibana dashboards and visualizations
  • 🔲 Automated report generation
  • 🔲 Timeline correlation with other artifacts

Contributing

Contributions welcome! Areas for enhancement:

  • Additional Volatility plugin support
  • Machine learning for anomaly detection
  • Kibana dashboard templates
  • Integration with other forensic tools (Plaso, Autopsy)
  • Automated hunting playbooks

License

MIT License - See LICENSE file for details

References

Version

Memory Forensics Parser v2.0

Changelog

v2.0 - Tier 2 Implementation (Current)

  • ✅ Added Filebeat integration for SOF-ELK compatibility
  • ✅ Added windows.netscan parser (network connections + GeoIP)
  • ✅ Added windows.cmdline parser (50+ attack pattern detections)
  • ✅ Added windows.netstat parser (netstat-style enumeration)
  • ✅ Updated batch_process.py for 6 plugins
  • ✅ Corrected architecture documentation
  • ✅ Comprehensive installation guide

v1.0 - Initial Release

  • ✅ Support for windows.pslist, windows.pstree, windows.psscan
  • ✅ Hidden process detection
  • ✅ Process classification and suspicious indicators
  • ✅ Batch processing capabilities

Built with ❤️ for the memory forensics community

About

Memory forensics parsers for ingesting Volatility3 output into SOF-ELK with automated enrichment and threat detection

Topics

dfir threat-hunting elk-stack memory-forensics volatility3 incident-reponse sof-elk

Resources

Readme

License

MIT license
Activity

Stars

1 star

Watchers

0 watching

Forks

0 forks
Report repository

Releases

No releases published

Packages

 
 
 

Contributors

Languages