A professional-grade, modular Python framework designed for Digital Signal Processing (DSP), Statistical Feature Extraction, and High-Dimensional Data Analysis. This project provides a robust pipeline to transform raw temporal signals into meaningful feature vectors suitable for Machine Learning models.
The framework implements advanced analytical techniques to decompose and interpret raw signals:
- FFT (Fast Fourier Transform): Converts signals from the time domain to the frequency domain, enabling the identification of dominant frequency components and signal periodicity.
- Correlation Analysis: Quantifies relationships between signals using Pearson and Spearman coefficients, crucial for identifying multi-collinearity in feature sets.
- KDE (Kernel Density Estimation): A non-parametric way to estimate the Probability Density Function (PDF) of a random variable. It is essential when the underlying distribution of the data is unknown and cannot be modeled by simple parametric functions.
- PCA (Principal Component Analysis): Uses orthogonal transformation to convert a set of possibly correlated variables into a set of values of linearly uncorrelated variables called principal components. This reduces noise and simplifies complex datasets.
| Module | Purpose |
|---|---|
DSPAnalyzer.py |
Spectral analysis, FFT, and correlation metrics. |
KDEstimator.py |
Advanced PDF/CDF estimation using multiple backends (Scipy, Scikit-learn, Statsmodels). |
Features.py |
Parallelized extraction pipeline utilizing concurrent.futures. |
PCAnalyzer.py |
Data normalization and projection into Principal Components. |
- Python 3.8+
- Dependencies:
numpy,pandas,scipy,scikit-learn,statsmodels,matplotlib.
pip install numpy pandas scipy scikit-learn statsmodels matplotlibTo start using the framework, you can either import the necessary modules into your own Python scripts or execute the main.py entry point directly to run the default pipeline.
python main.pyThe Features_Extraction class is designed to handle parallel processing of your input signals using a thread pool, significantly reducing computation time when dealing with large datasets.
The framework follows a pipeline-based approach to transform raw data into actionable insights:
-
Input: Raw data is loaded and fed into the
Featuresmodule. -
Parallel Processing: Using
ThreadPoolExecutorinFeatures.py, statistical feature extraction is performed concurrently to maximize CPU utilization. -
DSP Analysis:
DSPAnalyzercomputes correlation indices and performs frequency-domain analysis via FFT (Fast Fourier Transform). -
Statistical Estimation:
KDEstimatormodels the underlying data distribution non-parametrically using Kernel Density Estimation (KDE). -
Reduction:
PCAnalyzerprojects extracted features into Principal Components, ready for visualization or predictive modeling via PCA (Principal Component Analysis).
To introduce new extraction logic, add a new static method to the Features_Extraction class in Features.py and register it within the ThreadPoolExecutor block:
@staticmethod
def extract_new_feature(data):
"""
Your custom feature extraction logic here.
Example: Calculate signal entropy or peak-to-peak amplitude.
"""
# Implementation
return feature_valueYou can extend the Signal_Analysis class in DSPAnalyzer.py to include domain-specific metrics, such as:
- Signal Energy: Integrating power spectral density to quantify the total signal strength.
- Spectral Entropy: Measuring the frequency distribution complexity to assess signal regularity.
- Wavelet Transforms: Implementing multi-resolution time-frequency analysis for non-stationary signal characterization.
Licensed under the BSD 2-Clause License. See the License file for details.
Developed to support complex data science workflows with efficiency and technical rigor.