This project implements and evaluates Support Vector Machine (SVM) classification using the Letter Recognition Dataset, based on the AAAI research paper:
Zhai, Z., Gu, B., Li, X., & Huang, H. (2020). Safe Sample Screening for Robust Support Vector Machine. Proceedings of AAAI, 34(04), 6981–6988.
The main objective is to evaluate different SVM models and analyze their performance in relation to the research paper.
SVM is used because it is the core machine-learning method investigated in the selected research paper. It is also well suited to the Letter Recognition dataset because it can construct effective decision boundaries for multi-class classification and supports both linear and nonlinear kernels.
This project therefore evaluates Linear, RBF, Polynomial, and Optimized RBF SVM models.
The Letter Recognition Dataset contains:
- 20,000 observations
- 16 numerical features
- 26 classes (A–Z)
- No missing values
The dataset also appears as a benchmark dataset in the selected AAAI research paper. The implementation confirmed the dataset shape as 20,000 × 17, consisting of 16 input features and one target column.
Dataset: https://archive.ics.uci.edu/dataset/59/letter+recognition
| Model | Purpose |
|---|---|
| Linear SVM | Baseline linear classifier |
| RBF SVM | Nonlinear classification |
| Polynomial SVM | Alternative nonlinear classifier |
| Optimized RBF SVM | RBF with hyperparameter tuning |
GridSearchCV was used to optimize the RBF SVM parameters.
The following results were obtained from the final experiments in the notebook.
| Model | Accuracy | Precision | Recall | F1 Score |
|---|---|---|---|---|
| Linear SVM | 85.25% | 85.47% | 85.25% | 85.28% |
| RBF SVM | 94.85% | 94.95% | 94.85% | 94.86% |
| Polynomial SVM | 89.08% | 91.39% | 89.08% | 89.74% |
| Optimized RBF SVM | 97.73% | 97.74% | 97.73% | 97.73% |
The Optimized RBF SVM achieved the best overall performance:
- Accuracy: 97.73%
- Precision: 97.74%
- Recall: 97.73%
- F1 Score: 97.73%
The notebook identifies the Optimized RBF SVM as the best-performing model.
The results show a clear improvement when moving from a linear to a nonlinear SVM.
The Linear SVM achieved 85.25% accuracy, providing the baseline performance.
The RBF SVM increased accuracy to 94.85%, showing that nonlinear decision boundaries are substantially more effective for this dataset.
The Polynomial SVM achieved 89.08%, performing better than the linear model but below the RBF model.
After hyperparameter optimization, the Optimized RBF SVM achieved 97.73%, an improvement of approximately 2.88 percentage points over the standard RBF SVM and 12.48 percentage points over the Linear SVM.
This demonstrates that both kernel selection and hyperparameter tuning have a significant effect on SVM classification performance.
The comparison must be interpreted carefully because the two experiments have different objectives.
| Aspect | Research Paper | Our Implementation |
|---|---|---|
| Main focus | Robust SVM efficiency | SVM classification performance |
| Dataset | Letter + other benchmarks | Letter Recognition |
| SVM type | Robust SVM (RSVM) | Conventional SVM |
| Main metrics | Training time & screening rate | Accuracy, Precision, Recall, F1 |
| Kernels | Linear & Gaussian | Linear, RBF & Polynomial |
| Optimization | CCCP + safe screening | GridSearchCV |
| Safe screening | Implemented | Not implemented |
The research paper focuses on safe sample screening for Robust SVM, with the goal of reducing the computational cost of optimization.
Our implementation instead focuses on predictive classification performance.
Therefore, the 97.73% accuracy obtained in this project should not be interpreted as a direct replication of the paper's numerical results. Instead, the project demonstrates the classification performance of conventional SVM on the same benchmark dataset.
The research paper proposes safe screening techniques that identify samples that can safely be removed from the optimization process.
The authors report that their proposed approach can reduce computational time by almost 50% in most experimental settings, while preserving the solution of the Robust SVM optimization problem.
This differs from the objective of this project:
The research paper focuses on computational efficiency, while this implementation focuses on classification performance.
The two approaches therefore provide complementary evaluations of SVM-based methods.
-
RBF SVM significantly outperformed Linear SVM, indicating that nonlinear decision boundaries are important for the Letter Recognition dataset.
-
Polynomial SVM improved over the linear baseline, but did not perform as well as the RBF kernel.
-
Hyperparameter optimization substantially improved RBF performance, increasing accuracy from 94.85% to 97.73%.
-
The Optimized RBF SVM was the best-performing model, achieving approximately 97.73% across accuracy, recall, and F1 score.
-
The research paper demonstrates a different type of improvement: reducing the computational cost of Robust SVM through safe sample screening.
The experiments demonstrate that SVM is highly effective for multi-class Letter Recognition classification.
Among the evaluated models, the Optimized RBF SVM achieved the strongest performance with 97.73% accuracy.
The comparison with the AAAI research paper highlights two different aspects of SVM research: classification performance in this implementation and computational efficiency in the research paper.
The project therefore provides a practical implementation of SVM while demonstrating how kernel selection and hyperparameter optimization can significantly improve classification results.
SVM-AAAI-Implementation/
│
├── SVM.ipynb
└── README.md
Zhai, Z., Gu, B., Li, X., & Huang, H. (2020). Safe Sample Screening for Robust Support Vector Machine. Proceedings of the AAAI Conference on Artificial Intelligence, 34(04), 6981–6988.
https://doi.org/10.1609/aaai.v34i04.6182
UCI Machine Learning Repository — Letter Recognition Dataset.