Deep Learning Library

A from-scratch deep learning framework in modern C++ with Python bindings.

Motivation

Built to understand deep learning frameworks from first principles - from computational graphs to gradient computation to optimization algorithms.

Running examples

For some examples on Python interface, see tests/python.

Features

• Computational Graph: Dynamic graph construction with automatic differentiation
• Core Components:
  • Automatic differentiation (autograd)
  • Backpropagation engine
  • Neural network layers
  • Training framework (optimizers, loss functions, layers, and networks)
• Example code: Full MNIST dataset training example
• Python Interface: Seamless integration via Boost.Python
• Clean Architecture: Modular design, maintainable and extensible
• CI/CD: Automated testing with GTest and GitHub Actions
• CUDA implementation: Full end-to-end CUDA implementation for better HW utilization

Tech Stack

• C++17/20/23
• CMake build system
• Boost.Python for Python bindings
• Python 3 for library interface and examples
• Google Test (GTest) and PyTest for unit testing
• GitHub Actions for CI/CD
• CUDA

Current Status

🚧 Work in Progress - Implementing additional layers and optimizations

Roadmap:

• Python Binding Unit Tests
• Optimizers and training framework
• MNIST example
• CUDA mode for operations
• CUDA optimization for optimal run-times
• CPU optimization for optimal run-times
• Additional layer types (Conv2D, Dropout, etc.)

Building

mkdir build && cd build
cmake ..
make

Building with CUDA

Project automatically detects whether CUDA is installed, and compiles with it. If CUDA compilation not desired you can switch it off via

cmake --DCUDA=Off ..

Running Unit Tests

Compile with building tests enabled:

mkdir build && cd build
cmake -DBUILD_TESTS=On ..
make
ctest .

Some hints on performance tuning

There are several knobs for tuning, and this currently has been adjusted using my personal hardware. Some things you can do right away and experiment with are:

1. Choose an appropriate AVX version, see also flag ${AVX_VERSION} in the base CMakeLists.txt
2. Enable CUDA if you can for the biggest wins
3. Some of the CPU instructions keep the hardware layout in mind. You can adapt using file 'memory_layout.h' for a start and then experiment.
4. If you want to go into the code, I plan to blog about profiler driven optimizations and tips and tricks on my personal blog

Required

• Compiler capable of C++23 at least (we test with gcc 13.3.0)
• Boost Python
• Cmake > 3.28
• Python 3 (we test with 3.10, but it should work with any version)
• numpy 1.26.4
• pytest and GTest for unit tests (we use pytest=9.0.2)
• Google Benchmark for benchmarking
• CUDA (we use CUDA 13.1 on an RTX-5050)

Troubleshooting

Building on Windows

The implementation of the Python wrapper does not work on MSVC6/7 in its current form. This is due to an issue that arises from Boost Python in combination with these compilers. Workarounds are proposed, but not implemented. More information here here.

License

MIT