ampy-proto

Canonical Protocol Buffer schemas for financial data processing

ampy-proto provides type-safe, cross-language schemas for financial market data including bars, ticks, orders, fills, and more. Built for high-precision trading systems with support for Go, Python, and C++.

What We're Solving

AmpyFin is building a self-learning, modular trading system that needs to handle data from multiple sources (DataBento, Tiingo, yfinance, etc.) across different languages (Go, Python, C++). This project provides:

• Unified schemas for all financial data (bars, ticks, fundamentals, news, etc.)
• Cross-language compatibility with generated code for Go, Python, and C++
• Precision guarantees using scaled decimal types instead of floating-point
• Time discipline with UTC timestamps and clear event/ingest time separation
• Versioned contracts that evolve safely without breaking consumers

🚀 Quick Start

Python

# Install from PyPI
pip install ampy-proto==2.1.1

# Or install locally
pip install -e .

from ampy.bars.v1 import bars_pb2
from ampy.common.v1 import common_pb2

# Create a bar
bar = bars_pb2.Bar()
bar.security.symbol = "AAPL"
bar.security.mic = "XNAS"
bar.open.scaled = 1923450
bar.open.scale = 4
bar.high.scaled = 1925600
bar.high.scale = 4
bar.low.scaled = 1922200
bar.low.scale = 4
bar.close.scaled = 1924100
bar.close.scale = 4
bar.volume = 184230

print(f"AAPL bar: ${bar.close.scaled / (10 ** bar.close.scale):.2f}")

Go

# Add to your go.mod
go get github.com/AmpyFin/ampy-proto/v2@v2.1.1

package main

import (
    "fmt"
    "time"
    
    "github.com/AmpyFin/ampy-proto/v2/pkg/ampy"
    bars "github.com/AmpyFin/ampy-proto/v2/gen/go/ampy/bars/v1"
    common "github.com/AmpyFin/ampy-proto/v2/gen/go/ampy/common/v1"
    "google.golang.org/protobuf/proto"
    "google.golang.org/protobuf/types/known/timestamppb"
)

func main() {
    bar := &bars.Bar{
        Security: &common.SecurityId{
            Symbol: "AAPL",
            Mic:    "XNAS",
        },
        Start: timestamppb.New(time.Now().Add(-time.Minute)),
        End:   timestamppb.New(time.Now()),
        Open: &common.Decimal{
            Scaled: 1923450,  // 192.3450 with scale 4
            Scale:  4,
        },
        Close: &common.Decimal{
            Scaled: 1924100,  // 192.4100 with scale 4
            Scale:  4,
        },
        Volume: 184230,
    }

    data, _ := proto.Marshal(bar)
    fmt.Printf("Serialized %d bytes using ampy-proto v%s\n", len(data), ampy.Version)
}

C++

Prerequisites: Install latest versions of Abseil and Protobuf:

# Install Abseil (latest version)
git clone https://github.com/abseil/abseil-cpp.git
cd abseil-cpp
mkdir build && cd build
cmake -DCMAKE_CXX_STANDARD=17 -DCMAKE_POSITION_INDEPENDENT_CODE=ON ..
make -j$(nproc)
sudo make install

# Install Protobuf (latest version)
git clone https://github.com/protobuf/protobuf.git
cd protobuf
git submodule update --init --recursive
mkdir build && cd build
cmake -Dprotobuf_BUILD_TESTS=OFF -DCMAKE_CXX_STANDARD=17 ..
make -j$(nproc)
sudo make install

Build the library:

# Build the library
cd gen/cpp
mkdir build && cd build
cmake .. && make

#include <iostream>
#include "ampy/bars/v1/bars.pb.h"

int main() {
    ampy::bars::v1::Bar bar;
    bar.mutable_security()->set_symbol("AAPL");
    bar.mutable_security()->set_mic("XNAS");
    bar.mutable_open()->set_scaled(1923450);
    bar.mutable_open()->set_scale(4);
    bar.mutable_high()->set_scaled(1925600);
    bar.mutable_high()->set_scale(4);
    bar.mutable_low()->set_scaled(1922200);
    bar.mutable_low()->set_scale(4);
    bar.mutable_close()->set_scaled(1924100);
    bar.mutable_close()->set_scale(4);
    bar.set_volume(184230);

    std::cout << "AAPL bar: $" 
              << (bar.close().scaled() / std::pow(10, bar.close().scale()))
              << std::endl;
    return 0;
}

📊 Available Schemas

Domain	Purpose	Key Messages
bars	OHLCV price bars	Bar, BarBatch
ticks	Trade and quote data	Tick, TickBatch
fundamentals	Financial statements	Fundamental, FundamentalBatch
news	Market news and sentiment	NewsItem
fx	Foreign exchange rates	FxRate
corporate_actions	Splits, dividends	CorporateAction
universe	Tradable securities lists	Universe
signals	Model outputs and signals	Signal
orders	Order management	Order, OrderRequest
fills	Trade executions	Fill
positions	Portfolio positions	Position
metrics	Operational metrics	Metric
common	Shared types	Decimal, Money, SecurityId, Meta

🎯 Key Design Principles

1. Precision with Scaled Decimals

Instead of floating-point numbers, we use scaled decimals:

message Decimal {
  int64 scaled = 1;  // The actual value
  int32 scale = 2;   // Decimal places (e.g., 4 = 4 decimal places)
}

Example: scaled: 1923450, scale: 4 represents 192.3450

2. Time Discipline

All timestamps are UTC with clear semantics:

• event_time: When the market event actually happened
• ingest_time: When our system received it
• as_of: Logical timestamp for downstream processing

3. Security Identification

Use proper security identifiers, not just tickers:

message SecurityId {
  string symbol = 1;    // e.g., "AAPL"
  string mic = 2;       // Market identifier code, e.g., "XNAS"
  string figi = 3;      // Financial Instrument Global Identifier (optional)
  string isin = 4;      // International Securities Identification Number (optional)
}

4. Metadata for Traceability

Every message includes metadata for lineage:

message Meta {
  string run_id = 1;           // Unique run identifier
  string source = 2;           // Data source (e.g., "yfinance-go")
  string producer = 3;         // Producer instance ID
  string schema_version = 4;   // Schema version used
  string checksum = 5;         // Optional message checksum
}

🛠️ Development

Prerequisites

• Buf for protobuf management
• Go 1.23+ for Go code generation
• Python 3.9+ for Python code generation
• CMake and C++17 compiler for C++ code generation

Building

# Generate code for all languages
make gen

# Lint protobuf files
make lint

# Run tests
make test

# Build Python package
make py-build

Testing

# Test Go imports
go run examples/go/smoke/main.go

# Test Python imports
python examples/python/smoke.py

# Test C++ compilation
cd examples/cpp && make && ./build/smoke_test

# Run roundtrip tests
make test

📈 Versioning

This project follows semantic versioning:

• Major versions (v2, v3): Breaking changes requiring migration
• Minor versions (v2.1, v2.2): Additive changes, backward compatible
• Patch versions (v2.1.0, v2.1.1): Bug fixes, backward compatible

Current version: v2.1.1

🔧 Requirements

Go

require (
    github.com/AmpyFin/ampy-proto/v2 v2.1.1+
    google.golang.org/protobuf v1.36.8
)

Python

# pyproject.toml
dependencies = ["protobuf>=6.32.0,<7"]

🤝 Contributing

1. Fork the repository
2. Create a feature branch: git checkout -b feature/your-feature
3. Make your changes and add tests
4. Run make lint and make test
5. Submit a pull request

Schema Evolution Rules

• ✅ Add new optional fields with default values
• ✅ Add new enum values (append only)
• ✅ Add new messages or services
• ❌ Never change field numbers of existing fields
• ❌ Never change field types of existing fields
• ❌ Never remove fields (mark as deprecated instead)
• ❌ Never renumber enum values

📄 License

MIT License - see LICENSE file for details.

🆘 Support

• Issues: GitHub Issues
• Documentation: Project Wiki

---

Part of the AmpyFin ecosystem - building the future of algorithmic trading.