BentoML is an open-source platform that streamlines machine-learning model deployment and runs high-performance serving workload at scale.

• 🍱 BentoML: The Unified Model Serving Framework
• 🦄️ yatai: Run BentoML workflow at scale on Kubernetes
• 🚀 bentoctl: Fast model deployment with BentoML on cloud platforms

Join us in our community Slack to receive project updates and get involved with latest development.

🍱 BentoML repo | 🎨 Gallery Projects | 📖 Documentation

BentoML provides high-level APIs for packaging ML models and defining ML Services. From the ML service definition, BentoML allows building versioned archive(aka Bento) capturing all required dependencies, automatically generate HTTP Server and create Docker Image for production-grade deployment.

Key Features:

• Support multiple ML frameworks including PyTorch, TensorFlow, Scikit-Learn, XGBoost, and many more
• Support Adaptive Batching which dynamically group inference requets into small batches in real-time for better performance
• Build inference graph composed from multiple models and execute them in parallel
• Automatic Docker image can be generated for production deployment
• Automatically generate REST API spec in Swagger/OpenAPI format

🦄️ Yatai repo | 👩‍🚀 Administrator's Guide | ⎈ Helm Chart

Yatai helps ML teams to run BentoML at scale on Kubernetes. It makes it easy for teams to manage all their ML assets in one place and operate large scale model serving workloads on Kubernetes.

• Bento Registry - manage all your team's Bentos and Models, backed by cloud blob storage(S3, MinIO)
• Deployment Automation - deploy Bentos as auto-scaling API endpoints on Kubernetes and easily rollout new versions
• Observability - monitoring dashboard helping users to identify model performance issues
• CI/CD - flexible APIs for integrating with your training and CI pipelines

🚀 bentoctl source

bentoctl is a CLI tool for deploying your BentoML packaged ML models as API endpoint on popular cloud platforms. It automates Bento docker image build, interactes with cloud platform APIs, and allow users to easily manage their deployment.

Supported platforms:

• AWS EC2
• AWS Lambda
• AWS SageMaker
• Azure Functions
• Azure Container Instances
• Google Cloud Run
• Google Compute Engine
• Heroku

Custom deploy target is also supported by building your own bentoctl plugin from the deployment operator template.