ametnes/litellm

LiteLLM - Unified Interface for 100+ LLMs

![Documentation]([***]

LiteLLM is a production-ready LLM Gateway and unified interface that allows you to call 100+ Large Language Models (LLMs) using a consistent OpenAI-compatible API format. Whether you're using OpenAI, Anthropic, Azure, Bedrock, Vertex AI, or any other provider, LiteLLM translates your requests and provides consistent responses.

🚀 What is LiteLLM?

LiteLLM is an AI Gateway (Proxy Server) that provides:

• Unified API: Call 100+ LLMs (OpenAI, Anthropic, Azure, Bedrock, Vertex AI, Groq, etc.) using the same OpenAI format
• Production Features: Authentication, rate limiting, cost tracking, load balancing, and observability
• Multi-tenant Support: Virtual keys, user management, team-based access control
• Admin Dashboard: Web UI for monitoring, managing models, and tracking usage
• Enterprise Ready: High performance (8ms P95 latency at 1K RPS), horizontal scaling, and production best practices

📦 Image Tags

Recommended: Use main-stable or version-specific -stable tags for production deployments.

🏃 Quick Start

Basic Usage

docker run -p 4000:4000 \
  -e OPENAI_API_KEY=your-openai-key \
  berriai/litellm:main-stable \
  --model gpt-4o### With Configuration File

docker run -p 4000:4000 \
  -v $(pwd)/config.yaml:/app/config.yaml \
  -e DATABASE_URL=postgresql://user:pass@host:5432/litellm \
  berriai/litellm:main-stable \
  --config /app/config.yaml

Using Docker Compose

ml
services:
  litellm:
    image: berriai/litellm:main-stable
    ports:
      - "4000:4000"
    environment:
      DATABASE_URL: postgresql://llmproxy:password@db:5432/litellm
      STORE_MODEL_IN_DB: "True"
    depends_on:
      - db
  
  db:
    image: postgres:16
    environment:
      POSTGRES_DB: litellm
      POSTGRES_USER: llmproxy
      POSTGRES_PASSWORD: password## 🔧 Configuration

Environment Variables

Command Line Arguments

docker run berriai/litellm:main-stable
--port 4000 \ # Server port (default: 4000) --config /app/config.yaml \ # Path to config file --num_workers 4 \ # Number of worker processes --run_gunicorn \ # Use Gunicorn instead of Uvicorn --max_requests_before_restart *** # Worker recycling### Example Config File

Create config.yaml:

model_list:

• model_name: gpt-4o litellm_params: model: openai/gpt-4o api_key: os.environ/OPENAI_API_KEY

general_settings: master_key: os.environ/LITELLM_MASTER_KEY database_connection_pool_limit: 10 proxy_batch_write_at: 60

router_settings: redis_host: os.environ/REDIS_HOST redis_port: os.environ/REDIS_PORT redis_password: os.environ/REDIS_PASSWORD

litellm_settings: cache: true cache_params: type: redis host: os.environ/REDIS_HOST port: os.environ/REDIS_PORT password: os.environ/REDIS_PASSWORD## 📚 Key Features

1. Unified API for 100+ LLMs

Call any LLM provider using OpenAI's format:

• OpenAI, Anthropic, Azure, Bedrock, Vertex AI
• Groq, Cohere, Hugging Face, Replicate
• And 90+ more providers

2. Production-Ready Gateway

• Authentication & Authorization: Virtual keys, user management, team-based access
• Rate Limiting: Per-key, per-user, per-team rate limits
• Cost Tracking: Real-time spend tracking and budget management
• Load Balancing: Intelligent routing across multiple deployments
• Observability: Prometheus metrics, logging, tracing

3. Admin Dashboard

Access the web UI at http://localhost:4000/ui to:

• Manage models and deployments
• Create and manage API keys
• Monitor usage and costs
• Configure teams and permissions

4. High Performance

• 8ms P95 latency at 1K RPS
• Horizontal scaling support
• Redis caching for 60-80% DB load reduction
• Optimized for production workloads

🎯 Use Cases

• LLM Gateway: Centralized API gateway for multiple LLM providers
• Cost Management: Track and control LLM spending across teams/projects
• Load Balancing: Distribute traffic across multiple deployments
• Multi-tenant Applications: Isolated access control per user/team
• Observability: Centralized logging and monitoring for LLM usage

📖 Documentation

• Full Documentation
• Quick Start Guide
• Production Best Practices
• Supported Providers
• Configuration Reference

🔗 Related Resources

• GitHub: https://github.com/BerriAI/litellm
• Website: [***]
• *******: Join our community
• Documentation: [***]

📝 Example: Making Your First Request

python
import openai

client = openai.OpenAI(
    api_key="your-virtual-key",  # Created via admin UI
    base_url="http://localhost:4000"
)

response = client.chat.completions.create(
    model="gpt-4o",
    messages=[{"role": "user", "content": "Hello!"}]
)

print(response.choices[0].message.content)## 🛠️ Production Deployment

For production deployments, see our Production Guide which covers:

• Recommended machine specifications
• Kubernetes deployment
• Database and Redis setup
• Monitoring and alerting
• Security best practices

📊 Performance Benchmarks

• Median Latency: 100ms at 1K RPS
• P95 Latency: 150ms at 1K RPS
• P99 Latency: 240ms at 1K RPS
• Throughput: 1,170 RPS per instance (4 vCPU, 8GB RAM)

See Benchmarks for detailed performance metrics.

🤝 Support

• Documentation: [***]
• GitHub Issues: https://github.com/BerriAI/litellm/issues
• *******: Community support
• Enterprise Support: Contact us

📄 License

LiteLLM is licensed under the MIT License. See https://github.com/BerriAI/litellm/blob/main/LICENSE for details.

---

Made with ❤️ by BerriAI