portkeyai/gateway

Portkey's AI Gateway is the interface between your app and hosted LLMs. It streamlines API requests to OpenAI, Anthropic, Mistral, LLama2, Anyscale, Google Gemini and more with a unified API.

✅  Blazing fast (9.9x faster) with a tiny footprint (~45kb installed)
✅  Load balance across multiple models, providers, and keys
✅  Fallbacks make sure your app stays resilient
✅  Automatic Retries with exponential fallbacks come by default
✅  Plug-in middleware as needed
✅  Battle tested over 100B tokens

Deploy using Docker

Run using Docker directly:

docker run -d -p 8787:8787 portkeyai/gateway:latest

For more information on the Docker image, check https://hub.docker.com/r/portkeyai/gateway

Deploy using Docker Compose

1. Download compose file from the repo:

wget "https://raw.githubusercontent.com/Portkey-AI/gateway/main/docker-compose.yaml"

2. Run:

docker compose up -d

3. The service is now running and listening on port 8787

Usage

Let's try making a chat completions call to OpenAI through the AI gateway:

bash
curl '127.0.0.1:8787/v1/chat/completions' \
  -H 'x-portkey-provider: openai' \
  -H "Authorization: Bearer $OPENAI_KEY" \
  -H 'Content-Type: application/json' \
  -d '{"messages": [{"role": "user","content": "Say this is test."}], "max_tokens": 20, "model": "gpt-4"}'

Full list of supported SDKs

Supported Providers

View the complete list of 100+ supported models here

Features

 <br><br>
</td>
<td>
  <h4>Fallback</h4>
  Don't let failures stop you. The Fallback feature allows you to specify a list of Language Model APIs (LLMs) in a prioritized order. If the primary LLM fails to respond or encounters an error, Portkey will automatically fallback to the next LLM in the list, ensuring your application's robustness and reliability.
  <br><br>
  
</td>

</td>
<td>
  <h4>Load Balancing</h4>
  Distribute load effectively across multiple API keys or providers based on custom weights. This ensures high availability and optimal performance of your generative AI apps, preventing any single LLM from becoming a performance bottleneck.
  <br><br>
  
</td>

Supported SDKs

Community

Join our growing community around the world, for help, ideas, and discussions on AI.

• View our official Blog
• Chat live with us on ***
• Follow us on ***
• Connect with us on LinkedIn

!https://github.com/Portkey-AI/gateway/assets/971978/89d6f0af-a95d-4402-b451-***c40d03f