tytn/fabulinus

TitanML | Takeoff Server

About   |   Features    

🎯 About

This is the Docker repository for the repository for the community edition of the TitanML Takeoff server. This is a server designed for optimized inference of large language models.

The takeoff server consists of two components, a command line manager interface, called iris and a server image, called fabulinus. The fabulinus images are stored here.

For usage information, tutorials, and usage examples, see the docs.

✨ Community Features

✔️ Easy deployment and streaming response

✔️ Optimized int8 quantization

✔️ Chat and playground-like interface

✔️ Support for encoder-decoder (T5 family) and decoder models

For the pro edition, including multi-gpu inference, int4 quantization, and more. contact us

🚊 Usage

iris

The easiest way to use the inference server is via the iris launcher. To install iris, run

pip install titan-iris

Then, to launch an inference server with a model, run

iris takeoff --model tiiuae/falcon-7b-instruct --device cpu --port 8000

You'll be prompted to login. To run with GPU access, add --device cuda instead.

fabulinus

Images are tagged with their version number, and an optional -cpu prefix. Behaviour of both images should be the same, except that the -cpu images are smaller (since they're built without GPU support).

To run the latest server images,

docker run -e TAKEOFF_MODEL_NAME=<model_name> -p 8000:80 tytn/fabulinus:latest-cpu

or, for GPU support

docker run --gpus all -e TAKEOFF_MODEL_NAME=<model_name> -e TAKEOFF_DEVICE=cuda -p 8000:80 tytn/fabulinus:latest

To experiment with the resulting server, navigate to http://localhost:8000/demos/playground, or http://localhost:8000/demos/chat. To see docs on how to query the model, navigate to http://localhost:8000/docs

ℹ️ Contributing

See our https://github.com/titanml/takeoff for more information about how to contribute.