diff --git a/.gitignore b/.gitignore index a7779be..cbc8662 100644 --- a/.gitignore +++ b/.gitignore @@ -8,4 +8,5 @@ *.sqlite-wal .vs/ **appsettings.Development.json -!**/appsettings.Example.json \ No newline at end of file +!**/appsettings.Example.json +debug/** \ No newline at end of file diff --git a/README.md b/README.md index 284165b..f5339ed 100644 --- a/README.md +++ b/README.md @@ -1,10 +1,38 @@ # ReverseLlama -(Eigenentwicklung; weitesgehend vibecoded) +ReverseLlama is a small outbound HTTP tunnel for running Ollama (or vLLM, etc.) on GPU workstations while exposing the API from a server that cannot reach those workstations directly. -ReverseLlama is a small outbound HTTP tunnel for testing Ollama on GPU workstations while exposing the API from a server that cannot reach those workstations directly. +The client opens and maintains a WebSocket connection to the server. The server accepts normal HTTP requests and forwards them through that WebSocket to the client. The client then calls a local upstream such as `http://localhost:11434` and streams the response back. -The client opens a WebSocket connection to the server. The server accepts normal HTTP requests and forwards them through that WebSocket to the client. The client then calls a local upstream such as `http://localhost:11434` and streams the response back. +
![]() |
++ +The server provides +- An API with + - Authentication via API keys + - Authorization (planned) +- Load balancing (Scale your AI strategy horizontally!) +- (Ollama-only) Model management (install, remove, load, unload models) +- Client monitoring + - Who is active + - What models are running + - How many requests is each client processing + - How many requests has each client processed +- Group management (planned) + - Who can access which models + - What clients are mapped to which groups +- Billing (planned) + - (planned) Price per model per thousand tokens + - (planned) Usage per API user + - (planned) Rate limiting + +The client provides a persistent outbound connection to the server and forwards requests to the local Ollama (or vLLM, etc.) instance. Responses stream back through the tunnel with minimal overhead. + + | +