11 Commits
Author SHA1 Message Date
lucretia d33a25bfa5 fix(server): replaces text logo with proper logo
Build & Deploy / build (push) Successful in 1m42s
2026-08-02 01:26:12 +02:00
lucretia b849cfd4b7 feat(client): file logging + llama.cpp fallback cooldown and parallel options
Build & Deploy / build (push) Successful in 2m8s
2026-08-02 01:24:09 +02:00
lucretia 44b09ffa86 fix(server): fixes favicon not showing
Build & Deploy / build (push) Successful in 1m55s
2026-08-01 22:44:04 +02:00
lucretia 2b56bc53b6 fix: recovers from dead llama.cpp containers and serves /api/version
Build & Deploy / build (push) Successful in 3m10s
2026-08-01 19:09:32 +02:00
LD50andGitHub faba272480 Merge pull request #33 from LD-Reborn/21-create-an-app-icon
Build & Deploy / build (push) Successful in 2m2s
docs: adds images
2026-07-31 17:28:17 +02:00
lucretia 9361aaf2fc docs: adds images 2026-07-31 17:27:53 +02:00
lucretia ded9030ede feat(deploy): adds uninstaller, docs: fixes wording and adds llama.cpp installation examples
Build & Deploy / build (push) Successful in 1m48s
2026-07-31 16:03:49 +02:00
lucretia afc85e4802 Merge branch 'main' of https://github.com/LD-Reborn/Ngino
Build & Deploy / build (push) Successful in 1m43s
2026-07-30 19:34:57 +02:00
LD50andGitHub 7a76acc247 Merge pull request #31 from LD-Reborn/10-change-disable-1h-to-disable-temporary
10 change disable 1h to disable temporary
2026-07-29 21:07:28 +02:00
lucretia 0c136f86cb fix(Server): fixes temporary disable input visibility, adds reason text
Build & Deploy / build (push) Successful in 1m43s
2026-07-29 20:18:35 +02:00
lucretia 9d01eb5311 feat(server): adds temporary disabling functionality
Build & Deploy / build (push) Successful in 1m55s
2026-07-29 20:06:52 +02:00
24 changed files with 1459 additions and 101 deletions
+24 -5
View File
@@ -1,8 +1,10 @@
# Ngino # <img src="docs/images/Ngino_logo_symbol.png" width="25" alt="logo symbol"/> Ngino
Ngino is a small outbound HTTP tunnel for running Ollama (or vLLM, etc.) on GPU workstations while exposing the API from a server that cannot reach those workstations directly. Ngino is a small outbound HTTP tunnel for running Ollama, llama.cpp, vLLM, etc. on GPU workstations while exposing the API from a server that cannot reach those workstations directly.
The client opens and maintains a WebSocket connection to the server. The server accepts normal HTTP requests and forwards them through that WebSocket to the client. The client then calls a local upstream such as `http://localhost:11434` and streams the response back. <img src="docs/images/Ngino_logo_full.png" alt="logo symbol"/>
The client opens and maintains a WebSocket connection to the server. The server accepts normal HTTP requests from users and forwards them through that WebSocket to the client. The client then calls a local upstream such as `http://localhost:11434` and streams the response back.
<table> <table>
<tr> <tr>
@@ -15,8 +17,8 @@ The server provides
- An API with - An API with
- Authentication via user keys - Authentication via user keys
- Authorization (planned) - Authorization (planned)
- Load balancing (Scale your AI strategy horizontally!) - Load balancing (Scale your AI inferencing horizontally by adding more nodes!)
- (Ollama-only) Model management (install, remove, load, unload models) - (Ollama-based) GGUF Model management (install, remove, load, unload models)
- Client monitoring - Client monitoring
- Who is active - Who is active
- What models are running - What models are running
@@ -129,6 +131,14 @@ Linux:
```bash ```bash
sudo bash deploy/install-client.sh --server http://your-server:5050 --token "change-me" sudo bash deploy/install-client.sh --server http://your-server:5050 --token "change-me"
``` ```
or using ollama models with llama.cpp backend using ROCm:
```bash
sudo bash deploy/install-client.sh --server https://ai.domain.tld --token "change-me" --use-llama-cpp-via-docker --use-ollama-models-path /usr/share/ollama/.ollama/models --llama-cpp-docker-image ghcr.io/ggml-org/llama.cpp:server-rocm --llama-cpp-base-port 8081 --llama-cpp-parallel 128
```
or using ollama models with llama.cpp backend using CUDA:
```bash
sudo bash deploy/install-client.sh --server https://ai.domain.tld --token "change-me" --use-llama-cpp-via-docker --use-ollama-models-path /usr/share/ollama/.ollama/models --llama-cpp-docker-image ghcr.io/ggml-org/llama.cpp:server-cuda --llama-cpp-base-port 8081 --llama-cpp-parallel 128
```
Options: `--server`, `--token` (required); `--client-id`, `--upstream`, `--install-dir`, `--service-name`, `--no-ollama` (optional). Missing required values are prompted interactively. Options: `--server`, `--token` (required); `--client-id`, `--upstream`, `--install-dir`, `--service-name`, `--no-ollama` (optional). Missing required values are prompted interactively.
@@ -140,6 +150,15 @@ llama.cpp via Docker options (replaces Ollama for inferencing):
| `--use-ollama-models-path <dir>` | Path to Ollama models directory (`manifests/blobs`); required with the flag above | | `--use-ollama-models-path <dir>` | Path to Ollama models directory (`manifests/blobs`); required with the flag above |
| `--llama-cpp-docker-image <img>` | Docker image; defaults to auto-detected (rocm/cuda/cpu) | | `--llama-cpp-docker-image <img>` | Docker image; defaults to auto-detected (rocm/cuda/cpu) |
| `--llama-cpp-base-port <num>` | Base port for containers; defaults to `8081` | | `--llama-cpp-base-port <num>` | Base port for containers; defaults to `8081` |
| `--llama-cpp-parallel <num>` | llama.cpp parallel slots per container; if unset, llama.cpp's own default is used (which is `1`) |
| `--llama-cpp-fallback-cooldown <sec>` | Seconds before llama.cpp is retried after a failed container start; defaults to `180` |
| `--log-dir <dir>` | Directory for log files; defaults to `<app dir>/Logs` |
### llama.cpp fallback to Ollama
Models are served via llama.cpp Docker containers. If a container cannot be started for a model (for example, the model's GGUF blob is incompatible with the llama.cpp build), the client falls back to the Ollama upstream for that model. Transient start failures are remembered for `--llama-cpp-fallback-cooldown` seconds (default 3 minutes) and then retried; a container that starts but exits before becoming ready marks the model as falling back until it is unloaded. `load`/`unload` model commands and on-demand request routing are all covered; a failed container start is detected quickly by watching the container state, and the container log tail is written to the client log to aid debugging.
Note: some hybrid SSM/attention models (e.g. `qwen3.5-coder-next`) are converted by Ollama into a GGUF tensor layout that stock llama.cpp cannot load (`missing tensor 'blk.0.ssm_dt.bias'` and similar). Such models are served via the Ollama fallback above. If you want them to run on llama.cpp instead, use a Hugging Face-converted GGUF (e.g. `unsloth/Qwen3-Coder-Next-GGUF`) rather than the Ollama blob.
The script ensures .NET 10 and Ollama are installed, builds the client self-contained, installs it to `/opt/Ngino-client`, and creates a systemd service (`Ngino-client`). Logs: `journalctl -u Ngino-client -f`. The script ensures .NET 10 and Ollama are installed, builds the client self-contained, installs it to `/opt/Ngino-client`, and creates a systemd service (`Ngino-client`). Logs: `journalctl -u Ngino-client -f`.
+22 -1
View File
@@ -20,6 +20,9 @@ USE_LLAMA_CPP_VIA_DOCKER=false
USE_OLLAMA_MODELS_PATH="" USE_OLLAMA_MODELS_PATH=""
LLAMA_CPP_DOCKER_IMAGE="" LLAMA_CPP_DOCKER_IMAGE=""
LLAMA_CPP_BASE_PORT="" LLAMA_CPP_BASE_PORT=""
LLAMA_CPP_PARALLEL=""
LLAMA_CPP_FALLBACK_COOLDOWN=""
LOG_DIR=""
# ── Colors ──────────────────────────────────────────────────────────────────── # ── Colors ────────────────────────────────────────────────────────────────────
RED='\033[0;31m' RED='\033[0;31m'
@@ -57,13 +60,19 @@ Optional:
llama.cpp Docker image; defaults to auto-detected (rocm/cuda/cpu) llama.cpp Docker image; defaults to auto-detected (rocm/cuda/cpu)
--llama-cpp-base-port <num> --llama-cpp-base-port <num>
Base port for llama.cpp containers; defaults to 8081 Base port for llama.cpp containers; defaults to 8081
--llama-cpp-parallel <num>
llama.cpp parallel slots per container; if unset, llama.cpp's own default is used
--llama-cpp-fallback-cooldown <sec>
Seconds before llama.cpp is retried after a failed container start; defaults to 180
--log-dir <dir> Directory for log files; defaults to <install-dir>/Logs
-h, --help Show this help message -h, --help Show this help message
Examples: Examples:
$0 --server http://gpu-server:5050 --token "my-secret" $0 --server http://gpu-server:5050 --token "my-secret"
$0 --server http://gpu-server:5050 --token "my-secret" --no-ollama $0 --server http://gpu-server:5050 --token "my-secret" --no-ollama
$0 --server http://gpu-server:5050 --token "my-secret" \\ $0 --server http://gpu-server:5050 --token "my-secret" \\
--use-llama-cpp-via-docker --use-ollama-models-path /usr/share/ollama/.ollama/models --use-llama-cpp-via-docker --use-ollama-models-path /usr/share/ollama/.ollama/models \\
--llama-cpp-parallel 128
EOF EOF
exit 0 exit 0
} }
@@ -82,6 +91,9 @@ while [[ $# -gt 0 ]]; do
--use-ollama-models-path) USE_OLLAMA_MODELS_PATH="$2"; shift 2 ;; --use-ollama-models-path) USE_OLLAMA_MODELS_PATH="$2"; shift 2 ;;
--llama-cpp-docker-image) LLAMA_CPP_DOCKER_IMAGE="$2"; shift 2 ;; --llama-cpp-docker-image) LLAMA_CPP_DOCKER_IMAGE="$2"; shift 2 ;;
--llama-cpp-base-port) LLAMA_CPP_BASE_PORT="$2"; shift 2 ;; --llama-cpp-base-port) LLAMA_CPP_BASE_PORT="$2"; shift 2 ;;
--llama-cpp-parallel) LLAMA_CPP_PARALLEL="$2"; shift 2 ;;
--llama-cpp-fallback-cooldown) LLAMA_CPP_FALLBACK_COOLDOWN="$2"; shift 2 ;;
--log-dir) LOG_DIR="$2"; shift 2 ;;
-h|--help) usage ;; -h|--help) usage ;;
*) die "Unknown option: $1" ;; *) die "Unknown option: $1" ;;
esac esac
@@ -311,6 +323,15 @@ mkdir -p "$ENV_DIR"
if [[ -n "$LLAMA_CPP_BASE_PORT" ]]; then if [[ -n "$LLAMA_CPP_BASE_PORT" ]]; then
printf 'NGINO_LLAMA_CPP_BASE_PORT=%s\n' "$LLAMA_CPP_BASE_PORT" printf 'NGINO_LLAMA_CPP_BASE_PORT=%s\n' "$LLAMA_CPP_BASE_PORT"
fi fi
if [[ -n "$LLAMA_CPP_PARALLEL" ]]; then
printf 'NGINO_LLAMA_CPP_PARALLEL=%s\n' "$LLAMA_CPP_PARALLEL"
fi
if [[ -n "$LLAMA_CPP_FALLBACK_COOLDOWN" ]]; then
printf 'NGINO_LLAMA_CPP_FALLBACK_COOLDOWN_SECONDS=%s\n' "$LLAMA_CPP_FALLBACK_COOLDOWN"
fi
fi
if [[ -n "$LOG_DIR" ]]; then
printf 'NGINO_LOG_DIR=%s\n' "$LOG_DIR"
fi fi
} > "$ENV_DIR/env" } > "$ENV_DIR/env"
chmod 600 "$ENV_DIR/env" chmod 600 "$ENV_DIR/env"
+42
View File
@@ -0,0 +1,42 @@
#Requires -Version 5.1
#Requires -RunAsAdministrator
[CmdletBinding()]
param(
[string]$InstallDir = "$env:ProgramFiles\Ngino Client",
[string]$ServiceName = "NginoClient"
)
$ErrorActionPreference = "Stop"
function Write-Info([string]$Message) { Write-Host "[INFO] $Message" -ForegroundColor Green }
if ($ServiceName -notmatch '^[A-Za-z0-9_.-]+$') { throw "ServiceName contains unsupported characters." }
# ── Stop and delete the Windows service ───────────────────────────────────────
$service = Get-Service -Name $ServiceName -ErrorAction SilentlyContinue
if ($service) {
if ($service.Status -ne "Stopped") {
Write-Info "Stopping service $ServiceName..."
Stop-Service -Name $ServiceName -Force
$service.WaitForStatus("Stopped", [TimeSpan]::FromSeconds(30))
}
Write-Info "Deleting service $ServiceName..."
& sc.exe delete $ServiceName
if ($LASTEXITCODE -ne 0) { throw "Could not delete Windows service $ServiceName." }
} else {
Write-Info "Service $ServiceName does not exist; skipping."
}
# ── Remove install directory ──────────────────────────────────────────────────
if (Test-Path -LiteralPath $InstallDir) {
Write-Info "Removing install directory $InstallDir..."
Remove-Item -LiteralPath $InstallDir -Recurse -Force
} else {
Write-Info "Install directory $InstallDir does not exist; skipping."
}
Write-Host ""
Write-Info "Uninstall complete."
Write-Host " Service: $ServiceName (stopped and deleted)"
Write-Host " Install dir: $InstallDir (removed)"
+95
View File
@@ -0,0 +1,95 @@
#!/usr/bin/env bash
set -euo pipefail
# ── Defaults ──────────────────────────────────────────────────────────────────
DEFAULT_INSTALL_DIR="/opt/ngino-client"
DEFAULT_SERVICE_NAME="ngino-client"
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
INSTALL_DIR="$DEFAULT_INSTALL_DIR"
SERVICE_NAME="$DEFAULT_SERVICE_NAME"
# ── Colors ────────────────────────────────────────────────────────────────────
RED='\033[0;31m'
GREEN='\033[0;32m'
YELLOW='\033[0;33m'
NC='\033[0m'
info() { echo -e "${GREEN}[INFO]${NC} $*"; }
warn() { echo -e "${YELLOW}[WARN]${NC} $*"; }
err() { echo -e "${RED}[ERROR]${NC} $*" >&2; }
die() { err "$@"; exit 1; }
# ── Usage ─────────────────────────────────────────────────────────────────────
usage() {
cat <<EOF
Usage: $0 [OPTIONS]
Uninstalls the Ngino client systemd service and removes its files.
Does NOT remove .NET SDK, Ollama, or Docker.
Optional:
--install-dir <dir> Install directory; defaults to $DEFAULT_INSTALL_DIR
--service-name <n> systemd service name; defaults to $DEFAULT_SERVICE_NAME
-h, --help Show this help message
Examples:
$0
$0 --install-dir /opt/ngino-client --service-name ngino-client
EOF
exit 0
}
# ── Argument parsing ──────────────────────────────────────────────────────────
while [[ $# -gt 0 ]]; do
case "$1" in
--install-dir) INSTALL_DIR="$2"; shift 2 ;;
--service-name) SERVICE_NAME="$2"; shift 2 ;;
-h|--help) usage ;;
*) die "Unknown option: $1" ;;
esac
done
# ── Root check ────────────────────────────────────────────────────────────────
if [[ $EUID -ne 0 ]]; then
die "This script must be run as root (or with sudo)."
fi
SERVICE_FILE="/etc/systemd/system/${SERVICE_NAME}.service"
ENV_DIR="/etc/ngino-client"
# ── Stop and disable the service ──────────────────────────────────────────────
if systemctl list-unit-files "$SERVICE_NAME.service" &>/dev/null 2>&1; then
info "Stopping service $SERVICE_NAME..."
systemctl stop "$SERVICE_NAME" 2>/dev/null || true
info "Disabling service $SERVICE_NAME..."
systemctl disable "$SERVICE_NAME" 2>/dev/null || true
fi
# ── Remove service unit file ──────────────────────────────────────────────────
if [[ -f "$SERVICE_FILE" ]]; then
info "Removing service unit file $SERVICE_FILE..."
rm -f "$SERVICE_FILE"
fi
systemctl daemon-reload
# ── Remove install directory ──────────────────────────────────────────────────
if [[ -d "$INSTALL_DIR" ]]; then
info "Removing install directory $INSTALL_DIR..."
rm -rf "$INSTALL_DIR"
fi
# ── Remove environment file ───────────────────────────────────────────────────
if [[ -d "$ENV_DIR" ]]; then
info "Removing environment directory $ENV_DIR..."
rm -rf "$ENV_DIR"
fi
# ── Done ──────────────────────────────────────────────────────────────────────
echo
info "Uninstall complete."
echo " Service: $SERVICE_NAME (stopped, disabled, unit removed)"
echo " Install dir: $INSTALL_DIR (removed)"
echo " Env dir: $ENV_DIR (removed)"
+332
View File
@@ -0,0 +1,332 @@
<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<svg
viewBox="0 0 1280 720"
role="img"
aria-labelledby="title desc"
version="1.1"
id="svg10"
sodipodi:docname="Ngino_logo.svg"
xml:space="preserve"
inkscape:version="1.4.3 (0d15f75042, 2025-12-25)"
inkscape:export-filename="Ngino_logo_full_dark.png"
inkscape:export-xdpi="125.532"
inkscape:export-ydpi="125.532"
xmlns:inkscape="http://www.inkscape.org/namespaces/inkscape"
xmlns:sodipodi="http://sodipodi.sourceforge.net/DTD/sodipodi-0.dtd"
xmlns:xlink="http://www.w3.org/1999/xlink"
xmlns="http://www.w3.org/2000/svg"
xmlns:svg="http://www.w3.org/2000/svg"
xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#"
xmlns:cc="http://creativecommons.org/ns#"
xmlns:dc="http://purl.org/dc/elements/1.1/"><sodipodi:namedview
id="namedview10"
pagecolor="#000000"
bordercolor="#000000"
borderopacity="1"
inkscape:showpageshadow="2"
inkscape:pageopacity="0.0"
inkscape:pagecheckerboard="0"
inkscape:deskcolor="#000000"
inkscape:zoom="0.7671875"
inkscape:cx="391.69043"
inkscape:cy="406.02851"
inkscape:window-width="2560"
inkscape:window-height="1412"
inkscape:window-x="0"
inkscape:window-y="0"
inkscape:window-maximized="1"
inkscape:current-layer="layer4"
inkscape:export-bgcolor="#00000000" /><title
id="title">Ngino logo</title><desc
id="desc">A fully vector llama outline with one input connection branching to three AI endpoints, beside the Ngino wordmark.</desc><defs
id="defs2"><linearGradient
id="swatch181"><stop
style="stop-color:#000000;stop-opacity:1;"
offset="0"
id="stop181" /></linearGradient><clipPath
clipPathUnits="userSpaceOnUse"
id="clipPath15"><rect
style="fill:none;stroke:#000000;stroke-linecap:butt;stroke-linejoin:bevel;paint-order:stroke markers fill;stop-color:#000000"
id="rect15-7"
width="45.034248"
height="38.955639"
x="262.48285"
y="121.00073" /></clipPath><linearGradient
id="swatch7"
inkscape:swatch="solid"><stop
style="stop-color:#000000;stop-opacity:1;"
offset="0"
id="stop7" /></linearGradient><linearGradient
id="linearGradient1"
inkscape:swatch="gradient"><stop
style="stop-color:#000000;stop-opacity:0;"
offset="0"
id="stop3" /><stop
style="stop-color:#000000;stop-opacity:0;"
offset="1"
id="stop4" /></linearGradient><linearGradient
id="routeGradient"
x1="95"
y1="0"
x2="430"
y2="0"
gradientUnits="userSpaceOnUse"><stop
offset="0"
stop-color="#0697b5"
id="stop1" /><stop
offset="1"
stop-color="#45d05f"
id="stop2" /></linearGradient><linearGradient
inkscape:collect="always"
xlink:href="#routeGradient"
id="linearGradient10"
gradientUnits="userSpaceOnUse"
x1="95"
y1="0"
x2="430"
y2="0"
gradientTransform="translate(50.835031)" /><linearGradient
inkscape:collect="always"
xlink:href="#routeGradient"
id="linearGradient24"
gradientUnits="userSpaceOnUse"
x1="95"
y1="0"
x2="430"
y2="0" /><linearGradient
inkscape:collect="always"
xlink:href="#routeGradient"
id="linearGradient25"
gradientUnits="userSpaceOnUse"
x1="95"
y1="0"
x2="430"
y2="0" /><linearGradient
inkscape:collect="always"
xlink:href="#routeGradient"
id="linearGradient26"
gradientUnits="userSpaceOnUse"
x1="95"
y1="0"
x2="430"
y2="0" /><linearGradient
inkscape:collect="always"
xlink:href="#routeGradient"
id="linearGradient27"
gradientUnits="userSpaceOnUse"
x1="95"
y1="0"
x2="430"
y2="0" /><linearGradient
inkscape:collect="always"
xlink:href="#routeGradient"
id="linearGradient28"
gradientUnits="userSpaceOnUse"
x1="95"
y1="0"
x2="430"
y2="0"
gradientTransform="translate(18.248473,-20.855397)" /><linearGradient
inkscape:collect="always"
xlink:href="#routeGradient"
id="linearGradient29"
gradientUnits="userSpaceOnUse"
x1="95"
y1="0"
x2="430"
y2="0"
gradientTransform="translate(18.248473,-20.855397)" /><linearGradient
inkscape:collect="always"
xlink:href="#routeGradient"
id="linearGradient30"
gradientUnits="userSpaceOnUse"
x1="95"
y1="0"
x2="430"
y2="0"
gradientTransform="translate(18.248473,-20.855397)" /><linearGradient
inkscape:collect="always"
xlink:href="#routeGradient"
id="linearGradient31"
gradientUnits="userSpaceOnUse"
x1="95"
y1="0"
x2="430"
y2="0"
gradientTransform="translate(18.248473,-20.855397)" /><filter
inkscape:collect="always"
style="color-interpolation-filters:sRGB"
id="filter215"
x="-0.0021624053"
y="-0.0047179751"
width="1.0043248"
height="1.009436"><feGaussianBlur
inkscape:collect="always"
stdDeviation="1.0812026"
id="feGaussianBlur215" /></filter><filter
inkscape:collect="always"
style="color-interpolation-filters:sRGB"
id="filter215-8"
x="-0.003844276"
y="-0.0074139609"
width="1.0076886"
height="1.0148279"><feGaussianBlur
inkscape:collect="always"
stdDeviation="1.0812026"
id="feGaussianBlur215-8" /></filter><filter
inkscape:collect="always"
style="color-interpolation-filters:sRGB"
id="filter215-6"
x="-0.0051897726"
y="-0.0047179751"
width="1.0103795"
height="1.009436"><feGaussianBlur
inkscape:collect="always"
stdDeviation="1.0812026"
id="feGaussianBlur215-0" /></filter></defs><!-- Transparent background; no embedded raster image. --><rect
style="display:inline;opacity:1;fill:#ffffff;fill-opacity:1;stroke:none;stroke-width:50;stroke-linecap:round;stroke-miterlimit:0.8;stroke-dasharray:none;stroke-opacity:1;filter:url(#filter215)"
id="rect1"
width="1200"
height="550"
x="26.069246"
y="20.855396"
inkscape:label="background_full"
rx="100"
inkscape:highlight-color="#000000"
inkscape:export-filename="Ngino_logo_full.png"
inkscape:export-xdpi="153.60001"
inkscape:export-ydpi="153.60001" /><g
inkscape:groupmode="layer"
id="layer1"
inkscape:label="text"
style="display:inline"
inkscape:export-filename="Ngino_logo_text.png"
inkscape:export-xdpi="125.532"
inkscape:export-ydpi="125.532"><rect
style="display:inline;fill:#ffffff;fill-opacity:1;stroke:none;stroke-width:50;stroke-linecap:round;stroke-miterlimit:0.8;stroke-dasharray:none;stroke-opacity:1;filter:url(#filter215-8)"
id="rect1-1"
width="675"
height="350"
x="458.96759"
y="152.75374"
inkscape:label="background_text"
rx="100"
inkscape:highlight-color="#000000"
inkscape:export-filename="Ngino_logo_full.png"
inkscape:export-xdpi="153.60001"
inkscape:export-ydpi="153.60001" /><text
xml:space="preserve"
style="font-style:normal;font-variant:normal;font-weight:normal;font-stretch:normal;font-size:37.3333px;line-height:1.25;font-family:Sans;-inkscape-font-specification:'Sans, Normal';font-variant-ligatures:normal;font-variant-caps:normal;font-variant-numeric:normal;font-variant-east-asian:normal;display:inline;fill:#000000;fill-opacity:1"
x="503.97342"
y="430.72964"
id="text2"
inkscape:label="sub"><tspan
sodipodi:role="line"
id="tspan2"
x="503.97342"
y="430.72964">R E V E R S E P R O X Y F O R A I</tspan></text><text
xml:space="preserve"
style="font-weight:900;font-size:170.667px;line-height:1.25;font-family:Montserrat;-inkscape-font-specification:'Montserrat, Heavy';display:inline;fill:#000000;fill-opacity:1"
x="496.78931"
y="357.61456"
id="text1"
inkscape:label="main"
inkscape:export-filename="Ngino_logo_text.png"
inkscape:export-xdpi="125.532"
inkscape:export-ydpi="125.532"><tspan
sodipodi:role="line"
id="tspan1"
x="496.78931"
y="357.61456">Ngino</tspan></text></g><!-- Connection nodes --><!-- Wordmark --><rect
style="display:inline;fill:#ffffff;fill-opacity:1;stroke:none;stroke-width:50;stroke-linecap:round;stroke-miterlimit:0.8;stroke-dasharray:none;stroke-opacity:1;filter:url(#filter215-6)"
id="rect1-8"
width="500"
height="550"
x="28.825022"
y="21.104044"
inkscape:label="background_logo"
rx="100"
inkscape:highlight-color="#000000"
inkscape:export-filename="Ngino_logo_full.png"
inkscape:export-xdpi="153.60001"
inkscape:export-ydpi="153.60001" /><g
inkscape:groupmode="layer"
id="layer2"
inkscape:label="Logo"
style="display:inline"
inkscape:export-filename="Ngino_Logo_Symbol.png"
inkscape:export-xdpi="125.532"
inkscape:export-ydpi="125.532"><g
inkscape:groupmode="layer"
id="layer3"
inkscape:label="Body"
style="display:inline"><path
d="m 115.82078,398.64766 c -22.393081,-46.4277 -14.055,-69.1446 2.64154,-97.02036 C 143.97964,277.26884 147,275.30346 190,275.30346 h 56 c 36,0 53.48269,-18.60692 63.48269,-46.60692 6,-16 4,-43 5,-65 0,-20 6,-37.30346 16,-52.30346 l 0.39308,-26.965381 C 333.17923,71.427699 347,53 356,48 l 2,47 c 6,-19 17,-34 31,-40 l -1.69653,38.268839 c 13,3 22.69653,17.517311 28.69653,28.517311 18,3 29,6.17923 31,22.17923 0,14 -3.78615,18.08961 -16.78615,20.08961 l -37,7.21385 c 12,24 28.91039,76.76578 36.91039,108.76578 14,58 -6.24848,146.8554 -29.24848,202.8554 l -13,29 h -47 l -8,-35 c -4,-22 -23,-38 -48,-38 h -53.50305 c -27,0 -45,16 -49,40 l -6,33 h -47.12424 z"
id="path2"
sodipodi:nodetypes="ccsscccccccccccccccccsscccc"
style="display:inline;fill:none;stroke:url(#linearGradient27)"
stroke-width="13"
inkscape:label="body" /></g><g
inkscape:groupmode="layer"
id="layer4"
inkscape:label="Symbol"
style="display:inline"><g
fill="url(#routeGradient)"
id="g9"
style="display:inline;fill:url(#linearGradient27)"
transform="translate(18.248473,-20.855397)"
inkscape:label="Symbol_circles"><circle
cx="354"
cy="398"
r="15"
id="circle9"
style="fill:url(#linearGradient26)"
inkscape:label="ball_fork_down" /><circle
cx="354"
cy="356"
r="15"
id="circle8"
style="display:inline;fill:url(#linearGradient25)"
inkscape:label="ball_fork_mid" /><circle
cx="354"
cy="314"
r="15"
id="circle7"
style="display:inline;fill:url(#linearGradient24)"
inkscape:label="ball_fork_up" /><circle
cx="130.83504"
cy="356"
r="15"
id="circle6"
style="display:inline;fill:url(#linearGradient10)"
inkscape:label="ball_connector" /></g><g
fill="none"
stroke="url(#routeGradient)"
stroke-width="13"
stroke-linecap="round"
stroke-linejoin="round"
id="g6"
style="display:inline"
inkscape:label="Symbol_paths"><!-- Llama silhouette --><!-- One input, three routed endpoints --><path
d="m 254.24847,335.1446 c 29,0 42,42 78,42 h 40"
id="path6"
style="display:inline;stroke:url(#linearGradient31)"
inkscape:label="fork_down" /><path
d="m 254.24847,335.1446 h 118"
id="path5"
style="display:inline;stroke:url(#linearGradient28)"
inkscape:label="fork_mid" /><path
d="m 254.24847,335.1446 c 29,0 42,-42 78,-42 h 40"
id="path4"
style="display:inline;stroke:url(#linearGradient30)"
inkscape:label="fork_up" /><path
d="M 150.38696,335.1446 H 254.24847"
id="path3"
sodipodi:nodetypes="cc"
style="display:inline;stroke:url(#linearGradient29)"
inkscape:label="connector"
inkscape:export-filename="Ngino_logo_symbol.png"
inkscape:export-xdpi="153.60001"
inkscape:export-ydpi="153.60001" /></g></g></g><metadata
id="metadata215"><rdf:RDF><cc:Work
rdf:about=""><dc:title>Ngino logo</dc:title></cc:Work></rdf:RDF></metadata></svg>

After

Width:  |  Height:  |  Size: 13 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 95 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 56 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 38 KiB

+20 -1
View File
@@ -28,6 +28,12 @@ internal sealed class ClientOptions
public int LlamaCppBasePort { get; init; } = 8081; public int LlamaCppBasePort { get; init; } = 8081;
public int? LlamaCppParallel { get; init; }
public TimeSpan LlamaCppFallbackCooldown { get; init; } = TimeSpan.FromMinutes(3);
public string? LogDirectory { get; init; }
public Uri TunnelUri public Uri TunnelUri
{ {
get get
@@ -68,7 +74,10 @@ internal sealed class ClientOptions
UseLlamaCppViaDocker = ReadBool(values, false, "use-llama-cpp-via-docker", "NGINO_USE_LLAMA_CPP_VIA_DOCKER"), UseLlamaCppViaDocker = ReadBool(values, false, "use-llama-cpp-via-docker", "NGINO_USE_LLAMA_CPP_VIA_DOCKER"),
UseOllamaModelsPath = NormalizeDirectoryPath(Read(values, "use-ollama-models-path", "NGINO_USE_OLLAMA_MODELS_PATH")), UseOllamaModelsPath = NormalizeDirectoryPath(Read(values, "use-ollama-models-path", "NGINO_USE_OLLAMA_MODELS_PATH")),
LlamaCppDockerImage = Read(values, "llama-cpp-docker-image", "NGINO_LLAMA_CPP_DOCKER_IMAGE"), LlamaCppDockerImage = Read(values, "llama-cpp-docker-image", "NGINO_LLAMA_CPP_DOCKER_IMAGE"),
LlamaCppBasePort = ReadInt(values, 8081, "llama-cpp-base-port", "NGINO_LLAMA_CPP_BASE_PORT") LlamaCppBasePort = ReadInt(values, 8081, "llama-cpp-base-port", "NGINO_LLAMA_CPP_BASE_PORT"),
LlamaCppParallel = ReadOptionalInt(values, "llama-cpp-parallel", "NGINO_LLAMA_CPP_PARALLEL"),
LlamaCppFallbackCooldown = TimeSpan.FromSeconds(ReadInt(values, 180, "llama-cpp-fallback-cooldown", "NGINO_LLAMA_CPP_FALLBACK_COOLDOWN_SECONDS")),
LogDirectory = NormalizeDirectoryPath(Read(values, "log-dir", "NGINO_LOG_DIR"))
}; };
} }
@@ -87,6 +96,10 @@ internal sealed class ClientOptions
--use-ollama-models-path <dir> Path to Ollama models directory (manifests/blobs), required with --use-llama-cpp-via-docker --use-ollama-models-path <dir> Path to Ollama models directory (manifests/blobs), required with --use-llama-cpp-via-docker
--llama-cpp-docker-image <img> llama.cpp Docker image; defaults to auto-detected (rocm/cuda/cpu) --llama-cpp-docker-image <img> llama.cpp Docker image; defaults to auto-detected (rocm/cuda/cpu)
--llama-cpp-base-port <num> Base port for llama.cpp containers; defaults to 8081 --llama-cpp-base-port <num> Base port for llama.cpp containers; defaults to 8081
--llama-cpp-parallel <num> llama.cpp parallel slots per container; if unset, llama.cpp's own default is used
--llama-cpp-fallback-cooldown <sec>
Seconds before llama.cpp is retried after a failed container start; defaults to 180
--log-dir <dir> Directory for log files; defaults to <app dir>/Logs
"""; """;
private static Dictionary<string, string> ParseArgs(string[] args) private static Dictionary<string, string> ParseArgs(string[] args)
@@ -160,6 +173,12 @@ internal sealed class ClientOptions
return int.TryParse(value, out var parsed) && parsed > 0 ? parsed : fallback; return int.TryParse(value, out var parsed) && parsed > 0 ? parsed : fallback;
} }
private static int? ReadOptionalInt(Dictionary<string, string> values, params string[] keys)
{
var value = Read(values, keys);
return int.TryParse(value, out var parsed) && parsed > 0 ? parsed : null;
}
private static bool ReadBool(Dictionary<string, string> values, bool fallback, params string[] keys) private static bool ReadBool(Dictionary<string, string> values, bool fallback, params string[] keys)
{ {
var value = Read(values, keys); var value = Read(values, keys);
+113
View File
@@ -0,0 +1,113 @@
using System.Text;
using Microsoft.Extensions.Logging;
namespace Ngino.Client;
internal sealed class FileLoggerProvider : ILoggerProvider
{
private const long DefaultMaxFileSizeBytes = 5L * 1024 * 1024;
private const string LogFileName = "ngino-client.log";
private const string RotatedLogFileName = "ngino-client.log.1";
private readonly string _directory;
private readonly long _maxFileSizeBytes;
private readonly object _lock = new();
private StreamWriter _writer = null!;
private string _currentFile = null!;
public FileLoggerProvider(string directory, long maxFileSizeBytes = DefaultMaxFileSizeBytes)
{
_directory = directory;
_maxFileSizeBytes = maxFileSizeBytes;
Directory.CreateDirectory(directory);
OpenFile();
}
public string LogDirectory => _directory;
public ILogger CreateLogger(string categoryName) => new FileLogger(this, categoryName);
public void WriteLog(DateTime timestamp, LogLevel level, string category, string message)
{
lock (_lock)
{
var line = $"{timestamp:yyyy-MM-dd HH:mm:ss.fff} [{level}] {category}: {message}";
if (_writer.BaseStream.Length + line.Length + 2 > _maxFileSizeBytes)
{
RotateFile();
}
_writer.WriteLine(line);
}
}
public void Dispose()
{
lock (_lock)
{
_writer.Dispose();
}
}
private void OpenFile()
{
_currentFile = Path.Combine(_directory, LogFileName);
_writer = new StreamWriter(
new FileStream(_currentFile, FileMode.Append, FileAccess.Write, FileShare.ReadWrite),
new UTF8Encoding(encoderShouldEmitUTF8Identifier: false))
{
AutoFlush = true
};
}
private void RotateFile()
{
_writer.Dispose();
var rotatedFile = Path.Combine(_directory, RotatedLogFileName);
try
{
File.Delete(rotatedFile);
if (File.Exists(_currentFile))
{
File.Move(_currentFile, rotatedFile);
}
}
catch (IOException)
{
}
catch (UnauthorizedAccessException)
{
}
OpenFile();
}
}
internal sealed class FileLogger(FileLoggerProvider provider, string category) : ILogger
{
public IDisposable? BeginScope<TState>(TState state) where TState : notnull => null;
public bool IsEnabled(LogLevel logLevel) => logLevel >= LogLevel.Trace;
public void Log<TState>(
LogLevel logLevel,
EventId eventId,
TState state,
Exception? exception,
Func<TState, Exception?, string> formatter)
{
if (!IsEnabled(logLevel))
{
return;
}
var message = formatter(state, exception);
if (exception is not null)
{
message += Environment.NewLine + exception;
}
provider.WriteLog(DateTime.Now, logLevel, category, message);
}
}
+354 -29
View File
@@ -1,5 +1,7 @@
using System.Collections.Concurrent; using System.Collections.Concurrent;
using System.Diagnostics; using System.Diagnostics;
using System.Net.Http;
using System.Net.Sockets;
using System.Text.Json; using System.Text.Json;
using System.Text.RegularExpressions; using System.Text.RegularExpressions;
using Microsoft.Extensions.Logging; using Microsoft.Extensions.Logging;
@@ -13,24 +15,36 @@ internal sealed partial class LlamaCppManager : IAsyncDisposable
private const int DefaultBasePort = 8081; private const int DefaultBasePort = 8081;
private const string NginoContainerLabel = "ngino-llamacpp"; private const string NginoContainerLabel = "ngino-llamacpp";
private static readonly TimeSpan DockerTimeout = TimeSpan.FromSeconds(60); private static readonly TimeSpan DockerTimeout = TimeSpan.FromSeconds(60);
private static readonly TimeSpan ContainerStartTimeout = TimeSpan.FromMinutes(5);
private static readonly TimeSpan DefaultFallbackCooldown = TimeSpan.FromMinutes(3);
private readonly string _blobsPath; private readonly string _blobsPath;
private readonly string _manifestsPath; private readonly string _manifestsPath;
private readonly string _dockerImage; private readonly string _dockerImage;
private readonly int _basePort; private readonly int _basePort;
private readonly int? _parallel;
private readonly TimeSpan _fallbackCooldown;
private readonly ILogger _logger; private readonly ILogger _logger;
private readonly ConcurrentDictionary<string, int> _modelPorts = new(StringComparer.OrdinalIgnoreCase); private readonly ConcurrentDictionary<string, int> _modelPorts = new(StringComparer.OrdinalIgnoreCase);
private readonly ConcurrentDictionary<int, byte> _reservedPorts = new();
private readonly ConcurrentDictionary<string, DateTime> _fallbackModels = new(StringComparer.OrdinalIgnoreCase);
private readonly ConcurrentDictionary<string, SemaphoreSlim> _modelStartLocks = new(StringComparer.OrdinalIgnoreCase);
private readonly object _portAllocationLock = new();
public LlamaCppManager( public LlamaCppManager(
string ollamaModelsPath, string ollamaModelsPath,
string? dockerImage, string? dockerImage,
int? basePort, int? basePort,
ILogger? logger = null) ILogger? logger = null,
TimeSpan fallbackCooldown = default,
int? parallel = null)
{ {
_manifestsPath = Path.Combine(ollamaModelsPath, "manifests"); _manifestsPath = Path.Combine(ollamaModelsPath, "manifests");
_blobsPath = Path.Combine(ollamaModelsPath, "blobs"); _blobsPath = Path.Combine(ollamaModelsPath, "blobs");
_dockerImage = dockerImage ?? GetDefaultDockerImage(); _dockerImage = dockerImage ?? GetDefaultDockerImage();
_basePort = basePort ?? DefaultBasePort; _basePort = basePort ?? DefaultBasePort;
_fallbackCooldown = fallbackCooldown > TimeSpan.Zero ? fallbackCooldown : DefaultFallbackCooldown;
_parallel = parallel is > 0 ? parallel : null;
_logger = logger ?? NullLogger<LlamaCppManager>.Instance; _logger = logger ?? NullLogger<LlamaCppManager>.Instance;
} }
@@ -75,6 +89,82 @@ internal sealed partial class LlamaCppManager : IAsyncDisposable
return _modelPorts.ContainsKey(ollamaModelName); return _modelPorts.ContainsKey(ollamaModelName);
} }
public void MarkModelAsFallback(string ollamaModelName)
{
_fallbackModels[ollamaModelName] = DateTime.UtcNow.Add(_fallbackCooldown);
_logger.LogWarning(
"Model {Model} will fall back to the Ollama upstream for {Cooldown} before llama.cpp is retried.",
ollamaModelName, _fallbackCooldown);
}
public void MarkModelAsPermanentFallback(string ollamaModelName)
{
_fallbackModels[ollamaModelName] = DateTime.MaxValue;
_logger.LogError(
"Model {Model} exited its llama.cpp container before becoming ready. It will fall back to the Ollama upstream until it is unloaded.",
ollamaModelName);
}
public bool IsModelOnFallback(string ollamaModelName)
{
if (_fallbackModels.TryGetValue(ollamaModelName, out var expiresAt))
{
if (expiresAt > DateTime.UtcNow)
{
return true;
}
_fallbackModels.TryRemove(ollamaModelName, out _);
}
return false;
}
public void ClearModelFallback(string ollamaModelName)
{
if (_fallbackModels.TryRemove(ollamaModelName, out _))
{
_logger.LogInformation("Cleared llama.cpp fallback marker for model {Model}.", ollamaModelName);
}
}
public async Task<bool> IsContainerRunningAsync(string ollamaModelName)
{
if (!_modelPorts.ContainsKey(ollamaModelName))
{
return false;
}
var containerName = SanitizeContainerName($"ngino-llamacpp-{ollamaModelName}");
var (exitCode, output) = await RunDockerWithOutputAsync(
["ps", "--filter", $"name=^{containerName}$", "--format", "{{.ID}}"],
CancellationToken.None);
var running = exitCode == 0 && !string.IsNullOrWhiteSpace(output);
if (!running)
{
_logger.LogWarning(
"llama.cpp container {ContainerName} is no longer running. Invalidating cached port for {Model}.",
containerName, ollamaModelName);
RemoveModelPort(ollamaModelName);
}
return running;
}
public bool RemoveModelMapping(string ollamaModelName)
{
if (RemoveModelPort(ollamaModelName))
{
_logger.LogWarning(
"Removed stale llama.cpp port mapping for {Model}.",
ollamaModelName);
return true;
}
return false;
}
public Uri? GetUpstream(string ollamaModelName) public Uri? GetUpstream(string ollamaModelName)
{ {
if (_modelPorts.TryGetValue(ollamaModelName, out var port)) if (_modelPorts.TryGetValue(ollamaModelName, out var port))
@@ -94,62 +184,130 @@ internal sealed partial class LlamaCppManager : IAsyncDisposable
return false; return false;
} }
var startLock = _modelStartLocks.GetOrAdd(ollamaName, static _ => new SemaphoreSlim(1, 1));
await startLock.WaitAsync(cancellationToken);
try
{
return await StartModelContainerCoreAsync(model, cancellationToken);
}
finally
{
startLock.Release();
}
}
private async Task<bool> StartModelContainerCoreAsync(LlamaCppModel model, CancellationToken cancellationToken)
{
var ollamaName = model.OllamaName;
if (string.IsNullOrWhiteSpace(ollamaName))
{
return false;
}
if (_modelPorts.ContainsKey(ollamaName)) if (_modelPorts.ContainsKey(ollamaName))
{ {
_logger.LogInformation("Model {Model} already has a running container", ollamaName); _logger.LogInformation("Model {Model} already has a running container", ollamaName);
return true; return true;
} }
if (IsModelOnFallback(ollamaName))
{
_logger.LogInformation(
"Model {Model} previously failed to load via llama.cpp. Skipping container start.",
ollamaName);
return false;
}
if (!File.Exists(model.BlobPath)) if (!File.Exists(model.BlobPath))
{ {
_logger.LogError("Model blob not found: {BlobPath}", model.BlobPath); _logger.LogError("Model blob not found: {BlobPath}", model.BlobPath);
MarkModelAsFallback(ollamaName);
return false; return false;
} }
var port = FindAvailablePort(); var port = FindAvailablePort();
var containerName = SanitizeContainerName($"ngino-llamacpp-{ollamaName}"); var containerName = SanitizeContainerName($"ngino-llamacpp-{ollamaName}");
var existingPort = await FindExistingContainerPortAsync(containerName);
if (existingPort.HasValue)
{
_logger.LogInformation(
"Reusing existing container for {Model} on port {Port}", ollamaName, existingPort.Value);
_modelPorts[ollamaName] = existingPort.Value;
return true;
}
await RunDockerAsync(["rm", "-f", containerName], CancellationToken.None);
var args = BuildDockerRunArgs(containerName, model, port);
_logger.LogInformation(
"Starting llama.cpp container for {Model} on port {Port}: docker {Args}",
ollamaName, port, string.Join(" ", args));
try try
{ {
var existingPort = await FindExistingContainerPortAsync(containerName);
if (existingPort.HasValue)
{
_logger.LogInformation(
"Reusing existing container for {Model} on port {Port}", ollamaName, existingPort.Value);
_modelPorts[ollamaName] = existingPort.Value;
return true;
}
await RunDockerAsync(["rm", "-f", containerName], CancellationToken.None);
var args = BuildDockerRunArgs(containerName, model, port);
_logger.LogInformation(
"Starting llama.cpp container for {Model} on port {Port}: docker {Args}",
ollamaName, port, string.Join(" ", args));
var (exitCode, output) = await RunDockerWithOutputAsync(args, cancellationToken); var (exitCode, output) = await RunDockerWithOutputAsync(args, cancellationToken);
if (exitCode != 0) if (exitCode != 0)
{ {
_logger.LogError( _logger.LogError(
"Failed to start llama.cpp container for {Model}, exit code: {ExitCode}, output: {Output}", "Failed to start llama.cpp container for {Model}, exit code: {ExitCode}, output: {Output}",
ollamaName, exitCode, output); ollamaName, exitCode, output);
MarkModelAsFallback(ollamaName);
return false;
}
_logger.LogInformation(
"llama.cpp container for {Model} started on port {Port}. Waiting for it to become ready...",
ollamaName, port);
var result = await WaitForServerReadyAsync("localhost", port, containerName, cancellationToken);
if (result != ContainerStartResult.Ready)
{
if (result == ContainerStartResult.ContainerExited)
{
MarkModelAsPermanentFallback(ollamaName);
}
else
{
MarkModelAsFallback(ollamaName);
}
_logger.LogError(
"llama.cpp container for {Model} did not become ready on port {Port} within {Timeout} ({Result}). Stopping it.",
ollamaName, port, ContainerStartTimeout, result);
try
{
await RunDockerAsync(["stop", "--time", "10", containerName], CancellationToken.None);
await RunDockerAsync(["rm", "-f", containerName], CancellationToken.None);
}
catch (Exception cleanupException)
{
_logger.LogWarning(cleanupException, "Failed to clean up container {ContainerName}", containerName);
}
return false; return false;
} }
_modelPorts[ollamaName] = port; _modelPorts[ollamaName] = port;
_logger.LogInformation("llama.cpp container for {Model} started on port {Port}", ollamaName, port); _logger.LogInformation("llama.cpp container for {Model} is ready on port {Port}.", ollamaName, port);
return true; return true;
} }
catch (Exception ex) catch (Exception ex)
{ {
_logger.LogError(ex, "Failed to start llama.cpp container for {Model}", ollamaName); _logger.LogError(ex, "Failed to start llama.cpp container for {Model}", ollamaName);
MarkModelAsFallback(ollamaName);
return false; return false;
} }
finally
{
ReleaseReservedPort(port);
}
} }
public async Task<bool> StopModelContainerAsync(string ollamaModelName, CancellationToken cancellationToken) public async Task<bool> StopModelContainerAsync(string ollamaModelName, CancellationToken cancellationToken)
{ {
if (!_modelPorts.TryRemove(ollamaModelName, out _)) if (!RemoveModelPort(ollamaModelName))
{ {
_logger.LogWarning("No running container found for model {Model}", ollamaModelName); _logger.LogWarning("No running container found for model {Model}", ollamaModelName);
return false; return false;
@@ -163,6 +321,7 @@ internal sealed partial class LlamaCppManager : IAsyncDisposable
{ {
await RunDockerAsync(["stop", "--time", "10", containerName], cancellationToken); await RunDockerAsync(["stop", "--time", "10", containerName], cancellationToken);
await RunDockerAsync(["rm", "-f", containerName], cancellationToken); await RunDockerAsync(["rm", "-f", containerName], cancellationToken);
ClearModelFallback(ollamaModelName);
return true; return true;
} }
catch (Exception ex) catch (Exception ex)
@@ -198,6 +357,8 @@ internal sealed partial class LlamaCppManager : IAsyncDisposable
} }
_modelPorts.Clear(); _modelPorts.Clear();
_reservedPorts.Clear();
_fallbackModels.Clear();
} }
public async Task<bool> TestDockerAsync() public async Task<bool> TestDockerAsync()
@@ -325,7 +486,6 @@ internal sealed partial class LlamaCppManager : IAsyncDisposable
{ {
"run", "run",
"-d", "-d",
"--rm",
"--label", $"{NginoContainerLabel}=true", "--label", $"{NginoContainerLabel}=true",
"--name", containerName, "--name", containerName,
"-p", $"{port}:{port}", "-p", $"{port}:{port}",
@@ -349,9 +509,12 @@ internal sealed partial class LlamaCppManager : IAsyncDisposable
args.Add("-m"); args.Add("-m");
args.Add($"/models/blobs/{blobFile}"); args.Add($"/models/blobs/{blobFile}");
args.Add("-ngl"); args.Add("-ngl");
args.Add("999"); args.Add("auto");
args.Add("--parallel"); if (_parallel.HasValue)
args.Add("4"); {
args.Add("--parallel");
args.Add(_parallel.Value.ToString());
}
args.Add("--host"); args.Add("--host");
args.Add("0.0.0.0"); args.Add("0.0.0.0");
args.Add("--port"); args.Add("--port");
@@ -412,15 +575,170 @@ internal sealed partial class LlamaCppManager : IAsyncDisposable
private int FindAvailablePort() private int FindAvailablePort()
{ {
var usedPorts = new HashSet<int>(_modelPorts.Values); lock (_portAllocationLock)
var port = _basePort;
while (usedPorts.Contains(port))
{ {
port++; var usedPorts = new HashSet<int>(_modelPorts.Values);
foreach (var reservedPort in _reservedPorts.Keys)
{
usedPorts.Add(reservedPort);
}
var port = _basePort;
while (usedPorts.Contains(port))
{
port++;
}
_reservedPorts[port] = 0;
return port;
}
}
private void ReleaseReservedPort(int port)
{
_reservedPorts.TryRemove(port, out _);
}
private bool RemoveModelPort(string ollamaModelName)
{
if (_modelPorts.TryRemove(ollamaModelName, out var port))
{
ReleaseReservedPort(port);
return true;
} }
return port; return false;
}
private async Task<ContainerStartResult> WaitForServerReadyAsync(
string host, int port, string containerName, CancellationToken cancellationToken)
{
var deadline = DateTime.UtcNow + ContainerStartTimeout;
var tcpResult = await WaitForTcpPortAsync(host, port, containerName, deadline, cancellationToken);
if (tcpResult != ContainerStartResult.Ready)
{
return tcpResult;
}
using var handler = new SocketsHttpHandler
{
ConnectTimeout = TimeSpan.FromSeconds(3),
UseProxy = false
};
using var httpClient = new HttpClient(handler) { Timeout = TimeSpan.FromSeconds(10) };
while (DateTime.UtcNow < deadline)
{
cancellationToken.ThrowIfCancellationRequested();
if (!await IsDockerContainerRunningAsync(containerName))
{
await LogContainerOutputAsync(containerName);
return ContainerStartResult.ContainerExited;
}
try
{
using var response = await httpClient.GetAsync(
$"http://{host}:{port}/health", cancellationToken);
if (response.IsSuccessStatusCode)
{
return ContainerStartResult.Ready;
}
}
catch (OperationCanceledException) when (!cancellationToken.IsCancellationRequested)
{
}
catch (HttpRequestException ex)
{
_logger.LogDebug(ex, "Health probe of llama.cpp container {ContainerName} failed; retrying.", containerName);
}
catch (IOException ex)
{
_logger.LogDebug(ex, "Health probe of llama.cpp container {ContainerName} failed; retrying.", containerName);
}
await Task.Delay(TimeSpan.FromSeconds(2), cancellationToken);
}
return ContainerStartResult.TimedOut;
}
private async Task<ContainerStartResult> WaitForTcpPortAsync(
string host, int port, string containerName, DateTime deadline, CancellationToken cancellationToken)
{
while (DateTime.UtcNow < deadline)
{
cancellationToken.ThrowIfCancellationRequested();
if (!await IsDockerContainerRunningAsync(containerName))
{
await LogContainerOutputAsync(containerName);
return ContainerStartResult.ContainerExited;
}
try
{
using var client = new TcpClient();
await client.ConnectAsync(host, port, cancellationToken);
return ContainerStartResult.Ready;
}
catch (OperationCanceledException)
{
throw;
}
catch (SocketException)
{
}
await Task.Delay(TimeSpan.FromSeconds(2), cancellationToken);
}
return ContainerStartResult.TimedOut;
}
private async Task<bool> IsDockerContainerRunningAsync(string containerName)
{
try
{
var (exitCode, output) = await RunDockerWithOutputAsync(
["inspect", "-f", "{{.State.Running}}", containerName],
CancellationToken.None);
return exitCode == 0 && string.Equals(output.Trim(), "true", StringComparison.OrdinalIgnoreCase);
}
catch
{
return true;
}
}
private async Task LogContainerOutputAsync(string containerName)
{
try
{
var (_, output) = await RunDockerWithOutputAsync(
["logs", "--tail", "100", containerName],
CancellationToken.None);
if (!string.IsNullOrWhiteSpace(output))
{
_logger.LogError(
"llama.cpp container {ContainerName} exited before becoming ready. Last output:\n{Output}",
containerName, output);
}
else
{
_logger.LogError(
"llama.cpp container {ContainerName} exited before becoming ready, but produced no output.",
containerName);
}
}
catch (Exception ex)
{
_logger.LogWarning(ex, "Failed to read logs of container {ContainerName}", containerName);
}
} }
private static string SanitizeContainerName(string name) private static string SanitizeContainerName(string name)
@@ -520,6 +838,13 @@ internal sealed partial class LlamaCppManager : IAsyncDisposable
[GeneratedRegex(@"[^a-zA-Z0-9_.-]")] [GeneratedRegex(@"[^a-zA-Z0-9_.-]")]
private static partial Regex InvalidContainerNameChars(); private static partial Regex InvalidContainerNameChars();
private enum ContainerStartResult
{
Ready,
ContainerExited,
TimedOut
}
} }
internal sealed record LlamaCppModel internal sealed record LlamaCppModel
+8
View File
@@ -22,8 +22,15 @@ try
Console.WriteLine($" ollama models path: {options.UseOllamaModelsPath ?? "(not set)"}"); Console.WriteLine($" ollama models path: {options.UseOllamaModelsPath ?? "(not set)"}");
Console.WriteLine($" llama.cpp docker image: {options.LlamaCppDockerImage ?? "(auto)"}"); Console.WriteLine($" llama.cpp docker image: {options.LlamaCppDockerImage ?? "(auto)"}");
Console.WriteLine($" llama.cpp base port: {options.LlamaCppBasePort}"); Console.WriteLine($" llama.cpp base port: {options.LlamaCppBasePort}");
Console.WriteLine(
options.LlamaCppParallel.HasValue
? $" llama.cpp parallel slots: {options.LlamaCppParallel.Value}"
: " llama.cpp parallel slots: (llama.cpp default)");
} }
var logDirectory = options.LogDirectory ?? Path.Combine(AppContext.BaseDirectory, "Logs");
Console.WriteLine($" log directory: {logDirectory}");
// Args are parsed by ClientOptions; keep them away from the host configuration. // Args are parsed by ClientOptions; keep them away from the host configuration.
var builder = Host.CreateApplicationBuilder(new HostApplicationBuilderSettings { Args = [] }); var builder = Host.CreateApplicationBuilder(new HostApplicationBuilderSettings { Args = [] });
builder.Services.AddSingleton(options); builder.Services.AddSingleton(options);
@@ -32,6 +39,7 @@ try
builder.Services.AddWindowsService(service => service.ServiceName = "NginoClient"); builder.Services.AddWindowsService(service => service.ServiceName = "NginoClient");
// The EventLog provider defaults to Warning; connection state is worth seeing there. // The EventLog provider defaults to Warning; connection state is worth seeing there.
builder.Logging.AddFilter<Microsoft.Extensions.Logging.EventLog.EventLogLoggerProvider>("Ngino.Client", LogLevel.Information); builder.Logging.AddFilter<Microsoft.Extensions.Logging.EventLog.EventLogLoggerProvider>("Ngino.Client", LogLevel.Information);
builder.Logging.AddProvider(new FileLoggerProvider(logDirectory));
await builder.Build().RunAsync(); await builder.Build().RunAsync();
return 0; return 0;
+69 -26
View File
@@ -47,7 +47,9 @@ internal sealed class TunnelClient
_options.UseOllamaModelsPath, _options.UseOllamaModelsPath,
_options.LlamaCppDockerImage, _options.LlamaCppDockerImage,
_options.LlamaCppBasePort, _options.LlamaCppBasePort,
_logger); _logger,
_options.LlamaCppFallbackCooldown,
_options.LlamaCppParallel);
} }
} }
@@ -477,13 +479,16 @@ internal sealed class TunnelClient
var started = await _llamaCppManager.StartModelContainerAsync(model, cancellationToken); var started = await _llamaCppManager.StartModelContainerAsync(model, cancellationToken);
if (!started) if (!started)
{ {
return new TunnelMessage _logger.LogWarning(
{ "Unable to load model '{Model}' via llama.cpp. Falling back to Ollama upstream.",
Type = TunnelMessageTypes.ModelCommandResult, modelName);
RequestId = message.RequestId,
StatusCode = 500, using var fallbackRequest = BuildModelCommandRequest(_options.Upstream, "load", modelName);
Error = $"Failed to start llama.cpp container for model '{modelName}'." using var fallbackResponse = await _httpClient.SendAsync(
}; fallbackRequest, HttpCompletionOption.ResponseHeadersRead, cancellationToken);
var fallbackBody = await fallbackResponse.Content.ReadAsByteArrayAsync(cancellationToken);
return BuildModelCommandResult(message.RequestId, fallbackResponse, fallbackBody);
} }
return BuildModelCommandResult(message.RequestId, 200, "OK", []); return BuildModelCommandResult(message.RequestId, 200, "OK", []);
@@ -494,13 +499,18 @@ internal sealed class TunnelClient
var stopped = await _llamaCppManager!.StopModelContainerAsync(modelName!, cancellationToken); var stopped = await _llamaCppManager!.StopModelContainerAsync(modelName!, cancellationToken);
if (!stopped) if (!stopped)
{ {
return new TunnelMessage _logger.LogWarning(
{ "No running llama.cpp container for model '{Model}'. Falling back to Ollama upstream to unload it.",
Type = TunnelMessageTypes.ModelCommandResult, modelName);
RequestId = message.RequestId,
StatusCode = 404, _llamaCppManager.ClearModelFallback(modelName!);
Error = $"No running llama.cpp container for model '{modelName}'."
}; using var fallbackRequest = BuildModelCommandRequest(_options.Upstream, "unload", modelName);
using var fallbackResponse = await _httpClient.SendAsync(
fallbackRequest, HttpCompletionOption.ResponseHeadersRead, cancellationToken);
var fallbackBody = await fallbackResponse.Content.ReadAsByteArrayAsync(cancellationToken);
return BuildModelCommandResult(message.RequestId, fallbackResponse, fallbackBody);
} }
return BuildModelCommandResult(message.RequestId, 200, "OK", []); return BuildModelCommandResult(message.RequestId, 200, "OK", []);
@@ -694,28 +704,54 @@ internal sealed class TunnelClient
if (modelName is not null) if (modelName is not null)
{ {
effectiveUpstream = _llamaCppManager.GetUpstream(modelName); effectiveUpstream = _llamaCppManager.GetUpstream(modelName);
if (effectiveUpstream is not null)
{
var running = await _llamaCppManager.IsContainerRunningAsync(modelName);
if (!running)
{
_logger.LogWarning(
"Cached llama.cpp container for model '{Model}' is not running anymore. Starting a fresh one on demand...",
modelName);
effectiveUpstream = null;
}
}
if (effectiveUpstream is null) if (effectiveUpstream is null)
{ {
_logger.LogInformation(
"Request for model '{Model}' but no llama.cpp container is running. Starting one on demand...",
modelName);
var model = _llamaCppManager.DiscoverModelsWithBlob() var model = _llamaCppManager.DiscoverModelsWithBlob()
.FirstOrDefault(m => string.Equals(m.OllamaName, modelName, StringComparison.OrdinalIgnoreCase)); .FirstOrDefault(m => string.Equals(m.OllamaName, modelName, StringComparison.OrdinalIgnoreCase));
if (model is not null) if (model is not null)
{ {
var started = await _llamaCppManager.StartModelContainerAsync(model, cancellationToken); if (_llamaCppManager.IsModelOnFallback(modelName))
if (started)
{ {
effectiveUpstream = _llamaCppManager.GetUpstream(modelName); _logger.LogInformation(
"Model '{Model}' previously failed to load via llama.cpp. Routing directly to the Ollama upstream.",
modelName);
}
else
{
_logger.LogInformation(
"Request for model '{Model}' but no llama.cpp container is running. Starting one on demand...",
modelName);
var started = await _llamaCppManager.StartModelContainerAsync(model, cancellationToken);
if (started)
{
effectiveUpstream = _llamaCppManager.GetUpstream(modelName);
}
else
{
_logger.LogWarning(
"Unable to load model '{Model}' via llama.cpp. Falling back to the Ollama upstream.",
modelName);
}
} }
} }
else
if (effectiveUpstream is null)
{ {
_logger.LogWarning( _logger.LogWarning(
"Failed to start llama.cpp container for model '{Model}'. Falling back to default upstream.", "Model '{Model}' was not found in the Ollama models path. Falling back to default upstream.",
modelName); modelName);
} }
} }
@@ -766,7 +802,14 @@ internal sealed class TunnelClient
effectiveUpstream: effectiveUpstream, effectiveUpstream: effectiveUpstream,
responseHandler: responseHandler, responseHandler: responseHandler,
pathTransform: pathTransform, pathTransform: pathTransform,
bodyTransform: bodyTransform); bodyTransform: bodyTransform,
onConnectionRefused: () =>
{
if (_llamaCppManager is not null && modelName is not null)
{
_llamaCppManager.RemoveModelMapping(modelName);
}
});
if (!_activeRequests.TryAdd(message.RequestId, request)) if (!_activeRequests.TryAdd(message.RequestId, request))
{ {
+35 -1
View File
@@ -1,4 +1,5 @@
using System.Net.Http.Headers; using System.Net.Http.Headers;
using System.Net.Sockets;
using System.Threading.Channels; using System.Threading.Channels;
using Ngino.Protocol; using Ngino.Protocol;
@@ -39,6 +40,7 @@ internal sealed class UpstreamRequest
private readonly Func<HttpResponseMessage, CancellationToken, Task>? _responseHandler; private readonly Func<HttpResponseMessage, CancellationToken, Task>? _responseHandler;
private readonly Func<string, string?>? _pathTransform; private readonly Func<string, string?>? _pathTransform;
private readonly Func<byte[], byte[]>? _bodyTransform; private readonly Func<byte[], byte[]>? _bodyTransform;
private readonly Action? _onConnectionRefused;
private readonly List<byte[]> _bufferedBody = []; private readonly List<byte[]> _bufferedBody = [];
public UpstreamRequest( public UpstreamRequest(
@@ -51,7 +53,8 @@ internal sealed class UpstreamRequest
Uri? effectiveUpstream = null, Uri? effectiveUpstream = null,
Func<HttpResponseMessage, CancellationToken, Task>? responseHandler = null, Func<HttpResponseMessage, CancellationToken, Task>? responseHandler = null,
Func<string, string?>? pathTransform = null, Func<string, string?>? pathTransform = null,
Func<byte[], byte[]>? bodyTransform = null) Func<byte[], byte[]>? bodyTransform = null,
Action? onConnectionRefused = null)
{ {
_httpClient = httpClient; _httpClient = httpClient;
_initialMessage = initialMessage; _initialMessage = initialMessage;
@@ -61,6 +64,7 @@ internal sealed class UpstreamRequest
_responseHandler = responseHandler; _responseHandler = responseHandler;
_pathTransform = pathTransform; _pathTransform = pathTransform;
_bodyTransform = bodyTransform; _bodyTransform = bodyTransform;
_onConnectionRefused = onConnectionRefused;
_cancellationTokenSource = CancellationTokenSource.CreateLinkedTokenSource(cancellationToken); _cancellationTokenSource = CancellationTokenSource.CreateLinkedTokenSource(cancellationToken);
if (!initialMessage.HasBody) if (!initialMessage.HasBody)
@@ -143,6 +147,17 @@ internal sealed class UpstreamRequest
} }
catch (Exception exception) catch (Exception exception)
{ {
if (IsConnectionRefused(exception))
{
try
{
_onConnectionRefused?.Invoke();
}
catch
{
}
}
await SendErrorAsync(exception); await SendErrorAsync(exception);
} }
finally finally
@@ -296,6 +311,25 @@ internal sealed class UpstreamRequest
} }
} }
internal static bool IsConnectionRefused(Exception exception)
{
for (var current = exception; current is not null; current = current.InnerException)
{
if (current is SocketException socketException
&& socketException.SocketErrorCode == SocketError.ConnectionRefused)
{
return true;
}
if (current.Message.Contains("Connection refused", StringComparison.OrdinalIgnoreCase))
{
return true;
}
}
return false;
}
private static List<HeaderPair> CollectResponseHeaders(HttpResponseMessage response) private static List<HeaderPair> CollectResponseHeaders(HttpResponseMessage response)
{ {
var headers = new List<HeaderPair>(); var headers = new List<HeaderPair>();
+10 -3
View File
@@ -167,11 +167,14 @@ internal static class AdminEndpoints
try try
{ {
var manual = string.Equals(request.Mode, "manual", StringComparison.OrdinalIgnoreCase); var manual = string.Equals(request.Mode, "manual", StringComparison.OrdinalIgnoreCase);
TimeSpan? duration = manual TimeSpan? duration = manual
? null ? null
: TimeSpan.FromMinutes(Math.Clamp(request.DurationMinutes ?? 60, 1, 24 * 60)); : request.DurationMinutes is { } minutes
? TimeSpan.FromMinutes(Math.Clamp(minutes, 1, 24 * 60))
: null;
store.DisableClient(clientId, duration, manual, request.Reason); store.DisableClient(clientId, duration, manual, request.Reason, request.StartAtUtc, request.UntilUtc);
return Results.Ok(new { clientId, disabled = true }); return Results.Ok(new { clientId, disabled = true });
} }
catch (Exception exception) catch (Exception exception)
@@ -673,6 +676,7 @@ internal static class AdminEndpoints
snapshot?.ActiveModels ?? [], snapshot?.ActiveModels ?? [],
snapshot?.ModelsUpdatedAt, snapshot?.ModelsUpdatedAt,
access.IsDisabled, access.IsDisabled,
access.DisabledFromUtc,
access.DisabledUntilUtc, access.DisabledUntilUtc,
access.DisabledManually, access.DisabledManually,
access.DisabledReason, access.DisabledReason,
@@ -880,7 +884,9 @@ internal static class AdminEndpoints
internal sealed record DisableClientRequest( internal sealed record DisableClientRequest(
string? Mode, string? Mode,
int? DurationMinutes, int? DurationMinutes,
string? Reason); string? Reason,
DateTimeOffset? StartAtUtc,
DateTimeOffset? UntilUtc);
internal sealed record ModelActionRequest( internal sealed record ModelActionRequest(
string ClientId, string ClientId,
@@ -929,6 +935,7 @@ internal sealed record ClientSummary(
IReadOnlyList<string> ActiveModels, IReadOnlyList<string> ActiveModels,
DateTimeOffset? ModelsUpdatedAt, DateTimeOffset? ModelsUpdatedAt,
bool Disabled, bool Disabled,
DateTimeOffset? DisabledFromUtc,
DateTimeOffset? DisabledUntilUtc, DateTimeOffset? DisabledUntilUtc,
bool DisabledManually, bool DisabledManually,
string? DisabledReason, string? DisabledReason,
+49 -15
View File
@@ -378,7 +378,7 @@ internal sealed class ManagementStore
using var connection = OpenConnection(); using var connection = OpenConnection();
using var command = connection.CreateCommand(); using var command = connection.CreateCommand();
command.CommandText = """ command.CommandText = """
SELECT disabled_until_utc, disabled_manually, disabled_reason SELECT disabled_until_utc, disabled_manually, disabled_reason, disabled_from_utc
FROM client_controls FROM client_controls
WHERE client_id = $client_id WHERE client_id = $client_id
"""; """;
@@ -393,18 +393,22 @@ internal sealed class ManagementStore
var disabledUntil = ReadNullableDateTimeOffset(reader, 0); var disabledUntil = ReadNullableDateTimeOffset(reader, 0);
var disabledManually = reader.GetInt32(1) != 0; var disabledManually = reader.GetInt32(1) != 0;
var reason = reader.IsDBNull(2) ? null : reader.GetString(2); var reason = reader.IsDBNull(2) ? null : reader.GetString(2);
var disabledFrom = ReadNullableDateTimeOffset(reader, 3);
if (disabledManually) var now = DateTimeOffset.UtcNow;
var isScheduled = disabledFrom > now;
if (disabledManually && !isScheduled)
{ {
return new ClientAccess(true, null, true, reason); return new ClientAccess(true, disabledFrom, null, true, reason);
} }
if (disabledUntil is { } until && until > DateTimeOffset.UtcNow) if (!isScheduled && disabledUntil is { } until && until > now)
{ {
return new ClientAccess(true, until, false, reason); return new ClientAccess(true, disabledFrom, until, false, reason);
} }
return ClientAccess.Enabled; return new ClientAccess(false, disabledFrom, disabledUntil, false, reason);
} }
} }
@@ -421,7 +425,7 @@ internal sealed class ManagementStore
{ {
using var connection = OpenConnection(); using var connection = OpenConnection();
using var command = connection.CreateCommand(); using var command = connection.CreateCommand();
command.CommandText = "SELECT client_id, disabled_until_utc, disabled_manually, disabled_reason FROM client_controls"; command.CommandText = "SELECT client_id, disabled_until_utc, disabled_manually, disabled_reason, disabled_from_utc FROM client_controls";
using var reader = command.ExecuteReader(); using var reader = command.ExecuteReader();
while (reader.Read()) while (reader.Read())
@@ -430,19 +434,23 @@ internal sealed class ManagementStore
var disabledUntil = ReadNullableDateTimeOffset(reader, 1); var disabledUntil = ReadNullableDateTimeOffset(reader, 1);
var disabledManually = reader.GetInt32(2) != 0; var disabledManually = reader.GetInt32(2) != 0;
var reason = reader.IsDBNull(3) ? null : reader.GetString(3); var reason = reader.IsDBNull(3) ? null : reader.GetString(3);
var disabledFrom = ReadNullableDateTimeOffset(reader, 4);
result[clientId] = disabledManually var now = DateTimeOffset.UtcNow;
? new ClientAccess(true, null, true, reason) var isScheduled = disabledFrom > now;
: disabledUntil is { } until && until > DateTimeOffset.UtcNow
? new ClientAccess(true, until, false, reason) result[clientId] = disabledManually && !isScheduled
: ClientAccess.Enabled; ? new ClientAccess(true, disabledFrom, null, true, reason)
: !isScheduled && disabledUntil is { } until && until > now
? new ClientAccess(true, disabledFrom, until, false, reason)
: new ClientAccess(false, disabledFrom, disabledUntil, disabledManually, reason);
} }
} }
return result; return result;
} }
public void DisableClient(string clientId, TimeSpan? duration, bool manually, string? reason) public void DisableClient(string clientId, TimeSpan? duration, bool manually, string? reason, DateTimeOffset? startAtUtc = null, DateTimeOffset? untilUtc = null)
{ {
EnsureAvailable(); EnsureAvailable();
@@ -452,7 +460,9 @@ internal sealed class ManagementStore
} }
var now = DateTimeOffset.UtcNow; var now = DateTimeOffset.UtcNow;
var disabledUntil = manually ? null : now.Add(duration ?? TimeSpan.FromHours(1)).ToString("O"); var disabledFrom = startAtUtc?.ToString("O");
var disabledUntil = untilUtc?.ToString("O")
?? (manually ? null : now.Add(duration ?? TimeSpan.FromHours(1)).ToString("O"));
lock (_lock) lock (_lock)
{ {
@@ -462,23 +472,27 @@ internal sealed class ManagementStore
INSERT INTO client_controls ( INSERT INTO client_controls (
client_id, client_id,
disabled_until_utc, disabled_until_utc,
disabled_from_utc,
disabled_manually, disabled_manually,
disabled_reason, disabled_reason,
updated_at_utc) updated_at_utc)
VALUES ( VALUES (
$client_id, $client_id,
$disabled_until_utc, $disabled_until_utc,
$disabled_from_utc,
$disabled_manually, $disabled_manually,
$disabled_reason, $disabled_reason,
$updated_at_utc) $updated_at_utc)
ON CONFLICT(client_id) DO UPDATE SET ON CONFLICT(client_id) DO UPDATE SET
disabled_until_utc = excluded.disabled_until_utc, disabled_until_utc = excluded.disabled_until_utc,
disabled_from_utc = excluded.disabled_from_utc,
disabled_manually = excluded.disabled_manually, disabled_manually = excluded.disabled_manually,
disabled_reason = excluded.disabled_reason, disabled_reason = excluded.disabled_reason,
updated_at_utc = excluded.updated_at_utc updated_at_utc = excluded.updated_at_utc
"""; """;
command.Parameters.AddWithValue("$client_id", clientId); command.Parameters.AddWithValue("$client_id", clientId);
command.Parameters.AddWithValue("$disabled_until_utc", (object?)disabledUntil ?? DBNull.Value); command.Parameters.AddWithValue("$disabled_until_utc", (object?)disabledUntil ?? DBNull.Value);
command.Parameters.AddWithValue("$disabled_from_utc", (object?)disabledFrom ?? DBNull.Value);
command.Parameters.AddWithValue("$disabled_manually", manually ? 1 : 0); command.Parameters.AddWithValue("$disabled_manually", manually ? 1 : 0);
command.Parameters.AddWithValue("$disabled_reason", string.IsNullOrWhiteSpace(reason) ? DBNull.Value : reason.Trim()); command.Parameters.AddWithValue("$disabled_reason", string.IsNullOrWhiteSpace(reason) ? DBNull.Value : reason.Trim());
command.Parameters.AddWithValue("$updated_at_utc", now.ToString("O")); command.Parameters.AddWithValue("$updated_at_utc", now.ToString("O"));
@@ -503,17 +517,20 @@ internal sealed class ManagementStore
INSERT INTO client_controls ( INSERT INTO client_controls (
client_id, client_id,
disabled_until_utc, disabled_until_utc,
disabled_from_utc,
disabled_manually, disabled_manually,
disabled_reason, disabled_reason,
updated_at_utc) updated_at_utc)
VALUES ( VALUES (
$client_id, $client_id,
NULL, NULL,
NULL,
0, 0,
NULL, NULL,
$updated_at_utc) $updated_at_utc)
ON CONFLICT(client_id) DO UPDATE SET ON CONFLICT(client_id) DO UPDATE SET
disabled_until_utc = NULL, disabled_until_utc = NULL,
disabled_from_utc = NULL,
disabled_manually = 0, disabled_manually = 0,
disabled_reason = NULL, disabled_reason = NULL,
updated_at_utc = excluded.updated_at_utc updated_at_utc = excluded.updated_at_utc
@@ -1997,6 +2014,7 @@ internal sealed class ManagementStore
CREATE TABLE IF NOT EXISTS client_controls ( CREATE TABLE IF NOT EXISTS client_controls (
client_id TEXT NOT NULL PRIMARY KEY, client_id TEXT NOT NULL PRIMARY KEY,
disabled_until_utc TEXT NULL, disabled_until_utc TEXT NULL,
disabled_from_utc TEXT NULL,
disabled_manually INTEGER NOT NULL DEFAULT 0, disabled_manually INTEGER NOT NULL DEFAULT 0,
disabled_reason TEXT NULL, disabled_reason TEXT NULL,
updated_at_utc TEXT NOT NULL updated_at_utc TEXT NOT NULL
@@ -2306,6 +2324,21 @@ internal sealed class ManagementStore
} }
} }
using (var migrate = connection.CreateCommand())
{
migrate.CommandText = """
SELECT COUNT(*) FROM pragma_table_info('client_controls') WHERE name = 'disabled_from_utc'
""";
var hasFromColumn = (long)migrate.ExecuteScalar()! > 0;
if (!hasFromColumn)
{
using var addFromCol = connection.CreateCommand();
addFromCol.CommandText = "ALTER TABLE client_controls ADD COLUMN disabled_from_utc TEXT NULL";
addFromCol.ExecuteNonQuery();
}
}
_logger.LogInformation( _logger.LogInformation(
"Loaded {ClientKeyCount} client key(s) from {DatabasePath}.", "Loaded {ClientKeyCount} client key(s) from {DatabasePath}.",
_clientKeysByHash.Count, _clientKeysByHash.Count,
@@ -2406,11 +2439,12 @@ internal sealed class ManagementStore
internal sealed record ClientAccess( internal sealed record ClientAccess(
bool IsDisabled, bool IsDisabled,
DateTimeOffset? DisabledFromUtc,
DateTimeOffset? DisabledUntilUtc, DateTimeOffset? DisabledUntilUtc,
bool DisabledManually, bool DisabledManually,
string? DisabledReason) string? DisabledReason)
{ {
public static ClientAccess Enabled { get; } = new(false, null, false, null); public static ClientAccess Enabled { get; } = new(false, null, null, false, null);
} }
internal sealed record UserKeyInfo( internal sealed record UserKeyInfo(
+5
View File
@@ -269,6 +269,11 @@ app.UseStaticFiles();
app.MapAdminEndpoints(settings); app.MapAdminEndpoints(settings);
app.MapGet("/favicon.ico", () =>
Results.File(
Path.Combine(app.Environment.WebRootPath, "favicon.ico"),
"image/x-icon"));
app.MapGet("/", () => Results.Redirect("/admin")); app.MapGet("/", () => Results.Redirect("/admin"));
app.MapGet(settings.StatusPath, (HttpContext context, TunnelHub hub, ServerSettings serverSettings, EmbeddingCache embeddingCache, ManagementStore managementStore) => app.MapGet(settings.StatusPath, (HttpContext context, TunnelHub hub, ServerSettings serverSettings, EmbeddingCache embeddingCache, ManagementStore managementStore) =>
+33 -4
View File
@@ -10,6 +10,8 @@ internal static class ReverseProxyEndpoint
{ {
private const string UnauthorizedMessage = "Missing or invalid Ngino token."; private const string UnauthorizedMessage = "Missing or invalid Ngino token.";
private const string OllamaVersion = "0.32.5";
private static readonly HashSet<string> HopByHopHeaders = new(StringComparer.OrdinalIgnoreCase) private static readonly HashSet<string> HopByHopHeaders = new(StringComparer.OrdinalIgnoreCase)
{ {
"Connection", "Connection",
@@ -72,6 +74,12 @@ internal static class ReverseProxyEndpoint
return; return;
} }
if (IsVersionRequest(context.Request, proxyPath))
{
await WriteVersionResponseAsync(context);
return;
}
if (TryGetClientAddress(proxyPath, out var pathClientId, out var clientPath)) if (TryGetClientAddress(proxyPath, out var pathClientId, out var clientPath))
{ {
if (!groupAccess.IsClientAllowed(pathClientId)) if (!groupAccess.IsClientAllowed(pathClientId))
@@ -317,6 +325,15 @@ internal static class ReverseProxyEndpoint
HttpMethods.IsGet(request.Method) HttpMethods.IsGet(request.Method)
&& string.Equals(proxyPath.Value, "/api/tags", StringComparison.OrdinalIgnoreCase); && string.Equals(proxyPath.Value, "/api/tags", StringComparison.OrdinalIgnoreCase);
private static bool IsVersionRequest(HttpRequest request, PathString proxyPath) =>
HttpMethods.IsGet(request.Method)
&& string.Equals(proxyPath.Value, "/api/version", StringComparison.OrdinalIgnoreCase);
private static Task WriteVersionResponseAsync(HttpContext context) =>
context.Response.WriteAsJsonAsync(
new { version = OllamaVersion },
context.RequestAborted);
private static async Task HandleTagsAsync( private static async Task HandleTagsAsync(
HttpContext context, HttpContext context,
TunnelHub hub, TunnelHub hub,
@@ -373,14 +390,26 @@ internal static class ReverseProxyEndpoint
private static string GetClientDisabledMessage(string clientId, ClientAccess access) private static string GetClientDisabledMessage(string clientId, ClientAccess access)
{ {
var reason = string.IsNullOrWhiteSpace(access.DisabledReason)
? ""
: $" Reason: {access.DisabledReason.Trim()}.";
if (access.DisabledManually) if (access.DisabledManually)
{ {
return $"Tunnel client '{clientId}' is disabled until it is enabled manually."; return $"Tunnel client '{clientId}' is disabled until it is enabled manually.{reason}";
} }
return access.DisabledUntilUtc is { } disabledUntil if (access.DisabledUntilUtc is { } disabledUntil)
? $"Tunnel client '{clientId}' is disabled until {disabledUntil:O}." {
: $"Tunnel client '{clientId}' is disabled."; var from = access.DisabledFromUtc is { } fromUtc
? $" (scheduled from {fromUtc:O})"
: "";
return $"Tunnel client '{clientId}' is disabled until {disabledUntil:O}.{from}{reason}";
}
return access.DisabledFromUtc is { } fromUtc2
? $"Tunnel client '{clientId}' is disabled (scheduled from {fromUtc2:O}).{reason}"
: $"Tunnel client '{clientId}' is disabled.{reason}";
} }
private static async Task<string?> GetRequestedModelAsync(HttpRequest request, PathString proxyPath) private static async Task<string?> GetRequestedModelAsync(HttpRequest request, PathString proxyPath)
Binary file not shown.

After

Width:  |  Height:  |  Size: 56 KiB

+66 -5
View File
@@ -58,16 +58,12 @@ textarea {
} }
.brand-mark { .brand-mark {
display: grid;
place-items: center;
width: 34px; width: 34px;
height: 34px; height: 34px;
border: 1px solid #4c5d74; border: 1px solid #4c5d74;
border-radius: 6px; border-radius: 6px;
background: #223044; background: #223044;
color: #c8f2df; object-fit: cover;
font-weight: 800;
letter-spacing: 0;
} }
.brand strong, .brand strong,
@@ -592,6 +588,71 @@ tr:last-child td {
} }
} }
.modal-overlay {
position: fixed;
inset: 0;
z-index: 100;
display: grid;
place-items: center;
background: rgba(0, 0, 0, 0.35);
opacity: 0;
transition: opacity 0.15s;
}
.modal-overlay.visible {
opacity: 1;
}
.modal-dialog {
width: min(480px, calc(100vw - 32px));
max-height: calc(100vh - 32px);
overflow-y: auto;
display: grid;
gap: 14px;
padding: 20px;
border-radius: 10px;
background: #ffffff;
box-shadow: 0 20px 60px rgba(0, 0, 0, 0.2);
}
.modal-dialog h3 {
margin: 0;
font-size: 16px;
overflow-wrap: anywhere;
}
.field-label {
color: var(--muted);
font-size: 12px;
font-weight: 600;
}
.radio-row {
display: flex;
gap: 14px;
align-items: center;
}
.radio-row label {
display: flex;
align-items: center;
gap: 5px;
font-size: 14px;
cursor: pointer;
}
.field-row {
display: flex;
gap: 8px;
align-items: center;
}
.modal-actions {
display: flex;
gap: 8px;
justify-content: flex-end;
}
@media (max-width: 520px) { @media (max-width: 520px) {
.nav { .nav {
grid-template-columns: 1fr; grid-template-columns: 1fr;
+155 -10
View File
@@ -94,13 +94,21 @@ content.addEventListener("click", async (event) => {
try { try {
setBusy(button, true); setBusy(button, true);
if (action === "disable-hour") { if (action === "disable-temporary") {
await api(`/clients/${encodeURIComponent(clientId)}/disable`, { setBusy(button, false);
method: "POST", try {
body: { mode: "duration", durationMinutes: 60 } const body = await showDisableModal(clientId);
}); setBusy(button, true);
setNotice(`Disabled ${clientId} for one hour.`); await api(`/clients/${encodeURIComponent(clientId)}/disable`, {
await refresh(); method: "POST",
body
});
setNotice(`Disabled ${clientId}.`);
await refresh();
} catch {
// cancelled
}
return;
} }
if (action === "disable-manual") { if (action === "disable-manual") {
@@ -498,6 +506,7 @@ function clientsTable(clients) {
${client.disabled ? badge(client.disabledManually ? "Disabled manual" : "Disabled timed", "bad") : badge("Enabled", "good")} ${client.disabled ? badge(client.disabledManually ? "Disabled manual" : "Disabled timed", "bad") : badge("Enabled", "good")}
</div> </div>
${client.disabled ? `<div class="cell-sub">${escapeHtml(disabledText(client))}</div>` : ""} ${client.disabled ? `<div class="cell-sub">${escapeHtml(disabledText(client))}</div>` : ""}
${isScheduled(client) ? `<div class="cell-sub">${escapeHtml(disabledText(client))}</div>` : ""}
</td> </td>
<td>${number(client.pendingRequests)}</td> <td>${number(client.pendingRequests)}</td>
<td> <td>
@@ -513,7 +522,7 @@ function clientsTable(clients) {
</td> </td>
<td> <td>
<div class="actions"> <div class="actions">
<button class="button warning" data-action="disable-hour" data-client-id="${escapeAttr(client.id)}" ${client.disabled ? "disabled" : ""}>Disable 1h</button> <button class="button warning" data-action="disable-temporary" data-client-id="${escapeAttr(client.id)}" ${client.disabled ? "disabled" : ""}>Disable temporary</button>
<button class="button warning" data-action="disable-manual" data-client-id="${escapeAttr(client.id)}" ${client.disabled ? "disabled" : ""}>Disable</button> <button class="button warning" data-action="disable-manual" data-client-id="${escapeAttr(client.id)}" ${client.disabled ? "disabled" : ""}>Disable</button>
<button class="button secondary" data-action="enable-client" data-client-id="${escapeAttr(client.id)}" ${client.disabled ? "" : "disabled"}>Enable</button> <button class="button secondary" data-action="enable-client" data-client-id="${escapeAttr(client.id)}" ${client.disabled ? "" : "disabled"}>Enable</button>
</div> </div>
@@ -1657,11 +1666,24 @@ function emptyState(text) {
} }
function disabledText(client) { function disabledText(client) {
var text = "";
if (client.disabledManually) { if (client.disabledManually) {
return "Until enabled manually"; text = "Until enabled manually";
} else if (client.disabledUntilUtc) {
text = `Until ${formatDate(client.disabledUntilUtc)}`;
} else if (isScheduled(client)) {
text = `Scheduled from ${formatDate(client.disabledFromUtc)}`;
} else {
text = "Disabled";
} }
if (client.disabledReason) {
text += `${escapeHtml(client.disabledReason)}`;
}
return text;
}
return client.disabledUntilUtc ? `Until ${formatDate(client.disabledUntilUtc)}` : "Disabled"; function isScheduled(client) {
return client.disabledFromUtc && new Date(client.disabledFromUtc) > new Date();
} }
function formatDate(value) { function formatDate(value) {
@@ -1723,4 +1745,127 @@ function escapeAttr(value) {
return escapeHtml(value); return escapeHtml(value);
} }
function showDisableModal(clientId) {
return new Promise((resolve, reject) => {
const overlay = document.createElement("div");
overlay.className = "modal-overlay";
overlay.innerHTML = `
<div class="modal-dialog">
<h3>Disable client &quot;${escapeHtml(clientId)}&quot;</h3>
<div class="field">
<span class="field-label">When to disable</span>
<div class="radio-row">
<label><input type="radio" name="d-when" value="now" checked> Now</label>
<label><input type="radio" name="d-when" value="later"> Later</label>
</div>
<input type="datetime-local" id="d-from" class="input" disabled>
</div>
<div class="field">
<span class="field-label">For how long</span>
<div class="radio-row">
<label><input type="radio" name="d-for" value="timespan" checked> Timespan</label>
<label><input type="radio" name="d-for" value="until"> Until</label>
</div>
<div id="d-timespan-group" class="field-row">
<input type="number" id="d-duration" class="input" value="1" min="1" style="width:100px">
<select id="d-unit" class="select" style="width:auto">
<option value="1">minute(s)</option>
<option value="60" selected>hour(s)</option>
<option value="1440">day(s)</option>
</select>
</div>
<input type="datetime-local" id="d-until" class="input" style="display:none" disabled>
</div>
<div class="field">
<label for="d-reason">Reason (optional)</label>
<input type="text" id="d-reason" class="input" placeholder="e.g. maintenance">
</div>
<div class="form-row modal-actions">
<button class="button secondary" id="d-cancel" type="button">Cancel</button>
<button class="button warning" id="d-confirm" type="button">Disable</button>
</div>
</div>
`;
document.body.appendChild(overlay);
requestAnimationFrame(() => overlay.classList.add("visible"));
const whenRadios = overlay.querySelectorAll('input[name="d-when"]');
const forRadios = overlay.querySelectorAll('input[name="d-for"]');
const fromInput = overlay.querySelector("#d-from");
const timespanGroup = overlay.querySelector("#d-timespan-group");
const untilInput = overlay.querySelector("#d-until");
function updateWhen() {
const val = overlay.querySelector('input[name="d-when"]:checked').value;
fromInput.style.display = val === "later" ? "" : "none";
fromInput.disabled = val !== "later";
if (val === "now") fromInput.value = "";
}
function updateFor() {
const val = overlay.querySelector('input[name="d-for"]:checked').value;
timespanGroup.style.display = val === "timespan" ? "flex" : "none";
untilInput.style.display = val === "until" ? "" : "none";
untilInput.disabled = val !== "until";
if (val === "timespan") untilInput.value = "";
}
whenRadios.forEach(r => r.addEventListener("change", updateWhen));
forRadios.forEach(r => r.addEventListener("change", updateFor));
updateFor();
overlay.querySelector("#d-confirm").addEventListener("click", () => {
const when = overlay.querySelector('input[name="d-when"]:checked').value;
const forVal = overlay.querySelector('input[name="d-for"]:checked').value;
const reason = overlay.querySelector("#d-reason").value.trim() || null;
if (when === "later" && !fromInput.value) {
setNotice("Please select a date and time for the disable.", true);
return;
}
if (forVal === "until" && !untilInput.value) {
setNotice("Please select a date and time for the end.", true);
return;
}
if (forVal === "timespan") {
const val = parseInt(overlay.querySelector("#d-duration").value);
if (!val || val < 1) {
setNotice("Please enter a valid duration.", true);
return;
}
}
const body = { reason };
if (when === "later") {
body.startAtUtc = new Date(fromInput.value).toISOString();
}
if (forVal === "timespan") {
const val = parseInt(overlay.querySelector("#d-duration").value) || 1;
const unit = parseInt(overlay.querySelector("#d-unit").value);
body.durationMinutes = val * unit;
} else {
body.untilUtc = new Date(untilInput.value).toISOString();
}
overlay.remove();
resolve(body);
});
overlay.querySelector("#d-cancel").addEventListener("click", () => {
overlay.remove();
reject(new Error("Cancelled"));
});
overlay.addEventListener("click", (e) => {
if (e.target === overlay) {
overlay.remove();
reject(new Error("Cancelled"));
}
});
});
}
boot(); boot();
+1 -1
View File
@@ -10,7 +10,7 @@
<div class="app-shell"> <div class="app-shell">
<aside class="sidebar"> <aside class="sidebar">
<div class="brand"> <div class="brand">
<span class="brand-mark">RL</span> <img class="brand-mark" src="/admin/Ngino_logo_symbol.png" alt="Ngino">
<span> <span>
<strong>Ngino</strong> <strong>Ngino</strong>
<small>Admin</small> <small>Admin</small>
Binary file not shown.

Before

Width:  |  Height:  |  Size: 1.4 KiB

After

Width:  |  Height:  |  Size: 4.2 KiB

@@ -1,3 +1,4 @@
using System.Net.Sockets;
using Ngino.Client; using Ngino.Client;
using Xunit; using Xunit;
@@ -32,4 +33,29 @@ public sealed class UpstreamRequestTests
Assert.Contains("origin-form path", exception.Message); Assert.Contains("origin-form path", exception.Message);
} }
[Fact]
public void IsConnectionRefused_DetectsSocketConnectionRefused()
{
var socketException = new SocketException((int)SocketError.ConnectionRefused);
var exception = new HttpRequestException("Connection refused", socketException);
Assert.True(UpstreamRequest.IsConnectionRefused(exception));
}
[Fact]
public void IsConnectionRefused_DetectsConnectionRefusedByMessage()
{
var exception = new HttpRequestException("Connection refused (localhost:8081)");
Assert.True(UpstreamRequest.IsConnectionRefused(exception));
}
[Fact]
public void IsConnectionRefused_IgnoresUnrelatedFailures()
{
var exception = new HttpRequestException("Connection reset by peer");
Assert.False(UpstreamRequest.IsConnectionRefused(exception));
}
} }