macandClaude Opus 5.5
702c1b8eae
The launcher passed --no-mamba-tail-checkpoint, so no recurrent state was saved at the end of a prompt and a follow-up chat turn could not reuse the previous prompt (prefix-cache hits stayed at 0 until a third request with the same prefix). Default to the tail checkpoint; MAMBA_TAIL_CHECKPOINT=0 restores the old behavior. On 4x V100 32 GB a 21K-token second turn went from 5.8 s to 1.3 s TTFT. Add a Dockerfile that installs the SM70 wheel on Ubuntu 24.04 (torch cu128 wheels carry the CUDA runtime) with an entrypoint that finds the GGUF, MTP and vision weights under /models. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
16 lines
891 B
Bash
16 lines
891 B
Bash
#!/usr/bin/env bash
|
|
# Resolve model paths under $MODEL_DIR unless given explicitly, then run the launcher.
|
|
set -euo pipefail
|
|
: "${VLLM_API_KEY:?Set VLLM_API_KEY (docker run -e VLLM_API_KEY=...)}"
|
|
export MODEL_FILE=${MODEL_FILE:-$(ls "$MODEL_DIR"/gguf/*-00001-of-*.gguf 2>/dev/null | head -1)}
|
|
export MTP_DIR=${MTP_DIR:-$MODEL_DIR/mtp}
|
|
if [[ -z "${VISION_DIR+x}" && -f "$MODEL_DIR/vision/visual.safetensors" ]]; then
|
|
export VISION_DIR=$MODEL_DIR/vision
|
|
fi
|
|
export EXPERT_PROFILE=${EXPERT_PROFILE:-/app/expert_profile.pt}
|
|
export PYTHON_BIN=/opt/venv/bin/python GPU_IDS=${GPU_IDS:-0,1,2,3}
|
|
[[ -f "$MODEL_FILE" ]] || { echo "No GGUF found: mount the model at $MODEL_DIR/gguf"; exit 1; }
|
|
[[ -f "$MTP_DIR/model.safetensors" ]] || { echo "No MTP weights at $MTP_DIR"; exit 1; }
|
|
echo "model=$MODEL_FILE mtp=$MTP_DIR vision=${VISION_DIR:-off} port=$PORT"
|
|
exec bash /app/serve_flashnext_gguf_256k.sh
|