claude e77c880ad8 [Deploy] Keep a 64K-token hot cache for two to four requests ...

A 96K hot cache at MAX_NUM_SEQS=4 left under 100 MiB on 16 GB cards and a
mixed prefill step ran out of memory during a soak; 64K keeps the margin
and still holds 4 x 32K contexts within 1-5 ms of 4K.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

2026-10-22 08:14:06 +08:00

..

applications

…

basic

…

deployment

[Core] Size GGUF dense scratch to admitted shapes and single-request graphs

2026-10-11 09:14:50 +08:00

disaggregated

…

features

…

generate

…

observability

…

pooling

…

ray_serving

…

reasoning

…

rl

…

speech_to_text

…

tool_calling

…

serve_flashnext_gguf_256k.sh

[Deploy] Keep a 64K-token hot cache for two to four requests

2026-10-22 08:14:06 +08:00

template_alpaca.jinja

…

template_baichuan.jinja

…

template_chatglm2.jinja

…

template_chatglm.jinja

…

template_chatml.jinja

…

template_falcon_180b.jinja

…

template_falcon.jinja

…

template_inkbot.jinja

…

template_teleflm.jinja

…

tool_chat_template_apertus.jinja

…

tool_chat_template_deepseekr1.jinja

…

tool_chat_template_deepseekv3.jinja

…

tool_chat_template_deepseekv31.jinja

…

tool_chat_template_functiongemma.jinja

…

tool_chat_template_gemma3_pythonic.jinja

…

tool_chat_template_gemma4.jinja

…

tool_chat_template_glm4.jinja

…

tool_chat_template_granite_20b_fc.jinja

…

tool_chat_template_granite.jinja

…

tool_chat_template_hermes.jinja

…

tool_chat_template_hunyuan_a13b.jinja

…

tool_chat_template_internlm2_tool.jinja

…

tool_chat_template_llama3.1_json.jinja

…

tool_chat_template_llama3.2_json.jinja

…

tool_chat_template_llama3.2_pythonic.jinja

…

tool_chat_template_llama4_json.jinja

…

tool_chat_template_llama4_pythonic.jinja

…

tool_chat_template_minimax_m1.jinja

…

tool_chat_template_mistral3.jinja

…

tool_chat_template_mistral_parallel.jinja

…

tool_chat_template_mistral.jinja

…

tool_chat_template_phi4_mini.jinja

…

tool_chat_template_qwen3coder.jinja

…

tool_chat_template_toolace.jinja

…

tool_chat_template_xlam_llama.jinja

…

tool_chat_template_xlam_qwen.jinja

…