diff --git a/roadmaps/inference-engineering/content/constrained-decoding@E8mQE9mlIPW6wpmJBQe0b.md b/roadmaps/inference-engineering/content/constrained-decoding@E8mQE9mlIPW6wpmJBQe0b.md new file mode 100644 index 000000000..e43edeb45 --- /dev/null +++ b/roadmaps/inference-engineering/content/constrained-decoding@E8mQE9mlIPW6wpmJBQe0b.md @@ -0,0 +1,10 @@ +# Constrained Decoding + +Constrained decoding is the mechanism that enforces structured outputs at the token level. At each sampling step, logits for tokens that would violate the active schema or grammar are masked to zero before sampling, so only valid continuations can be selected. Grammar compilation adds overhead to time to first token, and the interaction with speculative decoding is important to note: some engines disable speculation under hard constraints because draft token acceptance rates drop sharply when only a narrow set of tokens is valid at each step. . + +Visit the following resources to learn more: + +- [@opensource@Guidance: guided generation](https://github.com/guidance-ai/guidance) +- [@opensource@Outlines: structured generation](https://github.com/dottxt-ai/outlines) +- [@article@A Guide to Structured Generation Using Constrained Decoding](https://www.aidancooper.co.uk/constrained-decoding/) +- [@article@Constrained decoding definition](https://www.sanity.io/glossary/constrained-decoding) \ No newline at end of file diff --git a/roadmaps/inference-engineering/content/multi-cloud@d78UiI_FZyLsU6O52btXg.md b/roadmaps/inference-engineering/content/multi-cloud@d78UiI_FZyLsU6O52btXg.md index 3105dd8d2..559782324 100644 --- a/roadmaps/inference-engineering/content/multi-cloud@d78UiI_FZyLsU6O52btXg.md +++ b/roadmaps/inference-engineering/content/multi-cloud@d78UiI_FZyLsU6O52btXg.md @@ -4,8 +4,7 @@ Multi-cloud inference distributes GPU capacity across providers and regions to m Visit the following resources to learn more: -- [@roadmap@Visit the Dedicated Cloud Computing Roadmap](https://roadmap.sh/cloud-computing) - [@article@How to Do Agentic AI Inference in a Multicloud, Multi-Model World](https://blog.equinix.com/blog/2025/03/19/how-to-do-agentic-ai-inference-in-a-multicloud-multi-model-world/) - [@article@What is multicloud?](https://cloud.google.com/learn/what-is-multicloud?hl=en) - [@article@Multi-Cloud GPU Strategy: How to Avoid AI Infrastructure Vendor Lock-In](https://lyceum.technology/magazine/multi-cloud-gpu-avoid-vendor-lock-in/) -- [@video@What is Multicloud? How Do You Manage It?](https://www.youtube.com/watch?v=AjtdZ3gFRjU) +- [@video@What is Multicloud? How Do You Manage It?](https://www.youtube.com/watch?v=AjtdZ3gFRjU) \ No newline at end of file diff --git a/roadmaps/inference-engineering/content/structured-outputs@HnJ29eZchSHMvU-mSIXCJ.md b/roadmaps/inference-engineering/content/structured-outputs@HnJ29eZchSHMvU-mSIXCJ.md new file mode 100644 index 000000000..57b19c405 --- /dev/null +++ b/roadmaps/inference-engineering/content/structured-outputs@HnJ29eZchSHMvU-mSIXCJ.md @@ -0,0 +1,11 @@ +# Structured Outputs + +Structured outputs constrain model generation to a defined schema — typically JSON, a specific data type, or a function signature — so that downstream code can parse the result directly without a repair pass. Major inference engines including vLLM and SGLang ship structured output as a first-class feature. The tradeoff is that a schema guarantees the shape of the output, not its correctness: if the model's best answer does not naturally conform to the schema, forcing conformance can affect response quality. + +Visit the following resources to learn more: + +- [@official@vLLM Structured Outputs](https://docs.vllm.ai/en/latest/features/structured_outputs.html) +- [@official@SGlang Structured Outputs](https://docs.sglang.ai/advanced_features/structured_outputs.html) +- [@opensource@Outlines: structured generation](https://github.com/dottxt-ai/outlines) +- [@opensource@Guidance: guided generation](https://github.com/guidance-ai/guidance) +- [@article@Structured Output Engineering for Production LLMs: Beyond JSON Mode](https://www.bigyan.dev/blog/structured-output-engineering-production-llm/) \ No newline at end of file