Files
9cf2d94a75 [Bugfix][DeepSeek-V4] Keep hc_head finite under float16 for attention-sink rows (#715)
#658 saturates the float16 stores of the mHC pre/post kernels; the fused
hc_head kernel still wrote its float32 weighted sum straight into the
float16 output. For an attention-sink row, whose four streams add up past
65504, the head input became inf. Clamp the store to the float16 range
under use_fp16, the same as the other mHC stores.

The new test builds a sink row whose exact hc_head output exceeds float16
and checks that the kernel stays finite, saturates that row and matches a
float32 reference elsewhere; on main it fails with inf in the sink row.

Signed-off-by: Peuqui <peuqui@github.com>
Co-authored-by: Peuqui <peuqui@github.com>
Co-authored-by: Claude <noreply@anthropic.com>
2026-10-28 10:23:38 +08:00
..