1 Commits
Author SHA1 Message Date
jzzliu 73ff04d77f Add continuous-batching engine with variable-length KV cache
Requests of different lengths share one left-padded cache with explicit per-row RoPE
positions and a key mask. Requests join mid-flight and leave on EOS/length, and unused
padding columns are trimmed. Greedy output is verified identical to solo generation.
2026-11-07 20:27:21 -07:00