jzzliu/microgpt
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
4 Commits 1 Branch 0 Tags
a603d953c1c1456d0ce009f7cd86c4fd04b64c69
Commit Graph
2 Commits
Author SHA1 Message Date
jzzliu a603d953c1 Add grouped-query attention with parity tests against multi-head 
GQA/MQA via GPTConfig.n_kv_head; the cache stores only the KV heads. Attention
also accepts per-row positions and a key padding mask, used by batched decoding.
 2026-11-07 20:25:52 -07:00
jzzliu 2ce8b739ed Add microgpt: from-scratch GPT with RoPE, KV cache, sampling, tests and benchmark 2026-11-07 20:19:10 -07:00