Mixture of Experts on The Coders Bloghttps://thecodersblog.com/tag/mixture-of-experts/Recent content in Mixture of Experts on The Coders BlogHugoen-usThu, 07 May 2026 11:51:42 +0000ZAYA1-8B: Efficient Large Language Models with MoEhttps://thecodersblog.com/zaya1-8b-a-new-moe-llm-2026/Thu, 07 May 2026 11:51:42 +0000https://thecodersblog.com/zaya1-8b-a-new-moe-llm-2026/<p>Forget scaling up parameter counts; the future of LLMs is about <em>intelligence density</em>, and ZAYA1-8B is the latest, and perhaps most compelling, testament to this shift. Zyphra’s new 8.4 billion total parameter model, with a mere 760 million active parameters per token, doesn&rsquo;t just tread water – it sprints ahead in crucial areas, particularly mathematical and coding reasoning. This isn&rsquo;t just another incremental improvement; it’s a statement piece that challenges the established dogma of &ldquo;bigger is always better.&rdquo;</p>