The GigaChat team has unveiled GigaChat 3.5 Ultra, making it publicly accessible as open source under the MIT license. This latest version represents a new 432 billion parameter model and is a pioneering open-source hybrid of GatedDeltaNet and MLA, developed through extensive refinement over 1,500 experimental iterations. Notably, despite its advancements in coding, mathematical capabilities, agent scenarios, and application areas, this model is 40% smaller than its predecessor, GigaChat 3.1 Ultra. Key features include: - A unique hybrid architecture combining MLA and GatedDeltaNet, supported by a specialized stabilization framework that ensures reliable training at this scale. - Gated Attention technology, which allows the model to reduce the impact of excessively strong signals from the attention mechanism. - GatedNorm, a normalization technique incorporating an explicit gate that manages signal strength across various features. - Approximately four times lower KV cache usage per token, enabling the model to accommodate 2.14 times longer context while achieving a 20% increase in throughput under heavy loads. - Two MTP heads that facilitate up to 2.2 times faster generation speeds. - The use of FP8 throughout all training phases without sacrificing quality compared to bf16, made possible by custom Triton and CUDA kernels. - An added online reinforcement learning stage following SFT and DPO. In terms of performance: - GigaChat 3.5 Ultra Base has been shown to outperform both DeepSeek V3.2 Exp Base and DeepSeek V4 Flash Base on a variety of general, mathematical, and coding benchmarks. - GigaChat 3.5 Ultra Instruct achieves comparable average scores to DeepSeek V3.2, despite being only half its size. - According to assessments by the MiniMax-M2.7 LLM judge, GigaChat 3.5 Ultra boasts a 75.9% average win rate against GigaChat 3.1 Ultra and 68.7% against GPT-5. The comprehensive system—including data sourced from a custom LLM-filtered Common Crawl and over 600 programming languages—was entirely developed by the GigaChat team.
Informational material. 18+.
