# How Deepseek's Manifold Constrained Hyper Connections (MHC) Stabilized Large-Scale Training

![h:300](https://cdn.qr-code-generator.com/qrcode-preview?background_color=%23ffffff&foreground_color=%23000000&marker_left_inner_color=%23000000&marker_left_outer_color=%23000000&marker_right_inner_color=%23000000&marker_right_outer_color=%23000000&marker_bottom_inner_color=%23000000&marker_bottom_outer_color=%23000000&marker_left_template=version17&marker_right_template=version17&marker_bottom_template=version17&qr_code_text=https%3A%2F%2Fqrco.de%2FbgbZKv&frame_name=no-frame&Expires=1770397646&Signature=pC1ITi-RcEv4szzwjIFFHOEQaL2XOMnLGtoG7KTRovAlulJDJur0HpGHsTkNteC~VNHDELsQG0bnZke89tQB2~~v125493FVCj7coObWePnLmjqPOhe0zHFf-sQbzK9sRoV3APRo1B5nhaQelQ2go1x8iUg844jWbdXlQq2HxZ-i~9Kbtr5CTT5HCVQLjGLqRF-3UvCLDeASoGj1xkY5criXQxXmMzFebmA1XbaxIlga1zWK0hLcGzpChB8YrpV6IBRiWEaxIVOA3pCTVmH7kAJziIRs7bD7RQ2Z91a5Ry0vrBrLmpogsxeD4UTwTrulEGzzbmfz6aHvY942otHvyA__&Key-Pair-Id=KKMPOJU8AYATR) https://qrco.de/bgbZKv

--- # Quick LLM architecture overview

- **Embedding Layer** - Tokenization, vectorization, embed semantics and positional encoding - **Transformation Blocks** - Composes transformation layers. There are dozens. - **Transformation Layers** - Transform vectors in n-dimensional latent space. - MHA, FFN, MoE, residual connections, layer norm blocks, etc. - **Output** - Prediction produced - **Backpropagation & Gradient Descent** - Calculates how to adjust weights for better predictions.

![h:600](https://i.ibb.co/FkJm3Wkw/Screenshot-2026-02-01-221919.jpg)

--- # Gradients Vanish and Explode During Backpropagation

- During **backpropagation**, we calculate **gradients** to determine how to adjust layer **weights** for learning during **gradient descent**. - As we add more layers, gradients become more unstable, reaching extreme values at either end (**gradient vanishing / explosion**), breaking training.

![h:250](https://media.geeksforgeeks.org/wp-content/uploads/20250701163824448467/Backpropagation-in-Neural-Network-1.webp) ![h:300](https://datamites.com/blog/uploads/images/202403/image_750x_660501cb801fd.jpg)

--- # Residual Connections Solve Gradient Vanishing By Preserving Signal ![h:200](https://arxiv.org/html/2409.19606v3/x5.png)

- Joins the residual (original) stream and the transformation stream. - In short, causes transformation to be additive rather than replace; we don't lose the original signal if the transformed one zeroes out. - Paired with layer normalization block for further stabilization and address **gradient explosion**. - **See-saw problem**: Tradeoff between meaningful development of deep layers and gradient stability; related to overall amount of residual and transformed signal. Traditional approaches changed positioning of layer normalization to address this.

--- # Hyper Connections Optimize Connection Weights

- **Hyper Connections**: Parallel streams that can be mixed in various combinations with tunable weights. - Model can learn optimal mix of weight ratios and connection architectures to address seesaw issue and optimize layer connectivity. - A later layer might need more of some signal from some early layer. - **Issue**: Gradients still destabilized around 11K steps because of their unconstrained nature.

![](https://arxiv.org/html/2409.19606v3/x5.png) ![](https://arxiv.org/html/2409.19606v3/x6.png)

--- # Manifold Constrained Hyper Connections (mHC)

- Deepseek found that unconstrained connection evolution broke the **identity mapping** property required for gradient stability. - **Preserve signal magnitude** by normalizing connection weights using a doubly stochastic matrix; this is projected onto a manifold (Birkhoff Polytope) using the Sinkhorn-Knopp Algorithm. - **Gradients stabilized**; signal gain dropped by three orders of magnitude.

![mHC Visualization](https://substackcdn.com/image/fetch/f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc54ec8ee-b5f9-4179-8496-f7b7d4fa7a8e_1072x463.png) $$ \begin{pmatrix} 0.2 & 0.5 & 0.3 \\ 0.4 & 0.1 & 0.5 \\ 0.4 & 0.4 & 0.2 \end{pmatrix} $$ Doubly stochastic matrix - columns and rows sum to 1

--- # Summary, Outcomes, Benchmarks

- Residual connections + layer normalization blocks **stabilized training deep models**. - Hyper connections provided a cost-efficient means of **improving model performance via learnable connection architecture** rather than scaling parameters, but this destabilized with deep models. - Manifold constraints **stabilized hyper connections** in exchange for **~6-7% compute overhead**, making it **feasible with larger models** + modest benchmark improvements. - Peer researchers speculate possible **scaling through topological complexity**. Currently facing memory bottlenecks.

![](https://miro.medium.com/v2/resize:fit:1400/1*hHao6kI3HF1wfFV00XKxLw.png) ![Benchmarks](https://substackcdn.com/image/fetch/f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F48a8f687-818c-47ae-a1f8-c13e2ff91bf8_1072x360.png)

--- # References - [The Manifold Dial: Visualizing Why DeepSeek's mHC Stabilizes Deep Networks](https://subhadipmitra.com/blog/2026/deepseek-mhc-manifold-constrained-hyper-connections/) - [mHC: Manifold-Constrained Hyper-Connections](https://arxiv.org/pdf/2512.24880) - [Hyper-Connections](https://arxiv.org/html/2409.19606v3) - [Paper page - Frac-Connections: Fractional Extension of Hyper-Connections](https://huggingface.co/papers/2503.14125) - [Dr. Robert Li | Manifold-Constrained Hyper-Connections (mHC): A Summary](https://drli.blog/posts/analysis-mhc-deepseekai/) - [A Detailed Guide to the Architecture of LLMs](https://medium.com/@anish.chitturu/a-detailed-guide-to-the-architecture-of-large-language-models-llms-6dfbe7351903) - [[Paper Reading]: mHC Manifold-Constrained Hyper-Connections](https://www.youtube.com/watch?v=O7hAML3vvRY)