← Back to Research

Interactive Research Report

DeepSeek V4: Architecture & Performance Deep Dive

By Yumei Dou ·

← Back to Research

Executive Summary

On April 24, 2026, DeepSeek released V4, its fourth-generation large language model, representing a significant leap in efficient AI architecture design. The model introduces a hybrid attention mechanism combining Chunked Shared Attention (CSA) and Hash-based Chunked Attention (HCA), a novel Mixture-of-Experts configuration with 384 experts, and aggressive KV cache compression that enables 1 million token context windows on commodity hardware.

V4-Pro Parameters

1.6T total

49B active per forward pass

V4-Flash Parameters

284B total

13B active per forward pass

Context Window

1M tokens

Both Pro and Flash variants

V4-Pro Pricing

$1.74/M input

$6.94/M output tokens

V4-Flash Pricing

$0.14/M input

$0.55/M output tokens

Launch Date

Apr 24, 2026

Open-weight release

Key Innovations

CSA/HCA Hybrid Attention mHC Residual Connection Muon Optimizer 384 MoE Experts (6+1 active) TileLang Kernel DSL GRPO Post-Training FP4 Quantization Multi-Latent Attention (MLA)

Architecture Visualization

DeepSeek V4's architecture represents a carefully designed system that decouples computational expressiveness from memory bandwidth requirements. The following interactive diagrams illustrate the key architectural innovations.

Interactive Architecture Blueprint

This diagram reconstructs the full Prefill → Decoding pipeline from the DeepSeek-V4-Pro open-source config. Click on any layer or stage to see quantitative details. Toggle between Prefill and Decoding views.

Decoding · Memory-Bound
Input Prompt: n tokens (Token 1 … Token n-1 → Prefill | Token n → Decoding) PREFILL: tokens 1…n-1 (Compute-Bound) DECODING: token n (Memory-Bound) Layer 1 / 61 Tokens 1…n-1 Batch embed → X K,V Projection W_k: 7168→512 W_v: 7168→512 Compress CSA: 4x HCA: 128x compress_ratios[i] Compute-bound: batch matmul all tokens GPU Utilization ~70-85% KV Cache (Layer 1) chunks_group(n-1)_(k,v) | compressed At 1M tokens: ~604MB/layer (MLA) vs ~29GB/layer (MHA) Token n embed → 7168-dim Q/K/V via MLA q: 7168→512→16384 k,v: 7168→512 (latent) ~80.7M FLOPs Hash Retrieval TopK=1024 Attention Q_n · K_chunks → softmax → V | ~134.2M FLOPs Attn(q, topk_chunks_kv) · W_output_proj Residual + RMSNorm (mHC) MoE / FFN Router → 6+1 / 384 experts SwiGLU: gate·up·down | ~924.8M FLOPs Residual + Norm → H(1,n) | Update KV Cache write n_(k,v) H(1,n) → Layer 2 input Layer 2 / 61 Same pipeline: Q/K/V → Hash → Attn → mHC → MoE → Residual Input: H(1,n) | Output: H(2,n) | Uses Layer 2's own KV cache compress_ratio[2] = 4 (CSA) | ~1.14G FLOPs per layer ⋮ Layers 3 – 60 ⋮ Alternating: HCA(128x) ↔ CSA(4x) compression per layer Layer 61 / 61 compress_ratio = 0 (Full Attention) Final layer uses uncompressed attention for maximum expressiveness LM Head → Softmax → Token n+1 H(61,n) → Vocab projection (7168 × 129280) → sample Autoregressive Loop: n+1 → new n Per-Layer Quantitative Projection FLOPs: 80.7M Attention FLOPs: 134.2M MoE FFN FLOPs: 924.8M Total per layer: ~1.14G FLOPs × 61 layers = ~69.5G FLOPs/token KV Cache/layer (1M): ~604MB (MLA) Baseline (MHA): ~29GB (57× more) Bottleneck Analysis FLOPs model: 61 × (Proj + Attn + FFN) KV Cache: n × dim × 61 (mem killer) Bandwidth Bottleneck: Latency = (Weights + KV) / Mem_Bandwidth

CSA/HCA Compression Ratios by Layer

DeepSeek V4 uses a heterogeneous compression strategy across its 61 layers. HCA layers (128x compression) use locality-sensitive hashing for coarse retrieval. CSA layers (4x compression) provide fine-grained attention. The final layer (ratio=0) uses full attention for maximum expressiveness at the output stage.

HCA (128x compression)
CSA (4x compression)
Full Attention (0)

Per-Layer Pipeline Steps

Step Operation FLOPs Memory Access Note
1 Projection (X_n → Q,K,V) ~80.7M Read weights (~147MB) Current token only
2 Hash Retrieval (TopK) O(1) Read KV Cache Memory-intensive
3 Attention (Q_n · K_chunks) ~134.2M Read KV Cache (top-1024) Dot product + softmax
4 MoE FFN (active experts) ~924.8M Read expert weights Compute-intensive
5 Residual + Norm (mHC) O(d) Negligible Constant time
6 Cache Write (compress KV) O(d) Write KV Cache Append compressed

MLA vs MHA: The Dimension Transform

Subscriber Content

Continue reading with a subscription.

You are reading a free preview. The full analysis, unit economics tables, and investment-relevant conclusions are available to Research and Full Quant Intelligence subscribers.

Subscribe from $15/moAlready subscribed? Sign in

This research was produced by InAI Capital Advisor as part of our ongoing coverage of the global AI investment landscape. The analysis represents proprietary research conducted through expert network consultations and primary technical evaluation.