Production Deployment of DeepSeek-V4 and Kimi K3: 32-GPU Topologies, Memory Math, and Cluster Engineering

[!NOTE] Executive Summary (TL;DR)

• Architecture & Checkpoint Formats: DeepSeek-V4 employs hybrid CSA (Compressed Sparse Attention) + HCA (Heavily Compressed Attention) with FP8/FP4 sparse MoE layers; Kimi K3 (official repository moonshotai/Kimi-K3) features a 93-layer hybrid design (69 KDA recurrent linear attention layers + 24 Gated MLA layers) with native MXFP4 weights and MXFP8 activations.
• Cluster Topology & Interconnect: Hosting 600B+ MoE architectures requires intra-node TP8 across NVLink (900 GB/s bidirectional) combined with cross-node PP2/PP4 and Expert Parallelism (EP). Inter-node fabrics strictly mandate 8x 400Gbps InfiniBand (NDR) or RoCEv2 rail-optimized topologies; cross-node TP is strictly prohibited due to communication bottlenecks.
• Engine Parameters & Runtime Nuances: In vLLM, --dtype does not accept fp8 (use --dtype auto --quantization fp8); KDA operators demand specialized Triton or FlashKDA backends; multi-node Ray deployments require explicit /dev/shm IPC expansion.
• Cross-Disciplinary Deep Dives: Fabric architecture is detailed in InfiniBand Production Tuning and Lossless RoCEv2 Deployments; underlying attention mechanics are analyzed in KDA vs. DeepSeek MLA; runtime serving optimizations are covered in vLLM Production Serving and SGLang vs. vLLM Architecture Comparison.

Introduction: The Infrastructure Reality of Trillion-Parameter MoE Models

By 2026, state-of-the-art open-weights foundation models have universally converged on ultra-sparse Mixture-of-Experts (MoE) and hybrid recurrent/compressed attention mechanisms. Flagship architectures such as DeepSeek-V4-Pro and Moonshot's Kimi K3 achieve reasoning parity with proprietary frontier models in mathematical synthesis, extended code generation, and autonomous agent loops.

However, operating these models under stringent enterprise SLAs exposes three fundamental infrastructure challenges:

1. Memory Capacity vs. Interconnect Disparity: Trillion-parameter checkpoints cannot reside on a single 8-GPU node. Without precise parallel sharding (TP, PP, and EP), cross-node network traffic saturates InfiniBand fabrics, collapsing inference throughput.
2. Non-Standard Kernel Execution: DeepSeek-V4 replaces homogeneous attention with a heterogeneous pipeline of Compressed Sparse Attention (CSA) and Heavily Compressed Attention (HCA); Kimi K3 pairs 69 layers of recurrent linear attention (KDA) with 24 layers of Gated MLA. Off-the-shelf standard attention kernels cannot execute these dynamic computation graphs efficiently.
3. Enterprise TCO and Provisioning Economics: Provisioning dedicated 4-node (32 GPU) or 8-node (64 GPU) clusters incurs substantial CapEx amortization, power draw, and orchestration overhead. Architecture teams require rigorous token volume break-even models before committing to private hosting.

This guide provides an end-to-end engineering blueprint covering hardware topologies, memory calculations, distributed Ray setups, production startup scripts, Kubernetes manifests, and cost models.

1. Architectural Foundations and Native Checkpoint Formats

Before provisioning hardware, infrastructure architects must inspect the mathematical mechanics of both target models.

Architectural Overview

graph TD
    subgraph DeepSeek_V4["DeepSeek-V4-Pro (61 Layers)"]
        D_In["Input Tokens"] --> D_CSA["CSA: Compressed Sparse Attention<br/>(Local dense + sparse global tokens)"]
        D_CSA --> D_HCA["HCA: Heavily Compressed Attention<br/>(Ultra-low-rank latent KV projections)"]
        D_HCA --> D_MoE["MoE Routing Layer<br/>(Fine-grained Top-K, FP4/FP8 experts)"]
    end

    subgraph Kimi_K3["Kimi K3 (93 Layers)"]
        K_In["Input Tokens"] --> K_KDA["69 Layers of KDA<br/>(Recurrent state updates, sublinear KV growth)"]
        K_KDA --> K_GMLA["24 Layers of Gated MLA<br/>(Precision long-range multi-head associative retrieval)"]
        K_GMLA --> K_MoE["MoE Routing Layer<br/>(MXFP4 weights, MXFP8 activations)"]
    end

• DeepSeek-V4-Pro (61 Layers):
  • Employs a layered combination of Compressed Sparse Attention (CSA) and Heavily Compressed Attention (HCA). CSA maintains local sliding-window attention alongside dynamically sampled global anchors, while HCA applies aggressive low-rank latent projections to background context.
  • The MoE layer routes to fine-grained experts, activating ~49B parameters per step while keeping the full 1.6T parameter footprint resident across cluster VRAM.
• Kimi K3 (93 Layers):
  • Features a hybrid stack of 69 KDA layers + 24 Gated MLA layers. KDA (Kimi Dynamic Attention) maintains recurrent linear states whose memory footprints remain bounded regardless of sequence depth. The subsequent 24 Gated MLA layers handle precise global associative recall.
  • Built on quantization-aware training (QAT) with native MXFP4 weights and MXFP8 activations, dramatically reducing raw model byte volume.

Official Specifications Comparison

[!CAUTION] Crucial Capacity Realities

1. Active Parameters Do Not Dictate Memory Allocation: An active budget of 49B or 104B reflects forward FLOPs, not resident weights. Because tokens dynamically route to arbitrary experts, all expert parameters must reside permanently in VRAM to avoid catastrophic host-memory swapping delays.
2. Canonical Repositories: The official open-weights repository is moonshotai/Kimi-K3; verify signatures before pulling third-party repackaged containers.

2. Hardware Topologies and Fabric Interconnects

Trillion-parameter distributed serving exposes an order-of-magnitude bandwidth disparity: intra-node NVLink delivers 900 GB/s bidirectional bandwidth per GPU, whereas inter-node NDR InfiniBand delivers ~50 GB/s (400 Gbps).

4-Node (32x H100) Cluster Interconnect Topology

Production deployments typically rely on 4 HGX H100/H800 (8x SXM5 80GB) nodes wired in a rail-optimized non-blocking fabric:

flowchart TB
    subgraph Fabric["Non-Blocking Datacenter InfiniBand Fabric (Leaf-Spine)"]
        SW1["Spine Switch 1"]
        SW2["Spine Switch 2"]
        Leaf1["Leaf Switch 1"]
        Leaf2["Leaf Switch 2"]
        SW1 --- Leaf1 & Leaf2
        SW2 --- Leaf1 & Leaf2
    end

    subgraph Node0["Node 0 (Head Node - 8x H100 SXM5)"]
        GPU0["GPU 0..7 (TP=8)"]
        NVS0["NVSwitch (900 GB/s)"]
        NIC0["8x ConnectX-7 400G NICs"]
        GPU0 <--> NVS0
        GPU0 <--> NIC0
    end

    subgraph Node1["Node 1 (Worker 1 - 8x H100 SXM5)"]
        GPU1["GPU 8..15 (TP=8)"]
        NVS1["NVSwitch (900 GB/s)"]
        NIC1["8x ConnectX-7 400G NICs"]
        GPU1 <--> NVS1
        GPU1 <--> NIC1
    end

    subgraph Node2["Node 2 (Worker 2 - 8x H100 SXM5)"]
        GPU2["GPU 16..23 (TP=8)"]
        NVS2["NVSwitch (900 GB/s)"]
        NIC2["8x ConnectX-7 400G NICs"]
        GPU2 <--> NVS2
        GPU2 <--> NIC2
    end

    subgraph Node3["Node 3 (Worker 3 - 8x H100 SXM5)"]
        GPU3["GPU 24..31 (TP=8)"]
        NVS3["NVSwitch (900 GB/s)"]
        NIC3["8x ConnectX-7 400G NICs"]
        GPU3 <--> NVS3
        GPU3 <--> NIC3
    end

    NIC0 <==> Leaf1
    NIC1 <==> Leaf1
    NIC2 <==> Leaf2
    NIC3 <==> Leaf2

Parallel Partitioning Strategy

1. Intra-Node Tensor Parallelism (TP = 8): Matrix multiplications across self-attention projections and MLP dense layers are sharded across 8 intra-node GPUs. Because every token step executes multiple All-Reduce collectives, these must remain on NVLink (3.2 TB/s aggregate per node). Setting TP > 8 across physical nodes is strictly prohibited; doing so pushes All-Reduce over network NICs, increasing per-token latency from milliseconds to seconds.
2. Inter-Node Pipeline Parallelism (PP = 2 or 4): The transformer layer stack is partitioned across physical nodes. Inter-node transfers are restricted to boundary activation tensors, reducing network volume to megabytes per step, which is easily handled by 400Gbps fabrics.
3. Expert Parallelism (EP): MoE experts are distributed across nodes. Inter-node token dispatching occurs via All-to-All collectives, which modern engines overlap with dense attention computation.

3. Per-Rank GPU Memory Math and Budget Modeling

Capacity planning must evaluate the most constrained single GPU rank rather than aggregate cluster numbers.

Mathematical Breakdown

The memory consumption on any GPU rank $i$ follows:

$$M_{\text{rank}} = M_{\text{weight}} + M_{\text{activation}} + M_{\text{kv_cache}} + M_{\text{comm_buffer}} + M_{\text{cuda_overhead}}$$

1. Static Weights ($M_{\text{weight}}$): Quantized weights sharded across TP and PP/EP partitions.
2. Peak Activations ($M_{\text{activation}}$): Transient buffers generated during forward attention passes, governed by batch size and chunk budgets.
3. Communication Ring Buffers ($M_{\text{comm_buffer}}$): Pinned host/device memory for NCCL All-to-All and All-Reduce operations (typically 2GB–6GB).
4. CUDA Runtime & Graph Pool ($M_{\text{cuda_overhead}}$): Driver contexts, cuBLAS/cuDNN workspaces, and captured CUDAGraph execution plans (1.5GB–3.5GB).
5. Dynamic KV Cache Pool ($M_{\text{kv_cache}}$): Residual memory allocated to PagedAttention block pools.

Single-Rank Budget Allocation (32x H100 80GB)

Budget allocation for DeepSeek-V4-Pro under a 32-GPU hybrid topology (TP8 x PP4):

[!NOTE] Capacity Assessment
With ~26.0 GB of dedicated KV Cache memory per GPU, and benefiting from DeepSeek-V4's CSA+HCA low-rank latent compression (reducing KV memory by ~80% relative to standard MHA), a 32-GPU cluster comfortably supports over 160,000 active concurrent tokens.

4. Multi-Node Ray Cluster and Production Startup Scripts

Production deployment across 4 physical nodes (Node 0: 192.168.10.101, Node 1: 192.168.10.102, Node 2: 192.168.10.103, Node 3: 192.168.10.104).

Step 1: Pre-Flight Hardware Checks

Execute on all nodes:

# 1. Verify 8 GPUs and NVLink topology
nvidia-smi
nvidia-smi topo -m

# 2. Verify all 8 ConnectX-7 NICs are Active at 400 Gbps (NDR)
ibstat | grep -E "(CA '|State:|Physical state:|Rate:)"

# 3. Verify shared memory allocation (requires >= 64GB)
df -h /dev/shm

Step 2: Initialize Ray Multi-Node Cluster

Execute on Node 0 (Head Node):

#!/usr/bin/env bash
ray stop --force
ray start --head \
    --port=6379 \
    --node-ip-address=192.168.10.101 \
    --num-gpus=8 \
    --object-store-memory=68719476736 \
    --block

Execute on Worker Nodes (Nodes 1, 2, and 3):

#!/usr/bin/env bash
ray stop --force
ray start \
    --address='192.168.10.101:6379' \
    --node-ip-address=$(hostname -I | awk '{print $1}') \
    --num-gpus=8 \
    --object-store-memory=68719476736 \
    --block

Validate cluster topology on the Head Node with ray status (must report 4 Nodes, 32 GPUs).

Step 3: Production vLLM Deployment Script

Launch serving on Node 0:

#!/usr/bin/env bash
set -eo pipefail

export CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7
export NCCL_DEBUG=INFO
export NCCL_IB_DISABLE=0
export NCCL_IB_HCA=mlx5_0,mlx5_1,mlx5_2,mlx5_3,mlx5_4,mlx5_5,mlx5_6,mlx5_7
export NCCL_IB_GID_INDEX=3
export NCCL_CROSS_NIC=1
export NCCL_SOCKET_IFNAME=eth0
export VLLM_HOST_IP=192.168.10.101

# Launch DeepSeek-V4-Pro Production Service
vllm serve deepseek-ai/DeepSeek-V4-Pro \
    --host 0.0.0.0 \
    --port 8000 \
    --tensor-parallel-size 8 \
    --pipeline-parallel-size 4 \
    --distributed-executor-backend ray \
    --dtype auto \
    --quantization fp8 \
    --gpu-memory-utilization 0.90 \
    --max-model-len 32768 \
    --max-num-seqs 256 \
    --enable-chunked-prefill \
    --max-num-batched-tokens 4096 \
    --trust-remote-code \
    --disable-log-requests

[!TIP] Parameter Best Practices

• --dtype auto and --quantization fp8: Do not specify --dtype fp8. Internal arithmetic is maintained in native precision (auto), while FP8 quantization scales are loaded via --quantization fp8.
• --enable-chunked-prefill: Caps single-step prefill chunks at 4096 tokens, preventing long prompts from degrading Time-Per-Output-Token (TPOT) on concurrent streams.
• --pipeline-parallel-size 4: Intra-node TP8 combined with inter-node PP4 scales across all 32 GPUs without pushing tensor parallelism across network interfaces.

Step 4: Kimi K3 Specialized Serving with FlashKDA

For Kimi K3, specify the dedicated recurrent kernel backend:

#!/usr/bin/env bash
vllm serve moonshotai/Kimi-K3 \
    --host 0.0.0.0 \
    --port 8000 \
    --tensor-parallel-size 8 \
    --pipeline-parallel-size 4 \
    --distributed-executor-backend ray \
    --dtype auto \
    --kda-prefill-backend triton \
    --gpu-memory-utilization 0.92 \
    --max-model-len 65536 \
    --trust-remote-code

5. Enterprise Kubernetes Deployment with KubeRay

In containerized Kubernetes environments, clusters are managed declaratively using the KubeRay Operator:

apiVersion: ray.io/v1
kind: RayCluster
metadata:
  name: deepseek-v4-cluster
  namespace: llm-serving
spec:
  rayVersion: '2.38.0'
  headGroupSpec:
    rayStartParams:
      dashboard-host: '0.0.0.0'
      num-gpus: '8'
    template:
      spec:
        nodeSelector:
          node.kubernetes.io/instance-type: hgx-h100-8gpu
        containers:
        - name: ray-head
          image: vllm/vllm-openai:v0.6.3
          resources:
            limits:
              nvidia.com/gpu: 8
              memory: 512Gi
              cpu: "64"
            requests:
              nvidia.com/gpu: 8
              memory: 256Gi
              cpu: "32"
          volumeMounts:
          - mountPath: /dev/shm
            name: dshm
          - mountPath: /root/.cache/huggingface
            name: model-cache
        volumes:
        - name: dshm
          emptyDir:
            medium: Memory
            sizeLimit: 128Gi
        - name: model-cache
          persistentVolumeClaim:
            claimName: nfs-model-weights-pvc
  workerGroupSpecs:
  - groupName: gpu-workers
    replicas: 3
    minReplicas: 3
    maxReplicas: 3
    rayStartParams:
      num-gpus: '8'
    template:
      spec:
        nodeSelector:
          node.kubernetes.io/instance-type: hgx-h100-8gpu
        containers:
        - name: ray-worker
          image: vllm/vllm-openai:v0.6.3
          resources:
            limits:
              nvidia.com/gpu: 8
              memory: 512Gi
              cpu: "64"
          volumeMounts:
          - mountPath: /dev/shm
            name: dshm
          - mountPath: /root/.cache/huggingface
            name: model-cache
        volumes:
        - name: dshm
          emptyDir:
            medium: Memory
            sizeLimit: 128Gi
        - name: model-cache
          persistentVolumeClaim:
            claimName: nfs-model-weights-pvc

[!IMPORTANT] Shared Memory Configuration
Mount /dev/shm backed by medium: Memory (>= 128Gi). NCCL intra-node collectives rely on shared memory for GPU-to-GPU transfers. Default container limits (64MB) trigger immediate Bus error (core dumped) crashes.

6. Benchmarks and Production SLA Target Baselines

Verify latency and throughput under production traffic using vLLM's benchmark_serving.py:

python3 -m vllm.benchmarks.benchmark_serving \
    --backend vllm \
    --model deepseek-ai/DeepSeek-V4-Pro \
    --dataset-name sharegpt \
    --dataset-path ./ShareGPT_V3_unfiltered_cleaned_split.json \
    --num-prompts 1000 \
    --request-rate 16 \
    --max-concurrency 64

Production SLA Baseline Thresholds

A healthy 32-GPU H100 cluster must achieve the following metrics:

==================== vLLM Serving Benchmark Summary ====================
Successful requests:             1000 / 1000 (100%)
Benchmark duration:              62.5 s
Total input tokens:              238,400
Total generated tokens:          196,800
Request throughput:              16.0 req/s
Token throughput (total):        6,963.2 tokens/s
Token throughput (generation):   3,148.8 tokens/s

[Latency Metrics]
Time to First Token (TTFT):
  Mean:                          48.2 ms
  P50:                           36.1 ms
  P95:                           112.5 ms
  P99:                           168.0 ms  <-- Target: < 250 ms

Time per Output Token (TPOT):
  Mean:                          18.4 ms
  P50:                           16.2 ms
  P95:                           24.8 ms
  P99:                           31.5 ms  <-- Target: < 40 ms (>= 25 tokens/s)
========================================================================

7. Enterprise TCO Economics: Self-Hosting vs. Commercial APIs

flowchart LR
    A["Daily Enterprise Token Volume"] --> B{"Is daily volume sustained<br/>above 1 Billion tokens?"}
    B -- "No (Early Stage / Bursty)" --> C["Commercial API Strategy<br/>(Zero CapEx, pay-as-you-go)"]
    B -- "Yes (Core High-Scale Services)" --> D{"Strict Data Sovereignty<br/>or Custom Weight Needs?"}
    D -- "Yes" --> E["Self-Hosted 32-GPU Cluster<br/>(On-prem isolation, dedicated compute)"]
    D -- "No" --> F{"Evaluate Duty Cycle<br/>(Peak-to-Trough Variance)"}
    F -- "High idle nights (>70%)" --> C
    F -- "Sustained high load" --> E

Total Cost of Ownership (32x H100 80GB)

Break-Even Volume Analysis

Assuming commercial APIs average $2.50 per million blended tokens (incorporating weighted prompt caching and completion pricing):

$$\text{Monthly Break-Even Volume} = \frac{\$57,000}{\$2.50 / 10^6} = 22,800,000,000 \text{ tokens (22.8 Billion/month)}$$

$$\text{Daily Break-Even Volume} \approx 760,000,000 \text{ tokens/day}$$

Decision Matrix

• Below 500M Tokens/Day: Commercial APIs remain significantly more cost-effective, eliminating hardware idle losses and operations overhead.
• Above 1B–2B Tokens/Day with High Utilization: Dedicated 32-GPU self-hosting reduces effective token costs by over 40% while delivering guaranteed latency predictability and data isolation.

Frequently Asked Questions (FAQ)

Q1: Can a 32-card H100 80GB cluster host Kimi K3 in full FP8 precision?

No. Kimi K3 contains 2.8T parameters. In FP8 precision (1 byte per parameter), raw uncompressed weights alone consume 2.8TB, exceeding the nominal physical memory of 32x 80GB GPUs ($32 \times 80 = 2560\text{GB} = 2.56\text{TB}$). This leaves zero memory for CUDA contexts, collective buffers, activations, and KV Cache pools. A 32-GPU deployment is made feasible strictly through Kimi K3's native MXFP4 checkpoint format (~0.5 bytes per parameter, ~1.4TB total weight footprint), preserving ~1TB of cluster memory for KV Cache pools.

Q2: Why is setting Tensor Parallelism (TP > 8) across physical nodes prohibited?

Tensor Parallelism requires every GPU in the TP group to perform two high-frequency All-Reduce collectives per transformer layer. Intra-node, this runs across NVLink at 900 GB/s with microsecond latency. Pushing TP across network nodes limits bandwidth to 400Gbps (~50 GB/s)—18 times slower than NVLink—introducing substantial packet processing overhead. This starves GPU Tensor Cores, collapsing generation speeds. Multi-node setups must isolate TP to individual nodes (TP=8) and scale out via Pipeline Parallelism (PP) or Expert Parallelism (EP).

Q3: What is the architectural difference between DeepSeek-V4's CSA+HCA and V3's MLA?

DeepSeek-V3's Multi-Head Latent Attention (MLA) compresses keys and values into a single low-rank latent vector $c_t^{KV}$, mathematically absorbing projection matrices during decoding. DeepSeek-V4-Pro's Compressed Sparse Attention (CSA) and Heavily Compressed Attention (HCA) form a heterogeneous tiered pipeline: CSA preserves localized sliding-window context and sparse global anchors, while HCA applies ultra-low-rank compression to long-range background tokens. This hybrid approach slashes KV memory requirements further while boosting prefill processing throughput across 128k contexts.

Q4: How should infrastructure teams detect and resolve MoE Expert Routing Imbalance?

When tokens concentrate disproportionately on specific experts, the GPU ranks hosting those experts experience high compute latency, stalling the global All-to-All collective. Infrastructure teams should monitor per-expert token dispatch counters via Prometheus. If significant routing skew emerges, verify that system prompts do not contain bias-inducing repetitive keywords, configure dynamic auxiliary loss coefficients in the engine scheduler, or deploy duplicated redundant instances for heavily loaded experts.

Q5: How do you troubleshoot intermittent NCCL Watchdog caught collective operation timeout errors?

Collective operation timeouts indicate network degradation or a stalled GPU rather than internal NCCL engine bugs. Troubleshoot via:

1. Enable Detailed Tracing: Set NCCL_DEBUG=INFO and NCCL_DEBUG_SUBSYS=INIT,ENV,COLL to identify the failing collective (AllGather vs. AlltoAll).
2. Verify IB Ports & GID: Run ibstat across all nodes to confirm ports are in the Active state with identical NCCL_IB_GID_INDEX configurations.
3. Execute Point-to-Point Benchmarks: Run ib_write_bw across every pair of nodes to detect asymmetric bandwidth drops (e.g., ports degrading from 400G to 100G due to optical transceiver issues).
4. Audit Shared Memory Allocation: Verify /dev/shm utilization on all nodes; shared memory exhaustion halts intra-node IPC handshakes.

Originally published at LLM Systems Architecture on blog.llmgo.top.