Qwen3.8-2.4T-A95B¶

1 简介¶

Qwen3.8-2.4T-A95B 是一个大规模 MoE 模型，总参数为 24000 亿，激活参数为 950 亿。它基于 Qwen3.5 架构构建，并提升了编码、专业工作、科学研究以及长时程智能体任务的能力。

本文档描述该模型的主要验证步骤，包括支持的功能、前提条件、安装、多节点部署、功能验证、精度与性能评估、性能调优以及常见问题。已验证的配置涵盖 Atlas A3 和 Atlas A2 部署。

本文档基于 vLLM-Ascend 0.23.0 进行验证和编写。当前模型（Qwen3.8-2.4T-A95B）首次在该版本中获得支持。

2 支持的功能¶

参见支持模型以获取模型支持矩阵。

参见功能指南以获取功能配置详情。

3 前提条件¶

3.1 模型权重¶

以下模型权重可用：

权重版本	硬件要求	下载链接
`Qwen3.8-2.4T-A95B` (FP16/BF16)	约 4.89 TB 的存储和权重内存	ModelScope
`Qwen3.8-2.4T-A95B-w8a8`	约 2.33 TiB 的存储和权重内存	ModelScope
`Qwen3.8-2.4T-A95B-w4a8`	约 1.21 TiB 的存储和权重内存	ModelScope

本指南包含以下已验证的部署配置：

平台	权重	部署	拓扑
4 × Atlas 800 A3 (64GB × 16)	W8A8	混合 Prefill/Decode 部署	DP4/TP16/EP64
8 × Atlas 800 A2 (64GB × 8)	W4A8	混合 Prefill/Decode 部署	DP8/TP8/EP64

在所有服务节点上，检查点和分词器目录必须位于相同路径。W8A8 部署使用惰性 Safetensors 策略，以避免从共享存储预取完整检查点。

建议将模型权重下载到多节点共享目录，例如 /root/.cache/。

路径说明：将模型权重下载到您选择的目录并记录该目录。确保后续部署命令中的模型路径与该目录一致。

3.2 验证多节点通信（可选）¶

如果您想在多节点环境中部署模型，请根据验证多节点通信环境验证通信环境。

4 安装¶

4.1 Docker 镜像安装¶

根据您的机型选择镜像，并在您的节点上启动 Docker 镜像。参见使用预构建镜像。

A3 系列A2 系列

在每个节点上启动 Docker 镜像。

export IMAGE=quay.io/ascend/vllm-ascend:qwen3.8-a3
export NAME=vllm-ascend

docker run --rm \
  --name $NAME \
  --net=host \
  --shm-size=1g \
  --device /dev/davinci0 \
  --device /dev/davinci1 \
  --device /dev/davinci2 \
  --device /dev/davinci3 \
  --device /dev/davinci4 \
  --device /dev/davinci5 \
  --device /dev/davinci6 \
  --device /dev/davinci7 \
  --device /dev/davinci8 \
  --device /dev/davinci9 \
  --device /dev/davinci10 \
  --device /dev/davinci11 \
  --device /dev/davinci12 \
  --device /dev/davinci13 \
  --device /dev/davinci14 \
  --device /dev/davinci15 \
  --device /dev/davinci_manager \
  --device /dev/devmm_svm \
  --device /dev/hisi_hdc \
  -v /usr/local/dcmi:/usr/local/dcmi \
  -v /usr/local/Ascend/driver/tools/hccn_tool:/usr/local/Ascend/driver/tools/hccn_tool \
  -v /usr/local/bin/npu-smi:/usr/local/bin/npu-smi \
  -v /usr/local/Ascend/driver/lib64/:/usr/local/Ascend/driver/lib64/ \
  -v /usr/local/Ascend/driver/version.info:/usr/local/Ascend/driver/version.info \
  -v /etc/ascend_install.info:/etc/ascend_install.info \
  -v /root/.cache:/root/.cache \
  -it $IMAGE bash

在每个节点上启动 Docker 镜像。

export IMAGE=quay.io/ascend/vllm-ascend:qwen3.8-a2
export NAME=vllm-ascend

docker run --rm \
  --name $NAME \
  --net=host \
  --shm-size=1g \
  --device /dev/davinci0 \
  --device /dev/davinci1 \
  --device /dev/davinci2 \
  --device /dev/davinci3 \
  --device /dev/davinci4 \
  --device /dev/davinci5 \
  --device /dev/davinci6 \
  --device /dev/davinci7 \
  --device /dev/davinci_manager \
  --device /dev/devmm_svm \
  --device /dev/hisi_hdc \
  -v /usr/local/dcmi:/usr/local/dcmi \
  -v /usr/local/Ascend/driver/tools/hccn_tool:/usr/local/Ascend/driver/tools/hccn_tool \
  -v /usr/local/bin/npu-smi:/usr/local/bin/npu-smi \
  -v /usr/local/Ascend/driver/lib64/:/usr/local/Ascend/driver/lib64/ \
  -v /usr/local/Ascend/driver/version.info:/usr/local/Ascend/driver/version.info \
  -v /etc/ascend_install.info:/etc/ascend_install.info \
  -v /root/.cache:/root/.cache \
  -it $IMAGE bash

进入容器后，验证 vLLM 和 vLLM-Ascend 是否可以导入：

python -c "import vllm, vllm_ascend; print('vllm and vllm_ascend are ready')"

预期输出：

vllm and vllm_ascend are ready

4.2 源码安装¶

您也可以从源码构建并安装 vllm-ascend。参见使用 Python 设置。

如果您想部署多节点服务，请在每个节点上安装相同版本的 vLLM 和 vLLM-Ascend。

5 在线服务部署¶

5.1 多节点部署¶

经过验证的混合部署在每个节点上运行一个 DP rank，在每个节点内使用张量并行，并使专家并行跨越所有节点。请选择与您平台匹配的选项卡：

A3 系列：四台 Atlas 800 A3（64GB × 16）节点，DP4/TP16/EP64，使用 W8A8 检查点。
A2 系列：八台 Atlas 800 A2（64GB × 8）节点，DP8/TP8/EP64，使用 W4A8 检查点。

启动服务前：

将模型路径、节点数、并行大小、本地 IP 地址、网络接口、服务端口和 RPC 端口替换为目标环境中的值。
NIC_NAME 必须是拥有 LOCAL_IP 的接口。
先启动节点 0。每个工作节点上的 NODE0_IP 必须等于节点 0 上的 LOCAL_IP。
为每个工作节点分配唯一的 DP_START_RANK。

A3 系列

节点 0节点 1-3

# Values that must be adapted to the target environment.
export MODEL_PATH=<QWEN3_8_MODEL_PATH>
export TOKENIZER_PATH=<QWEN3_8_TOKENIZER_PATH>
export LOCAL_IP=<NODE0_LOCAL_IP>
export NIC_NAME=<NODE0_NIC_NAME>
export PORT=<SERVICE_PORT>
export RPC_PORT=<DP_RPC_PORT>
export DP_SIZE=4
export TP_SIZE=16

export HCCL_BUFFSIZE_EP=2048
export VLLM_EXECUTE_MODEL_TIMEOUT_SECONDS=3000
export HCCL_BUFFSIZE=1024
export HCCL_IF_IP=$LOCAL_IP
export HCCL_INTRA_ROCE_ENABLE=0
export HCCL_OP_EXPANSION_MODE="AIV"
export HCCL_SOCKET_IFNAME=$NIC_NAME
export ASCEND_RT_VISIBLE_DEVICES=0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15
export GLOO_SOCKET_IFNAME=$NIC_NAME
export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True
export OPENBLAS_NUM_THREADS=1

# Ensure the model path matches the directory recorded during download
vllm serve $MODEL_PATH \
    --host 0.0.0.0 \
    --port $PORT \
    --served-model-name qwen3.8 \
    --tokenizer $TOKENIZER_PATH \
    --trust-remote-code \
    --quantization ascend \
    --safetensors-load-strategy lazy \
    --tensor-parallel-size $TP_SIZE \
    --data-parallel-size $DP_SIZE \
    --data-parallel-size-local 1 \
    --data-parallel-address $LOCAL_IP \
    --data-parallel-rpc-port $RPC_PORT \
    --enable-prefix-caching \
    --enable-expert-parallel \
    --max-model-len 131072 \
    --max-num-seqs 8 \
    --max-num-batched-tokens 16384 \
    --gpu-memory-utilization 0.85 \
    --speculative-config '{"method":"qwen3_5_mtp","num_speculative_tokens":1}' \
    --compilation-config '{"cudagraph_mode":"FULL_DECODE_ONLY"}' \
    --additional-config '{"enable_cpu_binding":true,"enable_fused_mc2":1}'

在每个工作节点上运行此命令。将 LOCAL_IP 和 NIC_NAME 设置为当前节点的值，并将 DP_START_RANK 设置为 1、2 或 3。

# Values that must be adapted to the target environment.
export MODEL_PATH=<QWEN3_8_MODEL_PATH>
export TOKENIZER_PATH=<QWEN3_8_TOKENIZER_PATH>
export LOCAL_IP=<WORKER_LOCAL_IP>
export NODE0_IP=<NODE0_LOCAL_IP>
export NIC_NAME=<WORKER_NIC_NAME>
export PORT=<SERVICE_PORT>
export RPC_PORT=<DP_RPC_PORT>
export DP_SIZE=4
export DP_START_RANK=<1_OR_2_OR_3>
export TP_SIZE=16

export HCCL_BUFFSIZE_EP=2048
export VLLM_EXECUTE_MODEL_TIMEOUT_SECONDS=3000
export HCCL_BUFFSIZE=1024
export HCCL_IF_IP=$LOCAL_IP
export HCCL_INTRA_ROCE_ENABLE=0
export HCCL_OP_EXPANSION_MODE="AIV"
export HCCL_SOCKET_IFNAME=$NIC_NAME
export ASCEND_RT_VISIBLE_DEVICES=0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15
export GLOO_SOCKET_IFNAME=$NIC_NAME
export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True
export OPENBLAS_NUM_THREADS=1

# Ensure the model path matches the directory recorded during download
vllm serve $MODEL_PATH \
    --headless \
    --host 0.0.0.0 \
    --port $PORT \
    --served-model-name qwen3.8 \
    --tokenizer $TOKENIZER_PATH \
    --trust-remote-code \
    --quantization ascend \
    --safetensors-load-strategy lazy \
    --tensor-parallel-size $TP_SIZE \
    --data-parallel-size $DP_SIZE \
    --data-parallel-size-local 1 \
    --data-parallel-start-rank $DP_START_RANK \
    --data-parallel-address $NODE0_IP \
    --data-parallel-rpc-port $RPC_PORT \
    --enable-prefix-caching \
    --enable-expert-parallel \
    --max-model-len 131072 \
    --max-num-seqs 8 \
    --max-num-batched-tokens 16384 \
    --gpu-memory-utilization 0.85 \
    --speculative-config '{"method":"qwen3_5_mtp","num_speculative_tokens":1}' \
    --compilation-config '{"cudagraph_mode":"FULL_DECODE_ONLY"}' \
    --additional-config '{"enable_cpu_binding":true,"enable_fused_mc2":1}'

以下值在主节点和工作节点之间有所不同：

设置	节点 0	节点 1-3	说明
`LOCAL_IP`	节点 0 IP	当前工作节点 IP	每个节点使用自己的通信 IP。
`NODE0_IP`	不需要	节点 0 IP	工作节点使用该地址加入 DP 组。
`--headless`	省略	启用	工作节点不暴露 API 端点。
`--data-parallel-address`	`$LOCAL_IP`	`$NODE0_IP`	始终解析为节点 0。
`--data-parallel-start-rank`	默认为 `0`	`1`、`2` 或 `3`	每个节点必须拥有唯一的 DP rank。

关键部署参数：

参数	说明
`--tensor-parallel-size 16`	使用单个 A3 节点中的全部 16 个 NPU 进行张量并行。

参数	描述
`--data-parallel-size 4`	在四个节点上创建四个全局 DP rank。
`--data-parallel-size-local 1`	在当前节点上运行一个 DP rank。
`--data-parallel-start-rank`	选择工作节点的全局 DP rank。
`--enable-expert-parallel`	为 MoE 层启用专家并行。
`--max-model-len 131072`	设置输入和输出的最大总长度。
`--quantization ascend`	启用 Ascend W8A8 量化。
`--safetensors-load-strategy lazy`	避免从 NFS 预取完整的 2.33 TiB 检查点。
`--max-num-seqs 8`	为每个 DP 组设置八个活跃序列，并在 DP4 中总计 32 个。
`--max-num-batched-tokens 16384`	控制调度器的 token 预算。
`--gpu-memory-utilization 0.85`	为运行时内存保留 HBM 余量。
`--enable-prefix-caching`	启用自动前缀缓存。
`--speculative-config`	启用一个 Qwen3.5 MTP 推测 token。
`--compilation-config`	使用 `FULL_DECODE_ONLY` ACL Graph 重放。
`--additional-config`	启用 CPU 绑定和 Fused MC2。

常见问题提示：如果工作节点立即退出，请确认节点 0 已在运行，
`--data-parallel-address` 解析到节点 0，所有节点使用相同的 RPC 端口，
并且每个工作节点使用唯一的 DP 起始 rank。

A2 系列

经过验证的混合部署使用八台 Atlas 800 A2 (64GB × 8) 节点，并采用 Qwen3.8-2.4T-A95B-w4a8 检查点。数据并行跨越八个节点，每个节点运行一个 DP rank，张量并行使用节点内全部 8 个 NPU，最终拓扑为 DP8/TP8/EP64。

在 A2 上使用 W4A8 检查点，以便为 95B 激活参数的大激活占用保留足够的 HBM。由于每个 token 的 MoE 中间缓冲区较大，与 A3 配置相比，--max-num-batched-tokens 降低到 4096， --gpu-memory-utilization 提高到 0.92。更小的 token 预算可将 MoE 激活峰值保持在足够低的水平，从而为 262144 上下文长度留出 KV cache 空间。

节点 0节点 1-7

# Values that must be adapted to the target environment.
export MODEL_PATH=<QWEN3_8_MODEL_PATH>
export TOKENIZER_PATH=<QWEN3_8_TOKENIZER_PATH>
export LOCAL_IP=<NODE0_LOCAL_IP>
export NIC_NAME=<NODE0_NIC_NAME>
export PORT=<SERVICE_PORT>
export RPC_PORT=<DP_RPC_PORT>
export DP_SIZE=8
export TP_SIZE=8

export VLLM_ENGINE_READY_TIMEOUT_S=7200
export VLLM_EXECUTE_MODEL_TIMEOUT_SECONDS=3000
export HCCL_BUFFSIZE=1024
export HCCL_BUFFSIZE_EP=2048
export HCCL_CONNECT_TIMEOUT=1800
export HCCL_EXEC_TIMEOUT=1800
export HCCL_IF_IP=$LOCAL_IP
export HCCL_INTRA_PCIE_ENABLE=1
export HCCL_INTRA_ROCE_ENABLE=0
export HCCL_OP_EXPANSION_MODE="AIV"
export HCCL_SOCKET_IFNAME=$NIC_NAME
export ASCEND_RT_VISIBLE_DEVICES=0,1,2,3,4,5,6,7
export GLOO_SOCKET_IFNAME=$NIC_NAME
export TP_SOCKET_IFNAME=$NIC_NAME
export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True
export TASK_QUEUE_ENABLE=1
export OMP_PROC_BIND=false
export OMP_NUM_THREADS=1
export OPENBLAS_NUM_THREADS=1

# Ensure the model path matches the directory recorded during download
vllm serve $MODEL_PATH \
    --host 0.0.0.0 \
    --port $PORT \
    --served-model-name qwen3.8 \
    --tokenizer $TOKENIZER_PATH \
    --trust-remote-code \
    --quantization ascend \
    --safetensors-load-strategy lazy \
    --tensor-parallel-size $TP_SIZE \
    --data-parallel-size $DP_SIZE \
    --data-parallel-size-local 1 \
    --data-parallel-address $LOCAL_IP \
    --data-parallel-rpc-port $RPC_PORT \
    --enable-prefix-caching \
    --enable-expert-parallel \
    --max-model-len 262144 \
    --max-num-seqs 8 \
    --max-num-batched-tokens 4096 \
    --gpu-memory-utilization 0.92 \
    --speculative-config '{"method":"qwen3_5_mtp","num_speculative_tokens":1}' \
    --compilation-config '{"cudagraph_mode":"FULL_DECODE_ONLY"}' \
    --additional-config '{"enable_cpu_binding":true,"enable_fused_mc2":1}'

在每个工作节点上运行此命令。将 LOCAL_IP 和 NIC_NAME 设置为当前节点，并将 DP_START_RANK 设置为 1 到 7 之间的唯一值。

# Values that must be adapted to the target environment.
export MODEL_PATH=<QWEN3_8_MODEL_PATH>
export TOKENIZER_PATH=<QWEN3_8_TOKENIZER_PATH>
export LOCAL_IP=<WORKER_LOCAL_IP>
export NODE0_IP=<NODE0_LOCAL_IP>
export NIC_NAME=<WORKER_NIC_NAME>
export PORT=<SERVICE_PORT>
export RPC_PORT=<DP_RPC_PORT>
export DP_SIZE=8
export DP_START_RANK=<1_TO_7>
export TP_SIZE=8

export VLLM_ENGINE_READY_TIMEOUT_S=7200
export VLLM_EXECUTE_MODEL_TIMEOUT_SECONDS=3000
export HCCL_BUFFSIZE=1024
export HCCL_BUFFSIZE_EP=2048
export HCCL_CONNECT_TIMEOUT=1800
export HCCL_EXEC_TIMEOUT=1800
export HCCL_IF_IP=$LOCAL_IP
export HCCL_INTRA_PCIE_ENABLE=1
export HCCL_INTRA_ROCE_ENABLE=0
export HCCL_OP_EXPANSION_MODE="AIV"
export HCCL_SOCKET_IFNAME=$NIC_NAME
export ASCEND_RT_VISIBLE_DEVICES=0,1,2,3,4,5,6,7
export GLOO_SOCKET_IFNAME=$NIC_NAME
export TP_SOCKET_IFNAME=$NIC_NAME
export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True
export TASK_QUEUE_ENABLE=1
export OMP_PROC_BIND=false
export OMP_NUM_THREADS=1
export OPENBLAS_NUM_THREADS=1

# Ensure the model path matches the directory recorded during download
vllm serve $MODEL_PATH \
    --headless \
    --host 0.0.0.0 \
    --port $PORT \
    --served-model-name qwen3.8 \
    --tokenizer $TOKENIZER_PATH \
    --trust-remote-code \
    --quantization ascend \
    --safetensors-load-strategy lazy \
    --tensor-parallel-size $TP_SIZE \
    --data-parallel-size $DP_SIZE \
    --data-parallel-size-local 1 \
    --data-parallel-start-rank $DP_START_RANK \
    --data-parallel-address $NODE0_IP \
    --data-parallel-rpc-port $RPC_PORT \
    --enable-prefix-caching \
    --enable-expert-parallel \
    --max-model-len 262144 \
    --max-num-seqs 8 \
    --max-num-batched-tokens 4096 \
    --gpu-memory-utilization 0.92 \
    --speculative-config '{"method":"qwen3_5_mtp","num_speculative_tokens":1}' \
    --compilation-config '{"cudagraph_mode":"FULL_DECODE_ONLY"}' \
    --additional-config '{"enable_cpu_binding":true,"enable_fused_mc2":1}'

, followed by the final answer. The service returns HTTP 200 and achoices` field containing generated text. Worker nodes are headless and do not accept HTTP requests directly.

The model officially supports these reasoning effort levels:

xhigh: default; deeper reasoning for complex tasks.
medium: balances accuracy and speed.
low: prioritizes speed and cost.

7 Accuracy Evaluation¶

7.1 Prepare the Evaluation Environment¶

Refer to Using AISBench for the evaluation environment and dataset preparation.

7.2 Using AISBench¶

Refer to Using AISBench for details.
After execution, you can get the result. Here are the GPQA Diamond results of Qwen3.8-2.4T-A95B-w4a8 and Qwen3.8-2.4T-A95B-w8a8 for reference only.

dataset	model	metric	mode	vllm-api-general-chat
GPQA Diamond	Qwen3.8-2.4T-A95B-w4a8	accuracy	gen	91.92%
GPQA Diamond	Qwen3.8-2.4T-A95B-w8a8	accuracy	gen	93.75%

8 Performance Evaluation¶

8.1 Install AISBench¶

Run AISBench in a separate environment or container so that the load generator does not affect the serving processes. Refer to Using AISBench for performance evaluation.

8.2 Performance Service Configuration¶

Use the deployment in Section 5 as the baseline. Change the following values on all four nodes:

Parameter	Standard deployment	Performance test
`--max-model-len`	131072	250000
`--max-num-batched-tokens`	16384	8192
`--gpu-memory-utilization`	0.85	0.95

8.3 Run the Tests¶

Run performance evaluation of Qwen3.8-2.4T-A95B as an example. Refer to vLLM benchmark for more details.

vllm bench serve \
  --model Qwen/Qwen3.8-2.4T-A95B \
  --served-model-name qwen3.8 \
  --base-url http://<server_ip>:8000 \
  --dataset-name random \
  --random-input-len 8192 \
  --random-output-len 1024 \
  --num-prompts 16 \
  --max-concurrency 4 \
  --save-result \
  --result-dir ./

Record the A3 node count, TP/PP/EP topology, context length, concurrency, reasoning effort, and weight revision together with the result."

Need translate prose. Headings:

5.2 Prefill-Decode Disaggregation -> ### 5.2 预填充-解码分离 {#prefill-decode-disaggregation}? Rule: for every heading append its original English anchor via attr_list, e.g. "## Install Guide" -> "## 安装指南 {#install-guide}" (slug = lowercase words joined by hyphens, from ORIGINAL English heading). Need include number? Original heading "### 5.2 Prefill-Decode Disaggregation". Slug from original English heading: lowercase words joined by hyphens. How to handle numbers and punctuation? Usually GitHub slug: remove punctuation, lowercase, spaces to hyphens. "5.2 Prefill-Decode Disaggregation" -> "52-prefill-decode-disaggregation"? The example says slug = lowercase words joined by hyphens, from original English heading. It might mean words only, no punctuation. For "5.2 Prefill-Decode Disaggregation", words: 5.2? If lowercase words joined by hyphens, maybe "52-prefill-decode-disaggregation" because dot removed? Need infer. They say e.g. "## Install Guide" -> "## 安装指南 {#install-guide}". For headings with numbers, likely anchor should be original English heading slug. In many docs, "### 5.2 Prefill-Decode Disaggregation" anchor is "52-prefill-decode-disaggregation". Need use that. For "## 6 Functional Verification" -> "6-functional-verification". "### 7.1 Prepare the Evaluation Environment" -> "71-prepare-the-evaluation-environment". "### 7.2 Using AISBench" -> "72-using-aisbench". "## 8 Performance Evaluation" -> "8-performance-evaluation". "### 8.1 Install AISBench" -> "81-install-aisbench". "### 8.2 Performance Service Configuration" -> "82-performance-service-configuration". "### 8.3 Run the Tests" -> "83-run-the-tests". Need preserve heading levels and append attr_list.¶

Need translate table cells. Keep inline code unchanged. For table cells with code, keep code. Translate prose in cells. Need preserve Markdown structure. The initial table is indented 4 spaces? It appears as code? In Markdown, four spaces before table may make it code block? The user input has table indented by 4 spaces under paragraph? Let's inspect: "The following values differ between the master and worker nodes:

| Setting | Node 0 | Nodes 1-7 | Description |
| --- | --- | --- | --- |

..." In Markdown, a line starting with 4 spaces is indented code block if not part of list? But table with 4 spaces may not render as table. However rule says keep Markdown structure unchanged. We should preserve indentation exactly? It says keep ALL code blocks, inline code, URLs, HTML tags and Markdown structure unchanged. Translate only prose. So preserve leading spaces before table lines. Need output same indentation. But if we translate table cells, keep pipes and indentation.

Need be careful with line breaks. We can output translated Markdown with same structure. Could preserve line wrapping? Not necessary but keep structure. Need no preamble.

Need translate: "The following values differ between the master and worker nodes:" -> "主节点和工作节点之间的以下值有所不同：" maybe master node? In context master and worker nodes. Use "主节点" and "工作节点". Node 0? Keep Node 0? It's a label, maybe translate "节点 0"? Rule says do not translate code identifiers, commands, file paths, API names, brand/product names. Node 0 is not code but can translate? Table header "Node 0" maybe keep? It's a node identifier. Could translate to "节点 0". But "Node 0" appears in prose and table. It's not code. We can translate to "节点 0". But in table cell Node 0 IP maybe "节点 0 IP". "Current worker IP" -> "当前工作节点 IP". "Each node uses its own communication IP." -> "每个节点使用自己的通信 IP。" NODE0_IP cell "Not required" -> "非必需". "Workers use this address to join the DP group." -> "工作节点使用该地址加入 DP 组。" DP is abbreviation maybe keep DP. --headless Omitted -> "省略". Enabled -> "启用". "Workers do not expose an API endpoint." -> "工作节点不暴露 API 端点。" --data-parallel-address $LOCAL_IP $NODE0_IP "Always resolves to Node 0." -> "始终解析为节点 0。" --data-parallel-start-rank 0 by default -> "默认 0"? But cell has inline code 0 by default. Need translate prose around code: "默认 0". 1 to 7 -> "1 到 7". "Every node must own a unique DP rank." -> "每个节点必须拥有唯一的 DP 排名。" Maybe DP rank -> DP 秩? In distributed inference, rank often "排名" or "秩". Use "DP 排名" maybe. "rank" in DP group. Could use "DP 秩" more technical. But Chinese docs often "DP rank" keep? Rule says don't translate API names? DP rank is concept. Could translate "DP 排名". Need consistent.

"Key deployment parameters:" -> "关键部署参数："

Table: --tensor-parallel-size 8 | Uses all 8 NPUs in one A2 node for tensor parallelism. -> "在一个 A2 节点上使用全部 8 个 NPU 进行张量并行。" Keep A2? product? A2 node maybe model? Keep A2. NPU keep. --data-parallel-size 8 | Creates eight global DP ranks across eight nodes. -> "在 8 个节点上创建 8 个全局 DP 排名。" --data-parallel-size-local 1 | Runs one DP rank on the current node. -> "在当前节点上运行 1 个 DP 排名。" --data-parallel-start-rank | Selects the global DP rank of a worker node. -> "选择工作节点的全局 DP 排名。" --enable-expert-parallel | Enables expert parallelism for the MoE layers. -> "为 MoE 层启用专家并行。" --max-model-len 262144 | Sets the maximum combined input and output length. -> "设置输入和输出的最大总长度。" --quantization ascend | Enables Ascend W4A8 quantization for the validated checkpoint. -> "为已验证的检查点启用 Ascend W4A8 量化。" --safetensors-load-strategy lazy | Avoids prefetching the complete checkpoint from NFS. -> "避免从 NFS 预取完整检查点。" --max-num-seqs 8 | Sets eight active sequences for each DP group. -> "为每个 DP 组设置 8 个活动序列。" --max-num-batched-tokens 4096 | Controls the scheduler token budget. Larger values increase the MoE activation peak and shrink the KV cache available for the 262144 context. -> "控制调度器的 Token 预算。更大的值会增加 MoE 激活峰值，并压缩 262144 上下文可用的 KV 缓存。" Token maybe keep Token? Rule says don't translate Token? It lists Token as brand/product? Actually says do not translate ... Token, Prompt, Claude Code. So keep Token. "token budget" -> "Token 预算". "KV cache" keep KV 缓存? KV is acronym, cache translate. "262144 context" -> "262144 上下文". --gpu-memory-utilization 0.92 | Reserves HBM headroom for the weights and activation peak on 64GB nodes. -> "在 64GB 节点上为权重和激活峰值预留 HBM 余量。" --enable-prefix-caching | Enables automatic prefix caching. -> "启用自动前缀缓存。" --speculative-config | Enables one Qwen3.5 MTP speculative token. -> "启用一个 Qwen3.5 MTP 推测 Token。" Keep Qwen3.5? product. MTP. speculative token -> 推测 Token. --compilation-config | Uses FULL_DECODE_ONLY ACL Graph replay. -> "使用 FULL_DECODE_ONLY ACL 图重放。" ACL Graph? ACL maybe acronym, Graph translate? "ACL 图重放". --additional-config | Enables CPU binding and Fused MC2. -> "启用 CPU 绑定和 Fused MC2。" Fused MC2 maybe keep.

"Common Issues Tip: If a worker exits immediately, confirm that Node 0 is already running, --data-parallel-address resolves to Node 0, all nodes use the same RPC port, and every worker uses a unique DP start rank." Need translate: "常见问题提示：如果工作节点立即退出，请确认节点 0 已在运行，--data-parallel-address 解析为节点 0，所有节点使用相同的 RPC 端口，并且每个工作节点使用唯一的 DP 起始排名。" Line breaks? Could keep as paragraph. Original has line breaks after Node 0 is etc. We can output as one paragraph or preserve line breaks? Markdown structure unchanged maybe line breaks not critical. But to be safe, preserve similar line breaks? We can keep as one paragraph; but original has hard line breaks? It's prose. It's okay.

"Wait until the engine finishes loading weights and graph capture. A successful startup includes output similar to the following:" -> "等待引擎完成权重加载和图捕获。成功启动会包含类似以下输出：" Code block unchanged.

Heading 5.2: "### 5.2 Prefill-Decode Disaggregation" -> "### 5.2 预填充-解码分离 {#52-prefill-decode-disaggregation}"? Need attr_list syntax: space before {#...}. Yes.

"PD disaggregation separates Prefill and Decode into different service groups. Prefill nodes process prompt chunks, Decode nodes serve token generation, and a proxy forwards requests between them." Translate: "PD 分离将 Prefill 和 Decode 划分为不同的服务组。Prefill 节点处理提示块，Decode 节点提供 Token 生成，代理在它们之间转发请求。" Keep Prefill/Decode? They are terms, maybe keep. PD disaggregation -> PD 分离. "prompt chunks" -> "提示块". "token generation" -> "Token 生成".

"Detailed commands and configuration for PD disaggregation will be provided in a future update. Stay tuned." -> "PD 分离的详细命令和配置将在后续更新中提供。敬请期待。"

Heading 6: "## 6 Functional Verification" -> "## 6 功能验证 {#6-functional-verification}"

"After all DP ranks are ready, send a text request to the Node 0 API endpoint. The open-weight model always uses thinking mode and does not support multimodal input." -> "所有 DP 排名就绪后，向节点 0 的 API 端点发送文本请求。开放权重模型始终使用思考模式，并且不支持多模态输入。" Code block unchanged.

"Expected Result: Each response begins with reasoning wrapped in <think>...</think>, followed by the final answer. The service returns HTTP 200 and a choices field containing generated text. Worker nodes are headless and do not accept HTTP requests directly." Need keep inline code <think>...</think> unchanged? It's inline code, yes. Translate: "预期结果：每个响应都以包裹在 <think>...</think> 中的推理开始，随后是最终答案。服务返回 HTTP 200 和一个包含生成文本的 choices 字段。工作节点是无头节点，不直接接受 HTTP 请求。" "headless" -> "无头" maybe. Could keep "headless"? It's a mode. But translate prose. "无头节点".

"The model officially supports these reasoning effort levels:" -> "该模型官方支持以下推理努力级别：" List: - xhigh: default; deeper reasoning for complex tasks. -> "- xhigh：默认；针对复杂任务进行更深入的推理。" - medium: balances accuracy and speed. -> "- medium：平衡准确性和速度。" - low: prioritizes speed and cost. -> "- low：优先考虑速度和成本。"

Heading 7: "## 7 Accuracy Evaluation" -> "## 7 准确性评估 {#7-accuracy-evaluation}"

Heading 7.1: "### 7.1 Prepare the Evaluation Environment" -> "### 7.1 准备评估环境 {#71-prepare-the-evaluation-environment}" "Refer to Using AISBench for the evaluation environment and dataset preparation." -> "有关评估环境和数据集准备，请参阅使用 AISBench。" Keep link text? It says translate prose inside table cells/list items. Link text is prose? Should translate link text? It says keep URLs unchanged, but link text can be translated? Usually yes, translate prose. But link text "Using AISBench" maybe title of doc. Could translate to "使用 AISBench". Keep URL unchanged. Need preserve Markdown link. So 使用 AISBench. Good.

Heading 7.2: "### 7.2 Using AISBench" -> "### 7.2 使用 AISBench {#72-using-aisbench}" List: 1. Refer to Using AISBench for details. -> "1. 有关详细信息，请参阅使用 AISBench。" 2. After execution, you can get the result. Here are the GPQA Diamond results of Qwen3.8... and Qwen3.8... for reference only. Translate: "2. 执行后，你可以获取结果。以下是 Qwen3.8-2.4T-A95B-w4a8 和 Qwen3.8-2.4T-A95B-w8a8 的 GPQA Diamond 结果，仅供参考。" Need preserve line breaks? Could output as one line. But original has line breaks after of and and. Markdown structure? We can keep as one paragraph. It's okay. But to be safe, preserve line breaks? Not necessary. But if we change line breaks, still Markdown structure? It's prose. I'd keep concise.

Table: | dataset | model | metric | mode | vllm-api-general-chat | Should translate header cells? They are prose? "dataset" -> "数据集", "model" -> "模型", "metric" -> "指标", "mode" -> "模式". Last "vllm-api-general-chat" maybe a column name? It's not code but a metric name? Could keep unchanged? It looks like a column identifier, maybe should not translate? Rule: translate prose text in table cells. But if it's a name, maybe keep. It is a column header, likely a benchmark mode name. Could translate to "vllm-api-general-chat"? It's not prose? It is a technical identifier. Keep unchanged to be safe. But header cells dataset/model/metric/mode are English words, translate. Table rows: GPQA Diamond, model names, accuracy, gen, percentages. "accuracy" and "gen" are values, maybe keep? They are metric/mode values, likely identifiers. Keep unchanged. If translate "accuracy" to "准确率"? But table values are data, not prose. Keep.

Heading 8: "## 8 Performance Evaluation" -> "## 8 性能评估 {#8-performance-evaluation}"

Heading 8.1: "### 8.1 Install AISBench" -> "### 8.1 安装 AISBench {#81-install-aisbench}" "Run AISBench in a separate environment or container so that the load generator does not affect the serving processes. Refer to Using AISBench for performance evaluation." -> "在独立的环境或容器中运行 AISBench，以免负载生成器影响服务进程。请参阅使用 AISBench 进行性能评估。" Keep URL fragment unchanged.

Heading 8.2: "### 8.2 Performance Service Configuration" -> "### 8.2 性能服务配置 {#82-performance-service-configuration}" "Use the deployment in Section 5 as the baseline. Change the following values on all four nodes:" -> "以第 5 节中的部署为基线。在所有四个节点上更改以下值：" Table: | Parameter | Standard deployment | Performance test | Translate headers: "参数" | "标准部署" | "性能测试". Rows code unchanged, numbers unchanged.

Heading 8.3: "### 8.3 Run the Tests" -> "### 8.3 运行测试 {#83-run-the-tests}" "Run performance evaluation of Qwen3.8-2.4T-A95B as an example. Refer to vLLM benchmark for more details." -> "以 Qwen3.8-2.4T-A95B 为例运行性能评估。有关更多详细信息，请参阅 vLLM 基准测试。" Link text "vLLM benchmark" -> "vLLM 基准测试". Keep URL. Code block unchanged.

"Record the A3 node count, TP/PP/EP topology, context length, concurrency, reasoning effort, and weight revision together with the result." -> "将 A3 节点数量、TP/PP/EP 拓扑、上下文长度、并发数、推理努力级别和权重修订版本与结果一起记录。" Maybe "weight revision" -> "权重版本"? "revision" maybe "修订版本". Use "权重修订版本".

Need ensure all headings have attr_list. What about no heading at top? The input starts with paragraph, no heading. Only headings listed. Need append anchors to every heading. There are headings: ### 5.2, ## 6, ## 7, ### 7.1, ### 7.2, ## 8, ### 8.1, ### 8.2, ### 8.3. Need ensure no other headings.

Need consider anchor slugs: - "5.2 Prefill-Decode Disaggregation": If using GitHub slug, "52-prefill-decode-disaggregation". But rule says slug = lowercase words joined by hyphens, from ORIGINAL English heading. It might mean remove punctuation and spaces to hyphens. For "5.2", word "5.2" lowercased, remove dot? "52". Yes. - "6 Functional Verification": "6-functional-verification". - "7 Accuracy Evaluation": "7-accuracy-evaluation". - "7.1 Prepare the Evaluation Environment": "71-prepare-the-evaluation-environment". - "7.2 Using AISBench": "72-using-aisbench". - "8 Performance Evaluation": "8-performance-evaluation". - "8.1 Install AISBench": "81-install-aisbench". - "8.2 Performance Service Configuration": "82-performance-service-configuration". - "8.3 Run the Tests": "83-run-the-tests".

Need maybe include original English anchor exactly? attr_list {#...}. Good.

Need preserve code blocks exactly. The shell code includes etc. Keep.

Potential issue: The initial table is indented by 4 spaces. If we output with 4 spaces, it may be interpreted as code block, but original had that. Need preserve. However if we translate and keep 4 spaces, okay. But if we include blank lines? Need same.

Let's draft final.

Need be careful with inline code in table: --data-parallel-start-rank | 0 by default | 1 to 7 -> translate to 0 默认? Better: "默认 0" and "1 到 7". In table cell, if we write "默认 0" okay. Original cell: 0 by default. Could be "默认 0". For 1 to 7: "1 到 7".

"Node 0 IP" -> "节点 0 IP". "Current worker IP" -> "当前工作节点 IP". "Not required" -> "非必需". "Omitted" -> "省略". "Enabled" -> "启用".

"Key deployment parameters:" maybe "关键部署参数：".

"Common Issues Tip:" -> "常见问题提示：".

"Wait until the engine finishes loading weights and graph capture." -> "等待引擎完成权重加载和图捕获。" Graph capture maybe "图捕获". Good.

"PD disaggregation separates Prefill and Decode into different service groups." Could translate "PD 分离将 Prefill 和 Decode 拆分到不同的服务组。" Good.

"Prefill nodes process prompt chunks, Decode nodes serve token generation, and a proxy forwards requests between them." -> "Prefill 节点处理提示块，Decode 节点提供 Token 生成，代理在它们之间转发请求。"

"Expected Result:" maybe "预期结果：".

"Worker nodes are headless and do not accept HTTP requests directly." -> "工作节点为无头节点，不直接接受 HTTP 请求。"

"The model officially supports these reasoning effort levels:" -> "该模型官方支持以下推理努力级别："

"Refer to Using AISBench for the evaluation environment and dataset preparation." -> "评估环境和数据集准备请参阅使用 AISBench。" Maybe more natural: "有关评估环境和数据集准备，请参阅使用 AISBench。"

List item 2: "After execution, you can get the result." -> "执行后，你可以获取结果。" "Here are the GPQA Diamond results of ... and ... for reference only." -> "以下是 ... 和 ... 的 GPQA Diamond 结果，仅供参考。"

Table header: "dataset" etc. If translate, row values "accuracy" and "gen" remain English. Could be inconsistent but okay. Maybe translate "accuracy" to "准确率"? But it's a metric value, likely should remain as data. The instruction says translate prose text in table cells. Values are not prose? They are data. Keep.

"vllm-api-general-chat" maybe a column name, keep.

"Run AISBench in a separate environment or container so that the load generator does not affect the serving processes." -> "在独立的环境或容器中运行 AISBench，以免负载生成器影响服务进程。"

"Use the deployment in Section 5 as the baseline." -> "以第 5 节中的部署为基线。"

"Change the following values on all four nodes:" -> "在所有四个节点上更改以下值："

"Run performance evaluation of Qwen3.8-2.4T-A95B as an example." -> "以 Qwen3.8-2.4T-A95B 为例运行性能评估。"

"Record the A3 node count, TP/PP/EP topology, context length, concurrency, reasoning effort, and weight revision together with the result." -> "将 A3 节点数量、TP/PP/EP 拓扑、上下文长度、并发数、推理努力级别和权重修订版本与结果一起记录。"

Need maybe "reasoning effort" -> "推理努力级别" consistent.

Need ensure no code fence around whole output. Final only Markdown.

Double-check anchors: attr_list syntax in Markdown: "## 6 功能验证 {#6-functional-verification}". Good.

Potential issue: The user's input includes a table indented under a paragraph. If we translate and keep indentation, the table may be code block. But original likely intended as indented table? In Markdown, a table cannot be indented 4 spaces unless inside list? It might be a code block. But rule says keep structure unchanged. We'll preserve 4 spaces. However if we preserve 4 spaces, the table lines are code block and not table. But original had that. It's okay.

Need maybe preserve blank lines exactly? We can output similar.

Let's construct final carefully.

Start: 主节点和工作节点之间的以下值有所不同：

| 设置 | 节点 0 | 节点 1-7 | 描述 |
| --- | --- | --- | --- |
| `LOCAL_IP` | 节点 0 IP | 当前工作节点 IP | 每个节点使用自己的通信 IP。 |
| `NODE0_IP` | 非必需 | 节点 0 IP | 工作节点使用该地址加入 DP 组。 |
| `--headless` | 省略 | 启用 | 工作节点不暴露 API 端点。 |
| `--data-parallel-address` | `$LOCAL_IP` | `$NODE0_IP` | 始终解析为节点 0。 |
| `--data-parallel-start-rank` | 默认 `0` | `1` 到 `7` | 每个节点必须拥有唯一的 DP 排名。 |

关键部署参数：

| 参数 | 描述 |

... etc.

Need "Nodes 1-7" -> "节点 1-7". Good.

In table, "Setting" -> "设置". "Description" -> "描述".

"Key deployment parameters:" line not indented? Original has 4 spaces? It is " Key deployment parameters:"? Let's check: after table blank line, " Key deployment parameters:" yes indented 4 spaces. Need preserve? It is prose, but indented. If we output with 4 spaces, it may be code block? Original likely all indented block. Need preserve indentation for that line and second table. The input: " | ... |

Key deployment parameters:

| Parameter | Description |

..." So preserve 4 spaces before "关键部署参数：" and table. Good.

"Common Issues Tip:" line is indented 4 spaces? Original: " Common Issues Tip: If a worker exits immediately, confirm that Node 0 is already running, ..." Yes 4 spaces. Need preserve indentation? It is prose inside indented block. We can output with 4 spaces. But if we output as one line with 4 spaces, it's code block? Original had 4 spaces. To preserve structure, keep 4 spaces. But line breaks? We can keep as: 常见问题提示：如果工作节点立即退出，请确认节点 0 已在运行，--data-parallel-address 解析为节点 0，所有节点使用相同的 RPC 端口，并且每个工作节点使用唯一的 DP 起始排名。 This is one line indented 4 spaces. Good.

Then blank line, "Wait until..." not indented. Good.

Need ensure code block after not indented. Good.

Potential issue: If we indent prose 4 spaces, Markdown treats as code block, but original did. It's okay.

Need maybe translate "master" as "主节点". Good.

Let's think about "DP rank" translation. Could use "DP 秩" more technical. Which is better? In Chinese ML docs, "rank" often "秩" or "排名". "DP rank" maybe "DP 秩". "start rank" -> "起始秩". "unique DP rank" -> "唯一的 DP 秩". "global DP rank" -> "全局 DP 秩". "DP start rank" -> "DP 起始秩". This sounds more technical. But "rank" in process group is "秩". Use "秩" for consistency. However "DP 排名" is understandable. Which is preferred? For tensor parallel rank, Chinese docs often "rank" kept or "秩". Example "DP rank" maybe "DP 秩". I'll use "DP 秩" to be technical. But list "reasoning effort levels" not rank. Need adjust: - "Every node must own a unique DP rank." -> "每个节点必须拥有唯一的 DP 秩。" - "Creates eight global DP ranks across eight nodes." -> "在 8 个节点上创建 8 个全局 DP 秩。" - "Runs one DP rank on the current node." -> "在当前节点上运行 1 个 DP 秩。" - "Selects the global DP rank of a worker node." -> "选择工作节点的全局 DP 秩。" - "After all DP ranks are ready" -> "所有 DP 秩就绪后" - "every worker uses a unique DP start rank" -> "每个工作节点使用唯一的 DP 起始秩". Good.

"DP group" -> "DP 组".

"token budget" -> "Token 预算". "MoE activation peak" -> "MoE 激活峰值". "KV cache" -> "KV 缓存". "HBM headroom" -> "HBM 余量". "weights and activation peak" -> "权重和激活峰值".

"Qwen3.5 MTP speculative token" -> "Qwen3.5 MTP 推测 Token". Good.

"ACL Graph replay" -> "ACL 图重放".

"Fused MC2" keep.

"open-weight model" -> "开放权重模型".

"thinking mode" -> "思考模式".

"multimodal input" -> "多模态输入".

"reasoning wrapped in <think>...</think>" -> "包裹在 <think>...</think> 中的推理". Good.

"headless" -> "无头".

"reasoning effort levels" -> "推理努力级别".

"accuracy evaluation" -> "准确性评估".

"performance service configuration" -> "性能服务配置".

"weight revision" -> "权重修订版本". Could be "权重版本". "revision" in model weights maybe "修订版本". Use "权重修订版本".

Need maybe translate "Standard deployment" -> "标准部署". "Performance test" -> "性能测试".

Need preserve table alignment? Original has | --- | ---: | ---: |. Keep.

Potential issue: The first table is indented, but if we translate header "Setting" to "设置", okay.

Need ensure no extra spaces in attr_list? "### 5.2 预填充-解码分离 {#52-prefill-decode-disaggregation}". Good.

Let's consider if anchor should include original English heading with number punctuation? The rule: "slug = lowercase words joined by hyphens, from the ORIGINAL English heading". For "5.2 Prefill-Decode Disaggregation", if words are "5.2", "Prefill-Decode", "Disaggregation". Lowercase words joined by hyphens: "5.2-prefill-decode-disaggregation"? But they say slug, likely remove punctuation. Example doesn't cover. In attr_list, anchor can be any? They likely expect GitHub-style slug. For heading "5.2 Prefill-Decode Disaggregation", GitHub slug is "52-prefill-decode-disaggregation". I think use that. For "7.1" -> "71". Good.

Need maybe include "prefill-decode-disaggregation" not "52"? If they say words joined by hyphens, number "5.2" is a word? If lowercased and punctuation removed, "52". Good.

Need maybe for "6 Functional Verification" slug "6-functional-verification". Good.

Now final output. Ensure no analysis in final. final only translated Markdown.

Double-check code blocks: The shell code includes JSON with English prompt "Write a Python function to merge two sorted linked lists." Should we translate inside code? Rule: Keep ALL code blocks unchanged. So do not translate. Good.

Text code block unchanged.

Links: 使用 AISBench. Good. For performance link: 使用 AISBench 进行性能评估. Good.

Need maybe translate "vLLM benchmark" link text to "vLLM 基准测试". Good.

Potential issue: "Qwen3.8" model names in prose? Keep. "Qwen3.5" keep.

Let's produce final. Ensure indentation of initial block. I will preserve 4 spaces for first table, key deployment parameters, second table, common issues tip. But if I put 4 spaces before "关键部署参数：" and table, it's okay. Need blank lines also indented? Original blank lines maybe no spaces. We can leave blank lines empty.

Let's write:

主节点和工作节点之间的以下值有所不同：

| 设置 | 节点 0 | 节点 1-7 | 描述 |

...

关键部署参数：

| 参数 | 描述 |

...

常见问题提示：如果工作节点立即退出，请确认节点 0 已在运行，`--data-parallel-address` 解析为节点 0，所有节点使用相同的 RPC 端口，并且每个工作节点使用唯一的 DP 起始秩。

等待引擎完成权重加载和图捕获。成功启动会包含类似以下输出：

...

5.2 预填充-解码分离¶

PD 分离将 Prefill 和 Decode 拆分到不同的服务组。Prefill 节点处理提示块，Decode 节点提供 Token 生成，代理在它们之间转发请求。

PD 分离的详细命令和配置将在后续更新中提供。敬请期待。

6 功能验证¶

所有 DP 秩就绪后，向节点 0 的 API 端点发送文本请求。开放权重模型始终使用思考模式，并且不支持多模态输入。

...

预期结果：每个响应都以包裹在 <think>...</think> 中的推理开始，随后是最终答案。服务返回 HTTP 200 和一个包含生成文本的 choices 字段。工作节点为无头节点，不直接接受 HTTP 请求。

该模型官方支持以下推理努力级别：

xhigh：默认；针对复杂任务进行更深入的推理。
medium：平衡准确性和速度。
low：优先考虑速度和成本。

7 准确性评估¶

7.1 准备评估环境¶

有关评估环境和数据集准备，请参阅使用 AISBench。

7.2 使用 AISBench¶

有关详细信息，请参阅使用 AISBench。
执行后，你可以获取结果。以下是 Qwen3.8-2.4T-A95B-w4a8 和 Qwen3.8-2.4T-A95B-w8a8 的 GPQA Diamond 结果，仅供参考。

数据集	模型	指标	模式	vllm-api-general-chat
GPQA Diamond	Qwen3.8-2.4T-A95B-w4a8	accuracy	gen	91.92%
GPQA Diamond	Qwen3.8-2.4T-A95B-w8a8	accuracy	gen	93.75%

8 性能评估¶

8.1 安装 AISBench¶

在独立的环境或容器中运行 AISBench，以免负载生成器影响服务进程。请参阅使用 AISBench 进行性能评估。

8.2 性能服务配置¶

以第 5 节中的部署为基线。在所有四个节点上更改以下值：

参数	标准部署	性能测试
`--max-model-len`	131072	250000
`--max-num-batched-tokens`	16384	8192
`--gpu-memory-utilization`	0.85	0.95

8.3 运行测试¶

以 Qwen3.8-2.4T-A95B 为例运行性能评估。有关更多详细信息，请参阅 vLLM 基准测试。

...

将 A3 节点数量、TP/PP/EP 拓扑、上下文长度、并发数、推理努力级别和权重修订版本与结果一起记录。

Need check if "vllm-api-general-chat" header should be translated? It's a column name maybe not. Keep.

One nuance: The first table is indented 4 spaces, but after translation, if there are blank lines, the indented table may be code block. But original. Good.

Need maybe preserve "Nodes 1-7" as "节点 1-7". Good.

Need maybe "Node 0 IP" in table cell: "节点 0 IP". Good.

Need maybe "Current worker IP" -> "当前工作节点 IP". Good.

Need maybe "Not required" -> "非必需". Good.

Need maybe "Omitted" -> "省略". Good.

Need maybe "Enabled" -> "启用". Good.

Need maybe "Always resolves to Node 0." -> "始终解析为节点 0。" Good.

Need maybe "Every node must own a unique DP rank." -> "每个节点必须拥有唯一的 DP 秩。" Good.

Need maybe "Key deployment parameters:" -> "关键部署参数：" Good.

Need maybe "Uses all 8 NPUs in one A2 node for tensor parallelism." -> "在一个 A2 节点上使用全部 8 个 NPU 进行张量并行。" Good.

Need maybe "Creates eight global DP ranks across eight nodes." -> "在 8 个节点上创建 8 个全局 DP 秩。" Good.

Need maybe "Runs one DP rank on the current node." -> "在当前节点上运行 1 个 DP 秩。" Good.

Need maybe "Selects the global DP rank of a worker node." -> "选择工作节点的全局 DP 秩。" Good.

Need maybe "Enables expert parallelism for the MoE layers." -> "为 MoE 层启用专家并行。" Good.

Need maybe "Sets the maximum combined input and output length." -> "设置输入和输出的最大总长度。" Good.

Need maybe "Enables Ascend W4A8 quantization for the validated checkpoint." -> "为已验证的检查点启用 Ascend W4A8 量化。" Good.

Need maybe "Avoids prefetching the complete checkpoint from NFS." -> "避免从 NFS 预取完整检查点。" Good.

Need maybe "Sets eight active sequences for each DP group." -> "为每个 DP 组设置 8 个活动序列。" Good.

Need maybe "Controls the scheduler token budget. Larger values increase the MoE activation peak and shrink the KV cache available for the 262144 context." -> "控制调度器的 Token 预算。更大的值会增加 MoE 激活峰值，并压缩 262144 上下文可用的 KV 缓存。" Good.

Need maybe "Reserves HBM headroom for the weights and activation peak on 64GB nodes." -> "在 64GB 节点上为权重和激活峰值预留 HBM 余量。" Good.

Need maybe "Enables automatic prefix caching." -> "启用自动前缀缓存。" Good.

Need maybe "Enables one Qwen3.5 MTP speculative token." -> "启用一个 Qwen3.5 MTP 推测 Token。" Good.

Need maybe "Uses FULL_DECODE_ONLY ACL Graph replay." -> "使用 FULL_DECODE_ONLY ACL 图重放。" Good.

Need maybe "Enables CPU binding and Fused MC2." -> "启用 CPU 绑定和 Fused MC2。" Good.

Need maybe "Common Issues Tip" -> "常见问题提示". Good.

Need maybe "If a worker exits immediately" -> "如果工作节点立即退出". Good.

Need maybe "confirm that Node 0 is already running" -> "请确认节点 0 已在运行". Good.

Need maybe "all nodes use the same RPC port" -> "所有节点使用相同的 RPC 端口". Good.

Need maybe "every worker uses a unique DP start rank" -> "每个工作节点使用唯一的 DP 起始秩". Good.

Need maybe "Wait until the engine finishes loading weights and graph capture." -> "等待引擎完成权重加载和图捕获。" Good.

Need maybe "A successful startup includes output similar to the following:" -> "成功启动会包含类似以下输出：" Good.

Need maybe "PD disaggregation separates Prefill and Decode into different service groups." -> "PD 分离将 Prefill 和 Decode 拆分到不同的服务组。" Good.

Need maybe "Prefill nodes process prompt chunks" -> "Prefill 节点处理提示块". Good.

Need maybe "Decode nodes serve token generation" -> "Decode 节点提供 Token 生成". Good.

Need maybe "a proxy forwards requests between them" -> "代理在它们之间转发请求". Good.

Need maybe "Detailed commands and configuration for PD disaggregation will be provided in a future update. Stay tuned." -> "PD 分离的详细命令和配置将在后续更新中提供。敬请期待。" Good.

Need maybe "After all DP ranks are ready" -> "所有 DP 秩就绪后". Good.

Need maybe "send a text request to the Node 0 API endpoint" -> "向节点 0 的 API 端点发送文本请求". Good.

Need maybe "The open-weight model always uses thinking mode and does not support multimodal input." -> "开放权重模型始终使用思考模式，并且不支持多模态输入。" Good.

Need maybe "Expected Result:" -> "预期结果：". Good.

Need maybe "Each response begins with reasoning wrapped in <think>...</think>, followed by the final answer." -> "每个响应都以包裹在 <think>...</think> 中的推理开始，随后是最终答案。" Good.

Need maybe "The service returns HTTP 200 and a choices field containing generated text." -> "服务返回 HTTP 200 和一个包含生成文本的 choices 字段。" Good.

Need maybe "Worker nodes are headless and do not accept HTTP requests directly." -> "工作节点为无头节点，不直接接受 HTTP 请求。" Good.

Need maybe "The model officially supports these reasoning effort levels:" -> "该模型官方支持以下推理努力级别：" Good.

Need maybe list translations. Good.

Need maybe "Refer to [Using AISBench]..." -> "有关评估环境和数据集准备，请参阅 [使用 AISBench]..." Good.

Need maybe "After execution, you can get the result." -> "执行后，你可以获取结果。" Good.

Need maybe "Here are the GPQA Diamond results of ... and ... for reference only." -> "以下是 ... 和 ... 的 GPQA Diamond 结果，仅供参考。" Good.

Need maybe table header "dataset" etc. Good.

Need maybe "Run AISBench in a separate environment or container so that the load generator does not affect the serving processes." -> "在独立的环境或容器中运行 AISBench，以免负载生成器影响服务进程。" Good.

Need maybe "Use the deployment in Section 5 as the baseline." -> "以第 5 节中的部署为基线。" Good.

Need maybe "Change the following values on all four nodes:" -> "在所有四个节点上更改以下值：" Good.

Need maybe "Run performance evaluation of Qwen3.8-2.4T-A95B as an example." -> "以 Qwen3.8-2.4T-A95B 为例运行性能评估。" Good.

Need maybe "Refer to [vLLM benchmark] for more details." -> "有关更多详细信息，请参阅 [vLLM 基准测试]。" Good.

Need maybe final record. Good.

Check if any heading missing anchor: The input has "### 5.2 Prefill-Decode Disaggregation" after code block. We add. "## 6 Functional Verification" add. "## 7 Accuracy Evaluation" add. "### 7.1 Prepare the Evaluation Environment" add. "### 7.2 Using AISBench" add. "## 8 Performance Evaluation" add. "### 8.1 Install AISBench" add. "### 8.2 Performance Service Configuration" add. "### 8.3 Run the Tests" add.

Need maybe "5.2" in heading translated? Keep number. Good.

Potential issue: attr_list anchor for "7.2 Using AISBench" if using "72-using-aisbench". Good.

Now final. Ensure no code fence around whole. Use final.

主节点和工作节点之间的以下值有所不同：

| 设置 | 节点 0 | 节点 1-7 | 描述 |
| --- | --- | --- | --- |
| `LOCAL_IP` | 节点 0 IP | 当前工作节点 IP | 每个节点使用自己的通信 IP。 |
| `NODE0_IP` | 非必需 | 节点 0 IP | 工作节点使用该地址加入 DP 组。 |
| `--headless` | 省略 | 启用 | 工作节点不暴露 API 端点。 |
| `--data-parallel-address` | `$LOCAL_IP` | `$NODE0_IP` | 始终解析为节点 0。 |
| `--data-parallel-start-rank` | 默认 `0` | `1` 到 `7` | 每个节点必须拥有唯一的 DP 秩。 |

关键部署参数：

| 参数 | 描述 |
| --- | --- |
| `--tensor-parallel-size 8` | 在一个 A2 节点上使用全部 8 个 NPU 进行张量并行。 |
| `--data-parallel-size 8` | 在 8 个节点上创建 8 个全局 DP 秩。 |
| `--data-parallel-size-local 1` | 在当前节点上运行 1 个 DP 秩。 |
| `--data-parallel-start-rank` | 选择工作节点的全局 DP 秩。 |
| `--enable-expert-parallel` | 为 MoE 层启用专家并行。 |
| `--max-model-len 262144` | 设置输入和输出的最大总长度。 |
| `--quantization ascend` | 为已验证的检查点启用 Ascend W4A8 量化。 |
| `--safetensors-load-strategy lazy` | 避免从 NFS 预取完整检查点。 |
| `--max-num-seqs 8` | 为每个 DP 组设置 8 个活动序列。 |
| `--max-num-batched-tokens 4096` | 控制调度器的 Token 预算。更大的值会增加 MoE 激活峰值，并压缩 262144 上下文可用的 KV 缓存。 |
| `--gpu-memory-utilization 0.92` | 在 64GB 节点上为权重和激活峰值预留 HBM 余量。 |
| `--enable-prefix-caching` | 启用自动前缀缓存。 |
| `--speculative-config` | 启用一个 Qwen3.5 MTP 推测 Token。 |
| `--compilation-config` | 使用 `FULL_DECODE_ONLY` ACL 图重放。 |
| `--additional-config` | 启用 CPU 绑定和 Fused MC2。 |

常见问题提示：如果工作节点立即退出，请确认节点 0 已在运行，`--data-parallel-address` 解析为节点 0，所有节点使用相同的 RPC 端口，并且每个工作节点使用唯一的 DP 起始秩。

等待引擎完成权重加载和图捕获。成功启动会包含类似以下输出：

INFO:     Started server process
INFO:     Waiting for application startup.
INFO:     Application startup complete.

5.2 预填充-解码分离¶

PD 分离将 Prefill 和 Decode 拆分到不同的服务组。Prefill 节点处理提示块，Decode 节点提供 Token 生成，代理在它们之间转发请求。

PD 分离的详细命令和配置将在后续更新中提供。敬请期待。

6 功能验证¶

所有 DP 秩就绪后，向节点 0 的 API 端点发送文本请求。开放权重模型始终使用思考模式，并且不支持多模态输入。

curl http://<server_ip>:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "qwen3.8",
    "messages": [
      {"role": "user", "content": "Write a Python function to merge two sorted linked lists."}
    ],
    "temperature": 1.0,
    "top_p": 0.95,
    "stream": true,
    "chat_template_kwargs": {
      "enable_thinking": true,
      "preserve_thinking": true
    }
  }'

预期结果：每个响应都以包裹在 <think>...</think> 中的推理开始，随后是最终答案。服务返回 HTTP 200 和一个包含生成文本的 choices 字段。工作节点为无头节点，不直接接受 HTTP 请求。

该模型官方支持以下推理努力级别：

xhigh：默认；针对复杂任务进行更深入的推理。
medium：平衡准确性和速度。
low：优先考虑速度和成本。

7 准确性评估¶

7.1 准备评估环境¶

有关评估环境和数据集准备，请参阅使用 AISBench。

7.2 使用 AISBench¶

有关详细信息，请参阅使用 AISBench。
执行后，你可以获取结果。以下是 Qwen3.8-2.4T-A95B-w4a8 和 Qwen3.8-2.4T-A95B-w8a8 的 GPQA Diamond 结果，仅供参考。

数据集	模型	指标	模式	vllm-api-general-chat
GPQA Diamond	Qwen3.8-2.4T-A95B-w4a8	accuracy	gen	91.92%
GPQA Diamond	Qwen3.8-2.4T-A95B-w8a8	accuracy	gen	93.75%

8 性能评估¶

8.1 安装 AISBench¶

在独立的环境或容器中运行 AISBench，以免负载生成器影响服务进程。请参阅使用 AISBench 进行性能评估。

8.2 性能服务配置¶

以第 5 节中的部署为基线。在所有四个节点上更改以下值：

参数	标准部署	性能测试
`--max-model-len`	131072	250000
`--max-num-batched-tokens`	16384	8192
`--gpu-memory-utilization`	0.85	0.95

8.3 运行测试¶

以 Qwen3.8-2.4T-A95B 为例运行性能评估。有关更多详细信息，请参阅 vLLM 基准测试。

vllm bench serve \
  --model Qwen/Qwen3.8-2.4T-A95B \
  --served-model-name qwen3.8 \
  --base-url http://<server_ip>:8000 \
  --dataset-name random \
  --random-input-len 8192 \
  --random-output-len 1024 \
  --num-prompts 16 \
  --max-concurrency 4 \
  --save-result \
  --result-dir ./

将 A3 节点数量、TP/PP/EP 拓扑、上下文长度、并发数、推理努力级别和权重修订版本与结果一起记录。

9 性能调优¶

9.1 推荐配置¶

以下配置已在特定测试环境中验证，仅供参考。最优配置取决于最大输入/输出长度、请求并发量、前缀缓存命中率、量化和工作负载特征。请根据实际工作负载调整第 9.2 节中的参数。

表 1：场景概览

场景	部署模式	*总 NPU 数	权重版本	拓扑
混合 Prefill/Decode	四个 Atlas 800 A3 节点	64	W8A8	DP4/TP16/EP64

*Total NPUs 表示所有节点使用的 NPU 总数。

表 2：详细配置

最大序列数	最大模型长度	最大批处理 Token 数	GPU 显存利用率	MTP Token 数	前缀缓存	主要优化
每 DP 8	131072	16384	0.85	1	开启	完整解码 ACL Graph、Fused MC2、CPU 绑定

完整的启动命令和参数说明，请参见第 5 章中的部署示例。

9.2 调优指南¶

9.2.1 通用调优参考¶

请参见公开性能调优文档了解通用调优方法，并参见功能矩阵了解功能说明。

推荐调优顺序：

首先设置部署拓扑。已验证的配置使用四个 A3 节点，拓扑为 DP4/TP16/EP64。
使用 --max-model-len 选择最大上下文长度。长上下文会增加 KV 缓存使用量，因此如果发生 OOM，请降低 --max-num-seqs 或 --gpu-memory-utilization。
调整 --max-num-batched-tokens。较大的值通常可提高 Prefill 吞吐量，但会增加激活内存。以解码为主的工作负载通常需要较小的值。
根据服务并发量调整 --max-num-seqs。超过该值的请求将在队列中等待，等待时间会计入 TTFT 和 TPOT。
调整 --gpu-memory-utilization。增大该值可提供更多 KV 缓存，但需为运行时内存波动和专家不均衡预留余量。
调整 --speculative-config。MTP 可提高解码吞吐量，但最优的 num_speculative_tokens 取决于接受率和工作负载。
调整 ACL Graph 捕获。解码推荐 FULL_DECODE_ONLY。如果手动设置 cudagraph_capture_sizes，请包含常见的解码批处理大小。

9.2.2 模型特定优化¶

优化项	启用方式	收益	说明
Chunked Prefill	由 vLLM V1 调度器启用	将长 Prefill 输入分块，以降低每步内存峰值。	使用 `--max-num-batched-tokens` 调整。
前缀缓存	`--enable-prefix-caching`	复用重复前缀的 KV 状态。	收益取决于前缀缓存命中率。
Qwen3.5 MTP 推测解码	`--speculative-config '{"method":"qwen3_5_mtp","num_speculative_tokens":1}'`	当接受率良好时，可提高解码吞吐量。	针对目标工作负载调整推测 Token 数。
完整解码 ACL Graph	`--compilation-config '{"cudagraph_mode":"FULL_DECODE_ONLY"}'`	降低解码期间的算子调度开销。	已验证配置中使用。
Fused MC2	`--additional-config '{"enable_fused_mc2":1}'`	启用融合 MoE 通信与计算。	已验证配置中使用。
Lazy Safetensors	`--safetensors-load-strategy lazy`	避免从共享存储预取完整检查点。	用于 W8A8 检查点。
CPU 绑定	`--additional-config '{"enable_cpu_binding":true}'`	降低 CPU 调度抖动。	在部署命令中显式启用。

10 常见问题¶

对于常见环境、安装和通用参数问题，请参见公开常见问题。

Q1：开放权重模型支持哪些输入？¶

Qwen3.8-2.4T-A95B 是纯文本模型。它不支持多模态输入。

Q2：能否禁用思考模式？¶

不能。请保持 enable_thinking=true。响应会先以 think 块中的推理开始，然后返回最终答案。

Q3：应如何选择并行大小？¶

应根据模型兼容性、权重内存、KV 缓存容量和通信性能，共同选择 TP、PP、DP 和 EP。请在目标集群上验证完整拓扑。

本页原文 Markdown：在 AtomGit 查看·内容源自开源项目 vl/vllm-ascend