DeepSeek-V4-Flash-Vision-Exp（实验性）¶

1 简介¶

DeepSeek-V4-Flash-Vision-Exp 是 DeepSeek-V4 系列中的多模态混合专家（mixture-of-experts）模型。它将 DeepSeek-V4 语言模型与视觉编码器和对齐器相结合，并通过兼容 OpenAI 的聊天 API 接受文本、单图像和多图像请求。

vLLM Ascend 上的支持目前为实验性，可在 main 分支上使用，并配有匹配的 vLLM 0.27.x 修订版本。本指南介绍在单台 Atlas 800 A3 服务器或两台 Atlas 800 A2 服务器上进行 Ascend W8A8 部署。目前仅支持共置部署：Prefill 和 Decode 在同一服务中运行。请勿将此模型用于 Prefill-Decode 分离部署。

2 支持的特性¶

完整的支持矩阵请参阅支持的模型，功能配置请参阅功能指南。

本指南中的 A3 配置已使用 W8A8 权重、TP4/DP4/EP16、自动前缀缓存和 FULL_DECODE_ONLY ACL Graph 完成验证。A2 配置在两台服务器上使用相同的全局 TP4/DP4/EP16 拓扑。本指南不涵盖 BF16 推理服务、DSA 上下文并行、FlashComm1 以及 Prefill-Decode 分离部署。

3 前提条件¶

3.1 模型权重与硬件¶

模型	下载	硬件要求
DeepSeek-V4-Flash-Vision-Exp-w8a8-QuaRot	ModelScope	一台 Atlas 800 A3 服务器（128GB × 8 个 NPU）或两台 Atlas 800 A2 服务器（各 64GB × 8 个 NPU）
DeepSeek-V4-Flash-Vision-Exp	Hugging Face	原始模型权重；本指南中的部署请使用 ModelScope W8A8 QuaRot checkpoint

下面的启动命令使用 ModelScope 上公开的 Ascend W8A8 QuaRot checkpoint。该 checkpoint 包含 ModelSlim 量化描述以及 vLLM Ascend 所需的多模态 bias_vl 张量。

请将 checkpoint 下载或准备到您选择的目录中，并记录绝对路径。示例中使用 <YOUR_MODEL_PATH>；请将其替换为您的实际路径，例如 /data/weights/DeepSeek-V4-Flash-Vision-Exp-w8a8-QuaRot。

3.2 软件¶

请为目标硬件使用已发布的镜像。每个镜像都包含匹配的 vLLM 和 vLLM Ascend 修订版本：

A3: quay.io/ascend/vllm-ascend:deepseekv4-flash-vision-exp-a3
A2: quay.io/ascend/vllm-ascend:deepseekv4-flash-vision-exp

请勿在容器内单独替换任一软件包。不支持混用其他 vLLM 和 vLLM Ascend 修订版本。

3.3 验证多节点通信（可选）¶

如果您需要部署多节点环境，请按照验证多节点通信环境中的说明验证多节点通信。

4 安装¶

4.1 Docker 镜像安装¶

请选择目标硬件对应的选项卡。请在两台服务器上均运行 A2 命令。

A3 系列A2 系列

A3 服务器有 8 个 NPU，在容器中暴露为 16 个逻辑设备（davinci0 到 davinci15）。

export IMAGE=quay.io/ascend/vllm-ascend:deepseekv4-flash-vision-exp-a3
export MODEL_ROOT="/data/weights"

docker pull "$IMAGE"

docker run --rm -it \
  --name dsv4-vision \
  --net=host \
  --shm-size=512g \
  --privileged=true \
  --device /dev/davinci0 \
  --device /dev/davinci1 \
  --device /dev/davinci2 \
  --device /dev/davinci3 \
  --device /dev/davinci4 \
  --device /dev/davinci5 \
  --device /dev/davinci6 \
  --device /dev/davinci7 \
  --device /dev/davinci8 \
  --device /dev/davinci9 \
  --device /dev/davinci10 \
  --device /dev/davinci11 \
  --device /dev/davinci12 \
  --device /dev/davinci13 \
  --device /dev/davinci14 \
  --device /dev/davinci15 \
  --device /dev/davinci_manager \
  --device /dev/devmm_svm \
  --device /dev/hisi_hdc \
  -v /usr/local/dcmi:/usr/local/dcmi \
  -v /usr/local/Ascend/driver/tools/hccn_tool:/usr/local/Ascend/driver/tools/hccn_tool \
  -v /usr/local/bin/npu-smi:/usr/local/bin/npu-smi \
  -v /usr/local/Ascend/driver/lib64/:/usr/local/Ascend/driver/lib64/ \
  -v /usr/local/Ascend/driver/version.info:/usr/local/Ascend/driver/version.info \
  -v /etc/ascend_install.info:/etc/ascend_install.info \
  -v /etc/hccn.conf:/etc/hccn.conf \
  -v "$MODEL_ROOT:$MODEL_ROOT" \
  "$IMAGE" bash

每台 A2 服务器暴露 8 个 NPU die（davinci0 到 davinci7）。

export IMAGE=quay.io/ascend/vllm-ascend:deepseekv4-flash-vision-exp
export MODEL_ROOT="/data/weights"

docker pull "$IMAGE"

docker run --rm -it \
  --name dsv4-vision \
  --net=host \
  --shm-size=512g \
  --privileged=true \
  --device /dev/davinci0 \
  --device /dev/davinci1 \
  --device /dev/davinci2 \
  --device /dev/davinci3 \
  --device /dev/davinci4 \
  --device /dev/davinci5 \
  --device /dev/davinci6 \
  --device /dev/davinci7 \
  --device /dev/davinci_manager \
  --device /dev/devmm_svm \
  --device /dev/hisi_hdc \
  -v /usr/local/dcmi:/usr/local/dcmi \
  -v /usr/local/Ascend/driver/tools/hccn_tool:/usr/local/Ascend/driver/tools/hccn_tool \
  -v /usr/local/bin/npu-smi:/usr/local/bin/npu-smi \
  -v /usr/local/Ascend/driver/lib64/:/usr/local/Ascend/driver/lib64/ \
  -v /usr/local/Ascend/driver/version.info:/usr/local/Ascend/driver/version.info \
  -v /etc/ascend_install.info:/etc/ascend_install.info \
  -v /etc/hccn.conf:/etc/hccn.conf \
  -v "$MODEL_ROOT:$MODEL_ROOT" \
  "$IMAGE" bash

4.2 验证与源码构建¶

如果 checkpoint 存储在其他位置，请修改 MODEL_ROOT。请保持容器内外的绝对模型路径一致，并在两台 A2 服务器上使用相同的路径。

由于这些命令会启动交互式 shell，请打开另一个主机终端并验证容器正在运行：

docker ps --filter name=^/dsv4-vision$ \
  --format 'table {% raw %}{{.Names}}\t{{.Image}}\t{{.Status}}{% endraw %}'

A3 的预期输出类似于以下内容。对于 A2，镜像名称没有 -a3 后缀。以 Up 开头的状态表示成功。

NAMES          IMAGE                                                                  STATUS
dsv4-vision    quay.io/ascend/vllm-ascend:deepseekv4-flash-vision-exp-a3              Up 30 seconds

进入每个容器后，验证已安装的软件包：

python -m pip show vllm vllm-ascend

两个软件包都必须存在。若要改为从源代码构建，请遵循软件环境安装指南，并使用 .github/vllm-main-verified.commit 中匹配的 vLLM 修订版本构建 main 分支。针对目标平台使用硬件特定的 Dockerfile。

5 在线服务部署¶

5.1 单节点 A3 共置部署¶

经过验证的拓扑使用四个数据并行引擎。每个引擎使用四个张量并行 rank，而专家并行跨越全部 16 个 rank。

# Replace <YOUR_MODEL_PATH> with the actual path recorded in Section 3.1.
export MODEL_PATH="<YOUR_MODEL_PATH>"
export ASCEND_RT_VISIBLE_DEVICES=0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15
export HCCL_BUFFSIZE=1024

vllm serve "$MODEL_PATH" \
  --served-model-name dsv4-vision \
  --max-model-len 130000 \
  --max-num-batched-tokens 4096 \
  --max-num-seqs 32 \
  --gpu-memory-utilization 0.9 \
  --data-parallel-size 4 \
  --tensor-parallel-size 4 \
  --enable-expert-parallel \
  --tokenizer-mode deepseek_v4 \
  --quantization ascend \
  --model-loader-extra-config '{"enable_multithread_load":true,"num_threads":32}' \
  --block-size 32 \
  --compilation-config '{"cudagraph_mode":"FULL_DECODE_ONLY"}' \
  --port 8000

关键参数：

--data-parallel-size 4 和 --tensor-parallel-size 4 会使用全部 16 个可见逻辑设备，这些设备由服务器的 8 个 NPU 提供支持。 --enable-expert-parallel 将 MoE 专家分布到各个 rank。
--max-model-len 130000 限制单个请求的输入和输出总长度。如果服务无法分配足够的 KV 缓存，请降低该值。
--max-num-seqs 32 是每个 DP 引擎调度的最大序列数。如果观察到运行时内存压力，请优先降低该值。
--block-size 32 是经过验证的 DeepSeek-V4 前缀缓存和稀疏注意力配置所必需的。
FULL_DECODE_ONLY 会捕获 decode 执行，同时保持 Prefill 在完整 decode 图之外。

等待全部四个 DP 引擎完成权重加载和图捕获。成功启动会包含类似以下输出：

INFO:     Started server process
INFO:     Waiting for application startup.
INFO:     Application startup complete.

对于一般启动问题，请参阅公共 FAQ。

5.2 双节点 A2 共置部署¶

A2 部署使用两台服务器，因为每台服务器提供 8 个 NPU die。每台服务器运行两个本地 DP rank，采用 TP4，使用全部 8 个 die。两台服务器共同组成全局 DP4/TP4/EP16 拓扑。

在启动服务之前，请遵循验证多节点通信。在两台服务器上使用相同的模型路径和 DP RPC 端口。将 NIC_NAME 设置为与 LOCAL_IP 关联的接口，并确保节点 1 能够通过 NODE0_IP 访问节点 0。

节点 0节点 1

# Replace the placeholders with the values for Node 0.
export MODEL_PATH="<YOUR_MODEL_PATH>"
export NIC_NAME="<NODE0_NIC_NAME>"
export LOCAL_IP="<NODE0_IP>"
export DP_RPC_PORT=13389

export ASCEND_RT_VISIBLE_DEVICES=0,1,2,3,4,5,6,7
export HCCL_IF_IP="$LOCAL_IP"
export GLOO_SOCKET_IFNAME="$NIC_NAME"
export TP_SOCKET_IFNAME="$NIC_NAME"
export HCCL_SOCKET_IFNAME="$NIC_NAME"
export HCCL_BUFFSIZE=1024
export HCCL_INTRA_PCIE_ENABLE=1
export HCCL_INTRA_ROCE_ENABLE=0

vllm serve "$MODEL_PATH" \
  --host 0.0.0.0 \
  --port 8000 \
  --served-model-name dsv4-vision \
  --max-model-len 130000 \
  --max-num-batched-tokens 4096 \
  --max-num-seqs 32 \
  --gpu-memory-utilization 0.9 \
  --data-parallel-size 4 \
  --data-parallel-size-local 2 \
  --data-parallel-start-rank 0 \
  --data-parallel-address "$LOCAL_IP" \
  --data-parallel-rpc-port "$DP_RPC_PORT" \
  --tensor-parallel-size 4 \
  --enable-expert-parallel \
  --tokenizer-mode deepseek_v4 \
  --quantization ascend \
  --model-loader-extra-config '{"enable_multithread_load":true,"num_threads":32}' \
  --block-size 32 \
  --compilation-config '{"cudagraph_mode":"FULL_DECODE_ONLY"}'

# Replace the placeholders with the values for Node 1 and Node 0.
export MODEL_PATH="<YOUR_MODEL_PATH>"
export NIC_NAME="<NODE1_NIC_NAME>"
export LOCAL_IP="<NODE1_IP>"
export NODE0_IP="<NODE0_IP>"
export DP_RPC_PORT=13389

export ASCEND_RT_VISIBLE_DEVICES=0,1,2,3,4,5,6,7
export HCCL_IF_IP="$LOCAL_IP"
export GLOO_SOCKET_IFNAME="$NIC_NAME"
export TP_SOCKET_IFNAME="$NIC_NAME"
export HCCL_SOCKET_IFNAME="$NIC_NAME"
export HCCL_BUFFSIZE=1024
export HCCL_INTRA_PCIE_ENABLE=1
export HCCL_INTRA_ROCE_ENABLE=0

vllm serve "$MODEL_PATH" \
  --headless \
  --host 0.0.0.0 \
  --port 8000 \
  --served-model-name dsv4-vision \
  --max-model-len 130000 \
  --max-num-batched-tokens 4096 \
  --max-num-seqs 32 \
  --gpu-memory-utilization 0.9 \
  --data-parallel-size 4 \
  --data-parallel-size-local 2 \
  --data-parallel-start-rank 2 \
  --data-parallel-address "$NODE0_IP" \
  --data-parallel-rpc-port "$DP_RPC_PORT" \
  --tensor-parallel-size 4 \
  --enable-expert-parallel \
  --tokenizer-mode deepseek_v4 \
  --quantization ascend \
  --model-loader-extra-config '{"enable_multithread_load":true,"num_threads":32}' \
  --block-size 32 \
  --compilation-config '{"cudagraph_mode":"FULL_DECODE_ONLY"}'

Start Node 0 first, then start Node 1. Only Node 0 exposes the API endpoint; Node 1 runs as a headless worker. The multi-node DP parameters have the following meanings:

--data-parallel-size 4 is the global DP size across both servers.
--data-parallel-size-local 2 creates two DP ranks on each server. Each DP rank uses four local dies through TP4, so each A2 server uses all eight dies.
--data-parallel-start-rank 0 assigns DP ranks 0 and 1 to Node 0, while --data-parallel-start-rank 2 assigns ranks 2 and 3 to Node 1.
--data-parallel-address points both servers to Node 0, and --data-parallel-rpc-port must have the same value on both servers.
--headless prevents Node 1 from starting another API server.

Wait until all four DP engines are ready before sending requests to Node 0.

5.3 Service Verification¶

For A3, run the following commands on the serving node. For A2, run them on Node 0; replace 127.0.0.1 with the Node 0 IP address when checking remotely.

Verify the health endpoint:

curl -sS -o /dev/null -w 'HTTP %{http_code}\n' \
  http://127.0.0.1:8000/health

Expected output:

HTTP 200

Verify that the configured model is available:

curl -sS http://127.0.0.1:8000/v1/models | \
  jq '{object, models: [.data[] | {id, object}]}'

Expected output:

{
  "object": "list",
  "models": [
    {
      "id": "dsv4-vision",
      "object": "model"
    }
  ]
}

An HTTP 200 response and a model entry whose id is dsv4-vision indicate that the service is ready.

5.4 Prefill-Decode Disaggregation¶

Prefill-Decode disaggregation is not currently supported for DeepSeek-V4-Flash-Vision-Exp. Use one of the colocated deployments in Sections 5.1 and 5.2.

6 Functional Verification¶

Set IMAGE_URL to an HTTP(S) URL that is reachable from the serving container, then send a multimodal chat-completions request. For the two-node A2 deployment, replace 127.0.0.1 with the Node 0 IP address if the request is sent remotely.

export IMAGE_URL="<YOUR_IMAGE_URL>"

curl -sS -o response.json -w '%{http_code}\n' \
  http://127.0.0.1:8000/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d "{
    \"model\": \"dsv4-vision\",
    \"messages\": [{
      \"role\": \"user\",
      \"content\": [
        {\"type\": \"image_url\", \"image_url\": {\"url\": \"${IMAGE_URL}\"}},
        {\"type\": \"text\", \"text\": \"Describe this image.\"}
      ]
    }],
    \"temperature\": 0,
    \"max_tokens\": 256
  }"

jq -e '.choices[0].message.content | length > 0' response.json

Expected output:

200
true

The generated description is stored in response.json under .choices[0].message.content. To use local image files, mount their parent directory into the container and add --allowed-local-media-path <DIRECTORY> to the serving command.

7 Accuracy Evaluation¶

The merged implementation was validated on OCRBench V1 with 1,000 samples. It completed without request errors and scored 826/1000 (82.6) using the W8A8 checkpoint, concurrency 64, temperature 1.0, top-p 0.95, maximum output length 8192, seed 7, and thinking enabled with high reasoning effort.

For reproducing an accuracy evaluation, refer to Using AISBench. Results depend on the checkpoint, prompt template, decoding parameters, and dataset version; record all of them when comparing runs.

8 Performance Evaluation¶

No production performance baseline is published for this experimental model. Use the AISBench performance evaluation guide with the deployment in Section 5.1, and report image resolution, input/output lengths, request concurrency, TTFT, TPOT, ITL, and throughput.

9 Performance Tuning¶

The values in Section 5.1 are the validated starting point, not globally optimal settings. Tune --max-num-seqs, --max-num-batched-tokens, and --gpu-memory-utilization together for the target image sizes and sequence lengths. Keep the single-node A3 TP4/DP4/EP topology and --block-size 32 until an alternative configuration has been validated.

Refer to the performance tuning guide for general tuning methods.

10 FAQ¶

For common environment, installation, and general parameter issues, please refer to the Public FAQs; this chapter only covers model-specific issues.

11 Limitations¶

Colocated serving is documented on either one Atlas 800 A3 server (128GB × 8 NPUs) or two Atlas 800 A2 servers (64GB × 8 NPUs each).
Prefill-Decode disaggregation, DSA context parallelism, and FlashComm1 are not supported in the documented configuration.
BF16 full-model validation and production performance qualification are not complete.
With TP4 ACL Graph, graph capture batch sizes must be multiples of four.
DSpark uses target-side multimodal Prefill followed by text-only speculative Decode; the draft model does not consume image embeddings or propose tokens inside image spans.

本页原文 Markdown：在 AtomGit 查看·内容源自开源项目 vl/vllm-ascend