华为910B 通过 vllm 部署 Qwen3.8 系列模型
一、环境说明
我这里的环境大概是这样：
- 机器：华为 Ascend 910B
- 系统：openEuler / CentOS 类 Linux
- 驱动和 CANN：已经安装好
- 模型：Qwen3.5-27B
- 部署方式：Docker Compose
- 推理框架：vllm-ascend
先确认宿主机能看到 NPU：
npu-smi info如果能看到设备列表，说明驱动基本没问题。
例如：
+------------------------------------------------------------------------------------------------+
| npu-smi 25.2.1 Version: 25.2.1 |
+---------------------------+---------------+----------------------------------------------------+
| NPU Name | Health | Power(W) Temp(C) Hugepages-Usage(page)|
| Chip | Bus-Id | AICore(%) Memory-Usage(MB) HBM-Usage(MB) |
+===========================+===============+====================================================+
| 0 910B2 | OK | 97.2 41 0 / 0 |
| 0 | 0000:C1:00.0 | 0 0 / 0 64231/ 65536 |
+===========================+===============+====================================================+
| 1 910B2 | OK | 93.3 43 0 / 0 |
| 0 | 0000:01:00.0 | 0 0 / 0 64231/ 65536 |
+===========================+===============+====================================================+
| 2 910B2 | OK | 93.5 39 0 / 0 |
| 0 | 0000:C2:00.0 | 0 0 / 0 64231/ 65536 |
+===========================+===============+====================================================+
| 3 910B2 | OK | 102.5 43 0 / 0 |
| 0 | 0000:02:00.0 | 0 0 / 0 64231/ 65536 |
+===========================+===============+====================================================+
| 4 910B2 | OK | 97.8 41 0 / 0 |
| 0 | 0000:81:00.0 | 0 0 / 0 3390 / 65536 |
+===========================+===============+====================================================+
| 5 910B2 | OK | 97.5 43 0 / 0 |
| 0 | 0000:41:00.0 | 0 0 / 0 3390 / 65536 |
+===========================+===============+====================================================+
| 6 910B2 | OK | 88.6 40 0 / 0 |
| 0 | 0000:82:00.0 | 0 0 / 0 3390 / 65536 |
+===========================+===============+====================================================+
| 7 910B2 | OK | 97.5 43 0 / 0 |
| 0 | 0000:42:00.0 | 0 0 / 0 3390 / 65536 |
+===========================+===============+====================================================+
+---------------------------+---------------+----------------------------------------------------+
| NPU Chip | Process id | Process name | Process memory(MB) |
+===========================+===============+====================================================+
| 0 0 | 1316011 | VLLMWorker_TP | 60891 |
+===========================+===============+====================================================+
| 1 0 | 1316012 | VLLMWorker_TP | 60891 |
+===========================+===============+====================================================+
| 2 0 | 1316013 | VLLMWorker_TP | 60891 |
+===========================+===============+====================================================+
| 3 0 | 1316014 | VLLMWorker_TP | 60891 |
+===========================+===============+====================================================+
| No running processes found in NPU 4 |
+===========================+===============+====================================================+
| No running processes found in NPU 5 |
+===========================+===============+====================================================+
| No running processes found in NPU 6 |
+===========================+===============+====================================================+
| No running processes found in NPU 7 |
+===========================+===============+====================================================+二、准备模型缓存
我不想让容器每次启动都重新下载模型，所以先在宿主机把模型缓存好。
我用的是 ModelScope：
pip install modelscope下载模型：
modelscope download --model Qwen/Qwen3.8-27B默认会下载到：
~/.cache/modelscope/hub/models/Qwen/Qwen3.8-27B如果你的磁盘空间比较紧张，也可以提前设置：
export MODELSCOPE_CACHE=/mnt/disk1/modelscope这样模型就会下载到你指定的目录。
三、docker-compose 配置
我最后用的是下面这份配置。
Ascend 文档明确说 Qwen3.8-27B 是 vLLM-Ascend 0.23.0 首次支持,所以使用最新版镜像 quay.io/ascend/vllm-ascend:v0.23.0
我使用 4 卡来运行，实测 2 卡也可以，但是速度较慢
services:
qwen38-27b:
image: quay.io/ascend/services:
qwen38-27b:
image: quay.io/ascend/vllm-ascend:v0.23.0
container_name: qwen38-27b
network_mode: host
shm_size: "1g"
restart: unless-stopped
environment:
VLLM_USE_MODELSCOPE: "true"
# 限制使用 0~3 号 NPU
ASCEND_RT_VISIBLE_DEVICES: "0,1,2,3"
# 减少 NPU 显存碎片
PYTORCH_NPU_ALLOC_CONF: "expandable_segments:True"
# HCCL 通信缓存
HCCL_BUFFSIZE: "512"
OMP_PROC_BIND: "false"
OMP_NUM_THREADS: "1"
TASK_QUEUE_ENABLE: "1"
devices:
- /dev/davinci0:/dev/davinci0
- /dev/davinci1:/dev/davinci1
- /dev/davinci2:/dev/davinci2
- /dev/davinci3:/dev/davinci3
- /dev/davinci_manager:/dev/davinci_manager
- /dev/devmm_svm:/dev/devmm_svm
- /dev/hisi_hdc:/dev/hisi_hdc
volumes:
# ModelScope 模型
- /mnt/disk1/modelscope/models:/root/.cache/modelscope/hub/models
# Ascend Driver
- /usr/local/dcmi:/usr/local/dcmi
- /usr/local/Ascend/driver/tools/hccn_tool:/usr/local/Ascend/driver/tools/hccn_tool
- /usr/local/bin/npu-smi:/usr/local/bin/npu-smi
- /usr/local/Ascend/driver/lib64/:/usr/local/Ascend/driver/lib64/
- /usr/local/Ascend/driver/version.info:/usr/local/Ascend/driver/version.info
- /etc/ascend_install.info:/etc/ascend_install.info
command:
- vllm
- serve
- /root/.cache/modelscope/hub/models/Qwen/Qwen3___8-27B
- --host
- 0.0.0.0
- --port
- "8000"
- --served-model-name
- Qwen
- --tensor-parallel-size
- "4"
- --max-model-len
- "262144"
- --max-num-batched-tokens
- "8192"
- --gpu-memory-utilization
- "0.90"
- --trust-remote-code
# Prefix KV Cache
- --enable-prefix-caching
# Qwen Thinking / Reasoning
- --reasoning-parser
- qwen3
# Function Calling
- --tool-call-parser
- qwen3_coder
- --enable-auto-tool-choice
# Ascend 异步调度
- --async-scheduling
# Ascend Graph
- --compilation-config
- '{"cudagraph_mode":"FULL_DECODE_ONLY"}'
container_name: qwen38-27b
network_mode: host
shm_size: "1g"
restart: unless-stopped
environment:
VLLM_USE_MODELSCOPE: "true"
# 限制使用 0~3 号 NPU
ASCEND_RT_VISIBLE_DEVICES: "0,1,2,3"
# 减少 NPU 显存碎片
PYTORCH_NPU_ALLOC_CONF: "expandable_segments:True"
# HCCL 通信缓存
HCCL_BUFFSIZE: "512"
OMP_PROC_BIND: "false"
OMP_NUM_THREADS: "1"
TASK_QUEUE_ENABLE: "1"
devices:
- /dev/davinci0:/dev/davinci0
- /dev/davinci1:/dev/davinci1
- /dev/davinci2:/dev/davinci2
- /dev/davinci3:/dev/davinci3
- /dev/davinci_manager:/dev/davinci_manager
- /dev/devmm_svm:/dev/devmm_svm
- /dev/hisi_hdc:/dev/hisi_hdc
volumes:
# ModelScope 模型
- /mnt/disk1/modelscope/models:/root/.cache/modelscope/hub/models
# Ascend Driver
- /usr/local/dcmi:/usr/local/dcmi
- /usr/local/Ascend/driver/tools/hccn_tool:/usr/local/Ascend/driver/tools/hccn_tool
- /usr/local/bin/npu-smi:/usr/local/bin/npu-smi
- /usr/local/Ascend/driver/lib64/:/usr/local/Ascend/driver/lib64/
- /usr/local/Ascend/driver/version.info:/usr/local/Ascend/driver/version.info
- /etc/ascend_install.info:/etc/ascend_install.info
command:
- vllm
- serve
- /root/.cache/modelscope/hub/models/Qwen/Qwen3___8-27B
- --host
- 0.0.0.0
- --port
- "8000"
- --served-model-name
- Qwen
- --tensor-parallel-size
- "4"
- --max-model-len
- "262144"
- --max-num-batched-tokens
- "8192"
- --gpu-memory-utilization
- "0.90"
- --trust-remote-code
# Prefix KV Cache
- --enable-prefix-caching
# Qwen Thinking / Reasoning
- --reasoning-parser
- qwen3
# Function Calling
- --tool-call-parser
- qwen3_coder
- --enable-auto-tool-choice
# Ascend 异步调度
- --async-scheduling
# Ascend Graph
- --compilation-config
- '{"cudagraph_mode":"FULL_DECODE_ONLY"}'启动：
docker-compose up -d查看日志：
docker-compose logs -f如果看到：
INFO: Application startup complete.说明服务已经起来了。
四、测试一下接口
服务起来以后，可以直接用 curl 测。
最简单的聊天：
curl http://127.0.0.1:8000/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{
"model": "Qwen",
"messages": [
{
"role": "user",
"content": "你好，介绍一下你自己"
}
]
}'五、测试视觉能力
Qwen3.5 是统一多模态模型，原生支持视觉(图片、视频输入)。
curl http://127.0.0.1:8000/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{
"model": "Qwen",
"messages": [
{
"role": "user",
"content": [
{
"type": "text",
"text": "图里有什么？"
},
{
"type": "image_url",
"image_url": {
"url": "https://your-image-url"
}
}
]
}
]
}'六、测试工具调用
curl http://127.0.0.1:8000/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{
"model": "Qwen",
"messages": [
{
"role": "user",
"content": "帮我查一下北京今天的天气"
}
],
"tools": [
{
"type": "function",
"function": {
"name": "get_weather",
"description": "获取天气",
"parameters": {
"type": "object",
"properties": {
"city": {
"type": "string"
}
},
"required": ["city"]
}
}
}
],
"tool_choice": "auto"
}'如果返回里出现：
"tool_calls": [
{
"function": {
"name": "get_weather",
"arguments": "{\"city\":\"北京\"}"
}
}
]说明工具调用能力已经正常了。
评论暂时无法加载。