Skip to content

Commit b70a4ef

Browse files
sjhxuedinge233
sjh
authored and
xuedinge233
committed
add doc for torchchat
1 parent 2fcc5ad commit b70a4ef

5 files changed

Lines changed: 298 additions & 1 deletion

File tree

‎index.rst‎

Lines changed: 19 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -37,6 +37,7 @@
3737
sources/trl/index.rst
3838
sources/opencompass/index.rst
3939
sources/lm_deploy/index.rst
40+
sources/torchchat/index.rst
4041

4142
.. warning::
4243

@@ -358,6 +359,24 @@
358359
<a href="sources/lm_deploy/quick_start.html">快速上手</a>
359360
</div>
360361
</div>
362+
<!-- Card 18 -->
363+
<div class="box rounded-lg p-4 flex flex-col items-center">
364+
<div class="flex items-center mb-4">
365+
<div class="img w-16 h-16 rounded-md mr-4" style="background-image: url('_static/images/pytorch.png')"></div>
366+
<div>
367+
<h2 class="text-lg font-semibold">torchchat</h2>
368+
<p class="text-gray-600 desc"></p>
369+
</div>
370+
</div>
371+
<div class="flex-grow"></div>
372+
<div class="flex space-x-4 text-blue-600">
373+
<a href="https://github.com/pytorch/torchchat">官方链接</a>
374+
<span class="split">|</span>
375+
<a href="sources/torchchat/install.html">安装指南</a>
376+
<span class="split">|</span>
377+
<a href="sources/torchchat/quick_start.html">快速上手</a>
378+
</div>
379+
</div>
361380

362381
</div>
363382
</div>

‎sources/llama_cpp/index.rst‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,4 @@
1-
llama.cpp
1+
Llama.cpp
22
===========
33

44
.. toctree::

‎sources/torchchat/index.rst‎

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,8 @@
1+
Torchchat
2+
===========
3+
4+
.. toctree::
5+
:maxdepth: 2
6+
7+
install.rst
8+
quick_start.rst

‎sources/torchchat/install.rst‎

Lines changed: 122 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,122 @@
1+
安装指南
2+
==============
3+
4+
本教程面向使用 torchchat & 昇腾的开发者，帮助完成昇腾环境下 torchchat 的安装。
5+
6+
昇腾环境安装
7+
------------
8+
9+
请根据已有昇腾产品型号及CPU架构等按照 :doc:`快速安装昇腾环境指引 <../ascend/quick_install>` 进行昇腾环境安装，或直接获取对应产品的昇腾环境镜像 `ascendai/cann <https://hub.docker.com/r/ascendai/cann/tags>`_ 。
10+
11+
.. warning::
12+
torchchat 依赖 CANN 最低版本为 8.0.0，安装 CANN 时，请同时安装 Kernel 算子包。
13+
14+
torchchat 安装
15+
----------------------
16+
17+
环境创建
18+
~~~~~~~~~~~~
19+
torchchat 依赖于 Python 3.10+，强烈建议使用 venv 或者 conda 创建 Python 环境。
20+
21+
- 使用 venv 创建 Python 环境
22+
23+
.. code-block:: shell
24+
:linenos:
25+
26+
# 创建名为 torchchat 的 python 3.10 的虚拟环境
27+
python -m venv .venv
28+
# 激活虚拟环境
29+
source .venv/bin/activate
30+
31+
- 使用 conda 创建 Python 环境
32+
33+
.. code-block:: shell
34+
:linenos:
35+
36+
# 安装miniconda
37+
mkdir -p ~/miniconda3
38+
wget https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-aarch64.sh -O ~/miniconda3/miniconda.sh
39+
bash ~/miniconda3/miniconda.sh -b -u -p ~/miniconda3
40+
rm ~/miniconda3/miniconda.sh
41+
42+
# 创建名为 torchchat 的 python 3.10 的虚拟环境
43+
conda create -y -n torchchat python=3.10
44+
# 激活虚拟环境
45+
conda activate torchchat
46+
47+
使用 git 获取源码
48+
~~~~~~~~~~~~~~~~~~~~
49+
50+
.. code-block:: shell
51+
:linenos:
52+
53+
git clone https://github.com/pytorch/torchchat.git
54+
cd torchchat
55+
56+
一键安装
57+
~~~~~~~~~~~~~
58+
torchchat 提供一键安装脚本，安装依赖包。
59+
60+
.. code-block:: shell
61+
:linenos:
62+
63+
# 安装依赖包
64+
./install/install_requirements.sh
65+
# 查看安装的依赖包
66+
pip list
67+
68+
以下为主要依赖包：
69+
70+
.. code-block:: shell
71+
:linenos:
72+
73+
Package Version
74+
------------------------ -------------------
75+
torch 2.7.0.dev20250310+cpu
76+
torchao 0.10.0+git711fa080
77+
torchdata 0.11.0
78+
torchtune 0.6.0
79+
torchvision 0.22.0.dev20250310
80+
81+
82+
安装 torch_npu
83+
~~~~~~~~~~~~~~~
84+
torch_npu 是昇腾 AI 计算框架的核心组件，提供了对昇腾硬件的支持，
85+
目前 torchchat 支持 2.7.0 版本的 torch_npu，安装时请确保 CANN 版本为 8.0.0 及以上。
86+
87+
nightly 版本的 torch_npu 需要通过源代码进行安装。
88+
89+
环境安装:
90+
91+
.. code-block:: shell
92+
93+
# 获取源代码
94+
git clone https://github.com/Ascend/pytorch.git
95+
cd pytorch
96+
97+
# 安装依赖包
98+
pip install wheel
99+
pip install -r requirements.txt
100+
101+
# 安装 torch_npu
102+
python setup.py build build_py
103+
104+
# 查看安装版本
105+
torch_version=$(python -c "import torch; print(torch.__version__)")
106+
torch_npu_version=$(python -c "import torch_npu; print(torch_npu.__version__)")
107+
echo "torch version: ${torch_version}"
108+
echo "torch_npu version: ${torch_npu_version}"
109+
110+
卸载 torch_npu
111+
~~~~~~~~~~~~~~~~
112+
torch_npu 依赖于 CANN 的安装路径，卸载时请确保 CANN 的安装路径正确。
113+
114+
.. code-block:: shell
115+
:linenos:
116+
117+
# 卸载 torch_npu
118+
cd pytorch
119+
python setup.py clean
120+
121+
# 卸载依赖包
122+
pip uninstall -y torch_npu

‎sources/torchchat/quick_start.rst‎

Lines changed: 148 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,148 @@
1+
快速开始
2+
==================
3+
4+
.. note::
5+
6+
阅读本篇前，请确保已按照 :doc:`安装教程 <./install>` 准备好昇腾环境及 torchchat ！
7+
8+
本篇教程将介绍如何使用 torchchat 进行快速开发，帮助您快速上手 torchchat。
9+
10+
查看帮助
11+
-----------------
12+
torchchat 提供了帮助命令，帮助用户快速了解 torchchat 的使用方法。
13+
14+
.. code-block:: shell
15+
:linenos:
16+
17+
# 查看帮助
18+
python torchchat.py --help
19+
20+
输出列表:
21+
22+
.. code-block:: shell
23+
:linenos:
24+
25+
usage: torchchat [-h] {chat,browser,generate,export,eval,download,list,remove,where,server} ...
26+
27+
positional arguments:
28+
{chat,browser,generate,export,eval,download,list,remove,where,server}
29+
The specific command to run
30+
chat Chat interactively with a model via the CLI
31+
generate Generate responses from a model given a prompt
32+
browser Chat interactively with a model in a locally hosted browser
33+
export Export a model artifact to AOT Inductor or ExecuTorch
34+
download Download model artifacts
35+
list List all supported models
36+
remove Remove downloaded model artifacts
37+
where Return directory containing downloaded model artifacts
38+
server [WIP] Starts a locally hosted REST server for model interaction
39+
eval Evaluate a model via lm-eval
40+
41+
options:
42+
-h, --help show this help message and exit
43+
44+
主要命令:
45+
46+
Inference (chat, generate, browser)
47+
- chat: 交互式聊天
48+
- generate: 生成响应
49+
- browser: 在本地浏览器中交互式聊天
50+
51+
Inventory Management (download, list, remove, where)
52+
- download: 下载模型
53+
- list: 列出所有支持的模型
54+
- remove: 删除下载的模型
55+
- where: 返回包含下载的模型的目录
56+
57+
下载模型
58+
-----------------
59+
torchchat 大多数模型使用 Hugging Face 作为分发渠道，因此需要创建一个 Hugging Face 帐户。
60+
按照 `此处 <https://huggingface.co/docs/hub/en/security-tokens>`_ 的说明使用write角色创建 Hugging Face 用户访问令牌。
61+
62+
登录 huggingface:
63+
64+
.. code-block:: shell
65+
:linenos:
66+
67+
# 登录 huggingface
68+
huggingface-cli login
69+
70+
查看支持模型列表:
71+
72+
.. code-block:: shell
73+
:linenos:
74+
75+
# 查看支持的模型列表
76+
python torchchat.py list
77+
78+
下载模型:
79+
80+
.. code-block:: shell
81+
:linenos:
82+
83+
# 下载模型
84+
python torchchat.py download <model_name>
85+
86+
# 例如下载 LLaMA 模型
87+
python torchchat.py download llama3.1
88+
89+
模型推理
90+
-----------------
91+
torchchat 支持多种推理方式，用户可以根据自己的需求选择合适的推理方式。
92+
93+
Chat
94+
以互动方式与模型进行聊天:
95+
96+
.. code-block:: shell
97+
:linenos:
98+
99+
# 交互式聊天
100+
python torchchat.py chat llama3.1
101+
102+
Generate
103+
根据输入提示生成文本:
104+
105+
.. code-block:: shell
106+
:linenos:
107+
108+
# 生成响应
109+
python torchchat.py generate llama3.1 --prompt "write me a story about a boy and his bear"
110+
111+
生成结果示例:
112+
113+
.. code-block:: shell
114+
:linenos:
115+
116+
Using device=npu Ascend910B3
117+
Loading model...
118+
Time to load model: 4.42 seconds
119+
-----------------------------------------------------------
120+
write me a story about a boy and his bear friend
121+
Once upon a time, in a dense forest, there lived a young boy named Timmy. Timmy was a curious and adventurous boy who loved exploring the woods behind his village. One day, while wandering deeper into the forest than he had ever gone before, Timmy stumbled upon a magnificent brown bear. The bear was enormous, with a thick coat of fur and piercing yellow eyes. At first, Timmy was frightened, but to his surprise, the bear didn't seem to be threatening him. Instead, the bear gently approached Timmy and began to sniff him.
122+
123+
As the days passed, Timmy and the bear, whom he named Boris, became inseparable friends. Boris was unlike any bear Timmy had ever seen before. He was incredibly intelligent and could understand human language. Boris would often sit by Timmy's side as he read books or helped with his chores. The villagers were initially wary of Boris, but as they saw how kind and gentle he was, they grew
124+
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
125+
Generated 199 tokens
126+
Time for inference 1: 13.3118 sec total
127+
Time to first token: 0.6189 sec with parallel prefill.
128+
129+
Total throughput: 15.0242 tokens/sec, 0.0666 s/token
130+
First token throughput: 1.6157 tokens/sec, 0.6189 s/token
131+
Next token throughput: 15.6781 tokens/sec, 0.0638 s/token
132+
133+
Bandwidth achieved: 241.30 GB/s
134+
*** This first iteration will include cold start effects for dynamic import, hardware caches. ***
135+
136+
========================================
137+
138+
139+
Warning: Excluding compile in calculations
140+
Average tokens/sec (total): 15.02
141+
Average tokens/sec (first token): 1.62
142+
Average tokens/sec (next tokens): 15.68
143+
144+
Memory used: 17.23 GB
145+
146+
如上所示，torchchat 会对输入的文本进行处理，并生成相应的文本，同时输出生成的文本长度、推理时间、带宽等信息，方便用户进行性能分析。
147+
148+
以上是 torchchat 的快速开始教程，更多其他昇腾原生支持功能请参考 :doc:`昇腾开源 <../../index>` 。

0 commit comments

Comments
 (0)