商元义
parent
d050bc667a
commit
39dea7f9e8
19
README.md
19
README.md
|
|
@ -317,7 +317,16 @@ ModelLink旨在为华为 [昇腾芯片](https://open.codehub.huawei.com/OpenBaiz
|
|||
<td> 【昇腾贡献】 </td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td rowspan="6"><a href="examples/qwen15/README.md">Qwen1.5</a></td>
|
||||
<td rowspan="7"><a href="examples/qwen15/README.md">Qwen1.5</a></td>
|
||||
<td>0.5B</td>
|
||||
<td> <a href="examples/qwen15/pretrain_qwen15_0point5b_ptd.sh"> pretrain </a> </td>
|
||||
<td> <a href="examples/qwen15/generate_qwen15_0point5b_ptd.sh"> generate </a> </td>
|
||||
<td> -- </td>
|
||||
<td> -- </td>
|
||||
<td> -- </td>
|
||||
<td> <a href="examples/qwen15/evaluate_qwen15_0point5b_ptd.sh"> eval </a> </td>
|
||||
<td rowspan="1"> 【社区贡献】 </td>
|
||||
<tr>
|
||||
<td>1.8B</td>
|
||||
<td> <a href="examples/qwen15/pretrain_qwen15_1point8b_ptd.sh"> pretrain </a> </td>
|
||||
<td> <a href="examples/qwen15/generate_qwen15_1point8b_ptd.sh"> generate </a> </td>
|
||||
|
|
@ -675,7 +684,13 @@ ModelLink旨在为华为 [昇腾芯片](https://open.codehub.huawei.com/OpenBaiz
|
|||
</tr>
|
||||
</tr>
|
||||
<tr>
|
||||
<td rowspan="6"><a href="examples/qwen15/README.md">Qwen1.5</a></td>
|
||||
<td rowspan="7"><a href="examples/qwen15/README.md">Qwen1.5</a></td>
|
||||
<td> 0.5B </td>
|
||||
<td> 1x8 </td>
|
||||
<td> BF16 </td>
|
||||
<td> 22834 </td>
|
||||
<td> 25306 </td>
|
||||
<tr>
|
||||
<td> 1.8B </td>
|
||||
<td> 1x8 </td>
|
||||
<td> BF16 </td>
|
||||
|
|
|
|||
19
README_en.md
19
README_en.md
|
|
@ -316,7 +316,16 @@ Current ModelLink supports pre-training and fine-tuning for the following models
|
|||
<td> 【Ascend】 </td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td rowspan="6"><a href="examples/qwen15/README.md">Qwen1.5</a></td>
|
||||
<td rowspan="7"><a href="examples/qwen15/README.md">Qwen1.5</a></td>
|
||||
<td>0.5B</td>
|
||||
<td> <a href="examples/qwen15/pretrain_qwen15_0point5b_ptd.sh"> pretrain </a> </td>
|
||||
<td> <a href="examples/qwen15/generate_qwen15_0point5b_ptd.sh"> generate </a> </td>
|
||||
<td> -- </td>
|
||||
<td> -- </td>
|
||||
<td> -- </td>
|
||||
<td> <a href="examples/qwen15/evaluate_qwen15_0point5b_ptd.sh"> eval </a> </td>
|
||||
<td rowspan="1"> 【Community】 </td>
|
||||
<tr>
|
||||
<td>1.8B</td>
|
||||
<td> <a href="examples/qwen15/pretrain_qwen15_1point8b_ptd.sh"> pretrain </a> </td>
|
||||
<td> <a href="examples/qwen15/generate_qwen15_1point8b_ptd.sh"> generate </a> </td>
|
||||
|
|
@ -671,7 +680,13 @@ For the supported models listed above, we provide training scripts and readme in
|
|||
<td> 345 </td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td rowspan="6"><a href="examples/qwen15/README.md">Qwen1.5</a></td>
|
||||
<td rowspan="7"><a href="examples/qwen15/README.md">Qwen1.5</a></td>
|
||||
<td> 0.5B </td>
|
||||
<td> 1x8 </td>
|
||||
<td> BF16 </td>
|
||||
<td> 22834 </td>
|
||||
<td> 25306 </td>
|
||||
<tr>
|
||||
<td> 1.8B </td>
|
||||
<td> 1x8 </td>
|
||||
<td> BF16 </td>
|
||||
|
|
|
|||
|
|
@ -6,6 +6,13 @@
|
|||
</p>
|
||||
|
||||
# 目录
|
||||
- [Qwen1.5-0.5B](#Qwen1.5-0.5b)
|
||||
- [训练-0.5B](#训练-0.5b)
|
||||
- [脚本-0.5B](#脚本-0.5b)
|
||||
- [性能-0.5B](#性能-0.5b)
|
||||
- [吞吐-0.5B](#吞吐-0.5b)
|
||||
- [推理-0.5B](#推理-0.5b)
|
||||
- [评估-0.5B](#评估-0.5b)
|
||||
- [Qwen1.5-1.8B](#Qwen1.5-1.8b)
|
||||
- [训练-1.8B](#训练-1.8b)
|
||||
- [脚本-1.8B](#脚本-1.8b)
|
||||
|
|
@ -48,6 +55,252 @@
|
|||
- [吞吐-72B](#吞吐-72b)
|
||||
- [推理-72B](#推理-72b)
|
||||
- [评估-72B](#评估-72b)
|
||||
# Qwen1.5-0.5B
|
||||
|
||||
## 训练-0.5B
|
||||
Qwen1.5-0.5B 训练的硬件配置:
|
||||
|
||||
| 硬件 | 配置 |
|
||||
|:---:|:---------------:|
|
||||
| NPU | 8 x Ascend NPUs |
|
||||
### 脚本-0.5B
|
||||
|
||||
1. 克隆仓库到本地服务器
|
||||
|
||||
```shell
|
||||
git clone https://gitee.com/ascend/ModelLink.git
|
||||
git clone https://github.com/NVIDIA/Megatron-LM.git
|
||||
cd Megatron-LM
|
||||
git checkout core_r0.6.0
|
||||
cp -r megatron ../ModelLink/
|
||||
cd ..
|
||||
cd ModelLink
|
||||
mkdir logs
|
||||
mkdir model_from_hf
|
||||
mkdir dataset
|
||||
mkdir ckpt
|
||||
```
|
||||
2. 搭建环境
|
||||
|
||||
```bash
|
||||
# python3.8
|
||||
conda create -n test python=3.8
|
||||
conda activate test
|
||||
|
||||
# 安装 torch 和 torch_npu
|
||||
pip install torch-2.2.0-cp38-cp38m-manylinux2014_aarch64.whl
|
||||
pip install torch_npu-2.2.0*-cp38-cp38m-linux_aarch64.whl
|
||||
pip install apex-0.1_ascend*-cp38-cp38m-linux_aarch64.whl
|
||||
|
||||
# 修改 ascend-toolkit 路径
|
||||
source /usr/local/Ascend/ascend-toolkit/set_env.sh
|
||||
|
||||
# 安装加速库
|
||||
git clone https://gitee.com/ascend/MindSpeed.git
|
||||
cd MindSpeed
|
||||
git checkout 2b0edd2
|
||||
pip install -r requirements.txt
|
||||
pip3 install -e .
|
||||
cd ..
|
||||
|
||||
# 安装其余依赖库
|
||||
pip install -r requirements.txt
|
||||
```
|
||||
3. 下载 Qwen1.5-0.5B 的 [预训练权重和词表](https://huggingface.co/Qwen/Qwen1.5-0.5B/tree/main)
|
||||
|
||||
```shell
|
||||
#!/bin/bash
|
||||
mkdir ./model_from_hf/qwen15-0.5b-hf/
|
||||
cd ./model_from_hf/qwen15-0.5b-hf/
|
||||
wget https://huggingface.co/Qwen/Qwen1.5-0.5B/resolve/main/config.json
|
||||
wget https://huggingface.co/Qwen/Qwen1.5-0.5B/resolve/main/generation_config.json
|
||||
wget https://huggingface.co/Qwen/Qwen1.5-0.5B/resolve/main/merges.txt
|
||||
wget https://huggingface.co/Qwen/Qwen1.5-0.5B/resolve/main/model.safetensors
|
||||
wget https://huggingface.co/Qwen/Qwen1.5-0.5B/resolve/main/tokenizer.json
|
||||
wget https://huggingface.co/Qwen/Qwen1.5-0.5B/resolve/main/tokenizer_config.json
|
||||
wget https://huggingface.co/Qwen/Qwen1.5-0.5B/resolve/main/vocab.json
|
||||
cd ../../
|
||||
```
|
||||
4. 权重转换
|
||||
|
||||
4.1 将权重从 huggingface 格式转化为 magatron 格式
|
||||
***（该场景一般用于使能开源的HuggingFace模型在Megatron上进行训练）***
|
||||
|
||||
```bash
|
||||
# 修改 ascend-toolkit 路径
|
||||
source /usr/local/Ascend/ascend-toolkit/set_env.sh
|
||||
|
||||
# 权重格式转换
|
||||
python tools/checkpoint/convert_ckpt.py \
|
||||
--model-type GPT \
|
||||
--loader llama2_hf \
|
||||
--saver megatron \
|
||||
--target-tensor-parallel-size 1 \
|
||||
--target-pipeline-parallel-size 1 \
|
||||
--add-qkv-bias \
|
||||
--load-dir ./model_from_hf/qwen15-0.5b-hf/ \
|
||||
--save-dir ./model_weights/qwen15-0.5b-hf-v0.1-tp1-pp1/ \
|
||||
--tokenizer-model ./model_from_hf/qwen15-0.5b-hf/tokenizer.json
|
||||
```
|
||||
|
||||
4.2 任意并行切分策略的 Megatron 权重 格式转化为 HuggingFace权重
|
||||
***（该场景一般用于将训练好的megatron模型重新转回HuggingFace格式）***
|
||||
|
||||
```shell
|
||||
# 请按照您的真实环境修改 set_env.sh 路径
|
||||
source /usr/local/Ascend/ascend-toolkit/set_env.sh
|
||||
python tools/checkpoint/convert_ckpt.py \
|
||||
--model-type GPT \
|
||||
--loader megatron \
|
||||
--saver megatron \
|
||||
--save-model-type save_huggingface_llama \
|
||||
--load-dir ./ckpt/ \
|
||||
--target-tensor-parallel-size 1 \
|
||||
--target-pipeline-parallel-size 1 \
|
||||
--add-qkv-bias \
|
||||
--save-dir ./model_from_hf/qwen15-0.5b-hf/ # <-- 需要填入原始HF模型路径，新权重会存于./model_from_hf/qwen15-0.5b-hf/mg2hg/
|
||||
```
|
||||
|
||||
权重转换适用于预训练、微调、推理和评估，根据任务不同调整参数 `target-tensor-parallel-size`和 `target-pipeline-parallel-size`。
|
||||
5. 预训练
|
||||
|
||||
5.1 准备数据集
|
||||
|
||||
下载 Qwen1.5-0.5B [数据集](https://huggingface.co/datasets/tatsu-lab/alpaca/resolve/main/data/train-00000-of-00001-a09b74b3ef9c3b56.parquet)
|
||||
|
||||
```shell
|
||||
# 下载数据
|
||||
cd ./dataset
|
||||
wget https://huggingface.co/datasets/tatsu-lab/alpaca/resolve/main/data/train-00000-of-00001-a09b74b3ef9c3b56.parquet
|
||||
cd ..
|
||||
# 处理数据
|
||||
mkdir ./dataset/qwen15-0.5b-hf/
|
||||
python ./tools/preprocess_data.py \
|
||||
--input ./dataset/train-00000-of-00001-a09b74b3ef9c3b56.parquet \
|
||||
--tokenizer-name-or-path ./model_from_hf/qwen15-0.5b-hf/ \
|
||||
--output-prefix ./dataset/qwen15-0.5b-hf/alpaca \
|
||||
--workers 4 \
|
||||
--log-interval 1000 \
|
||||
--tokenizer-type PretrainedFromHF
|
||||
```
|
||||
|
||||
5.2 预训练
|
||||
|
||||
```shell
|
||||
# 设置 ascend-toolkit 路径
|
||||
source /usr/local/Ascend/ascend-toolkit/set_env.sh
|
||||
|
||||
# 根据实际情况配置词表、数据集、模型参数保存路径
|
||||
CKPT_SAVE_DIR="./ckpt/qwen15-0.5b-hf/"
|
||||
TOKENIZER_PATH="./model_from_hf/qwen15-0.5b-hf" #词表路径
|
||||
DATA_PATH="./dataset/qwen15-0.5b-hf/alpaca_text_document" #数据集路径
|
||||
CKPT_LOAD_DIR="./model_weights/qwen15-0.5b-hf-v0.1-tp1-pp1"
|
||||
```
|
||||
启动 Qwen1.5-0.5B 预训练脚本: examples/qwen15/pretrain_qwen15_0point5b_ptd.sh
|
||||
|
||||
6. 微调
|
||||
|
||||
6.1 准备微调数据集
|
||||
下载微调数据集 [这里](https://huggingface.co/datasets/tatsu-lab/alpaca/resolve/main/data/train-00000-of-00001-a09b74b3ef9c3b56.parquet)
|
||||
|
||||
```shell
|
||||
# 下载数据集
|
||||
mkdir finetune_dataset
|
||||
cd ./finetune_dataset
|
||||
wget https://huggingface.co/datasets/tatsu-lab/alpaca/resolve/main/data/train-00000-of-00001-a09b74b3ef9c3b56.parquet
|
||||
cd ..
|
||||
|
||||
# 处理微调数据集
|
||||
mkdir ./finetune_dataset/qwen15-0.5b-hf/
|
||||
python ./tools/preprocess_data.py \
|
||||
--input ./dataset/train-00000-of-00001-a09b74b3ef9c3b56.parquet \
|
||||
--tokenizer-name-or-path ./model_from_hf/qwen15-0.5b-hf/ \
|
||||
--output-prefix ./finetune_dataset/qwen15-0.5b-hf/alpaca \
|
||||
--workers 4 \
|
||||
--log-interval 1000 \
|
||||
--tokenizer-type PretrainedFromHF \
|
||||
--handler-name GeneralInstructionHandler \
|
||||
--append-eod
|
||||
```
|
||||
|
||||
6.2 全参微调
|
||||
全参微调的配置脚本基本和预训练脚本一致. *区别是数据集，以及增加训练参数--is-instruction-dataset*
|
||||
|
||||
增加微调参数--finetune，参数如下：
|
||||
|
||||
```bash
|
||||
DATA_PATH="./finetune_dataset/qwen15-0.5b-hf/alpaca"
|
||||
TOKENIZER_PATH="./model_from_hf/qwen15-0.5b-hf/"
|
||||
CKPT_PATH="./model_weights/qwen15-0.5b-hf-v0.1-tp1-pp1/"
|
||||
--load ${CKPT_PATH} \
|
||||
--finetune \
|
||||
--is-instruction-dataset \
|
||||
--tokenizer-type PretrainedFromHF \
|
||||
--tokenizer-name-or-path ${TOKENIZER_PATH} \
|
||||
--tokenizer-not-use-fast \
|
||||
```
|
||||
### 性能-0.5B
|
||||
|
||||
#### 吞吐-0.5B
|
||||
|
||||
Qwen1.5-0.5B 在 **昇腾芯片** 和 **参考芯片** 上的性能对比：
|
||||
|
||||
| 设备 | 模型 | 迭代数 | tokens吞吐 (tokens/s/p) |
|
||||
| :--: |:------------:|:----:|:---------------------:|
|
||||
| NPUs | Qwen1.5-0.5B | 2000 | 22834 |
|
||||
| 参考 | Qwen1.5-0.5B | 2000 | 25306 |
|
||||
|
||||
## 推理-0.5B
|
||||
|
||||
配置Qwen1.5-0.5B 推理脚本: examples/qwen15/generate_qwen1.5_0point5b_ptd.sh
|
||||
|
||||
```bash
|
||||
# 根据您自己的 ascend-toolkit 路径，执行set_env.sh
|
||||
source /usr/local/Ascend/ascend-toolkit/set_env.sh
|
||||
|
||||
# 修改模型权重路径和词表路径
|
||||
CHECKPOINT="./model_weights/qwen15-0.5b-hf-v0.1-tp1-pp1"
|
||||
TOKENIZER_PATH="./model_from_hf/qwen15-0.5b-hf/"
|
||||
```
|
||||
|
||||
启动Qwen1.5-0.5B 推理脚本
|
||||
|
||||
```bash
|
||||
bash examples/qwen15/generate_qwen15_0point5b_ptd.sh
|
||||
```
|
||||
|
||||
推理示例如下：
|
||||
|
||||
|
||||
|
||||
## 评估-0.5B
|
||||
|
||||
使用 MMLU数据集评估模型. 数据集下载路径 [这里](https://huggingface.co/datasets/cais/mmlu).
|
||||
配置Qwen1.5-0.5B 评估脚本: examples/qwen15/evaluate_qwen1.5_0point5b_ptd.sh
|
||||
|
||||
```bash
|
||||
# ascend-toolkit 路径
|
||||
source /usr/local/Ascend/ascend-toolkit/set_env.sh
|
||||
|
||||
# 修改模型参数路径和词表路径
|
||||
TOKENIZER_PATH="./model_from_hf/qwen15-0.5b-hf/" #词表路径
|
||||
CHECKPOINT="./model_weights/qwen15-0.5b-hf-v0.1-tp1-pp1" #模型路径
|
||||
# 配置任务和数据集路径
|
||||
DATA_PATH="./mmlu/data/test/"
|
||||
TASK="mmlu"
|
||||
```
|
||||
|
||||
启动评估
|
||||
|
||||
```bash
|
||||
bash examples/qwen15/evaluate_qwen15_0point5b_ptd.sh
|
||||
```
|
||||
|
||||
评估结果如下
|
||||
|
||||
| 数据集 | 总学科数 | 总问题数 | 参考准确率 | NPU准确率 |
|
||||
| :----: | :------: | :------: |:-----:|:------:|
|
||||
| MMLU | 57 | 14042 | 0.318 | 0.318 |
|
||||
# Qwen1.5-1.8B
|
||||
|
||||
## 训练-1.8B
|
||||
|
|
@ -260,7 +513,7 @@ TOKENIZER_PATH="./model_from_hf/qwen15-1.8b-hf/"
|
|||
启动Qwen1.5-1.8B 推理脚本
|
||||
|
||||
```bash
|
||||
bash examples/qwen15/generate_qwen15_1.8b_ptd.sh
|
||||
bash examples/qwen15/generate_qwen15_1point8b_ptd.sh
|
||||
```
|
||||
|
||||
推理示例如下：
|
||||
|
|
@ -498,7 +751,7 @@ Qwen1.5-4B 在 **昇腾芯片** 和 **参考芯片** 上的性能对比：
|
|||
|
||||
## 推理-4B
|
||||
|
||||
配置Qwen1.5-4B 推理脚本: examples/qwen1.5/generate_qwen1.5_4b_ptd.sh
|
||||
配置Qwen1.5-4B 推理脚本: examples/qwen15/generate_qwen15_4b_ptd.sh
|
||||
|
||||
```bash
|
||||
# 根据您自己的 ascend-toolkit 路径，执行set_env.sh
|
||||
|
|
@ -522,7 +775,7 @@ bash examples/qwen15/generate_qwen15_4b_ptd.sh
|
|||
## 评估-4B
|
||||
|
||||
使用 MMLU数据集评估模型. 数据集下载路径 [这里](https://huggingface.co/datasets/cais/mmlu).
|
||||
配置Qwen1.5-4B 评估脚本: examples/qwen1.5/evaluate_qwen1.5_4b_ptd.sh
|
||||
配置Qwen1.5-4B 评估脚本: examples/qwen15/evaluate_qwen15_4b_ptd.sh
|
||||
|
||||
```bash
|
||||
# ascend-toolkit 路径
|
||||
|
|
@ -809,9 +1062,9 @@ TASK="mmlu" # ceval任务配置为 "ceval"
|
|||
bash examples/qwen15/evaluate_qwen15_7b_ptd.sh
|
||||
```
|
||||
|
||||
| 数据集 | 总学科数 | 总问题数 | 参考准确率 | NPU准确率 |
|
||||
|:-----:|:----:|:-----:|:-------------------------------------------------------:|:------:|
|
||||
| MMLU | 57 | 14042 | [61.0](https://qwenlm.github.io/zh/blog/qwen1.5) | 60.3 |
|
||||
| 数据集 | 总学科数 | 总问题数 | 参考准确率 | NPU准确率 |
|
||||
|:-----:|:----:|:-----:|:----------------------------------------------------:|:------:|
|
||||
| MMLU | 57 | 14042 | [61.0](https://qwenlm.github.io/zh/blog/qwen1.5) | 0.603 |
|
||||
|
||||
|
||||
# Qwen1.5-14B
|
||||
|
|
@ -1225,8 +1478,8 @@ bash examples/qwen15/evaluate_qwen15_14b_ptd.sh
|
|||
# 根据实际情况配置词表、数据集、模型参数保存路径
|
||||
CKPT_SAVE_DIR="./ckpt/Qwen1.5-32B/"
|
||||
TOKENIZER_PATH="./model_from_hf/Qwen1.5-32B/" #词表路径
|
||||
DATA_PATH="./dataset/Qwen1.5-32B-hf/alpaca_text_document" #数据集路径
|
||||
CKPT_LOAD_DIR="./model_weights/Qwen1.5-32B-v0.1-tp8-pp4-vpp2/"
|
||||
DATA_PATH="./dataset/qwen1.5-32B-hf/alpaca_text_document" #数据集路径
|
||||
CKPT_LOAD_DIR="./model_weights/qwen1.5-32B-v0.1-tp8-pp4-vpp2/"
|
||||
```
|
||||
|
||||
启动 Qwen1.5-32B 预训练脚本: examples/qwen15/pretrain_qwen15_32b_ptd.sh
|
||||
|
|
@ -1250,7 +1503,7 @@ bash examples/qwen15/evaluate_qwen15_14b_ptd.sh
|
|||
# 处理微调数据集
|
||||
mkdir ./finetune_dataset/qwen-1.5-32b-hf/
|
||||
python ./tools/preprocess_data.py \
|
||||
--input ./dataset/ train-00000-of-00001-a09b74b3ef9c3b56.parquet \
|
||||
--input ./finetune_dataset/ train-00000-of-00001-a09b74b3ef9c3b56.parquet \
|
||||
--tokenizer-name-or-path ./model_from_hf/Qwen1.5-32B/ \
|
||||
--output-prefix ./finetune_dataset/qwen-1.5-32b-hf/alpaca \
|
||||
--workers 4 \
|
||||
|
|
@ -1340,7 +1593,7 @@ bash ./examples/qwen15/generate_qwen15_32b_lora_chat_ptd.sh
|
|||
```
|
||||
|
||||
lora微调后的推理效果如下：
|
||||
|
||||
|
||||
|
||||
## 评估-32B
|
||||
|
||||
|
|
@ -1589,7 +1842,7 @@ Qwen1.5-72B 在 **昇腾芯片** 和 **参考芯片** 上的性能对比：
|
|||
|
||||
## 推理-72B
|
||||
|
||||
配置 qwen1.5-72b 推理脚本：examples/qwen15/generate_qwen15_72b_ptd.sh
|
||||
配置 qwen1.5-72b 推理脚本：examples/qwen15/generate_qwen15_72b_chat_ptd.sh
|
||||
|
||||
```bash
|
||||
# ascend-toolkit 路径
|
||||
|
|
@ -1623,7 +1876,7 @@ bash ./examples/qwen15/generate_qwen15_72b_lora_chat_ptd.sh
|
|||
```
|
||||
|
||||
lora微调后的推理效果如下：
|
||||
|
||||
|
||||
|
||||
## 评估-72B
|
||||
|
||||
|
|
|
|||
|
|
@ -5,6 +5,13 @@
|
|||
</p>
|
||||
|
||||
# Contents
|
||||
- [Qwen1.5-0.5B](#Qwen1.5-0.5b)
|
||||
- [Training-0.5B](#training-0.5b)
|
||||
- [Script-0.5B](#script-0.5b)
|
||||
- [Performance-0.5B](#performance-0.5b)
|
||||
- [Machine performance 0.5B](#machine-performance-0.5b)
|
||||
- [Inference-0.5B](#inference-0.5b)
|
||||
- [Evaluation-0.5B](#evaluation-0.5b)
|
||||
- [Qwen1.5-1.8B](#Qwen1.5-1.8b)
|
||||
- [Training-1.8B](#training-1.8b)
|
||||
- [Script-1.8B](#script-1.8b)
|
||||
|
|
@ -47,6 +54,250 @@
|
|||
- [Machine performance 72B](#machine-performance-72b)
|
||||
- [Inference-72B](#Inference-72b)
|
||||
- [Evaluation-72B](#Evaluation-72b)
|
||||
# Qwen1.5-0.5B
|
||||
|
||||
## Training-0.5B
|
||||
Here's a hardware summary of pre-training Qwen-0.5B:
|
||||
|
||||
| Hardware | Value |
|
||||
| :------: | :---------------------------------------------: |
|
||||
| NPU | 8 x Ascend NPUs |
|
||||
### Script
|
||||
|
||||
1. Clone the repository to your local server
|
||||
|
||||
```shell
|
||||
git clone https://gitee.com/ascend/ModelLink.git
|
||||
git clone https://github.com/NVIDIA/Megatron-LM.git
|
||||
cd Megatron-LM
|
||||
git checkout core_r0.6.0
|
||||
cp -r megatron ../ModelLink/
|
||||
cd ..
|
||||
cd ModelLink
|
||||
mkdir logs
|
||||
mkdir model_from_hf
|
||||
mkdir dataset
|
||||
mkdir ckpt
|
||||
```
|
||||
2. Build environment
|
||||
|
||||
```bash
|
||||
# python3.8
|
||||
conda create -n test python=3.8
|
||||
conda activate test
|
||||
|
||||
# install torch 和 torch_npu
|
||||
pip install torch-2.2.0-cp38-cp38m-manylinux2014_aarch64.whl
|
||||
pip install torch_npu-2.2.0*-cp38-cp38m-linux_aarch64.whl
|
||||
pip install apex-0.1_ascend*-cp38-cp38m-linux_aarch64.whl
|
||||
|
||||
# modify ascend-toolkit path
|
||||
source /usr/local/Ascend/ascend-toolkit/set_env.sh
|
||||
|
||||
# install MindSpeed
|
||||
git clone https://gitee.com/ascend/MindSpeed.git
|
||||
cd MindSpeed
|
||||
git checkout 2b0edd2
|
||||
pip install -r requirements.txt
|
||||
pip3 install -e .
|
||||
cd ..
|
||||
|
||||
# install other packages
|
||||
pip install -r requirements.txt
|
||||
```
|
||||
3. Download Qwen1.5-0.5B [pretrained weights and tokenizer](https://huggingface.co/Qwen/Qwen1.5-1.8B/tree/main)
|
||||
|
||||
```shell
|
||||
#!/bin/bash
|
||||
mkdir ./model_from_hf/qwen15-0.5b-hf/
|
||||
cd ./model_from_hf/qwen15-4b-hf/
|
||||
wget https://huggingface.co/Qwen/Qwen1.5-0.5B/resolve/main/config.json
|
||||
wget https://huggingface.co/Qwen/Qwen1.5-0.5B/resolve/main/generation_config.json
|
||||
wget https://huggingface.co/Qwen/Qwen1.5-0.5B/resolve/main/merges.txt
|
||||
wget https://huggingface.co/Qwen/Qwen1.5-0.5B/resolve/main/model.safetensors
|
||||
wget https://huggingface.co/Qwen/Qwen1.5-0.5B/resolve/main/tokenizer.json
|
||||
wget https://huggingface.co/Qwen/Qwen1.5-0.5B/resolve/main/tokenizer_config.json
|
||||
wget https://huggingface.co/Qwen/Qwen1.5-0.5B/resolve/main/vocab.json
|
||||
cd ../../
|
||||
```
|
||||
4. weight conversion in ptd mode
|
||||
|
||||
4.1 Convert weights from huggingface format to megatron format
|
||||
***(This scenario is generally used to train open-source HuggingFace models on Megatron)***
|
||||
|
||||
```bash
|
||||
# modify the script according to your own ascend-toolkit path
|
||||
source /usr/local/Ascend/ascend-toolkit/set_env.sh
|
||||
|
||||
# convert to ptd weights
|
||||
python tools/checkpoint/convert_ckpt.py \
|
||||
--model-type GPT \
|
||||
--loader llama2_hf \
|
||||
--saver megatron \
|
||||
--target-tensor-parallel-size 1 \
|
||||
--target-pipeline-parallel-size 1 \
|
||||
--params-dtype bf16 \
|
||||
--add-qkv-bias \
|
||||
--load-dir ./model_from_hf/qwen15-0.5b-hf/ \
|
||||
--save-dir ./model_weights/qwen15-0.5b-hf-v0.1-tp1-pp1/ \
|
||||
--tokenizer-model ./model_from_hf/qwen15-0.5b-hf/tokenizer.json
|
||||
```
|
||||
|
||||
4.2 Any Megatron weights with parallel slicing strategy --> Any Megatron weights with parallel slicing strategy
|
||||
***(This scenario is generally used to convert the trained megatron model back to the HuggingFace format)***
|
||||
|
||||
```shell
|
||||
# Modify the ascend-toolkit path
|
||||
source /usr/local/Ascend/ascend-toolkit/set_env.sh
|
||||
python tools/checkpoint/convert_ckpt.py \
|
||||
--model-type GPT \
|
||||
--loader megatron \
|
||||
--saver megatron \
|
||||
--save-model-type save_huggingface_llama \
|
||||
--load-dir ./ckpt/ \
|
||||
--target-tensor-parallel-size 1 \
|
||||
--target-pipeline-parallel-size 1 \
|
||||
--add-qkv-bias \
|
||||
--save-dir ./model_from_hf/qwen15-0.5b-hf/ # <-- Fill in the original HF model path here, new weights will be saved in ./model_from_hf/qwen15-0.5b-hf/mg2hg/
|
||||
```
|
||||
|
||||
5. pre-training
|
||||
|
||||
5.1 prepare dataset
|
||||
|
||||
Download the Qwen1.5-0.5B datasets from [here](https://huggingface.co/datasets/tatsu-lab/alpaca/resolve/main/data/train-00000-of-00001-a09b74b3ef9c3b56.parquet)
|
||||
|
||||
```shell
|
||||
# download datasets
|
||||
cd ./dataset
|
||||
wget https://huggingface.co/datasets/tatsu-lab/alpaca/resolve/main/data/train-00000-of-00001-a09b74b3ef9c3b56.parquet
|
||||
cd ..
|
||||
# process datasets
|
||||
mkdir ./dataset/qwen15-0.5b-hf/
|
||||
python ./tools/preprocess_data.py \
|
||||
--input ./dataset/train-00000-of-00001-a09b74b3ef9c3b56.parquet \
|
||||
--tokenizer-name-or-path ./model_from_hf/qwen15-0.5b-hf/ \
|
||||
--output-prefix ./dataset/qwen15-0.5b-hf/alpaca \
|
||||
--workers 4 \
|
||||
--log-interval 1000 \
|
||||
--tokenizer-type PretrainedFromHF
|
||||
```
|
||||
|
||||
5.2 pre-training
|
||||
|
||||
```shell
|
||||
# modify the script according to your own ascend-toolkit path
|
||||
source /usr/local/Ascend/ascend-toolkit/set_env.sh
|
||||
|
||||
# modify config according to your own actual situation
|
||||
CKPT_SAVE_DIR="./ckpt/qwen15-0.5b-hf/"
|
||||
TOKENIZER_PATH="./model_from_hf/qwen15-0.5b-hf" #tokenizer path
|
||||
DATA_PATH="./dataset/qwen15-0.5b-hf/alpaca_text_document" #processed dataset
|
||||
CKPT_LOAD_DIR="./model_weights/qwen15-0.5b-hf-v0.1-tp1-pp1"
|
||||
```
|
||||
Config Qwen1.5-0.5B pre-training script: examples/qwen15/pretrain_qwen15_0point5b_ptd.sh
|
||||
|
||||
6. fine-tuning
|
||||
|
||||
6.1 Prepare fine-tuning dataset
|
||||
Download the Qwen1.5-0.5B datasets from [here](https://huggingface.co/datasets/tatsu-lab/alpaca/resolve/main/data/train-00000-of-00001-a09b74b3ef9c3b56.parquet)
|
||||
|
||||
```shell
|
||||
# download datasets
|
||||
mkdir finetune_dataset
|
||||
cd ./finetune_dataset
|
||||
wget https://huggingface.co/datasets/tatsu-lab/alpaca/resolve/main/data/train-00000-of-00001-a09b74b3ef9c3b56.parquet
|
||||
cd ..
|
||||
|
||||
# process datasets
|
||||
mkdir ./finetune_dataset/qwen15-0.5b-hf/
|
||||
python ./tools/preprocess_data.py \
|
||||
--input ./dataset/train-00000-of-00001-a09b74b3ef9c3b56.parquet \
|
||||
--tokenizer-name-or-path ./model_from_hf/qwen15-0.5b-hf/ \
|
||||
--output-prefix ./finetune_dataset/qwen15-0.5b-hf/alpaca \
|
||||
--workers 4 \
|
||||
--log-interval 1000 \
|
||||
--tokenizer-type PretrainedFromHF \
|
||||
--handler-name GeneralInstructionHandler \
|
||||
--append-eod
|
||||
```
|
||||
|
||||
6.2 Full Parameters Fine-Tuning
|
||||
The configuration script for full parameters fine-tuning is basically the same as that for pretrain_qwen15_0point5b_ptd.sh.*The difference is that the dataset and the training parameter is-instruction-dataset are added.*
|
||||
|
||||
Add the fine-tuning parameter '--finetune' so that fine-tuning starts from the first step.
|
||||
|
||||
```bash
|
||||
DATA_PATH="./finetune_dataset/qwen15-0.5b-hf/alpaca"
|
||||
TOKENIZER_PATH="./model_from_hf/qwen15-0.5b-hf/"
|
||||
CKPT_PATH="./model_weights/qwen15-0.5b-hf-v0.1-tp1-pp1/"
|
||||
--load ${CKPT_PATH} \
|
||||
--finetune \
|
||||
--is-instruction-dataset \
|
||||
--tokenizer-type PretrainedFromHF \
|
||||
--tokenizer-name-or-path ${TOKENIZER_PATH} \
|
||||
--tokenizer-not-use-fast \
|
||||
```
|
||||
### Performance-0.5B
|
||||
|
||||
#### Machine performance
|
||||
|
||||
The performance of Qwen1.5-0.5B in **Ascend NPU** and **Reference**:
|
||||
|
||||
| Device | Model | total Iterations | throughput rate (tokens/s/p) |
|
||||
| :--: |:------------:|:----------------:|:----------------------------:|
|
||||
| NPUs | Qwen1.5-0.5B | 2000 | 22834 |
|
||||
| Reference | Qwen1.5-0.5B | 2000 | 25306 |
|
||||
|
||||
## Inference-0.5B
|
||||
|
||||
Config Qwen1.5-0.5B inference script: examples/qwen15/generate_qwen1.5_0point5b_ptd.sh
|
||||
|
||||
```bash
|
||||
# modify the script according to your own ascend-toolkit path
|
||||
source /usr/local/Ascend/ascend-toolkit/set_env.sh
|
||||
|
||||
# modify script model path and tokenizer path
|
||||
CHECKPOINT="./model_weights/qwen15-0.5b-hf-v0.1-tp1-pp1"
|
||||
TOKENIZER_PATH="./model_from_hf/qwen15-0.5b-hf/"
|
||||
```
|
||||
|
||||
Config Qwen1.5-0.5B inference script
|
||||
|
||||
```bash
|
||||
bash examples/qwen15/generate_qwen15_0point5b_ptd.sh
|
||||
```
|
||||
|
||||
Some inference samples are as follows:
|
||||
|
||||
|
||||
## Evaluation-0.5B
|
||||
|
||||
We use MMLU benchmark to evaluate our model. Benchmark Download [here](https://huggingface.co/datasets/cais/mmlu).
|
||||
Config Qwen1.5-0.5B evaluation script: examples/qwen15/evaluate_qwen15_0point5b_ptd.sh
|
||||
|
||||
```bash
|
||||
source /usr/local/Ascend/ascend-toolkit/set_env.sh
|
||||
|
||||
# modify script model path and tokenizer path
|
||||
TOKENIZER_PATH="./model_from_hf/qwen15-0.5b-hf/" #tokenizer path
|
||||
CHECKPOINT="./model_weights/qwen15-0.5b-hf-v0.1-tp1-pp1" #model path
|
||||
# configure task and data path
|
||||
DATA_PATH="./mmlu/data/test/"
|
||||
TASK="mmlu"
|
||||
```
|
||||
|
||||
Launch evaluation script:
|
||||
|
||||
```bash
|
||||
bash examples/qwen15/evaluate_qwen15_0point5b_ptd.sh
|
||||
```
|
||||
|
||||
Evaluation results
|
||||
|
||||
| dataset | subject_num | question_num | reference_acc | NPU acc |
|
||||
| :----: | :------: | :------: |:-------------:|:-------:|
|
||||
| MMLU | 57 | 14042 | 0.318 | 0.318 |
|
||||
# Qwen1.5-1.8B
|
||||
|
||||
## Training-1.8B
|
||||
|
|
@ -216,7 +467,7 @@ Here's a hardware summary of pre-training Qwen-1.8B:
|
|||
```
|
||||
|
||||
6.2 Full Parameters Fine-Tuning
|
||||
The configuration script for full parameters fine-tuning is basically the same as that for pretrain_qwen15_4b_ptd.sh.*The difference is that the dataset and the training parameter is-instruction-dataset are added.*
|
||||
The configuration script for full parameters fine-tuning is basically the same as that for pretrain_qwen15_1point8b_ptd.sh.*The difference is that the dataset and the training parameter is-instruction-dataset are added.*
|
||||
|
||||
Add the fine-tuning parameter '--finetune' so that fine-tuning starts from the first step.
|
||||
|
||||
|
|
@ -244,7 +495,7 @@ The performance of Qwen1.5-1.8B in **Ascend NPU** and **Reference**:
|
|||
|
||||
## Inference-1.8B
|
||||
|
||||
Config Qwen1.5-1.8B inference script: examples/qwen1.5/generate_qwen1.5_1point8b_ptd.sh
|
||||
Config Qwen1.5-1.8B inference script: examples/qwen1.5/generate_qwen15_1point8b_ptd.sh
|
||||
|
||||
```bash
|
||||
# modify the script according to your own ascend-toolkit path
|
||||
|
|
@ -262,7 +513,7 @@ bash examples/qwen1.5/generate_qwen1.5_1point8b_ptd.sh
|
|||
```
|
||||
|
||||
Some inference samples are as follows:
|
||||
|
||||
|
||||
|
||||
## Evaluation-1.8B
|
||||
|
||||
|
|
@ -288,9 +539,9 @@ bash examples/qwen15/evaluate_qwen15_1point8b_ptd.sh
|
|||
|
||||
Evaluation results
|
||||
|
||||
| dataset | subject_num | question_num | reference_acc | NPU acc |
|
||||
| :----: | :------: | :------: |:------------------------------------------------------:|:-------:|
|
||||
| MMLU | 57 | 14042 | [0.468](https://qwenlm.github.io/zh/blog/qwen1.5) | 0。462 |
|
||||
| dataset | subject_num | question_num | reference_acc | NPU acc |
|
||||
| :----: | :------: | :------: |:-------------------------------------------------:|:-------:|
|
||||
| MMLU | 57 | 14042 | [0.468](https://qwenlm.github.io/zh/blog/qwen1.5) | 0.462 |
|
||||
# Qwen1.5-4B
|
||||
|
||||
## Training-4B
|
||||
|
|
@ -492,7 +743,7 @@ The performance of Qwen1.5-4B in **Ascend NPU** and **Reference**:
|
|||
|
||||
## Inference-4B
|
||||
|
||||
Config Qwen1.5-4B inference script: examples/qwen1.5/generate_qwen1.5_4b_ptd.sh
|
||||
Config Qwen1.5-4B inference script: examples/qwen15/generate_qwen15_4b_ptd.sh
|
||||
|
||||
```bash
|
||||
# modify the script according to your own ascend-toolkit path
|
||||
|
|
@ -506,7 +757,7 @@ TOKENIZER_PATH="./model_from_hf/qwen15-4b-hf/"
|
|||
Config Qwen1.5-4B inference script
|
||||
|
||||
```bash
|
||||
bash examples/qwen1.5/generate_qwen1.5_4b_ptd.sh
|
||||
bash examples/qwen15/generate_qwen15_4b_ptd.sh
|
||||
```
|
||||
|
||||
Some inference samples are as follows:
|
||||
|
|
@ -770,7 +1021,7 @@ bash examples/qwen15/generate_qwen15_7b_ptd.sh
|
|||
```
|
||||
|
||||
Some inference samples are as follows:
|
||||
|
||||
|
||||
|
||||
## Evaluation
|
||||
|
||||
|
|
@ -1321,7 +1572,7 @@ bash ./examples/qwen15/generate_qwen15_32b_lora_chat_ptd.sh
|
|||
```
|
||||
|
||||
Some inference samples of Qwen1.5-32B after lora are as follows：
|
||||
|
||||
|
||||
|
||||
## Evaluation
|
||||
|
||||
|
|
@ -1566,7 +1817,7 @@ The performance of Qwen1.5-72B in **Ascend NPU** and **Reference**:
|
|||
|
||||
## Inference
|
||||
|
||||
Config qwen1.5-72b inference script:examples/qwen15/generate_qwen15_72b_ptd.sh
|
||||
Config qwen1.5-72b inference script:examples/qwen15/generate_qwen15_72b_chat_ptd.sh
|
||||
|
||||
```bash
|
||||
# ascend-toolkit path
|
||||
|
|
@ -1599,7 +1850,7 @@ bash ./examples/qwen15/generate_qwen15_72b_lora_chat_ptd.sh
|
|||
```
|
||||
|
||||
Some inference samples of Qwen1.5-72B after lora are as follows：
|
||||
|
||||
|
||||
|
||||
## Evaluation
|
||||
|
||||
|
|
|
|||
|
|
@ -0,0 +1,67 @@
|
|||
#!/bin/bash
|
||||
|
||||
# The number of parameters is not aligned
|
||||
export LD_LIBRARY_PATH=/usr/local/lib:/usr/local/lib:/root/miniconda3/lib:$LD_LIBRARY_PATH
|
||||
export HCCL_CONNECT_TIMEOUT=1200
|
||||
export COMBINED_ENABLE=1
|
||||
export CUDA_DEVICE_MAX_CONNECTIONS=1
|
||||
|
||||
# Change for multinode config
|
||||
MASTER_ADDR=localhost
|
||||
MASTER_PORT=6001
|
||||
NNODES=1
|
||||
NODE_RANK=0
|
||||
NPUS_PER_NODE=1
|
||||
WORLD_SIZE=$(($NPUS_PER_NODE*$NNODES))
|
||||
|
||||
# please fill these path configurations
|
||||
CHECKPOINT="your model ckpt path"
|
||||
TOKENIZER_PATH="your tokenizer path"
|
||||
DATA_PATH="your data path"
|
||||
TASK="mmlu"
|
||||
|
||||
TP=1
|
||||
PP=1
|
||||
|
||||
DISTRIBUTED_ARGS="
|
||||
--nproc_per_node $NPUS_PER_NODE \
|
||||
--nnodes $NNODES \
|
||||
--node_rank $NODE_RANK \
|
||||
--master_addr $MASTER_ADDR \
|
||||
--master_port $MASTER_PORT
|
||||
"
|
||||
|
||||
# Different task needs different max_new_tokens value, please follow the instruction in readme.
|
||||
torchrun $DISTRIBUTED_ARGS evaluation.py \
|
||||
--task-data-path $DATA_PATH \
|
||||
--task ${TASK} \
|
||||
--tensor-model-parallel-size ${TP} \
|
||||
--pipeline-model-parallel-size ${PP} \
|
||||
--seq-length 8192 \
|
||||
--max-new-tokens 1 \
|
||||
--max-position-embeddings 8192 \
|
||||
--num-layers 24 \
|
||||
--hidden-size 1024 \
|
||||
--ffn-hidden-size 2816 \
|
||||
--num-attention-heads 16 \
|
||||
--disable-bias-linear \
|
||||
--swiglu \
|
||||
--position-embedding-type rope \
|
||||
--load $CHECKPOINT \
|
||||
--normalization RMSNorm \
|
||||
--tokenizer-type PretrainedFromHF \
|
||||
--tokenizer-name-or-path ${TOKENIZER_PATH} \
|
||||
--tokenizer-not-use-fast \
|
||||
--micro-batch-size 1 \
|
||||
--exit-on-missing-checkpoint \
|
||||
--no-load-rng \
|
||||
--no-load-optim \
|
||||
--add-qkv-bias \
|
||||
--make-vocab-size-divisible-by 1 \
|
||||
--padded-vocab-size 151936 \
|
||||
--rotary-base 1000000 \
|
||||
--no-gradient-accumulation-fusion \
|
||||
--attention-softmax-in-fp32 \
|
||||
--seed 42 \
|
||||
--no-chat-template \
|
||||
| tee logs/eval_qwen15_0point5b_${TASK}.log
|
||||
|
|
@ -0,0 +1,63 @@
|
|||
#!/bin/bash
|
||||
|
||||
# The number of parameters is not aligned
|
||||
export LD_LIBRARY_PATH=/usr/local/lib:/usr/local/lib:/root/miniconda3/lib:$LD_LIBRARY_PATH
|
||||
export HCCL_CONNECT_TIMEOUT=1200
|
||||
export COMBINED_ENABLE=1
|
||||
export CUDA_DEVICE_MAX_CONNECTIONS=1
|
||||
|
||||
# please fill these path configurations
|
||||
CHECKPOINT="your model ckpt path"
|
||||
TOKENIZER_PATH="your tokenizer path"
|
||||
|
||||
# Change for multinode config
|
||||
MASTER_ADDR=localhost
|
||||
MASTER_PORT=6010
|
||||
NNODES=1
|
||||
NODE_RANK=0
|
||||
NPUS_PER_NODE=8
|
||||
WORLD_SIZE=$(($NPUS_PER_NODE*$NNODES))
|
||||
|
||||
TP=8
|
||||
PP=1
|
||||
|
||||
DISTRIBUTED_ARGS="
|
||||
--nproc_per_node $NPUS_PER_NODE \
|
||||
--nnodes $NNODES \
|
||||
--node_rank $NODE_RANK \
|
||||
--master_addr $MASTER_ADDR \
|
||||
--master_port $MASTER_PORT
|
||||
"
|
||||
|
||||
torchrun $DISTRIBUTED_ARGS inference.py \
|
||||
--tensor-model-parallel-size ${TP} \
|
||||
--pipeline-model-parallel-size ${PP} \
|
||||
--num-layers 24 \
|
||||
--hidden-size 1024 \
|
||||
--num-attention-heads 16 \
|
||||
--ffn-hidden-size 2816 \
|
||||
--max-position-embeddings 32768 \
|
||||
--seq-length 8192 \
|
||||
--make-vocab-size-divisible-by 1 \
|
||||
--padded-vocab-size 151936 \
|
||||
--rotary-base 1000000 \
|
||||
--micro-batch-size 1 \
|
||||
--swiglu \
|
||||
--disable-bias-linear \
|
||||
--tokenizer-type PretrainedFromHF \
|
||||
--tokenizer-name-or-path ${TOKENIZER_PATH} \
|
||||
--load ${CHECKPOINT} \
|
||||
--normalization RMSNorm \
|
||||
--position-embedding-type rope \
|
||||
--norm-epsilon 1e-6 \
|
||||
--hidden-dropout 0 \
|
||||
--attention-dropout 0 \
|
||||
--tokenizer-not-use-fast \
|
||||
--add-qkv-bias \
|
||||
--max-new-tokens 256 \
|
||||
--no-gradient-accumulation-fusion \
|
||||
--exit-on-missing-checkpoint \
|
||||
--attention-softmax-in-fp32 \
|
||||
--seed 42 \
|
||||
--bf16 \
|
||||
| tee logs/generate_qwen15_0point5b.log
|
||||
|
|
@ -0,0 +1,101 @@
|
|||
#!/bin/bash
|
||||
|
||||
export CUDA_DEVICE_MAX_CONNECTIONS=1
|
||||
export NPU_ASD_ENABLE=0
|
||||
|
||||
NPUS_PER_NODE=8
|
||||
MASTER_ADDR=localhost
|
||||
MASTER_PORT=6000
|
||||
NNODES=1
|
||||
NODE_RANK=0
|
||||
WORLD_SIZE=$(($NPUS_PER_NODE*$NNODES))
|
||||
|
||||
# please fill these path configurations
|
||||
CKPT_LOAD_DIR="your model ckpt path"
|
||||
CKPT_SAVE_DIR="your model save ckpt path"
|
||||
DATA_PATH="your data path"
|
||||
TOKENIZER_PATH="your tokenizer path"
|
||||
|
||||
TP=1
|
||||
PP=1
|
||||
|
||||
DISTRIBUTED_ARGS="
|
||||
--nproc_per_node $NPUS_PER_NODE \
|
||||
--nnodes $NNODES \
|
||||
--node_rank $NODE_RANK \
|
||||
--master_addr $MASTER_ADDR \
|
||||
--master_port $MASTER_PORT
|
||||
"
|
||||
|
||||
GPT_ARGS="
|
||||
--tensor-model-parallel-size ${TP} \
|
||||
--pipeline-model-parallel-size ${PP} \
|
||||
--sequence-parallel \
|
||||
--num-layers 24 \
|
||||
--hidden-size 1024 \
|
||||
--ffn-hidden-size 2816 \
|
||||
--num-attention-heads 16 \
|
||||
--load ${CKPT_LOAD_DIR} \
|
||||
--tokenizer-type PretrainedFromHF \
|
||||
--tokenizer-name-or-path ${TOKENIZER_PATH} \
|
||||
--seq-length 8192 \
|
||||
--max-position-embeddings 8192 \
|
||||
--micro-batch-size 1 \
|
||||
--global-batch-size 64 \
|
||||
--make-vocab-size-divisible-by 1 \
|
||||
--padded-vocab-size 151936 \
|
||||
--rotary-base 1000000 \
|
||||
--lr 1.25e-6 \
|
||||
--train-iters 5000 \
|
||||
--lr-decay-style cosine \
|
||||
--disable-bias-linear \
|
||||
--attention-dropout 0.0 \
|
||||
--init-method-std 0.01 \
|
||||
--hidden-dropout 0.0 \
|
||||
--position-embedding-type rope \
|
||||
--normalization RMSNorm \
|
||||
--use-fused-rmsnorm \
|
||||
--swiglu \
|
||||
--use-flash-attn \
|
||||
--use-fused-rotary-pos-emb \
|
||||
--use-rotary-position-embeddings \
|
||||
--use-fused-swiglu \
|
||||
--use-mc2 \
|
||||
--no-masked-softmax-fusion \
|
||||
--attention-softmax-in-fp32 \
|
||||
--min-lr 1.25e-7 \
|
||||
--weight-decay 1e-1 \
|
||||
--lr-warmup-fraction 0.01 \
|
||||
--clip-grad 1.0 \
|
||||
--adam-beta1 0.9 \
|
||||
--adam-beta2 0.95 \
|
||||
--add-qkv-bias \
|
||||
--initial-loss-scale 4096 \
|
||||
--use-distributed-optimizer \
|
||||
--no-gradient-accumulation-fusion \
|
||||
--no-load-optim \
|
||||
--no-load-rng \
|
||||
--seed 42 \
|
||||
--bf16
|
||||
"
|
||||
|
||||
DATA_ARGS="
|
||||
--data-path $DATA_PATH \
|
||||
--split 100,0,0
|
||||
"
|
||||
|
||||
OUTPUT_ARGS="
|
||||
--log-interval 1 \
|
||||
--save-interval 10000 \
|
||||
--eval-interval 10000 \
|
||||
--eval-iters 0 \
|
||||
"
|
||||
|
||||
torchrun $DISTRIBUTED_ARGS pretrain_gpt.py \
|
||||
$GPT_ARGS \
|
||||
$DATA_ARGS \
|
||||
$OUTPUT_ARGS \
|
||||
--distributed-backend nccl \
|
||||
--jit-compile \
|
||||
--save ${CKPT_SAVE_DIR} \
|
||||
| tee logs/train_qwen15_0point5b.log
|
||||
Binary file not shown.
|
After Width: | Height: | Size: 20 KiB |
Loading…
Reference in New Issue