diff --git a/README.md b/README.md
index 1ffca5d..c0e602b 100644
--- a/README.md
+++ b/README.md
@@ -317,7 +317,16 @@ ModelLink旨在为华为 [昇腾芯片](https://open.codehub.huawei.com/OpenBaiz
| 【昇腾贡献】 |
- | Qwen1.5 |
+ Qwen1.5 |
+ 0.5B |
+ pretrain |
+ generate |
+ -- |
+ -- |
+ -- |
+ eval |
+ 【社区贡献】 |
+
| 1.8B |
pretrain |
generate |
@@ -675,7 +684,13 @@ ModelLink旨在为华为 [昇腾芯片](https://open.codehub.huawei.com/OpenBaiz
- | Qwen1.5 |
+ Qwen1.5 |
+ 0.5B |
+ 1x8 |
+ BF16 |
+ 22834 |
+ 25306 |
+
| 1.8B |
1x8 |
BF16 |
diff --git a/README_en.md b/README_en.md
index 9b7e05a..64eb5ac 100644
--- a/README_en.md
+++ b/README_en.md
@@ -316,7 +316,16 @@ Current ModelLink supports pre-training and fine-tuning for the following models
【Ascend】 |
- | Qwen1.5 |
+ Qwen1.5 |
+ 0.5B |
+ pretrain |
+ generate |
+ -- |
+ -- |
+ -- |
+ eval |
+ 【Community】 |
+
| 1.8B |
pretrain |
generate |
@@ -671,7 +680,13 @@ For the supported models listed above, we provide training scripts and readme in
345 |
- | Qwen1.5 |
+ Qwen1.5 |
+ 0.5B |
+ 1x8 |
+ BF16 |
+ 22834 |
+ 25306 |
+
| 1.8B |
1x8 |
BF16 |
diff --git a/examples/qwen15/README.md b/examples/qwen15/README.md
index 68422fa..4b032b9 100644
--- a/examples/qwen15/README.md
+++ b/examples/qwen15/README.md
@@ -6,6 +6,13 @@
# 目录
+- [Qwen1.5-0.5B](#Qwen1.5-0.5b)
+ - [训练-0.5B](#训练-0.5b)
+ - [脚本-0.5B](#脚本-0.5b)
+ - [性能-0.5B](#性能-0.5b)
+ - [吞吐-0.5B](#吞吐-0.5b)
+ - [推理-0.5B](#推理-0.5b)
+ - [评估-0.5B](#评估-0.5b)
- [Qwen1.5-1.8B](#Qwen1.5-1.8b)
- [训练-1.8B](#训练-1.8b)
- [脚本-1.8B](#脚本-1.8b)
@@ -48,6 +55,252 @@
- [吞吐-72B](#吞吐-72b)
- [推理-72B](#推理-72b)
- [评估-72B](#评估-72b)
+# Qwen1.5-0.5B
+
+## 训练-0.5B
+Qwen1.5-0.5B 训练的硬件配置:
+
+| 硬件 | 配置 |
+|:---:|:---------------:|
+| NPU | 8 x Ascend NPUs |
+### 脚本-0.5B
+
+1. 克隆仓库到本地服务器
+
+ ```shell
+ git clone https://gitee.com/ascend/ModelLink.git
+ git clone https://github.com/NVIDIA/Megatron-LM.git
+ cd Megatron-LM
+ git checkout core_r0.6.0
+ cp -r megatron ../ModelLink/
+ cd ..
+ cd ModelLink
+ mkdir logs
+ mkdir model_from_hf
+ mkdir dataset
+ mkdir ckpt
+ ```
+2. 搭建环境
+
+ ```bash
+ # python3.8
+ conda create -n test python=3.8
+ conda activate test
+
+ # 安装 torch 和 torch_npu
+ pip install torch-2.2.0-cp38-cp38m-manylinux2014_aarch64.whl
+ pip install torch_npu-2.2.0*-cp38-cp38m-linux_aarch64.whl
+ pip install apex-0.1_ascend*-cp38-cp38m-linux_aarch64.whl
+
+ # 修改 ascend-toolkit 路径
+ source /usr/local/Ascend/ascend-toolkit/set_env.sh
+
+ # 安装加速库
+ git clone https://gitee.com/ascend/MindSpeed.git
+ cd MindSpeed
+ git checkout 2b0edd2
+ pip install -r requirements.txt
+ pip3 install -e .
+ cd ..
+
+ # 安装其余依赖库
+ pip install -r requirements.txt
+ ```
+3. 下载 Qwen1.5-0.5B 的 [预训练权重和词表](https://huggingface.co/Qwen/Qwen1.5-0.5B/tree/main)
+
+ ```shell
+ #!/bin/bash
+ mkdir ./model_from_hf/qwen15-0.5b-hf/
+ cd ./model_from_hf/qwen15-0.5b-hf/
+ wget https://huggingface.co/Qwen/Qwen1.5-0.5B/resolve/main/config.json
+ wget https://huggingface.co/Qwen/Qwen1.5-0.5B/resolve/main/generation_config.json
+ wget https://huggingface.co/Qwen/Qwen1.5-0.5B/resolve/main/merges.txt
+ wget https://huggingface.co/Qwen/Qwen1.5-0.5B/resolve/main/model.safetensors
+ wget https://huggingface.co/Qwen/Qwen1.5-0.5B/resolve/main/tokenizer.json
+ wget https://huggingface.co/Qwen/Qwen1.5-0.5B/resolve/main/tokenizer_config.json
+ wget https://huggingface.co/Qwen/Qwen1.5-0.5B/resolve/main/vocab.json
+ cd ../../
+ ```
+4. 权重转换
+
+ 4.1 将权重从 huggingface 格式转化为 magatron 格式
+ ***（该场景一般用于使能开源的HuggingFace模型在Megatron上进行训练）***
+
+ ```bash
+ # 修改 ascend-toolkit 路径
+ source /usr/local/Ascend/ascend-toolkit/set_env.sh
+
+ # 权重格式转换
+ python tools/checkpoint/convert_ckpt.py \
+ --model-type GPT \
+ --loader llama2_hf \
+ --saver megatron \
+ --target-tensor-parallel-size 1 \
+ --target-pipeline-parallel-size 1 \
+ --add-qkv-bias \
+ --load-dir ./model_from_hf/qwen15-0.5b-hf/ \
+ --save-dir ./model_weights/qwen15-0.5b-hf-v0.1-tp1-pp1/ \
+ --tokenizer-model ./model_from_hf/qwen15-0.5b-hf/tokenizer.json
+ ```
+
+ 4.2 任意并行切分策略的 Megatron 权重 格式转化为 HuggingFace权重
+ ***（该场景一般用于将训练好的megatron模型重新转回HuggingFace格式）***
+
+ ```shell
+ # 请按照您的真实环境修改 set_env.sh 路径
+ source /usr/local/Ascend/ascend-toolkit/set_env.sh
+ python tools/checkpoint/convert_ckpt.py \
+ --model-type GPT \
+ --loader megatron \
+ --saver megatron \
+ --save-model-type save_huggingface_llama \
+ --load-dir ./ckpt/ \
+ --target-tensor-parallel-size 1 \
+ --target-pipeline-parallel-size 1 \
+ --add-qkv-bias \
+ --save-dir ./model_from_hf/qwen15-0.5b-hf/ # <-- 需要填入原始HF模型路径，新权重会存于./model_from_hf/qwen15-0.5b-hf/mg2hg/
+ ```
+
+ 权重转换适用于预训练、微调、推理和评估，根据任务不同调整参数 `target-tensor-parallel-size`和 `target-pipeline-parallel-size`。
+5. 预训练
+
+ 5.1 准备数据集
+
+ 下载 Qwen1.5-0.5B [数据集](https://huggingface.co/datasets/tatsu-lab/alpaca/resolve/main/data/train-00000-of-00001-a09b74b3ef9c3b56.parquet)
+
+ ```shell
+ # 下载数据
+ cd ./dataset
+ wget https://huggingface.co/datasets/tatsu-lab/alpaca/resolve/main/data/train-00000-of-00001-a09b74b3ef9c3b56.parquet
+ cd ..
+ # 处理数据
+ mkdir ./dataset/qwen15-0.5b-hf/
+ python ./tools/preprocess_data.py \
+ --input ./dataset/train-00000-of-00001-a09b74b3ef9c3b56.parquet \
+ --tokenizer-name-or-path ./model_from_hf/qwen15-0.5b-hf/ \
+ --output-prefix ./dataset/qwen15-0.5b-hf/alpaca \
+ --workers 4 \
+ --log-interval 1000 \
+ --tokenizer-type PretrainedFromHF
+ ```
+
+ 5.2 预训练
+
+ ```shell
+ # 设置 ascend-toolkit 路径
+ source /usr/local/Ascend/ascend-toolkit/set_env.sh
+
+ # 根据实际情况配置词表、数据集、模型参数保存路径
+ CKPT_SAVE_DIR="./ckpt/qwen15-0.5b-hf/"
+ TOKENIZER_PATH="./model_from_hf/qwen15-0.5b-hf" #词表路径
+ DATA_PATH="./dataset/qwen15-0.5b-hf/alpaca_text_document" #数据集路径
+ CKPT_LOAD_DIR="./model_weights/qwen15-0.5b-hf-v0.1-tp1-pp1"
+ ```
+ 启动 Qwen1.5-0.5B 预训练脚本: examples/qwen15/pretrain_qwen15_0point5b_ptd.sh
+
+6. 微调
+
+ 6.1 准备微调数据集
+ 下载微调数据集 [这里](https://huggingface.co/datasets/tatsu-lab/alpaca/resolve/main/data/train-00000-of-00001-a09b74b3ef9c3b56.parquet)
+
+ ```shell
+ # 下载数据集
+ mkdir finetune_dataset
+ cd ./finetune_dataset
+ wget https://huggingface.co/datasets/tatsu-lab/alpaca/resolve/main/data/train-00000-of-00001-a09b74b3ef9c3b56.parquet
+ cd ..
+
+ # 处理微调数据集
+ mkdir ./finetune_dataset/qwen15-0.5b-hf/
+ python ./tools/preprocess_data.py \
+ --input ./dataset/train-00000-of-00001-a09b74b3ef9c3b56.parquet \
+ --tokenizer-name-or-path ./model_from_hf/qwen15-0.5b-hf/ \
+ --output-prefix ./finetune_dataset/qwen15-0.5b-hf/alpaca \
+ --workers 4 \
+ --log-interval 1000 \
+ --tokenizer-type PretrainedFromHF \
+ --handler-name GeneralInstructionHandler \
+ --append-eod
+ ```
+
+ 6.2 全参微调
+ 全参微调的配置脚本基本和预训练脚本一致. *区别是数据集，以及增加训练参数--is-instruction-dataset*
+
+ 增加微调参数--finetune，参数如下：
+
+ ```bash
+ DATA_PATH="./finetune_dataset/qwen15-0.5b-hf/alpaca"
+ TOKENIZER_PATH="./model_from_hf/qwen15-0.5b-hf/"
+ CKPT_PATH="./model_weights/qwen15-0.5b-hf-v0.1-tp1-pp1/"
+ --load ${CKPT_PATH} \
+ --finetune \
+ --is-instruction-dataset \
+ --tokenizer-type PretrainedFromHF \
+ --tokenizer-name-or-path ${TOKENIZER_PATH} \
+ --tokenizer-not-use-fast \
+ ```
+### 性能-0.5B
+
+#### 吞吐-0.5B
+
+Qwen1.5-0.5B 在 **昇腾芯片** 和 **参考芯片** 上的性能对比：
+
+| 设备 | 模型 | 迭代数 | tokens吞吐 (tokens/s/p) |
+| :--: |:------------:|:----:|:---------------------:|
+| NPUs | Qwen1.5-0.5B | 2000 | 22834 |
+| 参考 | Qwen1.5-0.5B | 2000 | 25306 |
+
+## 推理-0.5B
+
+配置Qwen1.5-0.5B 推理脚本: examples/qwen15/generate_qwen1.5_0point5b_ptd.sh
+
+```bash
+# 根据您自己的 ascend-toolkit 路径，执行set_env.sh
+source /usr/local/Ascend/ascend-toolkit/set_env.sh
+
+# 修改模型权重路径和词表路径
+CHECKPOINT="./model_weights/qwen15-0.5b-hf-v0.1-tp1-pp1"
+TOKENIZER_PATH="./model_from_hf/qwen15-0.5b-hf/"
+```
+
+启动Qwen1.5-0.5B 推理脚本
+
+```bash
+bash examples/qwen15/generate_qwen15_0point5b_ptd.sh
+```
+
+推理示例如下：
+
+
+
+## 评估-0.5B
+
+使用 MMLU数据集评估模型. 数据集下载路径 [这里](https://huggingface.co/datasets/cais/mmlu).
+配置Qwen1.5-0.5B 评估脚本: examples/qwen15/evaluate_qwen1.5_0point5b_ptd.sh
+
+```bash
+# ascend-toolkit 路径
+source /usr/local/Ascend/ascend-toolkit/set_env.sh
+
+# 修改模型参数路径和词表路径
+TOKENIZER_PATH="./model_from_hf/qwen15-0.5b-hf/" #词表路径
+CHECKPOINT="./model_weights/qwen15-0.5b-hf-v0.1-tp1-pp1" #模型路径
+# 配置任务和数据集路径
+DATA_PATH="./mmlu/data/test/"
+TASK="mmlu"
+```
+
+启动评估
+
+```bash
+bash examples/qwen15/evaluate_qwen15_0point5b_ptd.sh
+```
+
+评估结果如下
+
+| 数据集 | 总学科数 | 总问题数 | 参考准确率 | NPU准确率 |
+| :----: | :------: | :------: |:-----:|:------:|
+| MMLU | 57 | 14042 | 0.318 | 0.318 |
# Qwen1.5-1.8B
## 训练-1.8B
@@ -260,7 +513,7 @@ TOKENIZER_PATH="./model_from_hf/qwen15-1.8b-hf/"
启动Qwen1.5-1.8B 推理脚本
```bash
-bash examples/qwen15/generate_qwen15_1.8b_ptd.sh
+bash examples/qwen15/generate_qwen15_1point8b_ptd.sh
```
推理示例如下：
@@ -498,7 +751,7 @@ Qwen1.5-4B 在 **昇腾芯片** 和 **参考芯片** 上的性能对比：
## 推理-4B
-配置Qwen1.5-4B 推理脚本: examples/qwen1.5/generate_qwen1.5_4b_ptd.sh
+配置Qwen1.5-4B 推理脚本: examples/qwen15/generate_qwen15_4b_ptd.sh
```bash
# 根据您自己的 ascend-toolkit 路径，执行set_env.sh
@@ -522,7 +775,7 @@ bash examples/qwen15/generate_qwen15_4b_ptd.sh
## 评估-4B
使用 MMLU数据集评估模型. 数据集下载路径 [这里](https://huggingface.co/datasets/cais/mmlu).
-配置Qwen1.5-4B 评估脚本: examples/qwen1.5/evaluate_qwen1.5_4b_ptd.sh
+配置Qwen1.5-4B 评估脚本: examples/qwen15/evaluate_qwen15_4b_ptd.sh
```bash
# ascend-toolkit 路径
@@ -809,9 +1062,9 @@ TASK="mmlu" # ceval任务配置为 "ceval"
bash examples/qwen15/evaluate_qwen15_7b_ptd.sh
```
-| 数据集 | 总学科数 | 总问题数 | 参考准确率 | NPU准确率 |
-|:-----:|:----:|:-----:|:-------------------------------------------------------:|:------:|
-| MMLU | 57 | 14042 | [61.0](https://qwenlm.github.io/zh/blog/qwen1.5) | 60.3 |
+| 数据集 | 总学科数 | 总问题数 | 参考准确率 | NPU准确率 |
+|:-----:|:----:|:-----:|:----------------------------------------------------:|:------:|
+| MMLU | 57 | 14042 | [61.0](https://qwenlm.github.io/zh/blog/qwen1.5) | 0.603 |
# Qwen1.5-14B
@@ -1225,8 +1478,8 @@ bash examples/qwen15/evaluate_qwen15_14b_ptd.sh
# 根据实际情况配置词表、数据集、模型参数保存路径
CKPT_SAVE_DIR="./ckpt/Qwen1.5-32B/"
TOKENIZER_PATH="./model_from_hf/Qwen1.5-32B/" #词表路径
- DATA_PATH="./dataset/Qwen1.5-32B-hf/alpaca_text_document" #数据集路径
- CKPT_LOAD_DIR="./model_weights/Qwen1.5-32B-v0.1-tp8-pp4-vpp2/"
+ DATA_PATH="./dataset/qwen1.5-32B-hf/alpaca_text_document" #数据集路径
+ CKPT_LOAD_DIR="./model_weights/qwen1.5-32B-v0.1-tp8-pp4-vpp2/"
```
启动 Qwen1.5-32B 预训练脚本: examples/qwen15/pretrain_qwen15_32b_ptd.sh
@@ -1250,7 +1503,7 @@ bash examples/qwen15/evaluate_qwen15_14b_ptd.sh
# 处理微调数据集
mkdir ./finetune_dataset/qwen-1.5-32b-hf/
python ./tools/preprocess_data.py \
- --input ./dataset/ train-00000-of-00001-a09b74b3ef9c3b56.parquet \
+ --input ./finetune_dataset/ train-00000-of-00001-a09b74b3ef9c3b56.parquet \
--tokenizer-name-or-path ./model_from_hf/Qwen1.5-32B/ \
--output-prefix ./finetune_dataset/qwen-1.5-32b-hf/alpaca \
--workers 4 \
@@ -1340,7 +1593,7 @@ bash ./examples/qwen15/generate_qwen15_32b_lora_chat_ptd.sh
```
lora微调后的推理效果如下：
-
+
## 评估-32B
@@ -1589,7 +1842,7 @@ Qwen1.5-72B 在 **昇腾芯片** 和 **参考芯片** 上的性能对比：
## 推理-72B
-配置 qwen1.5-72b 推理脚本：examples/qwen15/generate_qwen15_72b_ptd.sh
+配置 qwen1.5-72b 推理脚本：examples/qwen15/generate_qwen15_72b_chat_ptd.sh
```bash
# ascend-toolkit 路径
@@ -1623,7 +1876,7 @@ bash ./examples/qwen15/generate_qwen15_72b_lora_chat_ptd.sh
```
lora微调后的推理效果如下：
-
+
## 评估-72B
diff --git a/examples/qwen15/README_en.md b/examples/qwen15/README_en.md
index deb80b5..d17824e 100644
--- a/examples/qwen15/README_en.md
+++ b/examples/qwen15/README_en.md
@@ -5,6 +5,13 @@
# Contents
+- [Qwen1.5-0.5B](#Qwen1.5-0.5b)
+ - [Training-0.5B](#training-0.5b)
+ - [Script-0.5B](#script-0.5b)
+ - [Performance-0.5B](#performance-0.5b)
+ - [Machine performance 0.5B](#machine-performance-0.5b)
+ - [Inference-0.5B](#inference-0.5b)
+ - [Evaluation-0.5B](#evaluation-0.5b)
- [Qwen1.5-1.8B](#Qwen1.5-1.8b)
- [Training-1.8B](#training-1.8b)
- [Script-1.8B](#script-1.8b)
@@ -47,6 +54,250 @@
- [Machine performance 72B](#machine-performance-72b)
- [Inference-72B](#Inference-72b)
- [Evaluation-72B](#Evaluation-72b)
+# Qwen1.5-0.5B
+
+## Training-0.5B
+Here's a hardware summary of pre-training Qwen-0.5B:
+
+| Hardware | Value |
+| :------: | :---------------------------------------------: |
+| NPU | 8 x Ascend NPUs |
+### Script
+
+1. Clone the repository to your local server
+
+ ```shell
+ git clone https://gitee.com/ascend/ModelLink.git
+ git clone https://github.com/NVIDIA/Megatron-LM.git
+ cd Megatron-LM
+ git checkout core_r0.6.0
+ cp -r megatron ../ModelLink/
+ cd ..
+ cd ModelLink
+ mkdir logs
+ mkdir model_from_hf
+ mkdir dataset
+ mkdir ckpt
+ ```
+2. Build environment
+
+ ```bash
+ # python3.8
+ conda create -n test python=3.8
+ conda activate test
+
+ # install torch 和 torch_npu
+ pip install torch-2.2.0-cp38-cp38m-manylinux2014_aarch64.whl
+ pip install torch_npu-2.2.0*-cp38-cp38m-linux_aarch64.whl
+ pip install apex-0.1_ascend*-cp38-cp38m-linux_aarch64.whl
+
+ # modify ascend-toolkit path
+ source /usr/local/Ascend/ascend-toolkit/set_env.sh
+
+ # install MindSpeed
+ git clone https://gitee.com/ascend/MindSpeed.git
+ cd MindSpeed
+ git checkout 2b0edd2
+ pip install -r requirements.txt
+ pip3 install -e .
+ cd ..
+
+ # install other packages
+ pip install -r requirements.txt
+ ```
+3. Download Qwen1.5-0.5B [pretrained weights and tokenizer](https://huggingface.co/Qwen/Qwen1.5-1.8B/tree/main)
+
+ ```shell
+ #!/bin/bash
+ mkdir ./model_from_hf/qwen15-0.5b-hf/
+ cd ./model_from_hf/qwen15-4b-hf/
+ wget https://huggingface.co/Qwen/Qwen1.5-0.5B/resolve/main/config.json
+ wget https://huggingface.co/Qwen/Qwen1.5-0.5B/resolve/main/generation_config.json
+ wget https://huggingface.co/Qwen/Qwen1.5-0.5B/resolve/main/merges.txt
+ wget https://huggingface.co/Qwen/Qwen1.5-0.5B/resolve/main/model.safetensors
+ wget https://huggingface.co/Qwen/Qwen1.5-0.5B/resolve/main/tokenizer.json
+ wget https://huggingface.co/Qwen/Qwen1.5-0.5B/resolve/main/tokenizer_config.json
+ wget https://huggingface.co/Qwen/Qwen1.5-0.5B/resolve/main/vocab.json
+ cd ../../
+ ```
+4. weight conversion in ptd mode
+
+ 4.1 Convert weights from huggingface format to megatron format
+ ***(This scenario is generally used to train open-source HuggingFace models on Megatron)***
+
+ ```bash
+ # modify the script according to your own ascend-toolkit path
+ source /usr/local/Ascend/ascend-toolkit/set_env.sh
+
+ # convert to ptd weights
+ python tools/checkpoint/convert_ckpt.py \
+ --model-type GPT \
+ --loader llama2_hf \
+ --saver megatron \
+ --target-tensor-parallel-size 1 \
+ --target-pipeline-parallel-size 1 \
+ --params-dtype bf16 \
+ --add-qkv-bias \
+ --load-dir ./model_from_hf/qwen15-0.5b-hf/ \
+ --save-dir ./model_weights/qwen15-0.5b-hf-v0.1-tp1-pp1/ \
+ --tokenizer-model ./model_from_hf/qwen15-0.5b-hf/tokenizer.json
+ ```
+
+ 4.2 Any Megatron weights with parallel slicing strategy --> Any Megatron weights with parallel slicing strategy
+ ***(This scenario is generally used to convert the trained megatron model back to the HuggingFace format)***
+
+ ```shell
+ # Modify the ascend-toolkit path
+ source /usr/local/Ascend/ascend-toolkit/set_env.sh
+ python tools/checkpoint/convert_ckpt.py \
+ --model-type GPT \
+ --loader megatron \
+ --saver megatron \
+ --save-model-type save_huggingface_llama \
+ --load-dir ./ckpt/ \
+ --target-tensor-parallel-size 1 \
+ --target-pipeline-parallel-size 1 \
+ --add-qkv-bias \
+ --save-dir ./model_from_hf/qwen15-0.5b-hf/ # <-- Fill in the original HF model path here, new weights will be saved in ./model_from_hf/qwen15-0.5b-hf/mg2hg/
+ ```
+
+5. pre-training
+
+ 5.1 prepare dataset
+
+ Download the Qwen1.5-0.5B datasets from [here](https://huggingface.co/datasets/tatsu-lab/alpaca/resolve/main/data/train-00000-of-00001-a09b74b3ef9c3b56.parquet)
+
+ ```shell
+ # download datasets
+ cd ./dataset
+ wget https://huggingface.co/datasets/tatsu-lab/alpaca/resolve/main/data/train-00000-of-00001-a09b74b3ef9c3b56.parquet
+ cd ..
+ # process datasets
+ mkdir ./dataset/qwen15-0.5b-hf/
+ python ./tools/preprocess_data.py \
+ --input ./dataset/train-00000-of-00001-a09b74b3ef9c3b56.parquet \
+ --tokenizer-name-or-path ./model_from_hf/qwen15-0.5b-hf/ \
+ --output-prefix ./dataset/qwen15-0.5b-hf/alpaca \
+ --workers 4 \
+ --log-interval 1000 \
+ --tokenizer-type PretrainedFromHF
+ ```
+
+ 5.2 pre-training
+
+ ```shell
+ # modify the script according to your own ascend-toolkit path
+ source /usr/local/Ascend/ascend-toolkit/set_env.sh
+
+ # modify config according to your own actual situation
+ CKPT_SAVE_DIR="./ckpt/qwen15-0.5b-hf/"
+ TOKENIZER_PATH="./model_from_hf/qwen15-0.5b-hf" #tokenizer path
+ DATA_PATH="./dataset/qwen15-0.5b-hf/alpaca_text_document" #processed dataset
+ CKPT_LOAD_DIR="./model_weights/qwen15-0.5b-hf-v0.1-tp1-pp1"
+ ```
+ Config Qwen1.5-0.5B pre-training script: examples/qwen15/pretrain_qwen15_0point5b_ptd.sh
+
+6. fine-tuning
+
+ 6.1 Prepare fine-tuning dataset
+ Download the Qwen1.5-0.5B datasets from [here](https://huggingface.co/datasets/tatsu-lab/alpaca/resolve/main/data/train-00000-of-00001-a09b74b3ef9c3b56.parquet)
+
+ ```shell
+ # download datasets
+ mkdir finetune_dataset
+ cd ./finetune_dataset
+ wget https://huggingface.co/datasets/tatsu-lab/alpaca/resolve/main/data/train-00000-of-00001-a09b74b3ef9c3b56.parquet
+ cd ..
+
+ # process datasets
+ mkdir ./finetune_dataset/qwen15-0.5b-hf/
+ python ./tools/preprocess_data.py \
+ --input ./dataset/train-00000-of-00001-a09b74b3ef9c3b56.parquet \
+ --tokenizer-name-or-path ./model_from_hf/qwen15-0.5b-hf/ \
+ --output-prefix ./finetune_dataset/qwen15-0.5b-hf/alpaca \
+ --workers 4 \
+ --log-interval 1000 \
+ --tokenizer-type PretrainedFromHF \
+ --handler-name GeneralInstructionHandler \
+ --append-eod
+ ```
+
+ 6.2 Full Parameters Fine-Tuning
+ The configuration script for full parameters fine-tuning is basically the same as that for pretrain_qwen15_0point5b_ptd.sh.*The difference is that the dataset and the training parameter is-instruction-dataset are added.*
+
+ Add the fine-tuning parameter '--finetune' so that fine-tuning starts from the first step.
+
+ ```bash
+ DATA_PATH="./finetune_dataset/qwen15-0.5b-hf/alpaca"
+ TOKENIZER_PATH="./model_from_hf/qwen15-0.5b-hf/"
+ CKPT_PATH="./model_weights/qwen15-0.5b-hf-v0.1-tp1-pp1/"
+ --load ${CKPT_PATH} \
+ --finetune \
+ --is-instruction-dataset \
+ --tokenizer-type PretrainedFromHF \
+ --tokenizer-name-or-path ${TOKENIZER_PATH} \
+ --tokenizer-not-use-fast \
+ ```
+### Performance-0.5B
+
+#### Machine performance
+
+The performance of Qwen1.5-0.5B in **Ascend NPU** and **Reference**:
+
+| Device | Model | total Iterations | throughput rate (tokens/s/p) |
+| :--: |:------------:|:----------------:|:----------------------------:|
+| NPUs | Qwen1.5-0.5B | 2000 | 22834 |
+| Reference | Qwen1.5-0.5B | 2000 | 25306 |
+
+## Inference-0.5B
+
+Config Qwen1.5-0.5B inference script: examples/qwen15/generate_qwen1.5_0point5b_ptd.sh
+
+```bash
+# modify the script according to your own ascend-toolkit path
+source /usr/local/Ascend/ascend-toolkit/set_env.sh
+
+# modify script model path and tokenizer path
+CHECKPOINT="./model_weights/qwen15-0.5b-hf-v0.1-tp1-pp1"
+TOKENIZER_PATH="./model_from_hf/qwen15-0.5b-hf/"
+```
+
+Config Qwen1.5-0.5B inference script
+
+```bash
+bash examples/qwen15/generate_qwen15_0point5b_ptd.sh
+```
+
+Some inference samples are as follows:
+
+
+## Evaluation-0.5B
+
+We use MMLU benchmark to evaluate our model. Benchmark Download [here](https://huggingface.co/datasets/cais/mmlu).
+Config Qwen1.5-0.5B evaluation script: examples/qwen15/evaluate_qwen15_0point5b_ptd.sh
+
+```bash
+source /usr/local/Ascend/ascend-toolkit/set_env.sh
+
+# modify script model path and tokenizer path
+TOKENIZER_PATH="./model_from_hf/qwen15-0.5b-hf/" #tokenizer path
+CHECKPOINT="./model_weights/qwen15-0.5b-hf-v0.1-tp1-pp1" #model path
+# configure task and data path
+DATA_PATH="./mmlu/data/test/"
+TASK="mmlu"
+```
+
+Launch evaluation script:
+
+```bash
+bash examples/qwen15/evaluate_qwen15_0point5b_ptd.sh
+```
+
+Evaluation results
+
+| dataset | subject_num | question_num | reference_acc | NPU acc |
+| :----: | :------: | :------: |:-------------:|:-------:|
+| MMLU | 57 | 14042 | 0.318 | 0.318 |
# Qwen1.5-1.8B
## Training-1.8B
@@ -216,7 +467,7 @@ Here's a hardware summary of pre-training Qwen-1.8B:
```
6.2 Full Parameters Fine-Tuning
- The configuration script for full parameters fine-tuning is basically the same as that for pretrain_qwen15_4b_ptd.sh.*The difference is that the dataset and the training parameter is-instruction-dataset are added.*
+ The configuration script for full parameters fine-tuning is basically the same as that for pretrain_qwen15_1point8b_ptd.sh.*The difference is that the dataset and the training parameter is-instruction-dataset are added.*
Add the fine-tuning parameter '--finetune' so that fine-tuning starts from the first step.
@@ -244,7 +495,7 @@ The performance of Qwen1.5-1.8B in **Ascend NPU** and **Reference**:
## Inference-1.8B
-Config Qwen1.5-1.8B inference script: examples/qwen1.5/generate_qwen1.5_1point8b_ptd.sh
+Config Qwen1.5-1.8B inference script: examples/qwen1.5/generate_qwen15_1point8b_ptd.sh
```bash
# modify the script according to your own ascend-toolkit path
@@ -262,7 +513,7 @@ bash examples/qwen1.5/generate_qwen1.5_1point8b_ptd.sh
```
Some inference samples are as follows:
-
+
## Evaluation-1.8B
@@ -288,9 +539,9 @@ bash examples/qwen15/evaluate_qwen15_1point8b_ptd.sh
Evaluation results
-| dataset | subject_num | question_num | reference_acc | NPU acc |
-| :----: | :------: | :------: |:------------------------------------------------------:|:-------:|
-| MMLU | 57 | 14042 | [0.468](https://qwenlm.github.io/zh/blog/qwen1.5) | 0。462 |
+| dataset | subject_num | question_num | reference_acc | NPU acc |
+| :----: | :------: | :------: |:-------------------------------------------------:|:-------:|
+| MMLU | 57 | 14042 | [0.468](https://qwenlm.github.io/zh/blog/qwen1.5) | 0.462 |
# Qwen1.5-4B
## Training-4B
@@ -492,7 +743,7 @@ The performance of Qwen1.5-4B in **Ascend NPU** and **Reference**:
## Inference-4B
-Config Qwen1.5-4B inference script: examples/qwen1.5/generate_qwen1.5_4b_ptd.sh
+Config Qwen1.5-4B inference script: examples/qwen15/generate_qwen15_4b_ptd.sh
```bash
# modify the script according to your own ascend-toolkit path
@@ -506,7 +757,7 @@ TOKENIZER_PATH="./model_from_hf/qwen15-4b-hf/"
Config Qwen1.5-4B inference script
```bash
-bash examples/qwen1.5/generate_qwen1.5_4b_ptd.sh
+bash examples/qwen15/generate_qwen15_4b_ptd.sh
```
Some inference samples are as follows:
@@ -770,7 +1021,7 @@ bash examples/qwen15/generate_qwen15_7b_ptd.sh
```
Some inference samples are as follows:
-
+
## Evaluation
@@ -1321,7 +1572,7 @@ bash ./examples/qwen15/generate_qwen15_32b_lora_chat_ptd.sh
```
Some inference samples of Qwen1.5-32B after lora are as follows：
-
+
## Evaluation
@@ -1566,7 +1817,7 @@ The performance of Qwen1.5-72B in **Ascend NPU** and **Reference**:
## Inference
-Config qwen1.5-72b inference script:examples/qwen15/generate_qwen15_72b_ptd.sh
+Config qwen1.5-72b inference script:examples/qwen15/generate_qwen15_72b_chat_ptd.sh
```bash
# ascend-toolkit path
@@ -1599,7 +1850,7 @@ bash ./examples/qwen15/generate_qwen15_72b_lora_chat_ptd.sh
```
Some inference samples of Qwen1.5-72B after lora are as follows：
-
+
## Evaluation
diff --git a/examples/qwen15/evaluate_qwen15_0point5b_ptd.sh b/examples/qwen15/evaluate_qwen15_0point5b_ptd.sh
new file mode 100644
index 0000000..81dc166
--- /dev/null
+++ b/examples/qwen15/evaluate_qwen15_0point5b_ptd.sh
@@ -0,0 +1,67 @@
+#!/bin/bash
+
+# The number of parameters is not aligned
+export LD_LIBRARY_PATH=/usr/local/lib:/usr/local/lib:/root/miniconda3/lib:$LD_LIBRARY_PATH
+export HCCL_CONNECT_TIMEOUT=1200
+export COMBINED_ENABLE=1
+export CUDA_DEVICE_MAX_CONNECTIONS=1
+
+# Change for multinode config
+MASTER_ADDR=localhost
+MASTER_PORT=6001
+NNODES=1
+NODE_RANK=0
+NPUS_PER_NODE=1
+WORLD_SIZE=$(($NPUS_PER_NODE*$NNODES))
+
+# please fill these path configurations
+CHECKPOINT="your model ckpt path"
+TOKENIZER_PATH="your tokenizer path"
+DATA_PATH="your data path"
+TASK="mmlu"
+
+TP=1
+PP=1
+
+DISTRIBUTED_ARGS="
+ --nproc_per_node $NPUS_PER_NODE \
+ --nnodes $NNODES \
+ --node_rank $NODE_RANK \
+ --master_addr $MASTER_ADDR \
+ --master_port $MASTER_PORT
+"
+
+# Different task needs different max_new_tokens value, please follow the instruction in readme.
+torchrun $DISTRIBUTED_ARGS evaluation.py \
+ --task-data-path $DATA_PATH \
+ --task ${TASK} \
+ --tensor-model-parallel-size ${TP} \
+ --pipeline-model-parallel-size ${PP} \
+ --seq-length 8192 \
+ --max-new-tokens 1 \
+ --max-position-embeddings 8192 \
+ --num-layers 24 \
+ --hidden-size 1024 \
+ --ffn-hidden-size 2816 \
+ --num-attention-heads 16 \
+ --disable-bias-linear \
+ --swiglu \
+ --position-embedding-type rope \
+ --load $CHECKPOINT \
+ --normalization RMSNorm \
+ --tokenizer-type PretrainedFromHF \
+ --tokenizer-name-or-path ${TOKENIZER_PATH} \
+ --tokenizer-not-use-fast \
+ --micro-batch-size 1 \
+ --exit-on-missing-checkpoint \
+ --no-load-rng \
+ --no-load-optim \
+ --add-qkv-bias \
+ --make-vocab-size-divisible-by 1 \
+ --padded-vocab-size 151936 \
+ --rotary-base 1000000 \
+ --no-gradient-accumulation-fusion \
+ --attention-softmax-in-fp32 \
+ --seed 42 \
+ --no-chat-template \
+ | tee logs/eval_qwen15_0point5b_${TASK}.log
\ No newline at end of file
diff --git a/examples/qwen15/generate_qwen15_0point5b_ptd.sh b/examples/qwen15/generate_qwen15_0point5b_ptd.sh
new file mode 100644
index 0000000..391dc52
--- /dev/null
+++ b/examples/qwen15/generate_qwen15_0point5b_ptd.sh
@@ -0,0 +1,63 @@
+#!/bin/bash
+
+# The number of parameters is not aligned
+export LD_LIBRARY_PATH=/usr/local/lib:/usr/local/lib:/root/miniconda3/lib:$LD_LIBRARY_PATH
+export HCCL_CONNECT_TIMEOUT=1200
+export COMBINED_ENABLE=1
+export CUDA_DEVICE_MAX_CONNECTIONS=1
+
+# please fill these path configurations
+CHECKPOINT="your model ckpt path"
+TOKENIZER_PATH="your tokenizer path"
+
+# Change for multinode config
+MASTER_ADDR=localhost
+MASTER_PORT=6010
+NNODES=1
+NODE_RANK=0
+NPUS_PER_NODE=8
+WORLD_SIZE=$(($NPUS_PER_NODE*$NNODES))
+
+TP=8
+PP=1
+
+DISTRIBUTED_ARGS="
+ --nproc_per_node $NPUS_PER_NODE \
+ --nnodes $NNODES \
+ --node_rank $NODE_RANK \
+ --master_addr $MASTER_ADDR \
+ --master_port $MASTER_PORT
+"
+
+torchrun $DISTRIBUTED_ARGS inference.py \
+ --tensor-model-parallel-size ${TP} \
+ --pipeline-model-parallel-size ${PP} \
+ --num-layers 24 \
+ --hidden-size 1024 \
+ --num-attention-heads 16 \
+ --ffn-hidden-size 2816 \
+ --max-position-embeddings 32768 \
+ --seq-length 8192 \
+ --make-vocab-size-divisible-by 1 \
+ --padded-vocab-size 151936 \
+ --rotary-base 1000000 \
+ --micro-batch-size 1 \
+ --swiglu \
+ --disable-bias-linear \
+ --tokenizer-type PretrainedFromHF \
+ --tokenizer-name-or-path ${TOKENIZER_PATH} \
+ --load ${CHECKPOINT} \
+ --normalization RMSNorm \
+ --position-embedding-type rope \
+ --norm-epsilon 1e-6 \
+ --hidden-dropout 0 \
+ --attention-dropout 0 \
+ --tokenizer-not-use-fast \
+ --add-qkv-bias \
+ --max-new-tokens 256 \
+ --no-gradient-accumulation-fusion \
+ --exit-on-missing-checkpoint \
+ --attention-softmax-in-fp32 \
+ --seed 42 \
+ --bf16 \
+ | tee logs/generate_qwen15_0point5b.log
\ No newline at end of file
diff --git a/examples/qwen15/pretrain_qwen15_0point5b_ptd.sh b/examples/qwen15/pretrain_qwen15_0point5b_ptd.sh
new file mode 100644
index 0000000..1e6f5ae
--- /dev/null
+++ b/examples/qwen15/pretrain_qwen15_0point5b_ptd.sh
@@ -0,0 +1,101 @@
+#!/bin/bash
+
+export CUDA_DEVICE_MAX_CONNECTIONS=1
+export NPU_ASD_ENABLE=0
+
+NPUS_PER_NODE=8
+MASTER_ADDR=localhost
+MASTER_PORT=6000
+NNODES=1
+NODE_RANK=0
+WORLD_SIZE=$(($NPUS_PER_NODE*$NNODES))
+
+# please fill these path configurations
+CKPT_LOAD_DIR="your model ckpt path"
+CKPT_SAVE_DIR="your model save ckpt path"
+DATA_PATH="your data path"
+TOKENIZER_PATH="your tokenizer path"
+
+TP=1
+PP=1
+
+DISTRIBUTED_ARGS="
+ --nproc_per_node $NPUS_PER_NODE \
+ --nnodes $NNODES \
+ --node_rank $NODE_RANK \
+ --master_addr $MASTER_ADDR \
+ --master_port $MASTER_PORT
+"
+
+GPT_ARGS="
+ --tensor-model-parallel-size ${TP} \
+ --pipeline-model-parallel-size ${PP} \
+ --sequence-parallel \
+ --num-layers 24 \
+ --hidden-size 1024 \
+ --ffn-hidden-size 2816 \
+ --num-attention-heads 16 \
+ --load ${CKPT_LOAD_DIR} \
+ --tokenizer-type PretrainedFromHF \
+ --tokenizer-name-or-path ${TOKENIZER_PATH} \
+ --seq-length 8192 \
+ --max-position-embeddings 8192 \
+ --micro-batch-size 1 \
+ --global-batch-size 64 \
+ --make-vocab-size-divisible-by 1 \
+ --padded-vocab-size 151936 \
+ --rotary-base 1000000 \
+ --lr 1.25e-6 \
+ --train-iters 5000 \
+ --lr-decay-style cosine \
+ --disable-bias-linear \
+ --attention-dropout 0.0 \
+ --init-method-std 0.01 \
+ --hidden-dropout 0.0 \
+ --position-embedding-type rope \
+ --normalization RMSNorm \
+ --use-fused-rmsnorm \
+ --swiglu \
+ --use-flash-attn \
+ --use-fused-rotary-pos-emb \
+ --use-rotary-position-embeddings \
+ --use-fused-swiglu \
+ --use-mc2 \
+ --no-masked-softmax-fusion \
+ --attention-softmax-in-fp32 \
+ --min-lr 1.25e-7 \
+ --weight-decay 1e-1 \
+ --lr-warmup-fraction 0.01 \
+ --clip-grad 1.0 \
+ --adam-beta1 0.9 \
+ --adam-beta2 0.95 \
+ --add-qkv-bias \
+ --initial-loss-scale 4096 \
+ --use-distributed-optimizer \
+ --no-gradient-accumulation-fusion \
+ --no-load-optim \
+ --no-load-rng \
+ --seed 42 \
+ --bf16
+"
+
+DATA_ARGS="
+ --data-path $DATA_PATH \
+ --split 100,0,0
+"
+
+OUTPUT_ARGS="
+ --log-interval 1 \
+ --save-interval 10000 \
+ --eval-interval 10000 \
+ --eval-iters 0 \
+"
+
+torchrun $DISTRIBUTED_ARGS pretrain_gpt.py \
+ $GPT_ARGS \
+ $DATA_ARGS \
+ $OUTPUT_ARGS \
+ --distributed-backend nccl \
+ --jit-compile \
+ --save ${CKPT_SAVE_DIR} \
+ | tee logs/train_qwen15_0point5b.log
\ No newline at end of file
diff --git a/sources/images/qwen15/qwen15_0point5b_inference.png b/sources/images/qwen15/qwen15_0point5b_inference.png
new file mode 100644
index 0000000..7f2f347
Binary files /dev/null and b/sources/images/qwen15/qwen15_0point5b_inference.png differ