modify README

This commit is contained in:
anzhengqi 2021-01-30 14:30:17 +08:00
parent d429ea3f7d
commit 1c35880d55
3 changed files with 18 additions and 12 deletions

View File

@ -171,7 +171,7 @@ optional arguments:
--device_target device where the code will be implemented: "Ascend", default is "Ascend"
--data_dir DATA_DIR Train dataset directory.
--per_batch_size PER_BATCH_SIZE
Batch size for Training. Default: 32.
Batch size for Training. Default: 8.
--pretrained_backbone PRETRAINED_BACKBONE
The ckpt file of CspDarkNet53. Default: "".
--resume_yolov4 RESUME_YOLOV4
@ -437,7 +437,7 @@ YOLOv4 on 20K images(The annotation and data format must be the same as coco tes
# [Description of Random Situation](#contents)
In dataset.py, we set the seed inside ```create_dataset``` function.
In var_init.py, we set seed for weight initilization
In var_init.py, we set seed for weight initialization
# [ModelZoo Homepage](#contents)

View File

@ -52,7 +52,7 @@ parser.add_argument('--device_target', type=str, default='Ascend',
# dataset related
parser.add_argument('--data_dir', type=str, help='Train dataset directory.')
parser.add_argument('--per_batch_size', default=8, type=int, help='Batch size for Training. Default: 32.')
parser.add_argument('--per_batch_size', default=8, type=int, help='Batch size for Training. Default: 8.')
# network related
parser.add_argument('--pretrained_backbone', default='', type=str,
@ -125,7 +125,7 @@ if args.is_distributed:
args.rank = get_rank()
args.group_size = get_group_size()
# select for master rank save ckpt or all rank save, compatiable for model parallel
# select for master rank save ckpt or all rank save, compatible for model parallel
args.rank_save_ckpt_flag = 0
if args.is_save_on_master:
if args.rank == 0:

View File

@ -8,16 +8,15 @@
- [Generate MindRecord](#generate-mindrecord)
- [Create MindDataset By MindRecord](#create-minddataset-by-mindrecord)
<!-- /TOC -->
## What does the example do
This example is used to read data from aclImdb dataset and generate mindrecord. It just transfers the aclImdb dataset to mindrecord without any data preprocessing. You can modify the example or follow the example to implement your own example.
This example is used to read data from aclImdb dataset and generate mindrecord. It just transfers the aclImdb dataset to mindrecord without any data preprocessing. You can modify the example or refer to the example to implement your own example.
1. run.sh: generate MindRecord entry script.
1. run.sh: generate MindRecord entry script.
- gen_mindrecord.py : read the aclImdb data and transfer it to mindrecord.
2. run_read.py: create MindDataset by MindRecord entry script.
2. run_read.py: create MindDataset by MindRecord entry script.
- create_dataset.py: use MindDataset to read MindRecord to generate dataset.
## How to use the example to generate MindRecord
@ -30,6 +29,7 @@ Download aclImdb dataset, transfer it to mindrecord, use MindDataset to read min
> [aclImdb dataset download address](http://ai.stanford.edu/~amaas/data/sentiment/) **-> Large Movie Review Dataset v1.0**
2. Unzip the training data to dir example/nlp_to_mindrecord/aclImdb/data.
```
tar -zxvf aclImdb_v1.tar.gz -C {your-mindspore}/example/nlp_to_mindrecord/aclImdb/data/
```
@ -37,11 +37,13 @@ Download aclImdb dataset, transfer it to mindrecord, use MindDataset to read min
### Generate MindRecord
1. Run the run.sh script.
```bash
bash run.sh
```
2. Output like this:
```
...
>> begin generate mindrecord by train data
@ -77,6 +79,7 @@ Download aclImdb dataset, transfer it to mindrecord, use MindDataset to read min
```
3. Generate mindrecord files
```
$ ls output/
aclImdb_test.mindrecord aclImdb_test.mindrecord.db aclImdb_train.mindrecord aclImdb_train.mindrecord.db README.md
@ -85,12 +88,14 @@ Download aclImdb dataset, transfer it to mindrecord, use MindDataset to read min
### Create MindDataset By MindRecord
1. Run the run_read.sh script.
```bash
bash run_read.sh
```
2. Output like this:
> Caution: field "review" which is string type output is displayed in type uint8.
```
...
example 2056: {'label': array(1, dtype=int32), 'score': array(4, dtype=int32), 'id': array(5871, dtype=int32), 'review': array([ 70, 111, 114, ..., 111, 110, 46], dtype=uint8)}
@ -98,7 +103,8 @@ Download aclImdb dataset, transfer it to mindrecord, use MindDataset to read min
example 2058: {'label': array(1, dtype=int32), 'score': array(4, dtype=int32), 'id': array(1357, dtype=int32), 'review': array([ 42, 109, 97, ..., 58, 32, 67], dtype=uint8)}
...
```
- id : the id "3219" is from review docs like **3219**_10.txt.
- label : indicates whether the review is positive or negative, positive: 0, negative: 1.
- score : the score "10" is from review docs like 3219_**10**.txt.
- review : the review is from the review dos's content.
- id : the id "3219" is from review docs like **3219**_10.txt.
- label : indicates whether the review is positive or negative, positive: 0, negative: 1.
- score : the score "10" is from review docs like 3219_**10**.txt.
- review : the review is from the review dos's content.