Merge pull request !46014 from xumengjuan1/code_docs_x76
This commit is contained in:
i-robot 2022-11-25 06:33:24 +00:00 committed by Gitee
commit c82263e31a
No known key found for this signature in database
GPG Key ID: 173E9B9CA92EEF8F
12 changed files with 176 additions and 168 deletions

View File

@ -191,7 +191,7 @@ Contributions of any kind are welcome!
- [STABLE] When using the map operation for dataset objects and the parameters like: num_parallel_workers > 1 and python_multiprocessing=True, the multi-process mechanism is optimized, so that the data channel and child processes are mapped one by one, avoiding excessive file handle occupation, and closing_pool interface is also deleted. - [STABLE] When using the map operation for dataset objects and the parameters like: num_parallel_workers > 1 and python_multiprocessing=True, the multi-process mechanism is optimized, so that the data channel and child processes are mapped one by one, avoiding excessive file handle occupation, and closing_pool interface is also deleted.
- [STABLE] Add a batch of Vision, Text and Audio data augmentation operations. - [STABLE] Add a batch of Vision, Text and Audio data augmentation operations.
- [STABLE] Fix a bug where the flat_map method of the Dataset class does not flatten the result. - [STABLE] Fix a bug where the flat_map method of the Dataset class does not flatten the result.
- [STABLE] Unify import paths of dataset augmentation APIs to provide more easier way to use. Refer to [latest api usages](https://www.mindspore.cn/docs/en/master/api_python/mindspore.dataset.vision.html). - [STABLE] Unify import paths of dataset augmentation APIs to provide more easier way to use. Refer to [latest api usages](https://www.mindspore.cn/docs/en/r1.8/api_python/mindspore.dataset.vision.html).
### API Change ### API Change

View File

@ -191,7 +191,7 @@ AGroupofProbiotocs, anzhengqi, askmiao, baihuawei, baiyangfan, bai-yangfan, bing
- [STABLE] 对于数据集对象使用map操作时同时num_parallel_workers>1并且python_multiprocessing=True时进行了多进程的机制优化使得数据通道与子进程一一映射避免了过多的文件句柄占用同时close_pool这个接口也被删除。 - [STABLE] 对于数据集对象使用map操作时同时num_parallel_workers>1并且python_multiprocessing=True时进行了多进程的机制优化使得数据通道与子进程一一映射避免了过多的文件句柄占用同时close_pool这个接口也被删除。
- [STABLE] 新增一批Vision、Text和Audio类数据增强操作。 - [STABLE] 新增一批Vision、Text和Audio类数据增强操作。
- [STABLE] 修复数据集类的flat_map方法未将结果展平的错误。 - [STABLE] 修复数据集类的flat_map方法未将结果展平的错误。
- [STABLE] 统一数据集增强API的导入路径提供更简单的使用方法请参阅[最新的API用法](https://www.mindspore.cn/docs/zh-CN/master/api_python/mindspore.dataset.vision.html)。 - [STABLE] 统一数据集增强API的导入路径提供更简单的使用方法请参阅[最新的API用法](https://www.mindspore.cn/docs/zh-CN/r1.8/api_python/mindspore.dataset.vision.html)。
### API变更 ### API变更

View File

@ -12,9 +12,9 @@ mindspore.dataset.Dataset.map
最后一个数据增强的输出列的列名由 `output_columns` 指定,如果没有指定 `output_columns` ,输出列名与 `input_columns` 一致。 最后一个数据增强的输出列的列名由 `output_columns` 指定,如果没有指定 `output_columns` ,输出列名与 `input_columns` 一致。
- 如果使用的是 `mindspore` `dataset` 提供的数据增强( - 如果使用的是 `mindspore` `dataset` 提供的数据增强(
`vision类 <https://www.mindspore.cn/docs/zh-CN/master/api_python/mindspore.dataset.vision.html>`_ `vision类 <https://mindspore.cn/docs/zh-CN/master/api_python/mindspore.dataset.transforms.html#视觉>`_
`nlp类 <https://www.mindspore.cn/docs/zh-CN/master/api_python/mindspore.dataset.text.html>`_ `nlp类 <https://mindspore.cn/docs/zh-CN/master/api_python/mindspore.dataset.transforms.html#文本>`_
`audio类 <https://www.mindspore.cn/docs/zh-CN/master/api_python/mindspore.dataset.audio.html>`_ ),请使用如下参数: `audio类 <https://mindspore.cn/docs/zh-CN/master/api_python/mindspore.dataset.transforms.html#音频>`_ ),请使用如下参数:
.. image:: map_parameter_cn.png .. image:: map_parameter_cn.png

View File

@ -38,9 +38,9 @@ mindspore.dataset
- 数据集操作filter/ skip用户通过数据集对象方法 `.shuffle` / `.filter` / `.skip` / `.split` / - 数据集操作filter/ skip用户通过数据集对象方法 `.shuffle` / `.filter` / `.skip` / `.split` /
`.take` / … 来实现数据集的进一步混洗、过滤、跳过、最多获取条数等操作; `.take` / … 来实现数据集的进一步混洗、过滤、跳过、最多获取条数等操作;
- 数据集样本增强操作map用户可以将数据增强操作 - 数据集样本增强操作map用户可以将数据增强操作
`vision类 <https://www.mindspore.cn/docs/zh-CN/master/api_python/mindspore.dataset.vision.html>`_ `vision类 <https://mindspore.cn/docs/zh-CN/master/api_python/mindspore.dataset.transforms.html#视觉>`_
`nlp类 <https://www.mindspore.cn/docs/zh-CN/master/api_python/mindspore.dataset.text.html>`_ `nlp类 <https://mindspore.cn/docs/zh-CN/master/api_python/mindspore.dataset.transforms.html#文本>`_
`audio类 <https://www.mindspore.cn/docs/zh-CN/master/api_python/mindspore.dataset.audio.html>`_ `audio类 <https://mindspore.cn/docs/zh-CN/master/api_python/mindspore.dataset.transforms.html#音频>`_
添加到map操作中执行数据预处理过程中可以定义多个map操作用于执行不同增强操作数据增强操作也可以是 添加到map操作中执行数据预处理过程中可以定义多个map操作用于执行不同增强操作数据增强操作也可以是
用户自定义增强的 `PyFunc` 用户自定义增强的 `PyFunc`
- 批batch用户在样本完成增强后使用 `.batch` 操作将多个样本组织成batch也可以通过batch的参数 `per_batch_map` - 批batch用户在样本完成增强后使用 `.batch` 操作将多个样本组织成batch也可以通过batch的参数 `per_batch_map`

View File

@ -1,8 +1,11 @@
mindspore mindspore
========= =========
数据表达
---------
张量 张量
------ ^^^^^
.. mscnautosummary:: .. mscnautosummary::
:toctree: mindspore :toctree: mindspore
@ -14,7 +17,7 @@ mindspore
mindspore.SparseTensor mindspore.SparseTensor
参数 参数
--------- ^^^^^
.. mscnautosummary:: .. mscnautosummary::
:toctree: mindspore :toctree: mindspore
@ -23,7 +26,7 @@ mindspore
mindspore.ParameterTuple mindspore.ParameterTuple
数据类型 数据类型
-------- ^^^^^^^^^
.. mscnautosummary:: .. mscnautosummary::
:toctree: mindspore :toctree: mindspore
@ -34,36 +37,6 @@ mindspore
mindspore.pytype_to_dtype mindspore.pytype_to_dtype
mindspore.get_py_obj_dtype mindspore.get_py_obj_dtype
随机种子
---------
.. mscnautosummary::
:toctree: mindspore
mindspore.set_seed
mindspore.get_seed
自动微分
----------------
.. mscnautosummary::
:toctree: mindspore
mindspore.grad
mindspore.value_and_grad
mindspore.jacfwd
mindspore.jacrev
mindspore.jvp
mindspore.vjp
自动向量化
----------------
.. mscnautosummary::
:toctree: mindspore
mindspore.vmap
运行环境 运行环境
--------- ---------
@ -83,23 +56,14 @@ mindspore
mindspore.get_algo_parameters mindspore.get_algo_parameters
mindspore.reset_algo_parameters mindspore.reset_algo_parameters
并行 随机种子
------------------- ---------
.. mscnautosummary:: .. mscnautosummary::
:toctree: mindspore :toctree: mindspore
mindspore.shard mindspore.set_seed
mindspore.get_seed
数据处理工具
-------------------
.. mscnautosummary::
:toctree: mindspore
mindspore.DatasetHelper
mindspore.connect_network_with_dataset
mindspore.data_sink
序列化 序列化
------- -------
@ -125,17 +89,37 @@ mindspore
mindspore.transform_checkpoint_by_rank mindspore.transform_checkpoint_by_rank
mindspore.transform_checkpoints mindspore.transform_checkpoints
调试调优 自动微分
---------- ----------------
.. mscnautosummary:: .. mscnautosummary::
:toctree: mindspore :toctree: mindspore
mindspore.Profiler mindspore.grad
mindspore.SummaryCollector mindspore.value_and_grad
mindspore.SummaryLandscape mindspore.jacfwd
mindspore.SummaryRecord mindspore.jacrev
mindspore.set_dump mindspore.jvp
mindspore.vjp
并行优化
---------
自动向量化
^^^^^^^^^^^
.. mscnautosummary::
:toctree: mindspore
mindspore.vmap
并行
^^^^^
.. mscnautosummary::
:toctree: mindspore
mindspore.shard
即时编译 即时编译
-------- --------
@ -148,10 +132,36 @@ mindspore
mindspore.jit_class mindspore.jit_class
mindspore.ms_class mindspore.ms_class
mindspore.ms_function mindspore.ms_function
mindspore.ms_memory_recycle
mindspore.mutable mindspore.mutable
工具
-----
数据处理工具
^^^^^^^^^^^^^
.. mscnautosummary::
:toctree: mindspore
mindspore.DatasetHelper
mindspore.connect_network_with_dataset
mindspore.data_sink
调试调优
^^^^^^^^^
.. mscnautosummary::
:toctree: mindspore
mindspore.Profiler
mindspore.SummaryCollector
mindspore.SummaryLandscape
mindspore.SummaryRecord
mindspore.set_dump
日志 日志
---- ^^^^^
.. mscnautosummary:: .. mscnautosummary::
:toctree: mindspore :toctree: mindspore
@ -161,26 +171,9 @@ mindspore
安装验证 安装验证
-------- ^^^^^^^^^
.. mscnautosummary:: .. mscnautosummary::
:toctree: mindspore :toctree: mindspore
mindspore.run_check mindspore.run_check
内存回收
----------
.. mscnautosummary::
:toctree: mindspore
mindspore.ms_memory_recycle
二阶优化
----------
.. mscnautosummary::
:toctree: mindspore
mindspore.ConvertModelUtils
mindspore.ConvertNetUtils

View File

@ -73,3 +73,12 @@ mindspore.train
mindspore.train.get_metric_fn mindspore.train.get_metric_fn
mindspore.train.names mindspore.train.names
mindspore.train.rearrange_inputs mindspore.train.rearrange_inputs
二阶优化
----------
.. mscnautosummary::
:toctree: mindspore
mindspore.train.ConvertModelUtils
mindspore.train.ConvertNetUtils

View File

@ -1,7 +1,7 @@
mindspore.ConvertModelUtils mindspore.train.ConvertModelUtils
================================ ==================================
.. py:class:: mindspore.ConvertModelUtils .. py:class:: mindspore.train.ConvertModelUtils
该接口用于增加计算图提升二阶算法THOR运行时的性能。 该接口用于增加计算图提升二阶算法THOR运行时的性能。

View File

@ -1,7 +1,7 @@
mindspore.ConvertNetUtils mindspore.train.ConvertNetUtils
================================ ================================
.. py:class:: mindspore.ConvertNetUtils .. py:class:: mindspore.train.ConvertNetUtils
将网络转换为thor层网络用于计算并存储二阶信息矩阵。 将网络转换为thor层网络用于计算并存储二阶信息矩阵。

View File

@ -1,8 +1,11 @@
mindspore mindspore
========= =========
Data Presentation
------------------
Tensor Tensor
------ ^^^^^^^
.. autosummary:: .. autosummary::
:toctree: mindspore :toctree: mindspore
@ -16,7 +19,7 @@ Tensor
mindspore.SparseTensor mindspore.SparseTensor
Parameter Parameter
--------- ^^^^^^^^^^
.. autosummary:: .. autosummary::
:toctree: mindspore :toctree: mindspore
@ -27,7 +30,7 @@ Parameter
mindspore.ParameterTuple mindspore.ParameterTuple
DataType DataType
-------- ^^^^^^^^^
.. class:: mindspore.dtype .. class:: mindspore.dtype
@ -141,42 +144,6 @@ DataType
mindspore.pytype_to_dtype mindspore.pytype_to_dtype
mindspore.get_py_obj_dtype mindspore.get_py_obj_dtype
Seed
----
.. autosummary::
:toctree: mindspore
:nosignatures:
:template: classtemplate.rst
mindspore.set_seed
mindspore.get_seed
Automatic Differentiation
---------------------------------
.. autosummary::
:toctree: mindspore
:nosignatures:
:template: classtemplate.rst
mindspore.grad
mindspore.value_and_grad
mindspore.jacfwd
mindspore.jacrev
mindspore.jvp
mindspore.vjp
Automatic Vectorization
---------------------------------
.. autosummary::
:toctree: mindspore
:nosignatures:
:template: classtemplate.rst
mindspore.vmap
Context Context
-------- --------
@ -198,27 +165,16 @@ Context
mindspore.get_algo_parameters mindspore.get_algo_parameters
mindspore.reset_algo_parameters mindspore.reset_algo_parameters
Parallel Seed
--------------- ----
.. autosummary:: .. autosummary::
:toctree: mindspore :toctree: mindspore
:nosignatures: :nosignatures:
:template: classtemplate.rst :template: classtemplate.rst
mindspore.shard mindspore.set_seed
mindspore.get_seed
Dataset Helper
---------------
.. autosummary::
:toctree: mindspore
:nosignatures:
:template: classtemplate.rst
mindspore.DatasetHelper
mindspore.connect_network_with_dataset
mindspore.data_sink
Serialization Serialization
------------- -------------
@ -246,6 +202,44 @@ Serialization
mindspore.transform_checkpoint_by_rank mindspore.transform_checkpoint_by_rank
mindspore.transform_checkpoints mindspore.transform_checkpoints
Automatic Differentiation
---------------------------------
.. autosummary::
:toctree: mindspore
:nosignatures:
:template: classtemplate.rst
mindspore.grad
mindspore.value_and_grad
mindspore.jacfwd
mindspore.jacrev
mindspore.jvp
mindspore.vjp
Parallel Optimization
-----------------------
Automatic Vectorization
^^^^^^^^^^^^^^^^^^^^^^^^^
.. autosummary::
:toctree: mindspore
:nosignatures:
:template: classtemplate.rst
mindspore.vmap
Parallel
^^^^^^^^^^
.. autosummary::
:toctree: mindspore
:nosignatures:
:template: classtemplate.rst
mindspore.shard
JIT JIT
--- ---
@ -259,31 +253,26 @@ JIT
mindspore.jit_class mindspore.jit_class
mindspore.ms_class mindspore.ms_class
mindspore.ms_function mindspore.ms_function
mindspore.ms_memory_recycle
mindspore.mutable mindspore.mutable
Log Tool
--- -----
Dataset Helper
^^^^^^^^^^^^^^^
.. autosummary:: .. autosummary::
:toctree: mindspore :toctree: mindspore
:nosignatures: :nosignatures:
:template: classtemplate.rst :template: classtemplate.rst
mindspore.get_level mindspore.DatasetHelper
mindspore.get_log_config mindspore.connect_network_with_dataset
mindspore.data_sink
Installation Verification
--------------------------
.. autosummary::
:toctree: mindspore
:nosignatures:
:template: classtemplate.rst
mindspore.run_check
Debugging and Tuning Debugging and Tuning
-------------------------- ^^^^^^^^^^^^^^^^^^^^^
.. autosummary:: .. autosummary::
:toctree: mindspore :toctree: mindspore
@ -294,25 +283,25 @@ Debugging and Tuning
mindspore.SummaryCollector mindspore.SummaryCollector
mindspore.SummaryLandscape mindspore.SummaryLandscape
mindspore.SummaryRecord mindspore.SummaryRecord
mindspore.set_dump mindspore.set_dump
Memory Recycle Log
-------------------------- ^^^^
.. autosummary:: .. autosummary::
:toctree: mindspore :toctree: mindspore
:nosignatures: :nosignatures:
:template: classtemplate.rst :template: classtemplate.rst
mindspore.ms_memory_recycle mindspore.get_level
mindspore.get_log_config
Thor Installation Verification
--------------- ^^^^^^^^^^^^^^^^^^^^^^^^^^
.. autosummary:: .. autosummary::
:toctree: mindspore :toctree: mindspore
:nosignatures: :nosignatures:
:template: classtemplate.rst :template: classtemplate.rst
mindspore.ConvertModelUtils mindspore.run_check
mindspore.ConvertNetUtils

View File

@ -77,3 +77,14 @@ Utils
mindspore.train.get_metric_fn mindspore.train.get_metric_fn
mindspore.train.names mindspore.train.names
mindspore.train.rearrange_inputs mindspore.train.rearrange_inputs
Thor
---------------
.. autosummary::
:toctree: mindspore
:nosignatures:
:template: classtemplate.rst
mindspore.train.ConvertModelUtils
mindspore.train.ConvertNetUtils

View File

@ -55,9 +55,12 @@ The specific steps are as follows:
- Dataset operation: The user uses the dataset object method `.shuffle` / `.filter` / `.skip` / `.split` / - Dataset operation: The user uses the dataset object method `.shuffle` / `.filter` / `.skip` / `.split` /
`.take` / ... to further shuffle, filter, skip, and obtain the maximum number of samples of datasets; `.take` / ... to further shuffle, filter, skip, and obtain the maximum number of samples of datasets;
- Dataset sample transform operation: The user can add data transform operations - Dataset sample transform operation: The user can add data transform operations
( `vision transform <https://www.mindspore.cn/docs/en/master/api_python/mindspore.dataset.vision.html>`_ , ( `vision transform <https://mindspore.cn/docs/en/master/api_python/mindspore.\
`NLP transform <https://www.mindspore.cn/docs/en/master/api_python/mindspore.dataset.text.html>`_ , dataset.transforms.html#module-mindspore.dataset.vision>`_ ,
`audio transform <https://www.mindspore.cn/docs/en/master/api_python/mindspore.dataset.audio.html>`_ ) to the map `NLP transform <https://mindspore.cn/docs/en/master/api_python/mindspore.\
dataset.transforms.html#module-mindspore.dataset.text>`_ ,
`audio transform <https://mindspore.cn/docs/en/master/api_python/mindspore.\
dataset.transforms.html#module-mindspore.dataset.audio>`_ ) to the map
operation to perform transformations. During data preprocessing, multiple map operations can be defined to operation to perform transformations. During data preprocessing, multiple map operations can be defined to
perform different transform operations to different fields. The data transform operation can also be a perform different transform operations to different fields. The data transform operation can also be a
user-defined transform `pyfunc` (Python function); user-defined transform `pyfunc` (Python function);

View File

@ -811,9 +811,12 @@ class Dataset:
`output_columns` , and if not specified, the column name of output column is same as that of `input_columns` . `output_columns` , and if not specified, the column name of output column is same as that of `input_columns` .
- If you use transformations ( - If you use transformations (
`vision transform <https://www.mindspore.cn/docs/en/master/api_python/mindspore.dataset.vision.html>`_ , `vision transform <https://mindspore.cn/docs/en/master/api_python/mindspore.\
`nlp transform <https://www.mindspore.cn/docs/en/master/api_python/mindspore.dataset.text.html>`_ , dataset.transforms.html#module-mindspore.dataset.vision>`_ ,
`audio transform <https://www.mindspore.cn/docs/en/master/api_python/mindspore.dataset.audio.html>`_ ) `nlp transform <https://mindspore.cn/docs/en/master/api_python/mindspore.\
dataset.transforms.html#module-mindspore.dataset.text>`_ ,
`audio transform <https://mindspore.cn/docs/en/master/api_python/mindspore.\
dataset.transforms.html#module-mindspore.dataset.audio>`_ )
provided by mindspore dataset, please use the following parameters: provided by mindspore dataset, please use the following parameters:
.. image:: map_parameter_en.png .. image:: map_parameter_en.png