mindspore-Ecosystem
/
mindspore
21
0
Fork 27
Code Issues Pull Requests 6 Projects Releases Wiki Activity

!45272 fix: md doc format 

Merge pull request !45272 from guozhijian/code_docs_md_format
This commit is contained in:
i-robot 2022-11-08 11:12:01 +00:00 committed by Gitee
parent 21d6e39fa1 42cfac317d
commit 0171878048
No known key found for this signature in database
GPG Key ID: 173E9B9CA92EEF8F
54 changed files with 379 additions and 375 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

4
docs/api/api_python/dataset/dataset_method/batch/mindspore.dataset.Dataset.batch.rst
View File

@ -3,9 +3,9 @@ mindspore.dataset.Dataset.batch
.. py:method:: mindspore.dataset.Dataset.batch(batch_size, drop_remainder=False, num_parallel_workers=None, **kwargs)
将数据集中连续 `batch_size` 条数据合并为一个批处理数据，其中batch成一个Tensor前可选择使用per_batch_map对样本进行处理。
将数据集中连续 `batch_size` 条数据合并为一个批处理数据，其中batch成一个Tensor前可选择使用 `per_batch_map` 对样本进行处理。
`batch` 操作要求每列中的数据具有相同的shape。如果指定了参数 `per_batch_map` ，该参数将作用于批处理后的数据。
`batch` 操作要求每列中的数据具有相同的shape。
执行流程参考下图：

2
docs/api/api_python/dataset/dataset_method/batch/mindspore.dataset.Dataset.padded_batch.rst
View File

@ -3,7 +3,7 @@ mindspore.dataset.Dataset.padded_batch
.. py:method:: mindspore.dataset.Dataset.padded_batch(batch_size, drop_remainder=False, num_parallel_workers=None, pad_info=None)
将数据集中连续 `batch_size` 条数据合并为一个批处理数据，其中batch成一个Tensor前可选择使用pad_info预先将样本补齐。
将数据集中连续 `batch_size` 条数据合并为一个批处理数据，其中batch成一个Tensor前可选择使用 `pad_info` 预先将样本补齐。
`batch` 操作要求每列中的数据具有相同的shape。

6
docs/api/api_python/dataset/dataset_method/operation/mindspore.dataset.Dataset.map.rst
View File

@ -12,9 +12,9 @@ mindspore.dataset.Dataset.map
最后一个数据增强的输出列的列名由 `output_columns` 指定，如果没有指定 `output_columns` ，输出列名与 `input_columns` 一致。
- 如果使用的是 `mindspore` `dataset` 提供的数据增强（
`vision类 <https://www.mindspore.cn/docs/zh-CN/master/api_python/mindspore.dataset.vision.html>`_，
`nlp类 <https://www.mindspore.cn/docs/zh-CN/master/api_python/mindspore.dataset.text.html>`_，
`audio类 <https://www.mindspore.cn/docs/zh-CN/master/api_python/mindspore.dataset.audio.html>`_），请使用如下参数：
`vision类 <https://www.mindspore.cn/docs/zh-CN/master/api_python/mindspore.dataset.vision.html>`_ ，
`nlp类 <https://www.mindspore.cn/docs/zh-CN/master/api_python/mindspore.dataset.text.html>`_ ，
`audio类 <https://www.mindspore.cn/docs/zh-CN/master/api_python/mindspore.dataset.audio.html>`_ ），请使用如下参数：
.. image:: map_parameter_cn.png

2
docs/api/api_python/dataset_audio/mindspore.dataset.audio.AmplitudeToDB.rst
View File

@ -10,7 +10,7 @@ mindspore.dataset.audio.AmplitudeToDB
参数：
- **stype** ( :class:`mindspore.dataset.audio.ScaleType` , 可选) - 输入音频的原始标度，取值可为ScaleType.MAGNITUDE或ScaleType.POWER。默认值：ScaleType.POWER。
- **ref_value** (float, 可选) - 系数参考值。默认值：1.0，用于计算分贝系数 `db_multiplier` ，公式为
:math:`db\_multiplier = Log10(max(ref\_value, amin))`。
:math:`db\_multiplier = Log10(max(ref\_value, amin))` 。
- **amin** (float, 可选) - 波形取值下界，低于该值的波形将会被裁切，取值必须大于0。默认值：1e-10。
- **top_db** (float, 可选) - 最小截止分贝值，取值为非负数。默认值：80.0。

2
docs/api/api_python/dataset_audio/mindspore.dataset.audio.ComputeDeltas.rst
View File

@ -14,7 +14,7 @@ mindspore.dataset.audio.ComputeDeltas
参数：
- **win_length** (int, 可选) - 用于计算delta值的窗口长度，必须不小于3。默认值：5。
- **pad_mode** (:class:`mindspore.dataset.audio.BorderType`, 可选) - 边界填充模式，可为
- **pad_mode** (:class:`mindspore.dataset.audio.BorderType` , 可选) - 边界填充模式，可为
BorderType.CONSTANT，BorderType.EDGE，BorderType.REFLECT或BorderType.SYMMETRIC。
默认值：BorderType.EDGE。

2
docs/api/api_python/dataset_audio/mindspore.dataset.audio.TimeStretch.rst
View File

@ -8,7 +8,7 @@ mindspore.dataset.audio.TimeStretch
.. note:: 待处理音频维度需为(..., freq, time, complex=2)。第0维代表实部，第1维代表虚部。
参数：
- **hop_length** (int, 可选) - STFT窗之间每跳的长度，即连续帧之间的样本数。默认值：None，表示取 `n_freq - 1`。
- **hop_length** (int, 可选) - STFT窗之间每跳的长度，即连续帧之间的样本数。默认值：None，表示取 `n_freq - 1` 。
- **n_freq** (int, 可选) - STFT中的滤波器组数。默认值：201。
- **fixed_rate** (float, 可选) - 频谱在时域加快或减缓的比例。默认值：None，表示保持原始速率。

4
docs/api/api_python/dataset_audio/mindspore.dataset.audio.create_dct.rst
View File

@ -3,7 +3,7 @@ mindspore.dataset.audio.create_dct
.. py:function:: mindspore.dataset.audio.create_dct(n_mfcc, n_mels, norm=NormMode.NONE)
创建一个shape为(`n_mels`, `n_mfcc`)的DCT变换矩阵，并根据范数进行标准化。
创建一个shape为( `n_mels` , `n_mfcc` )的DCT变换矩阵，并根据范数进行标准化。
参数：
- **n_mfcc** (int) - 要保留mfc系数的数量，该值必须大于0。
@ -11,4 +11,4 @@ mindspore.dataset.audio.create_dct
- **norm** (NormMode, 可选) - 标准化模式，可以是NormMode.NONE或NormMode.ORTHO。默认值：NormMode.NONE。
返回：
numpy.ndarray，shape为 ( `n_mels`, `n_mfcc` ) 的DCT转换矩阵。
numpy.ndarray，shape为 ( `n_mels` , `n_mfcc` ) 的DCT转换矩阵。

2
docs/api/api_python/dataset_text/mindspore.dataset.text.BasicTokenizer.rst
View File

@ -10,7 +10,7 @@
参数：
- **lower_case** (bool，可选) - 是否对字符串进行小写转换处理。若为True，会将字符串转换为小写并删除重音字符；若为False，将只对字符串进行规范化处理，其模式由 `normalization_form` 指定。默认值：False。
- **keep_whitespace** (bool，可选) - 是否在分词输出中保留空格。默认值：False。
- **normalization_form** (:class:`mindspore.dataset.text.NormalizeForm`，可选) - `Unicode规范化模式 <http://unicode.org/reports/tr15/>`_，仅当 `lower_case` 为False时生效，取值可为NormalizeForm.NONE、NormalizeForm.NFC、NormalizeForm.NFKC、NormalizeForm.NFD或NormalizeForm.NFKD。默认值：NormalizeForm.NONE。
- **normalization_form** (:class:`mindspore.dataset.text.NormalizeForm`，可选) - `Unicode规范化模式 <http://unicode.org/reports/tr15/>`_ ，仅当 `lower_case` 为False时生效，取值可为NormalizeForm.NONE、NormalizeForm.NFC、NormalizeForm.NFKC、NormalizeForm.NFD或NormalizeForm.NFKD。默认值：NormalizeForm.NONE。
- NormalizeForm.NONE：不进行规范化处理。
- NormalizeForm.NFC：先以标准等价方式分解，再以标准等价方式重组。

2
docs/api/api_python/dataset_text/mindspore.dataset.text.BertTokenizer.rst
View File

@ -14,7 +14,7 @@ mindspore.dataset.text.BertTokenizer
- **unknown_token** (str，可选) - 对未知词汇的分词输出。当设置为空字符串时，直接返回对应未知词汇作为分词输出；否则，返回该字符串作为分词输出。默认值：'[UNK]'。
- **lower_case** (bool，可选) - 是否对字符串进行小写转换处理。若为True，会将字符串转换为小写并删除重音字符；若为False，将只对字符串进行规范化处理，其模式由 `normalization_form` 指定。默认值：False。
- **keep_whitespace** (bool，可选) - 是否在分词输出中保留空格。默认值：False。
- **normalization_form** (:class:`mindspore.dataset.text.NormalizeForm`，可选) - `Unicode规范化模式 <http://unicode.org/reports/tr15/>`_，仅当 `lower_case` 为False时生效，取值可为NormalizeForm.NONE、NormalizeForm.NFC、NormalizeForm.NFKC、NormalizeForm.NFD或NormalizeForm.NFKD。默认值：NormalizeForm.NONE。
- **normalization_form** (:class:`mindspore.dataset.text.NormalizeForm`，可选) - `Unicode规范化模式 <http://unicode.org/reports/tr15/>`_ ，仅当 `lower_case` 为False时生效，取值可为NormalizeForm.NONE、NormalizeForm.NFC、NormalizeForm.NFKC、NormalizeForm.NFD或NormalizeForm.NFKD。默认值：NormalizeForm.NONE。
- NormalizeForm.NONE：不进行规范化处理。
- NormalizeForm.NFC：先以标准等价方式分解，再以标准等价方式重组。

2
docs/api/api_python/dataset_vision/mindspore.dataset.vision.RandomErasing.rst
View File

@ -5,7 +5,7 @@ mindspore.dataset.vision.RandomErasing
按照指定的概率擦除输入numpy.ndarray图像上随机矩形区域内的像素。
请参阅论文 `Random Erasing Data Augmentation <https://arxiv.org/pdf/1708.04896.pdf>`_。
请参阅论文 `Random Erasing Data Augmentation <https://arxiv.org/pdf/1708.04896.pdf>`_ 。
参数：
- **prob** (float，可选) - 执行随机擦除的概率，取值范围：[0.0, 1.0]。默认值：0.5。

2
docs/api/api_python/dataset_vision/mindspore.dataset.vision.Resize.rst
View File

@ -6,7 +6,7 @@ mindspore.dataset.vision.Resize
对输入图像使用给定的 :class:`mindspore.dataset.vision.Inter` 插值方式去调整为给定的尺寸大小。
参数：
- **size** (Union[int, Sequence[int]]) - 图像的输出尺寸大小。若输入整型，将调整图像的较短边长度为 `size`，且保持图像的宽高比不变；若输入是2元素组成的序列，其输入格式需要是 (高度, 宽度) 。
- **size** (Union[int, Sequence[int]]) - 图像的输出尺寸大小。若输入整型，将调整图像的较短边长度为 `size` ，且保持图像的宽高比不变；若输入是2元素组成的序列，其输入格式需要是 (高度, 宽度) 。
- **interpolation** (Inter, 可选) - 图像插值方式。它可以是 [Inter.LINEAR, Inter.NEAREST, Inter.BICUBIC, Inter.PILCUBIC] 中的任何一个。默认值：Inter.LINEAR。
- Inter.BILINEAR，双线性插值。

2
docs/api/api_python/dataset_vision/mindspore.dataset.vision.ResizeWithBBox.rst
View File

@ -6,7 +6,7 @@ mindspore.dataset.vision.ResizeWithBBox
将输入图像调整为给定的尺寸大小并相应地调整边界框的大小。
参数：
- **size** (Union[int, Sequence[int]]) - 图像的输出尺寸大小。若输入整型，将调整图像的较短边长度为 `size`，且保持图像的宽高比不变；若输入是2元素组成的序列，其输入格式需要是 (高度, 宽度) 。
- **size** (Union[int, Sequence[int]]) - 图像的输出尺寸大小。若输入整型，将调整图像的较短边长度为 `size` ，且保持图像的宽高比不变；若输入是2元素组成的序列，其输入格式需要是 (高度, 宽度) 。
- **interpolation** (Inter, 可选) - 图像插值方式。它可以是 [Inter.LINEAR, Inter.NEAREST, Inter.BICUBIC, Inter.PILCUBIC] 中的任何一个。默认值：Inter.LINEAR。
- Inter.LINEAR，双线性插值。

2
docs/api/api_python/dataset_vision/mindspore.dataset.vision.ToTensor.rst
View File

@ -6,7 +6,7 @@ mindspore.dataset.vision.ToTensor
将输入PIL图像或numpy.ndarray图像转换为指定类型的numpy.ndarray图像，图像的像素值范围将从[0, 255]放缩为[0.0, 1.0]，shape将从(H, W, C)调整为(C, H, W)。
参数：
- **output_type** (Union[mindspore.dtype, numpy.dtype]，可选) - 输出图像的数据类型。默认值：:class:`numpy.float32`。
- **output_type** (Union[mindspore.dtype, numpy.dtype]，可选) - 输出图像的数据类型。默认值：:class:`numpy.float32` 。
异常：
- **TypeError** - 当输入图像的类型不为 :class:`PIL.Image.Image` 或 :class:`numpy.ndarray` 。

2
docs/api/api_python/mindrecord/mindspore.mindrecord.Cifar100ToMR.rst
View File

@ -4,7 +4,7 @@
将CIFAR-100数据集转换为MindRecord格式数据集。
.. note::
示例的详细信息，请参见 `转换CIFAR-10数据集 <https://www.mindspore.cn/tutorials/zh-CN/master/advanced/dataset/record.html#转换cifar-10数据集>`_。
示例的详细信息，请参见 `转换CIFAR-10数据集 <https://www.mindspore.cn/tutorials/zh-CN/master/advanced/dataset/record.html#转换cifar-10数据集>`_ 。
参数：
- **source** (str) - 待转换的CIFAR-100数据集文件所在目录的路径。

2
docs/api/api_python/mindrecord/mindspore.mindrecord.Cifar10ToMR.rst
View File

@ -4,7 +4,7 @@
将CIFAR-10数据集转换为MindRecord格式数据集。
.. note::
示例的详细信息，请参见 `转换CIFAR-10数据集 <https://www.mindspore.cn/tutorials/zh-CN/master/advanced/dataset/record.html#转换cifar-10数据集>`_。
示例的详细信息，请参见 `转换CIFAR-10数据集 <https://www.mindspore.cn/tutorials/zh-CN/master/advanced/dataset/record.html#转换cifar-10数据集>`_ 。
参数：
- **source** (str) - 待转换的CIFAR-10数据集文件所在目录的路径。

2
docs/api/api_python/mindrecord/mindspore.mindrecord.CsvToMR.rst
View File

@ -4,7 +4,7 @@
将CSV格式数据集转换为MindRecord格式数据集。
.. note::
示例的详细信息，请参见 `转换CSV数据集 <https://www.mindspore.cn/tutorials/zh-CN/master/advanced/dataset/record.html#转换csv数据集>`_。
示例的详细信息，请参见 `转换CSV数据集 <https://www.mindspore.cn/tutorials/zh-CN/master/advanced/dataset/record.html#转换csv数据集>`_ 。
参数：
- **source** (str) - 待转换的CSV文件路径。

2
docs/api/api_python/mindrecord/mindspore.mindrecord.TFRecordToMR.rst
View File

@ -4,7 +4,7 @@
将TFRecord格式数据集转换为MindRecord格式数据集。
.. note::
示例的详细信息，请参见 `转换TFRecord数据集 <https://www.mindspore.cn/tutorials/zh-CN/master/advanced/dataset/record.html#转换tfrecord数据集>`_。
示例的详细信息，请参见 `转换TFRecord数据集 <https://www.mindspore.cn/tutorials/zh-CN/master/advanced/dataset/record.html#转换tfrecord数据集>`_ 。
参数：
- **source** (str) - 待转换的TFRecord文件路径。

2
docs/api/api_python/mindspore.dataset.audio.rst
View File

@ -5,7 +5,7 @@ mindspore.dataset.audio
数据增强操作可以放入数据处理Pipeline中执行，也可以Eager模式执行：
- Pipeline模式一般用于处理数据集，示例可参考 `数据处理Pipeline介绍 <https://www.mindspore.cn/docs/zh-CN/master/api_python/mindspore.dataset.html#数据处理pipeline介绍>`_。
- Pipeline模式一般用于处理数据集，示例可参考 `数据处理Pipeline介绍 <https://www.mindspore.cn/docs/zh-CN/master/api_python/mindspore.dataset.html#数据处理pipeline介绍>`_ 。
- Eager模式一般用于零散样本，音频预处理举例如下：
.. code-block::

12
docs/api/api_python/mindspore.dataset.rst
View File

@ -8,7 +8,7 @@ mindspore.dataset
大多数数据集可以通过指定参数 `cache` 启用缓存服务，以提升整体数据处理效率。
请注意Windows平台上还不支持缓存服务，因此在Windows上加载和处理数据时，请勿使用。更多介绍和限制，
请参考 `Single-Node Tensor Cache <https://www.mindspore.cn/tutorials/experts/zh-CN/master/dataset/cache.html>`_。
请参考 `Single-Node Tensor Cache <https://www.mindspore.cn/tutorials/experts/zh-CN/master/dataset/cache.html>`_ 。
在API示例中，常用的模块导入方法如下：
@ -38,18 +38,18 @@ mindspore.dataset
- 数据集操作（filter/ skip）：用户通过数据集对象方法 `.shuffle` / `.filter` / `.skip` / `.split` /
`.take` / … 来实现数据集的进一步混洗、过滤、跳过、最多获取条数等操作；
- 数据集样本增强操作（map）：用户可以将数据增强操作
（`vision类 <https://www.mindspore.cn/docs/zh-CN/master/api_python/mindspore.dataset.vision.html>`_，
`nlp类 <https://www.mindspore.cn/docs/zh-CN/master/api_python/mindspore.dataset.text.html>`_，
`audio类 <https://www.mindspore.cn/docs/zh-CN/master/api_python/mindspore.dataset.audio.html>`_）
（`vision类 <https://www.mindspore.cn/docs/zh-CN/master/api_python/mindspore.dataset.vision.html>`_ ，
`nlp类 <https://www.mindspore.cn/docs/zh-CN/master/api_python/mindspore.dataset.text.html>`_ ，
`audio类 <https://www.mindspore.cn/docs/zh-CN/master/api_python/mindspore.dataset.audio.html>`_ ）
添加到map操作中执行，数据预处理过程中可以定义多个map操作，用于执行不同增强操作，数据增强操作也可以是
用户自定义增强的 `PyFunc`；
用户自定义增强的 `PyFunc` ；
- 批（batch）：用户在样本完成增强后，使用 `.batch` 操作将多个样本组织成batch，也可以通过batch的参数 `per_batch_map`
来自定义batch逻辑；
- 迭代器（create_dict_iterator）：最后用户通过数据集对象方法 `create_dict_iterator` 来创建迭代器，
可以将预处理完成的数据循环输出。
数据处理Pipeline示例如下，完整示例请参考
`datasets_example.py <https://gitee.com/mindspore/mindspore/tree/master/docs/api/api_python/datasets_example.py>`_：
`datasets_example.py <https://gitee.com/mindspore/mindspore/tree/master/docs/api/api_python/datasets_example.py>`_ ：
.. code-block:: python

4
docs/api/api_python/mindspore.dataset.text.rst
View File

@ -14,7 +14,7 @@ mindspore.dataset.text
import mindspore.dataset as ds
from mindspore.dataset import text
更多详情请参考 `文本数据变换 <https://www.mindspore.cn/tutorials/zh-CN/master/beginner/transforms.html#text-transforms>`_。
更多详情请参考 `文本数据变换 <https://www.mindspore.cn/tutorials/zh-CN/master/beginner/transforms.html#text-transforms>`_ 。
常用数据处理术语说明如下：
@ -23,7 +23,7 @@ mindspore.dataset.text
数据增强操作可以放入数据处理Pipeline中执行，也可以Eager模式执行：
- Pipeline模式一般用于处理数据集，示例可参考 `数据处理Pipeline介绍 <https://www.mindspore.cn/docs/zh-CN/master/api_python/mindspore.dataset.html#数据处理pipeline介绍>`_。
- Pipeline模式一般用于处理数据集，示例可参考 `数据处理Pipeline介绍 <https://www.mindspore.cn/docs/zh-CN/master/api_python/mindspore.dataset.html#数据处理pipeline介绍>`_ 。
- Eager模式一般用于零散样本，文本预处理举例如下：
.. code-block::

4
docs/api/api_python/mindspore.dataset.vision.rst
View File

@ -19,7 +19,7 @@ API样例中常用的导入模块如下：
import mindspore.dataset.vision.py_transforms as py_vision
from mindspore.dataset.transforms import c_transforms
更多详情请参考 `视觉数据变换 <https://www.mindspore.cn/tutorials/zh-CN/master/beginner/transforms.html#vision-transforms>`_。
更多详情请参考 `视觉数据变换 <https://www.mindspore.cn/tutorials/zh-CN/master/beginner/transforms.html#vision-transforms>`_ 。
常用数据处理术语说明如下：
@ -28,7 +28,7 @@ API样例中常用的导入模块如下：
数据增强操作可以放入数据处理Pipeline中执行，也可以Eager模式执行：
- Pipeline模式一般用于处理数据集，示例可参考 `数据处理Pipeline介绍 <https://www.mindspore.cn/docs/zh-CN/master/api_python/mindspore.dataset.html#数据处理pipeline介绍>`_。
- Pipeline模式一般用于处理数据集，示例可参考 `数据处理Pipeline介绍 <https://www.mindspore.cn/docs/zh-CN/master/api_python/mindspore.dataset.html#数据处理pipeline介绍>`_ 。
- Eager模式一般用于零散样本，图像预处理举例如下：
.. code-block::

1
docs/api/api_python_en/dataset_list.rst
View File

@ -332,6 +332,7 @@ Batch
mindspore.dataset.Dataset.batch
mindspore.dataset.Dataset.bucket_batch_by_length
mindspore.dataset.Dataset.padded_batch
Iterator
---------

8
mindspore/python/mindspore/dataset/__init__.py
View File

@ -50,14 +50,14 @@ pipelines and transform samples in the dataset in the most efficient (multi-proc
The specific steps are as follows:
- Loading datasets: Users can easily load supported datasets using the `*Dataset` class, or load Python layer
customized datasets through `UDF Loader` + `GeneratorDataset`. At the same time, the loading class method can
customized datasets through `UDF Loader` + `GeneratorDataset` . At the same time, the loading class method can
accept a variety of parameters such as sampler, data slicing, and data shuffle;
- Dataset operation: The user uses the dataset object method `.shuffle` / `.filter` / `.skip` / `.split` /
`.take` / ... to further shuffle, filter, skip, and obtain the maximum number of samples of datasets;
- Dataset sample transform operation: The user can add data transform operations
(`vision transform <https://www.mindspore.cn/docs/en/master/api_python/mindspore.dataset.vision.html>`_,
`NLP transform <https://www.mindspore.cn/docs/en/master/api_python/mindspore.dataset.text.html>`_,
`audio transform <https://www.mindspore.cn/docs/en/master/api_python/mindspore.dataset.audio.html>`_) to the map
( `vision transform <https://www.mindspore.cn/docs/en/master/api_python/mindspore.dataset.vision.html>`_ ,
`NLP transform <https://www.mindspore.cn/docs/en/master/api_python/mindspore.dataset.text.html>`_ ,
`audio transform <https://www.mindspore.cn/docs/en/master/api_python/mindspore.dataset.audio.html>`_ ) to the map
operation to perform transformations. During data preprocessing, multiple map operations can be defined to
perform different transform operations to different fields. The data transform operation can also be a
user-defined transform `pyfunc` (Python function);

2
mindspore/python/mindspore/dataset/audio/__init__.py
View File

@ -41,7 +41,7 @@ The data transform operation can be executed in the data processing pipeline or
- Pipeline mode is generally used to process datasets. For examples, please refer to
`introduction to data processing pipeline <https://www.mindspore.cn/docs/en/master/api_python/
mindspore.dataset.html#introduction-to-data-processing-pipeline>`_.
mindspore.dataset.html#introduction-to-data-processing-pipeline>`_ .
- Eager mode is generally used for scattered samples. Examples of audio preprocessing are as follows:
.. code-block::

33
mindspore/python/mindspore/dataset/audio/transforms.py
View File

@ -122,16 +122,16 @@ class AmplitudeToDB(AudioTensorOperation):
stype (ScaleType, optional): Scale of the input waveform, which can be
ScaleType.POWER or ScaleType.MAGNITUDE. Default: ScaleType.POWER.
ref_value (float, optional): Multiplier reference value for generating
`db_multiplier`. Default: 1.0. The formula is
`db_multiplier` . Default: 1.0. The formula is
:math:`\text{db_multiplier} = Log10(max(\text{ref_value}, amin))`.
:math:`\text{db_multiplier} = Log10(max(\text{ref_value}, amin))` .
amin (float, optional): Lower bound to clamp the input waveform, which must
be greater than zero. Default: 1e-10.
top_db (float, optional): Minimum cut-off decibels, which must be non-negative. Default: 80.0.
Raises:
TypeError: If `stype` is not of type :class:`mindspore.dataset.audio.utils.ScaleType`.
TypeError: If `stype` is not of type :class:`mindspore.dataset.audio.utils.ScaleType` .
TypeError: If `ref_value` is not of type float.
ValueError: If `ref_value` is not a positive number.
TypeError: If `amin` is not of type float.
@ -1044,7 +1044,7 @@ class GriffinLim(AudioTensorOperation):
Compute waveform from a linear scale magnitude spectrogram using the Griffin-Lim transformation.
About Griffin-Lim please refer to `A fast Griffin-Lim algorithm <https://doi.org/10.1109/WASPAA.2013.6701851>`_
and `Signal estimation from modified short-time Fourier transform <https://doi.org/10.1109/ICASSP.1983.1172092>`_.
and `Signal estimation from modified short-time Fourier transform <https://doi.org/10.1109/ICASSP.1983.1172092>`_ .
Args:
n_fft (int, optional): Size of FFT. Default: 400.
@ -1062,8 +1062,8 @@ class GriffinLim(AudioTensorOperation):
Default: True.
Raises:
RuntimeError: If `n_fft` is not less than `length`.
RuntimeError: If `win_length` is not less than `n_fft`.
RuntimeError: If `n_fft` is not less than `length` .
RuntimeError: If `win_length` is not less than `n_fft` .
Examples:
>>> import numpy as np
@ -1137,7 +1137,7 @@ class InverseMelScale(AudioTensorOperation):
n_mels (int, optional): Number of mel filterbanks. Default: 128.
sample_rate (int, optional): Sample rate of audio signal. Default: 16000.
f_min (float, optional): Minimum frequency. Default: 0.0.
f_max (float, optional): Maximum frequency. Default: None, will be set to `sample_rate // 2`.
f_max (float, optional): Maximum frequency. Default: None, will be set to `sample_rate // 2` .
max_iter (int, optional): Maximum number of optimization iterations. Default: 100000.
tolerance_loss (float, optional): Value of loss to stop optimization at. Default: 1e-5.
tolerance_change (float, optional): Difference in losses to stop optimization at. Default: 1e-8.
@ -1306,7 +1306,7 @@ class Magphase(AudioTensorOperation):
class MaskAlongAxis(AudioTensorOperation):
"""
Apply a mask along `axis`. Mask will be applied from indices `[mask_start, mask_start + mask_width)`.
Apply a mask along `axis` . Mask will be applied from indices `[mask_start, mask_start + mask_width)` .
Args:
mask_start (int): Starting position of the mask, which must be non negative.
@ -1342,9 +1342,10 @@ class MaskAlongAxis(AudioTensorOperation):
class MaskAlongAxisIID(AudioTensorOperation):
"""
Apply a mask along `axis`. Mask will be applied from indices `[mask_start, mask_start + mask_width)`, where
`mask_width` is sampled from `uniform[0, mask_param]`, and `mask_start` from `uniform[0, max_length - mask_width]`,
`max_length` is the number of columns of the specified axis of the spectrogram.
Apply a mask along `axis` . Mask will be applied from indices `[mask_start, mask_start + mask_width)` , where
`mask_width` is sampled from `uniform[0, mask_param]` , and `mask_start` from
`uniform[0, max_length - mask_width]` , `max_length` is the number of columns of the specified axis
of the spectrogram.
Args:
mask_param (int): Number of columns to be masked, will be uniformly sampled from
@ -1395,7 +1396,7 @@ class MelScale(AudioTensorOperation):
n_mels (int, optional): Number of mel filterbanks. Default: 128.
sample_rate (int, optional): Sample rate of audio signal. Default: 16000.
f_min (float, optional): Minimum frequency. Default: 0.
f_max (float, optional): Maximum frequency. Default: None, will be set to `sample_rate // 2`.
f_max (float, optional): Maximum frequency. Default: None, will be set to `sample_rate // 2` .
n_stft (int, optional): Number of bins in STFT. Default: 201.
norm (NormType, optional): Type of norm, value should be NormType.SLANEY or NormType::NONE.
If norm is NormType.SLANEY, divide the triangular mel weight by the width of the mel band.
@ -1766,7 +1767,7 @@ class SpectralCentroid(TensorOperation):
ValueError: If `n_fft` is not a positive number.
TypeError: If `win_length` is not of type int.
ValueError: If `win_length` is not a positive number.
ValueError: If `win_length` is greater than `n_fft`.
ValueError: If `win_length` is greater than `n_fft` .
TypeError: If `hop_length` is not of type int.
ValueError: If `hop_length` is not a positive number.
TypeError: If `pad` is not of type int.
@ -1824,7 +1825,7 @@ class Spectrogram(TensorOperation):
ValueError: If `n_fft` is not a positive number.
TypeError: If `win_length` is not of type int.
ValueError: If `win_length` is not a positive number.
ValueError: If `win_length` is greater than `n_fft`.
ValueError: If `win_length` is greater than `n_fft` .
TypeError: If `hop_length` is not of type int.
ValueError: If `hop_length` is not a positive number.
TypeError: If `pad` is not of type int.
@ -1934,7 +1935,7 @@ class TimeStretch(AudioTensorOperation):
Args:
hop_length (int, optional): Length of hop between STFT windows, i.e. the number of samples
between consecutive frames. Default: None, will use `n_freq - 1`.
between consecutive frames. Default: None, will use `n_freq - 1` .
n_freq (int, optional): Number of filter banks from STFT. Default: 201.
fixed_rate (float, optional): Rate to speed up or slow down by. Default: None, will keep
the original rate.
@ -2080,7 +2081,7 @@ class Vad(AudioTensorOperation):
ValueError: If `noise_up_time` is a negative number.
TypeError: If `noise_down_time` is not of type float.
ValueError: If `noise_down_time` is a negative number.
ValueError: If `noise_up_time` is less than `noise_down_time`.
ValueError: If `noise_up_time` is less than `noise_down_time` .
TypeError: If `noise_reduction_amount` is not of type float.
ValueError: If `noise_reduction_amount` is a negative number.
TypeError: If `measure_freq` is not of type float.

22
mindspore/python/mindspore/dataset/callback/ds_callback.py
View File

@ -27,8 +27,8 @@ class DSCallback:
"""
Abstract base class used to build dataset callback classes.
Users can obtain the dataset pipeline context through `ds_run_context`, including
`cur_epoch_num`, `cur_step_num_in_epoch` and `cur_step_num`.
Users can obtain the dataset pipeline context through `ds_run_context` , including
`cur_epoch_num` , `cur_step_num_in_epoch` and `cur_step_num` .
Args:
step_size (int, optional): The number of steps between adjacent `ds_step_begin`/`ds_step_end`
@ -131,20 +131,22 @@ class WaitedDSCallback(Callback, DSCallback):
r"""
Abstract base class used to build dataset callback classes that are synchronized with the training callback class
`mindspore.train.Callback \
<https://www.mindspore.cn/docs/en/master/api_python/train/mindspore.train.Callback.html#mindspore.train.Callback>`_.
<https://www.mindspore.cn/docs/en/master/api_python/train/
mindspore.train.Callback.html#mindspore.train.Callback>`_ .
It can be used to execute a custom callback method before a step or an epoch, such as
updating the parameters of operations according to the loss of the previous training epoch in auto augmentation.
Users can obtain the network training context through `train_run_context`, such as
`network`, `train_network`, `epoch_num`, `batch_num`, `loss_fn`, `optimizer`, `parallel_mode`,
`device_number`, `list_callback`, `cur_epoch_num`, `cur_step_num`, `dataset_sink_mode`,
`net_outputs`, etc., see
Users can obtain the network training context through `train_run_context` , such as
`network` , `train_network` , `epoch_num` , `batch_num` , `loss_fn` , `optimizer` , `parallel_mode` ,
`device_number` , `list_callback` , `cur_epoch_num` , `cur_step_num` , `dataset_sink_mode` ,
`net_outputs` , etc., see
`mindspore.train.Callback \
<https://www.mindspore.cn/docs/en/master/api_python/train/mindspore.train.Callback.html#mindspore.train.Callback>`_.
<https://www.mindspore.cn/docs/en/master/api_python/train/
mindspore.train.Callback.html#mindspore.train.Callback>`_ .
Users can obtain the dataset pipeline context through `ds_run_context`, including
`cur_epoch_num`, `cur_step_num_in_epoch` and `cur_step_num`.
Users can obtain the dataset pipeline context through `ds_run_context` , including
`cur_epoch_num` , `cur_step_num_in_epoch` and `cur_step_num` .
Note:
Note that the call is triggered only at the beginning of the second step or epoch.

20
mindspore/python/mindspore/dataset/engine/datasets.py
View File

@ -535,7 +535,6 @@ class Dataset:
Combine batch_size number of consecutive rows into batch which apply per_batch_map to the samples first.
For any column, all the elements within that column must have the same shape.
If a per_batch_map callable is provided, it will be applied to the batches of tensors.
Refer to the following figure for the execution process:
@ -749,7 +748,7 @@ class Dataset:
Args:
func (function): A function that must take one `numpy.ndarray` as an argument and
return a `Dataset`.
return a `Dataset` .
Returns:
Dataset, dataset applied by the function.
@ -811,12 +810,12 @@ class Dataset:
columns of the previous operation are used as the input columns for the next operation.
The columns outputted by the very last operation will be assigned names specified by
`output_columns`, and if not specified, the column name of output column is same as that of `input_columns`.
`output_columns` , and if not specified, the column name of output column is same as that of `input_columns` .
- If you use transformations (
`vision transform <https://www.mindspore.cn/docs/en/master/api_python/mindspore.dataset.vision.html>`_,
`nlp transform <https://www.mindspore.cn/docs/en/master/api_python/mindspore.dataset.text.html>`_,
`audio transform <https://www.mindspore.cn/docs/en/master/api_python/mindspore.dataset.audio.html>`_)
`vision transform <https://www.mindspore.cn/docs/en/master/api_python/mindspore.dataset.vision.html>`_ ,
`nlp transform <https://www.mindspore.cn/docs/en/master/api_python/mindspore.dataset.text.html>`_ ,
`audio transform <https://www.mindspore.cn/docs/en/master/api_python/mindspore.dataset.audio.html>`_ )
provided by mindspore dataset, please use the following parameters:
.. image:: map_parameter_en.png
@ -860,7 +859,7 @@ class Dataset:
- Input `operations` accepts TensorOperations defined in mindspore.dataset part, plus user-defined
Python functions (PyFuncs).
- Do not add network computing operators from mindspore.nn and mindspore.ops or others into this
`operations`.
`operations` .
Returns:
Dataset, dataset after mapping operation.
@ -1271,7 +1270,7 @@ class Dataset:
Args:
apply_func (function): A function that must take one `Dataset` as an argument and
return a preprocessed `Dataset`.
return a preprocessed `Dataset` .
Returns:
Dataset, dataset applied by the function.
@ -1325,7 +1324,8 @@ class Dataset:
Save the dynamic data processed by the dataset pipeline in common dataset format.
Supported dataset formats: `mindrecord` only. And you can use `MindDataset` API to read the saved file(s).
Implicit type casting exists when saving data as `mindrecord`. The transform table shows how to do type casting.
Implicit type casting exists when saving data as `mindrecord` . The transform table shows how to do
type casting.
.. list-table:: Implicit Type Casting when Saving as `mindrecord`
:widths: 25 25 50
@ -1404,7 +1404,7 @@ class Dataset:
@check_tuple_iterator
def create_tuple_iterator(self, columns=None, num_epochs=-1, output_numpy=False, do_copy=True):
"""
Create an iterator over the dataset. The datatype retrieved back will be a list of `numpy.ndarray`.
Create an iterator over the dataset. The datatype retrieved back will be a list of `numpy.ndarray` .
To specify which columns to list and the order needed, use columns_list. If columns_list
is not provided, the order of the columns will remain unchanged.

40
mindspore/python/mindspore/dataset/engine/datasets_audio.py
View File

@ -36,7 +36,7 @@ class CMUArcticDataset(MappableDataset, AudioBaseDataset):
"""
A source dataset that reads and parses CMUArctic dataset.
The generated dataset has four columns: :py:obj:`["waveform", "sample_rate", "transcript", "utterance_id"]`.
The generated dataset has four columns: :py:obj:`["waveform", "sample_rate", "transcript", "utterance_id"]` .
The tensor of column :py:obj:`waveform` is of the float32 type.
The tensor of column :py:obj:`sample_rate` is of a scalar of uint32 type.
The tensor of column :py:obj:`transcript` is of a scalar of string type.
@ -57,7 +57,7 @@ class CMUArcticDataset(MappableDataset, AudioBaseDataset):
dataset. Default: None, expected order behavior shown in the table below.
num_shards (int, optional): Number of shards that the dataset will be divided into. Default: None.
When this argument is specified, `num_samples` reflects the max sample number of per shard.
shard_id (int, optional): The shard ID within `num_shards`. Default: None. This
shard_id (int, optional): The shard ID within `num_shards` . Default: None. This
argument can only be specified when `num_shards` is also specified.
cache (DatasetCache, optional): Use tensor caching service to speed up dataset processing. More details:
`Single-Node Data Cache <https://www.mindspore.cn/tutorials/experts/en/master/dataset/cache.html>`_ .
@ -75,7 +75,7 @@ class CMUArcticDataset(MappableDataset, AudioBaseDataset):
Note:
- CMUArctic dataset doesn't support PKSampler.
- This dataset can take in a `sampler`. `sampler` and `shuffle` are mutually exclusive.
- This dataset can take in a `sampler` . `sampler` and `shuffle` are mutually exclusive.
The table below shows what input arguments are allowed and their expected behavior.
.. list-table:: Expected Order Behavior of Using 'sampler' and 'shuffle'
@ -174,7 +174,7 @@ class GTZANDataset(MappableDataset, AudioBaseDataset):
"""
A source dataset that reads and parses GTZAN dataset.
The generated dataset has three columns: :py:obj:`["waveform", "sample_rate", "label"]`.
The generated dataset has three columns: :py:obj:`["waveform", "sample_rate", "label"]` .
The tensor of column :py:obj:`waveform` is of the float32 type.
The tensor of column :py:obj:`sample_rate` is of a scalar of uint32 type.
The tensor of column :py:obj:`label` is of a scalar of string type.
@ -193,7 +193,7 @@ class GTZANDataset(MappableDataset, AudioBaseDataset):
dataset. Default: None, expected order behavior shown in the table below.
num_shards (int, optional): Number of shards that the dataset will be divided into. Default: None.
When this argument is specified, `num_samples` reflects the max sample number of per shard.
shard_id (int, optional): The shard ID within `num_shards`. Default: None. This
shard_id (int, optional): The shard ID within `num_shards` . Default: None. This
argument can only be specified when `num_shards` is also specified.
cache (DatasetCache, optional): Use tensor caching service to speed up dataset processing. More details:
`Single-Node Data Cache <https://www.mindspore.cn/tutorials/experts/en/master/dataset/cache.html>`_ .
@ -211,7 +211,7 @@ class GTZANDataset(MappableDataset, AudioBaseDataset):
Note:
- GTZAN doesn't support PKSampler.
- This dataset can take in a `sampler`. `sampler` and `shuffle` are mutually exclusive.
- This dataset can take in a `sampler` . `sampler` and `shuffle` are mutually exclusive.
The table below shows what input arguments are allowed and their expected behavior.
.. list-table:: Expected Order Behavior of Using 'sampler' and 'shuffle'
@ -306,7 +306,7 @@ class LibriTTSDataset(MappableDataset, AudioBaseDataset):
A source dataset that reads and parses the LibriTTS dataset.
The generated dataset has seven columns :py:obj:`['waveform', 'sample_rate', 'original_text', 'normalized_text',
'speaker_id', 'chapter_id', 'utterance_id']`.
'speaker_id', 'chapter_id', 'utterance_id']` .
The tensor of column :py:obj:`waveform` is of the float32 type.
The tensor of column :py:obj:`sample_rate` is of a scalar of uint32 type.
The tensor of column :py:obj:`original_text` is of a scalar of string type.
@ -329,7 +329,7 @@ class LibriTTSDataset(MappableDataset, AudioBaseDataset):
dataset. Default: None, expected order behavior shown in the table below.
num_shards (int, optional): Number of shards that the dataset will be divided into. Default: None.
When this argument is specified, `num_samples` reflects the max sample number of per shard.
shard_id (int, optional): The shard ID within `num_shards`. Default: None. This
shard_id (int, optional): The shard ID within `num_shards` . Default: None. This
argument can only be specified when `num_shards` is also specified.
cache (DatasetCache, optional): Use tensor caching service to speed up dataset processing. More details:
`Single-Node Data Cache <https://www.mindspore.cn/tutorials/experts/en/master/dataset/cache.html>`_ .
@ -347,7 +347,7 @@ class LibriTTSDataset(MappableDataset, AudioBaseDataset):
Note:
- LibriTTS dataset doesn't support PKSampler.
- This dataset can take in a `sampler`. `sampler` and `shuffle` are mutually exclusive.
- This dataset can take in a `sampler` . `sampler` and `shuffle` are mutually exclusive.
The table below shows what input arguments are allowed and their expected behavior.
.. list-table:: Expected Order Behavior of Using 'sampler' and 'shuffle'
@ -491,7 +491,7 @@ class LJSpeechDataset(MappableDataset, AudioBaseDataset):
ValueError: If `shard_id` is invalid (< 0 or >= `num_shards`).
Note:
- This dataset can take in a `sampler`. `sampler` and `shuffle` are mutually exclusive.
- This dataset can take in a `sampler` . `sampler` and `shuffle` are mutually exclusive.
The table below shows what input arguments are allowed and their expected behavior.
.. list-table:: Expected Order Behavior of Using `sampler` and `shuffle`
@ -593,7 +593,7 @@ class SpeechCommandsDataset(MappableDataset, AudioBaseDataset):
"""
A source dataset that reads and parses the SpeechCommands dataset.
The generated dataset has five columns :py:obj:`[waveform, sample_rate, label, speaker_id, utterance_number]`.
The generated dataset has five columns :py:obj:`[waveform, sample_rate, label, speaker_id, utterance_number]` .
The tensor of column :py:obj:`waveform` is a vector of the float32 type.
The tensor of column :py:obj:`sample_rate` is a scalar of the int32 type.
The tensor of column :py:obj:`label` is a scalar of the string type.
@ -615,7 +615,7 @@ class SpeechCommandsDataset(MappableDataset, AudioBaseDataset):
Default: None, expected order behavior shown in the table below.
num_shards (int, optional): Number of shards that the dataset will be divided into. Default: None.
When this argument is specified, `num_samples` reflects the maximum sample number of per shard.
shard_id (int, optional): The shard ID within `num_shards`. Default: None. This argument can only be specified
shard_id (int, optional): The shard ID within `num_shards` . Default: None. This argument can only be specified
when `num_shards` is also specified.
cache (DatasetCache, optional): Use tensor caching service to speed up dataset processing. More details:
`Single-Node Data Cache <https://www.mindspore.cn/tutorials/experts/en/master/dataset/cache.html>`_ .
@ -631,7 +631,7 @@ class SpeechCommandsDataset(MappableDataset, AudioBaseDataset):
ValueError: If `shard_id` is invalid (< 0 or >= `num_shards`).
Note:
- This dataset can take in a `sampler`. `sampler` and `shuffle` are mutually exclusive.
- This dataset can take in a `sampler` . `sampler` and `shuffle` are mutually exclusive.
The table below shows what input arguments are allowed and their expected behavior.
.. list-table:: Expected Order Behavior of Using `sampler` and `shuffle`
@ -718,10 +718,10 @@ class TedliumDataset(MappableDataset, AudioBaseDataset):
The columns of generated dataset depend on the source SPH files and the corresponding STM files.
The generated dataset has six columns :py:obj:`[waveform, sample_rate, transcript, talk_id, speaker_id,
identifier]`.
identifier]` .
The data type of column `waveform` is float32, the data type of column `sample_rate` is int32,
and the data type of columns `transcript`, `talk_id`, `speaker_id` and `identifier` is string.
and the data type of columns `transcript` , `talk_id` , `speaker_id` and `identifier` is string.
Args:
dataset_dir (str): Path to the root directory that contains the dataset.
@ -746,7 +746,7 @@ class TedliumDataset(MappableDataset, AudioBaseDataset):
num_shards (int, optional): Number of shards that the dataset will be divided
into. Default: None. When this argument is specified, `num_samples` reflects
the maximum sample number of per shard.
shard_id (int, optional): The shard ID within `num_shards`. Default: None. This
shard_id (int, optional): The shard ID within `num_shards` . Default: None. This
argument can only be specified when `num_shards` is also specified.
cache (DatasetCache, optional): Use tensor caching service to speed up dataset processing. More details:
`Single-Node Data Cache <https://www.mindspore.cn/tutorials/experts/en/master/dataset/cache.html>`_ .
@ -762,7 +762,7 @@ class TedliumDataset(MappableDataset, AudioBaseDataset):
ValueError: If `shard_id` is invalid (< 0 or >= `num_shards`).
Note:
- This dataset can take in a `sampler`. `sampler` and `shuffle` are mutually exclusive.
- This dataset can take in a `sampler` . `sampler` and `shuffle` are mutually exclusive.
The table below shows what input arguments are allowed and their expected behavior.
.. list-table:: Expected Order Behavior of Using `sampler` and `shuffle`
@ -927,7 +927,7 @@ class YesNoDataset(MappableDataset, AudioBaseDataset):
"""
A source dataset that reads and parses the YesNo dataset.
The generated dataset has three columns :py:obj:`[waveform, sample_rate, labels]`.
The generated dataset has three columns :py:obj:`[waveform, sample_rate, labels]` .
The tensor of column :py:obj:`waveform` is a vector of the float32 type.
The tensor of column :py:obj:`sample_rate` is a scalar of the int32 type.
The tensor of column :py:obj:`labels` is a scalar of the int32 type.
@ -944,7 +944,7 @@ class YesNoDataset(MappableDataset, AudioBaseDataset):
dataset. Default: None, expected order behavior shown in the table below.
num_shards (int, optional): Number of shards that the dataset will be divided into. Default: None.
When this argument is specified, `num_samples` reflects the maximum sample number of per shard.
shard_id (int, optional): The shard ID within `num_shards`. Default: None. This argument can only
shard_id (int, optional): The shard ID within `num_shards` . Default: None. This argument can only
be specified when `num_shards` is also specified.
cache (DatasetCache, optional): Use tensor caching service to speed up dataset processing. More details:
`Single-Node Data Cache <https://www.mindspore.cn/tutorials/experts/en/master/dataset/cache.html>`_ .
@ -960,7 +960,7 @@ class YesNoDataset(MappableDataset, AudioBaseDataset):
ValueError: If `shard_id` is invalid (< 0 or >= `num_shards`).
Note:
- This dataset can take in a `sampler`. `sampler` and `shuffle` are mutually exclusive.
- This dataset can take in a `sampler` . `sampler` and `shuffle` are mutually exclusive.
The table below shows what input arguments are allowed and their expected behavior.
.. list-table:: Expected Order Behavior of Using `sampler` and `shuffle`

10
mindspore/python/mindspore/dataset/engine/datasets_standard_format.py
View File

@ -70,7 +70,7 @@ class CSVDataset(SourceDataset, UnionBaseDataset):
num_shards (int, optional): Number of shards that the dataset will be divided into. Default: None.
When this argument is specified, `num_samples` reflects the maximum sample number of per shard.
shard_id (int, optional): The shard ID within `num_shards`. Default: None. This
shard_id (int, optional): The shard ID within `num_shards` . Default: None. This
argument can only be specified when `num_shards` is also specified.
cache (DatasetCache, optional): Use tensor caching service to speed up dataset processing. More details:
`Single-Node Data Cache <https://www.mindspore.cn/tutorials/experts/en/master/dataset/cache.html>`_ .
@ -132,7 +132,7 @@ class MindDataset(MappableDataset, UnionBaseDataset):
num_shards (int, optional): Number of shards that the dataset will be divided into. Default: None.
When this argument is specified, 'num_samples' reflects the maximum sample number of per shard.
shard_id (int, optional): The shard ID within `num_shards`. Default: None. This
shard_id (int, optional): The shard ID within `num_shards` . Default: None. This
argument can only be specified when `num_shards` is also specified.
sampler (Sampler, optional): Object used to choose samples from the
dataset. Default: None, sampler is exclusive
@ -156,7 +156,7 @@ class MindDataset(MappableDataset, UnionBaseDataset):
ValueError: If `shard_id` is invalid (< 0 or >= `num_shards`).
Note:
- This dataset can take in a `sampler`. `sampler` and `shuffle` are mutually exclusive.
- This dataset can take in a `sampler` . `sampler` and `shuffle` are mutually exclusive.
The table below shows what input arguments are allowed and their expected behavior.
.. list-table:: Expected Order Behavior of Using `sampler` and `shuffle`
@ -271,7 +271,7 @@ class TFRecordDataset(SourceDataset, UnionBaseDataset):
num_shards (int, optional): Number of shards that the dataset will be divided
into. Default: None. When this argument is specified, `num_samples` reflects
the maximum sample number of per shard.
shard_id (int, optional): The shard ID within `num_shards`. Default: None. This
shard_id (int, optional): The shard ID within `num_shards` . Default: None. This
argument can only be specified when `num_shards` is also specified.
shard_equal_rows (bool, optional): Get equal rows for all shards. Default: False. If shard_equal_rows
is false, number of rows of each shard may be not equal, and may lead to a failure in distributed training.
@ -380,7 +380,7 @@ class OBSMindDataset(GeneratorDataset):
- It's necessary to create a synchronization directory on cloud storage in
advance which be defined by parameter: `sync_obs_path` .
- If training is offline(no cloud), it's recommended to set the
environment variable `BATCH_JOB_ID`.
environment variable `BATCH_JOB_ID` .
- In distributed training, if there are multiple nodes(servers), all 8
devices must be used in each node(server). If there is only one
node(server), there is no such restriction.

46
mindspore/python/mindspore/dataset/engine/datasets_text.py
View File

@ -249,7 +249,7 @@ class CLUEDataset(SourceDataset, TextBaseDataset):
num_shards (int, optional): Number of shards that the dataset will be divided into. Default: None.
When this argument is specified, `num_samples` reflects the maximum sample number of per shard.
shard_id (int, optional): The shard ID within `num_shards`. Default: None. This
shard_id (int, optional): The shard ID within `num_shards` . Default: None. This
argument can only be specified when `num_shards` is also specified.
cache (DatasetCache, optional): Use tensor caching service to speed up dataset processing. More details:
`Single-Node Data Cache <https://www.mindspore.cn/tutorials/experts/en/master/dataset/cache.html>`_ .
@ -461,8 +461,8 @@ class CoNLL2000Dataset(SourceDataset, TextBaseDataset):
"""
A source dataset that reads and parses CoNLL2000 chunking dataset.
The generated dataset has three columns: :py:obj:`[word, pos_tag, chunk_tag]`.
The tensors of column :py:obj:`word`, column :py:obj:`pos_tag`,
The generated dataset has three columns: :py:obj:`[word, pos_tag, chunk_tag]` .
The tensors of column :py:obj:`word` , column :py:obj:`pos_tag` ,
and column :py:obj:`chunk_tag` are of the string type.
Args:
@ -473,7 +473,7 @@ class CoNLL2000Dataset(SourceDataset, TextBaseDataset):
'all' will read from all 1,0948 samples. Default: None, read all samples.
num_samples (int, optional): Number of samples (rows) to be read. Default: None, read the full dataset.
shuffle (Union[bool, Shuffle], optional): Perform reshuffling of the data every epoch.
Default: `mindspore.dataset.Shuffle.GLOBAL`.
Default: `mindspore.dataset.Shuffle.GLOBAL` .
If shuffle is False, no shuffling will be performed.
If shuffle is True, performs global shuffle.
There are three levels of shuffling, desired shuffle enum defined by mindspore.dataset.Shuffle.
@ -483,7 +483,7 @@ class CoNLL2000Dataset(SourceDataset, TextBaseDataset):
num_shards (int, optional): Number of shards that the dataset will be divided into.
When this argument is specified, `num_samples` reflects the max sample number of per shard. Default: None.
shard_id (int, optional): The shard ID within `num_shards`. This
shard_id (int, optional): The shard ID within `num_shards` . This
argument can only be specified when `num_shards` is also specified. Default: None.
num_parallel_workers (int, optional): Number of workers to read the data.
Default: None, number set in the config.
@ -773,7 +773,7 @@ class IMDBDataset(MappableDataset, TextBaseDataset):
Note:
- The shape of the test column.
- This dataset can take in a `sampler`. `sampler` and `shuffle` are mutually exclusive.
- This dataset can take in a `sampler` . `sampler` and `shuffle` are mutually exclusive.
The table below shows what input arguments are allowed and their expected behavior.
.. list-table:: Expected Order Behavior of Using `sampler` and `shuffle`
@ -908,7 +908,7 @@ class IWSLT2016Dataset(SourceDataset, TextBaseDataset):
num_shards (int, optional): Number of shards that the dataset will be divided into. Default: None.
When this argument is specified, `num_samples` reflects the max sample number of per shard.
shard_id (int, optional): The shard ID within `num_shards`. Default: None. This
shard_id (int, optional): The shard ID within `num_shards` . Default: None. This
argument can only be specified when `num_shards` is also specified.
num_parallel_workers (int, optional): Number of workers to read the data.
Default: None, number set in the mindspore.dataset.config.
@ -1104,7 +1104,7 @@ class Multi30kDataset(SourceDataset, TextBaseDataset):
"""
A source dataset that reads and parses Multi30k dataset.
The generated dataset has two columns :py:obj:`[text, translation]`.
The generated dataset has two columns :py:obj:`[text, translation]` .
The tensor of column :py:obj:'text' is of the string type.
The tensor of column :py:obj:'translation' is of the string type.
@ -1130,7 +1130,7 @@ class Multi30kDataset(SourceDataset, TextBaseDataset):
num_shards (int, optional): Number of shards that the dataset will be divided
into. Default: None. When this argument is specified, `num_samples` reflects
the max sample number of per shard.
shard_id (int, optional): The shard ID within `num_shards`. Default: None. This
shard_id (int, optional): The shard ID within `num_shards` . Default: None. This
argument can only be specified when `num_shards` is also specified.
cache (DatasetCache, optional): Use tensor caching service to speed up dataset processing. More details:
`Single-Node Data Cache <https://www.mindspore.cn/tutorials/experts/en/master/dataset/cache.html>`_ .
@ -1302,7 +1302,7 @@ class SogouNewsDataset(SourceDataset, TextBaseDataset):
r"""
A source dataset that reads and parses Sogou News dataset.
The generated dataset has three columns: :py:obj:`[index, title, content]`,
The generated dataset has three columns: :py:obj:`[index, title, content]` ,
and the data type of three columns is string.
Args:
@ -1322,7 +1322,7 @@ class SogouNewsDataset(SourceDataset, TextBaseDataset):
- Shuffle.FILES: Shuffle files only.
num_shards (int, optional): Number of shards that the dataset will be divided into. Default: None.
When this argument is specified, `num_samples` reflects the max sample number of per shard.
shard_id (int, optional): The shard ID within `num_shards`. Default: None. This
shard_id (int, optional): The shard ID within `num_shards` . Default: None. This
argument can only be specified when `num_shards` is also specified.
num_parallel_workers (int, optional): Number of workers to read the data.
Default: None, number set in the config.
@ -1389,7 +1389,7 @@ class SQuADDataset(SourceDataset, TextBaseDataset):
A source dataset that reads and parses SQuAD 1.1 and SQuAD 2.0 datasets.
The generated dataset with different versions and usages has the same output columns:
:py:obj:`[context, question, text, answer_start]`.
:py:obj:`[context, question, text, answer_start]` .
The tensor of column :py:obj:`context` is of the string type.
The tensor of column :py:obj:`question` is of the string type.
The tensor of column :py:obj:`text` is the answer in the context of the string type.
@ -1415,7 +1415,7 @@ class SQuADDataset(SourceDataset, TextBaseDataset):
num_shards (int, optional): Number of shards that the dataset will be divided into. Default: None.
When this argument is specified, `num_samples` reflects the maximum sample number of per shard.
shard_id (int, optional): The shard ID within `num_shards`. Default: None. This
shard_id (int, optional): The shard ID within `num_shards` . Default: None. This
argument can only be specified when `num_shards` is also specified.
cache (DatasetCache, optional): Use tensor caching service to speed up dataset processing. More details:
`Single-Node Data Cache <https://www.mindspore.cn/tutorials/experts/en/master/dataset/cache.html>`_ .
@ -1509,7 +1509,7 @@ class TextFileDataset(SourceDataset, TextBaseDataset):
num_parallel_workers (int, optional): Number of workers to read the data.
Default: None, number set in the config.
shuffle (Union[bool, Shuffle], optional): Perform reshuffling of the data every epoch.
Default: `Shuffle.GLOBAL`. Bool type and Shuffle enum are both supported to pass in.
Default: `Shuffle.GLOBAL` . Bool type and Shuffle enum are both supported to pass in.
If shuffle is False, no shuffling will be performed.
If shuffle is True, performs global shuffle.
There are three levels of shuffling, desired shuffle enum defined by mindspore.dataset.Shuffle.
@ -1520,7 +1520,7 @@ class TextFileDataset(SourceDataset, TextBaseDataset):
num_shards (int, optional): Number of shards that the dataset will be divided into. Default: None.
When this argument is specified, `num_samples` reflects the maximum sample number of per shard.
shard_id (int, optional): The shard ID within `num_shards`. Default: None. This
shard_id (int, optional): The shard ID within `num_shards` . Default: None. This
argument can only be specified when `num_shards` is also specified.
cache (DatasetCache, optional): Use tensor caching service to speed up dataset processing. More details:
`Single-Node Data Cache <https://www.mindspore.cn/tutorials/experts/en/master/dataset/cache.html>`_ .
@ -1555,7 +1555,7 @@ class UDPOSDataset(SourceDataset, TextBaseDataset):
"""
A source dataset that reads and parses UDPOS dataset.
The generated dataset has three columns: :py:obj:`[word, universal, stanford]`,
The generated dataset has three columns: :py:obj:`[word, universal, stanford]` ,
and the data type of three columns is string.
Args:
@ -1576,7 +1576,7 @@ class UDPOSDataset(SourceDataset, TextBaseDataset):
num_shards (int, optional): Number of shards that the dataset will be divided into. Default: None.
When this argument is specified, `num_samples` reflects the max sample number of per shard.
shard_id (int, optional): The shard ID within `num_shards`. Default: None. This
shard_id (int, optional): The shard ID within `num_shards` . Default: None. This
argument can only be specified when `num_shards` is also specified.
num_parallel_workers (int, optional): Number of workers to read the data.
Default: None, number set in the config.
@ -1631,7 +1631,7 @@ class WikiTextDataset(SourceDataset, TextBaseDataset):
"""
A source dataset that reads and parses WikiText2 and WikiText103 datasets.
The generated dataset has one column :py:obj:`[text]`, and
The generated dataset has one column :py:obj:`[text]` , and
the tensor of column `text` is of the string type.
Args:
@ -1652,7 +1652,7 @@ class WikiTextDataset(SourceDataset, TextBaseDataset):
num_shards (int, optional): Number of shards that the dataset will be divided into. Default: None.
When this argument is specified, 'num_samples' reflects the max sample number of per shard.
shard_id (int, optional): The shard ID within `num_shards`. Default: None. This
shard_id (int, optional): The shard ID within `num_shards` . Default: None. This
argument can only be specified when `num_shards` is also specified.
cache (DatasetCache, optional): Use tensor caching service to speed up dataset processing. More details:
`Single-Node Data Cache <https://www.mindspore.cn/tutorials/experts/en/master/dataset/cache.html>`_ .
@ -1718,7 +1718,7 @@ class YahooAnswersDataset(SourceDataset, TextBaseDataset):
"""
A source dataset that reads and parses the YahooAnswers dataset.
The generated dataset has four columns :py:obj:`[class, title, content, answer]`, whose data type is string.
The generated dataset has four columns :py:obj:`[class, title, content, answer]` , whose data type is string.
Args:
dataset_dir (str): Path to the root directory that contains the dataset.
@ -1741,7 +1741,7 @@ class YahooAnswersDataset(SourceDataset, TextBaseDataset):
num_shards (int, optional): Number of shards that the dataset will be divided into. Default: None.
When this argument is specified, `num_samples` reflects the maximum sample number of per shard.
shard_id (int, optional): The shard ID within `num_shards`. Default: None. This
shard_id (int, optional): The shard ID within `num_shards` . Default: None. This
argument can only be specified when `num_shards` is also specified.
cache (DatasetCache, optional): Use tensor caching service to speed up dataset processing. More details:
`Single-Node Data Cache <https://www.mindspore.cn/tutorials/experts/en/master/dataset/cache.html>`_ .
@ -1812,7 +1812,7 @@ class YelpReviewDataset(SourceDataset, TextBaseDataset):
"""
A source dataset that reads and parses Yelp Review Polarity and Yelp Review Full dataset.
The generated dataset has two columns: :py:obj:`[label, text]`, and the data type of two columns is string.
The generated dataset has two columns: :py:obj:`[label, text]` , and the data type of two columns is string.
Args:
dataset_dir (str): Path to the root directory that contains the dataset.
@ -1833,7 +1833,7 @@ class YelpReviewDataset(SourceDataset, TextBaseDataset):
- Shuffle.FILES: Shuffle files only.
num_shards (int, optional): Number of shards that the dataset will be divided into. Default: None.
When this argument is specified, `num_samples` reflects the max sample number of per shard.
shard_id (int, optional): The shard ID within `num_shards`. Default: None. This
shard_id (int, optional): The shard ID within `num_shards` . Default: None. This
argument can only be specified when `num_shards` is also specified.
num_parallel_workers (int, optional): Number of workers to read the data.
Default: None, number set in the config.

10
mindspore/python/mindspore/dataset/engine/datasets_user_defined.py
View File

@ -523,7 +523,7 @@ class GeneratorDataset(MappableDataset, UnionBaseDataset):
num_shards (int, optional): Number of shards that the dataset will be divided into. Default: None.
Random accessible input is required. When this argument is specified, `num_samples` reflects the maximum
sample number of per shard.
shard_id (int, optional): The shard ID within `num_shards`. Default: None. This argument must be specified only
shard_id (int, optional): The shard ID within `num_shards` . Default: None. This argument must be specified only
when num_shards is also specified. Random accessible input is required.
python_multiprocessing (bool, optional): Parallelize Python operations with multiple worker process. This
option could be beneficial if the Python operation is computational heavy. Default: True.
@ -542,8 +542,8 @@ class GeneratorDataset(MappableDataset, UnionBaseDataset):
Note:
- Input `source` accepts user-defined Python functions (PyFuncs), Do not add network computing operators from
mindspore.nn and mindspore.ops or others into this `source`.
- This dataset can take in a `sampler`. `sampler` and `shuffle` are mutually exclusive.
mindspore.nn and mindspore.ops or others into this `source` .
- This dataset can take in a `sampler` . `sampler` and `shuffle` are mutually exclusive.
The table below shows what input arguments are allowed and their expected behavior.
.. list-table:: Expected Order Behavior of Using `sampler` and `shuffle`
@ -851,11 +851,11 @@ class NumpySlicesDataset(GeneratorDataset):
Default: None, expected order behavior shown in the table below.
num_shards (int, optional): Number of shards that the dataset will be divided into. Default: None.
When this argument is specified, `num_samples` reflects the max sample number of per shard.
shard_id (int, optional): The shard ID within `num_shards`. Default: None. This argument must be specified only
shard_id (int, optional): The shard ID within `num_shards` . Default: None. This argument must be specified only
when num_shards is also specified.
Note:
- This dataset can take in a `sampler`. `sampler` and `shuffle` are mutually exclusive.
- This dataset can take in a `sampler` . `sampler` and `shuffle` are mutually exclusive.
The table below shows what input arguments are allowed and their expected behavior.
.. list-table:: Expected Order Behavior of Using `sampler` and `shuffle`

164
mindspore/python/mindspore/dataset/engine/datasets_vision.py
View File

@ -110,11 +110,11 @@ class Caltech101Dataset(GeneratorDataset):
"""
A source dataset that reads and parses Caltech101 dataset.
The columns of the generated dataset depend on the value of `target_type`.
The columns of the generated dataset depend on the value of `target_type` .
- When `target_type` is 'category', the columns are :py:obj:`[image, category]`.
- When `target_type` is 'annotation', the columns are :py:obj:`[image, annotation]`.
- When `target_type` is 'all', the columns are :py:obj:`[image, category, annotation]`.
- When `target_type` is 'category', the columns are :py:obj:`[image, category]` .
- When `target_type` is 'annotation', the columns are :py:obj:`[image, annotation]` .
- When `target_type` is 'all', the columns are :py:obj:`[image, category, annotation]` .
The tensor of column :py:obj:`image` is of the uint8 type.
The tensor of column :py:obj:`category` is of the uint32 type.
@ -139,7 +139,7 @@ class Caltech101Dataset(GeneratorDataset):
num_shards (int, optional): Number of shards that the dataset will be divided
into. Default: None. When this argument is specified, `num_samples` reflects
the maximum sample number of per shard.
shard_id (int, optional): The shard ID within `num_shards`. Default: None. This
shard_id (int, optional): The shard ID within `num_shards` . Default: None. This
argument can only be specified when `num_shards` is also specified.
Raises:
@ -153,7 +153,7 @@ class Caltech101Dataset(GeneratorDataset):
ValueError: If `num_parallel_workers` exceeds the max thread numbers.
Note:
- This dataset can take in a `sampler`. `sampler` and `shuffle` are mutually exclusive.
- This dataset can take in a `sampler` . `sampler` and `shuffle` are mutually exclusive.
The table below shows what input arguments are allowed and their expected behavior.
.. list-table:: Expected Order Behavior of Using `sampler` and `shuffle`
@ -282,7 +282,7 @@ class Caltech256Dataset(MappableDataset, VisionBaseDataset):
"""
A source dataset that reads and parses Caltech256 dataset.
The generated dataset has two columns: :py:obj:`[image, label]`.
The generated dataset has two columns: :py:obj:`[image, label]` .
The tensor of column :py:obj:`image` is of the uint8 type.
The tensor of column :py:obj:`label` is of the uint32 type.
@ -300,7 +300,7 @@ class Caltech256Dataset(MappableDataset, VisionBaseDataset):
num_shards (int, optional): Number of shards that the dataset will be divided
into. Default: None. When this argument is specified, `num_samples` reflects
the maximum sample number of per shard.
shard_id (int, optional): The shard ID within `num_shards`. Default: None. This
shard_id (int, optional): The shard ID within `num_shards` . Default: None. This
argument can only be specified when `num_shards` is also specified.
cache (DatasetCache, optional): Use tensor caching service to speed up dataset processing. More details:
`Single-Node Data Cache <https://www.mindspore.cn/tutorials/experts/en/master/dataset/cache.html>`_ .
@ -317,7 +317,7 @@ class Caltech256Dataset(MappableDataset, VisionBaseDataset):
ValueError: If `num_parallel_workers` exceeds the max thread numbers.
Note:
- This dataset can take in a `sampler`. `sampler` and `shuffle` are mutually exclusive.
- This dataset can take in a `sampler` . `sampler` and `shuffle` are mutually exclusive.
The table below shows what input arguments are allowed and their expected behavior.
.. list-table:: Expected Order Behavior of Using `sampler` and `shuffle`
@ -411,7 +411,7 @@ class CelebADataset(MappableDataset, VisionBaseDataset):
A source dataset that reads and parses CelebA dataset.
Only support to read `list_attr_celeba.txt` currently, which is the attribute annotations of the dataset.
The generated dataset has two columns: :py:obj:`[image, attr]`.
The generated dataset has two columns: :py:obj:`[image, attr]` .
The tensor of column :py:obj:`image` is of the uint8 type.
The tensor of column :py:obj:`attr` is of the uint32 type and one hot encoded.
@ -430,7 +430,7 @@ class CelebADataset(MappableDataset, VisionBaseDataset):
num_shards (int, optional): Number of shards that the dataset will be divided
into. Default: None. When this argument is specified, `num_samples` reflects
the maximum sample number of per shard.
shard_id (int, optional): The shard ID within `num_shards`. Default: None. This
shard_id (int, optional): The shard ID within `num_shards` . Default: None. This
argument can only be specified when `num_shards` is also specified.
cache (DatasetCache, optional): Use tensor caching service to speed up dataset processing. More details:
`Single-Node Data Cache <https://www.mindspore.cn/tutorials/experts/en/master/dataset/cache.html>`_ .
@ -449,7 +449,7 @@ class CelebADataset(MappableDataset, VisionBaseDataset):
ValueError: If `usage` is not 'train', 'valid', 'test' or 'all'.
Note:
- This dataset can take in a `sampler`. `sampler` and `shuffle` are mutually exclusive.
- This dataset can take in a `sampler` . `sampler` and `shuffle` are mutually exclusive.
The table below shows what input arguments are allowed and their expected behavior.
.. list-table:: Expected Order Behavior of Using `sampler` and `shuffle`
@ -581,7 +581,7 @@ class Cifar10Dataset(MappableDataset, VisionBaseDataset):
A source dataset that reads and parses Cifar10 dataset.
This api only supports parsing Cifar10 file in binary version now.
The generated dataset has two columns :py:obj:`[image, label]`.
The generated dataset has two columns :py:obj:`[image, label]` .
The tensor of column :py:obj:`image` is of the uint8 type.
The tensor of column :py:obj:`label` is a scalar of the uint32 type.
@ -601,7 +601,7 @@ class Cifar10Dataset(MappableDataset, VisionBaseDataset):
num_shards (int, optional): Number of shards that the dataset will be divided
into. Default: None. When this argument is specified, `num_samples` reflects
the maximum sample number of per shard.
shard_id (int, optional): The shard ID within `num_shards`. Default: None. This
shard_id (int, optional): The shard ID within `num_shards` . Default: None. This
argument can only be specified when `num_shards` is also specified.
cache (DatasetCache, optional): Use tensor caching service to speed up dataset processing. More details:
`Single-Node Data Cache <https://www.mindspore.cn/tutorials/experts/en/master/dataset/cache.html>`_ .
@ -618,7 +618,7 @@ class Cifar10Dataset(MappableDataset, VisionBaseDataset):
ValueError: If `usage` is not 'train', 'test' or 'all'.
Note:
- This dataset can take in a `sampler`. `sampler` and `shuffle` are mutually exclusive.
- This dataset can take in a `sampler` . `sampler` and `shuffle` are mutually exclusive.
The table below shows what input arguments are allowed and their expected behavior.
.. list-table:: Expected Order Behavior of Using `sampler` and `shuffle`
@ -713,7 +713,7 @@ class Cifar100Dataset(MappableDataset, VisionBaseDataset):
"""
A source dataset that reads and parses Cifar100 dataset.
The generated dataset has three columns :py:obj:`[image, coarse_label, fine_label]`.
The generated dataset has three columns :py:obj:`[image, coarse_label, fine_label]` .
The tensor of column :py:obj:`image` is of the uint8 type.
The tensor of column :py:obj:`coarse_label` and :py:obj:`fine_labels` are each a scalar of uint32 type.
@ -733,7 +733,7 @@ class Cifar100Dataset(MappableDataset, VisionBaseDataset):
num_shards (int, optional): Number of shards that the dataset will be divided
into. Default: None. When this argument is specified, 'num_samples' reflects
the maximum sample number of per shard.
shard_id (int, optional): The shard ID within `num_shards`. Default: None. This
shard_id (int, optional): The shard ID within `num_shards` . Default: None. This
argument can only be specified when `num_shards` is also specified.
cache (DatasetCache, optional): Use tensor caching service to speed up dataset processing. More details:
`Single-Node Data Cache <https://www.mindspore.cn/tutorials/experts/en/master/dataset/cache.html>`_ .
@ -750,7 +750,7 @@ class Cifar100Dataset(MappableDataset, VisionBaseDataset):
ValueError: If `usage` is not 'train', 'test' or 'all'.
Note:
- This dataset can take in a `sampler`. `sampler` and `shuffle` are mutually exclusive.
- This dataset can take in a `sampler` . `sampler` and `shuffle` are mutually exclusive.
The table below shows what input arguments are allowed and their expected behavior.
.. list-table:: Expected Order Behavior of Using `sampler` and shuffle
@ -839,7 +839,7 @@ class CityscapesDataset(MappableDataset, VisionBaseDataset):
"""
A source dataset that reads and parses Cityscapes dataset.
The generated dataset has two columns :py:obj:`[image, task]`.
The generated dataset has two columns :py:obj:`[image, task]` .
The tensor of column :py:obj:`image` is of the uint8 type.
The tensor of column :py:obj:`task` is of the uint8 type if task is not 'polygon' otherwise task is
a string tensor with serialize json.
@ -863,7 +863,7 @@ class CityscapesDataset(MappableDataset, VisionBaseDataset):
num_shards (int, optional): Number of shards that the dataset will be divided
into. Default: None. When this argument is specified, `num_samples` reflects
the max sample number of per shard.
shard_id (int, optional): The shard ID within `num_shards`. Default: None. This
shard_id (int, optional): The shard ID within `num_shards` . Default: None. This
argument can only be specified when `num_shards` is also specified.
cache (DatasetCache, optional): Use tensor caching service to speed up dataset processing. More details:
`Single-Node Data Cache <https://www.mindspore.cn/tutorials/experts/en/master/dataset/cache.html>`_ .
@ -883,7 +883,7 @@ class CityscapesDataset(MappableDataset, VisionBaseDataset):
ValueError: If `shard_id` is invalid (< 0 or >= `num_shards`).
Note:
- This dataset can take in a `sampler`. `sampler` and `shuffle` are mutually exclusive.
- This dataset can take in a `sampler` . `sampler` and `shuffle` are mutually exclusive.
The table below shows what input arguments are allowed and their expected behavior.
.. list-table:: Expected Order Behavior of Using `sampler` and `shuffle`
@ -1031,13 +1031,13 @@ class CocoDataset(MappableDataset, VisionBaseDataset):
num_shards (int, optional): Number of shards that the dataset will be divided
into. Default: None. When this argument is specified, `num_samples` reflects
the maximum sample number of per shard.
shard_id (int, optional): The shard ID within `num_shards`. Default: None. This
shard_id (int, optional): The shard ID within `num_shards` . Default: None. This
argument can only be specified when `num_shards` is also specified.
cache (DatasetCache, optional): Use tensor caching service to speed up dataset processing. More details:
`Single-Node Data Cache <https://www.mindspore.cn/tutorials/experts/en/master/dataset/cache.html>`_ .
Default: None, which means no cache is used.
extra_metadata(bool, optional): Flag to add extra meta-data to row. If True, an additional column will be
output at the end :py:obj:`[_meta-filename, dtype=string]`. Default: False.
output at the end :py:obj:`[_meta-filename, dtype=string]` . Default: False.
decrypt (callable, optional): Image decryption function, which accepts the path of the encrypted image file
and returns the decrypted bytes data. Default: None, no decryption.
@ -1094,7 +1094,7 @@ class CocoDataset(MappableDataset, VisionBaseDataset):
- Column '[_meta-filename, dtype=string]' won't be output unless an explicit rename dataset op is added
to remove the prefix('_meta-').
- CocoDataset doesn't support PKSampler.
- This dataset can take in a `sampler`. `sampler` and `shuffle` are mutually exclusive.
- This dataset can take in a `sampler` . `sampler` and `shuffle` are mutually exclusive.
The table below shows what input arguments are allowed and their expected behavior.
.. list-table:: Expected Order Behavior of Using `sampler` and `shuffle`
@ -1256,7 +1256,7 @@ class DIV2KDataset(MappableDataset, VisionBaseDataset):
"""
A source dataset that reads and parses DIV2KDataset dataset.
The generated dataset has two columns :py:obj:`[hr_image, lr_image]`.
The generated dataset has two columns :py:obj:`[hr_image, lr_image]` .
The tensor of column :py:obj:`hr_image` and the tensor of column :py:obj:`lr_image` are of the uint8 type.
Args:
@ -1280,7 +1280,7 @@ class DIV2KDataset(MappableDataset, VisionBaseDataset):
num_shards (int, optional): Number of shards that the dataset will be divided
into. Default: None. When this argument is specified, `num_samples` reflects
the max sample number of per shard.
shard_id (int, optional): The shard ID within `num_shards`. Default: None. This
shard_id (int, optional): The shard ID within `num_shards` . Default: None. This
argument can only be specified when `num_shards` is also specified.
cache (DatasetCache, optional): Use tensor caching service to speed up dataset processing. More details:
`Single-Node Data Cache <https://www.mindspore.cn/tutorials/experts/en/master/dataset/cache.html>`_ .
@ -1302,7 +1302,7 @@ class DIV2KDataset(MappableDataset, VisionBaseDataset):
ValueError: If `shard_id` is invalid (< 0 or >= `num_shards`).
Note:
- This dataset can take in a `sampler`. `sampler` and `shuffle` are mutually exclusive.
- This dataset can take in a `sampler` . `sampler` and `shuffle` are mutually exclusive.
The table below shows what input arguments are allowed and their expected behavior.
.. list-table:: Expected Order Behavior of Using `sampler` and `shuffle`
@ -1479,7 +1479,7 @@ class EMnistDataset(MappableDataset, VisionBaseDataset):
ValueError: If `shard_id` is invalid (< 0 or >= `num_shards`).
Note:
- This dataset can take in a `sampler`. `sampler` and `shuffle` are mutually exclusive.
- This dataset can take in a `sampler` . `sampler` and `shuffle` are mutually exclusive.
The table below shows what input arguments are allowed and their expected behavior.
.. list-table:: Expected Order Behavior of Using `sampler` and `shuffle`
@ -1701,7 +1701,7 @@ class FashionMnistDataset(MappableDataset, VisionBaseDataset):
ValueError: If `shard_id` is invalid (< 0 or >= `num_shards`).
Note:
- This dataset can take in a `sampler`. `sampler` and `shuffle` are mutually exclusive.
- This dataset can take in a `sampler` . `sampler` and `shuffle` are mutually exclusive.
The table below shows what input arguments are allowed and their expected behavior.
.. list-table:: Expected Order Behavior of Using `sampler` and `shuffle`
@ -1788,7 +1788,7 @@ class FlickrDataset(MappableDataset, VisionBaseDataset):
"""
A source dataset that reads and parses Flickr8k and Flickr30k dataset.
The generated dataset has two columns :py:obj:`[image, annotation]`.
The generated dataset has two columns :py:obj:`[image, annotation]` .
The tensor of column :py:obj:`image` is of the uint8 type.
The tensor of column :py:obj:`annotation` is a tensor which contains 5 annotations string,
such as ["a", "b", "c", "d", "e"].
@ -1808,7 +1808,7 @@ class FlickrDataset(MappableDataset, VisionBaseDataset):
num_shards (int, optional): Number of shards that the dataset will be divided
into. Default: None. When this argument is specified, `num_samples` reflects
the max sample number of per shard.
shard_id (int, optional): The shard ID within `num_shards`. Default: None. This
shard_id (int, optional): The shard ID within `num_shards` . Default: None. This
argument can only be specified when `num_shards` is also specified.
cache (DatasetCache, optional): Use tensor caching service to speed up dataset processing. More details:
`Single-Node Data Cache <https://www.mindspore.cn/tutorials/experts/en/master/dataset/cache.html>`_ .
@ -1826,7 +1826,7 @@ class FlickrDataset(MappableDataset, VisionBaseDataset):
ValueError: If `shard_id` is invalid (< 0 or >= `num_shards`).
Note:
- This dataset can take in a `sampler`. `sampler` and `shuffle` are mutually exclusive.
- This dataset can take in a `sampler` . `sampler` and `shuffle` are mutually exclusive.
The table below shows what input arguments are allowed and their expected behavior.
.. list-table:: Expected Order Behavior of Using `sampler` and `shuffle`
@ -2195,7 +2195,7 @@ class ImageFolderDataset(MappableDataset, VisionBaseDataset):
A source dataset that reads images from a tree of directories.
All images within one folder have the same label.
The generated dataset has two columns: :py:obj:`[image, label]`.
The generated dataset has two columns: :py:obj:`[image, label]` .
The tensor of column :py:obj:`image` is of the uint8 type.
The tensor of column :py:obj:`label` is of a scalar of uint32 type.
@ -2219,7 +2219,7 @@ class ImageFolderDataset(MappableDataset, VisionBaseDataset):
num_shards (int, optional): Number of shards that the dataset will be divided
into. Default: None. When this argument is specified, `num_samples` reflects
the maximum sample number of per shard.
shard_id (int, optional): The shard ID within `num_shards`. Default: None. This
shard_id (int, optional): The shard ID within `num_shards` . Default: None. This
argument can only be specified when `num_shards` is also specified.
cache (DatasetCache, optional): Use tensor caching service to speed up dataset processing. More details:
`Single-Node Data Cache <https://www.mindspore.cn/tutorials/experts/en/master/dataset/cache.html>`_ .
@ -2239,7 +2239,7 @@ class ImageFolderDataset(MappableDataset, VisionBaseDataset):
Note:
- The shape of the image column is [image_size] if decode flag is False, or [H,W,C] otherwise.
- This dataset can take in a `sampler`. `sampler` and `shuffle` are mutually exclusive.
- This dataset can take in a `sampler` . `sampler` and `shuffle` are mutually exclusive.
The table below shows what input arguments are allowed and their expected behavior.
.. list-table:: Expected Order Behavior of Using `sampler` and `shuffle`
@ -2331,8 +2331,8 @@ class KITTIDataset(MappableDataset):
A source dataset that reads and parses the KITTI dataset.
When usage is "train", the generated dataset has multiple columns: :py:obj:`[image, label, truncated,
occluded, alpha, bbox, dimensions, location, rotation_y]`; When usage is "test", the generated dataset
has only one column: :py:obj:`[image]`.
occluded, alpha, bbox, dimensions, location, rotation_y]` ; When usage is "test", the generated dataset
has only one column: :py:obj:`[image]` .
The tensor of column :py:obj:`image` is of the uint8 type.
The tensor of column :py:obj:`label` is of the uint32 type.
The tensor of column :py:obj:`truncated` is of the float32 type.
@ -2345,8 +2345,8 @@ class KITTIDataset(MappableDataset):
Args:
dataset_dir (str): Path to the root directory that contains the dataset.
usage (str, optional): Usage of this dataset, can be `train` or `test`. `train` will read 7481
train samples, `test` will read from 7518 test samples without label. Default: None, will use `train`.
usage (str, optional): Usage of this dataset, can be `train` or `test` . `train` will read 7481
train samples, `test` will read from 7518 test samples without label. Default: None, will use `train` .
num_samples (int, optional): The number of images to be included in the dataset.
Default: None, will include all images.
num_parallel_workers (int, optional): Number of workers to read the data.
@ -2371,10 +2371,10 @@ class KITTIDataset(MappableDataset):
RuntimeError: If `num_shards` is specified but `shard_id` is None.
RuntimeError: If `shard_id` is specified but `num_shards` is None.
ValueError: If `dataset_dir` is not exist.
ValueError: If `shard_id` is invalid (< 0 or >= num_shards).
ValueError: If `shard_id` is invalid (< 0 or >= `num_shards` ).
Note:
- This dataset can take in a `sampler`. `sampler` and `shuffle` are mutually exclusive.
- This dataset can take in a `sampler` . `sampler` and `shuffle` are mutually exclusive.
The table below shows what input arguments are allowed and their expected behavior.
.. list-table:: Expected Order Behavior of Using `sampler` and `shuffle`
@ -2512,7 +2512,7 @@ class KMnistDataset(MappableDataset, VisionBaseDataset):
ValueError: If `shard_id` is invalid (< 0 or >= `num_shards`).
Note:
- This dataset can take in a `sampler`. `sampler` and `shuffle` are mutually exclusive.
- This dataset can take in a `sampler` . `sampler` and `shuffle` are mutually exclusive.
The table below shows what input arguments are allowed and their expected behavior.
.. list-table:: Expected Order Behavior of Using `sampler` and `shuffle`
@ -2598,7 +2598,7 @@ class LFWDataset(MappableDataset, VisionBaseDataset):
A source dataset that reads and parses the LFW dataset.
When task is 'people', the generated dataset has two columns: :py:obj:`[image, label]`;
When task is 'pairs', the generated dataset has three columns: :py:obj:`[image1, image2, label]`.
When task is 'pairs', the generated dataset has three columns: :py:obj:`[image1, image2, label]` .
The tensor of column :py:obj:`image` is of the uint8 type.
The tensor of column :py:obj:`image1` is of the uint8 type.
The tensor of column :py:obj:`image2` is of the uint8 type.
@ -2635,7 +2635,7 @@ class LFWDataset(MappableDataset, VisionBaseDataset):
RuntimeError: If sampler and sharding are specified at the same time.
RuntimeError: If num_shards is specified but shard_id is None.
RuntimeError: If shard_id is specified but num_shards is None.
ValueError: If shard_id is invalid (< 0 or >= num_shards).
ValueError: If shard_id is invalid (< 0 or >= `num_shards` ).
.. list-table:: Expected Order Behavior of Using 'sampler' and 'shuffle'
:widths: 25 25 50
@ -2752,14 +2752,14 @@ class LSUNDataset(MappableDataset, VisionBaseDataset):
"""
A source dataset that reads and parses the LSUN dataset.
The generated dataset has two columns: :py:obj:`[image, label]`.
The generated dataset has two columns: :py:obj:`[image, label]` .
The tensor of column :py:obj:`image` is of the uint8 type.
The tensor of column :py:obj:`label` is of a scalar of uint32 type.
Args:
dataset_dir (str): Path to the root directory that contains the dataset.
usage (str, optional): Usage of this dataset, can be `train`, `test`, `valid` or `all`
Default: None, will be set to `all`.
usage (str, optional): Usage of this dataset, can be `train` , `test` , `valid` or `all`
Default: None, will be set to `all` .
classes(Union[str, list[str]], optional): Choose the specific classes to load. Default: None, means loading
all classes in root directory.
num_samples (int, optional): The number of images to be included in the dataset.
@ -2785,7 +2785,7 @@ class LSUNDataset(MappableDataset, VisionBaseDataset):
RuntimeError: If 'sampler' and sharding are specified at the same time.
RuntimeError: If 'num_shards' is specified but 'shard_id' is None.
RuntimeError: If 'shard_id' is specified but 'num_shards' is None.
ValueError: If 'shard_id' is invalid (< 0 or >= num_shards).
ValueError: If 'shard_id' is invalid (< 0 or >= `num_shards` ).
ValueError: If 'usage' or 'classes' is invalid (not in specific types).
.. list-table:: Expected Order Behavior of Using 'sampler' and 'shuffle'
@ -2883,7 +2883,7 @@ class ManifestDataset(MappableDataset, VisionBaseDataset):
"""
A source dataset for reading images from a Manifest file.
The generated dataset has two columns: :py:obj:`[image, label]`.
The generated dataset has two columns: :py:obj:`[image, label]` .
The tensor of column :py:obj:`image` is of the uint8 type.
The tensor of column :py:obj:`label` is of a scalar of uint64 type.
@ -2905,7 +2905,7 @@ class ManifestDataset(MappableDataset, VisionBaseDataset):
num_shards (int, optional): Number of shards that the dataset will be divided
into. Default: None. When this argument is specified, `num_samples` reflects
the max number of samples per shard.
shard_id (int, optional): The shard ID within `num_shards`. Default: None. This
shard_id (int, optional): The shard ID within `num_shards` . Default: None. This
argument can only be specified when `num_shards` is also specified.
cache (DatasetCache, optional): Use tensor caching service to speed up dataset processing. More details:
`Single-Node Data Cache <https://www.mindspore.cn/tutorials/experts/en/master/dataset/cache.html>`_ .
@ -2923,7 +2923,7 @@ class ManifestDataset(MappableDataset, VisionBaseDataset):
Note:
- The shape of the image column is [image_size] if decode flag is False, or [H,W,C] otherwise.
- This dataset can take in a `sampler`. `sampler` and `shuffle` are mutually exclusive.
- This dataset can take in a `sampler` . `sampler` and `shuffle` are mutually exclusive.
The table below shows what input arguments are allowed and their expected behavior.
.. list-table:: Expected Order Behavior of Using `sampler` and `shuffle`
@ -3003,7 +3003,7 @@ class MnistDataset(MappableDataset, VisionBaseDataset):
"""
A source dataset that reads and parses the MNIST dataset.
The generated dataset has two columns :py:obj:`[image, label]`.
The generated dataset has two columns :py:obj:`[image, label]` .
The tensor of column :py:obj:`image` is of the uint8 type.
The tensor of column :py:obj:`label` is a scalar of the uint32 type.
@ -3022,7 +3022,7 @@ class MnistDataset(MappableDataset, VisionBaseDataset):
dataset. Default: None, expected order behavior shown in the table below.
num_shards (int, optional): Number of shards that the dataset will be divided into. Default: None.
When this argument is specified, `num_samples` reflects the maximum sample number of per shard.
shard_id (int, optional): The shard ID within `num_shards`. Default: None. This
shard_id (int, optional): The shard ID within `num_shards` . Default: None. This
argument can only be specified when `num_shards` is also specified.
cache (DatasetCache, optional): Use tensor caching service to speed up dataset processing. More details:
`Single-Node Data Cache <https://www.mindspore.cn/tutorials/experts/en/master/dataset/cache.html>`_ .
@ -3039,7 +3039,7 @@ class MnistDataset(MappableDataset, VisionBaseDataset):
ValueError: If `shard_id` is invalid (< 0 or >= `num_shards`).
Note:
- This dataset can take in a `sampler`. `sampler` and `shuffle` are mutually exclusive.
- This dataset can take in a `sampler` . `sampler` and `shuffle` are mutually exclusive.
The table below shows what input arguments are allowed and their expected behavior.
.. list-table:: Expected Order Behavior of Using `sampler` and `shuffle`
@ -3125,7 +3125,7 @@ class OmniglotDataset(MappableDataset):
"""
A source dataset that reads and parses the Omniglot dataset.
The generated dataset has two columns :py:obj:`[image, label]`.
The generated dataset has two columns :py:obj:`[image, label]` .
The tensor of column :py:obj:`image` is of the uint8 type.
The tensor of column :py:obj:`label` is a scalar of the uint32 type.
@ -3405,7 +3405,7 @@ class Places365Dataset(MappableDataset, VisionBaseDataset):
"""
A source dataset that reads and parses the Places365 dataset.
The generated dataset has two columns :py:obj:`[image, label]`.
The generated dataset has two columns :py:obj:`[image, label]` .
The tensor of column :py:obj:`image` is of the uint8 type.
The tensor of column :py:obj:`label` is of the uint32 type.
@ -3425,7 +3425,7 @@ class Places365Dataset(MappableDataset, VisionBaseDataset):
dataset. Default: None, expected order behavior shown in the table below.
num_shards (int, optional): Number of shards that the dataset will be divided into. Default: None.
When this argument is specified, `num_samples` reflects the max sample number of per shard.
shard_id (int, optional): The shard ID within `num_shards`. Default: None. This
shard_id (int, optional): The shard ID within `num_shards` . Default: None. This
argument can only be specified when `num_shards` is also specified.
cache (DatasetCache, optional): Use tensor caching service to speed up dataset processing. More details:
`Single-Node Data Cache <https://www.mindspore.cn/tutorials/experts/en/master/dataset/cache.html>`_ .
@ -3545,7 +3545,7 @@ class QMnistDataset(MappableDataset, VisionBaseDataset):
"""
A source dataset that reads and parses the QMNIST dataset.
The generated dataset has two columns :py:obj:`[image, label]`.
The generated dataset has two columns :py:obj:`[image, label]` .
The tensor of column :py:obj:`image` is of the uint8 type.
The tensor of column :py:obj:`label` is of the uint32 type.
@ -3565,7 +3565,7 @@ class QMnistDataset(MappableDataset, VisionBaseDataset):
dataset. Default: None, expected order behavior shown in the table below.
num_shards (int, optional): Number of shards that the dataset will be divided into. Default: None.
When this argument is specified, `num_samples` reflects the maximum sample number of per shard.
shard_id (int, optional): The shard ID within `num_shards`. Default: None. This
shard_id (int, optional): The shard ID within `num_shards` . Default: None. This
argument can only be specified when `num_shards` is also specified.
cache (DatasetCache, optional): Use tensor caching service to speed up dataset processing. More details:
`Single-Node Data Cache <https://www.mindspore.cn/tutorials/experts/en/master/dataset/cache.html>`_ .
@ -3581,7 +3581,7 @@ class QMnistDataset(MappableDataset, VisionBaseDataset):
ValueError: If `num_parallel_workers` exceeds the max thread numbers.
Note:
- This dataset can take in a `sampler`. `sampler` and `shuffle` are mutually exclusive.
- This dataset can take in a `sampler` . `sampler` and `shuffle` are mutually exclusive.
The table below shows what input arguments are allowed and their expected behavior.
.. list-table:: Expected Order Behavior of Using `sampler` and `shuffle`
@ -3691,7 +3691,7 @@ class RandomDataset(SourceDataset, VisionBaseDataset):
num_shards (int, optional): Number of shards that the dataset will be divided
into. Default: None. When this argument is specified, 'num_samples' reflects
the maximum sample number of per shard.
shard_id (int, optional): The shard ID within `num_shards`. Default: None. This
shard_id (int, optional): The shard ID within `num_shards` . Default: None. This
argument can only be specified when `num_shards` is also specified.
Raises:
@ -3823,7 +3823,7 @@ class SBDataset(GeneratorDataset):
num_shards (int, optional): Number of shards that the dataset will be divided
into. Default: None. When this argument is specified, `num_samples` reflects
the max sample number of per shard.
shard_id (int, optional): The shard ID within `num_shards`. Default: None. This
shard_id (int, optional): The shard ID within `num_shards` . Default: None. This
argument can only be specified when `num_shards` is also specified.
Raises:
@ -3935,7 +3935,7 @@ class SBUDataset(MappableDataset, VisionBaseDataset):
"""
A source dataset that reads and parses the SBU dataset.
The generated dataset has two columns :py:obj:`[image, caption]`.
The generated dataset has two columns :py:obj:`[image, caption]` .
The tensor of column :py:obj:`image` is of the uint8 type.
The tensor of column :py:obj:`caption` is of the string type.
@ -3952,7 +3952,7 @@ class SBUDataset(MappableDataset, VisionBaseDataset):
dataset. Default: None, expected order behavior shown in the table below.
num_shards (int, optional): Number of shards that the dataset will be divided into. Default: None.
When this argument is specified, `num_samples` reflects the max sample number of per shard.
shard_id (int, optional): The shard ID within `num_shards`. Default: None. This
shard_id (int, optional): The shard ID within `num_shards` . Default: None. This
argument can only be specified when `num_shards` is also specified.
cache (DatasetCache, optional): Use tensor caching service to speed up dataset processing. More details:
`Single-Node Data Cache <https://www.mindspore.cn/tutorials/experts/en/master/dataset/cache.html>`_ .
@ -4050,7 +4050,7 @@ class SemeionDataset(MappableDataset, VisionBaseDataset):
"""
A source dataset that reads and parses Semeion dataset.
The generated dataset has two columns :py:obj:`[image, label]`.
The generated dataset has two columns :py:obj:`[image, label]` .
The tensor of column :py:obj:`image` is of the uint8 type.
The tensor of column :py:obj:`label` is a scalar of the uint32 type.
@ -4067,7 +4067,7 @@ class SemeionDataset(MappableDataset, VisionBaseDataset):
num_shards (int, optional): Number of shards that the dataset will be divided
into. Default: None. When this argument is specified, `num_samples` reflects
the maximum sample number of per shard.
shard_id (int, optional): The shard ID within `num_shards`. Default: None. This
shard_id (int, optional): The shard ID within `num_shards` . Default: None. This
argument can only be specified when `num_shards` is also specified.
cache (DatasetCache, optional): Use tensor caching service to speed up dataset processing. More details:
`Single-Node Data Cache <https://www.mindspore.cn/tutorials/experts/en/master/dataset/cache.html>`_ .
@ -4083,7 +4083,7 @@ class SemeionDataset(MappableDataset, VisionBaseDataset):
ValueError: If `shard_id` is invalid (< 0 or >= `num_shards`).
Note:
- This dataset can take in a `sampler`. `sampler` and `shuffle` are mutually exclusive.
- This dataset can take in a `sampler` . `sampler` and `shuffle` are mutually exclusive.
The table below shows what input arguments are allowed and their expected behavior.
.. list-table:: Expected Order Behavior of Using `sampler` and `shuffle`
@ -4172,7 +4172,7 @@ class STL10Dataset(MappableDataset, VisionBaseDataset):
"""
A source dataset that reads and parses STL10 dataset.
The generated dataset has two columns: :py:obj:`[image, label]`.
The generated dataset has two columns: :py:obj:`[image, label]` .
The tensor of column :py:obj:`image` is of the uint8 type.
The tensor of column :py:obj:`label` is of a scalar of int32 type.
@ -4195,7 +4195,7 @@ class STL10Dataset(MappableDataset, VisionBaseDataset):
num_shards (int, optional): Number of shards that the dataset will be divided
into. Default: None. When this argument is specified, 'num_samples' reflects
the max sample number of per shard.
shard_id (int, optional): The shard ID within `num_shards`. Default: None. This
shard_id (int, optional): The shard ID within `num_shards` . Default: None. This
argument can only be specified when `num_shards` is also specified.
cache (DatasetCache, optional): Use tensor caching service to speed up dataset processing. More details:
`Single-Node Data Cache <https://www.mindspore.cn/tutorials/experts/en/master/dataset/cache.html>`_ .
@ -4344,7 +4344,7 @@ class SVHNDataset(GeneratorDataset):
"""
A source dataset that reads and parses SVHN dataset.
The generated dataset has two columns: :py:obj:`[image, label]`.
The generated dataset has two columns: :py:obj:`[image, label]` .
The tensor of column :py:obj:`image` is of the uint8 type.
The tensor of column :py:obj:`label` is of a scalar of uint32 type.
@ -4360,7 +4360,7 @@ class SVHNDataset(GeneratorDataset):
input is required. Default: None, expected order behavior shown in the table below.
num_shards (int, optional): Number of shards that the dataset will be divided into. Default: None.
When this argument is specified, 'num_samples' reflects the max sample number of per shard.
shard_id (int, optional): The shard ID within `num_shards`. Default: None. This argument must be specified only
shard_id (int, optional): The shard ID within `num_shards` . Default: None. This argument must be specified only
when num_shards is also specified.
Raises:
@ -4453,7 +4453,7 @@ class USPSDataset(SourceDataset, VisionBaseDataset):
"""
A source dataset that reads and parses the USPS dataset.
The generated dataset has two columns: :py:obj:`[image, label]`.
The generated dataset has two columns: :py:obj:`[image, label]` .
The tensor of column :py:obj:`image` is of the uint8 type.
The tensor of column :py:obj:`label` is of the uint32 type.
@ -4478,7 +4478,7 @@ class USPSDataset(SourceDataset, VisionBaseDataset):
num_shards (int, optional): Number of shards that the dataset will be divided into. Default: None.
When this argument is specified, `num_samples` reflects the max sample number of per shard.
shard_id (int, optional): The shard ID within `num_shards`. Default: None. This
shard_id (int, optional): The shard ID within `num_shards` . Default: None. This
argument can only be specified when `num_shards` is also specified.
cache (DatasetCache, optional): Use tensor caching service to speed up dataset processing. More details:
`Single-Node Data Cache <https://www.mindspore.cn/tutorials/experts/en/master/dataset/cache.html>`_ .
@ -4550,9 +4550,9 @@ class VOCDataset(MappableDataset, VisionBaseDataset):
The generated dataset with different task setting has different output columns:
- task = :py:obj:`Detection`, output columns: :py:obj:`[image, dtype=uint8]`, :py:obj:`[bbox, dtype=float32]`, \
:py:obj:`[label, dtype=uint32]`, :py:obj:`[difficult, dtype=uint32]`, :py:obj:`[truncate, dtype=uint32]`.
- task = :py:obj:`Segmentation`, output columns: :py:obj:`[image, dtype=uint8]`, :py:obj:`[target,dtype=uint8]`.
- task = :py:obj:`Detection` , output columns: :py:obj:`[image, dtype=uint8]` , :py:obj:`[bbox, dtype=float32]` , \
:py:obj:`[label, dtype=uint32]` , :py:obj:`[difficult, dtype=uint32]` , :py:obj:`[truncate, dtype=uint32]` .
- task = :py:obj:`Segmentation` , output columns: :py:obj:`[image, dtype=uint8]` , :py:obj:`[target,dtype=uint8]` .
Args:
dataset_dir (str): Path to the root directory that contains the dataset.
@ -4577,7 +4577,7 @@ class VOCDataset(MappableDataset, VisionBaseDataset):
num_shards (int, optional): Number of shards that the dataset will be divided
into. Default: None. When this argument is specified, `num_samples` reflects
the maximum sample number of per shard.
shard_id (int, optional): The shard ID within `num_shards`. Default: None. This
shard_id (int, optional): The shard ID within `num_shards` . Default: None. This
argument can only be specified when `num_shards` is also specified.
cache (DatasetCache, optional): Use tensor caching service to speed up dataset processing. More details:
`Single-Node Data Cache <https://www.mindspore.cn/tutorials/experts/en/master/dataset/cache.html>`_ .
@ -4590,8 +4590,8 @@ class VOCDataset(MappableDataset, VisionBaseDataset):
Raises:
RuntimeError: If `dataset_dir` does not contain data files.
RuntimeError: If xml of Annotations is an invalid format.
RuntimeError: If xml of Annotations loss attribution of `object`.
RuntimeError: If xml of Annotations loss attribution of `bndbox`.
RuntimeError: If xml of Annotations loss attribution of `object` .
RuntimeError: If xml of Annotations loss attribution of `bndbox` .
RuntimeError: If `sampler` and `shuffle` are specified at the same time.
RuntimeError: If `sampler` and `num_shards`/`shard_id` are specified at the same time.
RuntimeError: If `num_shards` is specified but `shard_id` is None.
@ -4605,7 +4605,7 @@ class VOCDataset(MappableDataset, VisionBaseDataset):
Note:
- Column '[_meta-filename, dtype=string]' won't be output unless an explicit rename dataset op
is added to remove the prefix('_meta-').
- This dataset can take in a `sampler`. `sampler` and `shuffle` are mutually exclusive.
- This dataset can take in a `sampler` . `sampler` and `shuffle` are mutually exclusive.
The table below shows what input arguments are allowed and their expected behavior.
.. list-table:: Expected Order Behavior of Using `sampler` and `shuffle`
@ -4775,7 +4775,7 @@ class WIDERFaceDataset(MappableDataset, VisionBaseDataset):
Default: None, expected order behavior shown in the table below.
num_shards (int, optional): Number of shards that the dataset will be divided into. Default: None.
When this argument is specified, `num_samples` reflects the maximum sample number of per shard.
shard_id (int, optional): The shard ID within `num_shards`. Default: None. This argument can only be specified
shard_id (int, optional): The shard ID within `num_shards` . Default: None. This argument can only be specified
when `num_shards` is also specified.
cache (DatasetCache, optional): Use tensor caching service to speed up dataset processing. More details:
`Single-Node Data Cache <https://www.mindspore.cn/tutorials/experts/en/master/dataset/cache.html>`_ .
@ -4794,7 +4794,7 @@ class WIDERFaceDataset(MappableDataset, VisionBaseDataset):
ValueError: If `dataset_dir` is not exist.
Note:
- This dataset can take in a `sampler`. `sampler` and `shuffle` are mutually exclusive.
- This dataset can take in a `sampler` . `sampler` and `shuffle` are mutually exclusive.
The table below shows what input arguments are allowed and their expected behavior.
.. list-table:: Expected Order Behavior of Using `sampler` and `shuffle`

42
mindspore/python/mindspore/dataset/engine/graphdata.py
View File

@ -38,7 +38,7 @@ from .datasets_user_defined import GeneratorDataset
class SamplingStrategy(IntEnum):
"""
Specifies the sampling strategy when execute `get_sampled_neighbors`.
Specifies the sampling strategy when execute `get_sampled_neighbors` .
- RANDOM: Random sampling with replacement.
- EDGE_WEIGHT: Sampling with edge weight as probability.
@ -55,7 +55,7 @@ DE_C_INTER_SAMPLING_STRATEGY = {
class OutputFormat(IntEnum):
"""
Specifies the output storage format when execute `get_all_neighbors`.
Specifies the output storage format when execute `get_all_neighbors` .
- NORMAL: Normal format.
- COO: COO format.
@ -229,7 +229,7 @@ class GraphData:
@check_gnn_get_all_neighbors
def get_all_neighbors(self, node_list, neighbor_type, output_format=OutputFormat.NORMAL):
"""
Get `neighbor_type` neighbors of the nodes in `node_list`.
Get `neighbor_type` neighbors of the nodes in `node_list` .
We try to use the following example to illustrate the definition of these formats. 1 represents connected
between two nodes, and 0 represents not connected.
@ -397,7 +397,7 @@ class GraphData:
@check_gnn_get_neg_sampled_neighbors
def get_neg_sampled_neighbors(self, node_list, neg_neighbor_num, neg_neighbor_type):
"""
Get `neg_neighbor_type` negative sampled neighbors of the nodes in `node_list`.
Get `neg_neighbor_type` negative sampled neighbors of the nodes in `node_list` .
Args:
node_list (Union[list, numpy.ndarray]): The given list of nodes.
@ -425,7 +425,7 @@ class GraphData:
@check_gnn_get_node_feature
def get_node_feature(self, node_list, feature_types):
"""
Get `feature_types` feature of the nodes in `node_list`.
Get `feature_types` feature of the nodes in `node_list` .
Args:
node_list (Union[list, numpy.ndarray]): The given list of nodes.
@ -454,7 +454,7 @@ class GraphData:
@check_gnn_get_edge_feature
def get_edge_feature(self, edge_list, feature_types):
"""
Get `feature_types` feature of the edges in `edge_list`.
Get `feature_types` feature of the edges in `edge_list` .
Args:
edge_list (Union[list, numpy.ndarray]): The given list of edges.
@ -532,8 +532,8 @@ class Graph(GraphData):
A graph object for storing Graph structure and feature data, and provide capabilities such as graph sampling.
This class supports init graph With input numpy array data, which represent node, edge and its features.
If working mode is `local`, there is no need to specify input arguments like `working_mode`, `hostname`, `port`,
`num_client`, `auto_shutdown`.
If working mode is `local` , there is no need to specify input arguments like `working_mode` , `hostname` , `port` ,
`num_client` , `auto_shutdown` .
Args:
edges(Union[list, numpy.ndarray]): edges of graph in COO format with shape [2, num_edges].
@ -722,7 +722,7 @@ class Graph(GraphData):
@check_gnn_get_all_neighbors
def get_all_neighbors(self, node_list, neighbor_type, output_format=OutputFormat.NORMAL):
"""
Get `neighbor_type` neighbors of the nodes in `node_list`.
Get `neighbor_type` neighbors of the nodes in `node_list` .
We try to use the following example to illustrate the definition of these formats. 1 represents connected
between two nodes, and 0 represents not connected.
@ -901,7 +901,7 @@ class Graph(GraphData):
@check_gnn_get_neg_sampled_neighbors
def get_neg_sampled_neighbors(self, node_list, neg_neighbor_num, neg_neighbor_type):
"""
Get `neg_neighbor_type` negative sampled neighbors of the nodes in `node_list`.
Get `neg_neighbor_type` negative sampled neighbors of the nodes in `node_list` .
Args:
node_list (Union[list, numpy.ndarray]): The given list of nodes.
@ -933,7 +933,7 @@ class Graph(GraphData):
@check_gnn_get_node_feature
def get_node_feature(self, node_list, feature_types):
"""
Get `feature_types` feature of the nodes in `node_list`.
Get `feature_types` feature of the nodes in `node_list` .
Args:
node_list (Union[list, numpy.ndarray]): The given list of nodes.
@ -969,7 +969,7 @@ class Graph(GraphData):
@check_gnn_get_edge_feature
def get_edge_feature(self, edge_list, feature_types):
"""
Get `feature_types` feature of the edges in `edge_list`.
Get `feature_types` feature of the edges in `edge_list` .
Args:
edge_list (Union[list, numpy.ndarray]): The given list of edges.
@ -1261,11 +1261,11 @@ class InMemoryGraphDataset(GeneratorDataset):
"""
Basic Dataset for loading graph into memory.
Recommended to Implement your own dataset with inheriting this class, and implement your own method like `process`,
`save` and `load`, refer source code of `ArgoverseDataset` for how to implement your own dataset. When init your own
dataset like ArgoverseDataset, The executed process like follows. Check if there are already processed data under
given `data_dir`, if so will call `load` method to load it directly, otherwise it will call `process` method to
create graphs and call `save` method to save the graphs into `save_dir`.
Recommended to Implement your own dataset with inheriting this class, and implement your own method like `process` ,
`save` and `load` , refer source code of `ArgoverseDataset` for how to implement your own dataset. When init your
own dataset like ArgoverseDataset, The executed process like follows. Check if there are already processed data
under given `data_dir` , if so will call `load` method to load it directly, otherwise it will call `process` method
to create graphs and call `save` method to save the graphs into `save_dir` .
You can access graph in created dataset using `graphs = my_dataset.graphs` and also you can iterate dataset
and get data using `my_dataset.create_tuple_iterator()` (in this way you need to implement methods like
@ -1275,10 +1275,10 @@ class InMemoryGraphDataset(GeneratorDataset):
Args:
data_dir (str): directory for loading dataset, here contains origin format data and will be loaded in
`process` method.
save_dir (str): relative directory for saving processed dataset, this directory is under `data_dir`.
save_dir (str): relative directory for saving processed dataset, this directory is under `data_dir` .
Default: './processed'.
column_names (Union[str, list[str]], optional): single column name or list of column names of the dataset,
num of column name should be equal to num of item in return data when implement method like `__getitem__`.
num of column name should be equal to num of item in return data when implement method like `__getitem__` .
Default: 'graph'.
num_samples (int, optional): The number of samples to be included in the dataset. Default: None, all samples.
num_parallel_workers (int, optional): Number of subprocesses used to fetch the dataset in parallel. Default: 1.
@ -1287,7 +1287,7 @@ class InMemoryGraphDataset(GeneratorDataset):
num_shards (int, optional): Number of shards that the dataset will be divided into. Default: None.
When this argument is specified, `num_samples` reflects the max
sample number of per shard.
shard_id (int, optional): The shard ID within `num_shards`. Default: None. This argument must be specified only
shard_id (int, optional): The shard ID within `num_shards` . Default: None. This argument must be specified only
when num_shards is also specified.
python_multiprocessing (bool, optional): Parallelize Python operations with multiple worker process. This
option could be beneficial if the Python operation is computational heavy. Default: True.
@ -1382,7 +1382,7 @@ class ArgoverseDataset(InMemoryGraphDataset):
data_dir (str): directory for loading dataset, here contains origin format data and will be loaded in
`process` method.
column_names (Union[str, list[str]], optional): single column name or list of column names of the dataset,
num of column name should be equal to num of item in return data when implement method like `__getitem__`,
num of column name should be equal to num of item in return data when implement method like `__getitem__` ,
recommend to specify it with
`column_names=["edge_index", "x", "y", "cluster", "valid_len", "time_step_len"]` like the following example.
num_parallel_workers (int, optional): Number of subprocesses used to fetch the dataset in parallel. Default: 1.

8
mindspore/python/mindspore/dataset/engine/samplers.py
View File

@ -328,11 +328,11 @@ class DistributedSampler(BuiltinSampler):
Args:
num_shards (int): Number of shards to divide the dataset into.
shard_id (int): Shard ID of the current shard, which should within the range of [0, `num_shards`-1].
shard_id (int): Shard ID of the current shard, which should within the range of [0, `num_shards` - 1].
shuffle (bool, optional): If True, the indices are shuffled, otherwise it will not be shuffled. Default: True.
num_samples (int, optional): The number of samples to draw. Default: None, which means sample all elements.
offset(int, optional): The starting shard ID where the elements in the dataset are sent to, which
should be no more than `num_shards`. This parameter is only valid when a ConcatDataset takes
should be no more than `num_shards` . This parameter is only valid when a ConcatDataset takes
a DistributedSampler as its sampler. It will affect the number of samples of per shard.
Default: -1, which means each shard has the same number of samples.
@ -344,8 +344,8 @@ class DistributedSampler(BuiltinSampler):
TypeError: If `offset` is not of type int.
ValueError: If `num_samples` is a negative value.
RuntimeError: If `num_shards` is not a positive value.
RuntimeError: If `shard_id` is smaller than 0 or equal to `num_shards` or larger than `num_shards`.
RuntimeError: If `offset` is greater than `num_shards`.
RuntimeError: If `shard_id` is smaller than 0 or equal to `num_shards` or larger than `num_shards` .
RuntimeError: If `offset` is greater than `num_shards` .
Examples:
>>> # creates a distributed sampler with 10 shards in total. This shard is shard 5.

2
mindspore/python/mindspore/dataset/engine/serializer_deserializer.py
View File

@ -64,7 +64,7 @@ def deserialize(input_dict=None, json_filepath=None):
Args:
input_dict (dict): A Python dictionary containing a serialized dataset graph. Default: None.
json_filepath (str): A path to the JSON file containing dataset graph.
User can obtain this file by calling API `mindspore.dataset.serialize()`. Default: None.
User can obtain this file by calling API `mindspore.dataset.serialize()` . Default: None.
Returns:
de.Dataset or None if error occurs.

4
mindspore/python/mindspore/dataset/engine/validators.py
View File

@ -1549,7 +1549,7 @@ def check_zip(method):
def check_zip_dataset(method):
"""check the input arguments of zip method in `Dataset`."""
"""check the input arguments of zip method in `Dataset` ."""
@wraps(method)
def new_method(self, *args, **kwargs):
@ -1562,7 +1562,7 @@ def check_zip_dataset(method):
def check_concat(method):
"""check the input arguments of concat method in `Dataset`."""
"""check the input arguments of concat method in `Dataset` ."""
@wraps(method)
def new_method(self, *args, **kwargs):

2
mindspore/python/mindspore/dataset/text/__init__.py
View File

@ -36,7 +36,7 @@ The data transform operation can be executed in the data processing pipeline or
- Pipeline mode is generally used to process datasets. For examples, please refer to
`introduction to data processing pipeline <https://www.mindspore.cn/docs/en/master/api_python/
mindspore.dataset.html#introduction-to-data-processing-pipeline>`_.
mindspore.dataset.html#introduction-to-data-processing-pipeline>`_ .
- Eager mode is generally used for scattered samples. Examples of text preprocessing are as follows:
.. code-block::

18
mindspore/python/mindspore/dataset/text/transforms.py
View File

@ -669,7 +669,7 @@ class WordpieceTokenizer(TextTensorOperation):
with_offsets (bool, optional): Whether to return the offsets of tokens. Default: False.
Raises:
TypeError: If `vocab` is not of type :class:`mindspore.dataset.text.Vocab`.
TypeError: If `vocab` is not of type :class:`mindspore.dataset.text.Vocab` .
TypeError: If `suffix_indicator` is not of type str.
TypeError: If `max_bytes_per_token` is not of type int.
TypeError: If `unknown_token` is not of type str.
@ -730,10 +730,10 @@ if platform.system().lower() != 'windows':
Args:
lower_case (bool, optional): Whether to perform lowercase processing on the text. If True, will fold the
text to lower case and strip accented characters. If False, will only perform normalization on the
text, with mode specified by `normalization_form`. Default: False.
text, with mode specified by `normalization_form` . Default: False.
keep_whitespace (bool, optional): If True, the whitespace will be kept in the output. Default: False.
normalization_form (NormalizeForm, optional):
`Unicode normalization forms <http://unicode.org/reports/tr15/>`_, only valid when `lower_case`
`Unicode normalization forms <http://unicode.org/reports/tr15/>`_ , only valid when `lower_case`
is False, can be NormalizeForm.NONE, NormalizeForm.NFC, NormalizeForm.NFKC, NormalizeForm.NFD or
NormalizeForm.NFKD. Default: NormalizeForm.NONE.
@ -750,7 +750,7 @@ if platform.system().lower() != 'windows':
Raises:
TypeError: If `lower_case` is not of type bool.
TypeError: If `keep_whitespace` is not of type bool.
TypeError: If `normalization_form` is not of type :class:`mindspore.dataset.text.NormalizeForm`.
TypeError: If `normalization_form` is not of type :class:`mindspore.dataset.text.NormalizeForm` .
TypeError: If `preserve_unused_token` is not of type bool.
TypeError: If `with_offsets` is not of type bool.
RuntimeError: If dtype of input Tensor is not str.
@ -817,10 +817,10 @@ if platform.system().lower() != 'windows':
output. Default: '[UNK]'.
lower_case (bool, optional): Whether to perform lowercase processing on the text. If True, will fold the
text to lower case and strip accented characters. If False, will only perform normalization on the
text, with mode specified by `normalization_form`. Default: False.
text, with mode specified by `normalization_form` . Default: False.
keep_whitespace (bool, optional): If True, the whitespace will be kept in the output. Default: False.
normalization_form (NormalizeForm, optional):
`Unicode normalization forms <http://unicode.org/reports/tr15/>`_, only valid when `lower_case`
`Unicode normalization forms <http://unicode.org/reports/tr15/>`_ , only valid when `lower_case`
is False, can be NormalizeForm.NONE, NormalizeForm.NFC, NormalizeForm.NFKC, NormalizeForm.NFD or
NormalizeForm.NFKD. Default: NormalizeForm.NONE.
@ -835,14 +835,14 @@ if platform.system().lower() != 'windows':
with_offsets (bool, optional): Whether to return the offsets of tokens. Default: False.
Raises:
TypeError: If `vocab` is not of type :class:`mindspore.dataset.text.Vocab`.
TypeError: If `vocab` is not of type :class:`mindspore.dataset.text.Vocab` .
TypeError: If `suffix_indicator` is not of type str.
TypeError: If `max_bytes_per_token` is not of type int.
ValueError: If `max_bytes_per_token` is negative.
TypeError: If `unknown_token` is not of type str.
TypeError: If `lower_case` is not of type bool.
TypeError: If `keep_whitespace` is not of type bool.
TypeError: If `normalization_form` is not of type :class:`mindspore.dataset.text.NormalizeForm`.
TypeError: If `normalization_form` is not of type :class:`mindspore.dataset.text.NormalizeForm` .
TypeError: If `preserve_unused_token` is not of type bool.
TypeError: If `with_offsets` is not of type bool.
@ -904,7 +904,7 @@ if platform.system().lower() != 'windows':
class CaseFold(TextTensorOperation):
"""
Apply case fold operation on UTF-8 string tensor, which is aggressive that can convert more characters into
lower case than :func:`str.lower`. For supported normalization forms, please refer to
lower case than :func:`str.lower` . For supported normalization forms, please refer to
`ICU_Normalizer2 <https://unicode-org.github.io/icu-docs/apidoc/released/icu4c/classicu_1_1Normalizer2.html>`_ .
Note:

18
mindspore/python/mindspore/dataset/text/utils.py
View File

@ -75,7 +75,7 @@ class FastText(cde.FastText):
Args:
file_path (str): Path of the file that contains the vectors. The shuffix of pre-trained vector sets
must be `*.vec`.
must be `*.vec` .
max_vectors (int, optional): This can be used to limit the number of pre-trained vectors loaded.
Most pre-trained vector sets are sorted in the descending order of word frequency. Thus, in
situations where the entire set doesn't fit in memory, or is not needed for another reason,
@ -111,7 +111,7 @@ class GloVe(cde.GloVe):
Args:
file_path (str): Path of the file that contains the vectors. The format of pre-trained vector sets
must be `glove.6B.*.txt`.
must be `glove.6B.*.txt` .
max_vectors (int, optional): This can be used to limit the number of pre-trained vectors loaded.
Most pre-trained vector sets are sorted in the descending order of word frequency. Thus, in
situations where the entire set doesn't fit in memory, or is not needed for another reason,
@ -136,7 +136,7 @@ class GloVe(cde.GloVe):
class JiebaMode(IntEnum):
"""
An enumeration for :class:`mindspore.dataset.text.JiebaTokenizer`.
An enumeration for :class:`mindspore.dataset.text.JiebaTokenizer` .
Possible enumeration values are: JiebaMode.MIX, JiebaMode.MP, JiebaMode.HMM.
@ -314,7 +314,7 @@ class SentencePieceVocab:
class SPieceTokenizerLoadType(IntEnum):
"""
An enumeration for loading type of :class:`mindspore.dataset.text.SentencePieceTokenizer`.
An enumeration for loading type of :class:`mindspore.dataset.text.SentencePieceTokenizer` .
Possible enumeration values are: SPieceTokenizerLoadType.FILE, SPieceTokenizerLoadType.MODEL.
@ -328,7 +328,7 @@ class SPieceTokenizerLoadType(IntEnum):
class SPieceTokenizerOutType(IntEnum):
"""
An enumeration for :class:`mindspore.dataset.text.SentencePieceTokenizer`.
An enumeration for :class:`mindspore.dataset.text.SentencePieceTokenizer` .
Possible enumeration values are: SPieceTokenizerOutType.STRING, SPieceTokenizerOutType.INT.
@ -591,14 +591,14 @@ class Vocab:
def to_bytes(array, encoding='utf8'):
"""
Convert NumPy array of `str` to array of `bytes` by encoding each element based on charset `encoding`.
Convert NumPy array of `str` to array of `bytes` by encoding each element based on charset `encoding` .
Args:
array (numpy.ndarray): Array of `str` type representing strings.
encoding (str): Indicating the charset for encoding. Default: 'utf8'.
Returns:
numpy.ndarray, NumPy array of `bytes`.
numpy.ndarray, NumPy array of `bytes` .
Examples:
>>> import numpy as np
@ -618,14 +618,14 @@ def to_bytes(array, encoding='utf8'):
def to_str(array, encoding='utf8'):
"""
Convert NumPy array of `bytes` to array of `str` by decoding each element based on charset `encoding`.
Convert NumPy array of `bytes` to array of `str` by decoding each element based on charset `encoding` .
Args:
array (numpy.ndarray): Array of `bytes` type representing strings.
encoding (str): Indicating the charset for decoding. Default: 'utf8'.
Returns:
numpy.ndarray, NumPy array of `str`.
numpy.ndarray, NumPy array of `str` .
Examples:
>>> import numpy as np

4
mindspore/python/mindspore/dataset/transforms/c_transforms.py
View File

@ -211,10 +211,10 @@ class Slice(TensorOperation):
`slice <https://docs.python.org/3.7/library/functions.html?highlight=slice#slice>`_ object along the
first dimension. Similar to start:stop:step.
4. :py:obj:`None`: Slice the whole dimension. Similar to :py:obj:`[:]` in Python indexing.
5. :py:obj:`Ellipsis`: Slice the whole dimension, same result with `None`.
5. :py:obj:`Ellipsis`: Slice the whole dimension, same result with `None` .
Raises:
TypeError: If `slices` is not of type int, list[int], :py:obj:`slice`, :py:obj:`None` or :py:obj:`Ellipsis`.
TypeError: If `slices` is not of type int, list[int], :py:obj:`slice` , :py:obj:`None` or :py:obj:`Ellipsis` .
Supported Platforms:
``CPU``

6
mindspore/python/mindspore/dataset/transforms/transforms.py
View File

@ -844,10 +844,10 @@ class Slice(TensorOperation):
`slice <https://docs.python.org/3.7/library/functions.html?highlight=slice#slice>`_ object along the
first dimension. Similar to start:stop:step.
4. :py:obj:`None`: Slice the whole dimension. Similar to :py:obj:`[:]` in Python indexing.
5. :py:obj:`Ellipsis`: Slice the whole dimension, same result with `None`.
5. :py:obj:`Ellipsis`: Slice the whole dimension, same result with `None` .
Raises:
TypeError: If `slices` is not of type int, list[int], :py:obj:`slice`, :py:obj:`None` or :py:obj:`Ellipsis`.
TypeError: If `slices` is not of type int, list[int], :py:obj:`slice` , :py:obj:`None` or :py:obj:`Ellipsis` .
Supported Platforms:
``CPU``
@ -893,7 +893,7 @@ class TypeCast(TensorOperation):
to be cast to.
Raises:
TypeError: If `data_type` is not of MindSpore data type bool, int, float, string or type :class:`numpy.dtype`.
TypeError: If `data_type` is not of MindSpore data type bool, int, float, string or type :class:`numpy.dtype` .
Supported Platforms:
``CPU`` ``Ascend`` ``GPU``

2
mindspore/python/mindspore/dataset/vision/__init__.py
View File

@ -44,7 +44,7 @@ The data transform operation can be executed in the data processing pipeline or
- Pipeline mode is generally used to process datasets. For examples, please refer to
`introduction to data processing pipeline <https://www.mindspore.cn/docs/en/master/api_python/
mindspore.dataset.html#introduction-to-data-processing-pipeline>`_.
mindspore.dataset.html#introduction-to-data-processing-pipeline>`_ .
- Eager mode is generally used for scattered samples. Examples of image preprocessing are as follows:
.. code-block::

38
mindspore/python/mindspore/dataset/vision/c_transforms.py
View File

@ -181,7 +181,7 @@ class AdjustGamma(ImageTensorOperation):
class AutoAugment(ImageTensorOperation):
"""
Apply AutoAugment data augmentation method based on
`AutoAugment: Learning Augmentation Strategies from Data <https://arxiv.org/pdf/1805.09501.pdf>`_.
`AutoAugment: Learning Augmentation Strategies from Data <https://arxiv.org/pdf/1805.09501.pdf>`_ .
This operation works only with 3-channel RGB images.
Args:
@ -298,7 +298,7 @@ class BoundingBoxAugment(ImageTensorOperation):
Raises:
TypeError: If `transform` is not an image processing operation
in :class:`mindspore.dataset.vision.c_transforms`.
in :class:`mindspore.dataset.vision.c_transforms` .
TypeError: If `ratio` is not of type float.
ValueError: If `ratio` is not in range [0.0, 1.0].
RuntimeError: If given bounding box is invalid.
@ -418,7 +418,7 @@ class ConvertColor(ImageTensorOperation):
- ConvertMode.COLOR_RGBA2GRAY, Convert RGBA image to GRAY image.
Raises:
TypeError: If `convert_mode` is not of type :class:`mindspore.dataset.vision.c_transforms.ConvertMode`.
TypeError: If `convert_mode` is not of type :class:`mindspore.dataset.vision.c_transforms.ConvertMode` .
RuntimeError: If given tensor shape is not <H, W> or <H, W, C>.
Supported Platforms:
@ -499,7 +499,7 @@ class CutMixBatch(ImageTensorOperation):
prob (float, optional): The probability by which CutMix is applied to each image, range: [0, 1]. Default: 1.0.
Raises:
TypeError: If `image_batch_format` is not of type :class:`mindspore.dataset.vision.ImageBatchFormat`.
TypeError: If `image_batch_format` is not of type :class:`mindspore.dataset.vision.ImageBatchFormat` .
TypeError: If `alpha` is not of type float.
TypeError: If `prob` is not of type float.
ValueError: If `alpha` is less than or equal 0.
@ -922,7 +922,7 @@ class Pad(ImageTensorOperation):
Raises:
TypeError: If `padding` is not of type int or Sequence[int].
TypeError: If `fill_value` is not of type int or tuple[int].
TypeError: If `padding_mode` is not of type :class:`mindspore.dataset.vision.Border`.
TypeError: If `padding_mode` is not of type :class:`mindspore.dataset.vision.Border` .
ValueError: If `padding` is negative.
ValueError: If `fill_value` is not in range [0, 255].
RuntimeError: If given tensor shape is not <H, W> or <H, W, C>.
@ -1034,7 +1034,7 @@ class RandomAffine(ImageTensorOperation):
TypeError: If `translate` is not of type sequence.
TypeError: If `scale` is not of type sequence.
TypeError: If `shear` is not of type int, float or sequence.
TypeError: If `resample` is not of type :class:`mindspore.dataset.vision.Inter`.
TypeError: If `resample` is not of type :class:`mindspore.dataset.vision.Inter` .
TypeError: If `fill_value` is not of type int or tuple[int].
ValueError: If `degrees` is negative.
ValueError: If `translate` is not in range [-1.0, 1.0].
@ -1303,7 +1303,7 @@ class RandomCrop(ImageTensorOperation):
TypeError: If `padding` is not of type int or Sequence[int].
TypeError: If `pad_if_needed` is not of type boolean.
TypeError: If `fill_value` is not of type int or tuple[int].
TypeError: If `padding_mode` is not of type :class:`mindspore.dataset.vision.Border`.
TypeError: If `padding_mode` is not of type :class:`mindspore.dataset.vision.Border` .
ValueError: If `size` is not positive.
ValueError: If `padding` is negative.
ValueError: If `fill_value` is not in range [0, 255].
@ -1346,7 +1346,7 @@ class RandomCrop(ImageTensorOperation):
class RandomCropDecodeResize(ImageTensorOperation):
"""
A combination of `Crop`, `Decode` and `Resize`. It will get better performance for JPEG images. This operation
A combination of `Crop` , `Decode` and `Resize` . It will get better performance for JPEG images. This operation
will crop the input image at a random location, decode the cropped image in RGB mode, and resize the decoded image.
Args:
@ -1378,7 +1378,7 @@ class RandomCropDecodeResize(ImageTensorOperation):
TypeError: If `size` is not of type int or Sequence[int].
TypeError: If `scale` is not of type tuple or list.
TypeError: If `ratio` is not of type tuple or list.
TypeError: If `interpolation` is not of type :class:`mindspore.dataset.vision.Inter`.
TypeError: If `interpolation` is not of type :class:`mindspore.dataset.vision.Inter` .
TypeError: If `max_attempts` is not of type int.
ValueError: If `size` is not positive.
ValueError: If `scale` is negative.
@ -1472,7 +1472,7 @@ class RandomCropWithBBox(ImageTensorOperation):
TypeError: If `padding` is not of type int or Sequence[int].
TypeError: If `pad_if_needed` is not of type boolean.
TypeError: If `fill_value` is not of type int or tuple[int].
TypeError: If `padding_mode` is not of type :class:`mindspore.dataset.vision.Border`.
TypeError: If `padding_mode` is not of type :class:`mindspore.dataset.vision.Border` .
ValueError: If `size` is not positive.
ValueError: If `padding` is negative.
ValueError: If `fill_value` is not in range [0, 255].
@ -1742,7 +1742,7 @@ class RandomResizedCrop(ImageTensorOperation):
TypeError: If `size` is not of type int or Sequence[int].
TypeError: If `scale` is not of type tuple or list.
TypeError: If `ratio` is not of type tuple or list.
TypeError: If `interpolation` is not of type :class:`mindspore.dataset.vision.Inter`.
TypeError: If `interpolation` is not of type :class:`mindspore.dataset.vision.Inter` .
TypeError: If `max_attempts` is not of type int.
ValueError: If `size` is not positive.
ValueError: If `scale` is negative.
@ -1808,7 +1808,7 @@ class RandomResizedCropWithBBox(ImageTensorOperation):
TypeError: If `size` is not of type int or Sequence[int].
TypeError: If `scale` is not of type tuple or list.
TypeError: If `ratio` is not of type tuple or list.
TypeError: If `interpolation` is not of type :class:`mindspore.dataset.vision.Inter`.
TypeError: If `interpolation` is not of type :class:`mindspore.dataset.vision.Inter` .
TypeError: If `max_attempts` is not of type int.
ValueError: If `size` is not positive.
ValueError: If `scale` is negative.
@ -1958,7 +1958,7 @@ class RandomRotation(ImageTensorOperation):
Raises:
TypeError: If `degrees` is not of type int, float or sequence.
TypeError: If `resample` is not of type :class:`mindspore.dataset.vision.Inter`.
TypeError: If `resample` is not of type :class:`mindspore.dataset.vision.Inter` .
TypeError: If `expand` is not of type boolean.
TypeError: If `center` is not of type tuple.
TypeError: If `fill_value` is not of type int or tuple[int].
@ -2243,7 +2243,7 @@ class Resize(ImageTensorOperation):
Raises:
TypeError: If `size` is not of type int or Sequence[int].
TypeError: If `interpolation` is not of type :class:`mindspore.dataset.vision.Inter`.
TypeError: If `interpolation` is not of type :class:`mindspore.dataset.vision.Inter` .
ValueError: If `size` is not positive.
RuntimeError: If given tensor shape is not <H, W> or <H, W, C>.
@ -2291,7 +2291,7 @@ class ResizeWithBBox(ImageTensorOperation):
Raises:
TypeError: If `size` is not of type int or Sequence[int].
TypeError: If `interpolation` is not of type :class:`mindspore.dataset.vision.Inter`.
TypeError: If `interpolation` is not of type :class:`mindspore.dataset.vision.Inter` .
ValueError: If `size` is not positive.
RuntimeError: If given tensor shape is not <H, W> or <H, W, C>.
@ -2375,7 +2375,7 @@ class Rotate(ImageTensorOperation):
Raises:
TypeError: If `degrees` is not of type int or float.
TypeError: If `resample` is not of type :class:`mindspore.dataset.vision.Inter`.
TypeError: If `resample` is not of type :class:`mindspore.dataset.vision.Inter` .
TypeError: If `expand` is not of type bool.
TypeError: If `center` is not of type tuple.
TypeError: If `fill_value` is not of type int or tuple[int].
@ -2435,7 +2435,7 @@ class SlicePatches(ImageTensorOperation):
Raises:
TypeError: If `num_height` is not of type int.
TypeError: If `num_width` is not of type int.
TypeError: If `slice_mode` is not of type :class:`mindspore.dataset.vision.Inter`.
TypeError: If `slice_mode` is not of type :class:`mindspore.dataset.vision.Inter` .
TypeError: If `fill_value` is not of type int.
ValueError: If `num_height` is not positive.
ValueError: If `num_width` is not positive.
@ -2472,7 +2472,7 @@ class SlicePatches(ImageTensorOperation):
class SoftDvppDecodeRandomCropResizeJpeg(ImageTensorOperation):
"""
A combination of `Crop`, `Decode` and `Resize` using the simulation algorithm of Ascend series chip DVPP module.
A combination of `Crop` , `Decode` and `Resize` using the simulation algorithm of Ascend series chip DVPP module.
The usage scenario is consistent with SoftDvppDecodeResizeJpeg.
The input image size should be in range [32*32, 8192*8192].
@ -2568,7 +2568,7 @@ class UniformAugment(ImageTensorOperation):
Raises:
TypeError: If `transform` is not an image processing operation
in :class:`mindspore.dataset.vision.c_transforms`.
in :class:`mindspore.dataset.vision.c_transforms` .
TypeError: If `num_ops` is not of type int.
ValueError: If `num_ops` is not positive.

72
mindspore/python/mindspore/dataset/vision/py_transforms.py
View File

@ -165,7 +165,7 @@ class CenterCrop(py_transforms.PyTensorOperation):
Args:
size (Union[int, Sequence[int, int]]): The size of the cropped image.
If int is provided, a square of size (`size`, `size`) will be cropped with this value.
If int is provided, a square of size `(size, size)` will be cropped with this value.
If Sequence[int, int] is provided, its two elements will be taken as the cropped height and width.
Raises:
@ -210,7 +210,7 @@ class Cutout(py_transforms.PyTensorOperation):
Randomly cut out a certain number of square patches on the input numpy.ndarray image,
setting the pixel values in the patch to zero.
See `Improved Regularization of Convolutional Neural Networks with Cutout <https://arxiv.org/pdf/1708.04552.pdf>`_.
See `Improved Regularization of Convolutional Neural Networks with Cutout <https://arxiv.org/pdf/1708.04552.pdf>`_ .
Args:
length (int): The side length of square patches to be cut out.
@ -361,7 +361,7 @@ class FiveCrop(py_transforms.PyTensorOperation):
Args:
size (Union[int, Sequence[int, int]]): The size of the cropped image.
If int is provided, a square of size (`size`, `size`) will be cropped with this value.
If int is provided, a square of size `(size, size)` will be cropped with this value.
If Sequence[int, int] is provided, its two elements will be taken as the cropped height and width.
Raises:
@ -500,7 +500,7 @@ class HWC2CHW(py_transforms.PyTensorOperation):
If the input image is of shape <H, W>, it will remain unchanged.
Raises:
TypeError: If the input image is not of type :class:`numpy.ndarray`.
TypeError: If the input image is not of type :class:`numpy.ndarray` .
TypeError: If shape of the input image is not <H, W> or <H, W, C>.
Supported Platforms:
@ -578,12 +578,12 @@ class LinearTransformation(py_transforms.PyTensorOperation):
Args:
transformation_matrix (numpy.ndarray): A square transformation matrix in shape of (D, D), where
:math:`D = C \times H \times W`.
mean_vector (numpy.ndarray): A mean vector in shape of (D,), where :math:`D = C \times H \times W`.
:math:`D = C \times H \times W` .
mean_vector (numpy.ndarray): A mean vector in shape of (D,), where :math:`D = C \times H \times W` .
Raises:
TypeError: If `transformation_matrix` is not of type :class:`numpy.ndarray`.
TypeError: If `mean_vector` is not of type :class:`numpy.ndarray`.
TypeError: If `transformation_matrix` is not of type :class:`numpy.ndarray` .
TypeError: If `mean_vector` is not of type :class:`numpy.ndarray` .
Supported Platforms:
``CPU``
@ -630,8 +630,8 @@ class MixUp(py_transforms.PyTensorOperation):
Randomly mix up a batch of images together with its labels.
Each image will be multiplied by a random weight :math:`lambda` generated from the Beta distribution and then added
to another image multiplied by :math:`1 - lambda`. The same transformation will be applied to their labels with the
same value of :math:`lambda`. Make sure that the labels are one-hot encoded in advance.
to another image multiplied by :math:`1 - lambda` . The same transformation will be applied to their labels with the
same value of :math:`lambda` . Make sure that the labels are one-hot encoded in advance.
Args:
batch_size (int): The number of images in a batch.
@ -718,7 +718,7 @@ class Normalize(py_transforms.PyTensorOperation):
and be arranged in channel order.
Raises:
TypeError: If the input image is not of type :class:`numpy.ndarray`.
TypeError: If the input image is not of type :class:`numpy.ndarray` .
TypeError: If dimension of the input image is not 3.
NotImplementedError: If dtype of the input image is int.
ValueError: If lengths of `mean` and `std` are not equal.
@ -786,7 +786,7 @@ class NormalizePad(py_transforms.PyTensorOperation):
dtype (str): The dtype of the output image. Only "float32" and "float16" are supported. Default: "float32".
Raises:
TypeError: If the input image is not of type :class:`numpy.ndarray`.
TypeError: If the input image is not of type :class:`numpy.ndarray` .
TypeError: If dimension of the input image is not 3.
NotImplementedError: If dtype of the input image is int.
ValueError: If lengths of `mean` and `std` are not equal.
@ -860,7 +860,7 @@ class Pad(py_transforms.PyTensorOperation):
Raises:
TypeError: If `padding` is not of type int or Sequence[int, int].
TypeError: If `fill_value` is not of type int or tuple[int, int, int].
TypeError: If `padding_mode` is not of type :class:`mindspore.dataset.vision.Border`.
TypeError: If `padding_mode` is not of type :class:`mindspore.dataset.vision.Border` .
ValueError: If `padding` is negative.
ValueError: If `fill_value` is not in range of [0, 255].
RuntimeError: If shape of the input image is not <H, W> or <H, W, C>.
@ -907,7 +907,7 @@ class RandomAffine(py_transforms.PyTensorOperation):
Args:
degrees (Union[float, Sequence[float, float]]): Range of degrees to select from.
If float is provided, the degree will be randomly selected from (-`degrees`, `degrees`).
If float is provided, the degree will be randomly selected from ( `-degrees` , `degrees` ).
If Sequence[float, float] is provided, it needs to be arranged in order of (min, max).
translate (Sequence[float, float], optional): Maximum absolute fraction sequence in shape of (tx, ty)
for horizontal and vertical translations. The horizontal and vertical shifts are randomly
@ -918,7 +918,7 @@ class RandomAffine(py_transforms.PyTensorOperation):
shear (Union[float, Sequence[float, float], Sequence[float, float, float, float]], optional):
Range of shear factor to select from.
If float is provided, a shearing parallel to X axis with a factor selected from
(- `shear` , `shear` ) will be applied.
( `-shear` , `shear` ) will be applied.
If Sequence[float, float] is provided, a shearing parallel to X axis with a factor selected
from ( `shear` [0], `shear` [1]) will be applied.
If Sequence[float, float, float, float] is provided, a shearing parallel to X axis with a factor selected
@ -942,7 +942,7 @@ class RandomAffine(py_transforms.PyTensorOperation):
TypeError: If `translate` is not of type Sequence[float, float].
TypeError: If `scale` is not of type Sequence[float, float].
TypeError: If `shear` is not of type float or Sequence[float, float].
TypeError: If `resample` is not of type :class:`mindspore.dataset.vision.Inter`.
TypeError: If `resample` is not of type :class:`mindspore.dataset.vision.Inter` .
TypeError: If `fill_value` is not of type int or tuple[int, int, int].
ValueError: If `degrees` is negative.
ValueError: If `translate` is not in range of [-1.0, 1.0].
@ -1062,20 +1062,20 @@ class RandomColorAdjust(py_transforms.PyTensorOperation):
brightness (Union[float, Sequence[float, float]], optional): Range of brightness adjustment factor
to select from, must be non negative.
If float is provided, the factor will be uniformly selected from
[max(0, 1 - `brightness`), 1 + `brightness`).
[max(0, 1 - `brightness` ), 1 + `brightness` ).
If Sequence[float, float] is provided, it should be arranged in order of (min, max). Default: (1, 1).
contrast (Union[float, Sequence[float, float]], optional): Range of contrast adjustment factor
to select from, must be non negative.
If float is provided, the factor will be uniformly selected from [max(0, 1 - `contrast`), 1 + `contrast`).
If float is provided, the factor will be uniformly selected from [max(0, 1 - `contrast` ), 1 + `contrast` ).
If Sequence[float, float] is provided, it should be arranged in order of (min, max). Default: (1, 1).
saturation (Union[float, Sequence[float, float]], optional): Range of saturation adjustment factor
to select from, must be non negative.
If float is provided, the factor will be uniformly selected from
[max(0, 1 - `saturation`), 1 + `saturation`).
[max(0, 1 - `saturation` ), 1 + `saturation` ).
If Sequence[float, float] is provided, it should be arranged in order of (min, max). Default: (1, 1).
hue (Union[float, Sequence[float, float]], optional): Range of hue adjustment factor to select from.
If float is provided, it must be in range of [0, 0.5], and the factor will be uniformly
selected from [-`hue`, `hue`).
selected from [ `-hue` , `hue` ).
If Sequence[float, float] is provided, the elements must be in range of [-0.5, 0.5] and arranged in
order of (min, max). Default: (0, 0).
@ -1130,7 +1130,7 @@ class RandomCrop(py_transforms.PyTensorOperation):
Args:
size (Union[int, Sequence[int, int]]): The size of the cropped image.
If int is provided, a square of size (`size`, `size`) will be cropped with this value.
If int is provided, a square of size `(size, size)` will be cropped with this value.
If Sequence[int, int] is provided, its two elements will be taken as the cropped height and width.
padding (Union[int, Sequence[int, int], Sequence[int, int, int, int]], optional): The number of pixels to pad
on each border. When specified, it will pad the image before random cropping.
@ -1164,7 +1164,7 @@ class RandomCrop(py_transforms.PyTensorOperation):
TypeError: If `padding` is not of type int, Sequence[int, int] or Sequence[int, int, int, int].
TypeError: If `pad_if_needed` is not of type bool.
TypeError: If `fill_value` is not of type int or tuple[int, int, int].
TypeError: If `padding_mode` is not of type :class:`mindspore.dataset.vision.Border`.
TypeError: If `padding_mode` is not of type :class:`mindspore.dataset.vision.Border` .
ValueError: If `size` is not positive.
ValueError: If `padding` is negative.
ValueError: If `fill_value` is not in range of [0, 255].
@ -1214,7 +1214,7 @@ class RandomErasing(py_transforms.PyTensorOperation):
"""
Randomly erase pixels within a random selected rectangle erea on the input numpy.ndarray image.
See `Random Erasing Data Augmentation <https://arxiv.org/pdf/1708.04896.pdf>`_.
See `Random Erasing Data Augmentation <https://arxiv.org/pdf/1708.04896.pdf>`_ .
Args:
prob (float, optional): Probability of performing erasing. Default: 0.5.
@ -1446,7 +1446,7 @@ class RandomPerspective(py_transforms.PyTensorOperation):
Raises:
TypeError: If `distortion_scale` is not of type float.
TypeError: If `prob` is not of type float.
TypeError: If `interpolation` is not of type :class:`mindspore.dataset.vision.Inter`.
TypeError: If `interpolation` is not of type :class:`mindspore.dataset.vision.Inter` .
ValueError: If `distortion_scale` is not in range of [0, 1].
ValueError: If `prob` is not in range of [0, 1].
@ -1496,7 +1496,7 @@ class RandomResizedCrop(py_transforms.PyTensorOperation):
Args:
size (Union[int, Sequence[int, int]]): The size of the cropped image.
If int is provided, a square of size (`size`, `size`) will be cropped with this value.
If int is provided, a square of size `(size, size)` will be cropped with this value.
If Sequence[int, int] is provided, its two elements will be taken as the cropped height and width.
scale (Sequence[float, float], optional): Range of area scale of the cropped area relative
to the original image to select from, arraged in order or (min, max). Default: (0.08, 1.0).
@ -1517,7 +1517,7 @@ class RandomResizedCrop(py_transforms.PyTensorOperation):
TypeError: If `size` is not of type int or Sequence[int, int].
TypeError: If `scale` is not of type Sequence[float, float].
TypeError: If `ratio` is not of type Sequence[float, float].
TypeError: If `interpolation` is not of type :class:`mindspore.dataset.vision.Inter`.
TypeError: If `interpolation` is not of type :class:`mindspore.dataset.vision.Inter` .
TypeError: If `max_attempts` is not of type int.
ValueError: If `size` is not positive.
ValueError: If `scale` is negative.
@ -1568,7 +1568,7 @@ class RandomRotation(py_transforms.PyTensorOperation):
Args:
degrees (Union[float, Sequence[float, float]]): Range of rotation degree to select from.
If int is provided, the rotation degree will be randomly selected from (-`degrees`, `degrees`).
If int is provided, the rotation degree will be randomly selected from ( `-degrees` , `degrees` ).
If Sequence[float, float] is provided, it should be arranged in order of (min, max).
resample (Inter, optional): Method of interpolation. It can be Inter.NEAREST,
Inter.BILINEAR or Inter.BICUBIC. If the input PIL Image is in mode of "1" or "P",
@ -1592,7 +1592,7 @@ class RandomRotation(py_transforms.PyTensorOperation):
Raises:
TypeError: If `degrees` is not of type float or Sequence[float, float].
TypeError: If `resample` is not of type :class:`mindspore.dataset.vision.Inter`.
TypeError: If `resample` is not of type :class:`mindspore.dataset.vision.Inter` .
TypeError: If `expand` is not of type bool.
TypeError: If `center` is not of type Sequence[int, int].
TypeError: If `fill_value` is not of type int or tuple[int, int, int].
@ -1746,7 +1746,7 @@ class Resize(py_transforms.PyTensorOperation):
Raises:
TypeError: If `size` is not of type int or Sequence[int, int].
TypeError: If `interpolation` is not of type :class:`mindspore.dataset.vision.Inter`.
TypeError: If `interpolation` is not of type :class:`mindspore.dataset.vision.Inter` .
ValueError: If `size` is not positive.
Supported Platforms:
@ -1879,7 +1879,7 @@ class TenCrop(py_transforms.PyTensorOperation):
Args:
size (Union[int, Sequence[int, int]]): The size of the cropped image.
If int is provided, a square of size (`size`, `size`) will be cropped with this value.
If int is provided, a square of size `(size, size)` will be cropped with this value.
If Sequence[int, int] is provided, its two elements will be taken as the cropped height and width.
use_vertical_flip (bool, optional): If True, flip the images vertically. Otherwise, flip them
horizontally. Default: False.
@ -1933,10 +1933,10 @@ class ToPIL(py_transforms.PyTensorOperation):
Convert the input decoded numpy.ndarray image to PIL Image.
Note:
The conversion mode will be determined by the data type using :class:`PIL.Image.fromarray`.
The conversion mode will be determined by the data type using :class:`PIL.Image.fromarray` .
Raises:
TypeError: If the input image is not of type :class:`numpy.ndarray` or :class:`PIL.Image.Image`.
TypeError: If the input image is not of type :class:`numpy.ndarray` or :class:`PIL.Image.Image` .
Supported Platforms:
``CPU``
@ -1977,10 +1977,10 @@ class ToTensor(py_transforms.PyTensorOperation):
from (H, W, C) to (C, H, W).
Args:
output_type (numpy.dtype, optional): The desired dtype of the output image. Default: :class:`numpy.float32`.
output_type (numpy.dtype, optional): The desired dtype of the output image. Default: :class:`numpy.float32` .
Raises:
TypeError: If the input image is not of type :class:`PIL.Image.Image` or :class:`numpy.ndarray`.
TypeError: If the input image is not of type :class:`PIL.Image.Image` or :class:`numpy.ndarray` .
TypeError: If dimension of the input image is not 2 or 3.
Supported Platforms:
@ -2021,10 +2021,10 @@ class ToType(py_transforms.PyTensorOperation):
Convert the input numpy.ndarray image to the desired dtype.
Args:
output_type (numpy.dtype): The desired dtype of the output image, e.g. :class:`numpy.float32`.
output_type (numpy.dtype): The desired dtype of the output image, e.g. :class:`numpy.float32` .
Raises:
TypeError: If the input image is not of type :class:`numpy.ndarray`.
TypeError: If the input image is not of type :class:`numpy.ndarray` .
Supported Platforms:
``CPU``

76
mindspore/python/mindspore/dataset/vision/transforms.py
View File

@ -370,7 +370,7 @@ class Affine(ImageTensorOperation):
scale (float): Scaling factor, which must be positive.
shear (Union[float, Sequence]): Shear angle value in degrees between -180 to 180.
If a number is provided, a shearing parallel to X axis with a factor selected from
(- `shear` , `shear` ) will be applied.
( `-shear` , `shear` ) will be applied.
If a sequence is provided, a shearing parallel to X axis with a factor selected
from ( `shear` [0], `shear` [1]) will be applied.
resample (Inter, optional): An optional resampling filter. Default: Inter.NEAREST.
@ -437,7 +437,7 @@ class Affine(ImageTensorOperation):
class AutoAugment(ImageTensorOperation):
"""
Apply AutoAugment data augmentation method based on
`AutoAugment: Learning Augmentation Strategies from Data <https://arxiv.org/pdf/1805.09501.pdf>`_.
`AutoAugment: Learning Augmentation Strategies from Data <https://arxiv.org/pdf/1805.09501.pdf>`_ .
This operation works only with 3-channel RGB images.
Args:
@ -469,8 +469,8 @@ class AutoAugment(ImageTensorOperation):
Default: 0.
Raises:
TypeError: If `policy` is not of type :class:`mindspore.dataset.vision.AutoAugmentPolicy`.
TypeError: If `interpolation` is not of type :class:`mindspore.dataset.vision.Inter`.
TypeError: If `policy` is not of type :class:`mindspore.dataset.vision.AutoAugmentPolicy` .
TypeError: If `interpolation` is not of type :class:`mindspore.dataset.vision.Inter` .
TypeError: If `fill_value` is not an integer or a tuple of length 3.
RuntimeError: If given tensor shape is not <H, W, C>.
@ -567,7 +567,7 @@ class BoundingBoxAugment(ImageTensorOperation):
Range: [0.0, 1.0]. Default: 0.3.
Raises:
TypeError: If `transform` is an image processing operation in :class:`mindspore.dataset.vision.transforms`.
TypeError: If `transform` is an image processing operation in :class:`mindspore.dataset.vision.transforms` .
TypeError: If `ratio` is not of type float.
ValueError: If `ratio` is not in range [0.0, 1.0].
RuntimeError: If given bounding box is invalid.
@ -701,7 +701,7 @@ class ConvertColor(ImageTensorOperation):
- ConvertMode.COLOR_RGBA2GRAY, Convert RGBA image to GRAY image.
Raises:
TypeError: If `convert_mode` is not of type :class:`mindspore.dataset.vision.transforms.ConvertMode`.
TypeError: If `convert_mode` is not of type :class:`mindspore.dataset.vision.transforms.ConvertMode` .
RuntimeError: If given tensor shape is not <H, W> or <H, W, C>.
Supported Platforms:
@ -785,7 +785,7 @@ class CutMixBatch(ImageTensorOperation):
which must be in range: [0.0, 1.0]. Default: 1.0.
Raises:
TypeError: If `image_batch_format` is not of type :class:`mindspore.dataset.vision.ImageBatchFormat`.
TypeError: If `image_batch_format` is not of type :class:`mindspore.dataset.vision.ImageBatchFormat` .
TypeError: If `alpha` is not of type float.
TypeError: If `prob` is not of type float.
ValueError: If `alpha` is less than or equal 0.
@ -861,7 +861,7 @@ class CutOut(ImageTensorOperation):
class Decode(ImageTensorOperation, PyTensorOperation):
"""
Decode the input image in RGB mode.
Supported image formats: JPEG, BMP, PNG, TIFF, GIF(need `to_pil=True`), WEBP(need `to_pil=True`).
Supported image formats: JPEG, BMP, PNG, TIFF, GIF(need `to_pil=True` ), WEBP(need `to_pil=True` ).
Args:
to_pil (bool, optional): decode to PIL Image. Default: False.
@ -1301,12 +1301,12 @@ class LinearTransformation(PyTensorOperation):
Args:
transformation_matrix (numpy.ndarray): A square transformation matrix in shape of (D, D), where
:math:`D = C \times H \times W`.
mean_vector (numpy.ndarray): A mean vector in shape of (D,), where :math:`D = C \times H \times W`.
:math:`D = C \times H \times W` .
mean_vector (numpy.ndarray): A mean vector in shape of (D,), where :math:`D = C \times H \times W` .
Raises:
TypeError: If `transformation_matrix` is not of type :class:`numpy.ndarray`.
TypeError: If `mean_vector` is not of type :class:`numpy.ndarray`.
TypeError: If `transformation_matrix` is not of type :class:`numpy.ndarray` .
TypeError: If `mean_vector` is not of type :class:`numpy.ndarray` .
Supported Platforms:
``CPU``
@ -1592,7 +1592,7 @@ class Pad(ImageTensorOperation, PyTensorOperation):
Raises:
TypeError: If `padding` is not of type int or Sequence[int, int], Sequence[int, int, int, int]].
TypeError: If `fill_value` is not of type int or tuple[int].
TypeError: If `padding_mode` is not of type :class:`mindspore.dataset.vision.Border`.
TypeError: If `padding_mode` is not of type :class:`mindspore.dataset.vision.Border` .
ValueError: If `padding` is negative.
ValueError: If `fill_value` is not in range [0, 255].
RuntimeError: If given tensor shape is not <H, W> or <H, W, C>.
@ -1661,7 +1661,7 @@ class PadToSize(ImageTensorOperation):
TypeError: If `size` is not of type int or Sequence[int, int].
TypeError: If `offset` is not of type int or Sequence[int, int].
TypeError: If `fill_value` is not of type int or tuple[int, int, int].
TypeError: If `padding_mode` is not of type :class:`mindspore.dataset.vision.Border`.
TypeError: If `padding_mode` is not of type :class:`mindspore.dataset.vision.Border` .
ValueError: If `size` is not positive.
ValueError: If `offset` is negative.
ValueError: If `fill_value` is not in range of [0, 255].
@ -1718,7 +1718,7 @@ class Perspective(ImageTensorOperation, PyTensorOperation):
TypeError: If element in `start_points` is not of type Sequence[int, int] of length 2.
TypeError: If `end_points` is not of type Sequence[Sequence[int, int]] of length 4.
TypeError: If element in `end_points` is not of type Sequence[int, int] of length 2.
TypeError: If `interpolation` is not of type :class:`mindspore.dataset.vision.Inter`.
TypeError: If `interpolation` is not of type :class:`mindspore.dataset.vision.Inter` .
RuntimeError: If given tensor shape is not <H, W> or <H, W, C>.
Supported Platforms:
@ -1795,7 +1795,7 @@ class Posterize(ImageTensorOperation):
class RandAugment(ImageTensorOperation):
"""
Apply RandAugment data augmentation method based on
`RandAugment: Learning Augmentation Strategies from Data <https://arxiv.org/pdf/1909.13719.pdf>`.
`RandAugment: Learning Augmentation Strategies from Data <https://arxiv.org/pdf/1909.13719.pdf>`_ .
This operation works only with 3-channel RGB images.
Args:
@ -1824,7 +1824,7 @@ class RandAugment(ImageTensorOperation):
TypeError: If `num_ops` is not of type int.
TypeError: If `magnitude` is not of type int.
TypeError: If `num_magnitude_bins` is not of type int.
TypeError: If `interpolation` not of type :class:`mindspore.dataset.vision.Inter`.
TypeError: If `interpolation` not of type :class:`mindspore.dataset.vision.Inter` .
TypeError: If `fill_value` is not an int or a tuple of length 3.
RuntimeError: If given tensor shape is not <H, W, C>.
@ -1914,7 +1914,7 @@ class RandomAffine(ImageTensorOperation, PyTensorOperation):
shear (Union[float, Sequence[float, float], Sequence[float, float, float, float]], optional):
Range of shear factor to select from.
If float is provided, a shearing parallel to X axis with a factor selected from
(- `shear` , `shear` ) will be applied.
( `-shear` , `shear` ) will be applied.
If Sequence[float, float] is provided, a shearing parallel to X axis with a factor selected
from ( `shear` [0], `shear` [1]) will be applied.
If Sequence[float, float, float, float] is provided, a shearing parallel to X axis with a factor selected
@ -1940,7 +1940,7 @@ class RandomAffine(ImageTensorOperation, PyTensorOperation):
TypeError: If `translate` is not of type sequence.
TypeError: If `scale` is not of type sequence.
TypeError: If `shear` is not of type int, float or sequence.
TypeError: If `resample` is not of type :class:`mindspore.dataset.vision.Inter`.
TypeError: If `resample` is not of type :class:`mindspore.dataset.vision.Inter` .
TypeError: If `fill_value` is not of type int or tuple[int].
ValueError: If `degrees` is negative.
ValueError: If `translate` is not in range [-1.0, 1.0].
@ -2256,7 +2256,7 @@ class RandomCrop(ImageTensorOperation, PyTensorOperation):
TypeError: If `padding` is not of type int or Sequence[int].
TypeError: If `pad_if_needed` is not of type boolean.
TypeError: If `fill_value` is not of type int or tuple[int].
TypeError: If `padding_mode` is not of type :class:`mindspore.dataset.vision.Border`.
TypeError: If `padding_mode` is not of type :class:`mindspore.dataset.vision.Border` .
ValueError: If `size` is not positive.
ValueError: If `padding` is negative.
ValueError: If `fill_value` is not in range [0, 255].
@ -2312,7 +2312,7 @@ class RandomCrop(ImageTensorOperation, PyTensorOperation):
class RandomCropDecodeResize(ImageTensorOperation):
"""
A combination of `Crop`, `Decode` and `Resize`. It will get better performance for JPEG images. This operation
A combination of `Crop` , `Decode` and `Resize` . It will get better performance for JPEG images. This operation
will crop the input image at a random location, decode the cropped image in RGB mode, and resize the decoded image.
Args:
@ -2344,7 +2344,7 @@ class RandomCropDecodeResize(ImageTensorOperation):
TypeError: If `size` is not of type int or Sequence[int].
TypeError: If `scale` is not of type tuple.
TypeError: If `ratio` is not of type tuple.
TypeError: If `interpolation` is not of type :class:`mindspore.dataset.vision.Inter`.
TypeError: If `interpolation` is not of type :class:`mindspore.dataset.vision.Inter` .
TypeError: If `max_attempts` is not of type integer.
ValueError: If `size` is not positive.
ValueError: If `scale` is negative.
@ -2434,7 +2434,7 @@ class RandomCropWithBBox(ImageTensorOperation):
TypeError: If `padding` is not of type int or Sequence[int].
TypeError: If `pad_if_needed` is not of type boolean.
TypeError: If `fill_value` is not of type int or tuple[int].
TypeError: If `padding_mode` is not of type :class:`mindspore.dataset.vision.Border`.
TypeError: If `padding_mode` is not of type :class:`mindspore.dataset.vision.Border` .
ValueError: If `size` is not positive.
ValueError: If `padding` is negative.
ValueError: If `fill_value` is not in range [0, 255].
@ -2513,7 +2513,7 @@ class RandomErasing(PyTensorOperation):
"""
Randomly erase pixels within a random selected rectangle erea on the input numpy.ndarray image.
See `Random Erasing Data Augmentation <https://arxiv.org/pdf/1708.04896.pdf>`_.
See `Random Erasing Data Augmentation <https://arxiv.org/pdf/1708.04896.pdf>`_ .
Args:
prob (float, optional): Probability of performing erasing, which
@ -2811,7 +2811,7 @@ class RandomPerspective(PyTensorOperation):
Raises:
TypeError: If `distortion_scale` is not of type float.
TypeError: If `prob` is not of type float.
TypeError: If `interpolation` is not of type :class:`mindspore.dataset.vision.Inter`.
TypeError: If `interpolation` is not of type :class:`mindspore.dataset.vision.Inter` .
ValueError: If `distortion_scale` is not in range of [0.0, 1.0].
ValueError: If `prob` is not in range of [0.0, 1.0].
@ -2897,7 +2897,7 @@ class RandomPosterize(ImageTensorOperation):
class RandomResizedCrop(ImageTensorOperation, PyTensorOperation):
"""
This operation will crop the input image randomly,
and resize the cropped image using a selected interpolation mode :class:`mindspore.dataset.vision.Inter`.
and resize the cropped image using a selected interpolation mode :class:`mindspore.dataset.vision.Inter` .
Note:
If the input image is more than one, then make sure that the image size is the same.
@ -2933,7 +2933,7 @@ class RandomResizedCrop(ImageTensorOperation, PyTensorOperation):
TypeError: If `size` is not of type int or Sequence[int].
TypeError: If `scale` is not of type tuple or list.
TypeError: If `ratio` is not of type tuple or list.
TypeError: If `interpolation` is not of type :class:`mindspore.dataset.vision.Inter`.
TypeError: If `interpolation` is not of type :class:`mindspore.dataset.vision.Inter` .
TypeError: If `max_attempts` is not of type int.
ValueError: If `size` is not positive.
ValueError: If `scale` is negative.
@ -3060,7 +3060,7 @@ class RandomResizedCropWithBBox(ImageTensorOperation):
class RandomResize(ImageTensorOperation):
"""
Resize the input image using :class:`mindspore.dataset.vision.Inter`, a randomly selected interpolation mode.
Resize the input image using :class:`mindspore.dataset.vision.Inter` , a randomly selected interpolation mode.
Args:
size (Union[int, Sequence[int]]): The output size of the resized image. The size value(s) must be positive.
@ -3485,7 +3485,7 @@ class Rescale(ImageTensorOperation):
class Resize(ImageTensorOperation, PyTensorOperation):
"""
Resize the input image to the given size with a given interpolation mode :class:`mindspore.dataset.vision.Inter`.
Resize the input image to the given size with a given interpolation mode :class:`mindspore.dataset.vision.Inter` .
Args:
size (Union[int, Sequence[int]]): The output size of the resized image. The size value(s) must be positive.
@ -3589,7 +3589,7 @@ class ResizedCrop(ImageTensorOperation):
TypeError: If `height` is not of type int.
TypeError: If `width` is not of type int.
TypeError: If `size` is not of type int or Sequence[int].
TypeError: If `interpolation` is not of type :class:`mindspore.dataset.vision.Inter`.
TypeError: If `interpolation` is not of type :class:`mindspore.dataset.vision.Inter` .
RuntimeError: If given tensor shape is not <H, W> or <H, W, C>.
Supported Platforms:
@ -3975,10 +3975,10 @@ class ToPIL(PyTensorOperation):
Convert the input decoded numpy.ndarray image to PIL Image.
Note:
The conversion mode will be determined by the data type using :class:`PIL.Image.fromarray`.
The conversion mode will be determined by the data type using :class:`PIL.Image.fromarray` .
Raises:
TypeError: If the input image is not of type :class:`numpy.ndarray` or :class:`PIL.Image.Image`.
TypeError: If the input image is not of type :class:`numpy.ndarray` or :class:`PIL.Image.Image` .
Supported Platforms:
``CPU``
@ -4020,10 +4020,10 @@ class ToTensor(ImageTensorOperation):
Args:
output_type (Union[mindspore.dtype, numpy.dtype], optional): The desired dtype of the output image.
Default: :class:`numpy.float32`.
Default: :class:`numpy.float32` .
Raises:
TypeError: If the input image is not of type :class:`PIL.Image.Image` or :class:`numpy.ndarray`.
TypeError: If the input image is not of type :class:`PIL.Image.Image` or :class:`numpy.ndarray` .
TypeError: If dimension of the input image is not 2 or 3.
Supported Platforms:
@ -4060,17 +4060,17 @@ class ToType(TypeCast):
"""
Cast the input to a given MindSpore data type or NumPy data type.
It is the same as that of :class:`mindspore.dataset.transforms.TypeCast`.
It is the same as that of :class:`mindspore.dataset.transforms.TypeCast` .
Note:
This operation supports running on Ascend or GPU platforms by Offload.
Args:
data_type (Union[mindspore.dtype, numpy.dtype]): The desired data type of the output image,
such as :class:`numpy.float32`.
such as :class:`numpy.float32` .
Raises:
TypeError: If `data_type` is not of type :class:`mindspore.dtype` or :class:`numpy.dtype`.
TypeError: If `data_type` is not of type :class:`mindspore.dtype` or :class:`numpy.dtype` .
Supported Platforms:
``CPU`` ``Ascend`` ``GPU``
@ -4092,7 +4092,7 @@ class ToType(TypeCast):
class TrivialAugmentWide(ImageTensorOperation):
"""
Apply TrivialAugmentWide data augmentation method based on
`TrivialAugmentWide: Tuning-free Yet State-of-the-Art Data Augmentation <https://arxiv.org/abs/2103.10158>`_.
`TrivialAugmentWide: Tuning-free Yet State-of-the-Art Data Augmentation <https://arxiv.org/abs/2103.10158>`_ .
This operation works only with 3-channel RGB images.
Args:

2
mindspore/python/mindspore/mindrecord/filereader.py
View File

@ -43,7 +43,7 @@ class FileReader:
operator (int, optional): Reserved parameter for operators. Default: None.
Raises:
ParamValueError: If `file_name`, `num_consumer` or `columns` is invalid.
ParamValueError: If `file_name` , `num_consumer` or `columns` is invalid.
Examples:
>>> from mindspore.mindrecord import FileReader

12
mindspore/python/mindspore/mindrecord/filewriter.py
View File

@ -163,7 +163,7 @@ class FileWriter:
The schema is added to describe the raw data to be written.
Note:
Please refer to the Examples of class: `mindspore.mindrecord.FileWriter`.
Please refer to the Examples of class: `mindspore.mindrecord.FileWriter` .
Args:
content (dict): Dictionary of schema content.
@ -186,14 +186,14 @@ class FileWriter:
def add_index(self, index_fields):
"""
Select index fields from schema to accelerate reading.
schema is added through `add_schema`.
schema is added through `add_schema` .
Note:
The index fields should be primitive type. e.g. int/float/str.
If the function is not called, the fields of the primitive type
in schema are set as indexes by default.
Please refer to the Examples of class: `mindspore.mindrecord.FileWriter`.
Please refer to the Examples of class: `mindspore.mindrecord.FileWriter` .
Args:
index_fields (list[str]): fields from schema.
@ -220,7 +220,7 @@ class FileWriter:
def open_and_set_header(self):
"""
Open writer and set header which stores meta information. The function is only used for parallel \
writing and is called before the `write_raw_data`.
writing and is called before the `write_raw_data` .
Returns:
MSRStatus, SUCCESS or FAILED.
@ -243,7 +243,7 @@ class FileWriter:
files after the raw data is verified against the schema.
Note:
Please refer to the Examples of class: `mindspore.mindrecord.FileWriter`.
Please refer to the Examples of class: `mindspore.mindrecord.FileWriter` .
Args:
raw_data (list[dict]): List of raw data.
@ -331,7 +331,7 @@ class FileWriter:
Flush data in memory to disk and generate the corresponding database files.
Note:
Please refer to the Examples of class: `mindspore.mindrecord.FileWriter`.
Please refer to the Examples of class: `mindspore.mindrecord.FileWriter` .
Returns:
MSRStatus, SUCCESS or FAILED.

6
mindspore/python/mindspore/mindrecord/mindpage.py
View File

@ -34,7 +34,7 @@ class MindPage:
It should not be smaller than 1 or larger than the number of processor cores.
Raises:
ParamValueError: If `file_name`, `num_consumer` or columns is invalid.
ParamValueError: If `file_name` , `num_consumer` or columns is invalid.
MRMInitSegmentError: If failed to initialize ShardSegment.
"""
@check_parameter
@ -128,7 +128,7 @@ class MindPage:
Query by category id in pagination.
Args:
category_id (int): Category id, referred to the return of `read_category_info`.
category_id (int): Category id, referred to the return of `read_category_info` .
page (int): Index of page.
num_row (int): Number of rows in a page.
@ -154,7 +154,7 @@ class MindPage:
Args:
category_name (str): String of category field's value,
referred to the return of `read_category_info`.
referred to the return of `read_category_info` .
page (int): Index of page.
num_row (int): Number of row in a page.

6
mindspore/python/mindspore/mindrecord/tools/cifar100_to_mr.py
View File

@ -40,7 +40,7 @@ class Cifar100ToMR:
Note:
For details about Examples, please refer to `Converting the CIFAR-10 Dataset <https://
www.mindspore.cn/tutorials/zh-CN/master/advanced/dataset/record.html#converting-the-cifar-10-dataset>`_.
www.mindspore.cn/tutorials/zh-CN/master/advanced/dataset/record.html#converting-the-cifar-10-dataset>`_ .
Args:
source (str): The cifar100 directory to be transformed.
@ -80,7 +80,7 @@ class Cifar100ToMR:
Args:
fields (list[str], optional):
A list of index field, e.g.["fine_label", "coarse_label"]. Default: None. For index
field settings, please refer to :func:`mindspore.mindrecord.FileWriter.add_index`.
field settings, please refer to :func:`mindspore.mindrecord.FileWriter.add_index` .
Returns:
MSRStatus, SUCCESS or FAILED.
@ -121,7 +121,7 @@ class Cifar100ToMR:
Args:
fields (list[str], optional):
A list of index field, e.g.["fine_label", "coarse_label"]. Default: None. For index
field settings, please refer to :func:`mindspore.mindrecord.FileWriter.add_index`.
field settings, please refer to :func:`mindspore.mindrecord.FileWriter.add_index` .
Returns:
MSRStatus, SUCCESS or FAILED.

6
mindspore/python/mindspore/mindrecord/tools/cifar10_to_mr.py
View File

@ -40,7 +40,7 @@ class Cifar10ToMR:
Note:
For details about Examples, please refer to `Converting the CIFAR-10 Dataset <https://
www.mindspore.cn/tutorials/zh-CN/master/advanced/dataset/record.html#converting-the-cifar-10-dataset>`_.
www.mindspore.cn/tutorials/zh-CN/master/advanced/dataset/record.html#converting-the-cifar-10-dataset>`_ .
Args:
source (str): The cifar10 directory to be transformed.
@ -79,7 +79,7 @@ class Cifar10ToMR:
Args:
fields (list[str], optional): A list of index fields. Default: None. For index field settings,
please refer to :func:`mindspore.mindrecord.FileWriter.add_index`.
please refer to :func:`mindspore.mindrecord.FileWriter.add_index` .
Returns:
MSRStatus, SUCCESS or FAILED.
@ -116,7 +116,7 @@ class Cifar10ToMR:
Args:
fields (list[str], optional): A list of index fields. Default: None. For index field settings,
please refer to :func:`mindspore.mindrecord.FileWriter.add_index`.
please refer to :func:`mindspore.mindrecord.FileWriter.add_index` .
Returns:
MSRStatus, SUCCESS or FAILED.

4
mindspore/python/mindspore/mindrecord/tools/csv_to_mr.py
View File

@ -36,7 +36,7 @@ class CsvToMR:
Note:
For details about Examples, please refer to `Converting CSV Dataset <https://
www.mindspore.cn/tutorials/zh-CN/master/advanced/dataset/record.html#converting-csv-dataset>`_.
www.mindspore.cn/tutorials/zh-CN/master/advanced/dataset/record.html#converting-csv-dataset>`_ .
Args:
source (str): The file path of csv.
@ -46,7 +46,7 @@ class CsvToMR:
partition_number (int, optional): The partition size, Default: 1.
Raises:
ValueError: If `source`, `destination`, `partition_number` is invalid.
ValueError: If `source` , `destination` , `partition_number` is invalid.
RuntimeError: If `columns_list` is invalid.
"""

2
mindspore/python/mindspore/mindrecord/tools/imagenet_to_mr.py
View File

@ -46,7 +46,7 @@ class ImageNetToMR:
partition_number (int, optional): The partition size. Default: 1.
Raises:
ValueError: If `map_file`, `image_dir` or `destination` is invalid.
ValueError: If `map_file` , `image_dir` or `destination` is invalid.
"""
def __init__(self, map_file, image_dir, destination, partition_number=1):

2
mindspore/python/mindspore/mindrecord/tools/mnist_to_mr.py
View File

@ -46,7 +46,7 @@ class MnistToMR:
partition_number (int, optional): The partition size. Default: 1.
Raises:
ValueError: If `source`, `destination`, `partition_number` is invalid.
ValueError: If `source` , `destination` , `partition_number` is invalid.
"""
def __init__(self, source, destination, partition_number=1):

2
mindspore/python/mindspore/mindrecord/tools/tfrecord_to_mr.py
View File

@ -69,7 +69,7 @@ class TFRecordToMR:
Note:
For details about Examples, please refer to `Converting TFRecord Dataset <https://
www.mindspore.cn/tutorials/zh-CN/master/advanced/dataset/record.html#converting-tfrecord-dataset>`_.
www.mindspore.cn/tutorials/zh-CN/master/advanced/dataset/record.html#converting-tfrecord-dataset>`_ .
Args:
source (str): TFRecord file to be transformed.