Nigel
/
mindspore
forked from mindspore-Ecosystem/mindspore
1
0
Fork 0
Code Issues Pull Requests Projects Releases Wiki Activity

!44217 [MD] Rectify terminology from operator to operation 

Merge pull request !44217 from xiaotianci/code_docs_terminology_rectification
This commit is contained in:
i-robot 2022-10-20 06:58:28 +00:00 committed by Gitee
parent 5499a734d5 560f3672c5
commit b7cc20f39a
No known key found for this signature in database
GPG Key ID: 173E9B9CA92EEF8F
63 changed files with 193 additions and 188 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

2
docs/api/api_python/dataset/dataset_method/operation/mindspore.dataset.Dataset.map.rst
View File

@ -23,7 +23,7 @@ mindspore.dataset.Dataset.map
.. image:: map_parameter_pyfunc_cn.png
参数：
- **operations** (Union[list[TensorOperation], list[functions]]) - 一组数据增强操作，支持数据集增强算子或者用户自定义的Python Callable对象。map操作将按顺序将一组数据增强作用在数据集对象上。
- **operations** (Union[list[TensorOperation], list[functions]]) - 一组数据增强操作，支持数据集增强操作或者用户自定义的Python Callable对象。map操作将按顺序将一组数据增强作用在数据集对象上。
- **input_columns** (Union[str, list[str]], 可选) - 第一个数据增强操作的输入数据列。此列表的长度必须与 `operations` 列表中第一个数据增强的预期输入列数相匹配。默认值：None。表示所有数据列都将传递给第一个数据增强操作。
- **output_columns** (Union[str, list[str]], 可选) - 最后一个数据增强操作的输出数据列。如果 `input_columns` 长度不等于 `output_columns` 长度，则必须指定此参数。列表的长度必须必须与最后一个数据增强的输出列数相匹配。默认值：None，输出列将与输入列具有相同的名称。
- **num_parallel_workers** (int, 可选) - 指定map操作的多进程/多线程并发数，加快处理速度。默认值：None，将使用 `set_num_parallel_workers` 设置的并发数。

2
docs/api/api_python/dataset/mindspore.dataset.DatasetCache.rst
View File

@ -14,7 +14,7 @@ mindspore.dataset.DatasetCache
- **hostname** (str, 可选) - 数据缓存服务客户端的主机IP。默认值：None，表示使用默认主机IP 127.0.0.1。
- **port** (int, 可选) - 指定连接到数据缓存服务端的端口号。默认值：None，表示端口为50052。
- **num_connections** (int, 可选) - TCP/IP连接数量。默认值：None，表示连接数量为12。
- **prefetch_size** (int, 可选) - 指定缓存队列大小，使用缓存功能算子时，将直接从缓存队列中获取数据。默认值：None，表示缓存队列大小为20。
- **prefetch_size** (int, 可选) - 指定缓存队列大小，使用缓存功能时，将直接从缓存队列中获取数据。默认值：None，表示缓存队列大小为20。
.. py:method:: get_stat()

2
docs/api/api_python/dataset/mindspore.dataset.WaitedDSCallback.rst
View File

@ -5,7 +5,7 @@ mindspore.dataset.WaitedDSCallback
阻塞式数据处理回调类的抽象基类，用于与训练回调类 `mindspore.train.Callback <https://www.mindspore.cn/docs/zh-CN/master/api_python/train/mindspore.train.Callback.html#mindspore.train.Callback>`_ 的同步。
可用于在step或epoch开始前执行自定义的回调方法，例如在自动数据增强中根据上一个epoch的loss值来更新增强算子参数配置。
可用于在step或epoch开始前执行自定义的回调方法，例如在自动数据增强中根据上一个epoch的loss值来更新增强操作参数配置。
用户可通过 `train_run_context` 获取网络训练相关信息，如 `network` 、 `train_network` 、 `epoch_num` 、 `batch_num` 、 `loss_fn` 、 `optimizer` 、 `parallel_mode` 、 `device_number` 、 `list_callback` 、 `cur_epoch_num` 、 `cur_step_num` 、 `dataset_sink_mode` 、 `net_outputs` 等，详见 `mindspore.train.Callback <https://www.mindspore.cn/docs/zh-CN/master/api_python/train/mindspore.train.Callback.html#mindspore.train.Callback>`_ 。

4
docs/api/api_python/dataset_transforms/mindspore.dataset.transforms.Compose.rst
View File

@ -3,10 +3,10 @@ mindspore.dataset.transforms.Compose
.. py:class:: mindspore.dataset.transforms.Compose(transforms)
将多个数据增强算子组合使用。
将多个数据增强操作组合使用。
.. note::
Compose可以将 `mindspore.dataset.transforms` / `mindspore.dataset.vision` 等模块中的数据增强算子以及用户自定义的Python可调用对象
Compose可以将 `mindspore.dataset.transforms` / `mindspore.dataset.vision` 等模块中的数据增强操作以及用户自定义的Python可调用对象
合并成单个数据增强。对于用户定义的Python可调用对象，要求其返回值是numpy.ndarray类型。
参数：

2
docs/api/api_python/dataset_vision/mindspore.dataset.vision.AutoAugment.rst
View File

@ -15,7 +15,7 @@ mindspore.dataset.vision.AutoAugment
- **AutoAugmentPolicy.CIFAR10**：表示应用在Cifar10数据集上学习的AutoAugment。
- **AutoAugmentPolicy.SVHN**：表示应用在SVHN数据集上学习的AutoAugment。
- **interpolation** (Inter, 可选) - 调整大小算子的图像插值模式，默认值：Inter.NEAREST。
- **interpolation** (Inter, 可选) - 图像插值方式，默认值：Inter.NEAREST。
可以是[Inter.NEAREST, Inter.BILINEAR, Inter.BICUBIC, Inter.AREA]中的任何一个。

4
docs/api/api_python/dataset_vision/mindspore.dataset.vision.RandomCropDecodeResize.rst
View File

@ -3,7 +3,7 @@ mindspore.dataset.vision.RandomCropDecodeResize
.. py:class:: mindspore.dataset.vision.RandomCropDecodeResize(size, scale=(0.08, 1.0), ratio=(3. / 4., 4. / 3.), interpolation=Inter.BILINEAR, max_attempts=10)
"裁剪"、"解码"和"调整尺寸大小"的组合处理。该算子将在随机位置裁剪输入图像，以 RGB 模式对裁剪后的图像进行解码，并调整解码图像的尺寸大小。针对 JPEG 图像进行了优化, 可以获得更好的性能。
"裁剪"、"解码"和"调整尺寸大小"的组合处理。该操作将在随机位置裁剪输入图像，以 RGB 模式对裁剪后的图像进行解码，并调整解码图像的尺寸大小。针对 JPEG 图像进行了优化, 可以获得更好的性能。
参数：
- **size** (Union[int, Sequence[int]]) - 调整后图像的输出尺寸大小。大小值必须为正。
@ -11,7 +11,7 @@ mindspore.dataset.vision.RandomCropDecodeResize
如果 size 是一个长度为 2 的序列，则以2个元素分别为高和宽放缩至(高度, 宽度)大小。
- **scale** (Union[list, tuple], 可选) - 要裁剪的原始尺寸大小的各个尺寸的范围[min, max)，必须为非负数，默认值：(0.08, 1.0)。
- **ratio** (Union[list, tuple], 可选) - 宽高比的范围 [min, max) 裁剪，必须为非负数，默认值：(3. / 4., 4. / 3.)。
- **interpolation** (Inter, 可选) - resize算子的图像插值方式。它可以是 [Inter.BILINEAR、Inter.NEAREST、Inter.BICUBIC、Inter.AREA、Inter.PILCUBIC] 中的任何一个，默认值：Inter.BILINEAR。
- **interpolation** (Inter, 可选) - 图像插值方式。它可以是 [Inter.BILINEAR、Inter.NEAREST、Inter.BICUBIC、Inter.AREA、Inter.PILCUBIC] 中的任何一个，默认值：Inter.BILINEAR。
- **Inter.BILINEAR**: 双线性插值。
- **Inter.NEAREST**: 最近邻插值。

2
docs/api/api_python/mindspore.dataset.audio.rst
View File

@ -3,7 +3,7 @@ mindspore.dataset.audio
.. include:: dataset_audio/mindspore.dataset.audio.rst
数据增强算子可以放入数据处理Pipeline中执行，也可以Eager模式执行：
数据增强操作可以放入数据处理Pipeline中执行，也可以Eager模式执行：
- Pipeline模式一般用于处理数据集，示例可参考 `数据处理Pipeline介绍 <https://www.mindspore.cn/docs/zh-CN/master/api_python/mindspore.dataset.html#数据处理pipeline介绍>`_。
- Eager模式一般用于零散样本，音频预处理举例如下：

4
docs/api/api_python/mindspore.dataset.rst
View File

@ -37,11 +37,11 @@ mindspore.dataset
`GeneratorDataset` 实现Python层自定义数据集的加载，同时加载类方法可以使用多种Sampler、数据分片、数据shuffle等功能；
- 数据集操作（filter/ skip）：用户通过数据集对象方法 `.shuffle` / `.filter` / `.skip` / `.split` /
`.take` / … 来实现数据集的进一步混洗、过滤、跳过、最多获取条数等操作；
- 数据集样本增强操作（map）：用户可以将数据增强算子
- 数据集样本增强操作（map）：用户可以将数据增强操作
（`vision类 <https://www.mindspore.cn/docs/zh-CN/master/api_python/mindspore.dataset.vision.html>`_，
`nlp类 <https://www.mindspore.cn/docs/zh-CN/master/api_python/mindspore.dataset.text.html>`_，
`audio类 <https://www.mindspore.cn/docs/zh-CN/master/api_python/mindspore.dataset.audio.html>`_）
添加到map操作来执行，数据预处理过程中可以定义多个map操作，用于执行不同增强操作，数据增强算子也可以是
添加到map操作中执行，数据预处理过程中可以定义多个map操作，用于执行不同增强操作，数据增强操作也可以是
用户自定义增强的 `PyFunc`；
- 批（batch）：用户在样本完成增强后，使用 `.batch` 操作将多个样本组织成batch，也可以通过batch的参数 `per_batch_map`
来自定义batch逻辑；

4
docs/api/api_python/mindspore.dataset.text.rst
View File

@ -14,12 +14,14 @@ mindspore.dataset.text
import mindspore.dataset as ds
from mindspore.dataset import text
更多详情请参考 `文本数据变换 <https://www.mindspore.cn/tutorials/zh-CN/master/beginner/transforms.html#text-transforms>`_。
常用数据处理术语说明如下：
- TensorOperation，所有C++实现的数据处理操作的基类。
- TextTensorOperation，所有文本数据处理操作的基类，派生自TensorOperation。
数据增强算子可以放入数据处理Pipeline中执行，也可以Eager模式执行：
数据增强操作可以放入数据处理Pipeline中执行，也可以Eager模式执行：
- Pipeline模式一般用于处理数据集，示例可参考 `数据处理Pipeline介绍 <https://www.mindspore.cn/docs/zh-CN/master/api_python/mindspore.dataset.html#数据处理pipeline介绍>`_。
- Eager模式一般用于零散样本，文本预处理举例如下：

2
docs/api/api_python/mindspore.dataset.transforms.rst
View File

@ -17,7 +17,7 @@ mindspore.dataset.transforms
from mindspore.dataset.transforms import c_transforms
from mindspore.dataset.transforms import py_transforms
更多详情请参考 `通用数据处理与增强 <https://www.mindspore.cn/tutorials/zh-CN/master/advanced/dataset/augment_common_data.html>`_ 。
更多详情请参考 `通用数据变换 <https://www.mindspore.cn/tutorials/zh-CN/master/beginner/transforms.html#common-transforms>`_ 。
常用数据处理术语说明如下：

4
docs/api/api_python/mindspore.dataset.vision.rst
View File

@ -19,14 +19,14 @@ API样例中常用的导入模块如下：
import mindspore.dataset.vision.py_transforms as py_vision
from mindspore.dataset.transforms import c_transforms
更多详情请参考 `图像数据加载与增强 <https://www.mindspore.cn/tutorials/zh-CN/master/advanced/dataset/augment_image_data.html>`_。
更多详情请参考 `视觉数据变换 <https://www.mindspore.cn/tutorials/zh-CN/master/beginner/transforms.html#vision-transforms>`_。
常用数据处理术语说明如下：
- TensorOperation，所有C++实现的数据处理操作的基类。
- PyTensorOperation，所有Python实现的数据处理操作的基类。
数据增强算子可以放入数据处理Pipeline中执行，也可以Eager模式执行：
数据增强操作可以放入数据处理Pipeline中执行，也可以Eager模式执行：
- Pipeline模式一般用于处理数据集，示例可参考 `数据处理Pipeline介绍 <https://www.mindspore.cn/docs/zh-CN/master/api_python/mindspore.dataset.html#数据处理pipeline介绍>`_。
- Eager模式一般用于零散样本，图像预处理举例如下：

22
mindspore/ccsrc/minddata/dataset/api/execute.cc
View File

@ -235,7 +235,7 @@ Status Execute::operator()(const mindspore::MSTensor &input, mindspore::MSTensor
}
*output = mindspore::MSTensor(std::make_shared<DETensor>(de_tensor));
} else if (device_type_ ==
MapTargetDevice::kAscend310) { // Ascend310 case, where we must set Ascend resource on each operators
MapTargetDevice::kAscend310) { // Ascend310 case, where we must set Ascend resource on each operations
#if defined(WITH_BACKEND) || defined(ENABLE_ACL)
CHECK_FAIL_RETURN_UNEXPECTED(device_resource_, "Device resource is nullptr which is illegal under case Ascend310.");
// Sink data from host into device
@ -243,7 +243,7 @@ Status Execute::operator()(const mindspore::MSTensor &input, mindspore::MSTensor
RETURN_IF_NOT_OK(device_resource_->Sink(input, &device_input));
for (auto &t : transforms_rt_) {
// Initialize AscendResource for each operators
// Initialize AscendResource for each operations
std::shared_ptr<DeviceTensor> device_output;
RETURN_IF_NOT_OK(t->SetAscendResource(device_resource_));
@ -313,7 +313,7 @@ Status Execute::operator()(const std::vector<MSTensor> &input_tensor_list, std::
}
CHECK_FAIL_RETURN_UNEXPECTED(!output_tensor_list->empty(), "Output Tensor is not valid.");
} else if (device_type_ ==
MapTargetDevice::kAscend310) { // Ascend310 case, where we must set Ascend resource on each operators
MapTargetDevice::kAscend310) { // Ascend310 case, where we must set Ascend resource on each operations
CHECK_FAIL_RETURN_UNEXPECTED(device_resource_, "Device resource is nullptr which is illegal under case Ascend310.");
for (auto &input_tensor : input_tensor_list) {
// Sink each data from host into device
@ -428,9 +428,9 @@ std::vector<uint32_t> AippSizeFilter(const std::vector<uint32_t> &resize_para, c
return aipp_size;
}
if (resize_para.empty()) { // If only Crop operator exists
if (resize_para.empty()) { // If only Crop operation exists
aipp_size = crop_para;
} else if (crop_para.empty()) { // If only Resize operator with 2 parameters exists
} else if (crop_para.empty()) { // If only Resize operation with 2 parameters exists
aipp_size = resize_para;
} else { // If both of them exist
if (resize_para.size() == 1) {
@ -450,7 +450,7 @@ std::vector<uint32_t> AippSizeFilter(const std::vector<uint32_t> &resize_para, c
std::vector<uint32_t> AippMeanFilter(const std::vector<uint32_t> &normalize_para) {
std::vector<uint32_t> aipp_mean;
if (normalize_para.size() == 6) { // If Normalize operator exist
if (normalize_para.size() == 6) { // If Normalize operation exist
std::transform(normalize_para.begin(), normalize_para.begin() + 3, std::back_inserter(aipp_mean),
[](uint32_t i) { return static_cast<uint32_t>(i / 10000); });
} else {
@ -461,7 +461,7 @@ std::vector<uint32_t> AippMeanFilter(const std::vector<uint32_t> &normalize_para
std::vector<float> AippStdFilter(const std::vector<uint32_t> &normalize_para) {
std::vector<float> aipp_std;
if (normalize_para.size() == 6) { // If Normalize operator exist
if (normalize_para.size() == 6) { // If Normalize operation exist
auto zeros = std::find(std::begin(normalize_para), std::end(normalize_para), 0);
if (zeros == std::end(normalize_para)) {
if (std::any_of(normalize_para.begin() + 3, normalize_para.end(), [](uint32_t i) { return i == 0; })) {
@ -538,23 +538,23 @@ std::string Execute::AippCfgGenerator() {
RETURN_SECOND_IF_ERROR(rc, "");
info_->init_with_shared_ptr_ = false;
}
std::vector<uint32_t> paras; // Record the parameters value of each Ascend operators
std::vector<uint32_t> paras; // Record the parameters value of each Ascend operations
for (int32_t i = 0; i < ops_.size(); i++) {
// Validate operator ir
// Validate operation ir
json ir_info;
if (ops_[i] == nullptr) {
MS_LOG(ERROR) << "Input TensorOperation[" + std::to_string(i) + "] is null.";
return "";
}
// Define map between operator name and parameter name
// Define map between operation name and parameter name
auto rc = ops_[i]->to_json(&ir_info);
if (rc.IsError()) {
MS_LOG(ERROR) << "IR information serialize to json failed, error msg is " << rc;
return "";
}
// Collect the information of operators
// Collect the information of operations
for (auto pos = info_->op2para_map_.equal_range(ops_[i]->Name()); pos.first != pos.second; ++pos.first) {
auto paras_key_word = pos.first->second;
paras = ir_info[paras_key_word].get<std::vector<uint32_t>>();

2
mindspore/ccsrc/minddata/dataset/include/dataset/config.h
View File

@ -75,7 +75,7 @@ int32_t DATASET_API get_prefetch_size();
/// \par Example
/// \code
/// // Set a new global configuration value for the number of parallel workers.
/// // Now parallel dataset operators will run with 16 workers.
/// // Now parallel dataset operations will run with 16 workers.
/// bool rc = config::set_num_parallel_workers(16);
/// \endcode
bool DATASET_API set_num_parallel_workers(int32_t num_parallel_workers);

36
mindspore/ccsrc/minddata/dataset/include/dataset/datasets.h
View File

@ -129,7 +129,7 @@ class DATASET_API Dataset : public std::enable_shared_from_this<Dataset> {
}
/// \brief Function to set runtime number of workers.
/// \param[in] num_workers The number of threads in this operator.
/// \param[in] num_workers The number of threads in this operation.
/// \return Shared pointer to the original object.
/// \par Example
/// \code
@ -186,8 +186,8 @@ class DATASET_API Dataset : public std::enable_shared_from_this<Dataset> {
/// \note Usage restrictions:
/// 1. Supported dataset formats: 'mindrecord' only.
/// 2. To save the samples in order, set dataset's shuffle to false and num_files to 1.
/// 3. Before calling the function, do not use batch operator, repeat operator or data augmentation operators
/// with random attribute in map operator.
/// 3. Before calling the function, do not use batch operation, repeat operation or data augmentation operations
/// with random attribute in map operation.
/// 4. Mindrecord does not support bool, uint64, multi-dimensional uint8(drop dimension) nor
/// multi-dimensional string.
/// \param[in] dataset_path Path to dataset file.
@ -374,7 +374,7 @@ class DATASET_API Dataset : public std::enable_shared_from_this<Dataset> {
/// are applied in the order they appear in this list.
/// \param[in] input_columns Vector of the names of the columns that will be passed to the first
/// operation as input. The size of this list must match the number of
/// input columns expected by the first operator. The default input_columns
/// input columns expected by the first operation. The default input_columns
/// is the first column.
/// \param[in] output_columns Vector of names assigned to the columns outputted by the last operation.
/// This parameter is mandatory if len(input_columns) != len(output_columns).
@ -426,7 +426,7 @@ class DATASET_API Dataset : public std::enable_shared_from_this<Dataset> {
/// Operations are applied in the order they appear in this list.
/// \param[in] input_columns Vector of the names of the columns that will be passed to the first
/// operation as input. The size of this list must match the number of
/// input columns expected by the first operator. The default input_columns
/// input columns expected by the first operation. The default input_columns
/// is the first column.
/// \param[in] output_columns Vector of names assigned to the columns outputted by the last operation.
/// This parameter is mandatory if len(input_columns) != len(output_columns).
@ -456,7 +456,7 @@ class DATASET_API Dataset : public std::enable_shared_from_this<Dataset> {
/// the order they appear in this list.
/// \param[in] input_columns Vector of the names of the columns that will be passed to the first
/// operation as input. The size of this list must match the number of
/// input columns expected by the first operator. The default input_columns
/// input columns expected by the first operation. The default input_columns
/// is the first column.
/// \param[in] output_columns Vector of names assigned to the columns outputted by the last operation.
/// This parameter is mandatory if len(input_columns) != len(output_columns).
@ -737,7 +737,7 @@ class DATASET_API SchemaObj {
};
/// \class BatchDataset
/// \brief The result of applying Batch operator to the input dataset.
/// \brief The result of applying Batch operation to the input dataset.
class DATASET_API BatchDataset : public Dataset {
public:
/// \brief Constructor of BatchDataset.
@ -755,7 +755,7 @@ class DATASET_API BatchDataset : public Dataset {
};
/// \class BucketBatchByLengthDataset
/// \brief The result of applying BucketBatchByLength operator to the input dataset.
/// \brief The result of applying BucketBatchByLength operation to the input dataset.
class DATASET_API BucketBatchByLengthDataset : public Dataset {
public:
/// \brief Constructor of BucketBatchByLengthDataset.
@ -796,7 +796,7 @@ class DATASET_API BucketBatchByLengthDataset : public Dataset {
};
/// \class ConcatDataset
/// \brief The result of applying concat dataset operator to the input Dataset.
/// \brief The result of applying Concat operation to the input Dataset.
class DATASET_API ConcatDataset : public Dataset {
public:
/// \brief Constructor of ConcatDataset.
@ -825,7 +825,7 @@ class DATASET_API FilterDataset : public Dataset {
};
/// \class MapDataset
/// \brief The result of applying the Map operator to the input Dataset.
/// \brief The result of applying the Map operation to the input Dataset.
class DATASET_API MapDataset : public Dataset {
public:
/// \brief Constructor of MapDataset.
@ -835,7 +835,7 @@ class DATASET_API MapDataset : public Dataset {
/// are applied in the order they appear in this list.
/// \param[in] input_columns Vector of the names of the columns that will be passed to the first
/// operation as input. The size of this list must match the number of
/// input columns expected by the first operator. The default input_columns
/// input columns expected by the first operation. The default input_columns
/// is the first column.
/// \param[in] output_columns Vector of names assigned to the columns outputted by the last operation.
/// This parameter is mandatory if len(input_columns) != len(output_columns).
@ -853,7 +853,7 @@ class DATASET_API MapDataset : public Dataset {
};
/// \class ProjectDataset
/// \brief The result of applying the Project operator to the input Dataset.
/// \brief The result of applying the Project operation to the input Dataset.
class DATASET_API ProjectDataset : public Dataset {
public:
/// \brief Constructor of ProjectDataset.
@ -867,7 +867,7 @@ class DATASET_API ProjectDataset : public Dataset {
};
/// \class RenameDataset
/// \brief The result of applying the Rename operator to the input Dataset.
/// \brief The result of applying the Rename operation to the input Dataset.
class DATASET_API RenameDataset : public Dataset {
public:
/// \brief Constructor of RenameDataset.
@ -883,7 +883,7 @@ class DATASET_API RenameDataset : public Dataset {
};
/// \class RepeatDataset
/// \brief The result of applying the Repeat operator to the input Dataset.
/// \brief The result of applying the Repeat operation to the input Dataset.
class DATASET_API RepeatDataset : public Dataset {
public:
/// \brief Constructor of RepeatDataset.
@ -897,7 +897,7 @@ class DATASET_API RepeatDataset : public Dataset {
};
/// \class ShuffleDataset
/// \brief The result of applying the Shuffle operator to the input Dataset.
/// \brief The result of applying the Shuffle operation to the input Dataset.
class DATASET_API ShuffleDataset : public Dataset {
public:
/// \brief Constructor of ShuffleDataset.
@ -911,7 +911,7 @@ class DATASET_API ShuffleDataset : public Dataset {
};
/// \class SkipDataset
/// \brief The result of applying the Skip operator to the input Dataset.
/// \brief The result of applying the Skip operation to the input Dataset.
class DATASET_API SkipDataset : public Dataset {
public:
/// \brief Constructor of SkipDataset.
@ -925,7 +925,7 @@ class DATASET_API SkipDataset : public Dataset {
};
/// \class TakeDataset
/// \brief The result of applying the Take operator to the input Dataset.
/// \brief The result of applying the Take operation to the input Dataset.
class DATASET_API TakeDataset : public Dataset {
public:
/// \brief Constructor of TakeDataset.
@ -939,7 +939,7 @@ class DATASET_API TakeDataset : public Dataset {
};
/// \class ZipDataset
/// \brief The result of applying the Zip operator to the input Dataset.
/// \brief The result of applying the Zip operation to the input Dataset.
class DATASET_API ZipDataset : public Dataset {
public:
/// \brief Constructor of ZipDataset.

4
mindspore/python/mindspore/dataset/__init__.py
View File

@ -54,12 +54,12 @@ The specific steps are as follows:
accept a variety of parameters such as sampler, data slicing, and data shuffle;
- Dataset operation: The user uses the dataset object method `.shuffle` / `.filter` / `.skip` / `.split` /
`.take` / ... to further shuffle, filter, skip, and obtain the maximum number of samples of datasets;
- Dataset sample transform operation: The user can add data transform operators
- Dataset sample transform operation: The user can add data transform operations
(`vision transform <https://www.mindspore.cn/docs/en/master/api_python/mindspore.dataset.vision.html>`_,
`NLP transform <https://www.mindspore.cn/docs/en/master/api_python/mindspore.dataset.text.html>`_,
`audio transform <https://www.mindspore.cn/docs/en/master/api_python/mindspore.dataset.audio.html>`_) to the map
operation to perform transformations. During data preprocessing, multiple map operations can be defined to
perform different transform operations to different fields. The data transform operator can also be a
perform different transform operations to different fields. The data transform operation can also be a
user-defined transform `pyfunc` (Python function);
- Batch: After the transformation of the samples, the user can use the batch operation to organize multiple samples
into batches, or use self-defined batch logic with the parameter `per_batch_map` applied;

2
mindspore/python/mindspore/dataset/audio/__init__.py
View File

@ -37,7 +37,7 @@ Descriptions of common data processing terms are as follows:
- TensorOperation, the base class of all data processing operations implemented in C++.
- AudioTensorOperation, the base class of all audio processing operations. It is a derived class of TensorOperation.
The data transform operator can be executed in the data processing pipeline or in the eager mode:
The data transform operation can be executed in the data processing pipeline or in the eager mode:
- Pipeline mode is generally used to process datasets. For examples, please refer to
`introduction to data processing pipeline <https://www.mindspore.cn/docs/en/master/api_python/

2
mindspore/python/mindspore/dataset/callback/ds_callback.py
View File

@ -134,7 +134,7 @@ class WaitedDSCallback(Callback, DSCallback):
<https://www.mindspore.cn/docs/en/master/api_python/train/mindspore.train.Callback.html#mindspore.train.Callback>`_.
It can be used to execute a custom callback method before a step or an epoch, such as
updating the parameters of operators according to the loss of the previous training epoch in auto augmentation.
updating the parameters of operations according to the loss of the previous training epoch in auto augmentation.
Users can obtain the network training context through `train_run_context`, such as
`network`, `train_network`, `epoch_num`, `batch_num`, `loss_fn`, `optimizer`, `parallel_mode`,

8
mindspore/python/mindspore/dataset/core/config.py
View File

@ -205,7 +205,7 @@ def set_num_parallel_workers(num):
Examples:
>>> # Set a new global configuration value for the number of parallel workers.
>>> # Now parallel dataset operators will run with 8 workers.
>>> # Now parallel dataset operations will run with 8 workers.
>>> ds.config.set_num_parallel_workers(8)
"""
if not isinstance(num, int) or isinstance(num, bool):
@ -244,7 +244,7 @@ def set_numa_enable(numa_enable):
Examples:
>>> # Set a new global configuration value for the state of numa enabled.
>>> # Now parallel dataset operators will run with numa bind function
>>> # Now parallel dataset operations will run with numa bind function
>>> ds.config.set_numa_enable(True)
"""
if not isinstance(numa_enable, bool):
@ -613,13 +613,13 @@ def get_enable_shared_mem():
def set_enable_shared_mem(enable):
"""
Set the default state of shared memory flag. If shared_mem_enable is True, will use shared memory queues
to pass data to processes that are created for operators that set python_multiprocessing=True.
to pass data to processes that are created for operations that set python_multiprocessing=True.
Note:
`set_enable_shared_mem` is not supported on Windows and MacOS platforms yet.
Args:
enable (bool): Whether to use shared memory in operators when python_multiprocessing=True.
enable (bool): Whether to use shared memory in operations when python_multiprocessing=True.
Raises:
TypeError: If `enable` is not a boolean data type.

14
mindspore/python/mindspore/dataset/core/validator_helpers.py
View File

@ -781,19 +781,19 @@ def check_dataset_num_shards_shard_id(num_shards, shard_id):
def deprecator_factory(version, old_module, new_module, substitute_name=None, substitute_module=None):
"""Decorator factory function for deprecated operator to log deprecation warning message.
"""Decorator factory function for deprecated operation to log deprecation warning message.
Args:
version (str): Version that the operator is deprecated.
old_module (str): Old module for deprecated operator.
new_module (str): New module for deprecated operator.
substitute_name (str, optional): The substitute name for deprecated operator.
substitute_module (str, optional): The substitute module for deprecated operator.
version (str): Version that the operation is deprecated.
old_module (str): Old module for deprecated operation.
new_module (str): New module for deprecated operation.
substitute_name (str, optional): The substitute name for deprecated operation.
substitute_module (str, optional): The substitute module for deprecated operation.
"""
def decorator(op):
def wrapper(*args, **kwargs):
# Get operator class name for operator class which applies decorator to __init__()
# Get operation class name for operation class which applies decorator to __init__()
name = str(op).split()[1].split(".")[0]
# Build message
message = f"'{name}' from " + f"{old_module}" + f" is deprecated from version " f"{version}" + \

80
mindspore/python/mindspore/dataset/engine/datasets.py
View File

@ -228,7 +228,7 @@ def _get_operator_process():
Inner implemented method, mainly for passing sub-process id in C layer
Returns:
dict, mapping dict of operator id and corresponding process id.
dict, mapping dict of operation id and corresponding process id.
"""
global _OP_PROCESS
process_info = _OP_PROCESS
@ -284,20 +284,20 @@ class Dataset:
|
MappableDataset
DatasetOperator: MapDataset(UnionBaseDataset)
BatchDataset(UnionBaseDataset)
PaddedBatchDataset(UnionBaseDataset)
BucketBatchByLengthDataset(UnionBaseDataset)
ShuffleDataset(UnionBaseDataset)
FilterDataset(UnionBaseDataset)
RepeatDataset(UnionBaseDataset)
SkipDataset(UnionBaseDataset)
TakeDataset(UnionBaseDataset)
ZipDataset(UnionBaseDataset)
ConcatDataset(UnionBaseDataset)
RenameDataset(UnionBaseDataset)
ProjectDataset(UnionBaseDataset)
SyncWaitDataset(UnionBaseDataset)
DatasetOperation: MapDataset(UnionBaseDataset)
BatchDataset(UnionBaseDataset)
PaddedBatchDataset(UnionBaseDataset)
BucketBatchByLengthDataset(UnionBaseDataset)
ShuffleDataset(UnionBaseDataset)
FilterDataset(UnionBaseDataset)
RepeatDataset(UnionBaseDataset)
SkipDataset(UnionBaseDataset)
TakeDataset(UnionBaseDataset)
ZipDataset(UnionBaseDataset)
ConcatDataset(UnionBaseDataset)
RenameDataset(UnionBaseDataset)
ProjectDataset(UnionBaseDataset)
SyncWaitDataset(UnionBaseDataset)
Impl Dataset - vision: ImageFolderDataset(MappableDataset, VisionBaseDataset)
USPSDataset(SourceDataset, VisionBaseDataset)
@ -349,7 +349,7 @@ class Dataset:
@staticmethod
def _get_operator_id(dataset):
"""
Internal method to iterate the tree and obtain op_id of each operator.
Internal method to iterate the tree and obtain op_id of each operation.
Returns:
Dataset, the root dataset of the tree.
@ -733,7 +733,7 @@ class Dataset:
Dataset, dataset shuffled.
Raises:
RuntimeError: If exist sync operators before shuffle.
RuntimeError: If exist sync operations before shuffle.
Examples:
>>> # dataset is an instance object of Dataset
@ -808,7 +808,7 @@ class Dataset:
Each operation will be passed one or more columns from the dataset as input, and one or
more columns will be outputted. The first operation will be passed the columns specified
in input_columns as input. If there is more than one operator in operations, the outputted
in input_columns as input. If there is more than one operation in operations, the outputted
columns of the previous operation are used as the input columns for the next operation.
The columns outputted by the very last operation will be assigned names specified by
@ -831,7 +831,7 @@ class Dataset:
applied on the dataset. Operations are applied in the order they appear in this list.
input_columns (Union[str, list[str]], optional): List of the names of the columns that will be passed to
the first operation as input. The size of this list must match the number of
input columns expected by the first operator. (default=None, the first
input columns expected by the first operation. (default=None, the first
operation will be passed however many columns that are required, starting from
the first column).
output_columns (Union[str, list[str]], optional): List of names assigned to the columns outputted by
@ -980,7 +980,7 @@ class Dataset:
>>> dataset = dataset.repeat(50)
>>>
>>> # Create a dataset where the dataset is first repeated for
>>> # 50 epochs before shuffling. The shuffle operator will treat
>>> # 50 epochs before shuffling. The shuffle operation will treat
>>> # the entire 50 epochs as one big dataset.
>>> dataset = dataset.repeat(50)
>>> dataset = dataset.shuffle(10)
@ -1374,8 +1374,8 @@ class Dataset:
Note:
1. To save the samples in order, set dataset's shuffle to False and num_files to 1.
2. Before calling the function, do not use batch operator, repeat operator or data augmentation operators
with random attribute in map operator.
2. Before calling the function, do not use batch operation, repeat operation or data augmentation operations
with random attribute in map operation.
3. When array dimension is variable, one-dimensional arrays or
multi-dimensional arrays with variable dimension 0 are supported.
4. Mindrecord does not support uint64, multi-dimensional uint8(drop dimension) nor
@ -1819,7 +1819,7 @@ class Dataset:
condition_name (str): The condition name that is used to toggle sending next row.
num_batch (Union[int, None]): The number of batches (rows) that are released.
When num_batch is None, it will default to the number specified by the
sync_wait operator (default=None).
sync_wait operation (default=None).
data (Any): The data passed to the callback, user defined (default=None).
"""
if (not isinstance(num_batch, int) and num_batch is not None) or \
@ -2339,7 +2339,7 @@ class MappableDataset(SourceDataset):
class BucketBatchByLengthDataset(UnionBaseDataset):
"""
The result of applying BucketBatchByLength operator to the input dataset.
The result of applying BucketBatchByLength operation to the input dataset.
"""
def __init__(self, input_dataset, column_names, bucket_boundaries, bucket_batch_sizes, element_length_function,
@ -2390,7 +2390,7 @@ def _check_shm_usage(num_worker, queue_size, max_rowsize, num_queues=1):
class BatchDataset(UnionBaseDataset):
"""
The result of applying Batch operator to the input dataset.
The result of applying Batch operation to the input dataset.
Args:
input_dataset (Dataset): Input Dataset to be batched.
@ -2515,7 +2515,7 @@ class BatchDataset(UnionBaseDataset):
class BatchInfo(cde.CBatchInfo):
"""
Only the batch size function and per_batch_map of the batch operator can dynamically adjust parameters
Only the batch size function and per_batch_map of the batch operation can dynamically adjust parameters
based on the number of batches and epochs during training.
"""
@ -2603,7 +2603,7 @@ class BlockReleasePair:
class PaddedBatchDataset(UnionBaseDataset):
"""
The result of applying Batch operator to the input dataset.
The result of applying Batch operation to the input dataset.
Args:
input_dataset (Dataset): Input Dataset to be batched.
@ -2746,14 +2746,14 @@ class SyncWaitDataset(UnionBaseDataset):
class ShuffleDataset(UnionBaseDataset):
"""
The result of applying Shuffle operator to the input Dataset.
The result of applying Shuffle operation to the input Dataset.
Args:
input_dataset (Dataset): Input Dataset to be shuffled.
buffer_size (int): Size of the buffer.
Raises:
RuntimeError: If exist sync operators before shuffle.
RuntimeError: If exist sync operations before shuffle.
"""
def __init__(self, input_dataset, buffer_size):
@ -3280,7 +3280,7 @@ class _PythonMultiprocessing(cde.PythonMultiprocessingRuntime):
class MapDataset(UnionBaseDataset):
"""
The result of applying the Map operator to the input Dataset.
The result of applying the Map operation to the input Dataset.
Args:
input_dataset (Dataset): Input Dataset to be mapped.
@ -3288,9 +3288,9 @@ class MapDataset(UnionBaseDataset):
to another nested structure of tensor (default=None).
input_columns (Union[str, list[str]]): List of names of the input columns
(default=None, the operations will be applied on the first columns in the dataset).
The size of the list should match the number of inputs of the first operator.
The size of the list should match the number of inputs of the first operation.
output_columns (Union[str, list[str]], optional): List of names of the output columns.
The size of the list should match the number of outputs of the last operator
The size of the list should match the number of outputs of the last operation
(default=None, output columns will be the input columns, i.e., the columns will
be replaced).
num_parallel_workers (int, optional): Number of workers to process the dataset
@ -3505,7 +3505,7 @@ class FilterDataset(UnionBaseDataset):
class RepeatDataset(UnionBaseDataset):
"""
The result of applying Repeat operator to the input Dataset.
The result of applying Repeat operation to the input Dataset.
Args:
input_dataset (Dataset): Input Dataset to be repeated.
@ -3522,7 +3522,7 @@ class RepeatDataset(UnionBaseDataset):
class SkipDataset(UnionBaseDataset):
"""
The result of applying Skip operator to the input Dataset.
The result of applying Skip operation to the input Dataset.
Args:
input_dataset (Dataset): Input dataset to have elements skipped.
@ -3539,7 +3539,7 @@ class SkipDataset(UnionBaseDataset):
class TakeDataset(UnionBaseDataset):
"""
The result of applying Take operator to the input Dataset.
The result of applying Take operation to the input Dataset.
Args:
input_dataset (Dataset): Input Dataset to have elements taken from.
@ -3556,7 +3556,7 @@ class TakeDataset(UnionBaseDataset):
class ZipDataset(UnionBaseDataset):
"""
The result of applying Zip operator to the input Dataset.
The result of applying Zip operation to the input Dataset.
Args:
datasets (tuple): A tuple of datasets to be zipped together.
@ -3577,7 +3577,7 @@ class ZipDataset(UnionBaseDataset):
class ConcatDataset(UnionBaseDataset):
"""
The result of applying concat dataset operator to the input Dataset.
The result of applying Concat operation to the input Dataset.
Args:
datasets (list): A list of datasets to be concatenated together.
@ -3688,7 +3688,7 @@ class ConcatDataset(UnionBaseDataset):
class RenameDataset(UnionBaseDataset):
"""
The result of applying Rename operator to the input Dataset.
The result of applying Rename operation to the input Dataset.
Args:
input_dataset (Dataset): Input Dataset to be Renamed.
@ -3717,7 +3717,7 @@ def to_list(items):
class ProjectDataset(UnionBaseDataset):
"""
The result of applying Project operator to the input Dataset.
The result of applying Project operation to the input Dataset.
Args:
input_dataset (Dataset): Input Dataset to be Projected.
@ -3796,7 +3796,7 @@ class _ToDevice:
class TransferDataset(Dataset):
"""
The result of applying TDT operator to the input Dataset.
The result of applying TDT operation to the input Dataset.
Args:
input_dataset (Dataset): Input Dataset to be transferred.

2
mindspore/python/mindspore/dataset/engine/samplers.py
View File

@ -778,7 +778,7 @@ class IterSampler(Sampler):
User provided an iterable object without inheriting from our Sampler class.
Note:
This class exists to allow handshake logic between dataset operators and user defined samplers.
This class exists to allow handshake logic between dataset operations and user defined samplers.
By constructing this object we avoid the user having to inherit from our Sampler class.
Args:

6
mindspore/python/mindspore/dataset/engine/validators.py
View File

@ -935,7 +935,7 @@ def check_lfw_dataset(method):
def check_save(method):
"""A wrapper that wraps a parameter checker around the saved operator."""
"""A wrapper that wraps a parameter checker around the saved operation."""
@wraps(method)
def new_method(self, *args, **kwargs):
@ -3021,8 +3021,8 @@ def deprecated(version, substitute=None):
"""deprecated warning
Args:
version (str): version that the operator or function is deprecated.
substitute (str): the substitute name for deprecated operator or function.
version (str): version that the operation or function is deprecated.
substitute (str): the substitute name for deprecated operation or function.
"""
def decorate(func):

5
mindspore/python/mindspore/dataset/text/__init__.py
View File

@ -24,12 +24,15 @@ Common imported modules in corresponding API examples are as follows:
import mindspore.dataset as ds
import mindspore.dataset.text as text
See `Text Transforms
<https://www.mindspore.cn/tutorials/en/master/beginner/transforms.html#text-transforms>`_ tutorial for more details.
Descriptions of common data processing terms are as follows:
- TensorOperation, the base class of all data processing operations implemented in C++.
- TextTensorOperation, the base class of all text processing operations. It is a derived class of TensorOperation.
The data transform operator can be executed in the data processing pipeline or in the eager mode:
The data transform operation can be executed in the data processing pipeline or in the eager mode:
- Pipeline mode is generally used to process datasets. For examples, please refer to
`introduction to data processing pipeline <https://www.mindspore.cn/docs/en/master/api_python/

6
mindspore/python/mindspore/dataset/text/transforms.py
View File

@ -31,7 +31,7 @@ Examples:
>>> tokenizer = text.UnicodeCharTokenizer()
>>> # Load vocabulary from list
>>> vocab = text.Vocab.from_list(word_list=['深', '圳', '欢', '迎', '您'])
>>> # Use Lookup operator to map tokens to ids
>>> # Use Lookup operation to map tokens to ids
>>> lookup = text.Lookup(vocab=vocab)
>>> text_file_dataset = text_file_dataset.map(operations=[tokenizer, lookup])
>>> # if text line in dataset_file is:
@ -294,7 +294,7 @@ class Lookup(TextTensorOperation):
Examples:
>>> # Load vocabulary from list
>>> vocab = text.Vocab.from_list(['深', '圳', '欢', '迎', '您'])
>>> # Use Lookup operator to map tokens to ids
>>> # Use Lookup operation to map tokens to ids
>>> lookup = text.Lookup(vocab)
>>> text_file_dataset = text_file_dataset.map(operations=[lookup])
"""
@ -551,7 +551,7 @@ class ToVectors(TextTensorOperation):
Examples:
>>> # Load vectors from file
>>> vectors = text.Vectors.from_file("/path/to/vectors/file")
>>> # Use ToVectors operator to map tokens to vectors
>>> # Use ToVectors operation to map tokens to vectors
>>> to_vectors = text.ToVectors(vectors)
>>> text_file_dataset = text_file_dataset.map(operations=[to_vectors])
"""

4
mindspore/python/mindspore/dataset/transforms/__init__.py
View File

@ -30,8 +30,8 @@ Note: Legacy c_transforms and py_transforms are deprecated but can still be impo
from mindspore.dataset.transforms import c_transforms
from mindspore.dataset.transforms import py_transforms
See `Common Data Processing and Augmentation
<https://www.mindspore.cn/tutorials/en/master/advanced/dataset/augment_common_data.html>`_ tutorial for more details.
See `Common Transforms
<https://www.mindspore.cn/tutorials/en/master/beginner/transforms.html#common-transforms>`_ tutorial for more details.
Descriptions of common data processing terms are as follows:

14
mindspore/python/mindspore/dataset/transforms/validators.py
View File

@ -245,7 +245,7 @@ def check_random_transform_ops(method):
def check_transform_op_type(ind, op):
"""Check the operator."""
"""Check the operation."""
# c_vision.HWC2CHW error
# py_vision.HWC2CHW error
if type(op) == type: # pylint: disable=unidiomatic-typecheck
@ -389,22 +389,22 @@ def check_type_cast(method):
def deprecated_c_transforms(substitute_name=None, substitute_module=None):
"""Decorator for version 1.8 deprecation warning for legacy mindspore.dataset.transforms.c_transforms operator.
"""Decorator for version 1.8 deprecation warning for legacy mindspore.dataset.transforms.c_transforms operation.
Args:
substitute_name (str, optional): The substitute name for deprecated operator.
substitute_module (str, optional): The substitute module for deprecated operator.
substitute_name (str, optional): The substitute name for deprecated operation.
substitute_module (str, optional): The substitute module for deprecated operation.
"""
return deprecator_factory("1.8", "mindspore.dataset.transforms.c_transforms", "mindspore.dataset.transforms",
substitute_name, substitute_module)
def deprecated_py_transforms(substitute_name=None, substitute_module=None):
"""Decorator for version 1.8 deprecation warning for legacy mindspore.dataset.transforms.py_transforms operator.
"""Decorator for version 1.8 deprecation warning for legacy mindspore.dataset.transforms.py_transforms operation.
Args:
substitute_name (str, optional): The substitute name for deprecated operator.
substitute_module (str, optional): The substitute module for deprecated operator.
substitute_name (str, optional): The substitute name for deprecated operation.
substitute_module (str, optional): The substitute module for deprecated operation.
"""
return deprecator_factory("1.8", "mindspore.dataset.transforms.py_transforms", "mindspore.dataset.transforms",
substitute_name, substitute_module)

6
mindspore/python/mindspore/dataset/vision/__init__.py
View File

@ -31,8 +31,8 @@ Note: Legacy c_transforms and py_transforms are deprecated but can still be impo
import mindspore.dataset.vision.c_transforms as c_vision
import mindspore.dataset.vision.py_transforms as py_vision
See `Image Data Processing and Augmentation
<https://www.mindspore.cn/tutorials/en/master/advanced/dataset/augment_image_data.html>`_ tutorial for more details.
See `Vision Transforms
<https://www.mindspore.cn/tutorials/en/master/beginner/transforms.html#vision-transforms>`_ tutorial for more details.
Descriptions of common data processing terms are as follows:
@ -40,7 +40,7 @@ Descriptions of common data processing terms are as follows:
- ImageTensorOperation, the base class of all image processing operations. It is a derived class of TensorOperation.
- PyTensorOperation, the base class of all data processing operations implemented in Python.
The data transform operator can be executed in the data processing pipeline or in the eager mode:
The data transform operation can be executed in the data processing pipeline or in the eager mode:
- Pipeline mode is generally used to process datasets. For examples, please refer to
`introduction to data processing pipeline <https://www.mindspore.cn/docs/en/master/api_python/

20
mindspore/python/mindspore/dataset/vision/c_transforms.py
View File

@ -196,7 +196,7 @@ class AutoAugment(ImageTensorOperation):
- AutoAugmentPolicy.SVHN, means to apply AutoAugment learned on SVHN dataset.
interpolation (Inter, optional): Image interpolation mode for Resize operator (default=Inter.NEAREST).
interpolation (Inter, optional): Image interpolation mode for Resize operation (default=Inter.NEAREST).
It can be any of [Inter.NEAREST, Inter.BILINEAR, Inter.BICUBIC, Inter.AREA].
- Inter.NEAREST: means interpolation method is nearest-neighbor interpolation.
@ -247,7 +247,7 @@ class AutoAugment(ImageTensorOperation):
class AutoContrast(ImageTensorOperation):
"""
Apply automatic contrast on input image. This operator calculates histogram of image, reassign cutoff percent
Apply automatic contrast on input image. This operation calculates histogram of image, reassign cutoff percent
of the lightest pixels from histogram to 255, and reassign cutoff percent of the darkest pixels from histogram to 0.
Args:
@ -490,7 +490,7 @@ class Crop(ImageTensorOperation):
class CutMixBatch(ImageTensorOperation):
"""
Apply CutMix transformation on input batch of images and labels.
Note that you need to make labels into one-hot format and batched before calling this operator.
Note that you need to make labels into one-hot format and batched before calling this operation.
Args:
image_batch_format (ImageBatchFormat): The method of padding. Can be any of
@ -737,7 +737,7 @@ class HWC2CHW(ImageTensorOperation):
class Invert(ImageTensorOperation):
"""
Apply invert on input image in RGB mode. This operator will reassign every pixel to (255 - pixel).
Apply invert on input image in RGB mode. This operation will reassign every pixel to (255 - pixel).
Raises:
RuntimeError: If given tensor shape is not <H, W, C>.
@ -768,7 +768,7 @@ class MixUpBatch(ImageTensorOperation):
The lambda is generated based on the specified alpha value. Two coefficients x1, x2 are randomly generated
in the range [alpha, 1], and lambda = (x1 / (x1 + x2)).
Note that you need to make labels into one-hot format and batched before calling this operator.
Note that you need to make labels into one-hot format and batched before calling this operation.
Args:
alpha (float, optional): Hyperparameter of beta distribution. The value must be positive (default = 1.0).
@ -802,7 +802,7 @@ class MixUpBatch(ImageTensorOperation):
class Normalize(ImageTensorOperation):
"""
Normalize the input image with respect to mean and standard deviation. This operator will normalize
Normalize the input image with respect to mean and standard deviation. This operation will normalize
the input image with: output[channel] = (input[channel] - mean[channel]) / std[channel], where channel >= 1.
Note:
@ -1346,7 +1346,7 @@ class RandomCrop(ImageTensorOperation):
class RandomCropDecodeResize(ImageTensorOperation):
"""
A combination of `Crop`, `Decode` and `Resize`. It will get better performance for JPEG images. This operator
A combination of `Crop`, `Decode` and `Resize`. It will get better performance for JPEG images. This operation
will crop the input image at a random location, decode the cropped image in RGB mode, and resize the decoded image.
Args:
@ -1357,7 +1357,7 @@ class RandomCropDecodeResize(ImageTensorOperation):
original size to be cropped, which must be non-negative (default=(0.08, 1.0)).
ratio (Union[list, tuple], optional): Range [min, max) of aspect ratio to be
cropped, which must be non-negative (default=(3. / 4., 4. / 3.)).
interpolation (Inter, optional): Image interpolation mode for resize operator(default=Inter.BILINEAR).
interpolation (Inter, optional): Image interpolation mode for resize operation (default=Inter.BILINEAR).
It can be any of [Inter.BILINEAR, Inter.NEAREST, Inter.BICUBIC, Inter.AREA, Inter.PILCUBIC].
- Inter.BILINEAR, means interpolation method is bilinear interpolation.
@ -1708,7 +1708,7 @@ class RandomPosterize(ImageTensorOperation):
class RandomResizedCrop(ImageTensorOperation):
"""
This operator will crop the input image randomly, and resize the cropped image using a selected interpolation mode.
This operation will crop the input image randomly, and resize the cropped image using a selected interpolation mode.
Note:
If the input image is more than one, then make sure that the image size is the same.
@ -2185,7 +2185,7 @@ class RandomVerticalFlipWithBBox(ImageTensorOperation):
class Rescale(ImageTensorOperation):
"""
Rescale the input image with the given rescale and shift. This operator will rescale the input image
Rescale the input image with the given rescale and shift. This operation will rescale the input image
with: output = image * rescale + shift.
Note:

24
mindspore/python/mindspore/dataset/vision/transforms.py
View File

@ -452,7 +452,7 @@ class AutoAugment(ImageTensorOperation):
- AutoAugmentPolicy.SVHN, means to apply AutoAugment learned on SVHN dataset.
interpolation (Inter, optional): Image interpolation mode for Resize operator (default=Inter.NEAREST).
interpolation (Inter, optional): Image interpolation mode for Resize operation (default=Inter.NEAREST).
It can be any of [Inter.NEAREST, Inter.BILINEAR, Inter.BICUBIC, Inter.AREA].
- Inter.NEAREST: means interpolation method is nearest-neighbor interpolation.
@ -504,7 +504,7 @@ class AutoAugment(ImageTensorOperation):
class AutoContrast(ImageTensorOperation, PyTensorOperation):
"""
Apply automatic contrast on input image. This operator calculates histogram of image, reassign cutoff percent
Apply automatic contrast on input image. This operation calculates histogram of image, reassign cutoff percent
of the lightest pixels from histogram to 255, and reassign cutoff percent of the darkest pixels from histogram to 0.
Args:
@ -775,7 +775,7 @@ class Crop(ImageTensorOperation):
class CutMixBatch(ImageTensorOperation):
"""
Apply CutMix transformation on input batch of images and labels.
Note that you need to make labels into one-hot format and batched before calling this operator.
Note that you need to make labels into one-hot format and batched before calling this operation.
Args:
image_batch_format (ImageBatchFormat): The method of padding. Can be any of
@ -1256,7 +1256,7 @@ class HWC2CHW(ImageTensorOperation):
class Invert(ImageTensorOperation, PyTensorOperation):
"""
Apply invert on input image in RGB mode. This operator will reassign every pixel to (255 - pixel).
Apply invert on input image in RGB mode. This operation will reassign every pixel to (255 - pixel).
Raises:
RuntimeError: If given tensor shape is not <H, W, C>.
@ -1429,7 +1429,7 @@ class MixUpBatch(ImageTensorOperation):
The lambda is generated based on the specified alpha value. Two coefficients x1, x2 are randomly generated
in the range [alpha, 1], and lambda = (x1 / (x1 + x2)).
Note that you need to make labels into one-hot format and batched before calling this operator.
Note that you need to make labels into one-hot format and batched before calling this operation.
Args:
alpha (float, optional): Hyperparameter of beta distribution. The value must be positive (default = 1.0).
@ -1464,7 +1464,7 @@ class MixUpBatch(ImageTensorOperation):
class Normalize(ImageTensorOperation):
"""
Normalize the input image with respect to mean and standard deviation. This operator will normalize
Normalize the input image with respect to mean and standard deviation. This operation will normalize
the input image with: output[channel] = (input[channel] - mean[channel]) / std[channel], where channel >= 1.
Note:
@ -1803,7 +1803,7 @@ class RandAugment(ImageTensorOperation):
of num_magnitude_bins. Default: 9.
num_magnitude_bins (int, optional): The number of different magnitude values. The number of different magnitude
values, must be greater than or equal to 2. Default: 31.
interpolation (Inter, optional): Image interpolation mode for Resize operator.
interpolation (Inter, optional): Image interpolation mode for Resize operation.
It can be any of [Inter.NEAREST, Inter.BILINEAR, Inter.BICUBIC, Inter.AREA]. Default: Inter.NEAREST.
- Inter.NEAREST: means interpolation method is nearest-neighbor interpolation.
@ -2311,7 +2311,7 @@ class RandomCrop(ImageTensorOperation, PyTensorOperation):
class RandomCropDecodeResize(ImageTensorOperation):
"""
A combination of `Crop`, `Decode` and `Resize`. It will get better performance for JPEG images. This operator
A combination of `Crop`, `Decode` and `Resize`. It will get better performance for JPEG images. This operation
will crop the input image at a random location, decode the cropped image in RGB mode, and resize the decoded image.
Args:
@ -2322,7 +2322,7 @@ class RandomCropDecodeResize(ImageTensorOperation):
original size to be cropped, which must be non-negative (default=(0.08, 1.0)).
ratio (Union[list, tuple], optional): Range [min, max) of aspect ratio to be
cropped, which must be non-negative (default=(3. / 4., 4. / 3.)).
interpolation (Inter, optional): Image interpolation mode for resize operator(default=Inter.BILINEAR).
interpolation (Inter, optional): Image interpolation mode for resize operation (default=Inter.BILINEAR).
It can be any of [Inter.BILINEAR, Inter.NEAREST, Inter.BICUBIC, Inter.AREA, Inter.PILCUBIC].
- Inter.BILINEAR, means interpolation method is bilinear interpolation.
@ -2889,7 +2889,7 @@ class RandomPosterize(ImageTensorOperation):
class RandomResizedCrop(ImageTensorOperation, PyTensorOperation):
"""
This operator will crop the input image randomly,
This operation will crop the input image randomly,
and resize the cropped image using a selected interpolation mode :class:`mindspore.dataset.vision.Inter`.
Note:
@ -3440,7 +3440,7 @@ class RandomVerticalFlipWithBBox(ImageTensorOperation):
class Rescale(ImageTensorOperation):
"""
Rescale the input image with the given rescale and shift. This operator will rescale the input image
Rescale the input image with the given rescale and shift. This operation will rescale the input image
with: output = image * rescale + shift.
Note:
@ -4089,7 +4089,7 @@ class TrivialAugmentWide(ImageTensorOperation):
Args:
num_magnitude_bins (int, optional): The number of different magnitude values,
must be greater than or equal to 2. Default: 31.
interpolation (Inter, optional): Image interpolation mode for Resize operator. Default: Inter.NEAREST.
interpolation (Inter, optional): Image interpolation mode for Resize operation. Default: Inter.NEAREST.
It can be any of [Inter.NEAREST, Inter.BILINEAR, Inter.BICUBIC, Inter.AREA].
- Inter.NEAREST: means interpolation method is nearest-neighbor interpolation.

12
mindspore/python/mindspore/dataset/vision/validators.py
View File

@ -1380,22 +1380,22 @@ def check_to_tensor(method):
def deprecated_c_vision(substitute_name=None, substitute_module=None):
"""Decorator for version 1.8 deprecation warning for legacy mindspore.dataset.vision.c_transforms operator.
"""Decorator for version 1.8 deprecation warning for legacy mindspore.dataset.vision.c_transforms operation.
Args:
substitute_name (str, optional): The substitute name for deprecated operator.
substitute_module (str, optional): The substitute module for deprecated operator.
substitute_name (str, optional): The substitute name for deprecated operation.
substitute_module (str, optional): The substitute module for deprecated operation.
"""
return deprecator_factory("1.8", "mindspore.dataset.vision.c_transforms", "mindspore.dataset.vision",
substitute_name, substitute_module)
def deprecated_py_vision(substitute_name=None, substitute_module=None):
"""Decorator for version 1.8 deprecation warning for legacy mindspore.dataset.vision.py_transforms operator.
"""Decorator for version 1.8 deprecation warning for legacy mindspore.dataset.vision.py_transforms operation.
Args:
substitute_name (str, optional): The substitute name for deprecated operator.
substitute_module (str, optional): The substitute module for deprecated operator.
substitute_name (str, optional): The substitute name for deprecated operation.
substitute_module (str, optional): The substitute module for deprecated operation.
"""
return deprecator_factory("1.8", "mindspore.dataset.vision.py_transforms", "mindspore.dataset.vision",
substitute_name, substitute_module)

2
tests/ut/cpp/dataset/c_api_vision_a_to_q_test.cc
View File

@ -2559,7 +2559,7 @@ TEST_F(MindDataTestPipeline, TestPosterizeParamCheck) {
}
/// Feature：AdjustHue op
/// Description: Test function of operator when hue_factor is 0.2
/// Description: Test function of operation when hue_factor is 0.2
/// Expectation: Create an ImageFolder dataset then do auto AjustHue on it
TEST_F(MindDataTestPipeline, TestAdjustHue) {
MS_LOG(INFO) << "Doing MindDataTestPipeline-TestAdjustHue.";

2
tests/ut/cpp/dataset/execution_tree_test.cc
View File

@ -68,7 +68,7 @@ TEST_F(MindDataTestExecutionTree, TestExecutionTree1) {
ASSERT_NE(root_op, nullptr);
// At this point, since move semantic was used,
// I don't have any operator access myself now.
// I don't have any operation access myself now.
// Ownership is fully transferred into the tree.
// explicitly drive tree destruction rather than

4
tests/ut/cpp/dataset/memory_pool_test.cc
View File

@ -41,7 +41,7 @@ TEST_F(MindDataTestMemoryPool, DumpPoolInfo) {
}
/// Feature: MemoryPool
/// Description: Test delete operator on heap
/// Description: Test delete operation on heap
/// Expectation: Runs successfully
TEST_F(MindDataTestMemoryPool, TestOperator1) {
Status rc;
@ -52,7 +52,7 @@ TEST_F(MindDataTestMemoryPool, TestOperator1) {
}
/// Feature: MemoryPool
/// Description: Test assignment operator on heap
/// Description: Test assignment operation on heap
/// Expectation: Runs successfully
TEST_F(MindDataTestMemoryPool, TestOperator3) {
Status rc;

4
tests/ut/cpp/dataset/tree_modifying_function_test.cc
View File

@ -439,8 +439,8 @@ TEST_F(MindDataTestTreeModifying, Drop03) {
* ds7 ds3 ds2
*
*
* ds4->Drop() will raise an error because we cannot add the children of an n-ary operator (ds4) to a unary operator
* (ds6).
* ds4->Drop() will raise an error because we cannot add the children of an n-ary operation (ds4) to a unary
* operation (ds6).
*
*/
std::string folder_path = datasets_root_path_ + "/testPK/data/";

2
tests/ut/python/dataset/test_cache_map.py
View File

@ -13,7 +13,7 @@
# limitations under the License.
# ==============================================================================
"""
Testing cache operator with mappable datasets
Testing cache operation with mappable datasets
"""
import os
import pytest

4
tests/ut/python/dataset/test_cache_nomap.py
View File

@ -13,7 +13,7 @@
# limitations under the License.
# ==============================================================================
"""
Testing cache operator with non-mappable datasets
Testing cache operation with non-mappable datasets
"""
import os
import itertools
@ -221,7 +221,7 @@ def test_cache_nomap_basic5():
Feature: DatasetCache op
Description: Test a TFReaderDataset (a non mappable dataset) with a Cache over it just after the leaf.
Same as test 3, but this one does not have Shuffle arg, causing TF to default to global
shuffle which attempts to inject a Shuffle operator. However, since there is a Cache
shuffle which attempts to inject a Shuffle operation. However, since there is a Cache
we do not need global shuffle, so the shuffle will not be built. It ends up being
identical to test basic 3, however we arrive at the same tree in different codepaths
(if there was no Cache, then the Shuffle is built)

2
tests/ut/python/dataset/test_datasets_caltech101.py
View File

@ -13,7 +13,7 @@
# limitations under the License.
# ==============================================================================
"""
Test Caltech101 dataset operators
Test Caltech101 dataset operations
"""
import os

2
tests/ut/python/dataset/test_datasets_caltech256.py
View File

@ -13,7 +13,7 @@
# limitations under the License.
# ==============================================================================
"""
Test Caltech256 dataset operators
Test Caltech256 dataset operations
"""
import numpy as np
import pytest

2
tests/ut/python/dataset/test_datasets_cifarop.py
View File

@ -13,7 +13,7 @@
# limitations under the License.
# ==============================================================================
"""
Test Cifar10 and Cifar100 dataset operators
Test Cifar10 and Cifar100 dataset operations
"""
import os
import pytest

2
tests/ut/python/dataset/test_datasets_cmu_arctic.py
View File

@ -13,7 +13,7 @@
# limitations under the License.
# ==============================================================================
"""
Test CMUArctic dataset operators
Test CMUArctic dataset operations
"""
import numpy as np
import pytest

2
tests/ut/python/dataset/test_datasets_emnist.py
View File

@ -13,7 +13,7 @@
# limitations under the License.
# ==============================================================================
"""
Test EMnist dataset operators
Test EMnist dataset operations
"""
import os

2
tests/ut/python/dataset/test_datasets_fake_image.py
View File

@ -13,7 +13,7 @@
# limitations under the License.
# ==============================================================================
"""
Test FakeImage dataset operators
Test FakeImage dataset operations
"""
import matplotlib.pyplot as plt

2
tests/ut/python/dataset/test_datasets_fashion_mnist.py
View File

@ -13,7 +13,7 @@
# limitations under the License.
# ==============================================================================
"""
Test FashionMnist dataset operators
Test FashionMnist dataset operations
"""
import os

2
tests/ut/python/dataset/test_datasets_flowers102.py
View File

@ -13,7 +13,7 @@
# limitations under the License.
# ==============================================================================
"""
Test Flowers102 dataset operators
Test Flowers102 dataset operations
"""
import os

2
tests/ut/python/dataset/test_datasets_gtzan.py
View File

@ -13,7 +13,7 @@
# limitations under the License.
# ==============================================================================
"""
Test Gtzan dataset operators.
Test Gtzan dataset operations.
"""
import numpy as np
import pytest

2
tests/ut/python/dataset/test_datasets_kmnist.py
View File

@ -13,7 +13,7 @@
# limitations under the License.
# ==============================================================================
"""
Test KMnist dataset operators
Test KMnist dataset operations
"""
import os

2
tests/ut/python/dataset/test_datasets_libri_tts.py
View File

@ -13,7 +13,7 @@
# limitations under the License.
# ==============================================================================
"""
Test LibriTTS dataset operators
Test LibriTTS dataset operations
"""
import numpy as np
import pytest

2
tests/ut/python/dataset/test_datasets_lj_speech.py
View File

@ -13,7 +13,7 @@
# limitations under the License.
# ==============================================================================
"""
Test LJSpeech dataset operators
Test LJSpeech dataset operations
"""
import numpy as np
import pytest

2
tests/ut/python/dataset/test_datasets_lsun.py
View File

@ -13,7 +13,7 @@
# limitations under the License.
# ==============================================================================
"""
Test LSUN dataset operators
Test LSUN dataset operations
"""
import pytest

2
tests/ut/python/dataset/test_datasets_mnist.py
View File

@ -13,7 +13,7 @@
# limitations under the License.
# ==============================================================================
"""
Test Mnist dataset operators
Test Mnist dataset operations
"""
import os
import pytest

2
tests/ut/python/dataset/test_datasets_obs_mindrecord.py
View File

@ -13,7 +13,7 @@
# limitations under the License.
# ==============================================================================
"""
Test OBSMindDataset operator
Test OBSMindDataset operations
"""
import pytest

2
tests/ut/python/dataset/test_datasets_omniglot.py
View File

@ -13,7 +13,7 @@
# limitations under the License.
# ==============================================================================
"""
Test Omniglot dataset operators
Test Omniglot dataset operations
"""
import mindspore.dataset as ds
import mindspore.dataset.vision as vision

2
tests/ut/python/dataset/test_datasets_photo_tour.py
View File

@ -13,7 +13,7 @@
# limitations under the License.
# ==============================================================================
"""
Test PhotoTour dataset operator
Test PhotoTour dataset operations
"""
import os

2
tests/ut/python/dataset/test_datasets_places365.py
View File

@ -13,7 +13,7 @@
# limitations under the License.
# ==============================================================================
"""
Test Places365 dataset operators
Test Places365 dataset operations
"""
import os

2
tests/ut/python/dataset/test_datasets_qmnist.py
View File

@ -13,7 +13,7 @@
# limitations under the License.
# ==============================================================================
"""
Test QMnistDataset operator
Test QMnistDataset operations
"""
import os

2
tests/ut/python/dataset/test_datasets_sbu.py
View File

@ -13,7 +13,7 @@
# limitations under the License.
# ==============================================================================
"""
Test USPS dataset operators
Test USPS dataset operations
"""
import os

2
tests/ut/python/dataset/test_datasets_speech_commands.py
View File

@ -13,7 +13,7 @@
# limitations under the License.
# ==============================================================================
"""
Test SpeechCommands dataset operators
Test SpeechCommands dataset operations
"""
import pytest
import numpy as np

2
tests/ut/python/dataset/test_datasets_stl10.py
View File

@ -13,7 +13,7 @@
# limitations under the License.
# ==============================================================================
"""
Test STL10 dataset operators
Test STL10 dataset operations
"""
import os

2
tests/ut/python/dataset/test_datasets_svhn.py
View File

@ -13,7 +13,7 @@
# limitations under the License.
# ==============================================================================
"""
Test SVHN dataset operators
Test SVHN dataset operations
"""
import os

2
tests/ut/python/dataset/test_datasets_usps.py
View File

@ -13,7 +13,7 @@
# limitations under the License.
# ==============================================================================
"""
Test USPS dataset operators
Test USPS dataset operations
"""
import os
from typing import cast

10
tests/ut/python/dataset/test_fade.py
View File

@ -43,7 +43,7 @@ def test_fade_linear():
for item in dataset.create_dict_iterator(num_epochs=1, output_numpy=True):
out_put = item["audio"]
# The result of the reference operator
# The result of the reference operation
expected_output = np.array([[0.0000000000000000000, 6.781666797905927e-06, 1.356333359581185e-05,
2.034499993897043e-05, 5.425333438324742e-05, 6.781666888855398e-05,
6.103533087298274e-05, 7.120789086911827e-05, 8.138045086525380e-05,
@ -72,7 +72,7 @@ def test_fade_exponential():
for item in dataset.create_dict_iterator(num_epochs=1, output_numpy=True):
out_put = item["audio"]
# The result of the reference operator
# The result of the reference operation
expected_output = np.array([[0.0000, 0.2071, 0.4823, 0.6657, 0.5743, 0.0000],
[0.0000, 0.7247, 0.4823, 12.9820, 0.9190, 0.0000]], dtype=np.float32)
assert np.mean(out_put - expected_output) < 0.0001
@ -96,7 +96,7 @@ def test_fade_logarithmic():
for item in dataset.create_dict_iterator(num_epochs=1, output_numpy=True):
out_put = item["audio"]
# The result of the reference operator
# The result of the reference operation
expected_output = np.array([[0.0000e+00, 9.4048e-03, 4.4193e-02,
-2.0599e-02, -3.5647e-02, 1.5389e-09]],
dtype=np.float32)
@ -122,7 +122,7 @@ def test_fade_quarter_sine():
for item in dataset.create_dict_iterator(num_epochs=1, output_numpy=True):
out_put = item["audio"]
# The result of the reference operator
# The result of the reference operation
expected_output = np.array([[0.0000, 0.5878, 1.4266, 1.9021, 1.4695, 0.0000],
[0.0000, 2.0572, 1.4266, 37.091, 2.3511, 0.0000],
[0.0000, 0.5878, 1.4266, 1.9021, 1.4695, 0.0000]], dtype=np.float64)
@ -149,7 +149,7 @@ def test_fade_half_sine():
for item in dataset.create_dict_iterator(num_epochs=1, output_numpy=True):
out_put = item["audio"]
# The result of the reference operator
# The result of the reference operation
expected_output = np.array([[0.0000, 0.0068, -0.0119, -0.0206, -0.0052, 0.0000],
[0.0000, 0.0303, 0.0500, 0.0131, -0.0098, -0.0000]], dtype=np.float32)
assert np.mean(out_put - expected_output) < 0.0001

2
tests/ut/python/dataset/test_onehot_op.py
View File

@ -63,7 +63,7 @@ def test_one_hot():
def test_one_hot_post_aug():
"""
Feature: OneHot Op
Description: Test C++ op with One Hot Encoding after Multiple Data Augmentation Operators
Description: Test C++ op with One Hot Encoding after Multiple Data Augmentation Operations
Expectation: Dataset pipeline runs successfully and results are verified
"""
logger.info("test_one_hot_post_aug")

2
tests/ut/python/dataset_deprecated/test_onehot_op.py
View File

@ -61,7 +61,7 @@ def test_one_hot():
def test_one_hot_post_aug():
"""
Feature: OneHot Op
Description: Test C++ op with One Hot Encoding after Multiple Data Augmentation Operators
Description: Test C++ op with One Hot Encoding after Multiple Data Augmentation Operations
Expectation: Dataset pipeline runs successfully and results are verified
"""
logger.info("test_one_hot_post_aug")