yystopf
/
mindspore
forked from mindspore-Ecosystem/mindspore
0
0
Fork 0
Code Issues Pull Requests Projects Releases Wiki Activity
45,876 Commits 23 Branches 0 Tags 3.9 GiB
Commit Graph

823 Commits

Author SHA1 Message Date
hesham 7120c66998 Add SubsetSampler 2021-01-27 15:00:18 -05:00
mindspore-ci-bot 4cd6588af0 !11393 Migrating cache transform pass from execution tree to IR tree 
From: @lixiachen
Reviewed-by: 
Signed-off-by:
 2021-01-27 22:10:13 +08:00
xulei2020 de630f1cb1 Add parameter verification for getROI function 
fix c++ ut compilation failed of Asan bug
 2021-01-27 09:06:59 +08:00
Lixia Chen 922e1f4f36 Migrate CacheTransformPass 2021-01-26 19:24:10 -05:00
Cathy Wong a3b10213b7 dataset: Resolve protected parent_access from DeepCopyPass 2021-01-22 15:56:32 -05:00
xiefangqi 3f76abcdee minddata err msg add file path 2021-01-22 16:38:46 +08:00
mindspore-ci-bot 2beb916bf5 !11427 TextOp decoupling 
From: @alexyuyue
Reviewed-by: 
Signed-off-by:
 2021-01-22 09:09:41 +08:00
mindspore-ci-bot fcc4a4eaea !11397 add GetDatasetSize to concat_node 
From: @mhmotallebi
Reviewed-by: 
Signed-off-by:
 2021-01-22 08:08:41 +08:00
mohammad b5e91139e2 add GetDatasetSize to concat node 2021-01-21 16:25:48 -05:00
xulei2020 de6356ebfa add warp affine 2021-01-21 14:20:14 +08:00
alex-yuyue a8675f8227 TextOp decoupling 
Signed-off-by: alex-yuyue <yue.yu1@huawei.com>
 2021-01-20 21:52:45 -05:00
mindspore-ci-bot c155dfee7a !11329 [MD] Rename SamplerObj::Build() to SamplerBuild() 
From: @lixiachen
Reviewed-by: 
Signed-off-by:
 2021-01-19 06:33:49 +08:00
Lixia Chen 273865a72b Rename SamplerObj::Build() to SamplerBuild() 2021-01-18 15:42:34 -05:00
Lixia Chen 3c5917a03d Fix IR node Drop() 2021-01-18 13:06:07 -05:00
mindspore-ci-bot cb58b5baa5 !11211 Provide functions for modifying IR tree (Insert/Drop/Append nodes) 
From: @lixiachen
Reviewed-by: 
Signed-off-by:
 2021-01-16 01:20:01 +08:00
Nat Sutyanyong e28fb6ce4d Tested with new test cases and all dataset UTs passed 2021-01-15 09:59:21 -05:00
mindspore-ci-bot dfa6daaa57 !11249 【MD】fix bug for extractchannel when type is UINT16 
From: @xulei2020
Reviewed-by: @heleiwang,@liucunwei
Signed-off-by: @liucunwei
 2021-01-15 14:33:16 +08:00
mindspore-ci-bot 581789c252 !11246 add num_epochs to CPP API Dataset::CreateIterator() 
From: @ziruiwu
Reviewed-by: 
Signed-off-by:
 2021-01-15 06:02:00 +08:00
xulei2020 5dd651f1a5 fix bug for extractchannel when type is UINT16 2021-01-14 09:26:18 +08:00
Cathy Wong 4b1ccf9dd5 dataset: Use int32_t for text's vocab_size 2021-01-13 18:12:50 -05:00
Zirui Wu fa553956ef add num_epochs to CPP API Dataset::CreateIterator(columns,num_epochs) 2021-01-13 12:13:40 -05:00
mohammad f48ab2b5c9 make MD public headers standalone 2021-01-11 06:35:53 -05:00
Zirui Wu 0e68575e77 port over getter pass 
port over tensor_op_fusion pass

add fusion support for prebuildOpeartion (TensorOpeartion)
 2021-01-07 10:52:49 -05:00
mindspore-ci-bot 5fd77356c9 !9687 Adding AddChild functionality to SamplerObj 
From: @mahdirahmanihanzaki
Reviewed-by: 
Signed-off-by:
 2021-01-06 01:46:08 +08:00
mindspore-ci-bot a5869f2984 !10142 dataset CPP UT: Updates for ExecTree to IRTree Support 
From: @cathwong
Reviewed-by: 
Signed-off-by:
 2021-01-05 06:53:14 +08:00
Nat Sutyanyong 7c5b2de1ec dataset CPP UT: Updates for ExecTree to IRTree Support 2021-01-04 16:00:30 -05:00
Zirui Wu 0a31f563a7 minor fixes 2020-12-23 10:06:01 -05:00
mindspore-ci-bot 20df37e428 !10287 Add ut for dataset to_number 
From: @luoyang42
Reviewed-by: @heleiwang
Signed-off-by:
 2020-12-23 21:14:42 +08:00
Zirui Wu 0e2f7a9e9e add validate param to base class DatasetNode 2020-12-21 11:14:37 -05:00
YangLuo da6de519ae Add ut for dataset to_number 2020-12-21 20:09:03 +08:00
Xiao Tianci 633f97bb48 fix get dataset size in distributed scenario 2020-12-15 20:41:45 +08:00
xiefangqi 3c6cc3c58c Add NormalizePadOp 2020-12-15 11:45:41 +08:00
Mahdi 0f2b5d8cac Changed SamplerObj validate params to return status and added AddChild to it 2020-12-10 15:39:59 -05:00
Zirui Wu 91a6b2b0ca Add checks to RandomData and LookUp 2020-12-09 17:09:34 -05:00
mindspore-ci-bot 317a97e6b9 !9336 auto num_parallel_workers setup 
From: @ziruiwu
Reviewed-by: 
Signed-off-by:
 2020-12-09 08:32:55 +08:00
mindspore-ci-bot a2c80435ce !9685 Fix a core dump in TreeConsumer::Terminate() plus minor cache fixes 
From: @lixiachen
Reviewed-by: @mikef,@nsyca
Signed-off-by: @nsyca
 2020-12-09 08:21:22 +08:00
Zirui Wu d6df1b0832 Implemented AutoNumWorker Pass which sets num_workers of selected parallel ops automatically if enabled 2020-12-08 17:49:05 -05:00
Lixia Chen 32b82c2737 Fix a core dump in TreeConsumer::Terminate() 
and other minor cache changes
 2020-12-08 15:38:40 -05:00
Xiao Tianci 0659473535 add TruncateSequencePair, ToNumber C++ API and enable three test cases 2020-12-08 11:12:16 +08:00
mindspore-ci-bot 49fd5308a4 !9564 Tidy up code in dataset compilation phase 
From: @nsyca
Reviewed-by: 
Signed-off-by:
 2020-12-08 05:11:29 +08:00
mindspore-ci-bot 6b5626634c !9530 add BasicTokenizer and BertTokenizer C++ API 
From: @tiancixiao
Reviewed-by: @mikef,@nsyca
Signed-off-by: @nsyca
 2020-12-08 04:09:40 +08:00
Nat Sutyanyong 41ea1621c5 Tidy up TreeAdapter::Compile 2020-12-07 14:57:22 -05:00
shenwei41 206ae1a7c3 Add Three text C++ API 2020-12-05 10:32:15 +08:00
Nat Sutyanyong 4cb78f2e03 Remove Repeat(1),Take(-1), and Skip(0) in NodeRemovalPass 2020-12-04 17:37:07 -05:00
Jesse Lee 0c8dfc68ec Fix hang on RequestFreeTag 2020-12-04 14:54:30 -05:00
Xiao Tianci 290cc72887 add BasicTokenizer and BertTokenizer C++ API 2020-12-04 21:51:49 +08:00
Nat Sutyanyong d69a29a44e Migrate 3 pre passes to IR optimizer, namely, cache_error_pass, epoch_injection, and removal_pass 2020-12-03 14:40:03 -05:00
mindspore-ci-bot 73c91e05b1 !9398 Change Build to return status 
From: @alex-yuyue
Reviewed-by: @mikef
Signed-off-by:
 2020-12-03 23:43:08 +08:00
alex-yuyue ccff3a5316 Change build to return status 
Signed-off-by: alex-yuyue <yue.yu1@huawei.com>
 2020-12-02 22:46:49 -05:00
YangLuo a2ec7ad5bd Fix concat issue & fix schema doc 2020-12-02 21:06:22 +08:00
mindspore-ci-bot df44e1339e !9249 Optimizing GetDatasetSize 
From: @mahdirahmanihanzaki
Reviewed-by: 
Signed-off-by:
 2020-12-02 08:26:34 +08:00
mindspore-ci-bot 9ed31ea2cb !9339 Disable getter optimization pass 
From: @ezphlow
Reviewed-by: @robingrosman,@nsyca
Signed-off-by: @nsyca
 2020-12-02 08:25:41 +08:00
Eric db2a8b5e1d Disable getter pass 2020-12-01 17:40:34 -05:00
Mahdi f80553bc6e Optimization for GetDatasetSize 2020-12-01 17:40:25 -05:00
mindspore-ci-bot da8d7d661a !9283 Modify the dataset ut log message 
From: @liucunwei
Reviewed-by: @pandoublefeng,@heleiwang,@jonyguo
Signed-off-by: @heleiwang,@jonyguo
 2020-12-01 16:28:32 +08:00
liucunwei 8ef4b2796e modify the dataset ut log message 2020-12-01 14:42:44 +08:00
shenwei41 3bea84d0f7 add four new text API 2020-12-01 10:21:47 +08:00
mindspore-ci-bot 7748ae823a !9141 Fix DatasetCache C++ API cache sharing behavior 
From: @lixiachen
Reviewed-by: @mikef,@nsyca
Signed-off-by: @nsyca
 2020-12-01 02:28:02 +08:00
mindspore-ci-bot a37e9710b5 !9038 port name functions from TensorOp to TensorOperation 
From: @mhmotallebi
Reviewed-by: 
Signed-off-by:
 2020-12-01 02:13:48 +08:00
mindspore-ci-bot a6845395e6 !9181 Add some ut pipeline test case for C++ dataset 
From: @liucunwei
Reviewed-by: @pandoublefeng,@jonyguo
Signed-off-by: @jonyguo
 2020-11-28 21:34:03 +08:00
liucunwei e612e23a05 Add some ut pipeline test case for C++ dataset 2020-11-28 18:14:58 +08:00
liucunwei 7f13f2ad0e add some pipeline ut test cases for c++ dataset 2020-11-28 16:59:38 +08:00
mohammad 9cfeb08c4d port name() to TensorOperation and its derived classes 2020-11-27 19:00:56 -05:00
Lixia Chen e8247eadef Same runtime CacheClient if c++ cache is shared 2020-11-27 18:52:15 -05:00
Zirui Wu 36bfd0d8ad address left over cmts from 8802 2020-11-27 16:54:28 -05:00
liucunwei 3e57236fe3 add some new ut test for minddataset 2020-11-27 20:39:13 +08:00
shenwei41 4e56618d18 Add text C++ API 2020-11-27 11:41:35 +08:00
Nat Sutyanyong 5e1bb0b697 boilerplate code for future IR optimizer 
add 2 test cases to IRNode Deepcopy()

address review cmts

fix ut

samplerObj copy

ci

fix ci

fix ci round III

address further review cmts

add a missing macro

fix merge conflict

fix complie err

fix lite compile err

fix compile err

fix lite compile round III

address an issue

fix minor comments
 2020-11-26 09:42:01 -05:00
Eric Zhang 809e1d5086 added python api based on cpp api 
1st draft of python iterator

Added Cifar10 and Cifar100 pybind port

Change pybind to use IR for Skip and Manifest

Signed-off-by: alex-yuyue <yue.yu1@huawei.com>

DatasetNode as a base for all IR nodes

namespace change

Fix the namespace issue and make ut tests work

Signed-off-by: alex-yuyue <yue.yu1@huawei.com>

Add VOCDataset

!63 Added RandomDataset
* Added RandomDataset

add imagefolder ir

Pybind switch: CelebA and UT

!61 CLUE example with class definition
* Merge branch 'python-api' of gitee.com:ezphlow/mindspore into clue_class_pybind
* Passing testcases
* Added CLUE, not working

add ManifestDataset IR

Signed-off-by: alex-yuyue <yue.yu1@huawei.com>

Update Coco & VOC & TFReader, Update clang-format, Reorder
datasets_binding

!69 Add Generator and move c_dataset.Iterator to dataset.Iterator
* Add GeneratorDataset to c_dataset
* Add GeneratorDataset to c_dataset

!67 Moving c_datasets and adding sampler wrapper
* Need to add create() method in datasets.py
* migration from c_dataset to dataset part 1

!71 Fix indent error
* Fix indentation error

!72 Fix c_api tests cases
* Fix c_api tests cases

!73 Added CSV Dataset
* Added CSVDataset

pybind switch: Take and CelebA fixes

!75 move c_dataset functionality to datasets
* Fixed existing testcases
* Added working clue and imagefolder
* Added sampler conversion from pybind
* Added sampler creation

!77 Add Python API tree
* Python API tree

add minddataset

TextFileDataset pybind

Rename to skip test_concat.py and test_minddataset_exception.py

!80 Add batch IR to python-api branch, most test cases work
* staging III
* staging, add pybind

Enable more c_api take and CelebA tests; delete util_c_api

!84 Schema changes in datasets.py
* Schema changes

!85 Remove input_indexes from sub-classes
* remove input_index from each subclass

!83 Remove C datasets
* Removed c_dataset package
* Remove c_datasets

!82  pybind switch: shuffle
* pybind switch: shuffle

!86 Add build_vocab
* Add build_vocab

Rebase with upstream/master
_shuffle conflict
BatchNode error

!88 Fix rebase problem
* fix rebase problem

Enable more unit tests; code typo/nit fixes

!91 Fix python vocag hang
* Fix python vocab hang

!89 Added BucketBatchByLength Pybind switch
* Added BucketBatchByLength

Update and enable more tet_c_api_*.py tests

!95 Add BuildSentencePeiceVocab
* - Add BuildSentencePeiceVocab

!96 Fix more tests
* - Fix some tests

- Enable more test_c_api_*
- Add syncwait

!99 pybind switch for device op
* pybind switch for device op

!93 Add getters to python API
* Add getters to python API

!101 Validate tree, error if graph
* - Add sync wait

!103 TFrecord/Random Datasets schema problem
* - TfRecord/Random schem aproblem

!102 Added filter pybind switch
* Added Filter pybind switch

!104 Fix num_samples
* - TfRecord/Random schem aproblem

!105 Fix to_device hang
* Fix to_device hang

!94 Adds Cache support for CLUE dataset
* Added cache for all dataset ops
* format change
* Added CLUE cache support
* Added Cache conversion

Add save pybind

fix compile err

init modify concat_node

!107 Fix some tests cases
* Fix tests cases

Enable and fix more tests

!109 pybind switch for get dataset size
* pybind_get_dataset_size

some check-code fixes for pylint, cpplint and clang-format

!113 Add callback
* revert
* dataset_sz 1 line
* fix typo
* get callback to work

!114 Make Android compile clean
* Make Android Compile Clean

Fix build issues due to rebase

!115 Fix more tests
* Fix tests cases
* !93 Add getters to python API

fix test_profiling.py

!116 fix get dataset size
* fix get dataset size

!117 GetColumnNames pybind switch
* Added GetColumnNames pybind switch

code-check fixes: clangformat, cppcheck, cpplint, pylint

Delete duplicate test_c_api_*.py files; more lint fixes

!121 Fix cpp tests
* Remove extra call to getNext in cpp tests

!122 Fix Schema with Generator
* Fix Schema with Generator

fix some cases of csv & mindrecord

!124 fix tfrecord get_dataset_size and add some UTs
* fix tfrecord get dataset size and add some ut for get_dataset_size

!125 getter separation
* Getter separation

!126 Fix sampler.GetNumSamples
* Fix sampler.GetNumSampler

!127 Assign runtime getter to each get function
* Assign runtime getter to each get function

Fix compile issues

!128 Match master code
* Match master code

!129 Cleanup DeviceOp/save code
* Cleanup ToDevice/Save code

!130 Add cache fix
* Added cache fix for map and image folder

!132 Fix testing team issues
* Pass queue_name from python to C++
* Add Schema.from_json

!131 Fix Cache op issues and delete de_pipeline
* Roll back C++ change
* Removed de_pipeline and passing all cache tests.
* fixed cache tests

!134 Cleanup datasets.py part1
* Cleanup dataset.py part1

!133 Updated validation for SentencePieceVocab.from_dataset
* Added type_check for column names in SentencePieceVocab.from_dataset

Rebase on master 181120 10:20

fix profiling

temporary solution of catching stauts from Node.Build()

!141 ToDevice Termination
* ToDevice termination

pylint fixes

!137 Fix test team issues and add some corresponding tests
* Fix test team issues and add some corresponding tests

!138 TreeGetter changes to use OptPass
* Getter changes to use OptPass (Zirui)

Rebase fix

!143 Fix cpplint issue
* Fix cpplint issue

pylint fixes in updated testcases

!145 Reset exceptions testcase
* reset exception test to master

!146 Fix Check_Pylint Error
* Fix Check_Pylint Error

!147 fix android
* fix android

!148 ToDevice changes
* Add ToDevice to the iterator List for cleanup at exit

!149 Pylint issue
* Add ToDevice to the iterator List for cleanup at exit

!150 Pylint 2
* Add ToDevice to the iterator List for cleanup at exit

!152 ExecutionTree error
* ET destructor error

!153 in getter_pass, only remove callback, without deleting map op
* getter pass no longer removes map

!156 early __del__ of iterator/to_device
* early __del__ of iterator

!155 Address review comments Eric 1
* Added one liner fix to validators.py
* roll back signature fix
* lint fix
* Eric Address comments 2
* C++ lint fix
* Address comments Eric 1

!158 Review rework for dataset bindings - part 1
* Reorder nodes repeat and rename
* Review rework for dataset bindings - part 1

!154 Fixing minor problems in the comments (datasets.py, python_tree_consumer.cc, iterators_bindings.cc, and iterators.py)
* Fixing minor problems in the comments (datasets.py, python_tree_consum…

!157 add replace none
* Add replace_none to datasets.py, address comments in tests

Trying to resolve copy

Override the deepcopy method of deviceop

Create_ir_tree method

Create_ir_tree method 2

Create_ir_tree method 2

del to_device if already exists

del to_device if already exists

cache getters shapes and types

Added yolov3 relaxation, to be rolled back

Get shapes and types together

bypass yolo

NumWorkers for MapOp

revert Yolo

revert Thor

Print more info

Debug code: Update LOG INFO to LOG ERROR

do not remove epochctrl for getter pass

Remove repeat(1)

pritn batch size

add log to tree_consumer and device_queue op

Revert PR 8744

Signed-off-by: alex-yuyue <yue.yu1@huawei.com>

__del__ toDEvice

__del__ toDevice2

!165 add ifndef ENABLE_ANDROID to device queue print
* Add ifndef ENABLE_ANDROID to device queue print

revert some changes

!166 getter: get_data_info
* getter: get_data_info

!168 add back tree print
* revert info to warnning in one log
* add back the missed print tree log

Release GIL in GetDataInfo
 2020-11-25 15:46:59 -05:00
shenwei41 4132228c10 Add c++ api of randomresizecropwithbbox 2020-11-21 17:06:56 +08:00
mindspore-ci-bot 812095de8f !8753 Adds execute testcase and fix get_dataset_size 
From: @ezphlow
Reviewed-by: 
Signed-off-by:
 2020-11-20 02:06:55 +08:00
Eric 45292abdfd Added execute test case 
Added execute tests for server
 2020-11-18 23:44:46 -05:00
mindspore-ci-bot c8047a35ab !8632 [MD] Add Multiply op for Lite Dataset 
From: @jiangzhiwen8
Reviewed-by: @liucunwei,@heleiwang
Signed-off-by: @liucunwei
 2020-11-19 09:58:43 +08:00
mindspore-ci-bot 215fce192e !8737 [MD] Fix the Pad function return false if bottom value is 0 
From: @jiangzhiwen8
Reviewed-by: @xulei2020,@liucunwei,@heleiwang
Signed-off-by: @liucunwei
 2020-11-19 09:07:25 +08:00
mindspore-ci-bot f352f9a54a !8511 [MD] Decouple more Transform Ops 
From: @luoyang42
Reviewed-by: 
Signed-off-by:
 2020-11-18 23:19:36 +08:00
jiangzhiwen bc91f7b3ac fix pad zero 2020-11-18 16:14:37 +08:00
luoyang d43155b5b0 [MD] Decouple more Transform Ops 2020-11-18 14:50:59 +08:00
mindspore-ci-bot e3b1814401 !8713 Fix for GetDatasetSize issue in TextFile 
From: @mahdirahmanihanzaki
Reviewed-by: @nsyca,@robingrosman
Signed-off-by: @nsyca
 2020-11-18 05:39:51 +08:00
Mahdi 449e1526dc Fixed GetDatasetSize for TextFile 2020-11-17 15:22:58 -05:00
Zirui Wu ff5999fc2f add removal pass for getters 
fix CI round I

fix ci round II

address review cmts

fix ci round II
 2020-11-17 10:49:23 -05:00
Mahdi ecde0f03a2 Updated GetNumSamples logic 2020-11-16 16:03:14 -05:00
jiangzhiwen b30c1045e1 add lite_mat multiply 2020-11-16 15:05:31 +08:00
mindspore-ci-bot edcb0cd86b !8370 [MD] Use neon instruction set to accelerate the Substract and Divide ops 
From: @jiangzhiwen8
Reviewed-by: @xulei2020
Signed-off-by:
 2020-11-13 16:14:00 +08:00
mindspore-ci-bot c54a4a4494 !8540 Added GetColumnNames to C++ 
From: @mahdirahmanihanzaki
Reviewed-by: @nsyca,@jonyguo
Signed-off-by: @nsyca
 2020-11-13 09:33:37 +08:00
Mahdi 5a7515e48f Added GetColumnNames to C++ 2020-11-12 16:36:59 -05:00
Zirui Wu 3555d19b07 fix get num_class in imagefolder 
address cmt

review cmts
 2020-11-12 14:26:54 -05:00
jiangzhiwen 03bbe5f2a8 neon acceleration for subtract 2020-11-12 16:30:40 +08:00
Eric a21eb2d527 Added random node fix 2020-11-11 17:24:48 -05:00
mindspore-ci-bot 70f5775711 !8387 Add support for GetClassIndexing in C++ API 
From: @alex-yuyue
Reviewed-by: 
Signed-off-by:
 2020-11-11 03:02:22 +08:00
alex-yuyue d7c3a9ba00 Add GetClassIndexing 
Signed-off-by: alex-yuyue <yue.yu1@huawei.com>
 2020-11-10 11:02:02 -05:00
xulei2020 f8d1b20873 init litemat from memory 2020-11-10 10:12:39 +08:00
mindspore-ci-bot 908a494add !8299 [MD] Skip returning IR when meet nullptr in ConcatDataset 
From: @luoyang42
Reviewed-by: @nsyca,@pandoublefeng,@jonyguo
Signed-off-by: @jonyguo
 2020-11-09 11:49:48 +08:00
mindspore-ci-bot 027916cb5f !8315 Cache server stability 
From: @jkl_lee
Reviewed-by: @mikef,@nsyca
Signed-off-by: @nsyca
 2020-11-08 20:03:23 +08:00
Lixia Chen f24a788eed Rebase up to 5792d50ca7 2020-11-06 18:39:05 -05:00
Zirui Wu 7538f82da9 add callback to map's c api 
fix test err

fix ci round 1

fix ci round 2

add col_order to batch_node

minor fix
 2020-11-06 16:36:47 -05:00
mindspore-ci-bot a9a9c45662 !8232 C++ api add Filter 
From: @tiancixiao
Reviewed-by: 
Signed-off-by:
 2020-11-07 00:32:15 +08:00
mindspore-ci-bot 97e1f6f438 !8248 Add macro and fix album testcase 
Merge pull request !8248 from EricZ/helper_dep_removal
 2020-11-06 20:33:20 +08:00
Xiao Tianci 4a5e287b92 add filter c api and ut 2020-11-06 15:49:23 +08:00
luoyang dcb758bc77 [MD] Skip returning IR when meet nullptr in ConcatDataset 2020-11-06 15:08:11 +08:00
Eric 394dbf50e1 Added testing macro and fix album_op test 2020-11-05 13:22:51 -05:00
mindspore-ci-bot 523c3f07e3 !8194 Optimizer on IR #2 - Call ValidateParams in PrePass 
Merge pull request !8194 from nsyca/opt3
 2020-11-05 22:52:37 +08:00
mindspore-ci-bot 099765bbb7 !8135 [MD] Transform api decouple 
Merge pull request !8135 from luoyang/c-api-pyfunc
 2020-11-05 22:41:44 +08:00
Yi Huaijie d7faa77b5e support int64 shape 2020-11-05 21:15:49 +08:00
luoyang 9a57cbacd9 [MD] Transform api decouple 2020-11-05 19:56:15 +08:00
mindspore-ci-bot 8709005a57 !8200 Add two transforms API 
Merge pull request !8200 from shenwei41/new_master
 2020-11-05 11:19:00 +08:00
Nat Sutyanyong a2a2223373 Add optimizer to IR tree #2 2020-11-04 21:44:17 -05:00
Eric 85f0a9bc8f Added dependency removal for data_helper 2020-11-04 12:15:32 -05:00
shenwei41 9ffdf0b6d4 Add transform API 2020-11-04 11:54:33 +08:00
Zirui Wu 5d1ada682c remove uncessary headers 
staging II

clean up

further clean up

fix merge conflict with 8139

fix complie err
 2020-11-03 11:59:55 -05:00
hesham 5169fb4c42 Remove api namespace 2020-11-02 20:09:37 -05:00
hesham 0ba0595caf DatasetNode as a base for all IR nodes 2020-11-01 16:57:10 -05:00
Mahdi 5005b434e6 Fixed repeat issue with GetDatasetSize 2020-10-30 19:36:56 -04:00
mindspore-ci-bot cacebd1211 !7933 Cache performance updates 
Merge pull request !7933 from Jamie/CacheOp_dev
 2020-10-31 04:25:47 +08:00
mindspore-ci-bot 7ef7bef1d8 !7796 [MD] Enable C++ cache api for non-mappable datasets 
Merge pull request !7796 from lixiachen/cache_c_api_nonmappable
 2020-10-30 22:11:13 +08:00
Lixia Chen 572c5e5f29 Rebase up to 542a52fbf8 2020-10-29 20:34:09 -04:00
Cathy Wong 18ecd65f93 C++ API: Delete ValidateParams in dataset APIs; update error input UTs 2020-10-29 16:29:16 -04:00
mindspore-ci-bot 8f44074796 !7866 [MD] C++ API support build_sentence_piece_vocab_node & sentence_piece_tokenizer 
Merge pull request !7866 from luoyang/c-api-pyfunc
 2020-10-29 14:26:13 +08:00
luoyang 52d278e858 [MD] C++ API support build_sentence_piece_vocab_node & sentence_piece_tokenizer 2020-10-29 11:05:28 +08:00
mindspore-ci-bot 2ba20dc8e7 !7848 Add five transforms API 
Merge pull request !7848 from shenwei41/new_master
 2020-10-29 10:18:57 +08:00
Zirui Wu 814044963b add validation check in tfrecord_node 
update ut
 2020-10-28 19:26:51 -04:00
Zirui Wu ea97197311 fix tfrecord_op core dump 
add test case

fix ci

fix ci round 2

address review cmts
 2020-10-28 17:38:43 -04:00
mindspore-ci-bot c39c3ccfbe !7734 Redesigned GetNumClasses 
Merge pull request !7734 from Alex Yuyue/IR_dataset_input
 2020-10-29 01:44:36 +08:00
shenwei41 779f0f387b Add transforms API 2020-10-28 22:06:58 +08:00
Xiao Tianci 823e52f1dc add params checking of offset in distributed_sampler 
add related comment to explain
add related UT
 2020-10-28 16:46:12 +08:00
Lixia Chen 02ddddfe7e Enable caching over non-mappable dataset c++ api 
and caching over Album dataset
 2020-10-27 23:02:13 -04:00
alex-yuyue 913a49f3af Add support for C++ API: GetNumClasses 
Signed-off-by: alex-yuyue <yue.yu1@huawei.com>
 2020-10-27 16:26:51 -04:00
mindspore-ci-bot 3376e0d034 !7704 [MD] Add testcases for cache c api 
Merge pull request !7704 from lixiachen/cache_c_api
 2020-10-26 14:31:09 +08:00
mindspore-ci-bot d81bd7b17b !7710 Add c++ API for GetRepeatCount and GetBatchSize 
Merge pull request !7710 from h.farahat/getBatchRepeat
 2020-10-26 09:25:21 +08:00
mindspore-ci-bot 0124eef185 !7729 [MD] Fix log of vision and dataset in C++ api 
Merge pull request !7729 from luoyang/pylint
 2020-10-26 09:23:22 +08:00
mindspore-ci-bot 7982f05038 !7730 [MD] Fix bugs in WeightedRandomSampler & SubsetRandomSampler 
Merge pull request !7730 from luoyang/c-api-pyfunc
 2020-10-26 09:22:14 +08:00
luoyang f0e976dbda [MD] Fix bugs in WeightedRandomSampler & SubsetRandomSampler 2020-10-25 17:58:51 +08:00
hesham f3880f8355 Add GetBatchSizer and GetRepeatCount to c++ API 2020-10-24 22:45:19 -04:00
YangLuo f3fa5a441d [MD] Fix log of vision and dataset in C++ api 2020-10-24 19:24:30 +08:00
mindspore-ci-bot 3d194137da !7702 Redesigned GetOutputType and GetOutputShape 
Merge pull request !7702 from Alex Yuyue/IR_dataset_input
 2020-10-24 14:42:29 +08:00
mindspore-ci-bot 48edd9708c !7690 [MD] fix num_sample in randomsampler and sequential sampler 
Merge pull request !7690 from liyong126/fix_md_num_samples
 2020-10-24 11:13:19 +08:00
alex-yuyue 2906659673 Add support for GetOutputTypes and GetOutputShapes 
Signed-off-by: alex-yuyue <yue.yu1@huawei.com>
 2020-10-23 22:21:52 -04:00
mindspore-ci-bot fe2852df82 !7703 C++ API: Add a parameter validation and fix incorrect format of error message for TFrecord 
Merge pull request !7703 from TinaMengtingZhang/fix_tfrecord_issue_I1YKU6_I1YKXL
 2020-10-24 07:33:26 +08:00
Lixia Chen bc396febae Add testcases for cache c api 2020-10-23 17:10:18 -04:00
tinazhang e7346dd3a9 add a validation code to fix issue I1YKU6 and corresponded ut, fix format issue I1YKXL 2020-10-23 17:07:51 -04:00
Mahdi 0e03f5b0dd Added GetDatasetSize 2020-10-23 16:29:11 -04:00
Zirui Wu a85a256749 fix complie err in c_api_dataset_test_save 2020-10-23 10:09:35 -04:00
mindspore-ci-bot 3355bb5f2f !7657 Leaf node IR breakdown, port to engine/ir/datasetops/source/ 
Merge pull request !7657 from ZiruiWu/ir_breakdown_leaf_nodes
 2020-10-23 20:32:31 +08:00
liyong ee042b90f7 fix num_sample in sequentialSampler and randomSampler 2020-10-23 19:26:10 +08:00
mindspore-ci-bot 9d35fdc4e5 !7653 [MD] C++ api SaveOp 
Merge pull request !7653 from luoyang/c-api-pyfunc
 2020-10-23 14:14:11 +08:00
Zirui Wu cfc50e231a Added IR breakdown for remaining ops after PR7596 
fix android build failure

fix include in datasets.cc

attempting fix android build

fix complie err x86
 2020-10-22 23:07:41 -04:00
luoyang 8e60cfe800 [MD] C++ api save 2020-10-23 10:27:35 +08:00
mindspore-ci-bot 19b10d474f !7530 [MD] Add divide image proccessing 
Merge pull request !7530 from jiangzhiwen/lite/image_divide
 2020-10-23 09:22:36 +08:00
Zirui Wu d471552fc5 shuffle take repeat bucketbatch buildvocab repeat shuffle take project concat rename node IR added 
concat, bucketbatch project rename

fix ci round 1

fix ci round 2

fix up

fix ci
 2020-10-21 18:41:34 -04:00
mindspore-ci-bot de7ea8e68c !7540 Extract ImageFolderNode and BatchNode into its own files from datasets.cc/.h 
Merge pull request !7540 from ZiruiWu/refactor_ir_demo
 2020-10-21 19:23:20 +08:00
mindspore-ci-bot a8dc0f5c1b !7557 [MD] Add merge image proccessing 
Merge pull request !7557 from jiangzhiwen/lite/merge
 2020-10-21 17:28:55 +08:00
jiangzhiwen 8e785648da add merge image processing 2020-10-21 14:44:22 +08:00
Zirui Wu 52eec881ec Refactor ImageFolderNode and BatchNode into its separate files 
fix ci

fix ci

address review cmmts, fix ci

remove dup code in datasets.cc
 2020-10-20 23:07:51 -04:00
mindspore-ci-bot 643366ea70 !7538 Add empty string tensor support for Album 
Merge pull request !7538 from EricZ/album_empty_string
 2020-10-21 08:41:29 +08:00
Eric 7e21648e16 Added Album empty string tensor support 2020-10-20 16:37:37 -04:00
Zirui Wu aa8442e0a3 add skeleton code for tree_adapter 
stage II tree_adapter

add test case for tree_adapter

use dfs to build tree

add more test case

add test case

fix ci

fix ci round 2

fix ci round 3

fix ci round 4

fix complie error

fix ci round 6

fix ci round 7

fix ci round 8

fix ci round 9
 2020-10-20 12:44:28 -04:00
jiangzhiwen fadc97a18f add divide op 2020-10-20 20:53:01 +08:00
mindspore-ci-bot 48601f51b0 !7295 [MD] C++ api add MindDataset 
Merge pull request !7295 from luoyang/c-api-pyfunc
 2020-10-20 19:45:45 +08:00
mindspore-ci-bot 8ff82843ea !7486 [MD] Add subtract image proccessing 
Merge pull request !7486 from jiangzhiwen/lite/image_subtract
 2020-10-20 19:39:11 +08:00
mindspore-ci-bot 947f6d96d1 !7414 C++ api add config_manager 
Merge pull request !7414 from xiaotianci/c_api_config
 2020-10-20 19:37:18 +08:00
mindspore-ci-bot 7707bb8540 !7439 [MD] Add YUV to BGR , Split, extractChannel ops 
Merge pull request !7439 from xulei/lite_test0905
 2020-10-20 19:36:19 +08:00
Xiao Tianci fb98fbcc9f add c api config 2020-10-20 16:32:57 +08:00
luoyang 2dc8e5f421 [MD] C++ api add MindDataset 2020-10-20 16:12:49 +08:00
xulei2020 442cee15ad add yuv420sp to bgr, split and extractchannel 2020-10-20 15:05:25 +08:00
jiangzhiwen 44f7978f7c add image subtract 2020-10-20 10:53:27 +08:00
Eric d3e89086e5 Removed check for normalize mean 2020-10-15 17:03:41 -04:00
mindspore-ci-bot 0357088a73 !7319 C++ API: vision ops - reorder and add gap UTs; correct parm desc style 
Merge pull request !7319 from cathwong/ckw_c_api_transforms_tests2
 2020-10-16 00:31:24 +08:00
Eric 359ad19884 Added float_array support 
Added fix, tests to come
 2020-10-15 00:18:30 -04:00
Cathy Wong 9f7c1a32e7 C++ API: vision ops - reorder and add gap UTs; correct parm desc style 2020-10-14 19:10:13 -04:00
mindspore-ci-bot 7e7d47db50 !7266 RandomResizedCrop for C API 
Merge pull request !7266 from MahdiRahmaniHanzaki/c-api
 2020-10-14 23:00:15 +08:00
shenwei41 87b108f4e0 C++ api add Rescale 2020-10-14 10:32:48 +08:00
Mahdi c8333accfc RandomResizedCrop C API 2020-10-13 17:27:29 -04:00
mindspore-ci-bot 4a077f6341 !7035 Add validation for input image size of affine transformation 
Merge pull request !7035 from jiangzhiwen/fix/check_affine_size
 2020-10-13 15:03:46 +08:00
jiangzhiwen 197f0cabe0 check affine input size 2020-10-12 17:06:31 +08:00
YangLuo 9e033bbe19 C++ api add TypeCaseOp & RandomCropDecodeResizeOp 2020-10-12 15:13:17 +08:00
Cathy Wong 2aa26d245c C++ API: Move vision ops to new vision files; create transforms namespace 2020-10-08 11:59:58 -04:00
Mahdi e92d4edceb Updated TensorRow conversion 2020-09-29 14:50:44 -04:00
mindspore-ci-bot 0d8015b043 !5300 [MD] Add nD Slice Support 
Merge pull request !5300 from nhussain/slice_op
 2020-09-25 20:38:19 +08:00
mindspore-ci-bot 70221f5261 !6307 Conversion to and from TensorRow in C++ 
Merge pull request !6307 from MahdiRahmaniHanzaki/c_func
 2020-09-25 04:14:05 +08:00
nhussain 77d507279c first working implementation 
add string implementation, can handle partial string slices

finish core implementation, added Index object

fix up the logic

add in new SliceOption object, cleaning up tests
 2020-09-24 15:39:28 -04:00
Lixia Chen 983827ec5c Rebase up to 88ded11f59 2020-09-24 14:09:08 -04:00
Mahdi 890598d112 Added TensorRow conversion 2020-09-24 11:12:22 -04:00
yanghaitao1 4ff4c17632 fix get_dataset_size in CelebADataset when usage is not all 2020-09-19 02:16:29 -04:00
luoyang 9064a5f731 add num_row check & fix api distributed sampler, lookup, default value 
of usage
 2020-09-17 20:31:26 +08:00
xulei2020 85962adffd add code 2020-09-16 17:01:30 +08:00
luoyang 581335453e Change return val of GetNextRow 2020-09-12 14:19:46 +08:00
mindspore-ci-bot a084196300 !6094 Adding input checks to C API 
Merge pull request !6094 from EricZ/add_input_check_capi
 2020-09-12 09:47:50 +08:00
eric cd5af9472e Added input check 2020-09-11 20:36:12 -04:00
xulei2020 72861a68f3 add code 2020-09-11 19:28:35 +08:00
jiangzhiwen d4ce668b9b add Affine ut 2020-09-10 21:50:56 +08:00
mindspore-ci-bot c882e4863c !5961 [Data]Add lite_cv lib for mindspore lite data preprocessing 
Merge pull request !5961 from xulei/lite_test0905
 2020-09-10 18:51:07 +08:00
xulei2020 b356211413 add init code 2020-09-10 15:35:08 +08:00
Zirui Wu 1bb93580dd Add usage flag to MNIST/CIFAR 
ci fix 1

fix ci 2

fix ci 3

fix ci 4

fix ci 5

fix ci 6

fix ci 7

change var name in CelebA from dataset_type to usage

address review cmts

fix cpp ut fail

change mode to usage in VOCDataset

fix cmts

fix ci test case fail
 2020-09-09 09:59:55 -04:00
Zirui Wu bde9f18f5a update lookup api to take in a type 
ci

add test case

address some of the review cmts

address review cmts
 2020-09-08 11:59:28 -04:00
mindspore-ci-bot d0c7b1ba24 !5731 Manifest Dataset check invalid filename string 
Merge pull request !5731 from jiangzhiwen/fix/invalid_filename_string
 2020-09-04 16:36:29 +08:00
mindspore-ci-bot 2f3684d4fa !4872 Add C++ API support for TFRecordDataset 
Merge pull request !4872 from TinaMengtingZhang/cpp-api-tfrecord-dataset
 2020-09-04 11:53:08 +08:00
jiangzhiwen 294f7b620a check invalid filename string 2020-09-03 22:09:26 +08:00
jiangzhiwen 1a1a8893a4 Align num_samples of CSV with other dataset 2020-09-03 17:30:39 +08:00
tinazhang e430b4056a add tfrecord dataset to cpp api 
fix to support schema=nullptr
 2020-09-02 08:55:47 -04:00
luoyang 8aba39a71b Modify VOCDataset doc & Fix BuildVocab bug 2020-09-02 16:54:50 +08:00
luoyang ce419404ce Add check for duplicate column for a-cpi 2020-09-02 10:41:52 +08:00
mindspore-ci-bot 0cb6d29f0c !4998 Shared memory pool clean up 
Merge pull request !4998 from JesseKLee/mem_pool
 2020-09-02 02:01:19 +08:00
mindspore-ci-bot dff02cc282 !5188 rename image files in imagefolder 
Merge pull request !5188 from tony_liu2/alt
 2020-09-01 11:16:30 +08:00
mindspore-ci-bot b99149dc37 !4901 Adding data_helper for minddata 
Merge pull request !4901 from EricZ/md_helper
 2020-09-01 11:14:11 +08:00
mindspore-ci-bot 3be136f6f0 !5354 Fix randomdata issue 
Merge pull request !5354 from xiefangqi/fix_randomdata_columnlist
 2020-09-01 11:03:54 +08:00
mahdi 84efa0b96b Added BucketBatchByLength for C API 2020-08-31 17:06:16 -04:00
Eric 65bf3ecc3e Adding data_helper class 2020-08-31 16:29:14 -04:00
Jesse Lee e89b4648f6 Memory pool clean up 2020-08-30 11:28:24 -04:00
xiefangqi e2085a9505 fix randomdata issue 2020-08-28 11:55:49 +08:00
tony_liu2 5e4c314130 rename dataset dir and image files 
change references to use path original

revert folder name

revert graphengine
 2020-08-27 16:49:23 -04:00
mindspore-ci-bot 716329df6c !5278 Change default value of sampler in c-api 
Merge pull request !5278 from luoyang/c-api
 2020-08-27 21:33:18 +08:00
YangLuo 419478b410 Change default value of sampler to seperate behaviour of default sampler 
& null sampler, add check for duplicate column name
 2020-08-27 21:24:37 +08:00
islam_amin 94a4a3f20b changing affine and posterize apis 2020-08-26 10:51:54 -04:00
Cathy Wong 46e9d79a50 C++ API: Lexicographical order support for CLUE, CSV & TextFile Datasets 2020-08-25 14:11:19 -04:00
luoyang b26f6b3d5e fix bug in c-api: rename, concat, take 2020-08-25 16:25:00 +08:00
mindspore-ci-bot 0bbce9367e !4772 Adding AlbumDataset for device training 
Merge pull request !4772 from EricZ/ms-album
 2020-08-24 08:27:14 +08:00
Eric c79db93c48 Initial commit for album 
Added linter fix for album dataset

Added testDataset

Adding signature

Added JsonDataset example API

Example dataset

Resolving format

More fixing

Refactor

Small fix

Added compiling album dataset

Running tests

Added linter fix #1

Passing UT

Added dataset API

Addressing clang

Clang part 2

Fixing pass

Fixed tree check

lint fix

Added lint fix part 2
 2020-08-22 11:07:10 -04:00
mindspore-ci-bot 9b503e4f38 !4955 Fixes for Dynamic Augmentation Ops 
Merge pull request !4955 from MahdiRahmaniHanzaki/dynamic-ops-fix
 2020-08-22 10:27:50 +08:00
Mahdi a5f9b8f92e Added fix for MixUpBatch and CutMixBatch and for RandomAffine 
updated c color op descriptions
 2020-08-21 17:40:08 -04:00
Zirui Wu 74c1e6da60 introducing pause and quit flags to DataBuffer 
fix review cmts

fix ci

fix Ci

fixci

address ci

ci

- add timeout
- add more test cases

fix CI

address review cmts
 2020-08-21 14:33:26 -04:00
mindspore-ci-bot 9a3baf4f6c !4314 [MD] remove defaults used in binding code, and remove bilinear resize op 
Merge pull request !4314 from nhussain/clean_up_bindings
 2020-08-20 22:16:56 +08:00
xiefangqi 7dc7990d75 add NumWorkers validate 2020-08-20 17:25:30 +08:00
mindspore-ci-bot 7b0a4e5929 !4796 Fix randomdata ut core dump problem 
Merge pull request !4796 from xiefangqi/fix_randomdata_ut
 2020-08-20 14:11:55 +08:00
xiefangqi c0d18d2517 fix randomdata ut problem 2020-08-20 11:13:02 +08:00
Jesse Lee 8a08d0c37b Phase 2 of CacheOp 2020-08-19 13:45:55 -04:00
Zirui Wu 78c1aa1d96 Implemented Callback for Dataset 
implment pause in MapOp, added more to callback

add ds_callback

- Initial drop of Python DSCallback

- Pybind DSCallback

- Pybind DSCallback

added callback to mapOp

- de_pipeline DSCallback

- de_pipeline DSCallback

add test case, segfault for now

fix seg fault

- de_pipeline DSCallback

remove 1 line

update callback test case, now works

use builder class for mapOp callback

- de_pipeline DSCallback

- de_pipeline DSCallback

- de_pipeline DSCallback

better test case

minor fix

add comments and minor clean ups

get rid of nullptr in MapOp, use other flag instead

fix a bug ParseMapOp only takes 1 callback

- Added WaitedDSCalabck

refactor callback param

fix text case incorrect number

- added testing

fix cpp test case

- added testing

- revert back lenet changes

- cleanup test_callbacks.py

- cleanup test_callbacks.py

fix CI stage I

fix CI stage II

fix CI and update epoch counter

- add validation
- add more testing  test_callbacks.py

use random data op to do tests

adjust when to call EpochBegin/End

- add repeat with callback

- addressing reviewers' comments

- docstring and CI fixes

- docstring and CI fixes

- docstring and CI fixes

- rebase with upstream/master

fix cpp test case

fix review comments

addr review cmts, add test case
 2020-08-19 12:31:04 -04:00
YangLuo b50ae27c72 !4597 [Dataset] C++ API Support for build_vocab 2020-08-19 22:47:47 +08:00
nhussain eb9a611041 remove old defaults 2020-08-19 10:11:19 -04:00
mindspore-ci-bot d541e261a0 !4696 C++ API Support for CSV Dataset 
Merge pull request !4696 from jiangzhiwen/jzw/c_api_csv
 2020-08-19 17:08:33 +08:00
mindspore-ci-bot 256dccc651 !4498 Gnn data processing supports distributed scenarios 
Merge pull request !4498 from heleiwang/gnn_distributed
 2020-08-19 16:09:05 +08:00
jiangzhiwen 39df25f803 c++ api for CSV Dataset 2020-08-19 16:02:11 +08:00
heleiwang 8ee4d8e92d Gnn data processing supports distributed scenarios 2020-08-19 11:09:01 +08:00
mindspore-ci-bot 6868b9b6fa !4422 Minddata new dataset api ==> RandomData 
Merge pull request !4422 from xiefangqi/xfq_c++api_randomdata
 2020-08-19 10:48:53 +08:00
xiefangqi b91e56375e add randomdataset and schema 2020-08-19 09:13:43 +08:00
nhussain 738ae2c78d change api to use std_vector 
try to display image
 2020-08-18 15:41:21 -04:00
islam_amin 979111d0ef Added Posterize Op 2020-08-18 13:55:47 -04:00
jiangzhiwen a88273bd3f c++ api support for Manifest Dataset 2020-08-18 09:48:55 +08:00
mindspore-ci-bot 113ff6ca24 !4367 CutMixBatch Augmentation Op 
Merge pull request !4367 from MahdiRahmaniHanzaki/cutmix
 2020-08-18 03:27:33 +08:00
mahdi 3ecc53fb4e Added CutMix 2020-08-17 13:01:47 -04:00
Cathy Wong 9c8af0d1cf C++ API: Provide validate param functions 2020-08-14 17:54:38 -04:00
mindspore-ci-bot 7d70fb4dc4 !3769 add RandomColorOp 
Merge pull request !3769 from Alexey_Shevlyakov/ColorOp
 2020-08-14 23:43:08 +08:00
mindspore-ci-bot ebdea65f70 !4402 [Dataset] Fix c_api bugs 
Merge pull request !4402 from luoyang/c-api-pyfunc
 2020-08-14 17:40:32 +08:00
Alexey Shevlyakov 8526d5414d RandomColor 2020-08-13 16:10:21 -04:00
mindspore-ci-bot 2953720169 !4322 Adding RGBA color conversion 
Merge pull request !4322 from EricZ/bitmap_support
 2020-08-14 01:58:33 +08:00
mindspore-ci-bot 7a10e07fc6 !4383 Add distributedSampler to concatOP 
Merge pull request !4383 from genglishuai/genglshOK
 2020-08-14 00:06:06 +08:00
eric 6a41b44724 Added wrapper around color change function 2020-08-13 11:39:56 -04:00
genglishuai 4d4df68d1e add DistributedSampler for Concat op 2020-08-13 22:52:02 +08:00
luoyang 373adaa553 !4402 [Dataset] Fix c_api bugs 2020-08-13 22:38:55 +08:00
tinazhang 0868720eb7 fix parameter type for repeat op in c++ api and added c++/python ut. 2020-08-12 17:23:42 -04:00
tony_liu2 deb1146474 reorganize headers and tests 
remove headers

fix iterator file

fix pr comments

fix new file
 2020-08-12 14:36:01 -04:00
mindspore-ci-bot 5adba834d0 !3819 C++ API Support for CLUE Dataset 
Merge pull request !3819 from jiangzhiwen/jzw/c_api_clue
 2020-08-12 14:21:19 +08:00
jiangzhiwen 5842a0212f c++ api support for CLUE Dataset 2020-08-12 09:55:43 +08:00
mindspore-ci-bot 9276a9d2fb !3771 C++ API Support for TextFile Dataset 
Merge pull request !3771 from cathwong/ckw_c_api_textfile_nosampler
 2020-08-12 04:32:05 +08:00
avakh 477528de7f random sharpness cpp op support 2020-08-11 15:30:40 -04:00
nhussain 7766efd58e add files 2020-08-11 11:54:14 -04:00
Cathy Wong 7f39b5cfd7 C++ API Support for TextFile Dataset and Unit Tests 2020-08-11 11:53:53 -04:00
mindspore-ci-bot f21ee64fd0 !4247 Add ChannelSwap operation 
Merge pull request !4247 from 章一智/channel_swap_op
 2020-08-11 09:59:55 +08:00
mindspore-ci-bot 2cab58fdbc !3850 Implementing RandomAffine C Op and a CPP UT for AutoContrast & Equalize 
Merge pull request !3850 from islam_amin/randomaffine_op
 2020-08-11 09:29:09 +08:00
mindspore-ci-bot 0bb39830ae !4250 fix cpp api iterator build function 
Merge pull request !4250 from anthonyaje/fix_cpp_api_iterator
 2020-08-11 07:21:27 +08:00
anthonyaje 9774505ecd Fixed dataset cpp api iterator build function 2020-08-10 18:35:59 -04:00
islam_amin e52d85ba9d Added RandomAffine Op 2020-08-10 17:52:50 -04:00
mindspore-ci-bot 612f533d15 !3917 Add c++ ops CreateTupleIterator & Code changes in VOC DatasetOp 
Merge pull request !3917 from luoyang/c-api
 2020-08-11 05:50:21 +08:00
ervinzhang 3a1d8b5f39 added channel swap operation in transforms.h 2020-08-10 17:32:50 -04:00
YangLuo e6a12946c8 Updata voc build() & add C++ Op CreateTupleIterator 2020-08-10 11:53:58 +08:00
Mahdi 1896950ae5 Added Mixup 2020-08-07 17:47:16 -04:00
tony_liu2 3a020fe877 split c_api_test 
split into datasets

review fix

fix merge issues

fixed minddata lite build issues
 2020-08-07 11:57:49 -04:00
xiefangqi 6276c0f22f new c++ api ==> CelebA 2020-08-07 09:23:20 +08:00
mindspore-ci-bot 7ec0b5857a !3977 [MD] Refactor Concatenate Op 
Merge pull request !3977 from nhussain/multi_dim_concat_2
 2020-08-06 02:53:06 +08:00
ervinzhang c65670b1a7 changed sampler parameter to support std::move 2020-08-05 13:56:52 -04:00
mindspore-ci-bot 6aa65da5d2 !3860 Minddata new dataset api ==> Coco 
Merge pull request !3860 from xiefangqi/md_new_api_cocodataset
 2020-08-06 00:34:28 +08:00
nhussain 61769b2dd9 fix concat 2020-08-05 11:28:56 -04:00
xiefangqi bb7bda1d6b new dataset api -- Coco 2020-08-05 10:54:37 +08:00
liubuyu d81862a916 decoupling core and context 2020-08-05 08:15:14 +08:00
tinazhang e57d849618 add concat op and extend zip support to cpp api 2020-07-31 12:57:50 -04:00
Eric 8c018da468 Fixing issues in distributed sampler 
Added Distributed sampler option

Fix style
 2020-07-31 11:12:26 -04:00
mindspore-ci-bot 634d797ca3 !3549 MapOp refactor for device augmentation (part 1) 
Merge pull request !3549 from anthonyaje/device_augment_mapop_refactor1
 2020-07-31 21:56:16 +08:00
dengyutao 388e43f7cb delete parameter block_reader for MindDataset 2020-07-31 11:30:05 +08:00
anthonyaje a7eaf2736a First drop of MapOp refactor to support device augment. 
Refactored MapOp to in preparation to support device augment

Minor fix
 2020-07-28 15:53:43 -04:00
YangLuo ae99a3a3e4 C++ API support for TakeDatasetOp and VOCDatasetOp 2020-07-28 23:09:57 +08:00
mindspore-ci-bot b6c7273bb6 !3551 [MD] Fix Broken C++ Only Build 
Merge pull request !3551 from nhussain/fix_cpp_only_build
 2020-07-28 20:12:13 +08:00
nhussain b65125b0a1 add more ifdef and include statments for pybind 2020-07-27 13:51:11 -04:00
mindspore-ci-bot ee67f70b73 !3449 Cifar100 for C API 
Merge pull request !3449 from MahdiRahmaniHanzaki/cifar100-c-api
 2020-07-28 00:09:41 +08:00
Mahdi 92568248c9 Added Cifar100 for C API 2020-07-27 10:35:24 -04:00
mindspore-ci-bot 84c8b85ddc !3452 [MD] Gets rid of uncessary print calls 
Merge pull request !3452 from nhussain/refactor_name_function
 2020-07-25 20:59:27 +08:00
mindspore-ci-bot 9c06a564d1 !2983 Dataset Tensor class cleanup 
Merge pull request !2983 from h.farahat/tensor_class
 2020-07-25 20:59:13 +08:00
Eric 465a69a70a Added crop signature, wip 
Added input check

Added SwapRedBlue
 2020-07-24 20:40:45 -04:00
hesham adfbc891d3 - Add checks and testing for empty tensors 
- cleanup work on createTensor and Tensor's constructors
 2020-07-24 17:23:18 -04:00
nhussain a306264030 remove print calls 2020-07-24 17:10:49 -04:00
Mahdi 2b71cbf1bd Added RenameOp 2020-07-23 12:22:55 -04:00
Cathy Wong 8921a6099c C++ API Support for Skip Dataset Op and UTs 2020-07-22 13:21:35 -04:00
mindspore-ci-bot 568e75fdd3 !3256 Zip C API 
Merge pull request !3256 from MahdiRahmaniHanzaki/zip-c-api
 2020-07-22 04:01:51 +08:00
Mahdi 91b4d90716 Added ZipOp 2020-07-21 10:24:38 -04:00
mindspore-ci-bot bfc18704d5 !3254 [MD]Add more strict upper bounds to CenterCropOp and RandomCropWithBboxOp 
Merge pull request !3254 from nhussain/padding_upperbound
 2020-07-21 10:18:56 +08:00
nhussain b064cbab5d fix upper bound 2020-07-20 15:26:49 -04:00
jiangzhiwen 2f506b7985 CSV dataset loader 2020-07-20 20:22:08 +08:00
xulei2020 18b519ae0f add sentence piece 2020-07-20 15:50:35 +08:00
anzhengqi 008b91b2a1 inject epoch ctrl op in the execution tree and send eos at the end of epoch 2020-07-20 13:02:47 +08:00
YangLuo 4136892a3e add SlidingWindow Op 2020-07-15 19:55:33 +08:00
mindspore-ci-bot ba0143402c !3067 Cleanup dataset UT: Remove unneeded tf data files and tests 
Merge pull request !3067 from cathwong/ckw_dataset_ut_cleanup6
 2020-07-15 04:44:52 +08:00
Cathy Wong ae1ed327ba Cleanup dataset UT: Remove unneeded data files and tests 2020-07-14 12:24:54 -04:00
liubuyu 43c79eb853 mindspore path adjust 2020-07-14 18:07:28 +08:00
Alexey Shevlyakov b930162e79 Decode + RandomCropAndResize = RandomCropDecodeResize 2020-07-13 20:20:27 -04:00
Jesse Lee 7c1bc5192b Initial Drop of CacheOp Phase I 2020-07-13 10:18:53 -04:00
mindspore-ci-bot 65c6e54e2e !2773 Adding cpp_api 
Merge pull request !2773 from 章一智/cpp_api
 2020-07-11 18:58:42 +08:00
mindspore-ci-bot 6284c42a76 !2941 MD tokenizer support output offsets 
Merge pull request !2941 from xiefangqi/md_add_offsets_feature
 2020-07-11 14:36:00 +08:00
xiefangqi 47060631e5 add offsets feature to tokenizer 2020-07-11 11:20:02 +08:00
ervinzhang bd5a777f81 introducing new C++ API 2020-07-10 16:32:38 -04:00
heleiwang 7a046a1d70 support get_edge_feature 2020-07-10 17:05:23 +08:00
mindspore-ci-bot 2c3c204b14 !2982 supporting cpp unit tests for randomresizewithbbox and resizewithbbox 
Merge pull request !2982 from ava/CPP_test3
 2020-07-10 04:56:36 +08:00
mindspore-ci-bot 02e797d130 !2947 CPP UTs for 3 BBoxAugOps + Fix to image_util.cc 
Merge pull request !2947 from danishnxt/AugOps-testing
 2020-07-10 03:31:52 +08:00
avakh b2bfb0342b supporting cpp unit tests for random_resize_with_bbox_op and resize_with_bbox_op. 2020-07-09 14:36:32 -04:00
Danish Farid 506f8f134e fix, cpp ut's 
fix cmakefile

fixed log statments in cpp UT tests
 2020-07-09 11:32:13 -04:00
mindspore-ci-bot 5355deca04 !2946 Added empty tensor support 
Merge pull request !2946 from EricZ/emtpy-tensor
 2020-07-09 09:38:32 +08:00
mindspore-ci-bot 3cc9217f06 !2926 [Dataset C++ UT] Implement C++ UTs for RandomHorizontalFlipWithBBox and BoundingBoxAugment 
Merge pull request !2926 from islam_amin/object_ops_cc_ut_floats
 2020-07-09 05:56:46 +08:00
Eric 79445012bd Tensor ut works 
new test case

Name change for empty check
 2020-07-08 17:28:59 -04:00
mindspore-ci-bot 915ddd25dd !2892 gnn random walk pr 1977 comment follow up 
Merge pull request !2892 from tony_liu2/master
 2020-07-09 03:35:55 +08:00
islam_amin 284cc910d5 C++ UTs for RandomHorizontalFlipWithBBox and Bounding Box Augment 2020-07-08 14:36:27 -04:00
tony_liu2 14899a1410 fix gnn random walk pr 1977 comments 
add fix to random resize decode crop test case

fix pylint issues
 2020-07-07 17:32:08 -04:00
mindspore-ci-bot 9ba937b1af !2920 Updating 3 BBox AugOps for Float + Common Files for BBox Float Support - (VertFlip/RandomCrop/RandomResizeCrop) 
Merge pull request !2920 from danishnxt/AugOps-M1-Python
 2020-07-08 05:18:58 +08:00
mindspore-ci-bot 6d946b6a19 !2890 Fix for Alexey's testcase 
Merge pull request !2890 from JesseKLee/queue_perf
 2020-07-08 03:21:50 +08:00
Danish Farid 78c370f72a First Commit - New Infrastructure - Python UT tests + Common Aug Files 
fix accidental change overwrite

fix 2

updated inits from 0 to 0.0 for float

improvd python ut comments

updated macro
 2020-07-07 14:50:34 -04:00
Jesse Lee e0ef91a676 Fix Queue::Reset 2020-07-06 11:36:14 -04:00
ale 383564c417 fix perf_data test 2020-07-02 13:24:45 -04:00
mindspore-ci-bot 709dfd7e81 !2738 [Dataset C++ UT] Implement C++ UT common class for object detection tests. 
Merge pull request !2738 from islam_amin/object_ops_ut_cc
 2020-07-03 01:23:50 +08:00
islam_amin 593ee1eb34 C++ UT common files for object detection tests 2020-06-30 17:54:35 -04:00
Eric 88bb65768e Clean up part 1 
Removed extra fields in schema

Fixed test cases

Fixing CI
 2020-06-29 15:57:53 -04:00
Alexey Shevlyakov 71cede8173 remove de_error.h 2020-06-29 14:01:50 -04:00
mindspore-ci-bot 9b4399480a !2321 Connector throughput performance metric 
Merge pull request !2321 from Alexey_Shevlyakov/connector_throughput_dev
 2020-06-29 23:54:42 +08:00
jonyguo 41d16a3c77 enhance: 1. fix api comment, 2. del dead code 2020-06-29 17:15:44 +08:00
Alexey Shevlyakov 698abf75fc Connector throughput + PerfData + CyclicArray 2020-06-26 14:45:01 -04:00
Jesse Lee 81bf4bde1d AutoIndexObj primary should start with 0 2020-06-26 12:22:44 -04:00
Jamie Nisbet c22eac743d subtree creation in python apis 
updates

fix

fix cpplint

fix
 2020-06-24 17:17:05 -04:00
mindspore-ci-bot d9b1825552 !2374 C++ UT for CutOut and Pad 
Merge pull request !2374 from MahdiRahmaniHanzaki/I1L892-pad-cutout-ut
 2020-06-24 22:56:17 +08:00
ms_yan e829535a86 change GetTensor into GetRow to avoid NullPtr 2020-06-24 15:07:32 +08:00
heleiwang 0d52888fc5 fix misspell and check parameters 2020-06-23 22:01:36 +08:00
Mahdi bca037e1be added cpp test cases for pad and cut out 2020-06-23 09:55:01 -04:00
ms_yan cf82aa9035 init remove storage op in c++ 
init remove storage op test case c++

remove source c++ files
 2020-06-20 13:52:07 +08:00
hesham 11826fb256 DuplicateOp 2020-06-18 19:47:27 -04:00
nhussain 5515016dba remove graphengine changes 
remove graphengine changes

concat op

Truncate Pair

concat_op

remove graph engine changes
 2020-06-18 17:17:50 -04:00
mindspore-ci-bot ffc8a3c362 !2274 add PadEndOp 
Merge pull request !2274 from xunxue/padend
 2020-06-19 05:05:14 +08:00
mindspore-ci-bot e4322baece !2316 [Dataset] Move python tree transformation into cpp pass 
Merge pull request !2316 from JunhanHu/alter_node_cpp
 2020-06-19 05:02:03 +08:00
xunxue 87aa9c8f7a PadEndOp 2020-06-18 16:07:34 -04:00
Junhan Hu 57f3732ac3 Move python tree transformation into cpp pass. 2020-06-18 15:52:38 -04:00
mindspore-ci-bot 5850b99161 !2211 Add TruncatePair Op to dataset 
Merge pull request !2211 from h.farahat/pair_truncate
 2020-06-18 20:58:40 +08:00
hesham b9495a9ccc Truncate Pair 2020-06-17 23:11:27 -04:00
mindspore-ci-bot a4048e192c !2256 Rename Sampler::Reset() to Sampler::ResetSampler() 
Merge pull request !2256 from JesseKLee/sampler
 2020-06-18 03:33:53 +08:00
mindspore-ci-bot ea829e89c3 !2198 Add MaskOp to dataset 
Merge pull request !2198 from h.farahat/mask_op
 2020-06-18 02:50:16 +08:00
Jesse Lee dcfdff6021 Rename Sampler Reset 2020-06-17 14:43:45 -04:00
qianlong 4f16f036be Add WhitespaceTokenizer and UnicodeScriptTokenizer for nlp 
add CaseFold, NormalizeUTF8

add RegexReplace

add RegexTokenizer

add BasicTokenizer

add WordpieceTokenizer

add BertTokenizer
 2020-06-17 15:47:04 +08:00
hesham f2462bb00d Mask Op 2020-06-16 23:51:46 -04:00
mindspore-ci-bot 625f2421b5 !1977 random walk in gnn node2vec 
Merge pull request !1977 from JonathanY/randomwalk
 2020-06-17 03:22:21 +08:00
mindspore-ci-bot a1f194c971 !2188 Remove B+ tree deadcode and add an additional output to Search function 
Merge pull request !2188 from JesseKLee/deadcode
 2020-06-17 01:53:01 +08:00
Jesse Lee 287ca71260 Remove dead code 2020-06-16 11:46:30 -04:00
hesham e2012a1de9 Slice Op 2020-06-16 02:32:40 -04:00
mindspore-ci-bot dceb3c2be4 !2026 Change C++ behavior in RandomCropAndResize 
Merge pull request !2026 from MahdiRahmaniHanzaki/I1K3FL-random-resized-crop
 2020-06-16 05:27:36 +08:00
Mahdi add29f95f5 resolved issues in the comments 2020-06-15 16:45:18 -04:00
mindspore-ci-bot 2cab44a637 !2028 FillOp for BERT 
Merge pull request !2028 from nhussain/fillOp_learning
 2020-06-16 02:45:45 +08:00
nhussain dd9bf09f0a added FillOp for #119 - special Ops 2020-06-15 11:30:43 -04:00
Jonathan Yan 87d2c27c7f random walk v1 2020-06-15 08:54:10 -04:00
mindspore-ci-bot a8cf83acf2 !1932 Add CLUE dataset 
Merge pull request !1932 from jiangzhiwen/dataset/clue
 2020-06-15 11:48:35 +08:00
jiangzhiwen e0e167a000 add CLUE dataset 2020-06-15 10:40:46 +08:00
mindspore-ci-bot c661087b99 !2029 Create TensorRow class that supports a row id 
Merge pull request !2029 from lixiachen/lixia/tensorrow
 2020-06-13 09:43:58 +08:00
Lixia Chen 6ae88c3987 Create TensorRow class that supports a row id. 2020-06-12 15:06:56 -04:00
hesham 0c5e47111b - Move MutableBuffer to be protected 
- Add new CreateTensors
 2020-06-11 17:32:03 -04:00
mindspore-ci-bot beefb20c01 !1983 Remove inheritance of Sampler from DatasetOp 
Merge pull request !1983 from JesseKLee/sampler
 2020-06-11 23:14:56 +08:00
mindspore-ci-bot b3da41bd7a !1913 CocoDataset implementation 
Merge pull request !1913 from xiefangqi/xfq_support_coco
 2020-06-11 14:37:44 +08:00
xiefangqi 097cbcc0cb CocoDataset implementation 2020-06-11 11:52:17 +08:00
Jesse Lee 255adf7c6e Remove inheritance of Sampler from DatasetOp 2020-06-10 18:10:51 -04:00
heleiwang 3ece8dd090 1. support get_all_edges, get_nodes_from_edge, get_sampled_neighbors, get_neg_sampled_neighbors and graph_info API 
2. mod cora and citeseer conversion script
 2020-06-10 15:52:26 +08:00
mindspore-ci-bot 5eb95599f6 !1874 Updates to string tensor 
Merge pull request !1874 from h.farahat/string_tensor_updates
 2020-06-10 00:59:17 +08:00
hesham f837ddc956 - Bug when empty strings sent to Python 
- Support accepting Numpy of str as input
- Support batching strings
- Core logic of batch&pad is static
- Make Pad a utility function
 2020-06-09 10:34:38 -04:00
Jamie Nisbet 51bc0c0460 consistent design for num_samples 
updates

more update

more work

more fixin

post rebase updates

clang formatting

code review recovery

ci fixes

updates

update

update

update
 2020-06-03 15:01:18 -04:00
Jesse Lee 3b80c10b1e Fix some timing hole in these two testcases 2020-05-27 14:22:37 -04:00
mindspore-ci-bot e311473401 !1453 Raise mse threshold in RandomCropDecodeResizeOp 
Merge pull request !1453 from Alexey_Shevlyakov/fix_random_crop_decode_resize_ut
 2020-05-27 23:58:58 +08:00
Jesse Lee a8fa847556 Allow join_all to be blocking 2020-05-25 15:24:36 -04:00
Alexey Shevlyakov 7a807cbff9 fix ut 2020-05-25 14:56:58 -04:00
Jesse Lee 641112a40c Initial Drop 2020-05-24 16:57:40 -04:00
mindspore-ci-bot 964a757db2 !1378 Just want to debug this case in Jenkin environment 
Merge pull request !1378 from JesseKLee/cond_var_debug
 2020-05-23 05:26:12 +08:00
Jesse Lee 06ae724261 Again 2020-05-22 16:38:03 -04:00
mindspore-ci-bot 458436186c !1365 Clean up work for text python sub-package 
Merge pull request !1365 from h.farahat/text_namespace
 2020-05-22 23:16:31 +08:00
mindspore-ci-bot 6f733ec113 !1308 Stage 2 of adding support for string Tensor 
Merge pull request !1308 from h.farahat/string_tensor2
 2020-05-22 21:17:53 +08:00
hesham 6c21e556c4 Clean up work for text python package 2020-05-22 04:35:58 -04:00
heleiwang 599a449e0b Support processing GNN data 2020-05-22 14:15:25 +08:00
hesham df361d1d26 Change mem layout of string tensor 
add support for MindRecord and TFRecord
----
optimize tensorshape

optimize tensorshape and FlatIndex

TFRecord and MindRecord support for string tensor

Modify mem layout
Add new constructor
Add method Allocate

Change some GetMutableBuffer usages to AllocateBuffer
 2020-05-22 01:22:29 -04:00
mindspore-ci-bot 2e3d55ed87 !1281 Implementation of SplitOp 
Merge pull request !1281 from Peilin/splitOp
 2020-05-22 09:29:03 +08:00
Peilin Wang 71e8bb1960 general split case done, chaining sampler (basic case) is working 
implementation 99% complete

everything and tested except for repeatable shuffling

tested most basic/typical split usecases

cleanup

some more cleanup

fix CI

more ci fix

more ci fixes

more ci fix

more ci fix

more ci fix

added more tests, fixed some bugs

some more clean up and test cases

added shard/shuffle before split warning/error

addressed code review comments and ci

fixed ci
 2020-05-21 20:42:51 -04:00
qianlong 451c20a6f5 Add UnicodeCharTokenizer for nlp 2020-05-21 09:22:45 +08:00
xulei2020 163b6b7ea7 add jieba c++ code 2020-05-20 15:55:12 +08:00
hesham e8ca243364 -Add DE_STRING 
-replace switch'case by indexing

- Add test case
- Add constructors
- Add getItem string

- Fix bugs
- Add more tests

- Tensor iterator
- asNumpy
- TextFileDataset

- Tensor(Numpy)

- Super > 2D
- Add more test cases for GeneratorDataset

- Change StartAddr to GetBuffer and GetMutableNuffer

- Raise an error if batch is used with strings

Clean-up work
 2020-05-15 20:33:28 -04:00
mindspore-ci-bot c680cfbf27 !1157 dataset: add concat operation for dataset 
Merge pull request !1157 from ms_yan/concat_dataset
 2020-05-15 16:07:19 +08:00
mindspore-ci-bot ab031ee9ea !1126 VOCDataset support object detection function 
Merge pull request !1126 from xiefangqi/voc_support_detection
 2020-05-15 15:56:39 +08:00
xiefangqi c937bad53f minddata support voc 2020-05-15 13:24:03 +08:00
ms_yan c0fa7b4b19 init commit of concat dataset 
change to use __add__ operation instead ds.concat
 2020-05-15 13:14:13 +08:00
mindspore-ci-bot 9e116e15fb !1139 Change Thread::Join to async call with time out 
Merge pull request !1139 from JesseKLee/timing_hole
 2020-05-14 09:07:06 +08:00
mindspore-ci-bot 1a052f7155 !1131 [Data]fix cmake build format 
Merge pull request !1131 from xulei/filter_op
 2020-05-14 08:24:35 +08:00
Jesse Lee f2be49d85e Fix a racing condition 2020-05-13 14:59:25 -04:00
xulei2020 903b64daa2 fix cmake build format 2020-05-13 17:00:35 +08:00
liyong aa3f89e74f mindrecord support read file list 2020-05-13 14:11:59 +08:00
Jesse Lee 3c643e7298 Move from dataset_dev branch to here 2020-05-11 15:11:53 -04:00
Zirui Wu c2d364a573 batch with padding implemented 
support for 1 specific dimension to be None, added validator

fix various CI complains

another round of CI fixes

ci

refactor parts of the code

code refactor

ci fix

comments added, fix bugs

address review comments

address review comments

review cmts

added simple perf test script

update pad code

perf imprv
 2020-05-07 11:18:42 -04:00
xiefangqi 9a2daae84c del unused change_mode_op 2020-05-06 11:54:05 +08:00
mindspore-ci-bot 2303453753 !869 Random data op 
Merge pull request !869 from JesseKLee/random_data_op
 2020-04-30 02:56:39 +08:00
Jesse Lee 270bf831a9 Random Data Op 2020-04-29 10:26:00 -04:00
mindspore-ci-bot 98939d839c !856 Fix skip op bug 
Merge pull request !856 from jiangzhiwen/dataset/skip_thread
 2020-04-29 21:47:50 +08:00
jiangzhiwen 34bfa2f7c9 fix skip 2020-04-29 20:12:37 +08:00
Jesse Lee 332398d20a Check return code from WaitPost::Register 2020-04-28 15:19:03 -04:00
mindspore-ci-bot 6369cf27bd !406 added first row crc check for when reading tfrecord files 
Merge pull request !406 from Peilin/first-row-crc-check
 2020-04-21 00:10:18 +08:00
xulei2020 c705ea5e5b add filterOp code 2020-04-20 22:32:19 +08:00
Peilin Wang 9bc2134cb7 added checking of first row crc to find invalid tfrecord files 
addressed code review comments. added check in python layer to exclude directories and to raise an error if a pattern does not match any file

fixed clang format

fixed cppcheck

fixed cppcheck (used std::accumulate and std::copy_if). regenerated tfrecord file to contain correct header, it was a dummy header before

fixed cppcheck: added const reference for string parameter for lambdas, fixed clang format: whitespace adjustments

more clang whitespace fixes...

changed print to logger.info
 2020-04-20 10:25:26 -04:00
mindspore-ci-bot d8176a77f4 !425 New api "TextFileDataset" 
Merge pull request !425 from yanghaitao/yht_textfiledatasetv2
 2020-04-20 22:14:26 +08:00
ms_yan c5cfb09e66 Repair some MS_LOG problem 2020-04-20 19:45:46 +08:00
yanghaitao 2795e492ff TextFileDataset 2020-04-20 11:18:55 +08:00
ms_yan f0c07c3fa6 Realize take op and add ut 2020-04-18 18:33:57 +08:00
jzw 3f7054dccb add skip dataset op 2020-04-17 19:48:25 +08:00
mindspore-ci-bot cf026096a6 !183 Mindspore.dataset CPP sampler for GeneratorDataset 
Merge pull request !183 from JunhanHu/cpp_sampler
 2020-04-16 22:30:59 +08:00
Junhan Hu 9739d3b048 Add CPP sampler support for GeneratorDataset 2020-04-14 12:09:03 -04:00
hesham 0fc23eee0f Support nested repeat 2020-04-14 10:13:18 -04:00
mindspore-ci-bot cc75cb357c !168 remove mindspore::make_unique and make_unique.h 
Merge pull request !168 from Alexey_Shevlyakov/remove_mindspore_make_unique
 2020-04-09 05:32:52 +08:00
jonyguo 20d1b64443 fix: error info is not exactly when column list invalid 2020-04-08 16:11:13 +08:00
Jonathan Yan 9d0fde29f4 remove ENABLE_MINDRECORD flag 2020-04-07 14:55:45 -04:00
Alexey Shevlyakov 84d780c1a4 remove make_unique.h 2020-04-07 14:00:09 -04:00
Alexey Shevlyakov d37b8c4f2d fix RandomCropDecodeResize test 2020-04-03 11:13:57 -04:00
Jonathan Yan 080bcda021 Replace std::cout with MS_LOG in dataset unit test 
Signed-off-by: Jonathan Yan <jonathan.yan1@huawei.com>
 2020-03-30 11:02:32 -04:00
zhunaipan 930a1fb0a8 initial version 
Signed-off-by: leonwanghui <leon.wanghui@huawei.com>
 2020-03-27 22:54:54 +08:00