yystopf
/
mindspore
forked from mindspore-Ecosystem/mindspore
0
0
Fork 0
Code Issues Pull Requests Projects Releases Wiki Activity
45,876 Commits 23 Branches 0 Tags 3.9 GiB
Commit Graph

1152 Commits

Author SHA1 Message Date
xiefangqi 41f3e02e87 fix pyfunc pickle issue 2021-03-12 15:17:08 +08:00
YangLuo 74d7bb41a6 fix minddata transform issue 2021-03-12 11:37:14 +08:00
mindspore-ci-bot f7040ca134 !13198 Fix get_dataset_size error with PKSampler 
From: @ziruiwu
Reviewed-by: 
Signed-off-by:
 2021-03-12 06:11:26 +08:00
mindspore-ci-bot 1bb789278d !13200 Fix CI warnings and test team issues 
From: @alexyuyue
Reviewed-by: @robingrosman,@nsyca
Signed-off-by: @robingrosman
 2021-03-12 05:38:01 +08:00
Zirui Wu 2692e3cc3d fix pk sampler's get_dataset error due to num_class unavaiable at pre-runtime 2021-03-11 15:43:02 -05:00
alex-yuyue 2457f52596 Fix CI warning and test team issues 
Signed-off-by: alex-yuyue <yue.yu1@huawei.com>
 2021-03-11 13:34:51 -05:00
RobinGrosman a0e75b845a Increase default level of parallelism in mindddata pipeline to 8. 
Adjust memory usage to not increase as parallelism increases.  it will
stay at same level it would be with 4 parallelism
 2021-03-10 18:38:11 -08:00
mindspore-ci-bot 3c7463bf2c !12943 adding fix to sample script 
From: @ezphlow
Reviewed-by: 
Signed-off-by:
 2021-03-10 10:22:19 +08:00
Eric ed373f61ac Adding inheritance to user defined sampler 
what if we just do nothing

trying wrapper approach

Fix yolo
 2021-03-09 18:14:49 -05:00
YangLuo 74431ad117 Fix bugs of minddata transforms 2021-03-06 16:41:51 +08:00
mindspore-ci-bot fe53c71b3b !12843 Add more callable test and clean the input check for tensor ops 
From: @alexyuyue
Reviewed-by: @robingrosman,@nsyca
Signed-off-by: @robingrosman
 2021-03-05 22:59:45 +08:00
alex-yuyue 6d7be32614 Add more callable test and clean the input check 
Signed-off-by: alex-yuyue <yue.yu1@huawei.com>
 2021-03-03 17:30:36 -05:00
mindspore-ci-bot 1c191a65fb !12765 Fix numpy input to samplers 
From: @hfarahat
Reviewed-by: 
Signed-off-by:
 2021-03-04 04:57:40 +08:00
mindspore-ci-bot c07371f5c9 !12763 Fix C++ API testcases 
From: @ezphlow
Reviewed-by: 
Signed-off-by:
 2021-03-04 04:57:28 +08:00
Eric f9a2379a70 Fixing api leftover 
Fixed compile error

Fixed more testcases
 2021-03-03 09:37:06 -05:00
alex-yuyue 66aff93940 Fix test team issue: ToNumber in eager mode. 
And support multiple tensor as input in call method

Signed-off-by: alex-yuyue <yue.yu1@huawei.com>
 2021-03-02 17:39:44 -05:00
hesham 616d1a1f3d Fix sampler supporting numpy input 2021-03-01 12:41:59 -05:00
hesham aa4298721a Fix sampler error messages 2021-02-23 23:58:19 -05:00
YangLuo d8ace8f041 fix minddata issues 2021-02-19 10:53:49 +08:00
mindspore-ci-bot c5df707b69 !12327 Block caching after random pyfunc cases 
From: @lixiachen
Reviewed-by: 
Signed-off-by:
 2021-02-17 08:56:10 +08:00
Lixia Chen 0667818d9a Block caching after random pyfunc 2021-02-12 14:10:00 -05:00
Mahdi 4b106f184c Fixed an issue with SequentialSampler 2021-02-10 12:55:10 -05:00
mindspore-ci-bot 260c77f4ad !12336 [MD] fix sampler log & transform log 
From: @luoyang42
Reviewed-by: 
Signed-off-by:
 2021-02-10 15:33:15 +08:00
luoyang 416ad23e9d fix resize & randomcoloradjust 
fix sampler
 2021-02-10 12:53:27 +08:00
mindspore-ci-bot 923dd47b0f !12331 Adding validators back to check method 
From: @ezphlow
Reviewed-by: @robingrosman,@nsyca,@robingrosman
Signed-off-by: @robingrosman
 2021-02-10 08:58:02 +08:00
Eric 5d91fa6d77 changed discription 
Added validator checks

Added builtin sampler check

Added proper input check for samplers
 2021-02-09 18:00:53 -05:00
hesham 58193bc469 User explicit deepcopy 2021-02-09 17:25:21 -05:00
luoyang cd86788cf3 fix bugs of execute & update doc description in minddata 2021-02-07 22:52:52 +08:00
TinaMengtingZhang ddab5ad417 Add support for vision op and py_transform op to serdes 2021-02-03 12:31:09 -05:00
mindspore-ci-bot e8f4cafd02 !11950 Overwrite default port/hostname from env variables for cache_admin 
From: @lixiachen
Reviewed-by: 
Signed-off-by:
 2021-02-03 23:38:48 +08:00
Lixia Chen 97cd142f8d Overwrite default port/hostname from env variables 
update copyright

add examples to CacheClient docstring
 2021-02-02 20:39:19 -05:00
hesham 1185218335 Support list of IDs as a sampler 2021-02-02 11:44:38 -05:00
luoyang fa4f67cbcd LookupOp: default value of unknown_token support NoneType 2021-02-02 16:17:02 +08:00
Nat Sutyanyong 5a7dc0accc Migrate repeat_pass.cc to IR optimizer and remove ExecTree optimizer 2021-02-01 12:30:16 -05:00
mindspore-ci-bot 320ea51308 !11853 Add call for decoupled image and text ops 
From: @alexyuyue
Reviewed-by: 
Signed-off-by:
 2021-02-01 19:47:00 +08:00
mindspore-ci-bot 794af6d094 !11834 [MD] Fix error msg description of dataset 
From: @xiefangqi
Reviewed-by: @liucunwei,@heleiwang
Signed-off-by: @liucunwei
 2021-01-30 14:49:24 +08:00
xiefangqi 3d72282f26 fix error msg issue 2021-01-30 11:03:53 +08:00
alex-yuyue 6fd58dc580 Add call for decoupled image and text ops 
Signed-off-by: alex-yuyue <yue.yu1@huawei.com>
 2021-01-29 21:36:31 -05:00
Mahdi 8f34faeb7a Changed bindings to SamplerObj 2021-01-29 14:20:05 -05:00
mindspore-ci-bot 20c1727b2c !11631 Add SubsetSampler 
From: @hfarahat
Reviewed-by: 
Signed-off-by:
 2021-01-29 03:48:45 +08:00
mindspore-ci-bot 8e4a6842ff !11681 gnn support edge weight sampling 
From: @heleiwang
Reviewed-by: @liucunwei
Signed-off-by: @liucunwei
 2021-01-28 19:15:37 +08:00
heleiwang 2dc9ba761c gnn support weight sampling neighbors. 2021-01-28 15:28:44 +08:00
hesham 7120c66998 Add SubsetSampler 2021-01-27 15:00:18 -05:00
mindspore-ci-bot 4cd6588af0 !11393 Migrating cache transform pass from execution tree to IR tree 
From: @lixiachen
Reviewed-by: 
Signed-off-by:
 2021-01-27 22:10:13 +08:00
mindspore-ci-bot f2fd357ffc !11623 fix issues of MD 
From: @luoyang42
Reviewed-by: 
Signed-off-by:
 2021-01-27 14:37:55 +08:00
luoyang 415c8b08a5 fix issues of MD: 
1. description of num_shard in datasets.py
2. add param validation for sampler
3. add description for add_child in sampler.py
 2021-01-27 09:30:42 +08:00
Lixia Chen 922e1f4f36 Migrate CacheTransformPass 2021-01-26 19:24:10 -05:00
mindspore-ci-bot 3708624a25 !11559 [MD] Change the default spill location of cache server 
From: @lixiachen
Reviewed-by: 
Signed-off-by:
 2021-01-27 03:12:24 +08:00
Lixia Chen be6d310946 Default to memory-only cache server 2021-01-25 18:38:24 -05:00
xiefangqi 3f76abcdee minddata err msg add file path 2021-01-22 16:38:46 +08:00
mindspore-ci-bot 2beb916bf5 !11427 TextOp decoupling 
From: @alexyuyue
Reviewed-by: 
Signed-off-by:
 2021-01-22 09:09:41 +08:00
mohammad b5e91139e2 add GetDatasetSize to concat node 2021-01-21 16:25:48 -05:00
alex-yuyue a8675f8227 TextOp decoupling 
Signed-off-by: alex-yuyue <yue.yu1@huawei.com>
 2021-01-20 21:52:45 -05:00
Lixia Chen 273865a72b Rename SamplerObj::Build() to SamplerBuild() 2021-01-18 15:42:34 -05:00
TinaMengtingZhang d3fdbceb61 [Part II] Push down json save logic to IR and add getter to each IR node 2021-01-15 18:03:37 -05:00
alex-yuyue b54e01bfb4 Tensor op decoupling stage 2 
Signed-off-by: alex-yuyue <yue.yu1@huawei.com>
 2021-01-14 11:51:16 -05:00
TinaMengtingZhang 4812fece80 [Part I] Push down json save logic to IR and add getter to each IR node 2021-01-12 11:17:45 -05:00
Nat Sutyanyong 686772cce7 Migrate GeneratorOp resetting logic to IR optimizer (incl. debug code) 
with comments from the peer view addressed
 2021-01-11 12:20:40 -05:00
mindspore-ci-bot 7957a3b6f5 !10988 Tensor op decoupling stage 1 
From: @ezphlow
Reviewed-by: 
Signed-off-by:
 2021-01-09 05:12:10 +08:00
Eric be46ccf721 Started vision decoupling 
map validate params

Changing bindings

Change behavior of vision.cc

Added signature and changed Cmake

Compiling

Need to fix compile

Added compiling random transforms

python changes for decoupling

two failed test case remaining

Compiling random choice

passes all ut

Changed assert

review comments

Added validate params

add back return value

Fix lint

py lint fix

pylint 2

Addressing comments
 2021-01-08 09:42:20 -05:00
Zirui Wu 0e68575e77 port over getter pass 
port over tensor_op_fusion pass

add fusion support for prebuildOpeartion (TensorOpeartion)
 2021-01-07 10:52:49 -05:00
mindspore-ci-bot 03e655f14a !10070 Fixing GetDatasetSize issue with chained samplers 
From: @mahdirahmanihanzaki
Reviewed-by: 
Signed-off-by:
 2021-01-06 05:11:51 +08:00
Xiao Tianci 31fed1a2f6 change code to import APIs from mindspore.dataset rather than mindspore.dataset.engine 2020-12-24 17:30:02 +08:00
mindspore-ci-bot 20df37e428 !10287 Add ut for dataset to_number 
From: @luoyang42
Reviewed-by: @heleiwang
Signed-off-by:
 2020-12-23 21:14:42 +08:00
Zirui Wu a1355a827c introduce a master switch to turn on and off profiler 2020-12-22 16:48:01 -05:00
YangLuo da6de519ae Add ut for dataset to_number 2020-12-21 20:09:03 +08:00
mindspore-ci-bot d99189f681 !10207 Fix some minor doc, typo related to AutoNumWorker 
From: @ziruiwu
Reviewed-by: @mikef
Signed-off-by:
 2020-12-19 06:35:59 +08:00
Zirui Wu 4ae7102fd3 address 2 tickets 2020-12-18 15:03:32 -05:00
Mahdi 98ea8fa6ea Added tests for chain samplers, fixed GetDatasetSize issue with chained samplers 2020-12-17 13:26:42 -05:00
mindspore-ci-bot fc11b7dd68 !10074 Fix a testcase error when shuffle is above cache 
From: @lixiachen
Reviewed-by: @liucunwei,@pandoublefeng
Signed-off-by: @liucunwei,@liucunwei
 2020-12-17 19:56:26 +08:00
mindspore-ci-bot 691e3baab6 !10109 [MD] Fix Decode doc & RandomNode log 
From: @luoyang42
Reviewed-by: @heleiwang,@liucunwei
Signed-off-by: @liucunwei
 2020-12-17 19:50:19 +08:00
mindspore-ci-bot 6035823c4b !10044 fix get num classes of concat 
From: @luoyang42
Reviewed-by: @liucunwei,@heleiwang,@liucunwei
Signed-off-by: @liucunwei,@liucunwei
 2020-12-17 19:30:50 +08:00
luoyang 595b7a782f [MD] Fix Decode doc & RandomNode log 2020-12-17 17:04:31 +08:00
Lixia Chen a7f0e13234 Remove special treatment for cache above repeat 2020-12-16 21:13:04 -05:00
YangLuo df9f4f41e8 fix get num classes of concat 2020-12-16 14:58:32 +08:00
Xiao Tianci 633f97bb48 fix get dataset size in distributed scenario 2020-12-15 20:41:45 +08:00
mindspore-ci-bot c4f284b928 !9910 [MD][Perf] MindData add NormalizePad for GPU performance 
From: @xiefangqi
Reviewed-by: 
Signed-off-by:
 2020-12-15 16:51:22 +08:00
xiefangqi 3c6cc3c58c Add NormalizePadOp 2020-12-15 11:45:41 +08:00
YangLuo 18e61a627a python eager add decode 2020-12-15 10:00:59 +08:00
mindspore-ci-bot ad6507d88a !9825 [MD] fix pad, normalize in c_trasnform 
From: @luoyang42
Reviewed-by: @heleiwang,@liucunwei
Signed-off-by: @liucunwei
 2020-12-14 10:48:10 +08:00
luoyang 1cc147df1d fix pad, normalize in c_trasnform 2020-12-11 17:46:29 +08:00
Zirui Wu 77521e78d2 staging 2020-12-10 14:45:04 -05:00
Zirui Wu 91a6b2b0ca Add checks to RandomData and LookUp 2020-12-09 17:09:34 -05:00
mindspore-ci-bot 317a97e6b9 !9336 auto num_parallel_workers setup 
From: @ziruiwu
Reviewed-by: 
Signed-off-by:
 2020-12-09 08:32:55 +08:00
mindspore-ci-bot a2c80435ce !9685 Fix a core dump in TreeConsumer::Terminate() plus minor cache fixes 
From: @lixiachen
Reviewed-by: @mikef,@nsyca
Signed-off-by: @nsyca
 2020-12-09 08:21:22 +08:00
Zirui Wu d6df1b0832 Implemented AutoNumWorker Pass which sets num_workers of selected parallel ops automatically if enabled 2020-12-08 17:49:05 -05:00
Lixia Chen 32b82c2737 Fix a core dump in TreeConsumer::Terminate() 
and other minor cache changes
 2020-12-08 15:38:40 -05:00
hesham ff65f6feb8 Remove batch_map multiprocess test case 2020-12-08 00:36:02 -05:00
mindspore-ci-bot aa5d4e08a9 !9544 Fix a possible seg fault in GenerateCRC caused by std::regex 
From: @lixiachen
Reviewed-by: @jonyguo,@heleiwang
Signed-off-by: @heleiwang
 2020-12-05 15:31:40 +08:00
Lixia Chen 27579fe858 Fix a seg fault caused by std::regex 2020-12-04 20:58:11 -05:00
Nat Sutyanyong 4cb78f2e03 Remove Repeat(1),Take(-1), and Skip(0) in NodeRemovalPass 2020-12-04 17:37:07 -05:00
mindspore-ci-bot 3280474d71 !9533 dataset Python Pushdown - more minor code rework; CSVDataset delim_field bug fix 
From: @cathwong
Reviewed-by: @nsyca,@mikef
Signed-off-by: @nsyca
 2020-12-05 06:01:13 +08:00
Jesse Lee 0c8dfc68ec Fix hang on RequestFreeTag 2020-12-04 14:54:30 -05:00
Cathy Wong 69b269c6cf dataset Python Pushdown - more minor code rework; CSVDataset delim_field bug fix 2020-12-04 14:39:52 -05:00
mindspore-ci-bot 4f607e9c26 !9335 Add multiprocessing support to per_batch_map 
From: @hfarahat
Reviewed-by: 
Signed-off-by:
 2020-12-04 15:13:46 +08:00
Nat Sutyanyong d69a29a44e Migrate 3 pre passes to IR optimizer, namely, cache_error_pass, epoch_injection, and removal_pass 2020-12-03 14:40:03 -05:00
alex-yuyue ccff3a5316 Change build to return status 
Signed-off-by: alex-yuyue <yue.yu1@huawei.com>
 2020-12-02 22:46:49 -05:00
hesham 809f9c91be Add multiprocessing support to per_batch_map 2020-12-02 15:40:14 -05:00
mindspore-ci-bot df44e1339e !9249 Optimizing GetDatasetSize 
From: @mahdirahmanihanzaki
Reviewed-by: 
Signed-off-by:
 2020-12-02 08:26:34 +08:00
mindspore-ci-bot 4ffd9b578f !9331 More cache minor fix for HQ testers 
From: @lixiachen
Reviewed-by: @mikef,@nsyca
Signed-off-by: @nsyca
 2020-12-02 06:45:15 +08:00
Mahdi f80553bc6e Optimization for GetDatasetSize 2020-12-01 17:40:25 -05:00
Lixia Chen 2ddb1fb75e More CRC fix and blocking BatchOp under cache 2020-12-01 15:20:33 -05:00
mindspore-ci-bot 920bbf1541 !9277 fix schema & zip validation 
From: @luoyang42
Reviewed-by: @pandoublefeng,@liucunwei
Signed-off-by: @liucunwei
 2020-12-01 16:28:48 +08:00
YangLuo 5cc3b76577 fix schema & zip validation 2020-12-01 14:23:56 +08:00
mindspore-ci-bot af3c27d354 !9244 Fix core dump with concat and getter_pass 
From: @ziruiwu
Reviewed-by: 
Signed-off-by:
 2020-12-01 06:56:41 +08:00
mindspore-ci-bot c3e7a8255c !9248 Add input check to nlp operators 
From: @ezphlow
Reviewed-by: @nsyca,@mikef
Signed-off-by: @nsyca
 2020-12-01 06:45:26 +08:00
Eric cf802c839c Added missing input check for nlp operator 2020-11-30 16:23:40 -05:00
Zirui Wu 23487fae09 fix get_shape/type and a segfault in getter_pass 
add test case
 2020-11-30 16:22:33 -05:00
mindspore-ci-bot 762793cb8f !9175 fix a bug in random resize crop op and some lint 
From: @tiancixiao
Reviewed-by: @liucunwei,@jonyguo,@liucunwei
Signed-off-by: @liucunwei
 2020-11-30 20:03:51 +08:00
Xiao Tianci 12da8717b7 1.correct param check for vision op 2.fix config review 3.fix api doc 2020-11-30 16:12:31 +08:00
luoyang aa4bf9a3f2 [MD] Support python-eager: Resize, Rescale, Normalize, HWC2CHW, Pad 2020-11-28 10:06:53 +08:00
tinazhang 6f31bdbf67 fix issues for hq testers: I277HD, I277HV 2020-11-27 17:01:23 -05:00
tinazhang 27698866d1 Block take/skip under cache and fix docstring 
and other minor changes per the request of HQ testers
 2020-11-26 17:44:19 -05:00
mindspore-ci-bot adc8e3e707 !8594 [MD] Pybind Pushdown Support for dataset 
From: @cathwong
Reviewed-by: 
Signed-off-by:
 2020-11-26 05:55:49 +08:00
Eric Zhang 809e1d5086 added python api based on cpp api 
1st draft of python iterator

Added Cifar10 and Cifar100 pybind port

Change pybind to use IR for Skip and Manifest

Signed-off-by: alex-yuyue <yue.yu1@huawei.com>

DatasetNode as a base for all IR nodes

namespace change

Fix the namespace issue and make ut tests work

Signed-off-by: alex-yuyue <yue.yu1@huawei.com>

Add VOCDataset

!63 Added RandomDataset
* Added RandomDataset

add imagefolder ir

Pybind switch: CelebA and UT

!61 CLUE example with class definition
* Merge branch 'python-api' of gitee.com:ezphlow/mindspore into clue_class_pybind
* Passing testcases
* Added CLUE, not working

add ManifestDataset IR

Signed-off-by: alex-yuyue <yue.yu1@huawei.com>

Update Coco & VOC & TFReader, Update clang-format, Reorder
datasets_binding

!69 Add Generator and move c_dataset.Iterator to dataset.Iterator
* Add GeneratorDataset to c_dataset
* Add GeneratorDataset to c_dataset

!67 Moving c_datasets and adding sampler wrapper
* Need to add create() method in datasets.py
* migration from c_dataset to dataset part 1

!71 Fix indent error
* Fix indentation error

!72 Fix c_api tests cases
* Fix c_api tests cases

!73 Added CSV Dataset
* Added CSVDataset

pybind switch: Take and CelebA fixes

!75 move c_dataset functionality to datasets
* Fixed existing testcases
* Added working clue and imagefolder
* Added sampler conversion from pybind
* Added sampler creation

!77 Add Python API tree
* Python API tree

add minddataset

TextFileDataset pybind

Rename to skip test_concat.py and test_minddataset_exception.py

!80 Add batch IR to python-api branch, most test cases work
* staging III
* staging, add pybind

Enable more c_api take and CelebA tests; delete util_c_api

!84 Schema changes in datasets.py
* Schema changes

!85 Remove input_indexes from sub-classes
* remove input_index from each subclass

!83 Remove C datasets
* Removed c_dataset package
* Remove c_datasets

!82  pybind switch: shuffle
* pybind switch: shuffle

!86 Add build_vocab
* Add build_vocab

Rebase with upstream/master
_shuffle conflict
BatchNode error

!88 Fix rebase problem
* fix rebase problem

Enable more unit tests; code typo/nit fixes

!91 Fix python vocag hang
* Fix python vocab hang

!89 Added BucketBatchByLength Pybind switch
* Added BucketBatchByLength

Update and enable more tet_c_api_*.py tests

!95 Add BuildSentencePeiceVocab
* - Add BuildSentencePeiceVocab

!96 Fix more tests
* - Fix some tests

- Enable more test_c_api_*
- Add syncwait

!99 pybind switch for device op
* pybind switch for device op

!93 Add getters to python API
* Add getters to python API

!101 Validate tree, error if graph
* - Add sync wait

!103 TFrecord/Random Datasets schema problem
* - TfRecord/Random schem aproblem

!102 Added filter pybind switch
* Added Filter pybind switch

!104 Fix num_samples
* - TfRecord/Random schem aproblem

!105 Fix to_device hang
* Fix to_device hang

!94 Adds Cache support for CLUE dataset
* Added cache for all dataset ops
* format change
* Added CLUE cache support
* Added Cache conversion

Add save pybind

fix compile err

init modify concat_node

!107 Fix some tests cases
* Fix tests cases

Enable and fix more tests

!109 pybind switch for get dataset size
* pybind_get_dataset_size

some check-code fixes for pylint, cpplint and clang-format

!113 Add callback
* revert
* dataset_sz 1 line
* fix typo
* get callback to work

!114 Make Android compile clean
* Make Android Compile Clean

Fix build issues due to rebase

!115 Fix more tests
* Fix tests cases
* !93 Add getters to python API

fix test_profiling.py

!116 fix get dataset size
* fix get dataset size

!117 GetColumnNames pybind switch
* Added GetColumnNames pybind switch

code-check fixes: clangformat, cppcheck, cpplint, pylint

Delete duplicate test_c_api_*.py files; more lint fixes

!121 Fix cpp tests
* Remove extra call to getNext in cpp tests

!122 Fix Schema with Generator
* Fix Schema with Generator

fix some cases of csv & mindrecord

!124 fix tfrecord get_dataset_size and add some UTs
* fix tfrecord get dataset size and add some ut for get_dataset_size

!125 getter separation
* Getter separation

!126 Fix sampler.GetNumSamples
* Fix sampler.GetNumSampler

!127 Assign runtime getter to each get function
* Assign runtime getter to each get function

Fix compile issues

!128 Match master code
* Match master code

!129 Cleanup DeviceOp/save code
* Cleanup ToDevice/Save code

!130 Add cache fix
* Added cache fix for map and image folder

!132 Fix testing team issues
* Pass queue_name from python to C++
* Add Schema.from_json

!131 Fix Cache op issues and delete de_pipeline
* Roll back C++ change
* Removed de_pipeline and passing all cache tests.
* fixed cache tests

!134 Cleanup datasets.py part1
* Cleanup dataset.py part1

!133 Updated validation for SentencePieceVocab.from_dataset
* Added type_check for column names in SentencePieceVocab.from_dataset

Rebase on master 181120 10:20

fix profiling

temporary solution of catching stauts from Node.Build()

!141 ToDevice Termination
* ToDevice termination

pylint fixes

!137 Fix test team issues and add some corresponding tests
* Fix test team issues and add some corresponding tests

!138 TreeGetter changes to use OptPass
* Getter changes to use OptPass (Zirui)

Rebase fix

!143 Fix cpplint issue
* Fix cpplint issue

pylint fixes in updated testcases

!145 Reset exceptions testcase
* reset exception test to master

!146 Fix Check_Pylint Error
* Fix Check_Pylint Error

!147 fix android
* fix android

!148 ToDevice changes
* Add ToDevice to the iterator List for cleanup at exit

!149 Pylint issue
* Add ToDevice to the iterator List for cleanup at exit

!150 Pylint 2
* Add ToDevice to the iterator List for cleanup at exit

!152 ExecutionTree error
* ET destructor error

!153 in getter_pass, only remove callback, without deleting map op
* getter pass no longer removes map

!156 early __del__ of iterator/to_device
* early __del__ of iterator

!155 Address review comments Eric 1
* Added one liner fix to validators.py
* roll back signature fix
* lint fix
* Eric Address comments 2
* C++ lint fix
* Address comments Eric 1

!158 Review rework for dataset bindings - part 1
* Reorder nodes repeat and rename
* Review rework for dataset bindings - part 1

!154 Fixing minor problems in the comments (datasets.py, python_tree_consumer.cc, iterators_bindings.cc, and iterators.py)
* Fixing minor problems in the comments (datasets.py, python_tree_consum…

!157 add replace none
* Add replace_none to datasets.py, address comments in tests

Trying to resolve copy

Override the deepcopy method of deviceop

Create_ir_tree method

Create_ir_tree method 2

Create_ir_tree method 2

del to_device if already exists

del to_device if already exists

cache getters shapes and types

Added yolov3 relaxation, to be rolled back

Get shapes and types together

bypass yolo

NumWorkers for MapOp

revert Yolo

revert Thor

Print more info

Debug code: Update LOG INFO to LOG ERROR

do not remove epochctrl for getter pass

Remove repeat(1)

pritn batch size

add log to tree_consumer and device_queue op

Revert PR 8744

Signed-off-by: alex-yuyue <yue.yu1@huawei.com>

__del__ toDEvice

__del__ toDevice2

!165 add ifndef ENABLE_ANDROID to device queue print
* Add ifndef ENABLE_ANDROID to device queue print

revert some changes

!166 getter: get_data_info
* getter: get_data_info

!168 add back tree print
* revert info to warnning in one log
* add back the missed print tree log

Release GIL in GetDataInfo
 2020-11-25 15:46:59 -05:00
Lixia Chen ab7427f1a9 Minor fix for DatasetCache param validation 2020-11-25 13:20:30 -05:00
mindspore-ci-bot f1ef84e1a6 !8714 dataset Python UT: Add Sampler Chain Tests, with get_dataset_size 
From: @cathwong
Reviewed-by: @nsyca,@mikef
Signed-off-by: @nsyca
 2020-11-19 03:26:44 +08:00
ms_yan 0cb5c47856 add err modify 2020-11-18 16:30:24 +08:00
Cathy Wong e5f62013f4 dataset Python UT: Add Sampler Chain Tests, with get_dataset_size 2020-11-17 15:24:44 -05:00
mindspore-ci-bot d2427b6e89 !8538 fix get_num_class in imagefolder 
From: @ziruiwu
Reviewed-by: @robingrosman,@nsyca
Signed-off-by: @nsyca
 2020-11-13 05:07:11 +08:00
Zirui Wu 3555d19b07 fix get num_class in imagefolder 
address cmt

review cmts
 2020-11-12 14:26:54 -05:00
Jesse Lee c795879313 CacheOp delivery for HQ tester 2020-11-12 08:48:29 -05:00
Xiao Tianci 4a5e287b92 add filter c api and ut 2020-11-06 15:49:23 +08:00
jonyguo 1b78e34310 add ANTIALIAS support for python progress 2020-11-03 11:03:46 +08:00
YangLuo 82cc7c3f63 [MD] Revert changes to weighted_random_sampler in PR7866 2020-10-31 12:56:17 +08:00
mindspore-ci-bot cacebd1211 !7933 Cache performance updates 
Merge pull request !7933 from Jamie/CacheOp_dev
 2020-10-31 04:25:47 +08:00
liyong 0d40abf5dd fix md padding bug 2020-10-30 12:00:44 +08:00
Lixia Chen 572c5e5f29 Rebase up to 542a52fbf8 2020-10-29 20:34:09 -04:00
Zirui Wu ea97197311 fix tfrecord_op core dump 
add test case

fix ci

fix ci round 2

address review cmts
 2020-10-28 17:38:43 -04:00
Xiao Tianci 823e52f1dc add params checking of offset in distributed_sampler 
add related comment to explain
add related UT
 2020-10-28 16:46:12 +08:00
mindspore-ci-bot c5867d2d7e !7676 [MD] fix arm compile error 
Merge pull request !7676 from liyong126/fix_arm_env
 2020-10-24 14:43:39 +08:00
liyong 1e7f46c998 fix unorder_map hash 2020-10-24 11:03:06 +08:00
liyong ee042b90f7 fix num_sample in sequentialSampler and randomSampler 2020-10-23 19:26:10 +08:00
xulei2020 8f392204be minddata build with mindspore-lite train 2020-10-22 21:34:56 +08:00
liyong fb2bd156c9 add unique op 2020-10-22 09:12:37 +08:00
Zirui Wu c42a1b4a2f fix batch core dump when col name doesn't exist 
fix ci
 2020-10-13 10:50:07 -04:00
mindspore-ci-bot 6f77ec45f1 !6905 [MD] Enable cache for more leaf datasets 
Merge pull request !6905 from lixiachen/CacheOp_dev
 2020-09-30 16:49:23 +08:00
mindspore-ci-bot 2a799fe90e !6018 Set the number of epochs in mode.train() non-sink Mode 
Merge pull request !6018 from h.farahat/ms_nonsink_epochs
 2020-09-28 22:15:54 +08:00
xiefangqi 82bc278e26 minddata use new ps api 2020-09-27 17:38:22 +08:00
jonyguo 19ca9762db skip graphdata distributed test case when asan mode 2020-09-27 16:24:20 +08:00
Lixia Chen 477808e622 Enable cache for other leaf ops 2020-09-25 13:11:04 -04:00
mindspore-ci-bot 0d8015b043 !5300 [MD] Add nD Slice Support 
Merge pull request !5300 from nhussain/slice_op
 2020-09-25 20:38:19 +08:00
mindspore-ci-bot b785b7d0ff !6813 dataset UT: check PIL version, delete test_callback_2maps 
Merge pull request !6813 from cathwong/ckw_dataset_cleanup8
 2020-09-25 20:34:55 +08:00
hesham 9cee0d2143 Add num_epochs to non-sink training 2020-09-24 23:24:43 -04:00
Cathy Wong f2b07d907a dataset UT: check PIL version, delete test_callback_2maps 2020-09-24 17:01:06 -04:00
mindspore-ci-bot 4c0b3c1bd3 !6017 per_batch_map needs to support x input columns and y output columns 
Merge pull request !6017 from ZiruiWu/per_batch_map_multi_col
 2020-09-25 04:08:01 +08:00
mindspore-ci-bot e88e114a50 !5930 Cache server phase 2 single node 
Merge pull request !5930 from Jamie/CacheOp_dev
 2020-09-25 03:54:55 +08:00
nhussain 77d507279c first working implementation 
add string implementation, can handle partial string slices

finish core implementation, added Index object

fix up the logic

add in new SliceOption object, cleaning up tests
 2020-09-24 15:39:28 -04:00
Lixia Chen 983827ec5c Rebase up to 88ded11f59 2020-09-24 14:09:08 -04:00
Zirui Wu 0cab373223 per_batch_map supports x num_columns to y num_columns 
decided to use project after batch as well

per_batch_map 1->x colunm works

add more test case

add test cases, project after map

col order hahaah

fix ci

add test cases

addr review cmts

ci

fix build err

fix test case failure

ci
 2020-09-24 12:01:22 -04:00
mindspore-ci-bot 2f3add4acd !6301 [MD] Combine c++ and python ops in map 
Merge pull request !6301 from nhussain/c_py_compose
 2020-09-24 23:10:47 +08:00
jonyguo 1c6c54ae0f fix: testcase test_cutmix_batch_op.py slowly 2020-09-23 10:24:39 +08:00
nhussain fda9462682 embed python compose op 
Leaf ops do self-reset.

embed python compose op

add implicit compose op

more tests
 2020-09-21 11:24:55 -04:00
shenwei41 f2e34b2eaf Clean up the view code 2020-09-19 19:02:04 +08:00
mindspore-ci-bot 53a82fa6e0 !6472 fix get_dataset_size error in CelebaDataset when usage is not 'all' 
Merge pull request !6472 from yanghaitao/yht_celeba_get_dataset_size
 2020-09-19 18:07:25 +08:00
mindspore-ci-bot 92aecf128f !6549 [MD] minddata add iterator parameter validation 
Merge pull request !6549 from xiefangqi/add_output_numpy_valid
 2020-09-19 14:30:10 +08:00
yanghaitao1 4ff4c17632 fix get_dataset_size in CelebADataset when usage is not all 2020-09-19 02:16:29 -04:00
xiefangqi a6360cb2e4 add output_numpy validation to iterator 2020-09-19 11:56:24 +08:00
yanghaitao1 dd0d9fe9ab fix ci error 2020-09-18 23:49:20 -04:00
yanghaitao1 da15495d7b fix bucket_batch ci error 2020-09-18 08:16:14 -04:00
mindspore-ci-bot bef2118253 !6300 [MD] fix get_dataset_size bug when set num_samples in DistributedSampler 
Merge pull request !6300 from liyong126/fix_md_num_samples
 2020-09-16 09:56:53 +08:00
mindspore-ci-bot 0d486adef1 !3974 Eliminate the pause for each cycle of the Repeat 
Merge pull request !3974 from lixiachen/repeat_task2
 2020-09-16 09:35:39 +08:00
liyong 930e85ed5c fix get_dataset_size in distributedSampler & num_samples 2020-09-15 22:53:57 +08:00
Cathy Wong f7adf648e9 dataset API docstring: Update datasets, samplers, graphdata and text 2020-09-14 18:10:44 -04:00
Lixia Chen b5b333726f Leaf ops do self-reset. 2020-09-14 09:48:14 -04:00
xiefangqi 653c79b4af md fix per_batch_map issue 2020-09-14 09:55:51 +08:00
liyong 16147669a6 fix num_samples in concatDataset 2020-09-12 15:06:18 +08:00
mindspore-ci-bot eb049b0e93 !5842 [Dataset] dataset op log changes 
Merge pull request !5842 from luoyang/pylint
 2020-09-12 09:42:42 +08:00
mindspore-ci-bot 28e2b0253e !6086 Order dict by column_id in create_dict_iterator() 
Merge pull request !6086 from ZiruiWu/order_dict_in_create_dict_iterator
 2020-09-12 08:41:46 +08:00
Zirui Wu d54ba374b9 order dict by column_id in create_dict_iterator() 
fix CI
 2020-09-11 15:33:47 -04:00
YangLuo f5a85510f2 [MD] dataset op log changes 2020-09-12 00:43:36 +08:00
nhussain 96de2ffdb6 fix errors from old compose call in Python 2020-09-11 10:30:23 -04:00
mindspore-ci-bot d8c09067ce !5801 [MD][Feature] minddata iterator output ms_tensor 
Merge pull request !5801 from xiefangqi/md_modify_output_tensor
 2020-09-11 20:19:06 +08:00
mindspore-ci-bot 5b4642d10a !4950 Fixing Minddata profiling monitor 
Merge pull request !4950 from anthonyaje/monitor_fix
 2020-09-11 14:34:27 +08:00
xiefangqi 9b3c33e157 minddata iterator output ms_tensor 2020-09-11 11:35:50 +08:00
mindspore-ci-bot cf7d6eddc4 !6019 [MD] Move Random(Choice/Apply/Order) from vision to transforms module 
Merge pull request !6019 from nhussain/move_random_choice_apply
 2020-09-11 10:37:21 +08:00
mindspore-ci-bot a778868a5a !5929 BucketBatchByLength column issue 
Merge pull request !5929 from MahdiRahmaniHanzaki/bucket_batch_by_length_fix
 2020-09-11 05:53:48 +08:00
nhussain f42f9ef9d7 move random apply order and choice ops 2020-09-10 15:39:56 -04:00
nhussain 92e99ff224 change map calls 2020-09-10 10:25:05 -04:00
anthonyaje 2d08d83247 Fixed connector_size and monitor. 2020-09-09 17:54:59 -04:00
Zirui Wu 1bb93580dd Add usage flag to MNIST/CIFAR 
ci fix 1

fix ci 2

fix ci 3

fix ci 4

fix ci 5

fix ci 6

fix ci 7

change var name in CelebA from dataset_type to usage

address review cmts

fix cpp ut fail

change mode to usage in VOCDataset

fix cmts

fix ci test case fail
 2020-09-09 09:59:55 -04:00
xiefangqi d0410d6191 md delete set_dataset_size interface 2020-09-09 09:56:03 +08:00
mindspore-ci-bot c45f79d36b !5384 [MD]-Api changes 
Merge pull request !5384 from nhussain/api_changes
 2020-09-09 09:49:05 +08:00
mahdi a152a65227 Fixed the column problem in BucketBatchByLength 2020-09-08 17:48:42 -04:00
Zirui Wu bde9f18f5a update lookup api to take in a type 
ci

add test case

address some of the review cmts

address review cmts
 2020-09-08 11:59:28 -04:00
nhussain 3bac9d3713 switch input columns and operation 
change ImagefolderDV2 name

change ds.transforms.vision to ds.vision

change batch api to match map api more closely

compose op changes

test_pylint

remove compose op from vision, move to transform module, refactor map and batch to use column_order
 2020-09-08 10:26:20 -04:00
mindspore-ci-bot d76ac7c6e8 !5648 GetColumnNames for Python 
Merge pull request !5648 from MahdiRahmaniHanzaki/get-col-name
 2020-09-03 22:37:42 +08:00
Mahdi 6e7aab8c4f Added GetColumnNames for Datasets 2020-09-02 17:12:48 -04:00
Cathy Wong 4d4c11b133 dataset API docstring cleanup: Standard product terms NumPy, Python 2020-09-02 10:42:12 -04:00
mindspore-ci-bot bb84f50407 !5473 optim pylint 
Merge pull request !5473 from jinyaohui/master
 2020-09-02 15:25:30 +08:00
Lixia Chen 27d5281641 Change epoch count to 1 for python testcases 2020-09-01 11:24:26 -04:00
mindspore-ci-bot 8f3ebfd469 !5624 Optimize the performance of GraphData.get_neg_sampled_neighbors 
Merge pull request !5624 from heleiwang/gnn_perf
 2020-09-01 19:22:53 +08:00
heleiwang 857cf2f77f Optimize the negative sampling process 2020-09-01 16:29:59 +08:00
mindspore-ci-bot 5453b40311 !4477 add dataset compare function to utils 
Merge pull request !4477 from tony_liu2/staging
 2020-09-01 11:15:18 +08:00
Cathy Wong 7f6782be2a dataset fixes: Update OneHot API docs; fixup Python UTs 2020-08-31 15:04:26 -04:00
jinyaohui a9972a7def optim pylint 2020-08-31 14:49:57 +08:00
tony_liu2 70bfd506a1 add function 
pr comment fix

revert graphengine
 2020-08-27 16:50:19 -04:00
mindspore-ci-bot 7aaf4118bf !5297 CutMixBatch and MixUpBatch fix for 2D one-hot labels 
Merge pull request !5297 from MahdiRahmaniHanzaki/cutmix-fix
 2020-08-27 21:32:09 +08:00
mahdi 7ad7024c45 Fixed 2D one-hot label problems in CutMix and MixUp 2020-08-26 16:17:21 -04:00
islam_amin 94a4a3f20b changing affine and posterize apis 2020-08-26 10:51:54 -04:00
mindspore-ci-bot e4d2f2fd3b !5030 Adding a BBox class and fixing Posterize C Op testing. 
Merge pull request !5030 from islam_amin/bbox_class
 2020-08-26 11:40:57 +08:00
mindspore-ci-bot 20b3134785 !5184 fix: padded dataset when no div and with repeat op 
Merge pull request !5184 from guozhijian/fix_padded_with_no_div_repeat
 2020-08-26 10:47:22 +08:00
mindspore-ci-bot f3fd7a5578 !5073 Add checks and exception handling DS callback 
Merge pull request !5073 from h.farahat/map_callback_end
 2020-08-26 10:40:40 +08:00
jonyguo 6df1a43ec2 fix: padded dataset with non div & repeat 2020-08-26 08:56:37 +08:00
mindspore-ci-bot 346b97acb8 !5156 fix softdvpp coredump 
Merge pull request !5156 from qianlong21st/master_fix_softdvpp_core
 2020-08-25 22:20:00 +08:00
qianlong 12dfca71f1 fix softdvpp coredump 2020-08-25 18:56:40 +08:00
YangLuo 67f5c89cd6 Check input image type for random posterize 2020-08-25 16:26:17 +08:00
islam_amin 641e751d5e adding bbox class 2020-08-24 18:22:58 -04:00
Mahdi a2c38d89f9 Added float32 support for CutMixBatch 2020-08-24 17:15:25 -04:00
hesham 8eeceb267b - release the wait if training is ended 2020-08-24 13:11:07 -04:00
mindspore-ci-bot ac81886328 !4916 fix generator_dataset hangs and test_graphdata_distributed.py failing randomly 
Merge pull request !4916 from heleiwang/gnn_fix_bug
 2020-08-22 10:51:01 +08:00
mindspore-ci-bot 9b503e4f38 !4955 Fixes for Dynamic Augmentation Ops 
Merge pull request !4955 from MahdiRahmaniHanzaki/dynamic-ops-fix
 2020-08-22 10:27:50 +08:00
Mahdi a5f9b8f92e Added fix for MixUpBatch and CutMixBatch and for RandomAffine 
updated c color op descriptions
 2020-08-21 17:40:08 -04:00
mindspore-ci-bot 9d7250c483 !4776 Introduce 2 extra ctrl flags to DataBuffer in dataset, address remaining cmts to PR4632 
Merge pull request !4776 from ZiruiWu/map_callback_follow_up
 2020-08-22 03:31:31 +08:00
Zirui Wu 74c1e6da60 introducing pause and quit flags to DataBuffer 
fix review cmts

fix ci

fix Ci

fixci

address ci

ci

- add timeout
- add more test cases

fix CI

address review cmts
 2020-08-21 14:33:26 -04:00
mindspore-ci-bot 82c888f065 !4930 Fix CI cifar hang issue 
Merge pull request !4930 from xiefangqi/fix_cifar_nofile_issue
 2020-08-21 20:49:40 +08:00
xiefangqi e3e7820413 fix cifar stuck problem 2020-08-21 19:35:07 +08:00
heleiwang 4870abc848 1. fix generator_dataset hangs 
2. fix test_graphdata_distributed.py failing randomly
 2020-08-21 16:03:06 +08:00
jonyguo 5b4b539751 fix: concat with none sample dataset 2020-08-21 09:19:50 +08:00
mindspore-ci-bot ed954dc407 !4782 Python UT fails on RandomPosterize 
Merge pull request !4782 from islam_amin/bbox_class
 2020-08-20 10:18:59 +08:00
islam_amin 85357238fe Fixing posterize python ut 2020-08-19 21:35:07 -04:00
Jesse Lee 8a08d0c37b Phase 2 of CacheOp 2020-08-19 13:45:55 -04:00
Zirui Wu 78c1aa1d96 Implemented Callback for Dataset 
implment pause in MapOp, added more to callback

add ds_callback

- Initial drop of Python DSCallback

- Pybind DSCallback

- Pybind DSCallback

added callback to mapOp

- de_pipeline DSCallback

- de_pipeline DSCallback

add test case, segfault for now

fix seg fault

- de_pipeline DSCallback

remove 1 line

update callback test case, now works

use builder class for mapOp callback

- de_pipeline DSCallback

- de_pipeline DSCallback

- de_pipeline DSCallback

better test case

minor fix

add comments and minor clean ups

get rid of nullptr in MapOp, use other flag instead

fix a bug ParseMapOp only takes 1 callback

- Added WaitedDSCalabck

refactor callback param

fix text case incorrect number

- added testing

fix cpp test case

- added testing

- revert back lenet changes

- cleanup test_callbacks.py

- cleanup test_callbacks.py

fix CI stage I

fix CI stage II

fix CI and update epoch counter

- add validation
- add more testing  test_callbacks.py

use random data op to do tests

adjust when to call EpochBegin/End

- add repeat with callback

- addressing reviewers' comments

- docstring and CI fixes

- docstring and CI fixes

- docstring and CI fixes

- rebase with upstream/master

fix cpp test case

fix review comments

addr review cmts, add test case
 2020-08-19 12:31:04 -04:00
mindspore-ci-bot 256dccc651 !4498 Gnn data processing supports distributed scenarios 
Merge pull request !4498 from heleiwang/gnn_distributed
 2020-08-19 16:09:05 +08:00
heleiwang 8ee4d8e92d Gnn data processing supports distributed scenarios 2020-08-19 11:09:01 +08:00
mindspore-ci-bot 50b5c8035b !4701 fix concat get_dataset_size error 
Merge pull request !4701 from guozhijian/fix_concat_get_dataset_size_err
 2020-08-19 09:14:30 +08:00
nhussain 738ae2c78d change api to use std_vector 
try to display image
 2020-08-18 15:41:21 -04:00
islam_amin 979111d0ef Added Posterize Op 2020-08-18 13:55:47 -04:00
jonyguo b0b370336f fix: concat get_dataset_size error 2020-08-18 23:40:17 +08:00
jonyguo 5f294fb90d fix: random affine parameter check 2020-08-18 21:58:53 +08:00
mindspore-ci-bot 113ff6ca24 !4367 CutMixBatch Augmentation Op 
Merge pull request !4367 from MahdiRahmaniHanzaki/cutmix
 2020-08-18 03:27:33 +08:00
mahdi 3ecc53fb4e Added CutMix 2020-08-17 13:01:47 -04:00
jonyguo 8bb4449fa8 add testcase for padded dataset with decode op 2020-08-17 16:43:23 +08:00
mayang c410c6a6e2 soft_dvpp_decode_resize_jpeg_op issue resolved, input size can be an int, break the constraints of input size being sequence (height, width) 2020-08-17 14:57:04 +08:00
Alexey Shevlyakov 8526d5414d RandomColor 2020-08-13 16:10:21 -04:00
mindspore-ci-bot 7a10e07fc6 !4383 Add distributedSampler to concatOP 
Merge pull request !4383 from genglishuai/genglshOK
 2020-08-14 00:06:06 +08:00
genglishuai 4d4df68d1e add DistributedSampler for Concat op 2020-08-13 22:52:02 +08:00
mindspore-ci-bot ed1244558c !4366 Fixing param type in c++ api and fixing python validator for Repeat Op 
Merge pull request !4366 from TinaMengtingZhang/cpp-api-repeat-count
 2020-08-13 22:03:39 +08:00
tinazhang 0868720eb7 fix parameter type for repeat op in c++ api and added c++/python ut. 2020-08-12 17:23:42 -04:00
anzhengqi 3e31ac6d62 all Dataset support get_dataset_size 2020-08-12 15:35:47 +08:00
mindspore-ci-bot 9276a9d2fb !3771 C++ API Support for TextFile Dataset 
Merge pull request !3771 from cathwong/ckw_c_api_textfile_nosampler
 2020-08-12 04:32:05 +08:00
avakh 477528de7f random sharpness cpp op support 2020-08-11 15:30:40 -04:00
nhussain 7766efd58e add files 2020-08-11 11:54:14 -04:00
Cathy Wong 7f39b5cfd7 C++ API Support for TextFile Dataset and Unit Tests 2020-08-11 11:53:53 -04:00
mindspore-ci-bot c2997845c2 !4289 Fix pad op validate issue 
Merge pull request !4289 from xiefangqi/fix_pad_validate
 2020-08-11 20:20:35 +08:00
xiefangqi 7ee1e46455 fix pad op validate issue 2020-08-11 17:28:08 +08:00
qianlong e9c4378ed8 add support of soft dvpp op 2020-08-11 15:46:07 +08:00
mindspore-ci-bot 266df56570 !4220 [MD] add num_samples support in minddataset pk sampler 
Merge pull request !4220 from liyong126/minddataset_pk_num_samples
 2020-08-11 11:22:28 +08:00
islam_amin e52d85ba9d Added RandomAffine Op 2020-08-10 17:52:50 -04:00
mindspore-ci-bot 9ad82f79fd !3979 Adding MixUp operation 
Merge pull request !3979 from MahdiRahmaniHanzaki/mixup
 2020-08-10 23:01:26 +08:00
liyong 7341421d3b fix num samples in pk sampler 2020-08-10 19:05:47 +08:00
xiefangqi 3d7826b6df test five crop problem 2020-08-10 16:59:24 +08:00
Mahdi 1896950ae5 Added Mixup 2020-08-07 17:47:16 -04:00
mindspore-ci-bot e19d382473 !3346 Maintain epoch/repeat count for ops 
Merge pull request !3346 from lixiachen/repeat_rework
 2020-08-07 20:39:19 +08:00
Lixia Chen ac85b77b76 Maintain epoch/repeat count for ops 2020-08-06 09:47:31 -04:00
mindspore-ci-bot 7ec0b5857a !3977 [MD] Refactor Concatenate Op 
Merge pull request !3977 from nhussain/multi_dim_concat_2
 2020-08-06 02:53:06 +08:00
nhussain 61769b2dd9 fix concat 2020-08-05 11:28:56 -04:00
mindspore-ci-bot 27e51c5a63 !3553 Unify np array comparisons 
Merge pull request !3553 from tony_liu2/staging
 2020-07-31 21:39:31 +08:00
dengyutao 388e43f7cb delete parameter block_reader for MindDataset 2020-07-31 11:30:05 +08:00
xiefangqi 6017b050ac add generatordataset support split 2020-07-30 15:51:28 +08:00
tony_liu2 269b477684 use np.testing.assert instead of asserting 
fix line lengths

fix spacing

reverse a few changes
 2020-07-29 14:53:01 -04:00
mindspore-ci-bot d06da1d270 !3603 Check that the number columns of names and default matches 
Merge pull request !3603 from jiangzhiwen/fix_column_names_exceeded
 2020-07-29 11:40:06 +08:00
jiangzhiwen 2cc6b5cb52 fix number of columns not match 2020-07-29 09:19:33 +08:00
guansongsong 1906ed1be8 fix cache description again 2020-07-28 13:26:29 +08:00
guansongsong b6529b9096 fix cache description 2020-07-28 13:03:06 +08:00
guansongsong 620d951e5e Merge branch 'ql/fix_cache_core' of https://gitee.com/qianlong21st/mindspore into ql/fix_cache_core 2020-07-28 11:37:06 +08:00
mindspore-ci-bot 9c06a564d1 !2983 Dataset Tensor class cleanup 
Merge pull request !2983 from h.farahat/tensor_class
 2020-07-25 20:59:13 +08:00
qianlong bf87b1d1fb fix cache op core dump 2020-07-25 18:37:02 +08:00
mindspore-ci-bot 0a2980ca74 !3440 [MD]Fix Segementation Falut when SentencepieceTokenizer Op before zipOp and ConcatOp 
Merge pull request !3440 from xulei/sentence_piece0715
 2020-07-25 09:19:34 +08:00
hesham adfbc891d3 - Add checks and testing for empty tensors 
- cleanup work on createTensor and Tensor's constructors
 2020-07-24 17:23:18 -04:00
mindspore-ci-bot 06ed9ffd6a !3183 Remove files on test fail for test_minddataset*.py 
Merge pull request !3183 from tony_liu2/master
 2020-07-24 23:47:29 +08:00
xulei2020 0af6d75761 add code 2020-07-24 19:51:27 +08:00
qianlong d28c63b6c0 fix diff workers disallow cache 2020-07-24 15:30:00 +08:00
mindspore-ci-bot 21a5f06e93 !3328 Add testcases 
Merge pull request !3328 from shenwei41/sw_master
 2020-07-23 09:57:22 +08:00
tony_liu2 1f4251a440 add try to minddataset python tests to remove files on fail 
delete unrelated file

change finally to except else

fix spacing alignment

fix indentation

fix indentation

fix indentation

add fix to new test case

add if main for pytest

fix spacing
 2020-07-22 16:34:10 -04:00
mindspore-ci-bot 859fe6bc41 !3342 C++ API Support for Skip Dataset Op plus UTs 
Merge pull request !3342 from cathwong/ckw_c_api_skip
 2020-07-23 04:28:38 +08:00
mindspore-ci-bot df1300d9cb !3294 Refactor opt/pre passes 
Merge pull request !3294 from nsyca/removal_pass
 2020-07-23 02:51:31 +08:00
Cathy Wong 8921a6099c C++ API Support for Skip Dataset Op and UTs 2020-07-22 13:21:35 -04:00
mindspore-ci-bot c24122bb55 !3303 fix validator that checks the vocab_size is positive in vocab.from_file 
Merge pull request !3303 from ZiruiWu/fix_vocab_validator
 2020-07-23 00:49:29 +08:00
Nat Sutyanyong 9fb1904ed8 Refactoring opt/pre 2020-07-22 11:13:42 -04:00
shenwei41 4eaa396ca7 Add testcases 2020-07-22 20:21:19 +08:00
Amir Lashkari 6f186aafa0 Update Python UA op, AutoContrast Op and UT 
Updated Validators.py

Fixed UA UT

Fixed UA UT

Fixed PyLint Errors

Added aditional changes

remove old golden AutoContrast UT file

updated Autocontrast python UT

Added golden files

Fixed issue

fixed UT
 2020-07-22 14:16:16 +08:00
Zirui Wu 40ea84ae81 fix from_file validator minor error, 1 line change 
add test case
 2020-07-21 23:17:26 -04:00
mindspore-ci-bot c451146b14 !3255 Added cpp Equalize op to vision api 
Merge pull request !3255 from alashkari/cpp_ops
 2020-07-21 21:56:41 +08:00
panfengfeng 5aad4bdfbf change size 2020-07-21 14:56:55 +08:00
xiefangqi 64a6ed56de fix annotation 2020-07-21 12:04:56 +08:00
mindspore-ci-bot ad651f38bf !3124 [MD] error occur when using numpy types 
Merge pull request !3124 from liyong126/fix_numpy_generic
 2020-07-21 11:44:34 +08:00
mindspore-ci-bot bfc18704d5 !3254 [MD]Add more strict upper bounds to CenterCropOp and RandomCropWithBboxOp 
Merge pull request !3254 from nhussain/padding_upperbound
 2020-07-21 10:18:56 +08:00
mindspore-ci-bot 0e27dccbcf !3093 VOCDataset output change to multi-columns 
Merge pull request !3093 from xiefangqi/md_voc_multi_columns
 2020-07-21 09:12:01 +08:00
liyong 28ebd73096 fix bug when int or float is numpy type 2020-07-21 09:07:29 +08:00
Amir Lashkari 572750cd40 Added cpp Equalize op 
adding missed files

Fixed Errors

fixed bugs

fixed bugs

fixed bugs

fixed bugs

fixed bugs

Updated Python UT for Equalize op

Added cpp Equalize op

adding missed files

Fixed Errors

fixed bugs

fixed bugs

fixed bugs

fixed bugs

fixed bugs

Updated Python UT for Equalize op

Fixed comment style

update files

deleted files

added files

added image_utils.h

Fixed PyLint and CPPLint Errors

updated tensor_op.h
 2020-07-20 21:03:26 -04:00
nhussain b064cbab5d fix upper bound 2020-07-20 15:26:49 -04:00
mindspore-ci-bot b091f74c23 !3016 Add CSV dataset loader 
Merge pull request !3016 from jiangzhiwen/dataset/csv
 2020-07-20 22:14:52 +08:00
jiangzhiwen 2f506b7985 CSV dataset loader 2020-07-20 20:22:08 +08:00
mindspore-ci-bot fa7fa8a162 !3234 [MD] fix bug when save tfrecord data 
Merge pull request !3234 from liyong126/fix_save_tfrecod
 2020-07-20 20:18:12 +08:00
liyong f521532a06 fix field_name probelem from tfrecord to mindrecord 2020-07-20 17:12:21 +08:00
xulei2020 18b519ae0f add sentence piece 2020-07-20 15:50:35 +08:00
xiefangqi fbd9e3d402 vocdataset output multi-columns 2020-07-20 14:40:30 +08:00
anzhengqi 008b91b2a1 inject epoch ctrl op in the execution tree and send eos at the end of epoch 2020-07-20 13:02:47 +08:00
mindspore-ci-bot 4945d34a41 !3133 [MD] fix num_sample in distributed sampler 
Merge pull request !3133 from liyong126/md_num_of_samples
 2020-07-18 15:36:33 +08:00
mindspore-ci-bot a84092e89d !3102 [MD] Add additional parameter checks in RandomCropWithBBoxOp and RandomResizeWithBbox 
Merge pull request !3102 from nhussain/random_crop_fixes
 2020-07-17 22:33:50 +08:00
liyong e2ea1fa0df activate num_samples in distributed samplers 2020-07-17 16:55:57 +08:00
mindspore-ci-bot 1bcbbb08d6 !2973 [MD] Add implementation of SlidingWindow TensorOp, c++ & python 
Merge pull request !2973 from luoyang/update
 2020-07-17 11:06:35 +08:00
mindspore-ci-bot 1a518e1cff !3144 [MD] Fix ResizeOp behaviour 
Merge pull request !3144 from nhussain/dataset_check
 2020-07-17 06:04:14 +08:00
nhussain c984b0fc88 fix size tuple 2020-07-16 17:02:34 -04:00
mindspore-ci-bot 35c0f1d2e6 !3143 Add python UT to MnistDataset op 
Merge pull request !3143 from TinaMengtingZhang/dataset-op-ut-mnist
 2020-07-17 04:40:26 +08:00
tinazhang ab37e87d7a adding Mnist python ut coverage 2020-07-16 14:14:20 -04:00
islam_amin 5647889c0d Added AutoContrast Op 2020-07-16 13:09:52 -04:00
mindspore-ci-bot 37e8439c0a !3077 supporting cpp invert operation 
Merge pull request !3077 from ava/invert
 2020-07-17 01:04:43 +08:00
mindspore-ci-bot c07a8d2033 !3137 Cleanup dataset UT: Remove deprecated save_and_check 
Merge pull request !3137 from cathwong/ckw_dataset_ut_cleanup7
 2020-07-17 00:45:56 +08:00
nhussain 5871ba8a05 more validator fixes 2020-07-16 10:10:36 -04:00
liyong bc676fe250 save op in minddataset 2020-07-16 15:10:48 +08:00
avakh 35c3a63701 support cpp invert operation 2020-07-15 23:45:45 -04:00
Cathy Wong 04eb4f89c7 Cleanup dataset UT: Replace save_and_check 2020-07-15 20:47:31 -04:00
Zirui Wu d233c54139 implemented cpp random choice, apply and compos 
python part of random ops

added random select sub policy

validators added

comments added, remaining issues addressed

add more python test cases

fix ci

fix CI

fix order of include files

addr review cmts

addr review cmts

reorg file

fix compile err

address review cmts

address review cmts
 2020-07-15 13:39:10 -04:00
YangLuo 4136892a3e add SlidingWindow Op 2020-07-15 19:55:33 +08:00
mindspore-ci-bot bc0a53cfb1 !3030 minddata support ps-lite feature 
Merge pull request !3030 from xiefangqi/md_support-pslite
 2020-07-15 09:11:37 +08:00
mindspore-ci-bot ba0143402c !3067 Cleanup dataset UT: Remove unneeded tf data files and tests 
Merge pull request !3067 from cathwong/ckw_dataset_ut_cleanup6
 2020-07-15 04:44:52 +08:00
mindspore-ci-bot 219a716eae !3066 fix some batch's get_dataset_size and some text validator inconsistency 
Merge pull request !3066 from ZiruiWu/fix_validator
 2020-07-15 03:11:31 +08:00
mindspore-ci-bot bed93a9ead !3039 [Dataset Python UT] adding coverage to Cifar Op and update test dataset 
Merge pull request !3039 from TinaMengtingZhang/dataset-op-ut-cifar
 2020-07-15 02:10:52 +08:00
Zirui Wu 63185cb20f fix some validators errors 
address review cmts

addr review cmts
 2020-07-14 14:04:40 -04:00
Cathy Wong ae1ed327ba Cleanup dataset UT: Remove unneeded data files and tests 2020-07-14 12:24:54 -04:00
nhussain 05b2a57d2a fix validation errors, and fix try catch error tests 2020-07-14 11:50:00 -04:00
tinazhang 340d98a4d1 added test case to cifar_op 
update cifar10 dataset
fixing missing error handling code in validator
 2020-07-14 11:22:38 -04:00
xiefangqi add19a591c md support ps-lite 2020-07-14 21:46:11 +08:00
mindspore-ci-bot 48654be14d !3040 BoundingBoxAugment op fix ratio 
Merge pull request !3040 from islam_amin/bounding_box_augment_bugs
 2020-07-14 09:36:34 +08:00
islam_amin 56da3b0ae1 Fixing ratio bug with BoundingBoxAugment 2020-07-13 18:11:19 -04:00
Jesse Lee 7c1bc5192b Initial Drop of CacheOp Phase I 2020-07-13 10:18:53 -04:00
mindspore-ci-bot 6284c42a76 !2941 MD tokenizer support output offsets 
Merge pull request !2941 from xiefangqi/md_add_offsets_feature
 2020-07-11 14:36:00 +08:00
xiefangqi 47060631e5 add offsets feature to tokenizer 2020-07-11 11:20:02 +08:00
mindspore-ci-bot 541456044d !2934 GraphData support random_walk and get_edge_feature 
Merge pull request !2934 from heleiwang/gnn_july
 2020-07-11 10:26:17 +08:00
mindspore-ci-bot 2f565f4c20 !2776 Update API of dataset.text.lookup 
Merge pull request !2776 from ZiruiWu/master
 2020-07-10 22:18:48 +08:00
heleiwang 7a046a1d70 support get_edge_feature 2020-07-10 17:05:23 +08:00
Zirui Wu 7b15e5a742 rework on lookup 
add test caser

fix ci

address review cmts

ci

addr review cmt

fix typo

address review cmts

add 2 more test cases

cpplint fix

addr cpplint

addr ci

fix tst case err

fix doc str
 2020-07-09 17:10:18 -04:00
nhussain 2c7fd248f8 fixes for PR-2908: avoid empty strings for column names 2020-07-09 11:44:33 -04:00
nhussain 6c37ea3be0 fix validators 
fixed random_apply tests

fix validators

fixed random_apply tests

fix engine validation
 2020-07-08 16:40:33 -04:00
mindspore-ci-bot 915ddd25dd !2892 gnn random walk pr 1977 comment follow up 
Merge pull request !2892 from tony_liu2/master
 2020-07-09 03:35:55 +08:00
avakh 421b80bf8f updating python unit tests based on floating point bounding boxes + adding coco dataset in unit tests 2020-07-07 18:05:49 -04:00
tony_liu2 14899a1410 fix gnn random walk pr 1977 comments 
add fix to random resize decode crop test case

fix pylint issues
 2020-07-07 17:32:08 -04:00
mindspore-ci-bot 9ba937b1af !2920 Updating 3 BBox AugOps for Float + Common Files for BBox Float Support - (VertFlip/RandomCrop/RandomResizeCrop) 
Merge pull request !2920 from danishnxt/AugOps-M1-Python
 2020-07-08 05:18:58 +08:00
mindspore-ci-bot c9929fd8a1 !2900 Update BoundingBoxAugment and RandomHorizontalFlipWithBBox to use float 
Merge pull request !2900 from islam_amin/object_ops_cc_float
 2020-07-08 03:53:39 +08:00
mindspore-ci-bot 92d2011b8b !2924 [Dataset Python UT] Disable md5 random perspective test 
Merge pull request !2924 from TinaMengtingZhang/redo-skip-md5-random-perspective-test
 2020-07-08 03:52:21 +08:00
Danish Farid 78c370f72a First Commit - New Infrastructure - Python UT tests + Common Aug Files 
fix accidental change overwrite

fix 2

updated inits from 0 to 0.0 for float

improvd python ut comments

updated macro
 2020-07-07 14:50:34 -04:00
tinazhang 8455f957ac disable md5 random perspective test case 2020-07-07 14:21:06 -04:00
islam_amin dfc3409f67 Update RandomHorizontalFlipWithBBox and BoundingBouxAugment C++ Ops to use floats 2020-07-06 23:49:21 -04:00
xiefangqi 62d55fdbce vocdataset support float bbox 2020-07-07 10:07:42 +08:00
mindspore-ci-bot b5066e81ee !2308 Unskip md5 python UT case for RandomPerspective 
Merge pull request !2308 from TinaMengtingZhang/unskip-case-random-perspective
 2020-07-04 03:37:58 +08:00
tinazhang dc03019200 unskip md5 testcase for RandomPerspective 2020-07-03 13:10:13 -04:00
Danish Farid 5a9673ee6f fix_annotate bug fix 2020-07-03 11:17:13 -04:00
peilin-wang 29aa589972 added check for invalid type for boolean args 2020-07-02 16:49:09 -04:00
yanghaitao 365c901ee0 get default value if num_parallel_worker is None 2020-06-30 21:10:37 +08:00
mindspore-ci-bot d08a89ab87 !2737 makes 0 an invaild bucket size 
Merge pull request !2737 from Peilin/zero-input-bucket-check
 2020-06-30 05:33:41 +08:00
mindspore-ci-bot 1227719cdd !2603 [Dataset Python UT] Cleanup mse and md5 validation 
Merge pull request !2603 from TinaMengtingZhang/I1LTCA-mse-check-cleanup
 2020-06-30 04:46:49 +08:00
peilin-wang 9468b49e28 bucket_batch_sizes must be strictly positive, 0 is not a valid batch size 2020-06-29 16:34:39 -04:00
mindspore-ci-bot c70e6987ba !2624 Fix config test 
Merge pull request !2624 from EricZ/fix_config_test
 2020-06-30 01:11:12 +08:00
mindspore-ci-bot 9b4399480a !2321 Connector throughput performance metric 
Merge pull request !2321 from Alexey_Shevlyakov/connector_throughput_dev
 2020-06-29 23:54:42 +08:00
tinazhang66 5cd3136355 remove local defined mse and add missing mse/md5 validation 2020-06-29 11:51:07 -04:00
ms_yan 7fa0d9e7e4 add paramter check for numpyslices and num_shards 2020-06-29 19:12:52 +08:00
mindspore-ci-bot 32a2d5d21c !2629 get monitor sampling interval from json file 
Merge pull request !2629 from yanghaitao/yht_sampling_interval
 2020-06-28 20:05:05 +08:00
mindspore-ci-bot 363489d00f !2580 BasicTokenizer do not case fold on preserved words 
Merge pull request !2580 from qianlong21st/fix_basic_tokenizer
 2020-06-28 19:36:28 +08:00
qianlong cae77c0c22 BasicTokenizer not case fold on preserverd words 2020-06-28 16:28:00 +08:00
liyong 68e2097897 fix split erroer message 2020-06-28 15:47:10 +08:00
yanghaitao fd701b8504 set monitor sampling interval through json file 2020-06-28 15:18:21 +08:00
Eric a4f5802924 Added Fix for test config 2020-06-27 15:20:48 -04:00
Alexey Shevlyakov 698abf75fc Connector throughput + PerfData + CyclicArray 2020-06-26 14:45:01 -04:00
mindspore-ci-bot a1b17bd2e2 !2593 python unit tests for randomResizeWithBBox and ResizeWithBBox 
Merge pull request !2593 from ava/python_ut_tests
 2020-06-27 02:31:31 +08:00
avakh 1e869146e9 applying comments removing VOC 2020-06-26 12:39:48 -04:00
Cathy Wong 277aba5326 dataset: Fixup docs; remove pylint disabled messages in UT 2020-06-26 12:26:41 -04:00
mindspore-ci-bot e11c953225 !2587 Implementing Python UTs for BoundingBoxAugment and RandomHorizontalFlipWithBBox 
Merge pull request !2587 from imaaamin/object_ops_ut
 2020-06-26 00:43:31 +08:00
mindspore-ci-bot 067592ccf9 !2590 Updated Python UTs for RandomVerticalFlipWithBBox, RandomCropWithBBox and RandomResizedCropWithBBox 
Merge pull request !2590 from danishnxt/AugOps2
 2020-06-25 23:40:34 +08:00
islam_amin 0f58f0338e updating ut for RandomHorizontalFlipWithBBox and BBoxAugment 2020-06-25 11:35:22 -04:00
mindspore-ci-bot b0a10c26a4 !2532 adding random-resize-with-bbox-op and resize-with-bbox-op 
Merge pull request !2532 from ava/NNNewMaster
 2020-06-25 20:42:00 +08:00
Danish Farid 3d1ecaaeb5 updated UT test for Python (3) AugOps with BBox - MD5 checks + imrpv comments 2020-06-25 03:19:14 -04:00
avakh 587e260236 addressing comments 2020-06-24 18:54:47 -04:00
Jamie Nisbet c22eac743d subtree creation in python apis 
updates

fix

fix cpplint

fix
 2020-06-24 17:17:05 -04:00
Danish Farid a8f392ca42 updated test util file + new BoundingBoxCheck + fixed VOCDataset annotations 
Style Error fix

fixed PyLint problem

reverting testVOC2012 due to CI break for existing test

reverting testVOC2012 due to CI break for existing test-2

updated old error messages to confirm with global standard

addressing PR 2355 Comments - 1

addressing PR 2355 Comments - 2

addressing PR 2355 Comments - 3
 2020-06-24 12:41:41 -04:00
YangLuo 4e3bfcf4c9 !2306 [Dataset] Code review & improve quality 2020-06-22 14:01:22 +08:00
mindspore-ci-bot beb436f457 !2398 Bug in Slice when multiple rows are used 
Merge pull request !2398 from h.farahat/slice_bug
 2020-06-22 09:23:44 +08:00
mindspore-ci-bot a9c309da4c !2404 [Dataset]rename input variable name to children and output variable to parent 
Merge pull request !2404 from xulei/input_output
 2020-06-22 09:22:33 +08:00
mindspore-ci-bot 1ea38eb60c !2375 Add Python Tokenizer 
Merge pull request !2375 from h.farahat/python_tokenizer
 2020-06-22 09:20:33 +08:00
liyong 2412ee09ce fix distributedSampler reshuffle and fix random_device failed 2020-06-21 18:23:47 +08:00
mindspore-ci-bot 78a8bc302d !2380 Fix CocoDataset issue 
Merge pull request !2380 from xiefangqi/xfq_fix_coco_issue_01
 2020-06-20 18:41:07 +08:00
xulei2020 3b71bd0d5d rename input to children, output to parent 2020-06-20 16:40:10 +08:00
hesham 68030e6a4b Slice Bug 2020-06-20 00:47:13 -04:00
qianlong 980ddd32a2 change output of WordpieceTokenizer and BertTokenizer to 1-D string tensors 2020-06-20 12:08:43 +08:00
xiefangqi 5703a10b8b fix CocoDataset issue 2020-06-20 09:51:10 +08:00
mindspore-ci-bot bbf69912be !2363 fix TextFildDataset and CLUEDataset does not support to_device 
Merge pull request !2363 from yanghaitao/yht_fix_textfiledataset
 2020-06-20 09:47:01 +08:00
hesham e981c67acd Python Tokenizer 
!38 Synchronize with latest Ascend software suite 17 Jun 2020
Merge pull request !38 from yanghaoran/master
 2020-06-19 19:57:11 -04:00
hesham 674415f7be Cleanup work for Concate, Mask, Slice, PadEnd and TruncatePair 2020-06-19 19:11:50 -04:00
mindspore-ci-bot bc4b1c2460 !2373 Comments + improved casting/parameters + UT tests for 3 AugOps (Augmentation Ops RandomCropWithBBox, Random Vertical FlipWithWithBBox and Random CropAndResizeWithWithBBox) 
Merge pull request !2373 from danishnxt/AugOpsD1
 2020-06-20 05:19:03 +08:00
mindspore-ci-bot 7bdcc319b2 !2325 Addressing comments from PR2314 
Merge pull request !2325 from imaaamin/object_ops_pr2
 2020-06-20 04:50:56 +08:00
Danish Farid 6442a85b75 comments added, ptrs to consts, UT tests for BBOps 
updated comments for 3 BB AugOps

addressing comments, void -> status, macros added, fixed test cmt

changed types -> value, fixed ut test funcs, grammar

bug type fix

fixed condition in tests for viz
 2020-06-19 16:09:16 -04:00
islam_amin edc42c5b85 addressed comments from reviews 2020-06-19 14:53:46 -04:00
nhussain d89101b95f add missing test 2020-06-19 11:44:41 -04:00
yanghaitao 9f336bb735 fix TextFileDataset and CLUEDataset failed with to_device 2020-06-19 17:28:05 +08:00
peilinwang 1e36b0649f remove graphengine changes 
remove graphengine changes

concat op

Truncate Pair

concat_op

remove graph engine changes

ToNumberOp implementation almost done

ToNumberOp complete

ci fix

ci fix

ci fix

ci fix

ci fix

ci fix

ci fix

ci fix

ci fix

ci fix

merge conflicts
 2020-06-18 20:07:35 -04:00
hesham 11826fb256 DuplicateOp 2020-06-18 19:47:27 -04:00
mindspore-ci-bot 90bb9320aa !2314 Implemented RandomHorizontalFlipWithBBox and BoundingBoxAugment C++ Ops 
Merge pull request !2314 from imaaamin/object_ops_pr
 2020-06-19 07:08:42 +08:00
mindspore-ci-bot a83baafbf6 !2253 Add ConcatOp to Dataset 
Merge pull request !2253 from nhussain/concat_op
 2020-06-19 06:32:15 +08:00
mindspore-ci-bot 3784220056 !2317 rework on dataset.text.vocab to support any user special_tokens 
Merge pull request !2317 from ZiruiWu/vocab_rework
 2020-06-19 05:25:31 +08:00
nhussain 5515016dba remove graphengine changes 
remove graphengine changes

concat op

Truncate Pair

concat_op

remove graph engine changes
 2020-06-18 17:17:50 -04:00
mindspore-ci-bot ffc8a3c362 !2274 add PadEndOp 
Merge pull request !2274 from xunxue/padend
 2020-06-19 05:05:14 +08:00
mindspore-ci-bot e4322baece !2316 [Dataset] Move python tree transformation into cpp pass 
Merge pull request !2316 from JunhanHu/alter_node_cpp
 2020-06-19 05:02:03 +08:00
islam_amin 0af156d169 added two ops for object detection, random horizontal flip with bbox and bouding box augment. along with their unit tests 2020-06-18 16:55:31 -04:00
xunxue 87aa9c8f7a PadEndOp 2020-06-18 16:07:34 -04:00
Junhan Hu 57f3732ac3 Move python tree transformation into cpp pass. 2020-06-18 15:52:38 -04:00
Zirui Wu b6e9504b31 phase I of Vocab rework 
phase II vocab rework

added more test cases

fix api doc string

address review cmts and fix CI

address ci complains

fix review cmts

ci
 2020-06-18 15:36:28 -04:00
Zirui Wu 8f2674850b address API doc style and content 
addr review cmts

revert

ci

private class

ci

fix ci

address ci

address review cmts

fix style err
 2020-06-18 12:23:42 -04:00
mindspore-ci-bot 5850b99161 !2211 Add TruncatePair Op to dataset 
Merge pull request !2211 from h.farahat/pair_truncate
 2020-06-18 20:58:40 +08:00
mindspore-ci-bot d0c5071cab !1993 [Dataset] Fix codedex. 
Merge pull request !1993 from luoyang/pylint
 2020-06-18 18:59:31 +08:00
ms_yan 8d1dae46ac Throw error when load config failed 2020-06-18 13:47:32 +08:00
hesham b9495a9ccc Truncate Pair 2020-06-17 23:11:27 -04:00
mindspore-ci-bot ea829e89c3 !2198 Add MaskOp to dataset 
Merge pull request !2198 from h.farahat/mask_op
 2020-06-18 02:50:16 +08:00
mindspore-ci-bot 808380a9f9 !2142 Cleanup dataset UT: refactor visualize() 
Merge pull request !2142 from Tinazhang/reorg-visualize
 2020-06-18 01:30:36 +08:00
ms_yan fdafc690f8 repair problem in NumpySlicesDataset 2020-06-17 21:49:02 +08:00
Tinazhang 169eaf73c7 reorganize util function visualize() 2020-06-17 09:30:05 -04:00
Yang dee8471d23 !1993 [Dataset] Fix codedex. 2020-06-17 16:25:38 +08:00
qianlong 4f16f036be Add WhitespaceTokenizer and UnicodeScriptTokenizer for nlp 
add CaseFold, NormalizeUTF8

add RegexReplace

add RegexTokenizer

add BasicTokenizer

add WordpieceTokenizer

add BertTokenizer
 2020-06-17 15:47:04 +08:00
hesham f2462bb00d Mask Op 2020-06-16 23:51:46 -04:00
mindspore-ci-bot ca94916ef9 !2184 CocoDataset add category_id check 
Merge pull request !2184 from xiefangqi/xfq_fix_coco_issue
 2020-06-17 10:06:41 +08:00
mindspore-ci-bot 625f2421b5 !1977 random walk in gnn node2vec 
Merge pull request !1977 from JonathanY/randomwalk
 2020-06-17 03:22:21 +08:00
mindspore-ci-bot 3adbde56b2 !2162 SliceOp 
Merge pull request !2162 from h.farahat/slice_op
 2020-06-17 01:22:47 +08:00
mindspore-ci-bot 898b2fdecd !2187 Fix comment display issues in BuildVocabDataset 
Merge pull request !2187 from ZiruiWu/vocab_rework
 2020-06-17 00:51:37 +08:00
Zirui Wu 2794883644 fix selected minor issues 
fix review comments
 2020-06-16 11:07:49 -04:00
xiefangqi aab190fcdb CocoDataset add category_id validation 2020-06-16 21:37:22 +08:00
liyong d4f8f57c7e add split in minddataset 2020-06-16 19:50:41 +08:00
hesham e2012a1de9 Slice Op 2020-06-16 02:32:40 -04:00
mindspore-ci-bot dceb3c2be4 !2026 Change C++ behavior in RandomCropAndResize 
Merge pull request !2026 from MahdiRahmaniHanzaki/I1K3FL-random-resized-crop
 2020-06-16 05:27:36 +08:00
Mahdi add29f95f5 resolved issues in the comments 2020-06-15 16:45:18 -04:00
mindspore-ci-bot 2cab44a637 !2028 FillOp for BERT 
Merge pull request !2028 from nhussain/fillOp_learning
 2020-06-16 02:45:45 +08:00
nhussain dd9bf09f0a added FillOp for #119 - special Ops 2020-06-15 11:30:43 -04:00
Peilin Wang 848e07d022 initial commit, start of BucketBatchByLengthOp 
c implementation done, just need to call batch/pad

added python api and validator

added pybind/de_pipeline stuff, fixed some compile errors, figure out how null py::function works

added tiny bit of doc

integrated with static batch methods

fixed some bugs

some more bug fixes and cleanup

ci fix

fix ci

ci fix

fix ci

added test_cases and debugged

addressed code review comments

addressed code review comments

ci fix

ci fix

addressed code review comments

addressed code review comments
 2020-06-15 10:20:01 -04:00
Jonathan Yan 87d2c27c7f random walk v1 2020-06-15 08:54:10 -04:00
mindspore-ci-bot a8cf83acf2 !1932 Add CLUE dataset 
Merge pull request !1932 from jiangzhiwen/dataset/clue
 2020-06-15 11:48:35 +08:00
jiangzhiwen e0e167a000 add CLUE dataset 2020-06-15 10:40:46 +08:00
hesham a22763b612 Bug in CIFAR after removing GetMutabble buffer 
Fixes # I1KIPC
 2020-06-14 13:27:17 -04:00
jonyguo bb2082daec fix: MindDataset padded log error 2020-06-13 19:46:24 +08:00
mindspore-ci-bot 3085e51e45 !1922 check parameter num_samples of sampler 
Merge pull request !1922 from yanghaitao/yht_check_num_samples_sampler
 2020-06-12 10:22:14 +08:00
mindspore-ci-bot 1b5d9a1194 !1981 Implemented from_dataset, a way to build a vocab from existing dataset 
Merge pull request !1981 from ZiruiWu/from_dataset_dev
 2020-06-12 08:08:19 +08:00
Zirui Wu 880ce5ea26 implemented from_dataset 
fix complie error

more tests

address CI complains

fix ci

adress review comments

address review cmts
 2020-06-11 16:20:19 -04:00
Alexey Shevlyakov 4e333a2f22 MindData profiler infrastructure 2020-06-11 14:52:36 -04:00
mindspore-ci-bot 0a95223f25 !1910 RandomColorAdjust error for grayscale images 
Merge pull request !1910 from MahdiRahmaniHanzaki/I1J9SQ-random-color-adjust-bug
 2020-06-11 23:15:19 +08:00
yanghaitao 8f16cff12e add para check for sampler 2020-06-11 22:26:06 +08:00
jonyguo 1de7271afc add floatxx test case 2020-06-11 19:26:54 +08:00
ms_yan bc22c172b8 add TensorDataset and its ut 2020-06-11 15:24:12 +08:00
mindspore-ci-bot 251a6667a5 !1989 fix MindDataset distribute shuffle error 
Merge pull request !1989 from guozhijian/fix_distribute_shuffle
 2020-06-11 14:51:17 +08:00
mindspore-ci-bot b3da41bd7a !1913 CocoDataset implementation 
Merge pull request !1913 from xiefangqi/xfq_support_coco
 2020-06-11 14:37:44 +08:00
xiefangqi 097cbcc0cb CocoDataset implementation 2020-06-11 11:52:17 +08:00
jonyguo 96859f44b6 fix: MindDataset distribute shuffle bug 2020-06-11 11:06:34 +08:00
Mahdi ba2e2dced4 fixed some warnings and typos 2020-06-10 17:18:56 -04:00
Tinazhang 2812cd57db fix pylint error 2020-06-10 15:55:11 -04:00
mindspore-ci-bot fe0c3e9a8f !1908 Add UT to six data vision transforms ops 
Merge pull request !1908 from Tinazhang/ut-normalize
 2020-06-11 00:43:57 +08:00
heleiwang 3ece8dd090 1. support get_all_edges, get_nodes_from_edge, get_sampled_neighbors, get_neg_sampled_neighbors and graph_info API 
2. mod cora and citeseer conversion script
 2020-06-10 15:52:26 +08:00
mindspore-ci-bot 5eb95599f6 !1874 Updates to string tensor 
Merge pull request !1874 from h.farahat/string_tensor_updates
 2020-06-10 00:59:17 +08:00
Tinazhang c3de5c7a54 adding new TCs to six ops and skip md5 case in RandomPerspective 2020-06-09 10:51:45 -04:00
hesham f837ddc956 - Bug when empty strings sent to Python 
- Support accepting Numpy of str as input
- Support batching strings
- Core logic of batch&pad is static
- Make Pad a utility function
 2020-06-09 10:34:38 -04:00
liyong feff8899ac support padding samples 2020-06-05 09:14:42 +08:00
mindspore-ci-bot 5c21616293 !1807 Implemented Ngram TensorOp for dataset 
Merge pull request !1807 from ZiruiWu/ngram_dev
 2020-06-05 01:04:27 +08:00
Zirui Wu dbf9936ec4 Implemented n-gram for dataset TensorOp 2020-06-04 09:57:44 -04:00
Jamie Nisbet 51bc0c0460 consistent design for num_samples 
updates

more update

more work

more fixin

post rebase updates

clang formatting

code review recovery

ci fixes

updates

update

update

update
 2020-06-03 15:01:18 -04:00
Tinazhang 33fe66fd1e fix error handling for bigger crop size case 2020-06-02 11:06:41 -04:00
Yang 5522edc4b5 !1713 [Dataset] Clean pylint. 2020-06-02 09:02:19 +08:00
liuwenhao4 7b911886ec Fixing some tiny faults about Pylint in my code(ops) 2020-05-30 16:55:37 +08:00
mindspore-ci-bot 7878743400 !1638 fixed rounding edge case in split 
Merge pull request !1638 from Peilin/splitOp-after-testing
 2020-05-30 05:31:33 +08:00
mindspore-ci-bot 0e3dd8149a !1703 Adding sync_wait input check 
Merge pull request !1703 from EricZ/master
 2020-05-30 05:29:38 +08:00
mindspore-ci-bot c366b3fb18 !1704 fix num rows bug in sampler 
Merge pull request !1704 from Peilin/fix-random-sampler-bug
 2020-05-30 05:27:20 +08:00
Eric 8e1b2107ce Added example for sync_wait bad batch_num 
Added broken test case for cv

Fixed error handling in sync_wait
 2020-05-29 16:30:12 -04:00
mindspore-ci-bot 976b4793d1 !1544 Python Unit Test for TenCrop 
Merge pull request !1544 from MahdiRahmaniHanzaki/I1IHNG-ten-crop
 2020-05-30 03:02:29 +08:00
Peilin Wang 3ef3d1433d fixed num_samples bug for RandomSampler 2020-05-29 13:09:26 -04:00
Tinazhang f4a61fdbb2 Adding new UT cases and missing TCs in PY Ops. 2020-05-29 12:12:17 -04:00
Peilin Wang 5469be2a97 fixed bug for split, RandomSampler and some other cleanup 
add another test case

typo

merge conflict

another PR changed testing behavior, updated test cases in this commit

added input check for use_sampler

addressed code review comments

fixed pylint, not related to my changes

fixed edge case of rounding in getting split sizes

fix pylint
 2020-05-29 11:27:32 -04:00
Mahdi dfc097019b Added TenCrop test 
Added unit tests for both testing the functinality of the TenCrop and
its error messages. Due to the similarity of this method to FiveCrop the
test cases are similar to FiveCrop test cases.
Signed-off-by: Mahdi <mahdi.rahmani.hanzaki@huawei.com>

added error_msg function call in the main method

refactored the test and added visual representation of the results

Separated the two error cases into two different functions and used the
visualize function in util.py to plot the result of TenCrop.
Signed-off-by: Mahdi <mahdi.rahmani.hanzaki@huawei.com>

Added new test cases

Added new test cases including test case for checking the error message
when the size variable is not a positive integer, test case for
rectangle crop, test case for vertical flip setting, and testing for
similarity of the result of TenCrop for the same input data in different runs.
Signed-off-by: Mahdi <mahdi.rahmani.hanzaki@huawei.com>

changed visualize in test_five_crop

Changed the visualize function in test_five_crop to use the already
existing function in util.py
Signed-off-by: Mahdi <mahdi.rahmani.hanzaki@huawei.com>

made generate_golden variable global
 2020-05-28 17:52:56 -04:00
mindspore-ci-bot 21da86b393 !1591 add get_dataset_size for CelebADataset 
Merge pull request !1591 from yanghaitao/yht_celeba_get_dataset_size
 2020-05-28 22:21:43 +08:00
yanghaitao cc6c7a3f60 add get_dataset_size to celebadataset 2020-05-28 17:00:14 +08:00
mindspore-ci-bot 0f4dc13e8e !1478 [Dataset] Fix CI warnings, clean pylint. 
Merge pull request !1478 from luoyang/pylint
 2020-05-28 15:54:52 +08:00
Yang 9b2a778d94 !1478 [Dataset] clean pylint. 2020-05-28 14:06:36 +08:00
xiefangqi 5e4728c50f add voc support split 2020-05-28 11:18:32 +08:00
Peilin Wang d4c93575e7 fixed bug for split, RandomSampler and some other cleanup 
add another test case

typo

merge conflict

another PR changed testing behavior, updated test cases in this commit

added input check for use_sampler

addressed code review comments

fixed pylint, not related to my changes
 2020-05-26 16:29:52 -04:00
Yang abca62f407 10:00 26/5 clean pylint 2020-05-26 10:05:12 +08:00
mindspore-ci-bot 0a8ef2fe72 !1455 Cleanup dataset UT: restore config support 
Merge pull request !1455 from cathwong/ckw_dataset_config_util
 2020-05-26 05:40:39 +08:00
Cathy Wong f891e1755c Cleanup dataset UT: restore config support 2020-05-25 15:10:03 -04:00
mindspore-ci-bot aeb4c52f2d !1379 Creating new UTs to three vision transformation ops 
Merge pull request !1379 from Tinazhang/linearTrans
 2020-05-25 23:10:30 +08:00
mindspore-ci-bot 718a96430f !1389 Security issues in recent string tensor PR 
Merge pull request !1389 from h.farahat/fix_security_issues
 2020-05-24 08:45:21 +08:00
mindspore-ci-bot a6b8451a33 !1380 make ShuffleOp have deterministic behavior for subsequent epochs 
Merge pull request !1380 from Peilin/shuffle-subsequent-epoch-deterministic
 2020-05-23 21:02:30 +08:00
heleiwang f28f883cac fix comments error and modify parameter check 2020-05-23 18:31:13 +08:00
hesham c0f0e154f5 Clean up work for text python package 2020-05-22 23:53:34 -04:00
mindspore-ci-bot 41456ac824 !1369 dataset: delete StorageDataset 
Merge pull request !1369 from ms_yan/del_storage
 2020-05-23 10:39:34 +08:00
Tinazhang 7322839b04 add UTs for LinearTransformation, ToPIL, ToType 2020-05-22 18:04:09 -04:00
Peilin Wang 0cbcc7200b made shuffle determinisitc for subsequent epochs 2020-05-22 17:03:19 -04:00
mindspore-ci-bot 61639be1e0 !1375 Cleanup dataset UT: remove num_parallel_workers=1 in test_exception 
Merge pull request !1375 from cathwong/ckw_dataset_ut_cleanup5
 2020-05-22 23:54:52 +08:00
mindspore-ci-bot a528797253 !1377 Code Fix for Uniform Augmentation 
Merge pull request !1377 from Tinazhang/cc
 2020-05-22 23:20:01 +08:00
mindspore-ci-bot 458436186c !1365 Clean up work for text python sub-package 
Merge pull request !1365 from h.farahat/text_namespace
 2020-05-22 23:16:31 +08:00
Tinazhang b390883c6a Bug fix 2020-05-22 10:56:56 -04:00
Cathy Wong 702005d403 Cleanup dataset UT: remove num_parallel_workers=1 in test_exception 2020-05-22 09:28:17 -04:00
ms_yan d5e896b51c delete storageDataset Op API and its test case 2020-05-22 21:23:24 +08:00
mindspore-ci-bot 6f733ec113 !1308 Stage 2 of adding support for string Tensor 
Merge pull request !1308 from h.farahat/string_tensor2
 2020-05-22 21:17:53 +08:00
xiefangqi 34236ce1f1 fix pylint 2020-05-22 20:18:21 +08:00
mindspore-ci-bot 3363d4e834 !1249 Add GNN dataset processing API 
Merge pull request !1249 from heleiwang/hlw_gnn_data
 2020-05-22 18:49:36 +08:00
hesham 6c21e556c4 Clean up work for text python package 2020-05-22 04:35:58 -04:00
heleiwang 599a449e0b Support processing GNN data 2020-05-22 14:15:25 +08:00
jinyaohui fbdba6e4da clean pylint 2020-05-22 13:23:13 +08:00
hesham df361d1d26 Change mem layout of string tensor 
add support for MindRecord and TFRecord
----
optimize tensorshape

optimize tensorshape and FlatIndex

TFRecord and MindRecord support for string tensor

Modify mem layout
Add new constructor
Add method Allocate

Change some GetMutableBuffer usages to AllocateBuffer
 2020-05-22 01:22:29 -04:00
mindspore-ci-bot 58e6d7d950 !1341 Added lookup and vocab to mindspore.dataset.text 
Merge pull request !1341 from ZiruiWu/vocab_and_lookup
 2020-05-22 10:19:42 +08:00
jonwe bb51bb88d7 add compress in mindrecord 2020-05-22 09:37:51 +08:00
mindspore-ci-bot 2e3d55ed87 !1281 Implementation of SplitOp 
Merge pull request !1281 from Peilin/splitOp
 2020-05-22 09:29:03 +08:00
mindspore-ci-bot 39b9aedf68 !1342 Bug fix on issue Core dump on GPU when train with lenet with AU 
Merge pull request !1342 from Tinazhang/cc
 2020-05-22 09:18:49 +08:00
Peilin Wang 71e8bb1960 general split case done, chaining sampler (basic case) is working 
implementation 99% complete

everything and tested except for repeatable shuffling

tested most basic/typical split usecases

cleanup

some more cleanup

fix CI

more ci fix

more ci fixes

more ci fix

more ci fix

more ci fix

added more tests, fixed some bugs

some more clean up and test cases

added shard/shuffle before split warning/error

addressed code review comments and ci

fixed ci
 2020-05-21 20:42:51 -04:00
Tinazhang e9e40b688b Bug fix 2020-05-21 18:20:00 -04:00
Zirui Wu 25ab2ef303 Implemented lookup and vocab 2020-05-21 17:17:24 -04:00
mindspore-ci-bot 46949fc327 !1307 Cleanup dataset UT: unskip and enhance TFRecord sharding tests 
Merge pull request !1307 from cathwong/ckw_dataset_ut_unskip2
 2020-05-22 03:21:45 +08:00
qianlong 451c20a6f5 Add UnicodeCharTokenizer for nlp 2020-05-21 09:22:45 +08:00
mindspore-ci-bot 93e7c97a96 !1272 [Dataset] MindData Tree Optimizer Infrastructure 
Merge pull request !1272 from JunhanHu/minddata_opt
 2020-05-21 05:29:00 +08:00
Cathy Wong b78894e02b Cleanup dataset UT: unskip and enhance TFRecord sharding tests 2020-05-20 17:05:38 -04:00
Junhan Hu f44d213503 MindData optimizer infrastructure. 2020-05-20 16:11:26 -04:00
xulei2020 163b6b7ea7 add jieba c++ code 2020-05-20 15:55:12 +08:00
Tinazhang 17cecf2cf5 Added TCs to RandomCrop and RandomCropAndResize and modified visalize() calling 2020-05-19 15:42:24 -04:00
jinyaohui 5a914994ba clean pylint 2020-05-18 16:42:35 +08:00
jinyaohui bcfaff97f9 clean pylint 2020-05-18 10:31:46 +08:00
hesham e8ca243364 -Add DE_STRING 
-replace switch'case by indexing

- Add test case
- Add constructors
- Add getItem string

- Fix bugs
- Add more tests

- Tensor iterator
- asNumpy
- TextFileDataset

- Tensor(Numpy)

- Super > 2D
- Add more test cases for GeneratorDataset

- Change StartAddr to GetBuffer and GetMutableNuffer

- Raise an error if batch is used with strings

Clean-up work
 2020-05-15 20:33:28 -04:00
jiangzhiwen cb2814b498 flat_map first commit 2020-05-15 17:45:39 +08:00
mindspore-ci-bot c680cfbf27 !1157 dataset: add concat operation for dataset 
Merge pull request !1157 from ms_yan/concat_dataset
 2020-05-15 16:07:19 +08:00
mindspore-ci-bot ab031ee9ea !1126 VOCDataset support object detection function 
Merge pull request !1126 from xiefangqi/voc_support_detection
 2020-05-15 15:56:39 +08:00
xiefangqi c937bad53f minddata support voc 2020-05-15 13:24:03 +08:00
ms_yan c0fa7b4b19 init commit of concat dataset 
change to use __add__ operation instead ds.concat
 2020-05-15 13:14:13 +08:00
jonyguo be2e7531ca fix: MindDataset parameter shard_id & num_shards check 2020-05-14 17:18:11 +08:00
Cathy Wong 913074e656 Cleanup dataset UT: resolve skipped test units 2020-05-13 14:41:57 -04:00
liyong aa3f89e74f mindrecord support read file list 2020-05-13 14:11:59 +08:00
Cathy Wong 49ef53f164 Cleanup dataset UT: util.py internals 2020-05-11 14:44:24 -04:00
mindspore-ci-bot 2860fd9338 !984 Add unit test case for HWC2CHW. 
Merge pull request !984 from Tinazhang/hwc2chw
 2020-05-09 05:02:41 +08:00
Tinazhang c8b5586c7f add unit test for HWC2CHWC 2020-05-08 13:17:20 -04:00
Cathy Wong 58226addd6 Cleanup dataset UT: use md5 npz in test_zip for images 2020-05-08 11:25:48 -04:00
mindspore-ci-bot 47f5abceb4 !960 Adding example for grayscale 
Merge pull request !960 from EricZ/grayscale_fix
 2020-05-08 05:08:15 +08:00
mindspore-ci-bot 078dd86cfe !507 Implemented padded_batch 
Merge pull request !507 from ZiruiWu/batch_with_padding
 2020-05-08 04:54:05 +08:00
mindspore-ci-bot de7625777f !951 fix: MindDataset with columns_name parameter cause errors in some scenes 
Merge pull request !951 from guozhijian/fix_read_by_columns
 2020-05-08 04:51:03 +08:00
eric 0f0548f21b Added test case for grayscale support 2020-05-07 15:09:57 -04:00
Zirui Wu c2d364a573 batch with padding implemented 
support for 1 specific dimension to be None, added validator

fix various CI complains

another round of CI fixes

ci

refactor parts of the code

code refactor

ci fix

comments added, fix bugs

address review comments

address review comments

review cmts

added simple perf test script

update pad code

perf imprv
 2020-05-07 11:18:42 -04:00
jonyguo d4d236bcce fix: use MindDataset by column_names get data error in some situation 2020-05-07 18:12:36 +08:00
liyong b520ca9087 fix pk sampler in mindrecord 2020-05-07 14:54:23 +08:00
Cathy Wong 772e6c1461 Cleanup dataset UT: test_batch, save_and_check support 2020-05-05 15:35:09 -04:00
eric 36fffb7706 Added example md5 generation 
Comparison example

Added md5 and comparison example for py_transforms

Added md5 check for images
 2020-05-04 21:15:33 -04:00
Junhan Hu 83c68ca2ef Skip pyfunc test case 2020-05-01 15:14:07 -04:00
eric 26cb3e8a5f Added test function to show that seed doesn't work. 
Added testcase to show that c image aug don't use seed properly

Added passing test cases

Added working testcases for using seed

Added additional test cases to show seed use

Added test case for seed
 2020-04-30 18:57:59 -04:00
ms_yan c56fe3aa2d modify take op with an operator 2020-04-30 10:16:36 +08:00
mindspore-ci-bot 8af10eb51e !875 Reject python OP in operations argument for C++ uniform augmentation OP 
Merge pull request !875 from AdelShafiei/ua_py
 2020-04-30 06:11:17 +08:00
Adel Shafiei d15bd04bfe added input validation to reject python op in C++ uniform augmentation operations list 2020-04-29 16:48:10 -04:00
mindspore-ci-bot a606c2e4da !872 [Dataset] Add schema support for GeneratorDataset 
Merge pull request !872 from JunhanHu/generator_schema
 2020-04-30 02:58:29 +08:00
mindspore-ci-bot 2303453753 !869 Random data op 
Merge pull request !869 from JesseKLee/random_data_op
 2020-04-30 02:56:39 +08:00
Junhan Hu c5a8ffe4f4 Add schema support for GeneratorDataset 2020-04-29 13:50:51 -04:00
Jesse Lee 5236d0c3c0 Replace print with logger.info 2020-04-29 12:44:04 -04:00
mindspore-ci-bot 8d3695f666 !672 Added UT for uniform augmentation C++ OP 
Merge pull request !672 from AdelShafiei/ua_ut
 2020-04-29 23:56:09 +08:00
Jesse Lee 270bf831a9 Random Data Op 2020-04-29 10:26:00 -04:00
jiangzhiwen 34bfa2f7c9 fix skip 2020-04-29 20:12:37 +08:00
Adel Shafiei 3322e65da9 added ut for uniform augment C++ op 2020-04-28 21:07:29 -04:00
mindspore-ci-bot b37db1edf5 !603 [MD] update pk sampler in minddataset 
Merge pull request !603 from liyong126/update_pk_sampler
 2020-04-27 23:17:38 +08:00
mindspore-ci-bot f82e63fecc !671 Added testcase for sync_wait 
Merge pull request !671 from EricZ/master
 2020-04-27 22:14:10 +08:00
mindspore-ci-bot 0e3054d527 !466 Deepcopy problem when pyfunc cannot be pickled 
Merge pull request !466 from h.farahat/deepcopy
 2020-04-27 22:07:55 +08:00
liyong bfba630aa2 update pK_sampler 2020-04-27 21:40:49 +08:00
Zirui Wu 8c3931cf1d fix first epoch always shuffle with default seed in random sampler 
fix ut
 2020-04-26 10:58:59 -04:00
eric 2d115cd04e Added example for multiple iterator 
Added new testcase for multi iterator

Addressing review

Fixed typo
 2020-04-24 17:28:40 -04:00
hesham a9e9266149 Deepcopy problem when pyfunc cannot be pickled 2020-04-22 11:55:27 -04:00
mindspore-ci-bot aad5771a62 !524 Added support for UA augmentation ops with tests 
Merge pull request !524 from alashkari/ua-ops
 2020-04-22 08:28:47 +08:00
Amir Lashkari 56e7a7deb5 Added UniformAugment + Python Augmentation Ops 2020-04-21 19:54:32 -04:00
mindspore-ci-bot dc0491caf9 !508 [Dataset] Adding sync_wait operator for dataset 
Merge pull request !508 from EricZ/master
 2020-04-22 06:15:24 +08:00
eric cd94518769 X# This is a combination of 2 commits. 
Initial commit for dataset op python

Added signature to barrier

Adde compiling barrier code

Rebasing, fixed new compile errors

Final fix for make_unique

Added pybind API for barrier

Fixed pyfunc invocation

python interface - sync_wait

!1 sync_wait python interface
* python interface - sync_wait

fix test

update test

update test

Added new test case

add test case

test for shuffle + batch

Added two-sync test case

Restrited that no shuffle after sync

Added sync to pipeline info

block first databuffer as well

Intelligently get batch size

Fix default case

Lock Pair shares among all iterators

Added fix for empty character

Fixed up test case formatting

Fix end of epoch in sync_wait

Fixing CI
 2020-04-21 17:42:13 -04:00
Junhan Hu 78001ac9e6 Add multiprocessing support for Mindspore.Dataset.GeneratorDataset 2020-04-21 16:03:43 -04:00
mindspore-ci-bot fb18671b28 !506 [Dataset] Multiprocessing support for Pyfunc 
Merge pull request !506 from JunhanHu/multiprocess_pyfunc
 2020-04-22 03:57:26 +08:00
Junhan Hu b13e7bc31a Add python multiprocessing support for Mindspore.dataset 2020-04-21 14:56:25 -04:00
qianlong db80f4ff92 The num_samples and numRows in schema for TFRecordDataset are conflict 2020-04-21 17:08:59 +08:00
mindspore-ci-bot d9e4dcc33b !483 Optimize skip dataset op 
Merge pull request !483 from jiangzhiwen/dataset/skip_opt
 2020-04-21 15:10:23 +08:00
liyong f1542a90a3 add pk sampler 2020-04-21 10:08:38 +08:00
jiangzhiwen e1b109e8b8 optimize skip dataset op 2020-04-21 09:16:37 +08:00
Cathy Wong 60df369100 Fixup py Normalize doc: takes input CHW 2020-04-20 15:17:30 -04:00
mindspore-ci-bot 6369cf27bd !406 added first row crc check for when reading tfrecord files 
Merge pull request !406 from Peilin/first-row-crc-check
 2020-04-21 00:10:18 +08:00
mindspore-ci-bot 98fbd30a5b !460 [Data]Add filter operation 
Merge pull request !460 from xulei/filter_master
 2020-04-20 23:33:44 +08:00
mindspore-ci-bot 822a3160e4 !404 [Dataset] Add Python Sampler support for CPP dataset 
Merge pull request !404 from JunhanHu/python_sampler_for_cpp_dataset
 2020-04-20 22:45:51 +08:00
xulei2020 c705ea5e5b add filterOp code 2020-04-20 22:32:19 +08:00
Peilin Wang 9bc2134cb7 added checking of first row crc to find invalid tfrecord files 
addressed code review comments. added check in python layer to exclude directories and to raise an error if a pattern does not match any file

fixed clang format

fixed cppcheck

fixed cppcheck (used std::accumulate and std::copy_if). regenerated tfrecord file to contain correct header, it was a dummy header before

fixed cppcheck: added const reference for string parameter for lambdas, fixed clang format: whitespace adjustments

more clang whitespace fixes...

changed print to logger.info
 2020-04-20 10:25:26 -04:00
yanghaitao 2795e492ff TextFileDataset 2020-04-20 11:18:55 +08:00
Junhan Hu 43a2e99833 Add python sampler support for CPP dataset 2020-04-19 20:36:37 -04:00
ms_yan f0c07c3fa6 Realize take op and add ut 2020-04-18 18:33:57 +08:00
mindspore-ci-bot 80333e9f55 !435 Fix dataset serialize and deserialize for MindDataset 
Merge pull request !435 from anthonyaje/fix_serdes_dataset
 2020-04-18 09:37:22 +08:00
mindspore-ci-bot 40f0a4a4f4 !333 Add skip op to Dataset 
Merge pull request !333 from jiangzhiwen/dataset/skip
 2020-04-18 09:19:07 +08:00
mindspore-ci-bot 9e1b5efd1d !434 Bug in cleaning dataset iterators 
Merge pull request !434 from h.farahat/multi_itr_bug
 2020-04-18 05:17:31 +08:00
anthonyaje ea297c0889 Fix dataset serdes for MindDataset 2020-04-17 13:24:02 -04:00
hesham 3c02c82771 Bug in weak reference. 
Add new test cases
 2020-04-17 11:36:32 -04:00
jzw 3f7054dccb add skip dataset op 2020-04-17 19:48:25 +08:00
mindspore-ci-bot cf026096a6 !183 Mindspore.dataset CPP sampler for GeneratorDataset 
Merge pull request !183 from JunhanHu/cpp_sampler
 2020-04-16 22:30:59 +08:00
Junhan Hu 9739d3b048 Add CPP sampler support for GeneratorDataset 2020-04-14 12:09:03 -04:00
mindspore-ci-bot 30de261c3c !243 Support nested repeat 
Merge pull request !243 from h.farahat/nested_repeat
 2020-04-14 23:07:32 +08:00
hesham 0fc23eee0f Support nested repeat 2020-04-14 10:13:18 -04:00
xiefangqi 1a1cbc6814 implemention of new api: apply 2020-04-14 21:43:53 +08:00
liyong 0ce83e39e1 fix TestShardSampleWrongNumber 
adjust internal api
 2020-04-13 21:31:45 +08:00
liyong 11403492ae add mindrecord subset random sampler 2020-04-11 18:48:23 +08:00
Cathy Wong 59a714c654 Correct shuffle UT buffer_size > #dataset-row as valid 2020-04-09 15:22:33 -04:00
jonyguo c688265671 fix: when use MindDataset block_reade=True hung 2020-04-07 17:53:23 +08:00
xiefangqi bc4602b58e fix and remove useless import of example, st, ut 2020-04-02 22:13:36 +08:00
mindspore-ci-bot 5c22c088bb !69 Enable skipped dataset zip operator python unit tests 
Merge pull request !69 from cathwong/ckw_dataset_py_ut1
 2020-04-01 22:20:37 +08:00
anzhengqi 6a1b865c91 check num_samples 2020-04-01 14:39:23 +08:00
Cathy Wong 2e881276ab Enable skipped dataset zip python unit tests 
Signed-off-by: Cathy Wong <cathy.wong@huawei.com>
 2020-03-31 16:47:13 -04:00
qianlong 8c88b39da1 Optimize the execution time of test case test_rgb_hsv.py 2020-03-31 09:23:20 +08:00