p47068915
/
mindspore
forked from mindspore-Ecosystem/mindspore
1
0
Fork 0
Code Issues Pull Requests Projects Releases Wiki Activity
876 Commits 23 Branches 0 Tags 3.9 GiB
Commit Graph

42 Commits

Author SHA1 Message Date
mindspore-ci-bot aad5771a62 !524 Added support for UA augmentation ops with tests 
Merge pull request !524 from alashkari/ua-ops
 2020-04-22 08:28:47 +08:00
Amir Lashkari 56e7a7deb5 Added UniformAugment + Python Augmentation Ops 2020-04-21 19:54:32 -04:00
mindspore-ci-bot dc0491caf9 !508 [Dataset] Adding sync_wait operator for dataset 
Merge pull request !508 from EricZ/master
 2020-04-22 06:15:24 +08:00
eric cd94518769 X# This is a combination of 2 commits. 
Initial commit for dataset op python

Added signature to barrier

Adde compiling barrier code

Rebasing, fixed new compile errors

Final fix for make_unique

Added pybind API for barrier

Fixed pyfunc invocation

python interface - sync_wait

!1 sync_wait python interface
* python interface - sync_wait

fix test

update test

update test

Added new test case

add test case

test for shuffle + batch

Added two-sync test case

Restrited that no shuffle after sync

Added sync to pipeline info

block first databuffer as well

Intelligently get batch size

Fix default case

Lock Pair shares among all iterators

Added fix for empty character

Fixed up test case formatting

Fix end of epoch in sync_wait

Fixing CI
 2020-04-21 17:42:13 -04:00
Junhan Hu 78001ac9e6 Add multiprocessing support for Mindspore.Dataset.GeneratorDataset 2020-04-21 16:03:43 -04:00
mindspore-ci-bot fb18671b28 !506 [Dataset] Multiprocessing support for Pyfunc 
Merge pull request !506 from JunhanHu/multiprocess_pyfunc
 2020-04-22 03:57:26 +08:00
Junhan Hu b13e7bc31a Add python multiprocessing support for Mindspore.dataset 2020-04-21 14:56:25 -04:00
qianlong db80f4ff92 The num_samples and numRows in schema for TFRecordDataset are conflict 2020-04-21 17:08:59 +08:00
mindspore-ci-bot d9e4dcc33b !483 Optimize skip dataset op 
Merge pull request !483 from jiangzhiwen/dataset/skip_opt
 2020-04-21 15:10:23 +08:00
liyong f1542a90a3 add pk sampler 2020-04-21 10:08:38 +08:00
jiangzhiwen e1b109e8b8 optimize skip dataset op 2020-04-21 09:16:37 +08:00
Cathy Wong 60df369100 Fixup py Normalize doc: takes input CHW 2020-04-20 15:17:30 -04:00
mindspore-ci-bot 6369cf27bd !406 added first row crc check for when reading tfrecord files 
Merge pull request !406 from Peilin/first-row-crc-check
 2020-04-21 00:10:18 +08:00
mindspore-ci-bot 98fbd30a5b !460 [Data]Add filter operation 
Merge pull request !460 from xulei/filter_master
 2020-04-20 23:33:44 +08:00
mindspore-ci-bot 822a3160e4 !404 [Dataset] Add Python Sampler support for CPP dataset 
Merge pull request !404 from JunhanHu/python_sampler_for_cpp_dataset
 2020-04-20 22:45:51 +08:00
xulei2020 c705ea5e5b add filterOp code 2020-04-20 22:32:19 +08:00
Peilin Wang 9bc2134cb7 added checking of first row crc to find invalid tfrecord files 
addressed code review comments. added check in python layer to exclude directories and to raise an error if a pattern does not match any file

fixed clang format

fixed cppcheck

fixed cppcheck (used std::accumulate and std::copy_if). regenerated tfrecord file to contain correct header, it was a dummy header before

fixed cppcheck: added const reference for string parameter for lambdas, fixed clang format: whitespace adjustments

more clang whitespace fixes...

changed print to logger.info
 2020-04-20 10:25:26 -04:00
yanghaitao 2795e492ff TextFileDataset 2020-04-20 11:18:55 +08:00
Junhan Hu 43a2e99833 Add python sampler support for CPP dataset 2020-04-19 20:36:37 -04:00
ms_yan f0c07c3fa6 Realize take op and add ut 2020-04-18 18:33:57 +08:00
mindspore-ci-bot 80333e9f55 !435 Fix dataset serialize and deserialize for MindDataset 
Merge pull request !435 from anthonyaje/fix_serdes_dataset
 2020-04-18 09:37:22 +08:00
mindspore-ci-bot 40f0a4a4f4 !333 Add skip op to Dataset 
Merge pull request !333 from jiangzhiwen/dataset/skip
 2020-04-18 09:19:07 +08:00
mindspore-ci-bot 9e1b5efd1d !434 Bug in cleaning dataset iterators 
Merge pull request !434 from h.farahat/multi_itr_bug
 2020-04-18 05:17:31 +08:00
anthonyaje ea297c0889 Fix dataset serdes for MindDataset 2020-04-17 13:24:02 -04:00
hesham 3c02c82771 Bug in weak reference. 
Add new test cases
 2020-04-17 11:36:32 -04:00
jzw 3f7054dccb add skip dataset op 2020-04-17 19:48:25 +08:00
mindspore-ci-bot cf026096a6 !183 Mindspore.dataset CPP sampler for GeneratorDataset 
Merge pull request !183 from JunhanHu/cpp_sampler
 2020-04-16 22:30:59 +08:00
Junhan Hu 9739d3b048 Add CPP sampler support for GeneratorDataset 2020-04-14 12:09:03 -04:00
mindspore-ci-bot 30de261c3c !243 Support nested repeat 
Merge pull request !243 from h.farahat/nested_repeat
 2020-04-14 23:07:32 +08:00
hesham 0fc23eee0f Support nested repeat 2020-04-14 10:13:18 -04:00
xiefangqi 1a1cbc6814 implemention of new api: apply 2020-04-14 21:43:53 +08:00
liyong 0ce83e39e1 fix TestShardSampleWrongNumber 
adjust internal api
 2020-04-13 21:31:45 +08:00
liyong 11403492ae add mindrecord subset random sampler 2020-04-11 18:48:23 +08:00
Cathy Wong 59a714c654 Correct shuffle UT buffer_size > #dataset-row as valid 2020-04-09 15:22:33 -04:00
jonyguo c688265671 fix: when use MindDataset block_reade=True hung 2020-04-07 17:53:23 +08:00
xiefangqi bc4602b58e fix and remove useless import of example, st, ut 2020-04-02 22:13:36 +08:00
mindspore-ci-bot 5c22c088bb !69 Enable skipped dataset zip operator python unit tests 
Merge pull request !69 from cathwong/ckw_dataset_py_ut1
 2020-04-01 22:20:37 +08:00
anzhengqi 6a1b865c91 check num_samples 2020-04-01 14:39:23 +08:00
Cathy Wong 2e881276ab Enable skipped dataset zip python unit tests 
Signed-off-by: Cathy Wong <cathy.wong@huawei.com>
 2020-03-31 16:47:13 -04:00
qianlong 8c88b39da1 Optimize the execution time of test case test_rgb_hsv.py 2020-03-31 09:23:20 +08:00
jonyguo 34e42bd6f9 1. add more log info for dataset & mindrecord, 2. add two new testcase for MindDataset 2020-03-30 17:24:24 +08:00
zhunaipan 930a1fb0a8 initial version 
Signed-off-by: leonwanghui <leon.wanghui@huawei.com>
 2020-03-27 22:54:54 +08:00