Nigel
/
mindspore
forked from mindspore-Ecosystem/mindspore
0
0
Fork 0
Code Issues Pull Requests Projects Releases Wiki Activity
1,883 Commits 24 Branches 0 Tags 3.9 GiB
Commit Graph

74 Commits

Author SHA1 Message Date
mindspore-ci-bot 2860fd9338 !984 Add unit test case for HWC2CHW. 
Merge pull request !984 from Tinazhang/hwc2chw
 2020-05-09 05:02:41 +08:00
Tinazhang c8b5586c7f add unit test for HWC2CHWC 2020-05-08 13:17:20 -04:00
Cathy Wong 58226addd6 Cleanup dataset UT: use md5 npz in test_zip for images 2020-05-08 11:25:48 -04:00
mindspore-ci-bot 47f5abceb4 !960 Adding example for grayscale 
Merge pull request !960 from EricZ/grayscale_fix
 2020-05-08 05:08:15 +08:00
mindspore-ci-bot 078dd86cfe !507 Implemented padded_batch 
Merge pull request !507 from ZiruiWu/batch_with_padding
 2020-05-08 04:54:05 +08:00
mindspore-ci-bot de7625777f !951 fix: MindDataset with columns_name parameter cause errors in some scenes 
Merge pull request !951 from guozhijian/fix_read_by_columns
 2020-05-08 04:51:03 +08:00
eric 0f0548f21b Added test case for grayscale support 2020-05-07 15:09:57 -04:00
Zirui Wu c2d364a573 batch with padding implemented 
support for 1 specific dimension to be None, added validator

fix various CI complains

another round of CI fixes

ci

refactor parts of the code

code refactor

ci fix

comments added, fix bugs

address review comments

address review comments

review cmts

added simple perf test script

update pad code

perf imprv
 2020-05-07 11:18:42 -04:00
jonyguo d4d236bcce fix: use MindDataset by column_names get data error in some situation 2020-05-07 18:12:36 +08:00
liyong b520ca9087 fix pk sampler in mindrecord 2020-05-07 14:54:23 +08:00
Cathy Wong 772e6c1461 Cleanup dataset UT: test_batch, save_and_check support 2020-05-05 15:35:09 -04:00
eric 36fffb7706 Added example md5 generation 
Comparison example

Added md5 and comparison example for py_transforms

Added md5 check for images
 2020-05-04 21:15:33 -04:00
Junhan Hu 83c68ca2ef Skip pyfunc test case 2020-05-01 15:14:07 -04:00
eric 26cb3e8a5f Added test function to show that seed doesn't work. 
Added testcase to show that c image aug don't use seed properly

Added passing test cases

Added working testcases for using seed

Added additional test cases to show seed use

Added test case for seed
 2020-04-30 18:57:59 -04:00
ms_yan c56fe3aa2d modify take op with an operator 2020-04-30 10:16:36 +08:00
mindspore-ci-bot 8af10eb51e !875 Reject python OP in operations argument for C++ uniform augmentation OP 
Merge pull request !875 from AdelShafiei/ua_py
 2020-04-30 06:11:17 +08:00
Adel Shafiei d15bd04bfe added input validation to reject python op in C++ uniform augmentation operations list 2020-04-29 16:48:10 -04:00
mindspore-ci-bot a606c2e4da !872 [Dataset] Add schema support for GeneratorDataset 
Merge pull request !872 from JunhanHu/generator_schema
 2020-04-30 02:58:29 +08:00
mindspore-ci-bot 2303453753 !869 Random data op 
Merge pull request !869 from JesseKLee/random_data_op
 2020-04-30 02:56:39 +08:00
Junhan Hu c5a8ffe4f4 Add schema support for GeneratorDataset 2020-04-29 13:50:51 -04:00
Jesse Lee 5236d0c3c0 Replace print with logger.info 2020-04-29 12:44:04 -04:00
mindspore-ci-bot 8d3695f666 !672 Added UT for uniform augmentation C++ OP 
Merge pull request !672 from AdelShafiei/ua_ut
 2020-04-29 23:56:09 +08:00
Jesse Lee 270bf831a9 Random Data Op 2020-04-29 10:26:00 -04:00
jiangzhiwen 34bfa2f7c9 fix skip 2020-04-29 20:12:37 +08:00
Adel Shafiei 3322e65da9 added ut for uniform augment C++ op 2020-04-28 21:07:29 -04:00
mindspore-ci-bot b37db1edf5 !603 [MD] update pk sampler in minddataset 
Merge pull request !603 from liyong126/update_pk_sampler
 2020-04-27 23:17:38 +08:00
mindspore-ci-bot f82e63fecc !671 Added testcase for sync_wait 
Merge pull request !671 from EricZ/master
 2020-04-27 22:14:10 +08:00
mindspore-ci-bot 0e3054d527 !466 Deepcopy problem when pyfunc cannot be pickled 
Merge pull request !466 from h.farahat/deepcopy
 2020-04-27 22:07:55 +08:00
liyong bfba630aa2 update pK_sampler 2020-04-27 21:40:49 +08:00
Zirui Wu 8c3931cf1d fix first epoch always shuffle with default seed in random sampler 
fix ut
 2020-04-26 10:58:59 -04:00
eric 2d115cd04e Added example for multiple iterator 
Added new testcase for multi iterator

Addressing review

Fixed typo
 2020-04-24 17:28:40 -04:00
hesham a9e9266149 Deepcopy problem when pyfunc cannot be pickled 2020-04-22 11:55:27 -04:00
mindspore-ci-bot aad5771a62 !524 Added support for UA augmentation ops with tests 
Merge pull request !524 from alashkari/ua-ops
 2020-04-22 08:28:47 +08:00
Amir Lashkari 56e7a7deb5 Added UniformAugment + Python Augmentation Ops 2020-04-21 19:54:32 -04:00
mindspore-ci-bot dc0491caf9 !508 [Dataset] Adding sync_wait operator for dataset 
Merge pull request !508 from EricZ/master
 2020-04-22 06:15:24 +08:00
eric cd94518769 X# This is a combination of 2 commits. 
Initial commit for dataset op python

Added signature to barrier

Adde compiling barrier code

Rebasing, fixed new compile errors

Final fix for make_unique

Added pybind API for barrier

Fixed pyfunc invocation

python interface - sync_wait

!1 sync_wait python interface
* python interface - sync_wait

fix test

update test

update test

Added new test case

add test case

test for shuffle + batch

Added two-sync test case

Restrited that no shuffle after sync

Added sync to pipeline info

block first databuffer as well

Intelligently get batch size

Fix default case

Lock Pair shares among all iterators

Added fix for empty character

Fixed up test case formatting

Fix end of epoch in sync_wait

Fixing CI
 2020-04-21 17:42:13 -04:00
Junhan Hu 78001ac9e6 Add multiprocessing support for Mindspore.Dataset.GeneratorDataset 2020-04-21 16:03:43 -04:00
mindspore-ci-bot fb18671b28 !506 [Dataset] Multiprocessing support for Pyfunc 
Merge pull request !506 from JunhanHu/multiprocess_pyfunc
 2020-04-22 03:57:26 +08:00
Junhan Hu b13e7bc31a Add python multiprocessing support for Mindspore.dataset 2020-04-21 14:56:25 -04:00
qianlong db80f4ff92 The num_samples and numRows in schema for TFRecordDataset are conflict 2020-04-21 17:08:59 +08:00
mindspore-ci-bot d9e4dcc33b !483 Optimize skip dataset op 
Merge pull request !483 from jiangzhiwen/dataset/skip_opt
 2020-04-21 15:10:23 +08:00
liyong f1542a90a3 add pk sampler 2020-04-21 10:08:38 +08:00
jiangzhiwen e1b109e8b8 optimize skip dataset op 2020-04-21 09:16:37 +08:00
Cathy Wong 60df369100 Fixup py Normalize doc: takes input CHW 2020-04-20 15:17:30 -04:00
mindspore-ci-bot 6369cf27bd !406 added first row crc check for when reading tfrecord files 
Merge pull request !406 from Peilin/first-row-crc-check
 2020-04-21 00:10:18 +08:00
mindspore-ci-bot 98fbd30a5b !460 [Data]Add filter operation 
Merge pull request !460 from xulei/filter_master
 2020-04-20 23:33:44 +08:00
mindspore-ci-bot 822a3160e4 !404 [Dataset] Add Python Sampler support for CPP dataset 
Merge pull request !404 from JunhanHu/python_sampler_for_cpp_dataset
 2020-04-20 22:45:51 +08:00
xulei2020 c705ea5e5b add filterOp code 2020-04-20 22:32:19 +08:00
Peilin Wang 9bc2134cb7 added checking of first row crc to find invalid tfrecord files 
addressed code review comments. added check in python layer to exclude directories and to raise an error if a pattern does not match any file

fixed clang format

fixed cppcheck

fixed cppcheck (used std::accumulate and std::copy_if). regenerated tfrecord file to contain correct header, it was a dummy header before

fixed cppcheck: added const reference for string parameter for lambdas, fixed clang format: whitespace adjustments

more clang whitespace fixes...

changed print to logger.info
 2020-04-20 10:25:26 -04:00
yanghaitao 2795e492ff TextFileDataset 2020-04-20 11:18:55 +08:00