Nigel
/
mindspore
forked from mindspore-Ecosystem/mindspore
0
0
Fork 0
Code Issues Pull Requests Projects Releases Wiki Activity
4,517 Commits 24 Branches 0 Tags 3.9 GiB
Commit Graph

252 Commits

Author SHA1 Message Date
nhussain 5515016dba remove graphengine changes 
remove graphengine changes

concat op

Truncate Pair

concat_op

remove graph engine changes
 2020-06-18 17:17:50 -04:00
mindspore-ci-bot ffc8a3c362 !2274 add PadEndOp 
Merge pull request !2274 from xunxue/padend
 2020-06-19 05:05:14 +08:00
mindspore-ci-bot e4322baece !2316 [Dataset] Move python tree transformation into cpp pass 
Merge pull request !2316 from JunhanHu/alter_node_cpp
 2020-06-19 05:02:03 +08:00
islam_amin 0af156d169 added two ops for object detection, random horizontal flip with bbox and bouding box augment. along with their unit tests 2020-06-18 16:55:31 -04:00
xunxue 87aa9c8f7a PadEndOp 2020-06-18 16:07:34 -04:00
Junhan Hu 57f3732ac3 Move python tree transformation into cpp pass. 2020-06-18 15:52:38 -04:00
Zirui Wu b6e9504b31 phase I of Vocab rework 
phase II vocab rework

added more test cases

fix api doc string

address review cmts and fix CI

address ci complains

fix review cmts

ci
 2020-06-18 15:36:28 -04:00
Zirui Wu 8f2674850b address API doc style and content 
addr review cmts

revert

ci

private class

ci

fix ci

address ci

address review cmts

fix style err
 2020-06-18 12:23:42 -04:00
mindspore-ci-bot 5850b99161 !2211 Add TruncatePair Op to dataset 
Merge pull request !2211 from h.farahat/pair_truncate
 2020-06-18 20:58:40 +08:00
mindspore-ci-bot d0c5071cab !1993 [Dataset] Fix codedex. 
Merge pull request !1993 from luoyang/pylint
 2020-06-18 18:59:31 +08:00
ms_yan 8d1dae46ac Throw error when load config failed 2020-06-18 13:47:32 +08:00
hesham b9495a9ccc Truncate Pair 2020-06-17 23:11:27 -04:00
mindspore-ci-bot ea829e89c3 !2198 Add MaskOp to dataset 
Merge pull request !2198 from h.farahat/mask_op
 2020-06-18 02:50:16 +08:00
mindspore-ci-bot 808380a9f9 !2142 Cleanup dataset UT: refactor visualize() 
Merge pull request !2142 from Tinazhang/reorg-visualize
 2020-06-18 01:30:36 +08:00
ms_yan fdafc690f8 repair problem in NumpySlicesDataset 2020-06-17 21:49:02 +08:00
Tinazhang 169eaf73c7 reorganize util function visualize() 2020-06-17 09:30:05 -04:00
Yang dee8471d23 !1993 [Dataset] Fix codedex. 2020-06-17 16:25:38 +08:00
qianlong 4f16f036be Add WhitespaceTokenizer and UnicodeScriptTokenizer for nlp 
add CaseFold, NormalizeUTF8

add RegexReplace

add RegexTokenizer

add BasicTokenizer

add WordpieceTokenizer

add BertTokenizer
 2020-06-17 15:47:04 +08:00
hesham f2462bb00d Mask Op 2020-06-16 23:51:46 -04:00
mindspore-ci-bot ca94916ef9 !2184 CocoDataset add category_id check 
Merge pull request !2184 from xiefangqi/xfq_fix_coco_issue
 2020-06-17 10:06:41 +08:00
mindspore-ci-bot 625f2421b5 !1977 random walk in gnn node2vec 
Merge pull request !1977 from JonathanY/randomwalk
 2020-06-17 03:22:21 +08:00
mindspore-ci-bot 3adbde56b2 !2162 SliceOp 
Merge pull request !2162 from h.farahat/slice_op
 2020-06-17 01:22:47 +08:00
mindspore-ci-bot 898b2fdecd !2187 Fix comment display issues in BuildVocabDataset 
Merge pull request !2187 from ZiruiWu/vocab_rework
 2020-06-17 00:51:37 +08:00
Zirui Wu 2794883644 fix selected minor issues 
fix review comments
 2020-06-16 11:07:49 -04:00
xiefangqi aab190fcdb CocoDataset add category_id validation 2020-06-16 21:37:22 +08:00
liyong d4f8f57c7e add split in minddataset 2020-06-16 19:50:41 +08:00
hesham e2012a1de9 Slice Op 2020-06-16 02:32:40 -04:00
mindspore-ci-bot dceb3c2be4 !2026 Change C++ behavior in RandomCropAndResize 
Merge pull request !2026 from MahdiRahmaniHanzaki/I1K3FL-random-resized-crop
 2020-06-16 05:27:36 +08:00
Mahdi add29f95f5 resolved issues in the comments 2020-06-15 16:45:18 -04:00
mindspore-ci-bot 2cab44a637 !2028 FillOp for BERT 
Merge pull request !2028 from nhussain/fillOp_learning
 2020-06-16 02:45:45 +08:00
nhussain dd9bf09f0a added FillOp for #119 - special Ops 2020-06-15 11:30:43 -04:00
Peilin Wang 848e07d022 initial commit, start of BucketBatchByLengthOp 
c implementation done, just need to call batch/pad

added python api and validator

added pybind/de_pipeline stuff, fixed some compile errors, figure out how null py::function works

added tiny bit of doc

integrated with static batch methods

fixed some bugs

some more bug fixes and cleanup

ci fix

fix ci

ci fix

fix ci

added test_cases and debugged

addressed code review comments

addressed code review comments

ci fix

ci fix

addressed code review comments

addressed code review comments
 2020-06-15 10:20:01 -04:00
Jonathan Yan 87d2c27c7f random walk v1 2020-06-15 08:54:10 -04:00
mindspore-ci-bot a8cf83acf2 !1932 Add CLUE dataset 
Merge pull request !1932 from jiangzhiwen/dataset/clue
 2020-06-15 11:48:35 +08:00
jiangzhiwen e0e167a000 add CLUE dataset 2020-06-15 10:40:46 +08:00
hesham a22763b612 Bug in CIFAR after removing GetMutabble buffer 
Fixes # I1KIPC
 2020-06-14 13:27:17 -04:00
jonyguo bb2082daec fix: MindDataset padded log error 2020-06-13 19:46:24 +08:00
mindspore-ci-bot 3085e51e45 !1922 check parameter num_samples of sampler 
Merge pull request !1922 from yanghaitao/yht_check_num_samples_sampler
 2020-06-12 10:22:14 +08:00
mindspore-ci-bot 1b5d9a1194 !1981 Implemented from_dataset, a way to build a vocab from existing dataset 
Merge pull request !1981 from ZiruiWu/from_dataset_dev
 2020-06-12 08:08:19 +08:00
Zirui Wu 880ce5ea26 implemented from_dataset 
fix complie error

more tests

address CI complains

fix ci

adress review comments

address review cmts
 2020-06-11 16:20:19 -04:00
Alexey Shevlyakov 4e333a2f22 MindData profiler infrastructure 2020-06-11 14:52:36 -04:00
mindspore-ci-bot 0a95223f25 !1910 RandomColorAdjust error for grayscale images 
Merge pull request !1910 from MahdiRahmaniHanzaki/I1J9SQ-random-color-adjust-bug
 2020-06-11 23:15:19 +08:00
yanghaitao 8f16cff12e add para check for sampler 2020-06-11 22:26:06 +08:00
jonyguo 1de7271afc add floatxx test case 2020-06-11 19:26:54 +08:00
ms_yan bc22c172b8 add TensorDataset and its ut 2020-06-11 15:24:12 +08:00
mindspore-ci-bot 251a6667a5 !1989 fix MindDataset distribute shuffle error 
Merge pull request !1989 from guozhijian/fix_distribute_shuffle
 2020-06-11 14:51:17 +08:00
mindspore-ci-bot b3da41bd7a !1913 CocoDataset implementation 
Merge pull request !1913 from xiefangqi/xfq_support_coco
 2020-06-11 14:37:44 +08:00
xiefangqi 097cbcc0cb CocoDataset implementation 2020-06-11 11:52:17 +08:00
jonyguo 96859f44b6 fix: MindDataset distribute shuffle bug 2020-06-11 11:06:34 +08:00
Mahdi ba2e2dced4 fixed some warnings and typos 2020-06-10 17:18:56 -04:00
Tinazhang 2812cd57db fix pylint error 2020-06-10 15:55:11 -04:00
mindspore-ci-bot fe0c3e9a8f !1908 Add UT to six data vision transforms ops 
Merge pull request !1908 from Tinazhang/ut-normalize
 2020-06-11 00:43:57 +08:00
heleiwang 3ece8dd090 1. support get_all_edges, get_nodes_from_edge, get_sampled_neighbors, get_neg_sampled_neighbors and graph_info API 
2. mod cora and citeseer conversion script
 2020-06-10 15:52:26 +08:00
mindspore-ci-bot 5eb95599f6 !1874 Updates to string tensor 
Merge pull request !1874 from h.farahat/string_tensor_updates
 2020-06-10 00:59:17 +08:00
Tinazhang c3de5c7a54 adding new TCs to six ops and skip md5 case in RandomPerspective 2020-06-09 10:51:45 -04:00
hesham f837ddc956 - Bug when empty strings sent to Python 
- Support accepting Numpy of str as input
- Support batching strings
- Core logic of batch&pad is static
- Make Pad a utility function
 2020-06-09 10:34:38 -04:00
liyong feff8899ac support padding samples 2020-06-05 09:14:42 +08:00
mindspore-ci-bot 5c21616293 !1807 Implemented Ngram TensorOp for dataset 
Merge pull request !1807 from ZiruiWu/ngram_dev
 2020-06-05 01:04:27 +08:00
Zirui Wu dbf9936ec4 Implemented n-gram for dataset TensorOp 2020-06-04 09:57:44 -04:00
Jamie Nisbet 51bc0c0460 consistent design for num_samples 
updates

more update

more work

more fixin

post rebase updates

clang formatting

code review recovery

ci fixes

updates

update

update

update
 2020-06-03 15:01:18 -04:00
Tinazhang 33fe66fd1e fix error handling for bigger crop size case 2020-06-02 11:06:41 -04:00
Yang 5522edc4b5 !1713 [Dataset] Clean pylint. 2020-06-02 09:02:19 +08:00
liuwenhao4 7b911886ec Fixing some tiny faults about Pylint in my code(ops) 2020-05-30 16:55:37 +08:00
mindspore-ci-bot 7878743400 !1638 fixed rounding edge case in split 
Merge pull request !1638 from Peilin/splitOp-after-testing
 2020-05-30 05:31:33 +08:00
mindspore-ci-bot 0e3dd8149a !1703 Adding sync_wait input check 
Merge pull request !1703 from EricZ/master
 2020-05-30 05:29:38 +08:00
mindspore-ci-bot c366b3fb18 !1704 fix num rows bug in sampler 
Merge pull request !1704 from Peilin/fix-random-sampler-bug
 2020-05-30 05:27:20 +08:00
Eric 8e1b2107ce Added example for sync_wait bad batch_num 
Added broken test case for cv

Fixed error handling in sync_wait
 2020-05-29 16:30:12 -04:00
mindspore-ci-bot 976b4793d1 !1544 Python Unit Test for TenCrop 
Merge pull request !1544 from MahdiRahmaniHanzaki/I1IHNG-ten-crop
 2020-05-30 03:02:29 +08:00
Peilin Wang 3ef3d1433d fixed num_samples bug for RandomSampler 2020-05-29 13:09:26 -04:00
Tinazhang f4a61fdbb2 Adding new UT cases and missing TCs in PY Ops. 2020-05-29 12:12:17 -04:00
Peilin Wang 5469be2a97 fixed bug for split, RandomSampler and some other cleanup 
add another test case

typo

merge conflict

another PR changed testing behavior, updated test cases in this commit

added input check for use_sampler

addressed code review comments

fixed pylint, not related to my changes

fixed edge case of rounding in getting split sizes

fix pylint
 2020-05-29 11:27:32 -04:00
Mahdi dfc097019b Added TenCrop test 
Added unit tests for both testing the functinality of the TenCrop and
its error messages. Due to the similarity of this method to FiveCrop the
test cases are similar to FiveCrop test cases.
Signed-off-by: Mahdi <mahdi.rahmani.hanzaki@huawei.com>

added error_msg function call in the main method

refactored the test and added visual representation of the results

Separated the two error cases into two different functions and used the
visualize function in util.py to plot the result of TenCrop.
Signed-off-by: Mahdi <mahdi.rahmani.hanzaki@huawei.com>

Added new test cases

Added new test cases including test case for checking the error message
when the size variable is not a positive integer, test case for
rectangle crop, test case for vertical flip setting, and testing for
similarity of the result of TenCrop for the same input data in different runs.
Signed-off-by: Mahdi <mahdi.rahmani.hanzaki@huawei.com>

changed visualize in test_five_crop

Changed the visualize function in test_five_crop to use the already
existing function in util.py
Signed-off-by: Mahdi <mahdi.rahmani.hanzaki@huawei.com>

made generate_golden variable global
 2020-05-28 17:52:56 -04:00
mindspore-ci-bot 21da86b393 !1591 add get_dataset_size for CelebADataset 
Merge pull request !1591 from yanghaitao/yht_celeba_get_dataset_size
 2020-05-28 22:21:43 +08:00
yanghaitao cc6c7a3f60 add get_dataset_size to celebadataset 2020-05-28 17:00:14 +08:00
mindspore-ci-bot 0f4dc13e8e !1478 [Dataset] Fix CI warnings, clean pylint. 
Merge pull request !1478 from luoyang/pylint
 2020-05-28 15:54:52 +08:00
Yang 9b2a778d94 !1478 [Dataset] clean pylint. 2020-05-28 14:06:36 +08:00
xiefangqi 5e4728c50f add voc support split 2020-05-28 11:18:32 +08:00
Peilin Wang d4c93575e7 fixed bug for split, RandomSampler and some other cleanup 
add another test case

typo

merge conflict

another PR changed testing behavior, updated test cases in this commit

added input check for use_sampler

addressed code review comments

fixed pylint, not related to my changes
 2020-05-26 16:29:52 -04:00
Yang abca62f407 10:00 26/5 clean pylint 2020-05-26 10:05:12 +08:00
mindspore-ci-bot 0a8ef2fe72 !1455 Cleanup dataset UT: restore config support 
Merge pull request !1455 from cathwong/ckw_dataset_config_util
 2020-05-26 05:40:39 +08:00
Cathy Wong f891e1755c Cleanup dataset UT: restore config support 2020-05-25 15:10:03 -04:00
mindspore-ci-bot aeb4c52f2d !1379 Creating new UTs to three vision transformation ops 
Merge pull request !1379 from Tinazhang/linearTrans
 2020-05-25 23:10:30 +08:00
mindspore-ci-bot 718a96430f !1389 Security issues in recent string tensor PR 
Merge pull request !1389 from h.farahat/fix_security_issues
 2020-05-24 08:45:21 +08:00
mindspore-ci-bot a6b8451a33 !1380 make ShuffleOp have deterministic behavior for subsequent epochs 
Merge pull request !1380 from Peilin/shuffle-subsequent-epoch-deterministic
 2020-05-23 21:02:30 +08:00
heleiwang f28f883cac fix comments error and modify parameter check 2020-05-23 18:31:13 +08:00
hesham c0f0e154f5 Clean up work for text python package 2020-05-22 23:53:34 -04:00
mindspore-ci-bot 41456ac824 !1369 dataset: delete StorageDataset 
Merge pull request !1369 from ms_yan/del_storage
 2020-05-23 10:39:34 +08:00
Tinazhang 7322839b04 add UTs for LinearTransformation, ToPIL, ToType 2020-05-22 18:04:09 -04:00
Peilin Wang 0cbcc7200b made shuffle determinisitc for subsequent epochs 2020-05-22 17:03:19 -04:00
mindspore-ci-bot 61639be1e0 !1375 Cleanup dataset UT: remove num_parallel_workers=1 in test_exception 
Merge pull request !1375 from cathwong/ckw_dataset_ut_cleanup5
 2020-05-22 23:54:52 +08:00
mindspore-ci-bot a528797253 !1377 Code Fix for Uniform Augmentation 
Merge pull request !1377 from Tinazhang/cc
 2020-05-22 23:20:01 +08:00
mindspore-ci-bot 458436186c !1365 Clean up work for text python sub-package 
Merge pull request !1365 from h.farahat/text_namespace
 2020-05-22 23:16:31 +08:00
Tinazhang b390883c6a Bug fix 2020-05-22 10:56:56 -04:00
Cathy Wong 702005d403 Cleanup dataset UT: remove num_parallel_workers=1 in test_exception 2020-05-22 09:28:17 -04:00
ms_yan d5e896b51c delete storageDataset Op API and its test case 2020-05-22 21:23:24 +08:00
mindspore-ci-bot 6f733ec113 !1308 Stage 2 of adding support for string Tensor 
Merge pull request !1308 from h.farahat/string_tensor2
 2020-05-22 21:17:53 +08:00
xiefangqi 34236ce1f1 fix pylint 2020-05-22 20:18:21 +08:00
mindspore-ci-bot 3363d4e834 !1249 Add GNN dataset processing API 
Merge pull request !1249 from heleiwang/hlw_gnn_data
 2020-05-22 18:49:36 +08:00
hesham 6c21e556c4 Clean up work for text python package 2020-05-22 04:35:58 -04:00
heleiwang 599a449e0b Support processing GNN data 2020-05-22 14:15:25 +08:00