jasder
/
mindspore
forked from OSSInnovation/mindspore
0
0
Fork 0
Code Issues Pull Requests Releases Wiki Activity
2,581 Commits 9 Branches 0 Tags 238 MiB
Commit Graph

119 Commits

Author SHA1 Message Date
mindspore-ci-bot 718a96430f !1389 Security issues in recent string tensor PR 
Merge pull request !1389 from h.farahat/fix_security_issues
 2020-05-24 08:45:21 +08:00
mindspore-ci-bot a6b8451a33 !1380 make ShuffleOp have deterministic behavior for subsequent epochs 
Merge pull request !1380 from Peilin/shuffle-subsequent-epoch-deterministic
 2020-05-23 21:02:30 +08:00
heleiwang f28f883cac fix comments error and modify parameter check 2020-05-23 18:31:13 +08:00
hesham c0f0e154f5 Clean up work for text python package 2020-05-22 23:53:34 -04:00
mindspore-ci-bot 41456ac824 !1369 dataset: delete StorageDataset 
Merge pull request !1369 from ms_yan/del_storage
 2020-05-23 10:39:34 +08:00
Peilin Wang 0cbcc7200b made shuffle determinisitc for subsequent epochs 2020-05-22 17:03:19 -04:00
mindspore-ci-bot 61639be1e0 !1375 Cleanup dataset UT: remove num_parallel_workers=1 in test_exception 
Merge pull request !1375 from cathwong/ckw_dataset_ut_cleanup5
 2020-05-22 23:54:52 +08:00
mindspore-ci-bot a528797253 !1377 Code Fix for Uniform Augmentation 
Merge pull request !1377 from Tinazhang/cc
 2020-05-22 23:20:01 +08:00
mindspore-ci-bot 458436186c !1365 Clean up work for text python sub-package 
Merge pull request !1365 from h.farahat/text_namespace
 2020-05-22 23:16:31 +08:00
Tinazhang b390883c6a Bug fix 2020-05-22 10:56:56 -04:00
Cathy Wong 702005d403 Cleanup dataset UT: remove num_parallel_workers=1 in test_exception 2020-05-22 09:28:17 -04:00
ms_yan d5e896b51c delete storageDataset Op API and its test case 2020-05-22 21:23:24 +08:00
mindspore-ci-bot 6f733ec113 !1308 Stage 2 of adding support for string Tensor 
Merge pull request !1308 from h.farahat/string_tensor2
 2020-05-22 21:17:53 +08:00
xiefangqi 34236ce1f1 fix pylint 2020-05-22 20:18:21 +08:00
mindspore-ci-bot 3363d4e834 !1249 Add GNN dataset processing API 
Merge pull request !1249 from heleiwang/hlw_gnn_data
 2020-05-22 18:49:36 +08:00
hesham 6c21e556c4 Clean up work for text python package 2020-05-22 04:35:58 -04:00
heleiwang 599a449e0b Support processing GNN data 2020-05-22 14:15:25 +08:00
jinyaohui fbdba6e4da clean pylint 2020-05-22 13:23:13 +08:00
hesham df361d1d26 Change mem layout of string tensor 
add support for MindRecord and TFRecord
----
optimize tensorshape

optimize tensorshape and FlatIndex

TFRecord and MindRecord support for string tensor

Modify mem layout
Add new constructor
Add method Allocate

Change some GetMutableBuffer usages to AllocateBuffer
 2020-05-22 01:22:29 -04:00
mindspore-ci-bot 58e6d7d950 !1341 Added lookup and vocab to mindspore.dataset.text 
Merge pull request !1341 from ZiruiWu/vocab_and_lookup
 2020-05-22 10:19:42 +08:00
jonwe bb51bb88d7 add compress in mindrecord 2020-05-22 09:37:51 +08:00
mindspore-ci-bot 2e3d55ed87 !1281 Implementation of SplitOp 
Merge pull request !1281 from Peilin/splitOp
 2020-05-22 09:29:03 +08:00
mindspore-ci-bot 39b9aedf68 !1342 Bug fix on issue Core dump on GPU when train with lenet with AU 
Merge pull request !1342 from Tinazhang/cc
 2020-05-22 09:18:49 +08:00
Peilin Wang 71e8bb1960 general split case done, chaining sampler (basic case) is working 
implementation 99% complete

everything and tested except for repeatable shuffling

tested most basic/typical split usecases

cleanup

some more cleanup

fix CI

more ci fix

more ci fixes

more ci fix

more ci fix

more ci fix

added more tests, fixed some bugs

some more clean up and test cases

added shard/shuffle before split warning/error

addressed code review comments and ci

fixed ci
 2020-05-21 20:42:51 -04:00
Tinazhang e9e40b688b Bug fix 2020-05-21 18:20:00 -04:00
Zirui Wu 25ab2ef303 Implemented lookup and vocab 2020-05-21 17:17:24 -04:00
mindspore-ci-bot 46949fc327 !1307 Cleanup dataset UT: unskip and enhance TFRecord sharding tests 
Merge pull request !1307 from cathwong/ckw_dataset_ut_unskip2
 2020-05-22 03:21:45 +08:00
qianlong 451c20a6f5 Add UnicodeCharTokenizer for nlp 2020-05-21 09:22:45 +08:00
mindspore-ci-bot 93e7c97a96 !1272 [Dataset] MindData Tree Optimizer Infrastructure 
Merge pull request !1272 from JunhanHu/minddata_opt
 2020-05-21 05:29:00 +08:00
Cathy Wong b78894e02b Cleanup dataset UT: unskip and enhance TFRecord sharding tests 2020-05-20 17:05:38 -04:00
Junhan Hu f44d213503 MindData optimizer infrastructure. 2020-05-20 16:11:26 -04:00
xulei2020 163b6b7ea7 add jieba c++ code 2020-05-20 15:55:12 +08:00
Tinazhang 17cecf2cf5 Added TCs to RandomCrop and RandomCropAndResize and modified visalize() calling 2020-05-19 15:42:24 -04:00
jinyaohui 5a914994ba clean pylint 2020-05-18 16:42:35 +08:00
jinyaohui bcfaff97f9 clean pylint 2020-05-18 10:31:46 +08:00
hesham e8ca243364 -Add DE_STRING 
-replace switch'case by indexing

- Add test case
- Add constructors
- Add getItem string

- Fix bugs
- Add more tests

- Tensor iterator
- asNumpy
- TextFileDataset

- Tensor(Numpy)

- Super > 2D
- Add more test cases for GeneratorDataset

- Change StartAddr to GetBuffer and GetMutableNuffer

- Raise an error if batch is used with strings

Clean-up work
 2020-05-15 20:33:28 -04:00
jiangzhiwen cb2814b498 flat_map first commit 2020-05-15 17:45:39 +08:00
mindspore-ci-bot c680cfbf27 !1157 dataset: add concat operation for dataset 
Merge pull request !1157 from ms_yan/concat_dataset
 2020-05-15 16:07:19 +08:00
mindspore-ci-bot ab031ee9ea !1126 VOCDataset support object detection function 
Merge pull request !1126 from xiefangqi/voc_support_detection
 2020-05-15 15:56:39 +08:00
xiefangqi c937bad53f minddata support voc 2020-05-15 13:24:03 +08:00
ms_yan c0fa7b4b19 init commit of concat dataset 
change to use __add__ operation instead ds.concat
 2020-05-15 13:14:13 +08:00
jonyguo be2e7531ca fix: MindDataset parameter shard_id & num_shards check 2020-05-14 17:18:11 +08:00
Cathy Wong 913074e656 Cleanup dataset UT: resolve skipped test units 2020-05-13 14:41:57 -04:00
liyong aa3f89e74f mindrecord support read file list 2020-05-13 14:11:59 +08:00
Cathy Wong 49ef53f164 Cleanup dataset UT: util.py internals 2020-05-11 14:44:24 -04:00
mindspore-ci-bot 2860fd9338 !984 Add unit test case for HWC2CHW. 
Merge pull request !984 from Tinazhang/hwc2chw
 2020-05-09 05:02:41 +08:00
Tinazhang c8b5586c7f add unit test for HWC2CHWC 2020-05-08 13:17:20 -04:00
Cathy Wong 58226addd6 Cleanup dataset UT: use md5 npz in test_zip for images 2020-05-08 11:25:48 -04:00
mindspore-ci-bot 47f5abceb4 !960 Adding example for grayscale 
Merge pull request !960 from EricZ/grayscale_fix
 2020-05-08 05:08:15 +08:00
mindspore-ci-bot 078dd86cfe !507 Implemented padded_batch 
Merge pull request !507 from ZiruiWu/batch_with_padding
 2020-05-08 04:54:05 +08:00