Nigel
/
mindspore
forked from mindspore-Ecosystem/mindspore
0
0
Fork 0
Code Issues Pull Requests Projects Releases Wiki Activity
56,551 Commits 24 Branches 0 Tags 3.9 GiB
Commit Graph

43 Commits

Author SHA1 Message Date
jonyguo d05ed11ac3 enable set_seed for distributed 2022-10-23 19:13:35 +08:00
Xiao Tianci 6a8aaf7b11 support processing bytes data in Dataset 2022-08-26 14:33:42 +08:00
jonyguo 4d1db74ff3 fix: read multi mindrecord files which contains one empty 2022-07-20 09:21:39 +08:00
David 229603b4b5 Added docs to ut part 4 2022-06-02 21:37:15 -04:00
Cathy Wong b9396a32b7 [MD] Update import transforms statements in dataset UT 2022-05-25 19:51:26 -04:00
jonyguo 8ba8c930e7 fix: md AddressSanitizer 2022-05-10 18:45:42 +08:00
Cathy Wong eb4fd20bac [MD] Transform Unification Feature - Phase 2A - test_[a-q]*.py Python UT updates 2022-05-04 15:24:18 -04:00
ms_yan a85113c093 change dataset_file into dataset_files for MIndDataset 2021-12-14 10:52:41 +08:00
liyong b9bad0bdbb fix md ut part06 2021-10-29 10:26:34 +08:00
liyong 7c9f6899a3 fix md ut part05 2021-10-21 20:40:09 +08:00
liyong fc5ea9cc1c fix dataset ut for running independently 2021-10-20 14:27:36 +08:00
liyong d9aa18dfca refactor md ut 2021-10-18 17:55:28 +08:00
hetshah 2c1666fd06 set num epochs for iterators that use the default value 
add explicit paramter for infinite iterators

set appropriate value for num_epoch param for iteratos

fix long lines

remove trailing whitespace

fixed typo

remove comment

fixed comments

fixed test
 2021-09-22 17:32:50 +00:00
jonyguo 4c915fdcbb fix: for loop dataset iterator will cause memory leak 2021-08-30 16:08:26 +08:00
ms_yan 36a8886ca2 Revert "[feat] [assistant] [I3T96T] add new Dataset operator CMUARCTICDataset" 
This reverts commit b077aa1cab.

Revert "[feat] [assistant] [I3T96X] add new Dataset operator LibriSpeechDataset"

This reverts commit 4e6f7dc97d.

delete pass_registry_test.cc

comment  hiai_nlu_model_multi.pb related  line
 2021-08-23 01:46:38 +08:00
djc 4e6f7dc97d [feat] [assistant] [I3T96X] add new Dataset operator LibriSpeechDataset 2021-08-22 13:39:37 +08:00
jonyguo 42b3512a1b fix: when field is none in mr 2021-08-11 17:11:38 +08:00
jonyguo 9452142b50 enhance: enable shuffle infile / files when mindrecord 2021-05-14 17:41:18 +08:00
liyong 930e85ed5c fix get_dataset_size in distributedSampler & num_samples 2020-09-15 22:53:57 +08:00
xiefangqi 9b3c33e157 minddata iterator output ms_tensor 2020-09-11 11:35:50 +08:00
nhussain 92e99ff224 change map calls 2020-09-10 10:25:05 -04:00
nhussain 3bac9d3713 switch input columns and operation 
change ImagefolderDV2 name

change ds.transforms.vision to ds.vision

change batch api to match map api more closely

compose op changes

test_pylint

remove compose op from vision, move to transform module, refactor map and batch to use column_order
 2020-09-08 10:26:20 -04:00
Lixia Chen 27d5281641 Change epoch count to 1 for python testcases 2020-09-01 11:24:26 -04:00
dengyutao 388e43f7cb delete parameter block_reader for MindDataset 2020-07-31 11:30:05 +08:00
tony_liu2 1f4251a440 add try to minddataset python tests to remove files on fail 
delete unrelated file

change finally to except else

fix spacing alignment

fix indentation

fix indentation

fix indentation

add fix to new test case

add if main for pytest

fix spacing
 2020-07-22 16:34:10 -04:00
liyong 28ebd73096 fix bug when int or float is numpy type 2020-07-21 09:07:29 +08:00
liyong e2ea1fa0df activate num_samples in distributed samplers 2020-07-17 16:55:57 +08:00
liyong 2412ee09ce fix distributedSampler reshuffle and fix random_device failed 2020-06-21 18:23:47 +08:00
jonyguo 1de7271afc add floatxx test case 2020-06-11 19:26:54 +08:00
jonyguo 96859f44b6 fix: MindDataset distribute shuffle bug 2020-06-11 11:06:34 +08:00
liyong feff8899ac support padding samples 2020-06-05 09:14:42 +08:00
Yang 9b2a778d94 !1478 [Dataset] clean pylint. 2020-05-28 14:06:36 +08:00
Yang abca62f407 10:00 26/5 clean pylint 2020-05-26 10:05:12 +08:00
hesham df361d1d26 Change mem layout of string tensor 
add support for MindRecord and TFRecord
----
optimize tensorshape

optimize tensorshape and FlatIndex

TFRecord and MindRecord support for string tensor

Modify mem layout
Add new constructor
Add method Allocate

Change some GetMutableBuffer usages to AllocateBuffer
 2020-05-22 01:22:29 -04:00
jonwe bb51bb88d7 add compress in mindrecord 2020-05-22 09:37:51 +08:00
jinyaohui 5a914994ba clean pylint 2020-05-18 16:42:35 +08:00
jinyaohui bcfaff97f9 clean pylint 2020-05-18 10:31:46 +08:00
liyong aa3f89e74f mindrecord support read file list 2020-05-13 14:11:59 +08:00
jonyguo d4d236bcce fix: use MindDataset by column_names get data error in some situation 2020-05-07 18:12:36 +08:00
jonyguo c688265671 fix: when use MindDataset block_reade=True hung 2020-04-07 17:53:23 +08:00
xiefangqi bc4602b58e fix and remove useless import of example, st, ut 2020-04-02 22:13:36 +08:00
jonyguo 34e42bd6f9 1. add more log info for dataset & mindrecord, 2. add two new testcase for MindDataset 2020-03-30 17:24:24 +08:00
zhunaipan 930a1fb0a8 initial version 
Signed-off-by: leonwanghui <leon.wanghui@huawei.com>
 2020-03-27 22:54:54 +08:00