Nigel
/
mindspore
forked from mindspore-Ecosystem/mindspore
0
0
Fork 0
Code Issues Pull Requests Projects Releases Wiki Activity
56,551 Commits 24 Branches 0 Tags 3.9 GiB
Commit Graph

1312 Commits

Author SHA1 Message Date
i-robot 7676d9958b
!34608 [MD] Adding docstrings for minddata UT Python 
Merge pull request !34608 from davidanugraha/add_dataset_test_comment
 2022-05-25 02:10:32 +00:00
David ee0a815990 Added docs for ut files 2022-05-24 17:43:53 -04:00
i-robot 1ade2e9f25
!34235 [assistant][MelScale] 
Merge pull request !34235 from QingfengLi/MelScale
 2022-05-24 12:08:00 +00:00
YangLuo e1df0d9688 output_shape add param estimate 2022-05-24 09:09:30 +08:00
chenx2ovo d62655e199 [fix] [assistant] [I4ZOCP] add validation for input param 2022-05-23 16:06:50 +08:00
jonyguo 0718a283c8 filter dumplicate log from python operation 2022-05-20 14:46:46 +08:00
i-robot 17fb1b2ea4
!34468 fix typecast on validate string 
Merge pull request !34468 from luoyang/fix_typecast
 2022-05-17 11:03:33 +00:00
luoyang bc4d57018c fix typecast on validate string 2022-05-16 23:21:06 +08:00
i-robot 4cde40189c
!34321 [MD] Normalize md5 tests - new golden files, unskip tests 
Merge pull request !34321 from cathwong/ckw_xtransuni_ut_new_golden_unskip_md5
 2022-05-13 21:00:38 +00:00
i-robot 2427626f10
!34250 Return -1 as unknown batch size for callable batch_size param 
Merge pull request !34250 from luoyang/get_batch_size
 2022-05-13 06:45:07 +00:00
Cathy Wong 1caa2012bb [MD] Normalize md5 tests - new golden files, unskip tests 2022-05-12 16:02:36 -04:00
Cathy Wong 942001ffc6 [MD] Tranform Unification - skip normalize md5 tests 2022-05-12 12:56:01 -04:00
i-robot 8d44248535
!34199 [MD] Transforms Unificiation Feature - Phase 1B - Minor Code Fixes and UT Updates 
Merge pull request !34199 from cathwong/ckw_xtranuni_code2_cutout_totensor
 2022-05-12 13:51:00 +00:00
luoyang 3cb854abc7 Return -1 as unknown batch size for callable batch_size param 2022-05-12 15:38:08 +08:00
fr199787 9e5eb88799 [faet] [assistant] [I4S2FH] add new operator GetImageSize 2022-05-12 11:17:38 +08:00
i-robot 5e31a54cbb
!32787 [assistant][ops]Add GetImageNumChannels 
Merge pull request !32787 from yangwm/channels
 2022-05-12 02:07:12 +00:00
Cathy Wong d08846176b [MD] Transforms Unificiation Feature - Phase 1B - Minor Code Fixes and UT updates 2022-05-11 16:21:35 -04:00
jonyguo 8ba8c930e7 fix: md AddressSanitizer 2022-05-10 18:45:42 +08:00
shenyu d69c2ce0d1 [feat] [assistant] [I4S2FG] add new operator GetImageNumChannels 2022-05-09 22:32:05 +08:00
i-robot 0828efe02e
!33811 [MD] Transform Unification Feature - Phase 2B - test_[r-z]*.py Python UT updates 
Merge pull request !33811 from cathwong/ckw_xtranuni_phase2b_uts_r_to_z
 2022-05-05 18:48:57 +00:00
Cathy Wong eb4fd20bac [MD] Transform Unification Feature - Phase 2A - test_[a-q]*.py Python UT updates 2022-05-04 15:24:18 -04:00
Cathy Wong 22adb200b2 [MD] Transform Unification Feature - Phase 2 - test_[r-q]*.py Python UT updates 2022-05-04 09:16:46 -04:00
i-robot fd22d69ca7
!33475 Remove test data containing sensitive word 
Merge pull request !33475 from xiaotianci/fix_examples
 2022-04-28 02:05:09 +00:00
i-robot 37a28be84a
!33574 [MD] fix test case 
Merge pull request !33574 from liyong126/fix_test_case1
 2022-04-27 20:59:48 +00:00
Xiao Tianci 0dc85a15b9 remove sensitive word in test data 2022-04-27 10:51:44 +08:00
liyong 29855385a5 fix random dataset 2022-04-26 19:48:17 +08:00
i-robot fcb0319747
!33270 [assistant][InverseMelScale] 
Merge pull request !33270 from chenchen/InverseMelScale
 2022-04-26 08:55:58 +00:00
chenx2ovo cd8395f352 [feat][assistant][I3CEG2] add op inversemelscale 2022-04-25 18:58:38 +08:00
Cathy Wong 907194dfb8 [MD] UT Updates - more eager tests, minor updates 2022-04-18 22:03:32 -04:00
liyong 830e76656a fix mindrecord rename issue 2022-04-12 19:36:24 +08:00
i-robot c1bacb39dc
!32644 [MD] fix doc issue2 in master 
Merge pull request !32644 from liyong126/fix_doc_issue2
 2022-04-08 06:32:34 +00:00
i-robot 0af661659c
!32534 [MD][Autotune] Add rank to filename 
Merge pull request !32534 from harshvardhangupta/rank_id_suffix_at_config
 2022-04-08 02:51:17 +00:00
liyong 2172934177 fix doc issue2 2022-04-08 10:46:08 +08:00
harshvardhangupta 6b8a76c59d add rank to filename 2022-04-07 16:12:06 -04:00
jonyguo b8205ee995 add testcase 2022-04-07 09:49:23 +08:00
liyong 3191160eca fix some issue 2022-04-06 17:48:55 +08:00
unknown a9e6a151da fix dataset two level pipeline hang problem 2022-04-06 12:39:48 +08:00
liu-yongqi-63 b125369f1e Fix the error is reported when the file name is directly transferred 2022-04-02 11:51:57 +08:00
i-robot d73ccd17f6
!32328 [MD][Offload] Reduce Offload UT Computation Time 
Merge pull request !32328 from alashkari/update-ut
 2022-03-31 14:55:28 +00:00
i-robot c3d7c425e9
!29856 [assistant][GriffinLim] 
Merge pull request !29856 from chenchen/g2
 2022-03-31 09:33:52 +00:00
yanghaoran c44de68591 takedown test_datasets_generator to ensure gate stability 2022-03-31 16:56:03 +08:00
chenx2ovo e19dc02b47 [feat][assistant][I3CECZ] add op griffinlim 2022-03-31 11:30:26 +08:00
Rescue 60d895fcd8 Decrease test_map_offload.py computation time. 2022-03-30 16:40:19 +00:00
jonyguo 4378014ffb fix: when return numpy in GeneratorDataset 2022-03-30 20:21:41 +08:00
liyong f703f66f6c fix gis issue 2022-03-29 11:17:01 +08:00
i-robot 3f18180606
!31742 [MD][Autotune] Change autotune config 
Merge pull request !31742 from harshvardhangupta/at_config_suggestion
 2022-03-25 21:00:12 +00:00
harshvardhangupta 0086f4ee05 Modify the Dataset AutoTune config #31742 2022-03-25 09:03:38 -04:00
i-robot 1af8b365de
!31880 [MD] add OBSMindDataset 
Merge pull request !31880 from liyong126/add_obs_mindrecord_dataset
 2022-03-25 06:50:58 +00:00
liyong 419109cf0a add OBSMindDataset 2022-03-25 11:03:53 +08:00
jonyguo 317c48a933 add log warning and op error check 2022-03-25 09:26:29 +08:00
liyong bff0fbc76f fix GIL acquire Misuse 2022-03-24 08:45:30 +08:00
i-robot 4110c3d6d6
!31646 [dataset][bugfix] 修复config 接口typeerror的问题 
Merge pull request !31646 from xiefangqi/md_fix_interface_comments
 2022-03-23 02:45:01 +00:00
xiefangqi f81c7967ec fix config interface comments 2022-03-22 10:04:12 +08:00
jonyguo 11079eabbb fix: code check 2022-03-17 08:23:12 +08:00
i-robot 1d6834ce4d
!31334 [MD][Autotune] Modify the generated autotune config to improve readability 
Merge pull request !31334 from harshvardhangupta/modify_at_config
 2022-03-16 18:18:57 +00:00
i-robot 72ea797e4d
!30339 [BUG][MD][FUNC]RandomInvert 
Merge pull request !30339 from yangwm/randominvert
 2022-03-16 06:39:53 +00:00
i-robot 8e8391dadb
!30429 BUG][MD][FUNC]RandomAutoContrast 
Merge pull request !30429 from yangwm/autocontrast
 2022-03-16 06:38:54 +00:00
harshvardhangupta accc32da6c modify the at config file 2022-03-15 13:49:00 -04:00
hesham 3d16a4dee2 Fix bug in terminating a failed pipeline 2022-03-15 11:35:28 -04:00
i-robot 313a247ae8
!31178 upgrade Ascend package 11 Mar 22 
Merge pull request !31178 from shenwei41/upgrade_ascend_20220311
 2022-03-15 11:08:32 +00:00
i-robot 3be552ca39
!31156 test centos ut 
Merge pull request !31156 from luoyang/test_centos
 2022-03-15 09:38:15 +00:00
shenwei41 f1fde6a262 upgrade Ascend package 11 Mar 22 2022-03-15 16:31:27 +08:00
i-robot d4fc47ef0c
!31211 Clean iterators before forking 
Merge pull request !31211 from h.farahat/pytest_problem
 2022-03-15 01:47:28 +00:00
i-robot 6be7c42052
!31191 [MD] more tests for failover reset 
Merge pull request !31191 from Mohammad Motallebi/failover_more_tests
 2022-03-14 21:50:37 +00:00
h.farahat 318e6b8803 fix segfault 2022-03-14 14:55:21 -04:00
mohammad 583eb32723 [MD] Add more UT and a ST for failover reset 2022-03-14 13:52:09 -04:00
luoyang 74d8969b91 ut failed with pillow>=9.0.0, need more validation 2022-03-14 23:58:09 +08:00
i-robot 8b7d7a1e1a
!31141 [MD][Offload] Offload RandomSharpness Op Pure Sharpness Support & UT 
Merge pull request !31141 from alashkari/fix-random-sharpness
 2022-03-11 21:06:13 +00:00
Rescue face5e51a0 Added sharpness without randomness support and sharpnes op ut. 
fix pad issue
 2022-03-11 17:30:49 +00:00
shenyu 443da2280d [FIX][BUG][MD][FUNC] RandomInvert 2022-03-11 11:53:04 +08:00
shenyu c14a8084c7 [FIX][BUG][MD][FUNC] RandomAuoContrast 2022-03-11 10:29:49 +08:00
liyong 50fc6f5736 Revert "!30810 [MD] add OBSMindDataset " 
This reverts commit 9fcab9184e, reversing
changes made to afb33fff62.
 2022-03-10 19:10:11 +08:00
liyong 10ada68fa9 add OBSMindDataset 2022-03-10 11:18:59 +08:00
i-robot 698b99aaa3
!18945 [assistant][ops]New operator implementation, include OmniglotDataset 
Merge pull request !18945 from Wangsong95/omniglot_dataset
 2022-03-09 06:51:15 +00:00
qkeys ce7597f500 [feat][assistant][I3J6VB] add new data operator OmniglotDataset 2022-03-08 21:36:22 +08:00
i-robot 2655d64720
!30865 [MD] Modify Batch number of workers with Python multiprocessing 
Merge pull request !30865 from cathwong/ckw_batch_workers_pymultip
 2022-03-07 18:21:16 +00:00
Cathy Wong bc29dd9d72 [MD] Modify Batch number of workers with Python multiprocessing 2022-03-07 10:39:05 -05:00
xiefangqi 2de6c8908b add a clean up subprocess to clean subprocess when main process exit 2022-03-05 17:24:11 +08:00
i-robot b90cf43562
!30553 Support dataset reset() to recover after failure 
Merge pull request !30553 from h.farahat/reset
 2022-03-04 02:19:45 +00:00
i-robot f01d841113
!27943 [MD][Autotune] Save/Load Autotune config 
Merge pull request !27943 from harshvardhangupta/save_load_at_config
 2022-03-03 20:42:23 +00:00
i-robot 7b26a32e98
!22608 [assistant][ops]New operator implementation, include KITTIDataset 
Merge pull request !22608 from Wangsong95/kitti_dataset
 2022-03-03 07:05:40 +00:00
h.farahat a3dec34833 Dataset failover reset 2022-03-02 23:53:56 -05:00
harshvardhangupta bd61adbb72 Implement save/load of autotune dataset pipeline configuration #27943 2022-03-02 15:08:08 -05:00
i-robot 70e61e9014
!30781 [MD] test_pyfunc_multiprocess_autotune.py - enable AutoTune 
Merge pull request !30781 from cathwong/ckw_ut_fixup_map_python_multiproc
 2022-03-02 19:50:29 +00:00
i-robot edcc6b790d
!30714 [MD][Offload] Add TypeCast op to offload 
Merge pull request !30714 from markuskunej/offload_typecast
 2022-03-02 19:48:58 +00:00
Cathy Wong eb931c7f88 [MD] test_pyfunc_multiprocess_autotune.py - enable AutoTune 2022-03-02 13:25:35 -05:00
zx 93617ce91e [feat][assistant][I3J6VO] add new data operator KITTI 2022-03-02 19:29:17 +08:00
Cathy Wong b151f90b8d [MD] Add tests for Python Multiprocessing with AutoTune 2022-03-01 20:20:59 -05:00
markuskunej 6de37045ff Added TypeCast op in dataset offload. 2022-03-01 21:55:27 +00:00
i-robot a8686ae3d9
!18827 [assistant][ops]New operator implementation, include LFWDataset 
Merge pull request !18827 from Wangsong95/lfw_dataset
 2022-02-28 08:31:11 +00:00
zx 2132f62d98 [feat][assistant][I3J6VQ] add new data operator LFW 2022-02-28 13:43:05 +08:00
danishfarid 6c4697fc8b reenable AT for non-sink 
tests re-enable

code check fix 1

remove self

lint fixing

lint fix 2

remove test as per req
 2022-02-25 00:02:58 +00:00
i-robot 2a00ffd3b1
!30259 [MD]Update set_autotune_enable API to add save filepath 
Merge pull request !30259 from cathwong/ckw_at_save_api
 2022-02-24 01:00:59 +00:00
Cathy Wong 46e223e569 [MD] Update set_autotune_enable API to add save filepath 2022-02-23 14:50:55 -05:00
i-robot eeb731ae3e
!18738 [assistant][ops]New operator implementation, include LSUNDataset 
Merge pull request !18738 from Wangsong95/lsun_dataset
 2022-02-23 02:28:40 +00:00
i-robot abb89d3f06
!29705 [MD][Offload] Support Multi-Column Datasets and Map Column Names to Index 
Merge pull request !29705 from markuskunej/offload_multi_col_ds
 2022-02-22 22:48:38 +00:00
liyong 3e1c668247 fix mnist dataset 2022-02-22 15:25:13 +08:00
markuskunej 87a45b239b Add offload support for multi-column datasets and map offload operations 
to specific columns.
 2022-02-18 20:09:49 +00:00
zhixinaa 4ce6104ff9 [feat][assistant][I3J6V9] add new data operator LSUN 2022-02-18 15:40:32 +08:00
i-robot c4c5a1c6da
!29862 MD Autotune supporting MindRecordOp 
Merge pull request !29862 from Mohammad Motallebi/autotune_addworker_mindrecord
 2022-02-16 23:34:11 +00:00
mohammad ddd3b07648 MD autotune supports mindrecord 2022-02-16 14:00:31 -05:00
Cathy Wong 6c5dbf937f [MD] Disable shared memory in Python Multiprocessing UTs for CI 2022-02-15 11:00:44 -05:00
i-robot c45e8b5c9b
!29797 [MD]Enhance Python Multiprocessing test coverage 
Merge pull request !29797 from cathwong/ckw_ut_python_multiproc
 2022-02-14 18:41:22 +00:00
Cathy Wong befe8f4837 [MD]Enhance Python Multiprocessing test coverage 2022-02-11 12:31:21 -05:00
i-robot 5858a06689
!29700 [MD] Tidy up UT and device queue info messages 
Merge pull request !29700 from cathwong/ckw_device_queue_probes
 2022-02-09 12:47:27 +00:00
Cathy Wong fb469f1ad2 [MD] Tidy up UT and device queue info messages 2022-02-08 17:51:11 -05:00
许志杰 ab9a32ad0d [feat] [assistant] [#I40GYE] add new loader SQuADDataset 2022-02-08 03:00:09 -08:00
luoyang c248274d91 fix vocab validation 2022-02-07 15:55:59 +08:00
i-robot a1b5f06407
!29535 [MD] Profiler Memory Tooling Support 
Merge pull request !29535 from cathwong/ckw_memory_tool
 2022-02-01 21:39:05 +00:00
Cathy Wong 3fe9a05341 [MD] Profiler Memory Tool Support 2022-01-31 17:46:17 -05:00
i-robot e12d6af257
!29358 MD Child Sampler Code and UT Issues 
Merge pull request !29358 from zetongzhao/sampler_AddChild
 2022-01-31 16:55:22 +00:00
zetongzhao ea9783e18c fix sampler add_child 2022-01-31 07:45:42 -05:00
i-robot 02d63d7ad9
!23008 [assistant][ops][FilterWikipediaXML] 
Merge pull request !23008 from Isaac/FilterWikipediaXML
 2022-01-29 08:35:29 +00:00
tophand 797cd52c96 [feat][assistant][I40GXK]add new data operator Filter_Wikipedia_XML 2022-01-28 02:26:20 -08:00
QRICK 52c73ff1b3 [assistant][ops][I3T96S] Add new data operator MaskAlongAxis 2022-01-28 14:53:38 +08:00
QRICK 9ed73eae94 [assistant][ops][I3T96P] Add new data operator MaskAlongAxisIID 2022-01-27 12:20:26 +08:00
i-robot 2b82ba1e49
!26244 [assistant][ops] Add new data PhaseVocoder 
Merge pull request !26244 from TR-nbu/PhaseVocoder
 2022-01-26 06:46:36 +00:00
donrichnx eee07047a0 [feat] [assistant] [I40GYJ] add Captioning task of CocoDataset 2022-01-24 17:24:16 +08:00
QRICK 859f90cb85 [assistant][ops][I3T96R] Add new data operator PhaseVocoder 2022-01-24 16:00:24 +08:00
i-robot 85ac17a8c6
!22736 [assistant][ops] Add new dataset operator LibriTTSDataset 
Merge pull request !22736 from TR-nbu/LibriTTSDataset
 2022-01-22 10:03:40 +00:00
ZJM 8d339ccfea [assistant][ops][22736] Add new dataset operator LibriTTSDataset 2022-01-21 17:34:42 +08:00
chenx2ovo bbba4fb940 [feat][assistant][I3CEG0] add op melscale 2022-01-21 11:30:34 +08:00
ZJM fff47f3886 [assistant][ops][I3T96T] Add new dataset operator CMUArcticDataset 2022-01-20 17:40:35 +08:00
i-robot 501614a61f
!22514 [assistant][ops] Add new data operator Multi30kDataset 
Merge pull request !22514 from 杨旭华/Multi30kDataset
 2022-01-20 02:13:17 +00:00
i-robot f4bd87380b
!29195 Update API docs for Audio and Callback to the previous modifications 
Merge pull request !29195 from xiaotianci/fix_chinese_api
 2022-01-20 01:17:02 +00:00
unknown c4ee548905 [feat][assistant][I40GY6]add new data operator Multi30kDataset 2022-01-19 18:33:13 +08:00
Xiao Tianci 1d628bfb85 Fix API docs of Audio and Callback 2022-01-19 15:23:50 +08:00
ZJM 21973bbbec [assistant][ops][I3T96W] Add new dataset operator GTZANDataset 2022-01-19 13:48:50 +08:00
1462492739@qq.com bf5f24d62f [assistant][ops][I3T96S] Add new data operator CreateFbMatrix 2022-01-14 21:24:17 +08:00
i-robot cfe5669de9
!28975 [MD] Chained Sampler Python UT fixes and updates 
Merge pull request !28975 from cathwong/ckw_ut_sampler_chain_fixes2
 2022-01-14 03:49:05 +00:00
Cathy Wong 9d9013ba79 [MD] Chained Sampler UT fixes and updates 2022-01-13 17:39:33 -05:00
luoyang e5f2c75233 fix minddata api doc 2022-01-13 16:39:08 +08:00
markuskunej 54e584c300 Added input column validation for offloaded map operations. 
Moved check_add_offload from dataset_helper.py to offload.py.
 2022-01-12 16:16:49 +00:00
i-robot fc4abc99f5
!28595 Fix some code review probelms 
Merge pull request !28595 from xiaotianci/update_code_review
 2022-01-12 09:30:28 +00:00
i-robot 5c961f9fb2
!28812 [MD UT] Misc AutoTune and Profiling UT updates 
Merge pull request !28812 from cathwong/ckw_prof_misc
 2022-01-12 03:03:13 +00:00
Cathy Wong 12b4e81f55 MD UT: Misc AutoTune and Profiling UT updates 2022-01-11 11:56:06 -05:00
Xiao Tianci 96a2e38d01 update review problems 2022-01-11 15:31:53 +08:00
i-robot b073af4b5d
!28559 [MD][Offload] Added dimension check to required Offload ops. 
Merge pull request !28559 from alashkari/offload-fix
 2022-01-05 07:38:31 +00:00
i-robot a1720c210d
!22348 [assistant][ops] Add new loader EnWik9Dataset 
Merge pull request !22348 from 杨旭华/EnWik9Dataset
 2022-01-05 07:36:48 +00:00
Rescue fac68dbb46 Added dimension check to required Offload ops. 2022-01-05 04:26:31 +00:00
i-robot de030c8d41
!28204 Improved HwcToChw support multi-channel input. 
Merge pull request !28204 from unseenme/mindcon_coding_hwc2chw
 2022-01-04 11:50:04 +00:00
unseenme 755cb0adfa Improved HwcToChw support multi-channel input. 2022-01-04 15:47:04 +09:00
Eden 63ab65e3a4 [feat] [assistant] [#I40GYF] add new loader EnWik9Dataset 2022-01-03 21:45:43 -08:00
vddong 7449b8364b [feat][assistant][I40GYQ] add new dataset operator WIDERFace 2021-12-31 13:49:47 +08:00
jingyangxiang a7f8b3af1c [feat][assistant][ops]New operator implementation, include IMDBDataset 
Signed-off-by: zjuter0126 <977180923@qq.com>
 2021-12-30 11:16:23 +08:00
zjuter0126 3c2ca41886 [feat][assistant][ops] add new data operator WikiTextDataset 2021-12-29 18:54:54 +08:00
i-robot ac9e545dc8 !22546 add new dataset loading operator CaltechDataset 
Merge pull request !22546 from 杨旭华/CaltechDataset
 2021-12-29 03:40:06 +00:00
donrichnx 9a0fc0ffc1 [feat] [assistant] [I40GYH] add new dataset loading operator CaltechDataset 2021-12-28 19:04:18 +08:00
i-robot 67e843ec7f !27487 [assistant] [ops][SpectralCentroid] 
Merge pull request !27487 from 杨旭华/SpectralCentroidOp
 2021-12-28 09:10:32 +00:00
i-robot 7faf774063 !22310 [assistant][ops] Add new loader SemeionDataset 
Merge pull request !22310 from 杨旭华/SemeionDataset
 2021-12-28 03:13:02 +00:00
i-robot bff8d6ae01 !28247 Fix Python Vocab validation 
Merge pull request !28247 from luoyang/vocab
 2021-12-28 02:58:25 +00:00
luqilin 896f10d6d5 [feat][assistant][I40GYK] add new dataset loading operator SemeionDataset 2021-12-27 19:49:06 +08:00
tongyue 9a1dcd60fc [fix] [assistant] [I40GXI] fix bug in the infer function of the operator SpectralCentroid 2021-12-27 01:41:31 -08:00
i-robot 69a6c7db98 !23380 [assistant][ops] Add new loader AmazonReview 
Merge pull request !23380 from 杨旭华/AmazonReview
 2021-12-27 09:36:32 +00:00
luoyang ca3de66559 Fix Python Vocab validation 2021-12-27 17:12:07 +08:00
luoyang 0288480722 Python Vocab refactoring and add ids_to_tokens & tokens_to_ids 2021-12-24 15:26:15 +08:00
chengchuanxing 07aa66ee43 [feat][assistant][I40GY2] add new loader AmazonReview 2021-12-23 16:32:31 +08:00
ms_yan 192983b5da check operator mixed or not 2021-12-23 10:22:03 +08:00
fattycat 289a7997df [feat] [assistant] [#I40GYB] add new loader CoNLL2000ChunkingDataset 2021-12-21 15:31:25 +08:00
i-robot f007b8a99f !27745 [MD][Offload] Move offload checks and split to the C pipeline 
Merge pull request !27745 from markuskunej/offload_column_pipeline
 2021-12-21 04:26:51 +00:00
i-robot 7331629ceb !22686 [assistant][ops][Dither] 
Merge pull request !22686 from Isaac/Dither
 2021-12-20 11:16:20 +00:00
haohaonannan97 f480be7c48 [feat][assistant][I40GXF] add new data operator Dither 2021-12-18 08:24:48 -08:00
王天浩 11cb3e4ef7 [fear][assistant][I40GYA] add new loader UDPOSDataset 2021-12-18 23:28:05 +08:00
markuskunej 0c10f52eb4 Moved the offload operation check and map operation split from python (datasets.py) to c++ (NodeOffloadPass.cc). 
Updated check for offload in python (dataset_helper.py and
iterators.py).

Updated auto_offload test in test_map_offload.py.

Changed MapNode.offload type from bool->int in bindings.cc.

Changed manual offload flag from int->enum.

Update offload end of pipeline detection to be column-specific.
 2021-12-17 14:35:17 +00:00
Carry955 2320bb7c70 [feat][assistant][I40GXW] add new loader YahooAnswers 2021-12-16 18:33:47 -08:00
i-robot d1dbdeffcb !27684 Update Python UT removal of generated files 
Merge pull request !27684 from zetongzhao/update_python_remove
 2021-12-17 01:09:17 +00:00
zetongzhao 7cb6989efd replace python removal with pytest tmp_path 2021-12-16 14:38:49 -05:00
i-robot 05336b03c5 !22332 [assistant][ops][SogouNews] 
Merge pull request !22332 from 杨旭华/SogouNews
 2021-12-16 01:07:21 +00:00
i-robot f6fe56c20d !27447 [MD][Offload] Fix Offload Normalize op bug, Added extra tests to Offload UT. 
Merge pull request !27447 from alashkari/offload-fix
 2021-12-15 16:00:26 +00:00
li-qiyao f4bfcb8c25 [feat][assistant][I40GXN] add new loader SogouNews 2021-12-14 22:53:38 -08:00
i-robot 83e9fe5c48 !27542 dataset: change dataset_file into dataset_files for MindDataset 
Merge pull request !27542 from ms_yan/datasetfiles_name
 2021-12-15 03:09:33 +00:00
i-robot dac636decd !22297 [assistant][ops] Add new loader IWSLTDataset 
Merge pull request !22297 from 杨旭华/IWSLTDataset
 2021-12-15 02:55:08 +00:00
Rescue dc5c388505 Fixed offload Normalize op, added offload Normalize and Rescale tests to offload UT. 2021-12-14 17:03:11 +00:00
ms_yan a85113c093 change dataset_file into dataset_files for MIndDataset 2021-12-14 10:52:41 +08:00
liannai a99b5461bd [feat][assistant][I40GXH]add new data operator Gain 2021-12-13 22:03:09 +08:00
lufei9026 30bc609ce6 [feat][assistant][I40GY8]add new loader IWSLTDataset 2021-12-13 21:47:55 +08:00
i-robot 2a2cd5e4ca !22222 [assistant][ops] Add new loader PennTreebankDataset 
Merge pull request !22222 from 杨旭华/PennTreebankDataset
 2021-12-13 11:43:00 +00:00
Megalomania 259854e040 [feat] [assistant] [I40GY5] add new loader PennTreebankDataset 2021-12-08 01:22:40 -08:00
xiefangqi bae1dd3bb0 Revert "Moved the offload operation check and map operation split from python (datasets.py) to c++ (NodeOffloadPass.cc)." 
This reverts commit 84d38e547a.
 2021-12-08 16:39:10 +08:00
i-robot 895256f69a !20043 [assistant][ops] Add new YelpReview 
Merge pull request !20043 from 杨旭华/YelpReview
 2021-12-08 08:13:34 +00:00
i-robot 75e35bded2 !27081 [MD][Offload] Move offload checks and split to the C pipeline 
Merge pull request !27081 from markuskunej/move_offload_check_to_c
 2021-12-07 15:55:16 +00:00
lufei9026 2947f418b1 [feat][assistant][I3T96N]add new loader YelpReview 2021-12-07 17:09:41 +08:00
i-robot 7a8e35b4f1 !26456 [assistant][ops]Add new operator Spectrogram 
Merge pull request !26456 from YJfuel123/Spectrogram
 2021-12-07 06:39:01 +00:00
RainWang6188 c4063f0f11 [feat][assistant][I3J6VI] add new data operator STL10 2021-12-07 01:21:11 +00:00
i-robot b1eae16a05 !22304 [assistant][ops]New operator implementation, include SVHNDataset 
Merge pull request !22304 from 王睿/svhn_dataset
 2021-12-06 12:30:13 +00:00
郑昊 4ae41350a6 [feat][assistant][I3ALD1] add new method Spectrogram 2021-12-06 19:24:37 +08:00
zhangyuqwer e934381b08 [feat][assistant][I3ZSQV] add new data operator CharNGram 2021-12-04 22:09:34 +08:00
RainWang6188 340b3b4d5d [feat][assistant][I3J6VJ] add new data operator SVHN 2021-12-04 08:19:47 +00:00
mohammad af6e587c28 MD Profiler: support multiple start/stop calls 2021-12-03 23:25:38 -05:00
danishfarid faaa5d5f0a autotune step 
fix testing script default

updates and changed sampling function

comments addressed

comments addressed

comments addressed

moving isSinkCheckAbove

update - move CheckSinkFunction

fix
 2021-12-03 20:29:35 +00:00
markuskunej 84d38e547a Moved the offload operation check and map operation split from python (datasets.py) to c++ (NodeOffloadPass.cc). 
Updated check for offload in python (dataset_helper.py and
iterators.py).

Updated auto_offload test in test_map_offload.py.

Changed MapNode.offload type from bool->int in bindings.cc.

Changed manual offload flag from int->enum.
 2021-12-03 14:40:01 +00:00
luoyang f3e1794a74 update log 2021-12-03 15:09:36 +08:00
i-robot 93db7f741f !20788 [assistant][ops] Add new dataset loading operator TedliumDataset 
Merge pull request !20788 from 杨旭华/TedliumDataset
 2021-12-03 06:29:29 +00:00
i-robot 9cec03e3a8 !27033 close autotune in non-sink mode 
Merge pull request !27033 from luoyang/close_autotune_nonsink
 2021-12-03 01:26:06 +00:00
luoyang 9cbb45bcc4 close autotune in non-sink mode 2021-12-03 08:46:14 +08:00
i-robot 9ddd3fb66f !27085 [MD][UT]: Use pytest.mark.forked for profiling tests 
Merge pull request !27085 from cathwong/ckw_ut_prof_forked
 2021-12-02 17:00:28 +00:00
Cathy Wong 367e5ed1dd MD UT: Use pytest.mark.forked for profiling tests 2021-12-02 10:55:05 -05:00
i-robot fa457b2bfa !17832 [assistant][ops]New operator implementation, include KMnistDataset 
Merge pull request !17832 from 张璇/kmnist_dataset
 2021-12-02 14:00:36 +00:00
i-robot 3e73e124ac !23225 [assistant][ops] Add Glove 
Merge pull request !23225 from 张渝/Glove
 2021-12-02 12:22:17 +00:00
i-robot eab0699c6d !26986 change MD error log 
Merge pull request !26986 from shenwei41/master
 2021-12-02 11:35:07 +00:00
i-robot 53f50c0d6b !26880 [MD] fix mindrecord log message part02 
Merge pull request !26880 from liyong126/fix_mindrecord_log_msg_2
 2021-12-02 06:48:15 +00:00
i-robot 3bff079969 !26896 Enhance error message of audio ops 
Merge pull request !26896 from xiaotianci/enhance_audio_err_message
 2021-12-02 03:42:35 +00:00
liyong d35e003def fix mindrecord log part02 2021-12-02 11:28:58 +08:00
luqilin 34bffbf768 [feat][assistant][I3T96H] add new dataset loading operator TedliumDataset 2021-12-02 09:53:39 +08:00
zy 8f9a09f79c [fix] [assistant] [I3ZSQR] add new data operator GloVe 2021-12-01 17:12:20 +08:00
shenwei41 5d48409395 change err info 2021-12-01 16:48:14 +08:00
i-robot 6924866a87 !26746 [MD] minddata error msg optimize 
Merge pull request !26746 from xiefangqi/md_datasetops_error_msg_opt
 2021-12-01 07:18:18 +00:00
zy 5ca98a494a [fix] [assistant] [I3ZSQS] add new data operator FastText 2021-12-01 11:12:08 +08:00
“uccInf” 92d6d31750 [feat][assistant][I3J6V7] add new data operator KMnist 2021-12-01 03:05:05 +00:00
i-robot eb9537af11 !21690 [assistant][ops] Add data operator Vectors 
Merge pull request !21690 from 张渝/Vectors
 2021-12-01 01:29:45 +00:00
Xiao Tianci b883498c93 enhance error message of audio ops 2021-12-01 09:27:04 +08:00
i-robot 1905b228b4 !26609 Update minddata log message 
Merge pull request !26609 from luoyang/log
 2021-12-01 01:25:11 +00:00
i-robot bf59d75604 !26611 [MD] fix mindrecord log message part01 
Merge pull request !26611 from liyong126/fix_mindrecord_log_msg
 2021-12-01 01:12:48 +00:00
xiefangqi f74b5cea0c optimize datasetops error msg 2021-12-01 09:11:22 +08:00
i-robot 2fd00935ed !20964 [assistant][ops] Add new data operator SpeechCommandsDataset 
Merge pull request !20964 from 杨旭华/SpeechCommandsDataset
 2021-11-30 09:33:03 +00:00
luoyang 1a6a42c083 Update minddata log message 2021-11-30 15:22:40 +08:00
zhangyuqwer 6a6e73ef82 [fix] [assistant] [I3ZSQM] add new data operator Vectors 2021-11-30 14:39:42 +08:00
liyong 2869352cc8 fix mindrecord log part01 2021-11-30 10:57:58 +08:00
vddong 91d3a7543f [feat][assistant][I3T96F] add new data operator SpeechCommands 2021-11-27 11:25:18 +08:00
i-robot d8fbc26419 !26700 [MD][Profiler][Autotune] Add test, logs, clean state 
Merge pull request !26700 from harshvardhangupta/add_test_at_prof
 2021-11-25 19:38:33 +00:00
harshvardhangupta 02f4a8068b add tests, clean states, print proper warn logs 2021-11-25 00:03:38 -05:00
jonyguo 58c9112c2e fix: generator dataset with source zip 2021-11-25 11:32:28 +08:00
i-robot 4f34c32c6d !26235 AutoTune v1 
Merge pull request !26235 from danishfarid/autotune
 2021-11-19 23:40:40 +00:00
Rescue da986710f0 Updated concat-zip check and UT. 2021-11-19 17:38:16 +00:00
hesham 40797c40da AutoTune 2021-11-19 09:27:12 -05:00
i-robot 70363899e7 !24080 [assistant][ops] Add new audio operator ComputeDeltas 
Merge pull request !24080 from Rainfor/compute_detlas
 2021-11-19 12:29:49 +00:00
i-robot c6821bde0a !21648 [assistant][ops] Add new data loading operator YesNoDataset 
Merge pull request !21648 from 杨旭华/YesNoDataset
 2021-11-19 09:05:07 +00:00
vddong f465cd0242 [feat][assistant][I3T96L]add new dataset operator YesNo 2021-11-19 13:36:24 +08:00
i-robot f38df5c888 !26234 New dataset config option, auto_offload 
Merge pull request !26234 from markuskunej/offload_config
 2021-11-19 03:44:56 +00:00
i-robot 3c39afad11 !24547 [MS][crowdfunding]New operator implementation, AutoAugment 
Merge pull request !24547 from yangwm/autoaugment
 2021-11-19 01:00:51 +00:00
markuskunej 29cf442521 Added dataset.config.set_auto_offload(), which will automatically attempt to offload dataset map operations if set to True. 2021-11-18 21:13:20 +00:00
zhuchuang 4c3931f5c9 [feat] [assistant] [I3CKEE] add new audio operator ComputeDeltas 2021-11-18 17:18:43 +08:00
despicablemme 09897ce7ec [feat] [assistant] [I40GZP] add new data op AutoAugment 2021-11-18 11:38:18 +08:00
Cathy Wong b1094b9d81 MD Profiling - Add Start Stop Python UT 2021-11-17 13:49:43 -05:00
wangkc123 706f9e2cbb [feat] [assistant] [I3CKEH] add new audio operator SlidingWindowCmn 2021-11-16 22:00:09 +08:00
i-robot 58b69a05ee !20576 [assistant][ops] add new dataset loading operator LJSpeechDataset 
Merge pull request !20576 from 杨旭华/LJSpeechDataset
 2021-11-16 07:00:20 +00:00
liyong 9663884db3 fix occasional failed in random sampler 2021-11-16 09:53:38 +08:00
mohammad 5c8ab5f60c add MD Profiler Save() 2021-11-15 13:18:36 -05:00
vddong 6fe2d35ae2 [feat][assistant][I3T96D] add new dataset loading operator LJSpeechDataset 2021-11-14 14:52:15 +08:00
Carry955 0d26c38693 [feat][assistant][I40GXT] add new loader DBpedia 2021-11-11 19:02:22 -08:00
i-robot 1792d5f8a4 !26131 Offload Hardware Acceleration Computing 
Merge pull request !26131 from alashkari/hw_accel_move_copy
 2021-11-11 13:18:07 +00:00
i-robot b910870ecc !19986 [assistant][ops] Add new dataset loading operator AGNEWS 
Merge pull request !19986 from 杨旭华/AGNEWS
 2021-11-11 06:22:25 +00:00
Megalomania c34bbabed9 [feat] [assistant] [I3T96M] add new loader AGNEWS 2021-11-09 23:51:19 -08:00
rescue 22e03ab12e Adding offload hardware accelerator. 2021-11-10 07:03:43 +00:00
i-robot f1f7731fb2 !25964 Fix serdes multithreading bug 
Merge pull request !25964 from zetongzhao/serdes_multithreading
 2021-11-09 15:48:06 +00:00
i-robot d89a39c661 !25851 MD Profiling Start and Stop Support 
Merge pull request !25851 from cathwong/ckw_prof_startstop4
 2021-11-08 16:16:42 +00:00
liyong 79891feadb fix shuffle and num_samples issue 2021-11-08 15:33:01 +08:00
Cathy Wong cdf3126459 MD Profiler Start Stop Support 2021-11-05 19:46:39 -04:00
zetongzhao 5e8377ea5c fix multithreading bug 2021-11-05 16:49:18 -04:00
Isaac 69f57bc5b1 [feat][assistant][I40GXD]add new data operator DBToAmplitude 2021-11-02 23:45:27 +08:00