yystopf
/
mindspore
forked from mindspore-Ecosystem/mindspore
0
0
Fork 0
Code Issues Pull Requests Projects Releases Wiki Activity
45,876 Commits 23 Branches 0 Tags 3.9 GiB
Commit Graph

1152 Commits

Author SHA1 Message Date
Xiao Tianci b883498c93 enhance error message of audio ops 2021-12-01 09:27:04 +08:00
i-robot 1905b228b4 !26609 Update minddata log message 
Merge pull request !26609 from luoyang/log
 2021-12-01 01:25:11 +00:00
i-robot bf59d75604 !26611 [MD] fix mindrecord log message part01 
Merge pull request !26611 from liyong126/fix_mindrecord_log_msg
 2021-12-01 01:12:48 +00:00
xiefangqi f74b5cea0c optimize datasetops error msg 2021-12-01 09:11:22 +08:00
i-robot 2fd00935ed !20964 [assistant][ops] Add new data operator SpeechCommandsDataset 
Merge pull request !20964 from 杨旭华/SpeechCommandsDataset
 2021-11-30 09:33:03 +00:00
luoyang 1a6a42c083 Update minddata log message 2021-11-30 15:22:40 +08:00
zhangyuqwer 6a6e73ef82 [fix] [assistant] [I3ZSQM] add new data operator Vectors 2021-11-30 14:39:42 +08:00
liyong 2869352cc8 fix mindrecord log part01 2021-11-30 10:57:58 +08:00
vddong 91d3a7543f [feat][assistant][I3T96F] add new data operator SpeechCommands 2021-11-27 11:25:18 +08:00
i-robot d8fbc26419 !26700 [MD][Profiler][Autotune] Add test, logs, clean state 
Merge pull request !26700 from harshvardhangupta/add_test_at_prof
 2021-11-25 19:38:33 +00:00
harshvardhangupta 02f4a8068b add tests, clean states, print proper warn logs 2021-11-25 00:03:38 -05:00
jonyguo 58c9112c2e fix: generator dataset with source zip 2021-11-25 11:32:28 +08:00
i-robot 4f34c32c6d !26235 AutoTune v1 
Merge pull request !26235 from danishfarid/autotune
 2021-11-19 23:40:40 +00:00
Rescue da986710f0 Updated concat-zip check and UT. 2021-11-19 17:38:16 +00:00
hesham 40797c40da AutoTune 2021-11-19 09:27:12 -05:00
i-robot 70363899e7 !24080 [assistant][ops] Add new audio operator ComputeDeltas 
Merge pull request !24080 from Rainfor/compute_detlas
 2021-11-19 12:29:49 +00:00
i-robot c6821bde0a !21648 [assistant][ops] Add new data loading operator YesNoDataset 
Merge pull request !21648 from 杨旭华/YesNoDataset
 2021-11-19 09:05:07 +00:00
vddong f465cd0242 [feat][assistant][I3T96L]add new dataset operator YesNo 2021-11-19 13:36:24 +08:00
i-robot f38df5c888 !26234 New dataset config option, auto_offload 
Merge pull request !26234 from markuskunej/offload_config
 2021-11-19 03:44:56 +00:00
i-robot 3c39afad11 !24547 [MS][crowdfunding]New operator implementation, AutoAugment 
Merge pull request !24547 from yangwm/autoaugment
 2021-11-19 01:00:51 +00:00
markuskunej 29cf442521 Added dataset.config.set_auto_offload(), which will automatically attempt to offload dataset map operations if set to True. 2021-11-18 21:13:20 +00:00
zhuchuang 4c3931f5c9 [feat] [assistant] [I3CKEE] add new audio operator ComputeDeltas 2021-11-18 17:18:43 +08:00
despicablemme 09897ce7ec [feat] [assistant] [I40GZP] add new data op AutoAugment 2021-11-18 11:38:18 +08:00
Cathy Wong b1094b9d81 MD Profiling - Add Start Stop Python UT 2021-11-17 13:49:43 -05:00
wangkc123 706f9e2cbb [feat] [assistant] [I3CKEH] add new audio operator SlidingWindowCmn 2021-11-16 22:00:09 +08:00
i-robot 58b69a05ee !20576 [assistant][ops] add new dataset loading operator LJSpeechDataset 
Merge pull request !20576 from 杨旭华/LJSpeechDataset
 2021-11-16 07:00:20 +00:00
liyong 9663884db3 fix occasional failed in random sampler 2021-11-16 09:53:38 +08:00
mohammad 5c8ab5f60c add MD Profiler Save() 2021-11-15 13:18:36 -05:00
vddong 6fe2d35ae2 [feat][assistant][I3T96D] add new dataset loading operator LJSpeechDataset 2021-11-14 14:52:15 +08:00
Carry955 0d26c38693 [feat][assistant][I40GXT] add new loader DBpedia 2021-11-11 19:02:22 -08:00
i-robot 1792d5f8a4 !26131 Offload Hardware Acceleration Computing 
Merge pull request !26131 from alashkari/hw_accel_move_copy
 2021-11-11 13:18:07 +00:00
i-robot b910870ecc !19986 [assistant][ops] Add new dataset loading operator AGNEWS 
Merge pull request !19986 from 杨旭华/AGNEWS
 2021-11-11 06:22:25 +00:00
Megalomania c34bbabed9 [feat] [assistant] [I3T96M] add new loader AGNEWS 2021-11-09 23:51:19 -08:00
rescue 22e03ab12e Adding offload hardware accelerator. 2021-11-10 07:03:43 +00:00
i-robot f1f7731fb2 !25964 Fix serdes multithreading bug 
Merge pull request !25964 from zetongzhao/serdes_multithreading
 2021-11-09 15:48:06 +00:00
i-robot d89a39c661 !25851 MD Profiling Start and Stop Support 
Merge pull request !25851 from cathwong/ckw_prof_startstop4
 2021-11-08 16:16:42 +00:00
liyong 79891feadb fix shuffle and num_samples issue 2021-11-08 15:33:01 +08:00
Cathy Wong cdf3126459 MD Profiler Start Stop Support 2021-11-05 19:46:39 -04:00
zetongzhao 5e8377ea5c fix multithreading bug 2021-11-05 16:49:18 -04:00
Isaac 69f57bc5b1 [feat][assistant][I40GXD]add new data operator DBToAmplitude 2021-11-02 23:45:27 +08:00
luoyang beba798e00 fix minddata ut core 2021-10-29 16:28:22 +08:00
i-robot 958c2a8e00 !25560 Support bytes input for decode op 
Merge pull request !25560 from luoyang/serving
 2021-10-29 06:47:02 +00:00
liyong b9bad0bdbb fix md ut part06 2021-10-29 10:26:34 +08:00
luoyang b47ac46e92 Support bytes input for decode op 2021-10-28 15:22:50 +08:00
i-robot 74977e548d !24624 [Autotune] Modify MD callback and AddWorker logic to ParallelOps 
Merge pull request !24624 from h.farahat/add_worker
 2021-10-27 00:41:20 +00:00
i-robot c0f6286fcb !25092 [MD] API for autotune 
Merge pull request !25092 from harshvardhangupta/auto_tune_api
 2021-10-26 18:14:43 +00:00
Cathy Wong e66aba9ef5 MD UT - Add missing set and restore seed. 
test_places365_content_check - delete visualize_dataset
 2021-10-26 13:19:47 -04:00
harshvardhangupta 1bb45142c9 Implement autotune api 2021-10-26 11:26:13 -04:00
i-robot c544af28b6 !25286 dataset: modify profiling ut for parallel running 
Merge pull request !25286 from ms_yan/ut_parallel
 2021-10-22 08:09:03 +00:00
ms_yan bb314ad07f make ut parallel 2021-10-22 13:17:43 +08:00
i-robot cf0029b33a !25277 [MD] fix mindrecord ut part05 
Merge pull request !25277 from liyong126/fix_md_ut_05
 2021-10-22 01:43:03 +00:00
i-robot 782874bf18 !21215 [MS][crowdfunding]New operator implementation, include phaser 
Merge pull request !21215 from Wangsong95/phaser
 2021-10-22 01:08:50 +00:00
hesham 85e077b60b Callback changes 
AddWorker logic to parallel ops
 2021-10-21 17:35:11 -04:00
liyong 7c9f6899a3 fix md ut part05 2021-10-21 20:40:09 +08:00
i-robot b7d8bcfe8d !25177 [MD] fix mindrecord ut part04 
Merge pull request !25177 from liyong126/fix_md_ut_part04
 2021-10-21 01:39:26 +00:00
i-robot 5d8be044c9 !18018 [assistant][ops]New operator implementation, include FashionMnistDataset 
Merge pull request !18018 from 张璇/fashionmnist_dataset
 2021-10-20 06:42:54 +00:00
i-robot f9bd640b09 !16269 [assistant][RandomLighting] 
Merge pull request !16269 from QingfengLi/randomlighting
 2021-10-20 06:42:12 +00:00
liyong fc5ea9cc1c fix dataset ut for running independently 2021-10-20 14:27:36 +08:00
“uccInf” fe3123cfe8 [feat][assistant][I3J6V4] add new data operator FashionMnist 2021-10-20 02:37:56 +00:00
i-robot 440036d331 !24362 [assistant][ops]New operator implementation, include PhotoTourDataset 
Merge pull request !24362 from Wangsong95/phototour_dataset
 2021-10-20 01:46:05 +00:00
yangyueren 6594b90a58 [feat][assistant][I3J6VE] add new data operator PhotoTour 2021-10-19 15:31:32 +00:00
chenx2ovo 77caf907c4 [feat][assistant][I3CEGR] add op random lighting 2021-10-19 20:38:01 +08:00
liyong a89741c32e fix md ut part03 2021-10-19 16:52:57 +08:00
i-robot 690642d672 !18664 [assistant][ops]New operator implementation, include Places365Dataset 
Merge pull request !18664 from Wangsong95/places365_dataset
 2021-10-19 07:06:35 +00:00
i-robot 1606f19dfb !18265 [assistant][ops] Add new audio operator CreateDct 
Merge pull request !18265 from 杨旭华/CreateDct
 2021-10-19 06:30:18 +00:00
tophand 4f4777f0da [feat][assistant][I3J6U2] Add new audio operator CreateDct 2021-10-18 03:23:18 -07:00
liyong d9aa18dfca refactor md ut 2021-10-18 17:55:28 +08:00
zx d13d2413f5 [feat][assistant][I3J6UV] add new audio operator Phaser 2021-10-16 08:45:12 +08:00
yangyueren 951193c308 [feat][assistant][I3J6VF] add new data operator Places365 2021-10-15 14:56:09 +00:00
Xiao Tianci 930a5b89c0 fix api docs of audio 2021-10-15 17:17:18 +08:00
robert_luo_yibo 75e2c3041c [feat][assistant][I3CEG3] add new data OP MuLawEncoding 2021-10-14 20:24:18 +08:00
i-robot 1b9b85bbb2 !17851 [assistant][ops]New operator implementation, include FakeImageDataset 
Merge pull request !17851 from 张璇/fakeimage_dataset
 2021-10-13 09:22:55 +00:00
G-Dragon-Liu d2f22a8726 [feat][assistant][I3J6V3] add new data operator FakeImage 2021-10-12 14:00:43 +00:00
zx 9b7d7be65c [feat][assistant][I3J6UU] add new audio operator Overdrive 2021-10-12 18:10:31 +08:00
li-qiyao 2283442855 [feat][assistant][I3J6U9] add new data operator Flanger 2021-10-08 05:03:31 -07:00
i-robot 8598d08351 !17820 [assistant][ops]New operator implementation, include EMnistDataset 
Merge pull request !17820 from 张璇/emnist_dataset
 2021-10-08 07:18:31 +00:00
G-Dragon-Liu 32baa520bf [feat][assistant][I3J6V2] add new data operator EMnist 2021-10-08 02:53:00 +00:00
Zhang Lian b7e6520e62 [feat] [assistant] [I3J6U8] Add a new audio operator DetectPitchFrequency 2021-09-29 15:12:58 +08:00
i-robot dd544a9693 !21406 [MS][crowdfunding]New operator implementation, include TrebleBiquad 
Merge pull request !21406 from 张璇/treble_biquad
 2021-09-27 14:18:00 +00:00
i-robot f16c003dfa !22610 [MS][crowdfunding]New operator implementation, RandomAdjustSharpness 
Merge pull request !22610 from yangwm/sharpness
 2021-09-27 09:20:05 +00:00
zx 3da0f0f487 [feat][assistant][I3J6V0] add new audio operator TrebleBiquad 2021-09-27 13:01:26 +08:00
despicablemme d072a4e4bf [feat] [assistant] [I40GZE] add new data operator RandomAdjustSharpness 2021-09-27 11:19:33 +08:00
i-robot ab567574c4 !21451 [MS][crowdfunding]New operator implementation, include RiaaBiquad 
Merge pull request !21451 from 张璇/riaa_biquad
 2021-09-27 03:14:16 +00:00
i-robot 2969688382 !20047 [assistant][ops]New operator implementation, include QmnistDataset 
Merge pull request !20047 from Wangsong95/qmnist_dataset
 2021-09-27 01:36:08 +00:00
despicablemme 7063907bdc [feat] [assidtant] [I40GZN] add new data ops RandomEqualize 2021-09-25 18:00:25 +08:00
i-robot d0023355a3 !23205 Fix bus error with multiprocessing enabled, large image, shared_memory, single device 
Merge pull request !23205 from RobinGrosman/mem_bus_failure
 2021-09-24 19:16:12 +00:00
RobinGrosman c90efc0403 add check for shm available when num_devices=1 to avoid bus errors 
-Required modifying tests to reduce memory usage to check doesn't fail during CI
-Also required fixing max_rowsize to be externalized from map+batch() function
 2021-09-24 10:24:31 -07:00
zx 1d95e6f480 [feat][assistant][I3J6UY] add new audio operator RiaaBiquad 2021-09-23 21:36:59 +08:00
despicablemme 7a6b75c2e8 [feat] [assistant] [I40GZJ] add new data op RandomAutoContrast 2021-09-23 20:17:39 +08:00
i-robot 0c26736da1 !22276 [MS][crowdfunding]New operator implementation, RandomInvert 
Merge pull request !22276 from yangwm/rinvert
 2021-09-23 08:12:57 +00:00
hetshah 2c1666fd06 set num epochs for iterators that use the default value 
add explicit paramter for infinite iterators

set appropriate value for num_epoch param for iteratos

fix long lines

remove trailing whitespace

fixed typo

remove comment

fixed comments

fixed test
 2021-09-22 17:32:50 +00:00
despicablemme b977d8723b [feat] [assistant] [I40GZC] add new data ops RandomInvert 2021-09-22 16:55:14 +08:00
ckczzj 001b68c286 [feat][assistant][I3J6VG] add new data operator QMnist 2021-09-22 12:13:27 +08:00
i-robot 6489dacdab !18114 [assistant][ops]Add new operator Magphase 
Merge pull request !18114 from YJfuel123/Magphase
 2021-09-18 07:26:34 +00:00
i-robot 92287b86d8 !23471 Fix white list error msg synax error 
Merge pull request !23471 from xiefangqi/md_white_list_optimization
 2021-09-18 05:03:40 +00:00
葛崇林 a05a5de073 [feat][assistant][I3ALC5] add new method Magphase 2021-09-17 16:02:46 +08:00
i-robot f5af432e04 !16922 [assistant][Vol] 
Merge pull request !16922 from QingfengLi/Vol
 2021-09-17 06:51:49 +00:00
chenx2ovo 9e62623b4a [feat][assistant][I3CEGI] add op vol 2021-09-17 09:34:20 +08:00
ms_yan 0a365e91bf modify log in kernel part 2021-09-16 22:09:21 +08:00
Guo Zhijian f49bacde09 fix: pyfunc hung when it's a yield func 2021-09-16 09:20:44 +08:00
i-robot 57b8c8e2d9 !23310 [MD] fix error message of mindrecord 
Merge pull request !23310 from liyong126/fix_log_msg
 2021-09-16 00:45:11 +00:00
i-robot e5af07d519 !23538 minddata error message optimize 
Merge pull request !23538 from luoyang/log_update
 2021-09-16 00:42:42 +00:00
i-robot 4a553c8d57 !16607 [assistant][Fade] 
Merge pull request !16607 from QingfengLi/Fade
 2021-09-15 14:54:16 +00:00
YangLuo 9d89de5765 minddata error message optimize 2021-09-15 18:05:02 +08:00
liyong 65f3365a2a fix mindrecord log 2021-09-15 16:18:40 +08:00
jonyguo c408867085 fix: hung when user return Tensor in generator 2021-09-15 14:13:33 +08:00
xiefangqi a91c87c581 fix some syntax error 2021-09-14 21:29:02 +08:00
chenx2ovo 1a3196b052 [feat][assistant][I3CEGF] add op fade 2021-09-14 19:08:37 +08:00
i-robot 32c2b77595 !19093 [assistant][ops] Add new audio operator Biquad 
Merge pull request !19093 from 杨旭华/Biquad
 2021-09-14 01:52:01 +00:00
i-robot 6dcfe64501 !23264 python serdes bug fix 
Merge pull request !23264 from zetongzhao/deserialize
 2021-09-13 15:53:05 +00:00
i-robot 0aa167cdf4 !23240 maybe max open files in mr or onehot fail in pytransform 
Merge pull request !23240 from guozhijian/fix_max_open_files_and_onehot
 2021-09-13 13:51:09 +00:00
i-robot 5abd4821ce !23357 Add DatasetName method for out developer 
Merge pull request !23357 from xiefangqi/md_white_list_optimization
 2021-09-13 12:06:49 +00:00
i-robot 3e1d1ff4d3 !18671 [assistant][ops]New operator implementation, include SBUDataset 
Merge pull request !18671 from Wangsong95/sbu_dataset
 2021-09-13 11:56:13 +00:00
liannai 631f49f044 [feat][assistant][I3T96A] Add new audio operator Biquad 2021-09-13 19:18:50 +08:00
xiefangqi 465dd25d74 add virtual DatasetName method 2021-09-13 15:17:32 +08:00
i-robot 87381a1cf9 !17806 [assistant][ops] DcShift Op 
Merge pull request !17806 from 杨旭华/DCShift
 2021-09-13 01:29:14 +00:00
i-robot c9ff7242a4 !23298 minddata engine optimize white list error msg 
Merge pull request !23298 from xiefangqi/md_white_list_optimization
 2021-09-13 01:19:15 +00:00
Carry955 0ba84345b0 [feat][assistant][I3J6U3] add new audio operator DCShift 2021-09-11 23:58:18 -07:00
jonyguo da78a17730 fix: may max openfiles in mindrecord and onehot error in py_transform 
and cache_admin rpath
 2021-09-12 00:25:07 +08:00
xiefangqi ff73e4ccd7 fix white list error msg 2021-09-11 17:36:23 +08:00
Isaac17555 53fc99b914 [feat][assistant][I3T96B]add new data operator LFilter 2021-09-11 02:05:43 -07:00
YJfuel123 16150d394f [feat][assistant][I3ALd1] add new operator EqualizerBiquad 2021-09-11 10:36:20 +08:00
ckczzj af97ba4a77 [feat][assistant][I3J6VH] add new data operator SBU 2021-09-11 09:58:17 +08:00
zetongzhao 719a207040 deserialize fix 2021-09-10 15:19:59 -04:00
i-robot 7b4f361ca7 !23232 Fix multiprocessing attr error 
Merge pull request !23232 from xiefangqi/md_fix_map_pool_issue
 2021-09-10 14:59:40 +00:00
i-robot 6d709c7254 !19624 [assistant][ops] Add new dataset operator DIV2K. 
Merge pull request !19624 from Rainfor/div2k
 2021-09-10 14:59:21 +00:00
xiefangqi 66c5f937dd fix attr failed problem 2021-09-10 18:01:04 +08:00
wangkc123 3daf3a4d5f [feat] [assistant] [I3CKEK] add new dataset operator DIV2K 2021-09-10 17:55:34 +08:00
robert_luo_yibo ce71699734 [feat][assistant][I3CEG5] add new data OP MuLawDecoding 2021-09-10 15:49:46 +08:00
i-robot 32cc5d79b0 !18670 [assistant][ops]New operator implementation, include USPSDataset 
Merge pull request !18670 from Wangsong95/usps_dataset
 2021-09-10 03:06:19 +00:00
i-robot f91e1440c8 !17810 [assistant][ops][DeemphBiquadOp] 
Merge pull request !17810 from Isaac/DeemphBiquad
 2021-09-10 03:05:41 +00:00
ckczzj d7c702e9d4 [feat][assistant][I3J6VL] add new data operator USPS 2021-09-09 21:46:57 +08:00
i-robot 936940da8e !22666 [assistant][ComplexNorm] 
Merge pull request !22666 from QingfengLi/ComplexNorm
 2021-09-09 06:41:53 +00:00
Isaac 5dc6c7fe96 [feat][assistant][I3J6U6]add new data operator DeemphBiquad 2021-09-09 14:34:24 +08:00
i-robot 9ad56e86ab !17846 [assistant][ops]Add new operator HighpassBiquad 
Merge pull request !17846 from YJfuel123/HighpassBiquad
 2021-09-09 02:06:41 +00:00
i-robot a918783f1a !22623 Add Random operation with Multi Field 
Merge pull request !22623 from shenwei41/operation_dev
 2021-09-09 01:49:04 +00:00
i-robot b21da536d3 !22784 serdes python operations by pyfunc 
Merge pull request !22784 from zetongzhao/deserialize
 2021-09-08 17:18:57 +00:00
zetongzhao 0d389f3f0c serdes python operations 2021-09-08 09:05:11 -04:00
shenwei41 a8d02ab782 Add Random operation with MultiField 2021-09-08 19:29:23 +08:00
i-robot 938324b397 !19362 [assistant][ops] Add new dataset operator Cityscapes. 
Merge pull request !19362 from Rainfor/cityscapes
 2021-09-08 09:14:43 +00:00
yaoyuan 85a1b55b47 [feat] [assistant] [I3ALCL] add new operator HighpassBiquad 2021-09-08 15:12:40 +08:00
i-robot b1be8dfd31 !22851 MD Profiling: Update Connector Init to Remove any existing file 
Merge pull request !22851 from cathwong/ckw_mon_seq_pipelines_fix
 2021-09-07 19:57:58 +00:00
xiefangqi 57f68d35bd fix generator 14 ut problem 2021-09-06 19:01:51 +08:00
Cathy Wong bc85c606b8 MD Profiling: Update Connector Init to Remove any existing file 
to fix sequential pipeline scenario.
 2021-09-03 15:45:38 -04:00
i-robot 3368287aab !22620 add util: browse dataset 
Merge pull request !22620 from luoyang/browse_dataset
 2021-09-02 01:00:46 +00:00
robert_luo_yibo 3d58da9023 [fix][assistant][I47V0O] fix bug about the wrong output in the operator ComplexNorm 2021-09-01 17:55:38 +08:00
YangLuo 97c1ba380f add util: browse dataset 2021-09-01 16:13:32 +08:00
wangkc123 70f741bcc1 [feat] [assistant] [I3CKEM] add new dataset operator Cityscapes 2021-08-31 20:03:37 +08:00
tongyue 329526dea8 [feat][assistant][I3J6U0] add new data operator Contrast 2021-08-30 19:34:37 -07:00
i-robot bbf9c6a9ad !22529 fix: for loop dataset iterator will cause memory leak 
Merge pull request !22529 from guozhijian/fix_for_loop_ds_iter_memory_leak
 2021-08-31 01:15:58 +00:00
i-robot a47ab8d643 !18073 [assistant][ops]Add new operator LowpassBiquad 
Merge pull request !18073 from YJfuel123/LowpassBiquad
 2021-08-30 08:33:39 +00:00
jonyguo 4c915fdcbb fix: for loop dataset iterator will cause memory leak 2021-08-30 16:08:26 +08:00
liyong 5a4ff8cfdb fix ci issue 2021-08-30 10:18:06 +08:00
i-robot 1639023909 !22263 Add convertcolor operation 
Merge pull request !22263 from shenwei41/operation_dev
 2021-08-27 12:33:43 +00:00
liyong c257cf36e2 refactor mindrecord 2021-08-27 09:15:05 +08:00
shenwei41 1295744524 Add ConvertColor Operation 2021-08-26 21:14:28 +08:00
haizhouye d96d9e64b0 [feat] [assistant] [I3ALCA] add new LowpassBiquad operator 
[feat] [assistant] [I3ALCA] add new LowpassBiquad operator

bugfix

fix

fix
 2021-08-26 18:49:21 +08:00
i-robot 956ae2bbfd !22223 [assistant][TimeMask and FreqMask] 
Merge pull request !22223 from QingfengLi/TimeFreqMask
 2021-08-25 10:43:24 +00:00
chenx2ovo 2f1d974eeb [fix] [assistant] [I465ZG] revise error message and change input param datatype 2021-08-25 10:47:27 +08:00
zhangyihui 3d19949eb4 device_id to rank_id 2021-08-25 09:52:23 +08:00
i-robot 6082835501 !22146 dataset: support return bytes in GeneratorDataset 
Merge pull request !22146 from ms_yan/decode_bytes
 2021-08-24 02:41:18 +00:00
i-robot db44b88e1e !22094 MD Profiling: For mismatch op info btwn files, skip bottleneck analysis 
Merge pull request !22094 from cathwong/ckw_mon_py_analyze_fixes3
 2021-08-23 18:28:51 +00:00
ms_yan 6ecfc34524 add np.frombuffer to support return bytes 2021-08-23 17:31:38 +08:00
ms_yan 36a8886ca2 Revert "[feat] [assistant] [I3T96T] add new Dataset operator CMUARCTICDataset" 
This reverts commit b077aa1cab.

Revert "[feat] [assistant] [I3T96X] add new Dataset operator LibriSpeechDataset"

This reverts commit 4e6f7dc97d.

delete pass_registry_test.cc

comment  hiai_nlu_model_multi.pb related  line
 2021-08-23 01:46:38 +08:00
djc b077aa1cab [feat] [assistant] [I3T96T] add new Dataset operator CMUARCTICDataset 2021-08-22 16:26:45 +08:00
djc 4e6f7dc97d [feat] [assistant] [I3T96X] add new Dataset operator LibriSpeechDataset 2021-08-22 13:39:37 +08:00
i-robot 0e7844daa9 !22093 revert merge request 21685 
Merge pull request !22093 from guozhijian/revert-merge-21685-master
 2021-08-21 09:25:32 +00:00
i-robot 4ba337e0f5 !16667 [assistant][ComplexNorm] 
Merge pull request !16667 from QingfengLi/ComplexNorm
 2021-08-21 02:44:18 +00:00
i-robot 5d1bb097e2 !22052 Fix probability core dump in MindData coverage test 
Merge pull request !22052 from luoyang/issues2
 2021-08-20 01:18:06 +00:00
robert_luo_yibo d837cbee0f [feat][assistant][I3CEG8] add new data operator ComplexNorm 2021-08-20 09:07:21 +08:00
Cathy Wong 85931175bc MD Profiling: For mismatch op info btwn files, skip bottleneck analysis. 
Add to summary output: per_pipeline_time and per_push_queue_time.
Enhance UT
 2021-08-19 12:19:59 -04:00
guozhijian e1df9a2152 revert 'Pull Request !21685 : add function to invert_op' 2021-08-19 22:49:51 +08:00
i-robot 4113aee7d6 !18668 [assistant][ops]New operator implementation, include Flowers102Dataset 
Merge pull request !18668 from Wangsong95/flowers102_dataset
 2021-08-19 08:41:44 +00:00
YangLuo 9352935ff6 Fix probability core dump in MindData coverage test 2021-08-19 14:34:57 +08:00
i-robot 3ad641cb65 !21685 add function to invert_op 
Merge pull request !21685 from Killer/fake_killer
 2021-08-19 05:13:39 +00:00
fake_killer b0bc4bf5da [feat] [assistant] [I44TRS] extend invert_op's function 2021-08-18 21:46:15 +08:00
Xiao Tianci aae886bcd7 fix crowd contributing code 2021-08-16 19:33:10 +08:00
i-robot f99effd430 !21755 Fix batch usability problem 
Merge pull request !21755 from xiefangqi/md_fix_batch_usability
 2021-08-13 07:21:49 +00:00
i-robot d1112f8e9c !17631 [assistant][FrequencyMasking] 
Merge pull request !17631 from QingfengLi/FreqMask
 2021-08-13 06:48:06 +00:00
xiefangqi e4894a1bf1 fix batch usability problem 2021-08-12 20:12:43 +08:00
chenx2ovo 53615010a3 [feat][assistant][I3CEGG] add new data OP FrequencyMasking 2021-08-12 15:28:30 +08:00
jonyguo 42b3512a1b fix: when field is none in mr 2021-08-11 17:11:38 +08:00
chenx2ovo e72acf362c [feat][assistant][I3CEGH] add op timemasking 2021-08-10 12:06:07 +08:00
i-robot dc626617fc !19526 [assistant][TimeStretch] 
Merge pull request !19526 from QingfengLi/TimeStretch
 2021-08-09 09:40:38 +00:00
shenwei41 59efbb88ec fix waring 2021-08-09 11:40:52 +08:00
chenx2ovo 04705e5b0d [feat][assistant][I3CEGC] add op timestretch 2021-08-07 22:53:41 +08:00
i-robot c84e09cf45 !21072 Deserialize - python binding 
Merge pull request !21072 from zetongzhao/deserialize_1
 2021-08-06 21:47:09 +00:00
zetongzhao 4f199b548c deserialize and tests 2021-08-06 13:35:27 -04:00
i-robot 51a1ca61f2 !21357 [MD] fix SlicePatches when dealing with some special tensor. 
Merge pull request !21357 from liyong126/fix_bug
 2021-08-06 08:58:27 +00:00
liyong 5fea72473c fix HW1 issue 2021-08-06 14:07:26 +08:00
ckczzj 9d9f33be88 [feat][assistant][I3J6VN] add new data operator flowers102 2021-08-06 10:55:12 +08:00
chenx2ovo aa45e2ebbc [feat] [assistant] [I3CEER] add op AmplitudeToDB 2021-08-06 10:07:29 +08:00
Carry955 98be266585 [feat][assistant][I3J6TQ] add a new operator Angle 2021-08-04 19:41:00 -07:00
i-robot 680e40a43a !16453 [assistant][BassBiquadOp] 
Merge pull request !16453 from StyleHang/BassBiquadOp
 2021-08-04 10:02:34 +00:00
i-robot f4e07c0dce !17486 [assistant][ops] Add new dataset operator Flickr. 
Merge pull request !17486 from Rainfor/wangkc
 2021-08-04 08:52:21 +00:00
i-robot d260f38936 !15722 [assistant][AdjustGamma] 
Merge pull request !15722 from QingfengLi/adjust_gamma
 2021-08-04 03:58:09 +00:00
li-qiyao ba326d76a8 [feat][assistant][I3J6TW] add new data operator BassBiquad 2021-08-03 19:40:12 -07:00
wangkc123 48f5fed1f8 [feat] [assistant] [I3CKEN] add new dataset operator Flickr 2021-08-04 10:20:43 +08:00
chenx2ovo 4191dde45f [feat][assistant][I3CEGM] add op adjust gamma 2021-08-04 02:32:13 +08:00
tongyue 08469e113a [feat][assistant][I3J6TV]add new audio operator BandrejectBiquad 2021-08-03 05:41:53 -07:00
i-robot 2c7b8e73df !16452 [assistant][BandpassBiquadOp] 
Merge pull request !16452 from StyleHang/BandpassBiquadOp
 2021-08-03 11:08:48 +00:00
i-robot 93af13f332 !19979 dataset: modify some error msg 
Merge pull request !19979 from ms_yan/err_msg
 2021-08-03 08:53:30 +00:00
i-robot ef77050348 !18892 [assistant][ops] Add new dataset operator SBD. 
Merge pull request !18892 from Rainfor/sbd
 2021-08-03 03:33:00 +00:00
ms_yan 65870045bc modify some error msg 2021-08-03 09:16:28 +08:00
administrator 29d57a89fb [feat][assistant][I3TDRD]add new audio operator BandpassBiquad 2021-08-02 07:08:47 -07:00
Isaac b49e1acc3e [feat] [assistant] [I3J6TM] add new audio operator AllpassBiquad 2021-07-31 16:15:17 +08:00
i-robot 9301de1b54 !21098 fix bug 
Merge pull request !21098 from shenwei41/fix_waring
 2021-07-31 01:12:21 +00:00
shenwei41 e3385e2564 fix waring to master 2021-07-30 16:37:27 +08:00
i-robot 6e84bf0f0f !16449 [assistant][ops] Add new audio operator BandBiquad 
Merge pull request !16449 from StyleHang/BandBiquadOp
 2021-07-30 03:35:07 +00:00
liannai 5a341e7e27 [feat] [assistant] [I3J6TS] add new audio operator BandBiquad 2021-07-29 17:57:27 +08:00
Xiao Tianci 53c01c437d fix NumpySlicesDataset not accept numpy 2021-07-22 15:46:37 +08:00
wangkc123 b381e0be9d [feat] [assistant] [I3CKER] add new dataset operator SBD 2021-07-17 21:02:30 +08:00
i-robot 8508c9eb45 !18179 [MD] add slice patches op in c++ and python 
Merge pull request !18179 from liyong126/slice_patches_op
 2021-07-05 07:04:45 +00:00
liyong 9a1de0af65 add slice patches in python and c++ 2021-07-05 09:16:47 +08:00
Xiao Tianci 8cdd0d8d90 fix param check for Rotate and GaussianBlur 2021-07-01 20:31:33 +08:00
YangLuo 17c121b087 Give concreted error when mappable dataset lack __len__ method 2021-06-29 11:13:12 +08:00
Lixia Chen 1f03b16e0e Fix cache_admin server start stderr output 
Fix TFRecord get_dataset_size() when cached
 2021-06-23 18:32:54 -04:00
Harshvardhan Gupta 6845a16b44 fix coredump with large max_rowsize and add arg validations 2021-06-23 09:31:11 -04:00
YangLuo 4e9f6d54f7 update cpp api & doc 2021-06-23 11:14:52 +08:00
chenx2ovo 4b98099dcb [feat][assistant][I3CEGQ] add op rgb2bgr 2021-06-21 16:16:50 +08:00
i-robot ab3259c43a !18292 MD Profiling UT: Check CPU util file. Add sequential pipeline UTs 
Merge pull request !18292 from cathwong/ckw_mon_py_analyze_ut4
 2021-06-17 21:47:10 +08:00
liyong 2989a5a23b fix some bugs 2021-06-17 11:56:10 +08:00
Cathy Wong c37eb350f6 MD Profiling UT: Check CPU util file. Add sequential pipeline UTs 2021-06-16 15:30:59 -04:00
Xiao Tianci a54e65a7ad add Crop Python API 2021-06-10 09:46:57 +08:00
i-robot e60f5f12f5 !17836 Add Rotate API for CV data processing 
Merge pull request !17836 from xiaotianci/rotate
 2021-06-10 09:29:18 +08:00
i-robot 50f11d5439 !17503 dataset: enhance the expression of error msg 
Merge pull request !17503 from ms_yan/log_modify
 2021-06-08 22:25:04 +08:00
Xiao Tianci 9c4c1c6a77 add Rotate Python API 2021-06-08 22:19:15 +08:00
i-robot 3a63a66d64 !17807 MD Profiling UT: Add MD Analyze UT for MinddataProfilingAnalyzer 
Merge pull request !17807 from cathwong/ckw_mon_py_analyze_ut3
 2021-06-08 21:52:55 +08:00
ms_yan be65ebcb28 complete dataset log 2021-06-08 16:45:47 +08:00
mindspore-ci-bot 8934e4a513 !17879 dynamic min max shape phase 2 
From: @luoyang42
Reviewed-by: @liucunwei,@jonyguo
Signed-off-by: @liucunwei
 2021-06-08 14:49:13 +08:00
Cathy Wong b7342efe12 MD Profiling UT: Add MD Analyze UT. Add more pipeline profiling UTs. 2021-06-07 11:00:34 -04:00
luoyang dad289c679 dynamic min max shape phase 2 2021-06-07 20:06:26 +08:00
Zhenglong Li 2c7d1bf603 Implement new gnn feature 2021-06-07 10:20:06 +08:00
mindspore-ci-bot b52f9ddb08 !17060 Add HorizontalFlip and VerticalFlip for CV data processing 
From: @tiancixiao
Reviewed-by: @liucunwei
Signed-off-by: @liucunwei
 2021-06-02 17:48:28 +08:00
mindspore-ci-bot c4757f0692 !17461 MD Profiling Analyze: Handle cpu util with missing ops. Enhance UTs. 
From: @cathwong
Reviewed-by: @robingrosman
Signed-off-by: @robingrosman
 2021-06-02 11:12:59 +08:00
mindspore-ci-bot ae2e7c1288 !17541 Fix MD Cache code check warnings [master] 
From: @lixiachen
Reviewed-by: @pandoublefeng,@robingrosman
Signed-off-by: @robingrosman
 2021-06-02 04:53:54 +08:00
Lixia Chen 8c1b561149 Fix cache code check warnings 2021-06-01 10:02:38 -04:00
Cathy Wong f8beb933ad MD Profiling Analyze: Handle cpu util with missing ops. Enhance UTs. 2021-05-31 20:33:19 -04:00
Xiao Tianci 9c74d55a49 add HorizontalFlip and VerticalFlip APIs 2021-05-31 15:55:21 +08:00
hesham 3a9695313b files_need_cleanup fixes 2021-05-30 15:44:08 -04:00
mindspore-ci-bot 2433778dac !15963 Support getting dynamic_min_max_shape of data 
From: @luoyang42
Reviewed-by: 
Signed-off-by:
 2021-05-26 09:41:02 +08:00
mindspore-ci-bot 730e211e52 !16702 Add GaussianBlur API for CV data processing 
From: @tiancixiao
Reviewed-by: @liucunwei,@jonyguo
Signed-off-by: @liucunwei
 2021-05-26 09:09:41 +08:00
mindspore-ci-bot d0978f709a !16167 CocoDataset add a new column according a new arg extra_metadata 
From: @xiefangqi
Reviewed-by: 
Signed-off-by:
 2021-05-25 19:55:41 +08:00
mindspore-ci-bot e513786709 !16565 Bug in Normalize when num_channels > 128 
From: @hfarahat
Reviewed-by: @robingrosman,@pandoublefeng
Signed-off-by: @robingrosman
 2021-05-25 10:24:25 +08:00
xiefangqi 53891dbdf7 coco add img_id column 2021-05-24 17:19:58 +08:00
luoyang 13ef5b61f3 support getting dynamic_min_max_shape of data 2021-05-24 17:04:50 +08:00
Xiao Tianci 3f495bbae1 add GaussianBlur API 2021-05-24 16:27:05 +08:00
mindspore-ci-bot 05bbd1d8fa !16426 Change the permission of cache shared memory to 600 for security 
From: @lixiachen
Reviewed-by: @robingrosman,@pandoublefeng
Signed-off-by: @robingrosman
 2021-05-20 02:36:26 +08:00
mindspore-ci-bot 1cb2d31642 !16509 GNN dataset new feature for invalid edge or node id case 
From: @lizhenglong1992
Reviewed-by: @liucunwei,@heleiwang
Signed-off-by: @liucunwei
 2021-05-19 09:20:24 +08:00
hesham bc92011b5f Bug in Normalize when num_channels > 128 2021-05-18 12:23:11 -04:00
Zhenglong Li 9284b6d6af Add special case returned value for get feature function 2021-05-18 15:53:01 +08:00
mindspore-ci-bot ec9fc5fb3b !16384 Fix python multiprocessing with shared memory for edge case 
From: @robingrosman
Reviewed-by: @mikef
Signed-off-by: @mikef
 2021-05-18 10:30:19 +08:00
Lixia Chen 8812111f94 change shared mem perm to 600 & minor other fixes 2021-05-17 17:19:55 -04:00
mindspore-ci-bot 601322356b !15638 VOCDataset need to add img_id for evaluation 
From: @xiefangqi
Reviewed-by: @liucunwei
Signed-off-by: @liucunwei
 2021-05-17 19:33:24 +08:00
xiefangqi 9b846c7873 voc add image_id column 2021-05-17 16:09:04 +08:00
RobinGrosman ca88d7d76a fix edge case and docstring in minddata multiprocessing shared memory code 2021-05-16 06:46:44 -07:00
jonyguo 9452142b50 enhance: enable shuffle infile / files when mindrecord 2021-05-14 17:41:18 +08:00
mindspore-ci-bot b5cddcb8cc !16036 Improve performance of multi-processing option for dataset 
From: @robingrosman
Reviewed-by: 
Signed-off-by:
 2021-05-13 01:41:16 +08:00
RobinGrosman 9a7d1fc034 Optimize multi-process in minddata by passing data via shared memory instead of multiprocessing.queue 2021-05-11 15:16:29 -07:00
mindspore-ci-bot ebed057e72 !16196 If Imagefolder input directory is not right, will occur out of memory error 
From: @xiefangqi
Reviewed-by: @ms_yan,@heleiwang,@jonyguo
Signed-off-by: @jonyguo
 2021-05-11 17:36:28 +08:00
mindspore-ci-bot d3a175e30f !16078 Modify Normalize to accept images with more than 3 channels 
From: @hfarahat
Reviewed-by: @mikef,@robingrosman
Signed-off-by: @robingrosman
 2021-05-11 02:03:34 +08:00
xiefangqi e0148f7181 fix minddata directory path problem 2021-05-10 22:16:37 +08:00
mindspore-ci-bot b86747b9a6 !15813 Convert some pipeline Ops to be inlined 
From: @hfarahat
Reviewed-by: 
Signed-off-by:
 2021-05-08 05:30:39 +08:00
hesham fd943bbdc0 Modify normalize to support multiple of channels 2021-05-07 17:06:26 -04:00
mindspore-ci-bot e0422e972e !16019 If dataset_dir is not a dir or something else validation is not right somehow 
From: @xiefangqi
Reviewed-by: @ms_yan,@liucunwei,@jonyguo
Signed-off-by: @liucunwei
 2021-05-07 11:57:31 +08:00
mindspore-ci-bot b85bf60eb8 !15727 VOCDataset read xml will filter invalid bbox automationally 
From: @xiefangqi
Reviewed-by: @heleiwang,@jonyguo
Signed-off-by: @jonyguo
 2021-05-07 09:50:53 +08:00
mindspore-ci-bot eb08d2e952 !16024 dataset: modify some error msg in dataset module 
From: @ms_yan
Reviewed-by: @jonyguo,@heleiwang
Signed-off-by: @jonyguo
 2021-05-07 09:12:46 +08:00
ms_yan 34dcd2b463 modify log of dataset part 2021-05-06 23:33:17 +08:00
xiefangqi b6e1fceedc minddata modify dir validation 2021-05-06 21:35:31 +08:00
Lixia Chen 5967cb2cd7 Unblock mindrecord under cache 2021-05-05 13:28:13 -04:00
hesham e4c0bd51a5 Modify concat to be an inlined Op 
Modify zip to be an inlined Op
Modify take to be an inlined Op
Modify skip to be an inlined Op
Modify rename to be an inlined Op
Modify filter to consume child's rows using one thread
 2021-05-04 12:45:18 -04:00
mindspore-ci-bot cefcf6c2ae !15931 Add eager tests for transform ops, and add a shape comparison in the EXPECT_MSTENSOR_EQ Macro 
From: @alexyuyue
Reviewed-by: @nsyca,@robingrosman
Signed-off-by: @robingrosman
 2021-04-30 06:15:12 +08:00
alex-yuyue 9ef4331f56 Add eager tests for transform ops 
Signed-off-by: alex-yuyue <yue.yu1@huawei.com>
 2021-04-29 13:19:24 -04:00
liyong 587247e793 add support save dynamic shape tensor 2021-04-27 19:15:10 +08:00
xiefangqi 458ff5fed9 md fix voc and manifest error scenario 2021-04-27 14:24:57 +08:00
dinglinhe 54fb72e2b3 Update normalization description at nn/layer/normalization.py 2021-04-20 12:30:37 +08:00
Cathy Wong 636809c255 dataset: C++ API ToNumber & Lookup: change data_type parm from string to DataType 2021-04-19 12:35:37 -04:00
mindspore-ci-bot f9c5060cf2 !14930 Handle ignored returned Status 
From: @hfarahat
Reviewed-by: @robingrosman,@pandoublefeng
Signed-off-by: @robingrosman
 2021-04-17 03:52:41 +08:00
hesham 6e6eaebb77 Fix unused return value in main code 2021-04-16 11:33:35 -04:00
alex-yuyue 97e3232b6d Add more ut for transform ops and subset sampler 
Signed-off-by: alex-yuyue <yue.yu1@huawei.com>
 2021-04-15 16:51:41 -04:00
mindspore-ci-bot 727203a498 !14911 dataset: Fill op: C++ API support, UTs and Pybind decoupling 
From: @cathwong
Reviewed-by: 
Signed-off-by:
 2021-04-14 05:32:11 +08:00
Cathy Wong 7e6a03487e dataset: Fill op: C++ API support, UTs and Pybind decoupling 2021-04-13 10:02:39 -04:00
mindspore-ci-bot e07a69d8f8 !14874 GNN dataset new feature 
From: @lizhenglong1992
Reviewed-by: 
Signed-off-by:
 2021-04-13 10:44:27 +08:00
Zhenglong Li f3895f2b09 Add get edges from nodes function for gnn network 2021-04-10 18:31:13 +08:00
mindspore-ci-bot f0f7bf1ed1 !14885 fix: wrong input order 
From: @jonyguo
Reviewed-by: @liucunwei,@heleiwang
Signed-off-by: @liucunwei
 2021-04-10 16:52:41 +08:00
jonyguo a5cc81dc97 fix: wrong input order in test_paddeddataset.py 2021-04-09 17:22:31 +08:00
Lixia Chen 1f2b178a16 Reset CacheClient's status for a new run 
Turn off lock only after all data is cached
 2021-04-08 15:30:48 -04:00
mindspore-ci-bot a9ba9b90f6 !14680 MindData Codespell correction 
From: @luoyang42
Reviewed-by: @heleiwang,@liucunwei
Signed-off-by: @liucunwei
 2021-04-07 18:28:55 +08:00
luoyang 624511337c MindData Codespell correction 2021-04-07 10:20:29 +08:00
hesham 8535e89e6c Remove DataBuffer 2021-04-05 17:55:59 -04:00
hesham 0d86cb1423 Remove rows_per_buffer 2021-04-04 00:22:16 -04:00
mindspore-ci-bot c99fe1e412 !12841 DataBuffer removal from DE 
From: @hfarahat
Reviewed-by: 
Signed-off-by:
 2021-04-04 03:28:13 +08:00
luoyang ec21d3f748 Fix validation of RGB2HSV/HSV2RGB/RandomResizedCrop in py_transform 
Fix error msg of pythontokenizer
 2021-04-01 19:09:07 +08:00
hesham a9a80693be Replace Databuffers with TensorRows 2021-03-31 17:00:36 -04:00
xulei2020 4122d1dcce fix code format bug 2021-03-25 09:18:36 +08:00
YangLuo f99204b292 fix python tokenizer 2021-03-18 15:47:09 +08:00
Lixia Chen 30017ea5a0 Fix codex issues 2021-03-15 11:15:15 -04:00
mindspore-ci-bot ac18da3770 !13061 [MD][bugfix] Fix to_json error: can't pickle pyfunc 
From: @xiefangqi
Reviewed-by: 
Signed-off-by:
 2021-03-13 12:48:26 +08:00
Xiao Tianci 4a7100c7f9 fix RandomAffine degree error and some examples in docs 2021-03-12 17:39:38 +08:00
mindspore-ci-bot 682a8926bd !13176 fix minddata transform issue 
From: @luoyang42
Reviewed-by: 
Signed-off-by:
 2021-03-12 17:01:56 +08:00