Nigel
/
mindspore
forked from mindspore-Ecosystem/mindspore
0
0
Fork 0
Code Issues Pull Requests Projects Releases Wiki Activity
42,838 Commits 24 Branches 0 Tags 3.9 GiB
Commit Graph

1084 Commits

Author SHA1 Message Date
i-robot f38df5c888 !26234 New dataset config option, auto_offload 
Merge pull request !26234 from markuskunej/offload_config
 2021-11-19 03:44:56 +00:00
i-robot 3c39afad11 !24547 [MS][crowdfunding]New operator implementation, AutoAugment 
Merge pull request !24547 from yangwm/autoaugment
 2021-11-19 01:00:51 +00:00
markuskunej 29cf442521 Added dataset.config.set_auto_offload(), which will automatically attempt to offload dataset map operations if set to True. 2021-11-18 21:13:20 +00:00
zhuchuang 4c3931f5c9 [feat] [assistant] [I3CKEE] add new audio operator ComputeDeltas 2021-11-18 17:18:43 +08:00
despicablemme 09897ce7ec [feat] [assistant] [I40GZP] add new data op AutoAugment 2021-11-18 11:38:18 +08:00
Cathy Wong b1094b9d81 MD Profiling - Add Start Stop Python UT 2021-11-17 13:49:43 -05:00
wangkc123 706f9e2cbb [feat] [assistant] [I3CKEH] add new audio operator SlidingWindowCmn 2021-11-16 22:00:09 +08:00
i-robot 58b69a05ee !20576 [assistant][ops] add new dataset loading operator LJSpeechDataset 
Merge pull request !20576 from 杨旭华/LJSpeechDataset
 2021-11-16 07:00:20 +00:00
liyong 9663884db3 fix occasional failed in random sampler 2021-11-16 09:53:38 +08:00
mohammad 5c8ab5f60c add MD Profiler Save() 2021-11-15 13:18:36 -05:00
vddong 6fe2d35ae2 [feat][assistant][I3T96D] add new dataset loading operator LJSpeechDataset 2021-11-14 14:52:15 +08:00
Carry955 0d26c38693 [feat][assistant][I40GXT] add new loader DBpedia 2021-11-11 19:02:22 -08:00
i-robot 1792d5f8a4 !26131 Offload Hardware Acceleration Computing 
Merge pull request !26131 from alashkari/hw_accel_move_copy
 2021-11-11 13:18:07 +00:00
i-robot b910870ecc !19986 [assistant][ops] Add new dataset loading operator AGNEWS 
Merge pull request !19986 from 杨旭华/AGNEWS
 2021-11-11 06:22:25 +00:00
Megalomania c34bbabed9 [feat] [assistant] [I3T96M] add new loader AGNEWS 2021-11-09 23:51:19 -08:00
rescue 22e03ab12e Adding offload hardware accelerator. 2021-11-10 07:03:43 +00:00
i-robot f1f7731fb2 !25964 Fix serdes multithreading bug 
Merge pull request !25964 from zetongzhao/serdes_multithreading
 2021-11-09 15:48:06 +00:00
i-robot d89a39c661 !25851 MD Profiling Start and Stop Support 
Merge pull request !25851 from cathwong/ckw_prof_startstop4
 2021-11-08 16:16:42 +00:00
liyong 79891feadb fix shuffle and num_samples issue 2021-11-08 15:33:01 +08:00
Cathy Wong cdf3126459 MD Profiler Start Stop Support 2021-11-05 19:46:39 -04:00
zetongzhao 5e8377ea5c fix multithreading bug 2021-11-05 16:49:18 -04:00
Isaac 69f57bc5b1 [feat][assistant][I40GXD]add new data operator DBToAmplitude 2021-11-02 23:45:27 +08:00
luoyang beba798e00 fix minddata ut core 2021-10-29 16:28:22 +08:00
i-robot 958c2a8e00 !25560 Support bytes input for decode op 
Merge pull request !25560 from luoyang/serving
 2021-10-29 06:47:02 +00:00
liyong b9bad0bdbb fix md ut part06 2021-10-29 10:26:34 +08:00
luoyang b47ac46e92 Support bytes input for decode op 2021-10-28 15:22:50 +08:00
i-robot 74977e548d !24624 [Autotune] Modify MD callback and AddWorker logic to ParallelOps 
Merge pull request !24624 from h.farahat/add_worker
 2021-10-27 00:41:20 +00:00
i-robot c0f6286fcb !25092 [MD] API for autotune 
Merge pull request !25092 from harshvardhangupta/auto_tune_api
 2021-10-26 18:14:43 +00:00
Cathy Wong e66aba9ef5 MD UT - Add missing set and restore seed. 
test_places365_content_check - delete visualize_dataset
 2021-10-26 13:19:47 -04:00
harshvardhangupta 1bb45142c9 Implement autotune api 2021-10-26 11:26:13 -04:00
i-robot c544af28b6 !25286 dataset: modify profiling ut for parallel running 
Merge pull request !25286 from ms_yan/ut_parallel
 2021-10-22 08:09:03 +00:00
ms_yan bb314ad07f make ut parallel 2021-10-22 13:17:43 +08:00
i-robot cf0029b33a !25277 [MD] fix mindrecord ut part05 
Merge pull request !25277 from liyong126/fix_md_ut_05
 2021-10-22 01:43:03 +00:00
i-robot 782874bf18 !21215 [MS][crowdfunding]New operator implementation, include phaser 
Merge pull request !21215 from Wangsong95/phaser
 2021-10-22 01:08:50 +00:00
hesham 85e077b60b Callback changes 
AddWorker logic to parallel ops
 2021-10-21 17:35:11 -04:00
liyong 7c9f6899a3 fix md ut part05 2021-10-21 20:40:09 +08:00
i-robot b7d8bcfe8d !25177 [MD] fix mindrecord ut part04 
Merge pull request !25177 from liyong126/fix_md_ut_part04
 2021-10-21 01:39:26 +00:00
i-robot 5d8be044c9 !18018 [assistant][ops]New operator implementation, include FashionMnistDataset 
Merge pull request !18018 from 张璇/fashionmnist_dataset
 2021-10-20 06:42:54 +00:00
i-robot f9bd640b09 !16269 [assistant][RandomLighting] 
Merge pull request !16269 from QingfengLi/randomlighting
 2021-10-20 06:42:12 +00:00
liyong fc5ea9cc1c fix dataset ut for running independently 2021-10-20 14:27:36 +08:00
“uccInf” fe3123cfe8 [feat][assistant][I3J6V4] add new data operator FashionMnist 2021-10-20 02:37:56 +00:00
i-robot 440036d331 !24362 [assistant][ops]New operator implementation, include PhotoTourDataset 
Merge pull request !24362 from Wangsong95/phototour_dataset
 2021-10-20 01:46:05 +00:00
yangyueren 6594b90a58 [feat][assistant][I3J6VE] add new data operator PhotoTour 2021-10-19 15:31:32 +00:00
chenx2ovo 77caf907c4 [feat][assistant][I3CEGR] add op random lighting 2021-10-19 20:38:01 +08:00
liyong a89741c32e fix md ut part03 2021-10-19 16:52:57 +08:00
i-robot 690642d672 !18664 [assistant][ops]New operator implementation, include Places365Dataset 
Merge pull request !18664 from Wangsong95/places365_dataset
 2021-10-19 07:06:35 +00:00
i-robot 1606f19dfb !18265 [assistant][ops] Add new audio operator CreateDct 
Merge pull request !18265 from 杨旭华/CreateDct
 2021-10-19 06:30:18 +00:00
tophand 4f4777f0da [feat][assistant][I3J6U2] Add new audio operator CreateDct 2021-10-18 03:23:18 -07:00
liyong d9aa18dfca refactor md ut 2021-10-18 17:55:28 +08:00
zx d13d2413f5 [feat][assistant][I3J6UV] add new audio operator Phaser 2021-10-16 08:45:12 +08:00
yangyueren 951193c308 [feat][assistant][I3J6VF] add new data operator Places365 2021-10-15 14:56:09 +00:00
Xiao Tianci 930a5b89c0 fix api docs of audio 2021-10-15 17:17:18 +08:00
robert_luo_yibo 75e2c3041c [feat][assistant][I3CEG3] add new data OP MuLawEncoding 2021-10-14 20:24:18 +08:00
i-robot 1b9b85bbb2 !17851 [assistant][ops]New operator implementation, include FakeImageDataset 
Merge pull request !17851 from 张璇/fakeimage_dataset
 2021-10-13 09:22:55 +00:00
G-Dragon-Liu d2f22a8726 [feat][assistant][I3J6V3] add new data operator FakeImage 2021-10-12 14:00:43 +00:00
zx 9b7d7be65c [feat][assistant][I3J6UU] add new audio operator Overdrive 2021-10-12 18:10:31 +08:00
li-qiyao 2283442855 [feat][assistant][I3J6U9] add new data operator Flanger 2021-10-08 05:03:31 -07:00
i-robot 8598d08351 !17820 [assistant][ops]New operator implementation, include EMnistDataset 
Merge pull request !17820 from 张璇/emnist_dataset
 2021-10-08 07:18:31 +00:00
G-Dragon-Liu 32baa520bf [feat][assistant][I3J6V2] add new data operator EMnist 2021-10-08 02:53:00 +00:00
Zhang Lian b7e6520e62 [feat] [assistant] [I3J6U8] Add a new audio operator DetectPitchFrequency 2021-09-29 15:12:58 +08:00
i-robot dd544a9693 !21406 [MS][crowdfunding]New operator implementation, include TrebleBiquad 
Merge pull request !21406 from 张璇/treble_biquad
 2021-09-27 14:18:00 +00:00
i-robot f16c003dfa !22610 [MS][crowdfunding]New operator implementation, RandomAdjustSharpness 
Merge pull request !22610 from yangwm/sharpness
 2021-09-27 09:20:05 +00:00
zx 3da0f0f487 [feat][assistant][I3J6V0] add new audio operator TrebleBiquad 2021-09-27 13:01:26 +08:00
despicablemme d072a4e4bf [feat] [assistant] [I40GZE] add new data operator RandomAdjustSharpness 2021-09-27 11:19:33 +08:00
i-robot ab567574c4 !21451 [MS][crowdfunding]New operator implementation, include RiaaBiquad 
Merge pull request !21451 from 张璇/riaa_biquad
 2021-09-27 03:14:16 +00:00
i-robot 2969688382 !20047 [assistant][ops]New operator implementation, include QmnistDataset 
Merge pull request !20047 from Wangsong95/qmnist_dataset
 2021-09-27 01:36:08 +00:00
despicablemme 7063907bdc [feat] [assidtant] [I40GZN] add new data ops RandomEqualize 2021-09-25 18:00:25 +08:00
i-robot d0023355a3 !23205 Fix bus error with multiprocessing enabled, large image, shared_memory, single device 
Merge pull request !23205 from RobinGrosman/mem_bus_failure
 2021-09-24 19:16:12 +00:00
RobinGrosman c90efc0403 add check for shm available when num_devices=1 to avoid bus errors 
-Required modifying tests to reduce memory usage to check doesn't fail during CI
-Also required fixing max_rowsize to be externalized from map+batch() function
 2021-09-24 10:24:31 -07:00
zx 1d95e6f480 [feat][assistant][I3J6UY] add new audio operator RiaaBiquad 2021-09-23 21:36:59 +08:00
despicablemme 7a6b75c2e8 [feat] [assistant] [I40GZJ] add new data op RandomAutoContrast 2021-09-23 20:17:39 +08:00
i-robot 0c26736da1 !22276 [MS][crowdfunding]New operator implementation, RandomInvert 
Merge pull request !22276 from yangwm/rinvert
 2021-09-23 08:12:57 +00:00
hetshah 2c1666fd06 set num epochs for iterators that use the default value 
add explicit paramter for infinite iterators

set appropriate value for num_epoch param for iteratos

fix long lines

remove trailing whitespace

fixed typo

remove comment

fixed comments

fixed test
 2021-09-22 17:32:50 +00:00
despicablemme b977d8723b [feat] [assistant] [I40GZC] add new data ops RandomInvert 2021-09-22 16:55:14 +08:00
ckczzj 001b68c286 [feat][assistant][I3J6VG] add new data operator QMnist 2021-09-22 12:13:27 +08:00
i-robot 6489dacdab !18114 [assistant][ops]Add new operator Magphase 
Merge pull request !18114 from YJfuel123/Magphase
 2021-09-18 07:26:34 +00:00
i-robot 92287b86d8 !23471 Fix white list error msg synax error 
Merge pull request !23471 from xiefangqi/md_white_list_optimization
 2021-09-18 05:03:40 +00:00
葛崇林 a05a5de073 [feat][assistant][I3ALC5] add new method Magphase 2021-09-17 16:02:46 +08:00
i-robot f5af432e04 !16922 [assistant][Vol] 
Merge pull request !16922 from QingfengLi/Vol
 2021-09-17 06:51:49 +00:00
chenx2ovo 9e62623b4a [feat][assistant][I3CEGI] add op vol 2021-09-17 09:34:20 +08:00
ms_yan 0a365e91bf modify log in kernel part 2021-09-16 22:09:21 +08:00
Guo Zhijian f49bacde09 fix: pyfunc hung when it's a yield func 2021-09-16 09:20:44 +08:00
i-robot 57b8c8e2d9 !23310 [MD] fix error message of mindrecord 
Merge pull request !23310 from liyong126/fix_log_msg
 2021-09-16 00:45:11 +00:00
i-robot e5af07d519 !23538 minddata error message optimize 
Merge pull request !23538 from luoyang/log_update
 2021-09-16 00:42:42 +00:00
i-robot 4a553c8d57 !16607 [assistant][Fade] 
Merge pull request !16607 from QingfengLi/Fade
 2021-09-15 14:54:16 +00:00
YangLuo 9d89de5765 minddata error message optimize 2021-09-15 18:05:02 +08:00
liyong 65f3365a2a fix mindrecord log 2021-09-15 16:18:40 +08:00
jonyguo c408867085 fix: hung when user return Tensor in generator 2021-09-15 14:13:33 +08:00
xiefangqi a91c87c581 fix some syntax error 2021-09-14 21:29:02 +08:00
chenx2ovo 1a3196b052 [feat][assistant][I3CEGF] add op fade 2021-09-14 19:08:37 +08:00
i-robot 32c2b77595 !19093 [assistant][ops] Add new audio operator Biquad 
Merge pull request !19093 from 杨旭华/Biquad
 2021-09-14 01:52:01 +00:00
i-robot 6dcfe64501 !23264 python serdes bug fix 
Merge pull request !23264 from zetongzhao/deserialize
 2021-09-13 15:53:05 +00:00
i-robot 0aa167cdf4 !23240 maybe max open files in mr or onehot fail in pytransform 
Merge pull request !23240 from guozhijian/fix_max_open_files_and_onehot
 2021-09-13 13:51:09 +00:00
i-robot 5abd4821ce !23357 Add DatasetName method for out developer 
Merge pull request !23357 from xiefangqi/md_white_list_optimization
 2021-09-13 12:06:49 +00:00
i-robot 3e1d1ff4d3 !18671 [assistant][ops]New operator implementation, include SBUDataset 
Merge pull request !18671 from Wangsong95/sbu_dataset
 2021-09-13 11:56:13 +00:00
liannai 631f49f044 [feat][assistant][I3T96A] Add new audio operator Biquad 2021-09-13 19:18:50 +08:00
xiefangqi 465dd25d74 add virtual DatasetName method 2021-09-13 15:17:32 +08:00
i-robot 87381a1cf9 !17806 [assistant][ops] DcShift Op 
Merge pull request !17806 from 杨旭华/DCShift
 2021-09-13 01:29:14 +00:00
i-robot c9ff7242a4 !23298 minddata engine optimize white list error msg 
Merge pull request !23298 from xiefangqi/md_white_list_optimization
 2021-09-13 01:19:15 +00:00
Carry955 0ba84345b0 [feat][assistant][I3J6U3] add new audio operator DCShift 2021-09-11 23:58:18 -07:00
jonyguo da78a17730 fix: may max openfiles in mindrecord and onehot error in py_transform 
and cache_admin rpath
 2021-09-12 00:25:07 +08:00
xiefangqi ff73e4ccd7 fix white list error msg 2021-09-11 17:36:23 +08:00
Isaac17555 53fc99b914 [feat][assistant][I3T96B]add new data operator LFilter 2021-09-11 02:05:43 -07:00
YJfuel123 16150d394f [feat][assistant][I3ALd1] add new operator EqualizerBiquad 2021-09-11 10:36:20 +08:00
ckczzj af97ba4a77 [feat][assistant][I3J6VH] add new data operator SBU 2021-09-11 09:58:17 +08:00
zetongzhao 719a207040 deserialize fix 2021-09-10 15:19:59 -04:00
i-robot 7b4f361ca7 !23232 Fix multiprocessing attr error 
Merge pull request !23232 from xiefangqi/md_fix_map_pool_issue
 2021-09-10 14:59:40 +00:00
i-robot 6d709c7254 !19624 [assistant][ops] Add new dataset operator DIV2K. 
Merge pull request !19624 from Rainfor/div2k
 2021-09-10 14:59:21 +00:00
xiefangqi 66c5f937dd fix attr failed problem 2021-09-10 18:01:04 +08:00
wangkc123 3daf3a4d5f [feat] [assistant] [I3CKEK] add new dataset operator DIV2K 2021-09-10 17:55:34 +08:00
robert_luo_yibo ce71699734 [feat][assistant][I3CEG5] add new data OP MuLawDecoding 2021-09-10 15:49:46 +08:00
i-robot 32cc5d79b0 !18670 [assistant][ops]New operator implementation, include USPSDataset 
Merge pull request !18670 from Wangsong95/usps_dataset
 2021-09-10 03:06:19 +00:00
i-robot f91e1440c8 !17810 [assistant][ops][DeemphBiquadOp] 
Merge pull request !17810 from Isaac/DeemphBiquad
 2021-09-10 03:05:41 +00:00
ckczzj d7c702e9d4 [feat][assistant][I3J6VL] add new data operator USPS 2021-09-09 21:46:57 +08:00
i-robot 936940da8e !22666 [assistant][ComplexNorm] 
Merge pull request !22666 from QingfengLi/ComplexNorm
 2021-09-09 06:41:53 +00:00
Isaac 5dc6c7fe96 [feat][assistant][I3J6U6]add new data operator DeemphBiquad 2021-09-09 14:34:24 +08:00
i-robot 9ad56e86ab !17846 [assistant][ops]Add new operator HighpassBiquad 
Merge pull request !17846 from YJfuel123/HighpassBiquad
 2021-09-09 02:06:41 +00:00
i-robot a918783f1a !22623 Add Random operation with Multi Field 
Merge pull request !22623 from shenwei41/operation_dev
 2021-09-09 01:49:04 +00:00
i-robot b21da536d3 !22784 serdes python operations by pyfunc 
Merge pull request !22784 from zetongzhao/deserialize
 2021-09-08 17:18:57 +00:00
zetongzhao 0d389f3f0c serdes python operations 2021-09-08 09:05:11 -04:00
shenwei41 a8d02ab782 Add Random operation with MultiField 2021-09-08 19:29:23 +08:00
i-robot 938324b397 !19362 [assistant][ops] Add new dataset operator Cityscapes. 
Merge pull request !19362 from Rainfor/cityscapes
 2021-09-08 09:14:43 +00:00
yaoyuan 85a1b55b47 [feat] [assistant] [I3ALCL] add new operator HighpassBiquad 2021-09-08 15:12:40 +08:00
i-robot b1be8dfd31 !22851 MD Profiling: Update Connector Init to Remove any existing file 
Merge pull request !22851 from cathwong/ckw_mon_seq_pipelines_fix
 2021-09-07 19:57:58 +00:00
xiefangqi 57f68d35bd fix generator 14 ut problem 2021-09-06 19:01:51 +08:00
Cathy Wong bc85c606b8 MD Profiling: Update Connector Init to Remove any existing file 
to fix sequential pipeline scenario.
 2021-09-03 15:45:38 -04:00
i-robot 3368287aab !22620 add util: browse dataset 
Merge pull request !22620 from luoyang/browse_dataset
 2021-09-02 01:00:46 +00:00
robert_luo_yibo 3d58da9023 [fix][assistant][I47V0O] fix bug about the wrong output in the operator ComplexNorm 2021-09-01 17:55:38 +08:00
YangLuo 97c1ba380f add util: browse dataset 2021-09-01 16:13:32 +08:00
wangkc123 70f741bcc1 [feat] [assistant] [I3CKEM] add new dataset operator Cityscapes 2021-08-31 20:03:37 +08:00
tongyue 329526dea8 [feat][assistant][I3J6U0] add new data operator Contrast 2021-08-30 19:34:37 -07:00
i-robot bbf9c6a9ad !22529 fix: for loop dataset iterator will cause memory leak 
Merge pull request !22529 from guozhijian/fix_for_loop_ds_iter_memory_leak
 2021-08-31 01:15:58 +00:00
i-robot a47ab8d643 !18073 [assistant][ops]Add new operator LowpassBiquad 
Merge pull request !18073 from YJfuel123/LowpassBiquad
 2021-08-30 08:33:39 +00:00
jonyguo 4c915fdcbb fix: for loop dataset iterator will cause memory leak 2021-08-30 16:08:26 +08:00
liyong 5a4ff8cfdb fix ci issue 2021-08-30 10:18:06 +08:00
i-robot 1639023909 !22263 Add convertcolor operation 
Merge pull request !22263 from shenwei41/operation_dev
 2021-08-27 12:33:43 +00:00
liyong c257cf36e2 refactor mindrecord 2021-08-27 09:15:05 +08:00
shenwei41 1295744524 Add ConvertColor Operation 2021-08-26 21:14:28 +08:00
haizhouye d96d9e64b0 [feat] [assistant] [I3ALCA] add new LowpassBiquad operator 
[feat] [assistant] [I3ALCA] add new LowpassBiquad operator

bugfix

fix

fix
 2021-08-26 18:49:21 +08:00
i-robot 956ae2bbfd !22223 [assistant][TimeMask and FreqMask] 
Merge pull request !22223 from QingfengLi/TimeFreqMask
 2021-08-25 10:43:24 +00:00
chenx2ovo 2f1d974eeb [fix] [assistant] [I465ZG] revise error message and change input param datatype 2021-08-25 10:47:27 +08:00
zhangyihui 3d19949eb4 device_id to rank_id 2021-08-25 09:52:23 +08:00
i-robot 6082835501 !22146 dataset: support return bytes in GeneratorDataset 
Merge pull request !22146 from ms_yan/decode_bytes
 2021-08-24 02:41:18 +00:00
i-robot db44b88e1e !22094 MD Profiling: For mismatch op info btwn files, skip bottleneck analysis 
Merge pull request !22094 from cathwong/ckw_mon_py_analyze_fixes3
 2021-08-23 18:28:51 +00:00
ms_yan 6ecfc34524 add np.frombuffer to support return bytes 2021-08-23 17:31:38 +08:00
ms_yan 36a8886ca2 Revert "[feat] [assistant] [I3T96T] add new Dataset operator CMUARCTICDataset" 
This reverts commit b077aa1cab.

Revert "[feat] [assistant] [I3T96X] add new Dataset operator LibriSpeechDataset"

This reverts commit 4e6f7dc97d.

delete pass_registry_test.cc

comment  hiai_nlu_model_multi.pb related  line
 2021-08-23 01:46:38 +08:00
djc b077aa1cab [feat] [assistant] [I3T96T] add new Dataset operator CMUARCTICDataset 2021-08-22 16:26:45 +08:00
djc 4e6f7dc97d [feat] [assistant] [I3T96X] add new Dataset operator LibriSpeechDataset 2021-08-22 13:39:37 +08:00
i-robot 0e7844daa9 !22093 revert merge request 21685 
Merge pull request !22093 from guozhijian/revert-merge-21685-master
 2021-08-21 09:25:32 +00:00
i-robot 4ba337e0f5 !16667 [assistant][ComplexNorm] 
Merge pull request !16667 from QingfengLi/ComplexNorm
 2021-08-21 02:44:18 +00:00
i-robot 5d1bb097e2 !22052 Fix probability core dump in MindData coverage test 
Merge pull request !22052 from luoyang/issues2
 2021-08-20 01:18:06 +00:00
robert_luo_yibo d837cbee0f [feat][assistant][I3CEG8] add new data operator ComplexNorm 2021-08-20 09:07:21 +08:00
Cathy Wong 85931175bc MD Profiling: For mismatch op info btwn files, skip bottleneck analysis. 
Add to summary output: per_pipeline_time and per_push_queue_time.
Enhance UT
 2021-08-19 12:19:59 -04:00
guozhijian e1df9a2152 revert 'Pull Request !21685 : add function to invert_op' 2021-08-19 22:49:51 +08:00
i-robot 4113aee7d6 !18668 [assistant][ops]New operator implementation, include Flowers102Dataset 
Merge pull request !18668 from Wangsong95/flowers102_dataset
 2021-08-19 08:41:44 +00:00
YangLuo 9352935ff6 Fix probability core dump in MindData coverage test 2021-08-19 14:34:57 +08:00
i-robot 3ad641cb65 !21685 add function to invert_op 
Merge pull request !21685 from Killer/fake_killer
 2021-08-19 05:13:39 +00:00
fake_killer b0bc4bf5da [feat] [assistant] [I44TRS] extend invert_op's function 2021-08-18 21:46:15 +08:00
Xiao Tianci aae886bcd7 fix crowd contributing code 2021-08-16 19:33:10 +08:00
i-robot f99effd430 !21755 Fix batch usability problem 
Merge pull request !21755 from xiefangqi/md_fix_batch_usability
 2021-08-13 07:21:49 +00:00
i-robot d1112f8e9c !17631 [assistant][FrequencyMasking] 
Merge pull request !17631 from QingfengLi/FreqMask
 2021-08-13 06:48:06 +00:00
xiefangqi e4894a1bf1 fix batch usability problem 2021-08-12 20:12:43 +08:00
chenx2ovo 53615010a3 [feat][assistant][I3CEGG] add new data OP FrequencyMasking 2021-08-12 15:28:30 +08:00
jonyguo 42b3512a1b fix: when field is none in mr 2021-08-11 17:11:38 +08:00
chenx2ovo e72acf362c [feat][assistant][I3CEGH] add op timemasking 2021-08-10 12:06:07 +08:00
i-robot dc626617fc !19526 [assistant][TimeStretch] 
Merge pull request !19526 from QingfengLi/TimeStretch
 2021-08-09 09:40:38 +00:00
shenwei41 59efbb88ec fix waring 2021-08-09 11:40:52 +08:00
chenx2ovo 04705e5b0d [feat][assistant][I3CEGC] add op timestretch 2021-08-07 22:53:41 +08:00
i-robot c84e09cf45 !21072 Deserialize - python binding 
Merge pull request !21072 from zetongzhao/deserialize_1
 2021-08-06 21:47:09 +00:00
zetongzhao 4f199b548c deserialize and tests 2021-08-06 13:35:27 -04:00
i-robot 51a1ca61f2 !21357 [MD] fix SlicePatches when dealing with some special tensor. 
Merge pull request !21357 from liyong126/fix_bug
 2021-08-06 08:58:27 +00:00
liyong 5fea72473c fix HW1 issue 2021-08-06 14:07:26 +08:00
ckczzj 9d9f33be88 [feat][assistant][I3J6VN] add new data operator flowers102 2021-08-06 10:55:12 +08:00
chenx2ovo aa45e2ebbc [feat] [assistant] [I3CEER] add op AmplitudeToDB 2021-08-06 10:07:29 +08:00
Carry955 98be266585 [feat][assistant][I3J6TQ] add a new operator Angle 2021-08-04 19:41:00 -07:00
i-robot 680e40a43a !16453 [assistant][BassBiquadOp] 
Merge pull request !16453 from StyleHang/BassBiquadOp
 2021-08-04 10:02:34 +00:00
i-robot f4e07c0dce !17486 [assistant][ops] Add new dataset operator Flickr. 
Merge pull request !17486 from Rainfor/wangkc
 2021-08-04 08:52:21 +00:00
i-robot d260f38936 !15722 [assistant][AdjustGamma] 
Merge pull request !15722 from QingfengLi/adjust_gamma
 2021-08-04 03:58:09 +00:00
li-qiyao ba326d76a8 [feat][assistant][I3J6TW] add new data operator BassBiquad 2021-08-03 19:40:12 -07:00
wangkc123 48f5fed1f8 [feat] [assistant] [I3CKEN] add new dataset operator Flickr 2021-08-04 10:20:43 +08:00
chenx2ovo 4191dde45f [feat][assistant][I3CEGM] add op adjust gamma 2021-08-04 02:32:13 +08:00
tongyue 08469e113a [feat][assistant][I3J6TV]add new audio operator BandrejectBiquad 2021-08-03 05:41:53 -07:00
i-robot 2c7b8e73df !16452 [assistant][BandpassBiquadOp] 
Merge pull request !16452 from StyleHang/BandpassBiquadOp
 2021-08-03 11:08:48 +00:00
i-robot 93af13f332 !19979 dataset: modify some error msg 
Merge pull request !19979 from ms_yan/err_msg
 2021-08-03 08:53:30 +00:00
i-robot ef77050348 !18892 [assistant][ops] Add new dataset operator SBD. 
Merge pull request !18892 from Rainfor/sbd
 2021-08-03 03:33:00 +00:00
ms_yan 65870045bc modify some error msg 2021-08-03 09:16:28 +08:00
administrator 29d57a89fb [feat][assistant][I3TDRD]add new audio operator BandpassBiquad 2021-08-02 07:08:47 -07:00
Isaac b49e1acc3e [feat] [assistant] [I3J6TM] add new audio operator AllpassBiquad 2021-07-31 16:15:17 +08:00
i-robot 9301de1b54 !21098 fix bug 
Merge pull request !21098 from shenwei41/fix_waring
 2021-07-31 01:12:21 +00:00
shenwei41 e3385e2564 fix waring to master 2021-07-30 16:37:27 +08:00
i-robot 6e84bf0f0f !16449 [assistant][ops] Add new audio operator BandBiquad 
Merge pull request !16449 from StyleHang/BandBiquadOp
 2021-07-30 03:35:07 +00:00
liannai 5a341e7e27 [feat] [assistant] [I3J6TS] add new audio operator BandBiquad 2021-07-29 17:57:27 +08:00
Xiao Tianci 53c01c437d fix NumpySlicesDataset not accept numpy 2021-07-22 15:46:37 +08:00
wangkc123 b381e0be9d [feat] [assistant] [I3CKER] add new dataset operator SBD 2021-07-17 21:02:30 +08:00
i-robot 8508c9eb45 !18179 [MD] add slice patches op in c++ and python 
Merge pull request !18179 from liyong126/slice_patches_op
 2021-07-05 07:04:45 +00:00
liyong 9a1de0af65 add slice patches in python and c++ 2021-07-05 09:16:47 +08:00
Xiao Tianci 8cdd0d8d90 fix param check for Rotate and GaussianBlur 2021-07-01 20:31:33 +08:00
YangLuo 17c121b087 Give concreted error when mappable dataset lack __len__ method 2021-06-29 11:13:12 +08:00
Lixia Chen 1f03b16e0e Fix cache_admin server start stderr output 
Fix TFRecord get_dataset_size() when cached
 2021-06-23 18:32:54 -04:00
Harshvardhan Gupta 6845a16b44 fix coredump with large max_rowsize and add arg validations 2021-06-23 09:31:11 -04:00
YangLuo 4e9f6d54f7 update cpp api & doc 2021-06-23 11:14:52 +08:00
chenx2ovo 4b98099dcb [feat][assistant][I3CEGQ] add op rgb2bgr 2021-06-21 16:16:50 +08:00
i-robot ab3259c43a !18292 MD Profiling UT: Check CPU util file. Add sequential pipeline UTs 
Merge pull request !18292 from cathwong/ckw_mon_py_analyze_ut4
 2021-06-17 21:47:10 +08:00
liyong 2989a5a23b fix some bugs 2021-06-17 11:56:10 +08:00
Cathy Wong c37eb350f6 MD Profiling UT: Check CPU util file. Add sequential pipeline UTs 2021-06-16 15:30:59 -04:00
Xiao Tianci a54e65a7ad add Crop Python API 2021-06-10 09:46:57 +08:00
i-robot e60f5f12f5 !17836 Add Rotate API for CV data processing 
Merge pull request !17836 from xiaotianci/rotate
 2021-06-10 09:29:18 +08:00
i-robot 50f11d5439 !17503 dataset: enhance the expression of error msg 
Merge pull request !17503 from ms_yan/log_modify
 2021-06-08 22:25:04 +08:00
Xiao Tianci 9c4c1c6a77 add Rotate Python API 2021-06-08 22:19:15 +08:00
i-robot 3a63a66d64 !17807 MD Profiling UT: Add MD Analyze UT for MinddataProfilingAnalyzer 
Merge pull request !17807 from cathwong/ckw_mon_py_analyze_ut3
 2021-06-08 21:52:55 +08:00
ms_yan be65ebcb28 complete dataset log 2021-06-08 16:45:47 +08:00
mindspore-ci-bot 8934e4a513 !17879 dynamic min max shape phase 2 
From: @luoyang42
Reviewed-by: @liucunwei,@jonyguo
Signed-off-by: @liucunwei
 2021-06-08 14:49:13 +08:00
Cathy Wong b7342efe12 MD Profiling UT: Add MD Analyze UT. Add more pipeline profiling UTs. 2021-06-07 11:00:34 -04:00
luoyang dad289c679 dynamic min max shape phase 2 2021-06-07 20:06:26 +08:00
Zhenglong Li 2c7d1bf603 Implement new gnn feature 2021-06-07 10:20:06 +08:00
mindspore-ci-bot b52f9ddb08 !17060 Add HorizontalFlip and VerticalFlip for CV data processing 
From: @tiancixiao
Reviewed-by: @liucunwei
Signed-off-by: @liucunwei
 2021-06-02 17:48:28 +08:00
mindspore-ci-bot c4757f0692 !17461 MD Profiling Analyze: Handle cpu util with missing ops. Enhance UTs. 
From: @cathwong
Reviewed-by: @robingrosman
Signed-off-by: @robingrosman
 2021-06-02 11:12:59 +08:00
mindspore-ci-bot ae2e7c1288 !17541 Fix MD Cache code check warnings [master] 
From: @lixiachen
Reviewed-by: @pandoublefeng,@robingrosman
Signed-off-by: @robingrosman
 2021-06-02 04:53:54 +08:00
Lixia Chen 8c1b561149 Fix cache code check warnings 2021-06-01 10:02:38 -04:00
Cathy Wong f8beb933ad MD Profiling Analyze: Handle cpu util with missing ops. Enhance UTs. 2021-05-31 20:33:19 -04:00
Xiao Tianci 9c74d55a49 add HorizontalFlip and VerticalFlip APIs 2021-05-31 15:55:21 +08:00
hesham 3a9695313b files_need_cleanup fixes 2021-05-30 15:44:08 -04:00
mindspore-ci-bot 2433778dac !15963 Support getting dynamic_min_max_shape of data 
From: @luoyang42
Reviewed-by: 
Signed-off-by:
 2021-05-26 09:41:02 +08:00
mindspore-ci-bot 730e211e52 !16702 Add GaussianBlur API for CV data processing 
From: @tiancixiao
Reviewed-by: @liucunwei,@jonyguo
Signed-off-by: @liucunwei
 2021-05-26 09:09:41 +08:00
mindspore-ci-bot d0978f709a !16167 CocoDataset add a new column according a new arg extra_metadata 
From: @xiefangqi
Reviewed-by: 
Signed-off-by:
 2021-05-25 19:55:41 +08:00
mindspore-ci-bot e513786709 !16565 Bug in Normalize when num_channels > 128 
From: @hfarahat
Reviewed-by: @robingrosman,@pandoublefeng
Signed-off-by: @robingrosman
 2021-05-25 10:24:25 +08:00
xiefangqi 53891dbdf7 coco add img_id column 2021-05-24 17:19:58 +08:00
luoyang 13ef5b61f3 support getting dynamic_min_max_shape of data 2021-05-24 17:04:50 +08:00
Xiao Tianci 3f495bbae1 add GaussianBlur API 2021-05-24 16:27:05 +08:00
mindspore-ci-bot 05bbd1d8fa !16426 Change the permission of cache shared memory to 600 for security 
From: @lixiachen
Reviewed-by: @robingrosman,@pandoublefeng
Signed-off-by: @robingrosman
 2021-05-20 02:36:26 +08:00
mindspore-ci-bot 1cb2d31642 !16509 GNN dataset new feature for invalid edge or node id case 
From: @lizhenglong1992
Reviewed-by: @liucunwei,@heleiwang
Signed-off-by: @liucunwei
 2021-05-19 09:20:24 +08:00
hesham bc92011b5f Bug in Normalize when num_channels > 128 2021-05-18 12:23:11 -04:00
Zhenglong Li 9284b6d6af Add special case returned value for get feature function 2021-05-18 15:53:01 +08:00
mindspore-ci-bot ec9fc5fb3b !16384 Fix python multiprocessing with shared memory for edge case 
From: @robingrosman
Reviewed-by: @mikef
Signed-off-by: @mikef
 2021-05-18 10:30:19 +08:00
Lixia Chen 8812111f94 change shared mem perm to 600 & minor other fixes 2021-05-17 17:19:55 -04:00
mindspore-ci-bot 601322356b !15638 VOCDataset need to add img_id for evaluation 
From: @xiefangqi
Reviewed-by: @liucunwei
Signed-off-by: @liucunwei
 2021-05-17 19:33:24 +08:00
xiefangqi 9b846c7873 voc add image_id column 2021-05-17 16:09:04 +08:00
RobinGrosman ca88d7d76a fix edge case and docstring in minddata multiprocessing shared memory code 2021-05-16 06:46:44 -07:00
jonyguo 9452142b50 enhance: enable shuffle infile / files when mindrecord 2021-05-14 17:41:18 +08:00
mindspore-ci-bot b5cddcb8cc !16036 Improve performance of multi-processing option for dataset 
From: @robingrosman
Reviewed-by: 
Signed-off-by:
 2021-05-13 01:41:16 +08:00
RobinGrosman 9a7d1fc034 Optimize multi-process in minddata by passing data via shared memory instead of multiprocessing.queue 2021-05-11 15:16:29 -07:00
mindspore-ci-bot ebed057e72 !16196 If Imagefolder input directory is not right, will occur out of memory error 
From: @xiefangqi
Reviewed-by: @ms_yan,@heleiwang,@jonyguo
Signed-off-by: @jonyguo
 2021-05-11 17:36:28 +08:00
mindspore-ci-bot d3a175e30f !16078 Modify Normalize to accept images with more than 3 channels 
From: @hfarahat
Reviewed-by: @mikef,@robingrosman
Signed-off-by: @robingrosman
 2021-05-11 02:03:34 +08:00
xiefangqi e0148f7181 fix minddata directory path problem 2021-05-10 22:16:37 +08:00
mindspore-ci-bot b86747b9a6 !15813 Convert some pipeline Ops to be inlined 
From: @hfarahat
Reviewed-by: 
Signed-off-by:
 2021-05-08 05:30:39 +08:00
hesham fd943bbdc0 Modify normalize to support multiple of channels 2021-05-07 17:06:26 -04:00
mindspore-ci-bot e0422e972e !16019 If dataset_dir is not a dir or something else validation is not right somehow 
From: @xiefangqi
Reviewed-by: @ms_yan,@liucunwei,@jonyguo
Signed-off-by: @liucunwei
 2021-05-07 11:57:31 +08:00
mindspore-ci-bot b85bf60eb8 !15727 VOCDataset read xml will filter invalid bbox automationally 
From: @xiefangqi
Reviewed-by: @heleiwang,@jonyguo
Signed-off-by: @jonyguo
 2021-05-07 09:50:53 +08:00
mindspore-ci-bot eb08d2e952 !16024 dataset: modify some error msg in dataset module 
From: @ms_yan
Reviewed-by: @jonyguo,@heleiwang
Signed-off-by: @jonyguo
 2021-05-07 09:12:46 +08:00
ms_yan 34dcd2b463 modify log of dataset part 2021-05-06 23:33:17 +08:00