yystopf
/
mindspore
forked from mindspore-Ecosystem/mindspore
0
0
Fork 0
Code Issues Pull Requests Projects Releases Wiki Activity
45,876 Commits 23 Branches 0 Tags 3.9 GiB
Commit Graph

1152 Commits

Author SHA1 Message Date
i-robot cf0029b33a !25277 [MD] fix mindrecord ut part05 
Merge pull request !25277 from liyong126/fix_md_ut_05
 2021-10-22 01:43:03 +00:00
i-robot 782874bf18 !21215 [MS][crowdfunding]New operator implementation, include phaser 
Merge pull request !21215 from Wangsong95/phaser
 2021-10-22 01:08:50 +00:00
hesham 85e077b60b Callback changes 
AddWorker logic to parallel ops
 2021-10-21 17:35:11 -04:00
liyong 7c9f6899a3 fix md ut part05 2021-10-21 20:40:09 +08:00
i-robot b7d8bcfe8d !25177 [MD] fix mindrecord ut part04 
Merge pull request !25177 from liyong126/fix_md_ut_part04
 2021-10-21 01:39:26 +00:00
i-robot 5d8be044c9 !18018 [assistant][ops]New operator implementation, include FashionMnistDataset 
Merge pull request !18018 from 张璇/fashionmnist_dataset
 2021-10-20 06:42:54 +00:00
i-robot f9bd640b09 !16269 [assistant][RandomLighting] 
Merge pull request !16269 from QingfengLi/randomlighting
 2021-10-20 06:42:12 +00:00
liyong fc5ea9cc1c fix dataset ut for running independently 2021-10-20 14:27:36 +08:00
“uccInf” fe3123cfe8 [feat][assistant][I3J6V4] add new data operator FashionMnist 2021-10-20 02:37:56 +00:00
i-robot 440036d331 !24362 [assistant][ops]New operator implementation, include PhotoTourDataset 
Merge pull request !24362 from Wangsong95/phototour_dataset
 2021-10-20 01:46:05 +00:00
yangyueren 6594b90a58 [feat][assistant][I3J6VE] add new data operator PhotoTour 2021-10-19 15:31:32 +00:00
chenx2ovo 77caf907c4 [feat][assistant][I3CEGR] add op random lighting 2021-10-19 20:38:01 +08:00
liyong a89741c32e fix md ut part03 2021-10-19 16:52:57 +08:00
i-robot 690642d672 !18664 [assistant][ops]New operator implementation, include Places365Dataset 
Merge pull request !18664 from Wangsong95/places365_dataset
 2021-10-19 07:06:35 +00:00
i-robot 1606f19dfb !18265 [assistant][ops] Add new audio operator CreateDct 
Merge pull request !18265 from 杨旭华/CreateDct
 2021-10-19 06:30:18 +00:00
tophand 4f4777f0da [feat][assistant][I3J6U2] Add new audio operator CreateDct 2021-10-18 03:23:18 -07:00
liyong d9aa18dfca refactor md ut 2021-10-18 17:55:28 +08:00
zx d13d2413f5 [feat][assistant][I3J6UV] add new audio operator Phaser 2021-10-16 08:45:12 +08:00
yangyueren 951193c308 [feat][assistant][I3J6VF] add new data operator Places365 2021-10-15 14:56:09 +00:00
Xiao Tianci 930a5b89c0 fix api docs of audio 2021-10-15 17:17:18 +08:00
robert_luo_yibo 75e2c3041c [feat][assistant][I3CEG3] add new data OP MuLawEncoding 2021-10-14 20:24:18 +08:00
i-robot 1b9b85bbb2 !17851 [assistant][ops]New operator implementation, include FakeImageDataset 
Merge pull request !17851 from 张璇/fakeimage_dataset
 2021-10-13 09:22:55 +00:00
G-Dragon-Liu d2f22a8726 [feat][assistant][I3J6V3] add new data operator FakeImage 2021-10-12 14:00:43 +00:00
zx 9b7d7be65c [feat][assistant][I3J6UU] add new audio operator Overdrive 2021-10-12 18:10:31 +08:00
li-qiyao 2283442855 [feat][assistant][I3J6U9] add new data operator Flanger 2021-10-08 05:03:31 -07:00
i-robot 8598d08351 !17820 [assistant][ops]New operator implementation, include EMnistDataset 
Merge pull request !17820 from 张璇/emnist_dataset
 2021-10-08 07:18:31 +00:00
G-Dragon-Liu 32baa520bf [feat][assistant][I3J6V2] add new data operator EMnist 2021-10-08 02:53:00 +00:00
Zhang Lian b7e6520e62 [feat] [assistant] [I3J6U8] Add a new audio operator DetectPitchFrequency 2021-09-29 15:12:58 +08:00
i-robot dd544a9693 !21406 [MS][crowdfunding]New operator implementation, include TrebleBiquad 
Merge pull request !21406 from 张璇/treble_biquad
 2021-09-27 14:18:00 +00:00
i-robot f16c003dfa !22610 [MS][crowdfunding]New operator implementation, RandomAdjustSharpness 
Merge pull request !22610 from yangwm/sharpness
 2021-09-27 09:20:05 +00:00
zx 3da0f0f487 [feat][assistant][I3J6V0] add new audio operator TrebleBiquad 2021-09-27 13:01:26 +08:00
despicablemme d072a4e4bf [feat] [assistant] [I40GZE] add new data operator RandomAdjustSharpness 2021-09-27 11:19:33 +08:00
i-robot ab567574c4 !21451 [MS][crowdfunding]New operator implementation, include RiaaBiquad 
Merge pull request !21451 from 张璇/riaa_biquad
 2021-09-27 03:14:16 +00:00
i-robot 2969688382 !20047 [assistant][ops]New operator implementation, include QmnistDataset 
Merge pull request !20047 from Wangsong95/qmnist_dataset
 2021-09-27 01:36:08 +00:00
despicablemme 7063907bdc [feat] [assidtant] [I40GZN] add new data ops RandomEqualize 2021-09-25 18:00:25 +08:00
i-robot d0023355a3 !23205 Fix bus error with multiprocessing enabled, large image, shared_memory, single device 
Merge pull request !23205 from RobinGrosman/mem_bus_failure
 2021-09-24 19:16:12 +00:00
RobinGrosman c90efc0403 add check for shm available when num_devices=1 to avoid bus errors 
-Required modifying tests to reduce memory usage to check doesn't fail during CI
-Also required fixing max_rowsize to be externalized from map+batch() function
 2021-09-24 10:24:31 -07:00
zx 1d95e6f480 [feat][assistant][I3J6UY] add new audio operator RiaaBiquad 2021-09-23 21:36:59 +08:00
despicablemme 7a6b75c2e8 [feat] [assistant] [I40GZJ] add new data op RandomAutoContrast 2021-09-23 20:17:39 +08:00
i-robot 0c26736da1 !22276 [MS][crowdfunding]New operator implementation, RandomInvert 
Merge pull request !22276 from yangwm/rinvert
 2021-09-23 08:12:57 +00:00
hetshah 2c1666fd06 set num epochs for iterators that use the default value 
add explicit paramter for infinite iterators

set appropriate value for num_epoch param for iteratos

fix long lines

remove trailing whitespace

fixed typo

remove comment

fixed comments

fixed test
 2021-09-22 17:32:50 +00:00
despicablemme b977d8723b [feat] [assistant] [I40GZC] add new data ops RandomInvert 2021-09-22 16:55:14 +08:00
ckczzj 001b68c286 [feat][assistant][I3J6VG] add new data operator QMnist 2021-09-22 12:13:27 +08:00
i-robot 6489dacdab !18114 [assistant][ops]Add new operator Magphase 
Merge pull request !18114 from YJfuel123/Magphase
 2021-09-18 07:26:34 +00:00
i-robot 92287b86d8 !23471 Fix white list error msg synax error 
Merge pull request !23471 from xiefangqi/md_white_list_optimization
 2021-09-18 05:03:40 +00:00
葛崇林 a05a5de073 [feat][assistant][I3ALC5] add new method Magphase 2021-09-17 16:02:46 +08:00
i-robot f5af432e04 !16922 [assistant][Vol] 
Merge pull request !16922 from QingfengLi/Vol
 2021-09-17 06:51:49 +00:00
chenx2ovo 9e62623b4a [feat][assistant][I3CEGI] add op vol 2021-09-17 09:34:20 +08:00
ms_yan 0a365e91bf modify log in kernel part 2021-09-16 22:09:21 +08:00
Guo Zhijian f49bacde09 fix: pyfunc hung when it's a yield func 2021-09-16 09:20:44 +08:00
i-robot 57b8c8e2d9 !23310 [MD] fix error message of mindrecord 
Merge pull request !23310 from liyong126/fix_log_msg
 2021-09-16 00:45:11 +00:00
i-robot e5af07d519 !23538 minddata error message optimize 
Merge pull request !23538 from luoyang/log_update
 2021-09-16 00:42:42 +00:00
i-robot 4a553c8d57 !16607 [assistant][Fade] 
Merge pull request !16607 from QingfengLi/Fade
 2021-09-15 14:54:16 +00:00
YangLuo 9d89de5765 minddata error message optimize 2021-09-15 18:05:02 +08:00
liyong 65f3365a2a fix mindrecord log 2021-09-15 16:18:40 +08:00
jonyguo c408867085 fix: hung when user return Tensor in generator 2021-09-15 14:13:33 +08:00
xiefangqi a91c87c581 fix some syntax error 2021-09-14 21:29:02 +08:00
chenx2ovo 1a3196b052 [feat][assistant][I3CEGF] add op fade 2021-09-14 19:08:37 +08:00
i-robot 32c2b77595 !19093 [assistant][ops] Add new audio operator Biquad 
Merge pull request !19093 from 杨旭华/Biquad
 2021-09-14 01:52:01 +00:00
i-robot 6dcfe64501 !23264 python serdes bug fix 
Merge pull request !23264 from zetongzhao/deserialize
 2021-09-13 15:53:05 +00:00
i-robot 0aa167cdf4 !23240 maybe max open files in mr or onehot fail in pytransform 
Merge pull request !23240 from guozhijian/fix_max_open_files_and_onehot
 2021-09-13 13:51:09 +00:00
i-robot 5abd4821ce !23357 Add DatasetName method for out developer 
Merge pull request !23357 from xiefangqi/md_white_list_optimization
 2021-09-13 12:06:49 +00:00
i-robot 3e1d1ff4d3 !18671 [assistant][ops]New operator implementation, include SBUDataset 
Merge pull request !18671 from Wangsong95/sbu_dataset
 2021-09-13 11:56:13 +00:00
liannai 631f49f044 [feat][assistant][I3T96A] Add new audio operator Biquad 2021-09-13 19:18:50 +08:00
xiefangqi 465dd25d74 add virtual DatasetName method 2021-09-13 15:17:32 +08:00
i-robot 87381a1cf9 !17806 [assistant][ops] DcShift Op 
Merge pull request !17806 from 杨旭华/DCShift
 2021-09-13 01:29:14 +00:00
i-robot c9ff7242a4 !23298 minddata engine optimize white list error msg 
Merge pull request !23298 from xiefangqi/md_white_list_optimization
 2021-09-13 01:19:15 +00:00
Carry955 0ba84345b0 [feat][assistant][I3J6U3] add new audio operator DCShift 2021-09-11 23:58:18 -07:00
jonyguo da78a17730 fix: may max openfiles in mindrecord and onehot error in py_transform 
and cache_admin rpath
 2021-09-12 00:25:07 +08:00
xiefangqi ff73e4ccd7 fix white list error msg 2021-09-11 17:36:23 +08:00
Isaac17555 53fc99b914 [feat][assistant][I3T96B]add new data operator LFilter 2021-09-11 02:05:43 -07:00
YJfuel123 16150d394f [feat][assistant][I3ALd1] add new operator EqualizerBiquad 2021-09-11 10:36:20 +08:00
ckczzj af97ba4a77 [feat][assistant][I3J6VH] add new data operator SBU 2021-09-11 09:58:17 +08:00
zetongzhao 719a207040 deserialize fix 2021-09-10 15:19:59 -04:00
i-robot 7b4f361ca7 !23232 Fix multiprocessing attr error 
Merge pull request !23232 from xiefangqi/md_fix_map_pool_issue
 2021-09-10 14:59:40 +00:00
i-robot 6d709c7254 !19624 [assistant][ops] Add new dataset operator DIV2K. 
Merge pull request !19624 from Rainfor/div2k
 2021-09-10 14:59:21 +00:00
xiefangqi 66c5f937dd fix attr failed problem 2021-09-10 18:01:04 +08:00
wangkc123 3daf3a4d5f [feat] [assistant] [I3CKEK] add new dataset operator DIV2K 2021-09-10 17:55:34 +08:00
robert_luo_yibo ce71699734 [feat][assistant][I3CEG5] add new data OP MuLawDecoding 2021-09-10 15:49:46 +08:00
i-robot 32cc5d79b0 !18670 [assistant][ops]New operator implementation, include USPSDataset 
Merge pull request !18670 from Wangsong95/usps_dataset
 2021-09-10 03:06:19 +00:00
i-robot f91e1440c8 !17810 [assistant][ops][DeemphBiquadOp] 
Merge pull request !17810 from Isaac/DeemphBiquad
 2021-09-10 03:05:41 +00:00
ckczzj d7c702e9d4 [feat][assistant][I3J6VL] add new data operator USPS 2021-09-09 21:46:57 +08:00
i-robot 936940da8e !22666 [assistant][ComplexNorm] 
Merge pull request !22666 from QingfengLi/ComplexNorm
 2021-09-09 06:41:53 +00:00
Isaac 5dc6c7fe96 [feat][assistant][I3J6U6]add new data operator DeemphBiquad 2021-09-09 14:34:24 +08:00
i-robot 9ad56e86ab !17846 [assistant][ops]Add new operator HighpassBiquad 
Merge pull request !17846 from YJfuel123/HighpassBiquad
 2021-09-09 02:06:41 +00:00
i-robot a918783f1a !22623 Add Random operation with Multi Field 
Merge pull request !22623 from shenwei41/operation_dev
 2021-09-09 01:49:04 +00:00
i-robot b21da536d3 !22784 serdes python operations by pyfunc 
Merge pull request !22784 from zetongzhao/deserialize
 2021-09-08 17:18:57 +00:00
zetongzhao 0d389f3f0c serdes python operations 2021-09-08 09:05:11 -04:00
shenwei41 a8d02ab782 Add Random operation with MultiField 2021-09-08 19:29:23 +08:00
i-robot 938324b397 !19362 [assistant][ops] Add new dataset operator Cityscapes. 
Merge pull request !19362 from Rainfor/cityscapes
 2021-09-08 09:14:43 +00:00
yaoyuan 85a1b55b47 [feat] [assistant] [I3ALCL] add new operator HighpassBiquad 2021-09-08 15:12:40 +08:00
i-robot b1be8dfd31 !22851 MD Profiling: Update Connector Init to Remove any existing file 
Merge pull request !22851 from cathwong/ckw_mon_seq_pipelines_fix
 2021-09-07 19:57:58 +00:00
xiefangqi 57f68d35bd fix generator 14 ut problem 2021-09-06 19:01:51 +08:00
Cathy Wong bc85c606b8 MD Profiling: Update Connector Init to Remove any existing file 
to fix sequential pipeline scenario.
 2021-09-03 15:45:38 -04:00
i-robot 3368287aab !22620 add util: browse dataset 
Merge pull request !22620 from luoyang/browse_dataset
 2021-09-02 01:00:46 +00:00
robert_luo_yibo 3d58da9023 [fix][assistant][I47V0O] fix bug about the wrong output in the operator ComplexNorm 2021-09-01 17:55:38 +08:00
YangLuo 97c1ba380f add util: browse dataset 2021-09-01 16:13:32 +08:00
wangkc123 70f741bcc1 [feat] [assistant] [I3CKEM] add new dataset operator Cityscapes 2021-08-31 20:03:37 +08:00
tongyue 329526dea8 [feat][assistant][I3J6U0] add new data operator Contrast 2021-08-30 19:34:37 -07:00
i-robot bbf9c6a9ad !22529 fix: for loop dataset iterator will cause memory leak 
Merge pull request !22529 from guozhijian/fix_for_loop_ds_iter_memory_leak
 2021-08-31 01:15:58 +00:00
i-robot a47ab8d643 !18073 [assistant][ops]Add new operator LowpassBiquad 
Merge pull request !18073 from YJfuel123/LowpassBiquad
 2021-08-30 08:33:39 +00:00
jonyguo 4c915fdcbb fix: for loop dataset iterator will cause memory leak 2021-08-30 16:08:26 +08:00
liyong 5a4ff8cfdb fix ci issue 2021-08-30 10:18:06 +08:00
i-robot 1639023909 !22263 Add convertcolor operation 
Merge pull request !22263 from shenwei41/operation_dev
 2021-08-27 12:33:43 +00:00
liyong c257cf36e2 refactor mindrecord 2021-08-27 09:15:05 +08:00
shenwei41 1295744524 Add ConvertColor Operation 2021-08-26 21:14:28 +08:00
haizhouye d96d9e64b0 [feat] [assistant] [I3ALCA] add new LowpassBiquad operator 
[feat] [assistant] [I3ALCA] add new LowpassBiquad operator

bugfix

fix

fix
 2021-08-26 18:49:21 +08:00
i-robot 956ae2bbfd !22223 [assistant][TimeMask and FreqMask] 
Merge pull request !22223 from QingfengLi/TimeFreqMask
 2021-08-25 10:43:24 +00:00
chenx2ovo 2f1d974eeb [fix] [assistant] [I465ZG] revise error message and change input param datatype 2021-08-25 10:47:27 +08:00
zhangyihui 3d19949eb4 device_id to rank_id 2021-08-25 09:52:23 +08:00
i-robot 6082835501 !22146 dataset: support return bytes in GeneratorDataset 
Merge pull request !22146 from ms_yan/decode_bytes
 2021-08-24 02:41:18 +00:00
i-robot db44b88e1e !22094 MD Profiling: For mismatch op info btwn files, skip bottleneck analysis 
Merge pull request !22094 from cathwong/ckw_mon_py_analyze_fixes3
 2021-08-23 18:28:51 +00:00
ms_yan 6ecfc34524 add np.frombuffer to support return bytes 2021-08-23 17:31:38 +08:00
ms_yan 36a8886ca2 Revert "[feat] [assistant] [I3T96T] add new Dataset operator CMUARCTICDataset" 
This reverts commit b077aa1cab.

Revert "[feat] [assistant] [I3T96X] add new Dataset operator LibriSpeechDataset"

This reverts commit 4e6f7dc97d.

delete pass_registry_test.cc

comment  hiai_nlu_model_multi.pb related  line
 2021-08-23 01:46:38 +08:00
djc b077aa1cab [feat] [assistant] [I3T96T] add new Dataset operator CMUARCTICDataset 2021-08-22 16:26:45 +08:00
djc 4e6f7dc97d [feat] [assistant] [I3T96X] add new Dataset operator LibriSpeechDataset 2021-08-22 13:39:37 +08:00
i-robot 0e7844daa9 !22093 revert merge request 21685 
Merge pull request !22093 from guozhijian/revert-merge-21685-master
 2021-08-21 09:25:32 +00:00
i-robot 4ba337e0f5 !16667 [assistant][ComplexNorm] 
Merge pull request !16667 from QingfengLi/ComplexNorm
 2021-08-21 02:44:18 +00:00
i-robot 5d1bb097e2 !22052 Fix probability core dump in MindData coverage test 
Merge pull request !22052 from luoyang/issues2
 2021-08-20 01:18:06 +00:00
robert_luo_yibo d837cbee0f [feat][assistant][I3CEG8] add new data operator ComplexNorm 2021-08-20 09:07:21 +08:00
Cathy Wong 85931175bc MD Profiling: For mismatch op info btwn files, skip bottleneck analysis. 
Add to summary output: per_pipeline_time and per_push_queue_time.
Enhance UT
 2021-08-19 12:19:59 -04:00
guozhijian e1df9a2152 revert 'Pull Request !21685 : add function to invert_op' 2021-08-19 22:49:51 +08:00
i-robot 4113aee7d6 !18668 [assistant][ops]New operator implementation, include Flowers102Dataset 
Merge pull request !18668 from Wangsong95/flowers102_dataset
 2021-08-19 08:41:44 +00:00
YangLuo 9352935ff6 Fix probability core dump in MindData coverage test 2021-08-19 14:34:57 +08:00
i-robot 3ad641cb65 !21685 add function to invert_op 
Merge pull request !21685 from Killer/fake_killer
 2021-08-19 05:13:39 +00:00
fake_killer b0bc4bf5da [feat] [assistant] [I44TRS] extend invert_op's function 2021-08-18 21:46:15 +08:00
Xiao Tianci aae886bcd7 fix crowd contributing code 2021-08-16 19:33:10 +08:00
i-robot f99effd430 !21755 Fix batch usability problem 
Merge pull request !21755 from xiefangqi/md_fix_batch_usability
 2021-08-13 07:21:49 +00:00
i-robot d1112f8e9c !17631 [assistant][FrequencyMasking] 
Merge pull request !17631 from QingfengLi/FreqMask
 2021-08-13 06:48:06 +00:00
xiefangqi e4894a1bf1 fix batch usability problem 2021-08-12 20:12:43 +08:00
chenx2ovo 53615010a3 [feat][assistant][I3CEGG] add new data OP FrequencyMasking 2021-08-12 15:28:30 +08:00
jonyguo 42b3512a1b fix: when field is none in mr 2021-08-11 17:11:38 +08:00
chenx2ovo e72acf362c [feat][assistant][I3CEGH] add op timemasking 2021-08-10 12:06:07 +08:00
i-robot dc626617fc !19526 [assistant][TimeStretch] 
Merge pull request !19526 from QingfengLi/TimeStretch
 2021-08-09 09:40:38 +00:00
shenwei41 59efbb88ec fix waring 2021-08-09 11:40:52 +08:00
chenx2ovo 04705e5b0d [feat][assistant][I3CEGC] add op timestretch 2021-08-07 22:53:41 +08:00
i-robot c84e09cf45 !21072 Deserialize - python binding 
Merge pull request !21072 from zetongzhao/deserialize_1
 2021-08-06 21:47:09 +00:00
zetongzhao 4f199b548c deserialize and tests 2021-08-06 13:35:27 -04:00
i-robot 51a1ca61f2 !21357 [MD] fix SlicePatches when dealing with some special tensor. 
Merge pull request !21357 from liyong126/fix_bug
 2021-08-06 08:58:27 +00:00
liyong 5fea72473c fix HW1 issue 2021-08-06 14:07:26 +08:00
ckczzj 9d9f33be88 [feat][assistant][I3J6VN] add new data operator flowers102 2021-08-06 10:55:12 +08:00
chenx2ovo aa45e2ebbc [feat] [assistant] [I3CEER] add op AmplitudeToDB 2021-08-06 10:07:29 +08:00
Carry955 98be266585 [feat][assistant][I3J6TQ] add a new operator Angle 2021-08-04 19:41:00 -07:00
i-robot 680e40a43a !16453 [assistant][BassBiquadOp] 
Merge pull request !16453 from StyleHang/BassBiquadOp
 2021-08-04 10:02:34 +00:00
i-robot f4e07c0dce !17486 [assistant][ops] Add new dataset operator Flickr. 
Merge pull request !17486 from Rainfor/wangkc
 2021-08-04 08:52:21 +00:00
i-robot d260f38936 !15722 [assistant][AdjustGamma] 
Merge pull request !15722 from QingfengLi/adjust_gamma
 2021-08-04 03:58:09 +00:00
li-qiyao ba326d76a8 [feat][assistant][I3J6TW] add new data operator BassBiquad 2021-08-03 19:40:12 -07:00
wangkc123 48f5fed1f8 [feat] [assistant] [I3CKEN] add new dataset operator Flickr 2021-08-04 10:20:43 +08:00
chenx2ovo 4191dde45f [feat][assistant][I3CEGM] add op adjust gamma 2021-08-04 02:32:13 +08:00
tongyue 08469e113a [feat][assistant][I3J6TV]add new audio operator BandrejectBiquad 2021-08-03 05:41:53 -07:00
i-robot 2c7b8e73df !16452 [assistant][BandpassBiquadOp] 
Merge pull request !16452 from StyleHang/BandpassBiquadOp
 2021-08-03 11:08:48 +00:00
i-robot 93af13f332 !19979 dataset: modify some error msg 
Merge pull request !19979 from ms_yan/err_msg
 2021-08-03 08:53:30 +00:00
i-robot ef77050348 !18892 [assistant][ops] Add new dataset operator SBD. 
Merge pull request !18892 from Rainfor/sbd
 2021-08-03 03:33:00 +00:00
ms_yan 65870045bc modify some error msg 2021-08-03 09:16:28 +08:00
administrator 29d57a89fb [feat][assistant][I3TDRD]add new audio operator BandpassBiquad 2021-08-02 07:08:47 -07:00
Isaac b49e1acc3e [feat] [assistant] [I3J6TM] add new audio operator AllpassBiquad 2021-07-31 16:15:17 +08:00
i-robot 9301de1b54 !21098 fix bug 
Merge pull request !21098 from shenwei41/fix_waring
 2021-07-31 01:12:21 +00:00
shenwei41 e3385e2564 fix waring to master 2021-07-30 16:37:27 +08:00
i-robot 6e84bf0f0f !16449 [assistant][ops] Add new audio operator BandBiquad 
Merge pull request !16449 from StyleHang/BandBiquadOp
 2021-07-30 03:35:07 +00:00
liannai 5a341e7e27 [feat] [assistant] [I3J6TS] add new audio operator BandBiquad 2021-07-29 17:57:27 +08:00
Xiao Tianci 53c01c437d fix NumpySlicesDataset not accept numpy 2021-07-22 15:46:37 +08:00
wangkc123 b381e0be9d [feat] [assistant] [I3CKER] add new dataset operator SBD 2021-07-17 21:02:30 +08:00
i-robot 8508c9eb45 !18179 [MD] add slice patches op in c++ and python 
Merge pull request !18179 from liyong126/slice_patches_op
 2021-07-05 07:04:45 +00:00
liyong 9a1de0af65 add slice patches in python and c++ 2021-07-05 09:16:47 +08:00
Xiao Tianci 8cdd0d8d90 fix param check for Rotate and GaussianBlur 2021-07-01 20:31:33 +08:00
YangLuo 17c121b087 Give concreted error when mappable dataset lack __len__ method 2021-06-29 11:13:12 +08:00
Lixia Chen 1f03b16e0e Fix cache_admin server start stderr output 
Fix TFRecord get_dataset_size() when cached
 2021-06-23 18:32:54 -04:00
Harshvardhan Gupta 6845a16b44 fix coredump with large max_rowsize and add arg validations 2021-06-23 09:31:11 -04:00
YangLuo 4e9f6d54f7 update cpp api & doc 2021-06-23 11:14:52 +08:00
chenx2ovo 4b98099dcb [feat][assistant][I3CEGQ] add op rgb2bgr 2021-06-21 16:16:50 +08:00
i-robot ab3259c43a !18292 MD Profiling UT: Check CPU util file. Add sequential pipeline UTs 
Merge pull request !18292 from cathwong/ckw_mon_py_analyze_ut4
 2021-06-17 21:47:10 +08:00
liyong 2989a5a23b fix some bugs 2021-06-17 11:56:10 +08:00
Cathy Wong c37eb350f6 MD Profiling UT: Check CPU util file. Add sequential pipeline UTs 2021-06-16 15:30:59 -04:00
Xiao Tianci a54e65a7ad add Crop Python API 2021-06-10 09:46:57 +08:00
i-robot e60f5f12f5 !17836 Add Rotate API for CV data processing 
Merge pull request !17836 from xiaotianci/rotate
 2021-06-10 09:29:18 +08:00
i-robot 50f11d5439 !17503 dataset: enhance the expression of error msg 
Merge pull request !17503 from ms_yan/log_modify
 2021-06-08 22:25:04 +08:00
Xiao Tianci 9c4c1c6a77 add Rotate Python API 2021-06-08 22:19:15 +08:00
i-robot 3a63a66d64 !17807 MD Profiling UT: Add MD Analyze UT for MinddataProfilingAnalyzer 
Merge pull request !17807 from cathwong/ckw_mon_py_analyze_ut3
 2021-06-08 21:52:55 +08:00
ms_yan be65ebcb28 complete dataset log 2021-06-08 16:45:47 +08:00
mindspore-ci-bot 8934e4a513 !17879 dynamic min max shape phase 2 
From: @luoyang42
Reviewed-by: @liucunwei,@jonyguo
Signed-off-by: @liucunwei
 2021-06-08 14:49:13 +08:00
Cathy Wong b7342efe12 MD Profiling UT: Add MD Analyze UT. Add more pipeline profiling UTs. 2021-06-07 11:00:34 -04:00
luoyang dad289c679 dynamic min max shape phase 2 2021-06-07 20:06:26 +08:00
Zhenglong Li 2c7d1bf603 Implement new gnn feature 2021-06-07 10:20:06 +08:00
mindspore-ci-bot b52f9ddb08 !17060 Add HorizontalFlip and VerticalFlip for CV data processing 
From: @tiancixiao
Reviewed-by: @liucunwei
Signed-off-by: @liucunwei
 2021-06-02 17:48:28 +08:00
mindspore-ci-bot c4757f0692 !17461 MD Profiling Analyze: Handle cpu util with missing ops. Enhance UTs. 
From: @cathwong
Reviewed-by: @robingrosman
Signed-off-by: @robingrosman
 2021-06-02 11:12:59 +08:00
mindspore-ci-bot ae2e7c1288 !17541 Fix MD Cache code check warnings [master] 
From: @lixiachen
Reviewed-by: @pandoublefeng,@robingrosman
Signed-off-by: @robingrosman
 2021-06-02 04:53:54 +08:00
Lixia Chen 8c1b561149 Fix cache code check warnings 2021-06-01 10:02:38 -04:00
Cathy Wong f8beb933ad MD Profiling Analyze: Handle cpu util with missing ops. Enhance UTs. 2021-05-31 20:33:19 -04:00
Xiao Tianci 9c74d55a49 add HorizontalFlip and VerticalFlip APIs 2021-05-31 15:55:21 +08:00
hesham 3a9695313b files_need_cleanup fixes 2021-05-30 15:44:08 -04:00
mindspore-ci-bot 2433778dac !15963 Support getting dynamic_min_max_shape of data 
From: @luoyang42
Reviewed-by: 
Signed-off-by:
 2021-05-26 09:41:02 +08:00
mindspore-ci-bot 730e211e52 !16702 Add GaussianBlur API for CV data processing 
From: @tiancixiao
Reviewed-by: @liucunwei,@jonyguo
Signed-off-by: @liucunwei
 2021-05-26 09:09:41 +08:00
mindspore-ci-bot d0978f709a !16167 CocoDataset add a new column according a new arg extra_metadata 
From: @xiefangqi
Reviewed-by: 
Signed-off-by:
 2021-05-25 19:55:41 +08:00
mindspore-ci-bot e513786709 !16565 Bug in Normalize when num_channels > 128 
From: @hfarahat
Reviewed-by: @robingrosman,@pandoublefeng
Signed-off-by: @robingrosman
 2021-05-25 10:24:25 +08:00
xiefangqi 53891dbdf7 coco add img_id column 2021-05-24 17:19:58 +08:00
luoyang 13ef5b61f3 support getting dynamic_min_max_shape of data 2021-05-24 17:04:50 +08:00
Xiao Tianci 3f495bbae1 add GaussianBlur API 2021-05-24 16:27:05 +08:00
mindspore-ci-bot 05bbd1d8fa !16426 Change the permission of cache shared memory to 600 for security 
From: @lixiachen
Reviewed-by: @robingrosman,@pandoublefeng
Signed-off-by: @robingrosman
 2021-05-20 02:36:26 +08:00
mindspore-ci-bot 1cb2d31642 !16509 GNN dataset new feature for invalid edge or node id case 
From: @lizhenglong1992
Reviewed-by: @liucunwei,@heleiwang
Signed-off-by: @liucunwei
 2021-05-19 09:20:24 +08:00
hesham bc92011b5f Bug in Normalize when num_channels > 128 2021-05-18 12:23:11 -04:00
Zhenglong Li 9284b6d6af Add special case returned value for get feature function 2021-05-18 15:53:01 +08:00
mindspore-ci-bot ec9fc5fb3b !16384 Fix python multiprocessing with shared memory for edge case 
From: @robingrosman
Reviewed-by: @mikef
Signed-off-by: @mikef
 2021-05-18 10:30:19 +08:00
Lixia Chen 8812111f94 change shared mem perm to 600 & minor other fixes 2021-05-17 17:19:55 -04:00
mindspore-ci-bot 601322356b !15638 VOCDataset need to add img_id for evaluation 
From: @xiefangqi
Reviewed-by: @liucunwei
Signed-off-by: @liucunwei
 2021-05-17 19:33:24 +08:00
xiefangqi 9b846c7873 voc add image_id column 2021-05-17 16:09:04 +08:00
RobinGrosman ca88d7d76a fix edge case and docstring in minddata multiprocessing shared memory code 2021-05-16 06:46:44 -07:00
jonyguo 9452142b50 enhance: enable shuffle infile / files when mindrecord 2021-05-14 17:41:18 +08:00
mindspore-ci-bot b5cddcb8cc !16036 Improve performance of multi-processing option for dataset 
From: @robingrosman
Reviewed-by: 
Signed-off-by:
 2021-05-13 01:41:16 +08:00
RobinGrosman 9a7d1fc034 Optimize multi-process in minddata by passing data via shared memory instead of multiprocessing.queue 2021-05-11 15:16:29 -07:00
mindspore-ci-bot ebed057e72 !16196 If Imagefolder input directory is not right, will occur out of memory error 
From: @xiefangqi
Reviewed-by: @ms_yan,@heleiwang,@jonyguo
Signed-off-by: @jonyguo
 2021-05-11 17:36:28 +08:00
mindspore-ci-bot d3a175e30f !16078 Modify Normalize to accept images with more than 3 channels 
From: @hfarahat
Reviewed-by: @mikef,@robingrosman
Signed-off-by: @robingrosman
 2021-05-11 02:03:34 +08:00
xiefangqi e0148f7181 fix minddata directory path problem 2021-05-10 22:16:37 +08:00
mindspore-ci-bot b86747b9a6 !15813 Convert some pipeline Ops to be inlined 
From: @hfarahat
Reviewed-by: 
Signed-off-by:
 2021-05-08 05:30:39 +08:00
hesham fd943bbdc0 Modify normalize to support multiple of channels 2021-05-07 17:06:26 -04:00
mindspore-ci-bot e0422e972e !16019 If dataset_dir is not a dir or something else validation is not right somehow 
From: @xiefangqi
Reviewed-by: @ms_yan,@liucunwei,@jonyguo
Signed-off-by: @liucunwei
 2021-05-07 11:57:31 +08:00
mindspore-ci-bot b85bf60eb8 !15727 VOCDataset read xml will filter invalid bbox automationally 
From: @xiefangqi
Reviewed-by: @heleiwang,@jonyguo
Signed-off-by: @jonyguo
 2021-05-07 09:50:53 +08:00
mindspore-ci-bot eb08d2e952 !16024 dataset: modify some error msg in dataset module 
From: @ms_yan
Reviewed-by: @jonyguo,@heleiwang
Signed-off-by: @jonyguo
 2021-05-07 09:12:46 +08:00
ms_yan 34dcd2b463 modify log of dataset part 2021-05-06 23:33:17 +08:00
xiefangqi b6e1fceedc minddata modify dir validation 2021-05-06 21:35:31 +08:00
Lixia Chen 5967cb2cd7 Unblock mindrecord under cache 2021-05-05 13:28:13 -04:00
hesham e4c0bd51a5 Modify concat to be an inlined Op 
Modify zip to be an inlined Op
Modify take to be an inlined Op
Modify skip to be an inlined Op
Modify rename to be an inlined Op
Modify filter to consume child's rows using one thread
 2021-05-04 12:45:18 -04:00
mindspore-ci-bot cefcf6c2ae !15931 Add eager tests for transform ops, and add a shape comparison in the EXPECT_MSTENSOR_EQ Macro 
From: @alexyuyue
Reviewed-by: @nsyca,@robingrosman
Signed-off-by: @robingrosman
 2021-04-30 06:15:12 +08:00
alex-yuyue 9ef4331f56 Add eager tests for transform ops 
Signed-off-by: alex-yuyue <yue.yu1@huawei.com>
 2021-04-29 13:19:24 -04:00
liyong 587247e793 add support save dynamic shape tensor 2021-04-27 19:15:10 +08:00
xiefangqi 458ff5fed9 md fix voc and manifest error scenario 2021-04-27 14:24:57 +08:00
dinglinhe 54fb72e2b3 Update normalization description at nn/layer/normalization.py 2021-04-20 12:30:37 +08:00
Cathy Wong 636809c255 dataset: C++ API ToNumber & Lookup: change data_type parm from string to DataType 2021-04-19 12:35:37 -04:00
mindspore-ci-bot f9c5060cf2 !14930 Handle ignored returned Status 
From: @hfarahat
Reviewed-by: @robingrosman,@pandoublefeng
Signed-off-by: @robingrosman
 2021-04-17 03:52:41 +08:00
hesham 6e6eaebb77 Fix unused return value in main code 2021-04-16 11:33:35 -04:00
alex-yuyue 97e3232b6d Add more ut for transform ops and subset sampler 
Signed-off-by: alex-yuyue <yue.yu1@huawei.com>
 2021-04-15 16:51:41 -04:00
mindspore-ci-bot 727203a498 !14911 dataset: Fill op: C++ API support, UTs and Pybind decoupling 
From: @cathwong
Reviewed-by: 
Signed-off-by:
 2021-04-14 05:32:11 +08:00
Cathy Wong 7e6a03487e dataset: Fill op: C++ API support, UTs and Pybind decoupling 2021-04-13 10:02:39 -04:00
mindspore-ci-bot e07a69d8f8 !14874 GNN dataset new feature 
From: @lizhenglong1992
Reviewed-by: 
Signed-off-by:
 2021-04-13 10:44:27 +08:00
Zhenglong Li f3895f2b09 Add get edges from nodes function for gnn network 2021-04-10 18:31:13 +08:00
mindspore-ci-bot f0f7bf1ed1 !14885 fix: wrong input order 
From: @jonyguo
Reviewed-by: @liucunwei,@heleiwang
Signed-off-by: @liucunwei
 2021-04-10 16:52:41 +08:00
jonyguo a5cc81dc97 fix: wrong input order in test_paddeddataset.py 2021-04-09 17:22:31 +08:00
Lixia Chen 1f2b178a16 Reset CacheClient's status for a new run 
Turn off lock only after all data is cached
 2021-04-08 15:30:48 -04:00
mindspore-ci-bot a9ba9b90f6 !14680 MindData Codespell correction 
From: @luoyang42
Reviewed-by: @heleiwang,@liucunwei
Signed-off-by: @liucunwei
 2021-04-07 18:28:55 +08:00
luoyang 624511337c MindData Codespell correction 2021-04-07 10:20:29 +08:00
hesham 8535e89e6c Remove DataBuffer 2021-04-05 17:55:59 -04:00
hesham 0d86cb1423 Remove rows_per_buffer 2021-04-04 00:22:16 -04:00
mindspore-ci-bot c99fe1e412 !12841 DataBuffer removal from DE 
From: @hfarahat
Reviewed-by: 
Signed-off-by:
 2021-04-04 03:28:13 +08:00
luoyang ec21d3f748 Fix validation of RGB2HSV/HSV2RGB/RandomResizedCrop in py_transform 
Fix error msg of pythontokenizer
 2021-04-01 19:09:07 +08:00
hesham a9a80693be Replace Databuffers with TensorRows 2021-03-31 17:00:36 -04:00
xulei2020 4122d1dcce fix code format bug 2021-03-25 09:18:36 +08:00
YangLuo f99204b292 fix python tokenizer 2021-03-18 15:47:09 +08:00
Lixia Chen 30017ea5a0 Fix codex issues 2021-03-15 11:15:15 -04:00
mindspore-ci-bot ac18da3770 !13061 [MD][bugfix] Fix to_json error: can't pickle pyfunc 
From: @xiefangqi
Reviewed-by: 
Signed-off-by:
 2021-03-13 12:48:26 +08:00
Xiao Tianci 4a7100c7f9 fix RandomAffine degree error and some examples in docs 2021-03-12 17:39:38 +08:00
mindspore-ci-bot 682a8926bd !13176 fix minddata transform issue 
From: @luoyang42
Reviewed-by: 
Signed-off-by:
 2021-03-12 17:01:56 +08:00
xiefangqi 41f3e02e87 fix pyfunc pickle issue 2021-03-12 15:17:08 +08:00
YangLuo 74d7bb41a6 fix minddata transform issue 2021-03-12 11:37:14 +08:00
mindspore-ci-bot f7040ca134 !13198 Fix get_dataset_size error with PKSampler 
From: @ziruiwu
Reviewed-by: 
Signed-off-by:
 2021-03-12 06:11:26 +08:00
mindspore-ci-bot 1bb789278d !13200 Fix CI warnings and test team issues 
From: @alexyuyue
Reviewed-by: @robingrosman,@nsyca
Signed-off-by: @robingrosman
 2021-03-12 05:38:01 +08:00
Zirui Wu 2692e3cc3d fix pk sampler's get_dataset error due to num_class unavaiable at pre-runtime 2021-03-11 15:43:02 -05:00
alex-yuyue 2457f52596 Fix CI warning and test team issues 
Signed-off-by: alex-yuyue <yue.yu1@huawei.com>
 2021-03-11 13:34:51 -05:00
RobinGrosman a0e75b845a Increase default level of parallelism in mindddata pipeline to 8. 
Adjust memory usage to not increase as parallelism increases.  it will
stay at same level it would be with 4 parallelism
 2021-03-10 18:38:11 -08:00
mindspore-ci-bot 3c7463bf2c !12943 adding fix to sample script 
From: @ezphlow
Reviewed-by: 
Signed-off-by:
 2021-03-10 10:22:19 +08:00
Eric ed373f61ac Adding inheritance to user defined sampler 
what if we just do nothing

trying wrapper approach

Fix yolo
 2021-03-09 18:14:49 -05:00
YangLuo 74431ad117 Fix bugs of minddata transforms 2021-03-06 16:41:51 +08:00
mindspore-ci-bot fe53c71b3b !12843 Add more callable test and clean the input check for tensor ops 
From: @alexyuyue
Reviewed-by: @robingrosman,@nsyca
Signed-off-by: @robingrosman
 2021-03-05 22:59:45 +08:00
alex-yuyue 6d7be32614 Add more callable test and clean the input check 
Signed-off-by: alex-yuyue <yue.yu1@huawei.com>
 2021-03-03 17:30:36 -05:00
mindspore-ci-bot 1c191a65fb !12765 Fix numpy input to samplers 
From: @hfarahat
Reviewed-by: 
Signed-off-by:
 2021-03-04 04:57:40 +08:00
mindspore-ci-bot c07371f5c9 !12763 Fix C++ API testcases 
From: @ezphlow
Reviewed-by: 
Signed-off-by:
 2021-03-04 04:57:28 +08:00
Eric f9a2379a70 Fixing api leftover 
Fixed compile error

Fixed more testcases
 2021-03-03 09:37:06 -05:00
alex-yuyue 66aff93940 Fix test team issue: ToNumber in eager mode. 
And support multiple tensor as input in call method

Signed-off-by: alex-yuyue <yue.yu1@huawei.com>
 2021-03-02 17:39:44 -05:00
hesham 616d1a1f3d Fix sampler supporting numpy input 2021-03-01 12:41:59 -05:00
hesham aa4298721a Fix sampler error messages 2021-02-23 23:58:19 -05:00
YangLuo d8ace8f041 fix minddata issues 2021-02-19 10:53:49 +08:00
mindspore-ci-bot c5df707b69 !12327 Block caching after random pyfunc cases 
From: @lixiachen
Reviewed-by: 
Signed-off-by:
 2021-02-17 08:56:10 +08:00
Lixia Chen 0667818d9a Block caching after random pyfunc 2021-02-12 14:10:00 -05:00
Mahdi 4b106f184c Fixed an issue with SequentialSampler 2021-02-10 12:55:10 -05:00
mindspore-ci-bot 260c77f4ad !12336 [MD] fix sampler log & transform log 
From: @luoyang42
Reviewed-by: 
Signed-off-by:
 2021-02-10 15:33:15 +08:00
luoyang 416ad23e9d fix resize & randomcoloradjust 
fix sampler
 2021-02-10 12:53:27 +08:00
mindspore-ci-bot 923dd47b0f !12331 Adding validators back to check method 
From: @ezphlow
Reviewed-by: @robingrosman,@nsyca,@robingrosman
Signed-off-by: @robingrosman
 2021-02-10 08:58:02 +08:00
Eric 5d91fa6d77 changed discription 
Added validator checks

Added builtin sampler check

Added proper input check for samplers
 2021-02-09 18:00:53 -05:00
hesham 58193bc469 User explicit deepcopy 2021-02-09 17:25:21 -05:00
luoyang cd86788cf3 fix bugs of execute & update doc description in minddata 2021-02-07 22:52:52 +08:00
TinaMengtingZhang ddab5ad417 Add support for vision op and py_transform op to serdes 2021-02-03 12:31:09 -05:00
mindspore-ci-bot e8f4cafd02 !11950 Overwrite default port/hostname from env variables for cache_admin 
From: @lixiachen
Reviewed-by: 
Signed-off-by:
 2021-02-03 23:38:48 +08:00
Lixia Chen 97cd142f8d Overwrite default port/hostname from env variables 
update copyright

add examples to CacheClient docstring
 2021-02-02 20:39:19 -05:00
hesham 1185218335 Support list of IDs as a sampler 2021-02-02 11:44:38 -05:00
luoyang fa4f67cbcd LookupOp: default value of unknown_token support NoneType 2021-02-02 16:17:02 +08:00
Nat Sutyanyong 5a7dc0accc Migrate repeat_pass.cc to IR optimizer and remove ExecTree optimizer 2021-02-01 12:30:16 -05:00
mindspore-ci-bot 320ea51308 !11853 Add call for decoupled image and text ops 
From: @alexyuyue
Reviewed-by: 
Signed-off-by:
 2021-02-01 19:47:00 +08:00
mindspore-ci-bot 794af6d094 !11834 [MD] Fix error msg description of dataset 
From: @xiefangqi
Reviewed-by: @liucunwei,@heleiwang
Signed-off-by: @liucunwei
 2021-01-30 14:49:24 +08:00
xiefangqi 3d72282f26 fix error msg issue 2021-01-30 11:03:53 +08:00
alex-yuyue 6fd58dc580 Add call for decoupled image and text ops 
Signed-off-by: alex-yuyue <yue.yu1@huawei.com>
 2021-01-29 21:36:31 -05:00
Mahdi 8f34faeb7a Changed bindings to SamplerObj 2021-01-29 14:20:05 -05:00
mindspore-ci-bot 20c1727b2c !11631 Add SubsetSampler 
From: @hfarahat
Reviewed-by: 
Signed-off-by:
 2021-01-29 03:48:45 +08:00
mindspore-ci-bot 8e4a6842ff !11681 gnn support edge weight sampling 
From: @heleiwang
Reviewed-by: @liucunwei
Signed-off-by: @liucunwei
 2021-01-28 19:15:37 +08:00
heleiwang 2dc9ba761c gnn support weight sampling neighbors. 2021-01-28 15:28:44 +08:00
hesham 7120c66998 Add SubsetSampler 2021-01-27 15:00:18 -05:00
mindspore-ci-bot 4cd6588af0 !11393 Migrating cache transform pass from execution tree to IR tree 
From: @lixiachen
Reviewed-by: 
Signed-off-by:
 2021-01-27 22:10:13 +08:00
mindspore-ci-bot f2fd357ffc !11623 fix issues of MD 
From: @luoyang42
Reviewed-by: 
Signed-off-by:
 2021-01-27 14:37:55 +08:00
luoyang 415c8b08a5 fix issues of MD: 
1. description of num_shard in datasets.py
2. add param validation for sampler
3. add description for add_child in sampler.py
 2021-01-27 09:30:42 +08:00
Lixia Chen 922e1f4f36 Migrate CacheTransformPass 2021-01-26 19:24:10 -05:00
mindspore-ci-bot 3708624a25 !11559 [MD] Change the default spill location of cache server 
From: @lixiachen
Reviewed-by: 
Signed-off-by:
 2021-01-27 03:12:24 +08:00
Lixia Chen be6d310946 Default to memory-only cache server 2021-01-25 18:38:24 -05:00
xiefangqi 3f76abcdee minddata err msg add file path 2021-01-22 16:38:46 +08:00
mindspore-ci-bot 2beb916bf5 !11427 TextOp decoupling 
From: @alexyuyue
Reviewed-by: 
Signed-off-by:
 2021-01-22 09:09:41 +08:00
mohammad b5e91139e2 add GetDatasetSize to concat node 2021-01-21 16:25:48 -05:00
alex-yuyue a8675f8227 TextOp decoupling 
Signed-off-by: alex-yuyue <yue.yu1@huawei.com>
 2021-01-20 21:52:45 -05:00
Lixia Chen 273865a72b Rename SamplerObj::Build() to SamplerBuild() 2021-01-18 15:42:34 -05:00
TinaMengtingZhang d3fdbceb61 [Part II] Push down json save logic to IR and add getter to each IR node 2021-01-15 18:03:37 -05:00
alex-yuyue b54e01bfb4 Tensor op decoupling stage 2 
Signed-off-by: alex-yuyue <yue.yu1@huawei.com>
 2021-01-14 11:51:16 -05:00
TinaMengtingZhang 4812fece80 [Part I] Push down json save logic to IR and add getter to each IR node 2021-01-12 11:17:45 -05:00
Nat Sutyanyong 686772cce7 Migrate GeneratorOp resetting logic to IR optimizer (incl. debug code) 
with comments from the peer view addressed
 2021-01-11 12:20:40 -05:00
mindspore-ci-bot 7957a3b6f5 !10988 Tensor op decoupling stage 1 
From: @ezphlow
Reviewed-by: 
Signed-off-by:
 2021-01-09 05:12:10 +08:00
Eric be46ccf721 Started vision decoupling 
map validate params

Changing bindings

Change behavior of vision.cc

Added signature and changed Cmake

Compiling

Need to fix compile

Added compiling random transforms

python changes for decoupling

two failed test case remaining

Compiling random choice

passes all ut

Changed assert

review comments

Added validate params

add back return value

Fix lint

py lint fix

pylint 2

Addressing comments
 2021-01-08 09:42:20 -05:00
Zirui Wu 0e68575e77 port over getter pass 
port over tensor_op_fusion pass

add fusion support for prebuildOpeartion (TensorOpeartion)
 2021-01-07 10:52:49 -05:00
mindspore-ci-bot 03e655f14a !10070 Fixing GetDatasetSize issue with chained samplers 
From: @mahdirahmanihanzaki
Reviewed-by: 
Signed-off-by:
 2021-01-06 05:11:51 +08:00
Xiao Tianci 31fed1a2f6 change code to import APIs from mindspore.dataset rather than mindspore.dataset.engine 2020-12-24 17:30:02 +08:00
mindspore-ci-bot 20df37e428 !10287 Add ut for dataset to_number 
From: @luoyang42
Reviewed-by: @heleiwang
Signed-off-by:
 2020-12-23 21:14:42 +08:00
Zirui Wu a1355a827c introduce a master switch to turn on and off profiler 2020-12-22 16:48:01 -05:00
YangLuo da6de519ae Add ut for dataset to_number 2020-12-21 20:09:03 +08:00
mindspore-ci-bot d99189f681 !10207 Fix some minor doc, typo related to AutoNumWorker 
From: @ziruiwu
Reviewed-by: @mikef
Signed-off-by:
 2020-12-19 06:35:59 +08:00
Zirui Wu 4ae7102fd3 address 2 tickets 2020-12-18 15:03:32 -05:00
Mahdi 98ea8fa6ea Added tests for chain samplers, fixed GetDatasetSize issue with chained samplers 2020-12-17 13:26:42 -05:00
mindspore-ci-bot fc11b7dd68 !10074 Fix a testcase error when shuffle is above cache 
From: @lixiachen
Reviewed-by: @liucunwei,@pandoublefeng
Signed-off-by: @liucunwei,@liucunwei
 2020-12-17 19:56:26 +08:00
mindspore-ci-bot 691e3baab6 !10109 [MD] Fix Decode doc & RandomNode log 
From: @luoyang42
Reviewed-by: @heleiwang,@liucunwei
Signed-off-by: @liucunwei
 2020-12-17 19:50:19 +08:00
mindspore-ci-bot 6035823c4b !10044 fix get num classes of concat 
From: @luoyang42
Reviewed-by: @liucunwei,@heleiwang,@liucunwei
Signed-off-by: @liucunwei,@liucunwei
 2020-12-17 19:30:50 +08:00
luoyang 595b7a782f [MD] Fix Decode doc & RandomNode log 2020-12-17 17:04:31 +08:00
Lixia Chen a7f0e13234 Remove special treatment for cache above repeat 2020-12-16 21:13:04 -05:00
YangLuo df9f4f41e8 fix get num classes of concat 2020-12-16 14:58:32 +08:00
Xiao Tianci 633f97bb48 fix get dataset size in distributed scenario 2020-12-15 20:41:45 +08:00
mindspore-ci-bot c4f284b928 !9910 [MD][Perf] MindData add NormalizePad for GPU performance 
From: @xiefangqi
Reviewed-by: 
Signed-off-by:
 2020-12-15 16:51:22 +08:00
xiefangqi 3c6cc3c58c Add NormalizePadOp 2020-12-15 11:45:41 +08:00
YangLuo 18e61a627a python eager add decode 2020-12-15 10:00:59 +08:00
mindspore-ci-bot ad6507d88a !9825 [MD] fix pad, normalize in c_trasnform 
From: @luoyang42
Reviewed-by: @heleiwang,@liucunwei
Signed-off-by: @liucunwei
 2020-12-14 10:48:10 +08:00
luoyang 1cc147df1d fix pad, normalize in c_trasnform 2020-12-11 17:46:29 +08:00
Zirui Wu 77521e78d2 staging 2020-12-10 14:45:04 -05:00
Zirui Wu 91a6b2b0ca Add checks to RandomData and LookUp 2020-12-09 17:09:34 -05:00
mindspore-ci-bot 317a97e6b9 !9336 auto num_parallel_workers setup 
From: @ziruiwu
Reviewed-by: 
Signed-off-by:
 2020-12-09 08:32:55 +08:00
mindspore-ci-bot a2c80435ce !9685 Fix a core dump in TreeConsumer::Terminate() plus minor cache fixes 
From: @lixiachen
Reviewed-by: @mikef,@nsyca
Signed-off-by: @nsyca
 2020-12-09 08:21:22 +08:00
Zirui Wu d6df1b0832 Implemented AutoNumWorker Pass which sets num_workers of selected parallel ops automatically if enabled 2020-12-08 17:49:05 -05:00
Lixia Chen 32b82c2737 Fix a core dump in TreeConsumer::Terminate() 
and other minor cache changes
 2020-12-08 15:38:40 -05:00
hesham ff65f6feb8 Remove batch_map multiprocess test case 2020-12-08 00:36:02 -05:00
mindspore-ci-bot aa5d4e08a9 !9544 Fix a possible seg fault in GenerateCRC caused by std::regex 
From: @lixiachen
Reviewed-by: @jonyguo,@heleiwang
Signed-off-by: @heleiwang
 2020-12-05 15:31:40 +08:00
Lixia Chen 27579fe858 Fix a seg fault caused by std::regex 2020-12-04 20:58:11 -05:00
Nat Sutyanyong 4cb78f2e03 Remove Repeat(1),Take(-1), and Skip(0) in NodeRemovalPass 2020-12-04 17:37:07 -05:00
mindspore-ci-bot 3280474d71 !9533 dataset Python Pushdown - more minor code rework; CSVDataset delim_field bug fix 
From: @cathwong
Reviewed-by: @nsyca,@mikef
Signed-off-by: @nsyca
 2020-12-05 06:01:13 +08:00
Jesse Lee 0c8dfc68ec Fix hang on RequestFreeTag 2020-12-04 14:54:30 -05:00
Cathy Wong 69b269c6cf dataset Python Pushdown - more minor code rework; CSVDataset delim_field bug fix 2020-12-04 14:39:52 -05:00
mindspore-ci-bot 4f607e9c26 !9335 Add multiprocessing support to per_batch_map 
From: @hfarahat
Reviewed-by: 
Signed-off-by:
 2020-12-04 15:13:46 +08:00
Nat Sutyanyong d69a29a44e Migrate 3 pre passes to IR optimizer, namely, cache_error_pass, epoch_injection, and removal_pass 2020-12-03 14:40:03 -05:00
alex-yuyue ccff3a5316 Change build to return status 
Signed-off-by: alex-yuyue <yue.yu1@huawei.com>
 2020-12-02 22:46:49 -05:00
hesham 809f9c91be Add multiprocessing support to per_batch_map 2020-12-02 15:40:14 -05:00
mindspore-ci-bot df44e1339e !9249 Optimizing GetDatasetSize 
From: @mahdirahmanihanzaki
Reviewed-by: 
Signed-off-by:
 2020-12-02 08:26:34 +08:00
mindspore-ci-bot 4ffd9b578f !9331 More cache minor fix for HQ testers 
From: @lixiachen
Reviewed-by: @mikef,@nsyca
Signed-off-by: @nsyca
 2020-12-02 06:45:15 +08:00