Nigel
/
mindspore
forked from mindspore-Ecosystem/mindspore
0
0
Fork 0
Code Issues Pull Requests Projects Releases Wiki Activity
22,898 Commits 24 Branches 0 Tags 3.9 GiB
Commit Graph

641 Commits

Author SHA1 Message Date
alex-yuyue 6d7be32614 Add more callable test and clean the input check 
Signed-off-by: alex-yuyue <yue.yu1@huawei.com>
 2021-03-03 17:30:36 -05:00
mindspore-ci-bot 1c191a65fb !12765 Fix numpy input to samplers 
From: @hfarahat
Reviewed-by: 
Signed-off-by:
 2021-03-04 04:57:40 +08:00
mindspore-ci-bot c07371f5c9 !12763 Fix C++ API testcases 
From: @ezphlow
Reviewed-by: 
Signed-off-by:
 2021-03-04 04:57:28 +08:00
Eric f9a2379a70 Fixing api leftover 
Fixed compile error

Fixed more testcases
 2021-03-03 09:37:06 -05:00
alex-yuyue 66aff93940 Fix test team issue: ToNumber in eager mode. 
And support multiple tensor as input in call method

Signed-off-by: alex-yuyue <yue.yu1@huawei.com>
 2021-03-02 17:39:44 -05:00
hesham 616d1a1f3d Fix sampler supporting numpy input 2021-03-01 12:41:59 -05:00
hesham aa4298721a Fix sampler error messages 2021-02-23 23:58:19 -05:00
YangLuo d8ace8f041 fix minddata issues 2021-02-19 10:53:49 +08:00
mindspore-ci-bot c5df707b69 !12327 Block caching after random pyfunc cases 
From: @lixiachen
Reviewed-by: 
Signed-off-by:
 2021-02-17 08:56:10 +08:00
Lixia Chen 0667818d9a Block caching after random pyfunc 2021-02-12 14:10:00 -05:00
Mahdi 4b106f184c Fixed an issue with SequentialSampler 2021-02-10 12:55:10 -05:00
mindspore-ci-bot 260c77f4ad !12336 [MD] fix sampler log & transform log 
From: @luoyang42
Reviewed-by: 
Signed-off-by:
 2021-02-10 15:33:15 +08:00
luoyang 416ad23e9d fix resize & randomcoloradjust 
fix sampler
 2021-02-10 12:53:27 +08:00
mindspore-ci-bot 923dd47b0f !12331 Adding validators back to check method 
From: @ezphlow
Reviewed-by: @robingrosman,@nsyca,@robingrosman
Signed-off-by: @robingrosman
 2021-02-10 08:58:02 +08:00
Eric 5d91fa6d77 changed discription 
Added validator checks

Added builtin sampler check

Added proper input check for samplers
 2021-02-09 18:00:53 -05:00
hesham 58193bc469 User explicit deepcopy 2021-02-09 17:25:21 -05:00
luoyang cd86788cf3 fix bugs of execute & update doc description in minddata 2021-02-07 22:52:52 +08:00
TinaMengtingZhang ddab5ad417 Add support for vision op and py_transform op to serdes 2021-02-03 12:31:09 -05:00
mindspore-ci-bot e8f4cafd02 !11950 Overwrite default port/hostname from env variables for cache_admin 
From: @lixiachen
Reviewed-by: 
Signed-off-by:
 2021-02-03 23:38:48 +08:00
Lixia Chen 97cd142f8d Overwrite default port/hostname from env variables 
update copyright

add examples to CacheClient docstring
 2021-02-02 20:39:19 -05:00
hesham 1185218335 Support list of IDs as a sampler 2021-02-02 11:44:38 -05:00
luoyang fa4f67cbcd LookupOp: default value of unknown_token support NoneType 2021-02-02 16:17:02 +08:00
Nat Sutyanyong 5a7dc0accc Migrate repeat_pass.cc to IR optimizer and remove ExecTree optimizer 2021-02-01 12:30:16 -05:00
mindspore-ci-bot 320ea51308 !11853 Add call for decoupled image and text ops 
From: @alexyuyue
Reviewed-by: 
Signed-off-by:
 2021-02-01 19:47:00 +08:00
mindspore-ci-bot 794af6d094 !11834 [MD] Fix error msg description of dataset 
From: @xiefangqi
Reviewed-by: @liucunwei,@heleiwang
Signed-off-by: @liucunwei
 2021-01-30 14:49:24 +08:00
xiefangqi 3d72282f26 fix error msg issue 2021-01-30 11:03:53 +08:00
alex-yuyue 6fd58dc580 Add call for decoupled image and text ops 
Signed-off-by: alex-yuyue <yue.yu1@huawei.com>
 2021-01-29 21:36:31 -05:00
Mahdi 8f34faeb7a Changed bindings to SamplerObj 2021-01-29 14:20:05 -05:00
mindspore-ci-bot 20c1727b2c !11631 Add SubsetSampler 
From: @hfarahat
Reviewed-by: 
Signed-off-by:
 2021-01-29 03:48:45 +08:00
mindspore-ci-bot 8e4a6842ff !11681 gnn support edge weight sampling 
From: @heleiwang
Reviewed-by: @liucunwei
Signed-off-by: @liucunwei
 2021-01-28 19:15:37 +08:00
heleiwang 2dc9ba761c gnn support weight sampling neighbors. 2021-01-28 15:28:44 +08:00
hesham 7120c66998 Add SubsetSampler 2021-01-27 15:00:18 -05:00
mindspore-ci-bot 4cd6588af0 !11393 Migrating cache transform pass from execution tree to IR tree 
From: @lixiachen
Reviewed-by: 
Signed-off-by:
 2021-01-27 22:10:13 +08:00
mindspore-ci-bot f2fd357ffc !11623 fix issues of MD 
From: @luoyang42
Reviewed-by: 
Signed-off-by:
 2021-01-27 14:37:55 +08:00
luoyang 415c8b08a5 fix issues of MD: 
1. description of num_shard in datasets.py
2. add param validation for sampler
3. add description for add_child in sampler.py
 2021-01-27 09:30:42 +08:00
Lixia Chen 922e1f4f36 Migrate CacheTransformPass 2021-01-26 19:24:10 -05:00
mindspore-ci-bot 3708624a25 !11559 [MD] Change the default spill location of cache server 
From: @lixiachen
Reviewed-by: 
Signed-off-by:
 2021-01-27 03:12:24 +08:00
Lixia Chen be6d310946 Default to memory-only cache server 2021-01-25 18:38:24 -05:00
xiefangqi 3f76abcdee minddata err msg add file path 2021-01-22 16:38:46 +08:00
mindspore-ci-bot 2beb916bf5 !11427 TextOp decoupling 
From: @alexyuyue
Reviewed-by: 
Signed-off-by:
 2021-01-22 09:09:41 +08:00
mohammad b5e91139e2 add GetDatasetSize to concat node 2021-01-21 16:25:48 -05:00
alex-yuyue a8675f8227 TextOp decoupling 
Signed-off-by: alex-yuyue <yue.yu1@huawei.com>
 2021-01-20 21:52:45 -05:00
Lixia Chen 273865a72b Rename SamplerObj::Build() to SamplerBuild() 2021-01-18 15:42:34 -05:00
TinaMengtingZhang d3fdbceb61 [Part II] Push down json save logic to IR and add getter to each IR node 2021-01-15 18:03:37 -05:00
alex-yuyue b54e01bfb4 Tensor op decoupling stage 2 
Signed-off-by: alex-yuyue <yue.yu1@huawei.com>
 2021-01-14 11:51:16 -05:00
TinaMengtingZhang 4812fece80 [Part I] Push down json save logic to IR and add getter to each IR node 2021-01-12 11:17:45 -05:00
Nat Sutyanyong 686772cce7 Migrate GeneratorOp resetting logic to IR optimizer (incl. debug code) 
with comments from the peer view addressed
 2021-01-11 12:20:40 -05:00
mindspore-ci-bot 7957a3b6f5 !10988 Tensor op decoupling stage 1 
From: @ezphlow
Reviewed-by: 
Signed-off-by:
 2021-01-09 05:12:10 +08:00
Eric be46ccf721 Started vision decoupling 
map validate params

Changing bindings

Change behavior of vision.cc

Added signature and changed Cmake

Compiling

Need to fix compile

Added compiling random transforms

python changes for decoupling

two failed test case remaining

Compiling random choice

passes all ut

Changed assert

review comments

Added validate params

add back return value

Fix lint

py lint fix

pylint 2

Addressing comments
 2021-01-08 09:42:20 -05:00
Zirui Wu 0e68575e77 port over getter pass 
port over tensor_op_fusion pass

add fusion support for prebuildOpeartion (TensorOpeartion)
 2021-01-07 10:52:49 -05:00
mindspore-ci-bot 03e655f14a !10070 Fixing GetDatasetSize issue with chained samplers 
From: @mahdirahmanihanzaki
Reviewed-by: 
Signed-off-by:
 2021-01-06 05:11:51 +08:00
Xiao Tianci 31fed1a2f6 change code to import APIs from mindspore.dataset rather than mindspore.dataset.engine 2020-12-24 17:30:02 +08:00
mindspore-ci-bot 20df37e428 !10287 Add ut for dataset to_number 
From: @luoyang42
Reviewed-by: @heleiwang
Signed-off-by:
 2020-12-23 21:14:42 +08:00
Zirui Wu a1355a827c introduce a master switch to turn on and off profiler 2020-12-22 16:48:01 -05:00
YangLuo da6de519ae Add ut for dataset to_number 2020-12-21 20:09:03 +08:00
mindspore-ci-bot d99189f681 !10207 Fix some minor doc, typo related to AutoNumWorker 
From: @ziruiwu
Reviewed-by: @mikef
Signed-off-by:
 2020-12-19 06:35:59 +08:00
Zirui Wu 4ae7102fd3 address 2 tickets 2020-12-18 15:03:32 -05:00
Mahdi 98ea8fa6ea Added tests for chain samplers, fixed GetDatasetSize issue with chained samplers 2020-12-17 13:26:42 -05:00
mindspore-ci-bot fc11b7dd68 !10074 Fix a testcase error when shuffle is above cache 
From: @lixiachen
Reviewed-by: @liucunwei,@pandoublefeng
Signed-off-by: @liucunwei,@liucunwei
 2020-12-17 19:56:26 +08:00
mindspore-ci-bot 691e3baab6 !10109 [MD] Fix Decode doc & RandomNode log 
From: @luoyang42
Reviewed-by: @heleiwang,@liucunwei
Signed-off-by: @liucunwei
 2020-12-17 19:50:19 +08:00
mindspore-ci-bot 6035823c4b !10044 fix get num classes of concat 
From: @luoyang42
Reviewed-by: @liucunwei,@heleiwang,@liucunwei
Signed-off-by: @liucunwei,@liucunwei
 2020-12-17 19:30:50 +08:00
luoyang 595b7a782f [MD] Fix Decode doc & RandomNode log 2020-12-17 17:04:31 +08:00
Lixia Chen a7f0e13234 Remove special treatment for cache above repeat 2020-12-16 21:13:04 -05:00
YangLuo df9f4f41e8 fix get num classes of concat 2020-12-16 14:58:32 +08:00
Xiao Tianci 633f97bb48 fix get dataset size in distributed scenario 2020-12-15 20:41:45 +08:00
mindspore-ci-bot c4f284b928 !9910 [MD][Perf] MindData add NormalizePad for GPU performance 
From: @xiefangqi
Reviewed-by: 
Signed-off-by:
 2020-12-15 16:51:22 +08:00
xiefangqi 3c6cc3c58c Add NormalizePadOp 2020-12-15 11:45:41 +08:00
YangLuo 18e61a627a python eager add decode 2020-12-15 10:00:59 +08:00
mindspore-ci-bot ad6507d88a !9825 [MD] fix pad, normalize in c_trasnform 
From: @luoyang42
Reviewed-by: @heleiwang,@liucunwei
Signed-off-by: @liucunwei
 2020-12-14 10:48:10 +08:00
luoyang 1cc147df1d fix pad, normalize in c_trasnform 2020-12-11 17:46:29 +08:00
Zirui Wu 77521e78d2 staging 2020-12-10 14:45:04 -05:00
Zirui Wu 91a6b2b0ca Add checks to RandomData and LookUp 2020-12-09 17:09:34 -05:00
mindspore-ci-bot 317a97e6b9 !9336 auto num_parallel_workers setup 
From: @ziruiwu
Reviewed-by: 
Signed-off-by:
 2020-12-09 08:32:55 +08:00
mindspore-ci-bot a2c80435ce !9685 Fix a core dump in TreeConsumer::Terminate() plus minor cache fixes 
From: @lixiachen
Reviewed-by: @mikef,@nsyca
Signed-off-by: @nsyca
 2020-12-09 08:21:22 +08:00
Zirui Wu d6df1b0832 Implemented AutoNumWorker Pass which sets num_workers of selected parallel ops automatically if enabled 2020-12-08 17:49:05 -05:00
Lixia Chen 32b82c2737 Fix a core dump in TreeConsumer::Terminate() 
and other minor cache changes
 2020-12-08 15:38:40 -05:00
hesham ff65f6feb8 Remove batch_map multiprocess test case 2020-12-08 00:36:02 -05:00
mindspore-ci-bot aa5d4e08a9 !9544 Fix a possible seg fault in GenerateCRC caused by std::regex 
From: @lixiachen
Reviewed-by: @jonyguo,@heleiwang
Signed-off-by: @heleiwang
 2020-12-05 15:31:40 +08:00
Lixia Chen 27579fe858 Fix a seg fault caused by std::regex 2020-12-04 20:58:11 -05:00
Nat Sutyanyong 4cb78f2e03 Remove Repeat(1),Take(-1), and Skip(0) in NodeRemovalPass 2020-12-04 17:37:07 -05:00
mindspore-ci-bot 3280474d71 !9533 dataset Python Pushdown - more minor code rework; CSVDataset delim_field bug fix 
From: @cathwong
Reviewed-by: @nsyca,@mikef
Signed-off-by: @nsyca
 2020-12-05 06:01:13 +08:00
Jesse Lee 0c8dfc68ec Fix hang on RequestFreeTag 2020-12-04 14:54:30 -05:00
Cathy Wong 69b269c6cf dataset Python Pushdown - more minor code rework; CSVDataset delim_field bug fix 2020-12-04 14:39:52 -05:00
mindspore-ci-bot 4f607e9c26 !9335 Add multiprocessing support to per_batch_map 
From: @hfarahat
Reviewed-by: 
Signed-off-by:
 2020-12-04 15:13:46 +08:00
Nat Sutyanyong d69a29a44e Migrate 3 pre passes to IR optimizer, namely, cache_error_pass, epoch_injection, and removal_pass 2020-12-03 14:40:03 -05:00
alex-yuyue ccff3a5316 Change build to return status 
Signed-off-by: alex-yuyue <yue.yu1@huawei.com>
 2020-12-02 22:46:49 -05:00
hesham 809f9c91be Add multiprocessing support to per_batch_map 2020-12-02 15:40:14 -05:00
mindspore-ci-bot df44e1339e !9249 Optimizing GetDatasetSize 
From: @mahdirahmanihanzaki
Reviewed-by: 
Signed-off-by:
 2020-12-02 08:26:34 +08:00
mindspore-ci-bot 4ffd9b578f !9331 More cache minor fix for HQ testers 
From: @lixiachen
Reviewed-by: @mikef,@nsyca
Signed-off-by: @nsyca
 2020-12-02 06:45:15 +08:00
Mahdi f80553bc6e Optimization for GetDatasetSize 2020-12-01 17:40:25 -05:00
Lixia Chen 2ddb1fb75e More CRC fix and blocking BatchOp under cache 2020-12-01 15:20:33 -05:00
mindspore-ci-bot 920bbf1541 !9277 fix schema & zip validation 
From: @luoyang42
Reviewed-by: @pandoublefeng,@liucunwei
Signed-off-by: @liucunwei
 2020-12-01 16:28:48 +08:00
YangLuo 5cc3b76577 fix schema & zip validation 2020-12-01 14:23:56 +08:00
mindspore-ci-bot af3c27d354 !9244 Fix core dump with concat and getter_pass 
From: @ziruiwu
Reviewed-by: 
Signed-off-by:
 2020-12-01 06:56:41 +08:00
mindspore-ci-bot c3e7a8255c !9248 Add input check to nlp operators 
From: @ezphlow
Reviewed-by: @nsyca,@mikef
Signed-off-by: @nsyca
 2020-12-01 06:45:26 +08:00
Eric cf802c839c Added missing input check for nlp operator 2020-11-30 16:23:40 -05:00
Zirui Wu 23487fae09 fix get_shape/type and a segfault in getter_pass 
add test case
 2020-11-30 16:22:33 -05:00
mindspore-ci-bot 762793cb8f !9175 fix a bug in random resize crop op and some lint 
From: @tiancixiao
Reviewed-by: @liucunwei,@jonyguo,@liucunwei
Signed-off-by: @liucunwei
 2020-11-30 20:03:51 +08:00
Xiao Tianci 12da8717b7 1.correct param check for vision op 2.fix config review 3.fix api doc 2020-11-30 16:12:31 +08:00
luoyang aa4bf9a3f2 [MD] Support python-eager: Resize, Rescale, Normalize, HWC2CHW, Pad 2020-11-28 10:06:53 +08:00