yystopf
/
mindspore
forked from mindspore-Ecosystem/mindspore
0
0
Fork 0
Code Issues Pull Requests Projects Releases Wiki Activity
45,876 Commits 23 Branches 0 Tags 3.9 GiB
Commit Graph

823 Commits

Author SHA1 Message Date
i-robot 6e84bf0f0f !16449 [assistant][ops] Add new audio operator BandBiquad 
Merge pull request !16449 from StyleHang/BandBiquadOp
 2021-07-30 03:35:07 +00:00
liannai 5a341e7e27 [feat] [assistant] [I3J6TS] add new audio operator BandBiquad 2021-07-29 17:57:27 +08:00
zetongzhao 8693fd730a deserialize and tests 2021-07-21 08:32:00 -04:00
i-robot 884f5ae1de !19764 [MD] Remove Builder class phase 3 
Merge pull request !19764 from harshvardhangupta/rem-build-2
 2021-07-12 13:25:34 +00:00
Xiao Tianci a77ed2a5c0 fix the param check of center in Rotate, and change the default value of center from {-1, -1} to {} 2021-07-09 15:24:27 +08:00
harshvardhangupta 52d1212239 remove builder 2021-07-08 16:55:52 -04:00
i-robot a5118ae5f2 !19507 Remove builder from CacheOp, CacheLookupOp, CacheMergeOp 
Merge pull request !19507 from lixiachen/remove_cacheop_builders
 2021-07-07 21:25:43 +00:00
Lixia Chen 946693e860 Remove builder from cache ops 2021-07-06 17:29:03 -04:00
i-robot 0a6732f6dc !19047 [MD] Remove Builder 
Merge pull request !19047 from harshvardhangupta/remove_builder
 2021-07-06 17:34:35 +00:00
Harshvardhan Gupta 5bd8fb57fc remove builder 2021-07-05 11:17:37 -04:00
liyong 9a1de0af65 add slice patches in python and c++ 2021-07-05 09:16:47 +08:00
YangLuo 4e9f6d54f7 update cpp api & doc 2021-06-23 11:14:52 +08:00
chenx2ovo 4b98099dcb [feat][assistant][I3CEGQ] add op rgb2bgr 2021-06-21 16:16:50 +08:00
Xiao Tianci a54e65a7ad add Crop Python API 2021-06-10 09:46:57 +08:00
Xiao Tianci 9c4c1c6a77 add Rotate Python API 2021-06-08 22:19:15 +08:00
Zhenglong Li 2c7d1bf603 Implement new gnn feature 2021-06-07 10:20:06 +08:00
mindspore-ci-bot cf27d3c8dc !12972 Removing Builder from dataset ops phase 1 
From: @ezphlow
Reviewed-by: 
Signed-off-by:
 2021-06-05 03:51:49 +08:00
EricZ 16837a4639 Removing Builder from dataset ops phase 1 2021-06-03 09:09:29 +08:00
Xiao Tianci 9c74d55a49 add HorizontalFlip and VerticalFlip APIs 2021-05-31 15:55:21 +08:00
zetongzhao 728b180329 delete_extra files 2021-05-26 16:41:27 -04:00
mindspore-ci-bot 686225ad2a !16382 Unity Input Order 
From: @zetongzhao
Reviewed-by: 
Signed-off-by:
 2021-05-26 21:28:36 +08:00
zetongzhao 4cb53823fd all the updated input order 
Fix FillOperation to_json function

Fix FillOperation to_json function

serialize tensor without testing

fix unity input order bug, next step is to test the tensor serialization

serialize tensor, testing code in tensor_test.cc

change for tensor serialization

changes after review

changes after review

clang-format the files

rename
 2021-05-25 18:26:43 -04:00
Xiao Tianci 3f495bbae1 add GaussianBlur API 2021-05-24 16:27:05 +08:00
mindspore-ci-bot 601322356b !15638 VOCDataset need to add img_id for evaluation 
From: @xiefangqi
Reviewed-by: @liucunwei
Signed-off-by: @liucunwei
 2021-05-17 19:33:24 +08:00
xiefangqi 9b846c7873 voc add image_id column 2021-05-17 16:09:04 +08:00
jonyguo 9452142b50 enhance: enable shuffle infile / files when mindrecord 2021-05-14 17:41:18 +08:00
mindspore-ci-bot b5cddcb8cc !16036 Improve performance of multi-processing option for dataset 
From: @robingrosman
Reviewed-by: 
Signed-off-by:
 2021-05-13 01:41:16 +08:00
RobinGrosman 9a7d1fc034 Optimize multi-process in minddata by passing data via shared memory instead of multiprocessing.queue 2021-05-11 15:16:29 -07:00
mindspore-ci-bot 6fcd6cab68 !16119 Fix minddata issues 
From: @luoyang42
Reviewed-by: @jonyguo,@liucunwei
Signed-off-by: @jonyguo,@liucunwei
 2021-05-11 14:33:29 +08:00
mindspore-ci-bot d3a175e30f !16078 Modify Normalize to accept images with more than 3 channels 
From: @hfarahat
Reviewed-by: @mikef,@robingrosman
Signed-off-by: @robingrosman
 2021-05-11 02:03:34 +08:00
YangLuo 0a4da64e13 Fix minddata issues 2021-05-10 11:12:00 +08:00
mindspore-ci-bot b86747b9a6 !15813 Convert some pipeline Ops to be inlined 
From: @hfarahat
Reviewed-by: 
Signed-off-by:
 2021-05-08 05:30:39 +08:00
hesham fd943bbdc0 Modify normalize to support multiple of channels 2021-05-07 17:06:26 -04:00
Lixia Chen 5967cb2cd7 Unblock mindrecord under cache 2021-05-05 13:28:13 -04:00
hesham e4c0bd51a5 Modify concat to be an inlined Op 
Modify zip to be an inlined Op
Modify take to be an inlined Op
Modify skip to be an inlined Op
Modify rename to be an inlined Op
Modify filter to consume child's rows using one thread
 2021-05-04 12:45:18 -04:00
mindspore-ci-bot 9c7cb65c3b !15801 Split sampler and vision IR files 
From: @mhmotallebi
Reviewed-by: 
Signed-off-by:
 2021-05-01 05:19:41 +08:00
mindspore-ci-bot cefcf6c2ae !15931 Add eager tests for transform ops, and add a shape comparison in the EXPECT_MSTENSOR_EQ Macro 
From: @alexyuyue
Reviewed-by: @nsyca,@robingrosman
Signed-off-by: @robingrosman
 2021-04-30 06:15:12 +08:00
mohammad 9ce28cd34c split sampler vision IR files 2021-04-29 16:30:18 -04:00
alex-yuyue 9ef4331f56 Add eager tests for transform ops 
Signed-off-by: alex-yuyue <yue.yu1@huawei.com>
 2021-04-29 13:19:24 -04:00
YangLuo bf48400c10 unify include file of minddata 2021-04-22 10:59:36 +08:00
Cathy Wong 636809c255 dataset: C++ API ToNumber & Lookup: change data_type parm from string to DataType 2021-04-19 12:35:37 -04:00
mindspore-ci-bot bd69a86931 !15124 Remove unused #include in cc files in MindData 
From: @hfarahat
Reviewed-by: @nsyca,@mikef,@robingrosman
Signed-off-by: @robingrosman
 2021-04-17 02:19:27 +08:00
mindspore-ci-bot 9c81c5d2a0 !15189 Add more ut for transform ops and subset sampler 
From: @alexyuyue
Reviewed-by: 
Signed-off-by:
 2021-04-17 01:46:13 +08:00
hesham 98f0f36a20 Removed unused include 2021-04-16 11:40:56 -04:00
mindspore-ci-bot 5b7ed56d57 !12521 [WIP] MindData C++ Plugin infrastructure with GDAL decode 
From: @ziruiwu
Reviewed-by: 
Signed-off-by:
 2021-04-16 20:37:18 +08:00
Zirui Wu 8ebc512d25 plugin infrastruture for mindddata 
add single element string tensor support for plugin tensor
 2021-04-15 21:11:16 -04:00
alex-yuyue 97e3232b6d Add more ut for transform ops and subset sampler 
Signed-off-by: alex-yuyue <yue.yu1@huawei.com>
 2021-04-15 16:51:41 -04:00
Cathy Wong a2ee0cb3e5 dataset UT: Update UTs to check for return Status 2021-04-14 17:52:31 -04:00
mindspore-ci-bot 1d89109d19 !14929 Cleanup stale comments about DataBuffers 
From: @hfarahat
Reviewed-by: @pandoublefeng,@robingrosman
Signed-off-by: @pandoublefeng
 2021-04-14 05:33:40 +08:00
mindspore-ci-bot 727203a498 !14911 dataset: Fill op: C++ API support, UTs and Pybind decoupling 
From: @cathwong
Reviewed-by: 
Signed-off-by:
 2021-04-14 05:32:11 +08:00
Cathy Wong 7e6a03487e dataset: Fill op: C++ API support, UTs and Pybind decoupling 2021-04-13 10:02:39 -04:00
mindspore-ci-bot e07a69d8f8 !14874 GNN dataset new feature 
From: @lizhenglong1992
Reviewed-by: 
Signed-off-by:
 2021-04-13 10:44:27 +08:00
hesham 15354569f2 Cleanup stale comments about DataBuffers 2021-04-12 14:26:59 -04:00
ms_yan ecb8d6bc08 add pil version cubic resize 2021-04-12 21:56:14 +08:00
Zhenglong Li f3895f2b09 Add get edges from nodes function for gnn network 2021-04-10 18:31:13 +08:00
mindspore-ci-bot 74f4c85315 !14750 Address some code issues in MD 
From: @hfarahat
Reviewed-by: @robingrosman,@mikef
Signed-off-by: @robingrosman
 2021-04-10 03:55:16 +08:00
hesham 97cf4e516e Address 3 code issues: 
1- access out of bound memory in toFlatIndex
2- CreateFromNPArray function is 50>
3- duplicate variable init
 2021-04-08 21:50:19 -04:00
Cathy Wong 82f4b3f757 dataset: C++ API TypeCast: change data_type parm from string to DataType 2021-04-08 11:47:58 -04:00
mindspore-ci-bot bdc5a9c88b !14579 TensorOp decoupling stage 3 (transform ops) 
From: @alexyuyue
Reviewed-by: 
Signed-off-by:
 2021-04-08 21:41:26 +08:00
mindspore-ci-bot b6cf16d8aa !14389 fix codex warning 
From: @luoyang42
Reviewed-by: @heleiwang,@liucunwei
Signed-off-by: @liucunwei
 2021-04-08 14:22:39 +08:00
alex-yuyue 83aaed09bb TensorOp decoupling stage 3 (transform ops) 
Signed-off-by: alex-yuyue <yue.yu1@huawei.com>
 2021-04-07 18:46:23 -04:00
mindspore-ci-bot a9ba9b90f6 !14680 MindData Codespell correction 
From: @luoyang42
Reviewed-by: @heleiwang,@liucunwei
Signed-off-by: @liucunwei
 2021-04-07 18:28:55 +08:00
luoyang 624511337c MindData Codespell correction 2021-04-07 10:20:29 +08:00
mindspore-ci-bot 3d85930cfe !14639 Remove DataBuffer class 
From: @hfarahat
Reviewed-by: @pandoublefeng,@robingrosman
Signed-off-by: @pandoublefeng
 2021-04-06 23:49:30 +08:00
hesham 8535e89e6c Remove DataBuffer 2021-04-05 17:55:59 -04:00
mindspore-ci-bot 0cfd8b7c9e !13364 Add WordpieceTokenizer and AddDict to Jieba 
From: @alexyuyue
Reviewed-by: 
Signed-off-by:
 2021-04-06 05:04:13 +08:00
hesham 0d86cb1423 Remove rows_per_buffer 2021-04-04 00:22:16 -04:00
hesham a9a80693be Replace Databuffers with TensorRows 2021-03-31 17:00:36 -04:00
liyong d5b7a67c71 fix asan issue 2021-03-31 14:18:50 +08:00
YangLuo b85a905eaf fix codex warning 2021-03-30 17:00:37 +08:00
Eric 38944eafc4 Added get dataset size and fix macro 2021-03-29 15:49:29 -04:00
mindspore-ci-bot a91e9c9294 !14106 reduce ut time of cutmix_batch and ConcatOp 
From: @luoyang42
Reviewed-by: @pandoublefeng,@liucunwei
Signed-off-by: @pandoublefeng,@liucunwei
 2021-03-25 22:58:35 +08:00
YangLuo f7ea18944d reduce ut time of cutmix_batch 2021-03-25 21:59:44 +08:00
mindspore-ci-bot 5926f7c386 !13906 【MD】【BUG】Conventional judgment on the parameter h w not less than zero 
From: @xulei2020
Reviewed-by: 
Signed-off-by:
 2021-03-25 14:10:42 +08:00
xulei2020 4122d1dcce fix code format bug 2021-03-25 09:18:36 +08:00
mindspore-ci-bot b1b99b97d7 !13750 Add check of ResizePreserveARWithFiller operation 
From: @shenwei41
Reviewed-by: 
Signed-off-by:
 2021-03-24 15:27:16 +08:00
shenwei41 d3265f0591 Add check of ResizePreserveARWithFiller operation 2021-03-23 23:30:53 +08:00
mindspore-ci-bot 06892b83ff !13575 Add check to RgbToGray 
From: @shenwei41
Reviewed-by: @tiancixiao,@liucunwei
Signed-off-by: @liucunwei
 2021-03-22 12:11:39 +08:00
Xiao Tianci b88a68c9e8 fix canny when size is 7 2021-03-19 17:26:59 +08:00
mindspore-ci-bot 0451a800bc !13571 Added seed to prevent random failures in ut 
From: @ezphlow
Reviewed-by: 
Signed-off-by:
 2021-03-19 17:17:42 +08:00
shenwei41 8a1f283efc Add check to RgbToGray 2021-03-19 10:35:35 +08:00
Eric a3c98d9d59 Added fix for random core dump 2021-03-18 21:15:50 -04:00
alex-yuyue 2d6d608fae Add WordpieceTokenizer to API and IR. Add AddDict to C++ API 
Signed-off-by: alex-yuyue <yue.yu1@huawei.com>
 2021-03-18 16:42:31 -04:00
mindspore-ci-bot b45f289801 !13504 Add check of GetPerspectiveTransform and GetAffineTransform 
From: @shenwei41
Reviewed-by: @liucunwei,@pandoublefeng
Signed-off-by: @liucunwei
 2021-03-18 09:57:52 +08:00
mindspore-ci-bot dd90f7f055 !13506 dataset UT: Reinstate INFO logging and data verification - part 3 
From: @cathwong
Reviewed-by: @robingrosman,@liucunwei
Signed-off-by: @robingrosman
 2021-03-18 09:35:22 +08:00
Cathy Wong 1775c4e83d dataset: Reinstate INFO logging and data verification - part 3 
Updates: GetItemAt, CreateFromVector, CreateScalar.
Add TEST_MS_LOG_MSTENSOR.
Wrap ASSERT_OK around Tensor function calls.
 2021-03-17 13:48:04 -04:00
shenwei41 89f109a3f3 add check of operation 2021-03-17 20:29:17 +08:00
Eric a7da38feeb Added example for ngram 2021-03-16 23:45:04 -04:00
mindspore-ci-bot c0aa5f8132 !13365 Fix random rotation input check 
From: @ezphlow
Reviewed-by: 
Signed-off-by:
 2021-03-17 09:42:37 +08:00
Eric c730df23e8 Added json fix 2 2021-03-16 18:32:42 -04:00
Eric 4be0ffb9ba removing type_id.h from header files 2021-03-16 16:51:10 -04:00
shenwei41 d1bb9d470e Add resizefiller 2021-03-16 20:35:27 +08:00
Eric 9b921953f9 misc issues 2021-03-15 22:30:04 -04:00
Lixia Chen 30017ea5a0 Fix codex issues 2021-03-15 11:15:15 -04:00
mindspore-ci-bot 0cf6fb5310 !13264 [MD] Fix input check bug in RandomSelectSubpolicyOperation and codex issues 
From: @tina_mengting_zhang
Reviewed-by: @robingrosman,@nsyca
Signed-off-by: @robingrosman
 2021-03-15 20:29:56 +08:00
mindspore-ci-bot 98a87ad969 !13220 Add data helper to C++ API 
From: @ezphlow
Reviewed-by: 
Signed-off-by:
 2021-03-13 09:52:54 +08:00
TinaMengtingZhang ac05546aeb fix input check bug in RandomSelectSubpolicy and codex issue 2021-03-12 16:04:38 -05:00
Eric 82f7c6f2c7 Moving data_helper 
exposing json

WIP fix for removing templates

Removing templates part 1

replacing core/constants.h

removed template part 2

Changed fake data to filler data

Fixing CI

Fix CI part 2

CI fix part 2

Fix CI 3

Fixed lizard for data_util

Fixed clang

Fixing CI part 4
 2021-03-12 15:30:34 -05:00
mindspore-ci-bot 483bb9de60 !13078 dataset UT: Reinstate INFO logging and data verification - part 2 
From: @cathwong
Reviewed-by: 
Signed-off-by:
 2021-03-13 03:55:34 +08:00
Cathy Wong 95bbaaf9f6 dataset: Reinstate INFO logging and data verification - part 2 
for image/vision datasets
 2021-03-12 10:07:45 -05:00
mindspore-ci-bot f7040ca134 !13198 Fix get_dataset_size error with PKSampler 
From: @ziruiwu
Reviewed-by: 
Signed-off-by:
 2021-03-12 06:11:26 +08:00
Zirui Wu 2692e3cc3d fix pk sampler's get_dataset error due to num_class unavaiable at pre-runtime 2021-03-11 15:43:02 -05:00
mindspore-ci-bot 236c874e64 !12862 Increase default level of parallelism in minddata pipeline 
From: @robingrosman
Reviewed-by: 
Signed-off-by:
 2021-03-12 04:23:02 +08:00
RobinGrosman a0e75b845a Increase default level of parallelism in mindddata pipeline to 8. 
Adjust memory usage to not increase as parallelism increases.  it will
stay at same level it would be with 4 parallelism
 2021-03-10 18:38:11 -08:00
mindspore-ci-bot 0febd9c891 !13017 Add RGB2GRAY Operation 
From: @shenwei41
Reviewed-by: 
Signed-off-by:
 2021-03-11 10:08:25 +08:00
Cathy Wong 71ad3bb808 dataset: Specify Sampler and TensorTransform C++ API classes as final 2021-03-10 15:25:20 -05:00
shenwei41 7b56d1772e Add RGB2GRAY operation 2021-03-10 20:59:50 +08:00
jonyguo c426cd5434 fix: increace deviation_ratio for gnn test case 2021-03-10 15:58:46 +08:00
mindspore-ci-bot 62a6f783b3 !12809 Added pull-based prototype 
From: @mahdirahmanihanzaki
Reviewed-by: 
Signed-off-by:
 2021-03-09 22:31:38 +08:00
Mahdi 9c3c36217e Added pull based prototype 2021-03-08 15:00:14 -05:00
YangLuo 74431ad117 Fix bugs of minddata transforms 2021-03-06 16:41:51 +08:00
Zirui Wu 8022001450 Remove datasets.h dependency on tensor.h and other related dependencies 
Replace std::function<TensorRow(TensorRow)> with std::function<MSTensorVec(MSTensor)> by using std::bind

Replace pad_info with MSTensor

Switch TensorShape/dataset::DataType to vector<int64> /mindspore::DataType

Change TypeId to mindspore::DataType in SchemaObj

address review cmts, add CreateFromMSTensor() in tensor.h/.cc
 2021-03-05 16:26:58 -05:00
mindspore-ci-bot ea8db7baf4 !12936 dataset UT: Reinstate INFO logging and data verification - part 1 
From: @cathwong
Reviewed-by: @robingrosman,@nsyca
Signed-off-by: @robingrosman
 2021-03-06 04:36:14 +08:00
Cathy Wong 521c313a3b dataset: Reinstate INFO logging and data verification - part1 2021-03-05 10:13:09 -05:00
mindspore-ci-bot 9baec79337 !12881 add GaussianBlur and Canny for lite_cv 
From: @tiancixiao
Reviewed-by: 
Signed-off-by:
 2021-03-05 17:33:20 +08:00
TinaMengtingZhang 6280582315 fix testcases in c_api_random_test.cc 
fix testcases in c_api_transforms_test.cc
 2021-03-04 11:20:23 -05:00
Xiao Tianci 449452c7ef add gaussian blur and canny for lite cv process 2021-03-04 22:27:48 +08:00
Eric f9a2379a70 Fixing api leftover 
Fixed compile error

Fixed more testcases
 2021-03-03 09:37:06 -05:00
mindspore-ci-bot 4f504ad2a3 !12760 Fix include file(samplers.h) contains internal head file & Fix execute ut 
From: @luoyang42
Reviewed-by: @pandoublefeng,@heleiwang,@liucunwei
Signed-off-by: @pandoublefeng,@liucunwei
 2021-03-03 15:02:52 +08:00
mindspore-ci-bot 7f71525b0a !12810 dataset: Add IR vision error input tests, plus other UT updates 
From: @cathwong
Reviewed-by: 
Signed-off-by:
 2021-03-03 06:24:20 +08:00
Cathy Wong 253f389817 dataset: Add IR vision error input tests, plus other UT updates 2021-03-02 16:33:28 -05:00
YangLuo eecbc99b2e fix minddata include internal head file & fix execute ut 2021-03-02 14:12:33 +08:00
TinaMengtingZhang ccefa8b32e Make Parse() as protected 2021-03-01 15:07:36 -05:00
Cathy Wong 2f1f0c438d dataset: C++ API Vision support for Ops with TensorTransform op input 
plus UTs
 2021-02-26 16:25:30 -05:00
mindspore-ci-bot f1c7aeaa6b !12555 Create C++ API classes for Samplers 
From: @mhmotallebi
Reviewed-by: 
Signed-off-by:
 2021-02-27 02:54:19 +08:00
mindspore-ci-bot 5556b12da4 !12480 Adding Affine API 
From: @ezphlow
Reviewed-by: 
Signed-off-by:
 2021-02-26 21:35:05 +08:00
mohammad f73def5933 make C++ API classes for samplers 2021-02-25 17:29:43 -05:00
Eric d328d6eb54 Adding affine 
Need to move Affine to image_utilC

Refactored affine

Added affine API

Added API class impl

Removed default from vision ir

Compiling

removed extra file

Spell check
 2021-02-25 16:55:02 -05:00
TinaMengtingZhang 518828b639 Support 3 composed data ops and cleanup code 2021-02-24 11:05:44 -05:00
Zhenglong Li 581f1e82d4 API merge with new API class and new IR class 2021-02-24 10:55:18 +08:00
mindspore-ci-bot 182d928379 !12553 dataset: Split c_api_vision_test.cc 
From: @cathwong
Reviewed-by: @robingrosman,@nsyca
Signed-off-by: @robingrosman
 2021-02-24 02:01:30 +08:00
Cathy Wong 6112738fea dataset: Split c_api_vision_test.cc 2021-02-23 11:04:29 -05:00
shenwei41 d8d56b5374 GetAffineTransform 2021-02-22 15:22:34 +08:00
Cathy Wong 3fc204e3bc dataset: Create C++ API classes for vision transform ops 
Review rework

Add overloaded API Map function

Overload Execute, update st test_de.cc and enhance ut execute_test.cc;
More TensorTransform support for BoundingBoxAugment and UniformAug

Execute support for auto case; UT updates

Rebase updates, including fix CI Ascend compilations

test_de.cc - comment out dvpp tests

Rebase updates

Added back testcase and constructor

Added back constructor

added missing macro

Fixed Macro

removed decode
 2021-02-20 15:29:33 -05:00
mindspore-ci-bot 98f6d78778 !12361 Provide C++ API class for text and data transforms ops 
From: @tina_mengting_zhang
Reviewed-by: 
Signed-off-by:
 2021-02-20 04:38:26 +08:00
TinaMengtingZhang c5aa3eeef2 Add API class for data transform ops except Compose, RandomApply, RandomChoice 
Add API class for text transforms ops
 2021-02-18 15:58:44 -05:00
mohammad 9e6fcc7f23 return status from sampler IR functions 2021-02-17 12:01:34 -05:00
mindspore-ci-bot c8332c16f7 !11957 getRotationMatrix2D and getPerspectiveTransform 
From: @shenwei41
Reviewed-by: 
Signed-off-by:
 2021-02-11 00:08:56 +08:00
shenwei41 737e828f3d getRotationMatrix2D and getPerspectiveTransform 2021-02-10 12:35:56 +08:00
Zhenglong Li e2d3495925 Dvpp refactor and 6 Dvpp operators 2021-02-10 10:34:38 +08:00
Eric 9953757ff4 Fixed tenor compile error for MD 2021-02-09 14:56:31 +08:00
Cathy Wong 3e6e35d749 dataset: delete minddata/dataset/include/status.h 2021-02-04 10:19:08 -05:00
hesham 715d70f397 Remove setting num_parallel_worker in UT, since it would fail for machines with less than 16 cores. 2021-02-02 11:38:27 -05:00
lixian 7d2fd6e76c cxx api refactor: tensor/status/model 2021-02-02 09:15:56 +08:00
Nat Sutyanyong 5a7dc0accc Migrate repeat_pass.cc to IR optimizer and remove ExecTree optimizer 2021-02-01 12:30:16 -05:00
mindspore-ci-bot f980561fe1 !11598 [MD] add common validation functions for transform 
From: @luoyang42
Reviewed-by: 
Signed-off-by:
 2021-01-30 14:32:44 +08:00
YangLuo 1893ae8662 [MD] add common validation functions for transform 2021-01-29 16:29:47 +08:00
mindspore-ci-bot 20c1727b2c !11631 Add SubsetSampler 
From: @hfarahat
Reviewed-by: 
Signed-off-by:
 2021-01-29 03:48:45 +08:00
mindspore-ci-bot 8e4a6842ff !11681 gnn support edge weight sampling 
From: @heleiwang
Reviewed-by: @liucunwei
Signed-off-by: @liucunwei
 2021-01-28 19:15:37 +08:00
heleiwang 2dc9ba761c gnn support weight sampling neighbors. 2021-01-28 15:28:44 +08:00
hesham 7120c66998 Add SubsetSampler 2021-01-27 15:00:18 -05:00
mindspore-ci-bot 4cd6588af0 !11393 Migrating cache transform pass from execution tree to IR tree 
From: @lixiachen
Reviewed-by: 
Signed-off-by:
 2021-01-27 22:10:13 +08:00
xulei2020 de630f1cb1 Add parameter verification for getROI function 
fix c++ ut compilation failed of Asan bug
 2021-01-27 09:06:59 +08:00
Lixia Chen 922e1f4f36 Migrate CacheTransformPass 2021-01-26 19:24:10 -05:00
Cathy Wong a3b10213b7 dataset: Resolve protected parent_access from DeepCopyPass 2021-01-22 15:56:32 -05:00
xiefangqi 3f76abcdee minddata err msg add file path 2021-01-22 16:38:46 +08:00
mindspore-ci-bot 2beb916bf5 !11427 TextOp decoupling 
From: @alexyuyue
Reviewed-by: 
Signed-off-by:
 2021-01-22 09:09:41 +08:00
mindspore-ci-bot fcc4a4eaea !11397 add GetDatasetSize to concat_node 
From: @mhmotallebi
Reviewed-by: 
Signed-off-by:
 2021-01-22 08:08:41 +08:00
mohammad b5e91139e2 add GetDatasetSize to concat node 2021-01-21 16:25:48 -05:00
xulei2020 de6356ebfa add warp affine 2021-01-21 14:20:14 +08:00
alex-yuyue a8675f8227 TextOp decoupling 
Signed-off-by: alex-yuyue <yue.yu1@huawei.com>
 2021-01-20 21:52:45 -05:00
mindspore-ci-bot c155dfee7a !11329 [MD] Rename SamplerObj::Build() to SamplerBuild() 
From: @lixiachen
Reviewed-by: 
Signed-off-by:
 2021-01-19 06:33:49 +08:00
Lixia Chen 273865a72b Rename SamplerObj::Build() to SamplerBuild() 2021-01-18 15:42:34 -05:00
Lixia Chen 3c5917a03d Fix IR node Drop() 2021-01-18 13:06:07 -05:00
mindspore-ci-bot cb58b5baa5 !11211 Provide functions for modifying IR tree (Insert/Drop/Append nodes) 
From: @lixiachen
Reviewed-by: 
Signed-off-by:
 2021-01-16 01:20:01 +08:00
Nat Sutyanyong e28fb6ce4d Tested with new test cases and all dataset UTs passed 2021-01-15 09:59:21 -05:00
mindspore-ci-bot dfa6daaa57 !11249 【MD】fix bug for extractchannel when type is UINT16 
From: @xulei2020
Reviewed-by: @heleiwang,@liucunwei
Signed-off-by: @liucunwei
 2021-01-15 14:33:16 +08:00
mindspore-ci-bot 581789c252 !11246 add num_epochs to CPP API Dataset::CreateIterator() 
From: @ziruiwu
Reviewed-by: 
Signed-off-by:
 2021-01-15 06:02:00 +08:00
xulei2020 5dd651f1a5 fix bug for extractchannel when type is UINT16 2021-01-14 09:26:18 +08:00
Cathy Wong 4b1ccf9dd5 dataset: Use int32_t for text's vocab_size 2021-01-13 18:12:50 -05:00
Zirui Wu fa553956ef add num_epochs to CPP API Dataset::CreateIterator(columns,num_epochs) 2021-01-13 12:13:40 -05:00
mohammad f48ab2b5c9 make MD public headers standalone 2021-01-11 06:35:53 -05:00
Zirui Wu 0e68575e77 port over getter pass 
port over tensor_op_fusion pass

add fusion support for prebuildOpeartion (TensorOpeartion)
 2021-01-07 10:52:49 -05:00
mindspore-ci-bot 5fd77356c9 !9687 Adding AddChild functionality to SamplerObj 
From: @mahdirahmanihanzaki
Reviewed-by: 
Signed-off-by:
 2021-01-06 01:46:08 +08:00
mindspore-ci-bot a5869f2984 !10142 dataset CPP UT: Updates for ExecTree to IRTree Support 
From: @cathwong
Reviewed-by: 
Signed-off-by:
 2021-01-05 06:53:14 +08:00
Nat Sutyanyong 7c5b2de1ec dataset CPP UT: Updates for ExecTree to IRTree Support 2021-01-04 16:00:30 -05:00
Zirui Wu 0a31f563a7 minor fixes 2020-12-23 10:06:01 -05:00
mindspore-ci-bot 20df37e428 !10287 Add ut for dataset to_number 
From: @luoyang42
Reviewed-by: @heleiwang
Signed-off-by:
 2020-12-23 21:14:42 +08:00
Zirui Wu 0e2f7a9e9e add validate param to base class DatasetNode 2020-12-21 11:14:37 -05:00
YangLuo da6de519ae Add ut for dataset to_number 2020-12-21 20:09:03 +08:00
Xiao Tianci 633f97bb48 fix get dataset size in distributed scenario 2020-12-15 20:41:45 +08:00
xiefangqi 3c6cc3c58c Add NormalizePadOp 2020-12-15 11:45:41 +08:00
Mahdi 0f2b5d8cac Changed SamplerObj validate params to return status and added AddChild to it 2020-12-10 15:39:59 -05:00
Zirui Wu 91a6b2b0ca Add checks to RandomData and LookUp 2020-12-09 17:09:34 -05:00
mindspore-ci-bot 317a97e6b9 !9336 auto num_parallel_workers setup 
From: @ziruiwu
Reviewed-by: 
Signed-off-by:
 2020-12-09 08:32:55 +08:00
mindspore-ci-bot a2c80435ce !9685 Fix a core dump in TreeConsumer::Terminate() plus minor cache fixes 
From: @lixiachen
Reviewed-by: @mikef,@nsyca
Signed-off-by: @nsyca
 2020-12-09 08:21:22 +08:00
Zirui Wu d6df1b0832 Implemented AutoNumWorker Pass which sets num_workers of selected parallel ops automatically if enabled 2020-12-08 17:49:05 -05:00
Lixia Chen 32b82c2737 Fix a core dump in TreeConsumer::Terminate() 
and other minor cache changes
 2020-12-08 15:38:40 -05:00
Xiao Tianci 0659473535 add TruncateSequencePair, ToNumber C++ API and enable three test cases 2020-12-08 11:12:16 +08:00
mindspore-ci-bot 49fd5308a4 !9564 Tidy up code in dataset compilation phase 
From: @nsyca
Reviewed-by: 
Signed-off-by:
 2020-12-08 05:11:29 +08:00
mindspore-ci-bot 6b5626634c !9530 add BasicTokenizer and BertTokenizer C++ API 
From: @tiancixiao
Reviewed-by: @mikef,@nsyca
Signed-off-by: @nsyca
 2020-12-08 04:09:40 +08:00
Nat Sutyanyong 41ea1621c5 Tidy up TreeAdapter::Compile 2020-12-07 14:57:22 -05:00
shenwei41 206ae1a7c3 Add Three text C++ API 2020-12-05 10:32:15 +08:00
Nat Sutyanyong 4cb78f2e03 Remove Repeat(1),Take(-1), and Skip(0) in NodeRemovalPass 2020-12-04 17:37:07 -05:00
Jesse Lee 0c8dfc68ec Fix hang on RequestFreeTag 2020-12-04 14:54:30 -05:00
Xiao Tianci 290cc72887 add BasicTokenizer and BertTokenizer C++ API 2020-12-04 21:51:49 +08:00
Nat Sutyanyong d69a29a44e Migrate 3 pre passes to IR optimizer, namely, cache_error_pass, epoch_injection, and removal_pass 2020-12-03 14:40:03 -05:00
mindspore-ci-bot 73c91e05b1 !9398 Change Build to return status 
From: @alex-yuyue
Reviewed-by: @mikef
Signed-off-by:
 2020-12-03 23:43:08 +08:00
alex-yuyue ccff3a5316 Change build to return status 
Signed-off-by: alex-yuyue <yue.yu1@huawei.com>
 2020-12-02 22:46:49 -05:00
YangLuo a2ec7ad5bd Fix concat issue & fix schema doc 2020-12-02 21:06:22 +08:00
mindspore-ci-bot df44e1339e !9249 Optimizing GetDatasetSize 
From: @mahdirahmanihanzaki
Reviewed-by: 
Signed-off-by:
 2020-12-02 08:26:34 +08:00
mindspore-ci-bot 9ed31ea2cb !9339 Disable getter optimization pass 
From: @ezphlow
Reviewed-by: @robingrosman,@nsyca
Signed-off-by: @nsyca
 2020-12-02 08:25:41 +08:00
Eric db2a8b5e1d Disable getter pass 2020-12-01 17:40:34 -05:00
Mahdi f80553bc6e Optimization for GetDatasetSize 2020-12-01 17:40:25 -05:00
mindspore-ci-bot da8d7d661a !9283 Modify the dataset ut log message 
From: @liucunwei
Reviewed-by: @pandoublefeng,@heleiwang,@jonyguo
Signed-off-by: @heleiwang,@jonyguo
 2020-12-01 16:28:32 +08:00
liucunwei 8ef4b2796e modify the dataset ut log message 2020-12-01 14:42:44 +08:00
shenwei41 3bea84d0f7 add four new text API 2020-12-01 10:21:47 +08:00
mindspore-ci-bot 7748ae823a !9141 Fix DatasetCache C++ API cache sharing behavior 
From: @lixiachen
Reviewed-by: @mikef,@nsyca
Signed-off-by: @nsyca
 2020-12-01 02:28:02 +08:00
mindspore-ci-bot a37e9710b5 !9038 port name functions from TensorOp to TensorOperation 
From: @mhmotallebi
Reviewed-by: 
Signed-off-by:
 2020-12-01 02:13:48 +08:00
mindspore-ci-bot a6845395e6 !9181 Add some ut pipeline test case for C++ dataset 
From: @liucunwei
Reviewed-by: @pandoublefeng,@jonyguo
Signed-off-by: @jonyguo
 2020-11-28 21:34:03 +08:00
liucunwei e612e23a05 Add some ut pipeline test case for C++ dataset 2020-11-28 18:14:58 +08:00
liucunwei 7f13f2ad0e add some pipeline ut test cases for c++ dataset 2020-11-28 16:59:38 +08:00
mohammad 9cfeb08c4d port name() to TensorOperation and its derived classes 2020-11-27 19:00:56 -05:00
Lixia Chen e8247eadef Same runtime CacheClient if c++ cache is shared 2020-11-27 18:52:15 -05:00
Zirui Wu 36bfd0d8ad address left over cmts from 8802 2020-11-27 16:54:28 -05:00
liucunwei 3e57236fe3 add some new ut test for minddataset 2020-11-27 20:39:13 +08:00
shenwei41 4e56618d18 Add text C++ API 2020-11-27 11:41:35 +08:00
Nat Sutyanyong 5e1bb0b697 boilerplate code for future IR optimizer 
add 2 test cases to IRNode Deepcopy()

address review cmts

fix ut

samplerObj copy

ci

fix ci

fix ci round III

address further review cmts

add a missing macro

fix merge conflict

fix complie err

fix lite compile err

fix compile err

fix lite compile round III

address an issue

fix minor comments
 2020-11-26 09:42:01 -05:00
Eric Zhang 809e1d5086 added python api based on cpp api 
1st draft of python iterator

Added Cifar10 and Cifar100 pybind port

Change pybind to use IR for Skip and Manifest

Signed-off-by: alex-yuyue <yue.yu1@huawei.com>

DatasetNode as a base for all IR nodes

namespace change

Fix the namespace issue and make ut tests work

Signed-off-by: alex-yuyue <yue.yu1@huawei.com>

Add VOCDataset

!63 Added RandomDataset
* Added RandomDataset

add imagefolder ir

Pybind switch: CelebA and UT

!61 CLUE example with class definition
* Merge branch 'python-api' of gitee.com:ezphlow/mindspore into clue_class_pybind
* Passing testcases
* Added CLUE, not working

add ManifestDataset IR

Signed-off-by: alex-yuyue <yue.yu1@huawei.com>

Update Coco & VOC & TFReader, Update clang-format, Reorder
datasets_binding

!69 Add Generator and move c_dataset.Iterator to dataset.Iterator
* Add GeneratorDataset to c_dataset
* Add GeneratorDataset to c_dataset

!67 Moving c_datasets and adding sampler wrapper
* Need to add create() method in datasets.py
* migration from c_dataset to dataset part 1

!71 Fix indent error
* Fix indentation error

!72 Fix c_api tests cases
* Fix c_api tests cases

!73 Added CSV Dataset
* Added CSVDataset

pybind switch: Take and CelebA fixes

!75 move c_dataset functionality to datasets
* Fixed existing testcases
* Added working clue and imagefolder
* Added sampler conversion from pybind
* Added sampler creation

!77 Add Python API tree
* Python API tree

add minddataset

TextFileDataset pybind

Rename to skip test_concat.py and test_minddataset_exception.py

!80 Add batch IR to python-api branch, most test cases work
* staging III
* staging, add pybind

Enable more c_api take and CelebA tests; delete util_c_api

!84 Schema changes in datasets.py
* Schema changes

!85 Remove input_indexes from sub-classes
* remove input_index from each subclass

!83 Remove C datasets
* Removed c_dataset package
* Remove c_datasets

!82  pybind switch: shuffle
* pybind switch: shuffle

!86 Add build_vocab
* Add build_vocab

Rebase with upstream/master
_shuffle conflict
BatchNode error

!88 Fix rebase problem
* fix rebase problem

Enable more unit tests; code typo/nit fixes

!91 Fix python vocag hang
* Fix python vocab hang

!89 Added BucketBatchByLength Pybind switch
* Added BucketBatchByLength

Update and enable more tet_c_api_*.py tests

!95 Add BuildSentencePeiceVocab
* - Add BuildSentencePeiceVocab

!96 Fix more tests
* - Fix some tests

- Enable more test_c_api_*
- Add syncwait

!99 pybind switch for device op
* pybind switch for device op

!93 Add getters to python API
* Add getters to python API

!101 Validate tree, error if graph
* - Add sync wait

!103 TFrecord/Random Datasets schema problem
* - TfRecord/Random schem aproblem

!102 Added filter pybind switch
* Added Filter pybind switch

!104 Fix num_samples
* - TfRecord/Random schem aproblem

!105 Fix to_device hang
* Fix to_device hang

!94 Adds Cache support for CLUE dataset
* Added cache for all dataset ops
* format change
* Added CLUE cache support
* Added Cache conversion

Add save pybind

fix compile err

init modify concat_node

!107 Fix some tests cases
* Fix tests cases

Enable and fix more tests

!109 pybind switch for get dataset size
* pybind_get_dataset_size

some check-code fixes for pylint, cpplint and clang-format

!113 Add callback
* revert
* dataset_sz 1 line
* fix typo
* get callback to work

!114 Make Android compile clean
* Make Android Compile Clean

Fix build issues due to rebase

!115 Fix more tests
* Fix tests cases
* !93 Add getters to python API

fix test_profiling.py

!116 fix get dataset size
* fix get dataset size

!117 GetColumnNames pybind switch
* Added GetColumnNames pybind switch

code-check fixes: clangformat, cppcheck, cpplint, pylint

Delete duplicate test_c_api_*.py files; more lint fixes

!121 Fix cpp tests
* Remove extra call to getNext in cpp tests

!122 Fix Schema with Generator
* Fix Schema with Generator

fix some cases of csv & mindrecord

!124 fix tfrecord get_dataset_size and add some UTs
* fix tfrecord get dataset size and add some ut for get_dataset_size

!125 getter separation
* Getter separation

!126 Fix sampler.GetNumSamples
* Fix sampler.GetNumSampler

!127 Assign runtime getter to each get function
* Assign runtime getter to each get function

Fix compile issues

!128 Match master code
* Match master code

!129 Cleanup DeviceOp/save code
* Cleanup ToDevice/Save code

!130 Add cache fix
* Added cache fix for map and image folder

!132 Fix testing team issues
* Pass queue_name from python to C++
* Add Schema.from_json

!131 Fix Cache op issues and delete de_pipeline
* Roll back C++ change
* Removed de_pipeline and passing all cache tests.
* fixed cache tests

!134 Cleanup datasets.py part1
* Cleanup dataset.py part1

!133 Updated validation for SentencePieceVocab.from_dataset
* Added type_check for column names in SentencePieceVocab.from_dataset

Rebase on master 181120 10:20

fix profiling

temporary solution of catching stauts from Node.Build()

!141 ToDevice Termination
* ToDevice termination

pylint fixes

!137 Fix test team issues and add some corresponding tests
* Fix test team issues and add some corresponding tests

!138 TreeGetter changes to use OptPass
* Getter changes to use OptPass (Zirui)

Rebase fix

!143 Fix cpplint issue
* Fix cpplint issue

pylint fixes in updated testcases

!145 Reset exceptions testcase
* reset exception test to master

!146 Fix Check_Pylint Error
* Fix Check_Pylint Error

!147 fix android
* fix android

!148 ToDevice changes
* Add ToDevice to the iterator List for cleanup at exit

!149 Pylint issue
* Add ToDevice to the iterator List for cleanup at exit

!150 Pylint 2
* Add ToDevice to the iterator List for cleanup at exit

!152 ExecutionTree error
* ET destructor error

!153 in getter_pass, only remove callback, without deleting map op
* getter pass no longer removes map

!156 early __del__ of iterator/to_device
* early __del__ of iterator

!155 Address review comments Eric 1
* Added one liner fix to validators.py
* roll back signature fix
* lint fix
* Eric Address comments 2
* C++ lint fix
* Address comments Eric 1

!158 Review rework for dataset bindings - part 1
* Reorder nodes repeat and rename
* Review rework for dataset bindings - part 1

!154 Fixing minor problems in the comments (datasets.py, python_tree_consumer.cc, iterators_bindings.cc, and iterators.py)
* Fixing minor problems in the comments (datasets.py, python_tree_consum…

!157 add replace none
* Add replace_none to datasets.py, address comments in tests

Trying to resolve copy

Override the deepcopy method of deviceop

Create_ir_tree method

Create_ir_tree method 2

Create_ir_tree method 2

del to_device if already exists

del to_device if already exists

cache getters shapes and types

Added yolov3 relaxation, to be rolled back

Get shapes and types together

bypass yolo

NumWorkers for MapOp

revert Yolo

revert Thor

Print more info

Debug code: Update LOG INFO to LOG ERROR

do not remove epochctrl for getter pass

Remove repeat(1)

pritn batch size

add log to tree_consumer and device_queue op

Revert PR 8744

Signed-off-by: alex-yuyue <yue.yu1@huawei.com>

__del__ toDEvice

__del__ toDevice2

!165 add ifndef ENABLE_ANDROID to device queue print
* Add ifndef ENABLE_ANDROID to device queue print

revert some changes

!166 getter: get_data_info
* getter: get_data_info

!168 add back tree print
* revert info to warnning in one log
* add back the missed print tree log

Release GIL in GetDataInfo
 2020-11-25 15:46:59 -05:00
shenwei41 4132228c10 Add c++ api of randomresizecropwithbbox 2020-11-21 17:06:56 +08:00
mindspore-ci-bot 812095de8f !8753 Adds execute testcase and fix get_dataset_size 
From: @ezphlow
Reviewed-by: 
Signed-off-by:
 2020-11-20 02:06:55 +08:00
Eric 45292abdfd Added execute test case 
Added execute tests for server
 2020-11-18 23:44:46 -05:00
mindspore-ci-bot c8047a35ab !8632 [MD] Add Multiply op for Lite Dataset 
From: @jiangzhiwen8
Reviewed-by: @liucunwei,@heleiwang
Signed-off-by: @liucunwei
 2020-11-19 09:58:43 +08:00
mindspore-ci-bot 215fce192e !8737 [MD] Fix the Pad function return false if bottom value is 0 
From: @jiangzhiwen8
Reviewed-by: @xulei2020,@liucunwei,@heleiwang
Signed-off-by: @liucunwei
 2020-11-19 09:07:25 +08:00
mindspore-ci-bot f352f9a54a !8511 [MD] Decouple more Transform Ops 
From: @luoyang42
Reviewed-by: 
Signed-off-by:
 2020-11-18 23:19:36 +08:00
jiangzhiwen bc91f7b3ac fix pad zero 2020-11-18 16:14:37 +08:00
luoyang d43155b5b0 [MD] Decouple more Transform Ops 2020-11-18 14:50:59 +08:00
mindspore-ci-bot e3b1814401 !8713 Fix for GetDatasetSize issue in TextFile 
From: @mahdirahmanihanzaki
Reviewed-by: @nsyca,@robingrosman
Signed-off-by: @nsyca
 2020-11-18 05:39:51 +08:00
Mahdi 449e1526dc Fixed GetDatasetSize for TextFile 2020-11-17 15:22:58 -05:00
Zirui Wu ff5999fc2f add removal pass for getters 
fix CI round I

fix ci round II

address review cmts

fix ci round II
 2020-11-17 10:49:23 -05:00
Mahdi ecde0f03a2 Updated GetNumSamples logic 2020-11-16 16:03:14 -05:00
jiangzhiwen b30c1045e1 add lite_mat multiply 2020-11-16 15:05:31 +08:00
mindspore-ci-bot edcb0cd86b !8370 [MD] Use neon instruction set to accelerate the Substract and Divide ops 
From: @jiangzhiwen8
Reviewed-by: @xulei2020
Signed-off-by:
 2020-11-13 16:14:00 +08:00
mindspore-ci-bot c54a4a4494 !8540 Added GetColumnNames to C++ 
From: @mahdirahmanihanzaki
Reviewed-by: @nsyca,@jonyguo
Signed-off-by: @nsyca
 2020-11-13 09:33:37 +08:00
Mahdi 5a7515e48f Added GetColumnNames to C++ 2020-11-12 16:36:59 -05:00
Zirui Wu 3555d19b07 fix get num_class in imagefolder 
address cmt

review cmts
 2020-11-12 14:26:54 -05:00
jiangzhiwen 03bbe5f2a8 neon acceleration for subtract 2020-11-12 16:30:40 +08:00
Eric a21eb2d527 Added random node fix 2020-11-11 17:24:48 -05:00
mindspore-ci-bot 70f5775711 !8387 Add support for GetClassIndexing in C++ API 
From: @alex-yuyue
Reviewed-by: 
Signed-off-by:
 2020-11-11 03:02:22 +08:00
alex-yuyue d7c3a9ba00 Add GetClassIndexing 
Signed-off-by: alex-yuyue <yue.yu1@huawei.com>
 2020-11-10 11:02:02 -05:00
xulei2020 f8d1b20873 init litemat from memory 2020-11-10 10:12:39 +08:00
mindspore-ci-bot 908a494add !8299 [MD] Skip returning IR when meet nullptr in ConcatDataset 
From: @luoyang42
Reviewed-by: @nsyca,@pandoublefeng,@jonyguo
Signed-off-by: @jonyguo
 2020-11-09 11:49:48 +08:00
mindspore-ci-bot 027916cb5f !8315 Cache server stability 
From: @jkl_lee
Reviewed-by: @mikef,@nsyca
Signed-off-by: @nsyca
 2020-11-08 20:03:23 +08:00
Lixia Chen f24a788eed Rebase up to 5792d50ca7 2020-11-06 18:39:05 -05:00
Zirui Wu 7538f82da9 add callback to map's c api 
fix test err

fix ci round 1

fix ci round 2

add col_order to batch_node

minor fix
 2020-11-06 16:36:47 -05:00
mindspore-ci-bot a9a9c45662 !8232 C++ api add Filter 
From: @tiancixiao
Reviewed-by: 
Signed-off-by:
 2020-11-07 00:32:15 +08:00
mindspore-ci-bot 97e1f6f438 !8248 Add macro and fix album testcase 
Merge pull request !8248 from EricZ/helper_dep_removal
 2020-11-06 20:33:20 +08:00
Xiao Tianci 4a5e287b92 add filter c api and ut 2020-11-06 15:49:23 +08:00
luoyang dcb758bc77 [MD] Skip returning IR when meet nullptr in ConcatDataset 2020-11-06 15:08:11 +08:00
Eric 394dbf50e1 Added testing macro and fix album_op test 2020-11-05 13:22:51 -05:00
mindspore-ci-bot 523c3f07e3 !8194 Optimizer on IR #2 - Call ValidateParams in PrePass 
Merge pull request !8194 from nsyca/opt3
 2020-11-05 22:52:37 +08:00
mindspore-ci-bot 099765bbb7 !8135 [MD] Transform api decouple 
Merge pull request !8135 from luoyang/c-api-pyfunc
 2020-11-05 22:41:44 +08:00
Yi Huaijie d7faa77b5e support int64 shape 2020-11-05 21:15:49 +08:00
luoyang 9a57cbacd9 [MD] Transform api decouple 2020-11-05 19:56:15 +08:00
mindspore-ci-bot 8709005a57 !8200 Add two transforms API 
Merge pull request !8200 from shenwei41/new_master
 2020-11-05 11:19:00 +08:00
Nat Sutyanyong a2a2223373 Add optimizer to IR tree #2 2020-11-04 21:44:17 -05:00
Eric 85f0a9bc8f Added dependency removal for data_helper 2020-11-04 12:15:32 -05:00
shenwei41 9ffdf0b6d4 Add transform API 2020-11-04 11:54:33 +08:00
Zirui Wu 5d1ada682c remove uncessary headers 
staging II

clean up

further clean up

fix merge conflict with 8139

fix complie err
 2020-11-03 11:59:55 -05:00
hesham 5169fb4c42 Remove api namespace 2020-11-02 20:09:37 -05:00
hesham 0ba0595caf DatasetNode as a base for all IR nodes 2020-11-01 16:57:10 -05:00
Mahdi 5005b434e6 Fixed repeat issue with GetDatasetSize 2020-10-30 19:36:56 -04:00
mindspore-ci-bot cacebd1211 !7933 Cache performance updates 
Merge pull request !7933 from Jamie/CacheOp_dev
 2020-10-31 04:25:47 +08:00
mindspore-ci-bot 7ef7bef1d8 !7796 [MD] Enable C++ cache api for non-mappable datasets 
Merge pull request !7796 from lixiachen/cache_c_api_nonmappable
 2020-10-30 22:11:13 +08:00
Lixia Chen 572c5e5f29 Rebase up to 542a52fbf8 2020-10-29 20:34:09 -04:00
Cathy Wong 18ecd65f93 C++ API: Delete ValidateParams in dataset APIs; update error input UTs 2020-10-29 16:29:16 -04:00
mindspore-ci-bot 8f44074796 !7866 [MD] C++ API support build_sentence_piece_vocab_node & sentence_piece_tokenizer 
Merge pull request !7866 from luoyang/c-api-pyfunc
 2020-10-29 14:26:13 +08:00
luoyang 52d278e858 [MD] C++ API support build_sentence_piece_vocab_node & sentence_piece_tokenizer 2020-10-29 11:05:28 +08:00
mindspore-ci-bot 2ba20dc8e7 !7848 Add five transforms API 
Merge pull request !7848 from shenwei41/new_master
 2020-10-29 10:18:57 +08:00
Zirui Wu 814044963b add validation check in tfrecord_node 
update ut
 2020-10-28 19:26:51 -04:00
Zirui Wu ea97197311 fix tfrecord_op core dump 
add test case

fix ci

fix ci round 2

address review cmts
 2020-10-28 17:38:43 -04:00
mindspore-ci-bot c39c3ccfbe !7734 Redesigned GetNumClasses 
Merge pull request !7734 from Alex Yuyue/IR_dataset_input
 2020-10-29 01:44:36 +08:00
shenwei41 779f0f387b Add transforms API 2020-10-28 22:06:58 +08:00
Xiao Tianci 823e52f1dc add params checking of offset in distributed_sampler 
add related comment to explain
add related UT
 2020-10-28 16:46:12 +08:00
Lixia Chen 02ddddfe7e Enable caching over non-mappable dataset c++ api 
and caching over Album dataset
 2020-10-27 23:02:13 -04:00
alex-yuyue 913a49f3af Add support for C++ API: GetNumClasses 
Signed-off-by: alex-yuyue <yue.yu1@huawei.com>
 2020-10-27 16:26:51 -04:00
mindspore-ci-bot 3376e0d034 !7704 [MD] Add testcases for cache c api 
Merge pull request !7704 from lixiachen/cache_c_api
 2020-10-26 14:31:09 +08:00
mindspore-ci-bot d81bd7b17b !7710 Add c++ API for GetRepeatCount and GetBatchSize 
Merge pull request !7710 from h.farahat/getBatchRepeat
 2020-10-26 09:25:21 +08:00
mindspore-ci-bot 0124eef185 !7729 [MD] Fix log of vision and dataset in C++ api 
Merge pull request !7729 from luoyang/pylint
 2020-10-26 09:23:22 +08:00
mindspore-ci-bot 7982f05038 !7730 [MD] Fix bugs in WeightedRandomSampler & SubsetRandomSampler 
Merge pull request !7730 from luoyang/c-api-pyfunc
 2020-10-26 09:22:14 +08:00
luoyang f0e976dbda [MD] Fix bugs in WeightedRandomSampler & SubsetRandomSampler 2020-10-25 17:58:51 +08:00
hesham f3880f8355 Add GetBatchSizer and GetRepeatCount to c++ API 2020-10-24 22:45:19 -04:00
YangLuo f3fa5a441d [MD] Fix log of vision and dataset in C++ api 2020-10-24 19:24:30 +08:00
mindspore-ci-bot 3d194137da !7702 Redesigned GetOutputType and GetOutputShape 
Merge pull request !7702 from Alex Yuyue/IR_dataset_input
 2020-10-24 14:42:29 +08:00
mindspore-ci-bot 48edd9708c !7690 [MD] fix num_sample in randomsampler and sequential sampler 
Merge pull request !7690 from liyong126/fix_md_num_samples
 2020-10-24 11:13:19 +08:00
alex-yuyue 2906659673 Add support for GetOutputTypes and GetOutputShapes 
Signed-off-by: alex-yuyue <yue.yu1@huawei.com>
 2020-10-23 22:21:52 -04:00
mindspore-ci-bot fe2852df82 !7703 C++ API: Add a parameter validation and fix incorrect format of error message for TFrecord 
Merge pull request !7703 from TinaMengtingZhang/fix_tfrecord_issue_I1YKU6_I1YKXL
 2020-10-24 07:33:26 +08:00
Lixia Chen bc396febae Add testcases for cache c api 2020-10-23 17:10:18 -04:00
tinazhang e7346dd3a9 add a validation code to fix issue I1YKU6 and corresponded ut, fix format issue I1YKXL 2020-10-23 17:07:51 -04:00
Mahdi 0e03f5b0dd Added GetDatasetSize 2020-10-23 16:29:11 -04:00
Zirui Wu a85a256749 fix complie err in c_api_dataset_test_save 2020-10-23 10:09:35 -04:00
mindspore-ci-bot 3355bb5f2f !7657 Leaf node IR breakdown, port to engine/ir/datasetops/source/ 
Merge pull request !7657 from ZiruiWu/ir_breakdown_leaf_nodes
 2020-10-23 20:32:31 +08:00
liyong ee042b90f7 fix num_sample in sequentialSampler and randomSampler 2020-10-23 19:26:10 +08:00
mindspore-ci-bot 9d35fdc4e5 !7653 [MD] C++ api SaveOp 
Merge pull request !7653 from luoyang/c-api-pyfunc
 2020-10-23 14:14:11 +08:00
Zirui Wu cfc50e231a Added IR breakdown for remaining ops after PR7596 
fix android build failure

fix include in datasets.cc

attempting fix android build

fix complie err x86
 2020-10-22 23:07:41 -04:00
luoyang 8e60cfe800 [MD] C++ api save 2020-10-23 10:27:35 +08:00
mindspore-ci-bot 19b10d474f !7530 [MD] Add divide image proccessing 
Merge pull request !7530 from jiangzhiwen/lite/image_divide
 2020-10-23 09:22:36 +08:00
Zirui Wu d471552fc5 shuffle take repeat bucketbatch buildvocab repeat shuffle take project concat rename node IR added 
concat, bucketbatch project rename

fix ci round 1

fix ci round 2

fix up

fix ci
 2020-10-21 18:41:34 -04:00
mindspore-ci-bot de7ea8e68c !7540 Extract ImageFolderNode and BatchNode into its own files from datasets.cc/.h 
Merge pull request !7540 from ZiruiWu/refactor_ir_demo
 2020-10-21 19:23:20 +08:00
mindspore-ci-bot a8dc0f5c1b !7557 [MD] Add merge image proccessing 
Merge pull request !7557 from jiangzhiwen/lite/merge
 2020-10-21 17:28:55 +08:00