Commit Graph

2688 Commits

Author SHA1 Message Date
mindspore-ci-bot 3bed28822d !9024 Add four text C++ API
From: @shenwei41
Reviewed-by: 
Signed-off-by:
2020-11-27 18:39:25 +08:00
mindspore-ci-bot aa2296bcc1 !9094 Blocking TakeOp/SkipOp under Cache
From: @lixiachen
Reviewed-by: @nsyca,@mikef,@nsyca,@mikef
Signed-off-by: @nsyca
2020-11-27 18:01:49 +08:00
shenwei41 4e56618d18 Add text C++ API 2020-11-27 11:41:35 +08:00
mindspore-ci-bot 3fa7233f68 !9072 fix topk fussion help tensor size = 4096*2
From: @jjfeing
Reviewed-by: @chujinjin,@kisnwang
Signed-off-by:
2020-11-27 10:03:48 +08:00
tinazhang 27698866d1 Block take/skip under cache and fix docstring
and other minor changes per the request of HQ testers
2020-11-26 17:44:19 -05:00
Nat Sutyanyong 5e1bb0b697 boilerplate code for future IR optimizer
add 2 test cases to IRNode Deepcopy()

address review cmts

fix ut

samplerObj copy

ci

fix ci

fix ci round III

address further review cmts

add a missing macro

fix merge conflict

fix complie err

fix lite compile err

fix compile err

fix lite compile round III

address an issue

fix minor comments
2020-11-26 09:42:01 -05:00
jjfeing 8fb7d11ecb fix topk help 4096 2020-11-26 19:16:37 +08:00
mindspore-ci-bot 421cad24ef !8715 Add Gamma distribution
From: @peixu_ren
Reviewed-by: 
Signed-off-by:
2020-11-26 09:57:31 +08:00
mindspore-ci-bot adc8e3e707 !8594 [MD] Pybind Pushdown Support for dataset
From: @cathwong
Reviewed-by: 
Signed-off-by:
2020-11-26 05:55:49 +08:00
Eric Zhang 809e1d5086 added python api based on cpp api
1st draft of python iterator

Added Cifar10 and Cifar100 pybind port

Change pybind to use IR for Skip and Manifest

Signed-off-by: alex-yuyue <yue.yu1@huawei.com>

DatasetNode as a base for all IR nodes

namespace change

Fix the namespace issue and make ut tests work

Signed-off-by: alex-yuyue <yue.yu1@huawei.com>

Add VOCDataset

!63 Added RandomDataset
* Added RandomDataset

add imagefolder ir

Pybind switch: CelebA and UT

!61 CLUE example with class definition
* Merge branch 'python-api' of gitee.com:ezphlow/mindspore into clue_class_pybind
* Passing testcases
* Added CLUE, not working

add ManifestDataset IR

Signed-off-by: alex-yuyue <yue.yu1@huawei.com>

Update Coco & VOC & TFReader, Update clang-format, Reorder
datasets_binding

!69 Add Generator and move c_dataset.Iterator to dataset.Iterator
* Add GeneratorDataset to c_dataset
* Add GeneratorDataset to c_dataset

!67 Moving c_datasets and adding sampler wrapper
* Need to add create() method in datasets.py
* migration from c_dataset to dataset part 1

!71 Fix indent error
* Fix indentation error

!72 Fix c_api tests cases
* Fix c_api tests cases

!73 Added CSV Dataset
* Added CSVDataset

pybind switch: Take and CelebA fixes

!75 move c_dataset functionality to datasets
* Fixed existing testcases
* Added working clue and imagefolder
* Added sampler conversion from pybind
* Added sampler creation

!77 Add Python API tree
* Python API tree

add minddataset

TextFileDataset pybind

Rename to skip test_concat.py and test_minddataset_exception.py

!80 Add batch IR to python-api branch, most test cases work
* staging III
* staging, add pybind

Enable more c_api take and CelebA tests; delete util_c_api

!84 Schema changes in datasets.py
* Schema changes

!85 Remove input_indexes from sub-classes
* remove input_index from each subclass

!83 Remove C datasets
* Removed c_dataset package
* Remove c_datasets

!82  pybind switch: shuffle
* pybind switch: shuffle

!86 Add build_vocab
* Add build_vocab

Rebase with upstream/master
_shuffle conflict
BatchNode error

!88 Fix rebase problem
* fix rebase problem

Enable more unit tests; code typo/nit fixes

!91 Fix python vocag hang
* Fix python vocab hang

!89 Added BucketBatchByLength Pybind switch
* Added BucketBatchByLength

Update and enable more tet_c_api_*.py tests

!95 Add BuildSentencePeiceVocab
* - Add BuildSentencePeiceVocab

!96 Fix more tests
* - Fix some tests

- Enable more test_c_api_*
- Add syncwait

!99 pybind switch for device op
* pybind switch for device op

!93 Add getters to python API
* Add getters to python API

!101 Validate tree, error if graph
* - Add sync wait

!103 TFrecord/Random Datasets schema problem
* - TfRecord/Random schem aproblem

!102 Added filter pybind switch
* Added Filter pybind switch

!104 Fix num_samples
* - TfRecord/Random schem aproblem

!105 Fix to_device hang
* Fix to_device hang

!94 Adds Cache support for CLUE dataset
* Added cache for all dataset ops
* format change
* Added CLUE cache support
* Added Cache conversion

Add save pybind

fix compile err

init modify concat_node

!107 Fix some tests cases
* Fix tests cases

Enable and fix more tests

!109 pybind switch for get dataset size
* pybind_get_dataset_size

some check-code fixes for pylint, cpplint and clang-format

!113 Add callback
* revert
* dataset_sz 1 line
* fix typo
* get callback to work

!114 Make Android compile clean
* Make Android Compile Clean

Fix build issues due to rebase

!115 Fix more tests
* Fix tests cases
* !93 Add getters to python API

fix test_profiling.py

!116 fix get dataset size
* fix get dataset size

!117 GetColumnNames pybind switch
* Added GetColumnNames pybind switch

code-check fixes: clangformat, cppcheck, cpplint, pylint

Delete duplicate test_c_api_*.py files; more lint fixes

!121 Fix cpp tests
* Remove extra call to getNext in cpp tests

!122 Fix Schema with Generator
* Fix Schema with Generator

fix some cases of csv & mindrecord

!124 fix tfrecord get_dataset_size and add some UTs
* fix tfrecord get dataset size and add some ut for get_dataset_size

!125 getter separation
* Getter separation

!126 Fix sampler.GetNumSamples
* Fix sampler.GetNumSampler

!127 Assign runtime getter to each get function
* Assign runtime getter to each get function

Fix compile issues

!128 Match master code
* Match master code

!129 Cleanup DeviceOp/save code
* Cleanup ToDevice/Save code

!130 Add cache fix
* Added cache fix for map and image folder

!132 Fix testing team issues
* Pass queue_name from python to C++
* Add Schema.from_json

!131 Fix Cache op issues and delete de_pipeline
* Roll back C++ change
* Removed de_pipeline and passing all cache tests.
* fixed cache tests

!134 Cleanup datasets.py part1
* Cleanup dataset.py part1

!133 Updated validation for SentencePieceVocab.from_dataset
* Added type_check for column names in SentencePieceVocab.from_dataset

Rebase on master 181120 10:20

fix profiling

temporary solution of catching stauts from Node.Build()

!141 ToDevice Termination
* ToDevice termination

pylint fixes

!137 Fix test team issues and add some corresponding tests
* Fix test team issues and add some corresponding tests

!138 TreeGetter changes to use OptPass
* Getter changes to use OptPass (Zirui)

Rebase fix

!143 Fix cpplint issue
* Fix cpplint issue

pylint fixes in updated testcases

!145 Reset exceptions testcase
* reset exception test to master

!146 Fix Check_Pylint Error
* Fix Check_Pylint Error

!147 fix android
* fix android

!148 ToDevice changes
* Add ToDevice to the iterator List for cleanup at exit

!149 Pylint issue
* Add ToDevice to the iterator List for cleanup at exit

!150 Pylint 2
* Add ToDevice to the iterator List for cleanup at exit

!152 ExecutionTree error
* ET destructor error

!153 in getter_pass, only remove callback, without deleting map op
* getter pass no longer removes map

!156 early __del__ of iterator/to_device
* early __del__ of iterator

!155 Address review comments Eric 1
* Added one liner fix to validators.py
* roll back signature fix
* lint fix
* Eric Address comments 2
* C++ lint fix
* Address comments Eric 1

!158 Review rework for dataset bindings - part 1
* Reorder nodes repeat and rename
* Review rework for dataset bindings - part 1

!154 Fixing minor problems in the comments (datasets.py, python_tree_consumer.cc, iterators_bindings.cc, and iterators.py)
* Fixing minor problems in the comments (datasets.py, python_tree_consum…

!157 add replace none
* Add replace_none to datasets.py, address comments in tests

Trying to resolve copy

Override the deepcopy method of deviceop

Create_ir_tree method

Create_ir_tree method 2

Create_ir_tree method 2

del to_device if already exists

del to_device if already exists

cache getters shapes and types

Added yolov3 relaxation, to be rolled back

Get shapes and types together

bypass yolo

NumWorkers for MapOp

revert Yolo

revert Thor

Print more info

Debug code: Update LOG INFO to LOG ERROR

do not remove epochctrl for getter pass

Remove repeat(1)

pritn batch size

add log to tree_consumer and device_queue op

Revert PR 8744

Signed-off-by: alex-yuyue <yue.yu1@huawei.com>

__del__ toDEvice

__del__ toDevice2

!165 add ifndef ENABLE_ANDROID to device queue print
* Add ifndef ENABLE_ANDROID to device queue print

revert some changes

!166 getter: get_data_info
* getter: get_data_info

!168 add back tree print
* revert info to warnning in one log
* add back the missed print tree log

Release GIL in GetDataInfo
2020-11-25 15:46:59 -05:00
mindspore-ci-bot 979bb26cba !9031 Fix DatasetCache parameter check
From: @lixiachen
Reviewed-by: @nsyca,@mikef
Signed-off-by: @nsyca
2020-11-26 03:55:02 +08:00
Lixia Chen ab7427f1a9 Minor fix for DatasetCache param validation 2020-11-25 13:20:30 -05:00
peixu_ren 93d4ef428f Add Gamma distribution 2020-11-25 09:40:46 -05:00
mindspore-ci-bot 22d683a805 !8920 Adapt ops LinSpace for Ascend.
From: @liu_xiao_93
Reviewed-by: @liangchenghui,@linqingke,@liangchenghui
Signed-off-by: @liangchenghui,@liangchenghui
2020-11-25 12:03:56 +08:00
mindspore-ci-bot d9b4b5c750 !8954 Add SequenceMask operator.
From: @liangzhibo
Reviewed-by: @chenfei52,@zh_qh
Signed-off-by: @zh_qh
2020-11-25 11:41:59 +08:00
liuxiao93 584e241e29 Adapt ops LinSpace for Ascend. 2020-11-25 09:50:03 +08:00
peixu_ren 01f5da0a14 Add Poisson distribution 2020-11-24 11:55:39 -05:00
anancds 7c73bc50f4 fixed codex 2020-11-24 21:04:04 +08:00
l00591931 17c0810f9e add SequenceMask operator 2020-11-24 19:47:28 +08:00
zhoufeng d532af3a9a new hccl interface
Signed-off-by: zhoufeng <zhoufeng54@huawei.com>
2020-11-24 17:34:35 +08:00
mindspore-ci-bot 0d49650bd5 !8888 Add PyNative Device Profiling
From: @jojobugfree
Reviewed-by: @kisnwang,@jjfeing
Signed-off-by: @jjfeing
2020-11-24 10:50:43 +08:00
mindspore-ci-bot 3d6d820612 !8154 Add nn.MultiFieldEmbedding for the embedding lookup opearations
From: @huangxinjing
Reviewed-by: 
Signed-off-by:
2020-11-24 09:49:39 +08:00
mindspore-ci-bot d915fd9b96 !8592 change repeat_elements to a composite op
From: @tom__chen
Reviewed-by: 
Signed-off-by:
2020-11-23 23:15:23 +08:00
huangxinjing b0deb7a289 Add dense embedding 2020-11-23 20:21:58 +08:00
mindspore-ci-bot ab26dd71dc !8789 Enable value inference in Concat; Support const input in graph_ir convertor.
From: @c_34
Reviewed-by: @liangchenghui,@guoqi1024
Signed-off-by: @liangchenghui
2020-11-23 19:54:32 +08:00
caifubi 12643a33dd Add profiling reporter 2020-11-23 16:53:11 +08:00
mindspore-ci-bot 2eee1c06f0 !8842 Change Ones/Zeros operator
From: @liangzhibo
Reviewed-by: @chenfei52
Signed-off-by:
2020-11-23 12:37:19 +08:00
mindspore-ci-bot 7f725b93a3 !8234 Remove expm1_generic and log1p_generic from PP utils
From: @peixu_ren
Reviewed-by: 
Signed-off-by:
2020-11-23 09:56:07 +08:00
l00591931 f1e92d0ea7 Change Ones/zeros 2020-11-23 09:33:38 +08:00
mindspore-ci-bot e3371a01d5 !8751 Fix minor bugs in bijector and distribution
From: @shallydeng
Reviewed-by: @zichun_ye,@sunnybeike
Signed-off-by:
2020-11-23 08:50:57 +08:00
shenwei41 4132228c10 Add c++ api of randomresizecropwithbbox 2020-11-21 17:06:56 +08:00
mindspore-ci-bot f6d7fc7eeb !8746 fixed get post message return type
From: @anancds
Reviewed-by: 
Signed-off-by:
2020-11-20 14:59:09 +08:00
mindspore-ci-bot cfb9fb40d0 !8794 Add nn.Interpolate() function.
From: @liangzhibo
Reviewed-by: @chenfei52,@zh_qh
Signed-off-by: @zh_qh
2020-11-20 14:55:21 +08:00
l00591931 05e26f4739 Add nn.interpolate() 2020-11-20 10:00:47 +08:00
anancds 69338af0f5 fixed get post message return type 2020-11-20 09:27:56 +08:00
mindspore-ci-bot 81dfd70ece !8785 Add Parallel Operator UnsortedSegmentMax
From: @huangxinjing
Reviewed-by: @yangzhenzhang,@stsuteng,@kisnwang
Signed-off-by: @stsuteng
2020-11-20 09:23:30 +08:00
tom__chen a52cd685fc change repeat_element op to a composite op 2020-11-19 15:42:22 -05:00
mindspore-ci-bot 812095de8f !8753 Adds execute testcase and fix get_dataset_size
From: @ezphlow
Reviewed-by: 
Signed-off-by:
2020-11-20 02:06:55 +08:00
chenhaozhe b3add83bf0 support const input in graph_ir convertor, add value inference in Concat 2020-11-19 20:34:49 +08:00
mindspore-ci-bot 8980bc3de7 !8675 support allreduce prod
From: @yao_yf
Reviewed-by: @stsuteng,@kisnwang
Signed-off-by: @stsuteng
2020-11-19 20:21:27 +08:00
mindspore-ci-bot 39352ca658 !8580 Add Operation AllSwap Primitive
From: @huangxinjing
Reviewed-by: 
Signed-off-by:
2020-11-19 19:41:06 +08:00
huangxinjing 89e7778497 Add UnsortedSegmentMax Operation 2020-11-19 17:41:54 +08:00
mindspore-ci-bot fa2185b8ef !8661 Add nn.Tril() and nn.Triu()
From: @liangzhibo
Reviewed-by: 
Signed-off-by:
2020-11-19 16:34:32 +08:00
mindspore-ci-bot f827c18f1d !8289 Add Parallel Implements of Slice Op
From: @huangxinjing
Reviewed-by: 
Signed-off-by:
2020-11-19 15:54:20 +08:00
Eric 45292abdfd Added execute test case
Added execute tests for server
2020-11-18 23:44:46 -05:00
mindspore-ci-bot a3ae3b9794 !8695 add AdamOffload optimizer
From: @wangnan39
Reviewed-by: @stsuteng,@stsuteng,@kingxian
Signed-off-by: @kingxian
2020-11-19 10:42:14 +08:00
l00591931 e1dba1337c Add nn.Tril function 2020-11-19 10:31:30 +08:00
Xun Deng b5e05472ce fix minor bugs in bijector and distribution utils, fix docs issues 2020-11-18 21:13:47 -05:00
mindspore-ci-bot c8047a35ab !8632 [MD] Add Multiply op for Lite Dataset
From: @jiangzhiwen8
Reviewed-by: @liucunwei,@heleiwang
Signed-off-by: @liucunwei
2020-11-19 09:58:43 +08:00
mindspore-ci-bot 215fce192e !8737 [MD] Fix the Pad function return false if bottom value is 0
From: @jiangzhiwen8
Reviewed-by: @xulei2020,@liucunwei,@heleiwang
Signed-off-by: @liucunwei
2020-11-19 09:07:25 +08:00