p47068915
/
mindspore
forked from mindspore-Ecosystem/mindspore
1
0
Fork 0
Code Issues Pull Requests Projects Releases Wiki Activity
11,530 Commits 23 Branches 0 Tags 3.9 GiB
Commit Graph

139 Commits

Author SHA1 Message Date
islam_amin 56da3b0ae1 Fixing ratio bug with BoundingBoxAugment 2020-07-13 18:11:19 -04:00
Jesse Lee 7c1bc5192b Initial Drop of CacheOp Phase I 2020-07-13 10:18:53 -04:00
heleiwang 7a046a1d70 support get_edge_feature 2020-07-10 17:05:23 +08:00
mindspore-ci-bot 2c3c204b14 !2982 supporting cpp unit tests for randomresizewithbbox and resizewithbbox 
Merge pull request !2982 from ava/CPP_test3
 2020-07-10 04:56:36 +08:00
avakh b2bfb0342b supporting cpp unit tests for random_resize_with_bbox_op and resize_with_bbox_op. 2020-07-09 14:36:32 -04:00
Danish Farid 506f8f134e fix, cpp ut's 
fix cmakefile

fixed log statments in cpp UT tests
 2020-07-09 11:32:13 -04:00
mindspore-ci-bot 3cc9217f06 !2926 [Dataset C++ UT] Implement C++ UTs for RandomHorizontalFlipWithBBox and BoundingBoxAugment 
Merge pull request !2926 from islam_amin/object_ops_cc_ut_floats
 2020-07-09 05:56:46 +08:00
islam_amin 284cc910d5 C++ UTs for RandomHorizontalFlipWithBBox and Bounding Box Augment 2020-07-08 14:36:27 -04:00
avakh 421b80bf8f updating python unit tests based on floating point bounding boxes + adding coco dataset in unit tests 2020-07-07 18:05:49 -04:00
mindspore-ci-bot 9ba937b1af !2920 Updating 3 BBox AugOps for Float + Common Files for BBox Float Support - (VertFlip/RandomCrop/RandomResizeCrop) 
Merge pull request !2920 from danishnxt/AugOps-M1-Python
 2020-07-08 05:18:58 +08:00
Danish Farid 78c370f72a First Commit - New Infrastructure - Python UT tests + Common Aug Files 
fix accidental change overwrite

fix 2

updated inits from 0 to 0.0 for float

improvd python ut comments

updated macro
 2020-07-07 14:50:34 -04:00
islam_amin dfc3409f67 Update RandomHorizontalFlipWithBBox and BoundingBouxAugment C++ Ops to use floats 2020-07-06 23:49:21 -04:00
tinazhang dc03019200 unskip md5 testcase for RandomPerspective 2020-07-03 13:10:13 -04:00
mindspore-ci-bot 3197f9fb35 !2733 Schema clean up 
Merge pull request !2733 from EricZ/schema_fix
 2020-06-30 05:31:27 +08:00
Eric 88bb65768e Clean up part 1 
Removed extra fields in schema

Fixed test cases

Fixing CI
 2020-06-29 15:57:53 -04:00
tinazhang66 5cd3136355 remove local defined mse and add missing mse/md5 validation 2020-06-29 11:51:07 -04:00
mindspore-ci-bot 32a2d5d21c !2629 get monitor sampling interval from json file 
Merge pull request !2629 from yanghaitao/yht_sampling_interval
 2020-06-28 20:05:05 +08:00
qianlong cae77c0c22 BasicTokenizer not case fold on preserverd words 2020-06-28 16:28:00 +08:00
yanghaitao fd701b8504 set monitor sampling interval through json file 2020-06-28 15:18:21 +08:00
mindspore-ci-bot a1b17bd2e2 !2593 python unit tests for randomResizeWithBBox and ResizeWithBBox 
Merge pull request !2593 from ava/python_ut_tests
 2020-06-27 02:31:31 +08:00
avakh 1e869146e9 applying comments removing VOC 2020-06-26 12:39:48 -04:00
Cathy Wong 277aba5326 dataset: Fixup docs; remove pylint disabled messages in UT 2020-06-26 12:26:41 -04:00
mindspore-ci-bot e11c953225 !2587 Implementing Python UTs for BoundingBoxAugment and RandomHorizontalFlipWithBBox 
Merge pull request !2587 from imaaamin/object_ops_ut
 2020-06-26 00:43:31 +08:00
islam_amin 0f58f0338e updating ut for RandomHorizontalFlipWithBBox and BBoxAugment 2020-06-25 11:35:22 -04:00
Danish Farid 3d1ecaaeb5 updated UT test for Python (3) AugOps with BBox - MD5 checks + imrpv comments 2020-06-25 03:19:14 -04:00
Danish Farid a8f392ca42 updated test util file + new BoundingBoxCheck + fixed VOCDataset annotations 
Style Error fix

fixed PyLint problem

reverting testVOC2012 due to CI break for existing test

reverting testVOC2012 due to CI break for existing test-2

updated old error messages to confirm with global standard

addressing PR 2355 Comments - 1

addressing PR 2355 Comments - 2

addressing PR 2355 Comments - 3
 2020-06-24 12:41:41 -04:00
ms_yan cf82aa9035 init remove storage op in c++ 
init remove storage op test case c++

remove source c++ files
 2020-06-20 13:52:07 +08:00
mindspore-ci-bot 90bb9320aa !2314 Implemented RandomHorizontalFlipWithBBox and BoundingBoxAugment C++ Ops 
Merge pull request !2314 from imaaamin/object_ops_pr
 2020-06-19 07:08:42 +08:00
islam_amin 0af156d169 added two ops for object detection, random horizontal flip with bbox and bouding box augment. along with their unit tests 2020-06-18 16:55:31 -04:00
Zirui Wu b6e9504b31 phase I of Vocab rework 
phase II vocab rework

added more test cases

fix api doc string

address review cmts and fix CI

address ci complains

fix review cmts

ci
 2020-06-18 15:36:28 -04:00
mindspore-ci-bot 2ecd5bdf25 !2239 [MD] convert csv to mindrecord 
Merge pull request !2239 from liyong126/csv_to_mindrecord
 2020-06-18 18:56:33 +08:00
ms_yan 8d1dae46ac Throw error when load config failed 2020-06-18 13:47:32 +08:00
liyong 7369950a93 convert csv to mindrecord 2020-06-18 12:55:22 +08:00
qianlong 4f16f036be Add WhitespaceTokenizer and UnicodeScriptTokenizer for nlp 
add CaseFold, NormalizeUTF8

add RegexReplace

add RegexTokenizer

add BasicTokenizer

add WordpieceTokenizer

add BertTokenizer
 2020-06-17 15:47:04 +08:00
mindspore-ci-bot ca94916ef9 !2184 CocoDataset add category_id check 
Merge pull request !2184 from xiefangqi/xfq_fix_coco_issue
 2020-06-17 10:06:41 +08:00
mindspore-ci-bot 625f2421b5 !1977 random walk in gnn node2vec 
Merge pull request !1977 from JonathanY/randomwalk
 2020-06-17 03:22:21 +08:00
xiefangqi aab190fcdb CocoDataset add category_id validation 2020-06-16 21:37:22 +08:00
jonyguo 3353a20d8b add tool: tfrecord to mindrecord 2020-06-16 12:09:21 +08:00
mindspore-ci-bot dceb3c2be4 !2026 Change C++ behavior in RandomCropAndResize 
Merge pull request !2026 from MahdiRahmaniHanzaki/I1K3FL-random-resized-crop
 2020-06-16 05:27:36 +08:00
Mahdi add29f95f5 resolved issues in the comments 2020-06-15 16:45:18 -04:00
Jonathan Yan 87d2c27c7f random walk v1 2020-06-15 08:54:10 -04:00
jiangzhiwen e0e167a000 add CLUE dataset 2020-06-15 10:40:46 +08:00
ms_yan bc22c172b8 add TensorDataset and its ut 2020-06-11 15:24:12 +08:00
mindspore-ci-bot b3da41bd7a !1913 CocoDataset implementation 
Merge pull request !1913 from xiefangqi/xfq_support_coco
 2020-06-11 14:37:44 +08:00
xiefangqi 097cbcc0cb CocoDataset implementation 2020-06-11 11:52:17 +08:00
mindspore-ci-bot fe0c3e9a8f !1908 Add UT to six data vision transforms ops 
Merge pull request !1908 from Tinazhang/ut-normalize
 2020-06-11 00:43:57 +08:00
heleiwang 3ece8dd090 1. support get_all_edges, get_nodes_from_edge, get_sampled_neighbors, get_neg_sampled_neighbors and graph_info API 
2. mod cora and citeseer conversion script
 2020-06-10 15:52:26 +08:00
Tinazhang c3de5c7a54 adding new TCs to six ops and skip md5 case in RandomPerspective 2020-06-09 10:51:45 -04:00
mindspore-ci-bot 976b4793d1 !1544 Python Unit Test for TenCrop 
Merge pull request !1544 from MahdiRahmaniHanzaki/I1IHNG-ten-crop
 2020-05-30 03:02:29 +08:00
Tinazhang f4a61fdbb2 Adding new UT cases and missing TCs in PY Ops. 2020-05-29 12:12:17 -04:00
Mahdi dfc097019b Added TenCrop test 
Added unit tests for both testing the functinality of the TenCrop and
its error messages. Due to the similarity of this method to FiveCrop the
test cases are similar to FiveCrop test cases.
Signed-off-by: Mahdi <mahdi.rahmani.hanzaki@huawei.com>

added error_msg function call in the main method

refactored the test and added visual representation of the results

Separated the two error cases into two different functions and used the
visualize function in util.py to plot the result of TenCrop.
Signed-off-by: Mahdi <mahdi.rahmani.hanzaki@huawei.com>

Added new test cases

Added new test cases including test case for checking the error message
when the size variable is not a positive integer, test case for
rectangle crop, test case for vertical flip setting, and testing for
similarity of the result of TenCrop for the same input data in different runs.
Signed-off-by: Mahdi <mahdi.rahmani.hanzaki@huawei.com>

changed visualize in test_five_crop

Changed the visualize function in test_five_crop to use the already
existing function in util.py
Signed-off-by: Mahdi <mahdi.rahmani.hanzaki@huawei.com>

made generate_golden variable global
 2020-05-28 17:52:56 -04:00
Yang abca62f407 10:00 26/5 clean pylint 2020-05-26 10:05:12 +08:00
mindspore-ci-bot aeb4c52f2d !1379 Creating new UTs to three vision transformation ops 
Merge pull request !1379 from Tinazhang/linearTrans
 2020-05-25 23:10:30 +08:00
Tinazhang 7322839b04 add UTs for LinearTransformation, ToPIL, ToType 2020-05-22 18:04:09 -04:00
Peilin Wang 0cbcc7200b made shuffle determinisitc for subsequent epochs 2020-05-22 17:03:19 -04:00
mindspore-ci-bot 458436186c !1365 Clean up work for text python sub-package 
Merge pull request !1365 from h.farahat/text_namespace
 2020-05-22 23:16:31 +08:00
mindspore-ci-bot 6f733ec113 !1308 Stage 2 of adding support for string Tensor 
Merge pull request !1308 from h.farahat/string_tensor2
 2020-05-22 21:17:53 +08:00
hesham 6c21e556c4 Clean up work for text python package 2020-05-22 04:35:58 -04:00
heleiwang 599a449e0b Support processing GNN data 2020-05-22 14:15:25 +08:00
hesham df361d1d26 Change mem layout of string tensor 
add support for MindRecord and TFRecord
----
optimize tensorshape

optimize tensorshape and FlatIndex

TFRecord and MindRecord support for string tensor

Modify mem layout
Add new constructor
Add method Allocate

Change some GetMutableBuffer usages to AllocateBuffer
 2020-05-22 01:22:29 -04:00
mindspore-ci-bot 58e6d7d950 !1341 Added lookup and vocab to mindspore.dataset.text 
Merge pull request !1341 from ZiruiWu/vocab_and_lookup
 2020-05-22 10:19:42 +08:00
jonwe bb51bb88d7 add compress in mindrecord 2020-05-22 09:37:51 +08:00
Zirui Wu 25ab2ef303 Implemented lookup and vocab 2020-05-21 17:17:24 -04:00
qianlong 451c20a6f5 Add UnicodeCharTokenizer for nlp 2020-05-21 09:22:45 +08:00
xulei2020 163b6b7ea7 add jieba c++ code 2020-05-20 15:55:12 +08:00
Tinazhang 17cecf2cf5 Added TCs to RandomCrop and RandomCropAndResize and modified visalize() calling 2020-05-19 15:42:24 -04:00
jinyaohui bcfaff97f9 clean pylint 2020-05-18 10:31:46 +08:00
jiangzhiwen cb2814b498 flat_map first commit 2020-05-15 17:45:39 +08:00
mindspore-ci-bot ab031ee9ea !1126 VOCDataset support object detection function 
Merge pull request !1126 from xiefangqi/voc_support_detection
 2020-05-15 15:56:39 +08:00
xiefangqi c937bad53f minddata support voc 2020-05-15 13:24:03 +08:00
Cathy Wong 913074e656 Cleanup dataset UT: resolve skipped test units 2020-05-13 14:41:57 -04:00
Cathy Wong 49ef53f164 Cleanup dataset UT: util.py internals 2020-05-11 14:44:24 -04:00
mindspore-ci-bot 2860fd9338 !984 Add unit test case for HWC2CHW. 
Merge pull request !984 from Tinazhang/hwc2chw
 2020-05-09 05:02:41 +08:00
Tinazhang c8b5586c7f add unit test for HWC2CHWC 2020-05-08 13:17:20 -04:00
Cathy Wong 58226addd6 Cleanup dataset UT: use md5 npz in test_zip for images 2020-05-08 11:25:48 -04:00
Cathy Wong 772e6c1461 Cleanup dataset UT: test_batch, save_and_check support 2020-05-05 15:35:09 -04:00
eric 36fffb7706 Added example md5 generation 
Comparison example

Added md5 and comparison example for py_transforms

Added md5 check for images
 2020-05-04 21:15:33 -04:00
Jesse Lee 270bf831a9 Random Data Op 2020-04-29 10:26:00 -04:00
yanghaitao 1cb3bead32 skip black line for TextFileDataset 2020-04-27 14:54:26 +08:00
qianlong db80f4ff92 The num_samples and numRows in schema for TFRecordDataset are conflict 2020-04-21 17:08:59 +08:00
liyong f1542a90a3 add pk sampler 2020-04-21 10:08:38 +08:00
mindspore-ci-bot 6369cf27bd !406 added first row crc check for when reading tfrecord files 
Merge pull request !406 from Peilin/first-row-crc-check
 2020-04-21 00:10:18 +08:00
xulei2020 c705ea5e5b add filterOp code 2020-04-20 22:32:19 +08:00
Peilin Wang 9bc2134cb7 added checking of first row crc to find invalid tfrecord files 
addressed code review comments. added check in python layer to exclude directories and to raise an error if a pattern does not match any file

fixed clang format

fixed cppcheck

fixed cppcheck (used std::accumulate and std::copy_if). regenerated tfrecord file to contain correct header, it was a dummy header before

fixed cppcheck: added const reference for string parameter for lambdas, fixed clang format: whitespace adjustments

more clang whitespace fixes...

changed print to logger.info
 2020-04-20 10:25:26 -04:00
yanghaitao 2795e492ff TextFileDataset 2020-04-20 11:18:55 +08:00
Cathy Wong 59a714c654 Correct shuffle UT buffer_size > #dataset-row as valid 2020-04-09 15:22:33 -04:00
yanzhenxiang2020 5a4f17bfb6 fix mindrecord ut long time 2020-04-02 19:18:49 +08:00
Cathy Wong 2e881276ab Enable skipped dataset zip python unit tests 
Signed-off-by: Cathy Wong <cathy.wong@huawei.com>
 2020-03-31 16:47:13 -04:00
zhunaipan 930a1fb0a8 initial version 
Signed-off-by: leonwanghui <leon.wanghui@huawei.com>
 2020-03-27 22:54:54 +08:00