Commit Graph

21 Commits

Author SHA1 Message Date
z00505269 87668d6ea2 remove predict 2020-08-01 11:45:19 +08:00
shenwei41 051c290d8b Modify patches and alerts 2020-07-29 19:26:06 +08:00
xulei2020 18b519ae0f add sentence piece 2020-07-20 15:50:35 +08:00
ZPaC 4645a43e08 Add ps module in batches 2020-07-09 15:45:56 +08:00
qianlong 0c35247b58 change icu4c compile way 2020-07-07 14:54:49 +08:00
jonyguo d0cd4cb83c del: zhwiki & enwiki & CLUERNER2020 preprocess script 2020-07-02 08:59:17 +08:00
ZPaC a057639abc Add pslite dependency 2020-07-01 16:26:24 +08:00
qianlong 4f16f036be Add WhitespaceTokenizer and UnicodeScriptTokenizer for nlp
add CaseFold, NormalizeUTF8

add RegexReplace

add RegexTokenizer

add BasicTokenizer

add WordpieceTokenizer

add BertTokenizer
2020-06-17 15:47:04 +08:00
Yang d7d8688ec4 !2118 update sqlite & patch 2020-06-16 09:31:12 +08:00
jonyguo 2c0258d373 add example for enwiki -> mindrecord 2020-06-11 19:00:11 +08:00
jonyguo 56d03f9eb9 enhance: add example for zhwiki and CLUERNER2020 to mindrecord 2020-06-04 14:24:06 +08:00
jonyguo 698c88f306 update sqlite patch 2020-05-25 11:06:57 +08:00
xulei2020 163b6b7ea7 add jieba c++ code 2020-05-20 15:55:12 +08:00
dengwentao 08968c2744 modify tvm build 2020-04-21 20:54:40 +08:00
mindspore-ci-bot 76c700fb35 !432 Support for patching and compiling sqlite on windows
Merge pull request !432 from ZhouFeng/windows-sqlite-patch
2020-04-18 16:52:00 +08:00
zhoufeng 638b119099 windows sqlite patch 2020-04-18 15:10:16 +08:00
chenjianping 780aa6f87b Add license head in build.bat 2020-04-18 01:41:25 +00:00
chenjianping 523fde0e25 eliminate securec warning 2020-04-17 02:33:37 +00:00
chenjianping c8da1cf8e9 support building on windows|support package 2020-04-16 13:44:20 +00:00
chenjianping 1286767d0e support building on windows 2020-04-15 09:36:48 +00:00
zhunaipan 930a1fb0a8 initial version
Signed-off-by: leonwanghui <leon.wanghui@huawei.com>
2020-03-27 22:54:54 +08:00