yystopf
/
mindspore
forked from mindspore-Ecosystem/mindspore
0
0
Fork 0
Code Issues Pull Requests Projects Releases Wiki Activity
mindspore/tests/ut/data/dataset/testTokenizerData/regex_tokenizer.txt

3 lines
70 B
Plaintext
Raw Normal View History

Add WhitespaceTokenizer and UnicodeScriptTokenizer for nlp add CaseFold, NormalizeUTF8 add RegexReplace add RegexTokenizer add BasicTokenizer add WordpieceTokenizer add BertTokenizer
2020-05-05 16:51:05 +08:00
Welcome to Shenzhen!
北京欢迎您!Welcome to Beijing!
12￥+36￥=?