yystopf
/
mindspore
forked from mindspore-Ecosystem/mindspore
0
0
Fork 0
Code Issues Pull Requests Projects Releases Wiki Activity
mindspore/tests/ut/data/dataset/testTokenizerData/normalize.txt

6 lines
30 B
Plaintext
Raw Normal View History

Add WhitespaceTokenizer and UnicodeScriptTokenizer for nlp add CaseFold, NormalizeUTF8 add RegexReplace add RegexTokenizer add BasicTokenizer add WordpieceTokenizer add BertTokenizer
2020-05-05 16:51:05 +08:00
ṩ
ḍ̇
q̣̇
fi
2⁵
ẛ̣