mindspore/tests/ut/data/dataset/testTokenizerData/wordpiece_tokenizer.txt

forked from mindspore-Ecosystem/mindspore
my
favorite
book
is
love
during
the
cholera
era
what
我
最
喜
欢
的
书
是
霍
乱
时
期
的
爱
情
您
Add WhitespaceTokenizer and UnicodeScriptTokenizer for nlp add CaseFold, NormalizeUTF8 add RegexReplace add RegexTokenizer add BasicTokenizer add WordpieceTokenizer add BertTokenizer 2020-05-05 16:51:05 +08:00			`my`
			`favorite`
			`book`
			`is`
			`love`
			`during`
			`the`
			`cholera`
			`era`
			`what`
			`我`
			`最`
			`喜`
			`欢`
			`的`
			`书`
			`是`
			`霍`
			`乱`
			`时`
			`期`
			`的`
			`爱`
			`情`
			`您`