mindspore/tests/ut/data/dataset/testTokenizerData/basic_tokenizer.txt

7 lines
502 B
Plaintext
Raw Blame History

This file contains ambiguous Unicode characters

This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.

Welcome to Beijing北京欢迎您
長風破浪會有時,直掛雲帆濟滄海
😀嘿嘿😃哈哈😄大笑😁嘻嘻
明朝1368—1644年和清朝1644—1911年是中国封建王朝史上最后两个朝代
明代1368-1644と清代1644-1911は、中国の封建王朝の歴史における最後の2つの王朝でした
명나라 (1368-1644)와 청나라 (1644-1911)는 중국 봉건 왕조의 역사에서 마지막 두 왕조였다
Tĥïŝ ĩš â fůňķŷ Šťŕĭńġ