6 Commits

Author	SHA1	Message	Date
hesham	adfbc891d3	- Add checks and testing for empty tensors ... - cleanup work on createTensor and Tensor's constructors	2020-07-24 17:23:18 -04:00
liubuyu	43c79eb853	mindspore path adjust	2020-07-14 18:07:28 +08:00
xiefangqi	47060631e5	add offsets feature to tokenizer	2020-07-11 11:20:02 +08:00
qianlong	4f16f036be	Add WhitespaceTokenizer and UnicodeScriptTokenizer for nlp ... add CaseFold, NormalizeUTF8 add RegexReplace add RegexTokenizer add BasicTokenizer add WordpieceTokenizer add BertTokenizer	2020-06-17 15:47:04 +08:00
hesham	6c21e556c4	Clean up work for text python package	2020-05-22 04:35:58 -04:00
qianlong	451c20a6f5	Add UnicodeCharTokenizer for nlp	2020-05-21 09:22:45 +08:00