\W in regex matches Chinese characters, causing all Chinese tokens
to be filtered out. Replaced with \p{P} (Unicode punctuation) to
correctly preserve Chinese tokens while filtering punctuation.
|
||
|---|---|---|
| .. | ||
| generate-doc-index.js | ||
\W in regex matches Chinese characters, causing all Chinese tokens
to be filtered out. Replaced with \p{P} (Unicode punctuation) to
correctly preserve Chinese tokens while filtering punctuation.
|
||
|---|---|---|
| .. | ||
| generate-doc-index.js | ||