Collectives™ on Stack Overflow
Find centralized, trusted content and collaborate around the technologies you use most.
Learn more about Collectives
Teams
Q&A for work
Connect and share knowledge within a single location that is structured and easy to search.
Learn more about Teams
I'm trying to make a language agnostic text parser in Scala/Spark. I was using regex to remove punctuation from large groups of text.
list_of_strings.map(_.replaceAll("""[\p{Punct}]"""))
However, I find a number of punctuation marks in Chinese text that are not removed. So far I have added [〝〞。，＂] to my regex for these characters. Notice that many of these are similar to latin punctuation marks, but not quite the same.
Is there a 'cannonical' function built into Java regex that will allow me to strip punctuation in other languages? Other than Chinese, I'm concerned about Arabic.
Also, I want to ensure that stripped punctuation in one language isn't a critical language component in another. I am not familiar with either the Chinese or Arabic languages, so I'm looking for something premade by a native speaker.
You do not need to add all those Unicode punctuation chars into the character class. Make the \p{Punct} Unicode-aware with the Pattern.UNICODE_CHARACTER_CLASS or an equivalent (?U) embedded flag option:
.replaceAll("""(?U)\p{Punct}+""", "")
See the Scala demo.
The + after \p{Punct} removes consecutive punctuation chars is chunks rather than one by one.
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
