Zie ook: AlpinoTokenizerPerl

installeren

ln -s /net/aistaff/vannoord/z/Alpino/Tokenization/libtok_no_breaks.c .
perl -p -e 's/QDATUM|new_t_accepts|qentry|qinit|qinsert|qpeek|qremove|queue|replace_from_queue|resize_buf|t_accepts|transition_struct|trans|unknown_symbol/$&1/g' \
    /net/aistaff/vannoord/z/Alpino/Tokenization/libtok.c > libtok1.c
python3 setup.py install --prefix $HOME

gebruik

   1 import AlpinoTokenizer
   2 
   3 zin = "Dit is een test. En dit ook! Etc., enz."
   4 
   5 print(AlpinoTokenizer.tokenize(zin))        # zonder newlines
   6 print(AlpinoTokenizer.tokenize(zin, False)) # zonder newlines
   7 print(AlpinoTokenizer.tokenize(zin, True))  # met newlines
   8 
   9 AlpinoTokenizer.test()

attachments

Download alles: AlpinoTokenizerPython.tar.gz

CategoryAlpino CategoryCorpora CategoryPython CategorySwig