epickh
/
Parcel
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

Improved tokenizer to make it possible to have digits in a word

This commit is contained in:
Eric-Paul Ickhorn 2023-11-28 16:00:10 +01:00
parent de80fc7b79
commit 138e361c54
1 changed files with 1 additions and 1 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

2
code/src/tokenizer.c
View File

@ -99,7 +99,7 @@ pac_tlist_s pac_tokenize_grammar(char *source, usz_t len_source)
++offset; ++offset;
subject = source[offset]; subject = source[offset];
if(!pac_rune_is_letter(subject) && (subject != '_')) if(!pac_rune_is_letter(subject) && !pac_rune_is_digit(subject) && (subject != '_'))
{ {
break; break;
} }