Tokenizer mem

It seems Tokenizer code read past the memory block.

TokenizerMem.txt

Should just give the tokenizer extra memory or fix it STOP past a memory point?

added 2 commits

• 8c7fe2ae - Make tokenizer keep a copy of string in normal memory
• da1b579c - Tokenizer_Open: pad the string with an NULL byte

Compare with previous version

added 1 commit

• b8023aee - Tokenizer: pad with 2 NULL bytes

Compare with previous version

marked this merge request as ready

merged

mentioned in commit e0749737