> When my code sees a '<', it checks whether the buffer has 3 additional
> characters (for the "!--"). If not, it reads more lines until it has the 3
Maybe you want to have the lexer check for {non-boundary}<--, then save
the fragment and run html_comment(), then append additional fragments to
the save buffer until the next word boundary is reached.