lexer mod

[Matthias Andree]

David Relson <relson at osagesoftware.com> writes:

> The patch below does the trick.  Not surprisingly it affects "make
> check".  If you're sure we want it, I'll add it and update the reference
> results so "make check" will be happy.

I'm not convinced it improves accuracy, but I'm convinced it keeps
unique tokens out of the data base, and it fits into the scheme of
discarding the Message-ID (likely we'd also need to scrap Outlook's
Thread-* stuff) and queue IDs from headers.

> diff -u -r1.55 token.c
> --- token.c	6 Sep 2003 20:50:39 -0000	1.55
> +++ token.c	9 Sep 2003 02:50:01 -0000
> @@ -77,6 +77,7 @@
>  	cls = lexer->yylex();
>  	yylval->leng = *lexer->yyleng;
>  	yylval->text = (unsigned char *)(*lexer->yytext);
> +	yylval->text[yylval->leng] = '\0';
>  
>  	if (DEBUG_TEXT(2)) { 
>  	    word_puts(yylval, 0, dbgout);

Do we need this diff? word_puts handles it.

-- 
Matthias Andree

Encrypt your mail: my GnuPG key ID is 0x052E7D95