problem email, bogofilter and bogolexer hang

[Greg Louis]

I've warned David about a problem I encountered trying to rebuild my
training db with 0.10.0: the 1047th email in my spam corpus, if
processed alone, causes bogofilter to hang without output; if processed
as part of the 14,262-email mbox file, it causes bogofilter to exit
with an "Invalid buffer size" message.  It seems that the problem is in
the mime-decoding stuff, so I thought I'd bring it to the -dev list --
it might interest more people than just David.

If I run bogolexer on the problem email, it prints:

content-transfer-encoding
base64
¥þÀ
¨t¦c°Ó«~¤j¯s½æ
¨ÎÄrÄ_¤k©Ê±mùÛ«o¾i¨t¦c
§Ú¬°§aºÆ¨g2-¤k©Ê³æ¥ó¦¡¤º¿Ç
¡ñ¡ð­ì¸Ë¶i¤f¤é¥»±¡½ì°Ó«~¯s½æ
¿Ë¿ËÄ_¨©¥d³q¬ÃÂÃÀ
¶i¤f
ºë½o­Ó©Ê¤b¦r¿Ç¤j½à
¥þ·s¶
§c»ùÀu´f¼Æ
¤dºØºë¬üªº°Ó«~¨Ñ±zºcºc¿ïÁÊ

at which point it hangs; no further output till ctrl-C is pressed, but
the cpu load is 1 (and the cpu is running hot).

Of course, 0.8.0 bogolexer stops at base64 and then exits without
printing anything further.

Copy of problem email (gzipped so it shouldn't hang anybody's
bogofilter) available on request.

-- 
| G r e g  L o u i s          | gpg public key:      |
|   http://www.bgl.nu/~glouis |   finger greg at bgl.nu |
| Help free our mailboxes. Include                   |
|        http://wecanstopspam.org in your signature. |