Hapax survival over time

[Bill McClain]

As part of my daily processing I've been capturing some data on hapaxes
(wordlist tokens that have been registered only once) and charting their
survival over time. I'm about to recreate my wordlist and change my
processing, so I thought I would report on what I've found, even though
the data is tentative and incomplete.

In the chart below:

* column 1 is the age in days
* column 2 is the sum of "day 0" hapaxes aggregated in this bucket
* column 3 is percent of original hapaxes remaining
* column 4 is a graphical representation of the same, where I've omitted
the left 50% of the bar to save space

 0 50667 100.0 **************************************************
 1 48010  98.1 ************************************************
 2 43689  96.7 **********************************************
 3 43525  95.7 *********************************************
 4 41579  93.6 *******************************************
 5 38564  92.2 ******************************************
 6 36477  91.6 *****************************************
 7 33590  90.5 ****************************************
 8 33510  90.5 ****************************************
 9 31241  91.4 *****************************************
10 28344  88.7 **************************************
11 26811  88.2 **************************************
12 24817  89.2 ***************************************
13 23466  89.2 ***************************************
14 21482  88.1 **************************************
15 19497  88.3 **************************************
16 17535  87.3 *************************************
17 15778  86.6 ************************************
18 14688  86.8 ************************************
19 12547  85.5 ***********************************
20 10830  85.7 ***********************************
21  9413  85.2 ***********************************
22  7232  82.6 ********************************
23  6017  82.9 ********************************
24  4598  84.1 **********************************
25  2658  77.1 ***************************
26  3553  81.1 *******************************
27  2024  84.1 **********************************

Notes:

* Counts should be strictly declining (we don't back-date tokens and
create new hapaxes in the past) but I am aggregating many days counts
together and do not yet have complete series of data, so the percentages
may fluctuate both ways. The smoothness and stability of the curve is
better higher in the chart, and worse lower where there are fewer reps.

* These are all spam hapaxes. I have more ham than spam messages in my
wordlist and so have not been registering ham during this period.

* There have been some bogofilter registration changes over this period
which may distort the data.

Conclusions:

None yet. I'm trying to determine how valuable old hapaxes are, to see
if they could be purged without harm. It will be interesting to see if
the decline levels off over a longer period of time.

-Bill
-- 
Sattre Press                              History of Astronomy 
http://sattre-press.com/               During the 19th Century
info at sattre-press.com                       by Agnes M. Clerke
                              http://sattre-press.com/han.html