2003-01-30  Brian Burton  <brian@burton-computer.com>

	* Updated version of spamprobe.el from Dave Pearson's web site.

	* Added README-mta-mda-mua.txt graciously contributed by Anto Veldre.

	* MessageFactory.cc (removeHTMLFromText): replaces all whitespace
	  with space characters to avoid wierd crash in regex routines on
	  RedHat 8 systems.

2003-01-28  Brian Burton  <brian@burton-computer.com>

	* spamprobe.cc (main): Added -M option to force a single message
	  per file (ignores content-length and From).

	* Fixed manpage.

	* Changed lock file mode to 0666 instead of 0600 so that shared
	  locks will work better.  TODO: need to eliminate the need for
	  the lock file altogether.

2002-12-29  Brian Burton  <brian@burton-computer.com>

	* MessageFactory.cc (expandCharsInURL): Added decoding of %xx encoded
	  characters in URLs.  Using this to prevent spammers from slipping
	  URLs through unchallenged by encoding them completely as hex.

2002-12-26  Brian Burton  <brian@burton-computer.com>

	* spamprobe.cc (main): SpamProbe now stores words and phrases in
	  the to, cc, and subject headers both normally and with a special
	  prefix to improve accuracy since some words are spammier in the
	  subject than in the message body.
	  (main): Added -p option to limit number of words per phrase.
	* Released version 0.8

2002-11-12  Brian Burton  <brian@burton-computer.com>

	* spamprobe.cc (classify_message): spam and good commands now
	  count the words from messages multiple times if necessary to
	  ensure that they are recalled correctly.  receive command does
	  not do this since it's decisions are not as reliable as manual
	  ones.  This is intended to improve overall accuracy by
	  maximizing recall and making it harder for "spams of the future"
	  to slip through the cracks because of their low word counts.

2002-10-28  Brian Burton  <brian@burton-computer.com>

	* spamprobe.cc (import_words): Fixed broken import command.

2002-10-27  Brian Burton  <brian@burton-computer.com>

	* spamprobe.cc (process_stream): Added summarize command to print
	  find-good style output for every message whether good or spam.

2002-10-26  Brian Burton  <brian@burton-computer.com>

	* MimeMessageReader.cc (readNextHeader): Uses inexact content
	  length in case the mbox has incorrect content-length values.

	* spamprobe.cc (process_stream): score and receive now print
	  message digest along with the score.
	  (main): all commands except receive look for digest in
	  X-SpamProbe header

	* MessageFactory.cc (assignDigestToMessage): message digest now
	  taken from header if available.

2002-10-24  Brian Burton  <brian@burton-computer.com>

	* spamprobe.cc (dump_words): flags now show all 8 digits in dump
	(process_stream): receive mode supports -T option

2002-10-22  Brian Burton  <brian@burton-computer.com>

	* WordData.h (class WordData): Modified database to store a
	  16 bit time stamp (days since August 12, 2002) instead of
	  using sweep count for database cleanup.

2002-10-20  Brian Burton  <brian@burton-computer.com>

	* MessageFactory.cc (addTextToMessage): Removed the to_lower() to
	  avoid unecessary string copying.  Made regex's case insensitive
	  so that they are not needed.

	* spamprobe.cc (import_words): Uses regular expression to parse
	  import lines instead of hard coded logic.

	* configure.in: Added test to verify existence of regex.h on
	  target system.

	* MimeHeader.cc (isFromLine): Uses regular expression to detect
	  From lines instead of the hard coded scans.

	* RegularExpression.h (class RegularExpression): Added
	  RegularExpression class as a front-end for POSIX regular
	  expression library.

	* MessageFactory.cc: Uses regular expressions instead of hardcoded
	  logic to detect html tags in pages and find urls inside of tags.

	* README.txt (including): Aded --enable-assert to configure script
	  so that assertions are off by default but can still be enabled
	  for debugging purposes.

2002-10-16  Brian Burton  <brian@burton-computer.com>

	* spamprobe.1: Changed version to just 0.7 so I don't have to keep
	  it up to date constantly.

	* contrib/spamprobe.el: Updated to latest version of spamprobe.el
	  Thanks Dave!

	* spamprobe.cc (main): Added --enable-8bit option to configure
	  script.

2002-10-15  Brian Burton  <brian@burton-computer.com>

	* configure.in (have_database): Moved berkeleydb tests into a
	  common spot.  Added -ldb3 to the list of libraries to check.

	* Switched to autoconf generated Makefile instead of the manual
	  one.  The original makefile is now named Makefile.orig.

	* Moved md5 files out of thirdparty and into the top level
	  directory.

2002-10-14  Brian Burton  <brian@burton-computer.com>

	* countscores.rb (goods): Changed to accomodate change to score
 	  output.

	* spamprobe.cc (process_stream): score command prints in same
	  format as receive command to simplify using it in procmailrc.
	  (find_message): Prints subject of message to make output more
	  human understandable.

	* SpamFilter.cc (normalScoreMessage): Fixed NAN bug if inner and
	  outer both approx. 0.  Returns 0.5 in that case to be safe.

	* spamprobe.cc (main): Added command name validation and support
	  for shared locks for read only commands.  Moved database lock
	  acquisition into the FrequencyDBImpl classes.

2002-10-11  Brian Burton  <brian@burton-computer.com>

	* FrequencyDBImpl_dual.h (class FrequencyDBImpl_dual): Added new
	  database impl class that uses a shared read only database and a
	  private read-write one.  Also added -D option to program to
	  allow user to specify the shared db dir and made numerous
	  changes to other classes to put this new option into effect.

	* spamprobe.cc (main): Modified to process multiple mboxes much
	  faster by opening and closing the database only once instead of
	  once per file.

	* Released SpamProbe-0.7d

	* spamprobe.cc (main): Added purge and edit-term commands.

	* FrequencyDBImpl_bdb.cc (sweepOutJunk): Added purge mode.

2002-10-08  Brian Burton  <brian@burton-computer.com>

	* FrequencyDB.h (class FrequencyDB, class FrequencyDBImpl*): added
	  sweepOutJunk method for use by cleanup function.

	* spamprobe.cc (cleanup_database): Added cleanup command to do a
	  mark and sweep database cleanup.

	* WordData.h (class WordData): Promoted WordData to its own
	  class so that the cleanup function could be implemented.

2002-10-06  Brian Burton  <brian@burton-computer.com>

	* MimeMessageReader.cc (getMD5Digest): Replaced sprintf call with
	  hex_digit() util.cc function call.

	* spamprobe.cc (import_words): import and export now use the
	  encode_string and decode_string functions from util.cc to
	  properly handle non-printable characters.
	  (main): Added -X option to rely almost exclusively on terms with
	  distance from mean >= 0.4 and allow word repeats of 5
	  (equivalent to -w 5 -r 5 -x)

2002-10-02  Brian Burton  <brian@burton-computer.com>

	* SpamFilter.cc (computeRatio): Fixed bug that returned word score
	  of 0.5 for messages which had 0 in one count.  Only happened if
	  corresponding message count was also zero.

	* FrequencyDB.cc: Removed uses of message id as database key.

	* spamprobe.cc (dump_words): spamprobe dump now prints word
	  probabilities in addition to counts.
	  (main): Added -x command line option to allow top terms array to
	  extend past size limit if there are more significant terms than
	  can fit.

2002-09-20  Brian Burton  <brian@burton-computer.com>

	* SpamFilter.cc (scoreMessage): Relaxed the maximum top terms
	  array size limit to allow more terms to be used if their
	  distance from the mean is at least 0.4.  This way emails with
	  many good and spam words get a more accurate evaluation since
	  the good terms don't squeeze all of the spammy words out.  Seems
	  to yield a slight improvement on new, difficult spams without
	  increasing false positives.

2002-09-19  Brian Burton  <brian@burton-computer.com>

	* NewPtr.h (class NewPtr): Added NewPtr class to use in place of
	  auto_ptr.  I'd rather follow the standard but some older
	  versions of g++ came with a broken auto_ptr.

	* FrequencyDBImpl_bdb.cc (open): Added #if condition to handle the
	  gratuitous api change made by SleepyCat to the open() function.

2002-09-17  Brian Burton  <brian@burton-computer.com>

	* Released 0.7b with better mbox support, domain name break down,
	  and md5 digests for message identification.

	* spamprobe.cc (import_words): Changed constructor arguments as
	  suggested by Xavier Nodet to work around problem with MSVC.
	  (set_headers): Added -H none command line option to ignore
	  headers when scoring a message.

	* FrequencyDB.cc and lots of other files: Added MD5 digest as
	  unique identifier for emails instead of using message-id.  For
	  now message-id is still used if digest not found but eventually
	  will remove it since digest is better identifier anyway.

2002-09-16  Brian Burton  <brian@burton-computer.com>

	* SpamFilter.cc (scoreMessage): Added code to put top tokens into
	  the Message object while scoring.

	* Message.h (class Message): Added code to store and retrieve top
	  tokens.

	* spamprobe.cc (main): Added -T command line option to print
	  top terms and their score and message count.

	* Tokenizer.cc (is_special_char): Removed ' from special chars
	  since it seemed to hurt accuracy to include it.

2002-09-15  Brian Burton  <brian@burton-computer.com>

	* MimeMessageReader.cc (readText): Content type is now returned
	  with each text block so that it can be added to the token list.

2002-09-13  Brian Burton  <brian@burton-computer.com>

	* MimeHeader.cc (isFromLine): Improved mbox reading logic by
	  incorporating the From line format specification as defined in
	  the qmail mbox man page.  Tried to be a little flexible for
	  flawed variations but still strict enough to not think a
	  sentence starting with From is a new message.

	* MessageFactory.cc (addWordPartsToMessage): Now breaks tokens
	  containing non-alnums into pieces adding each sub-word plus each
	  suffix.  This breaks host names down into their host and domain
	  names.  This seems to improve accurracy.

	* MimeHeader.cc (read): Added extra argument to control whether or
	  not to allow the header to begin with a From_ line.  This fixes
	  a bug causing SP to miss some emails in mboxes if the preceeding
	  email was multipart and did not have a terminator.

	* Released 0.7a with receive mode bug fix, solaris ctype functions
	  bug fix, and better tokenizer.

2002-09-12  Brian Burton  <brian@burton-computer.com>

	* Tokenizer.h (class Tokenizer): Changed tokenizing of text to
	  involve less copying.

	* util.h: Added ctype front-end functions to work around problems
	  on solaris with non-ascii chars.

	* MimeLineReader.cc (readLine): Rewrote loop to make it handle
	  lines terminated by only CR as well as CR or CRLF.

2002-09-11  Brian Burton  <brian@burton-computer.com>

	* MessageFactory.cc (addStringToMessage): Fixed bug that dropped 8
	  bit characters when m_replaceNonAsciiChars was false.

	* MimeLineReader.cc (readLine): Converts null bytes into spaces.

	* spamprobe.cc (import_words): Added import command to import
	  terms previously saved using export command.

2002-09-10  Brian Burton  <brian@burton-computer.com>

	* util.cc (is_all_digits): Added is_all_digits.

	* MessageFactory.cc (addWordToMessage): Fixed all digits token
	  removal so that IP addresses are added as tokens.

	* MimeMessageReader.cc (readToBoundary): When reading messages
	  from mboxes now honor content-length fields in the headers
	  unless -Y option was specified.

	* FrequencyDBImpl_cache.h (class FrequencyDBImpl_cache): Added
	  is_dirty flag to cache entries so that values that haven't
	  changed don't get written to the database.

	* spamprobe.cc (main): Added -S command line option to allow
	  messages per cache flush to be controlled from command line.
	  Some other code cleanup as well.

	* FrequencyDBImpl_cache.h (class FrequencyDBImpl_cache): Added a
	  caching proxy frequency db impl class that uses an STL map to
	  cache term counts to reduce disk i/o at the expense of more cpu
	  time and memory usage.

	* FrequencyDBImpl.h (class FrequencyDBImpl): Added an abstract
	  base class for frequency db impls so that I could have a caching
	  proxy.

	* SpamFilter.cc (token_qsort_criterion): Fixed incorrect sort
	  order that put spammy words ahead of good words in the tie
	  breaker.  Also imposed a limit on the term count when sorting
	  since counts above a certain number become basically identical.

	* FrequencyDB.h (class FrequencyDB): Modified to use an
	  implementation class for all database access.  This will make it
	  easier to plug in new ones later.

	* FrequencyDBImpl_bdb.h (class FrequencyDBImpl): Added berkeley db
	  based implementation class to isolate the rest of the code from
	  the choice of database.  This version uses btree files instead
	  of hash for better performance, smaller file sizes, and sorted
	  output during traversals.

	* FrequencyDBImpl_dbm.h (class FrequencyDBImpl): Added dbm based
	  implementation class to isolate the rest of the code from the
	  choice of database.

	* MessageFactory.cc (addWordToMessage): Fixed bug that allowed all
	  digit tokens to slip in.

2002-09-07  Brian Burton  <brian@burton-computer.com>

	* contrib/README-maildrop.txt: Added Matthias Andree's maildrop
	  howto to the contrib directory.

	* MimeLineReader.cc (readLine): Fixed to properly handle null
	  bytes in lines.  Not that those are valid but bugged mailer
	  sometimes embed them.

2002-09-06  Brian Burton  <brian@burton-computer.com>

	* spamprobe.cc: added two new commands: dump and export.

	* SpamFilter.cc: Fixed a memory leak in scoreToken().  Converted
	  to use only a single FrequencyDB now.  Added an accessor to
	  allow clients to get access to the db.  Changed comparisons to
	  zero to allow for inexact floating point differences.

	* FrequencyDB.cc: FrequencyDB modified to store both spam and good
	  word counts for each word in a single dbm file.  Added a pair of
	  traveral functions for the export command.

	* util.h: Moved iostream inclusion into util.h.  Also added cctype
	  include there at Matthias Andree's suggestion for better gcc 3
	  compatibility.

2002-09-05  Brian Burton  <brian@burton-computer.com>

	* FrequencyDB.h: Added hooks for switching to berkeley db in ndbm
	  compatibility mode.  GDBM does not scale well for large
	  databases.  Will continue to use GDBM until 0.7 but will switch
	  over at that release.

	* util.h: Added using namespace std to avoid problems on modern
	  C++ compilers.  Thanks to Matthias Andree for bug report.

	* README.txt: Put in fix to procmail recipe.  Thanks to Steven
 	  Grimm for bug report.

	* FrequencyDB.cc (removeMessage): Will not attempt to remove a
	  message which has no message id.
	  (addMessage): Will not attempt to add a message which has no
   	  message id.

	* MessageFactory.cc (initMessage): Fixed bug 604808 which caused
	  messages with no message id to not have their bodies read.
	  Thanks to Steven Grimm for bug report.

	* MimeMessageReader.cc (readText): Fixed potential bug which could
	  incorrectly skip part of message body for non-multipart
	  messages.

	* MessageFactory.cc (addWordToMessage): Allows leading $ in
	  tokens.  Ignores tokens consisting entirely of digits.

2002-09-03  Brian Burton  <brian@burton-computer.com>

	* spamprobe.cc (set_headers): Removed obsolete test cases.
	(process_stream): Added find-spam and find-good commands.

2002-09-02  Brian Burton  <brian@burton-computer.com>

	* Changed the use to string::find() == 0 to use a new inline
	  function that used strncmp() for greater efficiency.

	* SpamFilter.cc (SpamFilter): Changed default scoring params to
	  use top 27 words and max of 2 repeats.  Found this to be a good
	  option based on test runs with sample corpus.

	* spamprobe.cc (set_headers): Added -H command line option to
	  control which headers are parsed to find tokens.

2002-08-30  Brian Burton  <brian@burton-computer.com>

	* spamprobe.cc (main): Added -h command line option to retain html
	  tags when generating tokens.

	* MessageFactory.cc (expandEntitiesInHtml): When not removing html
	  we still expand any entities in the html.

	* SpamFilter.cc (token_qsort_criterion): Modified token sort
	  criteria to favor good words over spammy ones if their distance
	  from mean and counts are equal.

	* spamprobe.cc (process_test_case): Removed tune_1 test case.

	* MessageFactory.cc (MessageFactory): Changed default settings for
	  better spam detection.

	* SpamFilter.cc (SpamFilter): Changed default settings for better
	  spam detection.

	* MessageFactory.cc (initMessage): Scoring additional headers.
	  Scoring subject header twice for extra emphasis.

	* MessageFactory.h: Made the scoring parameters setting member
	  variables.

	* SpamFilter.h: Made the scoring parameters setting member variables.

	* MimeMessageReader.cc (readText): skips junk at end of message in
	  mime multipart messages.  Previously became confused by the
	  extra junk and generated spurious scores for some messages which
	  threw off accuracy.

	* Misc: Added scripts for testing accuracy of results.

