Skip to content

mail-server: Clear up Bogofilter logic

Clear up bogofilter training:

  • Remove -u flag causing classification to reinforce training
  • Record whether and how each email was used to train bogofilter
  • Fix training in case of repeated moves in-and-out of Junk
  • Reinforce spam training on incoming mail scored as spam (>99%)

Also fix:

  • Fix moves from Junk to Trash folder causing training as ham
  • Fix classification when the bogofilter database is still empty
  • Fix classification when Junk does not exist

The -u flag was unintended; with it, bogofilter training was silently reinforced from incoming mail based on the score given by bogofilter:

  • below the ham threshold (<45%): train as ham ham
  • above the spam threshold (>99%): train as spam
  • in the middle "unsure" zone: do not use for training

This training caused classification errors to reinforce over time. It helps refine and keep accuracy over time when previous training leads to accurate classification, but can drift if left unattended. It also made it hard to track whether and how each incoming mail was used for training.

Tracking training usage is important to take the correct actions on training triggered by manual user action:

  • Moving an email to Junk triggers spam training
  • Moving an email out of Junk triggers ham training

When an email was already used as training in the opposite category, that training must be undone first.

The training usage is recorded for each email with IMAP flags: $BogofilterSpam, $BogofilterHam, or neither.

Incoming email scoring as spam (>99%) is now still used to reinforce training automatically, but not email scoring as ham. This now comes with proper tracking, and can be done more confidently thanks to the high spam threshold and the likelihood of users fetching incorrectly classified emails back out of Junk.

Edited by Xavier Thompson

Merge request reports