Spam Classifier Adversarial Robustness

Adversarial machine learning studies how models can be deceived by inputs deliberately crafted to fool them. In NLP, that usually means perturbing text with word- or character-level tricks that a human reader barely notices but that flip a model's prediction. Adversarial training, in turn, is the practice of explicitly training on those adversarial examples to make a model harder to fool. We wanted to measure both halves of that story on a task everyone recognizes: spam email detection. First we'd see how badly a range of models could be fooled, then we'd see how much adversarial training actually helped.

We built on the Enron email dataset, a collection of emails from Enron Energy Company obtained by the Federal Energy Regulatory Commission during their investigation. A cleaned version we found online turned out to contain only 6 unique spam examples, so we went back to the original 2006 raw release hosted by the Athens University of Economics and Business and cleaned it ourselves: downloading and extracting the spam and ham archives (SSL had to be disabled to reach the source), parsing each SMTP message for its subject and body, and extracting text with Beautiful Soup 4 wherever an email's body was HTML. After removing duplicates and NA values, the final dataset held 18,429 ham (good) emails and 17,466 spam emails, each with a subject, body, and binary label.

To generate attacks we integrated our dataset with OpenAttack, a research-oriented adversarial attack toolkit that took some work to get running, including disabling SSL again and falling back to a deprecated Python and NLTK version for a handful of the attackers. Once wired up, our script took a labeled email and tried to perturb it just enough to flip the classifier's prediction, ideally from spam to ham or vice versa, while staying close to the original text.

Some attackers took over 30 minutes to generate a single adversarial example on our longer emails, which wasn't viable at full scale. So to compare attackers fairly, we filtered down to emails with below-median subject and content length (30 words and 949 characters respectively), cutting the dataset by about 65% to 12,222 entries. From an 80/20 train/test split of that smaller set we sampled 50 emails, combined each one's subject and body into a single string, and computed its TF-IDF representation for three vanilla classifiers. A successful attack was any perturbation that caused a model to misclassify the email.

OpenAttack offers 14 attackers; we ran 10 of them against three vanilla models (Multinomial Naive Bayes, SVM, Random Forest) and two complex ones (LSTM, BERT). We skipped four: SCPN had a source code error, BERT and Genetic were too slow to run at our scale, and FD needs an embedding layer none of our vanilla models have.

GAN, a black-box attacker that generates adversarial text through an encoder-decoder architecture, was the most consistent performer, always clearing a 50% success rate and running fast. Its main weakness was output quality: the perturbed text was often unintelligible, and against BERT it sometimes degenerated into near-nonsense like "and and to to to to a kind of , of that the film of of , be to the , , , the 're n't more of , of the" while still flipping the prediction. TextBugger and PWWS, which both read the model's output scores directly rather than treating it as a black box, performed almost as well on most models but fell off sharply against Random Forest. BAE was the clear worst performer everywhere, likely because its attack strategy leans on BERT embeddings that none of the vanilla models expose, and it was also by far the slowest, averaging over 40 minutes per example against Naive Bayes and SVM.

With TextBugger established as the strongest practical attacker against simple models, we used it to adversarially train a Multinomial Naive Bayes classifier. We built a test set from examples TextBugger could successfully flip against the pre-trained model, then after every epoch fed the model whatever adversarial examples TextBugger could still generate from the training set and measured accuracy against that fixed test set. Because each adversarial example has to be generated against the model's current weights, the process can't be parallelized. Even on a reduced subset (5,000 training points, 1,000 test points) nine epochs took roughly a week to run.

The pre-trained model scores 0% on the test set by construction, since it's built from examples the pre-trained model already got wrong. One epoch of adversarial training was enough to push accuracy to 90.4%, and it kept climbing to 94.7% by epoch nine. TextBugger's success rate against the training set fell in step, from 53.3% down to 29.8%, meaning the model was correctly defending against a majority of the perturbations it used to fall for. Adversarial training isn't free of tradeoffs, though: it can leave a model overfit to defending specific perturbations rather than to the features that separate clean spam from clean ham. Scaling this past our 6,000-point subset to the full ~35,000 emails, and to more of OpenAttack's other attackers, is the natural next step.

The write-up below covers everything in more depth: full success-rate and runtime tables for every attacker and model, and the future work we didn't get to. If it doesn't load, you can also open it directly.