Fool a Neural Network
A small neural network is running in your browser and reading the digit on the left. The middle panel shows the direction in pixel space that most increases its error, which is the network's own gradient turned against it. Slide ε up to add a small amount of that noise. The prediction changes even though the image looks the same to you. This is the Fast Gradient Sign Method, and attacks like it are part of why my PhD work was about making models more robust.
Click a confidence bar to aim the attack at that digit. You can also draw on the left image (right-drag erases; on a touchscreen, press and hold, then drag, to erase); the network reads it live and the attack adapts. The adversarially-trained model has seen attacks during training, so it is much harder to fool (its clean accuracy is a little lower in exchange).
FGSM takes one step of size ε along the gradient. PGD takes several smaller steps, re-reading the gradient after each one and never straying more than ε from the original image, so it usually finds a stronger attack for the same budget.
standard adversarially-trained dashed line = current ε
Rows: the model the attack was crafted on. Columns: the model it is tested against. Each cell shows this digit, then the flip rate over the 60 demo digits (untargeted, at the current ε). Attacks crafted against one model often fool the other, which is why black-box attacks work at all. Adversarial training reduces transfer but does not remove it.