Adversarial examples

Adversarial Examples are inputs to a software system which cause it to make mistakes. These mistakes can be caused by carefully crafted inputs, generally intended to exploit the system’s vulnerability or a specific class of vulnerability. The purpose of adversarial examples is to identify potential weaknesses in a system, or to demonstrate to a user or developer that such weaknesses may exist.

Adversarial examples are usually crafted by selecting input values very close to the system’s decision boundary, with the intent of circumventing a classifier’s decision logic. The target software system might be a computer vision algorithm, or a machine learning algorithm. Adversarial examples might also be crafted by looking for unusual inputs that deviate from the system’s expected input distribution.

Adversarial examples are of interest in the field of cybersecurity because they are a potential way to bypass standard detection or classification systems. For example, an attacker might present an email containing malicious code, but that code is disguised to look like normal content to an automated signature-based detection system.

Adversarial examples can also be used by researchers to identify potential problems in a machine learning classifier. By presenting a series of crafted inputs, researchers can determine whether a classifier behaves as expected.

Adversarial examples are also seen as a potential tool in adversarial machine learning – the use of machine learning techniques to adversarially attack a machine learning system or to deepen understanding of a system (or dataset) in order to strengthen it.

Adversarial examples have garnered attention and interest largely due to their potential implications in cybersecurity, data privacy, and information security. They are also seen as one possible avenue in the development of secure machine learning systems, helping to show that such systems remain robust against malicious inputs.