What is a false positive and what is a false negative?
Do you like to cook? If so, your fire alarm has probably sounded at some point, even though you may have only generated a little too much steam in the kitchen. As a result, you have not only created a hopefully tasty meal, but also a false positive. This blog article explains what false positives and false negatives mean for your email security.
A false positive is the result of a test with mostly binary classification, in which a match (positive) with previously defined criteria is determined falsely (false). The criterion of smoke development was met in the above example, but in a controlled environment and not as a symptom of an impending fire. So the fire extinguisher stays in the cupboard, the table has to be set. In this case, a false negative would be worse, i.e. a fire alarm that remains mute even though there is a fire in the kitchen.
There are many examples of false positives and false negatives, from medical misdiagnoses to motion detectors activated by cats to incorrect corona or pregnancy tests.
False positives and false negatives in email
When we talk about email security, we know a false positive email as an email that has been wrongly identified as spam, a phishing attempt or infected with malware. Accordingly, a false negative email is an email that has not been recognised as such, even though it is.
What is a False Positive?
A false positive is an error in binary classification where a test result falsely indicates the presence of a state when it is not present.
What is a False Negative?
A false negative is an error in binary classification where the test result falsely indicates the absence of a state even though it is present.
Welcome to the truth matrix
As already mentioned, false positives and false negatives mostly occur in tests with binary classification. The classifier is the criterion that is used to classify the object in question.
How well a classifier works can be seen by how often misclassifications occur. Binary classifications can also be formulated as a yes-no question: Is the kitchen on fire? Is the patient sick? Is this email spam or a newsletter that was previously subscribed to?
Accordingly, there are two possible misclassifications in spam detection, for example:
Of course, there are also two possible correct classifications:
Causes and consequences of false negatives and false positives
False positives and false negatives exist in all areas of life, from statistical studies to laboratory tests to interpersonal communication, e.g. does silence really mean agreement or is it just a character trait?
For this reason, there are also a wide variety of causes for misclassification: Technical defects in the analyzing device, test cases that have not been adapted, an incorrect comparison period and many others.
Filters that are responsible for classifying emails in email security software can also be responsible for false negatives or false positives. This is the case, for example, when emails that are actually harmless are classified as dangerous by one of your partners or when a contaminated email was delivered by a partner.
In the case of false positives, this can mean an interruption in communication with your partner; in the case of a false negative, it can mean, in the worst case, an infection of your IT infrastructure with all the other consequences such as data loss, blackmail with the help of ransomware and, in any case, financial damage.
Quarantine as a risk factor
With email security products that include a quarantine function, another risk arises. For example, an email containing an important invoice is assessed as spam and moved to the quarantine. It is now possible that this email is overlooked – and if the quarantine folder is automatically emptied after 30 days, the email is lost. This is followed by the reminder, which probably also ends up in the quarantine.
NoSpamProxy does not have a quarantine, but checks every email as soon as it is received and classifies it using different spam filters. If an email is classified as spam or potentially dangerous, NoSpamProxy does not accept this email. Only messages rated as trustworthy are delivered while all others are rejected and the sender is informed about the prevented delivery.
What to do if there’s a fire but no fire?
NoSpamProxy customers do not have to worry about misclassified emails: Virus Bulletin regularly confirms NoSpamProxy’s stellar spam detection rate of at least 99.5 % and a false positive rate of 0 % by awarding it the VBSpam+ award.
If a misclassification by NoSpamProxy has occurred, we recommend the following procedures:
False Negatives in NoSpamProxy
If you have received a contaminated email from one of your communication partners, we recommend that you block the affected email addresses. To ensure that we are informed about the incident, you should report the misclassification to spamreport@nospamproxy.de, an important contribution to product improvement. To do this, you should create a rule to block the email address (MAIL FROM, not Header-From!). You can find out how to create a rule-based blocklist in the NoSpamProxy documentation.
Of course, it is problematic to block communication with a partner even if this serves the security of your company. In the event that communication with the compromised partner is to be maintained, we have some recommendations for you.
False positives in NoSpamProxy
If a filter, for example an anti-spam filter, finds certain indicators in an email that trigger a misclassification, you can set the trust in the affected partner domain to 40 with the help of the Level of Trust. Emails will then be delivered in any case.
You can find further information on dealing with false positives and false negatives in the NoSpamProxy documentation.
A general recommendation
Some users try to prevent the detection of false positives by adjusting existing rules based on the given situation. For example, it has repeatedly happened that the threshold or the multiplier of a filter in the standard rule All other inbound emails was changed or that individual Realtime Blocklists (RBLs) in the filter were removed for each incident – with the consequence that this filter became increasingly ineffective over time.
Often, shortly after raising the threshold, another problem comes to light: emails that were previously recognised as spam and usually rated with 4 SCL points (Spam Confidence Level) are now delivered. In many cases, users now attempt to restore the originally good spam detection with the help of the multipliers after raising the threshold.
We strongly advise against changing the above rule. Instead, proceed as follows if you want to make adjustments:
- Duplicate the All other inbound emails rule.
- Restrict the rule to the affected sender domain of the email address.
- Adjust the index.
- Make the desired changes in the newly created rule. You can clarify which changes are necessary by analysing the respective message track. An example would be the adjustment of the reputation filter.
Not yet using NoSpamProxy?
With NoSpamProxy you can reliably protect your company from cyber attacks. Request your free trial version now!
