Phishing scenario metrics for awareness and Red Team assessments

tldr; I have an opinion about the effectiveness of phishing attacks to measure the state of security awareness within an organization. Measurability is great, but there are quite a few ifs and buts. By creating a tool, ethical hackers can make phishing emails more realistic and measurable.

When you think of phishing campaigns or security awareness programs, you might think of the obvious emails that appear in your emailbox once in a while. These emails were necessary to see whether everyone is still alert on phishing risks and can distinguish a phishing email from a legitimate email. The number of clicks on the link displayed in the emails, how often credentials have been entered or whether there may be recipients who have downloaded and exported a file from the attachment, is tracked at a time. Nice for the report to show that the organization is really doing everything to prevent you from being compromised by an attacker through this well-known way of social engineering. These figures are beautifully displayed in a report including colorful pie charts with an increasingly lower click rate and the organization has again met the compliance requirements. On to the next assessment!

Because that’s the goal of security awareness testing right? Increase the awareness of the employees and thereby reduce the click rates? Because the fewer people click on a link in a phishing email and enter data or execute malware, the better the prevention worked out against these types of attacks right? Of course: if nobody fills in his or her credentials on a page or executes that macro from a Word document on his or her system at all, you will avoid a lot of trouble. But fair; of course you cannot prevent it completely. In an earlier post (which is Dutch, but the paper is in English) was stated what the three main reasons are why someone interacts with a phishing email or website1: 1. because the person has insufficient knowledge about what a phishing email is, 2. because the phishing email looks so legitimate that the person cannot distinguish it from an actual legitimate email or 3. because they are not paying attention at the time of receiving the email. And the latter makes it difficult. Maybe you haven’t had your double espresso in the morning yet, you have been in long Teams meetings all day or you want to quickly clear your email before closing your laptop for the weekend: you are just not there with your head for a while and you click. And you enter your credentials. Or you download something. That can happen because that’s just human behavior. So using security awareness to completely prevent someone from clicking or filling in credentials or downloading malware is simply impossible. Fortunately, there are other preventive measures for this, such as proper separation of the network, separation of user roles and a bunch of detection mechanisms and a good response process.

The attentive reader saw, however, that can actually be improved to decrease click rates. And with the clickrate hopefully also the number of credentials entered and malware executed, because blindly staring at ‘the number of clicks on a phishing link’ may say something about awareness, but less about the risk that the organization runs: with only one click as an attacker, unless it contains an exotic browser attack (or that one that was targeted at security researchers not so long ago) for example. But under the guise of probability calculation and ‘return on investment’ you can leave that out in this context.

Employees don’t basic knowledge about phishing emails

Multiple security threat landscape reports, such as the Dutch one for 2020 state that organizations are still not sufficiently resistant to phishing, but that basic measures can limit the damage. If you want to make phishing less effective, an awareness campaign can ensure that these emails are detected faster and, above all, reported by employees. In practice you often see that during awareness moments (or interventions) certain characteristics about phishing emails are imparted to employees, such as checking out the sender address, inspecting the link (sometimes you even have to pay attention to whether there is a lock in front of it: really!) or the attachment and checking the spelling errors in the email. This indeed ensures that some (simple examples) of the phishing emails are detected.

Employees are not able to distinguish a legit looking phishing email from a real legit email

But attackers don’t sit still either. Since a large proportion of organizations provide proper information about the dangers of phishing and how to recognize these emails, attackers are busy making emails look as legitimate as possible, so that they pass this trained human detection. Emails don’t contain spelling mistakes anymore, addresses of the sender are very nicely typosquatted, are spoofed, sound legitimate within the context or are previously used expired domains, contain the correct corporate identity and possibly personal information. This information was obtained from public password breaches or simply from your public social media. In the latter cases, it might involve spear phishing, but a simple web scraper is also set up in such a way to personal information about you. If this attacker uses a credible sender, error-free text with any personal information of the victim and a copy of the corporate identity, the employee’s basic knowledge will not be sufficient to single out this email as a phishing email. And in that case you could also look at such an email differently.

Phishing emails that are sent all have one goal: they want something from you. Attackers want you to click a link and enter your credentials on a legitimate looking website, download a file from the website, or open the attachment in the email. With the obtained login data, these attackers will try to enter the internal network (via remote desktop or VPN solutions), cloud services such as Office 365 or try to gain access to your system by using malware. And in order to get what you want, you use a bit of psychology. Cialdini and Gragg2 have both researched which approaches work best in a phishing email and read as follows:

  1. The principle of commitment, reciprocation and consistency - Within this principle creating and maintaining a good relationship between the victim and the attacker is paramount. When Alice asks Bob to do something for her, Bob will do it (reciprocation) to maintain the right contact or relationship (commitment). Then Alice will do something in return for Bob in the future, because he helped her in the first instance (consistency). In phishing emails or on websites, you see this reflected in the form of coupon codes for example. You just made a large purchase at webshop X last week, so as a good customer you will receive a code for a discount on your next order. Since your purchase was a good experience last time, that coupon code gives you that last push you needed to order something new. Web store employees surely are smart people.

  2. The principle of social proof - This principle is all about peer pressure: if all your friends buy a new iPhone, big chance you also will. But it goes further than just friends and especially Booking.com is a rock star in doing this; it says on the website that the hotel you are looking at has just been booked by 4 other people from the same country, so you may also be tempted to book this hotel. Websites use locations from profiles and your location obtained from your IP address, among other things.

  3. The principle of liking, similarity and deception - People are quick to trust people they consider themselves to resemble; you soon trust someone who acts the same as you because you think it is the right thing to do. If Alice sends a message to Bob and they have the same background, the same pets or come from the same region, it creates a bond. If you use this information as an attacker, a victim will believe you more quickly. An example is sending an invitation to a conference: if you see that the sender works in the same sector in the same neighborhood, you will open that attachment in your email faster than if it is less relevant.

  4. The principle of authority - From childhood you are told to obey rules and obey authority such as the police. A victim is therefore more likely to respond to a message that appears to come from the police or the CEO of the company. In interventions, emphasize that the authorities and official governmental organisations never send you an email with a request for data or a payment.

  5. The principle of distraction - You soon notice when you have to multitask: you are just a lot better at finishing your work if you can focus on it. However, when you have a lot on your plate that you must complete at the same time, you are much more susceptible to techniques that attackers use to trick you. If there is something in an email or website such as ‘Confirm your purchase within a minute to take advantage of this discount’, you will have to take immediate action and you will be less critical of the email or website and what you actually want was doing.

This list actually lacks an important principle: the principle of curiosity. People are curious from nature. If the attacker sends an email to the victim that stimulates that curiosity, the victim can or will fall for it. Examples are an email reminding you that you have not yet paid for a product, a message from a person on a dating site or the launch of a new intranet.

In addition to telling employees about the basic characteristics of a phishing email, it is extremely useful to also tell something about the use of social engineering and how attackers use the mentioned principles to get things done. Consider carefully what feeling is evoked when you open an email. Do they offer you something for free? Is it an email saying you have been fined in an unknown place? Should you act right now? Or are they trying to take advantage of your goodness (helpfulness)? Be careful; ask someone else for help, browse to the website of the organization or webshop (by entering the URL yourself) or if necessary pick up the phone and contact the so-called sender (but of course not on the number in the email!). Set up two-factor authentication: if you have entered your credentials, it will be useless without second factor. And of course report the email. Do not make it too difficult for your employee: presenting a flow chart of what to do when you have received a phishing email, can even be done better by the Dutch Tax Authorities (their slogan: we cannot make it more fun, but we can make it easier). Options are reporting plugins in Outlook or an email address to which specifically phishing emails can be forwarded that is easy to remember.

Test this awareness in a reliable way and make it measurable over a longer period of time

Send an email with tips and tricks to all employees, organize a classroom training, show a demonstration where you show how easy it is to hack someone or let everyone do an e-learning. Or in short: increase the awareness of employees by intervening. Such an awareness session is good to get everyone on their toes, but must be repeated regularly to keep everyone alert. Research from German universities shows that intervention through interactive or video methods is the most effective and that it should be repeated twice a year3. In order to check whether the method of intervention has actually had an effect, there are awareness programs where a phishing email is sent by or from the organization once in a while. You can draw a conclusion about the effectiveness of these interventions over time. What is rarely looked at is what exactly this phishing test measures. Of course; you test the number of clicks, but to what extent does this actually say something about the awareness of the employees? Often, the exact content of all phishing emails within a program is not determined at the outset. Is the first email a scenario where you are asked by a spoofed address to go to an Office 365 login page clone and then download a fact sheet of a page that resembles Sharepoint and in the second email (with some spelling errors) asked by an unknown sender (price@flostracizedent.ru) if you want to download a file to see if you have won a new iPhone, yes of course the second time will show a decrease in clicks! Show the two reports to your CISO and you get a pat on the back because you managed to get the click rate down: awareness mission successful4.

A complex problem has been outlined, but what is a possible solution? By quantifying a phishing email by means of a model, it can be compared with each other. First of all, it is important to know how a (phishing) email can be recognized. Existing research has already made a model for sites5, but in this case it should be expanded with extra characteristics. Ultimately, an email consists of the context (who, what, where, how is it sent), content (the text, links, images), the domain name, the attachments and the use of security and confidentiality. Within these characteristics, choices could be made, these are based on analysis of a large dataset of phishing emails and advice from the Dutch Digital Trust Center and Safe Internet Platform. The result is presented as the model below (click on the image, is not phishing!):

You can subsequently apply this model to existing phishing emails. For example, look at the two examples below and compare them based on content. Both emails don’t contain crazy characters or typos and are in Dutch (free Dutch lesson!). The Netflix email uses a copied corporate identity (same logo and colors), does not contain any personal information and tries to respond to fear of the recipient (because money is being debited). The second email (from non-existent company PowerGym in The Hague), uses a neutral corporate identity, responds to the greed and an important value of the recipient - who is addressed by first name. In this way there is variation within the choices of the content of the email, without making it less or more advanced.

Netflix
Powergym

Result: the Netflix email scores at 84% and the PowerGym variant at 83% when using the model. The context and content parts have been filled in as shown in the image and, in addition, Netflix has assumed a typosquat domain and PowerGym a domain that contains the word PowerGym. Use your wild imagination for the latter because designing phishing emails on your free Saturday evening is a first, creating login pages is going a bit too far :). These 84% and 83% are the result of weightings based on a single dataset, available literature and own experience. The weighting may differ per organization, but the availability of more data sets will also lead to more reliable results. For the time being, with a little tweaking, the first results look really good.

Use of the model during Red Team assessments

Phishing tests are not the only way of testing where phishing attacks are being used: Red Team assignments also use this way of social engineering to obtain initial access to the network. Comparing and measuring Red Team assessments in not fully new. If you run two Red Team assessments over three years, you can compare the techniques used during these two Red Teams. For example, how high is the effort put into initial access / privilege escalation / lateral movement? And has this technique been detected by the Blue Team? Former colleague and brilliant security hero Francisco is one of the ‘founders’ of this project (including more cyber security organizations)to make this easier to measure and to give organizations a better idea of ​​their prevention, detection and response. If you zoom in considerably on phishing within the initial access phase, the tool could help indicate how high the effort was in this phase. A small side note: the tool only helps with the scenario itself. The degree to which the malware that is used is advanced and difficult to detect is not covered by this model. It can help in determining whether it is a low, medium or high effort technique.

Besides being able to be used after completion of the Red Team assessment, the model can also help with the preparation. In Threat Intelligence Based Ethical Red Teaming (TIBER), an attack by an APT (Advanced Persistent Threat) or other larger threat actor is simulated to see how well the organization can defend itself against it. If a specific group emerges from the Threat Intelligence report that is active in the sector of the organization being tested, the emails sent from this actor can be compared with the emails sent within TIBER. With keeping this in mind you have more certainty that the phishing email sent during this Red Team actually corresponds in terms of complexity with that of the threat actor.

Take for example TA505, known for the attack on Maastricht University in the Netherlands. When you grab two emails from this threat actor sent to Maastricht University (obtained from this forensic report) and a COVID-19-mail reported by Trend Micro. At first glance there are many similarities: the email is impersonal, addresses a recent event or current theme and contains a link to a page where a download is offered (a URL where a Microsoft-related service occurs). Both emails therefore come to a percentage of 68% (TA505 amateurs!!!). During a Red Team assessment you could therefore set the difficulty of the phishing email to this, but preferably do this slightly higher. An APT has months (or perhaps years) to attack an organization, which you often don’t have during a Red Team assessment (unfortunately). This allows them to vary with emails until this is successful.

But wait, there’s more than clicks

Some platforms offer insight into more than just clicks, but often only this measurement is used to draw conclusions over a longer period of time. Lower numbers would mean that awareness has increased. But there is more than just clicks or logins. The same number of clicks with two different (equivalent) phishing scenarios can mean that the awareness has remained the same, but what if a doubling has been seen in the second scenario in the number of reports submitted to the service desk? Employees may have clicked to inspect the page, then reported it, and then clicked away. That is not unsafe behavior (as described earlier). Also include this in the measurement of how effective the intervention was, but also consider the number of employees who show insecure behavior every time, how many employees have properly completed the e-learning about phishing or how much is posted on the intranet as a warning for fellow employees. Research organization SANS has defined a metrics matrix6 that helps to measure these points and to process them in the awareness roadmap. Think about what exactly you are going to measure, who is responsible for this, the goal and the target group. Target groups are groups within the organization to which a certain intervention applies. These are defined to create more focused awareness. After all, a recruiter or financial employee of an organization will have to deal more with emails with attachments (because: CVs or invoices) than someone with little customer contact. So consider for each metric to which target group it applies. Educating the entire organization on the basics of phishing plus additional training on Word macros, PDF files, and other file extensions in the attachment for specific groups could be a good approach.

Inspired to make everything measurable? The SANS institute has defined a Security Awareness Maturity Model in which phase 5 shows possibilities for a good measurable security awareness program. This can provide organisations tools for making not only phishing campaigns and their effectiveness measurable, but also other components such as safe use of passwords, safe internet browsing and safe device management. Just make everything measurable; to infinity and beyond!

  1. Why phishing works 

  2. Principles of Persuasion in Social Engineering and Their Use in Phishing 

  3. An investigation of phishing awareness and education over time: When and how to best remind users 

  4. Small nuance: of course programs have different learning objectives. You can also look at the awareness level by offering phishing emails that vary widely in level. This way you can also draw a conclusion to a certain extent. 

  5. Decisive Heuristics to Differentiate Legitimate from Phishing Sites 

  6. SANS Security Awareness Roadmap