Microsoft Defender for Office 365: Attack simulation training

With Microsoft Defender for Office 365, you can create an attack simulation training to identify vulnerable users and mitigate potential threats before they impact your organization.

Think before you click

Understanding the intricacies of cybersecurity is crucial in today’s digital landscape. Attack simulation training is indispensable for users as it provides hands-on experience in recognizing and defending against potential threats. This proactive approach empowers individuals to enhance their security awareness, identify vulnerabilities, and contribute to a more resilient organizational defense against cyber attacks.

Microsoft Defender for Office 365 provides an attack simulation training if you are licensed for Microsoft Defender for Office 365 Plan 2 (add-on licenses or included in subscriptions such as Microsoft 365 E5). Without the need for third-party phishing simulations, this attack simulation training can be easily set up in the Defender portal.

This blog focuses more on the user part of the attack simulation and is an extension of the Microsoft Learn documentation, which already provides a good explanation of how to set up the attack simulation in the Defender portal. We will definitely have a summary of the configuration part as well.

Requirements

1: Required license

All target users must have a Microsoft Defender for Office 365 Plan 2 (add-on licenses or included in subscriptions such as Microsoft 365 E5).

2: Report message button

You should start by giving your users the ability to report email messages, which is also necessary for this attack simulation training. To do so, you can activate the:

• Built-in Report button (supported versions of Outlook)
• Microsoft Report Message or Report Phishing add-ins (virtually all Outlook platforms)

Reported messages appear in the User Reported section of the Submissions page, your reporting mailbox, and are visible in the simulation report.

3: Required permissions

Required role: Attack Simulation Administrator

Users in this role can handle every facet of attack simulations, including creation, launch, scheduling, and result review. They have full access to all simulations within the tenant.

The role is available in the Microsoft Defender portal or in Entra ID (e.g. through PIM).

4: Remove external tagging for the default end user notifications address

Once the user clicked on the link and/or logged on to the phishing site, they received an email From notification@attacksimulationtraining.com. Exclude this email address or domain from your external tagging configuration (Exchange Online mail flow rules or Exchange Online’s External Email Tagging Feature).

You can also use Tenant notifications to change the From address to an internal address.

5: Turn on auditing

In order for Attack simulation training to have reporting capabilities, auditing needs to be enabled.

1. Connect to Exchange Online PowerShell
2. Enable Organization Customization by running: Enable-OrganizationCustomization
3. Then run the following PowerShell command to turn on auditing: Set-AdminAuditLogConfig -UnifiedAuditLogIngestionEnabled $true

Creating an attack simulation training

After the requirements are set, you can begin creating an attack simulation training in the Microsoft Defender Portal. You have the option to:

• Simulate a phishing attack
  • These simulations test your security policies and practices, as well as train your employees to increase their awareness and decrease their susceptibility to attacks.
• Using automated flows for Attack simulation
  • Creating a simulation automation is similar to creating an individual simulation, except for the ability to select multiple techniques, payloads, and the automation schedule.
• Payload automations for Attack simulation training
  • Payload automations, also known as payload harvesting, gather data from real-world phishing attacks reported by your organization’s users.
• Training campaigns for Attack simulation
  • Instead of creating and launching simulated phishing attacks that eventually lead to training, you can create and assign Training campaigns directly to users.

Personal Insight:

While automation can be a good fit for your organization, there are some considerations to keep in mind. For example, if you plan to automate a year-long simulation (which is also the maximum for an automation schedule), you won’t be able to edit the content after the initial setup, meaning you’ll be restricted to the choices made during configuration.

During setup, you have two options: you can manually select up to 20 payloads (both global and tenant-specific), or choose the randomize option, where Microsoft will randomly select the payloads for the simulation. When configuring the automation schedule, you can opt for a randomized schedule, which will start simulations randomly within your chosen days of the week, along with random send times. However, you can’t limit it to just one simulation per month—only to specific days of the week when simulations are allowed to start. Alternatively, with a fixed schedule, you can choose a weekly or monthly schedule, but the recurrence can only be set for a static day of the week or month, and you can’t set the send time. Each automated simulation will land in the Simulations tab with a naming convention such as: AutomatedSimulation_PayloadName [Technique]_date

My advice: Consider creating individual monthly simulations (e.g., 2 global payloads and 1 tenant payload as the interval) and plan a yearly schedule that varies the simulation dates and times each month. Keep in mind that random send times aren’t available with this method, but using individual monthly simulations will provide you with more control and flexibility.

The basic elements of a simulation are:

• Select a Social Engineering Technique, such as credential harvesting
• Select a Payload (phishing emails and web pages that you use to launch simulations)
  • Global Payloads: Includes built-in payloads, such as the Keep Office 365 Password payload
  • Tenant Payloads: Contains custom payloads, such as a fake email from an executive with an official company signature. You can also use ChatGPT (or any other AI tool) to prompt a tenant payload to generate an HTML email for SharePoint Online document sharing, for example. A prompt can be: Can you generate an HTML email template that looks like a Microsoft email to share a SharePoint document? This prompt leads to this tenant payload, the HTML content can be copied and pasted into the Configure Payload section and change the Dynamic tags such as ${firstName} and ${phishingUrl}.

Images that you use in tenant payloads may be blocked with a message that the sender is not in the Outlook Safe Senders list. This happens by default because Outlook is configured to block automatic image downloads in messages from the Internet.

• Select a Login Page (phish web login page for credential harvesting and link in attachment techniques)

  • Global Login Pages: Includes built-in login pages, such as the Microsoft login page
  • Tenant Login Pages: Includes custom login pages, such as a custom Microsoft login page with corporate branding

• Select a Phish Landing Page (provides a learning moment for the user after being phished)

  • Global Phish Landing Pages: Includes built-in phish landing pages
  • Tenant Phish Landing Pages: Includes custom landing pages, such as with corporate branding

• Select End user notifications

  • Global notifications: Includes built-in end user notifications send From notification@attacksimulationtraining.com
  • Tenant notifications: Includes custom end user notifications for branding and to set a different From address to an internal mailbox

• Who receives the simulated phishing message and on what schedule

  • All users* or specific users and groups
    • *All users are all mailboxes (user and shared) and resources (room and equipment) in Exchange Online
  • Supported groups: Microsoft 365 (dynamic supported), distribution list (dynamic not supported) and mail-enabled security.

The best practices of a simulation are:

• Target users: Include all users in your organization (Assuming all user mailboxes are licensed for Microsoft Defender for Office Plan 2)
  • If you want to target a specific department, you could import a CSV containing all the members of that department. To do this, run the following command in Graph PowerShell:

Get-MgUser -All -Property "Department,UserPrincipalname" | Where-Object {$_.Department -eq "DepartmentNameHere"} | Select-Object UserPrincipalname | Export-CSV -Path <PATH> -NoTypeInformation

• Exclude users: Import a CSV file that contains all your shared and room mailboxes (also specify your mail-enabled service accounts in this CSV file)
  • To export these RecipientTypes, you can run the following command in ExchangeOnline PowerShell:

Get-Mailbox -RecipientTypeDetails SharedMailbox, RoomMailbox -ResultSize Unlimited | Select-Object PrimarySmtpAddress | Export-CSV <PATH> -NoTypeInformation

Note: You may see that the excluded users end up in the report as FailedToDeliverEmail, this is because the given user is blocked from signing in, such as you shared mailbox identities. This is normal behavior and you can filter them out in the report.

• Training: Select the Microsoft training experience and let Microsoft assign training courses and modules based on a user’s previous simulation and training results learning paths.

Progress of the attack simulation

We will take a deep dive into a Credential Harvest simulation, one of the several social engineering techniques to choose from. Create the Credential Harvest simulation using the steps provided by Microsoft to simulate a phishing attack and select the global payload: Keep Office 365 Password. Upon completion, you should have a simulation in progress.

The attack simulation then begins with users receiving credential phishing emails with the selected payload.

Payload:

A user can click on the link, which creates an outbound connection to an adversary-in-the-middle (AiTM) phishing site (the connection does not raise an alert in Defender).

Login Page:

If the user logs in, they will land on the phish landing page that provides a learning moment to the user after getting phished.

Phish Landing Page:

Once the user clicked on the link and logged in, they received an email from notification@attacksimulationtraining.com for each action to complete a training course.

NOTE: You should remove the External tag for the notification email. To do so, run the cmdlet Set-ExternalInOutlook -AllowList @{Add="attacksimulationtraining.com"} by using the Exchange Online PowerShell module. You can also use Tenant notifcations to change the content or from address.

The link will take the user to the Defender portal to complete the courses at https://security.microsoft.com/trainingassignments.

The simulation report allows you to analyze how your users performed in the attack simulation.

Conlusion

Despite advanced security measures, phishing tactics continue to evolve, making it difficult to catch every attempt. Thereby user awareness is important because users play a critical role in identifying and avoiding potential threats.

Reference

• Attack simulation training documentation
• Attack simulation training deployment considerations and FAQ
• Turn on auditing