Whats HAFNIUM with Exchange On-Premises?

Scenario: Microsoft recently reported 4 new 0-day exploits that may impact Exchange On-Premises servers.

The vulnerabilities being exploited are CVE-2021-26855, CVE-2021-26857, CVE-2021-26858, and CVE-2021-27065

Patch your Exchange On-Premises servers with the latest security patch to protect your environment!!!!

To see if you were impacted or exploited, you can check your log files. Please see the reference article/URL listed at the bottom of this post. My commands used are built off of Microsofts PowerShell commands, except I needed my commands to run on multiple servers, in parallel, and I needed a faster way of evaluating my log files for CVE-2021-26855.

Here are my PowerShell notes that I used:

CVE-2021-26855 – Review the output file for Authenticateduser = $null and AnchorMailbox is like ServerInfo~*/*

$s = “ExOnPremSrv1″,”ExOnPremSrv2”

$S | %{
$Server= $_
$files = get-childitem -recurse -path “\$server\c$\PROGRAM FILES\Microsoft\Exchange Server\V15\Logging\HttpProxy” -Filter ‘*.log’
$FileCount = $Files.Count
$c = 0
$results = @()
$outfile = “\FileServer01\c$\temp\$server.txt”

$files.fullname | %{
    $f = $_
    "$c OF $FILECOUNT :::  $F"
    $temp =  findstr "ServerInfo" "$F"
    $temp | out-file $outfile -Append



$server = “ExOnPremSrv1″,”ExOnPremSrv2”
$server | %{
$s = $_
“Checking $S”
findstr /snip /c:”Download failed and temporary file” “\$s\c$\PROGRAM FILES\Microsoft\Exchange Server\V15\Logging\OABGeneratorLog*.log”


$server = “ExOnPremSrv1″,”ExOnPremSrv2”

$Server | %{
$s = $_
“Checking $S”
Get-EventLog -ComputerName $S -LogName Application -Source “MSExchange Unified Messaging” -EntryType Error | Where-Object { $_.Message -like “System.InvalidCastException” }


$server = “ExOnPremSrv1″,”ExOnPremSrv2”

$server | %{
$s = $_
“Checking $S”
Select-String -Path “\$s\c$\PROGRAM FILES\Microsoft\Exchange Server\V15\Logging\ECP\Server*.log” -Pattern ‘Set-.+VirtualDirectory’

Ref: https://www.microsoft.com/security/blog/2021/03/02/hafnium-targeting-exchange-servers/


Configure your Exchange Online PowerShell Script to leverage OAUTH/Modern Authentication and Authenticate Silently

Scenario: You need to connect to Exchange Online PowerShell via a script that will silently authenticate using Modern Authentication/OAuth.

Microsoft is deprecating Legacy/Basic Authentication when connecting to Exchange Online. Your existing scripts that leverages a username/password to authenticate silently, either by hardcoding a username or password into the script OR using encrypted keys that PowerShell calls in, will break when legacy authentication is officially disabled. You will need to convert the logic us to now start connecting to Exchange Online via Modern Authentication/Oauth.

If you haven’t yet asked ‘How to do we do that’? I am glad I asked for you….

1. You need the latest release of the ExchangeOnlineManagement (Connect-ExchangeOnline) module installed in PowerShell : Install-Module -name ExchangeOnlineManagement.

2. Setup App-only application in Azure.

2a. Register a new application object in Azure Active Directory

2b. Provide the Azure applications the following API permission: Exchange.ManageAsApp

2c. Create a self-signed cert in PowerShell that will be used to authenticate to the Azure App.
# Create certificate
$mycert = New-SelfSignedCertificate -DnsName “whatever.com” -CertStoreLocation “cert:\LocalMachine\My” -NotAfter (Get-Date).AddYears(3) -KeySpec KeyExchange

#Export certificate to .pfx file
$mycert | Export-PfxCertificate -FilePath c:\temp\mycert.pfx -Password $(ConvertTo-SecureString -String “PasswordForCert!” -Force -AsPlainText)

#Export certificate to .cer file
$mycert | Export-Certificate -FilePath c:\temp\mycert.cer

2d. Upload the Certificate in the Certificates & Secrets of the Azure App.

3. Assign the Exchange Administrator role (via Azure Roles or MSOL Roles) to the new Registered App.

4. Now that the app is configured with Exchange Permissions, Exchange Access, and the Certificate uploaded, connect to Exchange Online using the pfx Cert from PowerShell:

Connect-ExchangeOnline -CertificateFilePath “C:\temp\mycert.pfx” -CertificatePassword (ConvertTo-SecureString -String “PasswordForCert!” -AsPlainText -Force) -AppID “<AppID of your new registered app>” -Organization “<your tenant organization name>”

Running PowerShell commands against a large group of mailboxes or returning large data returns

Scenario: While attempting to run Exchange Online PowerShell commands against a large group of mailboxes and/or returning large datasets, you may run into mico delays or other errors once you breach any number of throttles that Microsoft Online imposes against your PowerShell session. Although throttles protect Microsoft Online for good reason, your still left with the question “WHY ME?” and how do we get around these strict restrictions.

Solutions: There’s a few ways to execute commands in Exchange Online PowerShell, but each have their own advantages and disadvantages. Below are a few examples with running the command: get-mobiledevicestastics which is naturally heavy in processing and subject to breaching throttles.

Standard Command – This may work if you only have a small number of mailboxes in your environment. Its quick, but also dangerous.
$AllMailboxes.alias | Get-mobiledevicestatistics -mailbox $_

ForEach with a Mini-Sleep – This command will help with some of the throttling, its not difficult to execute by implementing a 2-command loop and will help with the recharge rate for how many commands you are allowed to execute within a time period.

$AllMailboxes.alias | %{ Get-mobiledevicestatistics -mailbox $_; Start-Sleep -milliseconds 500}
$AllMailboxes.alias | ForEach { Get-mobiledevicestatistics -mailbox $_; Start-Sleep -milliseconds 500}

Invoke-Command to help with Large Data Return – Invoke with Select-Object forces the O365 servers run the command and limits the data returned. This takes your local client out of the loop helping to ease local client throttling. HOWEVER invoke command doesn’t allow complex PowerShell commands either.

Invoke-Command -session (Get-Pssession) -scriptblock {Get-mobiledevicestatistics -mailbox $_ | select-object identity, Device*,LastSuccessSync}
Invoke-Command -session (Get-Pssession) -scriptblock {Get-Mailbox -resultsize unlimited | select-object -property Displayname,Identity,PrimarySMTPAddress}

Microsoft Module: RobustCloudCommand – Microsoft created a module that can be used, originally it used to be a script. The module is designed to stay under the throttling limitations imposed by Microsoft. The module keeps an eye on the amount of commands issued, the amount of time or data collected, throttling errors, and so on. The module will tear down and rebuild the PowerShell session when needed and pickup where the script left off. AND one of the coolest things is that it supports Modern Auth and will renew your OAUTH tokens automatically. Let those big scripts fly! The only issue I see is that its limited in complex PowerShell commands. However if you want to run a single command, such as adding a BlockEverything Authentication Policy for every single one my theoretical 400K users, the workhorse below will do the trick

Install-Module -Name RobustCloudCommand #https://www.powershellgallery.com/packages/RobustCloudCommand

$users = Import-csv C:\temp\AllMailboxes.csv

Start-RobustCloudCommand -recipients $Users -logfile C:\temp\out.log -ScriptBlock {Get-user $input.userprincipalname -AuthenticationPolicy BlockEverything}

Determine when a user can start sending email again after hitting the 10,000 Recipient Rate Limit in Exchange Online

Scenario: Recently we had a user who hit the 10K Recipient Rate Limit in a 24 hour period within Exchange Online. This user was performing two types of emailing from their ExOnline mailbox: 1. Sending mass mailings from their Mailbox using their from address and 2. Sending mass mailings from their Mailbox and setting another From Address (Delegated Send-as Permissions for a Service Account). Both types of sending impacted the recipient rate limit for the persons mailbox.

BTW – Do not mass mail directly to recipients from an ExOnline mailbox. Instead use distribution groups or other third party mailing systems to send mass communications.

Scriptlet: Here is a scriptlet to help determine when the user should be able to send from their mailbox again. The results provide hourly recipient counts for the sender which may help in determine the timeframe when a user should no longer be impacted by the 24 hour period:

Edit the Variables in the scriptlet and paste it into Exchange Online PowerShell.

#Edit these variables
$sender= “SteveTheMassMailer@domain.com”
$start = “1/7/2021”
$end = “1/9/2021”
$pageSize = 1000
$P = 1
$messages = $null
$report = @()
$totalRecipients = 0

#Loop for All Messages
Write-Host “Message Trace – Page $P…”
$temp_Messages = Get-MessageTrace -senderaddress $sender -startdate $start -enddate $end -PageSize $pagesize -Page $P
$Messages += $temp_Messages
}until ($temp_Messages -eq $null)

#Display messages or Message Count

#Build the Loop Parameters
$Loop_start = get-date($messages | Sort Received | Select -expandproperty Received -First 1) -format “MM/dd/yyyy HH:00:00”
$Loop_end = get-date($messages | Sort Received | Select -expandproperty Received -last 1) -format “MM/dd/yyyy HH:00:00”
$1Hour_end = get-date($Loop_start)
$1Hour_end = $1Hour_end.AddHours(1)
$1Hour_end = get-date($1Hour_end) -format “MM/dd/yyyy HH:00:00”

#Now Loop It!
“Starting Loop for $sender”
“Checking $Loop_Start and $1Hour_end”
$RecipientCount = $messages | Where {($_.Received -gt $loop_start) -and ($_.Received -lt $1hour_end)} | Select RecipientAddress
$totalrecipients = $totalRecipients + $recipientCount.count
$recipientCount = $RecipientCount.count

#Report it
$obj = New-Object PSObject
$obj | Add-Member NoteProperty -Name StartTime -Value $Loop_Start
$obj | Add-Member NoteProperty -Name EndTime -Value $1hour_End
$obj | Add-Member NoteProperty -Name RecipientCount -Value $RecipientCount
$Report += $obj

#New Variables
$Loop_Start = $1hour_end
$1Hour_end = get-date($Loop_start)
$1Hour_end = $1Hour_end.AddHours(1)
$1Hour_end = get-date($1Hour_end) -format “MM/dd/yyyy HH:00:00”
}While( (get-date($Loop_start)) -lt (Get-date($Loop_End)))

“-Results are reported in GMT”
“- Total Recipient Count for $Sender between $start – $end : $TotalRecipients”

Microsoft Example of the 24 hour Recipient Rate Limit:
After the recipient rate limit is reached, messages can’t be sent from the mailbox until the number of recipients that were sent messages in the past 24 hours drops below the limit. For example, a user sends an email message to 5000 recipients at 09:00 AM, then sends another message to 2500 recipients at 10:00 AM, and then sends another message to 2500 recipients at 11:00 AM, hitting the limit of 10,000 messages. The user won’t be able to send messages again until 09:00 AM the next day. 

How many users are actually using the IMAP and POP protocols for Exchange On-Premises?

Scenario: There is a lot of mailbox in Exchange On-Premises that have IMAP and POP enabled when running get-casmailbox, but how many are actually using those protocols to connect to Exchange? Use the scriptlets below to help:

Scriptlets: Lets first grab the users who have IMAP and POP enabled in the environment and then compare it to the IMAP and POP Logs.

1. Filter For IMAP and POP Enabled Users

$ImapEnabled = Get-CASMailbox -Filter {Imapenabled -eq $true} -ResultSize unlimited
$PopEnabled = Get-CASMailbox -Filter {Popenabled -eq $true} -ResultSize unlimited

2. Parse through the IMAP and POP Logs to see which accounts are actually connecting with those protocols. (LogParser needs to be installed)

#Declare Variables
$files = @()
$servers = “ExServer1″,”ExServer2”

#Loop it to find log files
$servers | %{
“Searching Files on: $_”

$files += Get-childitem “\$_\c$\Program Files\Microsoft\Exchange Server\V15\Logging\Imap4\IMAP4.log” | Where name -notlike “BE

$files += Get-childitem “\$_\c$\Program Files\Microsoft\Exchange Server\V15\Logging\Pop3\POP3.log” | Where name -notlike “BE


#Separate the IMAP and POP Log Files
$IMAPFiles = $files | Where fullname -like “\IMAP4*” | Select -ExpandProperty fullname
$POPFiles = $files | Where fullname -like “\POP3*” | Select -ExpandProperty fullname

#Search IMAP logs and put all unique values into $IMAPAll
$IMAPLogs = @()
$IMAPfiles | %{
$1 = & “C:\Program Files (x86)\Log Parser 2.2\logparser.exe” -i:CSV -q:on -rtp:-1 -nskiplines:5 @”
Select user From ‘$_’
$IMAPLogs += $1
$IMAPLogs = $IMAPLogs | select -Unique

#Search POP logs and put all unique values into $POPAll
$POPLogs = @()
$Popfiles | %{
$2 = & “C:\Program Files (x86)\Log Parser 2.2\logparser.exe” -i:CSV -q:on -rtp:-1 -nskiplines:5 @”
Select user From ‘$_’
$POPLogs += $2
$POPLogs = $PopLogs | select -Unique

3. Review and Compare the Data Sets

Now you have two data sets that you can review and compare. The users that are actively using IMAP and POP are stored in the $IMAPLogs and $POPLogs. Everyone else found in $IMAPEnabled and $POPEnabled tha tare not in the $IMAPLogs and $POPLogs could have the protocols potentially disabled.

You could use Excel or commands similar to below:

Compare-Object $ImapLogs $($ImapEnabled.name) | Where sideindicator -eq “<=”
Compare-Object $POPLogs $($POPEnabled.name) | Where sideindicator -eq “<=”

The call to ‘https://mail.domain.com/EWS/mrsproxy.svc’ failed because no service was listening on the specified endpoint.

Scenario: When migrating (onboarding) to Exchange Online, you may receive a error message similar to this:

The call to ‘https://servername/EWS/mrsproxy.svc&#8217; failed because no service was listening on the specified endpoint. Error details: There was no endpoint listening at https://servername/EWS/mrsproxy.svc that could accept the message. This is often caused by an incorrect address or SOAP action. See InnerException, if present, for more details. –> The remote server returned an error: (404) Not Found.

Solution: Verify the ExchangeGuid is the same value:

Exchange OnPrem: Get-Mailbox <alias> | Select ExchangeGUID

Exchange Online: Get-mailuser <alias> | Select Exchange GUID

Fix – Exchange Online: Set-mailuser <alias> -exchangeguid <GUID of ExOnPrem user>