DeviceAccessState: Blocked DeviceAccessStateReason: CommandFrequency

Scenario: After a migration to Exchange Online, multiple mobile devices for a single user were receiving an error similar to: Cannot Get Mail – The connection to the server failed. When running this command: Get-mobiledevicestatistics -mailbox <user> | Select DeviceType, DeviceAccessState,LastSuccessSync we found that the device was blocked due to Command Frequency.

What was happening: It appears that Microsoft was throttling all ActiveSync requests to the server for this specific migrated mailbox. After reviewing and monitoring, the device would unblock itself and continue to successfully sync. The theory is that there were multiple devices attempting to synchronize the mailbox data at the exact same time. Plus the administrator was clicking in and out of folders on the mobile device during the initial sync.

Scriptlet: We were monitoring the mobile device status for the mailbox using a command similar to this:

Do{$d = get-date;
$d = $d.tostring();”
Checking devices at :$d
“;
Get-MobileDeviceStatistics -Mailbox usera | Where deviceuseragent -like Apple*|Select DeviceType,DeviceAccessState,lastsucce*;
sleep 15;}While($c -ne 10101)

Analyzing the Get-MobileDeviceStatistics -GetMailboxLog to determine Sync Errors

Scenario:  You have already enabled ActiveSync Debug Logging  (Set-CASMailbox steve -ActiveSyncDebugLogging:$true) and now you want to collect and analyze the logs for errors.  ActiveSync is chatty so it creates a TON of logging data to sift through. So how do we know which log entries are good and which are bad? Here is the dirty way we sifted through it.

Pull the ActiveSync Debug Log:

$1 = Get-MobileDeviceStatistics -Mailbox steve -GetMailboxLog

Lets take a quick glance at the status provided for each ActiveSync transaction provided in the MailboxLogReport

$1.MailboxLogReport | Out-file C:tempmobiledevicedebuglog.txt
$Text = “<status>”
$statuscheck = Get-Content C:tempmobiledevicedebuglog.txt | Select-String -Pattern $Text
$statuscheck

This should provide a long list of StatusCodes.  Usually a status of 1 (<status>1</status>) down the list is great!

BUT if you receive other statuses, then we may have to dive deeper into figuring out what the status means related to the ActiveSync command which caused it.  To do this, you will need to do the following:

  1. Navigate to the Microsoft document which provides a list of each ActiveSync Command and their status codes.  It can be found here.

2. Open the C:tempmobiledevicedebuglog.txt  which was created in previous steps and perform a find on that status code. In this example, we will use:  <status>2</status> as our questionable status.

3. Perform a search for that status and in the responsebody: found in the text file, that section should list the ActiveSync command that was attempted that produced a result.  For example our <status>2</status> was for the PING command.

Digging deeper, it appears the <status>2</status> for PING is fine because it just lets me know that: “Changes occurred in at least one of the monitored folders. The response specifies the changed folders.”  which is fine — no big deal. I found that status code  here.

BUT lets pretend I found a status 8 (<status>8</status>). This would be an issue that was caused/recognized by the Exchange  server.  Something like this could help in troubleshooting a server issue versus a client issue.

 

 

Troubleshooting Techniques: Verify ActiveSync is operational on a server

Scenario:  You received complaints that ActiveSync isn’t working for some users in your environment.  Your environment consists of large number of servers and you need a quick way of determining which server is not able to process ActiveSync requests.

Troubleshooting Steps:

  1. Check the server component state on each server: Get-exchangeserver EX* | Get-servercomponentstate | Where {($_.Component -eq “ActiveSyncProxy”) -and ($_.state -ne “Active”)}
  2. Test-ActiveSyncConnectivity:$1 = get-credential
    $Servers = Get-clientaccessserver esg* | Where AdminDisplayVersion -like *15*
    $final = @()
    $Servers | %{$final += Test-ActiveSyncConnectivity -AllowUnsecureAccess:$true -ClientAccessServer $_ -MailboxCredential $1 -lightmode}
  3. Check Logs on the server:
    1. Verify EAS connections are being logged in the IIS Logs: C:inetpublogslogfilesw3svc1
    2. Verify the EAS logs are generating activity. EAS logs should be generated every hour:  c:Program FilesMicrosoftExchange ServerV15LoggingHttpProxyEas
  4. Go to the URL and test.  A working server produces  DeviceIdMissingOrInvalid as an error after authentication:  https://<server fqdn>/Microsoft-server-activesync

Script to monitor the state of the ActiveSyncProxy server component and email an alert

Scenario:  You want to monitor the ActiveSyncProxy server component state. If ActiveSyncProxy becomes inactive, you want the script to automatically resolve the issue and to send an alert via email.

Script:

#Collect Servers
$1 = Get-exchangeserver Ex* | Get-servercomponentstate | Where {($_.Component -eq "ActiveSyncProxy") -and ($_.state -ne "Active")}

#Loop Servers for Component State
$1 | Select -ExpandProperty Identity | %{$s = [string]$_;Set-servercomponentstate $s -component ActiveSyncProxy -state Active -Requester HealthAPI}

#Send an Email
If($1 -ne $null){
$1_body = $1 | Out-String
#Email
$body =""
$smtp = "smtp.domain.com"
[string[]]$to = "Steve@Domain.com","Fred@Domain.com"
$from = "EASMonitor@domain.com"
$subject = "ActiveSync Monitor - Activesync is currently not working on a server" 
$body += "<b><Font color=Black>There could be an issue with ActiveSync as the server component state is not active on the following servers: </b></font><br><br>"
$body += "<Font color=red>$1_body </font><br><br><br>"
#Send Email
send-MailMessage -SmtpServer $smtp -To $to -From $from -Subject $subject -Body $body -BodyAsHtml -Priority high
}

 

Test-ActiveSyncConnectivity to verify EAS is working properly

Scenario: You want to test whether ActiveSync (EAS) is working properly.  You can use the following script to determine if EAS is failing on any of your Exchange Client Access Servers:

Script:

#Gather Credentials for the Test User into a variable
$1 = get-credential ("DomainEASTestuser")

#Gather Servers into a variable
$Servers = Get-clientaccessserver 

#Define your collection variable
$EAS = @()

#Loop through each server and test
$Servers | %{$EAS += Test-ActiveSyncConnectivity -AllowUnsecureAccess:$true -ClientAccessServer $_ -MailboxCredential $1 -lightmode}

Script to determine Exchange ActiveSync devices and email the report

Scenario:  You want a script that will perform the following:

  1. Provide all device statistics in csv format attached to the email report.
  2. Provide a total of Devices in the email report
  3. Provide a total of mailboxes that have a device in the email report
  4. The average number of devices per user in the email report
  5. Provide a list of devices that have not connected in over 6 months in the email report
  6. Provide a count for each ActiveSync policy applied on all mailboxes in the email report
  7. Provide a list of the top 10 device types in the email report
  8. Provide a list of the top 10 users based on device count in the email report

Script: 

#Define Variables
    $age = 180
    $final_DeviceCount = @()
    $report = @()
    $file = $(((get-date).ToUniversalTime()).ToString("yyyyMMddThhmmssZ"))
    $filename = "C:tempeas_stat_"+$file+".csv"
    $today = get-date
    $final = @()
    $BreakLine = "
    "
 $stats = @("FirstSyncTime",
        "LastPolicyUpdateTime",
        "LastSyncAttemptTime",
        "LastSuccessSync",
        "DeviceType",
        "DeviceID",
        "DeviceUserAgent",
        "DeviceWipeSentTime",
        "DeviceWipeRequestTime",
        "DeviceWipeAckTime",
        "LastPingHeartbeat",
        "RecoveryPassword",
        "DeviceModel",
        "DeviceImei",
        "DeviceFriendlyName",
        "DeviceOS",
        "DeviceOSLanguage",
        "DevicePhoneNumber",
        "MailboxLogReport",
        "DeviceEnableOutboundSMS",
        "DeviceMobileOperator",
        "Identity",
        "Guid",
        "IsRemoteWipeSupported",
        "Status",
        "StatusNote",
        "DeviceAccessState",
        "DeviceAccessStateReason",
        "DeviceAccessControlRule",
        "DevicePolicyApplied",
        "DevicePolicyApplicationStatus",
        "LastDeviceWipeRequestor",
        "ClientVersion",
        "NumberOfFoldersSynced",
        "SyncStateUpgradeTime",
        "ClientType",
        "IsValid",
        "ObjectState"
          )

#Define Email Variables:
    $smtp = "smtp.domain.com"
    [string[]]$to = "steve@domain.com"
    $from = "MobileDeviceStats@domain.com
    $subject = "Mobile Device Stats using ActiveSync"

#Define HTTP Style
$b = "<style>
        BODY{
        font-family: Arial; font-size: 10pt;
        }
        TABLE{
        border: 1px solid black; border-collapse: collapse;
        }
        TH{
        border: 1px solid black; background: #dddddd; padding: 5px;
        }
        TD{
        border: 1px solid black; padding: 5px;
        }
    </style>"
#=========Begin Script===========
#Calculate Mailboxes with EAS Device Partnership
$MailboxesWithEASDevices = @(Get-CASMailbox -Resultsize Unlimited -erroraction SilentlyContinue | Where {$_.HasActiveSyncDevicePartnership} | Select Identity,SAMAccountName)
$MailboxesWithEASDevices = $MailboxeswithEasDevices | Sort SAMAccountName
#Loop Through Each Mailbox
Foreach ($Mailbox in $MailboxesWithEASDevices)
{
$EASDeviceStats = @(Get-MobileDeviceStatistics -Mailbox $Mailbox.Identity)
Write-Host "$($Mailbox.SAMAccountName) has $($EASDeviceStats.Count) device(s)"
#Build the Array
   $ServerObj = New-Object PSObject
   $ServerObj | Add-Member NoteProperty -Name "Alias" -Value $Mailbox.SAMAccountName
   $ServerObj | Add-Member NoteProperty -Name "DeviceCount" -Value $EASDeviceStats.Count
   $Final_DeviceCount += $ServerObj   
$MailboxInfo = Get-Mailbox $Mailbox.Identity | Select DisplayName,PrimarySMTPAddress,Alias
Foreach ($EASDevice in $EASDeviceStats)
    {$LastSuccessSync = ($EASDevice.LastSuccessSync)
 if ($LastSuccessSync -eq $null)
        {
            $syncAge = "Never"
        }
        else
        {
            $syncAge = ($today - $LastSuccessSync).Days
        }
       $reportObj = New-Object PSObject
       $reportObj | Add-Member NoteProperty -Name "DisplayName" -Value $MailboxInfo.DisplayName
       $reportObj | Add-Member NoteProperty -Name "EmailAddress" -Value $MailboxInfo.PrimarySMTPAddress
       $reportObj | Add-Member NoteProperty -Name "Alias" -Value $MailboxInfo.Alias
       $reportObj | Add-Member NoteProperty -Name "SyncAgeInDays" -Value $syncAge
            Foreach ($stat in $stats)
            {
                $reportObj | Add-Member NoteProperty -Name $stat -Value $EASDevice.$stat
            }
            $report += $reportObj
        }    
}
$report | Export-csv $filename
$TotalDevices = $report.count
$TotalMailboxes = $MailboxesWithEASDevices.Count
$AvgPerMailbox = $TotalDevices / $TotalMailboxes
$LastSync180  = ($Report | Where SyncAgeInDays -ge $age).count
$Top10DeviceTypes = $Report | group DeviceType | Sort Count -Descending | Select Count,Name -first 10 | ConvertTo-Html -head $b
$Top10Users = $final_DeviceCount | Sort DeviceCount -Descending | Select Alias,DeviceCount -first 10 | ConvertTo-Html -head $b
$PolicyApplied = $report | Group DevicePolicyApplied | Sort Count -Descending | Select Count,Name | ConvertTo-Html -head $b
$body =""
$attachment = $filename
$body += "<Font color=black>Attached are the Mobile Device Statistics within the entire OnPremise Exchange Organization using ActiveSync.</font><br><br><br>"
$body += "<b><Font color=#00008B>Total devices:</b><Font color=black> $TotalDevices</font><br><br>"
$body += "<b><Font color=#00008B>Total mailboxes that have a device:</b><Font color=black> $TotalMailboxes</font><br><br>"
$body += "<b><Font color=#00008B>Average Devices Per User:</b><Font color=black> $AvgPerMailbox</font><br><br>"
$body += "<b><Font color=#00008B>Devices with last sync time older than $age days:</b><Font color=black> $LastSync180</font><br><br>"
$body += "<b><Font color=#00008B>ActiveSync policy applied on all mailboxes:</b> $PolicyApplied</font><br><br>"
$body += "<b><Font color=#00008B>Top 10 device types:</b></font><br><br>"
$body += "<Font color=black>$Top10DeviceTypes</font><br><br><br>"
$body += "<b><Font color=#00008B>Top 10 users by device count:</b></font><br><br>"
$body += "<Font color=#00008B>$Top10Users</font><br><br><br>"
send-MailMessage -SmtpServer $smtp -To $to -From $from -Subject $subject -Body $body -BodyAsHtml -Priority high -Attachments $attachment

#=========End Script===========

Configure Exchange Certificate Based Authentication for Exchange ActiveSync

Scenario:  You want to enable Certificate Based Authentication for Exchange ActiveSync. TMG is already setup and configured to accept/process he CBA ActiveSync request. Now we need to configure Exchange so it can successfully accept the trusted authenticated hand off from TMG.
Our Environment:
 Exchange: Exchange 2013 SP1
Server:  Windows Server 2012
MDM:  Airwatch (Needed to push out the Certificates)
 We currently have and will keep servers that will process activesync via Basic Authentication with the server name: mail.domain.com. We have separate servers will process activesync via CBA with the servername: mailcba.domain.com.
Configuration Steps:
Exchange Certificate:
1. Add the new external and internal server names that will be used to point activesync devices to EAS CBA as an subject alternate name to the existing server. Currently you may use mail.domain.com as the server name for Exchange ActiveSync that processes Basic Authentication.  You will need a separate server name, such as mailcba.domain.com, that will be used to allow the TMG accept and pass this traffic to the new servers.  Also you need to include the internal server name as an subject alternative name.
 2. Import the Exchange certificate in Exchange and apply the IIS service to it.
Add Roles to Server:
3. Install the Client Certificate Mapping Authentication Role. In PowerShell run:
Import-Module ServerManager
Add-WindowsFeature Web-Client-Auth
 
ActiveSync Virtual Directory:
4. Remove Basic Authentication and Select Accept Client Certificate.
     a. Open Exchange EAC.
b. Navigate to Servers–>Virtual Directories
c. Open the ActiveSync Virtual Directory on the server you wish to enable Certificate Based Authentication.
d. Uncheck Basic Authentication and mark Accept Client Certificates.
 5. Add the Internal and External URLs accordingly, example: internalurl/externalurl: mailcba.domain.com  (Note: Becareful of AutoDiscover. If you have the option of not including this server for Autodiscover lookups for ActiveSync, mobile devices should not receive these server settings when they want to authenticate via Basic.  Else, you could set the internalurl/externalurl on the CBA virtual directories to the same urls as the ones accepting basic auth.
IIS Manager:
5. Enable Active Directory Client Certificate Authentication on the Server.
a. Open IIS Manager.
b. Click on the Server Name.
c. Click on Authentication
d. Enable Active Directory Client Certificate Authentication
e. Restart IIS Admin Service in Services console.
 6. Enable Client Certificate Mapping Authentication on the ActiveSync Virtual Directory.
a. From an elevated command prompt, navigate to C:windowssystem32inetsrv.
b. Enter in the following:
APPCMD.EXE set config “Default Web Site/Microsoft-Se
rver-ActiveSync” -section:system.webServer/security/authentication/clientCertificateMappingAuthentication /enabled:”True” /commit:apphost
 7. Change the UploadReadAheadSize from its default value (0) to the max size you wish to send from a activesync device. For example, if your sending limits is 35MB, the value will be 36700160.
a. Open IIS Manager.
b. Navigate to the Microsoft-Server-ActiveSync virtual directory.
c. Click on Configuration Editor.
d. Navigate to system.webserver/serverRuntime.
e. Edit the Value of uploadReadAheadSize to 36700160 and apply it.
 8.  Enable Windows Authentication on the Microsoft-Server-ActiveSync virtual Directory. (this is for TMG)
a. Open IIS Manager.
b. Navigate to the Microsoft-Server-ActiveSync virtual directory.
c. Click on Authentication.
d. Enable Windows Authentication.
 9.  Restart World Wide Web services from the services console.

Emails in users “Inbox” are automatically moved to a folder (not created by user), labeled “Junk”

Scenario:

A new email arrives, it sits in the “Inbox” anywhere from a few seconds to a few minutes before it moves to a folder labeled “Junk”. This is not the default “Junk E-Mail” folder in Outlook/OWA.

Issue:

Client has a Samsung Galaxy device configured to view users mailbox.

Solution: 

  1. Go into the E-Mail application on the Samsung Galaxy
  2. Press the Menu Key, select Settings, then General Settings
  3. Select Spam addresses and remove any addresses that should not be in there, such as your corporate domain.

Recommendations to avoid IOS Calendar Corruption with Exchange Mailboxes

Scenario: An entry on an IOS calendar shows incorrect times for an appointment, has disappeared, or is missing information, although the calendar entries in OWA and Outlook is correct. Specifically a single occurrence of a repeat appointment/meeting has one of the symptoms from above.

Recommendations from support:


1. Microsoft recommends running the same version of Outlook on all the computers. Mailbox owners and any delegates need to be using the same version of Outlook with the latest updates on all the computers that are used for calendaring. If you are in a mixed environment of Mac, Windows and iOS devices, each platform needs to be using the same version and each device should have the latest updates. 
2. Only one person should process meeting requests. Other people, computers or devices that receive the meeting request should ignore them, they should not delete or process them. Users should have a maximum of 2 delegates.
3. Manage your calendar exclusively from Outlook or OWA. Don’t accept, decline, modify or invite others to appointments from your mobile device. You can create new appointments on your mobile device.
4. Verify that the device has the latest iOS version installed. (Please verify this before adding the Exchange account to the device. Sometimes, new devices are running an older version of iOS, so it is a good habit to confirm that all updates are applied before adding Exchange accounts)
5. To change an entire series of meetings, cancel the original meeting and create a new one. To change one instance, cancel just that meeting and create a new one to replace it. Always put an end date on a recurring meeting.
6. A “corrupt” meeting will remain that way until you delete it. If it is a recurring appointment, delete all occurrences and reschedule it.
7. When scheduling a recurring meeting, Microsoft recommends setting the end date no more than 6 months. If you need to schedule a meeting for a longer period, start a new recurring meeting.

8. Making multiple changes to recurring events can contribute to unexpected results.

Exchange Activesync Monitor for Specific Devices

Scenario:  Monitor specific ActiveSync Devices and report when a device has not made a successful ActiveSync connection for over an hour.  Report the time in local time and not Greenwich.  

Script: I ran the following Exchange PS script every hour . Depending on your requirements, you may need to manipulate or move the script around.

#Format Date to Greenwich
$currentdate = get-date
$currentdate = $currentdate.Addhours(-1)
$currentdate = $currentdate.touniversaltime()

#Pull the devices that have not connected to LastSuccessSync in over an hour
$devices = get-activesyncdevicestatistics DeviceID  | Where {$_.LastSuccessSync -lt $currentdate} | Sort LastSuccessSync | Select DeviceID, DeviceOS, deviceFriendlyName, LastSuccessSync, LastSyncAttemptTime, DeviceModel, Identity

#For the device(s) found, format the information
ForEach ($entry in $devices){
$Device = “Device: “+$entry.DeviceFriendlyName
$DeviceOS = “Device OS:   “+$entry.DeviceOS
$DeviceLastAttempt = “Last Sync Attempt (EST):   “+$entry.LastSyncAttemptTime.ToLocalTime()
$DeviceLastSync = “Last Success Sync (EST):   “+$entry.LastSuccessSync.ToLocalTime()
$DeviceModel = “Device Model:   “+$entry.DeviceModel
$DeviceIdentity = “DeviceID:   “+$entry.Identity
$DeviceIdentity = $DeviceIdentity -replace “Domain/OU/”,””
$DeviceIdentity = $DeviceIdentity -replace “/ExchangeActiveSyncDevices/”,”_”
}

#Email the results if there is a device that has not reported in over 1 hour.
If ($Devices -ne $null){
$SmtpClient = new-object system.net.mail.smtpClient 
$MailMessage = New-Object system.net.mail.mailmessage 
$SmtpClient.Host = “smtp.domain.com” 
$mailmessage.from = (“EASMonitoring@domain.com”) 
#$mailmessage.To.add(“User@domain.com”) 
$mailmessage.Subject = “Alert: A mobile device has not connected to e-mail in over 60 minutes.”
$mailmessage.Body = “The mobile device below has not connected to e-mail in over 60 minutes.
$DeviceIdentity
$Device
$DeviceOS
$DeviceLastAttempt
$DeviceLastSync

$smtpclient.Send($mailmessage)
}