Category: Exchange 2013

Troubleshooting a Managed Availability Probe Responder that has rebooted an Exchange 2013 Server

Scenario:  An Exchange 2013 server reboots unexpectedly and it is suspected that the Responder Engine component of Exchange 2013’s Managed Availability has performed this action. Troubleshooting steps are below to narrow down what actually happened.

What you need to know:  Exchange 2013 Managed Availability consists of 3 components; 1. the Probe Engine, 2. the monitor, 3. The Responder Engine.  The responder will try to resolve issues automatically by restarting the application pool, restarting the service, restarting the server, and finally taking the server offline so it no longer accepts traffic.  Here’s the TechNet article on Managed Availability.

Troubleshooting:
1. Get windows events for responder that forced the server to reboot by running in Powershell:
(Get-WinEvent -LogName Microsoft-Exchange-ManagedAvailability/* | % {[XML]$_.toXml()}).event.userData.eventXml| ?{$_.ActionID -like “*ForceReboot*”} | ft id,RequestorName,Endtime,result –AutoSize
2. Take the RequestorName from the step above and run the following to get more details about the responder:

(Get-WinEvent -LogName Microsoft-Exchange-ActiveMonitoring/responderdefinition | % {[XML]$_.toXml()}).event.userData.eventXml | ?{$_.Name -like “RequestorName“} | ft ServiceName,Name,Alertmask

The AlertMask shows which Probe is used by the Responder. A repetitive failed probe causes a monitor change and a recovery action is invoked.

3. Now we can run the following to check the error messages associated with the failed probe: (Note I removed the word ‘monitor’ and replaced it with ‘probe’ in the end of the powershell command. 
(Get-WinEvent -LogName Microsoft-Exchange-ActiveMonitoring/ProbeResult | % {[XML]$_.toXml()}).event.userData.eventXml | ?{($_.ResultType -eq 4) -and ($_.ResultName -like “* AlertMaskProbe*”)}

The results of this command may show you the error and give an understanding on why the probe failed and made the responder restart the server. If required, the responder can be disabled until the issue is resolved by running:  


Add-GlobalMonitoringOverride -Identity ExchangeRequestorName -ItemType Responder -PropertyName Enabled -PropertyValue 0 -ApplyVersion “15.0.775.38”

Exchange 2013 Content Indexing Failed After Reboot

Scenario:  After performing maintenance and rebooting a Exchange 2013 mailbox server, the Database Copy Status may show healthy, but the Content Index State may show failed. (you can check this by running Get-MailboxDatabaseCopyStatus <dbname> )

Resolution: Give it some time. About 1/4th of our databases Content Index was failed after server reboots. It took about 20 minutes and the Content Index automatically became healthy again.

If you have given it time and it has not became healthy, you can rebuild the Content Index by using the following command:   update-mailboxdatabasecopy <dbname><servername> -CatalogOnly. The database does not need to be dismounted in order for this to run and this is a relatively quick process.

Rolling and Redistributing Databases

Rolling Databases to other Servers

PS Script  – Note: we have a csv file that has the databasenames and the server we want to move to that the script calls. In the csv it has a db and a server column.

import-csv movedb_Ex2010.csv | foreach {Move-ActiveMailboxDatabase $_.DB -ActivateOnServer $_.Server -MountDialOverride:Lossless -confirm:$false}

Status of Script:  Check mounted databases
Get-MailboxServer | Where {$_.Name -like “Server*”} | Get-MailboxDatabaseCopyStatus | Where {$_.Status -like “Mounted”}

Troubleshooting:  

  • Check the health of the database copies: Get-MailboxDatabaseCopyStatus database
  • Manually roll a database, run the following command:   Move-ActiveMailboxDatabase databasename –ActivateOnServer servername –MountDialOverride:Lossless –confirm:$false

Redistributing Databases 

PS command: From the <exchange dir>scripts directory, run the following: 

.RedistributeActiveDatabases.ps1 -DagName dagname -BalanceDbsByActivationPreference –ShowFinalDatabaseDistribution –Confirm:$false


Troubleshooting: 
If the databases did not mount on the server they should have, check for databasecopystatus on that server or give it some time to try again.

You can run Get-MailboxDatabaseCopyStatus databasename to check to see the database copy health.

You can run Move-ActiveMailboxDatabase databasename –ActivateOnServer servername –MountDialOverride:Lossless –confirm:$false to manually move the database to another server.

If the ContentIndexState on the database copies is failed and it does become healthy after time, you could reseed just the ContentIndex by using this command: update-mailboxdatabasecopy databasenameservername –CatalogOnly

If the ContentIndexState is failed on the databasecopies with the same server, you may have to restart that server again.

To run a report on the Health of the databasecopies after a redistribute is finished, run the following:

Get-MailboxServer | Where {$_.name -like “server*”} | Get-MailboxDatabaseCopyStatus | Select Name, Status, ContentIndexState 

Export Message Tracking Log for a Specific Sender after a certain day

Scenario: Export the message tracking log for a specific sender after certain day to a .csv

Get-TransportServer | Get-MessageTrackingLog -Sender user@domain.com -Start 3/19/2014 | Where EventID -like “Send” | Select MessageSubject, ClientIP, ClientHostname, Timestamp, {$_.Recipients} | Export-csv C:usersusernamedesktopmessagetrackinglog.csv

Note: the Recipients is enclosed in {$_. }. This is because the recipients property is an array that can hold multiple values and it does not export correctly if referenced without the additional formatting.

Validate if an Exchange Retention Policy is applied to a mailbox item

Scenario:  You have applied a Exchange Retention Policy to a mailbox and the mailbox items that you expected to delete, may not be deleting right away.

To see if a mailbox item has a retention policy applied, perform the following:
1. Open MFCMAPI
2. Connect to your Outlook Profile via Session–>Logon.
3. Open your mailbox by right clicking your displayname and choose Open Store.
4. Expand out the Root Container –> Top of Information Store.
5. Right click on the mail folder that should have the retention applied and Open Contents Table.
6. Locate an email and find the PR_RETENTION_DATE Property.  The value should contain the date for when the retention will process on the item.
This may tell you the date for when the item may be deleted.  If you believe the retention policy is not getting applied, you could adjust the work cycle by extending it from 1 day to 3 days.  The work cycle is a background process that is throttled, therefore may not be able to get to all the mailboxes in a single days work cycle.

Exchange PowerShell script to search a mailbox based on Message Class

Scenario:  You want to search a mailbox via PowerShell for any messages of a specific message class.

The link below contains a script that can be downloaded. I ran it from PowerShell to look for a CommVault message class (IPM.Note.Commvault.Galaxy.Stub) that would show if we have any stubbed messages in a users mailbox with success. Note: I had to change the location of the webservices.dll in the script.

.Search-MailboxForMessageClass.ps1 user@domain.com IPM.Note.Commvault.Galaxy.STub

PowerShell: Search mailbox for items of a particular message class (ItemClass)

Exchange script to delete email items within a date range against a mailbox

Scenario:   You want to delete mail items (not calendar or contact items) from all mailbox folders in a mailbox. The script performs a query for all email items between a date range.

Script:
$startdate = ’01/01/1900′   #specifies start date
$enddate = (get-date).adddays(-60)  #specifies end date by subtracting 60 days
$enddate = $enddate.ToShortDateString()  #converts end date to string.

$users = Import-csv C:scriptusers.csv   #imports list of users with the column heading ‘name’

#Deletes email content between the two dates for each mailbox.
$users | ForEach {
search-mailbox $_.name -searchquery “kind:email AND Received:$startdate..$enddate” -deletecontent -force
}

Search-Mailbox has a 10,000 item limit that search-mailbox before it stops processing. Put it in a loop and let it run. The example below is for a single mailbox outside of the script above.

do {
 Write-Host $i
search-mailbox mailboxname -searchquery “kind:email AND Received:1/1/1900..12/31/2012” -deletecontent -force
 $i++
 }
 while ($i -le 30)

Powershell Script to check how many emails were Sent and Received by a specific user

Powershell Script to check how many emails were Sent and Received by a specific user:

[Int] $intSent = $intRec = 0

Get-TransportServer | Get-MessageTrackingLog -ResultSize Unlimited -Start “03/09/2014” -End “03/10/2014” -Sender “user@domain.com” -EventID RECEIVE | ? {$_.Source -eq “STOREDRIVER”} | ForEach { $intSent++ }

Get-TransportServer | Get-MessageTrackingLog -ResultSize Unlimited -Start “03/09/2014” -End “03/10/2014” -Recipients “user@domain.com” -EventID DELIVER | ForEach { $intRec++ }

Write-Host “E-mails sent:    “, $intSent

Write-Host “E-mails received:”, $intRec

Cannot install the Exchange 2013 Management Tools after Removing the Exchange 2010 Management Tools.

Scenario:  When attempting to install the Exchange 2013 Management Tools, the Exchange 2013 setup GUI shows the Management Tools as checked and grayed out even though the Exchange 2010 management tools are no longer installed on the computer.

Resolution:  Install the Management Tools from Powershell. From the Exchange Install directory, execute the following:

.setup.exe /Role:ManagementTools /IacceptExchangeServerLicenseTerms

Recommendations to avoid IOS Calendar Corruption with Exchange Mailboxes

Scenario: An entry on an IOS calendar shows incorrect times for an appointment, has disappeared, or is missing information, although the calendar entries in OWA and Outlook is correct. Specifically a single occurrence of a repeat appointment/meeting has one of the symptoms from above.

Recommendations from support:


1. Microsoft recommends running the same version of Outlook on all the computers. Mailbox owners and any delegates need to be using the same version of Outlook with the latest updates on all the computers that are used for calendaring. If you are in a mixed environment of Mac, Windows and iOS devices, each platform needs to be using the same version and each device should have the latest updates. 
2. Only one person should process meeting requests. Other people, computers or devices that receive the meeting request should ignore them, they should not delete or process them. Users should have a maximum of 2 delegates.
3. Manage your calendar exclusively from Outlook or OWA. Don’t accept, decline, modify or invite others to appointments from your mobile device. You can create new appointments on your mobile device.
4. Verify that the device has the latest iOS version installed. (Please verify this before adding the Exchange account to the device. Sometimes, new devices are running an older version of iOS, so it is a good habit to confirm that all updates are applied before adding Exchange accounts)
5. To change an entire series of meetings, cancel the original meeting and create a new one. To change one instance, cancel just that meeting and create a new one to replace it. Always put an end date on a recurring meeting.
6. A “corrupt” meeting will remain that way until you delete it. If it is a recurring appointment, delete all occurrences and reschedule it.
7. When scheduling a recurring meeting, Microsoft recommends setting the end date no more than 6 months. If you need to schedule a meeting for a longer period, start a new recurring meeting.

8. Making multiple changes to recurring events can contribute to unexpected results.