Category: Exchange 2013

Mailboxes were unable to connect after Exchange 2013 CU8 install

Scenario:  After upgrading our Exchange environment to CU8, users were reporting that they were unable to connect to their mailbox.  After investigation, it was narrowed down that only a few servers were showing this issue, and other recently updated servers were working fine.

Symptoms:

  • Outlook Anywhere was showing as disconnected.
  • OWA was taking mailboxes to a blank white page.
  • Mobile Devices were reporting that there were issues connecting to the server
  • Failure to connect to the Exchange Management Shell
  • Multiple events in the System Log of the affected servers: Event ID 15021 (Source: HttpEvent): An error occurred while using SSL configuration for endpoint 0.0.0.0:444.  The error status code is contained within the returned data.

Cause/Resolution: The problem was found to be that the Exchange Back End virtual directory in IIS was no longer bound to a SSL certificate; thus users were not able to establish a connection to their mailboxes.

  1.  Open IIS Manager
  2.  Expand the Server Name –> Sites.
  3.  Right click on the  Exchange Back End site.
  4.  Click on Edit Bindings.
  5.  Edit the entry for https port 444.
  6.  Select a valid SSL vertificate from the list. A certificate needs to be set and this was set  to none and was causing the issue.

 

Configure the Exchange Web.Config File for Lync via Powershell Script on Multiple Servers

Scenario:  During the OWA Integration of Exchange 2013 and Lync 2013 , one of the steps is to edit the web.config file on each Exchange Mailbox Server by adding a IMCertificateThumbprint and IMServerName.

The script below queries a list of Exchange servers, determines the certificate thumbprint needed, appends the required values to the web.config file, and then restarts the MSExchangeOWAAppPool and the MSExchangeUM* services.

$Servers= Get-ExchangeServer | Where AdminDisplayVersion -like "*15*"

$Servers | %{
$servername = $_.name 
Write-Host $servername

#Adding the UM Info into Web.Config
$UMThumb= (Get-ExchangeCertificate -server $servername | Where { $_.Services -like '*UMC*'}).ThumbPrint | Select -First 1

Write-Host "Configuring WebConfig for "$servername " with " $UMThumb

$WebConfigFile= "\$servernameC$Program FilesMicrosoftExchange ServerV15ClientAccessOwaweb.config"

Write-Host "Editing Web.Config in $WebConfigFile"

$wc= [XML](Get-Content $WebConfigFile)
$el= $wc.CreateElement("add")
$key= $wc.CreateAttribute( 'key')
$key.psbase.value = 'IMCertificateThumbprint'
$val= $wc.CreateAttribute('value')
$val.psbase.value= $UMThumb
$el.SetAttributeNode($key)
$el.SetAttributeNode($val)
$wc.configuration.appSettings.Appendchild( $el)
$el= $wc.CreateElement("add")
$key= $wc.CreateAttribute( 'key')
$key.psbase.value = 'IMServerName'
$val= $wc.CreateAttribute('value')
$val.psbase.value= 'LyncServerPool.Domain.Com'
$el.SetAttributeNode($key)
$el.SetAttributeNode($val)
$wc.configuration.appSettings.Appendchild( $el)
$wc.Save( $WebConfigFile)


#Restart the AppPool
$appPoolName = "MSExchangeOWAAppPool"
$appPool = get-wmiobject -computername $servername -namespace "rootMicrosoftIISv2" -class "IIsApplicationPool" -Authentication PacketPrivacy -Impersonation Impersonate | Where-Object {$_.Name -eq "W3SVC/APPPOOLS/$appPoolName"}
$appPool.Recycle()

#Restart the UM Services
Get-Service MSExchangeUM* -computername $servername | Restart-Service

}

 

Build a custom table in Powershell that includes mailbox count

Scenario:  You want to build a custom table to include the MailboxCount and other elements of the Get-ExchangeServer command.

$exchangeservers = Get-ExchangeServer | Where AdminDisplayVersion -like "*15*"

$final = @()

$Exchangeservers | %{

#Put the Server Name into a Variable
$server = $_.Name

#Grab the Mailbox Count Per server
$MBXCount = (Get-mailbox -server $_.Name).Count

#Version of Exchange
$AdminDisplayVersion = $_.AdminDisplayVersion

Write-Host "Checking $server"

#Grab the InstallPath
$ExchangeInstallPath = $null

$ExchangeInstallPath = Invoke-Command –Computername $server -ScriptBlock {$env:ExchangeInstallPath} -ErrorAction STOP

#Build the Array
 $ServerObj = New-Object PSObject
 $ServerObj | Add-Member NoteProperty -Name "ServerName" -Value $server
 $ServerObj | Add-Member NoteProperty -Name "InstallPath" -Value $ExchangeInstallPath
 $ServerObj | Add-Member NoteProperty -Name "MBXCount" -value $MBXCount
 $ServerObj | Add-Member NoteProperty -Name "Version" -value $AdminDisplayVersion
    $Final += $ServerObj    
}
$Final

 

 

Exchange 2013 Backup Event ID’s in Order

Here are the Exchange 2013 Backup Event ID’s in order to assist troubleshooting your backup related problems.

  • Event ID 2021  – MSExchangeRepl –  Successfully collected metadata document in preparation for backup.
  • Event ID 2110  – MSExchangeRepl –  Successfully prepared for a full or a copy backup of database MDB01.
  • Event ID 2023  – MSExchangeRepl –  VSS writer successfully prepared for backup.
  • Event ID 2005  – ESE –  Shadow copy instance started.
  • Event ID 2025  – MSExchangeRepl –  VSS successfully prepared for a snapshot.
  • Event ID 2001  – ESE –  MDB01 shadow copy freeze started.
  • Event ID 2027  – MSExchangeRepl –  VSS writer instance has successfully frozen the databases.
  • Event ID 2003  – ESE –  MDB01 shadow copy freeze ended.
  • Event ID 2029  – MSExchangeRepl –  VSS writer instance has successfully thawed the databases.
  • Event ID 2035  – MSExchangeRepl –  VSS writer has successfully processed the post – snapshot event.
  • Event ID 2021  – MSExchangeRepl –  VSS writer has successfully collected the metadata document in preparation for backup.
  • Event ID 224  – ESE –  MDB01 deleting log files C:ExchVolsMDB01Log FilesE0000000001.log to C:ExchVolsMDB01Log FilesE000000002B.log.
  • Event ID 225  – ESE –  MDB01—no log files can be truncated; will be logged instead of Event ID 224 when circular logging is used.
  • Event ID 2046  – MSExchangeRepl –  VSS writer has successfully completed the backup of database MDB01.
  • Event ID 2006  – ESE –  MDB01 shadow copy completed successfully.
  • Event ID 2033  – MSExchangeRepl –  VSS writer has successfully processed the backup completion event.
  • Event ID 2037  – MSExchangeRepl –  VSS writer backup has been successfully shut down.

Exchange Database Restore with CommVault

Here are the steps for performing the restore:

1. Create the Recovery Exchange Database in Exchange

2. Restore the Database from backup in CommVault

3. Restore the contents of the mailbox in the recovery database to a place holder mailbox in Exchange.

4. Create a PST of the of the mailbox content of the place holder mailbox in Exchange.

 

Create a Recovery Database in Exchange 2013:
The server MBX1 has a dedicated mount point for restoring databases in Exchange. The mountpoint is C:RecoveryDB on MBX1. Run the following powershell command:

New-MailboxDatabase -Recovery -Name RDB_DB01 -Server MBX1 -EdbFilePath “C:RecoveryDBRDBDB01.edb” -LogFolderPath “C:RecoveryDBRDB logs”

 

Restore to the Recovery Database in CommVault:

1. Ensure the database you want to restore is dismounted and marked for overwrite.

2. From the CommCell Console, navigate to Client Computers | <Exchange CommVault Client>. Right-click Exchange Database and then click All Tasks | Browse Backup Data.

3. Select your Browse option for when to Restore From. Select the date of the last known good backup of the data that needs to be restored.

4. In the left pane of the Client Browse window, navigate to Exchange Database | Microsoft Information Store | <Storage Group>. Select the database to be restored in the right pane and click Recover All Selected.

5. In the Restore Destination section, select the destination client that holds the Recovery Database; MBX1. In the Destination DB, select the new recovery database that was created; RDB_DB01. Click OK. You can monitor the restore in the Job Controller so you know when it is 100%.
Extract Content from Recovery Database:
1. Mount the Recovery Database so we can access the content by running the following in Exchange PowerShell:

Mount-database RDB_DB01

2. The following PowerShell command we can use to extract content of the mailboxes. The content can be extracted and placed into another mailbox. We will use a place holder mailbox called r_steve2010 and put all of the content into a recovery folder which will be created during the mailbox restore.

Exchange 2013 command:

New-MailboxRestoreRequest -SourceDatabase “RDB_DB01” -SourceStoreMailbox “<DisplayName of User on RDB Database>” -TargetMailbox “r_steve2010” -TargetRootFolder Recovery -AllowLegacyDNMismatch

Exchange 2010 Command:

Restore-Mailbox -Identity “R_Steve2010” -RecoveryDatabase “RDB_DB01” -RecoveryMailbox ‘steve2010’ -TargetFolder Recovery

3. From here we can either provide access the dummy account, or export the content from the dummy mailbox to a PST and give it to the customer. To export it to a pst, run the following Powershell Command:

New-MailboxExportRequest R_steve2010 –FilePath “\servernamesharenamefilename.pst” -acceptlargedataloss -baditemlimit 999
Now we have successfully restored and recovered content from a Commvault backup.

How to quickly gather IP Addresses for a list of Servers

Scenario: You want to quickly gather the IP addresses from a list of HostNames.  Gather your hostnames into a variable and run the following script:

#Gather into your Variable ( I am gathering a list of all Exchange 2010 servers) – You could also Import-CSV or other import types.

$Servers = Get-ExchangeServer ExSvr* | Where AdminDisplayversion -like *14* | Sort Name

#Loop It!  You can also write it out to a file as well by inserting Out-File with -append OR other export types.

$servers | %{
$IP = [System.Net.DNS]::GetHostAddresses($_.Name).IPAddressToString
$Name = $_.Name +":"+$IP
Write-Host $name
}

“The User Profile service service failed the logon. User profile cannot be loaded.” when sigining into a server.

Scenario: When trying to log into a 2012 server, you receive the following error:

The User Profile service service failed the logon. User profile cannot be loaded.

​This info is displayed in the application log (Event 1509).

Windows cannot copy file \?C:UsersDefaultAppDataLocalMicrosoftExchange Serverv15Configuration14720_100.sqm to location \?C:Users<username>AppDataLocalMicrosoftExchange Serverv15Configuration14720_100.sqm. This error may be caused by network problems or insufficient security rights.

DETAIL – Access is denied.

Resolution:  Go to the permissions of the source file and make sure you select the permissions to be inherited: C:usersdefault…Configuration14720_100.sqm

Exchange 2010 is no longer connecting with Outlook Anywhere

Scenario: After the installation of additional Exchange 2013 servers​, we noticed that Outlook Anywhere is broke in our Exchange 2010 environment. The Exchange 2013 environment and mailboxes are unaffected by the connection problems, but the Exchange 2010 mailboxes cannot use Outlook Anywhere. Ex2010 mailboxes have to make a connection directly to the CAS Array or connect via other protocols as workarounds.

Symptoms:

◦We are seeing 503 HTTP Statuses (Service Unavailable) in the Ex2013 IIS logs when trying to connect to Exchange 2010 for Outlook Anywhere.

◦Outlook Clients either cannot establish a Outlook Anywhere connection, or their connection failback’s to a standard TCP connection.

◦Exchange 2013 mailboxes may have problems connecting to Public Folders on Exchange 2010 via Outlook.

◦Running the powershell command below fails when trying to get a referral and gives a 0x0000006BA or 0x6Ba error:

test-outlookconnectivity -protocols HTTP -credential $mycreds -verbose

◦Using the following command, you are unable to make a connection to ports 6001, 6002, and 6004.

rpcping -t ncacn_http -o RpcProxy=mail.domain.com -P “user,domain,password” -H 1 -F 3 -a connect -u 9 -v 3 -s casarray.domain.com -I “user,domain,password” -e 6001

 

Cause:  Ex2010 has a limit of servers that can be entered into a ServerFarm value in the Registry.   The addition of the new Ex2013 servers took us over the 64 server limit. When this happened, Outlook Anywhere (RPC/HTTP) in Ex2010 broke. Troubleshooting various symptoms pointed to different areas of Exchange that could be the cause of this behavior.  Later it was discovered that it was a bug confirmed by Microsoft. The real kicker of this ‘bug’ is that the Exchange 2013 servers, regardless if their frontend (CAS) or backend (MBX) servers, gets populated in this Ex 2010 key because of the Ex2013 architectural differences.

 

2 Resolutions:

1. Microsoft has an IU (interium update) for Exchange 2010 to fix this issue. You just need to remove this IU before you proceed with other installs. Supposedly this issue will be fixed in Exchange 2010 RU9 for SP3.   This was our fix

Or

2. You can edit the registry manually. Note this worked for us for a while, but we did have problems with this and resorted to the IU method. Regardless , this method may get you by for a little while:

Turn the Polling for the RPCHttpConfigurator off by going setting the PeriodicPollingMinutes to 0:

HKEY_LOCAL_MACHINESYSTEMCurrentControlSetservicesMSExchangeServiceHostRpcHttpConfiguratorPeriodicPollingMinutes

Manually remove the Exchange 2013 servers from the ServerFarm Registry Key to take the number of entries below 64:

HKEY_LOCAL_MACHINESOFTWAREMicrosoftRpcRpcProxyLBSConfigurationca5b08e5-4a52-5701-0000-000000000000

Re-Activate inactive Server Component States for Exchange 2013

Scenario: After Exchange Patching/Upgrades (both failed and successful attempts) you may notice that the ​Server Component States become Inactive. For Example, even though the IMAP service is started, the server may be refusing the connection. To check to see the Server Component States state, run the following command:

get-servercomponentstate <servername>

To re-activate a Server Component State, run the following

set-servercomponentstate <servername> -component <component name> -State Active -Requester HealthAPI.

Note, other Requester’s you could try are:

Maintenance,  Sidelined, Functional, Deployment

Cert Work! Querying, Removing, Assigning Services to Exchange Servers via Powershell

Scenario:  You want to clean up Exchange Certificates on your Exchange Servers. The following steps are examples of querying and building your query to perform an action.

1. Check to see what Exchange Certs are on your server.

get-exchangecertificate -server  ExSvr1

2. Query a list of Certificates that have the subject mail.domain.com:

get-exchangecertificate – Server ExSvr1 | Where Subject -like CN=Mail.dom*

3.  Query a list of Certificates that have the subject mail.domain.com and have a Expiration less than a specific date:

Get-ExchangeCertificate -Server ExSvr1 | Where {($_.NotAfter -lt “3/22/2019”) -and ($_.Subject -like “CN=Mail.dom*”)} 

4.  Remove the list of Certificates that have the subject mail.domain.com and an expiration less than a specific date:

Get-ExchangeCertificate -Server ExSvr1 | Where {($_.NotAfter -lt “3/22/2019”) -and ($_.Subject -like “CN=Mail.dom*”)}  | Remove-Exchangecertificate -confirm:$false

 

Lets say you want to query all Ex2013 servers to find and remove the certs:

1. Gather your Servers into a Variable:

$Servers = Get-ExchangeServers | Where AdminDisplayVersion -like *15*

2. Use that variable in a loop to loop through the certs:

$Servers | %{

Write-Host $_.name;

Get-ExchangeCertificate -server $_.name | Where {($_.NotAfter -lt “3/22/2019”) -and ($_.Subject -like “CN=M*”)}  | Remove-Exchangecertificate -confirm:$false

}

 

How to move/enable services on an Exchange Certificate:

1. Determine the Thumbprints of the Certificate you want to move Exchange Services to:

Get-exchangecertificate -server ExSrv1

2. Move/Enable services on an Exchange Certificate

Enable-ExchangeCertificate -thumbprint <thumpbrint> -server ExSrv1 -services IIS,SMTP,POP,IMAP

 

Now lets say you want to loop it:

1. Gather your servers into a variable:

$Servers = Get-ExchangeServers | Where AdminDisplayVersion -like *15*

2. Enable Services on all your servers certs with a Loop:

$Servers | %{ 

Write-Host $_.name;

Enable-ExchangeCertificate -thumbprint <thumpbrint> -server $_.name -services IIS,SMTP,POP,IMAP

}