Category Archives: Exchange

Running Sync-ModernMailPublicFolders.ps1 with Modern Authentication

UPDATE: After years of Microsoft leaving users lost and confused, they finally decided to refactor and release a new version their script (only a few days after I published this article). Their new script is now based on the EXO PS v2 module which uses Graph API and no longer uses EXO PS Remoting.

As part of an Exchange Online migration I was re-running a Public Folder sync that I initially ran two years ago when we still had basic authentication enabled from internal networks for Exchange Online. When I went to re-run I realized it was not going to work because we completely disabled basic authentication. I had recently created an unattended script that connects to Exchange Online using MSAL and integrated Windows authentication (IWA) and was able to use this methodology in this script without making any modifications to the script.

To make this work we first need to install the MSAL.PS module:

Install-Module -Name 'MSAL.PS'

Next, we need to get an OAuth token for Exchange Online PowerShell, create a bearer token secure string, and create a basic credential from it along with our UPN. We will need to specify the client ID for Exchange Online PowerShell (612e1698-a44c-49a4-b66b-4188cc69cbaa) along with our tenant ID (I’m entering a fake one here). When prompted you will login with an account that has EXO administrative access:

$EXOToken = Get-MsalToken -ClientId 'a0c73c16-a7e3-4564-9a95-2bdf47383716' -TenantId '612e1698-a44c-49a4-b66b-4188cc69cbaa' -Scopes 'https://outlook.office365.com/.default'

$TokenSecureString = ConvertTo-SecureString "Bearer $($EXOToken.AccessToken)" -AsPlainText -Force

$Credential = New-Object System.Management.Automation.PSCredential($EXOToken.Account.Username, $TokenSecureString)

Now we can execute the script, but we will need to use an alternate ConnectionUri so that we can make EXO do the basic -> OAuth conversion:

.\Sync-ModernMailPublicFolders.ps1 -Credential $Credential -ConnectionUri 'https://outlook.office365.com/powershell-liveid?BasicAuthToOAuthConversion=true' -CsvSummaryFile:sync_summary.csv

The script should have executed without an issue. No need to re-enable basic authentication!

Exchange FIP-FS Scan Engine Update Issues: How to roll-back the update

UPDATE #3: There is the potential that after running Rollback-FIPFSEngine.ps1 that you may see the following error in the Application Event Logs:

Log Name:      Application
Source:        Microsoft-Filtering-FIPFS
Date:          01/04/2022 00:00:00
Event ID:      6027
Task Category: None
Level:         Error
Keywords:      
User:          NETWORK SERVICE
Computer:      exchange-server.domain.com
Description:
MS Filtering Engine Update process was unsuccessful to download the engine update for Microsoft from Custom Update Path.
Update Path:http://amupdatedl.microsoft.com/server/amupdate
UpdateVersion:0
Reason:"There was a catastrophic error while attempting to update the engine. Error: DownloadEngine failed and there are no further update paths available.Engine Id: 1 Engine Name: Microsoft"

If you are seeing the error above this could be an indication that there is an improper ACL on the ‘FIP-FS\Data\Engines’ directory. The Rollback-FIPFSEngine.ps1 script has been updated to set the proper ACL, but if you’ve previously run the script and need to resolve this issue you can run the following on your affected Exchange server(s) from an elevated PowerShell session:

$InstallFIPFSPath = "$($env:ExchangeInstallPath)FIP-FS"
$Sddl = 'O:SYG:SYD:PAI(A;OICI;FA;;;SY)(A;OICI;FA;;;LS)(A;OICI;FA;;;NS)(A;OICI;FA;;;BA)'
$NewSddl = Get-Acl -Path "$InstallFIPFSPath\Data\Engines"
$NewSddl.SetSecurityDescriptorSddlForm($Sddl)
Set-Acl -Path "$InstallFIPFSPath\Data\Engines" -AclObject $NewSddl

UPDATE #2: I have still been recommending this procedure since it doesn’t rely on deleting definitions and then waiting for the definition download from MS. With this workaround you are immediately in a fully working state.

If you have already used the Microsoft reset method on one or some server(s), you can use Rollback-FIPFSEngine.ps1 to copy definitions directly from the good server. This will speed up resolution time as the other servers will not have to wait for a definition download. Simply change the $BackupPathFIPFSPath variable to something like below and run on the other servers:

$BackupPathFIPFSPath = '\\GOODEXSERVER.domain.com\C$\Program Files\Microsoft\Exchange Server\V15\FIP-FS'

Note: Just make sure to re-enable auto-updates by running: Set-EngineUpdateCommonSettings -EnableUpdates $true

UPDATE: Microsoft has released a new engine update with a version number that resolves the issue. If you have already performed the fix above there is *no need* to perform the procedures in their article as you are already in a functional state. All you need to do is run Set-EngineUpdateCommonSettings -EnableUpdates $true to re-enable updates and your server(s) will download the latest update at the next check interval. Their script/procedure is for customers who are broken.


How to roll-back…

As almost anyone with an on-prem Exchange implementation probably knows there was an update released on 12/31/21 that caused email delivery issues globally. The core issue is the new version number is too large to fit in a long variable. Microsoft has acknowledged the issue, but the only current workaround is to disable the FIP-FS engine. The issue here is that certain transport rules also use this service, so if you have transport rules like these you may still have email delays even after disabling the engine. A better option would be to roll-back to the engine version that did not invoke the bug while Microsoft works this out.

I created a script (Rollback-FIPFSEngine.ps1) that makes the roll-back quick and easy. The one pre-requisite is that you need to restore your FIP-FS directory (ex. C:\Program Files\Microsoft\Exchange Server\V15\FIP-FS) from a restore. The restore should be from some point before the bad engine update. You can use the same restore for all Exchange servers assuming they are all running the same version of Exchange and have the same architecture (ex. amd64).

To preform the roll-back you must do the following:

  • Obtain a restore of the FIP-FS directory
  • Copy the Rollback-FIPFSEngine.ps1 script to the Exchange server(s) that need the rollback
  • Edit the $BackupPathFIPFSPath variable of the script to reflect the restored FIP-FS directory path
  • Open a new elevated (as Administrator) Exchange Management Shell PowerShell session on the server. Do not run this from a regular PowerShell session, it must be a Exchange session
  • Execute the script

The script should take a few minutes to execute and will do the following:

  • Re-enable FIP-FS. It doesn’t matter if you previously disabled FIP-FS using the Disable-Antimalwarescanning.ps1 script or Set-MalwareFilteringServer cmdlet. It will re-enable both
  • Globally disable FIP-FS engine updates (so that the problem updates do not return)
  • Stop necessary services (BITS, MSExchangeTransport, and MSExchangeAntispamUpdate)
  • Rename current engine files/directories
  • Copy engine files/directories from restore path
  • Start stopped services

After the script is run everything should be functioning normally. We’ve already run this in our environment and mail flow is now back to normal.

NOTE: After MS releases a fix you will need to re-enable updates by running: Set-EngineUpdateCommonSettings -EnableUpdates $true

FilteringServiceFailureException Error: Microsoft.Exchange.MessagingPolicies.Rules.FilteringServiceFailureException: FIPS text extraction failed with error: ‘WSM_Error: Scanning Process caught exception: (0x00000005) Access is denied

For some time we had been seeing these events in the event logs of our Exchange mailbox servers and the ‘UnifiedContent‘ directory (related to the Hub Transport role) has been growing:

Log Name:      Application
Source:        MSExchange Messaging Policies
Date:          10/26/2021 8:08:10 AM
Event ID:      4010
Task Category: Rules
Level:         Error
Keywords:      Classic
User:          N/A
Computer:      mbx1.domain.com
Description:
Transport engine failed to evaluate condition due to Filtering Service error. The rule is configured to ignore errors. Details: 'Organization: '' Message ID '<1ea41f5d-64ec-424a-b863-19d7fc2cf7d0@journal.report.generator>' Rule ID 'bcdf1c32-0249-4149-a91b-85ecabaeb695' Predicate '' Action ''. FilteringServiceFailureException Error: Microsoft.Exchange.MessagingPolicies.Rules.FilteringServiceFailureException: FIPS text extraction failed with error: 'WSM_Error: Scanning Process caught exception: 
Stream ID: <1ea41f5d-64ec-424a-b863-19d7fc2cf7d0@journal.report.generator>
ScanID: {E44453FB-B127-44F8-BEF0-357252C6DAA3}
(0x00000005) Access is denied.  Failed to open file: T:\TransportRoles\data\Temp\UnifiedContent\8bedad9e-130a-490e-be7a-af8a58758231'. See inner exception for details ---> Microsoft.Filtering.FilteringException: WSM_Error: Scanning Process caught exception: 
Stream ID: <1ea41f5d-64ec-424a-b863-19d7fc2cf7d0@journal.report.generator>
ScanID: {E44453FB-B127-44F8-BEF0-357252C6DAA3}
(0x00000005) Access is denied.  Failed to open file: T:\TransportRoles\data\Temp\UnifiedContent\8bedad9e-130a-490e-be7a-af8a58758231
   at Microsoft.Filtering.InteropUtils.ThrowPostScanErrorAsFilteringException(WSM_ReturnCode code, String message)
   at Microsoft.Filtering.FilteringService.EndScan(IAsyncResult ar)
   at Microsoft.Filtering.FipsDataStreamFilteringService.EndScan(IAsyncResult ar)
   at Microsoft.Exchange.MessagingPolicies.Rules.UnifiedContentServiceInvoker.TextExtractionComplete(IFipsDataStreamFilteringService textExtractionService, TextExtractionCompleteCallback textExtractionCompleteCallback, IAsyncResult asyncResult)
   --- End of inner exception stack trace ---
   at Microsoft.Exchange.MessagingPolicies.Rules.UnifiedContentServiceInvoker.GetUnifiedContentResults(FilteringServiceInvokerRequest filteringServiceInvokerRequest)
   at Microsoft.Exchange.MessagingPolicies.Rules.MailMessage.GetUnifiedContentResults()
   at Microsoft.Exchange.MessagingPolicies.Rules.MailMessage.GetAttachmentStreamIdentities()
   at Microsoft.Exchange.MessagingPolicies.Rules.MailMessage.GetAttachmentInfos()
   at Microsoft.Exchange.MessagingPolicies.Rules.MailMessage.get_AttachmentNames()
   at Microsoft.Exchange.MessagingPolicies.Rules.MessageProperty.OnGetValue(RulesEvaluationContext baseContext)
   at Microsoft.Exchange.MessagingPolicies.Rules.Property.GetValue(RulesEvaluationContext context)
   at Microsoft.Exchange.MessagingPolicies.Rules.TextMatchingPredicate.OnEvaluate(RulesEvaluationContext context)
   at Microsoft.Exchange.MessagingPolicies.Rules.PredicateCondition.Evaluate(RulesEvaluationContext context)
   at Microsoft.Exchange.MessagingPolicies.Rules.AndCondition.Evaluate(RulesEvaluationContext context)
   at Microsoft.Exchange.MessagingPolicies.Rules.RulesEvaluator.EvaluateCondition(Condition condition, RulesEvaluationContext evaluationContext)
   at Microsoft.Exchange.MessagingPolicies.Rules.TransportRulesEvaluator.EvaluateCondition(Condition condition, RulesEvaluationContext evaluationContext). Message-Id:<1ea41f5d-64ec-424a-b863-19d7fc2cf7d0@journal.report.generator>'

You may notice the ‘T:\TransportRoles\data‘ path above and this is due to the fact that we have our transport queue database path set to an alternate location. It is clear in the error that there is an access issue as it is is stating ‘(0x00000005) Access is denied. Failed to open file: T:\TransportRoles\data\Temp\UnifiedContent\8bedad9e-130a-490e-be7a-af8a58758231‘ as the core issue. Looking at the ‘Temp‘ directory ACL we saw the current permissions were:

  • LocalSystem – Full Control
  • Administrators – Full Control
  • NetworkService – Full Control

These permissions seem correct at face value, but when we look at the ACL of one of the files we actually found:

  • LocalSystem – Full Control
  • Administrators – Full Control
  • NetworkService – Full Control
  • LocalService – Full Control

If you look at a default Exchange installation you will also see the ACL above is how it is set. It seems that when using a non-default queue database location you are required to set the ACL yourself as it won’t be set automatically. After fixing the ACL we simply shut down the transport service, cleared the directory, and restarted the transport service:

Stop-Service MSExchangeTransport
Remove-Item -Path "T:\TransportRoles\data\Temp\UnifiedContent\*"
Start-Service MSExchangeTransport

After this change the ‘UnifiedContent‘ directories are no longer growing and the error we started with is no longer appearing in the event log.

Using Application Permissions (and client credentials grant flow) with Hybrid Exchange Graph API

We recently came across an application that uses Graph API and we wanted to start using it for some our on-prem mailboxes. Hybrid Graph API only supports delegated authentication flows and not application authentication flows. Just because something isn’t “supported” doesn’t mean you can’t make it work! There are two things that we’ll need to do to make this work.

First, any internet-facing Exchange server will need to have ‘V1S2SAppOnly‘ OAuth support added. You can do this by adding V1S2SAppOnly to the OAuthHttpModule.Profiles key in the REST web.config (ex. …\Program Files\Microsoft\Exchange Server\V15\FrontEnd\HttpProxy\rest). Once you add this value the key should look like this:

After this has been added either perform an iisreset or restart the ‘MSExchangeRestFrontEndAppPool‘ app pool in IIS for each server where you did this modification.

The next step is to add the appropriate ‘AppOnlyPermissions‘ to the Microsoft Graph partner application in Exchange/AD. First we’ll take a look at our Graph partner application:

Get-PartnerApplication | Where {$_.Name -like '*graph*'} | select *permissions* | fl

AppOnlyPermissions : 
ActAsPermissions   : {Mail.Read, Mail.Write, Mail.Send, Calendars.Read...}

The ‘AppOnlyPermissions‘ value should be blank. We need this to match the ‘ActAsPermissions‘ value. You’d think (and some other articles say) that you could just run Set-PartnerApplication -AppOnlyPermissions… but this was not a supported parameter for me. To set this we’ll have to edit AD directly. You’ll need to fire up something like ADSIEdit, load the AD configuration partition, and drill down to your Exchange org and partner application object. The path should be something like:

CN=Microsoft Graph,CN=Partner Applications,CN=Auth Configuration,CN=YOUR_EXCH_ORG,CN=Microsoft Exchange,CN=Services,CN=Configuration,DC=domain,DC=com

Once here you’ll need to open the ‘Microsoft Graph‘ object and copy the ‘msExchConfigurationXML‘ attribute value to the clipboard:

Next, we’ll use Notepad++ and the XML Tools plugin to manipulate this. MAKE SURE YOU BACKUP THIS VALUE. We want to convert this to a nicely formatted XML output so that it is easy to work with. To do this we use the ‘Pretty print’ option in XML Tools.

Once we have this we’ll need to duplicate all of the ‘ActAsPermissions’ lines and then use find and replace to convert those tags to be ‘AppOnlyPermissions’. Doing this will create a set of application permissions based on our delegated permissions (ActAsPermissions).

Once completed we need to linearize the output again so that we can copy it back into AD. We can use the ‘Linearize’ option in XML Tools for this:

Once we have the XML in the proper format we can put it back in the AD object:

Now that we’ve updated the AD object we can verify everything looks good by checking Exchange again (AppOnlyPermissions should have the same values as ActAsPermissions):

Get-PartnerApplication | Where {$_.Name -like '*graph*'} | select *permissions* | fl


AppOnlyPermissions : {Mail.Read, Mail.Write, Mail.Send, Calendars.Read...}
ActAsPermissions   : {Mail.Read, Mail.Write, Mail.Send, Calendars.Read...}

At this point you should be able to access on-prem Exchange resources using the supported Graph API functions with Application Permissions.

NOTE: One limitation of this is that application access policies (set in EXO) DO NOT apply and are ignored when accessing an on-prem mailbox.

Using an alternative address for Microsoft Bookings companies

UPDATE (11/2/22): I just noticed a parameter in OWA policies in EXO PowerShell the other day that allows you to set the default domain for Bookings calendars. I was able to set the default domain to bookings.company.com and it is taking effect for all newly created Bookings calendars. You need to run this for all OWA mailbox policies that apply to users who have access to create calendars. The command I used was:

Set-OwaMailboxPolicy -Identity OwaMailboxPolicy-Default -BookingsMailboxDomain bookings.company.com

During testing of Microsoft Bookings we found there was no way to adjust the sender email address for a Bookings company. This was frustrating as our default address was our default tenant address (which ended in onmicrosoft.com). This led me to ask how does this system even work? What is generating the emails? What hosts the email address?

Let’s start by looking for the object(s) hosting this service from an email perspective. To do this we can issue the command below in an Exchange Online PowerShell session. For each ‘company’ created in Bookings there is an associated mailbox created with the ‘RecipientTypeDetails‘ of ‘SchedulingMailbox‘. These mailboxes are not visible in the Exchange Online admin console.

Get-Mailbox -RecipientTypeDetails SchedulingMailbox

When the company is created, the user(s) with permissions to manage are added with ‘FullAccess‘ to the mailbox permissions. This is how Bookings company permissions are managed.

The mailbox is also configured to forward to the email address configured under ‘Business Information’ -> ‘Send customer replies to’.

There is also a Azure AD user account created for each company.

To avoid email address conflicts we decided we would create a new sub-domain for Bookings mailboxes called bookings.domain.com. To do this you would use the Microsoft 365 admin center to add a new domain. After this domain is added and proofed it can be used in Azure AD/O365. Make sure you configure on-premises Exchange routing for this new sub-domain if you are in a hybrid configuration. You can use the instructions under ‘Configure a group domain’ here to setup this routing.

Now that we have the new sub-domain we can use this in our Bookings company mailbox. The easiest way to do this is to adjust the UPN of the user account associated with the mailbox. This will automatically update the address.

After the Exchange Online directory syncs up with Azure AD the email address should be updated.

Now we need to unpublish and re-publish the company booking page so that it starts using the new address.

After re-publishing the page we need to wait a little bit for it to pick up the new email address. Eventually Bookings will start sending emails from the new email address. In testing it took around 10-15 minutes to take effect.

One more thing to note is that Bookings does not give you a method to delete companies after creating them. If you ever need to delete the company you simply need to delete the mailbox/user using the Remove-Mailbox cmdlet.

Looks like you don’t have permission to schedule meetings for this account: A deep-dive into Teams delegate meeting scheduling in a hybrid environment

After implementing Teams last year we were faced with a number of issues to work through. One of the issues we were having was the inability of mailbox delegates to schedule meetings on behalf of mailbox owners. If you are a hybrid Exchange user, the very first step is making sure you have OAuth configured between your on-premises environment and Exchange online. The Hybrid Configuration Wizard should be taking care of this for you. I’ve also written an article on properly securing it externally. You will also need create and configure a service principal to allow Teams to interact with your on-premises Exchange environment. Steps 2-3 in this article must be preformed to set this up. If one of these items are not configured, you will receive the error ‘Sorry, but we can’t connect to the server right now. Please try again later.

Teams delegate invitations through Outlook - Microsoft Community

There is already a lot of decent information around troubleshooting the OAuth side of things here, so I’m going to focus on the other pieces that aren’t always covered…

Once these pre-requisites are configured, you may find users are still receiving errors when trying to schedule Teams meetings on behalf of other users. An error that many of our users were experiencing was ‘Looks like you don’t have permission to schedule meetings for this account. Talk to the owner to get permission and try again.

Scheduling Teams Meetings as Delegate for another mailbox. - Microsoft Tech  Community

During testing we found that Teams was looking for a specific set of permissions on the owner’s mailbox when generating the Teams meeting in Outlook. In order to create a meeting as a delegate, the delegate must be a true ‘delegate’ under the owner’s mailbox AND have either ‘Author‘ or ‘Editor‘ access to the calendar like the example below. If the user is not present in this list or if they are present but show a calendar permission such as ‘Custom‘ in this dialog, they will receive an error like the one above when creating the Teams portion of the meeting even if they technically have all the required permissions to create a meeting on behalf of the organizer. Many times permissions are granted either by giving full access (Add-MailboxPermission) to the mailbox or by granting folder level permissions (Add-MailboxFolderPermission) and by granting Send-On-Behalf/SendAs permissions.

NOTE: If you are getting the error like the one below: ‘The user ‘Smith, John’ cannot be added. Non-local users cannot be given rights on this server‘ when trying to add an EXO user as a delegate to an on-prem mailbox you need to make the mailbox ACLable per this KB: A remote mailbox created in on-premises AD DS is not ACLable in Exchange Online.

Enabling Cross-premises delegate access - Microsoft Tech Community
Error encountered when adding an EXO user as a delegate to an on-prem mailbox

To understand why Teams requires such a specific configuration, let’s look at what is happening in the background… When you are creating a new Teams meeting via Outlook, the Teams back-end actually does a OAuth-authenticated EWS (Exchange Web Services) ‘GetDelegate‘ call to your on-premises Exchange environment to verify that the user creating the meeting actually has access to the mailbox. The request look like this:

<?xml version="1.0" encoding="utf-8"?>"
<soap:Envelope xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/"
               xmlns:t="http://schemas.microsoft.com/exchange/services/2006/types">
  <soap:Header>
    <t:RequestServerVersion Version="Exchange2016"/>
  </soap:Header>
  <soap:Body>
    <GetDelegate xmlns="http://schemas.microsoft.com/exchange/services/2006/messages"
                 xmlns:t="http://schemas.microsoft.com/exchange/services/2006/types"
                 IncludePermissions="true">
      <Mailbox>
        <t:EmailAddress>mailboxowner@domain.com</t:EmailAddress>
      </Mailbox>
    </GetDelegate>
  </soap:Body>
</soap:Envelope>

Exchange will respond with delegate list and associated permissions (similar to the view you see in Outlook).

<?xml version="1.0" encoding="utf-8"?>
<!-- Note: EwsEditor has replaced the "utf-16" text in the first line with"utf-8" in order for the XML to render in the response web control. -->
<s:Envelope xmlns:s="http://schemas.xmlsoap.org/soap/envelope/">
    <s:Header>
        <h:ServerVersionInfo MajorVersion="15" MinorVersion="1" MajorBuildNumber="1979" MinorBuildNumber="3" Version="V2017_07_11" xmlns:h="http://schemas.microsoft.com/exchange/services/2006/types" xmlns="http://schemas.microsoft.com/exchange/services/2006/types" xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" />
    </s:Header>
    <s:Body xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema">
        <GetDelegateResponse ResponseClass="Success" xmlns="http://schemas.microsoft.com/exchange/services/2006/messages">
            <ResponseCode>NoError</ResponseCode>
            <ResponseMessages>
                <DelegateUserResponseMessageType ResponseClass="Success">
                    <ResponseCode>NoError</ResponseCode>
                    <DelegateUser>
                        <UserId xmlns="http://schemas.microsoft.com/exchange/services/2006/types">
                            <SID>S-1-5-21-2061111111-1111111111-222222222-33333</SID>
                            <PrimarySmtpAddress>delegateuser@domain.com</PrimarySmtpAddress>
                            <DisplayName>User, Delegate</DisplayName>
                        </UserId>
                        <DelegatePermissions xmlns="http://schemas.microsoft.com/exchange/services/2006/types">
                            <CalendarFolderPermissionLevel>Editor</CalendarFolderPermissionLevel>
                            <TasksFolderPermissionLevel>None</TasksFolderPermissionLevel>
                            <InboxFolderPermissionLevel>None</InboxFolderPermissionLevel>
                            <ContactsFolderPermissionLevel>None</ContactsFolderPermissionLevel>
                            <NotesFolderPermissionLevel>None</NotesFolderPermissionLevel>
                        </DelegatePermissions>
                        <ReceiveCopiesOfMeetingMessages xmlns="http://schemas.microsoft.com/exchange/services/2006/types">false</ReceiveCopiesOfMeetingMessages>
                        <ViewPrivateItems xmlns="http://schemas.microsoft.com/exchange/services/2006/types">false</ViewPrivateItems>
                    </DelegateUser>
                </DelegateUserResponseMessageType>
            </ResponseMessages>
            <DeliverMeetingRequests>NoForward</DeliverMeetingRequests>
        </GetDelegateResponse>
    </s:Body>
</s:Envelope>

The important things here are that the user is present and has either ‘Author‘ or ‘Editor‘ permissions to the calendar. Remember, even having ‘Custom‘ permissions to the calendar will result in an error during meeting creation.

Another issue you can run into is one where you have already added the user as a delegate with the appropriate calendar permissions, but still receive the error ‘Looks like you don’t have permission to schedule meetings for this account. Talk to the owner to get permission and try again.‘ To troubleshoot this, let’s look at the mailbox from the perspective of Teams and perform the EWS ‘GetDelegate‘ call ourselves. We can use EwsEditor to do this.

In this example we receive an empty response from EWS even though we have properly configured the delegate permissions. This is indicative of mailbox corruption and to resolve this we have to understand how delegates are actually stored in the mailbox. Delegates actually translate to a number of things, but at the center of all of this is the ‘LocalFreebusy‘ object in the mailbox which can only be viewed with a tool like MFCMapi. The reason EWS is returning a blank list is because the object in the mailbox is corrupt and missing the delegate information. In order to resolve this we will use MFCMapi to delete the object. You can perform this on behalf of the user if you have full mailbox permissions and write access to their AD account, but in this example we will run as the mailbox owner using their default MAPI profile. To reset/repair the delegates list we will perform the following steps:

  • Before anything, record all of the users delegates from Outlook along with all the permissions for each one
  • Go to QuickStart -> Open Folder -> Local Freebusy
  • Choose the item with the subject ‘LocalFreebusy
  • Choose ‘Delete message‘ and when prompted with options choose ‘Permanent delete passing DELETE_HARD_DELETE…’
  • Go back into Outlook and re-create the user’s delegate(s)

After recreating the user’s delegates we can issue the EWS call again and see if they now show properly. The response should contain every delegate user along with their ‘Author‘ or ‘Editor‘ calendar permissions. Once you have verified EWS is returning the correct information you can try creating a new Teams meeting request.

Properly securing your on-prem Exchange 2016 environment when using Hybrid Modern Authentication

In the past many organizations completely blocked or limited external access to on-premises Exchange servers because of the lack of multi-factor authentication. Protocols like OutlookAnywhere (also known as RPC-over-HTTP, now MAPI-over-HTTP) and EWS had no native methods to accomplish multi-factor authentication. Failure to protect these protocols from external exposure has led to many breaches like FIN4 and London Blue.

HMA to the rescue… In 2017 Microsoft finally answered this deficiency with Hybrid Modern Authentication. I briefly touched on modern authentication in two previous articles (here and here). With Hybrid Modern Authentication Microsoft gave you the ability to use new technologies like modern authentication and conditional access for on-premises Exchange. Clients will connect using modern authentication by default once Exchange is on a supported version, supported clients are implemented, and the configuration is implemented. The issue here is that legacy Windows authentication is still available. You can simply disable modern authentication in the client or use a different client and you are now connected to on-premises Exchange with a simple username and password completely bypassing conditional access. Conditional access is only invoked when you are authenticating with modern authentication. Exchange 2019 implemented Authentication Policies which allow you turn off legacy authentication methods. If you are using Exchange 2019, you can use these to lock down your environment.

We were in the situation where we wanted to allow secure external access to Exchange (mainly for OutlookAnywhere, but also Outlook Mobile), but we couldn’t have any legacy authentication exposure. The solution we came up with was creating a set of externally facing Exchange 2016 mailbox servers (think Client Access Servers from the pre-Ex2016 days) that have all legacy authentication methods disabled (only OAuth available). These servers are the only ones exposed to the internet. The protocols we want to expose but lock down are ActiveSync (needed for Outlook Mobile), EWS (Exchange Web Services), MAPI, and OAB (Offline Address Book). To lock these down we ran the following against the externally facing servers:

$Servers = @(Get-MailboxServer excas01)
$Servers = $Servers + (Get-MailboxServer excas02)
$Servers | Get-ActiveSyncVirtualDirectory | Set-ActiveSyncVirtualDirectory -BasicAuthEnabled $false
$Servers | Get-WebServicesVirtualDirectory | Set-WebServicesVirtualDirectory -WindowsAuthentication $false
$Servers | Get-MapiVirtualDirectory | Set-MapiVirtualDirectory -IISAuthenticationMethods @('OAuth')
$Servers | Get-OabVirtualDirectory | Set-OabVirtualDirectory -WindowsAuthentication $false

After this is completed, Windows and basic authentication should now fail for these virtual directories.

IMPORTANT: It is VERY important to regularly check that these settings are still in place. You should always re-run these commands after any kind of Exchange update. If you do not do this, you could inadvertently expose your Exchange environment. A simple script could be run on a schedule to check and report on any changes to the authentication configuration of these virtual directories.

The second step is disabling or blocking the other virtual directories that do not need to be accessed externally. For us, these were ECP, OWA, PowerShell, and RPC. We have an on-premises load balancer with SSL bridging configured for our Exchange environment, so we used that to block access to these virtual directories. Another option is to use IP restrictions in IIS on these virtual directories. A third option is to disable the virtual directories via PowerShell. For those of you who want to allow secure access to OWA (Outlook Web Access) you can use Azure App Proxy to accomplish this or an ADC like NetScaler or F5 Big-IP.

The final step in this configuration is allowing the O365 servers to reach an unaltered version of EWS for the IntraOrganizationConnector used for Exchange Online to pull free/busy data (and other data like photos) from your on-premises environment. I found that for some reason the IntraOrganizationConnector fails to authenticate from EXO->on-premises when it uses the modified virtual directory even though all OAuth tests pass. I also use this configuration for my MRS endpoint when doing mailbox migrations since MRS wants to do traditional Windows authentication to EWS. If you are using the Microsoft Hybrid Agent, you shouldn’t have to do this since Azure App Proxy is taking care of the MRS and free/busy communication. I have still have an ongoing ticket open with Microsoft to understand the root cause of this. The workaround is fairly simple:

  • Create a namespace that can be used for EXO->on-premises communications. (Ex. exocomm.domain.com)
  • Configure this namespace to point to your regular INTERNAL and unaltered mailbox servers
  • Lock down this namespace in your firewall, so that ONLY Microsoft O365 servers can reach it. NOTE: This is very important and failure to do so will undermine all of the work done above and leave you exposed. We use a combination of PaloAlto firewalls and MineMeld to accomplish this, but this can be accomplished with a static/maintained ACL as well.
  • Configure the IntraOrganizationConnector in EXO to not use Autodiscover and to use this new namespace as its endpoint with the following commands:
Get-IntraOrganizationConnector | Set-IntraOrganizationConnector -TargetSharingEpr "https://exocomm.domain.com/ews/Exchange.asmx"
Get-IntraOrganizationConnector | Set-IntraOrganizationConnector -DiscoveryEndpoint $null

Outlook with ADAL + Hybrid Modern Authentication causing a white box and AADSTS500011 / 500011 errors in Azure AD

We are in the process of selectively turning on ADAL for Outlook clients. We have already gone through enabling Hybrid Modern Authentication for Exchange (https://docs.microsoft.com/en-us/exchange/configure-oauth-authentication-between-exchange-and-exchange-online-organizations-exchange-2013-help) a while back. We recently ran into an issue where specific users were getting a white box about a minute after launching Outlook. I have seen this issue where all of Outlook freezes, but this was not the same. They receive this error while Outlook continues to run in the background. The error is also accompanied by an Azure AD sign-in failure for the user. The error received is 500011. When looking this up in the documentation (https://login.microsoftonline.com/error?code=500011) you can see it is referring to the error ‘The resource principal named {name} was not found in the tenant named {tenant}‘.

I decided to do a Fiddler trace to get to the bottom of this and this is where the issue started becoming clearer. In the trace you see Outlook reaching out to autodiscover.domainname.com (which is on-prem), getting a 401 response, reaching out to login.windows.net/login.microsoftonline.com, and looping in this manner. This part of the capture aligned exactly with the mysterious white box.

In my case this specific set of users had a different primary SMTP address (and UPN) than the other users we had already enabled ADAL for and their autodiscover.domain.com URL was never added to our Azure AD service principals for the ‘Office 365 Exchange Online‘ application ID. Microsoft documentation talks about this in Step 5 of the link I added at the beginning of this post. Using the ‘MSOnline‘ PowerShell module I was able to add the URL to the service principal list.

$x = Get-MsolServicePrincipal -AppPrincipalId 00000002-0000-0ff1-ce00-000000000000
$x.ServicePrincipalnames.Add("https://autodiscover.domain.com/")
Set-MSOLServicePrincipal -AppPrincipalId 00000002-0000-0ff1-ce00-000000000000 -ServicePrincipalNames $x.ServicePrincipalNames

After adding the principal there were no more instances of the white box.

Creating group-based GPO without requiring a logoff/logon to take effect

As part of piloting O365 I was tasked with implementing hybrid modern authentication in our Exchange org in order to leverage functionality like the Outlook mobile application and MFA within the Windows version of Outlook for on-prem mailboxes. One caveat of enabling hybrid modern authentication in Exchange is that once this is flipped on any compatible client (ex. Outlook 2016) will begin using modern authentication (ADAL) exclusively by default. This switch can potentially be disruptive and we did not want to run into issues with the general user base. To do this we needed to disable modern authentication in Outlook on the client-side while being able to selectively enable it for certain users. This is easily handled with a ‘EnableADAL’ registry setting via GPO/Group Policy Preferences (GPP)/AD group. The issue is when you use an AD group with a group policy any member addition/removal needs to be coupled with a logoff/logon (or a reboot if it involves in a computer object in an AD group) to generate a new Kerberos token. I wanted to be able to quickly enable/disable ADAL for a user without requiring them to logoff/logon.

In order to get around this requirement I used GPP targeting with an LDAP query that looked for the group membership rather than standard group membership check. This LDAP query is completely dynamic and isn’t tied to the group list in user’s Kerberos token.

To do this you can do the following:

  • Create your GPP setting
  • Enable ‘Item-level targeting‘ on the setting
  • Create a new ‘LDAP Query‘ item
  • Create your filter using the distinguished name of your AD group and the ‘%LogonUser% variable
(&(objectCategory=user)(memberOf=GROUP DISTINGUISHED NAME)(sAMAccountName=%LogonUser%))
Create LDAP Query
Create LDAP Query condition
Retrieve group distinguishedName

This method could also be used for traditional GPO settings as well, but you’d have to use GPP to directly target GPO registry value(s) (ex. HKCU\Software\Policies\Microsoft\Windows\Control Panel\Desktop – ScreenSaveActive=0/1). This method could also be used for computer-based settings, but the LDAP query would have to be adjusted to target a ‘computerobjectCategory and the name of the computer (%ComputerName%). I wouldn’t use this method for everything, but can be very helpful for those one-off situations where you want a setting to take effect immediately without requiring a logoff/logon or reboot.

Journaling stops working after entering Exchange Online hybrid mode

We recently made changes to our on-prem Exchange org. Not long after we realized that any email flowing through Exchange Online to on-prem was not getting processed by our journaling configuration (per-database journaling). After digging and opening a case with Microsoft we found that Exchange Online was injecting this header:

X-MS-Exchange-Organization-Processed-By-Journaling: Journal Agent

This header tells Exchange that journaling was already processed. On-prem Exchange will then not process any journaling for that message. O365 apparently started injecting this header in the summer of 2018. The reason we did not run into the issue earlier is because until we were in hybrid mode (and ran the hybrid wizard) the Exchange header firewall was stripping this header as it arrived on-prem. They did release an article on this exact issue back in July 2016, but we didn’t come across it until Microsoft found the issue. The current fix is to duplicate all journal rules/settings in Exchange Online. According to Microsoft they are planning to add a warning in the hybrid wizard for this condition.