Windows Defender Causing iSCSI and Citrix VDA Issues

Over the last few months we have gone through a Defender pilot and rollout. I’ll admit I was surprised how relatively painless the rollout has been. Major problems were scarce and we have been able to enable most features for the general landscape. One feature that is not turned on by default that we decided to turn on was Network Protection. This features provides filtering and detection around web traffic. When we first enabled this in our pilot a number of us were on persistent Citrix VDIs and would randomly experience disconnects and reconnects to our machines. Sometimes this would go away for days or weeks and then randomly come back. Another issue we had was when we were trying to roll out Defender to some of our backup servers. These servers were physical servers with a number of iSCSI volumes mounted on them (using the Microsoft iSCSI Initiator). When the server would boot up they would hang at logon and the system event log would report a ton of iScsiPrt/Event ID: 9 errors around timeouts while trying to mount the storage. Both of these issues were immediately resolved simply by enabling asynchronous inspection. I have found almost nothing on the internet about this, but it resolved two major issues for us. This setting doesn’t seem to be exposed in GPO or other the other management methods and needs to be done manually on the machine. This setting can be enabled in PowerShell using this command:

Set-MpPreference -AllowSwitchToAsyncInspection $true

Note: In addition turning on asynchronous inspection you can obviously also disable network protection altogether while troubleshooting an issue.

The Microsoft documentation now briefly mentions this functionality, but states that it is on by default. We just had the issue within the last two weeks and needed to enable this on our backup servers. If I find anymore information on this I will post it here.

Running Sync-ModernMailPublicFolders.ps1 with Modern Authentication

As part of an Exchange Online migration I was re-running a Public Folder sync that I initially ran two years ago when we still had basic authentication enabled from internal networks for Exchange Online. When I went to re-run I realized it was not going to work because we completely disabled basic authentication. I had recently created an unattended script that connects to Exchange Online using MSAL and integrated Windows authentication (IWA) and was able to use this methodology in this script without making any modifications to the script.

To make this work we first need to install the MSAL.PS module:

Install-Module -Name 'MSAL.PS'

Next, we need to get an OAuth token for Exchange Online PowerShell, create a bearer token secure string, and create a basic credential from it along with our UPN. We will need to specify the client ID for Exchange Online PowerShell (612e1698-a44c-49a4-b66b-4188cc69cbaa) along with our tenant ID (I’m entering a fake one here). When prompted you will login with an account that has EXO administrative access:

$EXOToken = Get-MsalToken -ClientId 'a0c73c16-a7e3-4564-9a95-2bdf47383716' -TenantId '612e1698-a44c-49a4-b66b-4188cc69cbaa' -Scopes 'https://outlook.office365.com/.default'

$TokenSecureString = ConvertTo-SecureString "Bearer $($EXOToken.AccessToken)" -AsPlainText -Force

$Credential = New-Object System.Management.Automation.PSCredential($EXOToken.Account.Username, $TokenSecureString)

Now we can execute the script, but we will need to use an alternate ConnectionUri so that we can make EXO do the basic -> OAuth conversion:

.\Sync-ModernMailPublicFolders.ps1 -Credential $Credential -ConnectionUri 'https://outlook.office365.com/powershell-liveid?BasicAuthToOAuthConversion=true' -CsvSummaryFile:sync_summary.csv

The script should have executed without an issue. No need to re-enable basic authentication!

Exchange FIP-FS Scan Engine Update Issues: How to roll-back the update

UPDATE #3: There is the potential that after running Rollback-FIPFSEngine.ps1 that you may see the following error in the Application Event Logs:

Log Name:      Application
Source:        Microsoft-Filtering-FIPFS
Date:          01/04/2022 00:00:00
Event ID:      6027
Task Category: None
Level:         Error
Keywords:      
User:          NETWORK SERVICE
Computer:      exchange-server.domain.com
Description:
MS Filtering Engine Update process was unsuccessful to download the engine update for Microsoft from Custom Update Path.
Update Path:http://amupdatedl.microsoft.com/server/amupdate
UpdateVersion:0
Reason:"There was a catastrophic error while attempting to update the engine. Error: DownloadEngine failed and there are no further update paths available.Engine Id: 1 Engine Name: Microsoft"

If you are seeing the error above this could be an indication that there is an improper ACL on the ‘FIP-FS\Data\Engines’ directory. The Rollback-FIPFSEngine.ps1 script has been updated to set the proper ACL, but if you’ve previously run the script and need to resolve this issue you can run the following on your affected Exchange server(s) from an elevated PowerShell session:

$InstallFIPFSPath = "$($env:ExchangeInstallPath)FIP-FS"
$Sddl = 'O:SYG:SYD:PAI(A;OICI;FA;;;SY)(A;OICI;FA;;;LS)(A;OICI;FA;;;NS)(A;OICI;FA;;;BA)'
$NewSddl = Get-Acl -Path "$InstallFIPFSPath\Data\Engines"
$NewSddl.SetSecurityDescriptorSddlForm($Sddl)
Set-Acl -Path "$InstallFIPFSPath\Data\Engines" -AclObject $NewSddl

UPDATE #2: I have still been recommending this procedure since it doesn’t rely on deleting definitions and then waiting for the definition download from MS. With this workaround you are immediately in a fully working state.

If you have already used the Microsoft reset method on one or some server(s), you can use Rollback-FIPFSEngine.ps1 to copy definitions directly from the good server. This will speed up resolution time as the other servers will not have to wait for a definition download. Simply change the $BackupPathFIPFSPath variable to something like below and run on the other servers:

$BackupPathFIPFSPath = '\\GOODEXSERVER.domain.com\C$\Program Files\Microsoft\Exchange Server\V15\FIP-FS'

Note: Just make sure to re-enable auto-updates by running: Set-EngineUpdateCommonSettings -EnableUpdates $true

UPDATE: Microsoft has released a new engine update with a version number that resolves the issue. If you have already performed the fix above there is *no need* to perform the procedures in their article as you are already in a functional state. All you need to do is run Set-EngineUpdateCommonSettings -EnableUpdates $true to re-enable updates and your server(s) will download the latest update at the next check interval. Their script/procedure is for customers who are broken.


How to roll-back…

As almost anyone with an on-prem Exchange implementation probably knows there was an update released on 12/31/21 that caused email delivery issues globally. The core issue is the new version number is too large to fit in a long variable. Microsoft has acknowledged the issue, but the only current workaround is to disable the FIP-FS engine. The issue here is that certain transport rules also use this service, so if you have transport rules like these you may still have email delays even after disabling the engine. A better option would be to roll-back to the engine version that did not invoke the bug while Microsoft works this out.

I created a script (Rollback-FIPFSEngine.ps1) that makes the roll-back quick and easy. The one pre-requisite is that you need to restore your FIP-FS directory (ex. C:\Program Files\Microsoft\Exchange Server\V15\FIP-FS) from a restore. The restore should be from some point before the bad engine update. You can use the same restore for all Exchange servers assuming they are all running the same version of Exchange and have the same architecture (ex. amd64).

To preform the roll-back you must do the following:

  • Obtain a restore of the FIP-FS directory
  • Copy the Rollback-FIPFSEngine.ps1 script to the Exchange server(s) that need the rollback
  • Edit the $BackupPathFIPFSPath variable of the script to reflect the restored FIP-FS directory path
  • Open a new elevated (as Administrator) Exchange Management Shell PowerShell session on the server. Do not run this from a regular PowerShell session, it must be a Exchange session
  • Execute the script

The script should take a few minutes to execute and will do the following:

  • Re-enable FIP-FS. It doesn’t matter if you previously disabled FIP-FS using the Disable-Antimalwarescanning.ps1 script or Set-MalwareFilteringServer cmdlet. It will re-enable both
  • Globally disable FIP-FS engine updates (so that the problem updates do not return)
  • Stop necessary services (BITS, MSExchangeTransport, and MSExchangeAntispamUpdate)
  • Rename current engine files/directories
  • Copy engine files/directories from restore path
  • Start stopped services

After the script is run everything should be functioning normally. We’ve already run this in our environment and mail flow is now back to normal.

NOTE: After MS releases a fix you will need to re-enable updates by running: Set-EngineUpdateCommonSettings -EnableUpdates $true

FilteringServiceFailureException Error: Microsoft.Exchange.MessagingPolicies.Rules.FilteringServiceFailureException: FIPS text extraction failed with error: ‘WSM_Error: Scanning Process caught exception: (0x00000005) Access is denied

For some time we had been seeing these events in the event logs of our Exchange mailbox servers and the ‘UnifiedContent‘ directory (related to the Hub Transport role) has been growing:

Log Name:      Application
Source:        MSExchange Messaging Policies
Date:          10/26/2021 8:08:10 AM
Event ID:      4010
Task Category: Rules
Level:         Error
Keywords:      Classic
User:          N/A
Computer:      mbx1.domain.com
Description:
Transport engine failed to evaluate condition due to Filtering Service error. The rule is configured to ignore errors. Details: 'Organization: '' Message ID '<1ea41f5d-64ec-424a-b863-19d7fc2cf7d0@journal.report.generator>' Rule ID 'bcdf1c32-0249-4149-a91b-85ecabaeb695' Predicate '' Action ''. FilteringServiceFailureException Error: Microsoft.Exchange.MessagingPolicies.Rules.FilteringServiceFailureException: FIPS text extraction failed with error: 'WSM_Error: Scanning Process caught exception: 
Stream ID: <1ea41f5d-64ec-424a-b863-19d7fc2cf7d0@journal.report.generator>
ScanID: {E44453FB-B127-44F8-BEF0-357252C6DAA3}
(0x00000005) Access is denied.  Failed to open file: T:\TransportRoles\data\Temp\UnifiedContent\8bedad9e-130a-490e-be7a-af8a58758231'. See inner exception for details ---> Microsoft.Filtering.FilteringException: WSM_Error: Scanning Process caught exception: 
Stream ID: <1ea41f5d-64ec-424a-b863-19d7fc2cf7d0@journal.report.generator>
ScanID: {E44453FB-B127-44F8-BEF0-357252C6DAA3}
(0x00000005) Access is denied.  Failed to open file: T:\TransportRoles\data\Temp\UnifiedContent\8bedad9e-130a-490e-be7a-af8a58758231
   at Microsoft.Filtering.InteropUtils.ThrowPostScanErrorAsFilteringException(WSM_ReturnCode code, String message)
   at Microsoft.Filtering.FilteringService.EndScan(IAsyncResult ar)
   at Microsoft.Filtering.FipsDataStreamFilteringService.EndScan(IAsyncResult ar)
   at Microsoft.Exchange.MessagingPolicies.Rules.UnifiedContentServiceInvoker.TextExtractionComplete(IFipsDataStreamFilteringService textExtractionService, TextExtractionCompleteCallback textExtractionCompleteCallback, IAsyncResult asyncResult)
   --- End of inner exception stack trace ---
   at Microsoft.Exchange.MessagingPolicies.Rules.UnifiedContentServiceInvoker.GetUnifiedContentResults(FilteringServiceInvokerRequest filteringServiceInvokerRequest)
   at Microsoft.Exchange.MessagingPolicies.Rules.MailMessage.GetUnifiedContentResults()
   at Microsoft.Exchange.MessagingPolicies.Rules.MailMessage.GetAttachmentStreamIdentities()
   at Microsoft.Exchange.MessagingPolicies.Rules.MailMessage.GetAttachmentInfos()
   at Microsoft.Exchange.MessagingPolicies.Rules.MailMessage.get_AttachmentNames()
   at Microsoft.Exchange.MessagingPolicies.Rules.MessageProperty.OnGetValue(RulesEvaluationContext baseContext)
   at Microsoft.Exchange.MessagingPolicies.Rules.Property.GetValue(RulesEvaluationContext context)
   at Microsoft.Exchange.MessagingPolicies.Rules.TextMatchingPredicate.OnEvaluate(RulesEvaluationContext context)
   at Microsoft.Exchange.MessagingPolicies.Rules.PredicateCondition.Evaluate(RulesEvaluationContext context)
   at Microsoft.Exchange.MessagingPolicies.Rules.AndCondition.Evaluate(RulesEvaluationContext context)
   at Microsoft.Exchange.MessagingPolicies.Rules.RulesEvaluator.EvaluateCondition(Condition condition, RulesEvaluationContext evaluationContext)
   at Microsoft.Exchange.MessagingPolicies.Rules.TransportRulesEvaluator.EvaluateCondition(Condition condition, RulesEvaluationContext evaluationContext). Message-Id:<1ea41f5d-64ec-424a-b863-19d7fc2cf7d0@journal.report.generator>'

You may notice the ‘T:\TransportRoles\data‘ path above and this is due to the fact that we have our transport queue database path set to an alternate location. It is clear in the error that there is an access issue as it is is stating ‘(0x00000005) Access is denied. Failed to open file: T:\TransportRoles\data\Temp\UnifiedContent\8bedad9e-130a-490e-be7a-af8a58758231‘ as the core issue. Looking at the ‘Temp‘ directory ACL we saw the current permissions were:

  • LocalSystem – Full Control
  • Administrators – Full Control
  • NetworkService – Full Control

These permissions seem correct at face value, but when we look at the ACL of one of the files we actually found:

  • LocalSystem – Full Control
  • Administrators – Full Control
  • NetworkService – Full Control
  • LocalService – Full Control

If you look at a default Exchange installation you will also see the ACL above is how it is set. It seems that when using a non-default queue database location you are required to set the ACL yourself as it won’t be set automatically. After fixing the ACL we simply shut down the transport service, cleared the directory, and restarted the transport service:

Stop-Service MSExchangeTransport
Remove-Item -Path "T:\TransportRoles\data\Temp\UnifiedContent\*"
Start-Service MSExchangeTransport

After this change the ‘UnifiedContent‘ directories are no longer growing and the error we started with is no longer appearing in the event log.

Using Application Permissions (and client credentials grant flow) with Hybrid Exchange Graph API

We recently came across an application that uses Graph API and we wanted to start using it for some our on-prem mailboxes. Hybrid Graph API only supports delegated authentication flows and not application authentication flows. Just because something isn’t “supported” doesn’t mean you can’t make it work! There are two things that we’ll need to do to make this work.

First, any internet-facing Exchange server will need to have ‘V1S2SAppOnly‘ OAuth support added. You can do this by adding V1S2SAppOnly to the OAuthHttpModule.Profiles key in the REST web.config (ex. …\Program Files\Microsoft\Exchange Server\V15\FrontEnd\HttpProxy\rest). Once you add this value the key should look like this:

After this has been added either perform an iisreset or restart the ‘MSExchangeRestFrontEndAppPool‘ app pool in IIS for each server where you did this modification.

The next step is to add the appropriate ‘AppOnlyPermissions‘ to the Microsoft Graph partner application in Exchange/AD. First we’ll take a look at our Graph partner application:

Get-PartnerApplication | Where {$_.Name -like '*graph*'} | select *permissions* | fl

AppOnlyPermissions : 
ActAsPermissions   : {Mail.Read, Mail.Write, Mail.Send, Calendars.Read...}

The ‘AppOnlyPermissions‘ value should be blank. We need this to match the ‘ActAsPermissions‘ value. You’d think (and some other articles say) that you could just run Set-PartnerApplication -AppOnlyPermissions… but this was not a supported parameter for me. To set this we’ll have to edit AD directly. You’ll need to fire up something like ADSIEdit, load the AD configuration partition, and drill down to your Exchange org and partner application object. The path should be something like:

CN=Microsoft Graph,CN=Partner Applications,CN=Auth Configuration,CN=YOUR_EXCH_ORG,CN=Microsoft Exchange,CN=Services,CN=Configuration,DC=domain,DC=com

Once here you’ll need to open the ‘Microsoft Graph‘ object and copy the ‘msExchConfigurationXML‘ attribute value to the clipboard:

Next, we’ll use Notepad++ and the XML Tools plugin to manipulate this. MAKE SURE YOU BACKUP THIS VALUE. We want to convert this to a nicely formatted XML output so that it is easy to work with. To do this we use the ‘Pretty print’ option in XML Tools.

Once we have this we’ll need to duplicate all of the ‘ActAsPermissions’ lines and then use find and replace to convert those tags to be ‘AppOnlyPermissions’. Doing this will create a set of application permissions based on our delegated permissions (ActAsPermissions).

Once completed we need to linearize the output again so that we can copy it back into AD. We can use the ‘Linearize’ option in XML Tools for this:

Once we have the XML in the proper format we can put it back in the AD object:

Now that we’ve updated the AD object we can verify everything looks good by checking Exchange again (AppOnlyPermissions should have the same values as ActAsPermissions):

Get-PartnerApplication | Where {$_.Name -like '*graph*'} | select *permissions* | fl


AppOnlyPermissions : {Mail.Read, Mail.Write, Mail.Send, Calendars.Read...}
ActAsPermissions   : {Mail.Read, Mail.Write, Mail.Send, Calendars.Read...}

At this point you should be able to access on-prem Exchange resources using the supported Graph API functions with Application Permissions.

NOTE: One limitation of this is that application access policies (set in EXO) DO NOT apply and are ignored when accessing an on-prem mailbox.

Windows Autopilot with User-Driven Hybrid Azure AD Domain Join using Palo Alto GlobalProtect VPN: Part 2, using GlobalProtect PLAP with Basic Credentials

I recently had a call with another company attempting to setup Autopilot following my previous post (Windows Autopilot with User-Driven Hybrid Azure AD Domain Join using Palo Alto GlobalProtect VPN). While speaking to them I learned that are currently using basic credentials (LDAP+RADIUS) with GlobalProtect and are only attempting to setup certificate authentication to get Autopilot working. They were still planning on having the user perform two-factor basic authentication after the Autopilot-based deployment. This configuration was the perfect use-case for GlobalProtect’s new “Use Connect Before Logon” functionality. This functionality was introduced version 5.2 and works by registering a Pre-Login Access Provider (PLAP). With PLAP you now have interactive access to the GlobalProtect client at the logon screen. A huge plus with this method is that it requires NO back-end changes to your existing GlobalProtect configuration. The functionality is completely client-side and only really requires an additional step during installation. This PLAP functionality works with basic credentials, certificates, and even SAML! I will be using basic two factor credentials below.

The first step will be to create a new GlobalProtect package in Intune. I am using the newest version below, 5.2.7. You can use the same steps for creating the package that I laid out in my first post, but we will be using an alternate wrapper script, InstallGlobalProtect_PLAP.ps1. InstallGlobalProtect_PLAP.ps1, will install GlobalProtect, set our default GlobalProtect portal, and register the Pre-Login Access Provider (PLAP). Everything else non-certificate related in my original post will still apply (ex. IntuneHybridJoinHelperInstaller.ps1).

Once the machine has been deployed you will notice an extra button in the lower right. This is the PLAP.

When clicked, GlobalProtect will attempt to connect to the portal configured in the wrapper script and you will be presented with a screen like the one below. The prompts here will vary based on your authentication method. Here I am being prompted for my LDAP credentials to authenticate to the portal.

Once I passed the correct credentials here (and the correct second set of credentials at a second screen for two-factor) I was connected.

At this point you can click the ‘Back’ button and continue to log in to the device. That’s all there is to it! This is a great option for those of you who are lacking the desire to use certificates in your existing GlobalProtect configuration, but want to start using Autopilot.

Intune Android Enterprise Work Profile devices no longer launching dialer for telephone tel: links after installing Zoom

We recently came across an issue where after a recent Zoom app update anytime a user clicked a tel: link from within the work profile the Zoom app was launched instead of the native dialer. We first looked into what changed within the app in the release notes. It was obvious that a change in 5.6.3 caused this:

Support for tel: protocol links - Android
Users can tap a telephone number in an email or web page to call that number with Zoom Phone.

What is happening here is Zoom is now registered to handle tel: links and because it exists inside of the work profile it is taking precedence (the dialer is not really part of the work profile). This would have happened if we introduced ANY application that was registered to handle tel: links, but Zoom was the only one for us. What we need to do is bring the dialer into the work profile to give it precedence and to do that we use the ‘Android Enterprise system app‘ option to create instances of the native dialers in the environment. In our case we created one for the native (Google) Android dialer and one for the Samsung dialer. You may have other vendors in your environment so there may be others you need to add.

First, we need a target group of devices. We could simply deploy the applications to all devices and it will take effect where applicable, but it is better to be more targeted and specific when possible. We already have two Azure AD dynamic device groups that target our Android Enterprise devices. You can also get more specific using properties like deviceManufacturer to target specific vendor devices.

The first group is for our personally-owned work profile devices:

Name: dyn All BYOD-Enrolled Android Devices
Dynamic Membership rule: (device.deviceOSType -eq “AndroidForWork”)

The second group is for our corporate-owned devices (this technically will also target non-work profile Android Enterprise devices, but there is no harm in that):

Name: dyn All CORP-Enrolled Android Devices
Dynamic Membership rule: (device.deviceOSType -contains “AndroidEnterprise”)

Next, we need to create our dialer apps in Intune and assign them to our dynamic device groups. We are going to create one for the native Android dialer (com.google.android.dialer) and one for the Samsung dialer (com.samsung.android.dialer). The first time I attempted this I had an issue with the Samsung app installing because I had a capital ‘S’ in Samsung. Make sure the app id is all lowercase.

Once the new apps have have applied to the devices you will notice that the ‘Phone’ app shows inside the work profile app launcher. Once the ‘Phone’ app shows there you will be able to open tel: links in the dialer from within the work profile again.

TeamsFirewall – The open-source ethical firewall for Microsoft Teams

Some months back I was faced with an interesting challenge in Microsoft Teams. How do we prevent chats for users that have access to an external tenant? The background here is that certain industries like mine require you to capture and archive electronic communications. Depending on how you interpret the requirement this could include chats taking place in a tenant other than your own. An administrator naturally has zero access to the data in a tenant other than their own. You can use tenant restrictions to prevent a user (who you have some network or proxy control over) from accessing a tenant other than your own, but there are cases where a user needs this access for external collaboration. The challenge here was finding a way to prevent certain actions while working in another tenant… and so came TeamsFirewall.

I started analyzing Teams traffic in my free time and got a feel for how it works. I quickly realized that I could approach the issue with a scriptable proxy server that supported HTTPS. I chose mitmproxy for this task. In the first iteration of the product I broke traffic down to actions (like sending a message or deleting a message) and location (internal tenant or external tenant). After I had that working I wanted to expand the functionality to control actions based on not only location, but on other things like conversation type and participants. After more development I came up with a process that learns the environment by looking at action -> conversation -> participants. The system caches API tokens from the users it supervises to make requests on their behalf in order to learn what it does not know about the environment. The product does not need any credentials or direct access to a tenant to function. This data is then saved in the cache database (teamsfirewall_cache.db). The cache is used for lookups so that faster decisions can be made on the fly. Cache lifetime of both the user table and the conversation table can be configured via the config file.

An example of console output with core level logging set to debug

The rule engine of the product allows for extremely granular rulesets. You can get as granular as saying user A cannot edit messages sent to user B or as broad as user A/B/C cannot communicate with anyone @companyA.com. I would like to note that M365 Information Barriers perform some basic ethical wall functions, but it does not have much granularity and does not address examples like the the one above with external tenants.

Some next steps are improving the scalability of the product and developing an easily deployable package. I am currently looking at adding an option to use a central database for the cache database and using Docker containers with a load balancer to add more workers. You can see some of this upcoming work in the TODO file within the project repository (below).

All development and documentation will be maintained at the TeamsFirewall GitHub here: https://github.com/markdepalma/TeamsFirewall. Please feel free to post issues/questions and contribute!

Using an alternative address for Microsoft Bookings companies

UPDATE (11/2/22): I just noticed a parameter in OWA policies in EXO PowerShell the other day that allows you to set the default domain for Bookings calendars. I was able to set the default domain to bookings.company.com and it is taking effect for all newly created Bookings calendars. You need to run this for all OWA mailbox policies that apply to users who have access to create calendars. The command I used was:

Set-OwaMailboxPolicy -Identity OwaMailboxPolicy-Default -BookingsMailboxDomain bookings.company.com

During testing of Microsoft Bookings we found there was no way to adjust the sender email address for a Bookings company. This was frustrating as our default address was our default tenant address (which ended in onmicrosoft.com). This led me to ask how does this system even work? What is generating the emails? What hosts the email address?

Let’s start by looking for the object(s) hosting this service from an email perspective. To do this we can issue the command below in an Exchange Online PowerShell session. For each ‘company’ created in Bookings there is an associated mailbox created with the ‘RecipientTypeDetails‘ of ‘SchedulingMailbox‘. These mailboxes are not visible in the Exchange Online admin console.

Get-Mailbox -RecipientTypeDetails SchedulingMailbox

When the company is created, the user(s) with permissions to manage are added with ‘FullAccess‘ to the mailbox permissions. This is how Bookings company permissions are managed.

The mailbox is also configured to forward to the email address configured under ‘Business Information’ -> ‘Send customer replies to’.

There is also a Azure AD user account created for each company.

To avoid email address conflicts we decided we would create a new sub-domain for Bookings mailboxes called bookings.domain.com. To do this you would use the Microsoft 365 admin center to add a new domain. After this domain is added and proofed it can be used in Azure AD/O365. Make sure you configure on-premises Exchange routing for this new sub-domain if you are in a hybrid configuration. You can use the instructions under ‘Configure a group domain’ here to setup this routing.

Now that we have the new sub-domain we can use this in our Bookings company mailbox. The easiest way to do this is to adjust the UPN of the user account associated with the mailbox. This will automatically update the address.

After the Exchange Online directory syncs up with Azure AD the email address should be updated.

Now we need to unpublish and re-publish the company booking page so that it starts using the new address.

After re-publishing the page we need to wait a little bit for it to pick up the new email address. Eventually Bookings will start sending emails from the new email address. In testing it took around 10-15 minutes to take effect.

One more thing to note is that Bookings does not give you a method to delete companies after creating them. If you ever need to delete the company you simply need to delete the mailbox/user using the Remove-Mailbox cmdlet.

Looks like you don’t have permission to schedule meetings for this account: A deep-dive into Teams delegate meeting scheduling in a hybrid environment

After implementing Teams last year we were faced with a number of issues to work through. One of the issues we were having was the inability of mailbox delegates to schedule meetings on behalf of mailbox owners. If you are a hybrid Exchange user, the very first step is making sure you have OAuth configured between your on-premises environment and Exchange online. The Hybrid Configuration Wizard should be taking care of this for you. I’ve also written an article on properly securing it externally. You will also need create and configure a service principal to allow Teams to interact with your on-premises Exchange environment. Steps 2-3 in this article must be preformed to set this up. If one of these items are not configured, you will receive the error ‘Sorry, but we can’t connect to the server right now. Please try again later.

Teams delegate invitations through Outlook - Microsoft Community

There is already a lot of decent information around troubleshooting the OAuth side of things here, so I’m going to focus on the other pieces that aren’t always covered…

Once these pre-requisites are configured, you may find users are still receiving errors when trying to schedule Teams meetings on behalf of other users. An error that many of our users were experiencing was ‘Looks like you don’t have permission to schedule meetings for this account. Talk to the owner to get permission and try again.

Scheduling Teams Meetings as Delegate for another mailbox. - Microsoft Tech  Community

During testing we found that Teams was looking for a specific set of permissions on the owner’s mailbox when generating the Teams meeting in Outlook. In order to create a meeting as a delegate, the delegate must be a true ‘delegate’ under the owner’s mailbox AND have either ‘Author‘ or ‘Editor‘ access to the calendar like the example below. If the user is not present in this list or if they are present but show a calendar permission such as ‘Custom‘ in this dialog, they will receive an error like the one above when creating the Teams portion of the meeting even if they technically have all the required permissions to create a meeting on behalf of the organizer. Many times permissions are granted either by giving full access (Add-MailboxPermission) to the mailbox or by granting folder level permissions (Add-MailboxFolderPermission) and by granting Send-On-Behalf/SendAs permissions.

NOTE: If you are getting the error like the one below: ‘The user ‘Smith, John’ cannot be added. Non-local users cannot be given rights on this server‘ when trying to add an EXO user as a delegate to an on-prem mailbox you need to make the mailbox ACLable per this KB: A remote mailbox created in on-premises AD DS is not ACLable in Exchange Online.

Enabling Cross-premises delegate access - Microsoft Tech Community
Error encountered when adding an EXO user as a delegate to an on-prem mailbox

To understand why Teams requires such a specific configuration, let’s look at what is happening in the background… When you are creating a new Teams meeting via Outlook, the Teams back-end actually does a OAuth-authenticated EWS (Exchange Web Services) ‘GetDelegate‘ call to your on-premises Exchange environment to verify that the user creating the meeting actually has access to the mailbox. The request look like this:

<?xml version="1.0" encoding="utf-8"?>"
<soap:Envelope xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/"
               xmlns:t="http://schemas.microsoft.com/exchange/services/2006/types">
  <soap:Header>
    <t:RequestServerVersion Version="Exchange2016"/>
  </soap:Header>
  <soap:Body>
    <GetDelegate xmlns="http://schemas.microsoft.com/exchange/services/2006/messages"
                 xmlns:t="http://schemas.microsoft.com/exchange/services/2006/types"
                 IncludePermissions="true">
      <Mailbox>
        <t:EmailAddress>mailboxowner@domain.com</t:EmailAddress>
      </Mailbox>
    </GetDelegate>
  </soap:Body>
</soap:Envelope>

Exchange will respond with delegate list and associated permissions (similar to the view you see in Outlook).

<?xml version="1.0" encoding="utf-8"?>
<!-- Note: EwsEditor has replaced the "utf-16" text in the first line with"utf-8" in order for the XML to render in the response web control. -->
<s:Envelope xmlns:s="http://schemas.xmlsoap.org/soap/envelope/">
    <s:Header>
        <h:ServerVersionInfo MajorVersion="15" MinorVersion="1" MajorBuildNumber="1979" MinorBuildNumber="3" Version="V2017_07_11" xmlns:h="http://schemas.microsoft.com/exchange/services/2006/types" xmlns="http://schemas.microsoft.com/exchange/services/2006/types" xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" />
    </s:Header>
    <s:Body xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema">
        <GetDelegateResponse ResponseClass="Success" xmlns="http://schemas.microsoft.com/exchange/services/2006/messages">
            <ResponseCode>NoError</ResponseCode>
            <ResponseMessages>
                <DelegateUserResponseMessageType ResponseClass="Success">
                    <ResponseCode>NoError</ResponseCode>
                    <DelegateUser>
                        <UserId xmlns="http://schemas.microsoft.com/exchange/services/2006/types">
                            <SID>S-1-5-21-2061111111-1111111111-222222222-33333</SID>
                            <PrimarySmtpAddress>delegateuser@domain.com</PrimarySmtpAddress>
                            <DisplayName>User, Delegate</DisplayName>
                        </UserId>
                        <DelegatePermissions xmlns="http://schemas.microsoft.com/exchange/services/2006/types">
                            <CalendarFolderPermissionLevel>Editor</CalendarFolderPermissionLevel>
                            <TasksFolderPermissionLevel>None</TasksFolderPermissionLevel>
                            <InboxFolderPermissionLevel>None</InboxFolderPermissionLevel>
                            <ContactsFolderPermissionLevel>None</ContactsFolderPermissionLevel>
                            <NotesFolderPermissionLevel>None</NotesFolderPermissionLevel>
                        </DelegatePermissions>
                        <ReceiveCopiesOfMeetingMessages xmlns="http://schemas.microsoft.com/exchange/services/2006/types">false</ReceiveCopiesOfMeetingMessages>
                        <ViewPrivateItems xmlns="http://schemas.microsoft.com/exchange/services/2006/types">false</ViewPrivateItems>
                    </DelegateUser>
                </DelegateUserResponseMessageType>
            </ResponseMessages>
            <DeliverMeetingRequests>NoForward</DeliverMeetingRequests>
        </GetDelegateResponse>
    </s:Body>
</s:Envelope>

The important things here are that the user is present and has either ‘Author‘ or ‘Editor‘ permissions to the calendar. Remember, even having ‘Custom‘ permissions to the calendar will result in an error during meeting creation.

Another issue you can run into is one where you have already added the user as a delegate with the appropriate calendar permissions, but still receive the error ‘Looks like you don’t have permission to schedule meetings for this account. Talk to the owner to get permission and try again.‘ To troubleshoot this, let’s look at the mailbox from the perspective of Teams and perform the EWS ‘GetDelegate‘ call ourselves. We can use EwsEditor to do this.

In this example we receive an empty response from EWS even though we have properly configured the delegate permissions. This is indicative of mailbox corruption and to resolve this we have to understand how delegates are actually stored in the mailbox. Delegates actually translate to a number of things, but at the center of all of this is the ‘LocalFreebusy‘ object in the mailbox which can only be viewed with a tool like MFCMapi. The reason EWS is returning a blank list is because the object in the mailbox is corrupt and missing the delegate information. In order to resolve this we will use MFCMapi to delete the object. You can perform this on behalf of the user if you have full mailbox permissions and write access to their AD account, but in this example we will run as the mailbox owner using their default MAPI profile. To reset/repair the delegates list we will perform the following steps:

  • Before anything, record all of the users delegates from Outlook along with all the permissions for each one
  • Go to QuickStart -> Open Folder -> Local Freebusy
  • Choose the item with the subject ‘LocalFreebusy
  • Choose ‘Delete message‘ and when prompted with options choose ‘Permanent delete passing DELETE_HARD_DELETE…’
  • Go back into Outlook and re-create the user’s delegate(s)

After recreating the user’s delegates we can issue the EWS call again and see if they now show properly. The response should contain every delegate user along with their ‘Author‘ or ‘Editor‘ calendar permissions. Once you have verified EWS is returning the correct information you can try creating a new Teams meeting request.