As part of an Exchange Online migration I was re-running a Public Folder sync that I initially ran two years ago when we still had basic authentication enabled from internal networks for Exchange Online. When I went to re-run I realized it was not going to work because we completely disabled basic authentication. I had recently created an unattended script that connects to Exchange Online using MSAL and integrated Windows authentication (IWA) and was able to use this methodology in this script without making any modifications to the script.
To make this work we first need to install the MSAL.PS module:
Install-Module -Name 'MSAL.PS'
Next, we need to get an OAuth token for Exchange Online PowerShell, create a bearer token secure string, and create a basic credential from it along with our UPN. We will need to specify the client ID for Exchange Online PowerShell (612e1698-a44c-49a4-b66b-4188cc69cbaa) along with our tenant ID (I’m entering a fake one here). When prompted you will login with an account that has EXO administrative access:
For some time we had been seeing these events in the event logs of our Exchange mailbox servers and the ‘UnifiedContent‘ directory (related to the Hub Transport role) has been growing:
Log Name: Application
Source: MSExchange Messaging Policies
Date: 10/26/2021 8:08:10 AM
Event ID: 4010
Task Category: Rules
Transport engine failed to evaluate condition due to Filtering Service error. The rule is configured to ignore errors. Details: 'Organization: '' Message ID '<firstname.lastname@example.org>' Rule ID 'bcdf1c32-0249-4149-a91b-85ecabaeb695' Predicate '' Action ''. FilteringServiceFailureException Error: Microsoft.Exchange.MessagingPolicies.Rules.FilteringServiceFailureException: FIPS text extraction failed with error: 'WSM_Error: Scanning Process caught exception:
Stream ID: <email@example.com>
(0x00000005) Access is denied. Failed to open file: T:\TransportRoles\data\Temp\UnifiedContent\8bedad9e-130a-490e-be7a-af8a58758231'. See inner exception for details ---> Microsoft.Filtering.FilteringException: WSM_Error: Scanning Process caught exception:
Stream ID: <firstname.lastname@example.org>
(0x00000005) Access is denied. Failed to open file: T:\TransportRoles\data\Temp\UnifiedContent\8bedad9e-130a-490e-be7a-af8a58758231
at Microsoft.Filtering.InteropUtils.ThrowPostScanErrorAsFilteringException(WSM_ReturnCode code, String message)
at Microsoft.Filtering.FilteringService.EndScan(IAsyncResult ar)
at Microsoft.Filtering.FipsDataStreamFilteringService.EndScan(IAsyncResult ar)
at Microsoft.Exchange.MessagingPolicies.Rules.UnifiedContentServiceInvoker.TextExtractionComplete(IFipsDataStreamFilteringService textExtractionService, TextExtractionCompleteCallback textExtractionCompleteCallback, IAsyncResult asyncResult)
--- End of inner exception stack trace ---
at Microsoft.Exchange.MessagingPolicies.Rules.UnifiedContentServiceInvoker.GetUnifiedContentResults(FilteringServiceInvokerRequest filteringServiceInvokerRequest)
at Microsoft.Exchange.MessagingPolicies.Rules.MessageProperty.OnGetValue(RulesEvaluationContext baseContext)
at Microsoft.Exchange.MessagingPolicies.Rules.Property.GetValue(RulesEvaluationContext context)
at Microsoft.Exchange.MessagingPolicies.Rules.TextMatchingPredicate.OnEvaluate(RulesEvaluationContext context)
at Microsoft.Exchange.MessagingPolicies.Rules.PredicateCondition.Evaluate(RulesEvaluationContext context)
at Microsoft.Exchange.MessagingPolicies.Rules.AndCondition.Evaluate(RulesEvaluationContext context)
at Microsoft.Exchange.MessagingPolicies.Rules.RulesEvaluator.EvaluateCondition(Condition condition, RulesEvaluationContext evaluationContext)
at Microsoft.Exchange.MessagingPolicies.Rules.TransportRulesEvaluator.EvaluateCondition(Condition condition, RulesEvaluationContext evaluationContext). Message-Id:<email@example.com>'
You may notice the ‘T:\TransportRoles\data‘ path above and this is due to the fact that we have our transport queue database path set to an alternate location. It is clear in the error that there is an access issue as it is is stating ‘(0x00000005) Access is denied. Failed to open file: T:\TransportRoles\data\Temp\UnifiedContent\8bedad9e-130a-490e-be7a-af8a58758231‘ as the core issue. Looking at the ‘Temp‘ directory ACL we saw the current permissions were:
LocalSystem – Full Control
Administrators – Full Control
NetworkService – Full Control
These permissions seem correct at face value, but when we look at the ACL of one of the files we actually found:
LocalSystem – Full Control
Administrators – Full Control
NetworkService – Full Control
LocalService – Full Control
If you look at a default Exchange installation you will also see the ACL above is how it is set. It seems that when using a non-default queue database location you are required to set the ACL yourself as it won’t be set automatically. After fixing the ACL we simply shut down the transport service, cleared the directory, and restarted the transport service:
We recently came across an application that uses Graph API and we wanted to start using it for some our on-prem mailboxes. Hybrid Graph API only supports delegated authentication flows and not application authentication flows. Just because something isn’t “supported” doesn’t mean you can’t make it work! There are two things that we’ll need to do to make this work.
First, any internet-facing Exchange server will need to have ‘V1S2SAppOnly‘ OAuth support added. You can do this by adding V1S2SAppOnly to the OAuthHttpModule.Profiles key in the REST web.config (ex. …\Program Files\Microsoft\Exchange Server\V15\FrontEnd\HttpProxy\rest). Once you add this value the key should look like this:
After this has been added either perform an iisreset or restart the ‘MSExchangeRestFrontEndAppPool‘ app pool in IIS for each server where you did this modification.
The next step is to add the appropriate ‘AppOnlyPermissions‘ to the Microsoft Graph partner application in Exchange/AD. First we’ll take a look at our Graph partner application:
The ‘AppOnlyPermissions‘ value should be blank. We need this to match the ‘ActAsPermissions‘ value. You’d think (and some other articles say) that you could just run Set-PartnerApplication -AppOnlyPermissions… but this was not a supported parameter for me. To set this we’ll have to edit AD directly. You’ll need to fire up something like ADSIEdit, load the AD configuration partition, and drill down to your Exchange org and partner application object. The path should be something like:
Once here you’ll need to open the ‘Microsoft Graph‘ object and copy the ‘msExchConfigurationXML‘ attribute value to the clipboard:
Next, we’ll use Notepad++ and the XML Tools plugin to manipulate this. MAKE SURE YOU BACKUP THIS VALUE. We want to convert this to a nicely formatted XML output so that it is easy to work with. To do this we use the ‘Pretty print’ option in XML Tools.
Once we have this we’ll need to duplicate all of the ‘ActAsPermissions’ lines and then use find and replace to convert those tags to be ‘AppOnlyPermissions’. Doing this will create a set of application permissions based on our delegated permissions (ActAsPermissions).
Once completed we need to linearize the output again so that we can copy it back into AD. We can use the ‘Linearize’ option in XML Tools for this:
Once we have the XML in the proper format we can put it back in the AD object:
Now that we’ve updated the AD object we can verify everything looks good by checking Exchange again (AppOnlyPermissions should have the same values as ActAsPermissions):
I recently had a call with another company attempting to setup Autopilot following my previous post (Windows Autopilot with User-Driven Hybrid Azure AD Domain Join using Palo Alto GlobalProtect VPN). While speaking to them I learned that are currently using basic credentials (LDAP+RADIUS) with GlobalProtect and are only attempting to setup certificate authentication to get Autopilot working. They were still planning on having the user perform two-factor basic authentication after the Autopilot-based deployment. This configuration was the perfect use-case for GlobalProtect’s new “Use Connect Before Logon” functionality. This functionality was introduced version 5.2 and works by registering a Pre-Login Access Provider (PLAP). With PLAP you now have interactive access to the GlobalProtect client at the logon screen. A huge plus with this method is that it requires NO back-end changes to your existing GlobalProtect configuration. The functionality is completely client-side and only really requires an additional step during installation. This PLAP functionality works with basic credentials, certificates, and even SAML! I will be using basic two factor credentials below.
The first step will be to create a new GlobalProtect package in Intune. I am using the newest version below, 5.2.7. You can use the same steps for creating the package that I laid out in my first post, but we will be using an alternate wrapper script, InstallGlobalProtect_PLAP.ps1. InstallGlobalProtect_PLAP.ps1, will install GlobalProtect, set our default GlobalProtect portal, and register the Pre-Login Access Provider (PLAP). Everything else non-certificate related in my original post will still apply (ex. IntuneHybridJoinHelperInstaller.ps1).
Once the machine has been deployed you will notice an extra button in the lower right. This is the PLAP.
When clicked, GlobalProtect will attempt to connect to the portal configured in the wrapper script and you will be presented with a screen like the one below. The prompts here will vary based on your authentication method. Here I am being prompted for my LDAP credentials to authenticate to the portal.
Once I passed the correct credentials here (and the correct second set of credentials at a second screen for two-factor) I was connected.
At this point you can click the ‘Back’ button and continue to log in to the device. That’s all there is to it! This is a great option for those of you who are lacking the desire to use certificates in your existing GlobalProtect configuration, but want to start using Autopilot.
Some months back I was faced with an interesting challenge in Microsoft Teams. How do we prevent chats for users that have access to an external tenant? The background here is that certain industries like mine require you to capture and archive electronic communications. Depending on how you interpret the requirement this could include chats taking place in a tenant other than your own. An administrator naturally has zero access to the data in a tenant other than their own. You can use tenant restrictions to prevent a user (who you have some network or proxy control over) from accessing a tenant other than your own, but there are cases where a user needs this access for external collaboration. The challenge here was finding a way to prevent certain actions while working in another tenant… and so came TeamsFirewall.
I started analyzing Teams traffic in my free time and got a feel for how it works. I quickly realized that I could approach the issue with a scriptable proxy server that supported HTTPS. I chose mitmproxy for this task. In the first iteration of the product I broke traffic down to actions (like sending a message or deleting a message) and location (internal tenant or external tenant). After I had that working I wanted to expand the functionality to control actions based on not only location, but on other things like conversation type and participants. After more development I came up with a process that learns the environment by looking at action -> conversation -> participants. The system caches API tokens from the users it supervises to make requests on their behalf in order to learn what it does not know about the environment. The product does not need any credentials or direct access to a tenant to function. This data is then saved in the cache database (teamsfirewall_cache.db). The cache is used for lookups so that faster decisions can be made on the fly. Cache lifetime of both the user table and the conversation table can be configured via the config file.
The rule engine of the product allows for extremely granular rulesets. You can get as granular as saying user A cannot edit messages sent to user B or as broad as user A/B/C cannot communicate with anyone @companyA.com. I would like to note that M365 Information Barriers perform some basic ethical wall functions, but it does not have much granularity and does not address examples like the the one above with external tenants.
Some next steps are improving the scalability of the product and developing an easily deployable package. I am currently looking at adding an option to use a central database for the cache database and using Docker containers with a load balancer to add more workers. You can see some of this upcoming work in the TODO file within the project repository (below).
Back in April, at the beginning of the pandemic, I started putting a lot of focus into getting Windows Autopilot to work with Hybrid Join clients and Microsoft Always On VPN. I was looking at both for different reasons but also looking at them as a combined solution. The issue with Autopilot was that technically you were still required to have line of sight to a domain controller even though the domain join happened via an offline blob using the on-prem Intune connector. Sifting through logs I could see the only thing holding back a successful enrollment was a little function at the end of enrollment that was simply looking for a domain controller. I was able to sometimes get an enrollment to work via device tunnel MS VPN policies, but success wasn’t consistent and relied on policies/certificates coming down in a timely manner. In the logging I also saw references to a configuration parameter that would disable the DC check. Soon after, I found a post from Microsoft saying that they had this setting in private beta and would be releasing it in the coming months. After this I decided to put everything on the backburner and abandon MS VPN (I found the MS VPN solution using RRAS to be clunky and inconsistent with a lot to be desired).
Fast forward a few months and Microsoft finally released the new ‘functionality‘. At its core it is really just a flag telling OOBE not to perform a DC connectivity check. After enrollment is completed you are on your own to establish pre-login connectivity to facilitate an initial logon to your domain as there are no cached credentials yet on the machine. We are already a Palo Alto GlobalProtect customer and have been happy with the solution, so getting the two to work together just made sense. At the same time there has also been a push to implement a proper Always On VPN configuration. I’ll be writing a post dedicated to the full technical and security architecture around a cert-based Palo Alto Always On VPN configuration, so I’ll only briefly touch on the relevant parts here. Please refer to Palo Alto documentation on the missing pieces. There are a number of security aspects that should be taken into account like revocation, key storage, etc., and you should already have a proper certificate authority. I am not going over those in this article. You should also already have configured your Autopilot profiles, Intune Connector for Active Directory, etc. as per this document.
In this configuration I use a certificate-only approach (only using certificate profiles and no other authentication methods) for both the portal and the gateway. Remember, our first GlobalProtect connection after an Autopilot enrollment will be a pre-logon connection via certificate. There are no other authentication methods available for this first connection and the portal -> gateway authentication flow needs to support this. Before configuration of the portal and gateway you need to configure zones, interfaces, policies, and a certificate profile. These steps are documented here (steps 1-3 and 5-6).
Below is the portal config. Notice how there are no client authentication methods present. If you were to add any method here, it would be layered on top of the certificate authentication and would prevent the pre-logon connection. PAN-OS 9.0 has implemented mixed authentication support so that you can implement an either/or type of configuration here. The other important note is that your user connection (post-logon) will be connecting to the same portal/gateway and because of this we will be using certificates for the user as well.
The next step is to configure the agent settings within the portal config. Our config is being configured as Always On, but this is not technically required for Autopilot to work. If you do not want an Always On user connection, set ‘Connect Method‘ to ‘Pre-logon then On-Demand‘. Some Palo-Alto documents mention using multiple agent configurations for pre-logon and post-logon that use different connect methods, but this is not necessary here (and will not always work as expected due to the order of operations). The other important thing is to set ‘Client Certificate Store Lookup‘ to ‘User and Machine‘ so that the client will be able to use user and device certificate. The client seems to do a good job at using the proper certificate depending on if the connection is pre-logon or post-logon.
After the portal you will configure the gateway. Authentication will be identical to the portal to allow for a seamless authentication flow.
To deliver a device certificate to the device we will use an Intune PKCS certificate profile. I won’t go into great detail here as Microsoft has done a good job of documenting the steps involved. The profile will need to be assigned to the device properly, and the easiest way to do this is by using an Azure AD dynamic group. I am using a custom EKU value in the screenshot below (Extended Key Usage) which you do not need to replicate. I had a specific reason for doing this.
You will also need to deliver a user certificate to the device. You can use another Intune PKCS certificate profile to do this or you can use GPO/User Certificate Autoenrollment. I chose the latter because I like the granular control it provides. If you use an Intune profile, but target the machine, every user that logs on to the machine will get a certificate and VPN access. If I target the user, every Intune-managed machine they log on to will get a user certificate. Both of these were undesirable. With autoenrollment I’m only enabling autoenrollment for the computers I want (using a user policy and loopback processing), and I’m controlling the users that can enroll/autoenroll via the ACL on the certificate template itself.
The first wrapper script is InstallGlobalProtect.ps1. This is the one responsible for installing the MSI and pre-configuring some registry values. I thought this part would be very straightforward, but I had trouble getting the pre-logon credential provider to kick in initially when installing the client via Intune. If I manually installed the client, it worked the first time without an issue. It was only acting this way when being deployed via Intune during Autopilot. After a few hours of procmon traces and some reverse engineering of the client I figured out the issue. There is a post-setup process that runs that doesn’t process some registry changes correctly until the client is executed in the context of a user at least once. I use the wrapper to stage these two registry values (LogonFlag + LogonState) along with the others needed to make this configuration work. I also enable User-initiated Pre-Logon (via the ShowPrelogonButton value), so it gives the user a chance to verify they have internet connectivity and so that they can perform a retry of the pre-logon connection on demand. I later turn this off via GPO making pre-logon completely automatic after the first successful login. This value is totally optional.
The next wrapper script is a batch script that launches the script above. It is called InstallGlobalProtect.cmd. This script is needed because Intune will launch the installer in 32-bit mode and we want everything kicking off in 64-bit mode (mainly for the registry work above).
Once the three files are ready, we can create our package which generates a .intunewin file.
We then take this file and upload it to Intune to create our application. The application will need to be assigned to a group. Again, I am using a dynamic group that targets my Autopilot devices. As you can see below some of the MSI info is pulled in automatically because it was read by the Microsoft-Win32-Content-Prep-Tool utility. We must also change the ‘Install command‘ to point to the batch file we created earlier.
In testing I came across multiple issues due to machine GPO not being applied before the first login. One of these was loopback processing not applying which caused multiple user GPOs not to apply. Others were trusted root certs not installing (used for things like SSL decryption) and User Certificate Autoenrollment not working (I touched on this earlier). The trusted root issue actually caused my hybrid join to get stuck (SSL decryption is being used here). I decided to create IntuneHybridJoinHelperInstaller.ps1 to solve all of this.
Modifies the SACL of the directory to remove modify access from ‘Authenticated Users‘ (someone could use this directory to execute malicious code in the context of ‘LOCAL SYSTEM‘ if you do not do this)
Create a script in the directory above called IntuneHybridJoinHelper.ps1 with an accompanying scheduled task that executes at any user logon in the context of ‘LOCAL SYSTEM‘
At the next logon, this newly deployed script is triggered by the scheduled task, checking to see if the computer group policy cache has ever been provisioned (has ever received computer policies) and if not it will do the following:
Perform a gpupdate for computer policies
Get the interactive logged on user
Create a task to run gpupdate as the currently logged on user which will perform a gupdate of their user policies
Re-run ‘Automatic-Device-Join‘ task to complete the device registration in case it failed at logon
It is best to deploy this as a Win32 app, like the GlobalProtect client, so that we can ensure it is on the machine before the first logon. Like GlobalProtect, we are using a batch wrapper (IntuneHybridJoinHelperInstaller.cmd) to launch the PowerShell script as a 64-bit process. I used the same dynamic group that I used for the GlobalProtect client as the target here. I also used a dummy uninstall command since we never need to ever uninstall this. For install detection I am just using the script path (C:\Scripts\IntuneHybridJoinHelperInstaller.ps1). We will build our package using the utility like we did for GlobalProtect.
Now that we have everything in place we can test an enrollment. If everything is configured properly, you’ll be asked to sign-in to your corporate environment right after establishing network connectivity. After everything completes you should wind up at a logon screen. Because I am using User-initiated Pre-Logon I will need to switch to the GlobalProtect logon provider, click ‘Start GlobalProtect Connection’, and wait for the status to change to ‘Connected’.
After logging on you are presented with the User ESP (Enrollment Status Page). This is when our helper script kicks in to resolve GPO issues and moves our device registration along. This process can take a bit because after the ‘Automatic-Device-Join‘ completes you still have to wait for the on-prem computer object to sync up to Azure AD via AD Connect. Steve Prentice came up with a little script to help speed this up called SyncNewAutoPilotComputersandUsersToAAD.ps1. It just forces an AD Connect sync after computer object has its ‘userCertificate‘ attribute populated.
Once this is completed you should be left at a functioning desktop and GlobalProtect should have switched over to a full tunnel using the user certificate. At this point I would be using my primary endpoint management product, Ivanti Endpoint Manager, to perform any additional application installs/configurations. I have its agent being deployed via Win32 app as part of my Autopilot process.
In the past many organizations completely blocked or limited external access to on-premises Exchange servers because of the lack of multi-factor authentication. Protocols like OutlookAnywhere (also known as RPC-over-HTTP, now MAPI-over-HTTP) and EWS had no native methods to accomplish multi-factor authentication. Failure to protect these protocols from external exposure has led to many breaches like FIN4 and London Blue.
HMA to the rescue… In 2017 Microsoft finally answered this deficiency with Hybrid Modern Authentication. I briefly touched on modern authentication in two previous articles (here and here). With Hybrid Modern Authentication Microsoft gave you the ability to use new technologies like modern authentication and conditional access for on-premises Exchange. Clients will connect using modern authentication by default once Exchange is on a supported version, supported clients are implemented, and the configuration is implemented. The issue here is that legacy Windows authentication is still available. You can simply disable modern authentication in the client or use a different client and you are now connected to on-premises Exchange with a simple username and password completely bypassing conditional access. Conditional access is only invoked when you are authenticating with modern authentication. Exchange 2019 implemented Authentication Policies which allow you turn off legacy authentication methods. If you are using Exchange 2019, you can use these to lock down your environment.
We were in the situation where we wanted to allow secure external access to Exchange (mainly for OutlookAnywhere, but also Outlook Mobile), but we couldn’t have any legacy authentication exposure. The solution we came up with was creating a set of externally facing Exchange 2016 mailbox servers (think Client Access Servers from the pre-Ex2016 days) that have all legacy authentication methods disabled (only OAuth available). These servers are the only ones exposed to the internet. The protocols we want to expose but lock down are ActiveSync (needed for Outlook Mobile), EWS (Exchange Web Services), MAPI, and OAB (Offline Address Book). To lock these down we ran the following against the externally facing servers:
After this is completed, Windows and basic authentication should now fail for these virtual directories.
IMPORTANT: It is VERY important to regularly check that these settings are still in place. You should always re-run these commands after any kind of Exchange update. If you do not do this, you could inadvertently expose your Exchange environment. A simple script could be run on a schedule to check and report on any changes to the authentication configuration of these virtual directories.
The second step is disabling or blocking the other virtual directories that do not need to be accessed externally. For us, these were ECP, OWA, PowerShell, and RPC. We have an on-premises load balancer with SSL bridging configured for our Exchange environment, so we used that to block access to these virtual directories. Another option is to use IP restrictions in IIS on these virtual directories. A third option is to disable the virtual directories via PowerShell. For those of you who want to allow secure access to OWA (Outlook Web Access) you can use Azure App Proxy to accomplish this or an ADC like NetScaler or F5 Big-IP.
The final step in this configuration is allowing the O365 servers to reach an unaltered version of EWS for the IntraOrganizationConnector used for Exchange Online to pull free/busy data (and other data like photos) from your on-premises environment. I found that for some reason the IntraOrganizationConnector fails to authenticate from EXO->on-premises when it uses the modified virtual directory even though all OAuth tests pass. I also use this configuration for my MRS endpoint when doing mailbox migrations since MRS wants to do traditional Windows authentication to EWS. If you are using the Microsoft Hybrid Agent, you shouldn’t have to do this since Azure App Proxy is taking care of the MRS and free/busy communication. I have still have an ongoing ticket open with Microsoft to understand the root cause of this. The workaround is fairly simple:
Create a namespace that can be used for EXO->on-premises communications. (Ex. exocomm.domain.com)
Configure this namespace to point to your regular INTERNAL and unaltered mailbox servers
I have been working on implementing services like Azure AD and O365 in my work place. One of the services we are are adopting is Azure MFA. We currently use RSA SecurID company-wide for multiple remote access services and needed a way move users over in batches. One of the larger services to integrate Azure MFA with was Citrix NetScaler. Our original NSG (NetScaler Gateway) authentication configuration consisted of multiple LDAP policies and a set of RADIUS polices for RSA SecurID. Our goal was to create a configuration where we could control where users authenticate for their second factor via an Active Directory group. To do this kind of dynamic authentication in NSG we would have to move authentication from the basic model to an advanced nFactor-based configuration. The other requirement we had was to have a landing page before an Azure MFA authentication. We needed this because if your default sign-in method is set to ‘notification’ the RADIUS authentication could easily time out if the user doesn’t have their device running Microsoft Authenticator next to them. The page would let the user know to have their device ready before initiating authentication.
Below is a flowchart of the configuration end result. Objects of the same type share the same color (Gray – Authentication Virtual Server Profile, Purple – Authentication Virtual Server, Green – Advanced Authentication Policy, Orange – Advanced Authentication PolicyLabel, Red – Authentication Server (LDAP/RADIUS).
The first step in setting up Azure MFA is to stand up one or multiple NPS (Network Policy Server) instances and install the Azure MFA NPS Extension. I won’t go into the whole setup of this since it is documented, but I will comment on the policy config within NPS. It could be a little confusing because we are just going to pass the username to NPS, NPS will not be authenticating the user against AD. Azure MFA authentication in NPS happens AFTER NPS authenticates the user against AD. To make this work we will have to create a Connection Request Policy that just passes the user without authentication. The steps involved are as follows:
Create a new Connection Request Policy called ‘Default – Pass directly to Azure MFA‘
Add a Day and time restriction that covers 24 hours and 7 days
Set the ‘Authentication’ settings‘ to ‘Accept users without validating credentials‘. (This will not perform an AD authentication and just pass the username along to the Azure MFA NPS Extension)
Now we can add this new Azure MFA RADIUS server to NetScaler.
Go to ‘NetScaler -> System -> Authentication -> Basic Policies -> RADIUS‘
Create a new Server using the settings from the NPS server above
Next we will create our Login Schemas. We will need an initial logon schema for the LDAP credentials, a login schema for the RSA SecurID token, and a prompt only login schema for the page we want to show before authentication is passed to the NPS RADIUS server we setup earlier for Azure MFA.
Go to ‘NetScaler -> Security -> AAA – Application Traffic -> Login Schema‘
Create a login schema profile called ‘Username Password Login Schema‘
Set the authentication schema to ‘SingleAuth.xml‘
Set ‘User Credential Index‘ to 11
Set ‘Password Credential Index‘ to 12
Create a login schema policy called ‘Username Password Login Schema Policy‘
Set the profile to be the schema profile you just created above (‘Username Password Login Schema‘)
Set the ‘Rule’ expression to ‘REQ.HTTP.HEADER User-Agent NOTCONTAINS CitrixReceiver‘ (Receiver does not support the nFactor flow, this will be for web clients only)
Create a login schema profile called ‘Password Only Login Schema‘
Create a new XML file called ‘RSAOnly.xml‘ using the XML below
Upload this to ‘/flash/nsconfig/loginschema/LoginSchema‘
Set the user expression to ‘HTTP.REQ.USER.ATTRIBUTE(11)‘. (This will carry the username from the first logon page to the RSA logon page)
Create a login schema profile called ‘No Prompt Login Schema‘
Set authentication schema to ‘noschema‘
Leave everything else blank
Create a login schema profile called ‘Azure Confirmation Login Schema‘
Create a new XML file called ‘Azure.xml‘ using the XML below
Upload this to ‘/flash/nsconfig/loginschema/LoginSchema‘
Set the user expression to ‘HTTP.REQ.USER.NAME‘. (This will pass the UPN to NPS after acknowledging the prompt)
<?xml version="1.0" encoding="UTF-8"?>
<Requirement><Credential><Type>none</Type></Credential><Label><Text>Please have your Microsoft Authentcator app ready.</Text><Type>confirmation</Type></Label><Input /></Requirement>
Now we are going to create our Policy/PolicyLabel chain. The important thing here is to understand how they relate to each other. A Policy is exactly that… an authentication policy. You can define an expression and an authentication request server to use once the expression is matched. A PolicyLabel contains one or more Policy bindings with priorities, goto expressions (like NEXT/END), and a ‘Next Factor’ which gives you the ability to link to ANOTHER PoilcyLabel. First, we will start with our Policies.
Go to ‘NetScaler -> Security -> AAA – Application Traffic -> Policies -> Authentication -> Advanced Policies -> Policy‘
Create an LDAP authentication policy
Set Name to be specific to the domain it is servicing (ex. ‘DOMAINA LDAP Policy‘)
Set Action Type to ‘LDAP‘
Set Action to your existing LDAP server
Set Expression to ‘REQ.HTTP.HEADER User-Agent NOTCONTAINS CitrixReceiver‘
Create a Azure MFA RADIUS authentication policy
Set Name to ‘Azure MFA NPS Policy‘
Set Action Type to ‘RADIUS‘
Set Action to the NPS RADIUS server you created earlier
Set Expression to ‘true‘
Create a RSA RADIUS authentication policy
Set Name to ‘RSA Policy‘
Set Action Type to ‘RADIUS‘
Set Action to your existing RSA RADIUS server
Set Expression to ‘true‘
Create an Active Directory group that will contain the users you are cutting over to Azure MFA. I used the name ‘Enable Azure MFA 2FA Override’. We will use this group in the next two policies
Create the RSA check policy
Set Name to ‘RSA Group Check Policy‘
Set Action Type to ‘NO_AUTHN‘
Set Expression to ‘HTTP.REQ.USER.IS_MEMBER_OF(“Enable Azure MFA 2FA Override”).NOT‘
Create the Azure override check policy
Set Name to ‘Azure Override Group Check Policy‘
Set Action Type to ‘NO_AUTHN‘
Set Expression to ‘HTTP.REQ.USER.IS_MEMBER_OF(“Enable Azure MFA 2FA Override”)‘
Next we can create our PolicyLabels. These objects will reference Login Schemas/Policies and will chain together.
Go to ‘NetScaler -> Security -> AAA – Application Traffic -> Policies -> Authentication -> Advanced Policies -> PolicyLabel‘
Create the Azure PolicyLabel
Set Name to ‘Azure Auth PL‘
Set Login Schema to ‘Azure Confirmation Login Schema‘ (we created this earlier)
Set Feature Type to ‘AAATM_REQ‘
Add a policy binding for ‘Azure MFA NPS Policy‘ (we created this earlier)
Set Priority to 100
Set Goto Expression to NEXT
Leave Next Factor empty
Create the RSA Policy
LabelSet Name to ‘RSA Auth PL‘
Set Login Schema to ‘Password Only Login Schema ‘ (we created this earlier)
Set Feature Type to ‘AAATM_REQ‘
Add a policy binding for ‘RSA Policy‘ (we created this earlier)
Set Priority to 100
Set Goto Expression to NEXT
Leave Next Factor empty
Create the primary PolicyLabel
Set Name to ‘Azure_RSA Auth PL – START‘
Set Login Schema to ‘No Prompt Login Schema‘ (we created this earlier)
Set Feature Type to ‘AAATM_REQ‘
Add a policy binding for ‘RSA Group Check Policy‘ (we created this earlier)
Set Priority to 100
Set Goto Expression to NEXT
Set Next Factor to ‘RSA Auth PL‘ (we created this earlier)
Add a policy binding for ‘Azure Override Group Check Policy‘ (we created this earlier)
Set Priority to 110
Set Goto Expression to NEXT
Set Next Factor to ‘Azure Auth PL‘ (we created this earlier)
Now we are going to create an authentication vServer and profile.
Create the authentication vServer
Go to ‘NetScaler -> Security -> AAA – Application Traffic -> Authentication Virtual Servers‘ and click ‘Add‘
Set the name to ‘AD-RSA-Azure Auth vServer‘
Set ‘IP Address Type‘ to ‘Non Addressable‘. (This authentication vServer will only be used internally, so we don’t need an IP address)
Under Advanced Authentication Policies add a binding for the LDAP policy you created above (ex. DOMAINA LDAP Policy)
Set the priority to 100
Set the Goto Expression to NEXT
Set the Next Factor to ‘Azure_RSA Auth PL – START‘ PolicyLabel we created this above
Under Login Schemas add a binding for the ‘Username Password Login Schema Policy‘ policy we created this earlier
Create an authentication profile
Go to ‘NetScaler -> Security -> AAA – Application Traffic -> Authentication Profile‘ and click ‘Add‘
Set the name to ‘AD-RSA-Azure Auth vServer Profile‘
Set Authentication Host to ‘fake‘ (this won’t be used)
Set Virtual Server Type to ‘Authentication Virtual Server‘
Set the Authentication Virtual Server to the ‘AD-RSA-Azure Auth vServer‘ we created above
Leave Authentication Domain blank
Leave Authentication Level to set 0
Now that everything has been created all that is left is to configure the NetScaler Gateway Virtual Server. If you already have one setup you can simply remove all authentication profile bindings and add the Authentication Profile we just created above.
With everything created we can test the authentication flow…
This is the login flow when the user IS NOT a member of the ‘Enable Azure MFA 2FA Override‘ group.
This is the login flow when the user IS a member of the ‘Enable Azure MFA 2FA Override‘ group. Depending on what the Azure AD user’s ‘Default sign-in method’ is set to they may or may not receive a prompt after the 2nd prompt. If their default method is set to something like SMS or Authenticator App code they will receive a 3rd window asking for that code. This prompt is automatically generated, so we did not have to create it like we did for the RSA prompt. In the case below the user’s default method was set to ‘Microsoft Authenticator – notification’, so the login flow will wait until the prompt is acknowledged, denied, or times out on the user’s Microsoft Authenticator instance.
The last thing to mention is that a little extra configuration needs to be added to support Citrix Receiver clients (Windows/Mac/Android/iOS). In 11.1 52.x+ NetScaler AAA servers need to be configured with a separate set of polices for requests where the User-Agent header contains ‘CitrixReceiver’. It is fairly straightforward and Citrix has a write up on this, so I won’t write it all out. I will say the one thing to consider is that you will not be able to support multiple second factor methods using the AD group. For these clients you will have to choose one or the other. In our case we will be sticking with RSA until the majority of users are moved over meaning those users connecting Receiver directly to NSG will need to have that method available. Fortunately for us we do not have many users that access our environment this way. Citrix’s documentation on this can be found here: https://support.citrix.com/article/CTX223386.
In part two I covered how I was able to make use of the cameras by first attempting to crack a known root password hash (which wound up not working for this firmware) and then later re-flashing the camera with alternative firmware. Even though I accomplished my primary goal of making the Verizon-branded camera usable I still felt I could take this further by actually cracking the default admin credentials and avoiding the need to re-flash these cameras. I started by taking a closer look at the camera and its components. I had already identified and utilized the UART points, but hadn’t yet looked very closely at other items. One of the easiest things you can do is take FCC ID and look it up in the FCC database. I did this and although it didn’t provide me with anything especially helpful it was an interesting read. After this I decided to single out each component and research them. The picture below shows the front side of the main board. I’m particularly interested in the two chips in the top right corner of the board, but we’ll get there.
I started with the small 8-pin chip in the middle of the board. I had originally hoped this was a standard EEPROM chip as they are fairly easy to read, but taking a closer look it didn’t seem to be the case. The text read “ADE SLAVH“. I was unable to find anything on this, so it was clearly not an EEPROM.
I then moved to another 8-pin chip on the back of the board (next to the wireless interface module). After doing a little research on this chip it seemed to be a Renesas ISL 1208 RTC (real-time clock).
The next chip was one to the right on the front side of the board. It was a 16-pin chip, but I had trouble finding anything about this one. It was clear this wasn’t one that would be useful at this point, so I moved on.
The next chip I focused on was one of the two larger ones in the upper right of the front side of the board. This chip was labeled ESMT M12L128168A. It also had a matching partner on the direct opposite side of the board. When looking this up it wound up being a 16MB RAM chip. These two 16MB chips align with the 32MB of RAM in the boot output I observed via UART back in part one.
The last chip I moved to was one reading MX29LV320EBTI-70G. This one was sitting right next to the RAM chip above. A quick search revealed that this is actually a Macronix 4MB 48-PIN NOR parallel flash chip. This also aligned with the boot output from part one. During bootup the bootloader referenced a 4MB flash. This was the component to focus on.
Extracting a surface-mount device component
Now that I knew which chip was the flash chip it was time to attempt extracting it. I had never attempted the extraction of such a small component before, so I did some research into methods. I found that there are special low melting point alloys that are meant for just this type of task. I chose to go with SRA Fast Chip KIt SMD Removal Alloy. It was relatively cheap and available on eBay. Another popular choice for this task is Chip Quik which looks to be identical, but slightly more expensive.
To use these products you take the included flux (or a regular rosin flux) and cover all pins on the chip. You then melt a small amount of the alloy and cover all pins with it. Once everything is covered you try to keep the alloy melted by running the iron over the alloy with it while carefully extracting the chip with tweezers.
Once the chip has been pulled you can clean up the board with the iron (if you are planning to re-solder a chip there). The chip was a little more tedious to clean up. I carefully ran the iron’s flat tip over the pins while slowly dragging the chip across cardboard. The alloy came off in small streaks with each pass. The chip needs to be free of any bridging in order to be read properly.
Reading a NOR parallel flash chip
Reading a parallel flash chip can be a bit difficult. There are a lot of pins to communicate with and it all has be done in a specific order. Communicating with something like a standard 8-pin EEPROM is much easier. I was able to find someone using an Arduino to read a TSOP-48 NOR flash chip like this one, but I decided to take an easier route as I had already put a good amount of time into this project. I looked into different flash readers/writers and came across the FlashcatUSB xPort with the TSOP-48 adapter. It was relatively cheap and supported this chip along with many many others. I’m sure I’ll be able to make use of it in the future. Setup was easy and I was able to successfully pull an image from the chip.
Now that I had a binary image of the chip I could start digging for what I was after… the default username/password for the admin console. In part one we learned the partition layout from the bootup output:
The “CONFIG” partition is usually what holds the persistent device configuration in plain-text on these types of devices. I loaded the image into hexedit and started working my way down from 0x00010000… and there it was! Some developer at Verizon has a sense of humor. I finally had what I really wanted from the beginning… the default admin console username/password. I tested it immediately and it worked. I also verified that once in the console I could change the username/password along with any other settings and they persisted a reboot. The admin console was almost identical to the default Sercomm web console. All the same configuration options were present.
I also extracted the Squashfs filesystem for examination. I verified that the root password was definitely not the manufacturer default that I had cracked in part two. It is actually generated during the boot process. I may possibly go into breaking this down in a part four post, but for now I accomplished exactly what I set out to do. I obtained the default admin credentials for a camera that Verizon was trying to keep you locked out of. The cameras are now fully functional as-is no flash required.
In the last post I went over obtaining terminal access to the camera via the UART connection points on the main board. One I had terminal access I was still presented with a login that I needed to get past. With no other way at the time to get this I started doing more research on this and the other related camera models I mentioned previously. I was hoping to find firmware that could lead me to a static root password. I was unable to find any export of the Verizon firmware, but I did find others. With some Google searches I stumbled across unpublished links for source-code builds on the Sitecom web site for both the WL-404 and LN-406 cameras. The two archives I used were under: https://www.sitecom.com/documents/. Let’s take a look at GPL _ LN-406_WL-404 _ fw1_0_11.zip. If we dig into ZIP we can a lot of source code, but more importantly we see an /etc/passwd file. The contents of this file were:
The string we are concerned with is ‘9szj4G6pgOGeA‘. Taking a closer look at the string we see It is an old descrypt crypt(3) Unix password hash. The password itself has a maximum length of eight characters combined with a two character salt (‘9s‘). If you Google this hash you find that it actually is common among many of the camera models I mentioned in part 1, but I was unable to find any instance of this hash actually being cracked. Looking at the firmware it seemed that this was based on a core source code distribution that originally came from Sercomm (the actual manufacturer of the camera). I figured that there was a high probability of this being actual root password on the camera I was trying to crack. At this point I decided that cracking the root password may be the easiest way to get the web console admin password and get access to the camera.
Cracking a crypt(3) Unix Password
I had never attempted to crack this algorithm before and knowing the password wasn’t overly simple (as others have attempted to crack it) I knew I’d need external resources to accomplish this. I started off by calculating the approximate number of password combinations I’d have to go through. I know nothing about the password, so I’d at least have to assume it could have all printable characters with a length between one and eight characters. The math for this would be: 93^1 + 93^2 + 93^3 + 93^4 + 93^5 + 93^6 + 93^7 + 93^8 = 5656642206396600 combinations. I ran a quick Jonn the Ripper benchmark on a decent Azure VM (no GPU) and the descrypt performance on that box was only around 16385000 c/s. At that rate it would take ~11 years to crack. I found the most powerful video card I had lying around which was an NVIDIA GTX 750Ti (forgive me, I am not a gamer). The Hashcat benchmark for this card cracking descrypt was around 28989000 c/s. This was significantly faster, but still would take ~6 years. I started researching video cards and found that one of the more popular cards, the NVIDIA 1080 GTX Ti, had a descrypt crack rate of around 1227000000 c/s. At this rate it would take around ~53 days. I had a colleague who actually had this card at home and I figured I’d make a deal with him to run this crack for me during idle time if I couldn’t find anything else. I started researching heavy GPU cloud systems and the most powerful (GPU-wise) I found was the Amazon p3.16xlarge configuration which used EIGHT NVIDIA® V100 Tensor Core GPUs. These GPUs are commonly used in the machine learning space and are a bit more powerful than the top gaming GPUs. An instance with this configuration has a descrypt crack rate of around 15571600000 c/s and could crack the password in around ~4 days, but would cost $2000 for the privilege. As a last resort when looking at computing rentals I turned to Vast.ai. Playing with different GPU/server rental combinations I found the password could be cracked somewhere around $250, but I couldn’t get it any lower than this. Doing further research I came across and interesting site called Crack.sh which was founded by David Hulton (also a founder of ToorCon). He created a purpose-built system that uses a series of FPGAs and specializes in cracking DES. They rent it out for much less than any other cracking service comes close to and even offer some services for free. Because this system was specifically created for cracking DES it actually can crack a descrypt password in ~3 days which is even faster than the Amazon configuration I spec’d out. I decided to eat the cost in the name of the project and figured the cracked password would be useful to others as well. Once my crack job actually started after sitting in the queue for a while I had the password in around three days. The email read:
Crack.sh has found a password that works against your hash. The password match is included below both in ASCII and HEX:
Password (ASCII): "h@11oCAM"
Password (HEX): 684031316f43414d
This run took 313726 seconds. Thank you for using crack.sh, this concludes your job.
The password was a bit comical. I took the first instance of idle time I had, remotely connected to my lab environment, and attempted getting into the camera’s shell using the password. To my dismay I found that the password DID NOT WORK on this camera. I felt somewhat defeated at this point, but I had known when going this route that there was some percentage chance that the password was in the Verizon firmware was not the same as the other cameras using the same platform. Cracking what was LIKELY the root password was sort of an easy way out.
Flashing New Firmware
Since I had multiple versions of usable firmware (with a default web console password I actually knew) I decided to look into somehow flashing alternate firmware. I forgot to mention this in part 1, but during discovery I found that holding the reset button and powering the camera on put the camera into a download mode. I was able to see this through the UART connection. When in this mode the output showed this:
The bootloader version 4.09
Flash Size = 4M(8K x 8,64K x 63)
MAC address 00:0e:8f:7a:34:83
MAC address 00:0e:8f:7a:34:83
MAC address is 00:0e:8f:7a:34:83
DAVICOM driver ready
got PID in flash
The reason I didn’t further explore this was because at time I didn’t see how there was a way to upload this firmware. When in download mode the camera did not have an IP address and the UART connection didn’t seem to accept any input in this mode. What I didn’t realize at the time was that it was able to accept a firmware download over network in this mode, but not via TCP/IP. Further research led me to an old Sercomm tool that was used to flash a number of Sercomm-based products. I had seen references to this tool earlier on, but knowing it was a network-based utility I didn’t see how it could be used for this device. The filename/download for this tool is: Upgrade_207_XP.zip. The tool required a 32-bit OS and I didn’t have any instances running, so I decided to quickly spin up a Windows 98 VM. To my surprise the application immediately discovered the camera on the network.
I was curious what protocol this was using since it was not using IP, so I used the span port I setup earlier when I was performing the initial reconnaissance on the camera to sniff this traffic.
After the firmware update I was FINALLY able to get into the camera’s admin console making the camera completely usable. Though I technically accomplished my original goal I was not happy about spending time on the password crack that turned out to be a dead end and I still did not have or know the default admin console password (or root password) that the Verizon firmware was using. Also, my buddies would need to open and manually flash every camera before attempting to sell them. True to form I could not let it be and I will cover this in part 3!