Disabling Microsoft Office 365’s Exchange Web Services (EWS) Throttling
What are the Exchange Web Services (EWS) in Office 365 and why do they get throttled?
In the mid-2000’s, Microsoft released an Application Programming Interface (API) for the Microsoft Exchange 2007 mail platform known appropriately as Exchange Web Services (EWS). This API allowed various software applications to run requests from an Exchange server via commonly used web protocols like Secure HyperText Transfer Protocol (HTTPS) and Open Authentication (OAUTH), making widespread enhancement and adoption of far faster and more secure applications available to the masses.
Exchange Web Services offers a slew of different options and capabilities related to being able to access items contained within a mailbox, such as the email messages, calendar appointments and even the digital rolodex of Contacts. Lesser known functionalities includes the Availability services that can show if a user is active in using Outlook or OWA at the moment, as well as being able to configure the Out of Office (OOO) wizard when setting up automatic replies.
When lots of requests are made to an Exchange Server using EWS for an individual mailbox, an automatic throttling policy will kick in that can reduce the throughput (by limiting command and response action time) or postpone the requests and time out the connection outright. In these scenarios, certain processes that import, read or export data from Office 365 can fail. Take note that the issue here is limited to a single mailbox receiving too many requests; a single mailbox can fail but all others in the tenant may behave normally.
That single mailbox may fail or time out on an operation, which is known as throttling, for a number of reasons, but it is most often due to the sheer quantity of data held inside of it that is being read by an EWS-based application or being moved to it, in terms of emails, calendar appointments, Teams messages or even the number of folders. Microsoft has a hidden “budget” for how many Input/Output (I/O) requests that can be played against a single mailbox within a period of time for users held in Office 365; if a mailbox exceeds this budget, requests over EWS to it will get canceled or time out until that budget increments back up to a “usable” amount to complete the operation. Subsequent commands to the mailbox will still run, but may not make much progress due to the budget not having increased back to a sufficient quantity yet and will time out in short order.
Such throttling is instituted by Microsoft in order to prevent any single tenant from consuming an overwhelming share of resources and starving other tenants for performance. In most situations, this budget is never a factor, but we all know “that user” who holds onto every message or has subtree after subtree of folders with a handful of messages contained therein; this article is designed to help deal with the misbehaviors that can come up from those users using a tool that Microsoft provides for these exact scenarios.
Note: While EWS throttling exists in local Exchange deployments also, the commands to change the limits there are different and not within the scope of this article.
Note: Microsoft is largely moving EWS over to its successor application, Graph API, in Office 365, hybrid environments and possibly newer Exchange builds in the future. EWS is still supported at the time of the writing of this article, but as more applications transition to Graph as their ‘gateway’ to access data held in Office 365, this article may get superseded by any newer articles discussing Graph instead.
How does DataCove use EWS?
In DataCove’s context, EWS is used in a few different locations:
The Exchange Crawler uses EWS for logging into user mailboxes and reading their emails for import.
The Folder Synchronization process uses EWS for logging into user mailboxes and determining where archived/imported emails exist inside of their folder structure so that DataCove can mirror that for the My Mailbox end user view.
The Teams Archiving feature uses EWS for logging into user mailboxes and reading their Teams chats for import.
In most environments, this is a non-issue since most EWS throttling is handled on a per-mailbox basis, rather than a per-tenant basis, but for user accounts with utterly enormous amounts of data in them, timeouts may occur and can delay the data ingestion and synchronization processes.
Most of these functions are being transitioned over to Graph API or have already been moved, but some subservices of Graph may still rely on EWS functionality in the background as Microsoft rebuilds their command set to entirely remove EWS from the equation. In these scenarios, Tangent has seen enhanced performance from these processes by executing the Microsoft throttle bypass.
Temporarily Disabling EWS Throttling for Faster Throughput
Microsoft encounters this mailbox throttling situation often enough that they’ve created an automated tool that not only verifies if Exchange Web Services (EWS) operations have been throttled in the tenant, but also allows for an automatic implementation for a bypass of the throttling policy for 30 to 90 days, at the O365 Administrator’s discretion. Depending on much data is being imported or read for the first time, using this bypass can save a significant amount of time in getting data synchronized or imported to DataCove.
Begin by opening a web browser and navigating to Admin.Microsoft.Com.
Log in with administrative credentials to the Office 365 tenant.
Once in, select the navigation menu on the left hand side of the screen and select Show All (if this is not already part of the configured view).
Locate the Support menu and click on it to see the options it provides.
Select Help & Support.
Despite the name of Help & Support, this page will actually land on the View Service Requests page, since that is where Help requests are tracked.
However, it will open a How Can We Help pane on the right hand side of the screen, which is where our next step is performed.
In the search box under the How Can We Help page, type in Increase EWS Throttling Policy.
An automatic text finisher will help guide the command to the proper one seen below.
Once the command is selected, click on the Run Tests button.
The diagnostic battery to determine if your O365 tenant is being throttled over EWS will now launch and take about 20 seconds to complete.
Once the test battery finishes, a report will appear specifying whether or not the tenant is being throttled.
If the report comes back as affirmatively stating that the tenant is being throttled, proceed with the next steps.
If the report comes back with a statement that the tenant is not being throttled, any delays being experienced are not related to EWS and the next steps are not likely to help the problem being encountered. Contacting DataCove Support would be the recommended procedure if this occurs.
When EWS is detected by the diagnostics as being throttled, Microsoft provides a temporary bypass option to allow for greater resources to be allocated to the tenant in order to get through the work needed.
This bypass can be set in terms of a duration of days, with 30 days, 60 days and 90 days being available in a dropdown menu box a little further down the pane.
Select the desired duration necessary (30 days is usually enough for most organizations, but a longer period won’t cause any harm), check the Acknowledgement Box and then click on the Update button.
A scrolling progress bar will now appear (incorrectly labeled as the same “running diagnostic tests” from a prior step; Tangent has lodged a ‘correction’ ticket with Microsoft to fix their text on this) as the change gets implemented.
Once the change takes place, a “Success” notification will appear advising that the change has gone through successfully.
After approximately 15 minutes, then bypass policy will go live and EWS transactions should become a bit heartier in just how much traffic is allowed to be passed around.
If after the assigned duration the data import or synchronization processes are still not complete, performing this process again may be necessary in order to allow more ‘high throughput’ time with Microsoft.
It is recommended to consult with DataCove Support in the event of such a scenario to discuss other options at that time, if need be.