Troubleshooting BizTalk Performance Issues (Intro to the Throttle)
I’ll admit, before we started this, I didn’t even know BizTalk had a Throttle. I suppose it makes sense, it’s a feature of an expensive enterprise service bus that it’s “clever” in knowing about the state of resources around it and reacting accordingly.
Here’s what Microsoft have to say about BizTalk Throttling (from MSDN):
The BizTalk Server host throttling mechanism continually monitors for a throttling condition, calculates the severity of the throttling condition, and applies host throttling progressively depending on the calculated severity. The throttling mechanism is self tuning and the default configuration options are suitable for the majority of BizTalk Server processing scenarios. BizTalk Server host throttling exposes several configurable options that can be used to tune throttling for specific scenarios.
If you don’t know about it, it can appear that BizTalk is hanging, running slowly, or in some other way underperforming. You might even start to consider whether it is capable of handling the level of traffic you’re giving it.
It turns out that it almost certainly is capable, but for one reason or another it’s decided to slow itself down. When this happens (or you think it has happened), you need to know why: as you have a performance bottleneck somewhere in your system that at worst you need to know about, and at best, iron out.
There are loads of useful perfmon counters for BizTalk, but I’m going to concentrate on the following ones:
- BizTalk:MessageAgent
- DatabaseSize
- Message delivery incoming rate
- Message delivery outgoing rate
- Message delivery throttling state
- Message delivery throttling state duration
- Message publishing delay (ms)
- Message delivery delay (ms)
- BizTalk:MessageBox:General Counters
- Host Queue – Length
These are the ones I’m using at the moment, but there are plenty of others that can help you work out what’s going on.
The Message delivery throttling state is a good one to start with. Rather than being a scalar representation of some rate or value, this is one of a set of (I think) 7 values:
- 0: Not throttling
- 1: Throttling due to imbalanced message delivery rate (input rate exceeds output rate)
- 3: Throttling due to high in-process message count
- 4: Throttling due to process memory pressure
- 5: Throttling due to system memory pressure
- 9: Throttling due to high thread count
- 10: Throttling due to user override on delivery
So, if the value is 0 then no throttling is taking place. The Message delivery throttling state duration counter shows you how long (in seconds) BizTalk has been in the current throttling state. So, a well-running, happy system might look like this:
Any value other than 0 indicates some sort of throttling.
I don’t (personally) think that throttling always indicates a problem with a BizTalk setup or implementation. For instance, temporary throttling due to an imbalanced message delivery rate which lasts for a small amount of time and then subsides is just a good example of throttling stopping a problem from getting worse. Or it might be that BizTalk is catering for a bottleneck further down the line (for instance a slow receiving host) and is managing system flow to prevent problems.
However, some of the memory throttling states probably warrant some investigation if they are happening frequently.
When throttling is active, BizTalk literally slows down the rate at which it processes messages. You can see this happening by monitoring the Message publishing delay (ms) and Message processing delay (ms) counters.
Altering Throttling Conditions
You can adjust how, why and when BizTalk throttles, away from the default settings. I think you should be confident that you have a particularly unusual setup, or a good knowledge of BizTalk before going down this route: the defaults are presumably the result of lots of research and should probably be respected.
Rate-Based Throttling Settings
Right click the Host Instance and select Settings. Click the Host tab. There are 3 different tabs, for Resource-based, Rate-Based and Orchestration Throttling. Roughly: rate-based throttling is applied if there is a mismatch between the incoming and outgoing message speeds, resource-based identifies problems with threads, memory and database size, and orchestration-based allows you to define when/if dehydration occurs and subscription pausing/resuming. Each tab has specific sliders to alter how the throttling engine performs.
I’m the very first to admit I’m not the authority on BizTalk and it’s Throttling Engine, so here’s some of the MSDN pages I’ve found useful in understanding, identifiying and diagnosing our own particular bottleneck problem, which manifested itself initially in BizTalk throttling:
MSDN: How BizTalk Server Implements Host Throttling
MSDN: What is Host Throttling?
MSDN: Host Throttling Performance Counters
Written by Tom Morgan
Tom is a Microsoft Teams Platform developer and Microsoft MVP who has been blogging for over a decade. Find out more.
The throttling mechanism is self tuning and the default configuration options are suitable for the majority of BizTalk Server processing scenarios.
a BizTalk project created in Visual Studio 2010 in Visual Studio 2008 and just re-compile it.