Inexplicable delays when calling (OnPrem) Business Central API’s (and how to solve it)

So, I was reminded recently during a twitter thread about an issue we had been having that we investigated quite thoroughly to figure out what was going on.

The issue

We always try to upgrade all customers to latest release – and we encountered some delays in API communications, which seemed to got worse at every new upgrade. At some point, every API call took from 500 to 1000 milliseconds. I think we all agree that this is FAR to high. Even for a simple call, like getting the companies, it took so long.

Investigation

We really didn’t have any clue why this happened, so we started to list all customers, called some API endpoint that always existed (getting the companies), and started to log all durations for all these customers and all these calls. With that, we wanted to gather as much info as possible, which made this call specific, like authentication type, version of BC, the endpoint, number of companies and of course the durations.

And this gave us a pretty nice overview and pointed us in the right direction of what the problem was.
For the customers where we used “NavUserPassword” as authentication, some of them, we authenticated with a WebService Access Key (WAC) , and some simply with username/password (NUP) (basically the same user, but without setting up a Web Service Access Key on the User Card). And we could clearly see that WAC was so much faster than NUP, so we took the test even further:
We used the same endpoint – the very same, on the very same NST – and one call we did with NUP and the other with WAC. And it clearly indicated that there is a massive difference:

If you’re NOT using the Web Service Access Key, the webservice call can take up to 10 times more time! Or (to not use the double negative): When using the Web Service Access Key, the same call, same endpoint, same everything is about 10 times faster!

If you look at this table, you see that we mostly do 2 calls per customer: one with WAC, one for NUP (for the ones that were set up as NAVUserPassword – for the ones with windows authentication (WIN), we can only use WinAuth and we couldn’t test an alternative). And the durations are very VERY different (I circled a few examples, but actually, every single customer, on every single minor, .. acted the same):

Or if we sort it on the average timing, you see that WAC is on top, scoring the best.

Remember the scope

I have no explanation. None whatsoever. But please do remember the narrow scope for this situation:

• OnPrem
• ClientServicesCredentialType “NavUserPassword”
• Basic Authentication
• Business Central API calls

Conclusion

This post is by no means a recommendation to use basic authentication, merely a possible indication when you suffer from inexplicable delays in case of API communication (I expect this behavior to be the same in v21 as well). So, in case you’re still using Basic Authentication / NavUserPassword, it’s very much recommended to not use the password for the user, but set up a Web Service Access Key and use that in stead for all API calls!