Speed up your automated tests without really doing anything..

Here’s a Christmas present from me for you: a short, but may be fore some people a very liberating post on how to speed up your tests in your pipeline, without doing anything “real” ;-).

The problem

We have made “tests” mandatory in our company.  What that means: every single pullrequest is being code reviewed, and in the code review, we check for either a changed or new tests.  That has lead to quite a number of tests: 8500 for the product, and we even have a customer with (in the meantime) over 1500 tests.

Over the years, we noticed a growing performance problem when running tests.  For our product, running all tests for all apps, took about 3 hours.  But we were ok with that: we run all those tests at night.  If they take 2, 3 or 5 hours .. who cares?  Nobody is waiting for it.  And for pullrequests, we only run the tests from the app we changed, and that didn’t take that long (the product is split in 50 apps, so we have an average of 170 tests per app, which was quite ok).

But for this customer (and a few projects), it’s a different story, because every single pullrequest needs to execute all the tests.  And it took more than an hour to run the tests – so a pull request took quite a while.  Just imagine we need to do a hotfix: it makes all the sense we reduce that time of a build, and the most significant amount of time to reduce, is the test run!

Just look a this:

At that time, we had about 8 developers constantly developing for this customer, which means: multiple times 1h 20 minutes wait time for a pullrequest validation. That’s simply too much .. something had to happen!

The analysis

After careful analysis of the log file, my colleague noticed a trend that lead us to quite an interesting conclusion:

The more codeunits we had in a test suite, the  longer it took to run the suite.

I’ll say that again in other words:

Every run of a codeunit in one suite, took a fraction longer to run to more the test run advanced in that suite

Or yet again in other words:

Codeunit 1 was fast, codeunit 2 was a fraction slower .. and all following codeunits were (in general) slower than the previous .. 

And this phenomenon was true “per suite“.  So a second suite, first codeunit, would be fast again .. .

Another colleague had an explanation for it which I’m not able to verify (that I know):  the way the test-suite is run from the pipeline, is that it “mimics” a user using the webclient (remember that we need the webclient to be able to execute page testability).  On top of that, you need to be able to track results per codeunit (or even per function): so one way to do that, is loop your way through:

• Open suite
• Navigate down to the next codeunit/function
• Run codeunit/function
• Save results

Now just imagine you have a lot of codeunits.  “Navigating down” can take a while.. 🤷‍♂️.

Again: not sure that this is the way it works, but it would sure explain this behavior.

(Part of) the solution

We tested what the sweet spots was for the number of codeunits in one suite and the time-gain we had – which (for us) is 25 codeunits.  Now we simply consciously divide codeunits per suite.

We always had an install codeunit that created a test-suite for an app.  Now we simply create multiple suites per app.

Yep – hardcoded and manual.  It’s ok – it’s just a test app ;-).

In the pipeline, we simply run the test-step multiple times:

Where we simply loop this array (by using some simple yaml magic ;-)):

Result:

BOOM! 50 minutes time difference for 100% the same tests!

Writing testable code

You might have joined Vjeko’s and my session at BCTechDays – or you might have followed one of Vjeko’s workshops or sessions at Directions – or you might have read his blog posts about the topic, like:

• Testing, testability, and all things test – Vjeko.com
• Directions EMEA 2023 demo – decoupling base app – Vjeko.com
• Testing in isolation – Vjeko.com

Obviously, what I just shared is by no means a way to avoid “decent coding”, or a replacement for you for not having to dive into “writing testable code” better ;-).  By no means at all!

But it does make sense to never combine all your tests in one suite and run them that way in a pipeline 🤷‍♂️.

Happy Holidays!