Best Practice: Be Aware of the Load on Your Servers

  • 18 May 2021
  • 1 reply
  • 12 views

Userlevel 1
Badge +1

As you move into fully automated testing and builds, with tests running in parallel and against multiple device/browser/platform/operating system combinations, be aware of the load that this will place on both your CI/CD server and the site under test. A best practice in this situation is make sure that both the CI/CD server and the site under test are on machines that are not running additional processes or open to additional network traffic, and can handle the additional number of simultaneous jobs and tests.


1 reply

Userlevel 1
Badge +2

This is a great point!  I’ve debugged so many problems where moving from a local grid to Sauce Labs caused a customer’s tests to become flaky, and the underlying cause was performance issues.

 

Usually what would happen is that users would go from running single-threaded or low-concurrency tests on their local grid, to running tens of concurrent tests on Sauce Labs.  Their SUT would become overloaded because it lacked RAM or processing power, there was a bug, or their Database server pegged out… But it would appear like the tests themselves had started timing out.  They weren’t, but the browser was having to wait longer and longer for the server to complete its job.

 

One way to diagnose this is to go back to running tests at the same concurrency you were using against a local grid; Then try doubling that.  Then double again.  If the problem starts happening “Reliably” when you get above a certain load, it’s likely a system resource problem.  Of course, monitoring load directly is more useful again, because you can see what your SUT is doing… But sometimes, just varying the number of tests you run is easier ;P

Reply