Forum

More Testing is More Better

  • 65 Topics
  • 20 Replies

65 Topics

M
McbvolkCommunity Manager
published in Forum

Best Practice: Imperative v. Declarative Testing Scenarios

Imperative v. declarative test scenarios is a concept that is often discussed in the context of Cucumber and Behavior Driven Development (BDD), but it is applicable to all languages and test runners. A great post by Aslak, the author of Cucumber, gives a great description of his intentions along with a number of additional links at the bottom for further reading.Imperative testing or programming is essentially spelling out with as much detail as necessary how to accomplish something. Declarative testing or programming is only specifying (or declaring) what needs to be accomplished.This is seen acutely in BDD circles because the goal of BDD is to get all of the interested parties (Project, Dev, Test, Business, etc) to collaborate on the requirements of a feature before anyone begins working on the implementation. Many testers have latched on to BDD tools as glorified test runners rather than a way to actually facilitate BDD practices. This results in features that include actual code an

21150
M
11 months ago
M
McbvolkCommunity Manager
published in Forum

Best Practices: Use Small, Atomic, Autonomous Tests

When a test fails, the most important thing is knowing what went wrong so you can easily come up with a fix. The best way to know what went wrong is to keep three words in mind when designing your tests: Small, Atomic, and Autonomous.SmallSmall refers to the idea that your tests should be short and succinct. If you have a test suite of 100 tests running concurrently on 100 VMs, then the time it will take to run the entire suite will be determined by the longest/slowest test case. Keeping your tests small ensures that your suite will run efficiently and provide you with results faster. AtomicAn atomic test is one that focuses on testing a single feature, and which makes clear exactly what it is that you're testing. If the test fails, then you should also have a very clear idea of what needs to be fixed. AutonomousAn autonomous test is one that runs completely independently of other tests, and is not dependent on the results of one test to run successfully. In addition, an autonomous tes

9670
M
11 months ago
M
McbvolkCommunity Manager
published in Forum

Best Practice: Use Explicit Waits

As a best practice, use explicit waits on Selenium when you run into timeouts and failing tests.There are many situations in which your test script may run ahead of the website or application you're testing, resulting in timeouts and a failing test. For example, you may have a dynamic content element that, after a user clicks on it, a loading appears for five seconds. If your script isn't written in such a way as to account for that five second load time, it may fail because the next interactive element isn't available yet. The general advice from the Selenium community on how to handle this is to use explicit waits. While you could also use implicit waits, an implicit wait only waits for the appearance of certain elements on the page, while an explicit wait can be set to wait for broader conditions. Selenium guru Dave Haeffner provides an excellent example of why you should use explicit waits on his Elemental Selenium blog. Whether you use explicit or implicit waits, you should not mi

7790
M
11 months ago
dylanatsauce
dylanatsauceCommunity Manager
asked in Forum

Why are my tests getting a '500 Internal Server Error'?

The symptoms:Failing Test Commands Elements which can’t be located Screenshots that look like this:500 Internal Server Errors 

1
dylanatsauce
1 year ago
M
McbvolkCommunity Manager
published in Forum

Best Practice: Tips for Lean, Speedy Tests with Sauce Labs

It's a reality that tests run on Sauce Labs will always have some latency, compared to running locally, due to there being an Internet in the middle between your tests and our browsers. So, rather than running in the same machine, each Selenium request is sent through the wire to our VMs and receives a response back through the same channels. These are some other common factors that contribute to test duration, and a few tips for using your minutes efficiently.
 Identify Time SinksIf you're testing against a local application server, you will see some additional latency due to the distance from our browsers to your server. The same applies if you're using Sauce Connect to test against a firewalled server. When you're running Sauce Connect, it sends all requests from Sauce's browsers back through the machine running Sauce Connect, such that they appear to originate from that machine. This means that all data sent between the Sauce cloud and the site you're testing must travel first to y

5660
M
11 months ago
E
EvelynColemanCommunity Manager
published in Forum

Leveraging Sauce Labs with GitLabSolution Blog

GitLab, Meet Sauce Labs!This is a high level guide intended for anyone who is curious about how Gitlab’s Continuous Integration platform works with Sauce Labs. What is Gitlab How Sauce Labs fits in with Gitlab Sauce Connect and Gitlab Additional Resources  What is Gitlab:Gitlab is a collaboration platform that allows you to build a Continuous Integration (CI) pipeline as a series of steps in a .yml file. Steps can include things like launching a containerized test environment and starting your tests using your preferred test runner. Building a Gitlab .yml file to set up your pipeline is easy because Gitlab allows you to configure your environment the same way you would to run a shell script. Gitlab also has a simple way of launching docker images as “services”. This may be something to consider or explore when building your pipeline.  How Sauce Labs fits in with GitlabMostly, seamlessly.  There may be a few adjustments needed if you use Sauce Connect, or wish to run different tes

2780
E
1 year ago
M
McbvolkCommunity Manager
published in Forum

Best Practice: Avoid Dependencies Between Tests to Run Tests in Parallel

Dependencies between tests prevent tests from being able to run in parallel. And running tests in parallel is by far the best way to speed up the execution of your entire test suite. It's much easier to add a virtual machine than to try to figure out how to squeeze out another second of performance from a single test.What are dependencies? Imagine a test suite with two tests:Example (Java) @Testpublic void testLogin(){    // do some stuff to trigger a login    assertEquals("My Logged In Page", driver.getTitle());}@Testpublic void testUserOnlyFunctionality(){    driver.findElement(By.id("userOnlyButton")).click();    assertEquals("Result of clicking userOnlyButton", driver.findElement(By.id("some_result")));} Example (PHP)<?phpfunction testLogin(){    // do some stuff to trigger a login    $this->assertEquals("My Logged In Page", $this->title());}function testUserOnlyFunctionality(){    $this->byId('userOnlyButton')->click();    $this->assertTextPresent("Result of clic

2200
M
1 year ago
dylanatsauce
dylanatsauceCommunity Manager
published in Forum

Best Practice: Handling Authentication

There are several ways to handle authentication and security dialogs during testing. You might, for example, need to log a user in, or bypass a browser warning. The topics in this section cover cookie injection with Selenium, basic HTTP authentication, and other options.Basic HTTP AuthenticationBasic HTTP authentication supplies a username and password via URL, such as https://admin:admin@the-internet.herokuapp.com/basic_auth. This request sends the credentials in the standard HTTP "Authorization" header.Because browser support for basic HTTP authentication is limited, we recommend Injecting Cookies to Bypass Authentication Dialogs and Running an AutoIt Script as a Pre-run Executable to Handle Windows Security Authentication Dialogs as solutions for authentication while testing. Browser HTTP Authentication Support Chrome Supported Firefox Supported, but Firefox will display a prompt asking you to confirm. Safari Unsupported. See the Support KB article Setting Basi

2020
dylanatsauce
2 months ago
dylanatsauce
dylanatsauceCommunity Manager
published in Forum

Dynamically naming your tests with TestNGGuide

Test Names Should Identify TestsSo here’s something that seems obvious but is often overlooked: test name should let you identify an individual test. You might have multiple tests in a Class, and they all might be testing the same Class as well... but the name of the test should still let you identify which test is which, eg, it should be unique. Ideally, they’ll also tell you what is being tested, eg, be descriptive.Here’s an example of a unique names VS descriptive names. Both “TC1498245” and “assertAccountValid” are unique test case names… But only the latter also tells you what the test is doing.  That is, only the last one is descriptive. Without unique test names, your Sauce Labs dashboard can end up looking like this:5 Identically named tests, all testing different thingsNot only is this confusing, it also stops you making full use of Sauce Labs’ failure analysis and insights tooling. There’s three ways to change the names of a Sauce Labs test, but I prefer setting it in advance

1840
dylanatsauce
1 year ago
M
McbvolkCommunity Manager
published in Forum

Best Practice: Avoid "Brittle Locators"

WebDriver provides a number of locator strategies for accessing elements on a webpage. It's tempting to use complex XPath expressions like //body/div/div/*[@class="someClass"] or CSS selectors like #content .wrapper .main. While these might work when you are developing your tests, they will almost certainly break when you make unrelated refactoring changes to your HTML output.Instead, use sensible semantics for CSS IDs and form element names, and try to restrict yourself to using these semantic identifiers. For example, in Java you could designate elements with driver.findElement(By.id("someId")); or driver.findElement(By.name("someName")); or, in the example of PHP, you could use $this->byId() or $this->byName() . This makes it much less likely that you'll inadvertently break your page by shuffling around some lines of code.

1401
dylanatsauce
11 months ago
dylanatsauce
dylanatsauceCommunity Manager
published in Forum

Best Practise: How to Deal With Flaky Java Tests

Most testers experience having to deal with flaky tests whether or not they realize it. Learn how to deal with flaky Java tests to improve your testing.In the world of user interface (UI) testing we all have come across tests and UI behavior that is not 100% repeatable. In other words, sometimes tests are “flaky.” The challenge with these tests is in trying to determine what causes this flakiness, and finding ways to resolve it, so that we are able to test the UI in a reliable fashion without raising false alarms and creating test gaps.Failure ModesThe two most common situations that lead to flakiness are external factors over which one has no control, and the application or website you are testing not being in the expected state when you test it. Among the external factors, network conditions, different client types, and test infrastructure limitations are among the most common causes of flakiness.The way to diagnose these factors usually boils down to comparing the physical testing e

1210
dylanatsauce
2 months ago
M
McbvolkCommunity Manager
published in Forum

Best Practice: Avoid Leakage of Credentials

Sauce Labs test logs are securely stored, protecting them from external access. However, there are still some groups that can see test logs, including Sauce Support, your parent account, and other accounts in your company (depending on test privacy settings).Solution - Don't use real credentialsThe best way to avoid this is to avoid using "real" credentials in tests, through the creation of temporary accounts.Workaround - Transmit session tokens onlyYou can also avoid sensitive credentials using Selenium's ability to extract and inject cookies into accounts:Create a session in your environment, either directly in the application engine, or by using a local Selenium session or headless browser. Extract the session tokens (local storage objects, credentials, cookies, etc.). Use Selenium to push these objects and tokens into the browser under Sauce Labs' controlThis technique avoids sending plain text passwords, however, the sent tokens and cookies are still logged. If your session tokens

1150
M
11 months ago
aleks.community
aleks.communityNew Participant
asked in Forum

How can I inspect my elements and get the correct element identifier for my mobile test?

My tests fail sometimes because it seems like I am using the wrong identifier for my elements. Appium Desktop can be used for that, are there other ways to do the same? When using Appium Desktop, is there a simple step-by-step instruction on how to set it up and use it?

0
aleks.community
1 year ago
M
max.dobeckNew Participant
asked in Forum

How do I run an iOS Simulator test instead of a iOS Real Device test?

Like the title says, how can I force an iOS simulator test instead of an iOS real device test? What capabilities must change?

1
dylanatsauce
1 year ago
M
McbvolkCommunity Manager
published in Forum

Best Practice: Use Breakpoints to Diagnose Flaky Tests on Sauce Labs

A "flaky" test is one that normally runs without any issues, but every so often, maybe one in a hundred times, fails for seemingly random reasons. Debugging the issue with flaky tests is challenging, but by using breakpoints in flaky tests, you can identify the specific conditions that led to the flake, and dig into the problem more deeply using your developer tools. There are two ways to use breakpoints with your Sauce Labs test: through the sauce: break Selenium command, and by using the Pause button on the Test Details page while the test is running. sauce: breaksauce: break is a JavaScript statement that you can insert into your Selenium execute_script command to both identify and interrupt flaky tests for further diagnosis. You can find more information in the topics Annotating Tests with Selenium's JavaScript Executor and Live Testing on Virtual Mobile Devices.The Pause ButtonWhen your test is running, you can use the Pause button on the Test Details page to interrupt the test an

900
M
11 months ago
M
McbvolkCommunity Manager
published in Forum

Best Practice: Use the Latest Version of Selenium Client Bindings

The Selenium Project is always working to improve the functionality and performance of its client drivers for supported languages like Java, C#, Ruby, Python, and JavaScript, so you should always be using the latest version of the driver for your particular scripting language. You can find these on the Downloads page of the SeleniumHQ website, under Selenium Client & WebDriver Language Bindings. 

820
M
11 months ago
M
McbvolkCommunity Manager
published in Forum

Best Practice: Have a Retry Strategy for Handling Flakes

There will always be flaky tests, and tests that once breezed through with no problem can fail for what seems like no reason. The trick is figuring out whether a test that fails does so because it found a real problem in your app functionality, or because there was an issue with the test itself. The best way to handle this problem is to log your failing tests into a database and then analyze them. Even tests that fail intermittently with no apparent cause may turn out to have a pattern when you are able to analyze them in detail and as a larger data set. If this is beyond the scope of your testing setup, the next best strategy is to log your failing cases into a log file that records the browser, version, and operating system for those tests, and then retry those tests. If they continue to fail after a second or third retry, chances are that the issue is with the functionality you're testing, rather than the test itself. This isn't a total solution for dealing with flakes, but it shoul

810
M
11 months ago
dylanatsauce
dylanatsauceCommunity Manager
posted in Forum

Appium 2.0 looks awesome.

Just the other day, I discovered that Appium’s Roadmap is really well documented.  Perhaps I shouldn’t be surprised since, IMO, Appium is one of the best run OSS projects out there. One of the cool things on the roadmap is the proposal for Appium 2.0.  It was added to the roadmap in 2019, but there have been several updates since then; it looks like it’s going to be a major overhaul!  The primary change is that Appium is going to become a set of WebDriver specifications, a runner, and a tool for managing drivers.  According to the proposal, Appium itself won’t contain any drivers.  Instead, it will have an interface that lets you install and manage drivers and driver versions.  What is considered the main “Appium” package right now will become the driver runner.  Appium will also develop a set of libraries that are consumable by other projects to make it easier to write drivers themselves (Maybe Apple Watch support?!). There’s some changes to interfaces as well; Not only are they plann

0
dylanatsauce
1 year ago
dylanatsauce
dylanatsauceCommunity Manager
posted in Forum

I AM a Robot and it's fine: CAPTCHAs in automated tests

CAPTCHAs are the weird little challenges you get online which ask you to prove you’re a human by clicking a box, unscrambling a warped word, or finding all the cars in an image:reCAPTCHA from Google is a common sight onlineThese challenges are great for protecting your website from automated spam, bots and other non-human interaction.That’s why Saucebot hates them.  In the normal course of things, you want your site protected from automated interactions.  When testing though, you don’t.  Worse, since the entire point of these things is to prevent your service being used by bots, they can block automated testing entirely. We get customers asking us all the time; How can they get around the CAPTCHA during their automated tests?  The bad news is, well, you can’t.  Not directly.  Not letting people get around them is the entire reason they exist. We usually suggest customers disable them entirely for their test systems.  For live/production systems, you can try using IP detection to check

0
dylanatsauce
1 year ago
josh.grant
josh.grantCommunity Manager
posted in Forum

Robot Framework: The Good and the Bad

A test tool that seems to be growing in popularity is Robot Framework. This is an open source tool for writing keyword-based tests. There are also pre-written keyword libraries that have plenty of keywords ready to use written by the Robot Framework community. Keyword-driven testing is an interesting way of writing test automation with pros and cons. On the plus side, test cases are “human readable” and can be written by non-technical folks like product owners or business analysts in addition to testers and developers. On the minus side, keyword management can be difficult to maintain, increasing overall project maintenance. As well, the Robot Framework seems to have an active community and good library maintainers, which means the actual tools are kept up-to-date. If you have any experiences with Robot Framework (good, bad or ugly), I’d like to hear about them!

1
dylanatsauce
1 year ago
abrown6123
abrown6123New Participant
asked in Forum

Why aren't my tests passing in the Sauce Labs dashboard?

My tests appear to be successful when executed on Sauce Labs, but I’m not seeing that they are being updated with a green checkmark/passed status. The same occurs for failing tests. How do I update that? 

1
aleks.community
1 year ago
M
McbvolkCommunity Manager
published in Forum

Best Practice: Be Aware of the Load on Your Servers

As you move into fully automated testing and builds, with tests running in parallel and against multiple device/browser/platform/operating system combinations, be aware of the load that this will place on both your CI/CD server and the site under test. A best practice in this situation is make sure that both the CI/CD server and the site under test are on machines that are not running additional processes or open to additional network traffic, and can handle the additional number of simultaneous jobs and tests.

371
dylanatsauce
11 months ago
dylanatsauce
dylanatsauceCommunity Manager
published in Forum

Best Practise: Use Page Objects to Model Repeated Interactions and Elements

Within your applications and sites there are elements that your tests interact with, sometimes on a repeating basis. Rather than having to repeatedly code these interactions into your test, you can use page objects to abstract these interactions into a single functional unit. For example, your tests may require logging into a a site or application. Rather than coding all these interactions into your test, you can create a LoginPage object that contains these interactions, which you then refer to in your test. This means taking a sort of object-oriented approach to test construction that  enables you to simplify your test code and reduce duplication of effort. For more information, check out these references.The SeleniumHQ documentation of page objects, hosted on Google Code The documentation for the Intern testing framework provides a good explanation of page objects and an example in JavaScript The cheezy/page-object GitHub repository includes the page-object gem for Ruby, as well as

350
dylanatsauce
2 months ago
dylanatsauce
dylanatsauceCommunity Manager
published in Forum

SauceCon 2021 Registration is LIVE!SauceCon

About SauceCon Online 2021SauceCon brings together the global community of Sauce Labs users and automated testing experts. Join us online April 20-22, as teams from around the world will come together to learn from each other and level up their automated testing and continuous delivery skills. What can I expect?Two days of talks 33 world class speakers Access to Sauce Labs staff Best Practises for Testers, Developers, Coders, DevOps Leaders and teams like yours 6 In-Depth half-day workshops on API testing, Automation, WebDriverIO, Docker, Cypress and Test FrameworksYou can watch all the SauceCon Online 2020 sessions for free, right here to get a better idea of what SauceCon Online is like. When is it?April 20-22 What are the workshops?Check them out here. How Much Are Tickets?Tickets. Are. FREE.Tickets are tiered, with the “Free” tier getting you access to all Keynote Sessions and Industry Technical Presentations.Basic TicketsGive you access to everything in the Free Tier, plus breakou

340
dylanatsauce
1 year ago
M
McbvolkCommunity Manager
published in Forum

Best Practice: Keep Functional Tests Separate from Performance Tests

It's very important to maintain a distinction between functional tests of your web applications, front-end web performance, and tests of your network and servers under load.    Functional tests should, as the name indicates, test some functionality or feature of your application. The output of these tests should generally be a simple "pass" or "fail" - either your functionality worked as expected, or it didn't. While running functional tests, it can also be advantageous to run front end performance tests that can help identify any regressions in JavaScript logic executed in the browser. When you use Sauce Labs for functional testing, you can test your website under specific network conditions and collect network and application-related metrics.  Load tests, in contrast, should gauge and output network and server performance metrics. For example, can your application server handle a particular load, and does it behave as expected when you push it to its limit? These types of tests are b

330
M
1 year ago

Raise a Private Ticket

Prefer to wait, and let a Support Engineer check out your issue? Raise a Ticket.
Terms & ConditionsCookie settings