TestCafe: writing flexible tests that can run against multiple websites

Note: this repo that contains all examples: testcafe-multisite-example
See the repo’s README file for instructions on installing & running the tests.

When writing tests, it’s useful to write them flexibly so that you can:

• run all tests against one site
• run one test against all sites
• run any number of tests against any number of websites

Choose a good test site:

Before we get too far, we want to set up a couple of simple test cases that we can run against different websites. And since this is an automated testing exercise, rather than testing against production sites that you might not own or have rights to test against, let’s use a site that welcomes practice test automation. Nikolay Advolodkin has created a nice list of test automation websites that you can run your tests against. For our case, let’s choose one that has a home page with several different language options: PHP Travel demo website.

Simple test:

Let’s start with a simple fixture with two test cases:

// file: ./tests/simpleSingleSite.js
import { Selector } from "testcafe";

fixture(`Simple single site test`).page("https://www.phptravels.net/en");

test(`PHPTravels en_US - test 'BLOG' page`, async t => {
  // click the "Blog" link in the nav menu
  const link = Selector("li.go-right > a").withText("BLOG");
  await t.click(link);
});

test(`PHPTravels en_US - test 'OFFERS' page`, async t => {
  // click the "Offers" link in the nav menu
  const link = Selector("li.go-right > a").withText("OFFERS");
  await t.click(link);
});

• The first test case loads a US travel home page, then clicks on a ‘Blogs’ link, and goes to that page.
• The second test case loads a US travel home page, then clicks on a “Offers” link, and goes to that page.

Pretty simple! These test cases aren’t really doing anything useful, but they’re a good place to start for our examples.

Running specific tests:

TestCafe already nicely provides many command-line options including the ability to select individual fixtures and tests to run. For example, you can choose tests based on “fixture” name grep pattern:

> testcafe -e chrome tests --fixture-grep 'single site'
Running tests in:
- Chrome 70.0.3538 / Mac OS X 10.12.6
 
Simple single site test
✓ PHPTravels en_US - test 'BLOG' page
✓ PHPTravels en_US - test 'OFFERS' page

Or you can choose tests based on “test” name grep pattern:

> testcafe -e chrome tests/simpleSingleSite.js --test-grep 'BLOG'
Running tests in:
- Chrome 70.0.3538 / Mac OS X 10.12.6
 
Simple single site test
✓ PHPTravels en_US - test 'BLOG' page

These let us flexibly choose the tests that we want to run. But what about choosing the sites?

Refactor to support multiple sites:

The above test loads a website, clicks a couple links, and visits a couple pages. We’d like to run tests against multiple websites, so let’s refactor the test. We’ll pull the strings and links out of the tests and into constants, then refactor the code:

// file: ./tests/refactoredSingleSite.js
import { Selector } from "testcafe";

// refactored so strings, urls, and selector strings are in constants
const siteName = "PHPTravels en_US";
const url = "https://www.phptravels.net/en";
const navLinks = "li.go-right > a";
const blogLinkText = "BLOG";
const offersLinkText = "OFFERS";

fixture(`Refactored single site test`).page(url);

test(`${siteName} - test '${blogLinkText}' page`, async t => {
  // click the "Blog" link in the nav menu
  const link = Selector(navLinks).withText(blogLinkText);
  await t.click(link);
});

test(`${siteName} - test '${offersLinkText}' page`, async t => {
  // click the "Offers" link in the nav menu
  const link = Selector(navLinks).withText(offersLinkText);
  await t.click(link);
});

Now, the site details are separated from the tests. You could copy this file into multiple files (one for each site) and update the specific site details in each file, but this isn’t a good idea. It violates the Don’t Repeat Yourself principle.

For example, let’s say that you had multiple files for each site and later found that you needed to modify a test. With multiple files, you would need to modify the test in each file. This is redundant, prone to errors, and makes projects hard to support over time.

Instead, for the moment, let’s list (but comment out) the other sites we want to test: US, FR, RU, and SA

// file: ./tests/multiSite1.js
import { Selector } from "testcafe";

const siteName = "PHPTravels en_US";
const url = "https://www.phptravels.net/en";
const navLinks = "li.go-right > a";
const blogLinkText = "BLOG";
const offersLinkText = "OFFERS";

// const siteName = 'PHPTravels fr_FR';
// const url = 'https://www.phptravels.net/fr';
// const navLinks = 'li.go-right > a';
// const blogLinkText = "BLOG";
// const offersLinkText = "DES OFFRES";

// const siteName = "PHPTravels ru_RU";
// const url = "https://www.phptravels.net/ru";
// const navLinks = "li.go-right > a";
// const blogLinkText = "БЛОГ";
// const offersLinkText = "OFFERS";

// const siteName = "PHPTravels ar_SA";
// const url = "https://www.phptravels.net/ar";
// const navLinks = "li.go-right > a";
// const blogLinkText = "مقالات";
// const offersLinkText = "عروض";

fixture(`Multi-site test 1`).page(url);

test(`${siteName} - test '${blogLinkText}' page`, async t => {
  // click the "Blog" link in the nav menu
  const link = Selector(navLinks).withText(blogLinkText);
  await t.click(link);
});

test(`${siteName} - test '${offersLinkText}' page`, async t => {
  // click the "Offers" link in the nav menu
  const link = Selector(navLinks).withText(offersLinkText);
  await t.click(link);
});

With this test, you can uncomment each site & test it. This might be a good intermediate step, but isn’t a good place to leave our test, either. Many tests we’ll eventually want to run on a schedule (say via a continuous integration system or a cron job), so it wouldn’t be practical to edit the file before each run. Additionally, if the code is part of a code repository (as it should be!), we don’t want to modify it and check it into the repository before each test, either.

Instead, let’s rework this code so that the sites are defined in an array of objects:

// file: ./tests/multiSite2.js
import { Selector } from "testcafe";

const sites = [
  {
    name: "PHPTravels en_US",
    url: "https://www.phptravels.net/en",
    navLinks: "li.go-right > a",
    blogLinkText: "BLOG",
    offersLinkText: "OFFERS"
  },
  {
    name: "PHPTravels fr_FR",
    url: "https://www.phptravels.net/fr",
    navLinks: "li.go-right > a",
    blogLinkText: "BLOG",
    offersLinkText: "DES OFFRES"
  },
  {
    name: "PHPTravels ru_RU",
    url: "https://www.phptravels.net/ru",
    navLinks: "li.go-right > a",
    blogLinkText: "БЛОГ",
    offersLinkText: "OFFERS"
  },
  {
    name: "PHPTravels ar_SA",
    url: "https://www.phptravels.net/ar",
    navLinks: "li.go-right > a",
    blogLinkText: "مقالات",
    offersLinkText: "عروض"
  }
];

fixture(`Multi-site test 2`);

sites.forEach(site => {
  test(`${site.name} - test '${site.blogLinkText}' page`, async t => {
    // click the "Blog" link in the nav menu
    const link = Selector(site.navLinks).withText(site.blogLinkText);
    await t.navigateTo(site.url).click(link);
  });

  test(`${site.name} - test '${site.offersLinkText}' page`, async t => {
    // click the "Offers" link in the nav menu
    const link = Selector(site.navLinks).withText(site.offersLinkText);
    await t.navigateTo(site.url).click(link);
  });
});

Now, we have a sites array with all site details defined in each site object. The tests are also contained within a ‘sites.forEach()’ loop so when we run the test, tests run for all sites:

> testcafe -e chrome tests/multiSite2.js
Running tests in:
- Chrome 70.0.3538 / Mac OS X 10.12.6
 
Multi-site test 2
✓ PHPTravels en_US - test 'BLOG' page
✓ PHPTravels en_US - test 'OFFERS' page
✓ PHPTravels fr_FR - test 'BLOG' page
✓ PHPTravels fr_FR - test 'DES OFFRES' page
✓ PHPTravels ru_RU - test 'БЛОГ' page
✓ PHPTravels ru_RU - test 'OFFERS' page
✓ PHPTravels ar_SA - test 'مقالات' page
✓ PHPTravels ar_SA - test 'عروض' page 

That’s great! We have two tests that run against 4 different websites. Woo-hoo!

But there are some “gotchas”:

1. If we want to add/remove/modify any sites, then we have to modify this file. It would be better if we were able to add or modify sites without affecting the test file.
2. If there are many constants for each site, then the size of the file quickly gets big. Also, tests could get “buried” at the bottom of the file, below the site definitions which isn’t great.
3. Rather than always running all tests for all sites, it would be good if we could create the site definitions, but then at test run-time, specify which sites we want to use.

Separate site definitions and tests:

Let’s change the test so that the site definitions are in a different file than the tests:

// file: ./lib/getSitesStatic.js
function getSites() {
  const sites = [
    {
      name: "PHPTravels en_US",
      url: "https://www.phptravels.net/en",
      navLinks: "li.go-right > a",
      blogLinkText: "BLOG",
      offersLinkText: "OFFERS"
    },
    {
      name: "PHPTravels fr_FR",
      url: "https://www.phptravels.net/fr",
      navLinks: "li.go-right > a",
      blogLinkText: "BLOG",
      offersLinkText: "DES OFFRES"
    },
    {
      name: "PHPTravels ru_RU",
      url: "https://www.phptravels.net/ru",
      navLinks: "li.go-right > a",
      blogLinkText: "БЛОГ",
      offersLinkText: "OFFERS"
    },
    {
      name: "PHPTravels ar_SA",
      url: "https://www.phptravels.net/ar",
      navLinks: "li.go-right > a",
      blogLinkText: "مقالات",
      offersLinkText: "عروض"
    }
  ];

  return sites;
}

export { getSites };

// file: ./tests/multiSite3.js
import { Selector } from "testcafe";
import { getSites } from "../lib/getSitesStatic";

const sites = getSites();

fixture(`Multi-site test 3`);

sites.forEach(site => {
  test(`${site.name} - test '${site.blogLinkText}' page`, async t => {
    // click the "Blog" link in the nav menu
    const link = Selector(site.navLinks).withText(site.blogLinkText);
    await t.navigateTo(site.url).click(link);
  });

  test(`${site.name} - test '${site.offersLinkText}' page`, async t => {
    // click the "Offers" link in the nav menu
    const link = Selector(site.navLinks).withText(site.offersLinkText);
    await t.navigateTo(site.url).click(link);
  });
});

Not a big change here. We just moved the site definitions into a ‘getSites()’ function in a separate file.

But this separation gives us a lot of power & flexibility! The test fixture has no idea which sites, if any, are going to be returned from the getSites() function. But, it also doesn’t need to care. The test fixture just runs tests for each site passed into it. We can use this separation to write a smarter “getSites()” function that only returns the sites that the user requests.

Also, earlier, we described how you could select which tests to run using ‘–fixture-grep’ and ‘–test-grep’ command lines. What if we could similarly select the sites that we’d like to run the tests against? Something like:

> testcafe chrome tests/sometest.js --site US,FR,RU,SA

Now, that would give us the flexibility we want!

Handling command-line arguments:

We said that we would like to pass in a “–site” command-line argument. For Node.js projects, there are many ways for processing command-line arguments. For this project, we’re using yargs (because pirates, yo) like this:

const yargs = require("yargs");
const argv = yargs.options({
  site: {
    type:"array"
  }
}).argv;
console.log(`sites = ${argv.site}`;

This code looks for a command-line argument named “site” and saves all values in an array. (It also handles cases where multiple site values are specified by the user). The values can then be found in ‘argv.site’ variable.

Store sites as separate JSON files:

Let’s take our site definitions and store them as separate JSON files:

1. Create a “sites” subdirectory.
2. In the subdirectory, create a JSON file for each site and name it appropriately. For example:

# file: ./sites/PHPTravels-en_US.json
{
  "name": "PHPTravels en_US",
  "url": "https://www.phptravels.net/en",
  "navLinks": "li.go-right > a",
  "blogLinkText": "BLOG",
  "offersLinkText": "OFFERS"
}

Continue similarly for the other site definitions.

File globbing:

Next, we’ll take the “site” command-line argument that the user passed in and figure out how to load the matching site JSON files into a sites array. We could end up writing a lot of custom code to try to do this with string and regex matching, but why not leverage what others have written? Actually, this problem has already been solved by others! We are trying to load JSON site files (defined in the previous step) that match strings and wildcards. This is exactly what file globbing does(!):

    “glob patterns specify sets of filenames with wildcard characters“

If we use the “–site” string as a glob pattern, we can use a file globbing library to return a list of matching files.  If you search on npmjs.com, you’ll see several good “glob” packages. We’ll use globby npm module because it supports pattern matching, but it also supports negative pattern matching (i.e. in our case, telling which sites not to load), too. That way, we could run tests like these:

Run tests against the US site:

> testcafe chrome tests/sometest.js --site '**/*US*'

Run tests against all sites except the US site:

> testcafe chrome tests/sometest.js --site '**/*' '!**/*US*'

The final result:

Here are the final files:

# file: ./sites/PHPTravels-ar_SA.json
{
  "name": "PHPTravels ar_SA",
  "url": "https://www.phptravels.net/ar",
  "navLinks": "li.go-right > a",
  "blogLinkText": "مقالات",
  "offersLinkText": "عروض"
}

# file: ./sites/PHPTravels-en_US.json
{
  "name": "PHPTravels en_US",
  "url": "https://www.phptravels.net/en",
  "navLinks": "li.go-right > a",
  "blogLinkText": "BLOG",
  "offersLinkText": "OFFERS"
}

# file: ./sites/PHPTravels-fr_FR.json
{
  "name": "PHPTravels fr_FR",
  "url": "https://www.phptravels.net/fr",
  "navLinks": "li.go-right > a",
  "blogLinkText": "BLOG",
  "offersLinkText": "DES OFFRES"
}

# file: ./sites/PHPTravels-ru_RU.json
{
  "name": "PHPTravels ru_RU",
  "url": "https://www.phptravels.net/ru",
  "navLinks": "li.go-right > a",
  "blogLinkText": "БЛОГ",
  "offersLinkText": "OFFERS"
}

// file: ./lib/getSitesDynamic.js
/**
 * Utility function that returns an array of site definitions
 * to be tested.
 */

const fs = require("fs");
const yargs = require("yargs");
const globby = require("globby");

/**
 * Based on "site" command line argument, return an
 * array of sites that matches the user's requests
 * against sites found in the 'sites' subdirectory.
 */
function getSites() {
  // see if a "site" command-line arg passed in.
  // if not, default to an 'en_US' site
  const argv = yargs.options({
    site: {
      default: "**/*en_US*",
      type: "array"
    }
  }).argv;

  // create empty sites array that we will fill in
  let sites = [];

  // search in the 'sites' subdirectory
  // and return absolute paths back
  const options = {
    cwd: "sites",
    absolute: true
  };

  // use globby module to get all file names
  const files = globby.sync(argv.site, options);

  // load each file and add it to the sites array
  files.forEach(file => {
    try {
      const siteDefinition = JSON.parse(fs.readFileSync(file, "utf8"));
      sites.push(siteDefinition);
    } catch (error) {
      console.error(error);
    }
  });

  // finally, return the populated sites array
  return sites;
}

export { getSites };

// file: ./tests/multiSite4.js
import { Selector } from "testcafe";
import { getSites } from "./testcafe-getSitesDynamic";

const sites = getSites();
fixture(`Multi-site test 4`);

sites.forEach(site => {
  test(`${site.name} - test '${site.blogLinkText}' page`, async t => {
    // click the "Blog" link in the nav menu
    const link = Selector(site.navLinks).withText(site.blogLinkText);
    await t.navigateTo(site.url).click(link);
  });

  test(`${site.name} - test '${site.offersLinkText}' page`, async t => {
    // click the "Offers" link in the nav menu
    const link = Selector(site.navLinks).withText(site.offersLinkText);
    await t.navigateTo(site.url).click(link);
  });
});

The files include:

• 4 JSON “site” files (be sure to save these in a “sites” subdirectory)
• a “getSites()” function file
• a test fixture file

The new, dynamic getSites() function does several things:

• it retrieves the “–site” argument from yargs. If a site wasn’t specified, then it defaults to the “**/*en_US*” site.
• it looks for site files in the “sites” subdirectory.
• it uses “globby” to find all files specified by the “–site” command-line argument.
• it loops through all files, loading each one and converting it from JSON to a javascript object
• it adds the site’s javascript object to a “sites” array
• and finally, it returns the “sites” array to the caller

So now, we can run any test against any site:

1. Run all tests in a fixture against one site:

> testcafe -e chrome tests --fixture-grep 'Multi-site test 4' --site '**/*en_US*'
Running tests in:
- Chrome 70.0.3538 / Mac OS X 10.12.6
 
Multi-site test 4
✓ PHPTravels en_US - test 'BLOG' page
✓ PHPTravels en_US - test 'OFFERS' page

2. Run all tests in a fixture that have “OFF” in the test title against all sites:

> testcafe -e chrome tests --fixture-grep 'Multi-site test 4' --test-grep OFF --site '**/*'
Running tests in:
- Chrome 70.0.3538 / Mac OS X 10.12.6
 
Multi-site test 4
 ✓ PHPTravels en_US - test 'OFFERS' page
 ✓ PHPTravels fr_FR - test 'DES OFFRES' page
 ✓ PHPTravels ru_RU - test 'OFFERS' page

3. Run all tests in a fixture against all sites except US and FR:

> testcafe -e chrome tests --fixture-grep 'Multi-site test 4' --site '**/*' '!**/*US*' '!**/*FR*'
Running tests in:
- Chrome 70.0.3538 / Mac OS X 10.12.6

Multi-site test 4
 ✓ PHPTravels ar_SA - test 'مقالات' page
 ✓ PHPTravels ar_SA - test 'عروض' page
 ✓ PHPTravels ru_RU - test 'БЛОГ' page
 ✓ PHPTravels ru_RU - test 'OFFERS' page

Conclusion:

There are many approaches to writing & running automated tests. This is one way that has worked well for me.