Mozilla Internship: Diving into Test Automation with Python

Test Automation with Python

The first week of my internship was spent primarily setting up my dev environment on the laptop I received.

I had also never used a Mac before, so there were some basics to learn as well.

Although, since MacOS is Unix-based now, all of the important things were pretty much the same.

(I’m worthless with a Windows command line)

Once I had everything configured, I started fiddling with python and pytest.

I’ve done some development in python, like a scrabble-esque game, but I’d never written tests in python before.


In order to automate regression testing for accessibility, we need an API of some sort.

I did some research on the available web accessibility APIs before settling on the axe-core API.

The axe-core API was created by Deque, a company that specializes in accessibility.

Deque offers assessment services, certifications, and more.

axe-core is written in JavaScript and distributed as an an npm package.

Eventually, I will create a more seamless integration of axe-core into python, by writing a python library.

DequeLabs did this with Java in their tool axe-selenium-java.

To get started, though, I will be using Selenium’s execute_script() function to handle the JavaScript directly.

Testing with aXe

The aXe API tests a website against a configurable set of rules.

These rules are based on WCAG 2.0, Section 508, and best practices endorsed by Deque.

The test loads an instance of Firefox, injects the axe-core script, and then injects a custom script that runs the API.

I didn’t have a way to directly pass data from the JavaScript back to python (this will be addressed by the axe_selenium_python package I’m writing).

For the purpose of this test, I used JavaScript to write the JSON results to an element in the DOM.

I then grabbed the contents of that element using Selenium in my python test.

For my first iteration of the test, I simply asserted that there were no violations with an impact of critical.

class TestCriticalViolations:

    def test_accessibility_critical_violations(self):
        # List to hold info about critical violations
        criticalViolations = []
        data = json.load(open('./result.json'))
        # Iterate through violations
        for item in data['violations']:
            # Find all critical violations
            if item['impact'] == 'critical':
                # Add description to list

        # Assert that no critical violations are found
        assert len(criticalViolations) == 0, 'Critical Failures found'


// Get axe-core rules
// Default setting is all rules
// Run axe
// On success, process results
  // Get element
  var window = document.getElementsByTagName('html');
  // Create new div element
  var node = document.createElement('DIV');
  // Populate div with results text
  var textNode = document.createTextNode(JSON.stringify(result));
  // Add selector to element
  node.setAttribute('id', 'axe-result');
  // And append to element

This first version of my test used the python packages pytest and selenium.

I modified the test to use the pytest-selenium package, written by Dave Hunt of the Firefox Test Engineering team.

This package adds some additional functionality, and makes testing with selenium a little easier.

I added another package, pytest-html (also written by Dave Hunt), to generate an HTML report of the pytest results.

I also implemented the tests with tox, so that I could run the tests in both Python 2.7 and Python 3 simultaneously.

I then wrote a test for each rule used by axe-core, to get more meaningful output from the test suite.

Instead of seeing that critical violations were found, I am now able to see each test that passed and each test that failed.

. . .

def test_accesskeys(self):
    """Ensures every accesskey attribute value is unique."""
    assert test_results.get('accesskeys') is None, test_results['accesskeys'].help

def test_area_alt(self):
 elements of image maps have alternate text."""
    assert test_results.get('area-alt') is None, test_results['area-alt'].help

def test_aria_allowed_attr(self):
    """Ensures ARIA attributes are allowed for an element's role."""
    assert test_results.get('aria-allowed-attr') is None, test_results['aria-allowed-attr'].help

. . .

Now that I understand how to use pytest and Selenium, the next step will be learning how to write Python packages.

Syntax Highlighting by EnlighterJS

About the Author

Kimberly is currently a Test Automation Engineer for Mozilla, with a focus on automating web accessibility testing.

While she always enjoys learning new technologies, her current focus is python, and when she has free time (she doesn't), she enjoys JavaScript programming and learning about Data Analysis.

When not coding, Kimberly spends time with her three young boys, brand new baby girl, and her partner, in Durango, CO.