Mozilla Internship: Diving into Test Automation with Python

Test Automation with Python

The first week of my internship was spent primarily setting up my dev environment on the laptop I received.

I had also never used a Mac before, so there were some basics to learn as well.

Although, since MacOS is Unix-based now, all of the important things were pretty much the same.

(I’m worthless with a Windows command line)

Once I had everything configured, I started fiddling with python and pytest.

I’ve done some development in python, like a scrabble-esque game, but I’d never written tests in python before.

axe-core

In order to automate regression testing for accessibility, we need an API of some sort.

I did some research on the available web accessibility APIs before settling on the axe-core API.

The axe-core API was created by Deque, a company that specializes in accessibility.

Deque offers assessment services, certifications, and more.

axe-core is written in JavaScript and distributed as an an npm package.

Eventually, I will create a more seamless integration of axe-core into python, by writing a python library.

DequeLabs did this with Java in their tool axe-selenium-java.

To get started, though, I will be using Selenium’s execute_script() function to handle the JavaScript directly.

Testing with aXe

The aXe API tests a website against a configurable set of rules.

These rules are based on WCAG 2.0, Section 508, and best practices endorsed by Deque.

The test loads an instance of Firefox, injects the axe-core script, and then injects a custom script that runs the API.

I didn’t have a way to directly pass data from the JavaScript back to python (this will be addressed by the axe_selenium_python package I’m writing).

For the purpose of this test, I used JavaScript to write the JSON results to an element in the DOM.

I then grabbed the contents of that element using Selenium in my python test.

For my first iteration of the test, I simply asserted that there were no violations with an impact of critical.

test_critical_violations.py

class TestCriticalViolations:

    @pytest.mark.nondestructive
    def test_accessibility_critical_violations(self):
        # List to hold info about critical violations
        criticalViolations = []
        data = json.load(open('./result.json'))
        # Iterate through violations
        for item in data['violations']:
            # Find all critical violations
            if item['impact'] == 'critical':
                # Add description to list
                criticalViolations.append(item['help'])

        # Assert that no critical violations are found
        assert len(criticalViolations) == 0, 'Critical Failures found'

script.js

// Get axe-core rules
// Default setting is all rules
axe.getRules();
// Run axe
axe.run()
// On success, process results
.then(function(result){
  console.log(result);
  // Get element
  var window = document.getElementsByTagName('html');
  // Create new div element
  var node = document.createElement('DIV');
  // Populate div with results text
  var textNode = document.createTextNode(JSON.stringify(result));
  node.appendChild(textNode);
  // Add selector to element
  node.setAttribute('id', 'axe-result');
  // And append to element
  window[0].appendChild(node);
});

This first version of my test used the python packages pytest and selenium.

I modified the test to use the pytest-selenium package, written by Dave Hunt of the Firefox Test Engineering team.

This package adds some additional functionality, and makes testing with selenium a little easier.

I added another package, pytest-html (also written by Dave Hunt), to generate an HTML report of the pytest results.

I also implemented the tests with tox, so that I could run the tests in both Python 2.7 and Python 3 simultaneously.

I then wrote a test for each rule used by axe-core, to get more meaningful output from the test suite.

Instead of seeing that critical violations were found, I am now able to see each test that passed and each test that failed.

test_accessibility.py

. . .

@pytest.mark.nondestructive
def test_accesskeys(self):
    """Ensures every accesskey attribute value is unique."""
    assert test_results.get('accesskeys') is None, test_results['accesskeys'].help

@pytest.mark.nondestructive
def test_area_alt(self):
    """Ensures 
 elements of image maps have alternate text."""
    assert test_results.get('area-alt') is None, test_results['area-alt'].help

@pytest.mark.nondestructive
def test_aria_allowed_attr(self):
    """Ensures ARIA attributes are allowed for an element's role."""
    assert test_results.get('aria-allowed-attr') is None, test_results['aria-allowed-attr'].help

. . .

Now that I understand how to use pytest and Selenium, the next step will be learning how to write Python packages.

Syntax Highlighting by EnlighterJS

About the Author

Kimberly is a software engineer, and currently works as a Test Engineering Intern for Mozilla.

While she always enjoys learning new technologies, her current focus is python, and when she has free time (she doesn't), Angular & Node.

When not coding, Kimberly spends time with her three young boys in Durango, CO.