Mozilla Internship: Diving into Test Automation with Python

June 25, 2017

Test Automation with Python

The first week of my internship was spent primarily setting up my dev environment on the laptop I received.

I had also never used a Mac before, so there were some basics to learn as well.

Although, since MacOS is Unix-based now, all of the important things were pretty much the same.

(I'm worthless with a Windows command line)

Once I had everything configured, I started fiddling with python and pytest.

I've done some development in python, like a scrabble-esque game, but I'd never written tests in python before.

axe-core

In order to automate regression testing for accessibility, we need an API of some sort.

I did some research on the available web accessibility APIs before settling on the axe-core API.

The axe-core API was created by Deque, a company that specializes in accessibility.

Deque offers assessment services, certifications, and more.

axe-core is written in JavaScript and distributed as an an npm package.

Eventually, I will create a more seamless integration of axe-core into python, by writing a python library.

DequeLabs did this with Java in their tool axe-selenium-java.

To get started, though, I will be using Selenium's execute_script() function to handle the JavaScript directly.

Testing with aXe

The aXe API tests a website against a configurable set of rules.

These rules are based on WCAG 2.0, Section 508, and best practices endorsed by Deque.

The test loads an instance of Firefox, injects the axe-core script, and then injects a custom script that runs the API.

I didn't have a way to directly pass data from the JavaScript back to python (this will be addressed by the axe_selenium_python package I'm writing).

For the purpose of this test, I used JavaScript to write the JSON results to an element in the DOM.

I then grabbed the contents of that element using Selenium in my python test.

For my first iteration of the test, I simply asserted that there were no violations with an impact of critical.

test_critical_violations.py
[python]
class TestCriticalViolations:

@pytest.mark.nondestructive
def test_accessibility_critical_violations(self):
# List to hold info about critical violations
criticalViolations = []
data = json.load(open('./result.json'))
# Iterate through violations
for item in data['violations']:
# Find all critical violations
if item['impact'] == 'critical':
# Add description to list
criticalViolations.append(item['help'])

# Assert that no critical violations are found
assert len(criticalViolations) == 0, 'Critical Failures found'
[/python]

script.js
[js]
// Get axe-core rules
// Default setting is all rules
axe.getRules();
// Run axe
axe.run()
// On success, process results
.then(function(result){
console.log(result);
// Get element
var window = document.getElementsByTagName('html');
// Create new div element
var node = document.createElement('DIV');
// Populate div with results text
var textNode = document.createTextNode(JSON.stringify(result));
node.appendChild(textNode);
// Add selector to element
node.setAttribute('id', 'axe-result');
// And append to element
window[0].appendChild(node);
});
[/js]

This first version of my test used the python packages pytest and selenium.

I modified the test to use the pytest-selenium package, written by Dave Hunt of the Firefox Test Engineering team.

This package adds some additional functionality, and makes testing with selenium a little easier.

I added another package, pytest-html (also written by Dave Hunt), to generate an HTML report of the pytest results.

I also implemented the tests with tox, so that I could run the tests in both Python 2.7 and Python 3 simultaneously.

I then wrote a test for each rule used by axe-core, to get more meaningful output from the test suite.

Instead of seeing that critical violations were found, I am now able to see each test that passed and each test that failed.

test_accessibility.py
[python]
. . .

@pytest.mark.nondestructive
def test_accesskeys(self):
"""Ensures every accesskey attribute value is unique."""
assert test_results.get('accesskeys') is None, test_results['accesskeys'].help

@pytest.mark.nondestructive
def test_area_alt(self):
"""Ensures
elements of image maps have alternate text."""
assert test_results.get('area-alt') is None, test_results['area-alt'].help

@pytest.mark.nondestructive
def test_aria_allowed_attr(self):
"""Ensures ARIA attributes are allowed for an element's role."""
assert test_results.get('aria-allowed-attr') is None, test_results['aria-allowed-attr'].help

. . .
[/python]

[gallery columns="2" size="medium" link="file" ids="497,498"]

Now that I understand how to use pytest and Selenium, the next step will be learning how to write Python packages.