Noisy Air

A while back I had an idea to write a Python tool that would inspect the sourcecode of a project and report on any accessibility pitfalls it could determine. This was motivated by the trend in software development whereby the majority of the UI is specified in external files (e.g. NIB files in Cocoa, resource files in Android, etc). Since these files are XML-based, they can be parsed and compared against a profile of desirable accessiblity properties. For example, all fonts should be at least 14 pt, or all widgets should have an alternate description associted with them.

I have made some progress with this and plan to build on it when I have time. In the meantime, however, I've come across a great tool included in Mac OS called the Accessiblity Verifier. This tool was highlighted in a talk by Martin Pilkington at this year's NSConference. Doing a spotlight search for "Accessibility Verifier" should reveal the application. Once launched, the user selects a running application they wish to evaluate.

Mac OS Accessibility Verifier main window

Once an application is selected and the tests are run, a report is generated outlining accessibility errors and warnings. The following figure shows a report on Firefox 3.

Accessibility Verifier report output from the evaluation of Firefox 3

Pilkington's MDN blog post explains the four types of errors this tool checks for; they are summarized as follows:

• Parent / Child relationships: Ensuring that parent and children agree they are related to each other. Apparently these errors rarely occur.
• Window issues: Finding UI elements within a window and ensuring that these objects' attributes are set to the proper window.
• Missing descriptions and titles for UI elements.
• Role Verification: Making sure that UI elements are in the proper role. An example described by Pilkington would be ensuring a Textfield shows up as a Textfield and not an AXButton.

Testing

To assess the current state of accessibility on the Mac desktop platform, the Accessibility Verifier was used to evaluate various classes of sofware.

Web Browsers

The Accessibility Verifier was run on four Web browsers for Mac OS 10.5 (Leopard): Mozilla Firefox, Mozilla Camino, Opera and Safari. The following chart summarizes the results of the tool's reports:

A chart compraing Mac OS web browsers, using the Accessibility Verifier tool.Note that the chart uses a logrithmic scale on the y-axis. The most prominent observations include the following: First, Camino seems to have the fewest errors out of the entire lot. Second, Safari has the most errors out of the bunch. Finally, by far the most errors occured in the Role Verification group.

Word Processing (Native) Applications

Word processors were chosen to represent native Mac OS applications for testing purposes. The following figure shows the Accessibility Verifier results for three popular word processors.  One point of contention may be that Open Office isn't truely native; it is an application that requires the Java runtime.  Java based applications are dicussed in a subsequent section.

Accessibility Verifier results for three word processing applications, representative of native Mac OS applications.

One point of interest is the relatively high number of element retrieval errors present in the Microsoft product.  As in the previous figure, Role Verification is the most prominent error category.

Adobe AIR Applications

The Accessibility Verifier was also run on three representative Adobe AIR applications. AIR (Adobe Integrated Runtime) is a secondary cross platform runtime on which developers can distribute platform independant applications.

Accessibility Verifier results for 3 representative Adobe AIR applications.

Note the scale of the y-axis. AIR applications resulted in far fewer errors in all cases than did other categories of software.

Java Applications

Java programs are another, alternative form of software that requires its own runtime (in this case, a Java Virtual Machine). Three Java based applications were selected and evaluated with the Accessibility Verifier. The results are shown in the fullowing figure:

Accessibility Verifier results for three representative Java based applications.

This figure shows that all of the Java applications have many problems in the Role Verification category.

Discussion

The motivation for these tests lie in the persuit of classifying the various tiers of software that are emerging in the desktop and mobile environments, with respect to accessibility.  Native applications have long been used by developers to produce the most powerful software possible. However, other factors such as development cost, time, convenience and portability intice developers to create software at a higher tier, for example a Web application or a "hosted" application (by hosted I mean software that requires a runtime such as AIR or a JVM).

There are many conclusions that can be drawn from this data, as well as differing interpretations.  A first analysis suggests the following points:

Regarding Web browsers, Camino produced the fewest errors - even fewer than Apple's native Web browser (Safari).  This may be due to the fact that Camino is all around a simpler browser with fewer features than the rest. Further, this is ironic that Safari produced the most errors out of all the browsers, seeing as it is Apple's own Web browser.

Of the various word processors, MS Office produced a distinctly high number of element retrieval errors.  This may suggest a lack of conformance to more fundamental programming paradigms on the Mac platform, beyond accessibility related functionality. 

When it comes to hosted applications (those supported by AIR, JVM, ect) there is a large discrepancy in results.  AIR applications produced miniscule amounts of errors, while Java applications were three orders of magnitude larger.  That said, I believe these results are misleading.  Consider two applications: Eclipse (a Java application) and TweetDeck (an AIR application).  While Eclipse produced far more errors than TweetDeck, Eclipse seems to be the more accessible application.  For example, all of the buttons and widgets within Eclipse have associated tooltips, while TweetDeck does not.   My single opinion on the accessibility of these two applications should not be taken as fact; that wasn't the point.  My point is that hosted applications are hidden by their runtime environments, making tools like the Accessibility Verifier ineffective for evaluation.  This problem is even more present in the Web application domain.  Tools like Gmail, Facebook and Twitter are completely opaque to the Accessibility Verifier; the most it can do is evaulate the brower in which these applications are housed. 

In the mobile realm, the lines between Web app, native app and hosted are distinctively more blurred.  Many native applications include "Web views" whereby a portion of the native application is completely powered by the Web and a browser.  HTML 5 will bring many more capabilities to applications (native hardware support, 2D/3D graphics, geolocation).  This may in turn lead to more accessibility features for Web applications. 

When it comes to the Accessibility Verifier, it is limited to evaluating software in an active, running state.  This makes for difficulty when evaluating software targeted iPhone OS, as only one application can be running at one time.  One option may be to use the iPhone simulator (this is currently unverified).  In either case, it would be nice for developers to be able to evaluate software before runtime, as it would save time in many cases.