This article summarizes the results of a project conducted in the AFB TECH lab at the request of the Royal National Institute of Blind People (RNIB) in the United Kingdom. RNIB asked us to compare the performance of NonVisual Desktop Access (NVDA) and Thunder, two free screen readers for Windows computers. (Although we AFB TECH lab rats think very highly of Serotek's SA To Go and System Access screen readers, they were not a part of this study.) For this project, we compared the performance of NVDA and Thunder on several of today's common computer tasks. Read on for our findings and conclusions.
Product Descriptions
NVDA is a free, open-source screen reader that provides feedback via synthetic speech. It's compatible with Windows XP, Vista, and Windows 7, and is also compatible with refreshable braille displays. Since NVDA is an open source application, interested parties can contribute to the screen reader's improvement and development. For this project, we used NVDA version 2011.1. You can download NVDA and learn more about it at the NVDA website.
Similarly, the Thunder screen reader is free, provides feedback via synthetic speech, and is compatible with Windows 7, Vista, and XP. The documentation makes no mention of the screen reader's compatibility with refreshable braille displays, and it's not an open-source screen reader. For this project, we used Thunder version 2.021. You can download Thunder and learn more about it at the Thunder website.
Basis of Evaluation
We evaluated the speech output performance of NVDA and Thunder on several different Windows 7 and XP computers. We compared the performance of the two products in the following eight categories of computer use:
- Microsoft Word
- Microsoft Excel
- Online Banking
- iTunes
- Amazon and Bookshare
- UK National Rail Enquiries (Travel)
- General Web Browsing
We tested NVDA and Thunder with Microsoft's Outlook Express mail client, as well as with Web-based e-mail using AFB's Outlook Exchange system. We also tested NVDA with the full version of Outlook. Thunder does not support Outlook so we were unable to compare its performance with that program. (Thunder does support the Windows Mail client on Windows Vista PCs, but we do not have access to a Vista machine at the AFB TECH lab.)
Outlook Express, Version 6
Both screen readers worked to access menus, and to read, forward, reply to, and attach files to messages, but Thunder had problems with message composition. Thunder did not speak any of the form labels, such as To, CC, Subject, and Attach. Also, when tabbing into the body of the message, Thunder would say, "formatting bullets checkbox checked." Since Thunder does not speak any other field labels, this little quirk can be a helpful, if strange, indication that you are in the body of the message.
Outlook Exchange Web-Based E-mail
The log-in form was somewhat of a chore with Thunder. With Thunder, pressing the F6 key opens the form fields in a new window. There, we entered the user name in the user name form, then activated an "OK" button to get back to the mail page. After that, we had to find the log-in form again and repeat the same process to enter the password. When entering a password with Thunder, the form visually shows asterisks, but you can also set it to echo the characters as you type (something you should of course avoid when using speakers in a public place).
NVDA's table navigation functionality made it slightly easier to use than Thunder, since Outlook Exchange arranges messages in a table. That said, since each message is also tagged as a header, Thunder did not have a problem finding messages. Reading messages was easier with NVDA because a press of the N key jumps directly to the body of the message. To read messages with Thunder, it was necessary to browse down the page with the arrow keys or use the browser's search tool to find the body of the message.
We had to do a little bit of searching around the pages to find the correct buttons, but ultimately both screen readers worked for composing, replying to, forwarding, and attaching files to messages. (Though NVDA unfortunately read the "Attach" button as, simply, "Button.")
Microsoft Outlook 2003 and 2007
NVDA performed very well with Outlook 2007, but had a few problems with Outlook 2003. NVDA navigated to the attachments field in Outlook 2003, but did not read the listed attachments. We were unable to open the attachments directly from within the message, but we could save the attachments using the file menu. Attaching files to messages using the menus worked well with NVDA, but it only read "blank" when an attachment was selected from within a message.
While composing a message with Outlook 2003, NVDA did not read the auto-fill suggestions when entering an address into the "To:" or "cc:" fields. If auto-fill was not used, NVDA read the entered characters, but when we tabbed away and back again, NVDA read "blank" for those same entries.
In spite of these relatively minor problems most of the features and functions of both Outlook 2003 and 2007 were accessible with NVDA, including:
- Scrolling through the list of messages in a mailbox and learning sender, subject, date, and attachment and read/unread status for each message.
- Reading the content of each message either all at once or by line, word, character, etc.
- Accessing Outlook's menu system.
- Adjusting the font in a message.
- Replying to and forwarding messages.
- Accessing different mailboxes.
- Learning the total number of messages in a mailbox.
- Accessing and using Outlook's address book.
Microsoft Word 2003
NVDA and Thunder performed similarly when using Microsoft Word 2003, but NVDA did a better job of handling format attribute changes while reading a document.
Both screen readers did well opening, creating, saving, and printing documents. They worked well with Word's menu system and spell checker. Adjusting font and other formatting settings worked well, but Thunder had trouble with those format changes when reading a document. Thunder is designed to alert the user of format changes when scrolling through a document, and that can be very helpful when editing a document. It's not possible to mute this feature, however, and it can be very annoying and distracting when reading. In addition, when Thunder encountered a line that contained a format change, it would sometimes only read the new format, not the actual words on the line. We also found that Thunder sometimes reported all of the changed formatting (e.g., typeface, font style, size, etc.) and other times reported only a new font size.
NVDA's formatting reporting functionality, which is possible to silence, worked well.
As for navigating through documents, neither screen reader had a feature (like JAWS's quick keys) that allows you to quickly jump to various elements such as headings, tables and paragraphs in Word. There were available table navigation commands in both NVDA and Thunder, but neither was as robust as the JAWS or Window-Eyes table commands.
There were a few other anomalies to note with Word functionality. NVDA did not speak periods when moving by word through a sentence. It did read the periods in URLs and when moving by character. NVDA did not read dashes under any setting. Thunder did not accurately report letters deleted via the backspace key. Sometimes it reported the letter to the left of the one being deleted, and sometimes it simply said, "backspace." Thunder also did not report carriage returns when we arrowed to them.
Microsoft Excel 2003
NVDA was the clear winner in this category, performing similarly to the way JAWS does with Excel. NVDA worked well with all the features and functions we tested, including using Excel's menu system, opening, creating, saving, and printing spreadsheets, adjusting format settings, and creating and using formulas. It also worked well when the arrow keys were used to move around the grid to read cell contents and coordinates, and to edit cell contents.
Thunder worked well with many of the features and functions we tested, but it was inconsistent when navigating and reading spreadsheets. In our initial test, we opened a blank spreadsheet and checked to see if the arrows would move around the grid. They did, but Thunder reported only the column coordinates, not the row coordinates. Once we entered data into a couple cells, the arrows no longer worked to navigate the spreadsheet. By trial and error we found that Tab and Shift + Tab moved left and right, and Enter and Shift + Enter moved up and down. At that point, the Up arrow read the cell contents but instead of moving to the cell directly beneath the current cell, the Enter key moved to the far left column in the next row, making spreadsheet navigation cumbersome and frustrating.
When we tried again, Thunder did a much better job. Arrow keys worked to move around the grid, and Thunder reported proper cell coordinates. As with Word, Thunder read cell contents properly except when it encountered format changes and reported the format attribute change instead of the cell contents.
We then used a second computer to test Thunder with Excel and we experienced the same inconsistent behavior. Thunder's website does claim support for Excel, as does an article published in the National Federation of the Blind's Braille Forum magazine, but we found no resources for correcting this inconsistent behavior.
Online Banking
For this assessment, we used the Huntington National Bank website. This site works very well with other screen readers such as JAWS, Window-Eyes, and System Access. Thunder and NVDA completed every task we attempted, but NVDA was considerably faster and easier to use. NVDA and Thunder both had trouble reading the labels for the log-in fields, but when we checked the source code for the website, we found that the labels were coded incorrectly. We were able to pay bills and transfer funds with both screen readers, but NVDA was a lot more efficient in doing so. Thunder uses an application called WebbIE to browse the Internet (more on WebbIE below), and it did not always report the correct labels.
The Huntington website uses a lot of tables to check balances, pay bills, transfer funds, and review the check ledger. Thunder's lack of table navigation really slowed everything down to a crawl. Finding tables with Thunder required a lot of scrolling or searching. When in a table, Thunder read entire rows at once and sometimes mixed up rows.
NVDA was the clear winner with online banking, navigating the tables and forms for bill paying and transferring funds much more smoothly than Thunder. That said, NVDA's table navigation is not as strong as other screen readers like JAWS or Window-Eyes. While we were able to move up and down and left and right easily, NVDA did not report the row or column headers as we did so. NVDA also does not support jumping to the top or end of a table. When we used the T hotkey to jump to the next table, NVDA incorrectly said there weren't any tables on the page. We went to the source code and found that the tables were generated using JAVA, which may be the source of this problem. When we scrolled or searched to find the tables, NVDA navigated properly.
iTunes 10.2.1
Both screen readers were able to complete everything we needed to do in iTunes, such as navigating the interface, searching for and playing music and other media, using menus, creating play lists, and searching for and buying music in the iTunes store.
Thunder presented a few quirks when scrolling through the list of sources. Thunder consistently reported the previous source rather than the selected source. When we selected "movies," Thunder reported it as "music," for instance.
As with other screen readers, both NVDA and Thunder occasionally lost focus. This problem was easily fixed by pressing Tab + F6 to regain focus.
Amazon and Bookshare
We chose to evaluate these two sites because of their popularity for those of us interested in reading books and magazines. The assessment included signing in, searching for books, and downloading selections to a computer or memory card. On the whole, both screen readers completed the tasks on both websites without problems, though NVDA again performed better and more efficiently than Thunder.
Thunder took longer to fill out forms and navigate the webpages. Thunder had particular difficulty with search results on Bookshare, which are presented as linked titles. These links are also headings, so we used CTRL + H to scroll through a set of results. Unfortunately, Thunder put the text of the heading on one line, and the link associated with the heading on the next line, doubling the number of keystrokes it took to browse through the results.
UK National Rail Website
We assessed the National Rail website since it's widely used by RNIB's constituents.
Throughout our testing, NVDA consistently outperformed Thunder on webpages and there was no exception on the National Rail site. There were a few issues using NVDA, but we were able to complete all the necessary tasks to search for and purchase a ticket. Inefficiency with forms and difficulty with table navigation were significant problems on this site for Thunder, but the biggest problem was that we were unable to complete the ticket purchasing process.
Like most travel websites, the National Rail site can be very difficult for a screen reader to navigate, and searching for and purchasing a ticket is a very involved and cumbersome process. The site also uses an auto-fill function that visually presents a list of train stations as you type your origination or destination station in the form used to set up your travel. This list of station options could not be accessed by NVDA or Thunder, nor was it accessible using JAWS or Window-Eyes. This is not a major problem if you know the name of your desired station, but it would be a significant barrier if you were investigating or browsing options for travel. A combo box would work better for screen readers.
When filling in the forms, NVDA easily tabbed from form field to form field, but with Thunder we had to activate an "OK" button after each form field and then navigate back to the next form field. Using the calendar to choose a date for your trip worked with both screen readers, but once again the table was easier to navigate with NVDA.
General Web Browsing
Obviously NVDA handles the Internet in a more effective and efficient manner than Thunder. The NVDA Web browsing experience is very similar to that of JAWS and Window-Eyes. NVDA supports 26 hotkeys to quickly move to webpage elements. NVDA also has an elements list that provides access to all the links, headings, and ARIA landmarks on a page. NVDA's "focus mode" is used when interacting with forms, and it works much like JAWS's "forms mode" to effectively enter form data. Although most testing was done with Internet Explorer, NVDA also works with the Firefox browser with no discernable deficit to functionality.
Thunder uses an application tool called WebbIE, which works in conjunction with Internet Explorer to surf the Web. Although WebbIE does provide access to the Internet, it isn't nearly as feature-rich or efficient as NVDA's browsing functionality. WebbIE hotkeys are limited to forms, headings, and links. Although we were able to use Thunder to fill in form data, it was a much less efficient process.
NVDA is clearly stronger with table navigation, but neither screen reader was perfect in this area. NVDA lacks a command to move to the beginning or end of a table, which is a problem with large tables. In another glitch, sometimes NVDA responded incorrectly to the release of navigation keys.
As an example of how these two screen readers work with Web 2.0 elements, we tested the CVS website, which uses Adobe Flash. The site includes a banner ad that rotates through five different promotions, each containing marketing language and a link to further information. The banner was designed using Adobe's guidelines for making Flash accessible, and it worked well with JAWS.
NVDA was once again the clear winner. Although NVDA identifies Flash presentations generically by saying, "embedded object," entering on the object brings focus to the Flash presentation. NVDA then worked exactly like JAWS to read the content, move between the various promotions, and activate the links to further information. Thunder also reported Flash as an embedded object, but when we navigated into the actual presentation, it behaved inconsistently, sometimes moving to a new promotion, and never accessing any marketing information. In addition, we could not figure out a way to get out of the Flash presentation and back to the main webpage.
The Bottom Line
For those of us with vision loss, the fact that these two free screen readers (along with Serotek's SA To Go) exist is great news. We are appreciative of all who contributed to the development of both NVDA and Thunder.
Although we certainly appreciate what the people behind Thunder have done, in our assessment, NVDA proved to be the more robust screen reader. Although it had a few minor problems, NVDA is not that far behind JAWS and Window-Eyes in functionality, especially when it comes to Web browsing. One of our sighted lab rats, a computer science student at Marshall University, commented that he was very impressed by NVDA and its open source framework, and mentioned that he found the screen reader more efficient than using a mouse for many tasks. If you or someone you know is interested in beginning to investigate using a screen reader, we highly recommend NVDA .
