Introducting Grafx Bot

One of the new features of Firefox 4 is graphics hardware acceleration.   This, along with the new layers code, will help improve Firefox performance during things like page rendering and full-screen video.

Firefox’s hardware acceleration interacts with a machine’s graphics hardware via DirectX or OpenGL, depending on platform.  These interactions tend to be very sensitive to the graphics environment on the system (e.g., the specific video card(s) on the system, how much VRAM is available, the version of the video driver, the OS version, etc).  In fact, there are so many permutations of the relevant factors that we can’t test them all internally.  We need help from the community, so we can get exposure on as many unique hardware environments as possible.

To answer this need, I developed Grafx Bot.  It’s an add-on that end users can download, which runs a suite of automatic tests on their machine that exercises interesting aspects of hardware acceleration.  At the end of the tests, it allows users to post their results to Mozilla, where the data will be collected and analyzed, and hopefully lead to bug fixes and more reliable code for hardware acceleration than we’d otherwise have.

How it works

Grafx Bot is basically a new UI wrapped around Mozilla’s existing reftest framework.  Reftest works by comparing the visual output of two different pages, which are supposed to be rendered identically.  For example, this page and this page should be rendered identically, even though their markup is different.  If reftest detects that the pages are rendered differently, it fails the test.

Grafx Bot runs a series of reftests that are designed to exercise the hardware acceleration code.  Primarily, these are tests in the following categories:  css-transitions, layers, ogg-video, scrolling, svg, text, text-decoration, and z-index.  While the tests are executing, users will see images and colors flicker across their screen.

It’s possible that graphics hardware acceleration will produce small, but invisible changes in rendering between test pages.  We don’t really care about these, we only care when the differences are visually apparent.  Because of this, when Grafx Bot detects a reftest failure, it will ask the user if the differences are visible, by displaying both test images side-by-side.

Once the tests are complete, it will ask the user to click a button which submits the test results to Mozilla.  After the data is in our database, the results will be reviewed, and bugs will be filed as needed.

I recommend that users install Grafx Bot in a clean profile with no other add-ons or extensions installed.  This is because certain add-ons can interfere with Grafx Bot, or can cause it to run tests much slower than it would otherwise be able to.  For instructions on how to create a new profile in Firefox, see this article at support.mozilla.org.

Types of acceleration tested

There are multiple types of hardware acceleration being developed.  Here’s what Grafx Bot tests.

Grafx Bot runs the reftests with MOZ_ACCELERATED=11. This enables acceleration of layers via OpenGL (on Linux and Mac) and Direct3D (on Windows).  On Windows, Grafx Bot also toggles Direct2D acceleration.

The latter type of acceleration can be toggled dynamically, whereas the former cannot.  For this reason, on Windows only, Grafx Bot runs twice as many tests as on the other platforms.  It basically runs each test twice, with different settings, like so:

• test 1a: test file with D2D enabled vs reference file with D2D enabled
• test 1b: test file with D2D enabled vs test file with D2D disabled

Test data

Along with the results of all the reftests that Grafx Bot performs, we collect anonymous hardware information.  This allows us to associate test failures with specific hardware.  Users can see the system information we collect before they submit test results by clicking on the “System info” link on Grafx Bot’s homepage.  It looks something like this:

Even though this data is anonymous, this data is protected by Grafx Bot’s privacy policy (see link from the add-on’s homepage), and so is available only to Mozilla and to the user who submitted the test data.

Aggregate data, on the other hand, can be viewed by anyone.  Click here to peruse Grafx Bot data summaries.

More info

Want more information on hardware acceleration features in Firefox?  Check out these blog posts from Firefox graphics developers:

Hardware accelerating Firefox

Firefox video goes up to 11

Firefox and Direct2d: performance analysis

Direct2d: hardware rendering a browser