JGriffin's Blog
Is this thing on?
GrafxBot Results Update
Posted by on September 22, 2010
Thanks to the many thousands of you who have downloaded GrafxBot and submitted test results to Mozilla! In case you’re curious about what we’ve done with all that data, here are some statistics:
- According to AMO, GrafxBot has been downloaded about 4700 times.
- We have nearly 30,000 sets of test results that were submitted to our database, for a total of 3.7 million tests.
- The test results span 282 unique video cards on Windows, 30 on Mac, and 171 on Linux.
- The failure rate (which is somewhat subjective given the manual pass/fail mechanism) averages around 0.4% on Windows, 0.5% on Mac, and 1.1% on Linux.
- Test data has resulted in a total of 37 bugs being filed.
Aside from the raw test results, many of you have submitted useful comments. Some have noted that fonts look bad when Firefox is accelerated; others have described scrolling or other issues. Not all of these problems can be detected by GrafxBot, so if you notice problems like these when browsing, I encourage you to file a bug report in Bugzilla, under Core -> Graphics. If you submit a bug report, please include the details of your graphics hardware, and include a screenshot if possible.
GrafxBot continues to be updated along with Firefox betas, so I encourage interested folks to continue running GrafxBot each beta release. Thanks for all your help in making Firefox 4 the fastest ever!
I’m glad to see people volunteer in such experience.
In the last test I’ve run (it was using Firefox 3.6, maybe I should not run grafbox on it ?), I have got a pass ratio of 65%, this is much bigger than the ~1% average failure rate, isn’t it ?
After Grafbox has finished the tests, it often asks me to compare 2 images, and press success/fail. Did you tried to automate this process ? I think about images diff tools or saving the pictures (using canvas ?) then comparing the files.
There’s no need to run it on 3.6; the purpose of GrafxBot is to help test hardware acceleration, and hardware acceleration isn’t in 3.6 in any form.
I have tried to automate the pass/fail process. Currently, GrafxBot compares canvases, and if they’re different, it asks the user to judge. The reason we do this, instead of automatically failing, is that some differences will be just a couple of pixels (due to pixel rounding issues and the like), which won’t be visible to the user, and we don’t consider this a real fail. It’s possible we could apply a Gaussian blur to both images, and compare the results, in order to weed out problems like this, but then it’s possible we’d miss out on real, visually apparent failures. So there’s no great solution.
I have had a number of suggestions on how to improve the UI for this, however, and I hope to implement them at some point.
does grafxbot also do profiling? it would be nice to know how much of a speedup there is with acceleration.
Also on linux is it worth checking if there is a compositing window manager (compiz) running?
GrafxBot does a limited amount of profiling on Windows, but not on Mac or Linux, since on those platforms, it is impossible to dynamically toggle hardware acceleration without a browser restart.
How about update add-on to Firefox 4.0.* ?
Nice will be also creating version for 64bit Firefox on Win7 64bit.
Win7 64-bit isn’t officially supported for Firefox 4.0, so I’m not releasing a version of GrafxBot for that platform yet.
I’ll update the add-on version to support 4.0.* with the next GrafxBot update.
Hmm…
Odd, they seems to officially release 64bit builds here
http://nightly.mozilla.org/
and here too
https://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/latest-mozilla-central/
And even if they don’t, nice will be supporting this version, because more bugs can be found and fixed in the future.
We do make 64-bit Windows builds, but we don’t test them, and bugs only involving 64-bit Windows will receive no attention until post-4.0.