In a nice moment of synchronicity, Ben Goodger has a post on his blog isolating some of the problems the mozilla.org faces (growing pains, really), some of which i've been thinking about for the last week or two. A lot of the issues can, I believe, be solved by using more consistent systems of information delivery and documentation. I'm not a developer and can't speak at all about the systems that developers need, this is just my own point of view as a QA tester in New Zealand and hapless champion of Mozilla products. The two main things I would find most useful, and solve problems I have encountered, are Consistency and Documentation. Mostly I want these to achieve autonomous testing, whereby I know exactly where to go and what to do. The other point of view i suppose i'm also thinking about is for the merely curious, those who want to try out a nightly trunk or branch build to see where development has progressed.
Consistency
When I first started testing, the learning curve for getting involved with testing was made easier by the members of the QA team in the QA irc channel, as it still is today. However a number of time difference problems arise when you're living so far away from Mozilla HQ and you need information that can't be answered by one of the other channel participants, and is usually not found in Mozilla wikis or pages. In my mind it's the human Mozilla HQ participants that make all the difference. Why? Well, often testing and development mean that you need up to the second information - information that currently needs other human beings to answer.
Many of the problems I and others faced, initially, is simply put, the 'barrier to entry'. No matter whether you have just downloaded Firefox or Thunderbird, or you've been a hard-core user of either since their very first release - you should be, and mostly are capable of testing. Most of the standardized tests allow anyone to test candidates, but it's getting to the testing that is often the most painful when the information you need has to come from someone who knows where to go and what information you will need. There is no one consistent site of information for testing purposes. You can visit the Mozillazine QA site or you can scour various wikis or Mozilla web pages for information. Half of the problem lies with the fact that a great deal of that material remains static and the information is by no means comprehensive, it is also very time-consuming to find what you need.
Prime Example One: Which build needs testing?
Initial pointers to builds are often posted to the QA Mozillazine page, however, it is often the QA leaders who know where most effort is being placed for the particular candidate and platform - and problems, if any, associated with testing particular candidates. Lest we forget that product candidates are cross-platform. For example, i'm on a G4 using 10.4.6, but who is testing a PPC candidate on a Blue and White G3 using 10.3 or 10.2? And given the introduction of the Intel Macs, who on earth is testing the Universal build candidate - and for that matter, are they testing it on a PPC or Intel Mac? Now that Boot Camp and Parallels Workstation are here, the possibilities for more testing problems may or may not be introduced on the Mac. Information needed is sometimes knowing, are there any particular 'gotchas' with the build? For example, is it known amongst the testing group that this particular build has problems with SVG graphics or an error with a key-command? Instead of wasting time with something that is already known, or worse yet, filing a new bug that is already known about, something far simpler than searching bugzilla and/or searching for or visiting multiple web pages needs to exist.
Two things I would love to see created.
I would love to see an RSS feed that is directly linked to Litmus logins that give me an up to the minute itemized list of the following statistics:
- How many people are testing
- Which platform are they testing on
- What build are they using?
If I have that information I know that more emphasis needs to be placed on testing a Firefox (Universal or regular PPC) candidate or Thunderbird build. I would also like links that get updated, when needed, that point me to all of the builds that need testing. For example, I currently have Ubuntu on a different drive I can boot into, is there a PPC linux build of Firefox or Thunderbird that no one *at all* is testing? (sadly, not that there are any being built right now). My time is valuable, if there are 10 testers working on 'Mozilla/5.0 (Macintosh; U; PPC Mac OS X Mach-O; en-US; rv:1.8.0.2) Gecko/20060328 Firefox/1.5.0.2', then wouldn't I be better serving the testing community to work on a Linux build or something else?
The other thing i would like to see is the creation of a basic document (old-school, friendly FAQ) about testing for people who would like to help (i'd be more than happy to write this or contribute). This document would include, at a minimum:
- Static url's for testing candidates
- Explanation of Bugzilla
- Explanation of Litmus
- A preferred methodology of testing
- Reasons for testing candidates
- What is a Branch and what is a Trunk and why on earth do these numbers not match the version i'm testing?
I would also love to see a QA interim bug site before you even engage the Bugzilla beast. What I mean by this is a daily/build page that bug testers can contribute to, for example questions such as,
"Has anyone seen weird colour changes while typing in the find input field using the Universal build of Mozilla/5.0 (Macintosh; U; PPC Mac OS X Mach-O; en-US; rv:1.8.0.2) Gecko/20060328 Firefox/1.5.0.2?"
Information like this would benefit from being verified as a problem by at least one other person.
Often these kinds of questions need input from others, is this a bug or maybe something peculiar to this build, or even my computer? I can ask in the IRC channel, but depending on time differences and whether others are actually actively engaged with checking activity in the room, well, you might not get a response at all.
This page could also include those 'gotchas' with builds I mentioned previously. For example, if the testing group know that this particular build has problems with SVG graphics, it can be listed on the page.
These are just some things i would love to see happen and would like to help realise. I'd love some feedback on what other community folks think is wanted or needed.