Testing on multiple browsers - looking for opinions

[mgilly]

Almost a year ago, mcowan asked about testing a web app on multiple browsers. (See the thread here.) He wondered what was the best way to organize tests for reporting when it was possible for a test case to fail on one browser but pass on another.

Passmark replied, in part:

Yes configurations was meant for this type of thing. However we assumed that if a single config failed then the entire test was a fail until all configurations passed.

Passmark went on to recommend possibly creating a test for each browser, or creating a project for each browser.

I was just wondering if anyone else had any thoughts on this? I'm looking at exactly the same situation. I am going to be testing an e-commerce application on a multitude of browsers and operating systems. I'd like to be able to report on my status and show, for example, that the shopping cart test is passed on IE7, failed on IE6, and not yet attempted on Firefox.

My thoughts about a solution, in no particular order:

1. Create a separate project for each browser/OS combination (there could be many). This means duplication of the test plan, which IMO really isn't necessary.
2. Create generic test cases and then copy them to my project multiple times, naming each one for the browser used and applying the appropriate configuration.
3. Create suites within my project for each browser, and copy generic test cases there with appropriate configurations assigned. I think this may be the best solution because, since I can create suites within suites, I can still break my tests up into functional areas by suite, either above or below the level of the browser suite.

I actually would be willing to assume that a test case fails overall if it fails on one browser, but I still need to break things out by browser for time esimtation and reporting purposes. Having multiple configurations assigned to the same test cases presupposes that I'll be able to run my test case on each browser required, in sequence, and that almost certainly won't be possible. It's far more likely that I'll end up running most or all test cases on one browser, then starting a different VM and running them on another browser, and so on.

So, kind of "thinking out loud" here, but just wondered if any users had any advice.

BTW I'm running TestLog in evaluation now but I love it, and I'm definitely going to ask my project manager to buy a few licenses. It's a great tool for creating, organizing, and reporting on test cases and test plans, especially for those of us who will never have the budget for the "big guys" like Test Director, etc. Thanks for a great product!