QA/Firefox3/TestPlan

« QA/Firefox3

Top Critical issues for QA

This is a working list that QA is using to request be fixed or completed for the Upcoming Fx3 Milestones.

If you have untracked issues in these areas, please enter them here:

• Design still incomplete?
• Major bugs blocking your feature that should be addressed?
• Major bugs blocking overall regression of the trunk that should be fixed before shipping?

QA Beta 1 Relnotes Tracking
QA Beta 2 Relnotes Tracking

Must fix by Beta 3

• Top crashes
  • bug 414747 - Firefox crashes during Addons -> Find Update [@nsQueryInterface::operator()]. Update - seems to be caused by regressions from bug 391002. Approved by drivers for blocking b3
  • bug 411249 – Top crash [@ jsds_ScriptHookProc[@jsds_ScriptHookProc(JSDContext*, JSDScript*, int, void*)] on shutdown with Firebug 1.1.0b10 installed. Update - work in progress
• bug 414056 – Minotaur - Re-running tool on a second run does not output default bookmarks on FFx 3b2+. this blocks minotaur testing for b3.
• bug 414808 - ###!!! ASSERTION: Oops! You're asking for a weak reference to an object that doesn't support that.: 'factoryPtr'. Security impacts? Update - nominated for blocking b3
• bug 414930 - Unable to drag and drop URLs to Bookmarks Toolbar. This is a serious regression. Update - approved by drivers for blocking b3
• bug 414922 – RSS Subscription broken. This is a serious regression. Update - Fixed and verified for b3
• bug 405198 Cannot drag and drop folders / items in Places Organizer - this cause the problem that its not possible to move bookmarks from one folder to another folder oder to merge folder etc .... - Tomcat Update - moved to beta 4
• bug 414829 – Not possible to remove "Home" text label from the new Home icon on bookmarks toolbar. Is this a new design? this could get annoying, but open to more thoughts. Update - moved to beta 4
• bug 414933 – Return star to a one click bookmark action. Wasnt the original intent of starring to do with one-click? Update - need to flag for b3 blocking or not?
• bug 324857 – MathML all screwed up in Cairo builds. Blocking MathML usage. Lots of action in the bug, but no resolution yet? - Update - lots of patches landing. on track with fixes. Need to flag for b3 blocking or not?

Must Fix by Release Candidate

• Breakpad issues
  • bug 394247 – check that nsITransfer::Init succeeded (crash [@nsDownloadProxy::OnStateChange])
  • bug 412008 – Can't access report/list for 1.9 branch topcrashes
  • bug 411438 – Crash reporter app should offer restart Firefox button when it throws an error dialog
  • bug 378528 – crash reporter should attempt to resubmit pending reports. Needs to be rolled up into a queue for later if breakpad is down. This causes backlog of missing crash data that takes days later to resurface. Takes away time from researching critical issues. update - marcia to retest
• bug 402335 - Meta bug for memory leaks. Tracked by tomcat. Ongoing testing and bugfixes. -Tomcat- Triage of this bugs will start this week
• Critical Printing Bugs - These are printing crash bugs from printing. Bugs listed in this query are marked in Trunk, and have been confirmed by QA members.
• bug 391598 and bug 407780 SW update anti-fatigue. Need to improve modal dialogs for updates. Need to add to Fx3 testplan.
• bug 407459 - Hook up new allocator to our build. QA will probably not test this. noting it down in case we need to look at it.
• bug 404024 - Add AMO integration pane. Update - fixed landed for b3, in test now
• bug 401301 - firefox.exe process still around after all windows closed] Still investigating firefox hang process. Hard to repro, reported by accessibility team and important for the community to resolve.
• bug 404645 – memory usage spikes dramatically while doing nothing with fresh profile. High memory utilization reported by many blogs. Update - fixed landed for b3. Need testing from tomcat
• bug 403232 - Focused textfield loses focus when switching to another tab and back. UPDATE - fix landed for b3, to be tested by Marcia
• Accessibility issues
  • bug 393398 Add Bookmark Dialog Inaccessible: Needs to be classified as a dialog or a window in order to work properly with most AT. Needs more work and can be post bedta 2. One screen reader has severe problems with the panel being classified as a menu, the others we test don't. This is also true for the new AutoComplete panel, which is currently being classified as a menu with alist within. (MarcoZ)
    • Since bug 407359 landed, this must be revisited for only Window-Eyes. JAWS, Orca, and NVDA work fine now. MarcoZ
  • Bookmark/Places Panel not Accessible, need menu items to duplicate toolbar entries.
    • There are equivalence in the Bookmarks menu to the toolbar buttons--for some reason they weren't accessible from my profile. However, there are sec508 concerns with the Places Organizer contained in bug 400703 and bug 402104 which will require more extensive testing and should be resolved by beta 2, if at all possible. -TimK
    • Since neither of these bugs have patches yet, this should slip past beta 2, but is a sec508 issue that needs to be resolved before shipping. (MarcoZ)

RESOLVED FIXED bugs, needing more information or testcases

• Bug 375750 - PDFs generated by the trunk are huge. The patch seems to make the size smaller, but the size is still not in line with what the branch size is
• Bug 402000 - Cannot be verified according to dev

Overview

Firefox 3 QA Test Strategy

The goal of this Overall Test Plan is to lay out our testing strategy, communicate it to others, and track the status.

Major QA Activities:

• Litmus Cleanup
• Litmus Triage Team
• New feature testing - see below
• L10n testing - TBD
• Milestone testing - TBD

Firefox 3 Central

Requirements and Ownership

• Fx3 Product Requirements
• Fx3 Dev team up-to-date feature tracking status

Archives:

• Firefox 3 TestPlan Tracking page (Google Spreadsheet)
• Fx 3 PRD Wiki page

Test Areas

Feature Focused Areas

Key: L = Large (3-6 weeks) M = Medium (2-4 weeks) S = Small (1-2 weeks)

Test Plan Template

Feature Area and Spec	Test Plan Location	QA Contact	Size	Description
Addons	Addons Test Plan	tchung (new Addons Manager UI, Extensions), stephend (Unified Plugin Manager)	M	Add-on management system, plugin management system, localized user-facing addon text, configuration, updates, Installation, Discovery, FUEL 0.2
Content Handling	Microformat Test Plan Web Application Test Plan Download Manager Test Plan	ctalbert, stephend (Download Manager)	L	microformats, Content Type Processing, download manager, MIME type support, User Interface, Media Plugins (needs definition), firebug (at risk)
Partner Distribution	TBD	tracy, ctalbert	M	High-Touch/Low-Touch distros, regression, updates, test automation
OS Platform Integration	Vista Test Plan	Tomcat, marcia	M	Vista Support (At Risk), MSI bundling, Airbag / Breakpad, parental controls (At Risk), Notification, Vista Installer
Gecko Platform Integration	Gecko Platform Test Plan	martijn, marcia	M	Cairo graphics, nxTextFrame migration to Thebes, Reflow Branch, Caret painting rewrite, Cross-domain XMLHttpRequest (Cocoa widgets covered under Mac Test Plan)
Mac OS Platform Integration (includes Gecko)	Mac OS Integration Test Plan Leopard Test Plan	Tomcat, marcia	M	Leopard support, Cocoa implementation, widgets, form controls, Growl Integration on Mac, Mac Installer
Places	Places Test Plan	Tracy, ABillings	L	Bookmarks, History Indexing, Data Management, User Interface, Extensibility, Performance
Location Bar	Location Bar Test Plan	Stephend	S	new Location Bar design. Auto-complete
Search / Tabbed Browsing / Visual Refresh	Search, Tabs, Visual Refresh Test Plan Tab Switching Test Mac Theme Test Plan	Tomcat (search), Marcia (tabbed browsing)	S	User Interfacing, Animations, Data management
Security, Privacy / Password, Identity	Security / Privacy Test Plan Password Manager Testplan	Tomcat, juanb	L	password management, Phishing protection, Private web browsing, Blacklisting (At Risk), password Manager, Password Security
Breakpad Client/Server	Breakpad Crash Reporting Test Plan	ss	S	Crash reporting tests from client/server, analysis, load
User Support	TBD	Tomcat, ss	S	Help content, Reporter, feedback, Customization options Now owned by SUMO team - David Tensor.
L10N Testing	L10N Test Plan	ctalbert, tchung	S	test automation, new locales, string lengths, bookmarks, rss, search plugins
UI Accessibility	UI Accessibility Test Plan	MarcoZ	S	UI accessibility convention testing
Major Update Tests (2.0.x -> FF3)	Major Update Test Plan	tchung, juanb, tracy	M	update verification from 2.0.x builds to 3.0b1+. includes upgrades/downgrades/sidegrade cases
Memory Leaks	Memory Leaks Test Plan	tomcat	M	Checking for memory leaks and asserts on debug builds

Regression Areas

• Top Web accessibility Tests
  • Plugins (eg. Flash, java, AJAX)
  • Extensions (eg. Firebug, NoScript, Fotofox)
  • Themes (eg. Vista, Joga, Simpsons)
  • Top Site Tests (eg. Alexa top 50)
  • Financial Sites (eg. banks, online trading)
• Accessibility Tests
• User Performance Testing
• Major / Minor Update Testing
• L10N Testing (may coincide with Feature area)
• Distribution Testing (may coincide with Feature area)

Ownership Needed Litmus Test Case Areas

These are Litmus test case areas that are currently unowned.

• RSS->marcia
• Printing -> Tomcat, tchung
• Location Bar -> Stephend
• Find -> TBD
• Customize Toolbar -> TBD
• Mac Leopard -> marcia

Tiger Team Testing

Daily smoketests are ran against nightly trunk builds. Please see marcia for more information.

Link to Schedule and Assignments

Extensions

For Firefox 2 we had Bug 353206 as extension Tracking bug.

The following extensions were never fixed from their extension compatibly authors to be able to install in Firefox 2.

• Slim Extension List [1]
• Bookmarks Synchronizer 3 [2]
• AutoMarks [3]
• Extended Link Properties [4]
• Snapper [5]
• Cacheout[6]
• Resize Search Box [7]
• superT 0.7.9.1 https://addons.mozilla.org/en-US/firefox/addon/2224
• Reveal extension [8]

Top Crash reports

The main report is the easiest to view.

Note that of those in that list, only a few seem to be active lately. I filed bug 402661 for the crash in RtlpCoalesceFreeBlocks / XPCWrappedNative::CallMethod(XPCCallContext&, XPCWrappedNative::CallMode). I still need to file a bug for the crash in CallGetService(nsID const&, nsID const&, void**). Neither of these should be blocking beta given their (relative) infrequency.

Banking and Financial Sites

For Firefox 3, we’re asking community members and MoCo QA to state which banks and financial institutions they're willing to test. Please add your name to the list.

Review Feedback

In this section, we will roll up all common feedback pulled from external sources that appears common and should be tracked by QA. Feedback sources can include:

• Hendrix Feedback system (mozilla.feedback newsgroups)
• Bugzilla
• Breakpad Crash Reporter
• Mozillazine Forums

Feedback summary and review notes are posted here.

Flash Beta plugin Testing

• http://weblogs.macromedia.com/labs/archives/2007/11/flash_player_9_6.cfm
• Team members actively testing betas
  • marcia
  • abillings
  • stephend

Additional References

New FF3 Feature list - An incomplete list of what Firefox 3 developers are working on.