QA/Firefox3/TestPlan

« QA/Firefox3

Top Critical issues for QA

This is a working list that QA is using to request be fixed or completed for the Upcoming Fx3 Milestones.

If you have untracked issues in these areas, please enter them here:

• Design still incomplete?
• Major bugs blocking your feature that should be addressed?
• Major bugs blocking overall regression of the trunk that should be fixed before shipping?

QA Beta 1 Relnotes Tracking
QA Beta 2 Relnotes Tracking

Must fix by Beta 3

• bug 390568 - vista fails to submit breakpad reports
• bug 378528 – crash reporter should attempt to resubmit pending reports. Needs to be rolled up into a queue for later if breakpad is down.
• bug 324857 – MathML all screwed up in Cairo builds. Blocking MathML usage
• bug 411369 - Crash while trying to print [@ @0x0 @0x1000 _cairo_meta_surface_create] [@ _moz_cairo_surface_finish ] - This is a major regression that is blocking any printing on Vista and XP.

Must Fix by Release Candidate

• bug 405198 Cannot drag and drop folders / items in Places Organizer - this cause the problem that its not possible to move bookmarks from one folder to another folder oder to merge folder etc .... - Tomcat
• bug 402335 - Meta bug for memory leaks. Tracked by tomcat. Ongoing testing and bugfixes.
• bug 391598 and bug 407780 SW update anti-fatigue. Need to improve modal dialogs for updates. Need to add to Fx3 testplan.
• bug 407459 - Hook up new allocator to our build. QA will probably not test this. noting it down in case we need to look at it.
• Critical Printing Bugs - These are printing crash bugs from printing. Bugs listed in this query are marked in Trunk, and have been confirmed by QA members.
  • In general, QA doesn't yet have an idea what "improve printing" means for this release; is it a metric-based, top-site notion, or something that includes some subset of prior-known, broken-printing pages, or...?
    • - stephend
• bug 404024 - Add AMO integration pane. Testplan completed, but no date set yet. Tracking spec is at: http://wiki.mozilla.org/Update:Remora_Meeting_Notepad. For Addons Manager UI, last update was 4 checkins were pending UX review for b3.
• Bug 401301 - firefox.exe process still around after all windows closed Still investigating firefox hang process. Hard to repro, but working with accessibility.
• bug 404619 - Firefox memory usage zooms when Norton Antivirus is running. This is creating a high memory usage reported by many in blogs. Needs more investigation. Same as bug 404645
• UPDATE - narrowed to one applet. P4 priority Bug 401064 – SnowFX applet requires a click to display. Affects some java plugin sites, but not all. Needs to be fixed, but doesnt block java testing.
• Mac Focus issues still a problem; bug 354768.
• Accessibility: Download Manager, being redesigned; would like the UI solidified earlier rather than later.
  • bug 393248 The info button currently brings up a popup that is classed as a menu, which confuses AT. Also the search box has no accessible label. -TimK
  • In latest Trunk, the Info button is still there, and the search box still has no label. The changes have aparently not landed yet. (MarcoZ)

• I'd like to know in what form of autocomplete the final Firefox 3 location bar will/should take; again, I understand there are ongoing experiments in order to get the best fit--across the board in the application--I just want everyone to be aware that churn really takes a toll on solidifying testcase-writing/execution. (Boy, I really sound whiny, no? I honestly just want us all to be on the same page, QA feels really, really behind in most areas, testing-wise, but perhaps that's just my perception...)
  • - stephend

• Accessibility: bug 393398 Add Bookmark Dialog Inaccessible: Needs to be classified as a dialog or a window in order to work properly with most AT. Needs more work and can be post bedta 2. One screen reader has severe problems with the panel being classified as a menu, the others we test don't. This is also true for the new AutoComplete panel, which is currently being classified as a menu with alist within. (MarcoZ)
• Accessibility: Bookmark/Places Panel not Accessible, need menu items to duplicate toolbar entries.
  • DISCUSSION - beltzner mentions there is work being done already. TimK, please comment on what specifics are blocking.
    • There are equivalence in the Bookmarks menu to the toolbar buttons--for some reason they weren't accessible from my profile. However, there are sec508 concerns with the Places Organizer contained in bug 400703 and bug 402104 which will require more extensive testing and should be resolved by beta 2, if at all possible. -TimK
    • Since neither of these bugs have patches yet, this should slip past beta 2, but is a sec508 issue that needs to be resolved before shipping. (MarcoZ)

RESOLVED FIXED bugs, needing more information or testcases

• Bug 375750 - PDFs generated by the trunk are huge. The patch seems to make the size smaller, but the size is still not in line with what the branch size is
• Bug 402000 - Cannot be verified according to dev

Overview

Firefox 3 QA Test Strategy

The goal of this Overall Test Plan is to lay out our testing strategy, communicate it to others, and track the status.

Major QA Activities:

• Litmus Cleanup
• Litmus Triage Team
• New feature testing - see below
• L10n testing - TBD
• Milestone testing - TBD

Firefox 3 Central

Requirements and Ownership

• Fx3 Product Requirements
• Fx3 Dev team up-to-date feature tracking status

Archives:

• Firefox 3 TestPlan Tracking page (Google Spreadsheet)
• Fx 3 PRD Wiki page

Test Areas

Feature Focused Areas

Key: L = Large (3-6 weeks) M = Medium (2-4 weeks) S = Small (1-2 weeks)

Test Plan Template

Feature Area and Spec	Test Plan Location	QA Contact	Size	Description
Addons	Addons Test Plan	tchung (new Addons Manager UI, Extensions), stephend (Unified Plugin Manager)	M	Add-on management system, plugin management system, localized user-facing addon text, configuration, updates, Installation, Discovery, FUEL 0.2
Content Handling	Microformat Test Plan Web Application Test Plan Download Manager Test Plan	ctalbert, stephend (Download Manager)	L	microformats, Content Type Processing, download manager, MIME type support, User Interface, Media Plugins (needs definition), firebug (at risk)
Partner Distribution	TBD	tracy, ctalbert	M	High-Touch/Low-Touch distros, regression, updates, test automation
OS Platform Integration	Vista Test Plan	Tomcat, marcia	M	Vista Support (At Risk), MSI bundling, Airbag / Breakpad, parental controls (At Risk), Notification, Vista Installer
Gecko Platform Integration	Gecko Platform Test Plan	martijn, marcia	M	Cairo graphics, nxTextFrame migration to Thebes, Reflow Branch, Caret painting rewrite, Cross-domain XMLHttpRequest (Cocoa widgets covered under Mac Test Plan)
Mac OS Platform Integration (includes Gecko)	Mac OS Integration Test Plan Leopard Test Plan	Tomcat, marcia	M	Leopard support, Cocoa implementation, widgets, form controls, Growl Integration on Mac, Mac Installer
Places	Places Test Plan	Tracy, ABillings	L	Bookmarks, History Indexing, Data Management, User Interface, Extensibility, Performance
Location Bar	Location Bar Test Plan	Stephend	S	new Location Bar design. Auto-complete
Search / Tabbed Browsing / Visual Refresh	Search, Tabs, Visual Refresh Test Plan Tab Switching Test Mac Theme Test Plan	Tomcat (search), Marcia (tabbed browsing)	S	User Interfacing, Animations, Data management
Security, Privacy / Password, Identity	Security / Privacy Test Plan Password Manager Testplan	Tomcat, juanb	L	password management, Phishing protection, Private web browsing, Blacklisting (At Risk), password Manager, Password Security
User Support	TBD	Tomcat, ss	S	Help content, Reporter, feedback, Customization options Now owned by SUMO team - David Tensor.
L10N Testing	L10N Test Plan	ctalbert, tchung	S	test automation, new locales, string lengths, bookmarks, rss, search plugins
UI Accessibility	UI Accessibility Test Plan	MarcoZ	S	UI accessibility convention testing
Major Update Tests (2.0.x -> FF3)	Major Update Test Plan	tchung, juanb, tracy	M	update verification from 2.0.x builds to 3.0b1+. includes upgrades/downgrades/sidegrade cases
Memory Leaks	Memory Leaks Test Plan	tomcat	M	Checking for memory leaks and asserts on debug builds

Regression Areas

• Top Web accessibility Tests
  • Plugins (eg. Flash, java, AJAX)
  • Extensions (eg. Firebug, NoScript, Fotofox)
  • Themes (eg. Vista, Joga, Simpsons)
  • Top Site Tests (eg. Alexa top 50)
  • Financial Sites (eg. banks, online trading)
• Accessibility Tests
• User Performance Testing
• Major / Minor Update Testing
• L10N Testing (may coincide with Feature area)
• Distribution Testing (may coincide with Feature area)

Ownership Needed Litmus Test Case Areas

These are Litmus test case areas that are currently unowned.

• RSS->marcia
• Printing -> Tomcat, tchung
• Location Bar -> Stephend
• Find -> TBD
• Customize Toolbar -> TBD
• Mac Leopard -> marcia

Tiger Team Testing

Daily smoketests are ran against nightly trunk builds. Please see marcia for more information.

Link to Schedule and Assignments

Extensions

For Firefox 2 we had Bug 353206 as extension Tracking bug.

The following extensions were never fixed from their extension compatibly authors to be able to install in Firefox 2.

• Slim Extension List [1]
• Bookmarks Synchronizer 3 [2]
• AutoMarks [3]
• Extended Link Properties [4]
• Snapper [5]
• Cacheout[6]
• Resize Search Box [7]
• superT 0.7.9.1 https://addons.mozilla.org/en-US/firefox/addon/2224
• Reveal extension [8]

Top Crash reports

The main report is the easiest to view.

Note that of those in that list, only a few seem to be active lately. I filed bug 402661 for the crash in RtlpCoalesceFreeBlocks / XPCWrappedNative::CallMethod(XPCCallContext&, XPCWrappedNative::CallMode). I still need to file a bug for the crash in CallGetService(nsID const&, nsID const&, void**). Neither of these should be blocking beta given their (relative) infrequency.

Banking and Financial Sites

For Firefox 3, we’re asking community members and MoCo QA to state which banks and financial institutions they're willing to test. Please add your name to the list.

Review Feedback

In this section, we will roll up all common feedback pulled from external sources that appears common and should be tracked by QA. Feedback sources can include:

• Hendrix Feedback system (mozilla.feedback newsgroups)
• Bugzilla
• Breakpad Crash Reporter
• Mozillazine Forums

Feedback summary and review notes are posted here.

Flash Beta plugin Testing

• http://weblogs.macromedia.com/labs/archives/2007/11/flash_player_9_6.cfm
• Team members actively testing betas
  • marcia
  • abillings
  • stephend

Additional References

New FF3 Feature list - An incomplete list of what Firefox 3 developers are working on.