Releases/Firefox 14/Test Plan

« Firefox 13 | Firefox 14 | Firefox 15 »

Firefox 14 Test Plan

Summary

Milestone	Firefox 14
Status	Currently in Aurora -- merging to Beta on June 4, 2012
Lead	Juan Becerra (irc: juanb)
2nd	Anthony Hughes (irc: ashughes)
Builds	[ftp] [http] [web]

The following is the test plan for Firefox 14 from Nightly through to Release. Use this document as a reference for what is being tested to validate the quality of Firefox 14. After the release this document will be used as an archive of what was done to validate this release.

If you have some free time, please pick a task below and get in touch with one of the leads.

Details

The following is a description of the tasks which need to be completed throughout this release.

• 2012-03-13: [DONE] Nightly Merge
• 2012-04-24: [DONE] Aurora Merge
• 2012-06-05: {{|}} Pre-Beta feature sign-offs
• 2012-06-07: {{|}} Beta 1
• 2012-06-14: {{|}} Beta 2
• 2012-06-21: {{|}} Beta 3
• 2012-06-28: {{|}} Beta 4
• 2012-07-05: {{|}} Beta 5
• 2012-07-11: {{|}} Beta 6
• 2012-07-13: {{|}} Release Candidate
• 2012-07-17: {{|}} Final

Features

See the Feature Release Tracking wiki for a general overview

[DONE]

• Cycle Collector Perf improvements (Virgil Dicu); landed in Firefox 13

[ON TRACK]

• Windows8 Metro Firefox (Vlad Ghetiu)
• Hang Detector and Reporter (Simona Badau)
• URL Autocomplete (Vlad Ghetiu)
• Firefox Migrators (Paul Silaghi)
• Pseudo-Class Lock (Ioana Budnar)
• Support for Mouse Capture / Mouse Lock (Virgil Dicu)
• Opt-in activation for plugins (Paul Silaghi); Initial minimal implementation landed for 14 (opt-in), further work in 15 and 16.

[AT RISK]

[MISSED]

• Incremental garbage collection (Ioana Budnar); delayed to Firefox 15
• Panel-based Download Manager (Simona Badau) opt-in; delayed to Firefox 15
• Silent Update: Background updates (Vlad Ghetiu); delayed to Firefox 15
• Windows Start-up Performance Improvements (Jason Smith); delayed to Firefox 15
• Speedy Session Restore (Paul Silaghi); delayed to Firefox 15
• Silent Update: Updated workflow for users with incompatible add-ons (Ioana Budnar); delayed to Firefox 17
• Show PDF inline (Mihaela Velimiroviciu); delayed to Firefox 16
• Web Apps Integration to Desktop (Jason Smith); delayed to Firefox 15

Nightly Merge

Functional Automation - result

Update Spotchecks - result

Endurance Automation

The purpose of this section is to track any potential memory regressions via spikes reported in the dashboard.

What to Do

1. Check the Endurance Charts
2. If you see an anomalous event (ie. spike or dip), get the report for the specific event
3. If the event happens twice in a row, file a bug under Mozmill Tests
4. Notify Juan Becerra of the issue
5. Record the event in this wiki below

Current Status (Mihaela Velimiroviciu) - Last Updated

• bug 752887 - Increased memory usage for endurance tests on Linux Aurora branch

Functional Automation

The purpose of this section is to track any potential functional regressions via results reported in the dashboard.

What to Do

1. Check the testrun reports
2. If you see any failing tests, check that there are not any bugs on file for it already
3. If not, file a bug under Mozmill Tests
4. Notify Juan Becerra of the issue
5. Record the event in this wiki below

Current Status (Vlad Maniac / Alex Lakatos) - Last Updated

• No issues found

Unconfirmed Bug Triage

The purpose of this section is to triage UNCONFIRMED incoming bugs. The below charts provides a summary of the unconfirmed bugs currently in the Firefox product:

Bugzilla query error

GD extension must be loaded.1

P1 Task - Get UNCONFIRMED Bugs Out of Firefox General and Untriaged If Possible

1. Check this <insert link> for UNCONFIRMED bugs found for this Firefox Release's Beta Timeframe
2. Read each bug and determine what product and component they belong under in bugzilla
3. Move the bug to the right product and component
4. CC any applicable team members that may be interested in the bug

P2 Task - Confirm UNCONFIRMED Bugs in this 6-Week Cycle

1. Check the <insert link> bugs
2. Test the bugs in the latest Firefox Nightly
3. If you can't reproduce the bug, ask for the reporter to provide more information
4. If you reproduce the bug, add your testing information and mark the bug NEW
5. Notify Juan Becerra of the NEW issue

P3 Task - Confirm UNCONFIRMED Bugs from Other Releases

1. Check the UNCONFIRMED bugs
2. Test the bugs in the latest Firefox Nightly
3. If you can't reproduce the bug, ask for the reporter to provide more information
4. If you reproduce the bug, add your testing information and mark the bug NEW
5. Notify Juan Becerra of the NEW issue

Current Status (Vlad Ghetiu / Virgil Dicu) - Last Updated

Result Spreadsheet

Aurora Merge

The following are the detailed results of the automated BFT test runs.

Platform	Result: report
Windows XP	PASS - 2012-04-26
Windows Vista	PASS - 2012-04-26
Windows 7 32-bit	PASS - 2012-04-26
Windows 7 64-bit	PASS - 2012-04-26
Mac OS X 10.6	PASS - 2012-04-26
Mac OS X 10.7	PASS - 2012-04-26
Ubuntu 11.10 32-bit	PASS - 2012-04-26
Ubuntu 11.10 64-bit	PASS - 2012-04-26

Test Failures

• /testPasswordManager/testPasswordNotificationBar.js (bug 749156)
• /testPasswordManager/testPasswordSavedAndDeleted.js (bug 749206)
• /testSecurity/testBlueLarry.js (bug 748686)
• /testSecurity/testGreenLarry.js (bug 748686)
• /testSecurity/testGreyLarry.js (bug 748686)

Updates

The following are details of update verifications for Firefox 12 merge to Aurora

Auroratest

• [ON TRACK] [ashughes] Windows XP:
• [ON TRACK] [ashughes] Windows 7:
• [ON TRACK] [ashughes] Mac OS X 10.6:
• [ON TRACK] [ashughes] Ubuntu 11.04:

Aurora

• [ON TRACK] [ashughes] Windows XP:
• [ON TRACK] [ashughes] Windows 7:
• [ON TRACK] [ashughes] Mac OS X 10.6:
• [ON TRACK] [ashughes] Ubuntu 11.04:

Issues Found

Manual Testing

Smoketests

Run the Aurora Merge Smoketests in Litmus and report results below.

Platform	Assignee	Result
Windows XP	Ioana	PASS - 2012-04-27
Windows Vista	Simona	PASS - 2012-04-27
Windows 7 32-bit	Paul	PASS - 2012-04-27
Windows 7 64-bit	Simona	PASS - 2012-04-27
Mac OS X 10.6.8	Paul	PASS - 2012-04-27
Mac OS X 10.7.3	Paul	PASS - 2012-04-27
Ubuntu 11.10 32-bit	Virgil	PASS - 2012-04-27
Ubuntu 11.10 64-bit	Mihaela	PASS - 2012-04-27

Results in Litmus

Feature Sign-offs

Run the Aurora 14.0 sign-offs tests in Litmus and report results below.

Feature	Owner	Win32	Mac	Linux
Panel-based Download Manager	Simona Badau	PASS - 2012-04-27	PASS - 2012-04-27	PASS - 2012-04-27
Windows 8 Metro Firefox	Vlad Ghetiu	N/A	N/A	N/A
Cycle Collector Perf improvements	Virgil Dicu	PASS - 2012-04-27	PASS - 2012-04-30	PASS - 2012-04-30
Incremental Garbage Collection	Ioana Budnar	PASS - 2012-04-27	PASS - 2012-04-27	PASS - 2012-04-27
Hang Detector and Reporter	Simona Badau	N/A	N/A	N/A
URL Autocomplete	Vlad Ghetiu	PASS - 2012-04-27	PASS - 2012-04-27	PASS - 2012-04-27
Opt-in activation for plugins	Paul Silaghi	PASS - 2012-04-27	PASS - 2012-04-30	PASS - 2012-04-30
Show PDF Inline	Mihaela Velimiroviciu	N/A	N/A	N/A
Firefox migrators	Paul Silaghi	PASS - 2012-04-30	PASS - 2012-04-30	N/A
Silent Update: Background updates	Vlad Ghetiu	N/A	N/A	N/A
Web Apps integration	Jason Smith	FAIL - 2012-05-01	FAIL - 2012-05-01	N/A - 2012-05-01
Pseudo-Class Lock	Ioana Budnar	PASS - 2012-04-30	PASS - 2012-04-30	PASS - 2012-04-30
Support for Mouse Capture / Mouse Lock	Virgil Dicu	FAIL - 2012-04-30	FAIL - 2012-04-30	FAIL - 2012-04-30

Results in Litmus

Notes:

• Windows 8 Metro Firefox - still in the definition phase.
• Hang Detector and Reporter - only available on the Nightly Profiling branch.
• Silent Update: Background updates - didn't land in Aurora yet.
• Show PDF Inline - feature is not available in latest Aurora builds (since Apr 26)

Issues Found

Full Query

ID	Summary	Priority	Status
749579	Visual glitch when playing chrome angrybirds in SD	--	RESOLVED
750245	Mouse lock pointer not working for quake demo	--	RESOLVED

2 Total; 0 Open (0%); 2 Resolved (100%); 0 Verified (0%);

Known Issues

Full Query

ID	Summary	Priority	Status
564934	Implement new Download Manager UI for browser	--	RESOLVED
726447	[meta] Theming and visual design	--	NEW
726453	[meta] Improve accessibility and keyboard interaction	--	NEW
728692	Javascript becomes slow after Incremental GC gets disabled	--	RESOLVED
735183	No Home Page Selection screen on first run import wizard	--	RESOLVED
735944	about:support says "incremental gc: 1" when javascript.options.mem.gc_incremental is disabled	--	VERIFIED
743102	Desktop Click to play blocks Silverlight - prompted to Install	--	VERIFIED
745187	Click-to-activate plugins which are removed immediately after adding don't trigger the plugin notification (Google Earth Plugin doesn't work with click-to-play enabled)	P1	VERIFIED
746658	Failed downloads can't be removed from panel	--	VERIFIED
746739	The new download list (when downloads button is clicked) looks bad	--	RESOLVED
746743	The new download list doesn't scroll automatically to the newly started download	--	RESOLVED
746751	Download panel icon doesn't appear	--	RESOLVED
746760	Tooltip covers part of the download list	--	NEW
746772	Downloads button disappears from toolbar when customizing	--	VERIFIED
746783	"Open Containing Folder" doesn't appear in context menu of downloads	P2	RESOLVED
747310	Custom toolbar disappears when Edit Bookmarks Panel or Downloads Panel pop up.	--	RESOLVED
747909	Youtube.com In Full Screen mode - copy actions are not allowed on HTML5 videos	--	RESOLVED
748381	Downloads indicator should persist in the toolbar (not auto-hide)	--	VERIFIED

18 Total; 3 Open (16.67%); 9 Resolved (50%); 6 Verified (33.33%);

Pre-beta features sign offs

Run the Firefox 14.0 pre-beta sign off tests in Litmus and report results below.

Feature	Owner	Win32	Mac	Linux
URL Autocomplete	Vlad Ghetiu	PASS - 2012-05-31	PASS - 2012-05-31	PASS - 2012-05-31
Opt-in activation for plugins	Paul Silaghi	PASS - 2012-05-31	PASS - 2012-05-31	PASS - 2012-05-31
Firefox migrators - Safari Migrator	Paul Silaghi	PASS - 2012-05-31	PASS - 2012-05-31	N/A - 2012-05-31
Pseudo-Class Lock	Ioana Budnar	FAIL - 2012-05-31	PASS - 2012-05-31	PASS - 2012-05-31
Support for Mouse Capture / Mouse Lock	Virgil Dicu	PASS - 2012-05-31	PASS - 2012-05-31	PASS - 2012-05-31

Results in Litmus

Issues Found

Full Query

ID	Summary	Priority	Status
760171	The inspector node menu feature allows the user to remove tabbed browsing	P1	RESOLVED
760175	The :focus pseudoClass is not kept for the parent node when deleting a node	--	RESOLVED

2 Total; 0 Open (0%); 2 Resolved (100%); 0 Verified (0%);

Known Issues

Bugzilla query error

Array ( [type] => error [message] => http-bad-status [params] => Array ( [0] => 400 [1] => Bad Request ) ) 1

Beta 6

Builds	FTP, HTTP
ID	TBD
Changeset	89ec8943347a

Checklist

• [ON TRACK] Functional automated tests, synced to Litmus
• [ON TRACK] Manual tests
• [ON TRACK] Bug fix verifications (target 50%)
• [ON TRACK] Security fix verifications (target 25%)
• [ON TRACK] Update tests on releasetest
• [ON TRACK] Get final sign-off from Bob Moss
• [ON TRACK] Update tests on beta

Automated Tests

The following are the detailed results of the automated BFT test runs.

Platform	Result: report
Windows XP	RESULT - 2012-06-06
Windows Vista	RESULT - 2012-06-06
Windows 7 32-bit	RESULT - 2012-06-06
Windows 7 64-bit	RESULT - 2012-06-06
Mac OS X 10.6	RESULT - 2012-06-06
Mac OS X 10.7	RESULT - 2012-06-06
Ubuntu 11.10 32-bit	RESULT - 2012-06-06
Ubuntu 11.10 64-bit	RESULT - 2012-06-06

Test Failures (known bugs)

Manual Tests

The following are the detailed results of the manual testing.

Smoketests

Run the Firefox 14.0 Smoketests in Litmus and report results below.

• Working on cloning the branch as of 06-04 @11:50pm Pacific

Platform	Assignee	Result
Windows XP	Simona	PASS - 2012-06-06
Windows Vista	tester	RESULT - 2012-06-06
Windows 7 32-bit	tester	RESULT - 2012-06-06
Windows 7 64-bit	tester	RESULT - 2012-06-06
Mac OS X 10.6.8	tester	RESULT - 2012-06-06
Mac OS X 10.7.4	tester	RESULT - 2012-06-06
Ubuntu 12.04 32-bit	tester	RESULT - 2012-06-06
Ubuntu 11.10 64-bit	tester	RESULT - 2012-06-06
Windows 7 32-bit (es-AR)	Gabriela	PASS - 2012-06-06

Results in Litmus

Regression Testing

Run the Firefox 14.0 Regression Tests and Firefox 14.0 Web App Compatibility in Litmus and report the results below.

Platform	Assignee	Result: report
Windows 7 32-bit	tester	RESULT - 2012-06-06
Mac OS X 10.6.8	tester	RESULT - 2012-06-06
Ubuntu 11.10 64-bit	tester	RESULT - 2012-06-06
Windows XP	tester	RESULT - 2012-06-06

Exploratory Testing

1. [ON TRACK] Exploratory testing on plugins, addons, bookmarks etc Etherpad
2. [ON TRACK] Exploratory testing on the new features Etherpad

Feature Testing

Run the Firefox 14.0 Features Sign-off tests in Litmus and report the results below. Additionally do some exploratory testing to make sure no regressions have been introduced to related legacy features.

Feature	Owner	Win32	Mac	Linux
URL Autocomplete	Vlad Ghetiu	PASS - 2012-06-06	PASS - 2012-06-06	PASS - 2012-06-06
Opt-in activation for plugins	Paul Silaghi	PASS - 2012-06-06	PASS - 2012-06-06	PASS - 2012-06-06
Firefox migrators - Safari Migrator	Paul Silaghi	PASS - 2012-06-06	PASS - 2012-06-06	N/A
Pseudo-Class Lock	Ioana Budnar	PASS - 2012-06-06	PASS - 2012-06-06	PASS - 2012-06-06
Support for Mouse Capture / Mouse Lock	Virgil Dicu	PASS - 2012-06-06	PASS - 2012-06-06	PASS - 2012-06-06

Update Tests

The following are details of update verifications

Platform	Releasetest: report	Beta: report
Windows XP	RESULT - 2012-06-06	RESULT - 2012-06-06
Windows Vista	RESULT - 2012-06-06	RESULT - 2012-06-06
Windows 7 32-bit	RESULT - 2012-06-06	RESULT - 2012-06-06
Windows 7 64-bit	RESULT - 2012-06-06	RESULT - 2012-06-06
Mac OS X 10.6	RESULT - 2012-06-06	RESULT - 2012-06-06
Mac OS 10.7	RESULT - 2012-06-06	RESULT - 2012-06-06
Ubuntu 11.10 32-bit	RESULT - 2012-06-06	RESULT - 2012-06-06
Ubuntu 11.10 64-bit	RESULT - 2012-06-06	RESULT - 2012-06-06

Issues

Bug Fix Verifications

The following are the detailed results of verifications of bug fixes landed in Beta.

• Any fix you very should have the status-firefox13 flag set to verified
• If the fix has been verified on all affected versions of Firefox, change [qa+] in the whiteboard to [qa!]

Query	Triaged	Remaining	% Complete
QA+: for testers to verify functional fixes	9	41	22%
QA+ SG: for the Security team to verify security fixes	7	23	23%
status-firefox14:fixed: for QA lead to triage for potential QA+ bugs	297	0	100%

Softvision verifications etherpad

QAWANTED

QAWANTED Bugs

• Need monitoring of crash stats for bug 760190
• Need steps to reproduce bug 709860
• bug 755728 only happens on mobile
• bug 748047 needs verification (rest profile)
• bug 748569 needs verification (migration code)

Issues Found

Full Query

ID	Summary	Priority	Status
761990	On Windows the "Feedback" button's dropdown arrow overlaps the label	--	RESOLVED
762011	[highlighter] Node operation bar displayed but not accessible for on mouse hover elements	--	RESOLVED

2 Total; 0 Open (0%); 2 Resolved (100%); 0 Verified (0%);

Known Issues

Full Query

ID	Summary	Priority	Status
711742	HTML5 video's "Save Video As" saves an empty .webm file if you have not actually played the video	--	RESOLVED
747858	A horizontal line is displayed on the Play button from HTML5 videos	--	RESOLVED
754829	In Full Screen mode - YouTube videos player changes dimmension	--	RESOLVED
760171	The inspector node menu feature allows the user to remove tabbed browsing	P1	RESOLVED
760175	The :focus pseudoClass is not kept for the parent node when deleting a node	--	RESOLVED

5 Total; 0 Open (0%); 5 Resolved (100%); 0 Verified (0%);

Feedback

If you want to help test and just send us general feedback, please do so below. Areas we are looking for feedback include but are not limited to hangs, crashes, add-on compatibility, youtube, facebook, twitter, etc.

Builds can be found here: FTP, HTTP

You can post your feedback below or in this etherpad.

Be sure to install the Add-on Compatibility Reporter if you want to report incompatibilities.