Releases/Firefox 14/Test Plan: Difference between revisions
Line 509: | Line 509: | ||
| Gabriela | | Gabriela | ||
| '''PASS''' - 2012-06-06 | | '''PASS''' - 2012-06-06 | ||
|15: Pop-up Blocking: The test page leads to a 404 Not Found page. | |15: Pop-up Blocking: The test page leads to a 404 Not Found page. | ||
| 21: Add and Refresh a RSS feed (Live Bookmark): In step 3 I'm taken to a page with an | | 21: Add and Refresh a RSS feed (Live Bookmark): In step 3 I'm taken to a page with an | ||
|invalid URL. | |invalid URL. | ||
| 26: WebM video Playback: Both videos are unavailable. | | 26: WebM video Playback: Both videos are unavailable. | ||
|} | |} |
Revision as of 16:11, 6 June 2012
« Firefox 13 | Firefox 14 | Firefox 15 »
Firefox 14 Test Plan
Summary
Milestone | Firefox 14 |
Status | Currently in Aurora -- merging to Beta on June 4, 2012 |
Lead | Juan Becerra (irc: juanb) |
2nd | Anthony Hughes (irc: ashughes) |
Builds | [ftp] [http] [web] |
The following is the test plan for Firefox 14 from Nightly through to Release. Use this document as a reference for what is being tested to validate the quality of Firefox 14. After the release this document will be used as an archive of what was done to validate this release.
If you have some free time, please pick a task below and get in touch with one of the leads.
Details
The following is a description of the tasks which need to be completed throughout this release.
- 2012-03-13: [DONE] Nightly Merge
- 2012-04-24: [DONE] Aurora Merge
- 2012-06-05: {{|}} Pre-Beta feature sign-offs
- 2012-06-07: {{|}} Beta 1
- 2012-06-14: {{|}} Beta 2
- 2012-06-21: {{|}} Beta 3
- 2012-06-28: {{|}} Beta 4
- 2012-07-05: {{|}} Beta 5
- 2012-07-11: {{|}} Beta 6
- 2012-07-13: {{|}} Release Candidate
- 2012-07-17: {{|}} Final
Features
See the Feature Release Tracking wiki for a general overview
- [DONE]
- Cycle Collector Perf improvements (Virgil Dicu); landed in Firefox 13
- [ON TRACK]
- Windows8 Metro Firefox (Vlad Ghetiu)
- Hang Detector and Reporter (Simona Badau)
- URL Autocomplete (Vlad Ghetiu)
- Firefox Migrators (Paul Silaghi)
- Pseudo-Class Lock (Ioana Budnar)
- Support for Mouse Capture / Mouse Lock (Virgil Dicu)
- Opt-in activation for plugins (Paul Silaghi); Initial minimal implementation landed for 14 (opt-in), further work in 15 and 16.
- [AT RISK]
- [MISSED]
- Incremental garbage collection (Ioana Budnar); delayed to Firefox 15
- Panel-based Download Manager (Simona Badau) opt-in; delayed to Firefox 15
- Silent Update: Background updates (Vlad Ghetiu); delayed to Firefox 15
- Windows Start-up Performance Improvements (Jason Smith); delayed to Firefox 15
- Speedy Session Restore (Paul Silaghi); delayed to Firefox 15
- Silent Update: Updated workflow for users with incompatible add-ons (Ioana Budnar); delayed to Firefox 17
- Show PDF inline (Mihaela Velimiroviciu); delayed to Firefox 16
- Web Apps Integration to Desktop (Jason Smith); delayed to Firefox 15
Nightly Merge
- Functional Automation - result
- Update Spotchecks - result
Endurance Automation
The purpose of this section is to track any potential memory regressions via spikes reported in the dashboard.
- What to Do
- Check the Endurance Charts
- If you see an anomalous event (ie. spike or dip), get the report for the specific event
- If the event happens twice in a row, file a bug under Mozmill Tests
- Notify Juan Becerra of the issue
- Record the event in this wiki below
- Current Status (Mihaela Velimiroviciu) - Last Updated
- bug 752887 - Increased memory usage for endurance tests on Linux Aurora branch
Functional Automation
The purpose of this section is to track any potential functional regressions via results reported in the dashboard.
- What to Do
- Check the testrun reports
- If you see any failing tests, check that there are not any bugs on file for it already
- If not, file a bug under Mozmill Tests
- Notify Juan Becerra of the issue
- Record the event in this wiki below
- Current Status (Vlad Maniac / Alex Lakatos) - Last Updated
- No issues found
Unconfirmed Bug Triage
The purpose of this section is to triage UNCONFIRMED incoming bugs. The below charts provides a summary of the unconfirmed bugs currently in the Firefox product:
Bugzilla query error
GD extension must be loaded.1
- P1 Task - Get UNCONFIRMED Bugs Out of Firefox General and Untriaged If Possible
- Check this <insert link> for UNCONFIRMED bugs found for this Firefox Release's Beta Timeframe
- Read each bug and determine what product and component they belong under in bugzilla
- Move the bug to the right product and component
- CC any applicable team members that may be interested in the bug
- P2 Task - Confirm UNCONFIRMED Bugs in this 6-Week Cycle
- Check the <insert link> bugs
- Test the bugs in the latest Firefox Nightly
- If you can't reproduce the bug, ask for the reporter to provide more information
- If you reproduce the bug, add your testing information and mark the bug NEW
- Notify Juan Becerra of the NEW issue
- P3 Task - Confirm UNCONFIRMED Bugs from Other Releases
- Check the UNCONFIRMED bugs
- Test the bugs in the latest Firefox Nightly
- If you can't reproduce the bug, ask for the reporter to provide more information
- If you reproduce the bug, add your testing information and mark the bug NEW
- Notify Juan Becerra of the NEW issue
- Current Status (Vlad Ghetiu / Virgil Dicu) - Last Updated
Aurora Merge
The following are the detailed results of the automated BFT test runs.
Platform | Result: report |
Windows XP | PASS - 2012-04-26 |
Windows Vista | PASS - 2012-04-26 |
Windows 7 32-bit | PASS - 2012-04-26 |
Windows 7 64-bit | PASS - 2012-04-26 |
Mac OS X 10.6 | PASS - 2012-04-26 |
Mac OS X 10.7 | PASS - 2012-04-26 |
Ubuntu 11.10 32-bit | PASS - 2012-04-26 |
Ubuntu 11.10 64-bit | PASS - 2012-04-26 |
- Test Failures
- /testPasswordManager/testPasswordNotificationBar.js (bug 749156)
- /testPasswordManager/testPasswordSavedAndDeleted.js (bug 749206)
- /testSecurity/testBlueLarry.js (bug 748686)
- /testSecurity/testGreenLarry.js (bug 748686)
- /testSecurity/testGreyLarry.js (bug 748686)
Updates
The following are details of update verifications for Firefox 12 merge to Aurora
- Auroratest
- [ON TRACK] [ashughes] Windows XP:
- [ON TRACK] [ashughes] Windows 7:
- [ON TRACK] [ashughes] Mac OS X 10.6:
- [ON TRACK] [ashughes] Ubuntu 11.04:
- Aurora
- [ON TRACK] [ashughes] Windows XP:
- [ON TRACK] [ashughes] Windows 7:
- [ON TRACK] [ashughes] Mac OS X 10.6:
- [ON TRACK] [ashughes] Ubuntu 11.04:
- Issues Found
Manual Testing
Smoketests
Run the Aurora Merge Smoketests in Litmus and report results below.
Platform | Assignee | Result |
Windows XP | Ioana | PASS - 2012-04-27 |
Windows Vista | Simona | PASS - 2012-04-27 |
Windows 7 32-bit | Paul | PASS - 2012-04-27 |
Windows 7 64-bit | Simona | PASS - 2012-04-27 |
Mac OS X 10.6.8 | Paul | PASS - 2012-04-27 |
Mac OS X 10.7.3 | Paul | PASS - 2012-04-27 |
Ubuntu 11.10 32-bit | Virgil | PASS - 2012-04-27 |
Ubuntu 11.10 64-bit | Mihaela | PASS - 2012-04-27 |
Results in Litmus
Feature Sign-offs
Run the Aurora 14.0 sign-offs tests in Litmus and report results below.
Feature | Owner | Win32 | Mac | Linux |
Panel-based Download Manager | Simona Badau | PASS - 2012-04-27 | PASS - 2012-04-27 | PASS - 2012-04-27 |
Windows 8 Metro Firefox | Vlad Ghetiu | N/A | N/A | N/A |
Cycle Collector Perf improvements | Virgil Dicu | PASS - 2012-04-27 | PASS - 2012-04-30 | PASS - 2012-04-30 |
Incremental Garbage Collection | Ioana Budnar | PASS - 2012-04-27 | PASS - 2012-04-27 | PASS - 2012-04-27 |
Hang Detector and Reporter | Simona Badau | N/A | N/A | N/A |
URL Autocomplete | Vlad Ghetiu | PASS - 2012-04-27 | PASS - 2012-04-27 | PASS - 2012-04-27 |
Opt-in activation for plugins | Paul Silaghi | PASS - 2012-04-27 | PASS - 2012-04-30 | PASS - 2012-04-30 |
Show PDF Inline | Mihaela Velimiroviciu | N/A | N/A | N/A |
Firefox migrators | Paul Silaghi | PASS - 2012-04-30 | PASS - 2012-04-30 | N/A |
Silent Update: Background updates | Vlad Ghetiu | N/A | N/A | N/A |
Web Apps integration | Jason Smith | FAIL - 2012-05-01 | FAIL - 2012-05-01 | N/A - 2012-05-01 |
Pseudo-Class Lock | Ioana Budnar | PASS - 2012-04-30 | PASS - 2012-04-30 | PASS - 2012-04-30 |
Support for Mouse Capture / Mouse Lock | Virgil Dicu | FAIL - 2012-04-30 | FAIL - 2012-04-30 | FAIL - 2012-04-30 |
Results in Litmus
Notes:
- Windows 8 Metro Firefox - still in the definition phase.
- Hang Detector and Reporter - only available on the Nightly Profiling branch.
- Silent Update: Background updates - didn't land in Aurora yet.
- Show PDF Inline - feature is not available in latest Aurora builds (since Apr 26)
Issues Found
ID | Summary | Priority | Status |
---|---|---|---|
749579 | Visual glitch when playing chrome angrybirds in SD | -- | RESOLVED |
750245 | Mouse lock pointer not working for quake demo | -- | RESOLVED |
2 Total; 0 Open (0%); 2 Resolved (100%); 0 Verified (0%);
Known Issues
18 Total; 3 Open (16.67%); 9 Resolved (50%); 6 Verified (33.33%);
Pre-beta features sign offs
Run the Firefox 14.0 pre-beta sign off tests in Litmus and report results below.
Feature | Owner | Win32 | Mac | Linux |
URL Autocomplete | Vlad Ghetiu | PASS - 2012-05-31 | PASS - 2012-05-31 | PASS - 2012-05-31 |
Opt-in activation for plugins | Paul Silaghi | PASS - 2012-05-31 | PASS - 2012-05-31 | PASS - 2012-05-31 |
Firefox migrators - Safari Migrator | Paul Silaghi | PASS - 2012-05-31 | PASS - 2012-05-31 | N/A - 2012-05-31 |
Pseudo-Class Lock | Ioana Budnar | FAIL - 2012-05-31 | PASS - 2012-05-31 | PASS - 2012-05-31 |
Support for Mouse Capture / Mouse Lock | Virgil Dicu | PASS - 2012-05-31 | PASS - 2012-05-31 | PASS - 2012-05-31 |
Results in Litmus
Issues Found
ID | Summary | Priority | Status |
---|---|---|---|
760171 | The inspector node menu feature allows the user to remove tabbed browsing | P1 | RESOLVED |
760175 | The :focus pseudoClass is not kept for the parent node when deleting a node | -- | RESOLVED |
2 Total; 0 Open (0%); 2 Resolved (100%); 0 Verified (0%);
Known Issues
Bugzilla query error
Array ( [type] => error [message] => http-bad-status [params] => Array ( [0] => 400 [1] => Bad Request ) ) 1
Beta 6
Builds | FTP, HTTP |
ID | TBD |
Changeset | 89ec8943347a |
- Checklist
- [ON TRACK] Functional automated tests, synced to Litmus
- [ON TRACK] Manual tests
- [ON TRACK] Bug fix verifications (target 50%)
- [ON TRACK] Security fix verifications (target 25%)
- [ON TRACK] Update tests on releasetest
- [ON TRACK] Get final sign-off from Bob Moss
- [ON TRACK] Update tests on beta
Automated Tests
The following are the detailed results of the automated BFT test runs.
Platform | Result: report |
Windows XP | RESULT - 2012-06-06 |
Windows Vista | RESULT - 2012-06-06 |
Windows 7 32-bit | RESULT - 2012-06-06 |
Windows 7 64-bit | RESULT - 2012-06-06 |
Mac OS X 10.6 | RESULT - 2012-06-06 |
Mac OS X 10.7 | RESULT - 2012-06-06 |
Ubuntu 11.10 32-bit | RESULT - 2012-06-06 |
Ubuntu 11.10 64-bit | RESULT - 2012-06-06 |
- Test Failures (known bugs)
Manual Tests
The following are the detailed results of the manual testing.
Smoketests
Run the Firefox 14.0 Smoketests in Litmus and report results below.
- Working on cloning the branch as of 06-04 @11:50pm Pacific
Platform | Assignee | Result | ||||
Windows XP | Simona | PASS - 2012-06-06 | ||||
Windows Vista | tester | RESULT - 2012-06-06 | ||||
Windows 7 32-bit | tester | RESULT - 2012-06-06 | ||||
Windows 7 64-bit | tester | RESULT - 2012-06-06 | ||||
Mac OS X 10.6.8 | tester | RESULT - 2012-06-06 | ||||
Mac OS X 10.7.4 | tester | RESULT - 2012-06-06 | ||||
Ubuntu 12.04 32-bit | tester | RESULT - 2012-06-06 | ||||
Ubuntu 11.10 64-bit | tester | RESULT - 2012-06-06 | ||||
Windows 7 32-bit (es-AR) | Gabriela | PASS - 2012-06-06 | 15: Pop-up Blocking: The test page leads to a 404 Not Found page. | 21: Add and Refresh a RSS feed (Live Bookmark): In step 3 I'm taken to a page with an | invalid URL. | 26: WebM video Playback: Both videos are unavailable. |
Results in Litmus
Regression Testing
Run the Firefox 14.0 Regression Tests and Firefox 14.0 Web App Compatibility in Litmus and report the results below.
Platform | Assignee | Result: report |
Windows 7 32-bit | tester | RESULT - 2012-06-06 |
Mac OS X 10.6.8 | tester | RESULT - 2012-06-06 |
Ubuntu 11.10 64-bit | tester | RESULT - 2012-06-06 |
Windows XP | tester | RESULT - 2012-06-06 |
Exploratory Testing
- [ON TRACK] Exploratory testing on plugins, addons, bookmarks etc Etherpad
- [ON TRACK] Exploratory testing on the new features Etherpad
Feature Testing
Run the Firefox 14.0 Features Sign-off tests in Litmus and report the results below. Additionally do some exploratory testing to make sure no regressions have been introduced to related legacy features.
Feature | Owner | Win32 | Mac | Linux |
URL Autocomplete | Vlad Ghetiu | PASS - 2012-06-06 | PASS - 2012-06-06 | PASS - 2012-06-06 |
Opt-in activation for plugins | Paul Silaghi | PASS - 2012-06-06 | PASS - 2012-06-06 | PASS - 2012-06-06 |
Firefox migrators - Safari Migrator | Paul Silaghi | PASS - 2012-06-06 | PASS - 2012-06-06 | N/A |
Pseudo-Class Lock | Ioana Budnar | PASS - 2012-06-06 | PASS - 2012-06-06 | PASS - 2012-06-06 |
Support for Mouse Capture / Mouse Lock | Virgil Dicu | PASS - 2012-06-06 | PASS - 2012-06-06 | PASS - 2012-06-06 |
Update Tests
The following are details of update verifications
Platform | Releasetest: report | Beta: report |
Windows XP | RESULT - 2012-06-06 | RESULT - 2012-06-06 |
Windows Vista | RESULT - 2012-06-06 | RESULT - 2012-06-06 |
Windows 7 32-bit | RESULT - 2012-06-06 | RESULT - 2012-06-06 |
Windows 7 64-bit | RESULT - 2012-06-06 | RESULT - 2012-06-06 |
Mac OS X 10.6 | RESULT - 2012-06-06 | RESULT - 2012-06-06 |
Mac OS 10.7 | RESULT - 2012-06-06 | RESULT - 2012-06-06 |
Ubuntu 11.10 32-bit | RESULT - 2012-06-06 | RESULT - 2012-06-06 |
Ubuntu 11.10 64-bit | RESULT - 2012-06-06 | RESULT - 2012-06-06 |
- Issues
Bug Fix Verifications
The following are the detailed results of verifications of bug fixes landed in Beta.
- Any fix you very should have the status-firefox13 flag set to verified
- If the fix has been verified on all affected versions of Firefox, change [qa+] in the whiteboard to [qa!]
Query | Triaged | Remaining | % Complete |
QA+: for testers to verify functional fixes | 9 | 41 | 22% |
QA+ SG: for the Security team to verify security fixes | 7 | 23 | 23% |
status-firefox14:fixed: for QA lead to triage for potential QA+ bugs | 297 | 0 | 100% |
Softvision verifications etherpad
QAWANTED
- Need monitoring of crash stats for bug 760190
- Need steps to reproduce bug 709860
- bug 755728 only happens on mobile
- bug 748047 needs verification (rest profile)
- bug 748569 needs verification (migration code)
Issues Found
ID | Summary | Priority | Status |
---|---|---|---|
761990 | On Windows the "Feedback" button's dropdown arrow overlaps the label | -- | RESOLVED |
762011 | [highlighter] Node operation bar displayed but not accessible for on mouse hover elements | -- | RESOLVED |
2 Total; 0 Open (0%); 2 Resolved (100%); 0 Verified (0%);
Known Issues
5 Total; 0 Open (0%); 5 Resolved (100%); 0 Verified (0%);
Feedback
If you want to help test and just send us general feedback, please do so below. Areas we are looking for feedback include but are not limited to hangs, crashes, add-on compatibility, youtube, facebook, twitter, etc.
Builds can be found here: FTP, HTTP
You can post your feedback below or in this etherpad.
Be sure to install the Add-on Compatibility Reporter if you want to report incompatibilities.