QA/Firefox3/TestPlan: Difference between revisions

From MozillaWiki
< QA‎ | Firefox3
Jump to navigation Jump to search
mNo edit summary
 
(37 intermediate revisions by 6 users not shown)
Line 9: Line 9:
* Major bugs blocking overall regression of the trunk that should be fixed before shipping?
* Major bugs blocking overall regression of the trunk that should be fixed before shipping?


[[QA/Firefox3/TestResults/M9/Release Notes|QA Beta 1 Relnotes Tracking]]<BR>
''Update'' We've shipped Fx3.0! Next steps is to start tracking 3.1 and 3.0.1 bugs.
[[QA/Firefox3/TestResults/M10/Release Notes|QA Beta 2 Relnotes Tracking]]
 
;Must fix by Beta 3
* Top crashes
** {{Bug|411249}} – Top crash [@ jsds_ScriptHookProc[@jsds_ScriptHookProc(JSDContext*, JSDScript*, int, void*)] on shutdown with Firebug 1.1.0b10 installed
** {{Bug|411369}} - Crash while trying to print [@ @0x0 @0x1000 _cairo_meta_surface_create] [@ _moz_cairo_surface_finish ] - This is a major regression that is blocking any printing on Vista and XP.
* Breakpad issues
** {{Bug|412375}} - Older crashreports no longer visible ?.  Unable to retrieve reports prior to Jan 10th.  Webdev investigating. '''Update - data has been lost due to running out of space on db.  unrecoverable'''
** {{Bug|390568}} - vista fails to submit breakpad reports.  This has been broken since alpha8, and we are losing more crash data as long as this remains unfixed.
** {{Bug|378528}} – crash reporter should attempt to resubmit pending reports.  Needs to be rolled up into a queue for later if breakpad is down.  This causes backlog of missing crash data that takes days later to resurface. Takes away time from researching critical issues.
* {{Bug|405198}} Cannot drag and drop folders / items in Places Organizer - this cause the problem that its not possible to move bookmarks from one folder to another folder oder to merge folder etc .... - Tomcat  '''Why is this assigned to nobody?'''
* {{Bug|324857}} – MathML all screwed up in Cairo builds.  Blocking MathML usage.  Lots of action in the bug, but no resolution yet?
 
;Must Fix by Release Candidate
* Breakpad issues
** {{Bug|394247}} – check that nsITransfer::Init succeeded (crash [@nsDownloadProxy::OnStateChange])
** {{Bug|412008}} – Can't access report/list for 1.9 branch topcrashes
** {{Bug|411438}} – Crash reporter app should offer restart Firefox button when it throws an error dialog
* {{bug|402335}} - Meta bug for memory leaks.  Tracked by tomcat.  Ongoing testing and bugfixes.
* [http://tinyurl.com/2mvcdj Critical Printing Bugs] - These are printing crash bugs from printing.  Bugs listed in this query are marked in Trunk, and have been confirmed by QA members.
* {{Bug|391598}} and {{Bug|407780}} SW update anti-fatigue.  Need to improve modal dialogs for updates.  Need to add to Fx3 testplan.
* {{bug|407459}} - Hook up new allocator to our build.  QA will probably not test this.  noting it down in case we need to look at it.
* {{Bug|404024}} - Add AMO integration pane.  Testplan completed, but no date set yet.  Tracking spec is at: http://wiki.mozilla.org/Update:Remora_Meeting_Notepad.  For Addons Manager UI, last update was 4 checkins were pending UX review for b3.
* {{Bug|401301}} - firefox.exe process still around after all windows closed]  Still investigating firefox hang process.  Hard to repro, reported by accessibility team and important for the community to resolve.
* {{Bug|404645}} – memory usage spikes dramatically while doing nothing with fresh profile.  High memory utilization reported by many blogs.
* {{Bug|403232}} - Focused textfield loses focus when switching to another tab and back.  '''UPDATE - fix should land by b3'''
* Accessibility issues
** {{bug|393398}} Add Bookmark Dialog Inaccessible: Needs to be classified as a dialog or a window in order to work properly with most AT.  Needs more work and can be post bedta 2. One screen reader has severe problems with the panel being classified as a menu, the others we test don't. This is also true for the new AutoComplete panel, which is currently being classified as a menu with alist within. (MarcoZ)
** Bookmark/Places Panel not Accessible, need menu items to duplicate toolbar entries. 
*** There are equivalence in the Bookmarks menu to the toolbar buttons--for some reason they weren't accessible from my profile.  However, there are sec508 concerns with the Places Organizer contained in {{bug|400703}} and {{bug|402104}} which will require more extensive testing and should be resolved by beta 2, if at all possible.  -TimK
*** Since neither of these bugs have patches yet, this should slip past beta 2, but is a sec508 issue that needs to be resolved before shipping. (MarcoZ)
;RESOLVED FIXED bugs, needing more information or testcases
*[https://bugzilla.mozilla.org/show_bug.cgi?id=375750 Bug 375750] - PDFs generated by the trunk are huge. The patch seems to make the size smaller, but the size is still not in line with what the branch size is
* [https://bugzilla.mozilla.org/show_bug.cgi?id=402000 Bug 402000] - Cannot be verified according to dev


== Overview ==
== Overview ==
Line 114: Line 80:
|[[Firefox3/Product_Requirements_Document#Gecko_integration|Gecko Platform Integration]]
|[[Firefox3/Product_Requirements_Document#Gecko_integration|Gecko Platform Integration]]
|[[QA/Firefox3/TestPlan/Platform|Gecko Platform Test Plan]]
|[[QA/Firefox3/TestPlan/Platform|Gecko Platform Test Plan]]
[[QA/Firefox3/TestPlan/Offline_Apps|Offline Apps Test Plan]]
|martijn, marcia
|martijn, marcia
|M
|M
Line 152: Line 119:
|L
|L
|password management, Phishing protection, Private web browsing, Blacklisting (At Risk), password Manager, Password Security
|password management, Phishing protection, Private web browsing, Blacklisting (At Risk), password Manager, Password Security
|-
|Breakpad Client/Server
|[[QA/Firefox3/TestPlan/Breakpad|Breakpad Crash Reporting Test Plan]]
|ss
|S
|Crash reporting tests from client/server, analysis, load


|-
|-
Line 167: Line 141:
|S
|S
|test automation, new locales, string lengths, bookmarks, rss, search plugins
|test automation, new locales, string lengths, bookmarks, rss, search plugins
|-
|L10n Mozilla.com-website Testing
||[[QA/Firefox3/TestPlan/l10n-mozilla.com|L10n Mozilla.com Test Plan]]
|stephend, pascalc, tchung, timr
|M
|see test plan


|-
|-
Line 184: Line 165:
|-
|-
|Memory Leaks
|Memory Leaks
|[http://wiki.mozilla.org/MozillaQualityAssurance:Home_Page:Firefox_3.0_TestPlan:Leaks Memory Leaks Test Plan]
|[http://wiki.mozilla.org/QA:Home_Page:Firefox_3.0_TestPlan:Leaks Memory Leaks Test Plan]
|tomcat
|tomcat
|M
|M
Line 238: Line 219:


== Top Crash reports ==  
== Top Crash reports ==  
The [http://crash-stats.mozilla.com/?do_query=1&product=Firefox&branch=1.9&version=Firefox%3A3.0a9pre&query_search=signature&query_type=contains&query=&date=&range_value=1&range_unit=weeks main report] is the easiest to view.
The [http://crash-stats.mozilla.com/?do_query=1&product=Firefox&branch=1.9&version=Firefox%3A3.0b3pre&query_search=signature&query_type=contains&query=&date=&range_value=1&range_unit=weeks main report] is the easiest to view.


Note that of those in that list, only a few seem to be active lately. I filed [https://bugzilla.mozilla.org/show_bug.cgi?id=402661 bug 402661] for the crash in RtlpCoalesceFreeBlocks / XPCWrappedNative::CallMethod(XPCCallContext&, XPCWrappedNative::CallMode). I still need to file a bug for the crash in  CallGetService(nsID const&, nsID const&, void**). Neither of these should be blocking beta given their (relative) infrequency.
'''Tracking Top crashes from Beta 3'''
* ''High Priority for beta 4''
** '''Resolved Fixed''' {{bug|418377}} - crash [@ XPCWrappedNativeScope::FindInJSObjectScope(XPCCallContext&, JSObject*, int)]
** '''Nom for blocking? please approve''' {{bug|418378}} - crash [@ nsGlobalWindow::SaveWindowState(nsISupports**)]
** '''Nom for blocking? please approve''' {{bug|418381}} - crash [@ HashString(nsAString_internal const&)]
* ''Lower Priority''
** {{bug|391311}} - Topcrash: [@ nsChromeRegistry::CheckForNewChrome] during startup
** {{bug|418379}} - crash [@ nsNavHistoryFolderResultNode::FindChildById(__int64, unsigned int*)]
** {{bug|418382}} - crash [@ UniscribeItem::SaveGlyphs(gfxTextRun*)]
** {{bug|418384}} - crash [@ ClientData::GetOtlTable(long, unsigned char const**, unsigned long*)]


== Banking and Financial Sites ==
== Banking and Financial Sites ==
For Firefox 3, we’re asking community members and MoCo QA to [[QA/Firefox3/TestPlan/Banking|state which banks and financial institutions they're willing to test]]. Please add your name to the list.
For Firefox 3, we’re asking community members and MoCo QA to [[QA/Firefox3/TestPlan/Banking|state which banks and financial institutions they're willing to test]]. Please add your name to the list.
== L10n Testing with Third Parties ==
For additional string testing, Mozilla contracts a third party company to utilize language experts with knowledge of Testing.  Results are listed in the links here.
* [http://people.mozilla.com/~timr/FF3.0/L10n/ Beta 2 Results]


== Review Feedback ==
== Review Feedback ==
Line 261: Line 256:
**abillings
**abillings
**stephend
**stephend
== Post Mortem Notes ==
The following is a [[QA/Firefox3/TestPlan/PostMortem|collection of post mortem]] notes that the team has collected over the course of the release. 


== Additional References ==  
== Additional References ==  
[http://developer.mozilla.org/en/docs/Firefox_3_for_developers New FF3 Feature list] - An incomplete list of what Firefox 3 developers are working on.
[http://developer.mozilla.org/en/docs/Firefox_3_for_developers New FF3 Feature list] - An incomplete list of what Firefox 3 developers are working on.

Latest revision as of 21:03, 19 June 2008

« QA/Firefox3

Top Critical issues for QA

This is a working list that QA is using to request be fixed or completed for the Upcoming Fx3 Milestones.

If you have untracked issues in these areas, please enter them here:

  • Design still incomplete?
  • Major bugs blocking your feature that should be addressed?
  • Major bugs blocking overall regression of the trunk that should be fixed before shipping?

Update We've shipped Fx3.0! Next steps is to start tracking 3.1 and 3.0.1 bugs.

Overview

Firefox 3 QA Test Strategy

The goal of this Overall Test Plan is to lay out our testing strategy, communicate it to others, and track the status.

Major QA Activities:

Firefox 3 Central

Requirements and Ownership

Archives:

Test Areas

Feature Focused Areas

Key: L = Large (3-6 weeks) M = Medium (2-4 weeks) S = Small (1-2 weeks)

Test Plan Template

Feature Area and Spec Test Plan Location QA Contact Size Description
Addons Addons Test Plan tchung (new Addons Manager UI, Extensions), stephend (Unified Plugin Manager) M Add-on management system, plugin management system, localized user-facing addon text, configuration, updates, Installation, Discovery, FUEL 0.2
Content Handling Microformat Test Plan

Web Application Test Plan

Download Manager Test Plan 		ctalbert, stephend (Download Manager) L microformats, Content Type Processing, download manager, MIME type support, User Interface, Media Plugins (needs definition), firebug (at risk)
Partner Distribution TBD tracy, ctalbert M High-Touch/Low-Touch distros, regression, updates, test automation
OS Platform Integration Vista Test Plan Tomcat, marcia M Vista Support (At Risk), MSI bundling, Airbag / Breakpad, parental controls (At Risk), Notification, Vista Installer
Gecko Platform Integration Gecko Platform Test Plan

Offline Apps Test Plan

martijn, marcia M Cairo graphics, nxTextFrame migration to Thebes, Reflow Branch, Caret painting rewrite, Cross-domain XMLHttpRequest (Cocoa widgets covered under Mac Test Plan)
Mac OS Platform Integration (includes Gecko) Mac OS Integration Test Plan

Leopard Test Plan

Tomcat, marcia M Leopard support, Cocoa implementation, widgets, form controls, Growl Integration on Mac, Mac Installer
Places Places Test Plan Tracy, ABillings L Bookmarks, History Indexing, Data Management, User Interface, Extensibility, Performance
Location Bar Location Bar Test Plan Stephend S new Location Bar design. Auto-complete
Search / Tabbed Browsing / Visual Refresh Search, Tabs, Visual Refresh Test Plan

Tab Switching Test

Mac Theme Test Plan 		Tomcat (search), Marcia (tabbed browsing) S User Interfacing, Animations, Data management
Security, Privacy / Password, Identity Security / Privacy Test Plan

Password Manager Testplan 		Tomcat, juanb L password management, Phishing protection, Private web browsing, Blacklisting (At Risk), password Manager, Password Security
Breakpad Client/Server Breakpad Crash Reporting Test Plan ss S Crash reporting tests from client/server, analysis, load
User Support TBD Tomcat, ss S Help content, Reporter, feedback, Customization options


Now owned by SUMO team - David Tensor.

L10N Testing L10N Test Plan ctalbert, tchung S test automation, new locales, string lengths, bookmarks, rss, search plugins
L10n Mozilla.com-website Testing L10n Mozilla.com Test Plan stephend, pascalc, tchung, timr M see test plan
UI Accessibility UI Accessibility Test Plan MarcoZ S UI accessibility convention testing
Major Update Tests (2.0.x -> FF3) Major Update Test Plan tchung, juanb, tracy M update verification from 2.0.x builds to 3.0b1+. includes upgrades/downgrades/sidegrade cases
Memory Leaks Memory Leaks Test Plan tomcat M Checking for memory leaks and asserts on debug builds

Regression Areas

  • Top Web accessibility Tests
    • Plugins (eg. Flash, java, AJAX)
    • Extensions (eg. Firebug, NoScript, Fotofox)
    • Themes (eg. Vista, Joga, Simpsons)
    • Top Site Tests (eg. Alexa top 50)
    • Financial Sites (eg. banks, online trading)
  • Accessibility Tests
  • User Performance Testing
  • Major / Minor Update Testing
  • L10N Testing (may coincide with Feature area)
  • Distribution Testing (may coincide with Feature area)

Ownership Needed Litmus Test Case Areas

These are Litmus test case areas that are currently unowned.

  • RSS->marcia
  • Printing -> Tomcat, tchung
  • Location Bar -> Stephend
  • Find -> TBD
  • Customize Toolbar -> TBD
  • Mac Leopard -> marcia


Tiger Team Testing

Daily smoketests are ran against nightly trunk builds. Please see marcia for more information.

Link to Schedule and Assignments


Extensions

For Firefox 2 we had Bug 353206 as extension Tracking bug.

The following extensions were never fixed from their extension compatibly authors to be able to install in Firefox 2.

Top Crash reports

The main report is the easiest to view.

Tracking Top crashes from Beta 3

  • High Priority for beta 4
    • Resolved Fixed bug 418377 - crash [@ XPCWrappedNativeScope::FindInJSObjectScope(XPCCallContext&, JSObject*, int)]
    • Nom for blocking? please approve bug 418378 - crash [@ nsGlobalWindow::SaveWindowState(nsISupports**)]
    • Nom for blocking? please approve bug 418381 - crash [@ HashString(nsAString_internal const&)]
  • Lower Priority
    • bug 391311 - Topcrash: [@ nsChromeRegistry::CheckForNewChrome] during startup
    • bug 418379 - crash [@ nsNavHistoryFolderResultNode::FindChildById(__int64, unsigned int*)]
    • bug 418382 - crash [@ UniscribeItem::SaveGlyphs(gfxTextRun*)]
    • bug 418384 - crash [@ ClientData::GetOtlTable(long, unsigned char const**, unsigned long*)]

Banking and Financial Sites

For Firefox 3, we’re asking community members and MoCo QA to state which banks and financial institutions they're willing to test. Please add your name to the list.

L10n Testing with Third Parties

For additional string testing, Mozilla contracts a third party company to utilize language experts with knowledge of Testing. Results are listed in the links here.

Review Feedback

In this section, we will roll up all common feedback pulled from external sources that appears common and should be tracked by QA. Feedback sources can include:

Feedback summary and review notes are posted here.

Flash Beta plugin Testing

Post Mortem Notes

The following is a collection of post mortem notes that the team has collected over the course of the release.


Additional References

New FF3 Feature list - An incomplete list of what Firefox 3 developers are working on.

Retrieved from "https://dev.wikimo.nonprod.webservices.mozgcp.net/index.php?title=QA/Firefox3/TestPlan&oldid=97143"