QA/Restartless System Addons
This section describes the modifications that have been made to this wiki page. A new row has been completed each time the content of this document is updated (small corrections for typographical errors do not need to be recorded). The description of the modification contains the differences from the prior version, in terms of what sections were updated and to what extent.
Date | Version | Author | Description |
---|---|---|---|
09/19/2016 | 1.0 | Alexandru Simonca Cristian Comorasu |
Created first draft |
Overview
Purpose
Detail the purpose of this document. For example:
- The test scope, focus areas and objectives
- The test responsibilities
- The test strategy for the levels and types of test for this release
- The entry and exit criteria
- The basis of the test estimates
- Any risks, issues, assumptions and test dependencies
- The test schedule and major milestones
- The test deliverables
Scope
This wiki details the testing that will be performed by the project team for the Restartless System Add-ons in Firefox. Its purpose is to document:
- What will be tested
- How testing will be performed
Ownership
Developer:
[mailto:rhelmer@mozilla.com Robert Helmer (:rhelmer)
Leading QA efforts:
Alexandru Simonca (:asimonca)
Cristian Comorasu (:cristicomo)
Testing Summary
Scope of Testing
In Scope
The goal is for Firefox to update add-ons without the browser to restart in order to function properly.
Out of Scope
Testing functionality associated to specific system add-ons.
Requirements for testing
Environments
Platforms:
- Windows 7 x32
- Windows 8.1 x32
- Windows 10 x64
- Mac OS X 10.11
- Mac OX X 10.12.3
- Ubuntu 14.04 LTS x64
Update channels:
- release-sysaddon
- tbd
Test Strategy
Test Objectives
This section details the progression test objectives that will be covered.
Ref | Test Objective | Evaluation Criteria | Test Type | Owner |
---|---|---|---|---|
1 | Restartlessly install an update to a sys add-on | “about:support” is accurate | Manual | Desktop Team |
2 | Restartlessly rollback to a sys add-on | “about:support” is accurate | Manual | Desktop Team |
3 | Restartlessly remove a sys add-on | “about:support” is accurate | Manual | Desktop Team |
4 | Restartlessly install System Addons - special cases | “about:support” is accurate | Manual | Desktop Team |
Builds
This section should contain links for builds with the feature
Test Execution Schedule
The following table identifies the anticipated testing period available for test execution.
Project phase | Start Date | End Date |
---|---|---|
Start project | TBD | - |
Study documentation/specs received from developers | TBD | TBD |
QA - Test plan creation | TBD | TBD |
QA - Test cases/Env preparation | TBD | TBD |
QA - Nightly Testing | TBD | TBD |
QA - Aurora Testing | TBD | TBD |
QA - Beta Testing | TBD | TBD |
Release Date | TBD | TBD |
Testing Tools
Detail the tools to be used for testing, for example see the following table:
Process | Tool |
---|---|
Test plan creation | Mozilla wiki |
Test case creation | TestRail |
Test case execution | TestRail |
Bugs Management | Bugzilla |
Status
Overview
Track the date and build number when feature was merged to Aurora
Track the date and build number when feature was merged to Release/Beta
Risk Analysis
Identify the high-risk assumptions
Identify existing bugs on the feature with high risk
Identify if other areas are affected by the fix
Risk Areas | Mitigation |
---|---|
TBD | TBD |
References
- Meta bug: 1204156 - Allow updated system add-ons to install without a restart
- Additional information can be found here
Testcases
- Summary of testing scenarios
Test areas
Test Areas | Covered | Details |
---|---|---|
Private Window | no | |
Multi-process enabled | yes | |
Multi-process disabled | no | |
Feature upgrades/downgrades data as expected | yes |
Test Suite
Full Test Suite - Link with tests in TestRail
Bug Work
Tracking bug - meta bug
Sign Off
Criteria
- All test cases should be executed
- All blockers & critical must be fixed and verified or have an agreed-upon timeline for being fixed (as determined by engineering/RelMan/QA)
Results
- Mid-Aurora sign-off
- Pre-Beta sign-off
Checklist
Exit Criteria | Status | Details |
---|---|---|
Testing Prerequisites (specs, use cases) | [DONE] | |
Testing Infrastructure setup | ||
Test Plan Creation | [DONE] | |
Test Cases Creation | [DONE] | |
Full Functional Tests Execution | [DONE] | |
Automation Coverage | N/A | |
Performance Testing | N/A | |
All Defects Logged | ||
Critical/Blockers Fixed and Verified | N/A | |
Daily Status Report (email/etherpad statuses/ gdoc with results) | N/A | |
Metrics/Telemetry | N/A | |
QA Signoff - Nightly Release | MISSED | |
QA Aurora - Full Testing | [DONE] | |
QA Signoff - Aurora Release | [DONE] | |
QA Beta - Full Testing | [DONE] | |
QA Signoff - Beta Release | [DONE] |