QA/Restartless System Addons: Difference between revisions

From MozillaWiki
< QA
Jump to navigation Jump to search
No edit summary
(The Test Plan for "Restartless System Add-ons")
Line 33: Line 33:


'''Developer:'''
'''Developer:'''
Robert Helmer (rhelmer@mozilla.com) <br />
[mailto:rhelmer@mozilla.com Robert Helmer (:rhelmer) <br />


'''Leading QA efforts:'''
'''Leading QA efforts:'''
[mailto:alexandru.simonca@softvision.ro Alexandru Simonca (:asimonca)]
[mailto:alexandru.simonca@softvision.ro Alexandru Simonca (:asimonca)] <br />
[mailto:cristian.comorasu@softvision.ro Cristian Comorasu (:cristicomo)]
[mailto:cristian.comorasu@softvision.ro Cristian Comorasu (:cristicomo)] <br />
 
= Testing Summary =
 
== Scope of Testing ==
 
=== In Scope ===
 
The goal is for Firefox to update add-ons without the browser to restart in order to function properly.
 
=== Out of Scope ===
 
Testing functionality associated to specific system add-ons.
 
= Requirements for testing =
 
== Environments ==
 
Platforms:
* Windows 7 x32
* Windows 8.1 x32
* Windows 10 x64
* Mac OS X 10.11
* Mac OX X 10.12.3
* Ubuntu 14.04 LTS x64 <br />
 
Update channels:
* release-sysaddon
* tbd
 
= Test Strategy =
 
== Test Objectives ==
 
This section details the progression test objectives that will be covered.
 
{| class="wikitable"
|-
! Ref !! Test Objective !! Evaluation Criteria !! Test Type !! Owner
|-
| 1 || Restartlessly install an update to a sys add-on || “about:support” is accurate ||  Manual || Desktop Team
|-
| 2 || Restartlessly rollback to a sys add-on || “about:support” is accurate || Manual || Desktop Team
|-
| 3 || Restartlessly remove a sys add-on || “about:support” is accurate || Manual || Desktop Team
|-
| 4 || Restartlessly install System Addons - special cases || “about:support” is accurate || Manual || Desktop Team
|-
|}
 
== Builds ==
This section should contain links for builds with the feature
 
* [https://www.youtube.com/watch?v=3bHLifwHrN0 Link for Beta builds]
 
== Test Execution Schedule ==
The following table identifies the anticipated testing period available for test execution.
 
{| class="wikitable"
|-
! Project phase !! Start Date !! End Date
|-
| Start project || TBD || -
|-
| Study documentation/specs received from developers || TBD || TBD
|-
| QA - Test plan creation || TBD || TBD
|-
| QA - Test cases/Env preparation || TBD || TBD
|-
| QA - Nightly Testing || TBD || TBD
|-
| QA - Aurora Testing || TBD || TBD
|-
| QA - Beta Testing || TBD || TBD
|-
| Release Date || TBD || TBD
|}
 
== Testing Tools ==
 
Detail the tools to be used for testing, for example see the following table:
 
{| class="wikitable"
|-
! Process !! Tool
|-
| Test plan creation || Mozilla wiki
|-
| Test case creation || TestRail
|-
| Test case execution || TestRail
|-
| Bugs Management || Bugzilla
|}
 
 
=== Status ===
== Overview ==
Track the date and build number when feature was merged to Aurora<br />
Track the date and build number when feature was merged to Release/Beta<br />
 
== Risk Analysis ==
 
Identify the high-risk assumptions<br />
Identify existing bugs on the feature with high risk<br />
Identify if other areas are affected by the fix<br />
 
{| class="wikitable"
|-
! Risk Areas !! Mitigation
|-
|  ||
|}
 
= References =
 
* Meta bug: [https://bugzilla.mozilla.org/show_bug.cgi?id=1204156 1204156 - Allow updated system add-ons to install without a restart]
* Additional information can be found [https://gecko.readthedocs.io/en/latest/toolkit/mozapps/extensions/addon-manager/SystemAddons.html here]
 
= Testcases =
* Summary of testing scenarios
 
== Test areas ==
 
{| class="wikitable"
|-
! Test Areas !! Covered !! Details
|-
| Private Window || no ||
|-
| Multi-process enabled || yes ||
|-
| Multi-process disabled || no ||
|-
| Feature upgrades/downgrades data as expected || yes ||
|}
 
== Test Suite ==
 
Full Test Suite - Link with tests in [https://testrail.stage.mozaws.net/index.php?/suites/view/374&group_by=cases:section_id&group_order=asc TestRail]
 
 
= Bug Work =
Tracking bug - [https://bugzilla.mozilla.org/show_bug.cgi?id=1204156 meta bug]
 
= Sign Off =
== Criteria ==
 
* All test cases should be executed
* All blockers & critical must be fixed and verified or have an agreed-upon timeline for being fixed (as determined by engineering/RelMan/QA)
 
= Results =
* Mid-Aurora sign-off
** https://testrail.stage.mozaws.net/index.php?/plans/view/1737
* Pre-Beta sign-off
** https://testrail.stage.mozaws.net/index.php?/plans/view/1962
 
== Checklist ==
 
{| class="wikitable"
|-
! Exit Criteria !! Status !! Details
|-
| Testing Prerequisites (specs, use cases) || [DONE] ||
|-
| Testing Infrastructure setup || ||
|-
| Test Plan Creation || [DONE] ||
|-
| Test Cases Creation || [DONE] ||
|-
| Full Functional Tests Execution || [DONE] ||
|-
| Automation Coverage || N/A ||
|-
|Performance Testing || N/A ||
|-
| All Defects Logged ||  ||
|-
| Critical/Blockers Fixed and Verified || N/A ||
|-
| Daily Status Report (email/etherpad statuses/ gdoc with results) || N/A ||
|-
| Metrics/Telemetry || N/A ||
|-
| QA Signoff - Nightly Release || MISSED ||
|-
| QA Aurora - Full Testing || [DONE] ||
|-
| QA Signoff - Aurora Release || [DONE] ||
|-
| QA Beta - Full Testing || [DONE] ||
|-
| QA Signoff - Beta Release || [DONE] ||
|}

Revision as of 14:12, 13 February 2017

This section describes the modifications that have been made to this wiki page. A new row has been completed each time the content of this document is updated (small corrections for typographical errors do not need to be recorded). The description of the modification contains the differences from the prior version, in terms of what sections were updated and to what extent.

Date Version Author Description
09/19/2016 1.0 Alexandru Simonca
Cristian Comorasu		 Created first draft

Overview

Purpose

Detail the purpose of this document. For example:

  • The test scope, focus areas and objectives
  • The test responsibilities
  • The test strategy for the levels and types of test for this release
  • The entry and exit criteria
  • The basis of the test estimates
  • Any risks, issues, assumptions and test dependencies
  • The test schedule and major milestones
  • The test deliverables

Scope

This wiki details the testing that will be performed by the project team for the Restartless System Add-ons in Firefox. Its purpose is to document:

  • What will be tested
  • How testing will be performed

Ownership

Developer: [mailto:rhelmer@mozilla.com Robert Helmer (:rhelmer)

Leading QA efforts: Alexandru Simonca (:asimonca)
Cristian Comorasu (:cristicomo)

Testing Summary

Scope of Testing

In Scope

The goal is for Firefox to update add-ons without the browser to restart in order to function properly.

Out of Scope

Testing functionality associated to specific system add-ons.

Requirements for testing

Environments

Platforms:

  • Windows 7 x32
  • Windows 8.1 x32
  • Windows 10 x64
  • Mac OS X 10.11
  • Mac OX X 10.12.3
  • Ubuntu 14.04 LTS x64

Update channels:

  • release-sysaddon
  • tbd

Test Strategy

Test Objectives

This section details the progression test objectives that will be covered.

Ref Test Objective Evaluation Criteria Test Type Owner
1 Restartlessly install an update to a sys add-on “about:support” is accurate Manual Desktop Team
2 Restartlessly rollback to a sys add-on “about:support” is accurate Manual Desktop Team
3 Restartlessly remove a sys add-on “about:support” is accurate Manual Desktop Team
4 Restartlessly install System Addons - special cases “about:support” is accurate Manual Desktop Team

Builds

This section should contain links for builds with the feature

Test Execution Schedule

The following table identifies the anticipated testing period available for test execution.

Project phase Start Date End Date
Start project TBD -
Study documentation/specs received from developers TBD TBD
QA - Test plan creation TBD TBD
QA - Test cases/Env preparation TBD TBD
QA - Nightly Testing TBD TBD
QA - Aurora Testing TBD TBD
QA - Beta Testing TBD TBD
Release Date TBD TBD

Testing Tools

Detail the tools to be used for testing, for example see the following table:

Process Tool
Test plan creation Mozilla wiki
Test case creation TestRail
Test case execution TestRail
Bugs Management Bugzilla


Status

Overview

Track the date and build number when feature was merged to Aurora
Track the date and build number when feature was merged to Release/Beta

Risk Analysis

Identify the high-risk assumptions
Identify existing bugs on the feature with high risk
Identify if other areas are affected by the fix

Risk Areas Mitigation

References

Testcases

  • Summary of testing scenarios

Test areas

Test Areas Covered Details
Private Window no
Multi-process enabled yes
Multi-process disabled no
Feature upgrades/downgrades data as expected yes

Test Suite

Full Test Suite - Link with tests in TestRail


Bug Work

Tracking bug - meta bug

Sign Off

Criteria

  • All test cases should be executed
  • All blockers & critical must be fixed and verified or have an agreed-upon timeline for being fixed (as determined by engineering/RelMan/QA)

Results

Checklist

Exit Criteria Status Details
Testing Prerequisites (specs, use cases) [DONE]
Testing Infrastructure setup
Test Plan Creation [DONE]
Test Cases Creation [DONE]
Full Functional Tests Execution [DONE]
Automation Coverage N/A
Performance Testing N/A
All Defects Logged
Critical/Blockers Fixed and Verified N/A
Daily Status Report (email/etherpad statuses/ gdoc with results) N/A
Metrics/Telemetry N/A
QA Signoff - Nightly Release MISSED
QA Aurora - Full Testing [DONE]
QA Signoff - Aurora Release [DONE]
QA Beta - Full Testing [DONE]
QA Signoff - Beta Release [DONE]
Retrieved from "https://dev.wikimo.nonprod.webservices.mozgcp.net/index.php?title=QA/Restartless_System_Addons&oldid=1162743"