QA/Send Tab: Difference between revisions
(Created page with "'''Revision History''' This section describes the modifications that have been made to this wiki page. A new row has been completed each time the content of this document is...") |
|||
(42 intermediate revisions by 2 users not shown) | |||
Line 1: | Line 1: | ||
'''Approvals Required / Received''' | |||
The following individuals are required to/have approved this Test Plan: | |||
{| class="wikitable" | |||
|- | |||
! Name !! Title !! Department !! Approval Date !! Method | |||
|- | |||
| RyanVM || QA Reviewer ||Product Integrity || n/a|| Email | |||
|- | |||
|Edouard Oger || Software Engineer || Engineering || || Email | |||
|- | |||
| Julie McCracken || EPM || Product Management || || Email | |||
|} | |||
'''Revision History''' | '''Revision History''' | ||
Line 8: | Line 23: | ||
|- | |- | ||
| 7/31/2017 || 1.0 || Abe Masresha || Created first draft | | 7/31/2017 || 1.0 || Abe Masresha || Created first draft | ||
|- | |||
| 8/15/2017 || 1.1 || Abe Masresha || updated links and updated some sections | |||
|- | |||
| 8/17/2017 || 1.2 || Abe Masresha || updated links and risk assesment section | |||
|- | |- | ||
|} | |} | ||
Line 13: | Line 32: | ||
= Overview = | = Overview = | ||
== Purpose == | == Purpose == | ||
The purpose of this work is to make the send tab | The purpose of this work is to make the send tab discoverable to non-Sync/FxA users. | ||
== Scope == | == Scope == | ||
Line 28: | Line 46: | ||
*Automation is covered by the developer | *Automation is covered by the developer | ||
::: [Link to automation test script will be provided here] | ::: [Link to automation test script will be provided here] | ||
== Ownership == | == Ownership == | ||
'''Mozilla Development'''<br /> | '''Mozilla Development'''<br /> | ||
Edouard Oger<br /> | |||
'''QA Eng. Softvision Las Vegas'''<br /> | '''QA Eng. Softvision Las Vegas'''<br /> | ||
[mailto:amasresha@softvision.com Abe Masresha ]<br /> | |||
[mailto:amasresha@softvision.com Abe Masresha | |||
= Testing summary = | = Testing summary = | ||
== Scope of Testing == | == Scope of Testing == | ||
=== In Scope === | === In Scope === | ||
This feature will be tested manually on Windows, Mac, and Linux operating systems and on iOS and Android devices. | This feature will be tested manually on Windows, Mac, and Linux operating systems and on iOS and Android devices.<br> | ||
Automation will be handled by the developer<br /> | |||
=== Out of Scope === | === Out of Scope === | ||
In fennec, the send tab may not be discoverable to non-Sync/FxA users as it is not implemented yet. | |||
= Requirements for testing = | = Requirements for testing = | ||
Line 61: | Line 71: | ||
MacOSX 10.12 with Android device | MacOSX 10.12 with Android device | ||
Android with another Android device | Android with another Android device | ||
== Channel dependent settings (configs) and environment setups == | == Channel dependent settings (configs) and environment setups == | ||
=== Nightly === | === Nightly === | ||
Firefox Nightly 56.0a1, no special configuration is required<br> | |||
Firefox Nightly 56.0a1 <br | |||
=== Beta === | === Beta === | ||
Beta 56.0bx, no special configuration is required<br> | |||
Beta 56. | |||
< | |||
= Test Strategy = | = Test Strategy = | ||
== Risk Assessment and Coverage == | |||
{| class="wikitable" | |||
|- | |||
! ID !! Description / Threat Description !! Covered by Test Objective !! Magnitude !! Probability !! Priority !! Impact Score | |||
|- | |||
| RAC-1 || The send tab feature may not send tab || TO-1 || 3-High || 2-Possible || 3-High ||18 | |||
|- | |||
| RAC-2 ||The send tab may not be discoverable to non-Sync/FxA users (Desktop only - at this time) || TO-2 || 3-High || 2-Possible || 3-High ||18 | |||
|- | |||
| RAC-3 || The send tab feature may not implement as described in the mock. [https://docs.google.com/document/d/1jnOVFROc461SyYv1g7qOGfTWi2nOvPsPh8N7KJKFhe8/edit Mock] || TO-3 || 2-Moderate || 2-Possible || 2-Medium ||8 | |||
|} | |||
'''Values:''' | |||
* '''Magnitude:''' 1- Low , ''2-Moderate'', '''3-High''' | |||
* '''Probability:''' 1-Unlikely, ''2-Possible'', '''3-Almost Certain''' | |||
* '''Priority:''' 1 - Low, ''2-Medium'', '''3-High''' | |||
'''Impact Score Breakdown:''' | |||
* An impact value of 1, 2, 3, 4 would describe an area which although should be covered there aren't expected any discoveries of critical issues. | |||
* An impact value of 6, 8, 9, 12 would describe an area in which we expect to find issues but those issues are not expected to be critical. | |||
* An impact value of 18 or 27 would describe an area on which it is likely to find issues and those issues to be critical or blockers. | |||
== Test Objectives == | == Test Objectives == | ||
Test | |||
{| class="wikitable" | |||
|- | |||
! Ref !! Function !! Test Objective !! Evaluation Criteria !! Test Type !! RAC !! Owners | |||
|- | |||
| 1 || Unconfigured-Sync not loaded || Check "Send page to Device" options within 10 seconds of the browser launch|| Check if "Not Connected to Sync" is displayed || Manual || RAC-3 || Eng Team | |||
|- | |||
| 2 ||Unconfigured-Sync loaded || Check"Send Page to Device" options after 10 seconds of the browser launch|| Check if "Not Connected to Sync" is displayed || Manual || RAC-3|| Eng Team | |||
|- | |||
| 3 ||Signed-in - Sync not loaded || Signed-in and check "Send page to Device" options within 10 seconds || Check if "Syncing Devices ..." is displayed|| Manual || RAC-3 || Eng Team | |||
|- | |||
| 4 ||Signed-in - Sync loaded || Signed-in and check "Send page to Device" option after 10 seconds || Check if your synced devices are listed || Manual || RAC-3 || Eng Team | |||
|- | |||
| 5 ||Signed-in but unverified ||If the user does not confirm email, sync should notify the user to verify || Send tab feature notifies the user to verify email || Manual || RAC-3|| Eng Team | |||
|- | |||
| 6 ||Signed-in but password changed || Signed-in to more than one devices and change password from one of them. Other device should prompt to verify or sign-in || Check if user gets prompt from other device || Manual || RAC-3|| Eng Team | |||
|- | |||
| 7 ||Signed-in with only 1 device|| Signed-in with only 1 device and check if "No Devices Connected" is displayed ||"No Devices Connected" should be displayed || Manual || RAC-3 || Eng Team | |||
|- | |||
| 8 ||Signed-in (happy case) || Check if user able to send tab using send tab feature || Check if links are received by other devices || Manual || RAC1,RAC2 || Eng Team | |||
|- | |||
| 9 ||Signed-in / Unsendable link|| Some links like "about:support" are not sendable||check some links are unsendable || Manual || RAC-3 || Eng Team | |||
|} | |||
== Builds == | == Builds == | ||
This section should contain links for builds with the feature - | This section should contain links for builds with the feature - | ||
* Links for Nightly | * Links for Nightly build: use the latest official Nightly build | ||
* Links for Beta | * Links for Beta build: use the latest official beta build | ||
== Test Execution Schedule == | == Test Execution Schedule == | ||
Line 101: | Line 145: | ||
|- | |- | ||
| Start project | | Start project | ||
|style="text-align:center;" | 07/25/2017 || | |style="text-align:center;" | 07/25/2017 || {{in progress|}} | ||
|- | |- | ||
| Study documentation/specs received from developers | | Study documentation/specs received from developers | ||
|style="text-align:center;" | 07/25/2017 || | |style="text-align:center;" | 07/25/2017 || 07/27/2017 | ||
|- | |- | ||
| QA - Test plan creation | | QA - Test plan creation | ||
|style="text-align:center;" | 7/31/2017 || | |style="text-align:center;" | 7/31/2017 || 7/25/2017 | ||
|- | |- | ||
| QA - Test cases/Env preparation | | QA - Test cases/Env preparation | ||
|style="text-align:center;" | 7/25/2017 || | |style="text-align:center;" | 7/25/2017 || {{in progress|}} | ||
|- | |- | ||
| QA - Nightly Testing | | QA - Nightly Testing | ||
|style="text-align:center;" | 7/27/2017 || | |style="text-align:center;" | 7/27/2017 || 7/31/2017 | ||
|- | |- | ||
| QA - Beta Testing | | QA - Beta Testing | ||
|style="text-align:center;" | | |style="text-align:center;" | {{in progress|}} || {{in progress|}} | ||
|- | |- | ||
| Release Date | | Release Date | ||
Line 130: | Line 174: | ||
| Test plan creation || Mozilla wiki | | Test plan creation || Mozilla wiki | ||
|- | |- | ||
| Test case creation || [https://testrail.stage.mozaws.net/index.php?/suites/view/1304&group_by=cases:section_id&group_order=desc TestRail] | | Test case creation || [https://testrail.stage.mozaws.net/index.php?/suites/view/1304&group_by=cases:section_id&group_order=desc TestRail] | ||
|- | |- | ||
| Test case execution || [https://testrail.stage.mozaws.net/index.php?/plans/view/5365 TestRail] | | Test case execution || [https://testrail.stage.mozaws.net/index.php?/plans/view/5365 TestRail] | ||
|- | |- | ||
| Bugs management || Bugzilla | | Bugs management || [https://bugzilla.mozilla.org/ Bugzilla] | ||
|} | |} | ||
Line 144: | Line 188: | ||
= References = | = References = | ||
* [https://bugzilla.mozilla.org/show_bug.cgi?id=1359894 Main bug:1359894] <br> | * [https://bugzilla.mozilla.org/show_bug.cgi?id=1359894 Main bug:1359894] <br> | ||
* [https://bugzilla.mozilla.org/show_bug.cgi?id=1370574 | * [https://bugzilla.mozilla.org/show_bug.cgi?id=1370574 Send Tabs landing page Bug] <br> | ||
* [https://bugzilla.mozilla.org/show_bug.cgi?id=1383943 Feature Documentation Bug] | |||
= Testcases = | = Testcases = | ||
Line 151: | Line 196: | ||
|- | |- | ||
! Test Areas !! Covered !! Details | ! Test Areas !! Covered !! Details | ||
|- | |- | ||
| Multi-Process Enabled | | Multi-Process Enabled | ||
|style="text-align:center;" | Yes| | |style="text-align:center;" | Yes|| The feature should work proprely in e10s browser | ||
|- | |- | ||
| ''' Add-ons ''' | | ''' Add-ons ''' | ||
|- | |- | ||
| Testing with existing/popular addons | | Testing with existing/popular addons | ||
|style="text-align:center;" | yes|| Feature functionality will be tested with popular addons | |style="text-align:center;" | yes|| Feature functionality will be tested with popular addons | ||
|} | |} | ||
== Test suite == | == Test suite == | ||
Full Test suite - | Full Test suite -[https://testrail.stage.mozaws.net/index.php?/suites/view/1304&group_by=cases:section_id&group_order=desc Link to test rail] | ||
Smoke Test suite - | Smoke Test suite -Full test suite is used for pre-beta test cycle | ||
Regression Test suite - | Regression Test suite - Full test suite is used for pre-beta test cycle | ||
= Bug Work = | = Bug Work = | ||
Line 203: | Line 219: | ||
! Bug No !! Summary !! Status !! Firefox Version | ! Bug No !! Summary !! Status !! Firefox Version | ||
|- | |- | ||
|[https://bugzilla.mozilla.org/show_bug.cgi?id=1385440 1385440]|| Only one device receives the link when "Send to All Devices" is clicked|| | |[https://bugzilla.mozilla.org/show_bug.cgi?id=1385440 1385440]|| Only one device receives the link when "Send to All Devices" is clicked|| VERIFIED FIXED || 56 | ||
|- | |- | ||
|[https://bugzilla.mozilla.org/show_bug.cgi?id=1385442 1385442]|| Android device is not listed in sync'd device list on mac|| | |[https://bugzilla.mozilla.org/show_bug.cgi?id=1385442 1385442]|| Android device is not listed in sync'd device list on mac|| RESOLVED DUPLICATE || 56 | ||
|- | |- | ||
Line 214: | Line 230: | ||
|- | |- | ||
|[https://bugzilla.mozilla.org/show_bug.cgi?id= | |[https://bugzilla.mozilla.org/show_bug.cgi?id=1385458 1385458 ]|| Frozen spinner is shown when there are no devices to send tab to (iOS)|| NEW || 56 | ||
|- | |- | ||
|[https://bugzilla.mozilla.org/show_bug.cgi?id=1385442 1385442]|| In Android, Send Tab is not discoverable to non Sync/FxA users|| RESOLVED INVALID|| 56 | |[https://bugzilla.mozilla.org/show_bug.cgi?id=1385442 1385442]|| In Android, Send Tab is not discoverable to non Sync/FxA users|| RESOLVED INVALID|| 56 | ||
Line 227: | Line 243: | ||
Checklist | Checklist | ||
* All test cases should be executed | * All test cases should be executed | ||
* Has sufficient automated test coverage (as measured by code coverage tools) | * Has sufficient automated test coverage (as measured by code coverage tools) | ||
* All blockers, criticals must be fixed and verified or have an agreed-upon timeline for being fixed (as determined by engineering/RelMan/QA) | * All blockers, criticals must be fixed and verified or have an agreed-upon timeline for being fixed (as determined by engineering/RelMan/QA) | ||
== Results == | == Results == | ||
'''Nightly testing'''<br /> | '''Nightly testing'''<br /> | ||
List of OSes that will be covered by testing- listed in environments section<br /> | |||
List of OSes that will be covered by testing<br /> | **[https://testrail.stage.mozaws.net/index.php?/reports/view/483 Link for the tests run] | ||
* | ** Daily Smoke, not avialable (n/a) | ||
* | ** Regression Test suite: not avialable (n/a)<br /> | ||
** Daily Smoke, | |||
** Regression Test suite | |||
<br /> | |||
'''Merge to Beta Sign-off'''<br /> | '''Merge to Beta Sign-off'''<br /> | ||
Line 263: | Line 276: | ||
|style="text-align:center;" | {{mdone|}} || | |style="text-align:center;" | {{mdone|}} || | ||
|- | |- | ||
| Automation Coverage | | Automation Coverage | ||
|style="text-align:center;" | |style="text-align:center;" | | ||
|style="text-align:center;" | Will be covered by developer | |||
|style="text-align:center;" | | |||
|- | |- | ||
| | | Performance Testing | ||
|style="text-align:center;" | | |||
|style="text-align:center;" | Will be covered by developer | |||
|- | |- | ||
| | | All Defects Logged | ||
|style="text-align:center;" | | |style="text-align:center;" |{{mdone|}} | ||
|style="text-align:center;" | | |style="text-align:center;" | All found bugs are logged | ||
|- | |- | ||
| QA mid-Nightly Signoff|| | | Critical/Blockers Fixed and Verified | ||
|style="text-align:center;" | | |style="text-align:center;" |{{In progress|}} | ||
|style="text-align:center;" | Devs are fixing bugs | |||
|- | |||
| QA mid-Nightly Signoff | |||
|style="text-align:center;" |{{Missed |}} | |||
|style="text-align:center;"| Missed (test build and feature document were not avialable to QE) | |||
|- | |- | ||
| QA pre-Beta Signoff|| | | QA pre-Beta Signoff | ||
|style="text-align:center;"| | |style="text-align:center;" |{{mdone|}} | ||
|style="text-align:center;"| YELLOW (conditional sign-off) sent on 7/31/2017 | |||
|- | |- | ||
| QA Beta - Full Testing | | QA Beta - Full Testing | ||
|style="text-align:center;" | | |style="text-align:center;" |{{in progress|}} | ||
|style="text-align:center;" | | |||
|- | |- | ||
| QA pre-Release Signoff || | | QA pre-Release Signoff | ||
|style="text-align:center;" |{{in progress|}} | |||
|style="text-align:center;" | Email to be sent | |style="text-align:center;" | Email to be sent | ||
|} | |} |
Latest revision as of 14:50, 31 August 2017
Approvals Required / Received
The following individuals are required to/have approved this Test Plan:
Name | Title | Department | Approval Date | Method |
---|---|---|---|---|
RyanVM | QA Reviewer | Product Integrity | n/a | |
Edouard Oger | Software Engineer | Engineering | ||
Julie McCracken | EPM | Product Management |
Revision History
This section describes the modifications that have been made to this wiki page. A new row has been completed each time the content of this document is updated (small corrections for typographical errors do not need to be recorded). The description of the modification contains the differences from the prior version, in terms of what sections were updated and to what extent.
Date | Version | Author | Description |
---|---|---|---|
7/31/2017 | 1.0 | Abe Masresha | Created first draft |
8/15/2017 | 1.1 | Abe Masresha | updated links and updated some sections |
8/17/2017 | 1.2 | Abe Masresha | updated links and risk assesment section |
Overview
Purpose
The purpose of this work is to make the send tab discoverable to non-Sync/FxA users.
Scope
- Testing will focus on:
- Stability of the browser with this work.
- If send tab is discoverable to non-Sync/FxA users (Desktop only - at this time)
- If user able to send links to other devices when signed-in to sync
- How testing will be performed:
- Manual
- Automation is covered by the developer
- [Link to automation test script will be provided here]
Ownership
Mozilla Development
Edouard Oger
QA Eng. Softvision Las Vegas
Abe Masresha
Testing summary
Scope of Testing
In Scope
This feature will be tested manually on Windows, Mac, and Linux operating systems and on iOS and Android devices.
Automation will be handled by the developer
Out of Scope
In fennec, the send tab may not be discoverable to non-Sync/FxA users as it is not implemented yet.
Requirements for testing
Test Environment Combinations
Windows 10 with iOS device Windows 7 32bits with Linux OS Linux with Android device MacOSX 10.12 with Android device Android with another Android device
Channel dependent settings (configs) and environment setups
Nightly
Firefox Nightly 56.0a1, no special configuration is required
Beta
Beta 56.0bx, no special configuration is required
Test Strategy
Risk Assessment and Coverage
ID | Description / Threat Description | Covered by Test Objective | Magnitude | Probability | Priority | Impact Score |
---|---|---|---|---|---|---|
RAC-1 | The send tab feature may not send tab | TO-1 | 3-High | 2-Possible | 3-High | 18 |
RAC-2 | The send tab may not be discoverable to non-Sync/FxA users (Desktop only - at this time) | TO-2 | 3-High | 2-Possible | 3-High | 18 |
RAC-3 | The send tab feature may not implement as described in the mock. Mock | TO-3 | 2-Moderate | 2-Possible | 2-Medium | 8 |
Values:
- Magnitude: 1- Low , 2-Moderate, 3-High
- Probability: 1-Unlikely, 2-Possible, 3-Almost Certain
- Priority: 1 - Low, 2-Medium, 3-High
Impact Score Breakdown:
- An impact value of 1, 2, 3, 4 would describe an area which although should be covered there aren't expected any discoveries of critical issues.
- An impact value of 6, 8, 9, 12 would describe an area in which we expect to find issues but those issues are not expected to be critical.
- An impact value of 18 or 27 would describe an area on which it is likely to find issues and those issues to be critical or blockers.
Test Objectives
Ref | Function | Test Objective | Evaluation Criteria | Test Type | RAC | Owners |
---|---|---|---|---|---|---|
1 | Unconfigured-Sync not loaded | Check "Send page to Device" options within 10 seconds of the browser launch | Check if "Not Connected to Sync" is displayed | Manual | RAC-3 | Eng Team |
2 | Unconfigured-Sync loaded | Check"Send Page to Device" options after 10 seconds of the browser launch | Check if "Not Connected to Sync" is displayed | Manual | RAC-3 | Eng Team |
3 | Signed-in - Sync not loaded | Signed-in and check "Send page to Device" options within 10 seconds | Check if "Syncing Devices ..." is displayed | Manual | RAC-3 | Eng Team |
4 | Signed-in - Sync loaded | Signed-in and check "Send page to Device" option after 10 seconds | Check if your synced devices are listed | Manual | RAC-3 | Eng Team |
5 | Signed-in but unverified | If the user does not confirm email, sync should notify the user to verify | Send tab feature notifies the user to verify email | Manual | RAC-3 | Eng Team |
6 | Signed-in but password changed | Signed-in to more than one devices and change password from one of them. Other device should prompt to verify or sign-in | Check if user gets prompt from other device | Manual | RAC-3 | Eng Team |
7 | Signed-in with only 1 device | Signed-in with only 1 device and check if "No Devices Connected" is displayed | "No Devices Connected" should be displayed | Manual | RAC-3 | Eng Team |
8 | Signed-in (happy case) | Check if user able to send tab using send tab feature | Check if links are received by other devices | Manual | RAC1,RAC2 | Eng Team |
9 | Signed-in / Unsendable link | Some links like "about:support" are not sendable | check some links are unsendable | Manual | RAC-3 | Eng Team |
Builds
This section should contain links for builds with the feature -
- Links for Nightly build: use the latest official Nightly build
- Links for Beta build: use the latest official beta build
Test Execution Schedule
The following table identifies the anticipated testing period available for test execution.
Project phase | Start Date | End Date |
---|---|---|
Start project | 07/25/2017 | IN PROGRESS |
Study documentation/specs received from developers | 07/25/2017 | 07/27/2017 |
QA - Test plan creation | 7/31/2017 | 7/25/2017 |
QA - Test cases/Env preparation | 7/25/2017 | IN PROGRESS |
QA - Nightly Testing | 7/27/2017 | 7/31/2017 |
QA - Beta Testing | IN PROGRESS | IN PROGRESS |
Release Date |
Testing Tools
Detail the tools to be used for testing, for example see the following table:
Process | Tool |
---|---|
Test plan creation | Mozilla wiki |
Test case creation | TestRail |
Test case execution | TestRail |
Bugs management | Bugzilla |
Status
Overview
Landed on Nightly on: 7/19/2017
Merged to Beta on: [date]
References
Testcases
Test Areas
Test Areas | Covered | Details |
---|---|---|
Multi-Process Enabled | Yes | The feature should work proprely in e10s browser |
Add-ons | ||
Testing with existing/popular addons | yes | Feature functionality will be tested with popular addons |
Test suite
Full Test suite -Link to test rail Smoke Test suite -Full test suite is used for pre-beta test cycle Regression Test suite - Full test suite is used for pre-beta test cycle
Bug Work
Bug No | Summary | Status | Firefox Version |
---|---|---|---|
1385440 | Only one device receives the link when "Send to All Devices" is clicked | VERIFIED FIXED | 56 |
1385442 | Android device is not listed in sync'd device list on mac | RESOLVED DUPLICATE | 56 |
1385451 | in Android "send to other devices" is not displayed | NEW | 56 |
1385454 | on Android, "verify your account" option is not available when the user changes password from other device | NEW | 56 |
1385458 | Frozen spinner is shown when there are no devices to send tab to (iOS) | NEW | 56 |
1385442 | In Android, Send Tab is not discoverable to non Sync/FxA users | RESOLVED INVALID | 56 |
1385447 | All previously logged in devices are displayed in sync'd device list | RESOLVED WORKSFORME | 56 |
Sign off
Criteria
Checklist
- All test cases should be executed
- Has sufficient automated test coverage (as measured by code coverage tools)
- All blockers, criticals must be fixed and verified or have an agreed-upon timeline for being fixed (as determined by engineering/RelMan/QA)
Results
Nightly testing
List of OSes that will be covered by testing- listed in environments section
- Link for the tests run
- Daily Smoke, not avialable (n/a)
- Regression Test suite: not avialable (n/a)
Merge to Beta Sign-off
List of OSes that will be covered by testing
- Link for the tests run
- Full Test suite
Checklist
Exit Criteria | Status | Notes/Details |
---|---|---|
Testing Prerequisites (specs, use cases) | [DONE] | |
Testing Infrastructure setup | [DONE] | |
Test Plan Creation | [DONE] | |
Test Cases Creation | [DONE] | |
Automation Coverage | Will be covered by developer | |
Performance Testing | Will be covered by developer | |
All Defects Logged | [DONE] | All found bugs are logged |
Critical/Blockers Fixed and Verified | IN PROGRESS | Devs are fixing bugs |
QA mid-Nightly Signoff | [MISSED] | Missed (test build and feature document were not avialable to QE) |
QA pre-Beta Signoff | [DONE] | YELLOW (conditional sign-off) sent on 7/31/2017 |
QA Beta - Full Testing | IN PROGRESS | |
QA pre-Release Signoff | IN PROGRESS | Email to be sent |