QA/Sync/Test Plan

Overview

Outline Quality Assurance plan to adapt to structured development cycles for Sync client and Sync Servers. Both parts of Sync have moved to a weekly cycle of fixes and pushes. This plan shows how QA will handle those weekly trains.

Client

Weekly client pushes of s-c to m-c

Sync client fixes are made on the services-central branch. Once per week, on Monday, those fixes will be handed off to QA for testing. Unless specifically required by dev/ops, testing of client fixes should occur against the stable production servers.

Testing strategy

• Verify fixes per bug STR's
• Run the set of manual smoketests plus cross platforms to ensure basic Sync functionality has not regressed due to the new fixes.
• Against the Wednesday m-c (Nightly) build, re-verify fixes and run smoketests and Basic Functional Test groups related to current fixes to ensure no merge issues regressed Sync on Nightly channel

Schedule

• Builds to QA: Monday
• QA sign-off: EOD Tuesday
• s-c push to m-c: Tuesday EOD
• QA sign-off of m-c: Wednesday EOD

Channel progression

Being a built into Firefox service, Sync will follow along with FxN

Schedule

The schedule follows the FxN schedule through the channel progression of releases.

Testing strategy

• Nightly - See the s-c to m-c push plan above.
• Aurora - Run Sync Basic Functional Tests one week prior the channel merge.
  • note: handle specific direct landing to Aurora on a case by case basis.
• Beta - Run Sync smoketests post build.
• Release
  • Run Sync smoketests on release candidate builds.
  • Upgrade testing when available
  • Major Update testing when available
  • Locale testing - install and quick spot check a few Tier1 locales

Servers

Sync server fixes are made on the staging server(s). Once per week, on Wednesday, those fixes will be handed off to QA for testing. Testing for Sync server, registration server, PAKE server and Account Portal can be required for any or all of them depending on fixes going in that week. Note: per the release process docs, Crossweave and load testing will be performed by Server Ops as necessary. Testing approach to each piece is outlined below.

Sync server

• Staging server is https://stage-auth.services.mozilla.com/
• Verify fixes per bug STR's
• Run the set of manual smoketest plus cross platforms to ensure basic Sync functionality has not regressed due to the new fixes.

Registration Servers

• Staging server is https://stage-auth.services.mozilla.com/
• Verify fixes per bug STR's
• Run the set of manual smoketests plus cross platforms to ensure basic account registration functionality has not regressed due to the new fixes.

PAKE Server

• Update about:config (restart required) services.sync.jpake.serverURL to https://stage-setup.services.mozilla.com/
• Verify fixes per bug STR's
• Run manual easy setup smoketests plus across devices to ensure functionality has not regressed due to the new fixes.

Account Portal

• Stage Account Portal is https://stage-account.services.mozilla.com/
• Verify fixes per bug STR's
• Run the set of manual smoketest plus to ensure account management functionality has not regressed due to the new fixes.

Schedule

• Builds to QA: Wednesday
• QA sign-off: EOD Friday
• Push from stage to production will happen during the maintenance window on Mondays, 3-5pm Pacific time. (QA to be on hand for live spot checks to help verify the push was a success)

Load Testing

• Working with ops team to improve grinder tests to better mimic real world conditions
• Detailed testing strategies and processes can be found here:Grinder Tests