QA/IAM/TestPlan

Overview

• This test plan covers the general weekly testing that will happen against 'Identity and Access Management' product. The goal is to ensure a defined and consistent amount of quality and usability in the server and client side.

Background

• Mozilla Persona is a cross-browser login system for the Web, working on all major browsers
• Because the metrics shown that usage of persona.org is low and has not grown over the last two years, persona will no longer actively developed by Mozilla.
• The operational and security support of the persona.org services is provided until November 30th, 2016, when Mozilla will shut down the persona.org services (Persona.org and related domains will be taken offline)
• For more information, see this guide to migrating your site away from Persona: https://wiki.mozilla.org/Identity/Persona_Shutdown_Guidelines_for_Reliers

Objective

• Identity and Access Management work provides an identified replacement for persona which outlines a future integration of LDAP with mozillians.org
• Auth0 will replace persona, which offers single sign-on services, abstracting various login and identity services into a single API including public APIs like Facebook Connect and public or private instances of Active Directory and LDAP.

Strategy

• Create a consistent and repeatable set of tests for qualifying and releasing the builds
• The focus needs to be on quality, but also on providing accurate feedback on successes and issues to the team, so that the Production releases provide a useful product for the Mozilla and development communities.
  • Exploratory testing on the client and server side
  • Create useful client-side and server-side test cases
  • Put accent on security and privacy issues that might come up across OS, browsers, accounts(emails and passwords)
  • Identifying and tracking issues in GitHub and Bugzilla.
  • Provide useful metrics

Scope of Testing

• Client-side testing will cover the following areas: basic functionality and UI, accounts and emails, interaction with the server, security and privacy, usability and compatibility across OS and browsers.
• Server-side testing will cover the following areas: basic functionality, support for multiple client sites, user security and privacy, information handling and storage, information persistence across deployments, and logging.

Functional testing

• Will cover:
  • full UI and functionality of the application
  • both basic and edge, positive and negative cases
  • connectivity (Offline behavior, functionality on cellular networks)
• Functional tests will be written in Test Rail, providing detailed test steps, expected and actual results for each test case
• Manual functional testing will be applied at two levels:
  • Component testing – covers a specific feature, and aims at validating its implementation and providing a list of issues affecting that specific component
  • End-to-End testing – aims at delivering a clear view on the overall quality of the product, and at providing a comprehensive list of issues affecting the application
• The full set of tests will also serve as a base to derive other test sets like: Smoke, Acceptance, and Regression

Re-testing

• Will be done as needed, in parallel with Regression Testing, by re-executing previously failed tests, to make sure fixes are properly verified and the Full test set is properly updated according to the latest fixes, to provide a more accurate insight on the overall quality of the product

Smoke testing

• Will cover only the most basic functionality
• Will be performed on intermediary builds, to confirm that the build is testable and stable

Exploratory testing

• Examples of exploratory testing cover: investigating a certain area suspected of being more prone to issues, looking for bugs at a very early stage, investigating freshly implemented features to provide quick feedback to developers and the rest of the team

Regression testing

• Will cover basic cases and areas that are most affected by issues
• Its purpose is to identify potential regression caused by bug fixes and/or new features

Acceptance testing

• Will cover basic and critical functionality of the application
• It is aimed to prove that Release Candidate builds quality is acceptable for release

Automated testing

• TBD

API testing

• Will be performed with the purpose of verifying the sanity of the server (issues with the API will be identified early on, thus allowing the team to be informed and take necessary action)

Non-Functional testing

• Non-functional testing will be done at different stages in the product lifecycle, and will evaluate product/feature non-functional attributes including (but not limited to):
  • Efficiency – Performance, Load, Stress
  • Portability

Target Readership

• This document is written for the following readership:
  • Since this is a public-facing product, we need to make sure that the Mozilla community can access, test, and develop around a solid product
  • Development team
  • Engineering group
  • Management

General Test Information

Links and Documentation

• Auth0 Implementation Roadmap:
  • https://docs.google.com/presentation/d/1Cd_UDM-ncdiVDGrwuBSOl0d8YhiYTXE1QuXifTsj4zA/edit#slide=id.g1655979439_0_0
• Tracking work:
  • https://tree.taiga.io/project/pierros-mozilla-particiaption-systems/kanban

Weekly Test Schedules

• Unknown yet

Weekly Meetings

• Participation Systems Standup: every Tuesday, Thursday from 5pm to 5:15pm in Pierros's Vydio
• Sprint Review / Retro / Planning: every Monday from 3pm to 5:30pm in Henrik's Vidyo

Email and IRC

• Post
• email List: parsys@mozilla.com
• Google Group: https://groups.google.com/a/mozilla.com/forum/#!forum/parsys
• IRC: #parsys
• Team:
  • Henrik Mitsch(:hmitsch)
  • Arielle - currently not on the team, will be back 01 JAN 2017
  • John Giannelos(:nemo-yiannis) - development on reps.mozilla.org, mozillians.org and supporting the infrastructure
  • Nikos Roussos(:nikos) - front-end
  • Pierros Papadeas(:pierros) - eng management for the team
  • Anastasios Katsoulas(:tasos) - web dev on mozillians
  • Yousef Alam(:yalam96) -new infrastructure + community websites
  • Teodora Vermesan(:TeoVermesan) - QA Engineer
  • Ioana Chiorean (:ioanachiorean) - Release QA Mobile Team Lead
  • Florin Mezei ((:florinmezei) - Project Manager (Release QA, WebQA, BuildDuty)

Bugs and Open Issues

• Bugzilla: mozillians.org & reps.mozilla.org
• Github: mozmoderator

Client and Server Test Environments

• development: http://mozillians-dev.allizom.org
• staging: http://mozillians.allizom.org
• production: http://mozillians.org

Supported OS and Browsers

• All information about supported platforms, operating systems, browsers, mobile devices will be kept in a Google doc spreadsheet

Requirements for test=

Environmental needs

• target mostly desktop PC / laptop
• mobile devices

Major Areas Focus

• Sign Up:
  • Buttons:
    • Verify the page has a 'SIGN UP' button
    • Verify the page has an 'Email' field
    • Verify the page has a 'Password' field
    • Verify the page has both 'Submit' and 'Cancel'(x) buttons
    • Verify the page has 'SIGN UP WITH other apps' option
  • Form fields:
    • Verify that clicking submit button after entering all the required fields, submits the data
    • Verify that clicking cancel button after entering all the required fields, cancels the submit request and resets all the fields
    • Verify that not filling the mandatory fields and clicking submit button will lead to validation error: "Can't be blank"
    • Verify that sign up with other apps works as expected
    • Verify that sign up with an already verified email will lead to an error message: "The user already exists"
    • Verify sign-up with:
      • valid email, invalid password
      • valid email, valid password
      • invalid email, invalid password
      • invalid email, valid password
  • Email Field:
    • Verify email strings
    • Verify all legal combinations of characters
    • Copy/Pasting emails from other sources
    • Auto-completion of emails
    • Verify minimum/maximum sizes of emails (length)
  • Password field:
    • Verify password strings
    • Verify all legal combinations of characters
    • Copy/Pasting passwords from other sources
    • Verify minimum/maximum sizes of passwords (length)
    • Verify that passwords are stored if "remember password" option is chosen
    • Verify that passwords are not stored if "never remember password" option is chosen
  • Email notification:
    • Email notification for new accounts: verification email through email provider
    • Check functionality when the user can not verify by email (email provider is down or user can not access email account for some reason)
    • Check functionality when the user does not verify by email (skips, forgets)
• Login
  • Verify that 'email field, 'password' field, "Not your account?" link, "Don't remember your password?" link are present
  • Verify that if the user was already logged in with an account he can changed the account using the "Not your account" option or login with the previous one
  • Login with:
    • valid email, valid password
    • valid email, invalid password
    • invalid email ,invalid password
    • valid email and password
    • with other apps
    • simultaneously in two different browsers with the same account
    • with different emails in the same browser/different browser
    • an email if he did not confirm the used email
  • Verify that the log in is kept when restoring a session after a browser crash
  • Verify that a message gets displayed in case user leaves email or password field as blank
  • Verify that a message is displayed in case user exceeds the character limit of the user name and password fields
  • Verify that the password is in encrypted form when entered
  • Verify that there is limit on the total number of unsuccessful attempts
  • Verify that in case of incorrect credentials a message is displayed "incorrect username or password"
  • Verify if the password can be copy-pasted or not
  • Verify that once logged in, clicking back button doesn't logout user
• Logout
  • Verify application allows single sign off from all the devices.
  • Verify application let’s you sign off for multiple accounts.
  • Verify if application takes more time for logout at different connection speeds
  • Verify the logout page redirects to the page where it allows login or homepage
  • Verify the logout button or link works on all devices
• Network: Verify the impact of various ways to access the internet
  • Ethernet - personal vs. office, with and without VPN
  • Public WiFi
  • Private WiFi and other home setups
  • 3g, 4g
• Other:
  • Login to the application with multiple accounts at the same time
  • Check if everything works as expected in different browsers
  • Page crash should not reveal application or server info. Error page should be displayed for this
  • Error messages should not reveal any sensitive information

Entry criteria

• Testing will begin when the following criteria are met:
  • Development team delivers a feature complete build to the QA
  • The application is successfully deployed on the appropriate environment (including integrated components, depending on environment scope)
  • Test cases, estimations and schedule has been reviewed and approved

Exit criteria

• Testing will complete when:
  • All test cases have been executed
  • All non-closed defects are reviewed by the team and marked as follows for subsequent releases:Fixed, Regressed, Closed, Deferred
  • Summary test report is published