QA/IAM/TestPlan

Overview

• This test plan covers the general weekly testing that will happen against 'Identity and Access Management' product. The goal is to ensure a defined and consistent amount of quality and usability in the server and client side.

Background

• Mozilla Persona is a cross-browser login system for the Web, working on all major browsers
• Because the metrics shown that usage of persona.org is low and has not grown over the last two years, persona will no longer actively developed by Mozilla.
• The operational and security support of the persona.org services is provided until November 30th, 2016, when Mozilla will shut down the persona.org services (Persona.org and related domains will be taken offline)
• For more information: https://wiki.mozilla.org/Identity/Persona_Shutdown_Guidelines_for_Reliers

Objective

• Identity and Access Management work provides an identified replacement for persona which outlines a future integration of LDAP with mozillians.org

Strategy

• Create a consistent and repeatable set of tests for qualifying and releasing the builds
• The focus needs to be on quality, but also on providing accurate feedback on successes and issues to the team, so that the Production releases provide a useful product for the Mozilla and development communities.
  • Exploratory testing on the client and server side
  • Create useful client-side and server-side test cases
  • Put accent on security and privacy issues that might come up across OS, browsers, accounts(emails and passwords)
  • Identifying and tracking issues in GitHub and Bugzilla.
  • Provide useful metrics

Scope of Testing

• Client-side testing will cover the following areas: basic functionality and UI, accounts and emails, interaction with the server, security and privacy, usability and compatibility across OS and browsers.
• Server-side testing will cover the following areas: basic functionality, support for multiple client sites, user security and privacy, information handling and storage, information persistence across deployments, and logging.

Target Readership

• This document is written for the following readership:
  • Since this is a public-facing product, we need to make sure that the Mozilla community can access, test, and develop around a solid product
  • Development team
  • Engineering group
  • Management

General Test Information

Links and Documentation

/edit#slide=id.g1655979439_0_0

• Tracking work:
  • https://tree.taiga.io/project/pierros-mozilla-particiaption-systems/kanban

Weekly Test Schedules

• Unknown yet

Weekly Meetings

• Participation Systems Standup: every Tuesday, Thursday from 5pm to 5:15pm in Pierros's Vydio
• Sprint Review / Retro / Planning: every Monday from 3pm to 5:30pm in Henrik's Vidyo

Email and IRC

• Post to:
  • email List: parsys@mozilla.com
  • Google Group: https://groups.google.com/a/mozilla.com/forum/#!forum/parsys
  • IRC: #parsys
• Team:
  • Henrik Mitsch(:hmitsch) - owns Community Analytics and Volunteer Management Systems
  • Arielle - currently not on the team, will be back 01 JAN 2017
  • John Giannelos(:nemo-yiannis) - development on reps.mozilla.org, mozillians.org and owns infrastructure fortification
  • Nikos Roussos(:nikos) - front-end
  • Pierros Papadeas(:pierros) - eng management for the team and owns the IAM packages B and C
  • Anastasios Katsoulas(:tasos) - web dev on mozillians and owns technical debt reduction
  • Yousef Alam(:yalam96) - owns Participation Infrastructure
  • Teodora Vermesan(:TeoVermesan) - QA Engineer
    • Ioana Chiorean (:ioanachiorean) - Release QA Mobile Team Lead
    • Florin Mezei ((:florinmezei) - Project Manager (Release QA, WebQA, BuildDuty)

Bugs and Open Issues

• Bugzilla: mozillians.org & reps.mozilla.org
• Github: mozmoderator

Client and Server Test Environments

• development: http://mozillians-dev.allizom.org
• staging: http://mozillians.allizom.org
• production: http://mozillians.org

Supported OS and Browsers

• All information about supported platforms, operating systems, browsers, mobile devices will be kept in a Google doc spreadsheet

Requirements for test

Environmental needs

• target mostly desktop PC / laptop
• different mobile devices

Desktop

• Verify compatibility/stability/functionality across different OS and browsers
  • smoke/sanity/acceptance section
  • Basic Functional and UI tests
  • Click, zoom, scroll, multi-window
  • Compatibility - review actions and processes that are unique to mobile devices

Mobile

• Berify compatibility/stability/functionality across Android and iOS tablets and phones
  • smoke/sanity/acceptance section on phone/tablet
  • Basic Functional and UI tests on phone/tablet
  • Panning, zooming, scrolling, tab access and flow between tabs
  • Compatibility - review actions and processes that are unique to mobile devices

Network/Internet

• Verify the impact of various ways to access the internet: timing, communication, messaging, outages, errors, traffic
  • Ethernet - personal vs. office, with and without VPN
  • Public WiFi
  • Private WiFi and other home setups
  • 3g, 4g

Major Areas Focus

Functional testing

• Will cover:
  • full UI and functionality of the application
  • both basic and edge, positive and negative cases
• Functional tests will be written in Test Rail, providing detailed test steps, expected and actual results for each test case
• Manual functional testing will be applied at two levels:
  • Component testing – covers a specific feature and aims at validating its implementation and providing a list of issues affecting that specific component
  • End-to-End testing – aims at delivering a clear view on the overall quality of the product and at providing a comprehensive list of issues affecting the application
• The full set of tests will also serve as a base to derive other test sets like: Smoke, Acceptance and Regression

UI

• Sign Up:
  • 'SIGN UP' button
  • 'Email' field
  • 'Password' field
  • 'Submit' and 'Cancel'(x) buttons
  • 'SIGN UP WITH other apps' option
• Confirm your Email UI
  • verifying accurate "prove" link
  • confirm email verification from client-side and server-side
• Login:
  • Verify that 'email field, 'password' field, "Not your account?" link, "Don't remember your password?" link are present
  • 'Login with other apps' option is present
• Logout
  • 'Log out' button

Full Functional

• Sign Up:
  • Verify that clicking submit button after entering all the required fields, submits the data
  • Verify that clicking cancel button after entering all the required fields, cancels the submit request and resets all the fields
  • Verify that not filling the mandatory fields and clicking submit button will lead to validation error: "Can't be blank"
  • Verify that sign up with other apps works as expected
  • Verify that sign up with an already verified email will lead to an error message: "The user already exists"
  • Verify sign-up with:
    • valid email, invalid password
    • valid email, valid password
    • invalid email, invalid password
    • invalid email, valid password
  • Verify email strings and all legal combinations of characters
  • Copy/Pasting emails from other sources
  • Auto-completion of emails
  • Verify minimum/maximum sizes of emails (length)
  • Verify password strings and all legal combinations of characters
  • Copy/Pasting passwords from other sources
  • Verify minimum/maximum sizes of passwords (length)
  • Verify that passwords are stored if "remember password" option is chosen
  • Verify that passwords are not stored if "never remember password" option is chosen
  • Verify browser restart after creation of account or access of an account
• Login
  • Verify that if the user was already logged in with an account he can changed the account using the "Not your account" option or login with the previous one
  • Login with:
    • valid email, valid password
    • valid email, invalid password
    • invalid email ,invalid password
    • valid email and password
    • with other apps
    • simultaneously in two different browsers with the same account
    • with different emails in the same browser/different browser
    • an email if he did not confirm the used email
  • Verify that the log in is kept when restoring a session after a browser crash
  • Verify that a message gets displayed in case user leaves email or password field as blank
  • Verify that a message is displayed in case user exceeds the character limit of the user name and password fields
  • Verify that the password is in encrypted form when entered
  • Verify that there is limit on the total number of unsuccessful attempts
  • Verify that in case of incorrect credentials a message is displayed "incorrect username or password"
  • Verify if the password can be copy-pasted or not
  • Verify that once logged in, clicking back button doesn't logout user
    • Verify email strings and all legal combinations of characters
    • Copy/Pasting emails from other sources
    • Auto-completion of emails
    • Verify minimum/maximum sizes of emails (length)
    • Verify password strings and ll legal combinations of characters
    • Copy/Pasting passwords from other sources
    • Verify minimum/maximum sizes of passwords (length)
    • Verify that passwords are stored if "remember password" option is chosen
    • Verify that passwords are not stored if "never remember password" option is chosen
• Logout
  • Verify application allows single sign off from all the devices.
  • Verify application let’s you sign off for multiple accounts.
  • Verify if application takes more time for logout at different connection speeds
  • Verify the logout page redirects to the page where it allows login or homepage
  • Verify the logout button or link works on all devices
• Email notification:
  • • Email notification for new accounts: verification email through email provider
    • Check functionality when the user can not verify by email (email provider is down or user can not access email account for some reason)
    • Check functionality when the user does not verify by email (skips, forgets)
• Other:
  • Login to the application with multiple accounts at the same time
  • Check if everything works as expected in different browsers
  • Page crash should not reveal application or server info. Error page should be displayed for this
  • Error messages should not reveal any sensitive information

Re-testing

• Will be done as needed, in parallel with Regression Testing, by re-executing previously failed tests, to make sure fixes are properly verified and the full test set is properly updated according to the latest fixes, to provide a more accurate insight on the overall quality of the product

Smoke testing

• Will cover only the most basic functionality
• Will be performed on intermediary builds, to confirm that the build is testable and stable

Exploratory testing

• Examples of exploratory testing cover: investigating a certain area suspected of being more prone to issues, looking for bugs at a very early stage, investigating freshly implemented features to provide quick feedback to developers and the rest of the team

Regression testing

• Will cover basic cases and areas that are most affected by issues
• Its purpose is to identify potential regression caused by bug fixes and/or new features

Acceptance testing

• Will cover basic and critical functionality of the application
• It is aimed to prove that Release Candidate builds quality is acceptable for release

Automated testing

• TBD

API testing

• Will be performed with the purpose of verifying the sanity of the server (issues with the API will be identified early on, thus allowing the team to be informed and take necessary action)

Non-Functional testing

• Non-functional testing will be done at different stages in the product lifecycle, and will evaluate product/feature non-functional attributes including (but not limited to):
  • Efficiency – Performance, Load, Stress

Localization

• Verify the UI flow with specific focus on localization, all text-based content
• Quick verification of localized headers

Entry criteria

• Testing will begin when the following criteria are met:
  • Development team delivers a feature complete build to the QA
  • The application is successfully deployed on the appropriate environment (including integrated components, depending on environment scope)
  • Test cases, estimations and schedule has been reviewed and approved

Exit criteria

• Testing will complete when:
  • All test cases should be executed
  • All blockers, criticals must be fixed and verified or have an agreed-upon timeline for being fixed