QA/Test Automation/2011-06-01

previous meeting | Meetings | next meeting »

Dial in

 # 650-903-0800 or 650-215-1282 x92 Conf# 315 (US/INTL)
 # 1-800-707-2533 (pin 369) Conf# 315 (US)
 # irc.mozilla.org #mozmill for backchannel

Attendees

• Anthony, Henrik, Aaron, Cameron, Owen, Al, David

Last action items

• Geo: Work with Al and Anthony to get the on-demand update testing running on qa-set (April 27th)
• [DONE] Henrik: Talk to David for a meeting about handling patch reviews with Softvision (include Web Automation)
  • It will not happen before Henrik comes back from vacation
• [DONE] Henrik: Create automation scripts for handing new test-run called "remote" (May 19th)
• [DONE] Anthony: Get in contact with Vlad by end of today to kick of Discovery Pane tests (May 18th)
• Dave: Please reference last meeting notes from endurance meeting if you can't attend
• Al: Collect possible use cases and goals on-demand provisioning
• Al: Create remaining invalid certificates (May 20th)

Project Status

Redesign Shared Modules (Geo/Henrik)

• No further API updates since the last meeting
• Geo and Owen are working on porting browser tests so we can start creating the ui map soon
• Henrik is working on the documentation for window handling

On-demand release tests (Geo/Henrik)

• Anthony has discovered a "Connection Refused" issue with the daemon yesterday during release testing which prevents us from relaunching the daemon. We had to fallback to the old way of running update tests

Endurance Tests (Dave/Anthony)

• Fixed the failing Panorama test
• Stats are now precalculated in the automation script
• Test results now displayed on the dashboard
• Support for Firefox 6 is in review, dashboard patch will require updating
• We're back to not restarting by default due to the outstanding issue with disconnects

Others

Mozmill Crowd

• Patch for further launch points (add-on bar, Firefox button) has been landed
• Working on the documentation and the version checker for test environments

Dashboard

• Pushed some updates to the dashboard to make the display of failures more visually

Dashboard 0.4 (Owen)

• User Stories Finalized at: http://etherpad.mozilla.com:9000/mozmill-dashboard-elasticsearch
• Need to finalize document structure and then will begin web app development shortly.

Sync Server Load Testing (Owen)

• Developing Functional Grinder Scripts
• Creating a back end for generalized weave server intereaction

On Demand Provisioning (Cloud Automation) (Al/Henrik)

• Machine is set up in PHX Colo. We're waiting for VSphere access from Phong.
• Al will begin uploading and testing VMs as soon as access is available.

Tests / Shared Modules

General (Henrik)

• Check-ins since the last meeting

Broken Tests

• All broken functional tests have been fixed for branches down to 4.0
• Endurance test for Tab Switching in Panorama is still failing - we really have to ensure to get those kind of failures fixed faster
  • Update - this is now fixed.

Softvision (Anthony)

• Vlad and Alex working away on tests now that they are unblocked by Shared Modules
• Investigating unnecessary tests based on what is already covered by Selenium

TCM (Cameron/?)

• FC on .3 features
• Prioritized tasks for .4 features
• FC for .4 slated for July 15th
• I'm looking for a second on the project. Hopefully Matt Brandt
• Currently deployed on Oddsites. But I want to get a deployment at Mozilla on mozqa as soon as I can.

Shadow web server (Al/Henrik)

• Mozqa.com domain should be transferred to Mozilla ownership on 6/2 according to updates yesterday.
  • OV and DV certs will be issued when Mozilla owns domain and deployed to server.
  • Self-signed expired cert has been issued and will be deployed today.

Personal Status

For the personal status please check the weekly status updates:

• 110527 - 110527
• 110603 - 110603

Roundtable

General

• To ensure a fast progress please work on action items right after the meeting, or at least get the process triggered. Doing that 1 or 2 days before the next meeting will push us back by nearly 2 weeks. (Henrik)
• We have to get better in fixing broken tests. We can't leave broken ones failing for over two weeks! (Henrik)
  • Triage the results daily (Free for all, Assigned "sheriff", or Component "owners") and file bugs for failures not already on file. Initially assign to component "owner", if they can't fix in 24 hours, put call out to qa-auto@ and waverley@ to pick up. Should give us at least a week turn around time factoring in the review process (ashughes)

Meeting Notes

• Review Process
  • WebQA Review Process that we can use to start

• Tests for the Discovery Pane
  • Currently there are 13 selenium tests for discovery pane
  • In Mozmill tests we should only test what cannot be done with Selenium
    • Download and installation of Add-ons including behavioral changes of sections
    • General interaction with the browser
  • What process do we need to become faster
    • Before starting to write a test get in contact with David or Krupa and figure out which parts of a Litmus test do not need coverage and can be skipped

Action items

• Geo: Send pivotaltracker project link for API refactoring project to team members or add it to the project wiki
• David: Work with Geo and Anthony to get some first ideas for changes of the review process; we will schedule a meeting once Henrik is back (should happen in June)
• Anthony: Touch base with Geo to fix on-demand daemon issue
• Anthony: Review patch from Henrik tonight for remote tests
• Anthony: Move SSL tests over to mozqa.com once Al put them up on mozqa.com
• Anthony: Send email today to Dave Burns, Krupa, Vlad, Alex, Henrik to identify unnecessary tests inside a Litmus test which we have already started to automate. We need this information ASAP. Also make it clear how to handle new tests.
• [DONE] Henrik: Create necessary dummy patches for remote test-run
• [DONE] Henrik: Send Cameron the contacts from LibreOffice