QA/Mozmill Test Automation: Difference between revisions
Line 133: | Line 133: | ||
|[[QA/Mozmill_Test_Automation/Triggered_Update_Testing|Triggered Update Testing]] | |[[QA/Mozmill_Test_Automation/Triggered_Update_Testing|Triggered Update Testing]] | ||
|Perform nightly update testing triggered by landing of the build | |Perform nightly update testing triggered by landing of the build | ||
|- valign="top" | |||
|[[QA/Mozmill_Test_Automation/Triggered_Update_Testing|On-Demand Update Testing]] | |||
|Streamline how update testing is performed on releases | |||
|} | |} | ||
Revision as of 19:35, 26 January 2011
Overview
Mozmill is a relatively new test tool to run functional tests in an automated manner. It can be used by applications which are based on the Gecko platform or XulRunner.
Mozilla QA is using Mozmill to automate existing manual tests which were added to Litmus and folded into different testgroups, like Smoketests, Basic Functional Tests (BFT), and Full Funcional Tests (FFT). The primary goal is to lower the active testing time for any of our QA engineers and contributors. As a result we will be able to cover more than only one platform and locale for BFT testing. To get a better impression of the enhancements just compare those raw numbers: To run all the ~250 BFT tests manually it takes one man day while running all the BFT tests on every platform in parallel will eventually take about 40 minutes with Mozmill!
Team Details
Given the number of new features and regression fixes for any new Firefox major or security release, we will continuously enhance our functional testing framework by adding more tests of any type, increasing the number of platforms and locale builds to run the tests on, and by making the analysis of test reports more user friendly.
Team Members and Assignments
Name | Contact | Availability | Project Assignments |
Henrik Skupin | hskupin@mozilla.com | Employee | Team Lead (Details see Areas of Work) |
Anthony Hughes | ahughes@mozilla.com | Employee | Fixing failures and writing new tests |
Al Billings | abillings@mozilla.com | Employee aka Team Punching Bag | Writing new tests |
Geo Mealer | gmealer@mozilla.com | Employee (FNG) | Ramping up on MozMill testing, updating Shared APIs |
Dave Hunt | dhunt@mozilla.com | Employee | |
Aaron Train | atrain@mozilla.com | Contractor | Fixing test failures, writing new tests, testing frameworks |
You can also get in contact with us by subscribing to the mozmill-dev list on Google or by joining the #mozmill channel on IRC.
Calendar
You can subscribe to the Google calendar (ical) to see our events, milestones, and other deliverable dates.
Meetings, Work Weeks, and Dates
We have bi-monthly public meetings about ongoing and upcoming projects. If you want to join the meetings, please check the meeting page for details. The same applies for work weeks which will be held once or twice within a year.
Roadmap
While we wanna see as much as possible tests from all the Litmus testgroups automated, we do not have the needed man power to call it done in the next weeks or months. To get an idea how long it will take to reach the final goal some milestones have to be set for each of the upcoming quarters.
Lets summarize some facts about existing Litmus tests:
- Around 250 tests in the BFT testgroup
- Around 710 remaining tests in the FFT testgroup
- Other testgroups with important tests, e.g. Software Updates
Beside the test creation work we will also focus other areas of work. In general the following goals have been set for the next quarters:
10/Q4
- Release the Mozmill crowd-source extension to support localizers and add-on authors for Firefox 4.
- Implement Mozmill Dashboard views for software update and add-on tests
- Integrate local data-based Firefox 4 mozmill tests under Buildbot execution control and reporting
- Refactoring the shared module system based on the specification from Q3
- Collaborate with development team to expand Mozmill Panorama support to include an automated test suite for functional and performance testing of the Firefox 4 Panorama feature.
- Enlarge the amount of tests with local test data (Phase II)
Areas of Work
As you can imagine a project like Mozmill automation has a couple of areas where work is happening or at least planned for the future. The following chapter will give you a good overview about all of our projects.
Mozmill Tests
Project | Description |
Broken Tests | Making sure to always have a green test-run for Mozmill tests. |
Mozmill Tests | Create automated tests to minimize our manual testing efforts. |
Shared Modules | Simplify Mozmill test scripting by using shared modules. |
Shared Modules Refactor | Exploring new ways to present the Shared API to simplify test scripts even further. |
Software Update Tests | Create Mozmill tests to ensure lesser problems during an upgrade of Firefox. |
Accessibility Tests | Create Mozmill tests to ensure Firefox is accessible by everyone. |
L10n Tests | Create Mozmill tests to ensure that localized builds are not broken. |
Add-ons Tests | Create Mozmill tests to run functional tests against add-ons. |
Panorama Testing | Create a high-quality suite of performance and functional tests to support the Panorama feature of Firefox 4 |
Test Modules Refactor | Work in parallel with the Shared Modules Refactor project to simplify the test modules. |
Endurance Tests | Create Mozmill tests to ensure Firefox endurance and detection of behaviours under sustained use. |
Automated Testing
Project | Description |
Crowd Testing | Make it totally easy for everyone to run Mozmill tests from within Firefox. |
Automated Test-runs | Create scripts for a collection of automated test-runs. |
Triggered Update Testing | Perform nightly update testing triggered by landing of the build |
On-Demand Update Testing | Streamline how update testing is performed on releases |
Reporting
Project | Leads | Description |
Web Dashboard | Henrik Skupin Anthony Hughes |
Dashboard to visualize collected Mozmill test results from the automated test-runs and the Mozmill Crowd extension |
Documentation
Project | Description |
Shared Modules | Documentation on how to use shared modules and their feature set. |
Finished Projects
Project | Description |
Result Data | Collecting necessary result data of test-runs for each test type. |
Contribution
Each of the above mentioned sub-projects needs love from someone with passion to work on such a challenging task. There will be a project lead who will offer help in various ways.
If you are interested in helping us to push Mozmill a big step forward, please get in contact with Henrik Skupin. We will figure out what's the best way for your contribution.
First Steps
Joining an existing project can be hard especially when you see all the stuff in the mind map. To give you a handy guide for the first steps a rich documentation has been created. It covers the following areas:
- Installation of Mozmill
- Getting in touch with the mozmill-test repository
- Executing existent Mozmill tests
All those documents should bring you into a state to be able to play around with Mozmill. At least by running those tests against nightly or release builds of Firefox and reporting failed tests would be a great help for us.
Resources
There are a couple of resources which are related to Mozmill. A nearly complete list you will find below:
- Overview about the Mozmill project on QMO
- Complete documentation for Mozmill
- Documentation for Shared Modules
- How to use Mozmill and create tests for Firefox