Auto-tools/Projects/B2G Automation

This page is mostly obsolete. FirefoxOS Automation is now handled by the FirefoxOS Automation Team. See https://github.com/mozilla-b2g/automation-projects or irc.mozilla.org#fxos-automation

Etherpad

https://etherpad.mozilla.org/b2g-automation-meeting-notes

Goal 2013 Q2

Milestone 1 - 2 weeks (April 1 - 12)

• [jgriffin] [MISSED] Get Gaia Unit tests running in TBPL (blocked by bug 855049)
• [jgriffin] [DONE] Triage all existing B2G failures and identify a concrete set of problems that need to be addressed

1. test_click errors bug 838607 - dburns looking at it.
2. reftest/crashtest crash bug 818103 - assigned to mike habicher.
3. [DONE] exception in window_manager.js bug 851666 - WORKSFORME
4. [DONE] Timeout for WebAPI tests is too small; emulator callbacks can occasionally take 10-20s, leading to random timeouts. Rob will fix.
5. socket.timeout errors caused by actorPool error bug 859417
6. socket.timeout errors that occur at startup in newSession, before a new Gecko is copied - see logs in bug 829551
7. chrome tests occasionally can't find elements (timing or page loading error?) bug 859423
8. B2G crashes during WebAPI tests bug 859439 - will be easier to handle once crash detection is working here

• [jgriffin] [MISSED] Make mach runners for B2G mochitests and reftests
• [ahal] [DONE] Enable crash detection on mozilla-b2g18 for mochitest/reftest
• [ahal] [DONE] Add crash detection to Marionette tests
• [ahal] [MISSED] Write a script to download the latest emulator
• [mdas] [DONE] Use weakref in marionette-elements (bug 736592)
• [mdas] [ON TRACK] Work on getting Gaia UI tests running on b2g desktop builds
• [yiming] [DONE] Complete touch/mouse event code for action chains
• [yiming] [MISSED] Convert code from synthetic gestures to action chains
• [rwood] [DONE] Develop bluetooth Gaia UI test to pair unagi with host machine
• [rwood] [DONE] Finish integrating long running tests with datazilla

Milestone 2 - 2 weeks (April 22 - May 3)

• [jgriffin] [DONE] Get Gaia Unit tests running in TBPL (scheduled on cedar)
• [jgriffin] [DONE] Make mach runners for Marionette
• [jgriffin] [MISSED] Add mach to B2G so we can build B2G mach targets
• [jgriffin] [MISSED] Improve Marionette's error reporting with specific, clear error messages and stack traces
• [jgriffin] [MISSED] Integrate JS client Pt 1: Get a simple JS Marionette test working locally with the JS Marionette client and the Python testrunner
• [jgriffin] [MISSED] Get mozilla-central b2g desktop builds working with Marionette
• [ahal] [DONE] Write a B2G bootstrap script to make downloading and running tests easier
• [ahal] [MISSED] Use in-tree mozbase instead of puppetagain packages for B2G tests in buildbot
• [mdas] [MISSED] Work on refactoring touch events and getting the mapping between action chains and mouse events working.
• [mdas] [MISSED] Finish converting synthetic gestures to action chains
• [rwood] [DONE] Get gaia-ui endurance tests into gaia-ui github repo (working with dhunt on improvements as per his review) was pulled out of the repo, new location tbd
• [rwood] [ON TRACK] Improve gaia-ui endurance test stability
• [rwood] [DONE] Automate entire process via Jenkins

Milestone 3 - 2 weeks (May 13 - May 24)

• [ON TRACK] Work on top B2G test stability problems and either fix or get them to a state where developers can take over

1. [DONE] test_click errors bug 838607 - dburns looking at it.
2. [DONE] reftest/crashtest crash bug 818103 - assigned to mike habicher.
3. [DROPPED] socket.timeout errors caused by actorPool error bug 859417 - dropped due to marionette refactor
4. [MISSED] socket.timeout errors that occur at startup in newSession, before a new Gecko is copied - see logs in bug 829551

• [mdas] [DEFERRED] Finish converting synthetic gestures to action chains - may not be necessary.
• (carry over) [mdas] [DONE] On on refactoring touch events and getting the mapping between action chains and mouse events working.
• [mdas] [DONE] Develop manifest of stable gaia-ui-tests on b2g desktop builds
• [mdas] [ON TRACK] Work on improving Marionette docs
• [mdas] [ON TRACK] Fix tap/touch related bugs that are blocking webQA
• (carry over) [jgriffin] [DONE] Get mozilla-central b2g desktop builds working with Marionette
• [jgriffin] [DONE] Add manifest support to gaia-unit-tests
• [jgriffin] [DONE] write mozharness script for gaia-ui-tests on b2g desktop builds
• [DROPPED] Add debugger support to B2G testrunners
• (carry over) [ahal] [MISSED] Use in-tree mozbase instead of puppetagain packages for B2G tests in buildbot
• [wlach] [MISSED] Get rid of debugger server dependency bug 797529

Milestone 4 - 2 weeks (June 3 - June 14)

• (carry over) [wlach] [DONE] Get rid of debugger server dependency bug 797529
• [wlach] [ON TRACK] Integrate JS client Pt 1: Get a simple JS Marionette test working locally with the JS Marionette client and the Python testrunner
• (ongoing) [mdas] [MISSED] Resolve issues with tap
• (ongoing) [mdas] [ON TRACK] Update marionette docs
• (carry over) [ahal] [DONE] Get mozilla-central B2G unit tests to use in-tree mozbase instead of puppetagain
• [jgriffin] [MISSED] Disable gaia-unit-tests that are still failing on cedar
• [jgriffin + releng] [MISSED] Get gaia-unit-tests scheduled on all branches (depends on gecko-gaia coupling)
• [jgriffin] [MISSED] Update mozharness script for gaia-ui-tests
• [jgriffin + releng] [MISSED] Get gaia-ui-tests on b2g desktop builds scheduled on cedar (depends on gecko-gaia coupling, gaia-ui-tests to gaia coupling, and gaia-ui-tests branch renaming)
• [jgriffin] [MISSED] Identify current B2G top oranges
• [rwood] [DONE] Tryrel-eng m-c unagi-eng on gaia-ui-tests
• [rwood] [DONE] Develop some heuristics to help determine when gaia-ui-test Jenkins failures are "real" or reproducible enough to warrant sending out an e-mail notification to engineering managers

Milestone 5 - 2 weeks (June 24 - July 5)

• [wlach] [DROPPED] Add JS client Pt 1: Get a simple JS Marionette test working locally with the JS Marionette client and the Python testrunner
• (ongoing) [mdas] [ON TRACK] Resolve blocking issues with tap
• [mdas] [DEFER] Resolve blocking scroll issues - blocked on is_displayed fix
• (ongoing) [mdas] [DONE] Add some mechanisms to make debugging touch related problems easier
• [jgriffin] [ON TRACK] Disable gaia-unit-tests that are still failing on cedar
• [jgriffin + releng] [MISSED] Get gaia-unit-tests scheduled on all branches (depends on gecko-gaia coupling)
• [jgriffin] [DONE] Update mozharness script for gaia-ui-tests
• [jgriffin + releng] [DONE] Get gaia-ui-tests on b2g desktop builds scheduled on cedar (depends on gecko-gaia coupling, gaia-ui-tests to gaia coupling, and gaia-ui-tests branch renaming)
• [jgriffin] [DEFER] Identify current B2G top oranges
• [ahal] [DONE] Identify and fix issues with full-stack emulator builds
• [rwood] [DONE] Fix intermittent gaia-ui endurance test failures
• [rwood] [DONE] Expand gaia-ui endurance test coverage
• [rwood] [DONE] Write script to email/flag b2g memory spikes via gaia-ui endurance tests

Goal 2013 Q3

Milestone 6 - 2 weeks (July 8 - July 19)

• (carry over) [jgriffin + releng] [DONE] Get gaia-unit-tests scheduled on all branches
• [jgriffin] [MISSED] Implement script to compare two sources.xml
• [jgriffin] [DONE] Identify source of memory leak in LG automated tests bug 897684
• [mdas, davehunt, webqa] [DONE] Get gaia-ui-tests on cedar running stable green
• [mdas] [MISSED] Implement modal dialog support in Marionette
• (carry over) [mdas] [MISSED] Fix remaining blocking touch issues
• [wlach] [DONE] Implement get_active_frame command - bug 855327
• [rwood] [MISSED] Get games benchmarking tests running on Firefox OS
• [rwood] [ON TRACK] Investigate/fix intermittent WebAPI emulator failures
• [rwood] [ON TRACK] Maintain endurance tests as needed; expand checks for b2g memory changes over time
• [rwood] [DONE] Get endurance tests running on Leo on Master
• [rwood] [DEFER] Look into data requirements for gaia-ui heuristics

Milestone 7 - 2 weeks (July 29 - August 9)

• ongoing - patches in hand - [ahal] Add mach to B2G so we can build B2G mach targets
• scheduled on cedar - [mihnea] Get JSreftests working for B2G
• [jgriffin] [DONE] Implement script to compare two sources.xml
• [DONE] final patch up - (carry over) [mdas] Implement modal dialog support in Marionette
• current focus -> (carry over) [mdas] Fix remaining blocking touch issues
• (carry over) [rwood] Get games benchmarking tests running on Firefox OS
• (ongoing) [rwood] [ON TRACK] Investigate/fix intermittent WebAPI emulator failures
• (ongoing) [rwood] Maintain endurance tests as needed (Leo on v1-train); expand checks for b2g memory changes over time
• currently working with jlebar on this -> [rwood] Validate emulator as platform for AreWeSlimYet (AWSY) memory tests
• [rwood] Create app-launch and page-browser tsets for AWSY
• (carry over) [rwood] Look into data requirements for gaia-ui heuristics

Milestone 8 - 2 weeks (August 19 - August 30)

• (carry over) [ahal] Add mach to B2G so we can build B2G mach targets
• [DONE] scheduled on cedar - [mihnea] Get JSreftests working for B2G
• P1! [jgriffin + all] Update all testing docs to make it easy for developers to get started with tests
• [mdas] Investigate modal dialog test failure
• (carry over) [rwood] Get games benchmarking tests running on Firefox OS
• (ongoing) [rwood] [ON TRACK] Investigate/fix intermittent WebAPI emulator failures
• (ongoing) [rwood] Maintain endurance tests as needed (Leo on v1-train); expand checks for b2g memory changes over time
• currently working with jlebar on this -> [rwood] Validate emulator as platform for AreWeSlimYet (AWSY) memory tests
• [rwood] Create app-launch and page-browser tsets for AWSY

Sept 19 - Oct 2

• (jgriffin PTO)
• (carry over) [DONE] [mdas] land the modal dialog fix
• [mdas] [MISSED] fix the import script leak (landed, but backed out)
• [mdas] [DEFER] write up and add some marionette/gaiatest endurance unit tests to catch space consumption/performance regressions
• [ahal] [DONE] write mozharness script for running mochitests on B2G desktop builds
• [wlach] [DONE] finish implementation of npm mirror for rel-eng (re-writing in node.js and documenting)
• [rwood] [ON TRACK] Endurance test fixes/maintenance for inari master/1.3
• [rwood] [ON TRACK] Further work on AWSY emulator test driver
• [rwood] [ON TRACK] Continued support for games benchmarking on Firefox OS

Goal 2013 Q4

Oct 10 - Oct 24

• [jgriffin] [ON TRACK] investigate problem with B2G unit test timeouts not being caught by mozprocess
• [jgriffin] [DONE] add B2G support to existing Marionette mach command
• [mdas] [DONE] investigate issues with the modal dialog fix
• (carry over) [mdas] [DONE] fix the import script leak (landed, but backed out)
• [mdas] [ON TRACK] write up and add some marionette/gaiatest endurance unit tests to catch space consumption/performance regressions
• [ahal] [ON TRACK] get mochitests on b2g desktop builds rolled out everywhere
• [ahal] [ON TRACK] get B2G reftests running OOP
• [wlach] document Eideticker in such a way that it's easy for partners and people new to the project to set it up from scratch
• [wlach] [DONE] instrument better app startup times for B2G bug 888103
• [wlach] [ON TRACK] do some fps comparisons with 120fps camera
• (ongoing) [wlach] attend FxOS_Perf standups and continue to provide feedback wrt to power management testing
• [rwood] Endurance test fixes/maintenance for inari master/1.3
• [rwood] Get AWSY emulator test driver running locally
• [rwood] [ON TRACK] Set up Jenkins cluster on new VMs for AWSY tests
• [rwood] [DONE] Continued support for games benchmarking on Firefox OS

Oct 24 - Nov 7

[jgriffin]

• [DONE] get gaia-integration tests in TBPL (depends on rel-eng)
  • not running green, but Gareth Aye owns this
• [MISSED] get green set of debug emulator mochitests running everywhere
  • in progress on cedar; see https://tbpl.mozilla.org/?tree=Cedar&jobname=emulator&showall=1; devs working on greening up problems also on https://tbpl.mozilla.org/?tree=Pine

[mdas]

• [ON TRACK] refactor runtests.py so we can use a mixin pattern bug 925398
  • (carry over) (time consuming) [ON TRACK] write up and add some marionette/gaiatest endurance unit tests to catch space consumption/performance regressions bug 931044 and bug 931045
• [DONE] Stop making marionette.log when unnecessary bug 918408

[ahal]

• [DONE] get mochitests on b2g desktop builds rolled out everywhere
• [DONE] address B2G crash reporting problems

[wlach]

• Get the B2G Eideticker dashboard running
  • running and uploading data to dave's people account (http://people.mozilla.org/~dhunt/eideticker-ci-dashboard/), but we need to configure to upload/send to eideticker.mozilla.org, which is waiting on moving the setup to the london office
  • bug filed for this, wlach will look up and post here after meeting
  • also need to file some follow-up bugs on test reproducibility (e.g. for contact loading we don't last long enough to display all pictures)
    • the contacts issue at least is ours to fix: pictures loading slowly and asynchrously is "by design" according to gaia devs. will file bug with davehunt to not load avatar pictures for eideticker
• (carry over) do some fps comparisons with 120fps camera
  • should have some results shortly

[rwood]

• (carry over) Get a full set of AWSY emulator test driver running locally
  • done except now blocked by Bug 931198 about_memory fails with > 30 memory files
• Assist the Taipei team with MTBF tests
  • Leaving tomorrow eve
• Deliver a set of memory test reports to the AWSY maintainer
• Get AWSY prodution VMs setup (1 master, others as nodes)
• Get gaia-ui endurance tests running on hamachi b2g26/1.2
  • Up and running but now need to cherry-pick endurance test changes that arent in b2g26
• Fix/investigate gaia-ui endurance test failures

[gkw]

• Spoke about marifuzz, a project to use marionette for fuzzing, based on orangfuzz but using marionette instead.
  • Emailed discussion outcome (drawings from TW office board) to jgriffin, mdas and wlach

Nov 7 - Nov 21

[jgriffin]

• [MISSED] Finish adding OOP support to WebAPI tests bug 926280
  • This is ready to land, but I'm going to wait to land it until Malini's mixin refactor lands (bug 925398), since this will bitrot that patch.
• (carry over) [MISSED] get green set of debug emulator mochitests running everywhere
  • These are mostly green on cedar, but intermittents still appear. I've filed a bug to schedule these on all trunk branches, but we'll likely have to hide these initially until the intermittents fall to an acceptable level. bug 940161

[ahal]

• get emulator reftests running oop
  • should also help getting b2g desktop reftests going oop
  • done and running on cedar (if the implementation is ok)
  • still some problems (crashes, timeouts, and an error that happens between tests but doens't seem to affect them)
• [MISSED] get reftests running on b2g desktop builds
• investigate emulator startup crashes?

[mdas]

• (carry over) [MISSED] write up and add some marionette/gaiatest endurance unit tests to catch space consumption/performance regressions bug 931044 and bug 931045 (I was actually working on bug 925398 in order to address the other bugs, milestone updated for next sprint)

[wlach]

• [MISSED] Get the B2G Eideticker dashboard running
  • running and uploading data to dave's people account (http://people.mozilla.org/~dhunt/eideticker-ci-dashboard/), but we need to configure to upload/send to eideticker.mozilla.org, which is waiting on moving the setup to the london office
  • Still blocked on IT AFAIK
• (carry over) [DONE] do some fps comparisons with 120fps camera
  • DONE
• improve test stability (launch time tests)
  • Dave is on this
  • Stabilizing contacts app: https://bugzilla.mozilla.org/show_bug.cgi?id=937730
  • Fixing b2g settings scrolling -DONE https://bugzilla.mozilla.org/show_bug.cgi?id=939191
• Get Eideticker working with Hamachi - DONE (mostly)
  • https://bugzilla.mozilla.org/show_bug.cgi?id=936121 (eideticker) - DONE pending review
  • https://bugzilla.mozilla.org/show_bug.cgi?id=936120 (orangutan, dependency) - DONE pending review
• NEW: Improve fps metrics (917404 -- old bug, but issue is more pressing now)
• NEW: Figure out why captures with 150 fps camera are so oversaturated -- they aren't with pointgrey's utility... (no bug filed yet but will do tomorrow)

[ato]

• Updating Selenium Proxy (programming sitting on top of Marionette that provides Selenium-compatible HTTP server) with missing endpoints to run Selenium tests against Marionette
• Allows for better overview of our existing compatibility level
  • Will file bugs on incompatibilities

[rwood]

• Nov 4-8th was MTBF week in Taipei with Taiwan QA Team; good progress
• (carry over) Get a full set of AWSY emulator test driver running locally
  • DONE except now blocked by Bug 931198 about_memory fails with > 30 memory files
• [missed] Deliver a set of memory test reports to the AWSY maintainer
• [missed - in progress] Get AWSY prodution VMs setup (1 master, others as nodes)
  • blocked by Bug 931198 however get as much setup as possible outside of that
• [DONE] Get gaia-ui endurance tests running on hamachi b2g26/1.2
  • Up and running but now need to cherry-pick endurance test changes that arent in b2g26
• Fix/investigate gaia-ui endurance test failures

Nov 22 - Dec 5

[jgriffin]

• (carry over) Finish adding OOP support to WebAPI tests bug 926280
• (carry over) get green set of debug emulator mochitests running everywhere bug 916359

[ahal]

• (carry over) get emulator reftests running oop
  • should also help getting b2g desktop reftests going oop
  • done and running on cedar (if the implementation is ok)
  • still some problems (crashes, timeouts, and an error that happens between tests but doens't seem to affect them)
• get reftests running on b2g desktop builds
• investigate emulator startup crashes?

[mdas]

• Fix gaia testing blocking issue with getting tabs bug 941749
• (carry over) (time consuming, landing will take more time) refactor marionette/gaiatest to use mixins: bug 925398
  • (carry over) write up and add some marionette/gaiatest endurance unit tests to catch space consumption/performance regressions bug 931044 and bug 931045

[wlach]

• Get the B2G Eideticker dashboard running
  • running and uploading data to dave's people account (http://people.mozilla.org/~dhunt/eideticker-ci-dashboard/), but we need to configure to upload/send to eideticker.mozilla.org, which is waiting on moving the setup to the london office
• NEW: Improve fps metrics (917404 -- old bug, but issue is more pressing now)
• NEW: Figure out why captures with 150 fps camera are so oversaturated -- they aren't with pointgrey's utility... (no bug filed yet but will do tomorrow)

[ato]

• (ongoing) Updating Selenium Proxy (programming sitting on top of Marionette that provides Selenium-compatible HTTP server) with missing endpoints to run Selenium tests against Marionette
• Add screen orientation implementation, using emulator build - bug 804515

[rwood]

• (carry over) Deliver a set of memory test reports to the AWSY maintainer
• (carry over) Get AWSY prodution VMs setup (1 master, others as nodes)
• Get another hamachi and replace the Inari endurance test (on master) device with hamachi
• Fix endurance test failures (update tests according to latest gaia changes)

Future milestones

• Fix frequent Mn orange - bug 930430 (test_submit.py failures)
• add tests for modal dialog bug 906175
• Investigate B2G "timed out after 1000s" errors: bug 906716 - these seem to be different flavors of emulator startup problems. See also a related bug, bug 905324, " Intermittent B2G mochitest "AssertionError: Timed out waiting for port!", which is probably another manifestation of the same problem.
• Stand up new builds and tests for B2G - bug 916356
• Put the Eideticker backend on Datazilla
• Look into data requirements for gaia-ui heuristics
• Make B2G tests in TBPL more sheriffable, by providing a tool in TBPL (Treeherder?) to compare revisions of all B2G repos between arbitrary builds
• Stand up tests on new Android 4.2 emulator (this is potentially going to be very time consuming, depends on 4.2 emu builds in TBPL)
• Implement device test failure notification heuristics (depends on results being uploaded to TBPL)
• Fix more top oranges
• Write tests for new bluetooth support in emulator?
• improve server side logging bug 928491
• shear away some of Marionette's MDN documentation and make it more concise and relevant. The device specific instructions may need updating as well.
• [DROPPED] Finish integrating JS Marionette client into Python testrunner
• [DROPPED] Gaia bisection tool - implemented by naoki