Auto-tools/Projects/B2G Automation: Difference between revisions
< Auto-tools | Projects
Jump to navigation
Jump to search
Line 198: | Line 198: | ||
== Future milestones == | == Future milestones == | ||
* add tests for modal dialog {{Bug|906175}} | |||
* Schedule gaia-ui-tests on B2G desktop builds on OSX 10.8 - https://bugzilla.mozilla.org/show_bug.cgi?id=932504 | * Schedule gaia-ui-tests on B2G desktop builds on OSX 10.8 - https://bugzilla.mozilla.org/show_bug.cgi?id=932504 | ||
* Investigate B2G "timed out after 1000s" errors: {{bug|906716}} - these seem to be different flavors of emulator startup problems. See also a related bug, {{bug|905324}}, " Intermittent B2G mochitest "AssertionError: Timed out waiting for port!", which is probably another manifestation of the same problem. | * Investigate B2G "timed out after 1000s" errors: {{bug|906716}} - these seem to be different flavors of emulator startup problems. See also a related bug, {{bug|905324}}, " Intermittent B2G mochitest "AssertionError: Timed out waiting for port!", which is probably another manifestation of the same problem. |
Revision as of 22:50, 31 October 2013
Etherpad
https://etherpad.mozilla.org/b2g-automation-meeting-notes
Goal 2013 Q2
Milestone 1 - 2 weeks (April 1 - 12)
- [jgriffin] [MISSED] Get Gaia Unit tests running in TBPL (blocked by bug 855049)
- [jgriffin] [DONE] Triage all existing B2G failures and identify a concrete set of problems that need to be addressed
- test_click errors bug 838607 - dburns looking at it.
- reftest/crashtest crash bug 818103 - assigned to mike habicher.
- [DONE] exception in window_manager.js bug 851666 - WORKSFORME
- [DONE] Timeout for WebAPI tests is too small; emulator callbacks can occasionally take 10-20s, leading to random timeouts. Rob will fix.
- socket.timeout errors caused by actorPool error bug 859417
- socket.timeout errors that occur at startup in newSession, before a new Gecko is copied - see logs in bug 829551
- chrome tests occasionally can't find elements (timing or page loading error?) bug 859423
- B2G crashes during WebAPI tests bug 859439 - will be easier to handle once crash detection is working here
- [jgriffin] [MISSED] Make mach runners for B2G mochitests and reftests
- [ahal] [DONE] Enable crash detection on mozilla-b2g18 for mochitest/reftest
- [ahal] [DONE] Add crash detection to Marionette tests
- [ahal] [MISSED] Write a script to download the latest emulator
- [mdas] [DONE] Use weakref in marionette-elements (bug 736592)
- [mdas] [ON TRACK] Work on getting Gaia UI tests running on b2g desktop builds
- [yiming] [DONE] Complete touch/mouse event code for action chains
- [yiming] [MISSED] Convert code from synthetic gestures to action chains
- [rwood] [DONE] Develop bluetooth Gaia UI test to pair unagi with host machine
- [rwood] [DONE] Finish integrating long running tests with datazilla
Milestone 2 - 2 weeks (April 22 - May 3)
- [jgriffin] [DONE] Get Gaia Unit tests running in TBPL (scheduled on cedar)
- [jgriffin] [DONE] Make mach runners for Marionette
- [jgriffin] [MISSED] Add mach to B2G so we can build B2G mach targets
- [jgriffin] [MISSED] Improve Marionette's error reporting with specific, clear error messages and stack traces
- [jgriffin] [MISSED] Integrate JS client Pt 1: Get a simple JS Marionette test working locally with the JS Marionette client and the Python testrunner
- [jgriffin] [MISSED] Get mozilla-central b2g desktop builds working with Marionette
- [ahal] [DONE] Write a B2G bootstrap script to make downloading and running tests easier
- [ahal] [MISSED] Use in-tree mozbase instead of puppetagain packages for B2G tests in buildbot
- [mdas] [MISSED] Work on refactoring touch events and getting the mapping between action chains and mouse events working.
- [mdas] [MISSED] Finish converting synthetic gestures to action chains
- [rwood] [DONE] Get gaia-ui endurance tests into gaia-ui github repo (working with dhunt on improvements as per his review) was pulled out of the repo, new location tbd
- [rwood] [ON TRACK] Improve gaia-ui endurance test stability
- [rwood] [DONE] Automate entire process via Jenkins
Milestone 3 - 2 weeks (May 13 - May 24)
- [ON TRACK] Work on top B2G test stability problems and either fix or get them to a state where developers can take over
- [DONE] test_click errors bug 838607 - dburns looking at it.
- [DONE] reftest/crashtest crash bug 818103 - assigned to mike habicher.
- [DROPPED] socket.timeout errors caused by actorPool error bug 859417 - dropped due to marionette refactor
- [MISSED] socket.timeout errors that occur at startup in newSession, before a new Gecko is copied - see logs in bug 829551
- [mdas] [DEFERRED] Finish converting synthetic gestures to action chains - may not be necessary.
- (carry over) [mdas] [DONE] On on refactoring touch events and getting the mapping between action chains and mouse events working.
- [mdas] [DONE] Develop manifest of stable gaia-ui-tests on b2g desktop builds
- [mdas] [ON TRACK] Work on improving Marionette docs
- [mdas] [ON TRACK] Fix tap/touch related bugs that are blocking webQA
- (carry over) [jgriffin] [DONE] Get mozilla-central b2g desktop builds working with Marionette
- [jgriffin] [DONE] Add manifest support to gaia-unit-tests
- [jgriffin] [DONE] write mozharness script for gaia-ui-tests on b2g desktop builds
- [DROPPED] Add debugger support to B2G testrunners
- (carry over) [ahal] [MISSED] Use in-tree mozbase instead of puppetagain packages for B2G tests in buildbot
- [wlach] [MISSED] Get rid of debugger server dependency bug 797529
Milestone 4 - 2 weeks (June 3 - June 14)
- (carry over) [wlach] [DONE] Get rid of debugger server dependency bug 797529
- [wlach] [ON TRACK] Integrate JS client Pt 1: Get a simple JS Marionette test working locally with the JS Marionette client and the Python testrunner
- (ongoing) [mdas] [MISSED] Resolve issues with tap
- (ongoing) [mdas] [ON TRACK] Update marionette docs
- (carry over) [ahal] [DONE] Get mozilla-central B2G unit tests to use in-tree mozbase instead of puppetagain
- [jgriffin] [MISSED] Disable gaia-unit-tests that are still failing on cedar
- [jgriffin + releng] [MISSED] Get gaia-unit-tests scheduled on all branches (depends on gecko-gaia coupling)
- [jgriffin] [MISSED] Update mozharness script for gaia-ui-tests
- [jgriffin + releng] [MISSED] Get gaia-ui-tests on b2g desktop builds scheduled on cedar (depends on gecko-gaia coupling, gaia-ui-tests to gaia coupling, and gaia-ui-tests branch renaming)
- [jgriffin] [MISSED] Identify current B2G top oranges
- [rwood] [DONE] Tryrel-eng m-c unagi-eng on gaia-ui-tests
- [rwood] [DONE] Develop some heuristics to help determine when gaia-ui-test Jenkins failures are "real" or reproducible enough to warrant sending out an e-mail notification to engineering managers
Milestone 5 - 2 weeks (June 24 - July 5)
- [wlach] [DROPPED] Add JS client Pt 1: Get a simple JS Marionette test working locally with the JS Marionette client and the Python testrunner
- (ongoing) [mdas] [ON TRACK] Resolve blocking issues with tap
- [mdas] [DEFER] Resolve blocking scroll issues - blocked on is_displayed fix
- (ongoing) [mdas] [DONE] Add some mechanisms to make debugging touch related problems easier
- [jgriffin] [ON TRACK] Disable gaia-unit-tests that are still failing on cedar
- [jgriffin + releng] [MISSED] Get gaia-unit-tests scheduled on all branches (depends on gecko-gaia coupling)
- [jgriffin] [DONE] Update mozharness script for gaia-ui-tests
- [jgriffin + releng] [DONE] Get gaia-ui-tests on b2g desktop builds scheduled on cedar (depends on gecko-gaia coupling, gaia-ui-tests to gaia coupling, and gaia-ui-tests branch renaming)
- [jgriffin] [DEFER] Identify current B2G top oranges
- [ahal] [DONE] Identify and fix issues with full-stack emulator builds
- [rwood] [DONE] Fix intermittent gaia-ui endurance test failures
- [rwood] [DONE] Expand gaia-ui endurance test coverage
- [rwood] [DONE] Write script to email/flag b2g memory spikes via gaia-ui endurance tests
Goal 2013 Q3
Milestone 6 - 2 weeks (July 8 - July 19)
- (carry over) [jgriffin + releng] [DONE] Get gaia-unit-tests scheduled on all branches
- [jgriffin] [MISSED] Implement script to compare two sources.xml
- [jgriffin] [DONE] Identify source of memory leak in LG automated tests bug 897684
- [mdas, davehunt, webqa] [DONE] Get gaia-ui-tests on cedar running stable green
- [mdas] [MISSED] Implement modal dialog support in Marionette
- (carry over) [mdas] [MISSED] Fix remaining blocking touch issues
- [wlach] [DONE] Implement get_active_frame command - bug 855327
- [rwood] [MISSED] Get games benchmarking tests running on Firefox OS
- [rwood] [ON TRACK] Investigate/fix intermittent WebAPI emulator failures
- [rwood] [ON TRACK] Maintain endurance tests as needed; expand checks for b2g memory changes over time
- [rwood] [DONE] Get endurance tests running on Leo on Master
- [rwood] [DEFER] Look into data requirements for gaia-ui heuristics
Milestone 7 - 2 weeks (July 29 - August 9)
- ongoing - patches in hand - [ahal] Add mach to B2G so we can build B2G mach targets
- scheduled on cedar - [mihnea] Get JSreftests working for B2G
- [jgriffin] [DONE] Implement script to compare two sources.xml
- [DONE] final patch up - (carry over) [mdas] Implement modal dialog support in Marionette
- current focus -> (carry over) [mdas] Fix remaining blocking touch issues
- (carry over) [rwood] Get games benchmarking tests running on Firefox OS
- (ongoing) [rwood] [ON TRACK] Investigate/fix intermittent WebAPI emulator failures
- (ongoing) [rwood] Maintain endurance tests as needed (Leo on v1-train); expand checks for b2g memory changes over time
- currently working with jlebar on this -> [rwood] Validate emulator as platform for AreWeSlimYet (AWSY) memory tests
- [rwood] Create app-launch and page-browser tsets for AWSY
- (carry over) [rwood] Look into data requirements for gaia-ui heuristics
Milestone 8 - 2 weeks (August 19 - August 30)
- (carry over) [ahal] Add mach to B2G so we can build B2G mach targets
- [DONE] scheduled on cedar - [mihnea] Get JSreftests working for B2G
- P1! [jgriffin + all] Update all testing docs to make it easy for developers to get started with tests
- [mdas] Investigate modal dialog test failure
- (carry over) [rwood] Get games benchmarking tests running on Firefox OS
- (ongoing) [rwood] [ON TRACK] Investigate/fix intermittent WebAPI emulator failures
- (ongoing) [rwood] Maintain endurance tests as needed (Leo on v1-train); expand checks for b2g memory changes over time
- currently working with jlebar on this -> [rwood] Validate emulator as platform for AreWeSlimYet (AWSY) memory tests
- [rwood] Create app-launch and page-browser tsets for AWSY
Sept 19 - Oct 2
- (jgriffin PTO)
- (carry over) [DONE] [mdas] land the modal dialog fix
- [mdas] [MISSED] fix the import script leak (landed, but backed out)
- [mdas] [DEFER] write up and add some marionette/gaiatest endurance unit tests to catch space consumption/performance regressions
- [ahal] [DONE] write mozharness script for running mochitests on B2G desktop builds
- [wlach] [DONE] finish implementation of npm mirror for rel-eng (re-writing in node.js and documenting)
- [rwood] [ON TRACK] Endurance test fixes/maintenance for inari master/1.3
- [rwood] [ON TRACK] Further work on AWSY emulator test driver
- [rwood] [ON TRACK] Continued support for games benchmarking on Firefox OS
Goal 2013 Q4
Oct 10 - Oct 24
- [jgriffin] [ON TRACK] investigate problem with B2G unit test timeouts not being caught by mozprocess
- [jgriffin] [DONE] add B2G support to existing Marionette mach command
- [mdas] [DONE] investigate issues with the modal dialog fix
- (carry over) [mdas] [DONE] fix the import script leak (landed, but backed out)
- [mdas] [ON TRACK] write up and add some marionette/gaiatest endurance unit tests to catch space consumption/performance regressions
- [ahal] [ON TRACK] get mochitests on b2g desktop builds rolled out everywhere
- [ahal] [ON TRACK] get B2G reftests running OOP
- [wlach] document Eideticker in such a way that it's easy for partners and people new to the project to set it up from scratch
- [wlach] [DONE] instrument better app startup times for B2G bug 888103
- [wlach] [ON TRACK] do some fps comparisons with 120fps camera
- (ongoing) [wlach] attend FxOS_Perf standups and continue to provide feedback wrt to power management testing
- [rwood] Endurance test fixes/maintenance for inari master/1.3
- [rwood] Get AWSY emulator test driver running locally
- [rwood] [ON TRACK] Set up Jenkins cluster on new VMs for AWSY tests
- [rwood] [DONE] Continued support for games benchmarking on Firefox OS
Oct 24 - Nov 7
[jgriffin]
- get gaia-integration tests in TBPL (depends on rel-eng)
- waiting on rel-eng to install npm/node 0.10 on test slaves - bug 931875; dustin owns this
- get green set of debug emulator mochitests running everywhere
- in progress on cedar; see https://tbpl.mozilla.org/?tree=Cedar&jobname=emulator&showall=1; devs working on greening up problems also on https://tbpl.mozilla.org/?tree=Pine
[mdas]
- (carry over) (time consuming) write up and add some marionette/gaiatest endurance unit tests to catch space consumption/performance regressions bug 931044 and bug 931045
- [DONE] Stop making marionette.log when unnecessary bug 918408 - (done? waiting on merge into m-c, and to mozilla-b2g26)
- [DEFER] add tests for modal dialog bug 906175
[ahal]
- (carry over) get mochitests on b2g desktop builds rolled out everywhere
- blocked by bug 924681
- [DONE] address B2G crash reporting problems
- mostly done (I think)
[wlach]
- (carry over) [DONE] document Eideticker in such a way that it's easy for partners and people new to the project to set it up from scratch - (done -- already transcribed to wiki)
- Get the B2G Eideticker dashboard running
- running and uploading data to dave's people account (http://people.mozilla.org/~dhunt/eideticker-ci-dashboard/), but we need to configure to upload/send to eideticker.mozilla.org, which is waiting on moving the setup to the london office
- also need to file some follow-up bugs on test reproducibility (e.g. for contact loading we don't last long enough to display all pictures)
- (carry over) do some fps comparisons with 120fps camera
[rwood]
- (carry over) Get a full set of AWSY emulator test driver running locally
- done except now blocked by Bug 931198 about_memory fails with > 30 memory files
- Assist the Taipei team with MTBF tests
- Leaving tomorrow eve
- Deliver a set of memory test reports to the AWSY maintainer
- Get AWSY prodution VMs setup (1 master, others as nodes)
- Get gaia-ui endurance tests running on hamachi b2g26/1.2
- Up and running but now need to cherry-pick endurance test changes that arent in b2g26
- Fix/investigate gaia-ui endurance test failures
Future milestones
- add tests for modal dialog bug 906175
- Schedule gaia-ui-tests on B2G desktop builds on OSX 10.8 - https://bugzilla.mozilla.org/show_bug.cgi?id=932504
- Investigate B2G "timed out after 1000s" errors: bug 906716 - these seem to be different flavors of emulator startup problems. See also a related bug, bug 905324, " Intermittent B2G mochitest "AssertionError: Timed out waiting for port!", which is probably another manifestation of the same problem.
- Stand up new builds and tests for B2G - bug 916356
- Put the Eideticker backend on Datazilla
- Look into data requirements for gaia-ui heuristics
- Make B2G tests in TBPL more sheriffable, by providing a tool in TBPL (Treeherder?) to compare revisions of all B2G repos between arbitrary builds
- Stand up tests on new Android 4.2 emulator (this is potentially going to be very time consuming, depends on 4.2 emu builds in TBPL)
- Implement device test failure notification heuristics (depends on results being uploaded to TBPL)
- Fix more top oranges
- Write tests for new bluetooth support in emulator?
- improve server side logging bug 928491
- [DROPPED] Finish integrating JS Marionette client into Python testrunner
- [DROPPED] Gaia bisection tool - implemented by naoki