Auto-tools/Meetings/2012-08-27: Difference between revisions
< Auto-tools | Meetings
Jump to navigation
Jump to search
(→War on Orange: Add edmorley) |
|||
(17 intermediate revisions by 9 users not shown) | |||
Line 3: | Line 3: | ||
Below are our [https://wiki.mozilla.org/Auto-tools/Goals/2012Q3 top level goals]. Beneath each goal, note any progress made towards that goal this week. | Below are our [https://wiki.mozilla.org/Auto-tools/Goals/2012Q3 top level goals]. Beneath each goal, note any progress made towards that goal this week. | ||
* {{ok|Support the B2G and platform team by moving existing automation to that platform as well as expanding automation on B2G and B2G devices for basecamp deliverables.}} | * {{ok|Support the B2G and platform team by moving existing automation to that platform as well as expanding automation on B2G and B2G devices for basecamp deliverables.}} | ||
** [mihneadb] SUTAgent can be now built inside B2G | |||
** [mihneadb] SUTAgent can now start on boot | |||
** [jgriffin] Will try turning on Marionette tests in TBPL on desktop Firefox this week; will help prevent accidental Marionette breakage in B2G testing | |||
* {{ok|Expand Mobile testing platforms for B2G and Fennec Native to include user devices and NEON Chipsets (panda boards).}} | * {{ok|Expand Mobile testing platforms for B2G and Fennec Native to include user devices and NEON Chipsets (panda boards).}} | ||
* {{ok|Establish new Datazilla system as a useful tool for capturing and diagnosing talos performance regressions on a per-push basis, as well as capturing data for network and B2G performance tests.}} | * {{ok|Establish new Datazilla system as a useful tool for capturing and diagnosing talos performance regressions on a per-push basis, as well as capturing data for network and B2G performance tests.}} | ||
Line 8: | Line 11: | ||
* {{ok|Upgrade BMO to 4.2, deploy tracking flags optimizations, UX improvements (comment tagging, mid-air resolution)}} | * {{ok|Upgrade BMO to 4.2, deploy tracking flags optimizations, UX improvements (comment tagging, mid-air resolution)}} | ||
* {{ok|Make TBPL More Orange aware to make it easier to file intermittent orange bugs, and so that we can more adequately track (and therefore address) issues around android-related and memory leak intermittent failures}} | * {{ok|Make TBPL More Orange aware to make it easier to file intermittent orange bugs, and so that we can more adequately track (and therefore address) issues around android-related and memory leak intermittent failures}} | ||
** Have been working on issues that are causing more day-to-day aggravation (/time loss) than filing oranges the last week (eg infra load patches, tree status, anti-orange work given after today ESR17 orange will be with us until December 2013 etc), so apart from a few more discussions on implementation, didn't have any time to work on this around sheriffing. Making tree status live was one of the other proposed sheriffing goals for this quarter - can we just add it retrospectively, seeing as it is now live? :-) | |||
* {{ok|Enable QA and contributors to increase their skills and participation by crafting automated tests for B2G and desktop smoketests by providing examples, mentoring, and documentation}} | * {{ok|Enable QA and contributors to increase their skills and participation by crafting automated tests for B2G and desktop smoketests by providing examples, mentoring, and documentation}} | ||
Line 15: | Line 19: | ||
''This is a place for people to put announcements or to describe activities that they'd like to highlight. It's not the responsibility of the MC to find things to put here.'' | ''This is a place for people to put announcements or to describe activities that they'd like to highlight. It's not the responsibility of the MC to find things to put here.'' | ||
* [edmorley] [https://treestatus.mozilla.org/ Tree Status] now live! This switch means that we are one step closing to ditching tinderbox & makes the sheriff's lives much easier, since multiple trees can have their state changed simultaneously (eg for an infra-caused tree closure), status messages can now be saved/restored, changes are logged (so you know who to ping for questions) & we have a [https://treestatus.mozilla.org/help proper API]! | |||
== In-Depth Project Discussion == | == In-Depth Project Discussion == | ||
Line 29: | Line 34: | ||
=== [[BMO|Bugzilla]] === | === [[BMO|Bugzilla]] === | ||
* push: https://globau.wordpress.com/2012/08/23/happy-bmo-push-day-10/ | |||
* [dkl+glob] code and reviewing leading up the html bugmail | |||
* [dkl+glob] debugging and fixing issues following html bugmail | |||
=== [[Project_Eideticker|Eideticker]] === | === [[Project_Eideticker|Eideticker]] === | ||
* [wlach] Finishing up dashboard support for Galaxy Nexus | |||
* Integrated some great patches from Benoit Girard (from the graphics team) - zooming around results is now much easier, get a summary of datapoints on hover. More exciting stuff pending. | |||
=== [https://github.com/mozilla/Firebug-Test-Runner Firebug Testing] === | === [https://github.com/mozilla/Firebug-Test-Runner Firebug Testing] === | ||
Line 37: | Line 47: | ||
=== [[Auto-tools/Projects/Marionette|Marionette]] === | === [[Auto-tools/Projects/Marionette|Marionette]] === | ||
* [mihneadb] almost done on getting Marionette to use Moztest for Autolog posting | |||
=== Mobile Automation === | === Mobile Automation === | ||
* [ahal] Investigating reftest resolution problem | |||
* [ahal] Made progress triaging, filing and fixing reftests for b2g | |||
** Reftests don't draw to canvas properly when running out of process | |||
** Some tests fail when using the <iframe> browser api instead of the xul:browser one | |||
** See {{bug|773482}} | |||
* [mihneadb] started work on getting the new agent to pass the existing tests. | |||
=== [[Auto-tools/Projects/Mobile_Evangelism_Site_Scraper|Mobile Evangelism Site-Scraping Tool]] === | === [[Auto-tools/Projects/Mobile_Evangelism_Site_Scraper|Mobile Evangelism Site-Scraping Tool]] === | ||
Line 47: | Line 64: | ||
=== [[Auto-tools/Projects/Mozmill|Mozmill]] === | === [[Auto-tools/Projects/Mozmill|Mozmill]] === | ||
* [henrik] Version [https://groups.google.com/forum/?fromgroups=#!topic/mozmill-dev/MlfwtEpRxNM 1.5.18 released] | |||
=== [[Auto-tools/Projects/MozTrap|MozTrap]] === | === [[Auto-tools/Projects/MozTrap|MozTrap]] === | ||
=== [[Auto-tools/Projects/Datazilla|New Graphserver (Datazilla)]] === | === [[Auto-tools/Projects/Datazilla|New Graphserver (Datazilla)]] === | ||
* [jeads] The metrics team developed an exponential smoothing treatment for t-test results that is very effective in reducing noise in talos data. Working on integrating that into datazilla now. | |||
=== [[Auto-tools/Projects/Open_Web_Apps_Testing|Open Web Apps]] === | === [[Auto-tools/Projects/Open_Web_Apps_Testing|Open Web Apps]] === | ||
Line 67: | Line 86: | ||
=== Talos === | === Talos === | ||
* [jhammel] give xtalos a real API and use it | |||
* [jhammel] Talos is now sending production data to datazilla (with datazilla_client) | |||
=== Sheriffing & Trees === | === Sheriffing & Trees === | ||
Line 75: | Line 96: | ||
* [edmorley] Investigation & patch for {{bug|690672}} - retriggering builds manually should never coalesce. This issue is a significant pain point for sheriffs on a daily basis. | * [edmorley] Investigation & patch for {{bug|690672}} - retriggering builds manually should never coalesce. This issue is a significant pain point for sheriffs on a daily basis. | ||
* [edmorley] Reviewed the revised list of removed 10.5 builders in {{bug|773120}}. | * [edmorley] Reviewed the revised list of removed 10.5 builders in {{bug|773120}}. | ||
* [edmorley] After the high infra load / long Try wait times issue was partially blamed on wasting capacity (by releng), filed {{bug|784681}} to assess all hidden builds/tests on TBPL to (a) make sure bugs filed to fix or else switch off broken/ | * [edmorley] After the high infra load / long Try wait times issue was partially blamed on wasting capacity (by releng), filed {{bug|784681}} & dependants to assess all hidden builds/tests on TBPL to (a) make sure bugs filed to fix or else switch off broken/potentially unwanted suites, and (b) ensure releng realise that most are in fact either working or blocked on releng themselves & so that they can't use them as a reason not to prioritise adding extra capacity. | ||
* [edmorley] Put together an idea for saving infra load, by only building certain platforms if files used by them changed. eg: If a push only changes b2g/* then don't build | * [edmorley] Put together an idea for saving infra load, by only building certain platforms if files used by them changed. eg: If a push only changes b2g/* then don't build anything other than B2G! :-) (We already do something like this for spidermonkey builds triggered on changes to js/src/). Created a WIP for producing stats for how much load this would save - which when finished in the coming week will be posted to dev.platform and used as justification for why it would be a good use of releng's time (my gut feeling is that this will save us quite a bit). | ||
=== [https://treestatus.mozilla.org/ Treestatus] === | === [https://treestatus.mozilla.org/ Treestatus] === | ||
Line 102: | Line 123: | ||
** Landed: https://bugzil.la/782939,782953 | ** Landed: https://bugzil.la/782939,782953 | ||
* [edmorley] Intermittent orange work: | * [edmorley] Intermittent orange work: | ||
** Chased up more newly landed tests that are failing intermittently, CCed devs on more top, unowned oranges. | ** In run up to ESR17, putting a lot of effort into anti-orange work, since whatever is in mozilla17 we'll be stuck starring on TBPL until December 2013. Chased up more newly landed tests that are failing intermittently, CCed devs on more top, unowned oranges. | ||
** Tried to narrow the regression range for {{bug|784247}}. | ** Tried to narrow the regression range for {{bug|784247}}. | ||
Line 108: | Line 129: | ||
=== Automation Development === | === Automation Development === | ||
* [henrik] Office hours upcoming this week on Thursday | |||
* [rwood] Submitted more tests for B2G WebTelephony API | |||
== Upcoming Events == | == Upcoming Events == |
Latest revision as of 16:35, 27 August 2012
Goals Progress
Below are our top level goals. Beneath each goal, note any progress made towards that goal this week.
- [ON TRACK] Support the B2G and platform team by moving existing automation to that platform as well as expanding automation on B2G and B2G devices for basecamp deliverables.
- [mihneadb] SUTAgent can be now built inside B2G
- [mihneadb] SUTAgent can now start on boot
- [jgriffin] Will try turning on Marionette tests in TBPL on desktop Firefox this week; will help prevent accidental Marionette breakage in B2G testing
- [ON TRACK] Expand Mobile testing platforms for B2G and Fennec Native to include user devices and NEON Chipsets (panda boards).
- [ON TRACK] Establish new Datazilla system as a useful tool for capturing and diagnosing talos performance regressions on a per-push basis, as well as capturing data for network and B2G performance tests.
- NOTE: This is largely a change to the data harvesting and analysis side of the Talos system (even though some changes are required to the test harness and tests themselves). We will not be replacing existing Talos data harvesting system in Q3. By the end of Q3, we want to have the new performance data harvesting and analysis system Datazilla to a point at which it can shadow the old system and we can use Q4 to verify that the new approach captures just as many regressions as the old system does. Potential timeframe for conversion of the old system to the new Datazilla system is in Q1 2013 (earliest "thumb in the wind" estimate).
- [ON TRACK] Upgrade BMO to 4.2, deploy tracking flags optimizations, UX improvements (comment tagging, mid-air resolution)
- [ON TRACK] Make TBPL More Orange aware to make it easier to file intermittent orange bugs, and so that we can more adequately track (and therefore address) issues around android-related and memory leak intermittent failures
- Have been working on issues that are causing more day-to-day aggravation (/time loss) than filing oranges the last week (eg infra load patches, tree status, anti-orange work given after today ESR17 orange will be with us until December 2013 etc), so apart from a few more discussions on implementation, didn't have any time to work on this around sheriffing. Making tree status live was one of the other proposed sheriffing goals for this quarter - can we just add it retrospectively, seeing as it is now live? :-)
- [ON TRACK] Enable QA and contributors to increase their skills and participation by crafting automated tests for B2G and desktop smoketests by providing examples, mentoring, and documentation
Highlights
This is a place for people to put announcements or to describe activities that they'd like to highlight. It's not the responsibility of the MC to find things to put here.
- [edmorley] Tree Status now live! This switch means that we are one step closing to ditching tinderbox & makes the sheriff's lives much easier, since multiple trees can have their state changed simultaneously (eg for an infra-caused tree closure), status messages can now be saved/restored, changes are logged (so you know who to ping for questions) & we have a proper API!
In-Depth Project Discussion
Pick one project/presenter per week who will give a detail presentation on their project.
The Overview
Progress, notes, problems, and other statuses about current projects. Add any new projects to this template!
AutoPhone/Noah's Ark
Bughunter
Bugzilla
- push: https://globau.wordpress.com/2012/08/23/happy-bmo-push-day-10/
- [dkl+glob] code and reviewing leading up the html bugmail
- [dkl+glob] debugging and fixing issues following html bugmail
Eideticker
- [wlach] Finishing up dashboard support for Galaxy Nexus
- Integrated some great patches from Benoit Girard (from the graphics team) - zooming around results is now much easier, get a summary of datapoints on hover. More exciting stuff pending.
Firebug Testing
Jetperf
Marionette
- [mihneadb] almost done on getting Marionette to use Moztest for Autolog posting
Mobile Automation
- [ahal] Investigating reftest resolution problem
- [ahal] Made progress triaging, filing and fixing reftests for b2g
- Reftests don't draw to canvas properly when running out of process
- Some tests fail when using the <iframe> browser api instead of the xul:browser one
- See bug 773482
- [mihneadb] started work on getting the new agent to pass the existing tests.
Mobile Evangelism Site-Scraping Tool
Mochitest
MozBase
Mozmill
- [henrik] Version 1.5.18 released
MozTrap
New Graphserver (Datazilla)
- [jeads] The metrics team developed an exponential smoothing treatment for t-test results that is very effective in reducing noise in talos data. Working on integrating that into datazilla now.
Open Web Apps
Peptest
Powerball
Pulse
Signal From Noise
Speedtests
Systems
Talos
- [jhammel] give xtalos a real API and use it
- [jhammel] Talos is now sending production data to datazilla (with datazilla_client)
Sheriffing & Trees
- [edmorley] Away today (27th) due to UK bank holiday.
- [edmorley] Day-to-day sheriffing of the trees.
- [edmorley] Sheriffing/tree closure issues of note:
- Issues with talos-r4-snow-041 and high rate of new intermittent crashes from that machine.
- [edmorley] Investigation & patch for bug 690672 - retriggering builds manually should never coalesce. This issue is a significant pain point for sheriffs on a daily basis.
- [edmorley] Reviewed the revised list of removed 10.5 builders in bug 773120.
- [edmorley] After the high infra load / long Try wait times issue was partially blamed on wasting capacity (by releng), filed bug 784681 & dependants to assess all hidden builds/tests on TBPL to (a) make sure bugs filed to fix or else switch off broken/potentially unwanted suites, and (b) ensure releng realise that most are in fact either working or blocked on releng themselves & so that they can't use them as a reason not to prioritise adding extra capacity.
- [edmorley] Put together an idea for saving infra load, by only building certain platforms if files used by them changed. eg: If a push only changes b2g/* then don't build anything other than B2G! :-) (We already do something like this for spidermonkey builds triggered on changes to js/src/). Created a WIP for producing stats for how much load this would save - which when finished in the coming week will be posted to dev.platform and used as justification for why it would be a good use of releng's time (my gut feeling is that this will save us quite a bit).
Treestatus
- [edmorley] Tree Status now live! This switch means that we are one step closing to ditching tinderbox & makes the sheriff's lives much easier, since multiple trees can have their state changed simultaneously (eg for an infra-caused tree closure), status messages can now be saved/restored, changes are logged (so you know who to ping for questions) & we have a proper API!
- Coordinated rollout with IT + performed post-rollout testing.
- Posted notification in dev.platform and the sheriffpass bug.
- Updated the following wiki documentation:
- [edmorley] Filed followups for a few UX tweaks once we'd been using it a few days + suggestions from other sheriffs: Github issue #22, #23, #24 & #25
Tinderboxpushlog
- [edmorley] Gave feedback on buildbot patch part of bug 704006 (adding TBPL support for identifying self-serve cancelled jobs from those caused by infra issues).
- [edmorley] Filed TBPL followups to TreeStatus support: https://bugzil.la/784670,785051,785058
- [edmorley] Made further tweaks to README.vagrant in response to feedback from those unfamiliar with Vagrant (bug 764894).
- [edmorley] Put together a list of easier, but still highly wanted TBPL bugs, so that a new contributor could pick some of them to work on.
- [edmorley] Helped a contributor set up a local TBPL instance using Vagrant & debug networking issues due to a VirtualBox bug. Want to try and find the root cause, so we can make Vagrant work for everyone out of the box and not put off potential new TBPL contributors.
- [edmorley] Discussions about bug 713419 (Use sheriffs.mozilla.org instead of the Google calendar to get the sheriff's name).
- [edmorley] Testing of patch for bug 656155 (Add "go to build" link for non-Try builds), testing performance of alternatives & review comments.
War on Orange
- [edmorley] OrangeFactor:
- Landed: https://bugzil.la/782939,782953
- [edmorley] Intermittent orange work:
- In run up to ESR17, putting a lot of effort into anti-orange work, since whatever is in mozilla17 we'll be stuck starring on TBPL until December 2013. Chased up more newly landed tests that are failing intermittently, CCed devs on more top, unowned oranges.
- Tried to narrow the regression range for bug 784247.
Web QA
Automation Development
- [henrik] Office hours upcoming this week on Thursday
- [rwood] Submitted more tests for B2G WebTelephony API
Upcoming Events
Standing weekly meeting related to our projects should go here
Round Table
Put your questions and things to raise with the entire team here
Take notes here and copy and paste them to this section afterwards.
Notes
Holidays and Trips
- Today (27th) is a UK bank holiday (no more now until Christmas, you'll be glad to know :-P).
Next week MC
Misc
Find something you think is cool, interesting, funny, or exciting and put it here