EngineeringProductivity/Projects/Everything: Difference between revisions

From MozillaWiki
Jump to navigation Jump to search
m (typo)
 
(36 intermediate revisions by 7 users not shown)
Line 1: Line 1:


== Project Table ==
== Project Table ==
We have many projects in different states.  This table below should highlight what is being done and upcoming work in the future.  If you want to get involved, look for a higher [[https://wiki.mozilla.org/Auto-tools/Projects#Guidelines Friendly]] score!


* Priority score of 1 is the highest, 3 is the lowest
There is a project table that is a short list of the most active, projects: https://wiki.mozilla.org/EngineeringProductivity/Projects
* Friendly score of 10 is the highest, 1 is the lowest
Click [[https://wiki.mozilla.org/Auto-tools/Projects#Guidelines here]] to learn more about the guidelines and priority
 
{| class="wikitable sortable"
|-
! Project Name !! Link to Details/Docs !! Contact Info !! Friendly (1-10) !! Priority (1-3)
|-
| Alert Manager || [[https://wiki.mozilla.org/Auto-tools/Projects/AlertManager AlertManager Wiki]]|| dminor,jmaher || 8 || 2
|-
| Autophone || [[https://wiki.mozilla.org/Auto-tools/Projects/AutoPhone AutoPhone Wiki]] || bc || 6 || 2
|-
| Bugzilla || [[http://wiki.mozilla.org/BMO BMO Wiki]] || dkl,dylan,glob,mcote || 9 || 2
|-
| DevTools Harness || [[https://wiki.mozilla.org/Auto-tools/Projects/DevToolsHarness Wiki]] || ted || 6 || 1
|-
| m21s || [[https://wiki.mozilla.org/Auto-tools/Projects/m21s m21s Wiki]] || ahal,chmanchester,whimboo || 5 || 1
|-
| Marionette || [[https://wiki.mozilla.org/Auto-tools/Projects/Marionette Marionette Wiki]] || ato,AutomatedTester,jgriffin || 7 || 1
|-
| MozReview || [[https://wiki.mozilla.org/Auto-tools/Projects/CodeReviewTool MozReview Wiki]] || mcote || 6 || 1
|-
| MozRegression || [[https://wiki.mozilla.org/Auto-tools/Projects/Mozregression Mozregression Wiki]] || wlach || 6 || 3
|-
| Talos || [[https://wiki.mozilla.org/Auto-tools/Projects/Talos Talos Wiki]] || jmaher || 5 || 3
|-
| Test Informant || [[https://wiki.mozilla.org/Auto-tools/Projects/Test-Informant Test Informant Wiki]] || ahal || 6 || 2
|-
| Treeherder || [[https://wiki.mozilla.org/Auto-tools/Projects/Treeherder Treeherder Wiki]] || camd,edmorley,mdoglio || 7 || 1
|-
| Web-Platform-Tests || [[https://wiki.mozilla.org/Auto-tools/Projects/web-platform-tests web-platform-tests Wiki]] || jgraham || 5 || 3
|}
 
 
== Maintenance Project Table ==
We have many projects in different states.  This table below lists the projects we have done and left off in some state.  Most of these are still running and we are responsible for maintenance as it is needed.  There is no specific goal, deliverable, roadmap, milestones planned for these projects.
 
* Priority score of 1 is the highest, 3 is the lowest
* Friendly score of 10 is the highest, 1 is the lowest
Click [[https://wiki.mozilla.org/Auto-tools/Projects#Guidelines here]] to learn more about the guidelines and priority
 
{| class="wikitable sortable"
|-
! Project Name !! Link to Details/Docs !! Contact Info !! Friendly (1-10) !! Priority (1-3)
|-
| BugzFeed || [[https://wiki.mozilla.org/BMO/ChangeNotificationSystem BMO Change Notification]] || mcote || 4 || 3
|-
| BMO/ES || [[https://wiki.mozilla.org/BMO/ElasticSearch BMO Elasticsearch]] || ekyle || 6 || 3
|-
| Charts || [[https://wiki.mozilla.org/Auto-tools/Projects/Charts Charts Wiki]] || ekyle || 7 || 2
|-
| Datazilla || [[https://datazilla.readthedocs.org/en/latest/ Datazilla ReadTheDocs]] || camd,mdoglio,wlach || 4 || 3
|-
| dzAlerts || [[https://wiki.mozilla.org/Auto-tools/Projects/Alerts Alerts Wiki]] || ekyle || 4 || 2
|-
| Eideticker || [[https://wiki.mozilla.org/Project_Eideticker#Documentation Eideticker Wiki]] || davehunt,wlach || 3 || 3
|-
| Mozharness || [[https://wiki.mozilla.org/Auto-tools/Projects/Mozharness Mozharness Wiki]] || armenzg || 8 || 2
|-
| OrangeFactor || [[https://wiki.mozilla.org/Auto-tools/Projects/OrangeFactor OrangeFactor Wiki]] || jgriffin,mcote || 3 || 3
|-
| Ouija || [[https://wiki.mozilla.org/Auto-tools/Projects/Ouija Ouija Wiki]] || dminor,jmaher || 4 || 3
|-
| Pulse || [[https://wiki.mozilla.org/Auto-tools/Projects/Pulse Pulse Wiki]] || mcote || 5 || 3
|-
| Structured Logging || [[https://wiki.mozilla.org/Auto-tools/Projects/Structured_Logging Structured Logging Wiki]] || ahal,chmanchester,jgraham || 8 || 3
|-
| TBPL|| [[https://wiki.mozilla.org/Sheriffing/TBPL TBPL Wiki]] || edmorley || 4 || 2
|}


== Motivation ==  
== Motivation ==  


This page is to help diagram all the various things the A*Team has built and continue to support.
This page exists to provide a low-maintenance list of all our projects. I do not beleive anyone knows how they '''all''' interact, but at least this list ensures here are no hidden moving parts.


The plan is to draw high level components/systems, and links between them, so it is easier to see how all the parts work together.  Hopefully this diagram will be clickable so the reader can get more detail on each of the components.
''Template can be found below''


Something like [https://wiki.mozilla.org/images/f/f3/Releng_flow_onepage_treeclose_reasons.pdf what RelEng has], but without all the internals.
== The List ==


== Example ==
=== ActiveData ===
 
* '''Docs''' - https://wiki.mozilla.org/Auto-tools/Projects/ActiveData
Use the EMPTY TEMPLATE to add more entries.  Include ones you know about, even if you can not fill them.
* '''Code''' - https://github.com/klahnakoski/ActiveData
 
* '''Contact''' - klahnakoski@mozilla.com
=== EXAMPLE ===
* '''Inputs'''  
* '''Docs''' - some wiki, or read the docs to learn more
** Buildbot JSON logs
* '''Code''' - link to code, if it makes sense
** hg.mozilla.org
* '''Contact''' - for ekyle to contact if he has questions
** OrangeFactor
* '''Inputs''' -
** Perfherder
** automated resources
** Pulse exchange/build/normalized
** services consumed
** Structured Logs from Tests
* '''Outputs/Services'''
** Talos
** automated services provided
** Text logs (for buildbot and mozharness steps only)
** dashboards (and teams that consume them)
 
=== EMPTY TEMPLATE ===
* '''Docs''' -
* '''Code''' -
* '''Contact''' -
* '''Inputs'''
**
* '''Outputs/Services'''
* '''Outputs/Services'''
**
** Query endpoint: http://activedata.allizom.org/query
 
** Query Tool: http://activedata.allizom.org/tools/query.html
== The List ==


=== AlertManager ===
=== AlertManager ===
Line 118: Line 40:


=== Autoland ===
=== Autoland ===
* '''Docs''' - https://wiki.mozilla.org/Auto-tools/Projects/Autoland
* '''Docs''' - http://mozilla-version-control-tools.readthedocs.io/en/latest/mozreview/autoland.html
* '''Code''' - https://github.com/dminor/autoland
* '''Code''' - http://hg.mozilla.org/hgcustom/version-control-tools/file/tip/autoland
* '''Contact''' - [:dminor]
* '''Contact''' - [:mcote], [:glob]
* '''Inputs'''
* '''Inputs'''
** try jobs via pulse
** MozReview UI & MozReview repo
** job status via relengapi
* '''Outputs'''
* '''Outputs'''
** autoland comments posted to bugzilla
** Commits land on the repository of record (or try)
** patches landed via releng's 'hg transplant'
** Status reported to MozReview


=== Autophone ===
=== Autophone ===
Line 137: Line 58:
** http://phonedash.mozilla.org/ (used by who?)
** http://phonedash.mozilla.org/ (used by who?)


=== Bugzilla ===
=== Bugzilla / BMO ===
* '''Docs''' - https://bugzilla.readthedocs.org
* '''Docs''' - https://bmo.readthedocs.io
* '''Code''' - https://git.mozilla.org/?p=webtools/bmo/bugzilla.git
* '''Code''' - https://github.com/mozilla-bteam/bmo
* '''Contact''' - Mark Cote, Byron Jones, David Lawrence, Dylan Hardison
* '''Contact''' - Mark Cote, David Lawrence, Dylan Hardison
* '''Inputs'''
* '''Inputs'''
** Web UI
** Web UI
Line 146: Line 67:
* '''Outputs/Services'''
* '''Outputs/Services'''
** https://bugzilla.mozilla.org (bug reporters, developers, etc)
** https://bugzilla.mozilla.org (bug reporters, developers, etc)
** WebServices API (For REST https://bugzilla.mozilla.org/rest)
** Pulse, Bugzfeed (via the [[https://wiki.mozilla.org/BMO/ChangeNotificationSystem]])
** BMO Push service (custom notifications for specific applications)


=== BugzFeed ===
=== BugzFeed ===
Line 174: Line 96:
* '''Contact''' -  
* '''Contact''' -  
* '''Inputs'''
* '''Inputs'''
** hg.mozilla.org?
** hg.mozilla.org
* '''Outputs/Services'''
* '''Outputs/Services'''
** builder slaves
** builder slaves
** tester slaves
** tester slaves
** https://wiki.mozilla.org/ReleaseEngineering/BuildAPI ??
** https://wiki.mozilla.org/ReleaseEngineering/BuildAPI ??
** where do buildbot results go, a file?  picked up by TBPL
** JSON Logs: http://builddata.pub.build.mozilla.org/builddata/buildjson/


=== C++ Code Coverage ===
=== C++ Code Coverage ===
Line 185: Line 107:
* '''Docs''' - https://bugzilla.mozilla.org/show_bug.cgi?id=890116
* '''Docs''' - https://bugzilla.mozilla.org/show_bug.cgi?id=890116
* '''Contact''' - [:ahal]
* '''Contact''' - [:ahal]
* Used to be possible to trigger code coverage builds with 'try: -b o -p linux64-cc -u all -t none', but I think this is broken now. Probably isn't terribly hard to revive this. Needs further investigation.


=== charts.mozilla.org ===
=== charts.mozilla.org ===
Line 196: Line 119:
** http://charts.mozilla.org (for management)
** http://charts.mozilla.org (for management)


=== Datazilla ===
=== CodeCoverage ===
* '''Docs''' - https://wiki.mozilla.org/EngineeringProductivity/Projects/CodeCoverage
* '''Code'''
** '''UI''' - https://github.com/chinhodado/codecoverage_presenter
** '''ETL''' - https://github.com/klahnakoski/ActiveData-ETL/blob/dev/activedata_etl/transforms/jscov_to_es.py
* '''Contact''' - jmaher@mozilla.com, klahnakoski@mozilla.com, chmanchester@mozilla.com
* '''Inputs'''
** Various tests have been configured with a code coverage option
* '''Outputs/Services'''
** ActiveData - https://wiki.mozilla.org/Auto-tools/Projects/ActiveData
** http://chinhodado.github.io/codecoverage_presenter/
 
=== Datazilla (retired) ===


* '''Docs''' - https://datazilla.readthedocs.org/en/latest/
* '''Docs''' - https://datazilla.readthedocs.org/en/latest/
Line 206: Line 141:
** charts at https://datazilla.mozilla.org
** charts at https://datazilla.mozilla.org
** raw results at https://datazilla.mozilla.org/talos/refdata/objectstore/json_blob
** raw results at https://datazilla.mozilla.org/talos/refdata/objectstore/json_blob
=== DevelopmentMetrics ===
* '''Docs''' - https://wiki.mozilla.org/Auto-tools/Projects/DevelopmentMetrics
* '''Code''' - https://github.com/klahnakoski/MoDevMetrics
* '''Contact''' - klahnakoski@mozilla.com
* '''Inputs'''
** Bugzilla/ES Cluster
* '''Outputs/Services'''
** various charts and dashboards


=== DevTools Harness ===
=== DevTools Harness ===
Line 212: Line 157:
* '''Contact''' - [:ted.mielczarek]
* '''Contact''' - [:ted.mielczarek]


=== dzAlerts ===
=== dzAlerts (retired)===


* '''Docs''' - https://wiki.mozilla.org/Auto-tools/Projects/Alerts
* '''Docs''' - https://wiki.mozilla.org/Auto-tools/Projects/Alerts
* '''Code''' - https://github.com/klahnakoski/datazilla-alerts
* '''Code''' - https://github.com/klahnakoski/datazilla-alerts
* '''Contact' - klahnakoski@mozilla.com
* '''Contact''' - klahnakoski@mozilla.com
* '''Inputs'''
* '''Inputs'''
** Datazilla web service (https://datazilla.mozilla.org/talos/refdata/objectstore/json_blob)
** Datazilla web service (https://datazilla.mozilla.org/talos/refdata/objectstore/json_blob)
Line 223: Line 168:
** fxos-perf-alerts@mozilla.org (b2g sheriffs?)
** fxos-perf-alerts@mozilla.org (b2g sheriffs?)


 
=== Eideticker (retired) ===
=== Eideticker ===


* '''Docs''' - https://wiki.mozilla.org/Project_Eideticker#Documentation
* '''Docs''' - https://wiki.mozilla.org/Project_Eideticker#Documentation
Line 233: Line 177:
* '''Outputs/Services'''
* '''Outputs/Services'''
** Eideticker dashboard (http://eideticker.mozilla.org)
** Eideticker dashboard (http://eideticker.mozilla.org)
=== Janitor ===
* '''Docs''' - https://mana.mozilla.org/wiki/display/ateam/Janitor
* '''Code''' - https://github.com/jankeromnes/janitor, https://github.com/jankeromnes/dockerfiles
* '''Contact''' - Jan Keromnes [:janx]
* '''Inputs'''
** (none yet)
* '''Outputs/Services'''
** https://janitor.technology
** BMO: new patches uploaded by users
** Treeherder: new pushes from users
** Docker Hub: image updates https://hub.docker.com/u/janx/


=== m21s ===
=== m21s ===
Line 259: Line 215:


=== MozReview (Review Board) ===
=== MozReview (Review Board) ===
* '''Docs''' - [[Auto-tools/Projects/CodeReviewTool]]
* '''Docs''' - http://mozilla-version-control-tools.readthedocs.io/en/latest/mozreview.html
* '''Code''' - various pieces of https://hg.mozilla.org/hgcustom/version-control-tools; see docs for more.
* '''Code''' - various pieces of https://hg.mozilla.org/hgcustom/version-control-tools; see docs for more.
* '''Contact''' - mcote@mozilla.com
* '''Contact''' - mcote, glob, smacleod
* '''Inputs'''
* '''Inputs'''
** BMO
** BMO
** hg.mozilla.org
** reviewboard-hg.mozilla.org
* '''Outputs/Services'''
* '''Outputs/Services'''
** BMO
** BMO
** Various repositories via Autoland


=== OrangeFactor ===
=== OrangeFactor ===
Line 291: Line 248:
** tbpl.mozilla.org
** tbpl.mozilla.org
* '''Outputs/Services'''
* '''Outputs/Services'''
**
** http://54.215.155.53
 
=== Perfherder ===
* '''Docs''' - https://wiki.mozilla.org/EngineeringProductivity/Projects/Perfherder
* '''Code''' - https://github.com/mozilla/treeherder/tree/master/treeherder/perf
* '''Contact''' - :wlach
* '''Inputs'''
** PERFHERDER_DATA
*** schema  https://github.com/mozilla/treeherder/blob/master/schemas/performance-artifact.json
*** example http://wrla.ch/blog/2015/11/perfherder-onward/
* '''Outputs/Services'''
** https://treeherder.mozilla.org/perf.html#/graphs


=== Pulse ===
=== Pulse ===
* '''Docs''' - https://wiki.mozilla.org/Auto-tools/Projects/Pulse  
* '''Docs''' - https://wiki.mozilla.org/Auto-tools/Projects/Pulse  
* '''Code''' - https://hg.mozilla.org/automation/mozillapulse
* '''Code''' - development library: https://hg.mozilla.org/automation/mozillapulse
* '''Contact''' - Mark Côté
* '''Contact''' - mcote
* '''Inputs''' (exchanges? http://christian.legnitto.com/blog/2010/07/17/mozilla-pulse-and-rabbitmq/)
* '''Inputs'''
** a multitude of other systems  
** a multitude of other systems  
* '''Outputs/Services'''
* '''Outputs/Services'''
** a multitude of other systems
** a multitude of other systems
** https://pulse.mozilla.org/
** https://pulse.mozilla.org/
=== PulseGuardian ===
* '''Docs''' - https://wiki.mozilla.org/Auto-tools/Projects/Pulse/PulseGuardian
* '''Code''' - https://github.com/mozilla/pulseguardian/
* '''Contact''' - mcote
* '''Inputs'''
** Persona (for auth) (soon to be Okta or TaskCluster auth)
** Pulse
** Web UI
* '''Outputs/Services'''
** Pulse
** Email notifications
=== PulseTranslator ===
* '''Docs''' - https://github.com/mozilla/pulsetranslator/blob/master/README.md
* '''Code''' - https://github.com/mozilla/pulsetranslator
* '''Contact''' - jgriffin@mozilla.com
* '''Inputs'''
** Pulse exchange/build
* '''Outputs/Services'''
** Pulse exchange/build/normalized


=== Structured Logging ===
=== Structured Logging ===
Line 321: Line 310:
Can be referred to as what is run by buildbot or by tbpl since this is the only performance test run on our per revision CI system.  Other tools are run at a different cadence.
Can be referred to as what is run by buildbot or by tbpl since this is the only performance test run on our per revision CI system.  Other tools are run at a different cadence.


=== TBPL (for the sake of history) ===
=== TBPL (retired) ===


* '''Docs''' - [[Sheriffing/TBPL]]
* '''Docs''' - [[Sheriffing/TBPL]]
Line 362: Line 351:
** https://github.com/mozilla/gecko-dev/blob/master/testing/web-platform/tests/README.md  
** https://github.com/mozilla/gecko-dev/blob/master/testing/web-platform/tests/README.md  
* '''Contact''' - [:jgraham]
* '''Contact''' - [:jgraham]
== Example ==
Use the EMPTY TEMPLATE to add more entries.  Include ones you know about, even if you can not fill them.
=== EXAMPLE ===
* '''Docs''' - some wiki, or read the docs to learn more
* '''Code''' - link to code, if it makes sense
* '''Contact''' - for ekyle to contact if he has questions
* '''Inputs''' -
** automated resources
** services consumed
* '''Outputs/Services'''
** automated services provided
** dashboards (and teams that consume them)
=== EMPTY TEMPLATE ===
* '''Docs''' -
* '''Code''' -
* '''Contact''' -
* '''Inputs'''
**
* '''Outputs/Services'''
**

Latest revision as of 15:39, 19 October 2016

Project Table

There is a project table that is a short list of the most active, projects: https://wiki.mozilla.org/EngineeringProductivity/Projects

Motivation

This page exists to provide a low-maintenance list of all our projects. I do not beleive anyone knows how they all interact, but at least this list ensures here are no hidden moving parts.

Template can be found below

The List

ActiveData

AlertManager

Autoland

Autophone

Bugzilla / BMO

BugzFeed

Bugzilla/ES Cluster

Buildbot

C++ Code Coverage

  • Docs - https://bugzilla.mozilla.org/show_bug.cgi?id=890116
  • Contact - [:ahal]
  • Used to be possible to trigger code coverage builds with 'try: -b o -p linux64-cc -u all -t none', but I think this is broken now. Probably isn't terribly hard to revive this. Needs further investigation.

charts.mozilla.org

CodeCoverage

Datazilla (retired)

DevelopmentMetrics

DevTools Harness

  • Docs - Needs project page!
  • Contact - [:ted.mielczarek]

dzAlerts (retired)

Eideticker (retired)

Janitor

m21s

Marionette

Mozharness

MozReview (Review Board)

OrangeFactor

Ouija

Perfherder

Pulse

PulseGuardian

PulseTranslator

Structured Logging

Talos

Can be referred to as what is run by buildbot or by tbpl since this is the only performance test run on our per revision CI system. Other tools are run at a different cadence.

TBPL (retired)

Test Informant

Treeherder

web-platform-tests

Example

Use the EMPTY TEMPLATE to add more entries. Include ones you know about, even if you can not fill them.

EXAMPLE

  • Docs - some wiki, or read the docs to learn more
  • Code - link to code, if it makes sense
  • Contact - for ekyle to contact if he has questions
  • Inputs -
    • automated resources
    • services consumed
  • Outputs/Services
    • automated services provided
    • dashboards (and teams that consume them)

EMPTY TEMPLATE

  • Docs -
  • Code -
  • Contact -
  • Inputs
  • Outputs/Services
Retrieved from "https://dev.wikimo.nonprod.webservices.mozgcp.net/index.php?title=EngineeringProductivity/Projects/Everything&oldid=1151838"