TestEngineering/Performance/Talos: Difference between revisions

Jump to navigation Jump to search

TestEngineering/Performance/Talos (view source)

Revision as of 19:19, 9 August 2019

2,214 bytes removed ,  9 August 2019
no edit summary
No edit summary
No edit summary
Line 3: Line 3:
[[Image:Talos.jpg|frameless|right]]  
[[Image:Talos.jpg|frameless|right]]  


Talos is a Python performance testing framework that is usable on Windows, macOS and Linux. Talos is our versatile performance testing framework we use at Mozilla. It was created to serve as a test runner for the existing performance tests that Mozilla was running back in 2007 as well as providing an extensible framework for new tests as they were created.  
Talos is a versatile cross-platform Python performance testing framework. It was created to serve as a test runner for the performance tests that Mozilla was running back in 2007, as well as providing an extensible framework for new tests as they were created.  
 
* Contact: Rob Wood [rwood]
* Source code: https://searchfox.org/mozilla-central/source/testing/talos
* Good first bugs: https://codetribute.mozilla.org/projects/automation?project%3DTalos


So, why Talos? Talos is the bronze automaton of Greek myth. Talos protected the island of Crete, throwing giant boulders at unwary seamen. He's also purported to have heated himself glowing hot and then embraced his enemies. '''Basically, he was awesome.'''
So, why Talos? Talos is the bronze automaton of Greek myth. Talos protected the island of Crete, throwing giant boulders at unwary seamen. He's also purported to have heated himself glowing hot and then embraced his enemies. '''Basically, he was awesome.'''


== Talos Tests ==
= Documentation =
Information about each test is available in [[/Tests|tests]]
* [[/Tests/|Tests]]
 
* [[/Platforms/|Platforms]]
== Running Talos Tests ==
* [[/Running|Running tests]]
 
* [[/Adding tests/]]
[[/Running|See the page on running Talos tests]].
* [[/Data|Data format]]
 
== Lifecycle of a Performance Test, from push to perfherder ==
 
* Taskcluster schedules [https://searchfox.org/mozilla-central/source/taskcluster/ci/test/talos.yml talos jobs]
* Taskcluster runs a talos job on a hardware machine when one is available - this is bootstrapped by [https://searchfox.org/mozilla-central/source/testing/mozharness/mozharness/mozilla/testing/talos.py Mozharness]
** [https://searchfox.org/mozilla-central/source/testing/mozharness/mozharness/mozilla/testing/talos.py Mozharness\ downloads the build, talos.zip (found in [http://hg.mozilla.org/mozilla-central/tip/testing/talos/talos.json talos.json]), and creates a virtualenv for running the test.
** Mozharness [[/Running#How_Talos_is_Run_in_Production|configures the test and runs it]]
** After the test is completed the data is uploaded to [https://treeherder.mozilla.org/perf.html#/graphs PerfHerder]
* Treeherder displays a green (all OK) status and has a link to the [https://treeherder.mozilla.org/perf.html#/graphs PerfHerder]
* 13 pushes later, [http://hg.mozilla.org/graphs/file/tip/server/analysis/analyze_talos.py analyze_talos.py] is ran which compares your push to the previous 12 pushes and next 12 pushes to look for a [[/Data#Regressions|regression]]
** if a regression is found, it will be posted on [https://treeherder.mozilla.org/perf.html#/alerts Perfherder Alerts]
 
== Resources / How-to ==


=== Adding a new test ===
= {{wip|Reporting issues}} =
To create a new test you ideally need a webpage(s) that performs the test(s) desired. The results can be a simple alert.
 
Once you've got that, just talk to folks in #perftest (:davehunt, :rwood, :igoldan) and they will handle the majority of the [[/Misc#Adding_a_new_test|steps required]].
 
=== Bugs ===
* [https://bugzilla.mozilla.org/buglist.cgi?resolution=---&query_format=advanced&component=Talos&product=Testing Talos bugs] are filed under [https://bugzilla.mozilla.org/enter_bug.cgi?product=Testing&component=Talos&op_sys=All&rep_platorm=All Testing/Talos], such as requests for new tests or repairs to the talos code itself.
* [https://bugzilla.mozilla.org/buglist.cgi?resolution=---&query_format=advanced&component=Talos&product=Testing Talos bugs] are filed under [https://bugzilla.mozilla.org/enter_bug.cgi?product=Testing&component=Talos&op_sys=All&rep_platorm=All Testing/Talos], such as requests for new tests or repairs to the talos code itself.
* Perfherder bugs are filed under [https://bugzilla.mozilla.org/enter_bug.cgi?product=Tree%20Management&component=Perfherder].
* Perfherder bugs are filed under [https://bugzilla.mozilla.org/enter_bug.cgi?product=Tree%20Management&component=Perfherder].
* Talos machine maintenance bugs are filed under [https://bugzilla.mozilla.org/enter_bug.cgi?product=mozilla.org mozilla.org/Release Engineering], such as bugs having to do with the hardware that talos is run on or requests to run extra talos tests against a given build.
* Talos machine maintenance bugs are filed under [https://bugzilla.mozilla.org/enter_bug.cgi?product=mozilla.org mozilla.org/Release Engineering], such as bugs having to do with the hardware that talos is run on or requests to run extra talos tests against a given build.


=== Subpages ===
= Contributing =
* Definition of [[/Tests|tests]]
We welcome contributions. Please see [https://codetribute.mozilla.org/projects/automation?project%3DTalos codetribute] for some good first bugs to get you started!
* How to [[/Profiling|use Talos profiling]].
* How to [[/Running|Run Talos]], and how it is run in production.
** How to run Talos on [[Mobile/Fennec/Android#talos|mobile]]
* What our raw [[/Data|data]] looks like and what our formulas for filtering and summarization.
* History and other uses of [[/Misc|Talos]]


=== Blog posts about Talos ===
= Further reading =
* http://armenzg.blogspot.com/2011/12/taloszip-talosjson-and-you.html
* https://blog.mozilla.org/nfroyd/2012/09/19/looking-at-talos-differently/
* https://blog.mozilla.org/nfroyd/2012/09/19/looking-at-talos-differently/
* https://blog.mozilla.org/nfroyd/2012/10/05/looking-at-talos-differently-part-2/
* https://blog.mozilla.org/nfroyd/2012/10/05/looking-at-talos-differently-part-2/
* http://armenzg.blogspot.com/2011/12/taloszip-talosjson-and-you.html
* https://elvis314.wordpress.com/2012/03/12/reducing-the-noise-in-talos/
* http://ask.mozilla.org/question/45/how-can-i-know-if-my-push-to-try-regresses-talos-results/
* http://limpet.net/mbrubeck/2013/11/10/improving-regression-detection.html
* http://limpet.net/mbrubeck/2013/11/10/improving-regression-detection.html
Davehunt
Confirmed users
2,177

edits

Retrieved from "https://dev.wikimo.nonprod.webservices.mozgcp.net/Special:MobileDiff/1216245"

Navigation menu