Build:TryServer:Suggestions

Try Server Suggestions and Feature Requests

• upload script: allow multiple patches
• upload script: use reference to bugzilla attachments instead of uploading a patch
  • Direct integration into bugzilla? "Click here to test patch"?
• upload script: allow users to reference 1..N bug numbers as part of the build description
• upload script: allow users to check an "email me when this build is ready" box
• try script: email users who checked the "email me when this build is ready" box
• upload script: ability to see default .mozconfig
• allow patches compressed with bzip2 or gzip
• copy build logs (and maybe input patches) to output download directory
• give developers an option to make the try-server "rebase and push to mozilla-central" if the patch is green on try-server
  • or starred as "effectively green"
• Display the last green hg id somewhere (this is bug 570859).
• Ability to just build installer/zip/dmg/tar without uploading symbols, running tests, etc.
• Compare talos numbers relative to mozilla-central numbers in proximity hg DAG-wise, see http://blog.mozilla.com/axel/2010/06/07/trying-talos-on-node-js/ for a proof of concept.
• (roc) We have a problem that try server doesn't degrade gracefully. When it's loaded, you lose results, or the results lose relevance when they're really old and far away from where trunk is now. So we get into situations where people are pushing but all the work done for the push is basically wasted. In these kinds of situations we need some kind of admission control so that fewer people can push but those who can actually get useful results.
• (mrbkap) The ability to cancel a try server run once it's started would be useful. Especially after seeing some failures, I often notice that my patch is wrong in some way, and I don't want to continue to receive e-mails or see more results until I fix my patch and push again. This might also help me use less resources whenever I push to try.

Preventing you from using tryserver effectively for your needs

• (bz) Being able to actually get results in sane amounts of time. It's now been over 24 hours since I pushed a patch, and I have yet to see a single mac test result, Linux debug test result, 32-bit Linux optimized test result, anything at all on Win7. The Win2k3 optimized build died with an infrastructure error... In other words, a bunch of machine time was wasted, and I still can't get the data I wanted.
• (roc) Understanding tryserver performance results is hard. The tbpl comparator is somewhat helpful, but only if you can identify a changeset on the same page of results that would be a good baseline for comparison, and that is error-prone and often impossible. You also don't get a fine-grained breakdown of test results. I generally end up screen-scraping the old-style Tinderbox with a script I wrote. The process is even slower and harder when your test results appear over several days, if at all (see above).
• (karl) Sometimes tinderbox loses results of completed runs. I get emails telling me that they've completed but they do not show up on the tinderbox. A link in the email to the log (or even some other way to find the log) would be helpful. One time some results showed up temporarily in tbpl but then disappeared (and could not, or no longer, be found on tinderbox).