Auto-tools/Projects/Pulse

Introducing Pulse

Pulse is a managed RabbitMQ cluster designed to provide loose coupling between automation and infrastructure tools. The goal of Pulse is to add visibility to Mozilla's tools and systems and to eliminate polling and other brittle methods of scraping data. This allows more robust, dynamic, and informative tools.

Pulse is available at pulse.mozilla.org:5671 (AMQP over SSL). It is hosted by CloudAMQP.

PulseGuardian is a tool that manages Pulse's users and queues (and eventually exchanges). It is available at https://pulseguardian.mozilla.org and hosted by Heroku.

We have a discussion forum available via the standard trio of USENET newsgroup, mailing list, and Google Group.

File bugs under Webtools :: Pulse. We don't have a separate component for PulseGuardian; rather, we just start the summaries with "[PulseGuardian]".

Also see the Pulse Inspector web app, which displays Pulse messages in real time, and the (manually updated) list of Pulse exchanges.

System Description

Pulse isn't any one thing. At its heart, it is a RabbitMQ system with a particular configuration and a set of conventions for using it along with a management tool, PulseGuardian, to make Pulse as automated and self-serve as possible. Pulse follows the pub-sub pattern, in which publishers send messages to topic exchanges, and consumers create queues bound to these exchanges in order to subscribe to the publishers' messages. In general, publishers create and own exchanges, and consumers create and own queues.

Specification

Pulse is a managed AMQP 0-9-1 service with RabbitMQ extensions for publishing messages from Mozilla infrastructure. The aim is provide hooks that subscribers can use to integrate and extend Mozilla infrastructure.

Authentication

Pulse credentials are managed and issued by PulseGuardian, available at https://pulseguardian.mozilla.org. This service SHALL issue an accessToken for any clientId that is registered with authorized email address. The accessToken is strictly secret and MUST NOT be shared publicly. The clientId is not secret. When establishing an AMQP connection, the clientId and accessToken MUST be used as username and password, respectively.

Authorized Users

Pulse is intended to be open to all Mozillians who want to extend or integrate with Mozilla infrastructure. To guard against abuse PulseGuardian users MUST authenticate via Persona. PulseGuardian SHOULD verify that users have a vouched Mozillians profile.

Publishers

Publishers MUST name exchanges in the form exchange/<clientId>/<name> where clientId is the userid used to bind/connect to the server. Attempts to name an exchange otherwise SHALL result in an authorization error. Exchanges MUST be topic exchanges and they MUST be declared durable.

Messages MUST contain a UTF-8-encoded JSON payload, and their Content-Type MUST be application/json. Messages SHOULD NOT be larger than 8 kB; deviations may be feasible for low-traffic exchanges. Messages MUST NOT contain secret or sensitive information; all exchanges and messages SHALL be considered public.

A message SHOULD carry a routing key, in which fields have a fixed index from the left. Additionally, a message MAY be cced to multiple routing keys, using the RabbitMQ Sender-selected Distribution extension.

Messages SHOULD be durable and SHOULD be published over RabbitMQ confirm-publish channels. Otherwise, the documentation MUST clearly reflect that messages from the given exchange do not exhibit deliver at-least-once semantics.

Subscribers

Subscribers MUST name queues in the form queue/<clientId>/<name>; attempts to name a queue otherwise SHALL result in an authorization error. Queues MAY consume from any exchange prefixed exchange/; attempts to consume from any other exchange SHALL result in an authentication error.

Subscribers MAY limit the size of their queues using the RabbitMQ Queue Length Limit extension. Subscribers MUST NOT let their queues grow unbounded; if left unattended, Pulse SHALL notify the owner by email. Additionally, Pulse MAY delete a queue which exceeds defined limits. Subscribers SHOULD specify a prefetch limit using the RabbitMQ Consumer Prefetch limit extension.

Subscribers SHOULD use either durable queues or auto-delete queues. Implementors are recommended to aim for deliver-at-least-once semantics.

Appendix A: Everything in Bullet Points

This is a summary of the above.

Pulse:

• MUST offer registration at pulse.mozilla.org
• MUST support AMQP 0-9-1 and these RabbitMQ extensions:
  • Confirms
  • Consumer Prefetch
  • Queue Length Limit
  • Sender-selected Distribution
• SHOULD exhibit deliver-at-least-once semantics
• MAY delete queues that grows beyond Pulse defined limits
• SHALL notify owner by email when a queue grows close to Pulse-defined limits.

Publishers:

• SHOULD use confirm-publish channels

Exchanges:

• MUST be named exchange/<clientId>/<name>
• MUST be topic exchanges
• MUST be durable

Messages:

• MUST be UTF-8-encoded JSON
• MUST carry application/json as Content-Type
• SHOULD be durable
• SHOULD be less than 8 KiB (for good performance)
• MAY be CC'ed to multiple routing keys
• MUST NOT contain private or sensitive information
• SHOULD have a routing key where fields have a fixed index from the left

Subscribers:

• SHOULD specify a consumer prefetch limit

Queues:

• MUST be named queue/<clientId>/<name>
• MAY have a limited length
• MUST not grow unbounded

Let's Use It

There are currently two Pulse clients available. Please note that you can also connect to Pulse in other languages, provided you have an AMQP 0.9.1 library that will let you interact with AMQP exchanges. See https://github.com/rabbitmq/rabbitmq-tutorials#languages for example.

Python Pulse client library

The mozillapulse Python package provides classes for existing publishers, consumers, and messages so you can quickly build Pulse applications. See the README to get started (although note that the test publisher is currently offline; see bug 1218976. You can use another consumer, e.g. BuildConsumer, to verify your setup.).

This library is somewhat inflexible, however, and should be rewritten. One idea is to turn TaskCluster's Python client into a standalone package.

Go (golang) Pulse client library

This can be found here:

• http://taskcluster.github.io/pulse-go/

Extensions for TaskCluster exchanges here (see section "AMQP APIs"):

• http://taskcluster.github.io/taskcluster-client-go/

Contributing

To set up a local system for development, see the HACKING.md file included in the mozillapulse source.

The main Pulse library (mozillapulse) and publisher shims (pulseshims) are written in Python, although there is also a Go library as mentioned in the section above. We also want to provide a canonical JavaScript library at some point. To hack on the main Pulse library, you should be comfortable in Python, and it's helpful to understand the basics of AMQP. Knowledge of kombu is also useful.

To hack on PulseGuardian, you should know some Python and JavaScript. Experience with Flask, SQLAlchemy, and RabbitMQ are useful, but you can probably learn what you need as you fix bugs.

Feel free to stop by #pulse or #ateam with questions!

Here is the list of open, unassigned, mentored Pulse and PulseGuardian bugs to get you started.

Full Query

ID	Summary	Priority	Status
1071947	Support for notifying mailing lists	P5	NEW
1079523	[PulseGuardian] List exchanges with ability to delete	P5	NEW
1084706	API for listing queues by user (useful for bulk deletion after tests)	P5	NEW
1215520	[PulseGuardian] Handle auth failures gracefully	P5	NEW
1298929	Disaster Recovery plan	--	NEW
1346304	[PulseGuardian] Randomly generate passwords rather than prompting for them	P5	NEW
1347088	[PulseGuardian] "Queue is overgrowing" email needs adjustment for unbounded queues	--	NEW
1347093	[PulseGuardian] Add UI for allowing admins to mark queues as unbounded	--	NEW
1434385	[PulseGuardian] "My RabbitMQ Accounts" shows unowned accounts as directly belonging to admins	--	NEW
1509429	[PulseGuardian] JS errorMessage() function doesn't exist	--	NEW
1536698	implement additional alerts for pulse.m.o to check for a large volume of unacked alarms	--	NEW
1609989	pulseguardian cannot delete exclusive queues, doesn't log about it	--	NEW
1663374	Please disable mtrinkala's Pulse Guardian account	--	NEW
1875132	queues with high amounts of unconsumed messages can take down pulse	--	NEW
1875328	upgrade to latest rabbitmq version	--	NEW
1903235	Investigate isolation of taskcluster exchanges/queues from the rest by vhost	--	NEW

16 Total; 16 Open (100%); 0 Resolved (0%); 0 Verified (0%);

Once you have your feet wet and are ready to take on a more involved project, here is a list of all current Pulse bugs:

Full Query

ID	Summary	Priority	Status
1071947	Support for notifying mailing lists	P5	NEW
1079523	[PulseGuardian] List exchanges with ability to delete	P5	NEW
1084706	API for listing queues by user (useful for bulk deletion after tests)	P5	NEW
1215520	[PulseGuardian] Handle auth failures gracefully	P5	NEW
1298929	Disaster Recovery plan	--	NEW
1346304	[PulseGuardian] Randomly generate passwords rather than prompting for them	P5	NEW
1347088	[PulseGuardian] "Queue is overgrowing" email needs adjustment for unbounded queues	--	NEW
1347093	[PulseGuardian] Add UI for allowing admins to mark queues as unbounded	--	NEW
1434385	[PulseGuardian] "My RabbitMQ Accounts" shows unowned accounts as directly belonging to admins	--	NEW
1509429	[PulseGuardian] JS errorMessage() function doesn't exist	--	NEW
1536698	implement additional alerts for pulse.m.o to check for a large volume of unacked alarms	--	NEW
1609989	pulseguardian cannot delete exclusive queues, doesn't log about it	--	NEW
1663374	Please disable mtrinkala's Pulse Guardian account	--	NEW
1875132	queues with high amounts of unconsumed messages can take down pulse	--	NEW
1875328	upgrade to latest rabbitmq version	--	NEW
1903235	Investigate isolation of taskcluster exchanges/queues from the rest by vhost	--	NEW

16 Total; 16 Open (100%); 0 Resolved (0%); 0 Verified (0%);

For mentored bugs, we use the User Story to provide a link back to this page, as well as any extra information for contributors, such as required knowledge or tools. The basic text for mentored bugs should be "This is a mentored Pulse bug. For general information on Pulse, see https://wiki.mozilla.org/Auto-tools/Projects/Pulse, which includes a section on Contributing." An example of extra text is "This bug also requires you to have a working mail server."

Road Map

See the prioritized bug list for all open issues and feature requests.

Security Model

This is summarized in the formal Pulse specification above. What follows is the rationale and some technical implementation notes.

In order to have a reliable, well behaved system, the following assertions will need to be true.

• All users, publishers and consumers alike, must have their own accounts (no guest/public users).
• Only publishers should be able to declare exchanges.
• Only the publisher user account associated with a particular vhost should be allowed to publish messages to exchanges in the vhost. In other words, exactly one user account should be allowed to publish messages within a given vhost.
• Only the user that created a particular queue should be allowed to consume from it.

Since exchange and queue permissions go together, we'll need exchange and queue naming conventions mixed with restrictive permissions. Each user will be restricted to a particular exchange and queue naming prefix. Many users will be either consumers or publishers, but for simplicity, each user can do both. Users will have full permissions on "^exchange/<username>/.*$" and "^queue/<username>/.*$". They will also have read permissions to exchange/*. This will both prevent users from writing to other users' exchanges as well as prevent them from consuming from other users' queues. For convenience, if a consumer creates a nondurable queue, mozillapulse can assign a random suffix to the user's standard queue name prefix, i.e. queue/<username>/<random string>, since the user wouldn't be able to create nor access a completely random server-assigned name.

Note that this doesn't prevent a consumer from creating an exchange named as a queue, since the permission model doesn't distinguish between queues and exchanges, and consumers need the ability to create queues. This is not particularly problematic, since no one would have permission to use that exchange.

With this security model, we technically don't really need vhosts, since the names of the queues and exchanges the users can use are so specific. There may still be a benefit in allowing apps to use the same queue name for different exchanges, though, which would be possible if each exchange had its own vhost. The downside is that you cannot specify "all vhosts" when setting a user's permissions, so they would either have to list all vhosts they want to use when creating the user in PulseGuardian, and be able to update that list later, or PulseGuardian or some other app would have to automatically add new permissions to all users when a vhost is created.

Admin Procedures

dustin and the taskcluster team have access to the Pulse cluster on CloudAMQP and the following related services:

• PulseGuardian should be deleting queues that are too long. If you need to manually delete a queue, use the Management UI. Try to ping the queue owner first before killing if possible.

To upgrade a ssl certificate on pulse.mozilla.org

Open a bug with IT to generate a new certificate https://bugzilla.mozilla.org/enter_bug.cgi?product=Infrastructure%20%26%20Operations&component=SSL%20Certificates See bug 1532325 for an example.

IT needs to email support@cloudamqp.com with the new cert. The cloudampq support team will install it on all our of cloudampq nodes. After it has been installed, you can login to the administrative start the nodes one by one which will not result any downtime. (Ensure you wait for the node to restart before restarting another one.) Verify that the certs are installed on the nodes

• https://sslanalyzer.comodoca.com/?url=pulse.mozilla.org
• https://sslanalyzer.comodoca.com/?url=orange-antelope-01.rmq.cloudamqp.com
• https://sslanalyzer.comodoca.com/?url=orange-antelope-02.rmq.cloudamqp.com
• https://sslanalyzer.comodoca.com/?url=orange-antelope-01.rmq.cloudamqp.com

This should show the dates for the new certificate and that the cert is trusted by Mozilla and Microsoft.

Cloudampq updated their web page since we last did this so that you should be able to upload the cert yourself and have it propagate. See the admin console under "Certificate".

More reading

• Slides from a presentation on Pulse.
• Update on Pulse from 2015/02/16.

LegNeato also wrote several blog posts on Pulse as he was building it. They contain some more background if you're really interested. They are linked below, in chronological order.

• Mozilla Pulse and RabbitMQ
• Push notifications for Bugzilla
  • This idea has been shelved for the time being, both due to security issues and performance. See BMO/ChangeNotificationSystem for an alternative.
• bugzilla-amqp is now buzgilla-push, supports the STOMP protocol
  • See above note. Just keeping these links for interest's sake.
• What’s up with Pulse?
• GitHub AMQP integration service hook live!
  • Tangentially related to Pulse.
• Lots of Pulse changes, out of prototype mode this week!
• Another update on Pulse