# 35. Distributed Mailbox Listeners Configuration

Date: 2020-04-23

## Status

Proposed

Supercedes [26. Removing a configured additional MailboxListener](0026-removing-configured-additional-mailboxListeners.md)

Not implemented yet.

## Context

James enables a user to register additional mailbox listeners.

The distributed James server is handling mailbox event processing (mailboxListener execution) using a RabbitMQ work-queue
per listener.

Mailbox listeners can be registered to be triggered every time an event is generated by user interaction with their 
mailbox. They are being executed in a distributed fashion following the workqueue messaging pattern. The "group" is an 
attribute of the mailbox listener identifying to which work queue they belong.

More information about this component can be found in [ADR 0036](0037-eventbus.md).

Currently, mailbox listeners are determined by the guice bindings of the server and additional mailbox listeners defined
via configuration files.

While the configuration might be specific for each James server, what actually is defined in RabbitMQ is common. 
Heterogeneous configuration might then result in unpredictable RabbitMQ resource status. This was left as a limitation
of [26. Removing a configured additional MailboxListener](0026-removing-configured-additional-mailboxListeners.md).

## Decision

We need to centralize the definition of mailbox listeners.

An event sourcing system will track the configured mailbox listeners.

It will have the following commands:

 - **AddListener**: Add a given listener. This should be rejected if the group is already used.
 - **RemoveListener**: Remove a given listener.
 
Configuration changes are not supported. The administrator is expected to remove the given listener, then add it again.

It will have the following events:

 - **ListenerAdded**: A mailbox listener is added
 - **ListenerRemoved**: A mailbox listener is removed

A subscriber will react to these events to modify the RabbitMQ resource accordingly by adding queues, adding or removing
bindings.

This event sourcing system differs from the one defined in
[26. Removing a configured additional MailboxListener](0026-removing-configured-additional-mailboxListeners.md) by the
fact that we should also keep track of listener configuration.

Upon start, James will ensure the **configured mailbox listener event sourcing system** contains the guice injected 
listeners, and add them if missing (handling the RabbitMQ bindings by this mean), then starts the eventBus which will
consume the given queues.

If a listener is configured with a class unknown to James, the start-up fails and James starts in a degraded state 
allowing to unconfigure the faulty listener. This downgraded state will be described in a separate ADR and the link will
be updated here.

This differs from [26. Removing a configured additional MailboxListener](0026-removing-configured-additional-mailboxListeners.md)
by the fact we no longer need to register all listeners at once.

A WebAdmin endpoint will allow:

 - **to add a listener** to the one configured. Such a call:
    - Will fail if the listener class is not on the local classpath, or if the corresponding group already used within 
    the **configured mailbox listener aggregate**.
    - Upon success the listener is added to the **configured mailbox listener aggregate**, and the listener is 
    registered locally.
 - **to remove a listener**. Such a call:
    - Will fail if the listener is required by Guice bindings on the current server or if the listener is not configured.
    - Upon success, the listener is removed from the **configured mailbox listener aggregate**, and the listener is 
    unregistered locally.
    
A broadcast on the event bus will be attempted to propagate topology changes, by the mean of a common registrationKey 
to all nodes, a "TopologyChanged" event, and a mailbox listener starting the MailboxListeners on local node upon
topology changes. `registrationKey` concept is explained in [ADR 0036](0037-eventbus.md).
 
If a listener is added but is not in the classpath, an ERROR log is emitted. This can happen during a rolling upgrade,
which defines a new guice binding for a new mailbox listener. Events will still be emitted (and consumed by other James
servers) however a local James upgrade will be required to effectively be able to start processing these events. The 
binding will not need to be redefined.

We will also expose an endpoint listing the groups currently in use, and for each group the associated configuration, if 
any. This will query the **configured mailbox listener aggregate**.

We will introduce a health check to actually ensure that RabbitMQ resources match the configured listeners, and propose
a WebAdmin endpoint to add/remove bindings/queue in a similar fashion of what had been proposed in 
[26. Removing a configured additional MailboxListener](0026-removing-configured-additional-mailboxListeners.md). This 
can happen if the James server performing the listener registration fails to create the group/queue. This health check 
will also report if this James server does not succeed to run a given listener, for instance if its class is not on the 
classpath.

## Consequences

All products other than "Distributed James" are unchanged.

All the currently configured additional listeners will need to be registered.

The definition of mailbox listeners is thus centralized and we are not exposed to an heterogeneous configuration 
incident.

Mailbox listeners no longer required by guice will still need to be instantiable (even with empty content). They will 
be considered as additional listener thus requiring explicit admin unconfiguration, which will be mentioned in the 
related upgrade instructions. Read notes about [rolling upgrade scenarii](#rolling-upgrade-scenari).

[Deploying a new custom listener](#deploying-a-new-custom-listener) also describes how to deploy new custom listeners.

Integration tests relying on additional mailbox listeners of the distributed James product will require to be ported to 
perform additional mailbox listener registry with this WebAdmin endpoint. JMAP SpamAssassin, quota mailing tests are 
concerned.

## Notes

## Broadcast of topology changes

### Rolling upgrade scenarii

During a rolling upgrade, the james version is heterogeneous across the cluster, and so might be the mailbox listeners
required at the Guice level.

**case 1**: James Server version 1 does not require listener A, James server version 2 requires listener A.

Since listener A is registered, James server version 1 cannot be rebooted without being upgraded first. (As listener A 
cannot be instantiated)

**case 2**: James Server version 1 requires listener A, James server version 2 does not require listener A.

Upgrading to James version 2 means that listener A is still registered as an additional listener, it needs to be 
manually unconfigured once the rolling upgrade finished. Which is acceptable in upgrade instruction. We need to make 
sure the listeners could still be instantiated (even with empty code) for a transition period.

## Deploying a new custom listener

Given a new custom listener, not yet deployed in Distributed James cluster,

To deploy it, an admin needs to follow these steps:

 - Add the jar in `extension-jars` folder for each James server
     - As `extension-jars` is read at instantiation time, no reboot is required to instantiate the new listener.
 - Call the webadmin endpoint alongside with listener specific configuration to enable the given custom listener. 
The bindings for the new listener will be created and a listener will be consuming its queue on the James server that 
had been treating the request.
 - Broadcast of topology changes will ensure the new custom additional mailbox listener will then be instantiated 
everywhere without a reboot.
