# Upgrade instructions

This document covers every changes an Ops needs to be aware of when running James.

The following procedures are to take as it, and the Apache Software Foundation, nor its contributors, can not be 
responsible for any damages generated by following the below procedures.

Before performing these operations, you should ensure to have the skills to conduct the operations, and you should read other
software documentation. Do not follow this guide blindly!

## Unreleased

Note: this section is in progress. It will be updated during all the development process until the release.

Changes to apply between 3.3.x and 3.4.x will be reported here.

Change list:

 - [Upgrade to ElasticSearch 6.3](#upgrade-to-elasticsearch-63)
 - [Enqueuing several times a mail with the same name](#enqueuing-several-times-a-mail-with-the-same-name)
 - [RabbitMQ Mail Queue with multiple specific headers for a single recipient](#rabbitmq-mail-queue-with-multiple-specific-headers-for-a-single-recipient)

#### RabbitMQ Mail Queue with multiple specific headers for a single recipient

Date: 31/07/2019

SHA-1: f19642aef4a67cd6675f66278792ba4aa85d6d6e

JIRA: https://issues.apache.org/jira/browse/JAMES-2850

Concerned products: (experimental) RabbitMQ MailQueue

RabbitMQ MailQueue projection in Cassandra relies on a map allowing a single specific header per recipient header. This
limitation causes rejection of emails with multiple per-recipient headers (which happens when forwarding to a remote server
a mail checked against SpamAssassin).

In order to fix this issue, the structure of the underlying table was updated. A new table is created upon the first startup
following the upgrade, and the old one is ignored.

Impact: the mails enqueued before the update can not be browsed nor removed from the queue after the update.

Recommendation: Conduct this update with an empty mail queue.

To do so:

 - Given a distributed Guice James server
 - Stop incoming traffic
 - Monitor the mailQueues and wait them to be empty (spool & outgoing)
 - Once empty, upgrade James server and re-enable incoming traffic.

#### Enqueuing several times a mail with the same name

Date: XX/06/2019

SHA-1: XXXXX

JIRA: https://issues.apache.org/jira/browse/JAMES-2794

Concerned products: (experimental) RabbitMQ MailQueue

RabbitMQ mail queue combines RabbitMQ with projections in Cassandra to offer advanced management capabilities expected from
a mail queue (browse, delete, size, clear). In these projections, the mails are identified by there name. Thus enqueuing a
mail that had already been processed will lead the given email to be considered already deleted and it will be discarded 
and lost.

This is an issue, as several other components build features around submitting a mail several time with the name. 

For instance:

 - MailRepository reprocessing
 - RemoteDelivery bouncing under some configurations
 - RecipientRewriteTable rewriting to a remote server

We thus changed the table structure of RabbitMQ mail queue projections to be built around an EnqueueId. This additional 
level of indirection allows several enqueues with the same name.

Upgrade to the newest James server needs to be performed with an empty MailQueue.

To do so:

 - Given a distributed Guice James server
 - Stop incoming traffic
 - Monitor the mailQueues and wait them to be empty (spool & outgoing)
 - Once empty, upgrade James server and re-enable incoming traffic.

#### Upgrade to ElasticSearch 6.3

Date: 27/05/2019

SHA-1: bbdf88e56d7a22fe92e1360ef563004f3bc0dd98

JIRA: https://issues.apache.org/jira/browse/JAMES-2766

Concerned products: (experimental) Cassandra-guice products.

In version 3.3.0 indexing for the Cassandra product was handled using ElasticSearch 2.2 released on the 31 march 2016. Some major upgrades had been included in recent ElasticSearch version.

Note that ElasticSearch APIs had been undergoing some major changes, making a smooth migration hard to provide. We proposed 2 migration strategies. A
simple one leading to major search inconsistencies in the process, and another one mitigating these inconsistencies (but getting rid of them).

#### Configuration changes

ElasticSearch 6 driver is relying on the high-level REST client and no more on the internal transport protocol.

Thus, you need to update your configuration files accordingly:

In `elasticsearch.properties` modify the `elasticsearch.port` properties to reference the HTTP port of your ElasticSearch 
nodes (**9200** by default instead of the previous default value of **9300**).

##### Simple strategy

Procedure:
 - From a running James 3.3.0 cluster connected to a running ElacticSearch 2.2 cluster
 - Start an empty ElasticSearch 6.3 cluster
 - Shutdown James 3.3.0 cluster and start a James 3.4.0 cluster connected to ElasticSearch 6.3
 - Search result will then be empty and thus innacurate
 - Thus trigger a [Full ReIndexing](https://james.apache.org/server/manage-webadmin.html#ReIndexing_all_mails) to restore search consistency.

Keep in mind that full reIndexing needs to process all users email and thus can be slow.

Obviously this approach trades search consistency against ease of migration.

If search consistency during the migration is important for you, consider the next approach

##### Strategy for minimizing search inconsistency during the migration

Procedure:
 - From a running James 3.3.0 cluster connected to a running ElacticSearch 2.2 cluster
 - Start an empty ElasticSearch 6.3 cluster
 - Start a James 3.4.0 cluster connected to ElasticSearch 6.3 cluster as well as the Cassandra source of trust database. Traffic should be directed to the James 3.3.0 cluster.
 - Trigger an offline [Full ReIndexing](https://james.apache.org/server/manage-webadmin.html#ReIndexing_all_mails) on the James 3.4.0 cluster
 - Once done, direct the traffic to the James 3.4.0 cluster, and dispose the James 3.3.0 cluster as well as the ElasticSearch 2.2 cluster
 - Search result will omit changes that took place during the switching process (starting from the reIndexing start)
 - Thus trigger a [Full ReIndexing](https://james.apache.org/server/manage-webadmin.html#ReIndexing_all_mails) to restore search consistency.

Keep in mind that full reIndexing needs to process all users email and thus can be slow.

## 3.3.0 version

Changes to apply between 3.2.0 and 3.3.0 had been reported here.

Change list:

 - [Changes to the MailboxListener API](#changes-to-the-mailboxlistener-api)
 - [Changes in WebAdmin reIndexing API](#changes-in-webadmin-reindexing-api)
 - [Rename KEY column in JAMES_MAILBOX_ANNOTATION table](#james-mailbox-annotation)
 - [Mailet API changes](#mailet-api-changes)

### Changes to the MailboxListener API

#### Persistent MailboxId for MailDir

Date: 30/11/2018

SHA-1: 7e32da51a29bee1c732b2b13708bb4b986140119

JIRA: https://issues.apache.org/jira/browse/MAILBOX-292

MailboxId are now persisted in a `james-mailboxId` file. This file is created on the fly, so no action is required for users relying on
the MailDir mailbox.

#### Registration by MailboxId

Date: 30/11/2018

SHA-1: d9bcebc7dd546bd5f11f3d9b496491e7c9042fe2

JIRA: https://issues.apache.org/jira/browse/MAILBOX-354

Only user written components performing MailboxListener registration will be affected.

The MailboxPath is mutable and thus can be changed upon mailbox rename. This leads to significantly complex code with possible inconsistency windows.

Using the mailboxId, which is immutable, solves these issues.

### Changes in WebAdmin reIndexing API

Date: 05/12/2018

SHA-1: 985b9a4a75bfa75c331cba6cbf835c043185dbdb

JIRA: https://issues.apache.org/jira/browse/JAMES-2555

We made this API introduced in James 3.2.0 a bit more REST friendly. If you developed tools using this API, you will need to update them.

For more details please refer to [the latest WebAdmin documentation](https://github.com/apache/james-project/blob/master/src/site/markdown/server/manage-webadmin.md#ReIndexing).

### Rename KEY column in JAMES_MAILBOX_ANNOTATION table

Date: 19/12/2018

SHA-1: e25967664538be18ec29f47e73e661bdf29da41f

JIRA: https://issues.apache.org/jira/projects/MAILBOX/issues/MAILBOX-356

Required: Yes

Concerned products: all JPA related products

#### Upgrade procedure

Rename `KEY` column in `JAMES_MAILBOX_ANNOTATION` table. The syntax is:

##### In MySQL
```
ALTER TABLE JAMES_MAILBOX_ANNOTATION CHANGE KEY ANNOTATION_KEY varchar(200);
```

##### In MariaDB
```
ALTER TABLE JAMES_MAILBOX_ANNOTATION CHANGE COLUMN KEY ANNOTATION_KEY varchar(200);
```

_or the syntax corresponding to your database._

### Mailet API changes
In order to allow safe serialization and strong typing `org.apache.mailet.Mail` have changed.

These methods have been deprecated and replaced:

 * `getSender()` in favor of `getMaybeSender()`
 * `getAttribute(String)` in favor of `getAttribute(AttributeName)`
 * `setAttribute(String, Serializable)` in favor of `setAttribute(Attribute)`
 * `removeAttribute(String)` in favor of `removeAttribute(AttributeName)`
 * `getAttributeNames()` in favor of `attributeNames()` and `attributesMap()`

Some plain-string `AttributeName` have also been replaced:

  * `SMTP_AUTH_USER_ATTRIBUTE_NAME` in favor of `SMTP_AUTH_USER`
  * `MAILET_ERROR_ATTRIBUTE_NAME` in favor of `MAILET_ERROR`
  * `SENT_BY_MAILET` in favor of `SENT_BY_MAILET_ATTRIBUTE`'s name, it is recommended to directly set the `Attribute`.

## 3.2.0 version

Changes to apply between 3.1.0 and 3.2.0 had been reported here.

Changelist:

 - [JMAPFiltering mailet is required for JMAP capable servers](#jmapfiltering-mailet-is-required-for-jmap-capable-servers)
 - [Cassandra 3.11.3 upgrade](#cassandra-3113-upgrade)

### Noticeable changes in Mail API: Mail::getMaybeSender

Date: 31/10/2018

SHA-1: 485406252d82c2d23a4078c76b26d6fc8973bbd7

JIRA: https://issues.apache.org/jira/browse/JAMES-2557

Required: Yes

Concerned products: User developed extensions - mailet/matcher

As part of the SMTP protocol, a mail can be sent without sender. This was represented implicitly in James by a potentially null MailAddress
(`null` or `MailAddress.nullSender()`). This means that mailet/matcher implementers needs to be aware, and handle these cases. This implicit
handling makes nullSender hard to work with, and prooved to be error prone as part of the 3.2.0 development process.

Hence we propose an alternative API returning a `MaybeSender` object, requiring the caller to explicitly handle missing sender.

`Mail::getSender` had then been deprecated. We strongly encourage our users to rely on `Mail::getMaybeSender`.

Note: thanks to java-8 default API methods, this is not a breaking change.

### JMAPFiltering mailet is required for JMAP capable servers

Date: 30/08/2018

SHA-1: 9ba6a1dd270f99735c7f9d3d4b2adb5076583c10

JIRA: https://issues.apache.org/jira/browse/JAMES-2529

Required: Yes

Concerned products: Cassandra Guice products

This mailet allow users filtering rules to be applied for incoming emails.

#### Upgrade procedure

Add this line before the `LocalDelivery` mailet of your `transport` processor:

```
<mailet match="RecipientIsLocal" class="org.apache.james.jmap.mailet.filter.JMAPFiltering"/>
```

### Cassandra 3.11.3 upgrade

Date: 03/08/2018

SHA-1: de0fa8a3df69f50cbc0684dfb1b911ad497856d7

JIRA: https://issues.apache.org/jira/browse/JAMES-2514

Required: Yes

Concerned products: Cassandra Guice products

James Cassandra Guice now officially uses Cassandra 3.11.3 as a storage backend. After performing the upgrade, the team
did perform some breaking changes, detailed below. James Cassandra Guice products are no more tested against Cassandra 2.2.x. Thus we strongly
advise our users to upgrade.

#### Changes not compatible with Cassandra 2.2.x

Replace in default compaction strategies "DateTieredCompactionStrategy" by "TimeWindowCompactionStrategy".

This means you can no more start James on top of an empty Cassandra 2.2.x cluster, but existing deployments should not be impacted.

#### Upgrade procedure

We will assume that Cassandra had been installed with a debian package. Upgrade procedure stays similar in other cases.


1. Update Cassandra dists in `/etc/apt/sources.list.d/cassandra.list` to match 311x repository

```
deb http://www.apache.org/dist/cassandra/debian 311x main
```


2. Update Cassandra

```
$ apt-get update
$ apt-get install cassandra=3.11.3
```

3. Correct the configuration

Edit /etc/cassandra/cassandra.yaml and ensure to really specify the interface cassandra is listening on as seeds.

4. ReStart Cassandra

4.1. Drain data & stop

```
$ nodetool drain
$ nodetool stop
```

4.2. start Cassandra

5. Upgrade SSTable (live update, performance degradation to expect)

```
$ nodetool upgradesstables apache_james
```
