Open-Xchange - User contributions [en]

My.cnf

2019-02-14T14:45:12Z

Dominik.epple: /* Functional items */ Fix typo.

== Introduction ==

This page lists some performance tuning parameters which we recommend for tuning MySQL database services used for Open-Xchange installations.

We cannot guarantee this is an exhaustive list of required settings. So treat this list of tunings as probably required, but not necessarily sufficient settings for optimal MySQL performance.

Furthermore, as MySQL changes over time, settings which have been correct as of the time of writing may become incorrect later.

However, this list of settings is the result of internal performance testing and real world customer feedback, so it should be valid to some extent.

In the end, proper configuration of the database service for performance, but also consistency, durability and high availability is in the responsibility of the customer.

=== Performance items ===

* You should adjust the <code>innodb_buffer_pool_size</code> parameter for reasonable memory usage. Our DB sizing is mainly memory-driven and this is where most of the memory goes. Our standard DB machine sizing assumption is 32 GB if MySQL dedicated memory on a 48 GB total memory machine. On such a machine, you would configure 32 GB for the innodb_buffer_pool size, being aware that MySQL does also require memory for other things, in particular there are some also per-connection related memory spendings, which can become substantial if you allow for a lot of maximum concurrent connections. Please watch your memory configuration carefully, use monitoring and tools like mysqltuner.pl.

* On bigger installations you should use <code>innodb_file_per_table = 1</code>, which is creating single files instead of one big blob. If you change this parameter after the database initialization you have to recreate (like dump/drop and re-import) the tables.

* It can help to put different parts of the mysql datadir (iblog, ibdata) on different filesystems / storage devices. This depends on your infrastructure. Settings herefore are <code>datadir</code>, <code>innodb_data_home_dir</code>, <code>innodb_log_group_home_dir</code>.

* If your storage is fast (handle a lot of IOPS), you may want to adjust the <code>innodb_io_capacity</code> setting, which defines a limit for the IOPS MySQL will create. The default is 200, which is sensible for single spindle disks. But if you have storage appliances with a lot of fast SAS drives, or even SSDs, this limit can be increased greatly.

=== Functional items ===

* Query cache is to be switched off; as we found in our own benchmarks and as backed up by upstreams, this hurts performance in load situations with high concurrency. [https://dev.mysql.com/doc/refman/5.7/en/query-cache.htm The query cache is deprecated as of MySQL 5.7.20, and is removed in MySQL 8.0], so we recommend also to switch that off.
* Starting with App Suite 7.10.0, <code>character_set_server</code> must be set to <code>utf8mb4</code> and <code>collation_server</code> to <code>utf8mb4_general_ci</code>. For older versions it must be <code>utf8</code> and <code>utf8_general_ci</code> respectively.
* Starting with MySQL 5.7 <code>innodb_strict_mode</code> must be disabled.
* Starting with MySQL 5.6 and MariaDB 10.1 <code>sql_mode</code> must be configured according to belows matrix.

==== SQL mode matrix ====

The default for the <code>sql_mode</code> setting changes regularly with MariaDB and MySQL releases and is not even consistent anymore between the two derivates. SQL modes affect how data and queries are handled at runtime. Enabling strict modes might lead to errors in terms of failing queries or even update tasks. We strongly recommend the following configuration to avoid according runtime errors:

{| class="wikitable"
|-
! scope="col"| sql_mode
! scope="col"| App Suite 7.8.4
! scope="col"| App Suite 7.10.0
|-
! scope="row"| MySQL 5.6
| <code>NO_ENGINE_SUBSTITUTION</code>
| <code>NO_ENGINE_SUBSTITUTION</code>
|-
! scope="row"| MySQL 5.7
| <code>NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION</code>
| <code>NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION,ONLY_FULL_GROUP_BY</code>
|-
! scope="row"| MariaDB 10.1
| <code>NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION</code>
| <code>NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION</code>
|-
! scope="row"| MariaDB 10.2
| <code>NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION</code>
| <code>NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION</code>
|-
|}

=== Sample config files ===

For easy deployment we recommend not to edit existing the existing my.cnf file, rather assume the distro provides sane settings for most of the items and override items where needed.

As in MySQL there is a [https://dev.mysql.com/doc/refman/5.7/en/option-files.html last instance wins] semantic in options file parsing, we propose to create a custom include directory "ox.conf.d", put our custom config files therein, and include this directory as latest directory in /etc/mysql/my.cnf.

To put in that directory, we have one main tuning file called tunings.cnf, one galera-related file if galera is in use, and one more galera host-specific file which contains per-host settings if using galera (separate in a file of its own for easier configuration management).

So, start with adding to the existing my.cnf at the very bottom:

!includedir /etc/mysql/ox.conf.d/

Create that directory and put in there the generic ox tunings file /etc/mysql/ox.conf.d/tunings.cnf:

[mysqld]
bind-address = *

#innodb_use_native_aio = 0

table_open_cache = 3072
table_definition_cache = 4096
max_heap_table_size = 64M
tmp_table_size = 64M
max_connections = 505
max_user_connections = 500
max_allowed_packet = 16M
thread_cache_size = 32
query_cache_size = 0
query_cache_type = 0
innodb_buffer_pool_size = 32G
# the default value in MySQL 5.6.6 and higher is 8 when innodb_buffer_pool_size is greater than or equal to 1GB. Otherwise, the default is 1.
innodb_buffer_pool_instances = 32
innodb_data_file_path = ibdata1:128M:autoextend
innodb_file_per_table = 1
# innodb_log_file_size should be 25% of the innodb_buffer_pool_size
innodb_log_file_size = 4GB
# default and recommended value is 2
innodb_log_files_in_group = 2
# adjust according to your storage
#innodb_io_capacity = 1000

# we are unsure about this setting. Newer versions of MariaDB seem to be fine with low (=1) settings for this value.
# Traditionally we encountered values up to 4x the number of cores.
# Default seems to be number of cores, so let's stick the default
# In the end, we need to leave this setting up to you: if you dont get full cpu utilization in cpu-bound situations, this might be a setting to increase.
#thread_pool_size = 32

binlog_cache_size = 1M
sync_binlog = 8
binlog_format = row

character_set_server = utf8mb4
collation_server = utf8mb4_general_ci

# This was default_table_type previous to MySQL 5.5
default_storage_engine = InnoDB

innodb_autoinc_lock_mode = 2

# keep until 5.6, deprecated later
innodb_locks_unsafe_for_binlog = 1

# we found this has huge impact on (galera) performance
# default (consistent) setting of 1 greatly severs performance
# in galera (or async master-slave) deployments, you might be ok with setting this to 0 or 2,
# assuming our consistency / availability comes from replication / other cluster nodes
innodb_flush_log_at_trx_commit = 0

# for performance testing systems, to not use excessive disk space
#expire_logs_days = 1

# MySQL 5.7.7 has changed the default to 1. Disable it explicitly to prevent from errors based on invalid data stored by former App Suite or MySQL versions.
innodb_strict_mode = 0

# The following value refers to App Suite 7.10 on top of MySQL 5.7. For other combinations see the sql mode matrix at http://oxpedia.org/wiki/index.php?title=My.cnf.
sql_mode = NO_ENGINE_SUBSTITUTION,NO_AUTO_CREATE_USER,ONLY_FULL_GROUP_BY

If using galera, use the following galera configuration file <code>/etc/mysql/ox.conf.d/wsrep.cnf</code>. See the comments in that file for values to be adjusted.

[mysqld]
# adjust for your distros SO location
wsrep_provider=/usr/lib/libgalera_smm.so

# this is the big winner and enables us to switch off OX's replication monitor
wsrep_sync_wait=1

# pick a unique cluster name
wsrep_cluster_name=devcluster
# adjust for your IPs / hostnames
wsrep_cluster_address=gcomm://10.20.29.68,10.20.29.69,10.20.29.70

# put this in host.cnf
#wsrep_node_name=...
#wsrep_node_address=...

# For some MariaDB versions, xtrabackup-v2 no longer works, instead use "mariabackup"
# (needs to be installed separately, e.g. via the mariadb-backup-10.2 package)
# see upstream documentation for details:
# https://mariadb.com/kb/en/library/getting-started-with-mariadb-galera-cluster/#xtrabackup
#
# wsrep_sst_method=mariabackup
wsrep_sst_method=xtrabackup-v2

# wsrep_sst_auth if of format username:password
# pick whatever you configured on the donor node
wsrep_sst_auth=sstuser:...

# galera-specific tunings
wsrep_slave_threads = 32

# finally, enable wsrep: required for some MariaDB versions
wsrep_on=ON -- Enable wsrep replication (MariaDB starting 10.1.1)

Galera-related host-specific settings go in /etc/mysql/ox.conf.d/host.cnf:

[mysqld]
# the nodes hostname
wsrep_node_name=...
# and the IP of the wsrep relevant interface, if multiple
wsrep_node_address=10.20.29.68

Template:OXLoadBalancingClustering Database

2018-09-27T14:25:22Z

Dominik.epple: /* First node (continued) */

== Overview ==

You can choose between Galera or Master/Slave replication. We like to recommend to use Galera for higher redudancy, easier operations, und synchronous semantics (so you can run OX without our "replication monitor"). For POC or demo setups, a single standalone database setup might be sufficient.

== Standalone database setup ==

=== Preparations ===

Our configuration process includes wiping and reinitializing the datadir. This is usually not a problem in a fresh installation. If you want to upgrade an existing database server, please be prepared to wipe the datadir, i.e. take a <code>mysqldump</code> for later restoration into the properly configured master.

mysqldump --databases configdb oxdb_{5..14} > backup.sql

Be sure to verify the list of databases.

=== Installation ===

Note: the following list ist not an exclusive list or authorative statement about supported MySQL flavors / versions. Please consult the official support / system requirements statement.

Please follow the upstream docs for your preferred flavor to get the software installed on your system.

* MariaDB (10.1, 10.2): https://downloads.mariadb.org/
* Oracle MySQL Community Server (5.6, 5.7): https://dev.mysql.com/downloads/mysql/

Make sure to doublecheck the service is not running (or stop it) after installation as we need to perform some reconfigurations.

service mysql stop

=== Configuration ===

MySQL configuration advise is given in our [[My.cnf|MySQL configuration article]]. Please consult that page for configuration information and create configuration files as described there.

Some settings we recommend to change require that the database gets re-initialized. We assume you don't have data there (since we are covering a fresh install) or you have taken a backup for later restore as explained above in the Preparations section.

cd /var/lib/
mv mysql mysql.old.datadir
mkdir mysql
chown mysql.mysql mysql

# mariadb
mysql_install_db
# mariadb 10.2
mysql_install_db --user=mysql
# oracle 5.6
mysql_install_db -u mysql
# oracle 5.7
mysqld --initialize-insecure --user=mysql

(Don't be worried about the <code>insecure</code>, it just means we set the db root pw in the next steps.)

Start the service. The actual command depends on your OS and on the MySQL flavor.

service mysql start

Run <code>mysql_secure_installation</code> for a "secure by default" installation:

mysql_secure_installation

That tool will ask for the current root password (which is empty by default) and subsequently questions like:

Change the root password? [Y/n]
Remove anonymous users? [Y/n]
Disallow root login remotely? [Y/n]
Remove test database and access to it? [Y/n]
Reload privilege tables now? [Y/n]

You should answer all these questions with "yes".

Configure a strong password for the MySQL <code>root</code> user.

The further steps in this guide omit <code>-u -p</code> arguments to the MySQL client. Rather than passing them on the command line [https://dev.mysql.com/doc/refman/5.7/en/password-security-user.html] it is recommended to place the credentials in a file like <code>/root/.my.cnf</code> like

[client]
user=root
password=wip9Phae3Beijeed

Make sure the service is enabled by the OS's init system. The actual command depends on your OS and on the MySQL flavor.

systemctl enable mysql.service

You should now be able to restore your previously taken backup.

# If you took a dump for restore before
mysql < backup.sql

=== Configure OX to use with a standalone database ===

Not much special wisdom here. OX was designed to be used with master/slave databases, and a standalone master works just as well, if we register it as a master, and not registering a slave.

For the ConfigDB, <code>configdb.properties</code> allows configuration of a <code>writeUrl</code> (which is set to the correct values if you use <code>oxinstaller</code> with the correct argument <code>--configdb-writehost</code>).

The single database is then used for reading and writing.

For the individiual UserDBs, use <code>registerdatabase -m true</code>.

== Galera database setup ==

=== Preparations ===

Our configuration process includes wiping and reinitializing the datadir. This is usually not a problem in a fresh installation. If you want to upgrade an existing database to Galera cluster, please be prepared to wipe the datadir, i.e. take a <code>mysqldump</code> for later restoration into the properly configured master.

Depeding on the flavor of the current database, this can be something like

# mariadb or oracle mysql without GTIDs
mysqldump --databases configdb oxdb_{5..14} > backup.sql

# mysql 5.6 with GTIDs... we dont want GTIDs here
mysqldump --databases --set-gtid-purged=OFF configdb oxdb_{5..14} > backup.sql

Be sure to verify the list of databases.

=== Installation ===

Please follow the upstream docs for your preferred flavor to get the software installed on your system.

* Percona XtraDB Cluster (5.6, 5.7): https://www.percona.com/doc/percona-xtradb-cluster/LATEST/install/index.html
* MariaDB Galera Cluster (10.0, 10.1): https://mariadb.com/kb/en/library/getting-started-with-mariadb-galera-cluster/ (Note: with 10.0, socat is required, but not a package dependency, so you need to explicitly install also socat)

Make sure to doublecheck the service is not running (or stop it) after installation as we need to perform some reconfigurations.

service mysql stop

=== Configuration ===

Galera-specific MySQL configuration advise is included in our main [[My.cnf|MySQL configuration article]]. Please consult that page for configuration information.

That page suggests a setup were we add three custom config files to <code>/etc/mysql/ox.conf.d/</code>: <code>ox.cnf</code> for general tuning/sizing, <code>wsrep.cnf</code> for clusterwide galera configuration, and <code>host.cnf</code> for host-specific settings.

Adjust the general settings and tunings in <code>ox.cnf</code> according to your sizing etc.

Adjust <code>wsrep.cnf</code> to reflect local paths, cluster member addresses, etc.

Adjust <code>host.cnf</code> to give node-local IPs, etc.

Version-specific hints:

# percona 5.6: unknown variable 'pxc_strict_mode=ENFORCING' ... unset that one
# mariadb 10.1: add wsrep_on=ON
# mariadb 10.0 and 10.1: set wsrep_node_incoming_address=192.168.1.22:3306 in host.cnf, otherwise the status wsrep_incoming_addresses might not be shown correctly(?!)

Some settings we recommend to change require that the database gets re-initialized. We assume you don't have data there (since we are covering a fresh install) or you have taken a backup for later restore as explained above in the Preparations section.

cd /var/lib/
mv mysql mysql.old.datadir
mkdir mysql
chown mysql.mysql mysql

# mariadb 10.0 and 10.1
mysql_install_db
# mariadb 10.2
mysql_install_db --user=mysql
# percona 5.6
mysqld --user=mysql
# percona 5.7
mysqld --initialize-insecure --user=mysql

(Don't be worried about the <code>insecure</code>, it just means we set the db root pw in the next steps.)

=== Cluster startup ===

Typically on startup a Galera node tries to join a cluster, and if it fails, it will exit. Thus, when no cluster nodes are running, the first cluster node to be started needs to be told to not try to join a cluster, but rather bootstrap a new cluster. The exact arguments vary from version to version and from flavor to flavor.

==== First node ====

So we initialize the cluster bootstrap on the first node:

# percona 5.6, 5.7
service mysql bootstrap-pxc
# mariadb 10.0
service mysql bootstrap
# mariadb 10.1, 10.2
galera_new_cluster

Run <code>mysql_secure_installation</code> for a "secure by default" installation:

mysql_secure_installation

The further steps in this guide omit <code>-u -p</code> arguments to the MySQL client. Rather than passing them on the command line [https://dev.mysql.com/doc/refman/5.7/en/password-security-user.html] it is recommended to place the credentials in a file like <code>/root/.my.cnf</code> like

[client]
user=root
password=wip9Phae3Beijeed

We need a Galera replication user:

CREATE USER 'sstuser'@'localhost' IDENTIFIED BY 'OpIdjijwef0';
-- percona 5.6, mariadb 10.0
GRANT RELOAD, LOCK TABLES, REPLICATION CLIENT ON *.* TO 'sstuser'@'localhost';
-- percona 5.7, mariadb 10.1, 10.2
GRANT PROCESS, RELOAD, LOCK TABLES, REPLICATION CLIENT ON *.* TO 'sstuser'@'localhost';
FLUSH PRIVILEGES;

(Debian specific note: MariaDB provided startup scripts use the distro's mechanism of verifying startup/shutdown using a system user, so we create that as well:

# mariadb 10.0, 10.1, 10.2
GRANT ALL PRIVILEGES ON *.* TO "debian-sys-maint"@"localhost" IDENTIFIED BY "adBexthTsI5TaEps";

If you do this, yo need to synchronize the <code>/etc/mysql/debian.cnf</code> file from the first node to the other nodes as well.)

==== Other nodes ====

On the other nodes, we only need to restart the service now, to trigger a full state transfer from the first node to the other nodes.

We recommend to do this serially to let one state transfer complete before the second state transfer.

==== First node (continued) ====

Only applicable if you used <code>galera_new_cluster</code> before rather than the service script: In order to get the systemctl status consistent, restart the service on the first node:

# mariadb 10.1, 10.2: restart the service so that the systemctl status is consistent
mysqladmin shutdown
service mysql bootstrap

=== Verify the replication ===

The key tool to verify replication status is

mysql> show status like "%wsrep%";

This will give a lot of output. You want to verify in particular

+------------------------------+--------------------------------------+
| Variable_name | Value |
+------------------------------+--------------------------------------+
| wsrep_cluster_size | 3 |
| wsrep_cluster_status | Primary |
| wsrep_local_state | 4 |
| wsrep_local_state_comment | Synced |
| wsrep_ready | ON |
+------------------------------+--------------------------------------+

You can also explicitly verify replication by creating / inserting DBs, tables, rows on one node and select on other nodes.

==== Troubleshooting ====

The logs are helpful. Always.

Common mistakes are listed below.

If the Galera module does not get loaded at all:
* Configuration settings in ''my.cnf'' which are incompatible to Galera
* Wrong path of the shared object providing the Galera plugin in wsrep.cnf (wsrep_provider)

If the first node starts, but the second / third nodes can not be added to the cluster:
* User for the replication not created correctly on the first Galera node
* SST fails due to missing / wrong version prerequisite packages (not everything is hardcoded in package dependencies -- make sure you got percona-xtrabackup installed in the correct version, and also socat). If SST fails, do not only look into mysqls primary error logs, but also into logfiles from the SST tool in /var/lib/mysql on the donor node.

=== Notes about configuring OX for use with Galera ===

==== Write requests ====

Open-Xchange supports Galera as database backend only in the configuration where all writes are directed to one Galera node. For availability, it makes sense to not configure one Galera node's IP address directly, but rather employ some HA solution which offers active-passive functionality. Options therefore are discussed below.

==== Read requests ====

Read requests can be directed to any node in the Galera cluster. Our standard approach is to recommend to use a loadbalancer to implement round-robin over all nodes in a Galera cluster for the read requests. But you can also chose to use a dedicated read node (the same node, or a different node, than the write node). Each of the approaches has its own advantages.

* Load balancer based setup: Read requests get distributed round-robin between the Galera nodes. Theoretically by distributing the load of the read requests, you benefit from lower latencies and more throughput. But this has never been benchmarked yet. For a discussion of available loadbalances, see next section. OX-wise, in this configuration, you have two alternatives:
** The Galera option wsrep_causal_reads=1 option enables you to configure OX with its replication monitor disabled (com.openexchange.database.replicationMonitor=false in configdb.properties). This is the setup which seems to perform best according to our experience as turning off the replication monitor reduces the commits on the DB and thus the write operations per second on the underlying storage significantly, which outweights the drawback from having higher commit latency due to fully synchronous mode.
** Alternatively, you can run Galera with wsrep_causal_reads=0 when switching on OX builtin replication monitor. This is also a valid setup.
* Use a designated floating IP for the read requests: This eliminates the need of a load balancer. With this option you will not gain any performance, but the quantitative benefit is unclear anyhow.
* Use the floating IP for the writes also for the reads: In this scenario, you direct all database queries only to one Galera node, and the other two nodes are only getting queries in case of a failure of that node. In this case, you can even use wsrep_causal_reads=0 while still having OX builtin replication monitor switched off. However we do not expect this option to be superior to the round-robin loadbalancer approach.

=== Loadbalancer options ===

While the JDBC driver has some round-robin load balancing capabilities built-in, we don't recommend it for production use since it lacks possibilities to check the Galera nodes health states.

Loadbalancers used for OX -> Galera loadbalancing should be able to implement active-passive instances for the write requests, and active-active (round-robin) instances for the read requests. (If they cannot implement active-passive, you can still take a floating IP therefore.) Furthermore it is required to configure node health checks not only on the TCP level (by a simple connect), but to query the Galera health status periodically, evaluating Galera WSREP status variables. Otherwise split-brain scenarios or other bad states cannot be detected. For an example of such an health check, see our [[Clustercheck]] page.

Some customers use loadbalancing appliances. It is important to check that if the (virtual) infrastructure offers "loadbalancer" instances that they satisfy the given requirements. Often this is not the case. In particular, a simple "DNS round robin" approach is not viable.

==== LVS/ipvsadm/keepalived ====

If you want to create your own loadbalancers based on Linux, we usually recommend LVS (Linux Virtual Servers) controlled by Keepalived. LVS is a set of kernel modules implementing a L4 loadbalancer which performs quite well. Keepalived is a userspace daemon to control LVS rules, using health checks to reconfigure LVS rules if required. Keepalived / LVS requires one (or, for availability, two) dedicated linux nodes to run on. This can be a disadvantage for some installations, but usually, it pays off. We provide some configuration information on Keepalived [[Keepalived|here]].

==== MariaDB Maxscale ====

Since Maxscale has become GA in 2015, it seems to have undergone significant stability, performance and functional improvements. We are currently experimenting with Maxscale and share our installation / configuration knowledge [[Maxscale|here]]. It looks quite promising and might become ''the standard replacement'' for HAproxy, while we still presume Keepalived offers superior robustness and performance, coming with the cost of the requirement for one (or more) dedicated loadbalancer nodes.

==== HAproxy ====

In case where the Keepalived based approach is not feasible due to its requirements on the infrastructure, it is also possible to use a HAproxy based solution where HAproxy processes run on each of the OX nodes, configured for one round-robin and one active/passive instance. OX is then connecting to the local HAproxy instances. It is vital to configure HAproxy timeouts different from the defaults, otherwise HAproxy will kill active DB connections, causing errors. Be aware that in large installations the number of (distributed) HAproxy instances can get quite large. Some configuration hints for HAproxy are available [[HAproxy|here]].

== Master/Slave database setup ==

While we also support also "legacy" (pre-GTID) Master/Slave replication, we recommend to use GTID based replication, for easier setup and failure recovery. Support for GTID based replication has been added with OX 7.8.0.

GTID has been available since MySQL 5.6, so no 5.5 installation instructions below, sorry. We try to be generic in this documentation (thus, applicable to Oracle Community Edition and MariaDB) and point out differences where needed. Note: Instructions below include information about Oracle Community MySQL 5.7 which is not yet formally supported.

=== Preparations ===

Our configuration process includes wiping and reinitializing the datadir. This is usually not a problem in a fresh installation. If you want to upgrade an existing database to GTID master-slave, please be prepared to wipe the datadir, i.e. take a <code>mysqldump</code> for later restoration into the properly configured master.

Depeding on the flavor of the current database, this can be something like

# mariadb or oracle mysql without GTIDs
mysqldump --databases configdb oxdb_{5..14} > backup.sql

# mysql 5.6 with GTIDs... we dont want GTIDs here
mysqldump --databases --set-gtid-purged=OFF configdb oxdb_{5..14} > backup.sql

Be sure to verify the list of databases.

=== Installation ===

Software installation is identical for master and slave.

Please follow the instructions for installing from The vendors.

* Oracle Community Edition: https://dev.mysql.com/doc/mysql-apt-repo-quick-guide/en/
* MariaDB (10.0, 10.1): https://downloads.mariadb.org/mariadb/repositories/

Stop the service (if it is running):

service mysql stop

=== Configuration ===

Configuration as per configuration files is also identical for master and slave.

Consult [[My.cnf]] for general recommendations how to configure databases for usage with OX.

For GTID based replication, make sure you add some configurables to a new <code>/etc/mysql/ox.conf.d/gtid.cnf</code> file (assuming you are following our proposed schema of adding a <code>!includedir /etc/mysql/ox.conf.d/</code>" directive to <code>/etc/mysql/my.cnf</code>):

# GTID
log-bin=mysql-bin
server-id=...
log_slave_updates = ON

Oracle Community Edition: we need to add also

enforce_gtid_consistency = ON
gtid_mode = ON

(GTID mode is on by default on MariaDB.)

Use unique a <code>server-id</code> for each server; like <code>1</code> for the master, <code>2</code> for slave. For more complicated setups (like multiple slaves), adjust accordingly.

Since applying our configuration / sizing requires reinitialization of the MySQL datadir, we wipe/recreate it. Caution: this assumes we are running an empty database. If there is data in the database you want to keep, use mysqldump. See Preparation section above.

So, to initialize the datadir:

cd /var/lib/
mv mysql mysql.old.datadir
mkdir mysql
chown mysql.mysql mysql

(When coming from an existing installation, be sure to wipe also old binlogs. They can confuse the server on startup. Their location varies by configuration.)

The step to initialize the datadir is different for the different DBs:

# MariaDB 10.0, 10.1
mysql_install_db

# MariaDB 10.2
mysql_install_db --user=mysql

# Oracle 5.6
mysql_install_db -u mysql

# Oracle 5.7
mysqld --initialize-insecure --user=mysql

(Don't be worried about the <code>insecure</code>, it just means we set the db root pw in the next steps.)

Then:

service mysql restart
mysql_secure_installation

We want to emphasize the last step to run "secure".

Steps up to here apply to both the designated master and slave. The next steps will apply to the master.

=== Replication Setup ===

==== Master Setup ====

Create a replication user on the master (but, as always, pick your own password, and use the same password in the slave setup below):

mysql -e "CREATE USER 'repl'@'gtid-slave.localdomain' IDENTIFIED BY 'IvIjyoffod2'; GRANT REPLICATION SLAVE ON *.* TO 'repl'@'gtid-slave.localdomain';"

Now would also be the time to restore a previously created mysqldump, or add other users you need for adminstration, monitoring etc (like <code>debian-sys-maint@localhost</code>, for example). Adding the OX users is explained below ("Creating Open-Xchange user").

# If you took a dump for restore before
mysql < backup.sql

To prepare for the initial sync of the slave, set the master read-only:

mysql -e "SET @@global.read_only = ON;"

Create a dump to initialize the slave:

# MariaDB
mysqldump --all-databases --triggers --routines --events --master-data --gtid > master.sql

# Oracle
mysqldump --all-databases --triggers --routines --events --set-gtid-purged=ON > master.sql

Transfer to the slave:

scp master.sql gtid-slave:

==== Slave Setup ====

Configure the replication master settings. Note we don't need complicated binlog position settings etc with GTID.

Yet again DB-specific (use the repl user password from above):

# MariaDB
mysql -e 'CHANGE MASTER TO MASTER_HOST="gtid-master.localdomain", MASTER_USER="repl", MASTER_PASSWORD="IvIjyoffod2";'

# Oracle
mysql -e "CHANGE MASTER TO MASTER_HOST='gtid-master.localdomain', MASTER_USER='repl', MASTER_PASSWORD='IvIjyoffod2', MASTER_AUTO_POSITION=1;"
# https://www.percona.com/blog/2013/02/08/how-to-createrestore-a-slave-using-gtid-replication-in-mysql-5-6/
mysql -e "RESET MASTER;"

Read the master dump:

mysql < master.sql

Start replication on the slave:

mysql -e 'START SLAVE;'
mysql -e 'SHOW SLAVE STATUS\G'

==== Master Setup (continued) ====

Finally, unset read-only on the master:

# on the master
mysql -e "SET @@global.read_only = OFF;"

=== Configure OX to use with Master/Slave replication ===

Not much special wisdom here. OX was designed to be used with master/slave databases. For the ConfigDB, <code>configdb.properties</code> allows configuration of a <code>readUrl</code> and <code>writeUrl</code> (both of which are set to the correct values if you use <code>oxinstaller</code> with the correct arguments <code>--configdb-readhost</code>, <code>--configdb-writehost</code>).

(Obviously, the master is for writing and the slave is for reading.)

For the individiual UserDBs, use <code>registerdatabase -m true</code> for the masters and <code>registerdatabase -m false -M ...</code> for the respective slaves.

Be sure to have enabled the replication monitor in <code>configdb.properties</code>: <code>com.openexchange.database.replicationMonitor=true</code> (which it is by default); while GTID can show synchronous semantics, it is specified to silently fall back to asynchronous in certain circumstances, so synchronity is not guaranteed.

We recommend, though, to not register the databases directly by their native hostname or IP, but rather use some kind of HA system in order to be able to easily move a floating/failover IP from the master to the slave in case of master failure. Configuring and running such systems (like, corosync/pacemaker, keepalived, or whatever) is out of scope of this documentation, however.

== Creating Open-Xchange user ==

Now setup access for the Open-Xchange Server database user 'openexchange' to configdb and the oxdb for both groupware server addresses. These databases do not exist yet, but will be created during the Open-Xchange Server installation.

Notes:

* Please use a real password.
* The IPs in this example belong to the two different Open-Xchange Servers, please adjust them accordingly.
* If using a database on the same host as the middlware (usually done for POCs and demo installations), you need to grant also to the ''localhost'' host.
* Consult [[AppSuite:DB_user_privileges]] (or ''grep GRANT /opt/open-xchange/sbin/initconfigdb'') for an up-to-date list of required privileges. The following statement was correct as of the time of writing this section.

mysql> GRANT CREATE, LOCK TABLES, REFERENCES, INDEX, DROP, DELETE, ALTER, SELECT, UPDATE, INSERT, CREATE TEMPORARY TABLES, SHOW VIEW, SHOW DATABASES ON *.* TO 'openexchange'@'10.20.30.213' IDENTIFIED BY 'IntyoyntOat1' WITH GRANT OPTION;
mysql> GRANT CREATE, LOCK TABLES, REFERENCES, INDEX, DROP, DELETE, ALTER, SELECT, UPDATE, INSERT, CREATE TEMPORARY TABLES, SHOW VIEW, SHOW DATABASES ON *.* TO 'openexchange'@'10.20.30.215' IDENTIFIED BY 'IntyoyntOat1' WITH GRANT OPTION;

Template:OXLoadBalancingClustering Database

2018-09-27T14:24:56Z

Dominik.epple: /* First node */

== Overview ==

You can choose between Galera or Master/Slave replication. We like to recommend to use Galera for higher redudancy, easier operations, und synchronous semantics (so you can run OX without our "replication monitor"). For POC or demo setups, a single standalone database setup might be sufficient.

== Standalone database setup ==

=== Preparations ===

Our configuration process includes wiping and reinitializing the datadir. This is usually not a problem in a fresh installation. If you want to upgrade an existing database server, please be prepared to wipe the datadir, i.e. take a <code>mysqldump</code> for later restoration into the properly configured master.

mysqldump --databases configdb oxdb_{5..14} > backup.sql

Be sure to verify the list of databases.

=== Installation ===

Note: the following list ist not an exclusive list or authorative statement about supported MySQL flavors / versions. Please consult the official support / system requirements statement.

Please follow the upstream docs for your preferred flavor to get the software installed on your system.

* MariaDB (10.1, 10.2): https://downloads.mariadb.org/
* Oracle MySQL Community Server (5.6, 5.7): https://dev.mysql.com/downloads/mysql/

Make sure to doublecheck the service is not running (or stop it) after installation as we need to perform some reconfigurations.

service mysql stop

=== Configuration ===

MySQL configuration advise is given in our [[My.cnf|MySQL configuration article]]. Please consult that page for configuration information and create configuration files as described there.

Some settings we recommend to change require that the database gets re-initialized. We assume you don't have data there (since we are covering a fresh install) or you have taken a backup for later restore as explained above in the Preparations section.

cd /var/lib/
mv mysql mysql.old.datadir
mkdir mysql
chown mysql.mysql mysql

# mariadb
mysql_install_db
# mariadb 10.2
mysql_install_db --user=mysql
# oracle 5.6
mysql_install_db -u mysql
# oracle 5.7
mysqld --initialize-insecure --user=mysql

(Don't be worried about the <code>insecure</code>, it just means we set the db root pw in the next steps.)

Start the service. The actual command depends on your OS and on the MySQL flavor.

service mysql start

Run <code>mysql_secure_installation</code> for a "secure by default" installation:

mysql_secure_installation

That tool will ask for the current root password (which is empty by default) and subsequently questions like:

Change the root password? [Y/n]
Remove anonymous users? [Y/n]
Disallow root login remotely? [Y/n]
Remove test database and access to it? [Y/n]
Reload privilege tables now? [Y/n]

You should answer all these questions with "yes".

Configure a strong password for the MySQL <code>root</code> user.

The further steps in this guide omit <code>-u -p</code> arguments to the MySQL client. Rather than passing them on the command line [https://dev.mysql.com/doc/refman/5.7/en/password-security-user.html] it is recommended to place the credentials in a file like <code>/root/.my.cnf</code> like

[client]
user=root
password=wip9Phae3Beijeed

Make sure the service is enabled by the OS's init system. The actual command depends on your OS and on the MySQL flavor.

systemctl enable mysql.service

You should now be able to restore your previously taken backup.

# If you took a dump for restore before
mysql < backup.sql

=== Configure OX to use with a standalone database ===

Not much special wisdom here. OX was designed to be used with master/slave databases, and a standalone master works just as well, if we register it as a master, and not registering a slave.

For the ConfigDB, <code>configdb.properties</code> allows configuration of a <code>writeUrl</code> (which is set to the correct values if you use <code>oxinstaller</code> with the correct argument <code>--configdb-writehost</code>).

The single database is then used for reading and writing.

For the individiual UserDBs, use <code>registerdatabase -m true</code>.

== Galera database setup ==

=== Preparations ===

Our configuration process includes wiping and reinitializing the datadir. This is usually not a problem in a fresh installation. If you want to upgrade an existing database to Galera cluster, please be prepared to wipe the datadir, i.e. take a <code>mysqldump</code> for later restoration into the properly configured master.

Depeding on the flavor of the current database, this can be something like

# mariadb or oracle mysql without GTIDs
mysqldump --databases configdb oxdb_{5..14} > backup.sql

# mysql 5.6 with GTIDs... we dont want GTIDs here
mysqldump --databases --set-gtid-purged=OFF configdb oxdb_{5..14} > backup.sql

Be sure to verify the list of databases.

=== Installation ===

Please follow the upstream docs for your preferred flavor to get the software installed on your system.

* Percona XtraDB Cluster (5.6, 5.7): https://www.percona.com/doc/percona-xtradb-cluster/LATEST/install/index.html
* MariaDB Galera Cluster (10.0, 10.1): https://mariadb.com/kb/en/library/getting-started-with-mariadb-galera-cluster/ (Note: with 10.0, socat is required, but not a package dependency, so you need to explicitly install also socat)

Make sure to doublecheck the service is not running (or stop it) after installation as we need to perform some reconfigurations.

service mysql stop

=== Configuration ===

Galera-specific MySQL configuration advise is included in our main [[My.cnf|MySQL configuration article]]. Please consult that page for configuration information.

That page suggests a setup were we add three custom config files to <code>/etc/mysql/ox.conf.d/</code>: <code>ox.cnf</code> for general tuning/sizing, <code>wsrep.cnf</code> for clusterwide galera configuration, and <code>host.cnf</code> for host-specific settings.

Adjust the general settings and tunings in <code>ox.cnf</code> according to your sizing etc.

Adjust <code>wsrep.cnf</code> to reflect local paths, cluster member addresses, etc.

Adjust <code>host.cnf</code> to give node-local IPs, etc.

Version-specific hints:

# percona 5.6: unknown variable 'pxc_strict_mode=ENFORCING' ... unset that one
# mariadb 10.1: add wsrep_on=ON
# mariadb 10.0 and 10.1: set wsrep_node_incoming_address=192.168.1.22:3306 in host.cnf, otherwise the status wsrep_incoming_addresses might not be shown correctly(?!)

Some settings we recommend to change require that the database gets re-initialized. We assume you don't have data there (since we are covering a fresh install) or you have taken a backup for later restore as explained above in the Preparations section.

cd /var/lib/
mv mysql mysql.old.datadir
mkdir mysql
chown mysql.mysql mysql

# mariadb 10.0 and 10.1
mysql_install_db
# mariadb 10.2
mysql_install_db --user=mysql
# percona 5.6
mysqld --user=mysql
# percona 5.7
mysqld --initialize-insecure --user=mysql

(Don't be worried about the <code>insecure</code>, it just means we set the db root pw in the next steps.)

=== Cluster startup ===

Typically on startup a Galera node tries to join a cluster, and if it fails, it will exit. Thus, when no cluster nodes are running, the first cluster node to be started needs to be told to not try to join a cluster, but rather bootstrap a new cluster. The exact arguments vary from version to version and from flavor to flavor.

==== First node ====

So we initialize the cluster bootstrap on the first node:

# percona 5.6, 5.7
service mysql bootstrap-pxc
# mariadb 10.0
service mysql bootstrap
# mariadb 10.1, 10.2
galera_new_cluster

Run <code>mysql_secure_installation</code> for a "secure by default" installation:

mysql_secure_installation

The further steps in this guide omit <code>-u -p</code> arguments to the MySQL client. Rather than passing them on the command line [https://dev.mysql.com/doc/refman/5.7/en/password-security-user.html] it is recommended to place the credentials in a file like <code>/root/.my.cnf</code> like

[client]
user=root
password=wip9Phae3Beijeed

We need a Galera replication user:

CREATE USER 'sstuser'@'localhost' IDENTIFIED BY 'OpIdjijwef0';
-- percona 5.6, mariadb 10.0
GRANT RELOAD, LOCK TABLES, REPLICATION CLIENT ON *.* TO 'sstuser'@'localhost';
-- percona 5.7, mariadb 10.1, 10.2
GRANT PROCESS, RELOAD, LOCK TABLES, REPLICATION CLIENT ON *.* TO 'sstuser'@'localhost';
FLUSH PRIVILEGES;

(Debian specific note: MariaDB provided startup scripts use the distro's mechanism of verifying startup/shutdown using a system user, so we create that as well:

# mariadb 10.0, 10.1, 10.2
GRANT ALL PRIVILEGES ON *.* TO "debian-sys-maint"@"localhost" IDENTIFIED BY "adBexthTsI5TaEps";

If you do this, yo need to synchronize the <code>/etc/mysql/debian.cnf</code> file from the first node to the other nodes as well.)

==== Other nodes ====

On the other nodes, we only need to restart the service now, to trigger a full state transfer from the first node to the other nodes.

We recommend to do this serially to let one state transfer complete before the second state transfer.

==== First node (continued) ====

Only applicable if you used <code>galera_new_cluster</code> before rather than the service script: In order to get the systemctl status consistent, restart the service on the first node:

# mariadb 10.1: restart the service so that the systemctl status is consistent
mysqladmin shutdown
service mysql bootstrap

=== Verify the replication ===

The key tool to verify replication status is

mysql> show status like "%wsrep%";

This will give a lot of output. You want to verify in particular

+------------------------------+--------------------------------------+
| Variable_name | Value |
+------------------------------+--------------------------------------+
| wsrep_cluster_size | 3 |
| wsrep_cluster_status | Primary |
| wsrep_local_state | 4 |
| wsrep_local_state_comment | Synced |
| wsrep_ready | ON |
+------------------------------+--------------------------------------+

You can also explicitly verify replication by creating / inserting DBs, tables, rows on one node and select on other nodes.

==== Troubleshooting ====

The logs are helpful. Always.

Common mistakes are listed below.

If the Galera module does not get loaded at all:
* Configuration settings in ''my.cnf'' which are incompatible to Galera
* Wrong path of the shared object providing the Galera plugin in wsrep.cnf (wsrep_provider)

If the first node starts, but the second / third nodes can not be added to the cluster:
* User for the replication not created correctly on the first Galera node
* SST fails due to missing / wrong version prerequisite packages (not everything is hardcoded in package dependencies -- make sure you got percona-xtrabackup installed in the correct version, and also socat). If SST fails, do not only look into mysqls primary error logs, but also into logfiles from the SST tool in /var/lib/mysql on the donor node.

=== Notes about configuring OX for use with Galera ===

==== Write requests ====

Open-Xchange supports Galera as database backend only in the configuration where all writes are directed to one Galera node. For availability, it makes sense to not configure one Galera node's IP address directly, but rather employ some HA solution which offers active-passive functionality. Options therefore are discussed below.

==== Read requests ====

Read requests can be directed to any node in the Galera cluster. Our standard approach is to recommend to use a loadbalancer to implement round-robin over all nodes in a Galera cluster for the read requests. But you can also chose to use a dedicated read node (the same node, or a different node, than the write node). Each of the approaches has its own advantages.

* Load balancer based setup: Read requests get distributed round-robin between the Galera nodes. Theoretically by distributing the load of the read requests, you benefit from lower latencies and more throughput. But this has never been benchmarked yet. For a discussion of available loadbalances, see next section. OX-wise, in this configuration, you have two alternatives:
** The Galera option wsrep_causal_reads=1 option enables you to configure OX with its replication monitor disabled (com.openexchange.database.replicationMonitor=false in configdb.properties). This is the setup which seems to perform best according to our experience as turning off the replication monitor reduces the commits on the DB and thus the write operations per second on the underlying storage significantly, which outweights the drawback from having higher commit latency due to fully synchronous mode.
** Alternatively, you can run Galera with wsrep_causal_reads=0 when switching on OX builtin replication monitor. This is also a valid setup.
* Use a designated floating IP for the read requests: This eliminates the need of a load balancer. With this option you will not gain any performance, but the quantitative benefit is unclear anyhow.
* Use the floating IP for the writes also for the reads: In this scenario, you direct all database queries only to one Galera node, and the other two nodes are only getting queries in case of a failure of that node. In this case, you can even use wsrep_causal_reads=0 while still having OX builtin replication monitor switched off. However we do not expect this option to be superior to the round-robin loadbalancer approach.

=== Loadbalancer options ===

While the JDBC driver has some round-robin load balancing capabilities built-in, we don't recommend it for production use since it lacks possibilities to check the Galera nodes health states.

Loadbalancers used for OX -> Galera loadbalancing should be able to implement active-passive instances for the write requests, and active-active (round-robin) instances for the read requests. (If they cannot implement active-passive, you can still take a floating IP therefore.) Furthermore it is required to configure node health checks not only on the TCP level (by a simple connect), but to query the Galera health status periodically, evaluating Galera WSREP status variables. Otherwise split-brain scenarios or other bad states cannot be detected. For an example of such an health check, see our [[Clustercheck]] page.

Some customers use loadbalancing appliances. It is important to check that if the (virtual) infrastructure offers "loadbalancer" instances that they satisfy the given requirements. Often this is not the case. In particular, a simple "DNS round robin" approach is not viable.

==== LVS/ipvsadm/keepalived ====

If you want to create your own loadbalancers based on Linux, we usually recommend LVS (Linux Virtual Servers) controlled by Keepalived. LVS is a set of kernel modules implementing a L4 loadbalancer which performs quite well. Keepalived is a userspace daemon to control LVS rules, using health checks to reconfigure LVS rules if required. Keepalived / LVS requires one (or, for availability, two) dedicated linux nodes to run on. This can be a disadvantage for some installations, but usually, it pays off. We provide some configuration information on Keepalived [[Keepalived|here]].

==== MariaDB Maxscale ====

Since Maxscale has become GA in 2015, it seems to have undergone significant stability, performance and functional improvements. We are currently experimenting with Maxscale and share our installation / configuration knowledge [[Maxscale|here]]. It looks quite promising and might become ''the standard replacement'' for HAproxy, while we still presume Keepalived offers superior robustness and performance, coming with the cost of the requirement for one (or more) dedicated loadbalancer nodes.

==== HAproxy ====

In case where the Keepalived based approach is not feasible due to its requirements on the infrastructure, it is also possible to use a HAproxy based solution where HAproxy processes run on each of the OX nodes, configured for one round-robin and one active/passive instance. OX is then connecting to the local HAproxy instances. It is vital to configure HAproxy timeouts different from the defaults, otherwise HAproxy will kill active DB connections, causing errors. Be aware that in large installations the number of (distributed) HAproxy instances can get quite large. Some configuration hints for HAproxy are available [[HAproxy|here]].

== Master/Slave database setup ==

While we also support also "legacy" (pre-GTID) Master/Slave replication, we recommend to use GTID based replication, for easier setup and failure recovery. Support for GTID based replication has been added with OX 7.8.0.

GTID has been available since MySQL 5.6, so no 5.5 installation instructions below, sorry. We try to be generic in this documentation (thus, applicable to Oracle Community Edition and MariaDB) and point out differences where needed. Note: Instructions below include information about Oracle Community MySQL 5.7 which is not yet formally supported.

=== Preparations ===

Our configuration process includes wiping and reinitializing the datadir. This is usually not a problem in a fresh installation. If you want to upgrade an existing database to GTID master-slave, please be prepared to wipe the datadir, i.e. take a <code>mysqldump</code> for later restoration into the properly configured master.

Depeding on the flavor of the current database, this can be something like

# mariadb or oracle mysql without GTIDs
mysqldump --databases configdb oxdb_{5..14} > backup.sql

# mysql 5.6 with GTIDs... we dont want GTIDs here
mysqldump --databases --set-gtid-purged=OFF configdb oxdb_{5..14} > backup.sql

Be sure to verify the list of databases.

=== Installation ===

Software installation is identical for master and slave.

Please follow the instructions for installing from The vendors.

* Oracle Community Edition: https://dev.mysql.com/doc/mysql-apt-repo-quick-guide/en/
* MariaDB (10.0, 10.1): https://downloads.mariadb.org/mariadb/repositories/

Stop the service (if it is running):

service mysql stop

=== Configuration ===

Configuration as per configuration files is also identical for master and slave.

Consult [[My.cnf]] for general recommendations how to configure databases for usage with OX.

For GTID based replication, make sure you add some configurables to a new <code>/etc/mysql/ox.conf.d/gtid.cnf</code> file (assuming you are following our proposed schema of adding a <code>!includedir /etc/mysql/ox.conf.d/</code>" directive to <code>/etc/mysql/my.cnf</code>):

# GTID
log-bin=mysql-bin
server-id=...
log_slave_updates = ON

Oracle Community Edition: we need to add also

enforce_gtid_consistency = ON
gtid_mode = ON

(GTID mode is on by default on MariaDB.)

Use unique a <code>server-id</code> for each server; like <code>1</code> for the master, <code>2</code> for slave. For more complicated setups (like multiple slaves), adjust accordingly.

Since applying our configuration / sizing requires reinitialization of the MySQL datadir, we wipe/recreate it. Caution: this assumes we are running an empty database. If there is data in the database you want to keep, use mysqldump. See Preparation section above.

So, to initialize the datadir:

cd /var/lib/
mv mysql mysql.old.datadir
mkdir mysql
chown mysql.mysql mysql

(When coming from an existing installation, be sure to wipe also old binlogs. They can confuse the server on startup. Their location varies by configuration.)

The step to initialize the datadir is different for the different DBs:

# MariaDB 10.0, 10.1
mysql_install_db

# MariaDB 10.2
mysql_install_db --user=mysql

# Oracle 5.6
mysql_install_db -u mysql

# Oracle 5.7
mysqld --initialize-insecure --user=mysql

(Don't be worried about the <code>insecure</code>, it just means we set the db root pw in the next steps.)

Then:

service mysql restart
mysql_secure_installation

We want to emphasize the last step to run "secure".

Steps up to here apply to both the designated master and slave. The next steps will apply to the master.

=== Replication Setup ===

==== Master Setup ====

Create a replication user on the master (but, as always, pick your own password, and use the same password in the slave setup below):

mysql -e "CREATE USER 'repl'@'gtid-slave.localdomain' IDENTIFIED BY 'IvIjyoffod2'; GRANT REPLICATION SLAVE ON *.* TO 'repl'@'gtid-slave.localdomain';"

Now would also be the time to restore a previously created mysqldump, or add other users you need for adminstration, monitoring etc (like <code>debian-sys-maint@localhost</code>, for example). Adding the OX users is explained below ("Creating Open-Xchange user").

# If you took a dump for restore before
mysql < backup.sql

To prepare for the initial sync of the slave, set the master read-only:

mysql -e "SET @@global.read_only = ON;"

Create a dump to initialize the slave:

# MariaDB
mysqldump --all-databases --triggers --routines --events --master-data --gtid > master.sql

# Oracle
mysqldump --all-databases --triggers --routines --events --set-gtid-purged=ON > master.sql

Transfer to the slave:

scp master.sql gtid-slave:

==== Slave Setup ====

Configure the replication master settings. Note we don't need complicated binlog position settings etc with GTID.

Yet again DB-specific (use the repl user password from above):

# MariaDB
mysql -e 'CHANGE MASTER TO MASTER_HOST="gtid-master.localdomain", MASTER_USER="repl", MASTER_PASSWORD="IvIjyoffod2";'

# Oracle
mysql -e "CHANGE MASTER TO MASTER_HOST='gtid-master.localdomain', MASTER_USER='repl', MASTER_PASSWORD='IvIjyoffod2', MASTER_AUTO_POSITION=1;"
# https://www.percona.com/blog/2013/02/08/how-to-createrestore-a-slave-using-gtid-replication-in-mysql-5-6/
mysql -e "RESET MASTER;"

Read the master dump:

mysql < master.sql

Start replication on the slave:

mysql -e 'START SLAVE;'
mysql -e 'SHOW SLAVE STATUS\G'

==== Master Setup (continued) ====

Finally, unset read-only on the master:

# on the master
mysql -e "SET @@global.read_only = OFF;"

=== Configure OX to use with Master/Slave replication ===

Not much special wisdom here. OX was designed to be used with master/slave databases. For the ConfigDB, <code>configdb.properties</code> allows configuration of a <code>readUrl</code> and <code>writeUrl</code> (both of which are set to the correct values if you use <code>oxinstaller</code> with the correct arguments <code>--configdb-readhost</code>, <code>--configdb-writehost</code>).

(Obviously, the master is for writing and the slave is for reading.)

For the individiual UserDBs, use <code>registerdatabase -m true</code> for the masters and <code>registerdatabase -m false -M ...</code> for the respective slaves.

Be sure to have enabled the replication monitor in <code>configdb.properties</code>: <code>com.openexchange.database.replicationMonitor=true</code> (which it is by default); while GTID can show synchronous semantics, it is specified to silently fall back to asynchronous in certain circumstances, so synchronity is not guaranteed.

We recommend, though, to not register the databases directly by their native hostname or IP, but rather use some kind of HA system in order to be able to easily move a floating/failover IP from the master to the slave in case of master failure. Configuring and running such systems (like, corosync/pacemaker, keepalived, or whatever) is out of scope of this documentation, however.

== Creating Open-Xchange user ==

Now setup access for the Open-Xchange Server database user 'openexchange' to configdb and the oxdb for both groupware server addresses. These databases do not exist yet, but will be created during the Open-Xchange Server installation.

Notes:

* Please use a real password.
* The IPs in this example belong to the two different Open-Xchange Servers, please adjust them accordingly.
* If using a database on the same host as the middlware (usually done for POCs and demo installations), you need to grant also to the ''localhost'' host.
* Consult [[AppSuite:DB_user_privileges]] (or ''grep GRANT /opt/open-xchange/sbin/initconfigdb'') for an up-to-date list of required privileges. The following statement was correct as of the time of writing this section.

mysql> GRANT CREATE, LOCK TABLES, REFERENCES, INDEX, DROP, DELETE, ALTER, SELECT, UPDATE, INSERT, CREATE TEMPORARY TABLES, SHOW VIEW, SHOW DATABASES ON *.* TO 'openexchange'@'10.20.30.213' IDENTIFIED BY 'IntyoyntOat1' WITH GRANT OPTION;
mysql> GRANT CREATE, LOCK TABLES, REFERENCES, INDEX, DROP, DELETE, ALTER, SELECT, UPDATE, INSERT, CREATE TEMPORARY TABLES, SHOW VIEW, SHOW DATABASES ON *.* TO 'openexchange'@'10.20.30.215' IDENTIFIED BY 'IntyoyntOat1' WITH GRANT OPTION;

My.cnf

2018-09-27T14:24:07Z

Dominik.epple: /* Sample config files */

== Introduction ==

This page lists some performance tuning parameters which we recommend for tuning MySQL database services used for Open-Xchange installations.

We cannot guarantee this is an exhaustive list of required settings. So treat this list of tunings as probably required, but not necessarily sufficient settings for optimal MySQL performance.

Furthermore, as MySQL changes over time, settings which have been correct as of the time of writing may become incorrect later.

However, this list of settings is the result of internal performance testing and real world customer feedback, so it should be valid to some extent.

In the end, proper configuration of the database service for performance, but also consistency, durability and high availability is in the responsibility of the customer.

=== Performance items ===

* You should adjust the <code>innodb_buffer_pool_size</code> parameter for reasonable memory usage. Our DB sizing is mainly memory-driven and this is where most of the memory goes. Our standard DB machine sizing assumption is 32 GB if MySQL dedicated memory on a 48 GB total memory machine. On such a machine, you would configure 32 GB for the innodb_buffer_pool size, being aware that MySQL does also require memory for other things, in particular there are some also per-connection related memory spendings, which can become substantial if you allow for a lot of maximum concurrent connections. Please watch your memory configuration carefully, use monitoring and tools like mysqltuner.pl.

* On bigger installations you should use <code>innodb_file_per_table = 1</code>, which is creating single files instead of one big blob. If you change this parameter after the database initialization you have to recreate (like dump/drop and re-import) the tables.

* It can help to put different parts of the mysql datadir (iblog, ibdata) on different filesystems / storage devices. This depends on your infrastructure. Settings herefore are <code>datadir</code>, <code>innodb_data_home_dir</code>, <code>innodb_log_group_home_dir</code>.

* If your storage is fast (handle a lot of IOPS), you may want to adjust the <code>innodb_io_capacity</code> setting, which defines a limit for the IOPS MySQL will create. The default is 200, which is sensible for single spindle disks. But if you have storage appliances with a lot of fast SAS drives, or even SSDs, this limit can be increased greatly.

=== Functional items ===

* Query cache is to be switched off; as we found in our own benchmarks and as backed up by upstreams, this hurts performance in load situations with high concurrency. [https://dev.mysql.com/doc/refman/5.7/en/query-cache.htm The query cache is deprecated as of MySQL 5.7.20, and is removed in MySQL 8.0], so we recommend also to switch that off.
* Starting with App Suite 7.10.0, <code>character_set_server</code> must be set to <code>utf8mb4</code> and <code>collation_server</code> to <code>utf8mb4_general_ci</code>
* Starting with MySQL 5.7 <code>innodb_strict_mode</code> must be disabled.
* Starting with MySQL 5.7 and MariaDB 10.2 <code>sql_mode</code> must be configured according to belows matrix.

==== SQL mode matrix ====

The default for the <code>sql_mode</code> setting changes regularly with MariaDB and MySQL releases and is not even consistent anymore between the two derivates. SQL modes affect how data and queries are handled at runtime. Enabling strict modes might lead to errors in terms of failing queries or even update tasks. We strongly recommend the following configuration to avoid according runtime errors:

{| class="wikitable"
|-
! scope="col"| sql_mode
! scope="col"| App Suite 7.8.4
! scope="col"| App Suite 7.10.0
|-
! scope="row"| MySQL 5.6
| <code>NO_ENGINE_SUBSTITUTION</code>
| <code>NO_ENGINE_SUBSTITUTION</code>
|-
! scope="row"| MySQL 5.7
| <code>NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION</code>
| <code>NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION,ONLY_FULL_GROUP_BY</code>
|-
! scope="row"| MariaDB 10.1
| <code>NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION</code>
| <code>NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION</code>
|-
! scope="row"| MariaDB 10.2
| <code>NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION</code>
| <code>NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION</code>
|-
|}

=== Sample config files ===

For easy deployment we recommend not to edit existing the existing my.cnf file, rather assume the distro provides sane settings for most of the items and override items where needed.

As in MySQL there is a [https://dev.mysql.com/doc/refman/5.7/en/option-files.html last instance wins] semantic in options file parsing, we propose to create a custom include directory "ox.conf.d", put our custom config files therein, and include this directory as latest directory in /etc/mysql/my.cnf.

To put in that directory, we have one main tuning file called tunings.cnf, one galera-related file if galera is in use, and one more galera host-specific file which contains per-host settings if using galera (separate in a file of its own for easier configuration management).

So, start with adding to the existing my.cnf at the very bottom:

!includedir /etc/mysql/ox.conf.d/

Create that directory and put in there the generic ox tunings file /etc/mysql/ox.conf.d/tunings.cnf:

[mysqld]
bind-address = *

#innodb_use_native_aio = 0

table_open_cache = 3072
table_definition_cache = 4096
max_heap_table_size = 64M
tmp_table_size = 64M
max_connections = 505
max_user_connections = 500
max_allowed_packet = 16M
thread_cache_size = 32
query_cache_size = 0
query_cache_type = 0
default_storage_engine = InnoDB
innodb_buffer_pool_size = 32G
# the default value in MySQL 5.6.6 and higher is 8 when innodb_buffer_pool_size is greater than or equal to 1GB. Otherwise, the default is 1.
innodb_buffer_pool_instances = 32
innodb_data_file_path = ibdata1:128M:autoextend
innodb_file_per_table = 1
# innodb_log_file_size should be 25% of the innodb_buffer_pool_size
innodb_log_file_size = 4GB
# default and recommended value is 2
innodb_log_files_in_group = 2
# adjust according to your storage
#innodb_io_capacity = 1000

# we are unsure about this setting. Newer versions of MariaDB seem to be fine with low (=1) settings for this value.
# Traditionally we encountered values up to 4x the number of cores.
# Default seems to be number of cores, so let's stick the default
# In the end, we need to leave this setting up to you: if you dont get full cpu utilization in cpu-bound situations, this might be a setting to increase.
#thread_pool_size = 32

binlog_cache_size = 1M
sync_binlog = 8
binlog_format = row

character_set_server = utf8
collation_server = utf8_general_ci

# This was default_table_type previous to MySQL 5.5
default_storage_engine = InnoDB

innodb_autoinc_lock_mode = 2

# keep until 5.6, deprecated later
innodb_locks_unsafe_for_binlog = 1

# we found this has huge impact on (galera) performance
# default (consistent) setting of 1 greatly severs performance
# in galera (or async master-slave) deployments, you might be ok with setting this to 0 or 2,
# assuming our consistency / availability comes from replication / other cluster nodes
innodb_flush_log_at_trx_commit = 0

# for performance testing systems, to not use excessive disk space
#expire_logs_days = 1

# MySQL 5.7.7 has changed the default to 1. Disable it explicitly to prevent from errors based on invalid data stored by former App Suite or MySQL versions.
innodb_strict_mode = 0

# The following value refers to App Suite 7.10 on top of MySQL 5.7. For other combinations see the sql mode matrix at http://oxpedia.org/wiki/index.php?title=My.cnf.
sql_mode = NO_ENGINE_SUBSTITUTION,NO_AUTO_CREATE_USER,ONLY_FULL_GROUP_BY

If using galera, use the following galera configuration file <code>/etc/mysql/ox.conf.d/wsrep.cnf</code>. See the comments in that file for values to be adjusted.

[mysqld]
# adjust for your distros SO location
wsrep_provider=/usr/lib/libgalera_smm.so

# this is the big winner and enables us to switch off OX's replication monitor
wsrep_sync_wait=1

# pick a unique cluster name
wsrep_cluster_name=devcluster
# adjust for your IPs / hostnames
wsrep_cluster_address=gcomm://10.20.29.68,10.20.29.69,10.20.29.70

# put this in host.cnf
#wsrep_node_name=...
#wsrep_node_address=...

# For some MariaDB versions, xtrabackup-v2 no longer works, instead use "mariabackup"
# (needs to be installed separately, e.g. via the mariadb-backup-10.2 package)
# see upstream documentation for details:
# https://mariadb.com/kb/en/library/getting-started-with-mariadb-galera-cluster/#xtrabackup
#
# wsrep_sst_method=mariabackup
wsrep_sst_method=xtrabackup-v2

# wsrep_sst_auth if of format username:password
# pick whatever you configured on the donor node
wsrep_sst_auth=sstuser:...

# galera-specific tunings
wsrep_slave_threads = 32

# finally, enable wsrep: required for some MariaDB versions
wsrep_on=ON -- Enable wsrep replication (MariaDB starting 10.1.1)

Galera-related host-specific settings go in /etc/mysql/ox.conf.d/host.cnf:

[mysqld]
# the nodes hostname
wsrep_node_name=...
# and the IP of the wsrep relevant interface, if multiple
wsrep_node_address=10.20.29.68

Template:OXLoadBalancingClustering Database

2018-09-27T13:48:22Z

Dominik.epple: /* First node */

== Overview ==

You can choose between Galera or Master/Slave replication. We like to recommend to use Galera for higher redudancy, easier operations, und synchronous semantics (so you can run OX without our "replication monitor"). For POC or demo setups, a single standalone database setup might be sufficient.

== Standalone database setup ==

=== Preparations ===

Our configuration process includes wiping and reinitializing the datadir. This is usually not a problem in a fresh installation. If you want to upgrade an existing database server, please be prepared to wipe the datadir, i.e. take a <code>mysqldump</code> for later restoration into the properly configured master.

mysqldump --databases configdb oxdb_{5..14} > backup.sql

Be sure to verify the list of databases.

=== Installation ===

Note: the following list ist not an exclusive list or authorative statement about supported MySQL flavors / versions. Please consult the official support / system requirements statement.

Please follow the upstream docs for your preferred flavor to get the software installed on your system.

* MariaDB (10.1, 10.2): https://downloads.mariadb.org/
* Oracle MySQL Community Server (5.6, 5.7): https://dev.mysql.com/downloads/mysql/

Make sure to doublecheck the service is not running (or stop it) after installation as we need to perform some reconfigurations.

service mysql stop

=== Configuration ===

MySQL configuration advise is given in our [[My.cnf|MySQL configuration article]]. Please consult that page for configuration information and create configuration files as described there.

Some settings we recommend to change require that the database gets re-initialized. We assume you don't have data there (since we are covering a fresh install) or you have taken a backup for later restore as explained above in the Preparations section.

cd /var/lib/
mv mysql mysql.old.datadir
mkdir mysql
chown mysql.mysql mysql

# mariadb
mysql_install_db
# mariadb 10.2
mysql_install_db --user=mysql
# oracle 5.6
mysql_install_db -u mysql
# oracle 5.7
mysqld --initialize-insecure --user=mysql

(Don't be worried about the <code>insecure</code>, it just means we set the db root pw in the next steps.)

Start the service. The actual command depends on your OS and on the MySQL flavor.

service mysql start

Run <code>mysql_secure_installation</code> for a "secure by default" installation:

mysql_secure_installation

That tool will ask for the current root password (which is empty by default) and subsequently questions like:

Change the root password? [Y/n]
Remove anonymous users? [Y/n]
Disallow root login remotely? [Y/n]
Remove test database and access to it? [Y/n]
Reload privilege tables now? [Y/n]

You should answer all these questions with "yes".

Configure a strong password for the MySQL <code>root</code> user.

The further steps in this guide omit <code>-u -p</code> arguments to the MySQL client. Rather than passing them on the command line [https://dev.mysql.com/doc/refman/5.7/en/password-security-user.html] it is recommended to place the credentials in a file like <code>/root/.my.cnf</code> like

[client]
user=root
password=wip9Phae3Beijeed

Make sure the service is enabled by the OS's init system. The actual command depends on your OS and on the MySQL flavor.

systemctl enable mysql.service

You should now be able to restore your previously taken backup.

# If you took a dump for restore before
mysql < backup.sql

=== Configure OX to use with a standalone database ===

Not much special wisdom here. OX was designed to be used with master/slave databases, and a standalone master works just as well, if we register it as a master, and not registering a slave.

For the ConfigDB, <code>configdb.properties</code> allows configuration of a <code>writeUrl</code> (which is set to the correct values if you use <code>oxinstaller</code> with the correct argument <code>--configdb-writehost</code>).

The single database is then used for reading and writing.

For the individiual UserDBs, use <code>registerdatabase -m true</code>.

== Galera database setup ==

=== Preparations ===

Our configuration process includes wiping and reinitializing the datadir. This is usually not a problem in a fresh installation. If you want to upgrade an existing database to Galera cluster, please be prepared to wipe the datadir, i.e. take a <code>mysqldump</code> for later restoration into the properly configured master.

Depeding on the flavor of the current database, this can be something like

# mariadb or oracle mysql without GTIDs
mysqldump --databases configdb oxdb_{5..14} > backup.sql

# mysql 5.6 with GTIDs... we dont want GTIDs here
mysqldump --databases --set-gtid-purged=OFF configdb oxdb_{5..14} > backup.sql

Be sure to verify the list of databases.

=== Installation ===

Please follow the upstream docs for your preferred flavor to get the software installed on your system.

* Percona XtraDB Cluster (5.6, 5.7): https://www.percona.com/doc/percona-xtradb-cluster/LATEST/install/index.html
* MariaDB Galera Cluster (10.0, 10.1): https://mariadb.com/kb/en/library/getting-started-with-mariadb-galera-cluster/ (Note: with 10.0, socat is required, but not a package dependency, so you need to explicitly install also socat)

Make sure to doublecheck the service is not running (or stop it) after installation as we need to perform some reconfigurations.

service mysql stop

=== Configuration ===

Galera-specific MySQL configuration advise is included in our main [[My.cnf|MySQL configuration article]]. Please consult that page for configuration information.

That page suggests a setup were we add three custom config files to <code>/etc/mysql/ox.conf.d/</code>: <code>ox.cnf</code> for general tuning/sizing, <code>wsrep.cnf</code> for clusterwide galera configuration, and <code>host.cnf</code> for host-specific settings.

Adjust the general settings and tunings in <code>ox.cnf</code> according to your sizing etc.

Adjust <code>wsrep.cnf</code> to reflect local paths, cluster member addresses, etc.

Adjust <code>host.cnf</code> to give node-local IPs, etc.

Version-specific hints:

# percona 5.6: unknown variable 'pxc_strict_mode=ENFORCING' ... unset that one
# mariadb 10.1: add wsrep_on=ON
# mariadb 10.0 and 10.1: set wsrep_node_incoming_address=192.168.1.22:3306 in host.cnf, otherwise the status wsrep_incoming_addresses might not be shown correctly(?!)

Some settings we recommend to change require that the database gets re-initialized. We assume you don't have data there (since we are covering a fresh install) or you have taken a backup for later restore as explained above in the Preparations section.

cd /var/lib/
mv mysql mysql.old.datadir
mkdir mysql
chown mysql.mysql mysql

# mariadb 10.0 and 10.1
mysql_install_db
# mariadb 10.2
mysql_install_db --user=mysql
# percona 5.6
mysqld --user=mysql
# percona 5.7
mysqld --initialize-insecure --user=mysql

(Don't be worried about the <code>insecure</code>, it just means we set the db root pw in the next steps.)

=== Cluster startup ===

Typically on startup a Galera node tries to join a cluster, and if it fails, it will exit. Thus, when no cluster nodes are running, the first cluster node to be started needs to be told to not try to join a cluster, but rather bootstrap a new cluster. The exact arguments vary from version to version and from flavor to flavor.

==== First node ====

So we initialize the cluster bootstrap on the first node:

# percona 5.6, 5.7
service mysql bootstrap-pxc
# mariadb 10.0
service mysql bootstrap
# mariadb 10.1, 10.2
galera_new_cluster

Run <code>mysql_secure_installation</code> for a "secure by default" installation:

mysql_secure_installation

The further steps in this guide omit <code>-u -p</code> arguments to the MySQL client. Rather than passing them on the command line [https://dev.mysql.com/doc/refman/5.7/en/password-security-user.html] it is recommended to place the credentials in a file like <code>/root/.my.cnf</code> like

[client]
user=root
password=wip9Phae3Beijeed

We need a Galera replication user:

CREATE USER 'sstuser'@'localhost' IDENTIFIED BY 'OpIdjijwef0';
-- percona 5.6, mariadb 10.0
GRANT RELOAD, LOCK TABLES, REPLICATION CLIENT ON *.* TO 'sstuser'@'localhost';
-- percona 5.7, mariadb 10.1
GRANT PROCESS, RELOAD, LOCK TABLES, REPLICATION CLIENT ON *.* TO 'sstuser'@'localhost';
FLUSH PRIVILEGES;

(Debian specific note: MariaDB provided startup scripts use the distro's mechanism of verifying startup/shutdown using a system user, so we create that as well:

# mariadb 10.0, 10.1
GRANT ALL PRIVILEGES ON *.* TO "debian-sys-maint"@"localhost" IDENTIFIED BY "adBexthTsI5TaEps";

If you do this, yo need to synchronize the <code>/etc/mysql/debian.cnf</code> file from the first node to the other nodes as well.)

==== Other nodes ====

On the other nodes, we only need to restart the service now, to trigger a full state transfer from the first node to the other nodes.

We recommend to do this serially to let one state transfer complete before the second state transfer.

==== First node (continued) ====

Only applicable if you used <code>galera_new_cluster</code> before rather than the service script: In order to get the systemctl status consistent, restart the service on the first node:

# mariadb 10.1: restart the service so that the systemctl status is consistent
mysqladmin shutdown
service mysql bootstrap

=== Verify the replication ===

The key tool to verify replication status is

mysql> show status like "%wsrep%";

This will give a lot of output. You want to verify in particular

+------------------------------+--------------------------------------+
| Variable_name | Value |
+------------------------------+--------------------------------------+
| wsrep_cluster_size | 3 |
| wsrep_cluster_status | Primary |
| wsrep_local_state | 4 |
| wsrep_local_state_comment | Synced |
| wsrep_ready | ON |
+------------------------------+--------------------------------------+

You can also explicitly verify replication by creating / inserting DBs, tables, rows on one node and select on other nodes.

==== Troubleshooting ====

The logs are helpful. Always.

Common mistakes are listed below.

If the Galera module does not get loaded at all:
* Configuration settings in ''my.cnf'' which are incompatible to Galera
* Wrong path of the shared object providing the Galera plugin in wsrep.cnf (wsrep_provider)

If the first node starts, but the second / third nodes can not be added to the cluster:
* User for the replication not created correctly on the first Galera node
* SST fails due to missing / wrong version prerequisite packages (not everything is hardcoded in package dependencies -- make sure you got percona-xtrabackup installed in the correct version, and also socat). If SST fails, do not only look into mysqls primary error logs, but also into logfiles from the SST tool in /var/lib/mysql on the donor node.

=== Notes about configuring OX for use with Galera ===

==== Write requests ====

Open-Xchange supports Galera as database backend only in the configuration where all writes are directed to one Galera node. For availability, it makes sense to not configure one Galera node's IP address directly, but rather employ some HA solution which offers active-passive functionality. Options therefore are discussed below.

==== Read requests ====

Read requests can be directed to any node in the Galera cluster. Our standard approach is to recommend to use a loadbalancer to implement round-robin over all nodes in a Galera cluster for the read requests. But you can also chose to use a dedicated read node (the same node, or a different node, than the write node). Each of the approaches has its own advantages.

* Load balancer based setup: Read requests get distributed round-robin between the Galera nodes. Theoretically by distributing the load of the read requests, you benefit from lower latencies and more throughput. But this has never been benchmarked yet. For a discussion of available loadbalances, see next section. OX-wise, in this configuration, you have two alternatives:
** The Galera option wsrep_causal_reads=1 option enables you to configure OX with its replication monitor disabled (com.openexchange.database.replicationMonitor=false in configdb.properties). This is the setup which seems to perform best according to our experience as turning off the replication monitor reduces the commits on the DB and thus the write operations per second on the underlying storage significantly, which outweights the drawback from having higher commit latency due to fully synchronous mode.
** Alternatively, you can run Galera with wsrep_causal_reads=0 when switching on OX builtin replication monitor. This is also a valid setup.
* Use a designated floating IP for the read requests: This eliminates the need of a load balancer. With this option you will not gain any performance, but the quantitative benefit is unclear anyhow.
* Use the floating IP for the writes also for the reads: In this scenario, you direct all database queries only to one Galera node, and the other two nodes are only getting queries in case of a failure of that node. In this case, you can even use wsrep_causal_reads=0 while still having OX builtin replication monitor switched off. However we do not expect this option to be superior to the round-robin loadbalancer approach.

=== Loadbalancer options ===

While the JDBC driver has some round-robin load balancing capabilities built-in, we don't recommend it for production use since it lacks possibilities to check the Galera nodes health states.

Loadbalancers used for OX -> Galera loadbalancing should be able to implement active-passive instances for the write requests, and active-active (round-robin) instances for the read requests. (If they cannot implement active-passive, you can still take a floating IP therefore.) Furthermore it is required to configure node health checks not only on the TCP level (by a simple connect), but to query the Galera health status periodically, evaluating Galera WSREP status variables. Otherwise split-brain scenarios or other bad states cannot be detected. For an example of such an health check, see our [[Clustercheck]] page.

Some customers use loadbalancing appliances. It is important to check that if the (virtual) infrastructure offers "loadbalancer" instances that they satisfy the given requirements. Often this is not the case. In particular, a simple "DNS round robin" approach is not viable.

==== LVS/ipvsadm/keepalived ====

If you want to create your own loadbalancers based on Linux, we usually recommend LVS (Linux Virtual Servers) controlled by Keepalived. LVS is a set of kernel modules implementing a L4 loadbalancer which performs quite well. Keepalived is a userspace daemon to control LVS rules, using health checks to reconfigure LVS rules if required. Keepalived / LVS requires one (or, for availability, two) dedicated linux nodes to run on. This can be a disadvantage for some installations, but usually, it pays off. We provide some configuration information on Keepalived [[Keepalived|here]].

==== MariaDB Maxscale ====

Since Maxscale has become GA in 2015, it seems to have undergone significant stability, performance and functional improvements. We are currently experimenting with Maxscale and share our installation / configuration knowledge [[Maxscale|here]]. It looks quite promising and might become ''the standard replacement'' for HAproxy, while we still presume Keepalived offers superior robustness and performance, coming with the cost of the requirement for one (or more) dedicated loadbalancer nodes.

==== HAproxy ====

In case where the Keepalived based approach is not feasible due to its requirements on the infrastructure, it is also possible to use a HAproxy based solution where HAproxy processes run on each of the OX nodes, configured for one round-robin and one active/passive instance. OX is then connecting to the local HAproxy instances. It is vital to configure HAproxy timeouts different from the defaults, otherwise HAproxy will kill active DB connections, causing errors. Be aware that in large installations the number of (distributed) HAproxy instances can get quite large. Some configuration hints for HAproxy are available [[HAproxy|here]].

== Master/Slave database setup ==

While we also support also "legacy" (pre-GTID) Master/Slave replication, we recommend to use GTID based replication, for easier setup and failure recovery. Support for GTID based replication has been added with OX 7.8.0.

GTID has been available since MySQL 5.6, so no 5.5 installation instructions below, sorry. We try to be generic in this documentation (thus, applicable to Oracle Community Edition and MariaDB) and point out differences where needed. Note: Instructions below include information about Oracle Community MySQL 5.7 which is not yet formally supported.

=== Preparations ===

Our configuration process includes wiping and reinitializing the datadir. This is usually not a problem in a fresh installation. If you want to upgrade an existing database to GTID master-slave, please be prepared to wipe the datadir, i.e. take a <code>mysqldump</code> for later restoration into the properly configured master.

Depeding on the flavor of the current database, this can be something like

# mariadb or oracle mysql without GTIDs
mysqldump --databases configdb oxdb_{5..14} > backup.sql

# mysql 5.6 with GTIDs... we dont want GTIDs here
mysqldump --databases --set-gtid-purged=OFF configdb oxdb_{5..14} > backup.sql

Be sure to verify the list of databases.

=== Installation ===

Software installation is identical for master and slave.

Please follow the instructions for installing from The vendors.

* Oracle Community Edition: https://dev.mysql.com/doc/mysql-apt-repo-quick-guide/en/
* MariaDB (10.0, 10.1): https://downloads.mariadb.org/mariadb/repositories/

Stop the service (if it is running):

service mysql stop

=== Configuration ===

Configuration as per configuration files is also identical for master and slave.

Consult [[My.cnf]] for general recommendations how to configure databases for usage with OX.

For GTID based replication, make sure you add some configurables to a new <code>/etc/mysql/ox.conf.d/gtid.cnf</code> file (assuming you are following our proposed schema of adding a <code>!includedir /etc/mysql/ox.conf.d/</code>" directive to <code>/etc/mysql/my.cnf</code>):

# GTID
log-bin=mysql-bin
server-id=...
log_slave_updates = ON

Oracle Community Edition: we need to add also

enforce_gtid_consistency = ON
gtid_mode = ON

(GTID mode is on by default on MariaDB.)

Use unique a <code>server-id</code> for each server; like <code>1</code> for the master, <code>2</code> for slave. For more complicated setups (like multiple slaves), adjust accordingly.

Since applying our configuration / sizing requires reinitialization of the MySQL datadir, we wipe/recreate it. Caution: this assumes we are running an empty database. If there is data in the database you want to keep, use mysqldump. See Preparation section above.

So, to initialize the datadir:

cd /var/lib/
mv mysql mysql.old.datadir
mkdir mysql
chown mysql.mysql mysql

(When coming from an existing installation, be sure to wipe also old binlogs. They can confuse the server on startup. Their location varies by configuration.)

The step to initialize the datadir is different for the different DBs:

# MariaDB 10.0, 10.1
mysql_install_db

# MariaDB 10.2
mysql_install_db --user=mysql

# Oracle 5.6
mysql_install_db -u mysql

# Oracle 5.7
mysqld --initialize-insecure --user=mysql

(Don't be worried about the <code>insecure</code>, it just means we set the db root pw in the next steps.)

Then:

service mysql restart
mysql_secure_installation

We want to emphasize the last step to run "secure".

Steps up to here apply to both the designated master and slave. The next steps will apply to the master.

=== Replication Setup ===

==== Master Setup ====

Create a replication user on the master (but, as always, pick your own password, and use the same password in the slave setup below):

mysql -e "CREATE USER 'repl'@'gtid-slave.localdomain' IDENTIFIED BY 'IvIjyoffod2'; GRANT REPLICATION SLAVE ON *.* TO 'repl'@'gtid-slave.localdomain';"

Now would also be the time to restore a previously created mysqldump, or add other users you need for adminstration, monitoring etc (like <code>debian-sys-maint@localhost</code>, for example). Adding the OX users is explained below ("Creating Open-Xchange user").

# If you took a dump for restore before
mysql < backup.sql

To prepare for the initial sync of the slave, set the master read-only:

mysql -e "SET @@global.read_only = ON;"

Create a dump to initialize the slave:

# MariaDB
mysqldump --all-databases --triggers --routines --events --master-data --gtid > master.sql

# Oracle
mysqldump --all-databases --triggers --routines --events --set-gtid-purged=ON > master.sql

Transfer to the slave:

scp master.sql gtid-slave:

==== Slave Setup ====

Configure the replication master settings. Note we don't need complicated binlog position settings etc with GTID.

Yet again DB-specific (use the repl user password from above):

# MariaDB
mysql -e 'CHANGE MASTER TO MASTER_HOST="gtid-master.localdomain", MASTER_USER="repl", MASTER_PASSWORD="IvIjyoffod2";'

# Oracle
mysql -e "CHANGE MASTER TO MASTER_HOST='gtid-master.localdomain', MASTER_USER='repl', MASTER_PASSWORD='IvIjyoffod2', MASTER_AUTO_POSITION=1;"
# https://www.percona.com/blog/2013/02/08/how-to-createrestore-a-slave-using-gtid-replication-in-mysql-5-6/
mysql -e "RESET MASTER;"

Read the master dump:

mysql < master.sql

Start replication on the slave:

mysql -e 'START SLAVE;'
mysql -e 'SHOW SLAVE STATUS\G'

==== Master Setup (continued) ====

Finally, unset read-only on the master:

# on the master
mysql -e "SET @@global.read_only = OFF;"

=== Configure OX to use with Master/Slave replication ===

Not much special wisdom here. OX was designed to be used with master/slave databases. For the ConfigDB, <code>configdb.properties</code> allows configuration of a <code>readUrl</code> and <code>writeUrl</code> (both of which are set to the correct values if you use <code>oxinstaller</code> with the correct arguments <code>--configdb-readhost</code>, <code>--configdb-writehost</code>).

(Obviously, the master is for writing and the slave is for reading.)

For the individiual UserDBs, use <code>registerdatabase -m true</code> for the masters and <code>registerdatabase -m false -M ...</code> for the respective slaves.

Be sure to have enabled the replication monitor in <code>configdb.properties</code>: <code>com.openexchange.database.replicationMonitor=true</code> (which it is by default); while GTID can show synchronous semantics, it is specified to silently fall back to asynchronous in certain circumstances, so synchronity is not guaranteed.

We recommend, though, to not register the databases directly by their native hostname or IP, but rather use some kind of HA system in order to be able to easily move a floating/failover IP from the master to the slave in case of master failure. Configuring and running such systems (like, corosync/pacemaker, keepalived, or whatever) is out of scope of this documentation, however.

== Creating Open-Xchange user ==

Now setup access for the Open-Xchange Server database user 'openexchange' to configdb and the oxdb for both groupware server addresses. These databases do not exist yet, but will be created during the Open-Xchange Server installation.

Notes:

* Please use a real password.
* The IPs in this example belong to the two different Open-Xchange Servers, please adjust them accordingly.
* If using a database on the same host as the middlware (usually done for POCs and demo installations), you need to grant also to the ''localhost'' host.
* Consult [[AppSuite:DB_user_privileges]] (or ''grep GRANT /opt/open-xchange/sbin/initconfigdb'') for an up-to-date list of required privileges. The following statement was correct as of the time of writing this section.

mysql> GRANT CREATE, LOCK TABLES, REFERENCES, INDEX, DROP, DELETE, ALTER, SELECT, UPDATE, INSERT, CREATE TEMPORARY TABLES, SHOW VIEW, SHOW DATABASES ON *.* TO 'openexchange'@'10.20.30.213' IDENTIFIED BY 'IntyoyntOat1' WITH GRANT OPTION;
mysql> GRANT CREATE, LOCK TABLES, REFERENCES, INDEX, DROP, DELETE, ALTER, SELECT, UPDATE, INSERT, CREATE TEMPORARY TABLES, SHOW VIEW, SHOW DATABASES ON *.* TO 'openexchange'@'10.20.30.215' IDENTIFIED BY 'IntyoyntOat1' WITH GRANT OPTION;

Template:OXLoadBalancingClustering Database

2018-09-27T13:35:03Z

Dominik.epple: /* Configuration */

== Overview ==

You can choose between Galera or Master/Slave replication. We like to recommend to use Galera for higher redudancy, easier operations, und synchronous semantics (so you can run OX without our "replication monitor"). For POC or demo setups, a single standalone database setup might be sufficient.

== Standalone database setup ==

=== Preparations ===

Our configuration process includes wiping and reinitializing the datadir. This is usually not a problem in a fresh installation. If you want to upgrade an existing database server, please be prepared to wipe the datadir, i.e. take a <code>mysqldump</code> for later restoration into the properly configured master.

mysqldump --databases configdb oxdb_{5..14} > backup.sql

Be sure to verify the list of databases.

=== Installation ===

Note: the following list ist not an exclusive list or authorative statement about supported MySQL flavors / versions. Please consult the official support / system requirements statement.

Please follow the upstream docs for your preferred flavor to get the software installed on your system.

* MariaDB (10.1, 10.2): https://downloads.mariadb.org/
* Oracle MySQL Community Server (5.6, 5.7): https://dev.mysql.com/downloads/mysql/

Make sure to doublecheck the service is not running (or stop it) after installation as we need to perform some reconfigurations.

service mysql stop

=== Configuration ===

MySQL configuration advise is given in our [[My.cnf|MySQL configuration article]]. Please consult that page for configuration information and create configuration files as described there.

Some settings we recommend to change require that the database gets re-initialized. We assume you don't have data there (since we are covering a fresh install) or you have taken a backup for later restore as explained above in the Preparations section.

cd /var/lib/
mv mysql mysql.old.datadir
mkdir mysql
chown mysql.mysql mysql

# mariadb
mysql_install_db
# mariadb 10.2
mysql_install_db --user=mysql
# oracle 5.6
mysql_install_db -u mysql
# oracle 5.7
mysqld --initialize-insecure --user=mysql

(Don't be worried about the <code>insecure</code>, it just means we set the db root pw in the next steps.)

Start the service. The actual command depends on your OS and on the MySQL flavor.

service mysql start

Run <code>mysql_secure_installation</code> for a "secure by default" installation:

mysql_secure_installation

That tool will ask for the current root password (which is empty by default) and subsequently questions like:

Change the root password? [Y/n]
Remove anonymous users? [Y/n]
Disallow root login remotely? [Y/n]
Remove test database and access to it? [Y/n]
Reload privilege tables now? [Y/n]

You should answer all these questions with "yes".

Configure a strong password for the MySQL <code>root</code> user.

The further steps in this guide omit <code>-u -p</code> arguments to the MySQL client. Rather than passing them on the command line [https://dev.mysql.com/doc/refman/5.7/en/password-security-user.html] it is recommended to place the credentials in a file like <code>/root/.my.cnf</code> like

[client]
user=root
password=wip9Phae3Beijeed

Make sure the service is enabled by the OS's init system. The actual command depends on your OS and on the MySQL flavor.

systemctl enable mysql.service

You should now be able to restore your previously taken backup.

# If you took a dump for restore before
mysql < backup.sql

=== Configure OX to use with a standalone database ===

Not much special wisdom here. OX was designed to be used with master/slave databases, and a standalone master works just as well, if we register it as a master, and not registering a slave.

For the ConfigDB, <code>configdb.properties</code> allows configuration of a <code>writeUrl</code> (which is set to the correct values if you use <code>oxinstaller</code> with the correct argument <code>--configdb-writehost</code>).

The single database is then used for reading and writing.

For the individiual UserDBs, use <code>registerdatabase -m true</code>.

== Galera database setup ==

=== Preparations ===

Our configuration process includes wiping and reinitializing the datadir. This is usually not a problem in a fresh installation. If you want to upgrade an existing database to Galera cluster, please be prepared to wipe the datadir, i.e. take a <code>mysqldump</code> for later restoration into the properly configured master.

Depeding on the flavor of the current database, this can be something like

# mariadb or oracle mysql without GTIDs
mysqldump --databases configdb oxdb_{5..14} > backup.sql

# mysql 5.6 with GTIDs... we dont want GTIDs here
mysqldump --databases --set-gtid-purged=OFF configdb oxdb_{5..14} > backup.sql

Be sure to verify the list of databases.

=== Installation ===

Please follow the upstream docs for your preferred flavor to get the software installed on your system.

* Percona XtraDB Cluster (5.6, 5.7): https://www.percona.com/doc/percona-xtradb-cluster/LATEST/install/index.html
* MariaDB Galera Cluster (10.0, 10.1): https://mariadb.com/kb/en/library/getting-started-with-mariadb-galera-cluster/ (Note: with 10.0, socat is required, but not a package dependency, so you need to explicitly install also socat)

Make sure to doublecheck the service is not running (or stop it) after installation as we need to perform some reconfigurations.

service mysql stop

=== Configuration ===

Galera-specific MySQL configuration advise is included in our main [[My.cnf|MySQL configuration article]]. Please consult that page for configuration information.

That page suggests a setup were we add three custom config files to <code>/etc/mysql/ox.conf.d/</code>: <code>ox.cnf</code> for general tuning/sizing, <code>wsrep.cnf</code> for clusterwide galera configuration, and <code>host.cnf</code> for host-specific settings.

Adjust the general settings and tunings in <code>ox.cnf</code> according to your sizing etc.

Adjust <code>wsrep.cnf</code> to reflect local paths, cluster member addresses, etc.

Adjust <code>host.cnf</code> to give node-local IPs, etc.

Version-specific hints:

# percona 5.6: unknown variable 'pxc_strict_mode=ENFORCING' ... unset that one
# mariadb 10.1: add wsrep_on=ON
# mariadb 10.0 and 10.1: set wsrep_node_incoming_address=192.168.1.22:3306 in host.cnf, otherwise the status wsrep_incoming_addresses might not be shown correctly(?!)

Some settings we recommend to change require that the database gets re-initialized. We assume you don't have data there (since we are covering a fresh install) or you have taken a backup for later restore as explained above in the Preparations section.

cd /var/lib/
mv mysql mysql.old.datadir
mkdir mysql
chown mysql.mysql mysql

# mariadb 10.0 and 10.1
mysql_install_db
# mariadb 10.2
mysql_install_db --user=mysql
# percona 5.6
mysqld --user=mysql
# percona 5.7
mysqld --initialize-insecure --user=mysql

(Don't be worried about the <code>insecure</code>, it just means we set the db root pw in the next steps.)

=== Cluster startup ===

Typically on startup a Galera node tries to join a cluster, and if it fails, it will exit. Thus, when no cluster nodes are running, the first cluster node to be started needs to be told to not try to join a cluster, but rather bootstrap a new cluster. The exact arguments vary from version to version and from flavor to flavor.

==== First node ====

So we initialize the cluster bootstrap on the first node:

# percona 5.6, 5.7
service mysql bootstrap-pxc
# mariadb 10.0
service mysql bootstrap
# mariadb 10.1: service mysql bootstrap seems to be broken, does not pass the necessary options to mysqld
galera_new_cluster

Run <code>mysql_secure_installation</code> for a "secure by default" installation:

mysql_secure_installation

The further steps in this guide omit <code>-u -p</code> arguments to the MySQL client. Rather than passing them on the command line [https://dev.mysql.com/doc/refman/5.7/en/password-security-user.html] it is recommended to place the credentials in a file like <code>/root/.my.cnf</code> like

[client]
user=root
password=wip9Phae3Beijeed

We need a Galera replication user:

CREATE USER 'sstuser'@'localhost' IDENTIFIED BY 'OpIdjijwef0';
-- percona 5.6, mariadb 10.0
GRANT RELOAD, LOCK TABLES, REPLICATION CLIENT ON *.* TO 'sstuser'@'localhost';
-- percona 5.7, mariadb 10.1
GRANT PROCESS, RELOAD, LOCK TABLES, REPLICATION CLIENT ON *.* TO 'sstuser'@'localhost';
FLUSH PRIVILEGES;

(Debian specific note: MariaDB provided startup scripts use the distro's mechanism of verifying startup/shutdown using a system user, so we create that as well:

# mariadb 10.0, 10.1
GRANT ALL PRIVILEGES ON *.* TO "debian-sys-maint"@"localhost" IDENTIFIED BY "adBexthTsI5TaEps";

If you do this, yo need to synchronize the <code>/etc/mysql/debian.cnf</code> file from the first node to the other nodes as well.)

==== Other nodes ====

On the other nodes, we only need to restart the service now, to trigger a full state transfer from the first node to the other nodes.

We recommend to do this serially to let one state transfer complete before the second state transfer.

==== First node (continued) ====

Only applicable if you used <code>galera_new_cluster</code> before rather than the service script: In order to get the systemctl status consistent, restart the service on the first node:

# mariadb 10.1: restart the service so that the systemctl status is consistent
mysqladmin shutdown
service mysql bootstrap

=== Verify the replication ===

The key tool to verify replication status is

mysql> show status like "%wsrep%";

This will give a lot of output. You want to verify in particular

+------------------------------+--------------------------------------+
| Variable_name | Value |
+------------------------------+--------------------------------------+
| wsrep_cluster_size | 3 |
| wsrep_cluster_status | Primary |
| wsrep_local_state | 4 |
| wsrep_local_state_comment | Synced |
| wsrep_ready | ON |
+------------------------------+--------------------------------------+

You can also explicitly verify replication by creating / inserting DBs, tables, rows on one node and select on other nodes.

==== Troubleshooting ====

The logs are helpful. Always.

Common mistakes are listed below.

If the Galera module does not get loaded at all:
* Configuration settings in ''my.cnf'' which are incompatible to Galera
* Wrong path of the shared object providing the Galera plugin in wsrep.cnf (wsrep_provider)

If the first node starts, but the second / third nodes can not be added to the cluster:
* User for the replication not created correctly on the first Galera node
* SST fails due to missing / wrong version prerequisite packages (not everything is hardcoded in package dependencies -- make sure you got percona-xtrabackup installed in the correct version, and also socat). If SST fails, do not only look into mysqls primary error logs, but also into logfiles from the SST tool in /var/lib/mysql on the donor node.

=== Notes about configuring OX for use with Galera ===

==== Write requests ====

Open-Xchange supports Galera as database backend only in the configuration where all writes are directed to one Galera node. For availability, it makes sense to not configure one Galera node's IP address directly, but rather employ some HA solution which offers active-passive functionality. Options therefore are discussed below.

==== Read requests ====

Read requests can be directed to any node in the Galera cluster. Our standard approach is to recommend to use a loadbalancer to implement round-robin over all nodes in a Galera cluster for the read requests. But you can also chose to use a dedicated read node (the same node, or a different node, than the write node). Each of the approaches has its own advantages.

* Load balancer based setup: Read requests get distributed round-robin between the Galera nodes. Theoretically by distributing the load of the read requests, you benefit from lower latencies and more throughput. But this has never been benchmarked yet. For a discussion of available loadbalances, see next section. OX-wise, in this configuration, you have two alternatives:
** The Galera option wsrep_causal_reads=1 option enables you to configure OX with its replication monitor disabled (com.openexchange.database.replicationMonitor=false in configdb.properties). This is the setup which seems to perform best according to our experience as turning off the replication monitor reduces the commits on the DB and thus the write operations per second on the underlying storage significantly, which outweights the drawback from having higher commit latency due to fully synchronous mode.
** Alternatively, you can run Galera with wsrep_causal_reads=0 when switching on OX builtin replication monitor. This is also a valid setup.
* Use a designated floating IP for the read requests: This eliminates the need of a load balancer. With this option you will not gain any performance, but the quantitative benefit is unclear anyhow.
* Use the floating IP for the writes also for the reads: In this scenario, you direct all database queries only to one Galera node, and the other two nodes are only getting queries in case of a failure of that node. In this case, you can even use wsrep_causal_reads=0 while still having OX builtin replication monitor switched off. However we do not expect this option to be superior to the round-robin loadbalancer approach.

=== Loadbalancer options ===

While the JDBC driver has some round-robin load balancing capabilities built-in, we don't recommend it for production use since it lacks possibilities to check the Galera nodes health states.

Loadbalancers used for OX -> Galera loadbalancing should be able to implement active-passive instances for the write requests, and active-active (round-robin) instances for the read requests. (If they cannot implement active-passive, you can still take a floating IP therefore.) Furthermore it is required to configure node health checks not only on the TCP level (by a simple connect), but to query the Galera health status periodically, evaluating Galera WSREP status variables. Otherwise split-brain scenarios or other bad states cannot be detected. For an example of such an health check, see our [[Clustercheck]] page.

Some customers use loadbalancing appliances. It is important to check that if the (virtual) infrastructure offers "loadbalancer" instances that they satisfy the given requirements. Often this is not the case. In particular, a simple "DNS round robin" approach is not viable.

==== LVS/ipvsadm/keepalived ====

If you want to create your own loadbalancers based on Linux, we usually recommend LVS (Linux Virtual Servers) controlled by Keepalived. LVS is a set of kernel modules implementing a L4 loadbalancer which performs quite well. Keepalived is a userspace daemon to control LVS rules, using health checks to reconfigure LVS rules if required. Keepalived / LVS requires one (or, for availability, two) dedicated linux nodes to run on. This can be a disadvantage for some installations, but usually, it pays off. We provide some configuration information on Keepalived [[Keepalived|here]].

==== MariaDB Maxscale ====

Since Maxscale has become GA in 2015, it seems to have undergone significant stability, performance and functional improvements. We are currently experimenting with Maxscale and share our installation / configuration knowledge [[Maxscale|here]]. It looks quite promising and might become ''the standard replacement'' for HAproxy, while we still presume Keepalived offers superior robustness and performance, coming with the cost of the requirement for one (or more) dedicated loadbalancer nodes.

==== HAproxy ====

In case where the Keepalived based approach is not feasible due to its requirements on the infrastructure, it is also possible to use a HAproxy based solution where HAproxy processes run on each of the OX nodes, configured for one round-robin and one active/passive instance. OX is then connecting to the local HAproxy instances. It is vital to configure HAproxy timeouts different from the defaults, otherwise HAproxy will kill active DB connections, causing errors. Be aware that in large installations the number of (distributed) HAproxy instances can get quite large. Some configuration hints for HAproxy are available [[HAproxy|here]].

== Master/Slave database setup ==

While we also support also "legacy" (pre-GTID) Master/Slave replication, we recommend to use GTID based replication, for easier setup and failure recovery. Support for GTID based replication has been added with OX 7.8.0.

GTID has been available since MySQL 5.6, so no 5.5 installation instructions below, sorry. We try to be generic in this documentation (thus, applicable to Oracle Community Edition and MariaDB) and point out differences where needed. Note: Instructions below include information about Oracle Community MySQL 5.7 which is not yet formally supported.

=== Preparations ===

Our configuration process includes wiping and reinitializing the datadir. This is usually not a problem in a fresh installation. If you want to upgrade an existing database to GTID master-slave, please be prepared to wipe the datadir, i.e. take a <code>mysqldump</code> for later restoration into the properly configured master.

Depeding on the flavor of the current database, this can be something like

# mariadb or oracle mysql without GTIDs
mysqldump --databases configdb oxdb_{5..14} > backup.sql

# mysql 5.6 with GTIDs... we dont want GTIDs here
mysqldump --databases --set-gtid-purged=OFF configdb oxdb_{5..14} > backup.sql

Be sure to verify the list of databases.

=== Installation ===

Software installation is identical for master and slave.

Please follow the instructions for installing from The vendors.

* Oracle Community Edition: https://dev.mysql.com/doc/mysql-apt-repo-quick-guide/en/
* MariaDB (10.0, 10.1): https://downloads.mariadb.org/mariadb/repositories/

Stop the service (if it is running):

service mysql stop

=== Configuration ===

Configuration as per configuration files is also identical for master and slave.

Consult [[My.cnf]] for general recommendations how to configure databases for usage with OX.

For GTID based replication, make sure you add some configurables to a new <code>/etc/mysql/ox.conf.d/gtid.cnf</code> file (assuming you are following our proposed schema of adding a <code>!includedir /etc/mysql/ox.conf.d/</code>" directive to <code>/etc/mysql/my.cnf</code>):

# GTID
log-bin=mysql-bin
server-id=...
log_slave_updates = ON

Oracle Community Edition: we need to add also

enforce_gtid_consistency = ON
gtid_mode = ON

(GTID mode is on by default on MariaDB.)

Use unique a <code>server-id</code> for each server; like <code>1</code> for the master, <code>2</code> for slave. For more complicated setups (like multiple slaves), adjust accordingly.

Since applying our configuration / sizing requires reinitialization of the MySQL datadir, we wipe/recreate it. Caution: this assumes we are running an empty database. If there is data in the database you want to keep, use mysqldump. See Preparation section above.

So, to initialize the datadir:

cd /var/lib/
mv mysql mysql.old.datadir
mkdir mysql
chown mysql.mysql mysql

(When coming from an existing installation, be sure to wipe also old binlogs. They can confuse the server on startup. Their location varies by configuration.)

The step to initialize the datadir is different for the different DBs:

# MariaDB 10.0, 10.1
mysql_install_db

# MariaDB 10.2
mysql_install_db --user=mysql

# Oracle 5.6
mysql_install_db -u mysql

# Oracle 5.7
mysqld --initialize-insecure --user=mysql

(Don't be worried about the <code>insecure</code>, it just means we set the db root pw in the next steps.)

Then:

service mysql restart
mysql_secure_installation

We want to emphasize the last step to run "secure".

Steps up to here apply to both the designated master and slave. The next steps will apply to the master.

=== Replication Setup ===

==== Master Setup ====

Create a replication user on the master (but, as always, pick your own password, and use the same password in the slave setup below):

mysql -e "CREATE USER 'repl'@'gtid-slave.localdomain' IDENTIFIED BY 'IvIjyoffod2'; GRANT REPLICATION SLAVE ON *.* TO 'repl'@'gtid-slave.localdomain';"

Now would also be the time to restore a previously created mysqldump, or add other users you need for adminstration, monitoring etc (like <code>debian-sys-maint@localhost</code>, for example). Adding the OX users is explained below ("Creating Open-Xchange user").

# If you took a dump for restore before
mysql < backup.sql

To prepare for the initial sync of the slave, set the master read-only:

mysql -e "SET @@global.read_only = ON;"

Create a dump to initialize the slave:

# MariaDB
mysqldump --all-databases --triggers --routines --events --master-data --gtid > master.sql

# Oracle
mysqldump --all-databases --triggers --routines --events --set-gtid-purged=ON > master.sql

Transfer to the slave:

scp master.sql gtid-slave:

==== Slave Setup ====

Configure the replication master settings. Note we don't need complicated binlog position settings etc with GTID.

Yet again DB-specific (use the repl user password from above):

# MariaDB
mysql -e 'CHANGE MASTER TO MASTER_HOST="gtid-master.localdomain", MASTER_USER="repl", MASTER_PASSWORD="IvIjyoffod2";'

# Oracle
mysql -e "CHANGE MASTER TO MASTER_HOST='gtid-master.localdomain', MASTER_USER='repl', MASTER_PASSWORD='IvIjyoffod2', MASTER_AUTO_POSITION=1;"
# https://www.percona.com/blog/2013/02/08/how-to-createrestore-a-slave-using-gtid-replication-in-mysql-5-6/
mysql -e "RESET MASTER;"

Read the master dump:

mysql < master.sql

Start replication on the slave:

mysql -e 'START SLAVE;'
mysql -e 'SHOW SLAVE STATUS\G'

==== Master Setup (continued) ====

Finally, unset read-only on the master:

# on the master
mysql -e "SET @@global.read_only = OFF;"

=== Configure OX to use with Master/Slave replication ===

Not much special wisdom here. OX was designed to be used with master/slave databases. For the ConfigDB, <code>configdb.properties</code> allows configuration of a <code>readUrl</code> and <code>writeUrl</code> (both of which are set to the correct values if you use <code>oxinstaller</code> with the correct arguments <code>--configdb-readhost</code>, <code>--configdb-writehost</code>).

(Obviously, the master is for writing and the slave is for reading.)

For the individiual UserDBs, use <code>registerdatabase -m true</code> for the masters and <code>registerdatabase -m false -M ...</code> for the respective slaves.

Be sure to have enabled the replication monitor in <code>configdb.properties</code>: <code>com.openexchange.database.replicationMonitor=true</code> (which it is by default); while GTID can show synchronous semantics, it is specified to silently fall back to asynchronous in certain circumstances, so synchronity is not guaranteed.

We recommend, though, to not register the databases directly by their native hostname or IP, but rather use some kind of HA system in order to be able to easily move a floating/failover IP from the master to the slave in case of master failure. Configuring and running such systems (like, corosync/pacemaker, keepalived, or whatever) is out of scope of this documentation, however.

== Creating Open-Xchange user ==

Now setup access for the Open-Xchange Server database user 'openexchange' to configdb and the oxdb for both groupware server addresses. These databases do not exist yet, but will be created during the Open-Xchange Server installation.

Notes:

* Please use a real password.
* The IPs in this example belong to the two different Open-Xchange Servers, please adjust them accordingly.
* If using a database on the same host as the middlware (usually done for POCs and demo installations), you need to grant also to the ''localhost'' host.
* Consult [[AppSuite:DB_user_privileges]] (or ''grep GRANT /opt/open-xchange/sbin/initconfigdb'') for an up-to-date list of required privileges. The following statement was correct as of the time of writing this section.

mysql> GRANT CREATE, LOCK TABLES, REFERENCES, INDEX, DROP, DELETE, ALTER, SELECT, UPDATE, INSERT, CREATE TEMPORARY TABLES, SHOW VIEW, SHOW DATABASES ON *.* TO 'openexchange'@'10.20.30.213' IDENTIFIED BY 'IntyoyntOat1' WITH GRANT OPTION;
mysql> GRANT CREATE, LOCK TABLES, REFERENCES, INDEX, DROP, DELETE, ALTER, SELECT, UPDATE, INSERT, CREATE TEMPORARY TABLES, SHOW VIEW, SHOW DATABASES ON *.* TO 'openexchange'@'10.20.30.215' IDENTIFIED BY 'IntyoyntOat1' WITH GRANT OPTION;

My.cnf

2018-09-25T14:19:13Z

Dominik.epple:

== Introduction ==

This page lists some performance tuning parameters which we recommend for tuning MySQL database services used for Open-Xchange installations.

We cannot guarantee this is an exhaustive list of required settings. So treat this list of tunings as probably required, but not necessarily sufficient settings for optimal MySQL performance.

Furthermore, as MySQL changes over time, settings which have been correct as of the time of writing may become incorrect later.

However, this list of settings is the result of internal performance testing and real world customer feedback, so it should be valid to some extent.

In the end, proper configuration of the database service for performance, but also consistency, durability and high availability is in the responsibility of the customer.

=== Performance items ===

* You should adjust the <code>innodb_buffer_pool_size</code> parameter for reasonable memory usage. Our DB sizing is mainly memory-driven and this is where most of the memory goes. Our standard DB machine sizing assumption is 32 GB if MySQL dedicated memory on a 48 GB total memory machine. On such a machine, you would configure 32 GB for the innodb_buffer_pool size, being aware that MySQL does also require memory for other things, in particular there are some also per-connection related memory spendings, which can become substantial if you allow for a lot of maximum concurrent connections. Please watch your memory configuration carefully, use monitoring and tools like mysqltuner.pl.

* On bigger installations you should use <code>innodb_file_per_table = 1</code>, which is creating single files instead of one big blob. If you change this parameter after the database initialization you have to recreate (like dump/drop and re-import) the tables.

* It can help to put different parts of the mysql datadir (iblog, ibdata) on different filesystems / storage devices. This depends on your infrastructure. Settings herefore are <code>datadir</code>, <code>innodb_data_home_dir</code>, <code>innodb_log_group_home_dir</code>.

* If your storage is fast (handle a lot of IOPS), you may want to adjust the <code>innodb_io_capacity</code> setting, which defines a limit for the IOPS MySQL will create. The default is 200, which is sensible for single spindle disks. But if you have storage appliances with a lot of fast SAS drives, or even SSDs, this limit can be increased greatly.

=== Functional items ===

* Query cache is to be switched off; as we found in our own benchmarks and as backed up by upstreams, this hurts performance in load situations with high concurrency. [https://dev.mysql.com/doc/refman/5.7/en/query-cache.htm The query cache is deprecated as of MySQL 5.7.20, and is removed in MySQL 8.0], so we recommend also to switch that off.
* Starting with App Suite 7.10.0, <code>character_set_server</code> must be set to <code>utf8mb4</code> and <code>collation_server</code> to <code>utf8mb4_general_ci</code>
* Starting with MySQL 5.7 <code>innodb_strict_mode</code> must be disabled.
* Starting with MySQL 5.7 and MariaDB 10.2 <code>sql_mode</code> must be configured according to belows matrix.

==== SQL mode matrix ====

The default for the <code>sql_mode</code> setting changes regularly with MariaDB and MySQL releases and is not even consistent anymore between the two derivates. SQL modes affect how data and queries are handled at runtime. Enabling strict modes might lead to errors in terms of failing queries or even update tasks. We strongly recommend the following configuration to avoid according runtime errors:

{| class="wikitable"
|-
! scope="col"| sql_mode
! scope="col"| App Suite 7.8.4
! scope="col"| App Suite 7.10.0
|-
! scope="row"| MySQL 5.6
| <code>NO_ENGINE_SUBSTITUTION</code>
| <code>NO_ENGINE_SUBSTITUTION</code>
|-
! scope="row"| MySQL 5.7
| <code>NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION</code>
| <code>NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION,ONLY_FULL_GROUP_BY</code>
|-
! scope="row"| MariaDB 10.1
| <code>NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION</code>
| <code>NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION</code>
|-
! scope="row"| MariaDB 10.2
| <code>NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION</code>
| <code>NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION</code>
|-
|}

=== Sample config files ===

For easy deployment we recommend not to edit existing the existing my.cnf file, rather assume the distro provides sane settings for most of the items and override items where needed.

As in MySQL there is a [https://dev.mysql.com/doc/refman/5.7/en/option-files.html last instance wins] semantic in options file parsing, we propose to create a custom include directory "ox.conf.d", put our custom config files therein, and include this directory as latest directory in /etc/mysql/my.cnf.

To put in that directory, we have one main tuning file called tunings.cnf, one galera-related file if galera is in use, and one more galera host-specific file which contains per-host settings if using galera (separate in a file of its own for easier configuration management).

So, start with adding to the existing my.cnf at the very bottom:

!includedir /etc/mysql/ox.conf.d/

Create that directory and put in there the generic ox tunings file /etc/mysql/ox.conf.d/tunings.cnf:

[mysqld]
bind-address = *

#innodb_use_native_aio = 0

table_open_cache = 3072
table_definition_cache = 4096
max_heap_table_size = 64M
tmp_table_size = 64M
max_connections = 505
max_user_connections = 500
max_allowed_packet = 16M
thread_cache_size = 32
query_cache_size = 0
query_cache_type = 0
default_storage_engine = InnoDB
innodb_buffer_pool_size = 32G
# the default value in MySQL 5.6.6 and higher is 8 when innodb_buffer_pool_size is greater than or equal to 1GB. Otherwise, the default is 1.
innodb_buffer_pool_instances = 32
innodb_data_file_path = ibdata1:128M:autoextend
innodb_file_per_table = 1
# innodb_log_file_size should be 25% of the innodb_buffer_pool_size
innodb_log_file_size = 4GB
# default and recommended value is 2
innodb_log_files_in_group = 2
# adjust according to your storage
#innodb_io_capacity = 1000

# we are unsure about this setting. Newer versions of MariaDB seem to be fine with low (=1) settings for this value.
# Traditionally we encountered values up to 4x the number of cores.
# Default seems to be number of cores, so let's stick the default
# In the end, we need to leave this setting up to you: if you dont get full cpu utilization in cpu-bound situations, this might be a setting to increase.
#thread_pool_size = 32

binlog_cache_size = 1M
sync_binlog = 8
binlog_format = row

character_set_server = utf8
collation_server = utf8_general_ci

# This was default_table_type previous to MySQL 5.5
default_storage_engine = InnoDB

innodb_autoinc_lock_mode = 2

# keep until 5.6, deprecated later
innodb_locks_unsafe_for_binlog = 1

# we found this has huge impact on (galera) performance
# default (consistent) setting of 1 greatly severs performance
# in galera (or async master-slave) deployments, you might be ok with setting this to 0 or 2,
# assuming our consistency / availability comes from replication / other cluster nodes
innodb_flush_log_at_trx_commit = 0

# for performance testing systems, to not use excessive disk space
#expire_logs_days = 1

# MySQL 5.7.7 has changed the default to 1. Disable it explicitly to prevent from errors based on invalid data stored by former App Suite or MySQL versions.
innodb_strict_mode = 0

# The following value refers to App Suite 7.10 on top of MySQL 5.7. For other combinations see the sql mode matrix at http://oxpedia.org/wiki/index.php?title=My.cnf.
sql_mode = NO_ENGINE_SUBSTITUTION,NO_AUTO_CREATE_USER,ONLY_FULL_GROUP_BY

If using galera, use the following galera configuration file /etc/mysql/ox.conf.d/wsrep.cnf:

[mysqld]
# adjust for your distros SO location
wsrep_provider=/usr/lib/libgalera_smm.so

# this is the big winner and enables us to switch off OX's replication monitor
wsrep_sync_wait=1

# pick a unique cluster name
wsrep_cluster_name=devcluster
# adjust for your IPs / hostnames
wsrep_cluster_address=gcomm://10.20.29.68,10.20.29.69,10.20.29.70

# put this in host.cnf
#wsrep_node_name=...
#wsrep_node_address=...

wsrep_sst_method=xtrabackup-v2
# wsrep_sst_auth if of format username:password
# pick whatever you configured on the donor node
wsrep_sst_auth=sstuser:...

# galera-specific tunings
wsrep_slave_threads = 32

Galera-related host-specific settings go in /etc/mysql/ox.conf.d/host.cnf:

[mysqld]
# the nodes hostname
wsrep_node_name=...
# and the IP of the wsrep relevant interface, if multiple
wsrep_node_address=10.20.29.68

Template:OXLoadBalancingClustering Database

2018-09-06T13:50:40Z

Dominik.epple: /* Configuration */

== Overview ==

You can choose between Galera or Master/Slave replication. We like to recommend to use Galera for higher redudancy, easier operations, und synchronous semantics (so you can run OX without our "replication monitor"). For POC or demo setups, a single standalone database setup might be sufficient.

== Standalone database setup ==

=== Preparations ===

Our configuration process includes wiping and reinitializing the datadir. This is usually not a problem in a fresh installation. If you want to upgrade an existing database server, please be prepared to wipe the datadir, i.e. take a <code>mysqldump</code> for later restoration into the properly configured master.

mysqldump --databases configdb oxdb_{5..14} > backup.sql

Be sure to verify the list of databases.

=== Installation ===

Note: the following list ist not an exclusive list or authorative statement about supported MySQL flavors / versions. Please consult the official support / system requirements statement.

Please follow the upstream docs for your preferred flavor to get the software installed on your system.

* MariaDB (10.1, 10.2): https://downloads.mariadb.org/
* Oracle MySQL Community Server (5.6, 5.7): https://dev.mysql.com/downloads/mysql/

Make sure to doublecheck the service is not running (or stop it) after installation as we need to perform some reconfigurations.

service mysql stop

=== Configuration ===

MySQL configuration advise is given in our [[My.cnf|MySQL configuration article]]. Please consult that page for configuration information and create configuration files as described there.

Some settings we recommend to change require that the database gets re-initialized. We assume you don't have data there (since we are covering a fresh install) or you have taken a backup for later restore as explained above in the Preparations section.

cd /var/lib/
mv mysql mysql.old.datadir
mkdir mysql
chown mysql.mysql mysql

# mariadb
mysql_install_db
# mariadb 10.2
mysql_install_db --user=mysql
# oracle 5.6
mysql_install_db -u mysql
# oracle 5.7
mysqld --initialize-insecure --user=mysql

(Don't be worried about the <code>insecure</code>, it just means we set the db root pw in the next steps.)

Start the service. The actual command depends on your OS and on the MySQL flavor.

service mysql start

Run <code>mysql_secure_installation</code> for a "secure by default" installation:

mysql_secure_installation

That tool will ask for the current root password (which is empty by default) and subsequently questions like:

Change the root password? [Y/n]
Remove anonymous users? [Y/n]
Disallow root login remotely? [Y/n]
Remove test database and access to it? [Y/n]
Reload privilege tables now? [Y/n]

You should answer all these questions with "yes".

Configure a strong password for the MySQL <code>root</code> user.

The further steps in this guide omit <code>-u -p</code> arguments to the MySQL client. Rather than passing them on the command line [https://dev.mysql.com/doc/refman/5.7/en/password-security-user.html] it is recommended to place the credentials in a file like <code>/root/.my.cnf</code> like

[client]
user=root
password=wip9Phae3Beijeed

Make sure the service is enabled by the OS's init system. The actual command depends on your OS and on the MySQL flavor.

systemctl enable mysql.service

You should now be able to restore your previously taken backup.

# If you took a dump for restore before
mysql < backup.sql

=== Configure OX to use with a standalone database ===

Not much special wisdom here. OX was designed to be used with master/slave databases, and a standalone master works just as well, if we register it as a master, and not registering a slave.

For the ConfigDB, <code>configdb.properties</code> allows configuration of a <code>writeUrl</code> (which is set to the correct values if you use <code>oxinstaller</code> with the correct argument <code>--configdb-writehost</code>).

The single database is then used for reading and writing.

For the individiual UserDBs, use <code>registerdatabase -m true</code>.

== Galera database setup ==

=== Preparations ===

Our configuration process includes wiping and reinitializing the datadir. This is usually not a problem in a fresh installation. If you want to upgrade an existing database to Galera cluster, please be prepared to wipe the datadir, i.e. take a <code>mysqldump</code> for later restoration into the properly configured master.

Depeding on the flavor of the current database, this can be something like

# mariadb or oracle mysql without GTIDs
mysqldump --databases configdb oxdb_{5..14} > backup.sql

# mysql 5.6 with GTIDs... we dont want GTIDs here
mysqldump --databases --set-gtid-purged=OFF configdb oxdb_{5..14} > backup.sql

Be sure to verify the list of databases.

=== Installation ===

Please follow the upstream docs for your preferred flavor to get the software installed on your system.

* Percona XtraDB Cluster (5.6, 5.7): https://www.percona.com/doc/percona-xtradb-cluster/LATEST/install/index.html
* MariaDB Galera Cluster (10.0, 10.1): https://mariadb.com/kb/en/library/getting-started-with-mariadb-galera-cluster/ (Note: with 10.0, socat is required, but not a package dependency, so you need to explicitly install also socat)

Make sure to doublecheck the service is not running (or stop it) after installation as we need to perform some reconfigurations.

service mysql stop

=== Configuration ===

Galera-specific MySQL configuration advise is included in our main [[My.cnf|MySQL configuration article]]. Please consult that page for configuration information.

That page suggests a setup were we add three custom config files to <code>/etc/mysql/ox.conf.d/</pre>: <code>ox.cnf</code> for general tuning/sizing, <code>wsrep.cnf</code> for clusterwide galera configuration, and <code>host.cnf</code> for host-specific settings.

Adjust the general settings and tunings in <code>ox.cnf</code> according to your sizing etc.

Adjust <code>wsrep.cnf</code> to reflect local paths, cluster member addresses, etc.

Adjust <code>host.cnf</code> to give node-local IPs, etc.

Version-specific hints:

# percona 5.6: unknown variable 'pxc_strict_mode=ENFORCING' ... unset that one
# mariadb 10.1: add wsrep_on=ON
# mariadb 10.0 and 10.1: set wsrep_node_incoming_address=192.168.1.22:3306 in host.cnf, otherwise the status wsrep_incoming_addresses might not be shown correctly(?!)

Some settings we recommend to change require that the database gets re-initialized. We assume you don't have data there (since we are covering a fresh install) or you have taken a backup for later restore as explained above in the Preparations section.

cd /var/lib/
mv mysql mysql.old.datadir
mkdir mysql
chown mysql.mysql mysql

# mariadb 10.0 and 10.1
mysql_install_db
# mariadb 10.2
mysql_install_db --user=mysql
# percona 5.6
mysqld --user=mysql
# percona 5.7
mysqld --initialize-insecure --user=mysql

(Don't be worried about the <code>insecure</code>, it just means we set the db root pw in the next steps.)

=== Cluster startup ===

Typically on startup a Galera node tries to join a cluster, and if it fails, it will exit. Thus, when no cluster nodes are running, the first cluster node to be started needs to be told to not try to join a cluster, but rather bootstrap a new cluster. The exact arguments vary from version to version and from flavor to flavor.

==== First node ====

So we initialize the cluster bootstrap on the first node:

# percona 5.6, 5.7
service mysql bootstrap-pxc
# mariadb 10.0
service mysql bootstrap
# mariadb 10.1: service mysql bootstrap seems to be broken, does not pass the necessary options to mysqld
galera_new_cluster

Run <code>mysql_secure_installation</code> for a "secure by default" installation:

mysql_secure_installation

The further steps in this guide omit <code>-u -p</code> arguments to the MySQL client. Rather than passing them on the command line [https://dev.mysql.com/doc/refman/5.7/en/password-security-user.html] it is recommended to place the credentials in a file like <code>/root/.my.cnf</code> like

[client]
user=root
password=wip9Phae3Beijeed

We need a Galera replication user:

CREATE USER 'sstuser'@'localhost' IDENTIFIED BY 'OpIdjijwef0';
-- percona 5.6, mariadb 10.0
GRANT RELOAD, LOCK TABLES, REPLICATION CLIENT ON *.* TO 'sstuser'@'localhost';
-- percona 5.7, mariadb 10.1
GRANT PROCESS, RELOAD, LOCK TABLES, REPLICATION CLIENT ON *.* TO 'sstuser'@'localhost';
FLUSH PRIVILEGES;

(Debian specific note: MariaDB provided startup scripts use the distro's mechanism of verifying startup/shutdown using a system user, so we create that as well:

# mariadb 10.0, 10.1
GRANT ALL PRIVILEGES ON *.* TO "debian-sys-maint"@"localhost" IDENTIFIED BY "adBexthTsI5TaEps";

If you do this, yo need to synchronize the <code>/etc/mysql/debian.cnf</code> file from the first node to the other nodes as well.)

==== Other nodes ====

On the other nodes, we only need to restart the service now, to trigger a full state transfer from the first node to the other nodes.

We recommend to do this serially to let one state transfer complete before the second state transfer.

==== First node (continued) ====

Only applicable if you used <code>galera_new_cluster</code> before rather than the service script: In order to get the systemctl status consistent, restart the service on the first node:

# mariadb 10.1: restart the service so that the systemctl status is consistent
mysqladmin shutdown
service mysql bootstrap

=== Verify the replication ===

The key tool to verify replication status is

mysql> show status like "%wsrep%";

This will give a lot of output. You want to verify in particular

+------------------------------+--------------------------------------+
| Variable_name | Value |
+------------------------------+--------------------------------------+
| wsrep_cluster_size | 3 |
| wsrep_cluster_status | Primary |
| wsrep_local_state | 4 |
| wsrep_local_state_comment | Synced |
| wsrep_ready | ON |
+------------------------------+--------------------------------------+

You can also explicitly verify replication by creating / inserting DBs, tables, rows on one node and select on other nodes.

==== Troubleshooting ====

The logs are helpful. Always.

Common mistakes are listed below.

If the Galera module does not get loaded at all:
* Configuration settings in ''my.cnf'' which are incompatible to Galera
* Wrong path of the shared object providing the Galera plugin in wsrep.cnf (wsrep_provider)

If the first node starts, but the second / third nodes can not be added to the cluster:
* User for the replication not created correctly on the first Galera node
* SST fails due to missing / wrong version prerequisite packages (not everything is hardcoded in package dependencies -- make sure you got percona-xtrabackup installed in the correct version, and also socat). If SST fails, do not only look into mysqls primary error logs, but also into logfiles from the SST tool in /var/lib/mysql on the donor node.

=== Notes about configuring OX for use with Galera ===

==== Write requests ====

Open-Xchange supports Galera as database backend only in the configuration where all writes are directed to one Galera node. For availability, it makes sense to not configure one Galera node's IP address directly, but rather employ some HA solution which offers active-passive functionality. Options therefore are discussed below.

==== Read requests ====

Read requests can be directed to any node in the Galera cluster. Our standard approach is to recommend to use a loadbalancer to implement round-robin over all nodes in a Galera cluster for the read requests. But you can also chose to use a dedicated read node (the same node, or a different node, than the write node). Each of the approaches has its own advantages.

* Load balancer based setup: Read requests get distributed round-robin between the Galera nodes. Theoretically by distributing the load of the read requests, you benefit from lower latencies and more throughput. But this has never been benchmarked yet. For a discussion of available loadbalances, see next section. OX-wise, in this configuration, you have two alternatives:
** The Galera option wsrep_causal_reads=1 option enables you to configure OX with its replication monitor disabled (com.openexchange.database.replicationMonitor=false in configdb.properties). This is the setup which seems to perform best according to our experience as turning off the replication monitor reduces the commits on the DB and thus the write operations per second on the underlying storage significantly, which outweights the drawback from having higher commit latency due to fully synchronous mode.
** Alternatively, you can run Galera with wsrep_causal_reads=0 when switching on OX builtin replication monitor. This is also a valid setup.
* Use a designated floating IP for the read requests: This eliminates the need of a load balancer. With this option you will not gain any performance, but the quantitative benefit is unclear anyhow.
* Use the floating IP for the writes also for the reads: In this scenario, you direct all database queries only to one Galera node, and the other two nodes are only getting queries in case of a failure of that node. In this case, you can even use wsrep_causal_reads=0 while still having OX builtin replication monitor switched off. However we do not expect this option to be superior to the round-robin loadbalancer approach.

=== Loadbalancer options ===

While the JDBC driver has some round-robin load balancing capabilities built-in, we don't recommend it for production use since it lacks possibilities to check the Galera nodes health states.

Loadbalancers used for OX -> Galera loadbalancing should be able to implement active-passive instances for the write requests, and active-active (round-robin) instances for the read requests. (If they cannot implement active-passive, you can still take a floating IP therefore.) Furthermore it is required to configure node health checks not only on the TCP level (by a simple connect), but to query the Galera health status periodically, evaluating Galera WSREP status variables. Otherwise split-brain scenarios or other bad states cannot be detected. For an example of such an health check, see our [[Clustercheck]] page.

Some customers use loadbalancing appliances. It is important to check that if the (virtual) infrastructure offers "loadbalancer" instances that they satisfy the given requirements. Often this is not the case. In particular, a simple "DNS round robin" approach is not viable.

==== LVS/ipvsadm/keepalived ====

If you want to create your own loadbalancers based on Linux, we usually recommend LVS (Linux Virtual Servers) controlled by Keepalived. LVS is a set of kernel modules implementing a L4 loadbalancer which performs quite well. Keepalived is a userspace daemon to control LVS rules, using health checks to reconfigure LVS rules if required. Keepalived / LVS requires one (or, for availability, two) dedicated linux nodes to run on. This can be a disadvantage for some installations, but usually, it pays off. We provide some configuration information on Keepalived [[Keepalived|here]].

==== MariaDB Maxscale ====

Since Maxscale has become GA in 2015, it seems to have undergone significant stability, performance and functional improvements. We are currently experimenting with Maxscale and share our installation / configuration knowledge [[Maxscale|here]]. It looks quite promising and might become ''the standard replacement'' for HAproxy, while we still presume Keepalived offers superior robustness and performance, coming with the cost of the requirement for one (or more) dedicated loadbalancer nodes.

==== HAproxy ====

In case where the Keepalived based approach is not feasible due to its requirements on the infrastructure, it is also possible to use a HAproxy based solution where HAproxy processes run on each of the OX nodes, configured for one round-robin and one active/passive instance. OX is then connecting to the local HAproxy instances. It is vital to configure HAproxy timeouts different from the defaults, otherwise HAproxy will kill active DB connections, causing errors. Be aware that in large installations the number of (distributed) HAproxy instances can get quite large. Some configuration hints for HAproxy are available [[HAproxy|here]].

== Master/Slave database setup ==

While we also support also "legacy" (pre-GTID) Master/Slave replication, we recommend to use GTID based replication, for easier setup and failure recovery. Support for GTID based replication has been added with OX 7.8.0.

GTID has been available since MySQL 5.6, so no 5.5 installation instructions below, sorry. We try to be generic in this documentation (thus, applicable to Oracle Community Edition and MariaDB) and point out differences where needed. Note: Instructions below include information about Oracle Community MySQL 5.7 which is not yet formally supported.

=== Preparations ===

Our configuration process includes wiping and reinitializing the datadir. This is usually not a problem in a fresh installation. If you want to upgrade an existing database to GTID master-slave, please be prepared to wipe the datadir, i.e. take a <code>mysqldump</code> for later restoration into the properly configured master.

Depeding on the flavor of the current database, this can be something like

# mariadb or oracle mysql without GTIDs
mysqldump --databases configdb oxdb_{5..14} > backup.sql

# mysql 5.6 with GTIDs... we dont want GTIDs here
mysqldump --databases --set-gtid-purged=OFF configdb oxdb_{5..14} > backup.sql

Be sure to verify the list of databases.

=== Installation ===

Software installation is identical for master and slave.

Please follow the instructions for installing from The vendors.

* Oracle Community Edition: https://dev.mysql.com/doc/mysql-apt-repo-quick-guide/en/
* MariaDB (10.0, 10.1): https://downloads.mariadb.org/mariadb/repositories/

Stop the service (if it is running):

service mysql stop

=== Configuration ===

Configuration as per configuration files is also identical for master and slave.

Consult [[My.cnf]] for general recommendations how to configure databases for usage with OX.

For GTID based replication, make sure you add some configurables to a new <code>/etc/mysql/ox.conf.d/gtid.cnf</code> file (assuming you are following our proposed schema of adding a <code>!includedir /etc/mysql/ox.conf.d/</code>" directive to <code>/etc/mysql/my.cnf</code>):

# GTID
log-bin=mysql-bin
server-id=...
log_slave_updates = ON

Oracle Community Edition: we need to add also

enforce_gtid_consistency = ON
gtid_mode = ON

(GTID mode is on by default on MariaDB.)

Use unique a <code>server-id</code> for each server; like <code>1</code> for the master, <code>2</code> for slave. For more complicated setups (like multiple slaves), adjust accordingly.

Since applying our configuration / sizing requires reinitialization of the MySQL datadir, we wipe/recreate it. Caution: this assumes we are running an empty database. If there is data in the database you want to keep, use mysqldump. See Preparation section above.

So, to initialize the datadir:

cd /var/lib/
mv mysql mysql.old.datadir
mkdir mysql
chown mysql.mysql mysql

(When coming from an existing installation, be sure to wipe also old binlogs. They can confuse the server on startup. Their location varies by configuration.)

The step to initialize the datadir is different for the different DBs:

# MariaDB 10.0, 10.1
mysql_install_db

# MariaDB 10.2
mysql_install_db --user=mysql

# Oracle 5.6
mysql_install_db -u mysql

# Oracle 5.7
mysqld --initialize-insecure --user=mysql

(Don't be worried about the <code>insecure</code>, it just means we set the db root pw in the next steps.)

Then:

service mysql restart
mysql_secure_installation

We want to emphasize the last step to run "secure".

Steps up to here apply to both the designated master and slave. The next steps will apply to the master.

=== Replication Setup ===

==== Master Setup ====

Create a replication user on the master (but, as always, pick your own password, and use the same password in the slave setup below):

mysql -e "CREATE USER 'repl'@'gtid-slave.localdomain' IDENTIFIED BY 'IvIjyoffod2'; GRANT REPLICATION SLAVE ON *.* TO 'repl'@'gtid-slave.localdomain';"

Now would also be the time to restore a previously created mysqldump, or add other users you need for adminstration, monitoring etc (like <code>debian-sys-maint@localhost</code>, for example). Adding the OX users is explained below ("Creating Open-Xchange user").

# If you took a dump for restore before
mysql < backup.sql

To prepare for the initial sync of the slave, set the master read-only:

mysql -e "SET @@global.read_only = ON;"

Create a dump to initialize the slave:

# MariaDB
mysqldump --all-databases --triggers --routines --events --master-data --gtid > master.sql

# Oracle
mysqldump --all-databases --triggers --routines --events --set-gtid-purged=ON > master.sql

Transfer to the slave:

scp master.sql gtid-slave:

==== Slave Setup ====

Configure the replication master settings. Note we don't need complicated binlog position settings etc with GTID.

Yet again DB-specific (use the repl user password from above):

# MariaDB
mysql -e 'CHANGE MASTER TO MASTER_HOST="gtid-master.localdomain", MASTER_USER="repl", MASTER_PASSWORD="IvIjyoffod2";'

# Oracle
mysql -e "CHANGE MASTER TO MASTER_HOST='gtid-master.localdomain', MASTER_USER='repl', MASTER_PASSWORD='IvIjyoffod2', MASTER_AUTO_POSITION=1;"
# https://www.percona.com/blog/2013/02/08/how-to-createrestore-a-slave-using-gtid-replication-in-mysql-5-6/
mysql -e "RESET MASTER;"

Read the master dump:

mysql < master.sql

Start replication on the slave:

mysql -e 'START SLAVE;'
mysql -e 'SHOW SLAVE STATUS\G'

==== Master Setup (continued) ====

Finally, unset read-only on the master:

# on the master
mysql -e "SET @@global.read_only = OFF;"

=== Configure OX to use with Master/Slave replication ===

Not much special wisdom here. OX was designed to be used with master/slave databases. For the ConfigDB, <code>configdb.properties</code> allows configuration of a <code>readUrl</code> and <code>writeUrl</code> (both of which are set to the correct values if you use <code>oxinstaller</code> with the correct arguments <code>--configdb-readhost</code>, <code>--configdb-writehost</code>).

(Obviously, the master is for writing and the slave is for reading.)

For the individiual UserDBs, use <code>registerdatabase -m true</code> for the masters and <code>registerdatabase -m false -M ...</code> for the respective slaves.

Be sure to have enabled the replication monitor in <code>configdb.properties</code>: <code>com.openexchange.database.replicationMonitor=true</code> (which it is by default); while GTID can show synchronous semantics, it is specified to silently fall back to asynchronous in certain circumstances, so synchronity is not guaranteed.

We recommend, though, to not register the databases directly by their native hostname or IP, but rather use some kind of HA system in order to be able to easily move a floating/failover IP from the master to the slave in case of master failure. Configuring and running such systems (like, corosync/pacemaker, keepalived, or whatever) is out of scope of this documentation, however.

== Creating Open-Xchange user ==

Now setup access for the Open-Xchange Server database user 'openexchange' to configdb and the oxdb for both groupware server addresses. These databases do not exist yet, but will be created during the Open-Xchange Server installation.

Notes:

* Please use a real password.
* The IPs in this example belong to the two different Open-Xchange Servers, please adjust them accordingly.
* If using a database on the same host as the middlware (usually done for POCs and demo installations), you need to grant also to the ''localhost'' host.
* Consult [[AppSuite:DB_user_privileges]] (or ''grep GRANT /opt/open-xchange/sbin/initconfigdb'') for an up-to-date list of required privileges. The following statement was correct as of the time of writing this section.

mysql> GRANT CREATE, LOCK TABLES, REFERENCES, INDEX, DROP, DELETE, ALTER, SELECT, UPDATE, INSERT, CREATE TEMPORARY TABLES, SHOW VIEW, SHOW DATABASES ON *.* TO 'openexchange'@'10.20.30.213' IDENTIFIED BY 'IntyoyntOat1' WITH GRANT OPTION;
mysql> GRANT CREATE, LOCK TABLES, REFERENCES, INDEX, DROP, DELETE, ALTER, SELECT, UPDATE, INSERT, CREATE TEMPORARY TABLES, SHOW VIEW, SHOW DATABASES ON *.* TO 'openexchange'@'10.20.30.215' IDENTIFIED BY 'IntyoyntOat1' WITH GRANT OPTION;

Template:OXLoadBalancingClustering Database

2018-09-06T13:50:17Z

Dominik.epple: /* Configuration */

== Overview ==

You can choose between Galera or Master/Slave replication. We like to recommend to use Galera for higher redudancy, easier operations, und synchronous semantics (so you can run OX without our "replication monitor"). For POC or demo setups, a single standalone database setup might be sufficient.

== Standalone database setup ==

=== Preparations ===

Our configuration process includes wiping and reinitializing the datadir. This is usually not a problem in a fresh installation. If you want to upgrade an existing database server, please be prepared to wipe the datadir, i.e. take a <code>mysqldump</code> for later restoration into the properly configured master.

mysqldump --databases configdb oxdb_{5..14} > backup.sql

Be sure to verify the list of databases.

=== Installation ===

Note: the following list ist not an exclusive list or authorative statement about supported MySQL flavors / versions. Please consult the official support / system requirements statement.

Please follow the upstream docs for your preferred flavor to get the software installed on your system.

* MariaDB (10.1, 10.2): https://downloads.mariadb.org/
* Oracle MySQL Community Server (5.6, 5.7): https://dev.mysql.com/downloads/mysql/

Make sure to doublecheck the service is not running (or stop it) after installation as we need to perform some reconfigurations.

service mysql stop

=== Configuration ===

MySQL configuration advise is given in our [[My.cnf|MySQL configuration article]]. Please consult that page for configuration information and create configuration files as described there.

Some settings we recommend to change require that the database gets re-initialized. We assume you don't have data there (since we are covering a fresh install) or you have taken a backup for later restore as explained above in the Preparations section.

cd /var/lib/
mv mysql mysql.old.datadir
mkdir mysql
chown mysql.mysql mysql

# mariadb
mysql_install_db
# mariadb 10.2
mysql_install_db --user=mysql
# oracle 5.6
mysql_install_db -u mysql
# oracle 5.7
mysqld --initialize-insecure --user=mysql

(Don't be worried about the <code>insecure</code>, it just means we set the db root pw in the next steps.)

Start the service. The actual command depends on your OS and on the MySQL flavor.

service mysql start

Run <code>mysql_secure_installation</code> for a "secure by default" installation:

mysql_secure_installation

That tool will ask for the current root password (which is empty by default) and subsequently questions like:

Change the root password? [Y/n]
Remove anonymous users? [Y/n]
Disallow root login remotely? [Y/n]
Remove test database and access to it? [Y/n]
Reload privilege tables now? [Y/n]

You should answer all these questions with "yes".

Configure a strong password for the MySQL <code>root</code> user.

The further steps in this guide omit <code>-u -p</code> arguments to the MySQL client. Rather than passing them on the command line [https://dev.mysql.com/doc/refman/5.7/en/password-security-user.html] it is recommended to place the credentials in a file like <code>/root/.my.cnf</code> like

[client]
user=root
password=wip9Phae3Beijeed

Make sure the service is enabled by the OS's init system. The actual command depends on your OS and on the MySQL flavor.

systemctl enable mysql.service

You should now be able to restore your previously taken backup.

# If you took a dump for restore before
mysql < backup.sql

=== Configure OX to use with a standalone database ===

Not much special wisdom here. OX was designed to be used with master/slave databases, and a standalone master works just as well, if we register it as a master, and not registering a slave.

For the ConfigDB, <code>configdb.properties</code> allows configuration of a <code>writeUrl</code> (which is set to the correct values if you use <code>oxinstaller</code> with the correct argument <code>--configdb-writehost</code>).

The single database is then used for reading and writing.

For the individiual UserDBs, use <code>registerdatabase -m true</code>.

== Galera database setup ==

=== Preparations ===

Our configuration process includes wiping and reinitializing the datadir. This is usually not a problem in a fresh installation. If you want to upgrade an existing database to Galera cluster, please be prepared to wipe the datadir, i.e. take a <code>mysqldump</code> for later restoration into the properly configured master.

Depeding on the flavor of the current database, this can be something like

# mariadb or oracle mysql without GTIDs
mysqldump --databases configdb oxdb_{5..14} > backup.sql

# mysql 5.6 with GTIDs... we dont want GTIDs here
mysqldump --databases --set-gtid-purged=OFF configdb oxdb_{5..14} > backup.sql

Be sure to verify the list of databases.

=== Installation ===

Please follow the upstream docs for your preferred flavor to get the software installed on your system.

* Percona XtraDB Cluster (5.6, 5.7): https://www.percona.com/doc/percona-xtradb-cluster/LATEST/install/index.html
* MariaDB Galera Cluster (10.0, 10.1): https://mariadb.com/kb/en/library/getting-started-with-mariadb-galera-cluster/ (Note: with 10.0, socat is required, but not a package dependency, so you need to explicitly install also socat)

Make sure to doublecheck the service is not running (or stop it) after installation as we need to perform some reconfigurations.

service mysql stop

=== Configuration ===

Galera-specific MySQL configuration advise is included in our main [[My.cnf|MySQL configuration article]]. Please consult that page for configuration information.

That page suggests a setup were we add three custom config files to <code>/etc/mysql/ox.conf.d/</pre>: <code>ox.cnf</code> for general tuning/sizing, <code>wsrep.cnf</code> for clusterwide galera configuration, and <code>host.cnf</code> for host-specific settings.

Adjust the general settings and tunings in <code>ox.cnf</code> according to your sizing etc.

Adjust <code>wsrep.cnf</code> to reflect local paths, cluster member addresses, etc.

Adjust <code>host.cnf</code> to give node-local IPs, etc.

Version-specific hints:

# percona 5.6: unknown variable 'pxc_strict_mode=ENFORCING' ... unset that one
# mariadb 10.1: add wsrep_on=ON
# mariadb 10.0 and 10.1: set wsrep_node_incoming_address=192.168.1.22:3306 in host.cnf, otherwise the status wsrep_incoming_addresses might not be shown correctly(?!)

Some settings we recommend to change require that the database gets re-initialized. We assume you don't have data there (since we are covering a fresh install) or you have taken a backup for later restore as explained above in the Preparations section.

cd /var/lib/
mv mysql mysql.old.datadir
mkdir mysql
chown mysql.mysql mysql

# mariadb 10.0 and 10.1
mysql_install_db
# mariadb 10.2
mysql_install_db --user=mysql
# percona 5.6
mysqld --user=mysql
# percona 5.7
mysqld --initialize-insecure --user=mysql

(Don't be worried about the <code>insecure</code>, it just means we set the db root pw in the next steps.)

=== Cluster startup ===

Typically on startup a Galera node tries to join a cluster, and if it fails, it will exit. Thus, when no cluster nodes are running, the first cluster node to be started needs to be told to not try to join a cluster, but rather bootstrap a new cluster. The exact arguments vary from version to version and from flavor to flavor.

==== First node ====

So we initialize the cluster bootstrap on the first node:

# percona 5.6, 5.7
service mysql bootstrap-pxc
# mariadb 10.0
service mysql bootstrap
# mariadb 10.1: service mysql bootstrap seems to be broken, does not pass the necessary options to mysqld
galera_new_cluster

Run <code>mysql_secure_installation</code> for a "secure by default" installation:

mysql_secure_installation

The further steps in this guide omit <code>-u -p</code> arguments to the MySQL client. Rather than passing them on the command line [https://dev.mysql.com/doc/refman/5.7/en/password-security-user.html] it is recommended to place the credentials in a file like <code>/root/.my.cnf</code> like

[client]
user=root
password=wip9Phae3Beijeed

We need a Galera replication user:

CREATE USER 'sstuser'@'localhost' IDENTIFIED BY 'OpIdjijwef0';
-- percona 5.6, mariadb 10.0
GRANT RELOAD, LOCK TABLES, REPLICATION CLIENT ON *.* TO 'sstuser'@'localhost';
-- percona 5.7, mariadb 10.1
GRANT PROCESS, RELOAD, LOCK TABLES, REPLICATION CLIENT ON *.* TO 'sstuser'@'localhost';
FLUSH PRIVILEGES;

(Debian specific note: MariaDB provided startup scripts use the distro's mechanism of verifying startup/shutdown using a system user, so we create that as well:

# mariadb 10.0, 10.1
GRANT ALL PRIVILEGES ON *.* TO "debian-sys-maint"@"localhost" IDENTIFIED BY "adBexthTsI5TaEps";

If you do this, yo need to synchronize the <code>/etc/mysql/debian.cnf</code> file from the first node to the other nodes as well.)

==== Other nodes ====

On the other nodes, we only need to restart the service now, to trigger a full state transfer from the first node to the other nodes.

We recommend to do this serially to let one state transfer complete before the second state transfer.

==== First node (continued) ====

Only applicable if you used <code>galera_new_cluster</code> before rather than the service script: In order to get the systemctl status consistent, restart the service on the first node:

# mariadb 10.1: restart the service so that the systemctl status is consistent
mysqladmin shutdown
service mysql bootstrap

=== Verify the replication ===

The key tool to verify replication status is

mysql> show status like "%wsrep%";

This will give a lot of output. You want to verify in particular

+------------------------------+--------------------------------------+
| Variable_name | Value |
+------------------------------+--------------------------------------+
| wsrep_cluster_size | 3 |
| wsrep_cluster_status | Primary |
| wsrep_local_state | 4 |
| wsrep_local_state_comment | Synced |
| wsrep_ready | ON |
+------------------------------+--------------------------------------+

You can also explicitly verify replication by creating / inserting DBs, tables, rows on one node and select on other nodes.

==== Troubleshooting ====

The logs are helpful. Always.

Common mistakes are listed below.

If the Galera module does not get loaded at all:
* Configuration settings in ''my.cnf'' which are incompatible to Galera
* Wrong path of the shared object providing the Galera plugin in wsrep.cnf (wsrep_provider)

If the first node starts, but the second / third nodes can not be added to the cluster:
* User for the replication not created correctly on the first Galera node
* SST fails due to missing / wrong version prerequisite packages (not everything is hardcoded in package dependencies -- make sure you got percona-xtrabackup installed in the correct version, and also socat). If SST fails, do not only look into mysqls primary error logs, but also into logfiles from the SST tool in /var/lib/mysql on the donor node.

=== Notes about configuring OX for use with Galera ===

==== Write requests ====

Open-Xchange supports Galera as database backend only in the configuration where all writes are directed to one Galera node. For availability, it makes sense to not configure one Galera node's IP address directly, but rather employ some HA solution which offers active-passive functionality. Options therefore are discussed below.

==== Read requests ====

Read requests can be directed to any node in the Galera cluster. Our standard approach is to recommend to use a loadbalancer to implement round-robin over all nodes in a Galera cluster for the read requests. But you can also chose to use a dedicated read node (the same node, or a different node, than the write node). Each of the approaches has its own advantages.

* Load balancer based setup: Read requests get distributed round-robin between the Galera nodes. Theoretically by distributing the load of the read requests, you benefit from lower latencies and more throughput. But this has never been benchmarked yet. For a discussion of available loadbalances, see next section. OX-wise, in this configuration, you have two alternatives:
** The Galera option wsrep_causal_reads=1 option enables you to configure OX with its replication monitor disabled (com.openexchange.database.replicationMonitor=false in configdb.properties). This is the setup which seems to perform best according to our experience as turning off the replication monitor reduces the commits on the DB and thus the write operations per second on the underlying storage significantly, which outweights the drawback from having higher commit latency due to fully synchronous mode.
** Alternatively, you can run Galera with wsrep_causal_reads=0 when switching on OX builtin replication monitor. This is also a valid setup.
* Use a designated floating IP for the read requests: This eliminates the need of a load balancer. With this option you will not gain any performance, but the quantitative benefit is unclear anyhow.
* Use the floating IP for the writes also for the reads: In this scenario, you direct all database queries only to one Galera node, and the other two nodes are only getting queries in case of a failure of that node. In this case, you can even use wsrep_causal_reads=0 while still having OX builtin replication monitor switched off. However we do not expect this option to be superior to the round-robin loadbalancer approach.

=== Loadbalancer options ===

While the JDBC driver has some round-robin load balancing capabilities built-in, we don't recommend it for production use since it lacks possibilities to check the Galera nodes health states.

Loadbalancers used for OX -> Galera loadbalancing should be able to implement active-passive instances for the write requests, and active-active (round-robin) instances for the read requests. (If they cannot implement active-passive, you can still take a floating IP therefore.) Furthermore it is required to configure node health checks not only on the TCP level (by a simple connect), but to query the Galera health status periodically, evaluating Galera WSREP status variables. Otherwise split-brain scenarios or other bad states cannot be detected. For an example of such an health check, see our [[Clustercheck]] page.

Some customers use loadbalancing appliances. It is important to check that if the (virtual) infrastructure offers "loadbalancer" instances that they satisfy the given requirements. Often this is not the case. In particular, a simple "DNS round robin" approach is not viable.

==== LVS/ipvsadm/keepalived ====

If you want to create your own loadbalancers based on Linux, we usually recommend LVS (Linux Virtual Servers) controlled by Keepalived. LVS is a set of kernel modules implementing a L4 loadbalancer which performs quite well. Keepalived is a userspace daemon to control LVS rules, using health checks to reconfigure LVS rules if required. Keepalived / LVS requires one (or, for availability, two) dedicated linux nodes to run on. This can be a disadvantage for some installations, but usually, it pays off. We provide some configuration information on Keepalived [[Keepalived|here]].

==== MariaDB Maxscale ====

Since Maxscale has become GA in 2015, it seems to have undergone significant stability, performance and functional improvements. We are currently experimenting with Maxscale and share our installation / configuration knowledge [[Maxscale|here]]. It looks quite promising and might become ''the standard replacement'' for HAproxy, while we still presume Keepalived offers superior robustness and performance, coming with the cost of the requirement for one (or more) dedicated loadbalancer nodes.

==== HAproxy ====

In case where the Keepalived based approach is not feasible due to its requirements on the infrastructure, it is also possible to use a HAproxy based solution where HAproxy processes run on each of the OX nodes, configured for one round-robin and one active/passive instance. OX is then connecting to the local HAproxy instances. It is vital to configure HAproxy timeouts different from the defaults, otherwise HAproxy will kill active DB connections, causing errors. Be aware that in large installations the number of (distributed) HAproxy instances can get quite large. Some configuration hints for HAproxy are available [[HAproxy|here]].

== Master/Slave database setup ==

While we also support also "legacy" (pre-GTID) Master/Slave replication, we recommend to use GTID based replication, for easier setup and failure recovery. Support for GTID based replication has been added with OX 7.8.0.

GTID has been available since MySQL 5.6, so no 5.5 installation instructions below, sorry. We try to be generic in this documentation (thus, applicable to Oracle Community Edition and MariaDB) and point out differences where needed. Note: Instructions below include information about Oracle Community MySQL 5.7 which is not yet formally supported.

=== Preparations ===

Our configuration process includes wiping and reinitializing the datadir. This is usually not a problem in a fresh installation. If you want to upgrade an existing database to GTID master-slave, please be prepared to wipe the datadir, i.e. take a <code>mysqldump</code> for later restoration into the properly configured master.

Depeding on the flavor of the current database, this can be something like

# mariadb or oracle mysql without GTIDs
mysqldump --databases configdb oxdb_{5..14} > backup.sql

# mysql 5.6 with GTIDs... we dont want GTIDs here
mysqldump --databases --set-gtid-purged=OFF configdb oxdb_{5..14} > backup.sql

Be sure to verify the list of databases.

=== Installation ===

Software installation is identical for master and slave.

Please follow the instructions for installing from The vendors.

* Oracle Community Edition: https://dev.mysql.com/doc/mysql-apt-repo-quick-guide/en/
* MariaDB (10.0, 10.1): https://downloads.mariadb.org/mariadb/repositories/

Stop the service (if it is running):

service mysql stop

=== Configuration ===

Configuration as per configuration files is also identical for master and slave.

Consult [[My.cnf]] for general recommendations how to configure databases for usage with OX.

For GTID based replication, make sure you add some configurables to a new <code>/etc/mysql/ox.conf.d/gtid.cnf</code> file (assuming you are following our proposed schema of adding a <code>!includedir /etc/mysql/ox.conf.d/</code>" directive to <code>/etc/mysql/my.cnf</code>):

# GTID
log-bin=mysql-bin
server-id=...
log_slave_updates = ON

Oracle Community Edition: we need to add also

enforce_gtid_consistency = ON
gtid_mode = ON

(GTID mode is on by default on MariaDB.)

Use unique a <code>server-id</code> for each server; like <code>1</code> for the master, <code>2</code> for slave. For more complicated setups (like multiple slaves), adjust accordingly.

Since applying our configuration / sizing requires reinitialization of the MySQL datadir, we wipe/recreate it. Caution: this assumes we are running an empty database. If there is data in the database you want to keep, use mysqldump. See Preparation section above.

So, to initialize the datadir:

cd /var/lib/
mv mysql mysql.old.datadir
mkdir mysql
chown mysql.mysql mysql

(When coming from an existing installation, be sure to wipe also old binlogs. They can confuse the server on startup. Their location varies by configuration.)

The step to initialize the datadir is different for the different DBs:

# MariaDB 10.0, 10.1
mysql_install_db

# Oracle 5.6
mysql_install_db -u mysql

# Oracle 5.7
mysqld --initialize-insecure --user=mysql

(Don't be worried about the <code>insecure</code>, it just means we set the db root pw in the next steps.)

Then:

service mysql restart
mysql_secure_installation

We want to emphasize the last step to run "secure".

Steps up to here apply to both the designated master and slave. The next steps will apply to the master.

=== Replication Setup ===

==== Master Setup ====

Create a replication user on the master (but, as always, pick your own password, and use the same password in the slave setup below):

mysql -e "CREATE USER 'repl'@'gtid-slave.localdomain' IDENTIFIED BY 'IvIjyoffod2'; GRANT REPLICATION SLAVE ON *.* TO 'repl'@'gtid-slave.localdomain';"

Now would also be the time to restore a previously created mysqldump, or add other users you need for adminstration, monitoring etc (like <code>debian-sys-maint@localhost</code>, for example). Adding the OX users is explained below ("Creating Open-Xchange user").

# If you took a dump for restore before
mysql < backup.sql

To prepare for the initial sync of the slave, set the master read-only:

mysql -e "SET @@global.read_only = ON;"

Create a dump to initialize the slave:

# MariaDB
mysqldump --all-databases --triggers --routines --events --master-data --gtid > master.sql

# Oracle
mysqldump --all-databases --triggers --routines --events --set-gtid-purged=ON > master.sql

Transfer to the slave:

scp master.sql gtid-slave:

==== Slave Setup ====

Configure the replication master settings. Note we don't need complicated binlog position settings etc with GTID.

Yet again DB-specific (use the repl user password from above):

# MariaDB
mysql -e 'CHANGE MASTER TO MASTER_HOST="gtid-master.localdomain", MASTER_USER="repl", MASTER_PASSWORD="IvIjyoffod2";'

# Oracle
mysql -e "CHANGE MASTER TO MASTER_HOST='gtid-master.localdomain', MASTER_USER='repl', MASTER_PASSWORD='IvIjyoffod2', MASTER_AUTO_POSITION=1;"
# https://www.percona.com/blog/2013/02/08/how-to-createrestore-a-slave-using-gtid-replication-in-mysql-5-6/
mysql -e "RESET MASTER;"

Read the master dump:

mysql < master.sql

Start replication on the slave:

mysql -e 'START SLAVE;'
mysql -e 'SHOW SLAVE STATUS\G'

==== Master Setup (continued) ====

Finally, unset read-only on the master:

# on the master
mysql -e "SET @@global.read_only = OFF;"

=== Configure OX to use with Master/Slave replication ===

Not much special wisdom here. OX was designed to be used with master/slave databases. For the ConfigDB, <code>configdb.properties</code> allows configuration of a <code>readUrl</code> and <code>writeUrl</code> (both of which are set to the correct values if you use <code>oxinstaller</code> with the correct arguments <code>--configdb-readhost</code>, <code>--configdb-writehost</code>).

(Obviously, the master is for writing and the slave is for reading.)

For the individiual UserDBs, use <code>registerdatabase -m true</code> for the masters and <code>registerdatabase -m false -M ...</code> for the respective slaves.

Be sure to have enabled the replication monitor in <code>configdb.properties</code>: <code>com.openexchange.database.replicationMonitor=true</code> (which it is by default); while GTID can show synchronous semantics, it is specified to silently fall back to asynchronous in certain circumstances, so synchronity is not guaranteed.

We recommend, though, to not register the databases directly by their native hostname or IP, but rather use some kind of HA system in order to be able to easily move a floating/failover IP from the master to the slave in case of master failure. Configuring and running such systems (like, corosync/pacemaker, keepalived, or whatever) is out of scope of this documentation, however.

== Creating Open-Xchange user ==

Now setup access for the Open-Xchange Server database user 'openexchange' to configdb and the oxdb for both groupware server addresses. These databases do not exist yet, but will be created during the Open-Xchange Server installation.

Notes:

* Please use a real password.
* The IPs in this example belong to the two different Open-Xchange Servers, please adjust them accordingly.
* If using a database on the same host as the middlware (usually done for POCs and demo installations), you need to grant also to the ''localhost'' host.
* Consult [[AppSuite:DB_user_privileges]] (or ''grep GRANT /opt/open-xchange/sbin/initconfigdb'') for an up-to-date list of required privileges. The following statement was correct as of the time of writing this section.

mysql> GRANT CREATE, LOCK TABLES, REFERENCES, INDEX, DROP, DELETE, ALTER, SELECT, UPDATE, INSERT, CREATE TEMPORARY TABLES, SHOW VIEW, SHOW DATABASES ON *.* TO 'openexchange'@'10.20.30.213' IDENTIFIED BY 'IntyoyntOat1' WITH GRANT OPTION;
mysql> GRANT CREATE, LOCK TABLES, REFERENCES, INDEX, DROP, DELETE, ALTER, SELECT, UPDATE, INSERT, CREATE TEMPORARY TABLES, SHOW VIEW, SHOW DATABASES ON *.* TO 'openexchange'@'10.20.30.215' IDENTIFIED BY 'IntyoyntOat1' WITH GRANT OPTION;

Template:OXLoadBalancingClustering Database

2018-09-06T13:49:46Z

Dominik.epple: /* Configuration */

== Overview ==

You can choose between Galera or Master/Slave replication. We like to recommend to use Galera for higher redudancy, easier operations, und synchronous semantics (so you can run OX without our "replication monitor"). For POC or demo setups, a single standalone database setup might be sufficient.

== Standalone database setup ==

=== Preparations ===

Our configuration process includes wiping and reinitializing the datadir. This is usually not a problem in a fresh installation. If you want to upgrade an existing database server, please be prepared to wipe the datadir, i.e. take a <code>mysqldump</code> for later restoration into the properly configured master.

mysqldump --databases configdb oxdb_{5..14} > backup.sql

Be sure to verify the list of databases.

=== Installation ===

Note: the following list ist not an exclusive list or authorative statement about supported MySQL flavors / versions. Please consult the official support / system requirements statement.

Please follow the upstream docs for your preferred flavor to get the software installed on your system.

* MariaDB (10.1, 10.2): https://downloads.mariadb.org/
* Oracle MySQL Community Server (5.6, 5.7): https://dev.mysql.com/downloads/mysql/

Make sure to doublecheck the service is not running (or stop it) after installation as we need to perform some reconfigurations.

service mysql stop

=== Configuration ===

MySQL configuration advise is given in our [[My.cnf|MySQL configuration article]]. Please consult that page for configuration information and create configuration files as described there.

Some settings we recommend to change require that the database gets re-initialized. We assume you don't have data there (since we are covering a fresh install) or you have taken a backup for later restore as explained above in the Preparations section.

cd /var/lib/
mv mysql mysql.old.datadir
mkdir mysql
chown mysql.mysql mysql

# mariadb
mysql_install_db
# mariadb 10.2
mysql_install_db --user=mysql
# oracle 5.6
mysql_install_db -u mysql
# oracle 5.7
mysqld --initialize-insecure --user=mysql

(Don't be worried about the <code>insecure</code>, it just means we set the db root pw in the next steps.)

Start the service. The actual command depends on your OS and on the MySQL flavor.

service mysql start

Run <code>mysql_secure_installation</code> for a "secure by default" installation:

mysql_secure_installation

That tool will ask for the current root password (which is empty by default) and subsequently questions like:

Change the root password? [Y/n]
Remove anonymous users? [Y/n]
Disallow root login remotely? [Y/n]
Remove test database and access to it? [Y/n]
Reload privilege tables now? [Y/n]

You should answer all these questions with "yes".

Configure a strong password for the MySQL <code>root</code> user.

The further steps in this guide omit <code>-u -p</code> arguments to the MySQL client. Rather than passing them on the command line [https://dev.mysql.com/doc/refman/5.7/en/password-security-user.html] it is recommended to place the credentials in a file like <code>/root/.my.cnf</code> like

[client]
user=root
password=wip9Phae3Beijeed

Make sure the service is enabled by the OS's init system. The actual command depends on your OS and on the MySQL flavor.

systemctl enable mysql.service

You should now be able to restore your previously taken backup.

# If you took a dump for restore before
mysql < backup.sql

=== Configure OX to use with a standalone database ===

Not much special wisdom here. OX was designed to be used with master/slave databases, and a standalone master works just as well, if we register it as a master, and not registering a slave.

For the ConfigDB, <code>configdb.properties</code> allows configuration of a <code>writeUrl</code> (which is set to the correct values if you use <code>oxinstaller</code> with the correct argument <code>--configdb-writehost</code>).

The single database is then used for reading and writing.

For the individiual UserDBs, use <code>registerdatabase -m true</code>.

== Galera database setup ==

=== Preparations ===

Our configuration process includes wiping and reinitializing the datadir. This is usually not a problem in a fresh installation. If you want to upgrade an existing database to Galera cluster, please be prepared to wipe the datadir, i.e. take a <code>mysqldump</code> for later restoration into the properly configured master.

Depeding on the flavor of the current database, this can be something like

# mariadb or oracle mysql without GTIDs
mysqldump --databases configdb oxdb_{5..14} > backup.sql

# mysql 5.6 with GTIDs... we dont want GTIDs here
mysqldump --databases --set-gtid-purged=OFF configdb oxdb_{5..14} > backup.sql

Be sure to verify the list of databases.

=== Installation ===

Please follow the upstream docs for your preferred flavor to get the software installed on your system.

* Percona XtraDB Cluster (5.6, 5.7): https://www.percona.com/doc/percona-xtradb-cluster/LATEST/install/index.html
* MariaDB Galera Cluster (10.0, 10.1): https://mariadb.com/kb/en/library/getting-started-with-mariadb-galera-cluster/ (Note: with 10.0, socat is required, but not a package dependency, so you need to explicitly install also socat)

Make sure to doublecheck the service is not running (or stop it) after installation as we need to perform some reconfigurations.

service mysql stop

=== Configuration ===

Galera-specific MySQL configuration advise is included in our main [[My.cnf|MySQL configuration article]]. Please consult that page for configuration information.

That page suggests a setup were we add three custom config files to <code>/etc/mysql/ox.conf.d/</pre>: <code>ox.cnf</code> for general tuning/sizing, <code>wsrep.cnf</code> for clusterwide galera configuration, and <code>host.cnf</code> for host-specific settings.

Adjust the general settings and tunings in <code>ox.cnf</code> according to your sizing etc.

Adjust <code>wsrep.cnf</code> to reflect local paths, cluster member addresses, etc.

Adjust <code>host.cnf</code> to give node-local IPs, etc.

Version-specific hints:

# percona 5.6: unknown variable 'pxc_strict_mode=ENFORCING' ... unset that one
# mariadb 10.1: add wsrep_on=ON
# mariadb 10.0 and 10.1: set wsrep_node_incoming_address=192.168.1.22:3306 in host.cnf, otherwise the status wsrep_incoming_addresses might not be shown correctly(?!)

Some settings we recommend to change require that the database gets re-initialized. We assume you don't have data there (since we are covering a fresh install) or you have taken a backup for later restore as explained above in the Preparations section.

cd /var/lib/
mv mysql mysql.old.datadir
mkdir mysql
chown mysql.mysql mysql

# mariadb 10.0 and 10.1
mysql_install_db
# percona 5.6
mysqld --user=mysql
# percona 5.7
mysqld --initialize-insecure --user=mysql

(Don't be worried about the <code>insecure</code>, it just means we set the db root pw in the next steps.)

=== Cluster startup ===

Typically on startup a Galera node tries to join a cluster, and if it fails, it will exit. Thus, when no cluster nodes are running, the first cluster node to be started needs to be told to not try to join a cluster, but rather bootstrap a new cluster. The exact arguments vary from version to version and from flavor to flavor.

==== First node ====

So we initialize the cluster bootstrap on the first node:

# percona 5.6, 5.7
service mysql bootstrap-pxc
# mariadb 10.0
service mysql bootstrap
# mariadb 10.1: service mysql bootstrap seems to be broken, does not pass the necessary options to mysqld
galera_new_cluster

Run <code>mysql_secure_installation</code> for a "secure by default" installation:

mysql_secure_installation

The further steps in this guide omit <code>-u -p</code> arguments to the MySQL client. Rather than passing them on the command line [https://dev.mysql.com/doc/refman/5.7/en/password-security-user.html] it is recommended to place the credentials in a file like <code>/root/.my.cnf</code> like

[client]
user=root
password=wip9Phae3Beijeed

We need a Galera replication user:

CREATE USER 'sstuser'@'localhost' IDENTIFIED BY 'OpIdjijwef0';
-- percona 5.6, mariadb 10.0
GRANT RELOAD, LOCK TABLES, REPLICATION CLIENT ON *.* TO 'sstuser'@'localhost';
-- percona 5.7, mariadb 10.1
GRANT PROCESS, RELOAD, LOCK TABLES, REPLICATION CLIENT ON *.* TO 'sstuser'@'localhost';
FLUSH PRIVILEGES;

(Debian specific note: MariaDB provided startup scripts use the distro's mechanism of verifying startup/shutdown using a system user, so we create that as well:

# mariadb 10.0, 10.1
GRANT ALL PRIVILEGES ON *.* TO "debian-sys-maint"@"localhost" IDENTIFIED BY "adBexthTsI5TaEps";

If you do this, yo need to synchronize the <code>/etc/mysql/debian.cnf</code> file from the first node to the other nodes as well.)

==== Other nodes ====

On the other nodes, we only need to restart the service now, to trigger a full state transfer from the first node to the other nodes.

We recommend to do this serially to let one state transfer complete before the second state transfer.

==== First node (continued) ====

Only applicable if you used <code>galera_new_cluster</code> before rather than the service script: In order to get the systemctl status consistent, restart the service on the first node:

# mariadb 10.1: restart the service so that the systemctl status is consistent
mysqladmin shutdown
service mysql bootstrap

=== Verify the replication ===

The key tool to verify replication status is

mysql> show status like "%wsrep%";

This will give a lot of output. You want to verify in particular

+------------------------------+--------------------------------------+
| Variable_name | Value |
+------------------------------+--------------------------------------+
| wsrep_cluster_size | 3 |
| wsrep_cluster_status | Primary |
| wsrep_local_state | 4 |
| wsrep_local_state_comment | Synced |
| wsrep_ready | ON |
+------------------------------+--------------------------------------+

You can also explicitly verify replication by creating / inserting DBs, tables, rows on one node and select on other nodes.

==== Troubleshooting ====

The logs are helpful. Always.

Common mistakes are listed below.

If the Galera module does not get loaded at all:
* Configuration settings in ''my.cnf'' which are incompatible to Galera
* Wrong path of the shared object providing the Galera plugin in wsrep.cnf (wsrep_provider)

If the first node starts, but the second / third nodes can not be added to the cluster:
* User for the replication not created correctly on the first Galera node
* SST fails due to missing / wrong version prerequisite packages (not everything is hardcoded in package dependencies -- make sure you got percona-xtrabackup installed in the correct version, and also socat). If SST fails, do not only look into mysqls primary error logs, but also into logfiles from the SST tool in /var/lib/mysql on the donor node.

=== Notes about configuring OX for use with Galera ===

==== Write requests ====

Open-Xchange supports Galera as database backend only in the configuration where all writes are directed to one Galera node. For availability, it makes sense to not configure one Galera node's IP address directly, but rather employ some HA solution which offers active-passive functionality. Options therefore are discussed below.

==== Read requests ====

Read requests can be directed to any node in the Galera cluster. Our standard approach is to recommend to use a loadbalancer to implement round-robin over all nodes in a Galera cluster for the read requests. But you can also chose to use a dedicated read node (the same node, or a different node, than the write node). Each of the approaches has its own advantages.

* Load balancer based setup: Read requests get distributed round-robin between the Galera nodes. Theoretically by distributing the load of the read requests, you benefit from lower latencies and more throughput. But this has never been benchmarked yet. For a discussion of available loadbalances, see next section. OX-wise, in this configuration, you have two alternatives:
** The Galera option wsrep_causal_reads=1 option enables you to configure OX with its replication monitor disabled (com.openexchange.database.replicationMonitor=false in configdb.properties). This is the setup which seems to perform best according to our experience as turning off the replication monitor reduces the commits on the DB and thus the write operations per second on the underlying storage significantly, which outweights the drawback from having higher commit latency due to fully synchronous mode.
** Alternatively, you can run Galera with wsrep_causal_reads=0 when switching on OX builtin replication monitor. This is also a valid setup.
* Use a designated floating IP for the read requests: This eliminates the need of a load balancer. With this option you will not gain any performance, but the quantitative benefit is unclear anyhow.
* Use the floating IP for the writes also for the reads: In this scenario, you direct all database queries only to one Galera node, and the other two nodes are only getting queries in case of a failure of that node. In this case, you can even use wsrep_causal_reads=0 while still having OX builtin replication monitor switched off. However we do not expect this option to be superior to the round-robin loadbalancer approach.

=== Loadbalancer options ===

While the JDBC driver has some round-robin load balancing capabilities built-in, we don't recommend it for production use since it lacks possibilities to check the Galera nodes health states.

Loadbalancers used for OX -> Galera loadbalancing should be able to implement active-passive instances for the write requests, and active-active (round-robin) instances for the read requests. (If they cannot implement active-passive, you can still take a floating IP therefore.) Furthermore it is required to configure node health checks not only on the TCP level (by a simple connect), but to query the Galera health status periodically, evaluating Galera WSREP status variables. Otherwise split-brain scenarios or other bad states cannot be detected. For an example of such an health check, see our [[Clustercheck]] page.

Some customers use loadbalancing appliances. It is important to check that if the (virtual) infrastructure offers "loadbalancer" instances that they satisfy the given requirements. Often this is not the case. In particular, a simple "DNS round robin" approach is not viable.

==== LVS/ipvsadm/keepalived ====

If you want to create your own loadbalancers based on Linux, we usually recommend LVS (Linux Virtual Servers) controlled by Keepalived. LVS is a set of kernel modules implementing a L4 loadbalancer which performs quite well. Keepalived is a userspace daemon to control LVS rules, using health checks to reconfigure LVS rules if required. Keepalived / LVS requires one (or, for availability, two) dedicated linux nodes to run on. This can be a disadvantage for some installations, but usually, it pays off. We provide some configuration information on Keepalived [[Keepalived|here]].

==== MariaDB Maxscale ====

Since Maxscale has become GA in 2015, it seems to have undergone significant stability, performance and functional improvements. We are currently experimenting with Maxscale and share our installation / configuration knowledge [[Maxscale|here]]. It looks quite promising and might become ''the standard replacement'' for HAproxy, while we still presume Keepalived offers superior robustness and performance, coming with the cost of the requirement for one (or more) dedicated loadbalancer nodes.

==== HAproxy ====

In case where the Keepalived based approach is not feasible due to its requirements on the infrastructure, it is also possible to use a HAproxy based solution where HAproxy processes run on each of the OX nodes, configured for one round-robin and one active/passive instance. OX is then connecting to the local HAproxy instances. It is vital to configure HAproxy timeouts different from the defaults, otherwise HAproxy will kill active DB connections, causing errors. Be aware that in large installations the number of (distributed) HAproxy instances can get quite large. Some configuration hints for HAproxy are available [[HAproxy|here]].

== Master/Slave database setup ==

While we also support also "legacy" (pre-GTID) Master/Slave replication, we recommend to use GTID based replication, for easier setup and failure recovery. Support for GTID based replication has been added with OX 7.8.0.

GTID has been available since MySQL 5.6, so no 5.5 installation instructions below, sorry. We try to be generic in this documentation (thus, applicable to Oracle Community Edition and MariaDB) and point out differences where needed. Note: Instructions below include information about Oracle Community MySQL 5.7 which is not yet formally supported.

=== Preparations ===

Our configuration process includes wiping and reinitializing the datadir. This is usually not a problem in a fresh installation. If you want to upgrade an existing database to GTID master-slave, please be prepared to wipe the datadir, i.e. take a <code>mysqldump</code> for later restoration into the properly configured master.

Depeding on the flavor of the current database, this can be something like

# mariadb or oracle mysql without GTIDs
mysqldump --databases configdb oxdb_{5..14} > backup.sql

# mysql 5.6 with GTIDs... we dont want GTIDs here
mysqldump --databases --set-gtid-purged=OFF configdb oxdb_{5..14} > backup.sql

Be sure to verify the list of databases.

=== Installation ===

Software installation is identical for master and slave.

Please follow the instructions for installing from The vendors.

* Oracle Community Edition: https://dev.mysql.com/doc/mysql-apt-repo-quick-guide/en/
* MariaDB (10.0, 10.1): https://downloads.mariadb.org/mariadb/repositories/

Stop the service (if it is running):

service mysql stop

=== Configuration ===

Configuration as per configuration files is also identical for master and slave.

Consult [[My.cnf]] for general recommendations how to configure databases for usage with OX.

For GTID based replication, make sure you add some configurables to a new <code>/etc/mysql/ox.conf.d/gtid.cnf</code> file (assuming you are following our proposed schema of adding a <code>!includedir /etc/mysql/ox.conf.d/</code>" directive to <code>/etc/mysql/my.cnf</code>):

# GTID
log-bin=mysql-bin
server-id=...
log_slave_updates = ON

Oracle Community Edition: we need to add also

enforce_gtid_consistency = ON
gtid_mode = ON

(GTID mode is on by default on MariaDB.)

Use unique a <code>server-id</code> for each server; like <code>1</code> for the master, <code>2</code> for slave. For more complicated setups (like multiple slaves), adjust accordingly.

Since applying our configuration / sizing requires reinitialization of the MySQL datadir, we wipe/recreate it. Caution: this assumes we are running an empty database. If there is data in the database you want to keep, use mysqldump. See Preparation section above.

So, to initialize the datadir:

cd /var/lib/
mv mysql mysql.old.datadir
mkdir mysql
chown mysql.mysql mysql

(When coming from an existing installation, be sure to wipe also old binlogs. They can confuse the server on startup. Their location varies by configuration.)

The step to initialize the datadir is different for the different DBs:

# MariaDB 10.0, 10.1
mysql_install_db

# Oracle 5.6
mysql_install_db -u mysql

# Oracle 5.7
mysqld --initialize-insecure --user=mysql

(Don't be worried about the <code>insecure</code>, it just means we set the db root pw in the next steps.)

Then:

service mysql restart
mysql_secure_installation

We want to emphasize the last step to run "secure".

Steps up to here apply to both the designated master and slave. The next steps will apply to the master.

=== Replication Setup ===

==== Master Setup ====

Create a replication user on the master (but, as always, pick your own password, and use the same password in the slave setup below):

mysql -e "CREATE USER 'repl'@'gtid-slave.localdomain' IDENTIFIED BY 'IvIjyoffod2'; GRANT REPLICATION SLAVE ON *.* TO 'repl'@'gtid-slave.localdomain';"

Now would also be the time to restore a previously created mysqldump, or add other users you need for adminstration, monitoring etc (like <code>debian-sys-maint@localhost</code>, for example). Adding the OX users is explained below ("Creating Open-Xchange user").

# If you took a dump for restore before
mysql < backup.sql

To prepare for the initial sync of the slave, set the master read-only:

mysql -e "SET @@global.read_only = ON;"

Create a dump to initialize the slave:

# MariaDB
mysqldump --all-databases --triggers --routines --events --master-data --gtid > master.sql

# Oracle
mysqldump --all-databases --triggers --routines --events --set-gtid-purged=ON > master.sql

Transfer to the slave:

scp master.sql gtid-slave:

==== Slave Setup ====

Configure the replication master settings. Note we don't need complicated binlog position settings etc with GTID.

Yet again DB-specific (use the repl user password from above):

# MariaDB
mysql -e 'CHANGE MASTER TO MASTER_HOST="gtid-master.localdomain", MASTER_USER="repl", MASTER_PASSWORD="IvIjyoffod2";'

# Oracle
mysql -e "CHANGE MASTER TO MASTER_HOST='gtid-master.localdomain', MASTER_USER='repl', MASTER_PASSWORD='IvIjyoffod2', MASTER_AUTO_POSITION=1;"
# https://www.percona.com/blog/2013/02/08/how-to-createrestore-a-slave-using-gtid-replication-in-mysql-5-6/
mysql -e "RESET MASTER;"

Read the master dump:

mysql < master.sql

Start replication on the slave:

mysql -e 'START SLAVE;'
mysql -e 'SHOW SLAVE STATUS\G'

==== Master Setup (continued) ====

Finally, unset read-only on the master:

# on the master
mysql -e "SET @@global.read_only = OFF;"

=== Configure OX to use with Master/Slave replication ===

Not much special wisdom here. OX was designed to be used with master/slave databases. For the ConfigDB, <code>configdb.properties</code> allows configuration of a <code>readUrl</code> and <code>writeUrl</code> (both of which are set to the correct values if you use <code>oxinstaller</code> with the correct arguments <code>--configdb-readhost</code>, <code>--configdb-writehost</code>).

(Obviously, the master is for writing and the slave is for reading.)

For the individiual UserDBs, use <code>registerdatabase -m true</code> for the masters and <code>registerdatabase -m false -M ...</code> for the respective slaves.

Be sure to have enabled the replication monitor in <code>configdb.properties</code>: <code>com.openexchange.database.replicationMonitor=true</code> (which it is by default); while GTID can show synchronous semantics, it is specified to silently fall back to asynchronous in certain circumstances, so synchronity is not guaranteed.

We recommend, though, to not register the databases directly by their native hostname or IP, but rather use some kind of HA system in order to be able to easily move a floating/failover IP from the master to the slave in case of master failure. Configuring and running such systems (like, corosync/pacemaker, keepalived, or whatever) is out of scope of this documentation, however.

== Creating Open-Xchange user ==

Now setup access for the Open-Xchange Server database user 'openexchange' to configdb and the oxdb for both groupware server addresses. These databases do not exist yet, but will be created during the Open-Xchange Server installation.

Notes:

* Please use a real password.
* The IPs in this example belong to the two different Open-Xchange Servers, please adjust them accordingly.
* If using a database on the same host as the middlware (usually done for POCs and demo installations), you need to grant also to the ''localhost'' host.
* Consult [[AppSuite:DB_user_privileges]] (or ''grep GRANT /opt/open-xchange/sbin/initconfigdb'') for an up-to-date list of required privileges. The following statement was correct as of the time of writing this section.

mysql> GRANT CREATE, LOCK TABLES, REFERENCES, INDEX, DROP, DELETE, ALTER, SELECT, UPDATE, INSERT, CREATE TEMPORARY TABLES, SHOW VIEW, SHOW DATABASES ON *.* TO 'openexchange'@'10.20.30.213' IDENTIFIED BY 'IntyoyntOat1' WITH GRANT OPTION;
mysql> GRANT CREATE, LOCK TABLES, REFERENCES, INDEX, DROP, DELETE, ALTER, SELECT, UPDATE, INSERT, CREATE TEMPORARY TABLES, SHOW VIEW, SHOW DATABASES ON *.* TO 'openexchange'@'10.20.30.215' IDENTIFIED BY 'IntyoyntOat1' WITH GRANT OPTION;

Tune apache2 for more concurrent connections

2018-09-06T10:11:51Z

Dominik.epple:

= Apache Configuration =

== MPM Module ==

Please ensure that your apache is using the <code>mpm_worker</code> or <code>mpm_event</code> module. This allows us to serve lots of concurrent connections by using less RAM than with <code>mpm_prefork</code> as we are going to start much less processes. Unfortunately the default MPM module seems to differ from distro to distro, so doublecheck and make sure you are on the right setting.

The method to configure the MPM module differes of course also from distribution to distribution:

* CentOS7/RHEL7: adjust <code>/etc/httpd/conf.modules.d/00-mpm.conf</code>
* Debian: use <code>a2dismod</code> / <code>a2enmod</code> to disable the wrong and enable the desired MPM Module.

About <code>mpm_worker</code> vs <code>mpm_event</code>: we assume they are equivalent for our purpose. <code>mpm_worker</code> is the older one and just works fine. <code>mpm_event</code> should improve on it further but we have been unable to observe any actual advantages.

== Concurrent Connections ==

By default apache2 is configured to support 150 concurrent connections.
This forces all parallel requests beyond that limit to wait.
Especially if, for example, active sync clients maintain a permanent connection for push events to arrive.

This is an example configuration to provide 8000 concurrent connections.

<pre><nowiki>
<IfModule mpm_worker_module>
ServerLimit 250
StartServers 10
MinSpareThreads 75
MaxSpareThreads 250
ThreadLimit 64
ThreadsPerChild 32
MaxRequestWorkers 8000
MaxConnectionsPerChild 10000
</IfModule>
</nowiki></pre>

Note: MaxRequestWorkers was previously named MaxClients and MaxConnectionsPerChild was previously named MaxRequestsPerChild. If you are using old (pre 2.4) verions of apache you might need to use the old names.

Short explanation of the parameters:

<table border="1" cellpadding="5" cellspacing="0">
<tr>
<td><b>ServerLimit</b></td><td>Declares the maximum number of running apache processes. If you change this value you have to restart the daemon.</td>
</tr>
<tr>
<td><b>StartServers</b></td><td>The number of processes to start initially when starting the apache daemon.</td>
</tr>
<tr>
<td><b>MinSpareThreads/MaxSpareThreads</b></td><td>This regulates how many threads may stay idle without being killed. Apache regulates this on its own very well with default values.</td>
</tr>
<tr>
<td><b>ThreadsPerChild</b></td><td>How many threads can be created per process. Can be changed during a reload.</td>
</tr>
<tr>
<td><b>ThreadLimit</b></td><td><b>ThreadsPerChild</b> can be configured as high as this value during runtime. If you change this value you have to restart the daemon.</td>
</tr>
<tr>
<td><b>MaxRequestWorkers</b></td><td>This declares how many concurrent connections we provide. Devided by '''ThreadsPerChild''' you get the suitable ServerLimit value. May be less than '''ServerLimit * ThreadsPerChild''' to reserve some resources that can be engaged during runtime with increasing '''MaxRequestWorkers''' and reloading the configuration.</td>
</tr>
<tr>
<td><b>MaxConnectionsPerChild</b></td><td>Defines the number of Connections that a process can handle during its lifetime (keep-alives are counted once). After that it will be killed. This can be used to prevent possible apache memory leaks. If set to 0 the lifetime is infinite.</td>
</tr>
</table>

For further information on these parameters see http://httpd.apache.org/docs/2.4/mod/worker.html and http://httpd.apache.org/docs/2.4/mod/mpm_common.html.

--[[User:Steffen.templin|Steffen.templin]] 12:10, 23 May 2011 (UTC)

HAproxy

2018-07-18T14:24:08Z

Dominik.epple:

= HAproxy Loadbalancer =

== Introduction ==

HAproxy is a mature and popular option for userspace loadbalancing under Linux (and similar operating systems). Its main (original) purpose is probably HTTP load balancing.

In Open-Xchange installations, for loadbalancing (reverse proxying) user session HTTP traffic towards our middleware, we require Apache, for reasons of correct handling of HTTP sticky sessions. See the corresponding "Configure Services" sections in our
[[AppSuite:Main_Page_Quickinstall#Quick_Installation_Guide|Quickinstall Guides]] for reference, [[AppSuite:Open-Xchange_Installation_Guide_for_CentOS_7#Configure_services|e.g. the CentOS version]].

Thus, in Open-Xchange installations HAproxy thus typically acts in other roles:

* As front-level loadbalancer in front of the Apache instances (in enterprise installations, however, usually other software is employed for that purpose), or
* As loadbalancer for connecting the OX middleware to the database (Galera) instances, or
* As loadbalancer for connecting [[AppSuite:Dovecot_Antiabuse_Shield|Dovecot Anti Abuse Shield]] to Dovecot and App Suite

We will cover in this article the latter two use cases.

== Software Installation ==

This is generic for the different use cases and thus described generically here.

HAproxy should be shipped with the distribution.

Historical Wheezy note: haproxy is provided in wheezy-backports, see http://haproxy.debian.net/. More recent (Debian) distributions don't need this extra repo and provide it with their native repos.

# yum install haproxy
# apt-get install haproxy

== HAproxy for DAAS / wforce Loadbalancing ==

In the following we present a configuration which was feedbacked to us from a large customer installation. (Thanks a lot!)

Crucial are the following aspects:

* Detect capacity issues on the wforce backends to be resilient to DOS attacks
* In case of capacity issues, just do a HTTP deny. It makes no sense to use some cross-datacenter routing (if multiple datacenters are available) as the latency combined with the query volume makes this less then ideal. A tiny amount of unanswered calls is fine, and doesn't affect the auth process or user experience.
* Timeouts are such that HAproxy's timeout (35 msecs(!)) is smaller than the corresponding wforce service consumer timeout (e.g. <code>dovecot</code>: <code>auth_policy_server_timeout_msecs</code> would get a corresponding value of 100 msecs). In this way timeouts are a little more often, but graceful and have little to no impact on user experience.

=== Configuration ===

frontend wforce :8084
mode http
# detect capacity issues in wforce backend
acl local_wforce_not_enough_capacity nbsrv(wforce_backend_primary) lt 2
# deny request if primary backend doesn't have capacity
http-request deny if local_wforce_not_enough_capacity
default_backend wforce_backend_primary

backend wforce_backend_primary
balance leastconn
mode http
timeout connect 35
timeout server 30s
timeout check 0
option httpchk GET /?command=ping HTTP/1.0\r\nAuthorization:\ Basic\ --redacted--
option http-keep-alive
server wforce-01.example.net 192.168.0.1:8084 weight 10 check inter 5000 fall 5 rise 3
server wforce-02.example.net 192.168.0.2:8084 weight 10 check inter 5000 fall 5 rise 3
server wforce-03.example.net 192.168.0.3:8084 weight 10 check inter 5000 fall 5 rise 3

== HAproxy for Galera Loadbalancing ==

HAproxy is one of the options to use for [[OXLoadBalancingClustering_Database#Galera_database_setup|Galera]] loadbalancing. Please consider reading through [[OXLoadBalancingClustering_Database#Loadbalancer_options|the available options]] before deciding for one solution.

=== System Design ===

We present a solution where each OX node runs a HAproxy instance. This way we can implement a solution without the need for additional loadbalancer (virtual) machines.

We create two HAproxy "listener", one round-robin for the read requests, one active/passive for the write requests.

=== Configuration ===

The following is a HAproxy configuration file <code>/etc/haproxy/haproxy.cfg</code>, assuming the Galera nodes have the IPs 10.0.0.1..3.

global
log 127.0.0.1 local0
log 127.0.0.1 local1 notice
user haproxy
group haproxy
# this is not recommended by the haproxy authors, but seems to improve performance for me
#nbproc 4
maxconn 256000
spread-checks 5
daemon
stats socket /var/lib/haproxy/stats

defaults
log global
retries 3
maxconn 256000
timeout connect 60000
timeout client 20m
timeout server 20m
option dontlognull
option redispatch
# the http options are not needed here
# but may be reasonable if you use haproxy also for some OX HTTP proxying
mode http
no option httpclose

listen mysql-read
bind 127.0.0.1:3306
mode tcp
balance roundrobin
option httpchk GET /
server db1 10.0.0.1:3306 check port 9200 inter 6000 rise 3 fall 3
server db2 10.0.0.2:3306 check port 9200 inter 6000 rise 3 fall 3
server db3 10.0.0.3:3306 check port 9200 inter 6000 rise 3 fall 3

listen mysql-write
bind 127.0.0.1:3307
mode tcp
balance roundrobin
option httpchk GET /master
# maybe be more prudent with the master for the fall parameter
server db1 10.0.0.1:3306 check port 9200 inter 6000 rise 3 fall 1
server db2 10.0.0.2:3306 check port 9200 inter 6000 rise 3 fall 1
server db3 10.0.0.3:3306 check port 9200 inter 6000 rise 3 fall 1

#
# can configure a stats interface here, but if you do so,
# change the username / password
#
#listen stats
# bind 0.0.0.0:8080
# mode http
# stats enable
# stats uri /
# stats realm Strictly\ Private
# stats auth user:pass

Note 1: the timeout options may seem exaggerated high, but they are required to ensure that it is not the loadbalancer shutting down MySQL connections while the systems still use it. Cf. <code>configdb.properties</code>:

# Maximum time in milliseconds a connection will be used. After this time
# the connection get closed.
maxLifeTime=600000

We got a default of 10 minutes, so allowing for some extra time to allow running queries to finish plus some overhead, 20 minutes look like a reasonable value for the connection timeout here.

Note 2: If you are configuring a dedicated loadbalancer node which should loadbalancer for other clients on the network (rather than a distributed / colocated HAproxy instance which should only serve for localhost) change the <code>bind</code> parameters accordingly.

=== Health check service ===

As you can see we use the <code>httpchk</code> option, so we assume a health check service to be available. Please have a look at the [[Clustercheck]] page how to configure such a service. Please be aware that we assume you use our customized, improved [[Clustercheck|<code>clustercheck</code>]] script, so please don't use the standard one.

Contrary to other setup instructions which recommend to configure one node as ''regular'' node and the other two ones as <code>backup</code> nodes, we recommend to leverage a health check which declares only the node with <code>wsrep_local_index=0</code> as available. This way we ensure that even in corner cases, multiple distributed HAproxy instances can not end up with declaring different nodes as designated write nodes, which [[OXLoadBalancingClustering_Database#Write_requests|would be problematic]].

=== Monitoring ===

Besided using the Galera check service configured before, you can also speak to the stats socket of HAproxy using <code>socat</code>.

# echo "show stat" | socat unix-connect:/var/lib/haproxy/stats stdio

The output is a CSV with long lines unsuitable for pasting here. Please test on your own.

There are more commands available via this socket to enable / disable servers; see the haproxy documentation for details. (As of writing that documentation could be found here: http://cbonte.github.io/haproxy-dconv/configuration-1.5.html#9.2 that URL seems unstable.)

AppSuite:7 10 Database Migration

2018-07-06T08:43:59Z

Dominik.epple: /* Initial OX App Suite v7.10.0 Rollout */

{{Version|7.10.0}}

= Database Migration with OX App Suite v7.10.0 =

OX App Suite v7.10.0 introduces significant changes regarding the underlying MySQL database system that require special attention in case of upgrades from former versions. Please read this paper carefully to ensure a smooth and clean upgrade process.

== Change Overview ==

The most significant changes are:
* New version and configuration requirements.
* Most VARCHAR columns need to be migrated to utf8mb4 character encoding.
* A major rewrite of the calendar application requires full data migration.

We strongly recommend to thoroughly plan and test the upgrade procedure. To gain insights about update task runtimes and the expected load, our recommendation is to clone ConfigDB and the biggest UserDB and perform an isolated test upgrade that especially covers the calendar migration and character encoding changes. Update task durations can be significantly longer than with previous upgrades and the migrations might cause noticeable higher I/O load. Also some additional disk space is needed during and after the migrations.

== Database System ==

MySQL Server is supported in versions 5.6 and 5.7 with recent patch levels only and MariaDB Server 10.1 and 10.2 respectively. Support for 5.6/10.1 exists for compatibility reasons and is transitional. We recommend upgrading to 5.7/10.2 as soon as possible. Any database system upgrade must happen before App Suite is upgraded to OX App Suite v7.10.0. Please follow the respective guides of your database vendor carefully.

Different App Suite and database system versions require different configurations of the DBMS, please follow http://oxpedia.org/wiki/index.php?title=My.cnf to have your database configured in a sane way. Especially the following items require some attention with the upgrade to App Suite 7.10.0 and upgrades of the DBMS itself:

* Supported SQL modes are only <code>NO_ENGINE_SUBSTITUTION</code> and <code>NO_AUTO_CREATE_USER</code>. Starting with App Suite 7.10.0, <code>ONLY_FULL_GROUP_BY</code> is also supported for MySQL 5.7. It is still not with older versions of MySQL or any MariaDB version! To review the current value, use <code>SHOW GLOBAL VARIABLES WHERE Variable_name = 'sql_mode';</code>.
* Ensure that <code>character_set_server</code> is set to <code>utf8</code> for App Suite 7.8.x. Again the current global default value can be obtained via <code>SHOW GLOBAL VARIABLES WHERE Variable_name = 'character_set_server';</code>. With 7.10.0 being fully rolled out, this setting must then be changed to <code>utf8mb4</code>. If <code>collation_server</code> is configured explicitly, it must be set to a matching value according to the character set in either case!

Note that App Suite <= 7.8.4 did not support MySQL 5.7/MariaDB 10.2 so far. While we recommend it for 7.10.0, this leaves a lack of definition during the upgrade process. We consider running 7.8.4 on top of MySQL 5.7/MariaDB 10.2 a valid scenario as long as it is transitional during the upgrade phase. Please take our configuration recommendations seriously to mitigate potential user-facing issues as far as possible.

== Upgrade Procedure ==

The upgrade to OX App Suite v7.10.0 can be performed like any other major upgrade before. However, the duration of blocking database update tasks for the mentioned charset and calendar migrations could conflict with customers’ availability demands. Therefore it is possible to decouple these special time- and resource-intensive tasks from the plain version upgrade. In this section a multi-step approach is described that performs the version upgrade before and independently from the migrations. Every step always results in a working system that is ready to serve user traffic. Some functional implications that affect user experience are outlined in the according subsections.

'''Important:''' Even though the first step leads to a basically working OX App Suite v7.10.0 environment, the subsequent steps are not optional but mandatory! Skipping the migrations will leave a few calendar features dysfunctional and can lead to issues with certain SQL queries that use explicit collations for searching and sorting. Running OX App Suite v7.10.0 in production without having all parts of the migration fulfilled is not supported – OX support will request you to complete the migration tasks when reporting issues that are not related to the migration itself.

== Initial OX App Suite v7.10.0 Rollout ==
If not done so far, upgrade your MySQL installation to a version supported with OX App Suite v7.10.0 but configure it to be compatible with OX App Suite v7.8.x as described in the “Database System” section.

Despite the fact that the two special migrations for calendar and character sets are explicitly skipped, this section assumes that the “Rolling Upgrade with breaking Hazelcast upgrade” is applied as described in http://oxpedia.org/wiki/index.php?title=AppSuite:Running_a_cluster#Updating_a_Cluster. For the application server upgrade, the common guide from http://oxpedia.org/wiki/index.php?title=AppSuite:UpdatingOXPackages can be followed.

Prepare a dedicated App Suite middleware node that will be used to perform the database update tasks. The node must not be serving any user traffic and be prepared with

* OX App Suite v7.10.0 packages
* Configuration according to the user production nodes
* The Hazelcast rolling upgrade compatibility package (see http://oxpedia.org/wiki/index.php?title=AppSuite:Running_a_cluster#Rolling_Upgrade_with_breaking_Hazelcast_upgrade)
* Exclude the update tasks for both mentioned migrations (i.e. calendar and character encoding). See “Update Task Exclusion” for details.
* Execute update tasks according to your preferred strategy. More on this can be found at http://oxpedia.org/wiki/index.php?title=UpdateTasks.

By default (if using the "runallupdate" tool") update tasks operate on database schemas sequentially, one at a time. All users from all contexts of a given schema are logged out and locked out. Then, DB schema changes are executed. Finally, users are unlocked and able to login again. Schemas typically contain a few thousand users (if our recommended sizing is being followed) and thus executing update tasks means bunches of a few thousand users will be affected sequentially. You will not have a full downtime. For each update task execution process the following statements hold true:

* All users got service for nearly all the time (all the time but the time where their schema is upgraded)
* For each point in time, nearly all users got service (all but the ones from the currently updated schema)
* When update tasks are completed, all users will have been affected by one "logout" - "locked out" cycle

After complete and successful update task execution, roll out OX App Suite v7.10.0 to one node after another. After complete rollout, reconfigure MySQL if appropriate as described in the “Database System” section.

=== Update Task Exclusion ===

Add the following lines to <code>/opt/open-xchange/etc/excludedupdatetasks.properties</code> or remove the leading # character if already included, so that it contains these two lines:

# Character Encoding Migration
com.openexchange.groupware.update.excludedUpdateTasks=groupware.utf8mb4

# Calendar Migration
com.openexchange.chronos.storage.rdb.migration.ChronosStorageMigrationTask

The first line does not denote one dedicated update task, but a whole list of tasks. To make exclusion more convenient, the concept of update task namespaces has been introduced. All update tasks belonging to the character encoding migration are part of the <code>groupware.utf8mb4</code> namespace. The denoted property takes care of excluding them all at once. You can list all according tasks with the <code>/opt/open-xchange/sbin/listUpdateTaskNamespaces</code> tool.

== Character Encoding Migration ==

The default character encoding for Unicode (named character set by MySQL) of MySQL will become <code>utf8mb4</code> in the near future. MariaDB on Debian Stretch (9) already has an according default configuration set when installing it from distribution packages. So far all VARCHAR columns are supposed to store at max. 3-byte UTF-8 characters due to the nature of MySQL’s utf8 character encoding. This leads to the fact that for example emojis cannot be saved as part of any App Suite entities. In a mixed-mode scenario (App Suite considers MySQL to operate in utf8mb4 mode due to the <code>character_set_server</code> setting, while columns are specified with utf8 encoding), this leads to issues whenever certain collations during SELECT statements are enforced. To avoid such issues generally and also increase user experience by finally allowing characters from the Unicode astral plane, Open-Xchange has decided to migrate existing data structures to the utf8mb4 character encoding.

This migration can be executed before or after the calendar migration, while it is recommend to execute it before.

The upgrade procedure is basically the same as above in terms executing update tasks, while the server software is already up to date and needs no further upgrades:

* Again prepare one dedicated node that doesn’t serve any user traffic
* Remove the according namespace property from <code>excludedupdatetasks.properties</code> again, but still keep the “ChronosStorageMigrationTask”. Afterwards restart the open-xchange daemon.
* Execute update tasks according to your preferred strategy.

'''Important:''' The update tasks require a lot of tables to be copied and re-created, leading to high I/O and especially sequential read and write operations. For every copied table the needed MySQL disk space doubles during the update task, so ensure enough free space before running the migration.

== Calendar Data Migration ==

The new calendar stack in OX App Suite v7.10.0 comes along with a new data model using its very own tables in MySQL. To preserve users calendar data, an update task <code>com.openexchange.chronos.storage.rdb.migration.ChronosStorageMigrationTask</code> has been introduced, that reads all data from the old tables, applies transformations to match the new stack and writes it into the new tables. The approach and upgrade process is described in detail at https://documentation.open-xchange.com/7.10.0/middleware/components/calendar/data_migration.html. Please read that article carefully before continuing. Especially we want to emphasize again the recommendation to test the migration with a copy of your real data to exclude or determine any issues beforehand.

Before executing this update task, OX App Suite v7.10.0 uses the new calendar stack on top of the old database tables through a compatibility layer. As the old storage layout lacks certain functionality, not all features are functional in between the application upgrade and execution of the update task. Due to this fact, a few spots are affected where not all appointment data that the user interface allows to enter can be persisted. This includes:

* Reminders, where still only one notification prior the appointment start is possible
* Colors, that cannot be mapped to the previously used labels
* 4-byte UTF-8 characters (emojis) are not yet possible
* Secret appointments, that are still stored as private ones
* An appointment's end timezone can't be applied, if it's different from the start timezone

Operators of ''non-Galera'' MySQL setups - i.e. Master-Slave replication - can potentially speed up the migration by configuring <code>com.openexchange.calendar.migration.intermediateCommits = false</code>. Per default the migration is performed in batches that separately committed, as Galera does not cope well with large transactions. By changing the setting to <code>false</code>, a single transaction with batch-mode enabled is used.

'''Important:''' The migrating of calendar data actually leads to a duplication of that data. Also with OX App Suite v7.10.0 every new calendar data is written to both, the old and the new tables redundantly to preserve to ability to roll back to OX App Suite v7.8.4. However, only the parts can preserved that match the old data model. I.e. additional features like multiple reminders or subscriptions of external calendars (Google Calendar, SchedJoules) cannot be preserved during a rollback.

A repeated OX App Suite v7.10.0 upgrade after a former rollback to OX App Suite v7.8.4 requires to force re-execution of this update task!

AppSuite:7 10 Database Migration

2018-07-06T08:43:41Z

Dominik.epple: /* Initial OX App Suite v7.10.0 Rollout */

{{Version|7.10.0}}

= Database Migration with OX App Suite v7.10.0 =

OX App Suite v7.10.0 introduces significant changes regarding the underlying MySQL database system that require special attention in case of upgrades from former versions. Please read this paper carefully to ensure a smooth and clean upgrade process.

== Change Overview ==

The most significant changes are:
* New version and configuration requirements.
* Most VARCHAR columns need to be migrated to utf8mb4 character encoding.
* A major rewrite of the calendar application requires full data migration.

We strongly recommend to thoroughly plan and test the upgrade procedure. To gain insights about update task runtimes and the expected load, our recommendation is to clone ConfigDB and the biggest UserDB and perform an isolated test upgrade that especially covers the calendar migration and character encoding changes. Update task durations can be significantly longer than with previous upgrades and the migrations might cause noticeable higher I/O load. Also some additional disk space is needed during and after the migrations.

== Database System ==

MySQL Server is supported in versions 5.6 and 5.7 with recent patch levels only and MariaDB Server 10.1 and 10.2 respectively. Support for 5.6/10.1 exists for compatibility reasons and is transitional. We recommend upgrading to 5.7/10.2 as soon as possible. Any database system upgrade must happen before App Suite is upgraded to OX App Suite v7.10.0. Please follow the respective guides of your database vendor carefully.

Different App Suite and database system versions require different configurations of the DBMS, please follow http://oxpedia.org/wiki/index.php?title=My.cnf to have your database configured in a sane way. Especially the following items require some attention with the upgrade to App Suite 7.10.0 and upgrades of the DBMS itself:

* Supported SQL modes are only <code>NO_ENGINE_SUBSTITUTION</code> and <code>NO_AUTO_CREATE_USER</code>. Starting with App Suite 7.10.0, <code>ONLY_FULL_GROUP_BY</code> is also supported for MySQL 5.7. It is still not with older versions of MySQL or any MariaDB version! To review the current value, use <code>SHOW GLOBAL VARIABLES WHERE Variable_name = 'sql_mode';</code>.
* Ensure that <code>character_set_server</code> is set to <code>utf8</code> for App Suite 7.8.x. Again the current global default value can be obtained via <code>SHOW GLOBAL VARIABLES WHERE Variable_name = 'character_set_server';</code>. With 7.10.0 being fully rolled out, this setting must then be changed to <code>utf8mb4</code>. If <code>collation_server</code> is configured explicitly, it must be set to a matching value according to the character set in either case!

Note that App Suite <= 7.8.4 did not support MySQL 5.7/MariaDB 10.2 so far. While we recommend it for 7.10.0, this leaves a lack of definition during the upgrade process. We consider running 7.8.4 on top of MySQL 5.7/MariaDB 10.2 a valid scenario as long as it is transitional during the upgrade phase. Please take our configuration recommendations seriously to mitigate potential user-facing issues as far as possible.

== Upgrade Procedure ==

The upgrade to OX App Suite v7.10.0 can be performed like any other major upgrade before. However, the duration of blocking database update tasks for the mentioned charset and calendar migrations could conflict with customers’ availability demands. Therefore it is possible to decouple these special time- and resource-intensive tasks from the plain version upgrade. In this section a multi-step approach is described that performs the version upgrade before and independently from the migrations. Every step always results in a working system that is ready to serve user traffic. Some functional implications that affect user experience are outlined in the according subsections.

'''Important:''' Even though the first step leads to a basically working OX App Suite v7.10.0 environment, the subsequent steps are not optional but mandatory! Skipping the migrations will leave a few calendar features dysfunctional and can lead to issues with certain SQL queries that use explicit collations for searching and sorting. Running OX App Suite v7.10.0 in production without having all parts of the migration fulfilled is not supported – OX support will request you to complete the migration tasks when reporting issues that are not related to the migration itself.

== Initial OX App Suite v7.10.0 Rollout ==
If not done so far, upgrade your MySQL installation to a version supported with OX App Suite v7.10.0 but configure it to be compatible with OX App Suite v7.8.x as described in the “Database System” section.

Despite the fact that the two special migrations for calendar and character sets are explicitly skipped, this section assumes that the “Rolling Upgrade with breaking Hazelcast upgrade” is applied as described in http://oxpedia.org/wiki/index.php?title=AppSuite:Running_a_cluster#Updating_a_Cluster. For the application server upgrade, the common guide from http://oxpedia.org/wiki/index.php?title=AppSuite:UpdatingOXPackages can be followed.

Prepare a dedicated App Suite middleware node that will be used to perform the database update tasks. The node must not be serving any user traffic and be prepared with

* OX App Suite v7.10.0 packages
* Configuration according to the user production nodes
* The Hazelcast rolling upgrade compatibility package (see http://oxpedia.org/wiki/index.php?title=AppSuite:Running_a_cluster#Rolling_Upgrade_with_breaking_Hazelcast_upgrade)
* Exclude the update tasks for both mentioned migrations (i.e. calendar and character encoding). See “Update Task Exclusion” for details.
* Execute update tasks according to your preferred strategy. More on this can be found at http://oxpedia.org/wiki/index.php?title=UpdateTasks.

By default (if using the "runupdate" tool") update tasks operate on database schemas sequentially, one at a time. All users from all contexts of a given schema are logged out and locked out. Then, DB schema changes are executed. Finally, users are unlocked and able to login again. Schemas typically contain a few thousand users (if our recommended sizing is being followed) and thus executing update tasks means bunches of a few thousand users will be affected sequentially. You will not have a full downtime. For each update task execution process the following statements hold true:

* All users got service for nearly all the time (all the time but the time where their schema is upgraded)
* For each point in time, nearly all users got service (all but the ones from the currently updated schema)
* When update tasks are completed, all users will have been affected by one "logout" - "locked out" cycle

After complete and successful update task execution, roll out OX App Suite v7.10.0 to one node after another. After complete rollout, reconfigure MySQL if appropriate as described in the “Database System” section.

=== Update Task Exclusion ===

Add the following lines to <code>/opt/open-xchange/etc/excludedupdatetasks.properties</code> or remove the leading # character if already included, so that it contains these two lines:

# Character Encoding Migration
com.openexchange.groupware.update.excludedUpdateTasks=groupware.utf8mb4

# Calendar Migration
com.openexchange.chronos.storage.rdb.migration.ChronosStorageMigrationTask

The first line does not denote one dedicated update task, but a whole list of tasks. To make exclusion more convenient, the concept of update task namespaces has been introduced. All update tasks belonging to the character encoding migration are part of the <code>groupware.utf8mb4</code> namespace. The denoted property takes care of excluding them all at once. You can list all according tasks with the <code>/opt/open-xchange/sbin/listUpdateTaskNamespaces</code> tool.

== Character Encoding Migration ==

The default character encoding for Unicode (named character set by MySQL) of MySQL will become <code>utf8mb4</code> in the near future. MariaDB on Debian Stretch (9) already has an according default configuration set when installing it from distribution packages. So far all VARCHAR columns are supposed to store at max. 3-byte UTF-8 characters due to the nature of MySQL’s utf8 character encoding. This leads to the fact that for example emojis cannot be saved as part of any App Suite entities. In a mixed-mode scenario (App Suite considers MySQL to operate in utf8mb4 mode due to the <code>character_set_server</code> setting, while columns are specified with utf8 encoding), this leads to issues whenever certain collations during SELECT statements are enforced. To avoid such issues generally and also increase user experience by finally allowing characters from the Unicode astral plane, Open-Xchange has decided to migrate existing data structures to the utf8mb4 character encoding.

This migration can be executed before or after the calendar migration, while it is recommend to execute it before.

The upgrade procedure is basically the same as above in terms executing update tasks, while the server software is already up to date and needs no further upgrades:

* Again prepare one dedicated node that doesn’t serve any user traffic
* Remove the according namespace property from <code>excludedupdatetasks.properties</code> again, but still keep the “ChronosStorageMigrationTask”. Afterwards restart the open-xchange daemon.
* Execute update tasks according to your preferred strategy.

'''Important:''' The update tasks require a lot of tables to be copied and re-created, leading to high I/O and especially sequential read and write operations. For every copied table the needed MySQL disk space doubles during the update task, so ensure enough free space before running the migration.

== Calendar Data Migration ==

The new calendar stack in OX App Suite v7.10.0 comes along with a new data model using its very own tables in MySQL. To preserve users calendar data, an update task <code>com.openexchange.chronos.storage.rdb.migration.ChronosStorageMigrationTask</code> has been introduced, that reads all data from the old tables, applies transformations to match the new stack and writes it into the new tables. The approach and upgrade process is described in detail at https://documentation.open-xchange.com/7.10.0/middleware/components/calendar/data_migration.html. Please read that article carefully before continuing. Especially we want to emphasize again the recommendation to test the migration with a copy of your real data to exclude or determine any issues beforehand.

Before executing this update task, OX App Suite v7.10.0 uses the new calendar stack on top of the old database tables through a compatibility layer. As the old storage layout lacks certain functionality, not all features are functional in between the application upgrade and execution of the update task. Due to this fact, a few spots are affected where not all appointment data that the user interface allows to enter can be persisted. This includes:

* Reminders, where still only one notification prior the appointment start is possible
* Colors, that cannot be mapped to the previously used labels
* 4-byte UTF-8 characters (emojis) are not yet possible
* Secret appointments, that are still stored as private ones
* An appointment's end timezone can't be applied, if it's different from the start timezone

Operators of ''non-Galera'' MySQL setups - i.e. Master-Slave replication - can potentially speed up the migration by configuring <code>com.openexchange.calendar.migration.intermediateCommits = false</code>. Per default the migration is performed in batches that separately committed, as Galera does not cope well with large transactions. By changing the setting to <code>false</code>, a single transaction with batch-mode enabled is used.

'''Important:''' The migrating of calendar data actually leads to a duplication of that data. Also with OX App Suite v7.10.0 every new calendar data is written to both, the old and the new tables redundantly to preserve to ability to roll back to OX App Suite v7.8.4. However, only the parts can preserved that match the old data model. I.e. additional features like multiple reminders or subscriptions of external calendars (Google Calendar, SchedJoules) cannot be preserved during a rollback.

A repeated OX App Suite v7.10.0 upgrade after a former rollback to OX App Suite v7.8.4 requires to force re-execution of this update task!

User:Dominik.epple

2018-06-27T12:30:18Z

Dominik.epple: /* Registering stuff */

== Install Guide ==

=== About this document ===

The aim of this document is to improve on the existing quickinstall guides to be more structured, provide a more extensive view on "single node and beyond" topics, follow closer to existing "best practices" (also, but not only security-wise), and point out what needs to be changed in clustered installations.

Most of the commands given in this document thus assume a high level design of "single-node, all-in-one".

This document was created on Debian Stretch (which, as of time of writing, is not even supported yet), but it should work as-is also for jessie. Porting to RHEL/SLES/... is TODO.

=== Preparations ===

==== System update ====

You want to start on latest patchlevel of your OS:

apt-get update
apt-get dist-upgrade
apt-get install less vim pwgen apt-transport-https
# or
yum update
yum install https://dl.fedoraproject.org/pub/epel/epel-release-latest-7.noarch.rpm
yum install vim less pwgen wget

reboot

==== Pregenerate passwords ====

This guide shall feature copy-paste ready commands which create installations with no default passwords.

We will pre-generate some passwords which will live as dotfiles in /root.

pwgen -c -n 16 1 > /root/.oxmasterpw
pwgen -c -n 16 1 > /root/.oxadminpw
pwgen -c -n 16 1 > /root/.oxuserpw
pwgen -c -n 16 1 > /root/.dbpw
pwgen -c -n 16 1 > /root/.dbrootpw

==== Prepare database ====

In real-world installations this will probably be multiple galera clusters of a supported flavor and version. For educational purposes a standalone DB on our single-node machine is sufficient.

Even for single-node, don't forget to apply database tuning. See [[My.cnf|our oxpedia article]] for default tunings. Note that typically you need to re-initialize the MySQL datadir after changing InnoDB sizing values, and subsequently start the service:

mysql_install_db
service mysql restart

We aim to create secure-by-default documentation, so here we go: Run mysql_secure_installation, and chose every security relevant option, but let the root password empty in this step, as we set it in the next step:

# leave the root password empty in mysql_secure_installation as we set it in the subsequent step
mysql_secure_installation
# now, configure the password from /root/.dbrootpw
mysql -e "UPDATE mysql.user SET Password=PASSWORD('$(cat /root/.dbrootpw)') WHERE User='root'; FLUSH PRIVILEGES;"
cat >/root/.my.cnf <<EOF
[client]
user=root
password=$(cat /root/.dbrootpw)
EOF

MySQL 5.7: the aforementioned must be adjusted using

ALTER USER USER() IDENTIFIED BY 'tiez7EiNgaish0ee';

These credentials also needs to be put in /etc/mysql/debian.cnf.

<div style="padding: 20px; border-style: solid; border-width: thin;">
Cluster note

On multiple db clusters, do it per node analogously. Just be aware the copy-paste command above expects the /root/.dbrootpw file.
</div>

==== Prepare OX user ====

While the packages will create the user automatically if it does not exist, we want to prepare the filestore now, and we need the user therefore.

useradd -r open-xchange

<div style="padding: 20px; border-style: solid; border-width: thin;">
Cluster Note

You should hard-wire the userid and groupid to the same fixed value. Otherwise, if you want to use a NFS filestore, you'll run into permissions problems, unless you use nfs4/kerberos/idmapd.

groupadd -r -g 999 open-xchange
useradd -r -g 999 -u 999 open-xchange
</div>

==== Prepare filestore ====

There are several options here.

===== Single-Node: local directory =====

For a single-node installation, you can just prepare a local directory:

mkdir /var/opt/filestore
chown open-xchange:open-xchange /var/opt/filestore

===== NFS =====

If using NFS:

Setup on the NFS server:

apt-get install nfs-kernel-server
service nfs-kernel-server restart

Configure /etc/exports. This is for traditional ip based access control; krb5 or other security configuration is out of scope of this document.

mkdir /var/opt/filestore
chown open-xchange:open-xchange /var/opt/filestore
echo "/var/opt/filestore 192.168.1.0/24(rw,sync,fsid=0,no_subtree_check)" >> /etc/exports
exportfs -a

Clients can then mount using

mkdir /var/opt/filestore
mount -t nfs -o vers=4 nfs-server:/filestore /var/opt/filestore

Or using fstab entries like

nfs-server:/filestore /var/opt/filestore nfs4 defaults 0 0

===== Object Store =====

You can use an object store. For lab environments Ceph is a convenient option. For demo / educational purpuses a "single node Ceph cluster" even co-located on your "single-node machine" is reasonble, but its setup is out of scope of this document. If you want to use this, be prepared to provide information about endpoint, bucket name, access key, secret key.

===== No filestore =====

If you dont want to provide a filestore, you can configure OX later to run without filestore. (Q: do we still need a dummy registerfilestore on a local directory in that event?)

==== Prepare mail system ====

Formally out of scope of this document.

If you need to create a testing dovecot/postfix setup, you can use our [https://gitlab.open-xchange.com/qa/performance/tree/develop/com.openexchange.test.performance.gatling/src/site-preprocess/dovecot performance testing sample config].

=== Install OX software ===

You need an ldb user and password for updates and proprietary repos. If you dont have such a user, you can still install the free components. You'll get a lot of authentication failed warnings however from apt tools unless you deconfigure the closed repos.

wget http://software.open-xchange.com/oxbuildkey.pub -O - | apt-key add -
wget -O/etc/apt/sources.list.d/ox.list http://software.open-xchange.com/products/DebianJessie.list
ldbuser=...
ldbpassword=...
sed -i -e "s/LDBUSER:LDBPASSWORD/$ldbuser:$ldbpassword/" /etc/apt/sources.list.d/ox.list
apt-get update
apt-get install open-xchange open-xchange-authentication-database open-xchange-grizzly open-xchange-admin open-xchange-appsuite-backend open-xchange-appsuite-manifest open-xchange-appsuite
#
# or
#
wget -O /etc/yum.repos.d/ox.repo http://software.open-xchange.com/products/RHEL7.repo
ldbuser=...
ldbpassword=...
sed -i -e "s/LDBUSER:LDBPASSWORD/$ldbuser:$ldbpassword/" /etc/yum.repos.d/ox.repo
yum install open-xchange open-xchange-authentication-database open-xchange-grizzly open-xchange-admin open-xchange-appsuite-backend open-xchange-appsuite-manifest open-xchange-appsuite

<div style="padding: 20px; border-style: solid; border-width: thin;">
Cluster note

*if you want to have separate frontend (apache) and middleware (open-xchange) systems, make sure to install packages which require apache as dependency on the frontend nodes, and packages which require java as a dependency on the middleware nodes. Currently this results in the split
** Frontend nodes: open-xchange-appsuite
** Middleware nodes: everything else
* If you want to use an object store, install the corresponding open-xchange-filestore-xyz package, like open-xchange-filestore-s3
* For hazelcast session storage, install also open-xchange-sessionstorage-hazelcast
</div>

=== Install database schemas ===

If the DB runs on localhost and you have root access, you can use

/opt/open-xchange/sbin/initconfigdb --configdb-pass="$(cat /root/.dbpw)" -a

<div style="padding: 20px; border-style: solid; border-width: thin;">

Cluster note

Create the DB users on all write instances manually. https://oxpedia.org/wiki/index.php?title=Template:OXLoadBalancingClustering_Database#Creating_Open-Xchange_user

mysql -e "GRANT CREATE, LOCK TABLES, REFERENCES, INDEX, DROP, DELETE, ALTER, SELECT, UPDATE, INSERT, CREATE TEMPORARY TABLES, SHOW VIEW, SHOW DATABASES ON *.* to 'openexchange'@'%' identified by '$(cat /root/.dbpw)' WITH GRANT OPTION;"

Run initconfigdb with some more options:

/opt/open-xchange/sbin/initconfigdb --configdb-user=openexchange --configdb-pass="$(cat /root/.dbpw)" --configdb-host=configdb-writehost

(initconfigdb needs to be run only once on one cluster node)
</div>

=== Initial configuration ===

/opt/open-xchange/sbin/oxinstaller --add-license=YOUR-OX-LICENSE-CODE --servername=oxserver --configdb-pass="$(cat /root/.dbpw)" --master-pass="$(cat /root/.oxmasterpw)" --network-listener-host=localhost --servermemory 1024

'''servername''' is more like a ''clustername'' and needs to be the same for all nodes.

'''servermemory''' should be adjusted to reflect the expected number of concurrent active sessions; sizing assumption is 4MB per session.

<div style="padding: 20px; border-style: solid; border-width: thin;">
Cluster Note

--configdb-readhost=...
--configdb-writehost=...
--imapserver=...
--smtpserver=...
--mail-login-src=<login|mail|name>
--mail-server-src=<user|global>
--transport-server-src=<user|global>
--jkroute=APP1
--object-link-hostname=[service DNS name like ox.example.com]
--extras-link=[https://ox.example.com/]
--name-of-oxcluster=[something unique per cluster, like business-staging; see --servername]
--network-listener-host=<localhost|*>

oxinstaller needs to be run on each cluster node with identical options besides the jkroute, which must be unique per cluster node and match the corresponding apache option, see below.

In a cluster you also want to configure hazelcast; see [[AppSuite:Running_a_cluster#Configuration]]. Most prominent options are

com.openexchange.hazelcast.enabled=true
com.openexchange.hazelcast.group.name=<reasonable unique group name>
com.openexchange.hazelcast.group.password=<unique password, CHANGE THE SHIPPED DEFAULT PASSWORD!>
com.openexchange.hazelcast.network.join=static # static is recommended over multicast for robustness
com.openexchange.hazelcast.network.join.static.nodes=... # configure your nodes as a comma-separated list; a bootstrapping subset is acceptable
com.openexchange.hazelcast.network.interfaces=... # pick your subnet; Wildcards (*) and ranges (-) can be used
</div>

Start the service:

systemctl restart open-xchange

<div style="padding: 20px; border-style: solid; border-width: thin;">
Cluster Note

Start the service on every cluster node.
</div>

=== Registering stuff ===

Register the "server":

/opt/open-xchange/sbin/registerserver -n oxserver -A oxadminmaster -P "$(cat /root/.oxmasterpw)"

<div style="padding: 20px; border-style: solid; border-width: thin;">
Cluster Note

All the register* commands need to be issued only once per cluster, as the effect is to enter corresponding lines in the configdb.
</div>

And the filestore:

/opt/open-xchange/sbin/registerfilestore -A oxadminmaster -P "$(cat /root/.oxmasterpw)" -t file:/var/opt/filestore -s 1000000 -x 1000000

<div style="padding: 20px; border-style: solid; border-width: thin;">
Cluster Note

If you chose an object store, the corresponding registerfilestore line reads as follows (Ceph radosgw example):

/opt/open-xchange/sbin/registerfilestore -A oxadminmaster -P "$(cat /root/.oxmasterpw)" -t s3://radosgw -s 1000000 -x 1000000

It requires configuration of the object store in filestore-s3.properties:

com.openexchange.filestore.s3.radosgw.endpoint=http://localhost:7480
com.openexchange.filestore.s3.radosgw.bucketName=oxbucket
com.openexchange.filestore.s3.radosgw.region=eu-west-1
com.openexchange.filestore.s3.radosgw.pathStyleAccess=true
com.openexchange.filestore.s3.radosgw.accessKey=...
com.openexchange.filestore.s3.radosgw.secretKey=...
com.openexchange.filestore.s3.radosgw.encryption=none
com.openexchange.filestore.s3.radosgw.signerOverride=S3SignerType
com.openexchange.filestore.s3.radosgw.chunkSize=5MB

Changing this file needs another

service open-xchange restart
</div>

And the database:

/opt/open-xchange/sbin/registerdatabase -A oxadminmaster -P "$(cat /root/.oxmasterpw)" -n oxdb -p "$(cat /root/.dbpw)" -m true

<div style="padding: 20px; border-style: solid; border-width: thin;">
Cluster Note

You probably have multiple clusters with read and write URLs. Register them each like first registering the master, subsequently registering the slave URL with the corresponding master ID:

/opt/open-xchange/sbin/registerdatabase -A oxadminmaster -P "$(cat /root/.oxmasterpw)" -n oxdb -p "$(cat /root/.dbpw)" -m true

This command gives as output the id of the registered db master, like

database 3 registered

Here, the id is 3. Use this id as argument of the -M switch of the next command:

/opt/open-xchange/sbin/registerdatabase -A oxadminmaster -P "$(cat /root/.oxmasterpw)" -n oxdbr -p "$(cat /root/.dbpw)" -m false -M 3
</div>

=== Configure Apache ===

Create config files /etc/apache2/conf-enabled/proxy_http.conf, /etc/apache2/sites-enabled/000-default.conf by copy-pasting as explained in [[AppSuite:Open-Xchange_Installation_Guide_for_Debian_8.0#Configure_services]]

Make sure you are using mpm_event. Apply concurrent connections tuning as described in [[Tune_apache2_for_more_concurrent_connections]].

Configure modules and restart:

a2enmod proxy proxy_http proxy_balancer expires deflate headers rewrite mime setenvif lbmethod_byrequests
systemctl restart apache2

<div style="padding: 20px; border-style: solid; border-width: thin;">
Cluster Note

Make sure that each middleware node got its unique server.properties:com.openexchange.server.backendRoute, e.g. APP1, APP2, APP3, etc.

Configure them in the http proxy definitions like

BalancerMember http://ox1:8009 timeout=100 smax=0 ttl=60 retry=60 loadfactor=50 route=APP1
BalancerMember http://ox2:8009 timeout=100 smax=0 ttl=60 retry=60 loadfactor=50 route=APP2
BalancerMember http://ox3:8009 timeout=100 smax=0 ttl=60 retry=60 loadfactor=50 route=APP3

If you colocate apache on middleware nodes, you might want to minimize cross-node routing, by setting on each node for the local node a loadfactor=50 and for the other nodes a status=+H instead of the loadfactor. E.g. on node ox1:

BalancerMember http://ox1:8009 timeout=100 smax=0 ttl=60 retry=60 loadfactor=50 route=APP1
BalancerMember http://ox2:8009 timeout=100 smax=0 ttl=60 retry=60 status=+H route=APP2
BalancerMember http://ox3:8009 timeout=100 smax=0 ttl=60 retry=60 status=+H route=APP3

However this requires host-specific config files for apache http proxy.

Use touch-appsuite with one identical timestamp on each frontend node.

timestamp=$(date -u +%Y%m%d.%H%M%S)
for node in $all_frontend_nodes; do
ssh $node /opt/open-xchange/sbin/touch-appsuite --timestamp=$timestamp
done

</div>

=== Provision a Test User ===

Provision a sample context and user:

/opt/open-xchange/sbin/createcontext -c 1 -A oxadminmaster -P $(cat /root/.oxmasterpw) -N localdomain -u oxadmin -d "Admin User" -g Admin -s User -p $(cat /root/.oxadminpw) -e oxadmin@localdomain -q 100 --access-combination-name groupware_premium
/opt/open-xchange/sbin/createuser -c 1 -A oxadmin -P $(cat /root/.oxadminpw) -u testuser -d "Test User" -g Test -s User -p $(cat /root/.oxuserpw) -e testuser@localdomain --access-combination-name groupware_premium

[[Context_Preprovisioning#Sample_Script]] provides an example how to fast-mode provision a huge number of contexts quickly.

User:Dominik.epple

2018-06-27T12:29:53Z

Dominik.epple: /* Initial configuration */

== Install Guide ==

=== About this document ===

The aim of this document is to improve on the existing quickinstall guides to be more structured, provide a more extensive view on "single node and beyond" topics, follow closer to existing "best practices" (also, but not only security-wise), and point out what needs to be changed in clustered installations.

Most of the commands given in this document thus assume a high level design of "single-node, all-in-one".

This document was created on Debian Stretch (which, as of time of writing, is not even supported yet), but it should work as-is also for jessie. Porting to RHEL/SLES/... is TODO.

=== Preparations ===

==== System update ====

You want to start on latest patchlevel of your OS:

apt-get update
apt-get dist-upgrade
apt-get install less vim pwgen apt-transport-https
# or
yum update
yum install https://dl.fedoraproject.org/pub/epel/epel-release-latest-7.noarch.rpm
yum install vim less pwgen wget

reboot

==== Pregenerate passwords ====

This guide shall feature copy-paste ready commands which create installations with no default passwords.

We will pre-generate some passwords which will live as dotfiles in /root.

pwgen -c -n 16 1 > /root/.oxmasterpw
pwgen -c -n 16 1 > /root/.oxadminpw
pwgen -c -n 16 1 > /root/.oxuserpw
pwgen -c -n 16 1 > /root/.dbpw
pwgen -c -n 16 1 > /root/.dbrootpw

==== Prepare database ====

In real-world installations this will probably be multiple galera clusters of a supported flavor and version. For educational purposes a standalone DB on our single-node machine is sufficient.

Even for single-node, don't forget to apply database tuning. See [[My.cnf|our oxpedia article]] for default tunings. Note that typically you need to re-initialize the MySQL datadir after changing InnoDB sizing values, and subsequently start the service:

mysql_install_db
service mysql restart

We aim to create secure-by-default documentation, so here we go: Run mysql_secure_installation, and chose every security relevant option, but let the root password empty in this step, as we set it in the next step:

# leave the root password empty in mysql_secure_installation as we set it in the subsequent step
mysql_secure_installation
# now, configure the password from /root/.dbrootpw
mysql -e "UPDATE mysql.user SET Password=PASSWORD('$(cat /root/.dbrootpw)') WHERE User='root'; FLUSH PRIVILEGES;"
cat >/root/.my.cnf <<EOF
[client]
user=root
password=$(cat /root/.dbrootpw)
EOF

MySQL 5.7: the aforementioned must be adjusted using

ALTER USER USER() IDENTIFIED BY 'tiez7EiNgaish0ee';

These credentials also needs to be put in /etc/mysql/debian.cnf.

<div style="padding: 20px; border-style: solid; border-width: thin;">
Cluster note

On multiple db clusters, do it per node analogously. Just be aware the copy-paste command above expects the /root/.dbrootpw file.
</div>

==== Prepare OX user ====

While the packages will create the user automatically if it does not exist, we want to prepare the filestore now, and we need the user therefore.

useradd -r open-xchange

<div style="padding: 20px; border-style: solid; border-width: thin;">
Cluster Note

You should hard-wire the userid and groupid to the same fixed value. Otherwise, if you want to use a NFS filestore, you'll run into permissions problems, unless you use nfs4/kerberos/idmapd.

groupadd -r -g 999 open-xchange
useradd -r -g 999 -u 999 open-xchange
</div>

==== Prepare filestore ====

There are several options here.

===== Single-Node: local directory =====

For a single-node installation, you can just prepare a local directory:

mkdir /var/opt/filestore
chown open-xchange:open-xchange /var/opt/filestore

===== NFS =====

If using NFS:

Setup on the NFS server:

apt-get install nfs-kernel-server
service nfs-kernel-server restart

Configure /etc/exports. This is for traditional ip based access control; krb5 or other security configuration is out of scope of this document.

mkdir /var/opt/filestore
chown open-xchange:open-xchange /var/opt/filestore
echo "/var/opt/filestore 192.168.1.0/24(rw,sync,fsid=0,no_subtree_check)" >> /etc/exports
exportfs -a

Clients can then mount using

mkdir /var/opt/filestore
mount -t nfs -o vers=4 nfs-server:/filestore /var/opt/filestore

Or using fstab entries like

nfs-server:/filestore /var/opt/filestore nfs4 defaults 0 0

===== Object Store =====

You can use an object store. For lab environments Ceph is a convenient option. For demo / educational purpuses a "single node Ceph cluster" even co-located on your "single-node machine" is reasonble, but its setup is out of scope of this document. If you want to use this, be prepared to provide information about endpoint, bucket name, access key, secret key.

===== No filestore =====

If you dont want to provide a filestore, you can configure OX later to run without filestore. (Q: do we still need a dummy registerfilestore on a local directory in that event?)

==== Prepare mail system ====

Formally out of scope of this document.

If you need to create a testing dovecot/postfix setup, you can use our [https://gitlab.open-xchange.com/qa/performance/tree/develop/com.openexchange.test.performance.gatling/src/site-preprocess/dovecot performance testing sample config].

=== Install OX software ===

You need an ldb user and password for updates and proprietary repos. If you dont have such a user, you can still install the free components. You'll get a lot of authentication failed warnings however from apt tools unless you deconfigure the closed repos.

wget http://software.open-xchange.com/oxbuildkey.pub -O - | apt-key add -
wget -O/etc/apt/sources.list.d/ox.list http://software.open-xchange.com/products/DebianJessie.list
ldbuser=...
ldbpassword=...
sed -i -e "s/LDBUSER:LDBPASSWORD/$ldbuser:$ldbpassword/" /etc/apt/sources.list.d/ox.list
apt-get update
apt-get install open-xchange open-xchange-authentication-database open-xchange-grizzly open-xchange-admin open-xchange-appsuite-backend open-xchange-appsuite-manifest open-xchange-appsuite
#
# or
#
wget -O /etc/yum.repos.d/ox.repo http://software.open-xchange.com/products/RHEL7.repo
ldbuser=...
ldbpassword=...
sed -i -e "s/LDBUSER:LDBPASSWORD/$ldbuser:$ldbpassword/" /etc/yum.repos.d/ox.repo
yum install open-xchange open-xchange-authentication-database open-xchange-grizzly open-xchange-admin open-xchange-appsuite-backend open-xchange-appsuite-manifest open-xchange-appsuite

<div style="padding: 20px; border-style: solid; border-width: thin;">
Cluster note

*if you want to have separate frontend (apache) and middleware (open-xchange) systems, make sure to install packages which require apache as dependency on the frontend nodes, and packages which require java as a dependency on the middleware nodes. Currently this results in the split
** Frontend nodes: open-xchange-appsuite
** Middleware nodes: everything else
* If you want to use an object store, install the corresponding open-xchange-filestore-xyz package, like open-xchange-filestore-s3
* For hazelcast session storage, install also open-xchange-sessionstorage-hazelcast
</div>

=== Install database schemas ===

If the DB runs on localhost and you have root access, you can use

/opt/open-xchange/sbin/initconfigdb --configdb-pass="$(cat /root/.dbpw)" -a

<div style="padding: 20px; border-style: solid; border-width: thin;">

Cluster note

Create the DB users on all write instances manually. https://oxpedia.org/wiki/index.php?title=Template:OXLoadBalancingClustering_Database#Creating_Open-Xchange_user

mysql -e "GRANT CREATE, LOCK TABLES, REFERENCES, INDEX, DROP, DELETE, ALTER, SELECT, UPDATE, INSERT, CREATE TEMPORARY TABLES, SHOW VIEW, SHOW DATABASES ON *.* to 'openexchange'@'%' identified by '$(cat /root/.dbpw)' WITH GRANT OPTION;"

Run initconfigdb with some more options:

/opt/open-xchange/sbin/initconfigdb --configdb-user=openexchange --configdb-pass="$(cat /root/.dbpw)" --configdb-host=configdb-writehost

(initconfigdb needs to be run only once on one cluster node)
</div>

=== Initial configuration ===

/opt/open-xchange/sbin/oxinstaller --add-license=YOUR-OX-LICENSE-CODE --servername=oxserver --configdb-pass="$(cat /root/.dbpw)" --master-pass="$(cat /root/.oxmasterpw)" --network-listener-host=localhost --servermemory 1024

'''servername''' is more like a ''clustername'' and needs to be the same for all nodes.

'''servermemory''' should be adjusted to reflect the expected number of concurrent active sessions; sizing assumption is 4MB per session.

<div style="padding: 20px; border-style: solid; border-width: thin;">
Cluster Note

--configdb-readhost=...
--configdb-writehost=...
--imapserver=...
--smtpserver=...
--mail-login-src=<login|mail|name>
--mail-server-src=<user|global>
--transport-server-src=<user|global>
--jkroute=APP1
--object-link-hostname=[service DNS name like ox.example.com]
--extras-link=[https://ox.example.com/]
--name-of-oxcluster=[something unique per cluster, like business-staging; see --servername]
--network-listener-host=<localhost|*>

oxinstaller needs to be run on each cluster node with identical options besides the jkroute, which must be unique per cluster node and match the corresponding apache option, see below.

In a cluster you also want to configure hazelcast; see [[AppSuite:Running_a_cluster#Configuration]]. Most prominent options are

com.openexchange.hazelcast.enabled=true
com.openexchange.hazelcast.group.name=<reasonable unique group name>
com.openexchange.hazelcast.group.password=<unique password, CHANGE THE SHIPPED DEFAULT PASSWORD!>
com.openexchange.hazelcast.network.join=static # static is recommended over multicast for robustness
com.openexchange.hazelcast.network.join.static.nodes=... # configure your nodes as a comma-separated list; a bootstrapping subset is acceptable
com.openexchange.hazelcast.network.interfaces=... # pick your subnet; Wildcards (*) and ranges (-) can be used
</div>

Start the service:

systemctl restart open-xchange

<div style="padding: 20px; border-style: solid; border-width: thin;">
Cluster Note

Start the service on every cluster node.
</div>

=== Registering stuff ===

Register the "server":

/opt/open-xchange/sbin/registerserver -n oxserver -A oxadminmaster -P "$(cat /root/.oxpw)"

<div style="padding: 20px; border-style: solid; border-width: thin;">
Cluster Note

All the register* commands need to be issued only once per cluster, as the effect is to enter corresponding lines in the configdb.
</div>

And the filestore:

/opt/open-xchange/sbin/registerfilestore -A oxadminmaster -P "$(cat /root/.oxpw)" -t file:/var/opt/filestore -s 1000000 -x 1000000

<div style="padding: 20px; border-style: solid; border-width: thin;">
Cluster Note

If you chose an object store, the corresponding registerfilestore line reads as follows (Ceph radosgw example):

/opt/open-xchange/sbin/registerfilestore -A oxadminmaster -P "$(cat /root/.oxpw)" -t s3://radosgw -s 1000000 -x 1000000

It requires configuration of the object store in filestore-s3.properties:

com.openexchange.filestore.s3.radosgw.endpoint=http://localhost:7480
com.openexchange.filestore.s3.radosgw.bucketName=oxbucket
com.openexchange.filestore.s3.radosgw.region=eu-west-1
com.openexchange.filestore.s3.radosgw.pathStyleAccess=true
com.openexchange.filestore.s3.radosgw.accessKey=...
com.openexchange.filestore.s3.radosgw.secretKey=...
com.openexchange.filestore.s3.radosgw.encryption=none
com.openexchange.filestore.s3.radosgw.signerOverride=S3SignerType
com.openexchange.filestore.s3.radosgw.chunkSize=5MB

Changing this file needs another

service open-xchange restart
</div>

And the database:

/opt/open-xchange/sbin/registerdatabase -A oxadminmaster -P "$(cat /root/.oxpw)" -n oxdb -p "$(cat /root/.dbpw)" -m true

<div style="padding: 20px; border-style: solid; border-width: thin;">
Cluster Note

You probably have multiple clusters with read and write URLs. Register them each like first registering the master, subsequently registering the slave URL with the corresponding master ID:

/opt/open-xchange/sbin/registerdatabase -A oxadminmaster -P "$(cat /root/.oxpw)" -n oxdb -p "$(cat /root/.dbpw)" -m true

This command gives as output the id of the registered db master, like

database 3 registered

Here, the id is 3. Use this id as argument of the -M switch of the next command:

/opt/open-xchange/sbin/registerdatabase -A oxadminmaster -P "$(cat /root/.oxpw)" -n oxdbr -p "$(cat /root/.dbpw)" -m false -M 3
</div>

=== Configure Apache ===

Create config files /etc/apache2/conf-enabled/proxy_http.conf, /etc/apache2/sites-enabled/000-default.conf by copy-pasting as explained in [[AppSuite:Open-Xchange_Installation_Guide_for_Debian_8.0#Configure_services]]

Make sure you are using mpm_event. Apply concurrent connections tuning as described in [[Tune_apache2_for_more_concurrent_connections]].

Configure modules and restart:

a2enmod proxy proxy_http proxy_balancer expires deflate headers rewrite mime setenvif lbmethod_byrequests
systemctl restart apache2

<div style="padding: 20px; border-style: solid; border-width: thin;">
Cluster Note

Make sure that each middleware node got its unique server.properties:com.openexchange.server.backendRoute, e.g. APP1, APP2, APP3, etc.

Configure them in the http proxy definitions like

BalancerMember http://ox1:8009 timeout=100 smax=0 ttl=60 retry=60 loadfactor=50 route=APP1
BalancerMember http://ox2:8009 timeout=100 smax=0 ttl=60 retry=60 loadfactor=50 route=APP2
BalancerMember http://ox3:8009 timeout=100 smax=0 ttl=60 retry=60 loadfactor=50 route=APP3

If you colocate apache on middleware nodes, you might want to minimize cross-node routing, by setting on each node for the local node a loadfactor=50 and for the other nodes a status=+H instead of the loadfactor. E.g. on node ox1:

BalancerMember http://ox1:8009 timeout=100 smax=0 ttl=60 retry=60 loadfactor=50 route=APP1
BalancerMember http://ox2:8009 timeout=100 smax=0 ttl=60 retry=60 status=+H route=APP2
BalancerMember http://ox3:8009 timeout=100 smax=0 ttl=60 retry=60 status=+H route=APP3

However this requires host-specific config files for apache http proxy.

Use touch-appsuite with one identical timestamp on each frontend node.

timestamp=$(date -u +%Y%m%d.%H%M%S)
for node in $all_frontend_nodes; do
ssh $node /opt/open-xchange/sbin/touch-appsuite --timestamp=$timestamp
done

</div>

=== Provision a Test User ===

Provision a sample context and user:

/opt/open-xchange/sbin/createcontext -c 1 -A oxadminmaster -P $(cat /root/.oxmasterpw) -N localdomain -u oxadmin -d "Admin User" -g Admin -s User -p $(cat /root/.oxadminpw) -e oxadmin@localdomain -q 100 --access-combination-name groupware_premium
/opt/open-xchange/sbin/createuser -c 1 -A oxadmin -P $(cat /root/.oxadminpw) -u testuser -d "Test User" -g Test -s User -p $(cat /root/.oxuserpw) -e testuser@localdomain --access-combination-name groupware_premium

[[Context_Preprovisioning#Sample_Script]] provides an example how to fast-mode provision a huge number of contexts quickly.

User:Dominik.epple

2018-06-27T12:29:41Z

Dominik.epple: /* Pregenerate passwords */

== Install Guide ==

=== About this document ===

The aim of this document is to improve on the existing quickinstall guides to be more structured, provide a more extensive view on "single node and beyond" topics, follow closer to existing "best practices" (also, but not only security-wise), and point out what needs to be changed in clustered installations.

Most of the commands given in this document thus assume a high level design of "single-node, all-in-one".

This document was created on Debian Stretch (which, as of time of writing, is not even supported yet), but it should work as-is also for jessie. Porting to RHEL/SLES/... is TODO.

=== Preparations ===

==== System update ====

You want to start on latest patchlevel of your OS:

apt-get update
apt-get dist-upgrade
apt-get install less vim pwgen apt-transport-https
# or
yum update
yum install https://dl.fedoraproject.org/pub/epel/epel-release-latest-7.noarch.rpm
yum install vim less pwgen wget

reboot

==== Pregenerate passwords ====

This guide shall feature copy-paste ready commands which create installations with no default passwords.

We will pre-generate some passwords which will live as dotfiles in /root.

pwgen -c -n 16 1 > /root/.oxmasterpw
pwgen -c -n 16 1 > /root/.oxadminpw
pwgen -c -n 16 1 > /root/.oxuserpw
pwgen -c -n 16 1 > /root/.dbpw
pwgen -c -n 16 1 > /root/.dbrootpw

==== Prepare database ====

In real-world installations this will probably be multiple galera clusters of a supported flavor and version. For educational purposes a standalone DB on our single-node machine is sufficient.

Even for single-node, don't forget to apply database tuning. See [[My.cnf|our oxpedia article]] for default tunings. Note that typically you need to re-initialize the MySQL datadir after changing InnoDB sizing values, and subsequently start the service:

mysql_install_db
service mysql restart

We aim to create secure-by-default documentation, so here we go: Run mysql_secure_installation, and chose every security relevant option, but let the root password empty in this step, as we set it in the next step:

# leave the root password empty in mysql_secure_installation as we set it in the subsequent step
mysql_secure_installation
# now, configure the password from /root/.dbrootpw
mysql -e "UPDATE mysql.user SET Password=PASSWORD('$(cat /root/.dbrootpw)') WHERE User='root'; FLUSH PRIVILEGES;"
cat >/root/.my.cnf <<EOF
[client]
user=root
password=$(cat /root/.dbrootpw)
EOF

MySQL 5.7: the aforementioned must be adjusted using

ALTER USER USER() IDENTIFIED BY 'tiez7EiNgaish0ee';

These credentials also needs to be put in /etc/mysql/debian.cnf.

<div style="padding: 20px; border-style: solid; border-width: thin;">
Cluster note

On multiple db clusters, do it per node analogously. Just be aware the copy-paste command above expects the /root/.dbrootpw file.
</div>

==== Prepare OX user ====

While the packages will create the user automatically if it does not exist, we want to prepare the filestore now, and we need the user therefore.

useradd -r open-xchange

<div style="padding: 20px; border-style: solid; border-width: thin;">
Cluster Note

You should hard-wire the userid and groupid to the same fixed value. Otherwise, if you want to use a NFS filestore, you'll run into permissions problems, unless you use nfs4/kerberos/idmapd.

groupadd -r -g 999 open-xchange
useradd -r -g 999 -u 999 open-xchange
</div>

==== Prepare filestore ====

There are several options here.

===== Single-Node: local directory =====

For a single-node installation, you can just prepare a local directory:

mkdir /var/opt/filestore
chown open-xchange:open-xchange /var/opt/filestore

===== NFS =====

If using NFS:

Setup on the NFS server:

apt-get install nfs-kernel-server
service nfs-kernel-server restart

Configure /etc/exports. This is for traditional ip based access control; krb5 or other security configuration is out of scope of this document.

mkdir /var/opt/filestore
chown open-xchange:open-xchange /var/opt/filestore
echo "/var/opt/filestore 192.168.1.0/24(rw,sync,fsid=0,no_subtree_check)" >> /etc/exports
exportfs -a

Clients can then mount using

mkdir /var/opt/filestore
mount -t nfs -o vers=4 nfs-server:/filestore /var/opt/filestore

Or using fstab entries like

nfs-server:/filestore /var/opt/filestore nfs4 defaults 0 0

===== Object Store =====

You can use an object store. For lab environments Ceph is a convenient option. For demo / educational purpuses a "single node Ceph cluster" even co-located on your "single-node machine" is reasonble, but its setup is out of scope of this document. If you want to use this, be prepared to provide information about endpoint, bucket name, access key, secret key.

===== No filestore =====

If you dont want to provide a filestore, you can configure OX later to run without filestore. (Q: do we still need a dummy registerfilestore on a local directory in that event?)

==== Prepare mail system ====

Formally out of scope of this document.

If you need to create a testing dovecot/postfix setup, you can use our [https://gitlab.open-xchange.com/qa/performance/tree/develop/com.openexchange.test.performance.gatling/src/site-preprocess/dovecot performance testing sample config].

=== Install OX software ===

You need an ldb user and password for updates and proprietary repos. If you dont have such a user, you can still install the free components. You'll get a lot of authentication failed warnings however from apt tools unless you deconfigure the closed repos.

wget http://software.open-xchange.com/oxbuildkey.pub -O - | apt-key add -
wget -O/etc/apt/sources.list.d/ox.list http://software.open-xchange.com/products/DebianJessie.list
ldbuser=...
ldbpassword=...
sed -i -e "s/LDBUSER:LDBPASSWORD/$ldbuser:$ldbpassword/" /etc/apt/sources.list.d/ox.list
apt-get update
apt-get install open-xchange open-xchange-authentication-database open-xchange-grizzly open-xchange-admin open-xchange-appsuite-backend open-xchange-appsuite-manifest open-xchange-appsuite
#
# or
#
wget -O /etc/yum.repos.d/ox.repo http://software.open-xchange.com/products/RHEL7.repo
ldbuser=...
ldbpassword=...
sed -i -e "s/LDBUSER:LDBPASSWORD/$ldbuser:$ldbpassword/" /etc/yum.repos.d/ox.repo
yum install open-xchange open-xchange-authentication-database open-xchange-grizzly open-xchange-admin open-xchange-appsuite-backend open-xchange-appsuite-manifest open-xchange-appsuite

<div style="padding: 20px; border-style: solid; border-width: thin;">
Cluster note

*if you want to have separate frontend (apache) and middleware (open-xchange) systems, make sure to install packages which require apache as dependency on the frontend nodes, and packages which require java as a dependency on the middleware nodes. Currently this results in the split
** Frontend nodes: open-xchange-appsuite
** Middleware nodes: everything else
* If you want to use an object store, install the corresponding open-xchange-filestore-xyz package, like open-xchange-filestore-s3
* For hazelcast session storage, install also open-xchange-sessionstorage-hazelcast
</div>

=== Install database schemas ===

If the DB runs on localhost and you have root access, you can use

/opt/open-xchange/sbin/initconfigdb --configdb-pass="$(cat /root/.dbpw)" -a

<div style="padding: 20px; border-style: solid; border-width: thin;">

Cluster note

Create the DB users on all write instances manually. https://oxpedia.org/wiki/index.php?title=Template:OXLoadBalancingClustering_Database#Creating_Open-Xchange_user

mysql -e "GRANT CREATE, LOCK TABLES, REFERENCES, INDEX, DROP, DELETE, ALTER, SELECT, UPDATE, INSERT, CREATE TEMPORARY TABLES, SHOW VIEW, SHOW DATABASES ON *.* to 'openexchange'@'%' identified by '$(cat /root/.dbpw)' WITH GRANT OPTION;"

Run initconfigdb with some more options:

/opt/open-xchange/sbin/initconfigdb --configdb-user=openexchange --configdb-pass="$(cat /root/.dbpw)" --configdb-host=configdb-writehost

(initconfigdb needs to be run only once on one cluster node)
</div>

=== Initial configuration ===

/opt/open-xchange/sbin/oxinstaller --add-license=YOUR-OX-LICENSE-CODE --servername=oxserver --configdb-pass="$(cat /root/.dbpw)" --master-pass="$(cat /root/.oxpw)" --network-listener-host=localhost --servermemory 1024

'''servername''' is more like a ''clustername'' and needs to be the same for all nodes.

'''servermemory''' should be adjusted to reflect the expected number of concurrent active sessions; sizing assumption is 4MB per session.

<div style="padding: 20px; border-style: solid; border-width: thin;">
Cluster Note

--configdb-readhost=...
--configdb-writehost=...
--imapserver=...
--smtpserver=...
--mail-login-src=<login|mail|name>
--mail-server-src=<user|global>
--transport-server-src=<user|global>
--jkroute=APP1
--object-link-hostname=[service DNS name like ox.example.com]
--extras-link=[https://ox.example.com/]
--name-of-oxcluster=[something unique per cluster, like business-staging; see --servername]
--network-listener-host=<localhost|*>

oxinstaller needs to be run on each cluster node with identical options besides the jkroute, which must be unique per cluster node and match the corresponding apache option, see below.

In a cluster you also want to configure hazelcast; see [[AppSuite:Running_a_cluster#Configuration]]. Most prominent options are

com.openexchange.hazelcast.enabled=true
com.openexchange.hazelcast.group.name=<reasonable unique group name>
com.openexchange.hazelcast.group.password=<unique password, CHANGE THE SHIPPED DEFAULT PASSWORD!>
com.openexchange.hazelcast.network.join=static # static is recommended over multicast for robustness
com.openexchange.hazelcast.network.join.static.nodes=... # configure your nodes as a comma-separated list; a bootstrapping subset is acceptable
com.openexchange.hazelcast.network.interfaces=... # pick your subnet; Wildcards (*) and ranges (-) can be used
</div>

Start the service:

systemctl restart open-xchange

<div style="padding: 20px; border-style: solid; border-width: thin;">
Cluster Note

Start the service on every cluster node.
</div>

=== Registering stuff ===

Register the "server":

/opt/open-xchange/sbin/registerserver -n oxserver -A oxadminmaster -P "$(cat /root/.oxpw)"

<div style="padding: 20px; border-style: solid; border-width: thin;">
Cluster Note

All the register* commands need to be issued only once per cluster, as the effect is to enter corresponding lines in the configdb.
</div>

And the filestore:

/opt/open-xchange/sbin/registerfilestore -A oxadminmaster -P "$(cat /root/.oxpw)" -t file:/var/opt/filestore -s 1000000 -x 1000000

<div style="padding: 20px; border-style: solid; border-width: thin;">
Cluster Note

If you chose an object store, the corresponding registerfilestore line reads as follows (Ceph radosgw example):

/opt/open-xchange/sbin/registerfilestore -A oxadminmaster -P "$(cat /root/.oxpw)" -t s3://radosgw -s 1000000 -x 1000000

It requires configuration of the object store in filestore-s3.properties:

com.openexchange.filestore.s3.radosgw.endpoint=http://localhost:7480
com.openexchange.filestore.s3.radosgw.bucketName=oxbucket
com.openexchange.filestore.s3.radosgw.region=eu-west-1
com.openexchange.filestore.s3.radosgw.pathStyleAccess=true
com.openexchange.filestore.s3.radosgw.accessKey=...
com.openexchange.filestore.s3.radosgw.secretKey=...
com.openexchange.filestore.s3.radosgw.encryption=none
com.openexchange.filestore.s3.radosgw.signerOverride=S3SignerType
com.openexchange.filestore.s3.radosgw.chunkSize=5MB

Changing this file needs another

service open-xchange restart
</div>

And the database:

/opt/open-xchange/sbin/registerdatabase -A oxadminmaster -P "$(cat /root/.oxpw)" -n oxdb -p "$(cat /root/.dbpw)" -m true

<div style="padding: 20px; border-style: solid; border-width: thin;">
Cluster Note

You probably have multiple clusters with read and write URLs. Register them each like first registering the master, subsequently registering the slave URL with the corresponding master ID:

/opt/open-xchange/sbin/registerdatabase -A oxadminmaster -P "$(cat /root/.oxpw)" -n oxdb -p "$(cat /root/.dbpw)" -m true

This command gives as output the id of the registered db master, like

database 3 registered

Here, the id is 3. Use this id as argument of the -M switch of the next command:

/opt/open-xchange/sbin/registerdatabase -A oxadminmaster -P "$(cat /root/.oxpw)" -n oxdbr -p "$(cat /root/.dbpw)" -m false -M 3
</div>

=== Configure Apache ===

Create config files /etc/apache2/conf-enabled/proxy_http.conf, /etc/apache2/sites-enabled/000-default.conf by copy-pasting as explained in [[AppSuite:Open-Xchange_Installation_Guide_for_Debian_8.0#Configure_services]]

Make sure you are using mpm_event. Apply concurrent connections tuning as described in [[Tune_apache2_for_more_concurrent_connections]].

Configure modules and restart:

a2enmod proxy proxy_http proxy_balancer expires deflate headers rewrite mime setenvif lbmethod_byrequests
systemctl restart apache2

<div style="padding: 20px; border-style: solid; border-width: thin;">
Cluster Note

Make sure that each middleware node got its unique server.properties:com.openexchange.server.backendRoute, e.g. APP1, APP2, APP3, etc.

Configure them in the http proxy definitions like

BalancerMember http://ox1:8009 timeout=100 smax=0 ttl=60 retry=60 loadfactor=50 route=APP1
BalancerMember http://ox2:8009 timeout=100 smax=0 ttl=60 retry=60 loadfactor=50 route=APP2
BalancerMember http://ox3:8009 timeout=100 smax=0 ttl=60 retry=60 loadfactor=50 route=APP3

If you colocate apache on middleware nodes, you might want to minimize cross-node routing, by setting on each node for the local node a loadfactor=50 and for the other nodes a status=+H instead of the loadfactor. E.g. on node ox1:

BalancerMember http://ox1:8009 timeout=100 smax=0 ttl=60 retry=60 loadfactor=50 route=APP1
BalancerMember http://ox2:8009 timeout=100 smax=0 ttl=60 retry=60 status=+H route=APP2
BalancerMember http://ox3:8009 timeout=100 smax=0 ttl=60 retry=60 status=+H route=APP3

However this requires host-specific config files for apache http proxy.

Use touch-appsuite with one identical timestamp on each frontend node.

timestamp=$(date -u +%Y%m%d.%H%M%S)
for node in $all_frontend_nodes; do
ssh $node /opt/open-xchange/sbin/touch-appsuite --timestamp=$timestamp
done

</div>

=== Provision a Test User ===

Provision a sample context and user:

/opt/open-xchange/sbin/createcontext -c 1 -A oxadminmaster -P $(cat /root/.oxmasterpw) -N localdomain -u oxadmin -d "Admin User" -g Admin -s User -p $(cat /root/.oxadminpw) -e oxadmin@localdomain -q 100 --access-combination-name groupware_premium
/opt/open-xchange/sbin/createuser -c 1 -A oxadmin -P $(cat /root/.oxadminpw) -u testuser -d "Test User" -g Test -s User -p $(cat /root/.oxuserpw) -e testuser@localdomain --access-combination-name groupware_premium

[[Context_Preprovisioning#Sample_Script]] provides an example how to fast-mode provision a huge number of contexts quickly.

User:Dominik.epple

2018-06-27T12:29:21Z

Dominik.epple: /* Provision a Test User */

== Install Guide ==

=== About this document ===

The aim of this document is to improve on the existing quickinstall guides to be more structured, provide a more extensive view on "single node and beyond" topics, follow closer to existing "best practices" (also, but not only security-wise), and point out what needs to be changed in clustered installations.

Most of the commands given in this document thus assume a high level design of "single-node, all-in-one".

This document was created on Debian Stretch (which, as of time of writing, is not even supported yet), but it should work as-is also for jessie. Porting to RHEL/SLES/... is TODO.

=== Preparations ===

==== System update ====

You want to start on latest patchlevel of your OS:

apt-get update
apt-get dist-upgrade
apt-get install less vim pwgen apt-transport-https
# or
yum update
yum install https://dl.fedoraproject.org/pub/epel/epel-release-latest-7.noarch.rpm
yum install vim less pwgen wget

reboot

==== Pregenerate passwords ====

This guide shall feature copy-paste ready commands which create installations with no default passwords.

We will pre-generate some passwords which will live as dotfiles in /root.

pwgen -c -n 16 1 > /root/.oxpw
pwgen -c -n 16 1 > /root/.dbpw
pwgen -c -n 16 1 > /root/.dbrootpw

==== Prepare database ====

In real-world installations this will probably be multiple galera clusters of a supported flavor and version. For educational purposes a standalone DB on our single-node machine is sufficient.

Even for single-node, don't forget to apply database tuning. See [[My.cnf|our oxpedia article]] for default tunings. Note that typically you need to re-initialize the MySQL datadir after changing InnoDB sizing values, and subsequently start the service:

mysql_install_db
service mysql restart

We aim to create secure-by-default documentation, so here we go: Run mysql_secure_installation, and chose every security relevant option, but let the root password empty in this step, as we set it in the next step:

# leave the root password empty in mysql_secure_installation as we set it in the subsequent step
mysql_secure_installation
# now, configure the password from /root/.dbrootpw
mysql -e "UPDATE mysql.user SET Password=PASSWORD('$(cat /root/.dbrootpw)') WHERE User='root'; FLUSH PRIVILEGES;"
cat >/root/.my.cnf <<EOF
[client]
user=root
password=$(cat /root/.dbrootpw)
EOF

MySQL 5.7: the aforementioned must be adjusted using

ALTER USER USER() IDENTIFIED BY 'tiez7EiNgaish0ee';

These credentials also needs to be put in /etc/mysql/debian.cnf.

<div style="padding: 20px; border-style: solid; border-width: thin;">
Cluster note

On multiple db clusters, do it per node analogously. Just be aware the copy-paste command above expects the /root/.dbrootpw file.
</div>

==== Prepare OX user ====

While the packages will create the user automatically if it does not exist, we want to prepare the filestore now, and we need the user therefore.

useradd -r open-xchange

<div style="padding: 20px; border-style: solid; border-width: thin;">
Cluster Note

You should hard-wire the userid and groupid to the same fixed value. Otherwise, if you want to use a NFS filestore, you'll run into permissions problems, unless you use nfs4/kerberos/idmapd.

groupadd -r -g 999 open-xchange
useradd -r -g 999 -u 999 open-xchange
</div>

==== Prepare filestore ====

There are several options here.

===== Single-Node: local directory =====

For a single-node installation, you can just prepare a local directory:

mkdir /var/opt/filestore
chown open-xchange:open-xchange /var/opt/filestore

===== NFS =====

If using NFS:

Setup on the NFS server:

apt-get install nfs-kernel-server
service nfs-kernel-server restart

Configure /etc/exports. This is for traditional ip based access control; krb5 or other security configuration is out of scope of this document.

mkdir /var/opt/filestore
chown open-xchange:open-xchange /var/opt/filestore
echo "/var/opt/filestore 192.168.1.0/24(rw,sync,fsid=0,no_subtree_check)" >> /etc/exports
exportfs -a

Clients can then mount using

mkdir /var/opt/filestore
mount -t nfs -o vers=4 nfs-server:/filestore /var/opt/filestore

Or using fstab entries like

nfs-server:/filestore /var/opt/filestore nfs4 defaults 0 0

===== Object Store =====

You can use an object store. For lab environments Ceph is a convenient option. For demo / educational purpuses a "single node Ceph cluster" even co-located on your "single-node machine" is reasonble, but its setup is out of scope of this document. If you want to use this, be prepared to provide information about endpoint, bucket name, access key, secret key.

===== No filestore =====

If you dont want to provide a filestore, you can configure OX later to run without filestore. (Q: do we still need a dummy registerfilestore on a local directory in that event?)

==== Prepare mail system ====

Formally out of scope of this document.

If you need to create a testing dovecot/postfix setup, you can use our [https://gitlab.open-xchange.com/qa/performance/tree/develop/com.openexchange.test.performance.gatling/src/site-preprocess/dovecot performance testing sample config].

=== Install OX software ===

You need an ldb user and password for updates and proprietary repos. If you dont have such a user, you can still install the free components. You'll get a lot of authentication failed warnings however from apt tools unless you deconfigure the closed repos.

wget http://software.open-xchange.com/oxbuildkey.pub -O - | apt-key add -
wget -O/etc/apt/sources.list.d/ox.list http://software.open-xchange.com/products/DebianJessie.list
ldbuser=...
ldbpassword=...
sed -i -e "s/LDBUSER:LDBPASSWORD/$ldbuser:$ldbpassword/" /etc/apt/sources.list.d/ox.list
apt-get update
apt-get install open-xchange open-xchange-authentication-database open-xchange-grizzly open-xchange-admin open-xchange-appsuite-backend open-xchange-appsuite-manifest open-xchange-appsuite
#
# or
#
wget -O /etc/yum.repos.d/ox.repo http://software.open-xchange.com/products/RHEL7.repo
ldbuser=...
ldbpassword=...
sed -i -e "s/LDBUSER:LDBPASSWORD/$ldbuser:$ldbpassword/" /etc/yum.repos.d/ox.repo
yum install open-xchange open-xchange-authentication-database open-xchange-grizzly open-xchange-admin open-xchange-appsuite-backend open-xchange-appsuite-manifest open-xchange-appsuite

<div style="padding: 20px; border-style: solid; border-width: thin;">
Cluster note

*if you want to have separate frontend (apache) and middleware (open-xchange) systems, make sure to install packages which require apache as dependency on the frontend nodes, and packages which require java as a dependency on the middleware nodes. Currently this results in the split
** Frontend nodes: open-xchange-appsuite
** Middleware nodes: everything else
* If you want to use an object store, install the corresponding open-xchange-filestore-xyz package, like open-xchange-filestore-s3
* For hazelcast session storage, install also open-xchange-sessionstorage-hazelcast
</div>

=== Install database schemas ===

If the DB runs on localhost and you have root access, you can use

/opt/open-xchange/sbin/initconfigdb --configdb-pass="$(cat /root/.dbpw)" -a

<div style="padding: 20px; border-style: solid; border-width: thin;">

Cluster note

Create the DB users on all write instances manually. https://oxpedia.org/wiki/index.php?title=Template:OXLoadBalancingClustering_Database#Creating_Open-Xchange_user

mysql -e "GRANT CREATE, LOCK TABLES, REFERENCES, INDEX, DROP, DELETE, ALTER, SELECT, UPDATE, INSERT, CREATE TEMPORARY TABLES, SHOW VIEW, SHOW DATABASES ON *.* to 'openexchange'@'%' identified by '$(cat /root/.dbpw)' WITH GRANT OPTION;"

Run initconfigdb with some more options:

/opt/open-xchange/sbin/initconfigdb --configdb-user=openexchange --configdb-pass="$(cat /root/.dbpw)" --configdb-host=configdb-writehost

(initconfigdb needs to be run only once on one cluster node)
</div>

=== Initial configuration ===

/opt/open-xchange/sbin/oxinstaller --add-license=YOUR-OX-LICENSE-CODE --servername=oxserver --configdb-pass="$(cat /root/.dbpw)" --master-pass="$(cat /root/.oxpw)" --network-listener-host=localhost --servermemory 1024

'''servername''' is more like a ''clustername'' and needs to be the same for all nodes.

'''servermemory''' should be adjusted to reflect the expected number of concurrent active sessions; sizing assumption is 4MB per session.

<div style="padding: 20px; border-style: solid; border-width: thin;">
Cluster Note

--configdb-readhost=...
--configdb-writehost=...
--imapserver=...
--smtpserver=...
--mail-login-src=<login|mail|name>
--mail-server-src=<user|global>
--transport-server-src=<user|global>
--jkroute=APP1
--object-link-hostname=[service DNS name like ox.example.com]
--extras-link=[https://ox.example.com/]
--name-of-oxcluster=[something unique per cluster, like business-staging; see --servername]
--network-listener-host=<localhost|*>

oxinstaller needs to be run on each cluster node with identical options besides the jkroute, which must be unique per cluster node and match the corresponding apache option, see below.

In a cluster you also want to configure hazelcast; see [[AppSuite:Running_a_cluster#Configuration]]. Most prominent options are

com.openexchange.hazelcast.enabled=true
com.openexchange.hazelcast.group.name=<reasonable unique group name>
com.openexchange.hazelcast.group.password=<unique password, CHANGE THE SHIPPED DEFAULT PASSWORD!>
com.openexchange.hazelcast.network.join=static # static is recommended over multicast for robustness
com.openexchange.hazelcast.network.join.static.nodes=... # configure your nodes as a comma-separated list; a bootstrapping subset is acceptable
com.openexchange.hazelcast.network.interfaces=... # pick your subnet; Wildcards (*) and ranges (-) can be used
</div>

Start the service:

systemctl restart open-xchange

<div style="padding: 20px; border-style: solid; border-width: thin;">
Cluster Note

Start the service on every cluster node.
</div>

=== Registering stuff ===

Register the "server":

/opt/open-xchange/sbin/registerserver -n oxserver -A oxadminmaster -P "$(cat /root/.oxpw)"

<div style="padding: 20px; border-style: solid; border-width: thin;">
Cluster Note

All the register* commands need to be issued only once per cluster, as the effect is to enter corresponding lines in the configdb.
</div>

And the filestore:

/opt/open-xchange/sbin/registerfilestore -A oxadminmaster -P "$(cat /root/.oxpw)" -t file:/var/opt/filestore -s 1000000 -x 1000000

<div style="padding: 20px; border-style: solid; border-width: thin;">
Cluster Note

If you chose an object store, the corresponding registerfilestore line reads as follows (Ceph radosgw example):

/opt/open-xchange/sbin/registerfilestore -A oxadminmaster -P "$(cat /root/.oxpw)" -t s3://radosgw -s 1000000 -x 1000000

It requires configuration of the object store in filestore-s3.properties:

com.openexchange.filestore.s3.radosgw.endpoint=http://localhost:7480
com.openexchange.filestore.s3.radosgw.bucketName=oxbucket
com.openexchange.filestore.s3.radosgw.region=eu-west-1
com.openexchange.filestore.s3.radosgw.pathStyleAccess=true
com.openexchange.filestore.s3.radosgw.accessKey=...
com.openexchange.filestore.s3.radosgw.secretKey=...
com.openexchange.filestore.s3.radosgw.encryption=none
com.openexchange.filestore.s3.radosgw.signerOverride=S3SignerType
com.openexchange.filestore.s3.radosgw.chunkSize=5MB

Changing this file needs another

service open-xchange restart
</div>

And the database:

/opt/open-xchange/sbin/registerdatabase -A oxadminmaster -P "$(cat /root/.oxpw)" -n oxdb -p "$(cat /root/.dbpw)" -m true

<div style="padding: 20px; border-style: solid; border-width: thin;">
Cluster Note

You probably have multiple clusters with read and write URLs. Register them each like first registering the master, subsequently registering the slave URL with the corresponding master ID:

/opt/open-xchange/sbin/registerdatabase -A oxadminmaster -P "$(cat /root/.oxpw)" -n oxdb -p "$(cat /root/.dbpw)" -m true

This command gives as output the id of the registered db master, like

database 3 registered

Here, the id is 3. Use this id as argument of the -M switch of the next command:

/opt/open-xchange/sbin/registerdatabase -A oxadminmaster -P "$(cat /root/.oxpw)" -n oxdbr -p "$(cat /root/.dbpw)" -m false -M 3
</div>

=== Configure Apache ===

Create config files /etc/apache2/conf-enabled/proxy_http.conf, /etc/apache2/sites-enabled/000-default.conf by copy-pasting as explained in [[AppSuite:Open-Xchange_Installation_Guide_for_Debian_8.0#Configure_services]]

Make sure you are using mpm_event. Apply concurrent connections tuning as described in [[Tune_apache2_for_more_concurrent_connections]].

Configure modules and restart:

a2enmod proxy proxy_http proxy_balancer expires deflate headers rewrite mime setenvif lbmethod_byrequests
systemctl restart apache2

<div style="padding: 20px; border-style: solid; border-width: thin;">
Cluster Note

Make sure that each middleware node got its unique server.properties:com.openexchange.server.backendRoute, e.g. APP1, APP2, APP3, etc.

Configure them in the http proxy definitions like

BalancerMember http://ox1:8009 timeout=100 smax=0 ttl=60 retry=60 loadfactor=50 route=APP1
BalancerMember http://ox2:8009 timeout=100 smax=0 ttl=60 retry=60 loadfactor=50 route=APP2
BalancerMember http://ox3:8009 timeout=100 smax=0 ttl=60 retry=60 loadfactor=50 route=APP3

If you colocate apache on middleware nodes, you might want to minimize cross-node routing, by setting on each node for the local node a loadfactor=50 and for the other nodes a status=+H instead of the loadfactor. E.g. on node ox1:

BalancerMember http://ox1:8009 timeout=100 smax=0 ttl=60 retry=60 loadfactor=50 route=APP1
BalancerMember http://ox2:8009 timeout=100 smax=0 ttl=60 retry=60 status=+H route=APP2
BalancerMember http://ox3:8009 timeout=100 smax=0 ttl=60 retry=60 status=+H route=APP3

However this requires host-specific config files for apache http proxy.

Use touch-appsuite with one identical timestamp on each frontend node.

timestamp=$(date -u +%Y%m%d.%H%M%S)
for node in $all_frontend_nodes; do
ssh $node /opt/open-xchange/sbin/touch-appsuite --timestamp=$timestamp
done

</div>

=== Provision a Test User ===

Provision a sample context and user:

/opt/open-xchange/sbin/createcontext -c 1 -A oxadminmaster -P $(cat /root/.oxmasterpw) -N localdomain -u oxadmin -d "Admin User" -g Admin -s User -p $(cat /root/.oxadminpw) -e oxadmin@localdomain -q 100 --access-combination-name groupware_premium
/opt/open-xchange/sbin/createuser -c 1 -A oxadmin -P $(cat /root/.oxadminpw) -u testuser -d "Test User" -g Test -s User -p $(cat /root/.oxuserpw) -e testuser@localdomain --access-combination-name groupware_premium

[[Context_Preprovisioning#Sample_Script]] provides an example how to fast-mode provision a huge number of contexts quickly.

User:Dominik.epple

2018-06-15T14:09:10Z

Dominik.epple: /* Install OX software */

== Install Guide ==

=== About this document ===

The aim of this document is to improve on the existing quickinstall guides to be more structured, provide a more extensive view on "single node and beyond" topics, follow closer to existing "best practices" (also, but not only security-wise), and point out what needs to be changed in clustered installations.

Most of the commands given in this document thus assume a high level design of "single-node, all-in-one".

This document was created on Debian Stretch (which, as of time of writing, is not even supported yet), but it should work as-is also for jessie. Porting to RHEL/SLES/... is TODO.

=== Preparations ===

==== System update ====

You want to start on latest patchlevel of your OS:

apt-get update
apt-get dist-upgrade
apt-get install less vim pwgen apt-transport-https
# or
yum update
yum install https://dl.fedoraproject.org/pub/epel/epel-release-latest-7.noarch.rpm
yum install vim less pwgen wget

reboot

==== Pregenerate passwords ====

This guide shall feature copy-paste ready commands which create installations with no default passwords.

We will pre-generate some passwords which will live as dotfiles in /root.

pwgen -c -n 16 1 > /root/.oxpw
pwgen -c -n 16 1 > /root/.dbpw
pwgen -c -n 16 1 > /root/.dbrootpw

==== Prepare database ====

In real-world installations this will probably be multiple galera clusters of a supported flavor and version. For educational purposes a standalone DB on our single-node machine is sufficient.

Even for single-node, don't forget to apply database tuning. See [[My.cnf|our oxpedia article]] for default tunings. Note that typically you need to re-initialize the MySQL datadir after changing InnoDB sizing values, and subsequently start the service:

mysql_install_db
service mysql restart

We aim to create secure-by-default documentation, so here we go: Run mysql_secure_installation, and chose every security relevant option, but let the root password empty in this step, as we set it in the next step:

# leave the root password empty in mysql_secure_installation as we set it in the subsequent step
mysql_secure_installation
# now, configure the password from /root/.dbrootpw
mysql -e "UPDATE mysql.user SET Password=PASSWORD('$(cat /root/.dbrootpw)') WHERE User='root'; FLUSH PRIVILEGES;"
cat >/root/.my.cnf <<EOF
[client]
user=root
password=$(cat /root/.dbrootpw)
EOF

MySQL 5.7: the aforementioned must be adjusted using

ALTER USER USER() IDENTIFIED BY 'tiez7EiNgaish0ee';

These credentials also needs to be put in /etc/mysql/debian.cnf.

<div style="padding: 20px; border-style: solid; border-width: thin;">
Cluster note

On multiple db clusters, do it per node analogously. Just be aware the copy-paste command above expects the /root/.dbrootpw file.
</div>

==== Prepare OX user ====

While the packages will create the user automatically if it does not exist, we want to prepare the filestore now, and we need the user therefore.

useradd -r open-xchange

<div style="padding: 20px; border-style: solid; border-width: thin;">
Cluster Note

You should hard-wire the userid and groupid to the same fixed value. Otherwise, if you want to use a NFS filestore, you'll run into permissions problems, unless you use nfs4/kerberos/idmapd.

groupadd -r -g 999 open-xchange
useradd -r -g 999 -u 999 open-xchange
</div>

==== Prepare filestore ====

There are several options here.

===== Single-Node: local directory =====

For a single-node installation, you can just prepare a local directory:

mkdir /var/opt/filestore
chown open-xchange:open-xchange /var/opt/filestore

===== NFS =====

If using NFS:

Setup on the NFS server:

apt-get install nfs-kernel-server
service nfs-kernel-server restart

Configure /etc/exports. This is for traditional ip based access control; krb5 or other security configuration is out of scope of this document.

mkdir /var/opt/filestore
chown open-xchange:open-xchange /var/opt/filestore
echo "/var/opt/filestore 192.168.1.0/24(rw,sync,fsid=0,no_subtree_check)" >> /etc/exports
exportfs -a

Clients can then mount using

mkdir /var/opt/filestore
mount -t nfs -o vers=4 nfs-server:/filestore /var/opt/filestore

Or using fstab entries like

nfs-server:/filestore /var/opt/filestore nfs4 defaults 0 0

===== Object Store =====

You can use an object store. For lab environments Ceph is a convenient option. For demo / educational purpuses a "single node Ceph cluster" even co-located on your "single-node machine" is reasonble, but its setup is out of scope of this document. If you want to use this, be prepared to provide information about endpoint, bucket name, access key, secret key.

===== No filestore =====

If you dont want to provide a filestore, you can configure OX later to run without filestore. (Q: do we still need a dummy registerfilestore on a local directory in that event?)

==== Prepare mail system ====

Formally out of scope of this document.

If you need to create a testing dovecot/postfix setup, you can use our [https://gitlab.open-xchange.com/qa/performance/tree/develop/com.openexchange.test.performance.gatling/src/site-preprocess/dovecot performance testing sample config].

=== Install OX software ===

You need an ldb user and password for updates and proprietary repos. If you dont have such a user, you can still install the free components. You'll get a lot of authentication failed warnings however from apt tools unless you deconfigure the closed repos.

wget http://software.open-xchange.com/oxbuildkey.pub -O - | apt-key add -
wget -O/etc/apt/sources.list.d/ox.list http://software.open-xchange.com/products/DebianJessie.list
ldbuser=...
ldbpassword=...
sed -i -e "s/LDBUSER:LDBPASSWORD/$ldbuser:$ldbpassword/" /etc/apt/sources.list.d/ox.list
apt-get update
apt-get install open-xchange open-xchange-authentication-database open-xchange-grizzly open-xchange-admin open-xchange-appsuite-backend open-xchange-appsuite-manifest open-xchange-appsuite
#
# or
#
wget -O /etc/yum.repos.d/ox.repo http://software.open-xchange.com/products/RHEL7.repo
ldbuser=...
ldbpassword=...
sed -i -e "s/LDBUSER:LDBPASSWORD/$ldbuser:$ldbpassword/" /etc/yum.repos.d/ox.repo
yum install open-xchange open-xchange-authentication-database open-xchange-grizzly open-xchange-admin open-xchange-appsuite-backend open-xchange-appsuite-manifest open-xchange-appsuite

<div style="padding: 20px; border-style: solid; border-width: thin;">
Cluster note

*if you want to have separate frontend (apache) and middleware (open-xchange) systems, make sure to install packages which require apache as dependency on the frontend nodes, and packages which require java as a dependency on the middleware nodes. Currently this results in the split
** Frontend nodes: open-xchange-appsuite
** Middleware nodes: everything else
* If you want to use an object store, install the corresponding open-xchange-filestore-xyz package, like open-xchange-filestore-s3
* For hazelcast session storage, install also open-xchange-sessionstorage-hazelcast
</div>

=== Install database schemas ===

If the DB runs on localhost and you have root access, you can use

/opt/open-xchange/sbin/initconfigdb --configdb-pass="$(cat /root/.dbpw)" -a

<div style="padding: 20px; border-style: solid; border-width: thin;">

Cluster note

Create the DB users on all write instances manually. https://oxpedia.org/wiki/index.php?title=Template:OXLoadBalancingClustering_Database#Creating_Open-Xchange_user

mysql -e "GRANT CREATE, LOCK TABLES, REFERENCES, INDEX, DROP, DELETE, ALTER, SELECT, UPDATE, INSERT, CREATE TEMPORARY TABLES, SHOW VIEW, SHOW DATABASES ON *.* to 'openexchange'@'%' identified by '$(cat /root/.dbpw)' WITH GRANT OPTION;"

Run initconfigdb with some more options:

/opt/open-xchange/sbin/initconfigdb --configdb-user=openexchange --configdb-pass="$(cat /root/.dbpw)" --configdb-host=configdb-writehost

(initconfigdb needs to be run only once on one cluster node)
</div>

=== Initial configuration ===

/opt/open-xchange/sbin/oxinstaller --add-license=YOUR-OX-LICENSE-CODE --servername=oxserver --configdb-pass="$(cat /root/.dbpw)" --master-pass="$(cat /root/.oxpw)" --network-listener-host=localhost --servermemory 1024

'''servername''' is more like a ''clustername'' and needs to be the same for all nodes.

'''servermemory''' should be adjusted to reflect the expected number of concurrent active sessions; sizing assumption is 4MB per session.

<div style="padding: 20px; border-style: solid; border-width: thin;">
Cluster Note

--configdb-readhost=...
--configdb-writehost=...
--imapserver=...
--smtpserver=...
--mail-login-src=<login|mail|name>
--mail-server-src=<user|global>
--transport-server-src=<user|global>
--jkroute=APP1
--object-link-hostname=[service DNS name like ox.example.com]
--extras-link=[https://ox.example.com/]
--name-of-oxcluster=[something unique per cluster, like business-staging; see --servername]
--network-listener-host=<localhost|*>

oxinstaller needs to be run on each cluster node with identical options besides the jkroute, which must be unique per cluster node and match the corresponding apache option, see below.

In a cluster you also want to configure hazelcast; see [[AppSuite:Running_a_cluster#Configuration]]. Most prominent options are

com.openexchange.hazelcast.enabled=true
com.openexchange.hazelcast.group.name=<reasonable unique group name>
com.openexchange.hazelcast.group.password=<unique password, CHANGE THE SHIPPED DEFAULT PASSWORD!>
com.openexchange.hazelcast.network.join=static # static is recommended over multicast for robustness
com.openexchange.hazelcast.network.join.static.nodes=... # configure your nodes as a comma-separated list; a bootstrapping subset is acceptable
com.openexchange.hazelcast.network.interfaces=... # pick your subnet; Wildcards (*) and ranges (-) can be used
</div>

Start the service:

systemctl restart open-xchange

<div style="padding: 20px; border-style: solid; border-width: thin;">
Cluster Note

Start the service on every cluster node.
</div>

=== Registering stuff ===

Register the "server":

/opt/open-xchange/sbin/registerserver -n oxserver -A oxadminmaster -P "$(cat /root/.oxpw)"

<div style="padding: 20px; border-style: solid; border-width: thin;">
Cluster Note

All the register* commands need to be issued only once per cluster, as the effect is to enter corresponding lines in the configdb.
</div>

And the filestore:

/opt/open-xchange/sbin/registerfilestore -A oxadminmaster -P "$(cat /root/.oxpw)" -t file:/var/opt/filestore -s 1000000 -x 1000000

<div style="padding: 20px; border-style: solid; border-width: thin;">
Cluster Note

If you chose an object store, the corresponding registerfilestore line reads as follows (Ceph radosgw example):

/opt/open-xchange/sbin/registerfilestore -A oxadminmaster -P "$(cat /root/.oxpw)" -t s3://radosgw -s 1000000 -x 1000000

It requires configuration of the object store in filestore-s3.properties:

com.openexchange.filestore.s3.radosgw.endpoint=http://localhost:7480
com.openexchange.filestore.s3.radosgw.bucketName=oxbucket
com.openexchange.filestore.s3.radosgw.region=eu-west-1
com.openexchange.filestore.s3.radosgw.pathStyleAccess=true
com.openexchange.filestore.s3.radosgw.accessKey=...
com.openexchange.filestore.s3.radosgw.secretKey=...
com.openexchange.filestore.s3.radosgw.encryption=none
com.openexchange.filestore.s3.radosgw.signerOverride=S3SignerType
com.openexchange.filestore.s3.radosgw.chunkSize=5MB

Changing this file needs another

service open-xchange restart
</div>

And the database:

/opt/open-xchange/sbin/registerdatabase -A oxadminmaster -P "$(cat /root/.oxpw)" -n oxdb -p "$(cat /root/.dbpw)" -m true

<div style="padding: 20px; border-style: solid; border-width: thin;">
Cluster Note

You probably have multiple clusters with read and write URLs. Register them each like first registering the master, subsequently registering the slave URL with the corresponding master ID:

/opt/open-xchange/sbin/registerdatabase -A oxadminmaster -P "$(cat /root/.oxpw)" -n oxdb -p "$(cat /root/.dbpw)" -m true

This command gives as output the id of the registered db master, like

database 3 registered

Here, the id is 3. Use this id as argument of the -M switch of the next command:

/opt/open-xchange/sbin/registerdatabase -A oxadminmaster -P "$(cat /root/.oxpw)" -n oxdbr -p "$(cat /root/.dbpw)" -m false -M 3
</div>

=== Configure Apache ===

Create config files /etc/apache2/conf-enabled/proxy_http.conf, /etc/apache2/sites-enabled/000-default.conf by copy-pasting as explained in [[AppSuite:Open-Xchange_Installation_Guide_for_Debian_8.0#Configure_services]]

Make sure you are using mpm_event. Apply concurrent connections tuning as described in [[Tune_apache2_for_more_concurrent_connections]].

Configure modules and restart:

a2enmod proxy proxy_http proxy_balancer expires deflate headers rewrite mime setenvif lbmethod_byrequests
systemctl restart apache2

<div style="padding: 20px; border-style: solid; border-width: thin;">
Cluster Note

Make sure that each middleware node got its unique server.properties:com.openexchange.server.backendRoute, e.g. APP1, APP2, APP3, etc.

Configure them in the http proxy definitions like

BalancerMember http://ox1:8009 timeout=100 smax=0 ttl=60 retry=60 loadfactor=50 route=APP1
BalancerMember http://ox2:8009 timeout=100 smax=0 ttl=60 retry=60 loadfactor=50 route=APP2
BalancerMember http://ox3:8009 timeout=100 smax=0 ttl=60 retry=60 loadfactor=50 route=APP3

If you colocate apache on middleware nodes, you might want to minimize cross-node routing, by setting on each node for the local node a loadfactor=50 and for the other nodes a status=+H instead of the loadfactor. E.g. on node ox1:

BalancerMember http://ox1:8009 timeout=100 smax=0 ttl=60 retry=60 loadfactor=50 route=APP1
BalancerMember http://ox2:8009 timeout=100 smax=0 ttl=60 retry=60 status=+H route=APP2
BalancerMember http://ox3:8009 timeout=100 smax=0 ttl=60 retry=60 status=+H route=APP3

However this requires host-specific config files for apache http proxy.

Use touch-appsuite with one identical timestamp on each frontend node.

timestamp=$(date -u +%Y%m%d.%H%M%S)
for node in $all_frontend_nodes; do
ssh $node /opt/open-xchange/sbin/touch-appsuite --timestamp=$timestamp
done

</div>

=== Provision a Test User ===

Provision a sample context and user:

/opt/open-xchange/sbin/createcontext -c 1 -A oxadminmaster -P secret -N localdomain -u oxadmin -d "Admin User" -g Admin -s User -p secret -e oxadmin@localdomain -q 100 --access-combination-name groupware_premium
/opt/open-xchange/sbin/createuser -c 1 -A oxadmin -P secret -u testuser -d "Test User" -g Test -s User -p secret -e testuser@localdomain --access-combination-name groupware_premium

[[Context_Preprovisioning#Sample_Script]] provides an example how to fast-mode provision a huge number of contexts quickly.

User:Dominik.epple

2018-06-15T14:03:02Z

Dominik.epple: /* System update */

== Install Guide ==

=== About this document ===

The aim of this document is to improve on the existing quickinstall guides to be more structured, provide a more extensive view on "single node and beyond" topics, follow closer to existing "best practices" (also, but not only security-wise), and point out what needs to be changed in clustered installations.

Most of the commands given in this document thus assume a high level design of "single-node, all-in-one".

This document was created on Debian Stretch (which, as of time of writing, is not even supported yet), but it should work as-is also for jessie. Porting to RHEL/SLES/... is TODO.

=== Preparations ===

==== System update ====

You want to start on latest patchlevel of your OS:

apt-get update
apt-get dist-upgrade
apt-get install less vim pwgen apt-transport-https
# or
yum update
yum install https://dl.fedoraproject.org/pub/epel/epel-release-latest-7.noarch.rpm
yum install vim less pwgen wget

reboot

==== Pregenerate passwords ====

This guide shall feature copy-paste ready commands which create installations with no default passwords.

We will pre-generate some passwords which will live as dotfiles in /root.

pwgen -c -n 16 1 > /root/.oxpw
pwgen -c -n 16 1 > /root/.dbpw
pwgen -c -n 16 1 > /root/.dbrootpw

==== Prepare database ====

In real-world installations this will probably be multiple galera clusters of a supported flavor and version. For educational purposes a standalone DB on our single-node machine is sufficient.

Even for single-node, don't forget to apply database tuning. See [[My.cnf|our oxpedia article]] for default tunings. Note that typically you need to re-initialize the MySQL datadir after changing InnoDB sizing values, and subsequently start the service:

mysql_install_db
service mysql restart

We aim to create secure-by-default documentation, so here we go: Run mysql_secure_installation, and chose every security relevant option, but let the root password empty in this step, as we set it in the next step:

# leave the root password empty in mysql_secure_installation as we set it in the subsequent step
mysql_secure_installation
# now, configure the password from /root/.dbrootpw
mysql -e "UPDATE mysql.user SET Password=PASSWORD('$(cat /root/.dbrootpw)') WHERE User='root'; FLUSH PRIVILEGES;"
cat >/root/.my.cnf <<EOF
[client]
user=root
password=$(cat /root/.dbrootpw)
EOF

MySQL 5.7: the aforementioned must be adjusted using

ALTER USER USER() IDENTIFIED BY 'tiez7EiNgaish0ee';

These credentials also needs to be put in /etc/mysql/debian.cnf.

<div style="padding: 20px; border-style: solid; border-width: thin;">
Cluster note

On multiple db clusters, do it per node analogously. Just be aware the copy-paste command above expects the /root/.dbrootpw file.
</div>

==== Prepare OX user ====

While the packages will create the user automatically if it does not exist, we want to prepare the filestore now, and we need the user therefore.

useradd -r open-xchange

<div style="padding: 20px; border-style: solid; border-width: thin;">
Cluster Note

You should hard-wire the userid and groupid to the same fixed value. Otherwise, if you want to use a NFS filestore, you'll run into permissions problems, unless you use nfs4/kerberos/idmapd.

groupadd -r -g 999 open-xchange
useradd -r -g 999 -u 999 open-xchange
</div>

==== Prepare filestore ====

There are several options here.

===== Single-Node: local directory =====

For a single-node installation, you can just prepare a local directory:

mkdir /var/opt/filestore
chown open-xchange:open-xchange /var/opt/filestore

===== NFS =====

If using NFS:

Setup on the NFS server:

apt-get install nfs-kernel-server
service nfs-kernel-server restart

Configure /etc/exports. This is for traditional ip based access control; krb5 or other security configuration is out of scope of this document.

mkdir /var/opt/filestore
chown open-xchange:open-xchange /var/opt/filestore
echo "/var/opt/filestore 192.168.1.0/24(rw,sync,fsid=0,no_subtree_check)" >> /etc/exports
exportfs -a

Clients can then mount using

mkdir /var/opt/filestore
mount -t nfs -o vers=4 nfs-server:/filestore /var/opt/filestore

Or using fstab entries like

nfs-server:/filestore /var/opt/filestore nfs4 defaults 0 0

===== Object Store =====

You can use an object store. For lab environments Ceph is a convenient option. For demo / educational purpuses a "single node Ceph cluster" even co-located on your "single-node machine" is reasonble, but its setup is out of scope of this document. If you want to use this, be prepared to provide information about endpoint, bucket name, access key, secret key.

===== No filestore =====

If you dont want to provide a filestore, you can configure OX later to run without filestore. (Q: do we still need a dummy registerfilestore on a local directory in that event?)

==== Prepare mail system ====

Formally out of scope of this document.

If you need to create a testing dovecot/postfix setup, you can use our [https://gitlab.open-xchange.com/qa/performance/tree/develop/com.openexchange.test.performance.gatling/src/site-preprocess/dovecot performance testing sample config].

=== Install OX software ===

You need an ldb user and password for updates and proprietary repos. If you dont have such a user, you can still install the free components. You'll get a lot of authentication failed warnings however from apt tools unless you deconfigure the closed repos.

wget http://software.open-xchange.com/oxbuildkey.pub -O - | apt-key add -
wget -O/etc/apt/sources.list.d/ox.list http://software.open-xchange.com/products/DebianJessie.list
ldbuser=...
ldbpassword=...
sed -i -e "s/LDBUSER:LDBPASSWORD/$ldbuser:$ldbpassword/" /etc/apt/sources.list.d/ox.list
apt-get update
apt-get install open-xchange open-xchange-authentication-database open-xchange-grizzly open-xchange-admin open-xchange-appsuite-backend open-xchange-appsuite-manifest open-xchange-appsuite

<div style="padding: 20px; border-style: solid; border-width: thin;">
Cluster note

*if you want to have separate frontend (apache) and middleware (open-xchange) systems, make sure to install packages which require apache as dependency on the frontend nodes, and packages which require java as a dependency on the middleware nodes. Currently this results in the split
** Frontend nodes: open-xchange-appsuite
** Middleware nodes: everything else
* If you want to use an object store, install the corresponding open-xchange-filestore-xyz package, like open-xchange-filestore-s3
* For hazelcast session storage, install also open-xchange-sessionstorage-hazelcast
</div>

=== Install database schemas ===

If the DB runs on localhost and you have root access, you can use

/opt/open-xchange/sbin/initconfigdb --configdb-pass="$(cat /root/.dbpw)" -a

<div style="padding: 20px; border-style: solid; border-width: thin;">

Cluster note

Create the DB users on all write instances manually. https://oxpedia.org/wiki/index.php?title=Template:OXLoadBalancingClustering_Database#Creating_Open-Xchange_user

mysql -e "GRANT CREATE, LOCK TABLES, REFERENCES, INDEX, DROP, DELETE, ALTER, SELECT, UPDATE, INSERT, CREATE TEMPORARY TABLES, SHOW VIEW, SHOW DATABASES ON *.* to 'openexchange'@'%' identified by '$(cat /root/.dbpw)' WITH GRANT OPTION;"

Run initconfigdb with some more options:

/opt/open-xchange/sbin/initconfigdb --configdb-user=openexchange --configdb-pass="$(cat /root/.dbpw)" --configdb-host=configdb-writehost

(initconfigdb needs to be run only once on one cluster node)
</div>

=== Initial configuration ===

/opt/open-xchange/sbin/oxinstaller --add-license=YOUR-OX-LICENSE-CODE --servername=oxserver --configdb-pass="$(cat /root/.dbpw)" --master-pass="$(cat /root/.oxpw)" --network-listener-host=localhost --servermemory 1024

'''servername''' is more like a ''clustername'' and needs to be the same for all nodes.

'''servermemory''' should be adjusted to reflect the expected number of concurrent active sessions; sizing assumption is 4MB per session.

<div style="padding: 20px; border-style: solid; border-width: thin;">
Cluster Note

--configdb-readhost=...
--configdb-writehost=...
--imapserver=...
--smtpserver=...
--mail-login-src=<login|mail|name>
--mail-server-src=<user|global>
--transport-server-src=<user|global>
--jkroute=APP1
--object-link-hostname=[service DNS name like ox.example.com]
--extras-link=[https://ox.example.com/]
--name-of-oxcluster=[something unique per cluster, like business-staging; see --servername]
--network-listener-host=<localhost|*>

oxinstaller needs to be run on each cluster node with identical options besides the jkroute, which must be unique per cluster node and match the corresponding apache option, see below.

In a cluster you also want to configure hazelcast; see [[AppSuite:Running_a_cluster#Configuration]]. Most prominent options are

com.openexchange.hazelcast.enabled=true
com.openexchange.hazelcast.group.name=<reasonable unique group name>
com.openexchange.hazelcast.group.password=<unique password, CHANGE THE SHIPPED DEFAULT PASSWORD!>
com.openexchange.hazelcast.network.join=static # static is recommended over multicast for robustness
com.openexchange.hazelcast.network.join.static.nodes=... # configure your nodes as a comma-separated list; a bootstrapping subset is acceptable
com.openexchange.hazelcast.network.interfaces=... # pick your subnet; Wildcards (*) and ranges (-) can be used
</div>

Start the service:

systemctl restart open-xchange

<div style="padding: 20px; border-style: solid; border-width: thin;">
Cluster Note

Start the service on every cluster node.
</div>

=== Registering stuff ===

Register the "server":

/opt/open-xchange/sbin/registerserver -n oxserver -A oxadminmaster -P "$(cat /root/.oxpw)"

<div style="padding: 20px; border-style: solid; border-width: thin;">
Cluster Note

All the register* commands need to be issued only once per cluster, as the effect is to enter corresponding lines in the configdb.
</div>

And the filestore:

/opt/open-xchange/sbin/registerfilestore -A oxadminmaster -P "$(cat /root/.oxpw)" -t file:/var/opt/filestore -s 1000000 -x 1000000

<div style="padding: 20px; border-style: solid; border-width: thin;">
Cluster Note

If you chose an object store, the corresponding registerfilestore line reads as follows (Ceph radosgw example):

/opt/open-xchange/sbin/registerfilestore -A oxadminmaster -P "$(cat /root/.oxpw)" -t s3://radosgw -s 1000000 -x 1000000

It requires configuration of the object store in filestore-s3.properties:

com.openexchange.filestore.s3.radosgw.endpoint=http://localhost:7480
com.openexchange.filestore.s3.radosgw.bucketName=oxbucket
com.openexchange.filestore.s3.radosgw.region=eu-west-1
com.openexchange.filestore.s3.radosgw.pathStyleAccess=true
com.openexchange.filestore.s3.radosgw.accessKey=...
com.openexchange.filestore.s3.radosgw.secretKey=...
com.openexchange.filestore.s3.radosgw.encryption=none
com.openexchange.filestore.s3.radosgw.signerOverride=S3SignerType
com.openexchange.filestore.s3.radosgw.chunkSize=5MB

Changing this file needs another

service open-xchange restart
</div>

And the database:

/opt/open-xchange/sbin/registerdatabase -A oxadminmaster -P "$(cat /root/.oxpw)" -n oxdb -p "$(cat /root/.dbpw)" -m true

<div style="padding: 20px; border-style: solid; border-width: thin;">
Cluster Note

You probably have multiple clusters with read and write URLs. Register them each like first registering the master, subsequently registering the slave URL with the corresponding master ID:

/opt/open-xchange/sbin/registerdatabase -A oxadminmaster -P "$(cat /root/.oxpw)" -n oxdb -p "$(cat /root/.dbpw)" -m true

This command gives as output the id of the registered db master, like

database 3 registered

Here, the id is 3. Use this id as argument of the -M switch of the next command:

/opt/open-xchange/sbin/registerdatabase -A oxadminmaster -P "$(cat /root/.oxpw)" -n oxdbr -p "$(cat /root/.dbpw)" -m false -M 3
</div>

=== Configure Apache ===

Create config files /etc/apache2/conf-enabled/proxy_http.conf, /etc/apache2/sites-enabled/000-default.conf by copy-pasting as explained in [[AppSuite:Open-Xchange_Installation_Guide_for_Debian_8.0#Configure_services]]

Make sure you are using mpm_event. Apply concurrent connections tuning as described in [[Tune_apache2_for_more_concurrent_connections]].

Configure modules and restart:

a2enmod proxy proxy_http proxy_balancer expires deflate headers rewrite mime setenvif lbmethod_byrequests
systemctl restart apache2

<div style="padding: 20px; border-style: solid; border-width: thin;">
Cluster Note

Make sure that each middleware node got its unique server.properties:com.openexchange.server.backendRoute, e.g. APP1, APP2, APP3, etc.

Configure them in the http proxy definitions like

BalancerMember http://ox1:8009 timeout=100 smax=0 ttl=60 retry=60 loadfactor=50 route=APP1
BalancerMember http://ox2:8009 timeout=100 smax=0 ttl=60 retry=60 loadfactor=50 route=APP2
BalancerMember http://ox3:8009 timeout=100 smax=0 ttl=60 retry=60 loadfactor=50 route=APP3

If you colocate apache on middleware nodes, you might want to minimize cross-node routing, by setting on each node for the local node a loadfactor=50 and for the other nodes a status=+H instead of the loadfactor. E.g. on node ox1:

BalancerMember http://ox1:8009 timeout=100 smax=0 ttl=60 retry=60 loadfactor=50 route=APP1
BalancerMember http://ox2:8009 timeout=100 smax=0 ttl=60 retry=60 status=+H route=APP2
BalancerMember http://ox3:8009 timeout=100 smax=0 ttl=60 retry=60 status=+H route=APP3

However this requires host-specific config files for apache http proxy.

Use touch-appsuite with one identical timestamp on each frontend node.

timestamp=$(date -u +%Y%m%d.%H%M%S)
for node in $all_frontend_nodes; do
ssh $node /opt/open-xchange/sbin/touch-appsuite --timestamp=$timestamp
done

</div>

=== Provision a Test User ===

Provision a sample context and user:

/opt/open-xchange/sbin/createcontext -c 1 -A oxadminmaster -P secret -N localdomain -u oxadmin -d "Admin User" -g Admin -s User -p secret -e oxadmin@localdomain -q 100 --access-combination-name groupware_premium
/opt/open-xchange/sbin/createuser -c 1 -A oxadmin -P secret -u testuser -d "Test User" -g Test -s User -p secret -e testuser@localdomain --access-combination-name groupware_premium

[[Context_Preprovisioning#Sample_Script]] provides an example how to fast-mode provision a huge number of contexts quickly.

AppSuite:Open-Xchange Installation Guide for CentOS 7

2018-06-12T13:14:10Z

Dominik.epple:

{{Version|7.8.0}}

= Open-Xchange App Suite on CentOS7 Linux =

{{QuickInstIntro|release=appsuite}}

= Requirements =

* Plain installed CentOS7 with latest updates
* A configured internet connection
* <code>httpd</code> - Apache web server
* <code>vim</code> is not installed by default on CentOS7. If you want to copy & paste the commands from this article into a shell window, you need to <code>yum install vim</code> first.

= Database installation =

Please consult our [[OXLoadBalancingClustering_Database#Standalone_database_setup|database installation instructions]] for information on how to install a database on the local system.

Before proceeding, make sure the local machine has got a working MySQL service in one of the supported versions / flavors with the configuration / tunings applied as mentioned on our [[My.cnf|corresponding page]].

{{AddReposRHEL|rhelname=RHEL7|release=appsuite}}

= Updating repositories and installing packages =

It is highly recommended to import the Open-Xchange build key to your package systems trusted keyring in order to make sure only Open-Xchange packages with valid signing are installed on the system. Otherwise you'll encounter warnings about untrusted package sources. To import the Open-Xchange buildkey, please refer to this quick guide: [[Importing_OX_Buildkey#Importing_key_into_apt_based_systems|Importing OX Buildkey]].

The following command starts the download and installation process of all required package for Open-Xchange deployment:

{{OXPackageInstallation|installer=yum|release=appsuite}}

{{OXConfiguration}}

{{oxinstaller|connector=http}}

After initializing the configuration, start the Open-Xchange service by executing:

$ systemctl start open-xchange

{{OXRegister|globaldb=true}}

= Configure services =

{{ApacheOXModsIntro|connector=http}}

The default installation of the Apache webserver on CentOS provides a welcome screen which is not necessary for server operation, it can be removed by deleting the corresponding configuration file:

$ rm /etc/httpd/conf.d/welcome.conf

{{Template:ApacheAppSuiteConf|connector=http|connectorConf=/etc/httpd/conf.d/proxy_http.conf|apacheconf=/etc/httpd/conf.d/ox.conf|docroot=/var/www/html|loadmodule=LoadModule proxy_http_module modules/mod_proxy_http.so|syncProxyName=eas_oxcluster}}

After the configuration is done, restart the Apache webserver

$ systemctl start httpd

== Apache Setting for more concurrent Connections ==

By default apache2 is configured to support 150 concurrent connections. This forces all parallel requests beyond that limit to wait. Especially if, for example, active sync clients maintain a permanent connection for push events to arrive. The following article explains how that can be done

[[Tune_apache2_for_more_concurrent_connections|Apache Setting for more concurrent Connections]]

= Adding services to runlevels =

The new services are now installed and configured, but to make them start up on a server boot, they need to be added to some runlevels:

$ systemctl enable mariadb
$ systemctl enable httpd
$ systemctl enable open-xchange

{{ContextUserAndLogs}}

[[Category: AppSuite]]

AppSuite:Open-Xchange Installation Guide for RHEL7

2018-06-12T13:13:40Z

Dominik.epple:

{{Version|7.8.0}}

= Open-Xchange App Suite on RedHat Enterprise Linux 7 =

{{QuickInstIntro|release=appsuite}}

= Requirements =

* Plain installed RedHat Enterprise Linux 7 with latest updates
* Valid access to the RedHat Network
* A configured internet connection
* <code>vim</code> is not installed by default on RHEL7. If you want to copy & paste the commands from this article into a shell window, you need to <code>yum install vim</code> first.

= Database installation =

Please consult our [[OXLoadBalancingClustering_Database#Standalone_database_setup|database installation instructions]] for information on how to install a database on the local system.

Before proceeding, make sure the local machine has got a working MySQL service in one of the supported versions / flavors with the configuration / tunings applied as mentioned on our [[My.cnf|corresponding page]].

= Enabling required RedHat Repositories =

Ensure that your Redhat system is subscribed by e.g. following this article https://access.redhat.com/solutions/253273

{{AddReposRHEL|rhelname=RHEL7|release=appsuite}}

= Updating repositories and installing packages =

Reload the package index. This will download the package descriptions available at the software repositories:

$ yum update

The following command starts the download and installation process of all required package for Open-Xchange deployment:

{{OXPackageInstallation|installer=yum|release=appsuite}}

{{OXConfiguration}}

{{oxinstaller|connector=http}}

After initializing the configuration, start the Open-Xchange service by executing:

$ systemctl start open-xchange

{{OXRegister|globaldb=true}}

= Configure services =

{{ApacheOXModsIntro|connector=http}}

The default installation of the Apache webserver on RHEL provides a welcome screen which is not necessary for server operation, it can be removed by deleting the corresponding configuration file:

$ rm /etc/httpd/conf.d/welcome.conf

{{Template:ApacheAppSuiteConf|connector=http|connectorConf=/etc/httpd/conf.d/proxy_http.conf|apacheconf=/etc/httpd/conf.d/ox.conf|docroot=/var/www/html/|loadmodule=LoadModule proxy_http_module modules/mod_proxy_http.so|syncProxyName=eas_oxcluster}}

After the configuration is done, restart the Apache webserver

$ systemctl start httpd

== Apache Setting for more concurrent Connections ==

By default apache2 is configured to support 150 concurrent connections. This forces all parallel requests beyond that limit to wait. Especially if, for example, active sync clients maintain a permanent connection for push events to arrive. The following article explains how that can be done

[[Tune_apache2_for_more_concurrent_connections|Apache Setting for more concurrent Connections]]

= Adding services to runlevels =

The new services are now installed and configured, but to make them start up on a server boot, they need to be added to some runlevels:

$ systemctl enable mariadb
$ systemctl enable httpd
$ systemctl enable open-xchange

{{ContextUserAndLogs}}

= Installing Open-Xchange Update packages =

Please read [[UpdatingOXPackages]] on how to get access to the latest Open-Xchange packages.

[[Category: AppSuite]]

AppSuite:Open-Xchange Installation Guide for Debian 8.0

2018-06-12T13:13:02Z

Dominik.epple:

{{Version|7.8.0}}

= OX App Suite on Debian GNU/Linux 8.0 =

{{QuickInstIntro|release=appsuite}}

= Requirements =

* Plain installed Debian GNU/Linux 8.0, no graphical tools required
* A supported Java Virtual Machine ([http://oxpedia.org/wiki/index.php?title=AppSuite:OX_System_Requirements#Server_Platforms learn more])
* A working internet connection
* vim is not installed by default on Debian. If you want to copy & paste the commands from this article into a shell window, you need to <code>apt-get install vim</code> first.

= Database installation =

Please consult our [[OXLoadBalancingClustering_Database#Standalone_database_setup|database installation instructions]] for information on how to install a database on the local system.

Before proceeding, make sure the local machine has got a working MySQL service in one of the supported versions / flavors with the configuration / tunings applied as mentioned on our [[My.cnf|corresponding page]].

{{AddReposDebian|debname=jessie|debnameox=DebianJessie|release=appsuite}}

= Updating repositories and install packages =

It is highly recommended to import the Open-Xchange build key to your package systems trusted keyring in order to make sure only Open-Xchange packages with valid signing are installed on the system. Otherwise you'll encounter warnings about untrusted package sources. To import the Open-Xchange buildkey, please refer to this quick guide: [[Importing_OX_Buildkey#Importing_key_into_apt_based_systems|Importing OX Buildkey]].

Reload the package index. This will download the package descriptions available at the software repositories and will enable the Open-Xchange repository as a valid source for signed packages:

$ apt-get update

The following command starts the download and installation process of all required package for Open-Xchange deployment:

{{OXPackageInstallation|installer=apt-get|javavendor=sun|release=appsuite}}

{{OXConfiguration}}

{{oxinstaller|connector=http}}

After initializing the configuration, restart the Open-Xchange Administration service by executing:

$ systemctl restart open-xchange

{{OXRegister|globaldb=true}}

= Configure services =

{{ApacheOXModsDebian|connector=http|extramods=lbmethod_byrequests}}

{{Template:ApacheAppSuiteConf|connector=http|connectorConf=/etc/apache2/conf-available/proxy_http.conf|apacheconf=/etc/apache2/sites-enabled/000-default.conf|docroot=/var/www/html|loadmodule=|syncProxyName=eas_oxcluster}}

Enable the proxy configuration

$ a2enconf proxy_http.conf

After the configuration is done, restart the Apache webserver

$ systemctl restart apache2

== Upgrade from Debian 7 ==

If you updated from Debian 7 to Debian 8 the proxy_http.conf file is still located in the <tt>conf.d</tt> directory which is not read in Debian 8. So you have to move the file to <tt>conf-available</tt> and execute <tt>a2enconf proxy_http</tt> afterwards.

== Apache Setting for more concurrent Connections ==

By default apache2 is configured to support 150 concurrent connections. This forces all parallel requests beyond that limit to wait. Especially if, for example, active sync clients maintain a permanent connection for push events to arrive. The following article explains how that can be done

[[Tune_apache2_for_more_concurrent_connections|Apache Setting for more concurrent Connections]]

{{ContextUserAndLogs}}

[[Category: AppSuite]]

Template:OXLoadBalancingClustering Database

2018-06-06T14:20:19Z

Dominik.epple: /* Configuration */

== Overview ==

You can choose between Galera or Master/Slave replication. We like to recommend to use Galera for higher redudancy, easier operations, und synchronous semantics (so you can run OX without our "replication monitor"). For POC or demo setups, a single standalone database setup might be sufficient.

== Standalone database setup ==

=== Preparations ===

Our configuration process includes wiping and reinitializing the datadir. This is usually not a problem in a fresh installation. If you want to upgrade an existing database server, please be prepared to wipe the datadir, i.e. take a <code>mysqldump</code> for later restoration into the properly configured master.

mysqldump --databases configdb oxdb_{5..14} > backup.sql

Be sure to verify the list of databases.

=== Installation ===

Note: the following list ist not an exclusive list or authorative statement about supported MySQL flavors / versions. Please consult the official support / system requirements statement.

Please follow the upstream docs for your preferred flavor to get the software installed on your system.

* MariaDB (10.1, 10.2): https://downloads.mariadb.org/
* Oracle MySQL Community Server (5.6, 5.7): https://dev.mysql.com/downloads/mysql/

Make sure to doublecheck the service is not running (or stop it) after installation as we need to perform some reconfigurations.

service mysql stop

=== Configuration ===

MySQL configuration advise is given in our [[My.cnf|MySQL configuration article]]. Please consult that page for configuration information and create configuration files as described there.

Some settings we recommend to change require that the database gets re-initialized. We assume you don't have data there (since we are covering a fresh install) or you have taken a backup for later restore as explained above in the Preparations section.

cd /var/lib/
mv mysql mysql.old.datadir
mkdir mysql
chown mysql.mysql mysql

# mariadb
mysql_install_db
# oracle 5.6
mysql_install_db -u mysql
# oracle 5.7
mysqld --initialize-insecure --user=mysql

(Don't be worried about the <code>insecure</code>, it just means we set the db root pw in the next steps.)

Start the service. The actual command depends on your OS and on the MySQL flavor.

service mysql start

Run <code>mysql_secure_installation</code> for a "secure by default" installation:

mysql_secure_installation

That tool will ask for the current root password (which is empty by default) and subsequently questions like:

Change the root password? [Y/n]
Remove anonymous users? [Y/n]
Disallow root login remotely? [Y/n]
Remove test database and access to it? [Y/n]
Reload privilege tables now? [Y/n]

You should answer all these questions with "yes".

Configure a strong password for the MySQL <code>root</code> user.

The further steps in this guide omit <code>-u -p</code> arguments to the MySQL client. Rather than passing them on the command line [https://dev.mysql.com/doc/refman/5.7/en/password-security-user.html] it is recommended to place the credentials in a file like <code>/root/.my.cnf</code> like

[client]
user=root
password=wip9Phae3Beijeed

Make sure the service is enabled by the OS's init system. The actual command depends on your OS and on the MySQL flavor.

systemctl enable mysql.service

You should now be able to restore your previously taken backup.

# If you took a dump for restore before
mysql < backup.sql

=== Configure OX to use with a standalone database ===

Not much special wisdom here. OX was designed to be used with master/slave databases, and a standalone master works just as well, if we register it as a master, and not registering a slave.

For the ConfigDB, <code>configdb.properties</code> allows configuration of a <code>writeUrl</code> (which is set to the correct values if you use <code>oxinstaller</code> with the correct argument <code>--configdb-writehost</code>).

The single database is then used for reading and writing.

For the individiual UserDBs, use <code>registerdatabase -m true</code>.

== Galera database setup ==

=== Preparations ===

Our configuration process includes wiping and reinitializing the datadir. This is usually not a problem in a fresh installation. If you want to upgrade an existing database to Galera cluster, please be prepared to wipe the datadir, i.e. take a <code>mysqldump</code> for later restoration into the properly configured master.

Depeding on the flavor of the current database, this can be something like

# mariadb or oracle mysql without GTIDs
mysqldump --databases configdb oxdb_{5..14} > backup.sql

# mysql 5.6 with GTIDs... we dont want GTIDs here
mysqldump --databases --set-gtid-purged=OFF configdb oxdb_{5..14} > backup.sql

Be sure to verify the list of databases.

=== Installation ===

Please follow the upstream docs for your preferred flavor to get the software installed on your system.

* Percona XtraDB Cluster (5.6, 5.7): https://www.percona.com/doc/percona-xtradb-cluster/LATEST/install/index.html
* MariaDB Galera Cluster (10.0, 10.1): https://mariadb.com/kb/en/library/getting-started-with-mariadb-galera-cluster/ (Note: with 10.0, socat is required, but not a package dependency, so you need to explicitly install also socat)

Make sure to doublecheck the service is not running (or stop it) after installation as we need to perform some reconfigurations.

service mysql stop

=== Configuration ===

Galera-specific MySQL configuration advise is included in our main [[My.cnf|MySQL configuration article]]. Please consult that page for configuration information.

That page suggests a setup were we add three custom config files to <code>/etc/mysql/ox.conf.d/</pre>: <code>ox.cnf</code> for general tuning/sizing, <code>wsrep.cnf</code> for clusterwide galera configuration, and <code>host.cnf</code> for host-specific settings.

Adjust the general settings and tunings in <code>ox.cnf</code> according to your sizing etc.

Adjust <code>wsrep.cnf</code> to reflect local paths, cluster member addresses, etc.

Adjust <code>host.cnf</code> to give node-local IPs, etc.

Version-specific hints:

# percona 5.6: unknown variable 'pxc_strict_mode=ENFORCING' ... unset that one
# mariadb 10.1: add wsrep_on=ON
# mariadb 10.0 and 10.1: set wsrep_node_incoming_address=192.168.1.22:3306 in host.cnf, otherwise the status wsrep_incoming_addresses might not be shown correctly(?!)

Some settings we recommend to change require that the database gets re-initialized. We assume you don't have data there (since we are covering a fresh install) or you have taken a backup for later restore as explained above in the Preparations section.

cd /var/lib/
mv mysql mysql.old.datadir
mkdir mysql
chown mysql.mysql mysql

# mariadb 10.0 and 10.1
mysql_install_db
# percona 5.6
mysqld --user=mysql
# percona 5.7
mysqld --initialize-insecure --user=mysql

(Don't be worried about the <code>insecure</code>, it just means we set the db root pw in the next steps.)

=== Cluster startup ===

Typically on startup a Galera node tries to join a cluster, and if it fails, it will exit. Thus, when no cluster nodes are running, the first cluster node to be started needs to be told to not try to join a cluster, but rather bootstrap a new cluster. The exact arguments vary from version to version and from flavor to flavor.

==== First node ====

So we initialize the cluster bootstrap on the first node:

# percona 5.6, 5.7
service mysql bootstrap-pxc
# mariadb 10.0
service mysql bootstrap
# mariadb 10.1: service mysql bootstrap seems to be broken, does not pass the necessary options to mysqld
galera_new_cluster

Run <code>mysql_secure_installation</code> for a "secure by default" installation:

mysql_secure_installation

The further steps in this guide omit <code>-u -p</code> arguments to the MySQL client. Rather than passing them on the command line [https://dev.mysql.com/doc/refman/5.7/en/password-security-user.html] it is recommended to place the credentials in a file like <code>/root/.my.cnf</code> like

[client]
user=root
password=wip9Phae3Beijeed

We need a Galera replication user:

CREATE USER 'sstuser'@'localhost' IDENTIFIED BY 'OpIdjijwef0';
-- percona 5.6, mariadb 10.0
GRANT RELOAD, LOCK TABLES, REPLICATION CLIENT ON *.* TO 'sstuser'@'localhost';
-- percona 5.7, mariadb 10.1
GRANT PROCESS, RELOAD, LOCK TABLES, REPLICATION CLIENT ON *.* TO 'sstuser'@'localhost';
FLUSH PRIVILEGES;

(Debian specific note: MariaDB provided startup scripts use the distro's mechanism of verifying startup/shutdown using a system user, so we create that as well:

# mariadb 10.0, 10.1
GRANT ALL PRIVILEGES ON *.* TO "debian-sys-maint"@"localhost" IDENTIFIED BY "adBexthTsI5TaEps";

If you do this, yo need to synchronize the <code>/etc/mysql/debian.cnf</code> file from the first node to the other nodes as well.)

==== Other nodes ====

On the other nodes, we only need to restart the service now, to trigger a full state transfer from the first node to the other nodes.

We recommend to do this serially to let one state transfer complete before the second state transfer.

==== First node (continued) ====

Only applicable if you used <code>galera_new_cluster</code> before rather than the service script: In order to get the systemctl status consistent, restart the service on the first node:

# mariadb 10.1: restart the service so that the systemctl status is consistent
mysqladmin shutdown
service mysql bootstrap

=== Verify the replication ===

The key tool to verify replication status is

mysql> show status like "%wsrep%";

This will give a lot of output. You want to verify in particular

+------------------------------+--------------------------------------+
| Variable_name | Value |
+------------------------------+--------------------------------------+
| wsrep_cluster_size | 3 |
| wsrep_cluster_status | Primary |
| wsrep_local_state | 4 |
| wsrep_local_state_comment | Synced |
| wsrep_ready | ON |
+------------------------------+--------------------------------------+

You can also explicitly verify replication by creating / inserting DBs, tables, rows on one node and select on other nodes.

==== Troubleshooting ====

The logs are helpful. Always.

Common mistakes are listed below.

If the Galera module does not get loaded at all:
* Configuration settings in ''my.cnf'' which are incompatible to Galera
* Wrong path of the shared object providing the Galera plugin in wsrep.cnf (wsrep_provider)

If the first node starts, but the second / third nodes can not be added to the cluster:
* User for the replication not created correctly on the first Galera node
* SST fails due to missing / wrong version prerequisite packages (not everything is hardcoded in package dependencies -- make sure you got percona-xtrabackup installed in the correct version, and also socat). If SST fails, do not only look into mysqls primary error logs, but also into logfiles from the SST tool in /var/lib/mysql on the donor node.

=== Notes about configuring OX for use with Galera ===

==== Write requests ====

Open-Xchange supports Galera as database backend only in the configuration where all writes are directed to one Galera node. For availability, it makes sense to not configure one Galera node's IP address directly, but rather employ some HA solution which offers active-passive functionality. Options therefore are discussed below.

==== Read requests ====

Read requests can be directed to any node in the Galera cluster. Our standard approach is to recommend to use a loadbalancer to implement round-robin over all nodes in a Galera cluster for the read requests. But you can also chose to use a dedicated read node (the same node, or a different node, than the write node). Each of the approaches has its own advantages.

* Load balancer based setup: Read requests get distributed round-robin between the Galera nodes. Theoretically by distributing the load of the read requests, you benefit from lower latencies and more throughput. But this has never been benchmarked yet. For a discussion of available loadbalances, see next section. OX-wise, in this configuration, you have two alternatives:
** The Galera option wsrep_causal_reads=1 option enables you to configure OX with its replication monitor disabled (com.openexchange.database.replicationMonitor=false in configdb.properties). This is the setup which seems to perform best according to our experience as turning off the replication monitor reduces the commits on the DB and thus the write operations per second on the underlying storage significantly, which outweights the drawback from having higher commit latency due to fully synchronous mode.
** Alternatively, you can run Galera with wsrep_causal_reads=0 when switching on OX builtin replication monitor. This is also a valid setup.
* Use a designated floating IP for the read requests: This eliminates the need of a load balancer. With this option you will not gain any performance, but the quantitative benefit is unclear anyhow.
* Use the floating IP for the writes also for the reads: In this scenario, you direct all database queries only to one Galera node, and the other two nodes are only getting queries in case of a failure of that node. In this case, you can even use wsrep_causal_reads=0 while still having OX builtin replication monitor switched off. However we do not expect this option to be superior to the round-robin loadbalancer approach.

=== Loadbalancer options ===

While the JDBC driver has some round-robin load balancing capabilities built-in, we don't recommend it for production use since it lacks possibilities to check the Galera nodes health states.

Loadbalancers used for OX -> Galera loadbalancing should be able to implement active-passive instances for the write requests, and active-active (round-robin) instances for the read requests. (If they cannot implement active-passive, you can still take a floating IP therefore.) Furthermore it is required to configure node health checks not only on the TCP level (by a simple connect), but to query the Galera health status periodically, evaluating Galera WSREP status variables. Otherwise split-brain scenarios or other bad states cannot be detected. For an example of such an health check, see our [[Clustercheck]] page.

Some customers use loadbalancing appliances. It is important to check that if the (virtual) infrastructure offers "loadbalancer" instances that they satisfy the given requirements. Often this is not the case. In particular, a simple "DNS round robin" approach is not viable.

==== LVS/ipvsadm/keepalived ====

If you want to create your own loadbalancers based on Linux, we usually recommend LVS (Linux Virtual Servers) controlled by Keepalived. LVS is a set of kernel modules implementing a L4 loadbalancer which performs quite well. Keepalived is a userspace daemon to control LVS rules, using health checks to reconfigure LVS rules if required. Keepalived / LVS requires one (or, for availability, two) dedicated linux nodes to run on. This can be a disadvantage for some installations, but usually, it pays off. We provide some configuration information on Keepalived [[Keepalived|here]].

==== MariaDB Maxscale ====

Since Maxscale has become GA in 2015, it seems to have undergone significant stability, performance and functional improvements. We are currently experimenting with Maxscale and share our installation / configuration knowledge [[Maxscale|here]]. It looks quite promising and might become ''the standard replacement'' for HAproxy, while we still presume Keepalived offers superior robustness and performance, coming with the cost of the requirement for one (or more) dedicated loadbalancer nodes.

==== HAproxy ====

In case where the Keepalived based approach is not feasible due to its requirements on the infrastructure, it is also possible to use a HAproxy based solution where HAproxy processes run on each of the OX nodes, configured for one round-robin and one active/passive instance. OX is then connecting to the local HAproxy instances. It is vital to configure HAproxy timeouts different from the defaults, otherwise HAproxy will kill active DB connections, causing errors. Be aware that in large installations the number of (distributed) HAproxy instances can get quite large. Some configuration hints for HAproxy are available [[HAproxy|here]].

== Master/Slave database setup ==

While we also support also "legacy" (pre-GTID) Master/Slave replication, we recommend to use GTID based replication, for easier setup and failure recovery. Support for GTID based replication has been added with OX 7.8.0.

GTID has been available since MySQL 5.6, so no 5.5 installation instructions below, sorry. We try to be generic in this documentation (thus, applicable to Oracle Community Edition and MariaDB) and point out differences where needed. Note: Instructions below include information about Oracle Community MySQL 5.7 which is not yet formally supported.

=== Preparations ===

Our configuration process includes wiping and reinitializing the datadir. This is usually not a problem in a fresh installation. If you want to upgrade an existing database to GTID master-slave, please be prepared to wipe the datadir, i.e. take a <code>mysqldump</code> for later restoration into the properly configured master.

Depeding on the flavor of the current database, this can be something like

# mariadb or oracle mysql without GTIDs
mysqldump --databases configdb oxdb_{5..14} > backup.sql

# mysql 5.6 with GTIDs... we dont want GTIDs here
mysqldump --databases --set-gtid-purged=OFF configdb oxdb_{5..14} > backup.sql

Be sure to verify the list of databases.

=== Installation ===

Software installation is identical for master and slave.

Please follow the instructions for installing from The vendors.

* Oracle Community Edition: https://dev.mysql.com/doc/mysql-apt-repo-quick-guide/en/
* MariaDB (10.0, 10.1): https://downloads.mariadb.org/mariadb/repositories/

Stop the service (if it is running):

service mysql stop

=== Configuration ===

Configuration as per configuration files is also identical for master and slave.

Consult [[My.cnf]] for general recommendations how to configure databases for usage with OX.

For GTID based replication, make sure you add some configurables to a new <code>/etc/mysql/ox.conf.d/gtid.cnf</code> file (assuming you are following our proposed schema of adding a <code>!includedir /etc/mysql/ox.conf.d/</code>" directive to <code>/etc/mysql/my.cnf</code>):

# GTID
log-bin=mysql-bin
server-id=...
log_slave_updates = ON

Oracle Community Edition: we need to add also

enforce_gtid_consistency = ON
gtid_mode = ON

(GTID mode is on by default on MariaDB.)

Use unique a <code>server-id</code> for each server; like <code>1</code> for the master, <code>2</code> for slave. For more complicated setups (like multiple slaves), adjust accordingly.

Since applying our configuration / sizing requires reinitialization of the MySQL datadir, we wipe/recreate it. Caution: this assumes we are running an empty database. If there is data in the database you want to keep, use mysqldump. See Preparation section above.

So, to initialize the datadir:

cd /var/lib/
mv mysql mysql.old.datadir
mkdir mysql
chown mysql.mysql mysql

(When coming from an existing installation, be sure to wipe also old binlogs. They can confuse the server on startup. Their location varies by configuration.)

The step to initialize the datadir is different for the different DBs:

# MariaDB 10.0, 10.1
mysql_install_db

# Oracle 5.6
mysql_install_db -u mysql

# Oracle 5.7
mysqld --initialize-insecure --user=mysql

(Don't be worried about the <code>insecure</code>, it just means we set the db root pw in the next steps.)

Then:

service mysql restart
mysql_secure_installation

We want to emphasize the last step to run "secure".

Steps up to here apply to both the designated master and slave. The next steps will apply to the master.

=== Replication Setup ===

==== Master Setup ====

Create a replication user on the master (but, as always, pick your own password, and use the same password in the slave setup below):

mysql -e "CREATE USER 'repl'@'gtid-slave.localdomain' IDENTIFIED BY 'IvIjyoffod2'; GRANT REPLICATION SLAVE ON *.* TO 'repl'@'gtid-slave.localdomain';"

Now would also be the time to restore a previously created mysqldump, or add other users you need for adminstration, monitoring etc (like <code>debian-sys-maint@localhost</code>, for example). Adding the OX users is explained below ("Creating Open-Xchange user").

# If you took a dump for restore before
mysql < backup.sql

To prepare for the initial sync of the slave, set the master read-only:

mysql -e "SET @@global.read_only = ON;"

Create a dump to initialize the slave:

# MariaDB
mysqldump --all-databases --triggers --routines --events --master-data --gtid > master.sql

# Oracle
mysqldump --all-databases --triggers --routines --events --set-gtid-purged=ON > master.sql

Transfer to the slave:

scp master.sql gtid-slave:

==== Slave Setup ====

Configure the replication master settings. Note we don't need complicated binlog position settings etc with GTID.

Yet again DB-specific (use the repl user password from above):

# MariaDB
mysql -e 'CHANGE MASTER TO MASTER_HOST="gtid-master.localdomain", MASTER_USER="repl", MASTER_PASSWORD="IvIjyoffod2";'

# Oracle
mysql -e "CHANGE MASTER TO MASTER_HOST='gtid-master.localdomain', MASTER_USER='repl', MASTER_PASSWORD='IvIjyoffod2', MASTER_AUTO_POSITION=1;"
# https://www.percona.com/blog/2013/02/08/how-to-createrestore-a-slave-using-gtid-replication-in-mysql-5-6/
mysql -e "RESET MASTER;"

Read the master dump:

mysql < master.sql

Start replication on the slave:

mysql -e 'START SLAVE;'
mysql -e 'SHOW SLAVE STATUS\G'

==== Master Setup (continued) ====

Finally, unset read-only on the master:

# on the master
mysql -e "SET @@global.read_only = OFF;"

=== Configure OX to use with Master/Slave replication ===

Not much special wisdom here. OX was designed to be used with master/slave databases. For the ConfigDB, <code>configdb.properties</code> allows configuration of a <code>readUrl</code> and <code>writeUrl</code> (both of which are set to the correct values if you use <code>oxinstaller</code> with the correct arguments <code>--configdb-readhost</code>, <code>--configdb-writehost</code>).

(Obviously, the master is for writing and the slave is for reading.)

For the individiual UserDBs, use <code>registerdatabase -m true</code> for the masters and <code>registerdatabase -m false -M ...</code> for the respective slaves.

Be sure to have enabled the replication monitor in <code>configdb.properties</code>: <code>com.openexchange.database.replicationMonitor=true</code> (which it is by default); while GTID can show synchronous semantics, it is specified to silently fall back to asynchronous in certain circumstances, so synchronity is not guaranteed.

We recommend, though, to not register the databases directly by their native hostname or IP, but rather use some kind of HA system in order to be able to easily move a floating/failover IP from the master to the slave in case of master failure. Configuring and running such systems (like, corosync/pacemaker, keepalived, or whatever) is out of scope of this documentation, however.

== Creating Open-Xchange user ==

Now setup access for the Open-Xchange Server database user 'openexchange' to configdb and the oxdb for both groupware server addresses. These databases do not exist yet, but will be created during the Open-Xchange Server installation.

Notes:

* Please use a real password.
* The IPs in this example belong to the two different Open-Xchange Servers, please adjust them accordingly.
* If using a database on the same host as the middlware (usually done for POCs and demo installations), you need to grant also to the ''localhost'' host.
* Consult [[AppSuite:DB_user_privileges]] (or ''grep GRANT /opt/open-xchange/sbin/initconfigdb'') for an up-to-date list of required privileges. The following statement was correct as of the time of writing this section.

mysql> GRANT CREATE, LOCK TABLES, REFERENCES, INDEX, DROP, DELETE, ALTER, SELECT, UPDATE, INSERT, CREATE TEMPORARY TABLES, SHOW VIEW, SHOW DATABASES ON *.* TO 'openexchange'@'10.20.30.213' IDENTIFIED BY 'IntyoyntOat1' WITH GRANT OPTION;
mysql> GRANT CREATE, LOCK TABLES, REFERENCES, INDEX, DROP, DELETE, ALTER, SELECT, UPDATE, INSERT, CREATE TEMPORARY TABLES, SHOW VIEW, SHOW DATABASES ON *.* TO 'openexchange'@'10.20.30.215' IDENTIFIED BY 'IntyoyntOat1' WITH GRANT OPTION;

Template:OXPackageInstallation

2018-06-05T13:09:56Z

Dominik.epple:

{{#if:{{{release|}}}|
{{#ifeq:{{{release|}}}|6.22|

If you want to install everything on a single server, just run

$ {{{installer}}} install open-xchange open-xchange-authentication-database open-xchange-grizzly open-xchange-admin \
open-xchange-gui {{#if:{{{addonpkgs|}}}|{{{addonpkgs}}}|}}

'''Note 1:''' You have to choose between one of the available spamhandler and authentication packages depending on your requirements.

|{{#ifeq:{{{release|}}}|7|

If you want to install everything on a single server, just run

$ {{{installer}}} install open-xchange open-xchange-authentication-database \
open-xchange-ajp open-xchange-admin open-xchange-ui7

'''Note:''' You have to choose between one of the available authentication packages depending on your requirements.

|{{#ifeq:{{{release|}}}|appsuite|

If you want to install everything on a single server, just run

$ {{{installer}}} install open-xchange open-xchange-authentication-database open-xchange-grizzly \
open-xchange-admin open-xchange-appsuite \
open-xchange-appsuite-backend open-xchange-appsuite-manifest

'''Note 1:''' You have to choose between one of the available authentication packages depending on your requirements.

}}|}}|}}| {{#!:

== Using Meta packages ==

The following Meta packages are available:

{| border="1" cellpadding="3" cellspacing="0"
!style="width:230px" align="left" |Name
!style="width:230px" align="left" |Description
|-
|open-xchange-meta-admin
| all provisioning packages
|-
|open-xchange-meta-gui
| all gui packages
|-
|open-xchange-meta-messaging
| the complete messaging packages like unified inbox, twitter, facebook, etc.
|-
|open-xchange-meta-mobility
| OXtender for business mobility
|-
|open-xchange-meta-outlook
| Outlook OXtender Updater
|-
|open-xchange-meta-pubsub
| all publish/subscribe related packages
|-
|open-xchange-meta-server
| the server backend packages
|-
|open-xchange-meta-databaseonly
| all packages needed when open-xchange is only managed via database and not e.g. LDAP
|-
|open-xchange-meta-singleserver
| most of the above
|}

If you want to install everything on a single server, just run

$ {{{installer}}} install open-xchange-meta-singleserver \
open-xchange-authentication-database open-xchange-spamhandler-default {{#if:{{{addonpkgs|}}}|{{{addonpkgs}}}|}}

'''Note:''' You have to choose between one of the available spamhandler and authentication packages depending on your requirements. If you plan to only manage your open-xchange installation via database and do not plan to integrate e.g. with LDAP and [[OXLDAPSync_Guide|OXLDAPSync]], you might also want to install the package open-xchange-meta-databaseonly.''

Of course you can still install the single packages as usual to be able to select a specific set of functionality that you'd like to have, for example:

$ {{{installer}}} install \
{{OXPackageInstallation-Server|javavendor={{{javavendor}}}}} \
{{OXPackageInstallation-GUI}} {{#if:{{{addonpkgs|}}}|{{{addonpkgs}}}|}}

}}}}

Template:ContextUserAndLogs

2018-06-05T13:05:14Z

Dominik.epple:

= Creating contexts and users =

Now as the whole setup is complete and you already should get a login screen when accessing the server with a webbrowser, we have to setup a context and a default user as the last step of this tutorial.

The mapping ''defaultcontext'' will allow you to set this context as the default one of the entire system so that users which will be created within this context can login into Open-Xchange Server without specifying their domain at the login screen. Only one context can be specified as ''defaultcontext''. The ''oxadmin'' user that will be created by this command is the default admin of the created context. This account will gather additional functions that are also described in the administration manual. The ''context id'' parameter must to be unique and numeric, otherwise the server will complain when you try to create a context. New contexts must be created by the ''oxadminmaster'' user, user accounts inside a context are created with the credentials of the contexts ''oxadmin'' account. The ''access-combination-name'' property defines the set of available modules and functions for users of the context.

$ /opt/open-xchange/sbin/createcontext -A oxadminmaster -P admin_master_password -c 1 \
-u oxadmin -d "Context Admin" -g Admin -s User -p admin_password -L defaultcontext \
-e oxadmin@example.com -q 1024 --access-combination-name=groupware_standard

To create a user for testing purposes (Make sure the password you use here for the user is the same password as your email account or you will not be able to use the email module until it is set right):

$ /opt/open-xchange/sbin/createuser -c 1 -A oxadmin -P admin_password -u testuser \
-d "Test User" -g Test -s User -p secret -e testuser@example.com \
--imaplogin testuser --imapserver 127.0.0.1 --smtpserver 127.0.0.1

Now connect to the server with a webbrowser and login using the credentials testuser / secret.

A complete overview about the different parameter is provided at the [[OX_Permission_Level|permission matrix]]

If you need to migrate a batch of users and contexts at once, check the CSV Batch Import documentation [[Csv_import|page]].

= Log files and issue tracking =
== Default logging mechanism ==
Whenever unexpected or erroneous behavior takes place, it will be logged depending on the configured loglevel. All logfiles are stored at the operating systems default location. Events triggered by the Open-Xchange Groupware services are logged to a rotating file ''open-xchange.log'', events triggered by the Open-Xchange Administration service are logged to ''open-xchange-admin.log''. Those files are the very first place to monitor.

$ tail -f -n200 /var/log/open-xchange/open-xchange.log.0

== Alternative logging mechanisms ==
Apart from the default file logging mechanism, Open-Xchange supports logging via logback framework and therefore via syslog and/or logstash. This makes it possible to directly log to a local or remote syslog daemon or other services. Logback is highly customizable, please see the documentation below.

* [[AppSuite:OX_Logging|Logback configuration Guide OX App Suite]]

Template:OXConfiguration

2018-06-05T12:49:52Z

Dominik.epple: /* Open-Xchange configuration */

= Open-Xchange configuration =

To avoid confusion right at the start notice that Open-Xchange uses multiple administration levels and requires different credentials at some stages at the installation and server management. Note that the passwords chosen at this guide are weak and should be replaced by stronger passwords.
* The MySQL database user
** Username: openexchange
** Password used at this guide: db_password
** Responsibility: Execute all kinds of database operations

* The Open-Xchange Admin Master
** Username: oxadminmaster
** Password used at this guide: admin_master_password
** Responsibility: Manage contexts, manage all kinds of low level server configuration

* The Context Admin
** Username: oxadmin
** Password used at this guide: admin_password
** Responsibility: Manage users/groups/resources inside a context

As stated above we assume the MySQL service has been installed previously, and it is running and available.

A good idea is to add the Open-Xchange binaries to PATH:

$ echo PATH=$PATH:/opt/open-xchange/sbin/ >> ~/.bashrc && . ~/.bashrc

Now we have to initialize the Open-Xchange ''configdb'' database. This can all be done by executing the ''initconfigdb'' script.

$ /opt/open-xchange/sbin/initconfigdb --configdb-pass=db_password -a --mysql-root-passwd=root_password

Use the ''--mysql-root-passwd'' option to supply the MySQL root password as configured during database installation.

Add the ''-i'' option if you want to remove an already existing open-xchange configdb.

'''Note:''' The -a parameter adds an ''openexchange'' account to MySQL. This account will be used for database connections from the OX App Suite middleware and requires [[AppSuite:DB_user_privileges|some privileges]]. You can also create that account manually [[OXLoadBalancingClustering_Database#Creating_Open-Xchange_user|during database installation / configuration]], in which case you can (should) skip the ''-a'' parameter here.

Before starting any service, all basic configuration files need to be set up correctly. The ''--configdb-pass'' option indicates the password of the ''openexchange'' database user previously created, the ''--master-pass'' options specifies the password of the Open-Xchange ''adminmaster'' user that will be created when executing the ''oxinstaller'' script.

User:Dominik.epple

2018-06-05T12:49:28Z

Dominik.epple:

== Install Guide ==

=== About this document ===

The aim of this document is to improve on the existing quickinstall guides to be more structured, provide a more extensive view on "single node and beyond" topics, follow closer to existing "best practices" (also, but not only security-wise), and point out what needs to be changed in clustered installations.

Most of the commands given in this document thus assume a high level design of "single-node, all-in-one".

This document was created on Debian Stretch (which, as of time of writing, is not even supported yet), but it should work as-is also for jessie. Porting to RHEL/SLES/... is TODO.

=== Preparations ===

==== System update ====

You want to start on latest patchlevel of your OS:

apt-get update
apt-get dist-upgrade
apt-get install less vim pwgen apt-transport-https

reboot

==== Pregenerate passwords ====

This guide shall feature copy-paste ready commands which create installations with no default passwords.

We will pre-generate some passwords which will live as dotfiles in /root.

pwgen -c -n 16 1 > /root/.oxpw
pwgen -c -n 16 1 > /root/.dbpw
pwgen -c -n 16 1 > /root/.dbrootpw

==== Prepare database ====

In real-world installations this will probably be multiple galera clusters of a supported flavor and version. For educational purposes a standalone DB on our single-node machine is sufficient.

Even for single-node, don't forget to apply database tuning. See [[My.cnf|our oxpedia article]] for default tunings. Note that typically you need to re-initialize the MySQL datadir after changing InnoDB sizing values, and subsequently start the service:

mysql_install_db
service mysql restart

We aim to create secure-by-default documentation, so here we go: Run mysql_secure_installation, and chose every security relevant option, but let the root password empty in this step, as we set it in the next step:

# leave the root password empty in mysql_secure_installation as we set it in the subsequent step
mysql_secure_installation
# now, configure the password from /root/.dbrootpw
mysql -e "UPDATE mysql.user SET Password=PASSWORD('$(cat /root/.dbrootpw)') WHERE User='root'; FLUSH PRIVILEGES;"
cat >/root/.my.cnf <<EOF
[client]
user=root
password=$(cat /root/.dbrootpw)
EOF

MySQL 5.7: the aforementioned must be adjusted using

ALTER USER USER() IDENTIFIED BY 'tiez7EiNgaish0ee';

These credentials also needs to be put in /etc/mysql/debian.cnf.

<div style="padding: 20px; border-style: solid; border-width: thin;">
Cluster note

On multiple db clusters, do it per node analogously. Just be aware the copy-paste command above expects the /root/.dbrootpw file.
</div>

==== Prepare OX user ====

While the packages will create the user automatically if it does not exist, we want to prepare the filestore now, and we need the user therefore.

useradd -r open-xchange

<div style="padding: 20px; border-style: solid; border-width: thin;">
Cluster Note

You should hard-wire the userid and groupid to the same fixed value. Otherwise, if you want to use a NFS filestore, you'll run into permissions problems, unless you use nfs4/kerberos/idmapd.

groupadd -r -g 999 open-xchange
useradd -r -g 999 -u 999 open-xchange
</div>

==== Prepare filestore ====

There are several options here.

===== Single-Node: local directory =====

For a single-node installation, you can just prepare a local directory:

mkdir /var/opt/filestore
chown open-xchange:open-xchange /var/opt/filestore

===== NFS =====

If using NFS:

Setup on the NFS server:

apt-get install nfs-kernel-server
service nfs-kernel-server restart

Configure /etc/exports. This is for traditional ip based access control; krb5 or other security configuration is out of scope of this document.

mkdir /var/opt/filestore
chown open-xchange:open-xchange /var/opt/filestore
echo "/var/opt/filestore 192.168.1.0/24(rw,sync,fsid=0,no_subtree_check)" >> /etc/exports
exportfs -a

Clients can then mount using

mkdir /var/opt/filestore
mount -t nfs -o vers=4 nfs-server:/filestore /var/opt/filestore

Or using fstab entries like

nfs-server:/filestore /var/opt/filestore nfs4 defaults 0 0

===== Object Store =====

You can use an object store. For lab environments Ceph is a convenient option. For demo / educational purpuses a "single node Ceph cluster" even co-located on your "single-node machine" is reasonble, but its setup is out of scope of this document. If you want to use this, be prepared to provide information about endpoint, bucket name, access key, secret key.

===== No filestore =====

If you dont want to provide a filestore, you can configure OX later to run without filestore. (Q: do we still need a dummy registerfilestore on a local directory in that event?)

==== Prepare mail system ====

Formally out of scope of this document.

If you need to create a testing dovecot/postfix setup, you can use our [https://gitlab.open-xchange.com/qa/performance/tree/develop/com.openexchange.test.performance.gatling/src/site-preprocess/dovecot performance testing sample config].

=== Install OX software ===

You need an ldb user and password for updates and proprietary repos. If you dont have such a user, you can still install the free components. You'll get a lot of authentication failed warnings however from apt tools unless you deconfigure the closed repos.

wget http://software.open-xchange.com/oxbuildkey.pub -O - | apt-key add -
wget -O/etc/apt/sources.list.d/ox.list http://software.open-xchange.com/products/DebianJessie.list
ldbuser=...
ldbpassword=...
sed -i -e "s/LDBUSER:LDBPASSWORD/$ldbuser:$ldbpassword/" /etc/apt/sources.list.d/ox.list
apt-get update
apt-get install open-xchange open-xchange-authentication-database open-xchange-grizzly open-xchange-admin open-xchange-appsuite-backend open-xchange-appsuite-manifest open-xchange-appsuite

<div style="padding: 20px; border-style: solid; border-width: thin;">
Cluster note

*if you want to have separate frontend (apache) and middleware (open-xchange) systems, make sure to install packages which require apache as dependency on the frontend nodes, and packages which require java as a dependency on the middleware nodes. Currently this results in the split
** Frontend nodes: open-xchange-appsuite
** Middleware nodes: everything else
* If you want to use an object store, install the corresponding open-xchange-filestore-xyz package, like open-xchange-filestore-s3
* For hazelcast session storage, install also open-xchange-sessionstorage-hazelcast
</div>

=== Install database schemas ===

If the DB runs on localhost and you have root access, you can use

/opt/open-xchange/sbin/initconfigdb --configdb-pass="$(cat /root/.dbpw)" -a

<div style="padding: 20px; border-style: solid; border-width: thin;">

Cluster note

Create the DB users on all write instances manually. https://oxpedia.org/wiki/index.php?title=Template:OXLoadBalancingClustering_Database#Creating_Open-Xchange_user

mysql -e "GRANT CREATE, LOCK TABLES, REFERENCES, INDEX, DROP, DELETE, ALTER, SELECT, UPDATE, INSERT, CREATE TEMPORARY TABLES, SHOW VIEW, SHOW DATABASES ON *.* to 'openexchange'@'%' identified by '$(cat /root/.dbpw)' WITH GRANT OPTION;"

Run initconfigdb with some more options:

/opt/open-xchange/sbin/initconfigdb --configdb-user=openexchange --configdb-pass="$(cat /root/.dbpw)" --configdb-host=configdb-writehost

(initconfigdb needs to be run only once on one cluster node)
</div>

=== Initial configuration ===

/opt/open-xchange/sbin/oxinstaller --add-license=YOUR-OX-LICENSE-CODE --servername=oxserver --configdb-pass="$(cat /root/.dbpw)" --master-pass="$(cat /root/.oxpw)" --network-listener-host=localhost --servermemory 1024

'''servername''' is more like a ''clustername'' and needs to be the same for all nodes.

'''servermemory''' should be adjusted to reflect the expected number of concurrent active sessions; sizing assumption is 4MB per session.

<div style="padding: 20px; border-style: solid; border-width: thin;">
Cluster Note

--configdb-readhost=...
--configdb-writehost=...
--imapserver=...
--smtpserver=...
--mail-login-src=<login|mail|name>
--mail-server-src=<user|global>
--transport-server-src=<user|global>
--jkroute=APP1
--object-link-hostname=[service DNS name like ox.example.com]
--extras-link=[https://ox.example.com/]
--name-of-oxcluster=[something unique per cluster, like business-staging; see --servername]
--network-listener-host=<localhost|*>

oxinstaller needs to be run on each cluster node with identical options besides the jkroute, which must be unique per cluster node and match the corresponding apache option, see below.

In a cluster you also want to configure hazelcast; see [[AppSuite:Running_a_cluster#Configuration]]. Most prominent options are

com.openexchange.hazelcast.enabled=true
com.openexchange.hazelcast.group.name=<reasonable unique group name>
com.openexchange.hazelcast.group.password=<unique password, CHANGE THE SHIPPED DEFAULT PASSWORD!>
com.openexchange.hazelcast.network.join=static # static is recommended over multicast for robustness
com.openexchange.hazelcast.network.join.static.nodes=... # configure your nodes as a comma-separated list; a bootstrapping subset is acceptable
com.openexchange.hazelcast.network.interfaces=... # pick your subnet; Wildcards (*) and ranges (-) can be used
</div>

Start the service:

systemctl restart open-xchange

<div style="padding: 20px; border-style: solid; border-width: thin;">
Cluster Note

Start the service on every cluster node.
</div>

=== Registering stuff ===

Register the "server":

/opt/open-xchange/sbin/registerserver -n oxserver -A oxadminmaster -P "$(cat /root/.oxpw)"

<div style="padding: 20px; border-style: solid; border-width: thin;">
Cluster Note

All the register* commands need to be issued only once per cluster, as the effect is to enter corresponding lines in the configdb.
</div>

And the filestore:

/opt/open-xchange/sbin/registerfilestore -A oxadminmaster -P "$(cat /root/.oxpw)" -t file:/var/opt/filestore -s 1000000 -x 1000000

<div style="padding: 20px; border-style: solid; border-width: thin;">
Cluster Note

If you chose an object store, the corresponding registerfilestore line reads as follows (Ceph radosgw example):

/opt/open-xchange/sbin/registerfilestore -A oxadminmaster -P "$(cat /root/.oxpw)" -t s3://radosgw -s 1000000 -x 1000000

It requires configuration of the object store in filestore-s3.properties:

com.openexchange.filestore.s3.radosgw.endpoint=http://localhost:7480
com.openexchange.filestore.s3.radosgw.bucketName=oxbucket
com.openexchange.filestore.s3.radosgw.region=eu-west-1
com.openexchange.filestore.s3.radosgw.pathStyleAccess=true
com.openexchange.filestore.s3.radosgw.accessKey=...
com.openexchange.filestore.s3.radosgw.secretKey=...
com.openexchange.filestore.s3.radosgw.encryption=none
com.openexchange.filestore.s3.radosgw.signerOverride=S3SignerType
com.openexchange.filestore.s3.radosgw.chunkSize=5MB

Changing this file needs another

service open-xchange restart
</div>

And the database:

/opt/open-xchange/sbin/registerdatabase -A oxadminmaster -P "$(cat /root/.oxpw)" -n oxdb -p "$(cat /root/.dbpw)" -m true

<div style="padding: 20px; border-style: solid; border-width: thin;">
Cluster Note

You probably have multiple clusters with read and write URLs. Register them each like first registering the master, subsequently registering the slave URL with the corresponding master ID:

/opt/open-xchange/sbin/registerdatabase -A oxadminmaster -P "$(cat /root/.oxpw)" -n oxdb -p "$(cat /root/.dbpw)" -m true

This command gives as output the id of the registered db master, like

database 3 registered

Here, the id is 3. Use this id as argument of the -M switch of the next command:

/opt/open-xchange/sbin/registerdatabase -A oxadminmaster -P "$(cat /root/.oxpw)" -n oxdbr -p "$(cat /root/.dbpw)" -m false -M 3
</div>

=== Configure Apache ===

Create config files /etc/apache2/conf-enabled/proxy_http.conf, /etc/apache2/sites-enabled/000-default.conf by copy-pasting as explained in [[AppSuite:Open-Xchange_Installation_Guide_for_Debian_8.0#Configure_services]]

Make sure you are using mpm_event. Apply concurrent connections tuning as described in [[Tune_apache2_for_more_concurrent_connections]].

Configure modules and restart:

a2enmod proxy proxy_http proxy_balancer expires deflate headers rewrite mime setenvif lbmethod_byrequests
systemctl restart apache2

<div style="padding: 20px; border-style: solid; border-width: thin;">
Cluster Note

Make sure that each middleware node got its unique server.properties:com.openexchange.server.backendRoute, e.g. APP1, APP2, APP3, etc.

Configure them in the http proxy definitions like

BalancerMember http://ox1:8009 timeout=100 smax=0 ttl=60 retry=60 loadfactor=50 route=APP1
BalancerMember http://ox2:8009 timeout=100 smax=0 ttl=60 retry=60 loadfactor=50 route=APP2
BalancerMember http://ox3:8009 timeout=100 smax=0 ttl=60 retry=60 loadfactor=50 route=APP3

If you colocate apache on middleware nodes, you might want to minimize cross-node routing, by setting on each node for the local node a loadfactor=50 and for the other nodes a status=+H instead of the loadfactor. E.g. on node ox1:

BalancerMember http://ox1:8009 timeout=100 smax=0 ttl=60 retry=60 loadfactor=50 route=APP1
BalancerMember http://ox2:8009 timeout=100 smax=0 ttl=60 retry=60 status=+H route=APP2
BalancerMember http://ox3:8009 timeout=100 smax=0 ttl=60 retry=60 status=+H route=APP3

However this requires host-specific config files for apache http proxy.

Use touch-appsuite with one identical timestamp on each frontend node.

timestamp=$(date -u +%Y%m%d.%H%M%S)
for node in $all_frontend_nodes; do
ssh $node /opt/open-xchange/sbin/touch-appsuite --timestamp=$timestamp
done

</div>

=== Provision a Test User ===

Provision a sample context and user:

/opt/open-xchange/sbin/createcontext -c 1 -A oxadminmaster -P secret -N localdomain -u oxadmin -d "Admin User" -g Admin -s User -p secret -e oxadmin@localdomain -q 100 --access-combination-name groupware_premium
/opt/open-xchange/sbin/createuser -c 1 -A oxadmin -P secret -u testuser -d "Test User" -g Test -s User -p secret -e testuser@localdomain --access-combination-name groupware_premium

[[Context_Preprovisioning#Sample_Script]] provides an example how to fast-mode provision a huge number of contexts quickly.

Template:OXLoadBalancingClustering Database

2018-06-05T12:47:36Z

Dominik.epple: /* Creating Open-Xchange user */

== Overview ==

You can choose between Galera or Master/Slave replication. We like to recommend to use Galera for higher redudancy, easier operations, und synchronous semantics (so you can run OX without our "replication monitor"). For POC or demo setups, a single standalone database setup might be sufficient.

== Standalone database setup ==

=== Preparations ===

Our configuration process includes wiping and reinitializing the datadir. This is usually not a problem in a fresh installation. If you want to upgrade an existing database server, please be prepared to wipe the datadir, i.e. take a <code>mysqldump</code> for later restoration into the properly configured master.

mysqldump --databases configdb oxdb_{5..14} > backup.sql

Be sure to verify the list of databases.

=== Installation ===

Note: the following list ist not an exclusive list or authorative statement about supported MySQL flavors / versions. Please consult the official support / system requirements statement.

Please follow the upstream docs for your preferred flavor to get the software installed on your system.

* MariaDB (10.1, 10.2): https://downloads.mariadb.org/
* Oracle MySQL Community Server (5.6, 5.7): https://dev.mysql.com/downloads/mysql/

Make sure to doublecheck the service is not running (or stop it) after installation as we need to perform some reconfigurations.

service mysql stop

=== Configuration ===

MySQL configuration advise is given in our [[My.cnf|MySQL configuration article]]. Please consult that page for configuration information and create configuration files as described there.

Some settings we recommend to change require that the database gets re-initialized. We assume you don't have data there (since we are covering a fresh install) or you have taken a backup for later restore as explained above in the Preparations section.

cd /var/lib/
mv mysql mysql.old.datadir
mkdir mysql
chown mysql.mysql mysql

# mariadb
mysql_install_db
# oracle 5.6
mysql_install_db -u mysql
# oracle 5.7
mysqld --initialize-insecure --user=mysql

(Don't be worried about the <code>insecure</code>, it just means we set the db root pw in the next steps.)

Start the service, and run <code>mysql_secure_installation</code> for a "secure by default" installation:

service mysql start
mysql_secure_installation

Make sure the service is enabled by the OS's init system.

The further steps in this guide omit <code>-u -p</code> arguments to the MySQL client. Rather than passing them on the command line [https://dev.mysql.com/doc/refman/5.7/en/password-security-user.html] it is recommended to place the credentials in a file like <code>/root/.my.cnf</code> like

[client]
user=root
password=wip9Phae3Beijeed

You should now be able to restore your previously taken backup.

# If you took a dump for restore before
mysql < backup.sql

=== Configure OX to use with a standalone database ===

Not much special wisdom here. OX was designed to be used with master/slave databases, and a standalone master works just as well, if we register it as a master, and not registering a slave.

For the ConfigDB, <code>configdb.properties</code> allows configuration of a <code>writeUrl</code> (which is set to the correct values if you use <code>oxinstaller</code> with the correct argument <code>--configdb-writehost</code>).

The single database is then used for reading and writing.

For the individiual UserDBs, use <code>registerdatabase -m true</code>.

== Galera database setup ==

=== Preparations ===

Our configuration process includes wiping and reinitializing the datadir. This is usually not a problem in a fresh installation. If you want to upgrade an existing database to Galera cluster, please be prepared to wipe the datadir, i.e. take a <code>mysqldump</code> for later restoration into the properly configured master.

Depeding on the flavor of the current database, this can be something like

# mariadb or oracle mysql without GTIDs
mysqldump --databases configdb oxdb_{5..14} > backup.sql

# mysql 5.6 with GTIDs... we dont want GTIDs here
mysqldump --databases --set-gtid-purged=OFF configdb oxdb_{5..14} > backup.sql

Be sure to verify the list of databases.

=== Installation ===

Please follow the upstream docs for your preferred flavor to get the software installed on your system.

* Percona XtraDB Cluster (5.6, 5.7): https://www.percona.com/doc/percona-xtradb-cluster/LATEST/install/index.html
* MariaDB Galera Cluster (10.0, 10.1): https://mariadb.com/kb/en/library/getting-started-with-mariadb-galera-cluster/ (Note: with 10.0, socat is required, but not a package dependency, so you need to explicitly install also socat)

Make sure to doublecheck the service is not running (or stop it) after installation as we need to perform some reconfigurations.

service mysql stop

=== Configuration ===

Galera-specific MySQL configuration advise is included in our main [[My.cnf|MySQL configuration article]]. Please consult that page for configuration information.

That page suggests a setup were we add three custom config files to <code>/etc/mysql/ox.conf.d/</pre>: <code>ox.cnf</code> for general tuning/sizing, <code>wsrep.cnf</code> for clusterwide galera configuration, and <code>host.cnf</code> for host-specific settings.

Adjust the general settings and tunings in <code>ox.cnf</code> according to your sizing etc.

Adjust <code>wsrep.cnf</code> to reflect local paths, cluster member addresses, etc.

Adjust <code>host.cnf</code> to give node-local IPs, etc.

Version-specific hints:

# percona 5.6: unknown variable 'pxc_strict_mode=ENFORCING' ... unset that one
# mariadb 10.1: add wsrep_on=ON
# mariadb 10.0 and 10.1: set wsrep_node_incoming_address=192.168.1.22:3306 in host.cnf, otherwise the status wsrep_incoming_addresses might not be shown correctly(?!)

Some settings we recommend to change require that the database gets re-initialized. We assume you don't have data there (since we are covering a fresh install) or you have taken a backup for later restore as explained above in the Preparations section.

cd /var/lib/
mv mysql mysql.old.datadir
mkdir mysql
chown mysql.mysql mysql

# mariadb 10.0 and 10.1
mysql_install_db
# percona 5.6
mysqld --user=mysql
# percona 5.7
mysqld --initialize-insecure --user=mysql

(Don't be worried about the <code>insecure</code>, it just means we set the db root pw in the next steps.)

=== Cluster startup ===

Typically on startup a Galera node tries to join a cluster, and if it fails, it will exit. Thus, when no cluster nodes are running, the first cluster node to be started needs to be told to not try to join a cluster, but rather bootstrap a new cluster. The exact arguments vary from version to version and from flavor to flavor.

==== First node ====

So we initialize the cluster bootstrap on the first node:

# percona 5.6, 5.7
service mysql bootstrap-pxc
# mariadb 10.0
service mysql bootstrap
# mariadb 10.1: service mysql bootstrap seems to be broken, does not pass the necessary options to mysqld
galera_new_cluster

Run <code>mysql_secure_installation</code> for a "secure by default" installation:

mysql_secure_installation

The further steps in this guide omit <code>-u -p</code> arguments to the MySQL client. Rather than passing them on the command line [https://dev.mysql.com/doc/refman/5.7/en/password-security-user.html] it is recommended to place the credentials in a file like <code>/root/.my.cnf</code> like

[client]
user=root
password=wip9Phae3Beijeed

We need a Galera replication user:

CREATE USER 'sstuser'@'localhost' IDENTIFIED BY 'OpIdjijwef0';
-- percona 5.6, mariadb 10.0
GRANT RELOAD, LOCK TABLES, REPLICATION CLIENT ON *.* TO 'sstuser'@'localhost';
-- percona 5.7, mariadb 10.1
GRANT PROCESS, RELOAD, LOCK TABLES, REPLICATION CLIENT ON *.* TO 'sstuser'@'localhost';
FLUSH PRIVILEGES;

(Debian specific note: MariaDB provided startup scripts use the distro's mechanism of verifying startup/shutdown using a system user, so we create that as well:

# mariadb 10.0, 10.1
GRANT ALL PRIVILEGES ON *.* TO "debian-sys-maint"@"localhost" IDENTIFIED BY "adBexthTsI5TaEps";

If you do this, yo need to synchronize the <code>/etc/mysql/debian.cnf</code> file from the first node to the other nodes as well.)

==== Other nodes ====

On the other nodes, we only need to restart the service now, to trigger a full state transfer from the first node to the other nodes.

We recommend to do this serially to let one state transfer complete before the second state transfer.

==== First node (continued) ====

Only applicable if you used <code>galera_new_cluster</code> before rather than the service script: In order to get the systemctl status consistent, restart the service on the first node:

# mariadb 10.1: restart the service so that the systemctl status is consistent
mysqladmin shutdown
service mysql bootstrap

=== Verify the replication ===

The key tool to verify replication status is

mysql> show status like "%wsrep%";

This will give a lot of output. You want to verify in particular

+------------------------------+--------------------------------------+
| Variable_name | Value |
+------------------------------+--------------------------------------+
| wsrep_cluster_size | 3 |
| wsrep_cluster_status | Primary |
| wsrep_local_state | 4 |
| wsrep_local_state_comment | Synced |
| wsrep_ready | ON |
+------------------------------+--------------------------------------+

You can also explicitly verify replication by creating / inserting DBs, tables, rows on one node and select on other nodes.

==== Troubleshooting ====

The logs are helpful. Always.

Common mistakes are listed below.

If the Galera module does not get loaded at all:
* Configuration settings in ''my.cnf'' which are incompatible to Galera
* Wrong path of the shared object providing the Galera plugin in wsrep.cnf (wsrep_provider)

If the first node starts, but the second / third nodes can not be added to the cluster:
* User for the replication not created correctly on the first Galera node
* SST fails due to missing / wrong version prerequisite packages (not everything is hardcoded in package dependencies -- make sure you got percona-xtrabackup installed in the correct version, and also socat). If SST fails, do not only look into mysqls primary error logs, but also into logfiles from the SST tool in /var/lib/mysql on the donor node.

=== Notes about configuring OX for use with Galera ===

==== Write requests ====

Open-Xchange supports Galera as database backend only in the configuration where all writes are directed to one Galera node. For availability, it makes sense to not configure one Galera node's IP address directly, but rather employ some HA solution which offers active-passive functionality. Options therefore are discussed below.

==== Read requests ====

Read requests can be directed to any node in the Galera cluster. Our standard approach is to recommend to use a loadbalancer to implement round-robin over all nodes in a Galera cluster for the read requests. But you can also chose to use a dedicated read node (the same node, or a different node, than the write node). Each of the approaches has its own advantages.

* Load balancer based setup: Read requests get distributed round-robin between the Galera nodes. Theoretically by distributing the load of the read requests, you benefit from lower latencies and more throughput. But this has never been benchmarked yet. For a discussion of available loadbalances, see next section. OX-wise, in this configuration, you have two alternatives:
** The Galera option wsrep_causal_reads=1 option enables you to configure OX with its replication monitor disabled (com.openexchange.database.replicationMonitor=false in configdb.properties). This is the setup which seems to perform best according to our experience as turning off the replication monitor reduces the commits on the DB and thus the write operations per second on the underlying storage significantly, which outweights the drawback from having higher commit latency due to fully synchronous mode.
** Alternatively, you can run Galera with wsrep_causal_reads=0 when switching on OX builtin replication monitor. This is also a valid setup.
* Use a designated floating IP for the read requests: This eliminates the need of a load balancer. With this option you will not gain any performance, but the quantitative benefit is unclear anyhow.
* Use the floating IP for the writes also for the reads: In this scenario, you direct all database queries only to one Galera node, and the other two nodes are only getting queries in case of a failure of that node. In this case, you can even use wsrep_causal_reads=0 while still having OX builtin replication monitor switched off. However we do not expect this option to be superior to the round-robin loadbalancer approach.

=== Loadbalancer options ===

While the JDBC driver has some round-robin load balancing capabilities built-in, we don't recommend it for production use since it lacks possibilities to check the Galera nodes health states.

Loadbalancers used for OX -> Galera loadbalancing should be able to implement active-passive instances for the write requests, and active-active (round-robin) instances for the read requests. (If they cannot implement active-passive, you can still take a floating IP therefore.) Furthermore it is required to configure node health checks not only on the TCP level (by a simple connect), but to query the Galera health status periodically, evaluating Galera WSREP status variables. Otherwise split-brain scenarios or other bad states cannot be detected. For an example of such an health check, see our [[Clustercheck]] page.

Some customers use loadbalancing appliances. It is important to check that if the (virtual) infrastructure offers "loadbalancer" instances that they satisfy the given requirements. Often this is not the case. In particular, a simple "DNS round robin" approach is not viable.

==== LVS/ipvsadm/keepalived ====

If you want to create your own loadbalancers based on Linux, we usually recommend LVS (Linux Virtual Servers) controlled by Keepalived. LVS is a set of kernel modules implementing a L4 loadbalancer which performs quite well. Keepalived is a userspace daemon to control LVS rules, using health checks to reconfigure LVS rules if required. Keepalived / LVS requires one (or, for availability, two) dedicated linux nodes to run on. This can be a disadvantage for some installations, but usually, it pays off. We provide some configuration information on Keepalived [[Keepalived|here]].

==== MariaDB Maxscale ====

Since Maxscale has become GA in 2015, it seems to have undergone significant stability, performance and functional improvements. We are currently experimenting with Maxscale and share our installation / configuration knowledge [[Maxscale|here]]. It looks quite promising and might become ''the standard replacement'' for HAproxy, while we still presume Keepalived offers superior robustness and performance, coming with the cost of the requirement for one (or more) dedicated loadbalancer nodes.

==== HAproxy ====

In case where the Keepalived based approach is not feasible due to its requirements on the infrastructure, it is also possible to use a HAproxy based solution where HAproxy processes run on each of the OX nodes, configured for one round-robin and one active/passive instance. OX is then connecting to the local HAproxy instances. It is vital to configure HAproxy timeouts different from the defaults, otherwise HAproxy will kill active DB connections, causing errors. Be aware that in large installations the number of (distributed) HAproxy instances can get quite large. Some configuration hints for HAproxy are available [[HAproxy|here]].

== Master/Slave database setup ==

While we also support also "legacy" (pre-GTID) Master/Slave replication, we recommend to use GTID based replication, for easier setup and failure recovery. Support for GTID based replication has been added with OX 7.8.0.

GTID has been available since MySQL 5.6, so no 5.5 installation instructions below, sorry. We try to be generic in this documentation (thus, applicable to Oracle Community Edition and MariaDB) and point out differences where needed. Note: Instructions below include information about Oracle Community MySQL 5.7 which is not yet formally supported.

=== Preparations ===

Our configuration process includes wiping and reinitializing the datadir. This is usually not a problem in a fresh installation. If you want to upgrade an existing database to GTID master-slave, please be prepared to wipe the datadir, i.e. take a <code>mysqldump</code> for later restoration into the properly configured master.

Depeding on the flavor of the current database, this can be something like

# mariadb or oracle mysql without GTIDs
mysqldump --databases configdb oxdb_{5..14} > backup.sql

# mysql 5.6 with GTIDs... we dont want GTIDs here
mysqldump --databases --set-gtid-purged=OFF configdb oxdb_{5..14} > backup.sql

Be sure to verify the list of databases.

=== Installation ===

Software installation is identical for master and slave.

Please follow the instructions for installing from The vendors.

* Oracle Community Edition: https://dev.mysql.com/doc/mysql-apt-repo-quick-guide/en/
* MariaDB (10.0, 10.1): https://downloads.mariadb.org/mariadb/repositories/

Stop the service (if it is running):

service mysql stop

=== Configuration ===

Configuration as per configuration files is also identical for master and slave.

Consult [[My.cnf]] for general recommendations how to configure databases for usage with OX.

For GTID based replication, make sure you add some configurables to a new <code>/etc/mysql/ox.conf.d/gtid.cnf</code> file (assuming you are following our proposed schema of adding a <code>!includedir /etc/mysql/ox.conf.d/</code>" directive to <code>/etc/mysql/my.cnf</code>):

# GTID
log-bin=mysql-bin
server-id=...
log_slave_updates = ON

Oracle Community Edition: we need to add also

enforce_gtid_consistency = ON
gtid_mode = ON

(GTID mode is on by default on MariaDB.)

Use unique a <code>server-id</code> for each server; like <code>1</code> for the master, <code>2</code> for slave. For more complicated setups (like multiple slaves), adjust accordingly.

Since applying our configuration / sizing requires reinitialization of the MySQL datadir, we wipe/recreate it. Caution: this assumes we are running an empty database. If there is data in the database you want to keep, use mysqldump. See Preparation section above.

So, to initialize the datadir:

cd /var/lib/
mv mysql mysql.old.datadir
mkdir mysql
chown mysql.mysql mysql

(When coming from an existing installation, be sure to wipe also old binlogs. They can confuse the server on startup. Their location varies by configuration.)

The step to initialize the datadir is different for the different DBs:

# MariaDB 10.0, 10.1
mysql_install_db

# Oracle 5.6
mysql_install_db -u mysql

# Oracle 5.7
mysqld --initialize-insecure --user=mysql

(Don't be worried about the <code>insecure</code>, it just means we set the db root pw in the next steps.)

Then:

service mysql restart
mysql_secure_installation

We want to emphasize the last step to run "secure".

Steps up to here apply to both the designated master and slave. The next steps will apply to the master.

=== Replication Setup ===

==== Master Setup ====

Create a replication user on the master (but, as always, pick your own password, and use the same password in the slave setup below):

mysql -e "CREATE USER 'repl'@'gtid-slave.localdomain' IDENTIFIED BY 'IvIjyoffod2'; GRANT REPLICATION SLAVE ON *.* TO 'repl'@'gtid-slave.localdomain';"

Now would also be the time to restore a previously created mysqldump, or add other users you need for adminstration, monitoring etc (like <code>debian-sys-maint@localhost</code>, for example). Adding the OX users is explained below ("Creating Open-Xchange user").

# If you took a dump for restore before
mysql < backup.sql

To prepare for the initial sync of the slave, set the master read-only:

mysql -e "SET @@global.read_only = ON;"

Create a dump to initialize the slave:

# MariaDB
mysqldump --all-databases --triggers --routines --events --master-data --gtid > master.sql

# Oracle
mysqldump --all-databases --triggers --routines --events --set-gtid-purged=ON > master.sql

Transfer to the slave:

scp master.sql gtid-slave:

==== Slave Setup ====

Configure the replication master settings. Note we don't need complicated binlog position settings etc with GTID.

Yet again DB-specific (use the repl user password from above):

# MariaDB
mysql -e 'CHANGE MASTER TO MASTER_HOST="gtid-master.localdomain", MASTER_USER="repl", MASTER_PASSWORD="IvIjyoffod2";'

# Oracle
mysql -e "CHANGE MASTER TO MASTER_HOST='gtid-master.localdomain', MASTER_USER='repl', MASTER_PASSWORD='IvIjyoffod2', MASTER_AUTO_POSITION=1;"
# https://www.percona.com/blog/2013/02/08/how-to-createrestore-a-slave-using-gtid-replication-in-mysql-5-6/
mysql -e "RESET MASTER;"

Read the master dump:

mysql < master.sql

Start replication on the slave:

mysql -e 'START SLAVE;'
mysql -e 'SHOW SLAVE STATUS\G'

==== Master Setup (continued) ====

Finally, unset read-only on the master:

# on the master
mysql -e "SET @@global.read_only = OFF;"

=== Configure OX to use with Master/Slave replication ===

Not much special wisdom here. OX was designed to be used with master/slave databases. For the ConfigDB, <code>configdb.properties</code> allows configuration of a <code>readUrl</code> and <code>writeUrl</code> (both of which are set to the correct values if you use <code>oxinstaller</code> with the correct arguments <code>--configdb-readhost</code>, <code>--configdb-writehost</code>).

(Obviously, the master is for writing and the slave is for reading.)

For the individiual UserDBs, use <code>registerdatabase -m true</code> for the masters and <code>registerdatabase -m false -M ...</code> for the respective slaves.

Be sure to have enabled the replication monitor in <code>configdb.properties</code>: <code>com.openexchange.database.replicationMonitor=true</code> (which it is by default); while GTID can show synchronous semantics, it is specified to silently fall back to asynchronous in certain circumstances, so synchronity is not guaranteed.

We recommend, though, to not register the databases directly by their native hostname or IP, but rather use some kind of HA system in order to be able to easily move a floating/failover IP from the master to the slave in case of master failure. Configuring and running such systems (like, corosync/pacemaker, keepalived, or whatever) is out of scope of this documentation, however.

== Creating Open-Xchange user ==

Now setup access for the Open-Xchange Server database user 'openexchange' to configdb and the oxdb for both groupware server addresses. These databases do not exist yet, but will be created during the Open-Xchange Server installation.

Notes:

* Please use a real password.
* The IPs in this example belong to the two different Open-Xchange Servers, please adjust them accordingly.
* If using a database on the same host as the middlware (usually done for POCs and demo installations), you need to grant also to the ''localhost'' host.
* Consult [[AppSuite:DB_user_privileges]] (or ''grep GRANT /opt/open-xchange/sbin/initconfigdb'') for an up-to-date list of required privileges. The following statement was correct as of the time of writing this section.

mysql> GRANT CREATE, LOCK TABLES, REFERENCES, INDEX, DROP, DELETE, ALTER, SELECT, UPDATE, INSERT, CREATE TEMPORARY TABLES, SHOW VIEW, SHOW DATABASES ON *.* TO 'openexchange'@'10.20.30.213' IDENTIFIED BY 'IntyoyntOat1' WITH GRANT OPTION;
mysql> GRANT CREATE, LOCK TABLES, REFERENCES, INDEX, DROP, DELETE, ALTER, SELECT, UPDATE, INSERT, CREATE TEMPORARY TABLES, SHOW VIEW, SHOW DATABASES ON *.* TO 'openexchange'@'10.20.30.215' IDENTIFIED BY 'IntyoyntOat1' WITH GRANT OPTION;

Template:OXConfiguration

2018-06-05T12:36:07Z

Dominik.epple: /* Open-Xchange configuration */

= Open-Xchange configuration =

To avoid confusion right at the start notice that Open-Xchange uses multiple administration levels and requires different credentials at some stages at the installation and server management. Note that the passwords chosen at this guide are weak and should be replaced by stronger passwords.
* The MySQL database user
** Username: openexchange
** Password used at this guide: db_password
** Responsibility: Execute all kinds of database operations

* The Open-Xchange Admin Master
** Username: oxadminmaster
** Password used at this guide: admin_master_password
** Responsibility: Manage contexts, manage all kinds of low level server configuration

* The Context Admin
** Username: oxadmin
** Password used at this guide: admin_password
** Responsibility: Manage users/groups/resources inside a context

As stated above we assume the MySQL service has been installed previously, and it is running and available.

A good idea is to add the Open-Xchange binaries to PATH:

$ echo PATH=$PATH:/opt/open-xchange/sbin/ >> ~/.bashrc && . ~/.bashrc

Now we have to initialize the Open-Xchange ''configdb'' database. This can all be done by executing the ''initconfigdb'' script.

$ /opt/open-xchange/sbin/initconfigdb --configdb-pass=db_password -a --mysql-root-passwd=root_password

Use the ''--mysql-root-passwd'' option to supply the MySQL root password as configured during database installation.

Add the ''-i'' option if you want to remove an already existing open-xchange configdb.

'''Note:''' The -a parameter adds an ''openexchange'' account to MySQL. This account will be used for database connections from the OX App Suite middleware and requires ALL PRIVILEGES. You can also create that account manually [[OXLoadBalancingClustering_Database#Creating_Open-Xchange_user|during database installation / configuration]], in which case you can skip the ''-a'' parameter here.

Before starting any service, all basic configuration files need to be set up correctly. The ''--configdb-pass'' option indicates the password of the ''openexchange'' database user previously created, the ''--master-pass'' options specifies the password of the Open-Xchange ''adminmaster'' user that will be created when executing the ''oxinstaller'' script.

Template:QuickInstIntro

2018-06-05T10:27:33Z

Dominik.epple:

{{#ifeq:{{{release|}}}|6|
This article will guide you through the installation of the Open-Xchange server, it describes the basic configuration and software requirements. As it is intended as a quick walk-through it assumes an existing installation of the operating system and requires average system administration skills. More, this guide will show you how to setup a basic installation with none of the typically used distributed environment settings. The objective of this guide is:

* To setup a single server installation
* To setup a database for a single database service, no replication
* To setup a single Open-Xchange instance, no cluster
* To provide a basic configuration setup, no mailserver configuration
}}
{{#ifeq:{{{release|}}}|edp|
This article will guide you through the installation of the Open-Xchange server, it describes the basic configuration and software requirements. As it is intended as a quick walk-through it assumes an existing installation of the operating system and requires average system administration skills. More, this guide will show you how to setup a basic installation with none of the typically used distributed environment settings. The objective of this guide is:

* To setup a single server installation
* To setup a database for a single database service, no replication
* To setup a single Open-Xchange instance, no cluster
* To provide a basic configuration setup, no mailserver configuration
}}
{{#ifeq:{{{release|}}}|appsuite|
This article will guide you through the installation of OX App Suite, it describes the basic configuration and software requirements. As it is intended as a quick walk-through it assumes an existing installation of the operating system and requires average system administration skills. More, this guide will show you how to setup a basic installation with none of the typically used distributed environment settings. The objective of this guide is:

* To setup a single server installation
* To setup a database for a single database service, no replication
* To setup a single Open-Xchange instance, no cluster
* To provide a basic configuration setup, no mailserver configuration
}}

Template:OXLoadBalancingClustering Database

2018-06-05T10:23:01Z

Dominik.epple: /* Standalone database setup */

== Overview ==

You can choose between Galera or Master/Slave replication. We like to recommend to use Galera for higher redudancy, easier operations, und synchronous semantics (so you can run OX without our "replication monitor"). For POC or demo setups, a single standalone database setup might be sufficient.

== Standalone database setup ==

=== Preparations ===

Our configuration process includes wiping and reinitializing the datadir. This is usually not a problem in a fresh installation. If you want to upgrade an existing database server, please be prepared to wipe the datadir, i.e. take a <code>mysqldump</code> for later restoration into the properly configured master.

mysqldump --databases configdb oxdb_{5..14} > backup.sql

Be sure to verify the list of databases.

=== Installation ===

Note: the following list ist not an exclusive list or authorative statement about supported MySQL flavors / versions. Please consult the official support / system requirements statement.

Please follow the upstream docs for your preferred flavor to get the software installed on your system.

* MariaDB (10.1, 10.2): https://downloads.mariadb.org/
* Oracle MySQL Community Server (5.6, 5.7): https://dev.mysql.com/downloads/mysql/

Make sure to doublecheck the service is not running (or stop it) after installation as we need to perform some reconfigurations.

service mysql stop

=== Configuration ===

MySQL configuration advise is given in our [[My.cnf|MySQL configuration article]]. Please consult that page for configuration information and create configuration files as described there.

Some settings we recommend to change require that the database gets re-initialized. We assume you don't have data there (since we are covering a fresh install) or you have taken a backup for later restore as explained above in the Preparations section.

cd /var/lib/
mv mysql mysql.old.datadir
mkdir mysql
chown mysql.mysql mysql

# mariadb
mysql_install_db
# oracle 5.6
mysql_install_db -u mysql
# oracle 5.7
mysqld --initialize-insecure --user=mysql

(Don't be worried about the <code>insecure</code>, it just means we set the db root pw in the next steps.)

Start the service, and run <code>mysql_secure_installation</code> for a "secure by default" installation:

service mysql start
mysql_secure_installation

Make sure the service is enabled by the OS's init system.

The further steps in this guide omit <code>-u -p</code> arguments to the MySQL client. Rather than passing them on the command line [https://dev.mysql.com/doc/refman/5.7/en/password-security-user.html] it is recommended to place the credentials in a file like <code>/root/.my.cnf</code> like

[client]
user=root
password=wip9Phae3Beijeed

You should now be able to restore your previously taken backup.

# If you took a dump for restore before
mysql < backup.sql

=== Configure OX to use with a standalone database ===

Not much special wisdom here. OX was designed to be used with master/slave databases, and a standalone master works just as well, if we register it as a master, and not registering a slave.

For the ConfigDB, <code>configdb.properties</code> allows configuration of a <code>writeUrl</code> (which is set to the correct values if you use <code>oxinstaller</code> with the correct argument <code>--configdb-writehost</code>).

The single database is then used for reading and writing.

For the individiual UserDBs, use <code>registerdatabase -m true</code>.

== Galera database setup ==

=== Preparations ===

Our configuration process includes wiping and reinitializing the datadir. This is usually not a problem in a fresh installation. If you want to upgrade an existing database to Galera cluster, please be prepared to wipe the datadir, i.e. take a <code>mysqldump</code> for later restoration into the properly configured master.

Depeding on the flavor of the current database, this can be something like

# mariadb or oracle mysql without GTIDs
mysqldump --databases configdb oxdb_{5..14} > backup.sql

# mysql 5.6 with GTIDs... we dont want GTIDs here
mysqldump --databases --set-gtid-purged=OFF configdb oxdb_{5..14} > backup.sql

Be sure to verify the list of databases.

=== Installation ===

Please follow the upstream docs for your preferred flavor to get the software installed on your system.

* Percona XtraDB Cluster (5.6, 5.7): https://www.percona.com/doc/percona-xtradb-cluster/LATEST/install/index.html
* MariaDB Galera Cluster (10.0, 10.1): https://mariadb.com/kb/en/library/getting-started-with-mariadb-galera-cluster/ (Note: with 10.0, socat is required, but not a package dependency, so you need to explicitly install also socat)

Make sure to doublecheck the service is not running (or stop it) after installation as we need to perform some reconfigurations.

service mysql stop

=== Configuration ===

Galera-specific MySQL configuration advise is included in our main [[My.cnf|MySQL configuration article]]. Please consult that page for configuration information.

That page suggests a setup were we add three custom config files to <code>/etc/mysql/ox.conf.d/</pre>: <code>ox.cnf</code> for general tuning/sizing, <code>wsrep.cnf</code> for clusterwide galera configuration, and <code>host.cnf</code> for host-specific settings.

Adjust the general settings and tunings in <code>ox.cnf</code> according to your sizing etc.

Adjust <code>wsrep.cnf</code> to reflect local paths, cluster member addresses, etc.

Adjust <code>host.cnf</code> to give node-local IPs, etc.

Version-specific hints:

# percona 5.6: unknown variable 'pxc_strict_mode=ENFORCING' ... unset that one
# mariadb 10.1: add wsrep_on=ON
# mariadb 10.0 and 10.1: set wsrep_node_incoming_address=192.168.1.22:3306 in host.cnf, otherwise the status wsrep_incoming_addresses might not be shown correctly(?!)

Some settings we recommend to change require that the database gets re-initialized. We assume you don't have data there (since we are covering a fresh install) or you have taken a backup for later restore as explained above in the Preparations section.

cd /var/lib/
mv mysql mysql.old.datadir
mkdir mysql
chown mysql.mysql mysql

# mariadb 10.0 and 10.1
mysql_install_db
# percona 5.6
mysqld --user=mysql
# percona 5.7
mysqld --initialize-insecure --user=mysql

(Don't be worried about the <code>insecure</code>, it just means we set the db root pw in the next steps.)

=== Cluster startup ===

Typically on startup a Galera node tries to join a cluster, and if it fails, it will exit. Thus, when no cluster nodes are running, the first cluster node to be started needs to be told to not try to join a cluster, but rather bootstrap a new cluster. The exact arguments vary from version to version and from flavor to flavor.

==== First node ====

So we initialize the cluster bootstrap on the first node:

# percona 5.6, 5.7
service mysql bootstrap-pxc
# mariadb 10.0
service mysql bootstrap
# mariadb 10.1: service mysql bootstrap seems to be broken, does not pass the necessary options to mysqld
galera_new_cluster

Run <code>mysql_secure_installation</code> for a "secure by default" installation:

mysql_secure_installation

The further steps in this guide omit <code>-u -p</code> arguments to the MySQL client. Rather than passing them on the command line [https://dev.mysql.com/doc/refman/5.7/en/password-security-user.html] it is recommended to place the credentials in a file like <code>/root/.my.cnf</code> like

[client]
user=root
password=wip9Phae3Beijeed

We need a Galera replication user:

CREATE USER 'sstuser'@'localhost' IDENTIFIED BY 'OpIdjijwef0';
-- percona 5.6, mariadb 10.0
GRANT RELOAD, LOCK TABLES, REPLICATION CLIENT ON *.* TO 'sstuser'@'localhost';
-- percona 5.7, mariadb 10.1
GRANT PROCESS, RELOAD, LOCK TABLES, REPLICATION CLIENT ON *.* TO 'sstuser'@'localhost';
FLUSH PRIVILEGES;

(Debian specific note: MariaDB provided startup scripts use the distro's mechanism of verifying startup/shutdown using a system user, so we create that as well:

# mariadb 10.0, 10.1
GRANT ALL PRIVILEGES ON *.* TO "debian-sys-maint"@"localhost" IDENTIFIED BY "adBexthTsI5TaEps";

If you do this, yo need to synchronize the <code>/etc/mysql/debian.cnf</code> file from the first node to the other nodes as well.)

==== Other nodes ====

On the other nodes, we only need to restart the service now, to trigger a full state transfer from the first node to the other nodes.

We recommend to do this serially to let one state transfer complete before the second state transfer.

==== First node (continued) ====

Only applicable if you used <code>galera_new_cluster</code> before rather than the service script: In order to get the systemctl status consistent, restart the service on the first node:

# mariadb 10.1: restart the service so that the systemctl status is consistent
mysqladmin shutdown
service mysql bootstrap

=== Verify the replication ===

The key tool to verify replication status is

mysql> show status like "%wsrep%";

This will give a lot of output. You want to verify in particular

+------------------------------+--------------------------------------+
| Variable_name | Value |
+------------------------------+--------------------------------------+
| wsrep_cluster_size | 3 |
| wsrep_cluster_status | Primary |
| wsrep_local_state | 4 |
| wsrep_local_state_comment | Synced |
| wsrep_ready | ON |
+------------------------------+--------------------------------------+

You can also explicitly verify replication by creating / inserting DBs, tables, rows on one node and select on other nodes.

==== Troubleshooting ====

The logs are helpful. Always.

Common mistakes are listed below.

If the Galera module does not get loaded at all:
* Configuration settings in ''my.cnf'' which are incompatible to Galera
* Wrong path of the shared object providing the Galera plugin in wsrep.cnf (wsrep_provider)

If the first node starts, but the second / third nodes can not be added to the cluster:
* User for the replication not created correctly on the first Galera node
* SST fails due to missing / wrong version prerequisite packages (not everything is hardcoded in package dependencies -- make sure you got percona-xtrabackup installed in the correct version, and also socat). If SST fails, do not only look into mysqls primary error logs, but also into logfiles from the SST tool in /var/lib/mysql on the donor node.

=== Notes about configuring OX for use with Galera ===

==== Write requests ====

Open-Xchange supports Galera as database backend only in the configuration where all writes are directed to one Galera node. For availability, it makes sense to not configure one Galera node's IP address directly, but rather employ some HA solution which offers active-passive functionality. Options therefore are discussed below.

==== Read requests ====

Read requests can be directed to any node in the Galera cluster. Our standard approach is to recommend to use a loadbalancer to implement round-robin over all nodes in a Galera cluster for the read requests. But you can also chose to use a dedicated read node (the same node, or a different node, than the write node). Each of the approaches has its own advantages.

* Load balancer based setup: Read requests get distributed round-robin between the Galera nodes. Theoretically by distributing the load of the read requests, you benefit from lower latencies and more throughput. But this has never been benchmarked yet. For a discussion of available loadbalances, see next section. OX-wise, in this configuration, you have two alternatives:
** The Galera option wsrep_causal_reads=1 option enables you to configure OX with its replication monitor disabled (com.openexchange.database.replicationMonitor=false in configdb.properties). This is the setup which seems to perform best according to our experience as turning off the replication monitor reduces the commits on the DB and thus the write operations per second on the underlying storage significantly, which outweights the drawback from having higher commit latency due to fully synchronous mode.
** Alternatively, you can run Galera with wsrep_causal_reads=0 when switching on OX builtin replication monitor. This is also a valid setup.
* Use a designated floating IP for the read requests: This eliminates the need of a load balancer. With this option you will not gain any performance, but the quantitative benefit is unclear anyhow.
* Use the floating IP for the writes also for the reads: In this scenario, you direct all database queries only to one Galera node, and the other two nodes are only getting queries in case of a failure of that node. In this case, you can even use wsrep_causal_reads=0 while still having OX builtin replication monitor switched off. However we do not expect this option to be superior to the round-robin loadbalancer approach.

=== Loadbalancer options ===

While the JDBC driver has some round-robin load balancing capabilities built-in, we don't recommend it for production use since it lacks possibilities to check the Galera nodes health states.

Loadbalancers used for OX -> Galera loadbalancing should be able to implement active-passive instances for the write requests, and active-active (round-robin) instances for the read requests. (If they cannot implement active-passive, you can still take a floating IP therefore.) Furthermore it is required to configure node health checks not only on the TCP level (by a simple connect), but to query the Galera health status periodically, evaluating Galera WSREP status variables. Otherwise split-brain scenarios or other bad states cannot be detected. For an example of such an health check, see our [[Clustercheck]] page.

Some customers use loadbalancing appliances. It is important to check that if the (virtual) infrastructure offers "loadbalancer" instances that they satisfy the given requirements. Often this is not the case. In particular, a simple "DNS round robin" approach is not viable.

==== LVS/ipvsadm/keepalived ====

If you want to create your own loadbalancers based on Linux, we usually recommend LVS (Linux Virtual Servers) controlled by Keepalived. LVS is a set of kernel modules implementing a L4 loadbalancer which performs quite well. Keepalived is a userspace daemon to control LVS rules, using health checks to reconfigure LVS rules if required. Keepalived / LVS requires one (or, for availability, two) dedicated linux nodes to run on. This can be a disadvantage for some installations, but usually, it pays off. We provide some configuration information on Keepalived [[Keepalived|here]].

==== MariaDB Maxscale ====

Since Maxscale has become GA in 2015, it seems to have undergone significant stability, performance and functional improvements. We are currently experimenting with Maxscale and share our installation / configuration knowledge [[Maxscale|here]]. It looks quite promising and might become ''the standard replacement'' for HAproxy, while we still presume Keepalived offers superior robustness and performance, coming with the cost of the requirement for one (or more) dedicated loadbalancer nodes.

==== HAproxy ====

In case where the Keepalived based approach is not feasible due to its requirements on the infrastructure, it is also possible to use a HAproxy based solution where HAproxy processes run on each of the OX nodes, configured for one round-robin and one active/passive instance. OX is then connecting to the local HAproxy instances. It is vital to configure HAproxy timeouts different from the defaults, otherwise HAproxy will kill active DB connections, causing errors. Be aware that in large installations the number of (distributed) HAproxy instances can get quite large. Some configuration hints for HAproxy are available [[HAproxy|here]].

== Master/Slave database setup ==

While we also support also "legacy" (pre-GTID) Master/Slave replication, we recommend to use GTID based replication, for easier setup and failure recovery. Support for GTID based replication has been added with OX 7.8.0.

GTID has been available since MySQL 5.6, so no 5.5 installation instructions below, sorry. We try to be generic in this documentation (thus, applicable to Oracle Community Edition and MariaDB) and point out differences where needed. Note: Instructions below include information about Oracle Community MySQL 5.7 which is not yet formally supported.

=== Preparations ===

Our configuration process includes wiping and reinitializing the datadir. This is usually not a problem in a fresh installation. If you want to upgrade an existing database to GTID master-slave, please be prepared to wipe the datadir, i.e. take a <code>mysqldump</code> for later restoration into the properly configured master.

Depeding on the flavor of the current database, this can be something like

# mariadb or oracle mysql without GTIDs
mysqldump --databases configdb oxdb_{5..14} > backup.sql

# mysql 5.6 with GTIDs... we dont want GTIDs here
mysqldump --databases --set-gtid-purged=OFF configdb oxdb_{5..14} > backup.sql

Be sure to verify the list of databases.

=== Installation ===

Software installation is identical for master and slave.

Please follow the instructions for installing from The vendors.

* Oracle Community Edition: https://dev.mysql.com/doc/mysql-apt-repo-quick-guide/en/
* MariaDB (10.0, 10.1): https://downloads.mariadb.org/mariadb/repositories/

Stop the service (if it is running):

service mysql stop

=== Configuration ===

Configuration as per configuration files is also identical for master and slave.

Consult [[My.cnf]] for general recommendations how to configure databases for usage with OX.

For GTID based replication, make sure you add some configurables to a new <code>/etc/mysql/ox.conf.d/gtid.cnf</code> file (assuming you are following our proposed schema of adding a <code>!includedir /etc/mysql/ox.conf.d/</code>" directive to <code>/etc/mysql/my.cnf</code>):

# GTID
log-bin=mysql-bin
server-id=...
log_slave_updates = ON

Oracle Community Edition: we need to add also

enforce_gtid_consistency = ON
gtid_mode = ON

(GTID mode is on by default on MariaDB.)

Use unique a <code>server-id</code> for each server; like <code>1</code> for the master, <code>2</code> for slave. For more complicated setups (like multiple slaves), adjust accordingly.

Since applying our configuration / sizing requires reinitialization of the MySQL datadir, we wipe/recreate it. Caution: this assumes we are running an empty database. If there is data in the database you want to keep, use mysqldump. See Preparation section above.

So, to initialize the datadir:

cd /var/lib/
mv mysql mysql.old.datadir
mkdir mysql
chown mysql.mysql mysql

(When coming from an existing installation, be sure to wipe also old binlogs. They can confuse the server on startup. Their location varies by configuration.)

The step to initialize the datadir is different for the different DBs:

# MariaDB 10.0, 10.1
mysql_install_db

# Oracle 5.6
mysql_install_db -u mysql

# Oracle 5.7
mysqld --initialize-insecure --user=mysql

(Don't be worried about the <code>insecure</code>, it just means we set the db root pw in the next steps.)

Then:

service mysql restart
mysql_secure_installation

We want to emphasize the last step to run "secure".

Steps up to here apply to both the designated master and slave. The next steps will apply to the master.

=== Replication Setup ===

==== Master Setup ====

Create a replication user on the master (but, as always, pick your own password, and use the same password in the slave setup below):

mysql -e "CREATE USER 'repl'@'gtid-slave.localdomain' IDENTIFIED BY 'IvIjyoffod2'; GRANT REPLICATION SLAVE ON *.* TO 'repl'@'gtid-slave.localdomain';"

Now would also be the time to restore a previously created mysqldump, or add other users you need for adminstration, monitoring etc (like <code>debian-sys-maint@localhost</code>, for example). Adding the OX users is explained below ("Creating Open-Xchange user").

# If you took a dump for restore before
mysql < backup.sql

To prepare for the initial sync of the slave, set the master read-only:

mysql -e "SET @@global.read_only = ON;"

Create a dump to initialize the slave:

# MariaDB
mysqldump --all-databases --triggers --routines --events --master-data --gtid > master.sql

# Oracle
mysqldump --all-databases --triggers --routines --events --set-gtid-purged=ON > master.sql

Transfer to the slave:

scp master.sql gtid-slave:

==== Slave Setup ====

Configure the replication master settings. Note we don't need complicated binlog position settings etc with GTID.

Yet again DB-specific (use the repl user password from above):

# MariaDB
mysql -e 'CHANGE MASTER TO MASTER_HOST="gtid-master.localdomain", MASTER_USER="repl", MASTER_PASSWORD="IvIjyoffod2";'

# Oracle
mysql -e "CHANGE MASTER TO MASTER_HOST='gtid-master.localdomain', MASTER_USER='repl', MASTER_PASSWORD='IvIjyoffod2', MASTER_AUTO_POSITION=1;"
# https://www.percona.com/blog/2013/02/08/how-to-createrestore-a-slave-using-gtid-replication-in-mysql-5-6/
mysql -e "RESET MASTER;"

Read the master dump:

mysql < master.sql

Start replication on the slave:

mysql -e 'START SLAVE;'
mysql -e 'SHOW SLAVE STATUS\G'

==== Master Setup (continued) ====

Finally, unset read-only on the master:

# on the master
mysql -e "SET @@global.read_only = OFF;"

=== Configure OX to use with Master/Slave replication ===

Not much special wisdom here. OX was designed to be used with master/slave databases. For the ConfigDB, <code>configdb.properties</code> allows configuration of a <code>readUrl</code> and <code>writeUrl</code> (both of which are set to the correct values if you use <code>oxinstaller</code> with the correct arguments <code>--configdb-readhost</code>, <code>--configdb-writehost</code>).

(Obviously, the master is for writing and the slave is for reading.)

For the individiual UserDBs, use <code>registerdatabase -m true</code> for the masters and <code>registerdatabase -m false -M ...</code> for the respective slaves.

Be sure to have enabled the replication monitor in <code>configdb.properties</code>: <code>com.openexchange.database.replicationMonitor=true</code> (which it is by default); while GTID can show synchronous semantics, it is specified to silently fall back to asynchronous in certain circumstances, so synchronity is not guaranteed.

We recommend, though, to not register the databases directly by their native hostname or IP, but rather use some kind of HA system in order to be able to easily move a floating/failover IP from the master to the slave in case of master failure. Configuring and running such systems (like, corosync/pacemaker, keepalived, or whatever) is out of scope of this documentation, however.

== Creating Open-Xchange user ==

Now setup access for the Open-Xchange Server database user 'openexchange' to configdb and the oxdb for both groupware server addresses. These databases do not exist yet, but will be created during the Open-Xchange Server installation.

Note: The IPs in this example belong to the two different Open-Xchange Servers, please adjust them accordingly. And use a real password.

mysql> GRANT ALL PRIVILEGES ON *.* TO 'openexchange'@'10.20.30.213' IDENTIFIED BY 'IntyoyntOat1';
mysql> GRANT ALL PRIVILEGES ON *.* TO 'openexchange'@'10.20.30.215' IDENTIFIED BY 'IntyoyntOat1';

Template:OXLoadBalancingClustering Database

2018-05-16T10:22:32Z

Dominik.epple:

== Overview ==

You can choose between Galera or Master/Slave replication. We like to recommend to use Galera for higher redudancy, easier operations, und synchronous semantics (so you can run OX without our "replication monitor"). For POC or demo setups, a single standalone database setup might be sufficient.

== Standalone database setup ==

=== Preparations ===

Our configuration process includes wiping and reinitializing the datadir. This is usually not a problem in a fresh installation. If you want to upgrade an existing database server, please be prepared to wipe the datadir, i.e. take a <code>mysqldump</code> for later restoration into the properly configured master.

mysqldump --databases configdb oxdb_{5..14} > backup.sql

Be sure to verify the list of databases.

=== Installation ===

Please follow the upstream docs for your preferred flavor to get the software installed on your system.

Note: the Oracle flavor is also valid, but not yet documented here.

* Percona XtraDB (5.6, 5.7): https://www.percona.com/downloads/
* MariaDB Galera Cluster (10.0, 10.1): https://downloads.mariadb.org/

Make sure to doublecheck the service is not running (or stop it) after installation as we need to perform some reconfigurations.

service mysql stop

=== Configuration ===

MySQL configuration advise is given in our [[My.cnf|MySQL configuration article]]. Please consult that page for configuration information and create configuration files as described there.

Some settings we recommend to change require that the database gets re-initialized. We assume you don't have data there (since we are covering a fresh install) or you have taken a backup for later restore as explained above in the Preparations section.

cd /var/lib/
mv mysql mysql.old.datadir
mkdir mysql
chown mysql.mysql mysql

# mariadb 10.0 and 10.1
mysql_install_db
# percona 5.6
mysqld --user=mysql
# percona 5.7
mysqld --initialize-insecure --user=mysql

(Don't be worried about the <code>insecure</code>, it just means we set the db root pw in the next steps.)

Start the service, and run <code>mysql_secure_installation</code> for a "secure by default" installation:

service mysql start
mysql_secure_installation

Make sure the service is enabled by the OS's init system.

The further steps in this guide omit <code>-u -p</code> arguments to the MySQL client. Rather than passing them on the command line [https://dev.mysql.com/doc/refman/5.7/en/password-security-user.html] it is recommended to place the credentials in a file like <code>/root/.my.cnf</code> like

[client]
user=root
password=wip9Phae3Beijeed

You should now be able to restore your previously taken backup.

# If you took a dump for restore before
mysql < backup.sql

=== Configure OX to use with a standalone database ===

Not much special wisdom here. OX was designed to be used with master/slave databases, and a standalone master works just as well, if we register it as a master, and not registering a slave.

For the ConfigDB, <code>configdb.properties</code> allows configuration of a <code>writeUrl</code> (which is set to the correct values if you use <code>oxinstaller</code> with the correct argument <code>--configdb-writehost</code>).

The single database is then used for reading and writing.

For the individiual UserDBs, use <code>registerdatabase -m true</code>.

== Galera database setup ==

=== Preparations ===

Our configuration process includes wiping and reinitializing the datadir. This is usually not a problem in a fresh installation. If you want to upgrade an existing database to Galera cluster, please be prepared to wipe the datadir, i.e. take a <code>mysqldump</code> for later restoration into the properly configured master.

Depeding on the flavor of the current database, this can be something like

# mariadb or oracle mysql without GTIDs
mysqldump --databases configdb oxdb_{5..14} > backup.sql

# mysql 5.6 with GTIDs... we dont want GTIDs here
mysqldump --databases --set-gtid-purged=OFF configdb oxdb_{5..14} > backup.sql

Be sure to verify the list of databases.

=== Installation ===

Please follow the upstream docs for your preferred flavor to get the software installed on your system.

* Percona XtraDB Cluster (5.6, 5.7): https://www.percona.com/doc/percona-xtradb-cluster/LATEST/install/index.html
* MariaDB Galera Cluster (10.0, 10.1): https://mariadb.com/kb/en/library/getting-started-with-mariadb-galera-cluster/ (Note: with 10.0, socat is required, but not a package dependency, so you need to explicitly install also socat)

Make sure to doublecheck the service is not running (or stop it) after installation as we need to perform some reconfigurations.

service mysql stop

=== Configuration ===

Galera-specific MySQL configuration advise is included in our main [[My.cnf|MySQL configuration article]]. Please consult that page for configuration information.

That page suggests a setup were we add three custom config files to <code>/etc/mysql/ox.conf.d/</pre>: <code>ox.cnf</code> for general tuning/sizing, <code>wsrep.cnf</code> for clusterwide galera configuration, and <code>host.cnf</code> for host-specific settings.

Adjust the general settings and tunings in <code>ox.cnf</code> according to your sizing etc.

Adjust <code>wsrep.cnf</code> to reflect local paths, cluster member addresses, etc.

Adjust <code>host.cnf</code> to give node-local IPs, etc.

Version-specific hints:

# percona 5.6: unknown variable 'pxc_strict_mode=ENFORCING' ... unset that one
# mariadb 10.1: add wsrep_on=ON
# mariadb 10.0 and 10.1: set wsrep_node_incoming_address=192.168.1.22:3306 in host.cnf, otherwise the status wsrep_incoming_addresses might not be shown correctly(?!)

Some settings we recommend to change require that the database gets re-initialized. We assume you don't have data there (since we are covering a fresh install) or you have taken a backup for later restore as explained above in the Preparations section.

cd /var/lib/
mv mysql mysql.old.datadir
mkdir mysql
chown mysql.mysql mysql

# mariadb 10.0 and 10.1
mysql_install_db
# percona 5.6
mysqld --user=mysql
# percona 5.7
mysqld --initialize-insecure --user=mysql

(Don't be worried about the <code>insecure</code>, it just means we set the db root pw in the next steps.)

=== Cluster startup ===

Typically on startup a Galera node tries to join a cluster, and if it fails, it will exit. Thus, when no cluster nodes are running, the first cluster node to be started needs to be told to not try to join a cluster, but rather bootstrap a new cluster. The exact arguments vary from version to version and from flavor to flavor.

==== First node ====

So we initialize the cluster bootstrap on the first node:

# percona 5.6, 5.7
service mysql bootstrap-pxc
# mariadb 10.0
service mysql bootstrap
# mariadb 10.1: service mysql bootstrap seems to be broken, does not pass the necessary options to mysqld
galera_new_cluster

Run <code>mysql_secure_installation</code> for a "secure by default" installation:

mysql_secure_installation

The further steps in this guide omit <code>-u -p</code> arguments to the MySQL client. Rather than passing them on the command line [https://dev.mysql.com/doc/refman/5.7/en/password-security-user.html] it is recommended to place the credentials in a file like <code>/root/.my.cnf</code> like

[client]
user=root
password=wip9Phae3Beijeed

We need a Galera replication user:

CREATE USER 'sstuser'@'localhost' IDENTIFIED BY 'OpIdjijwef0';
-- percona 5.6, mariadb 10.0
GRANT RELOAD, LOCK TABLES, REPLICATION CLIENT ON *.* TO 'sstuser'@'localhost';
-- percona 5.7, mariadb 10.1
GRANT PROCESS, RELOAD, LOCK TABLES, REPLICATION CLIENT ON *.* TO 'sstuser'@'localhost';
FLUSH PRIVILEGES;

(Debian specific note: MariaDB provided startup scripts use the distro's mechanism of verifying startup/shutdown using a system user, so we create that as well:

# mariadb 10.0, 10.1
GRANT ALL PRIVILEGES ON *.* TO "debian-sys-maint"@"localhost" IDENTIFIED BY "adBexthTsI5TaEps";

If you do this, yo need to synchronize the <code>/etc/mysql/debian.cnf</code> file from the first node to the other nodes as well.)

==== Other nodes ====

On the other nodes, we only need to restart the service now, to trigger a full state transfer from the first node to the other nodes.

We recommend to do this serially to let one state transfer complete before the second state transfer.

==== First node (continued) ====

Only applicable if you used <code>galera_new_cluster</code> before rather than the service script: In order to get the systemctl status consistent, restart the service on the first node:

# mariadb 10.1: restart the service so that the systemctl status is consistent
mysqladmin shutdown
service mysql bootstrap

=== Verify the replication ===

The key tool to verify replication status is

mysql> show status like "%wsrep%";

This will give a lot of output. You want to verify in particular

+------------------------------+--------------------------------------+
| Variable_name | Value |
+------------------------------+--------------------------------------+
| wsrep_cluster_size | 3 |
| wsrep_cluster_status | Primary |
| wsrep_local_state | 4 |
| wsrep_local_state_comment | Synced |
| wsrep_ready | ON |
+------------------------------+--------------------------------------+

You can also explicitly verify replication by creating / inserting DBs, tables, rows on one node and select on other nodes.

==== Troubleshooting ====

The logs are helpful. Always.

Common mistakes are listed below.

If the Galera module does not get loaded at all:
* Configuration settings in ''my.cnf'' which are incompatible to Galera
* Wrong path of the shared object providing the Galera plugin in wsrep.cnf (wsrep_provider)

If the first node starts, but the second / third nodes can not be added to the cluster:
* User for the replication not created correctly on the first Galera node
* SST fails due to missing / wrong version prerequisite packages (not everything is hardcoded in package dependencies -- make sure you got percona-xtrabackup installed in the correct version, and also socat). If SST fails, do not only look into mysqls primary error logs, but also into logfiles from the SST tool in /var/lib/mysql on the donor node.

=== Notes about configuring OX for use with Galera ===

==== Write requests ====

Open-Xchange supports Galera as database backend only in the configuration where all writes are directed to one Galera node. For availability, it makes sense to not configure one Galera node's IP address directly, but rather employ some HA solution which offers active-passive functionality. Options therefore are discussed below.

==== Read requests ====

Read requests can be directed to any node in the Galera cluster. Our standard approach is to recommend to use a loadbalancer to implement round-robin over all nodes in a Galera cluster for the read requests. But you can also chose to use a dedicated read node (the same node, or a different node, than the write node). Each of the approaches has its own advantages.

* Load balancer based setup: Read requests get distributed round-robin between the Galera nodes. Theoretically by distributing the load of the read requests, you benefit from lower latencies and more throughput. But this has never been benchmarked yet. For a discussion of available loadbalances, see next section. OX-wise, in this configuration, you have two alternatives:
** The Galera option wsrep_causal_reads=1 option enables you to configure OX with its replication monitor disabled (com.openexchange.database.replicationMonitor=false in configdb.properties). This is the setup which seems to perform best according to our experience as turning off the replication monitor reduces the commits on the DB and thus the write operations per second on the underlying storage significantly, which outweights the drawback from having higher commit latency due to fully synchronous mode.
** Alternatively, you can run Galera with wsrep_causal_reads=0 when switching on OX builtin replication monitor. This is also a valid setup.
* Use a designated floating IP for the read requests: This eliminates the need of a load balancer. With this option you will not gain any performance, but the quantitative benefit is unclear anyhow.
* Use the floating IP for the writes also for the reads: In this scenario, you direct all database queries only to one Galera node, and the other two nodes are only getting queries in case of a failure of that node. In this case, you can even use wsrep_causal_reads=0 while still having OX builtin replication monitor switched off. However we do not expect this option to be superior to the round-robin loadbalancer approach.

=== Loadbalancer options ===

While the JDBC driver has some round-robin load balancing capabilities built-in, we don't recommend it for production use since it lacks possibilities to check the Galera nodes health states.

Loadbalancers used for OX -> Galera loadbalancing should be able to implement active-passive instances for the write requests, and active-active (round-robin) instances for the read requests. (If they cannot implement active-passive, you can still take a floating IP therefore.) Furthermore it is required to configure node health checks not only on the TCP level (by a simple connect), but to query the Galera health status periodically, evaluating Galera WSREP status variables. Otherwise split-brain scenarios or other bad states cannot be detected. For an example of such an health check, see our [[Clustercheck]] page.

Some customers use loadbalancing appliances. It is important to check that if the (virtual) infrastructure offers "loadbalancer" instances that they satisfy the given requirements. Often this is not the case. In particular, a simple "DNS round robin" approach is not viable.

==== LVS/ipvsadm/keepalived ====

If you want to create your own loadbalancers based on Linux, we usually recommend LVS (Linux Virtual Servers) controlled by Keepalived. LVS is a set of kernel modules implementing a L4 loadbalancer which performs quite well. Keepalived is a userspace daemon to control LVS rules, using health checks to reconfigure LVS rules if required. Keepalived / LVS requires one (or, for availability, two) dedicated linux nodes to run on. This can be a disadvantage for some installations, but usually, it pays off. We provide some configuration information on Keepalived [[Keepalived|here]].

==== MariaDB Maxscale ====

Since Maxscale has become GA in 2015, it seems to have undergone significant stability, performance and functional improvements. We are currently experimenting with Maxscale and share our installation / configuration knowledge [[Maxscale|here]]. It looks quite promising and might become ''the standard replacement'' for HAproxy, while we still presume Keepalived offers superior robustness and performance, coming with the cost of the requirement for one (or more) dedicated loadbalancer nodes.

==== HAproxy ====

In case where the Keepalived based approach is not feasible due to its requirements on the infrastructure, it is also possible to use a HAproxy based solution where HAproxy processes run on each of the OX nodes, configured for one round-robin and one active/passive instance. OX is then connecting to the local HAproxy instances. It is vital to configure HAproxy timeouts different from the defaults, otherwise HAproxy will kill active DB connections, causing errors. Be aware that in large installations the number of (distributed) HAproxy instances can get quite large. Some configuration hints for HAproxy are available [[HAproxy|here]].

== Master/Slave database setup ==

While we also support also "legacy" (pre-GTID) Master/Slave replication, we recommend to use GTID based replication, for easier setup and failure recovery. Support for GTID based replication has been added with OX 7.8.0.

GTID has been available since MySQL 5.6, so no 5.5 installation instructions below, sorry. We try to be generic in this documentation (thus, applicable to Oracle Community Edition and MariaDB) and point out differences where needed. Note: Instructions below include information about Oracle Community MySQL 5.7 which is not yet formally supported.

=== Preparations ===

Our configuration process includes wiping and reinitializing the datadir. This is usually not a problem in a fresh installation. If you want to upgrade an existing database to GTID master-slave, please be prepared to wipe the datadir, i.e. take a <code>mysqldump</code> for later restoration into the properly configured master.

Depeding on the flavor of the current database, this can be something like

# mariadb or oracle mysql without GTIDs
mysqldump --databases configdb oxdb_{5..14} > backup.sql

# mysql 5.6 with GTIDs... we dont want GTIDs here
mysqldump --databases --set-gtid-purged=OFF configdb oxdb_{5..14} > backup.sql

Be sure to verify the list of databases.

=== Installation ===

Software installation is identical for master and slave.

Please follow the instructions for installing from The vendors.

* Oracle Community Edition: https://dev.mysql.com/doc/mysql-apt-repo-quick-guide/en/
* MariaDB (10.0, 10.1): https://downloads.mariadb.org/mariadb/repositories/

Stop the service (if it is running):

service mysql stop

=== Configuration ===

Configuration as per configuration files is also identical for master and slave.

Consult [[My.cnf]] for general recommendations how to configure databases for usage with OX.

For GTID based replication, make sure you add some configurables to a new <code>/etc/mysql/ox.conf.d/gtid.cnf</code> file (assuming you are following our proposed schema of adding a <code>!includedir /etc/mysql/ox.conf.d/</code>" directive to <code>/etc/mysql/my.cnf</code>):

# GTID
log-bin=mysql-bin
server-id=...
log_slave_updates = ON

Oracle Community Edition: we need to add also

enforce_gtid_consistency = ON
gtid_mode = ON

(GTID mode is on by default on MariaDB.)

Use unique a <code>server-id</code> for each server; like <code>1</code> for the master, <code>2</code> for slave. For more complicated setups (like multiple slaves), adjust accordingly.

Since applying our configuration / sizing requires reinitialization of the MySQL datadir, we wipe/recreate it. Caution: this assumes we are running an empty database. If there is data in the database you want to keep, use mysqldump. See Preparation section above.

So, to initialize the datadir:

cd /var/lib/
mv mysql mysql.old.datadir
mkdir mysql
chown mysql.mysql mysql

(When coming from an existing installation, be sure to wipe also old binlogs. They can confuse the server on startup. Their location varies by configuration.)

The step to initialize the datadir is different for the different DBs:

# MariaDB 10.0, 10.1
mysql_install_db

# Oracle 5.6
mysql_install_db -u mysql

# Oracle 5.7
mysqld --initialize-insecure --user=mysql

(Don't be worried about the <code>insecure</code>, it just means we set the db root pw in the next steps.)

Then:

service mysql restart
mysql_secure_installation

We want to emphasize the last step to run "secure".

Steps up to here apply to both the designated master and slave. The next steps will apply to the master.

=== Replication Setup ===

==== Master Setup ====

Create a replication user on the master (but, as always, pick your own password, and use the same password in the slave setup below):

mysql -e "CREATE USER 'repl'@'gtid-slave.localdomain' IDENTIFIED BY 'IvIjyoffod2'; GRANT REPLICATION SLAVE ON *.* TO 'repl'@'gtid-slave.localdomain';"

Now would also be the time to restore a previously created mysqldump, or add other users you need for adminstration, monitoring etc (like <code>debian-sys-maint@localhost</code>, for example). Adding the OX users is explained below ("Creating Open-Xchange user").

# If you took a dump for restore before
mysql < backup.sql

To prepare for the initial sync of the slave, set the master read-only:

mysql -e "SET @@global.read_only = ON;"

Create a dump to initialize the slave:

# MariaDB
mysqldump --all-databases --triggers --routines --events --master-data --gtid > master.sql

# Oracle
mysqldump --all-databases --triggers --routines --events --set-gtid-purged=ON > master.sql

Transfer to the slave:

scp master.sql gtid-slave:

==== Slave Setup ====

Configure the replication master settings. Note we don't need complicated binlog position settings etc with GTID.

Yet again DB-specific (use the repl user password from above):

# MariaDB
mysql -e 'CHANGE MASTER TO MASTER_HOST="gtid-master.localdomain", MASTER_USER="repl", MASTER_PASSWORD="IvIjyoffod2";'

# Oracle
mysql -e "CHANGE MASTER TO MASTER_HOST='gtid-master.localdomain', MASTER_USER='repl', MASTER_PASSWORD='IvIjyoffod2', MASTER_AUTO_POSITION=1;"
# https://www.percona.com/blog/2013/02/08/how-to-createrestore-a-slave-using-gtid-replication-in-mysql-5-6/
mysql -e "RESET MASTER;"

Read the master dump:

mysql < master.sql

Start replication on the slave:

mysql -e 'START SLAVE;'
mysql -e 'SHOW SLAVE STATUS\G'

==== Master Setup (continued) ====

Finally, unset read-only on the master:

# on the master
mysql -e "SET @@global.read_only = OFF;"

=== Configure OX to use with Master/Slave replication ===

Not much special wisdom here. OX was designed to be used with master/slave databases. For the ConfigDB, <code>configdb.properties</code> allows configuration of a <code>readUrl</code> and <code>writeUrl</code> (both of which are set to the correct values if you use <code>oxinstaller</code> with the correct arguments <code>--configdb-readhost</code>, <code>--configdb-writehost</code>).

(Obviously, the master is for writing and the slave is for reading.)

For the individiual UserDBs, use <code>registerdatabase -m true</code> for the masters and <code>registerdatabase -m false -M ...</code> for the respective slaves.

Be sure to have enabled the replication monitor in <code>configdb.properties</code>: <code>com.openexchange.database.replicationMonitor=true</code> (which it is by default); while GTID can show synchronous semantics, it is specified to silently fall back to asynchronous in certain circumstances, so synchronity is not guaranteed.

We recommend, though, to not register the databases directly by their native hostname or IP, but rather use some kind of HA system in order to be able to easily move a floating/failover IP from the master to the slave in case of master failure. Configuring and running such systems (like, corosync/pacemaker, keepalived, or whatever) is out of scope of this documentation, however.

== Creating Open-Xchange user ==

Now setup access for the Open-Xchange Server database user 'openexchange' to configdb and the oxdb for both groupware server addresses. These databases do not exist yet, but will be created during the Open-Xchange Server installation.

Note: The IPs in this example belong to the two different Open-Xchange Servers, please adjust them accordingly. And use a real password.

mysql> GRANT ALL PRIVILEGES ON *.* TO 'openexchange'@'10.20.30.213' IDENTIFIED BY 'IntyoyntOat1';
mysql> GRANT ALL PRIVILEGES ON *.* TO 'openexchange'@'10.20.30.215' IDENTIFIED BY 'IntyoyntOat1';

Template:OXLoadBalancingClustering Database

2018-05-16T10:21:02Z

Dominik.epple:

== Overview ==

You can choose between Galera or Master/Slave replication. We like to recommend to use Galera for higher redudancy, easier operations, und synchronous semantics (so you can run OX without our "replication monitor"). For POC or demo setups, a single standalone database setup might be sufficient.

== Standalone database setup ==

=== Preparations ===

Our configuration process includes wiping and reinitializing the datadir. This is usually not a problem in a fresh installation. If you want to upgrade an existing database server, please be prepared to wipe the datadir, i.e. take a <code>mysqldump</code> for later restoration into the properly configured master.

mysqldump --databases configdb oxdb_{5..14} > backup.sql

Be sure to verify the list of databases.

=== Installation ===

Please follow the upstream docs for your preferred flavor to get the software installed on your system.

Note: the Oracle flavor is also valid, but not yet documented here.

* Percona XtraDB (5.6, 5.7): https://www.percona.com/downloads/
* MariaDB Galera Cluster (10.0, 10.1): https://downloads.mariadb.org/

Make sure to doublecheck the service is not running (or stop it) after installation as we need to perform some reconfigurations.

service mysql stop

=== Configuration ===

MySQL configuration advise is given in our [[My.cnf|MySQL configuration article]]. Please consult that page for configuration information and create configuration files as described there.

Some settings we recommend to change require that the database gets re-initialized. We assume you don't have data there (since we are covering a fresh install) or you have taken a backup for later restore as explained above in the Preparations section.

cd /var/lib/
mv mysql mysql.old.datadir
mkdir mysql
chown mysql.mysql mysql

# mariadb 10.0 and 10.1
mysql_install_db
# percona 5.6
mysqld --user=mysql
# percona 5.7
mysqld --initialize-insecure --user=mysql

(Don't be worried about the <code>insecure</code>, it just means we set the db root pw in the next steps.)

Run <code>mysql_secure_installation</code> for a "secure by default" installation:

mysql_secure_installation

The further steps in this guide omit <code>-u -p</code> arguments to the MySQL client. Rather than passing them on the command line [https://dev.mysql.com/doc/refman/5.7/en/password-security-user.html] it is recommended to place the credentials in a file like <code>/root/.my.cnf</code> like

[client]
user=root
password=wip9Phae3Beijeed

You should now be able to restore your previously taken backup.

# If you took a dump for restore before
mysql < backup.sql

=== Configure OX to use with a standalone database ===

Not much special wisdom here. OX was designed to be used with master/slave databases, and a standalone master works just as well, if we register it as a master, and not registering a slave.

For the ConfigDB, <code>configdb.properties</code> allows configuration of a <code>writeUrl</code> (which is set to the correct values if you use <code>oxinstaller</code> with the correct argument <code>--configdb-writehost</code>).

The single database is then used for reading and writing.

For the individiual UserDBs, use <code>registerdatabase -m true</code>.

== Galera database setup ==

=== Preparations ===

Our configuration process includes wiping and reinitializing the datadir. This is usually not a problem in a fresh installation. If you want to upgrade an existing database to Galera cluster, please be prepared to wipe the datadir, i.e. take a <code>mysqldump</code> for later restoration into the properly configured master.

Depeding on the flavor of the current database, this can be something like

# mariadb or oracle mysql without GTIDs
mysqldump --databases configdb oxdb_{5..14} > backup.sql

# mysql 5.6 with GTIDs... we dont want GTIDs here
mysqldump --databases --set-gtid-purged=OFF configdb oxdb_{5..14} > backup.sql

Be sure to verify the list of databases.

=== Installation ===

Please follow the upstream docs for your preferred flavor to get the software installed on your system.

* Percona XtraDB Cluster (5.6, 5.7): https://www.percona.com/doc/percona-xtradb-cluster/LATEST/install/index.html
* MariaDB Galera Cluster (10.0, 10.1): https://mariadb.com/kb/en/library/getting-started-with-mariadb-galera-cluster/ (Note: with 10.0, socat is required, but not a package dependency, so you need to explicitly install also socat)

Make sure to doublecheck the service is not running (or stop it) after installation as we need to perform some reconfigurations.

service mysql stop

=== Configuration ===

Galera-specific MySQL configuration advise is included in our main [[My.cnf|MySQL configuration article]]. Please consult that page for configuration information.

That page suggests a setup were we add three custom config files to <code>/etc/mysql/ox.conf.d/</pre>: <code>ox.cnf</code> for general tuning/sizing, <code>wsrep.cnf</code> for clusterwide galera configuration, and <code>host.cnf</code> for host-specific settings.

Adjust the general settings and tunings in <code>ox.cnf</code> according to your sizing etc.

Adjust <code>wsrep.cnf</code> to reflect local paths, cluster member addresses, etc.

Adjust <code>host.cnf</code> to give node-local IPs, etc.

Version-specific hints:

# percona 5.6: unknown variable 'pxc_strict_mode=ENFORCING' ... unset that one
# mariadb 10.1: add wsrep_on=ON
# mariadb 10.0 and 10.1: set wsrep_node_incoming_address=192.168.1.22:3306 in host.cnf, otherwise the status wsrep_incoming_addresses might not be shown correctly(?!)

Some settings we recommend to change require that the database gets re-initialized. We assume you don't have data there (since we are covering a fresh install) or you have taken a backup for later restore as explained above in the Preparations section.

cd /var/lib/
mv mysql mysql.old.datadir
mkdir mysql
chown mysql.mysql mysql

# mariadb 10.0 and 10.1
mysql_install_db
# percona 5.6
mysqld --user=mysql
# percona 5.7
mysqld --initialize-insecure --user=mysql

(Don't be worried about the <code>insecure</code>, it just means we set the db root pw in the next steps.)

=== Cluster startup ===

Typically on startup a Galera node tries to join a cluster, and if it fails, it will exit. Thus, when no cluster nodes are running, the first cluster node to be started needs to be told to not try to join a cluster, but rather bootstrap a new cluster. The exact arguments vary from version to version and from flavor to flavor.

==== First node ====

So we initialize the cluster bootstrap on the first node:

# percona 5.6, 5.7
service mysql bootstrap-pxc
# mariadb 10.0
service mysql bootstrap
# mariadb 10.1: service mysql bootstrap seems to be broken, does not pass the necessary options to mysqld
galera_new_cluster

Run <code>mysql_secure_installation</code> for a "secure by default" installation:

mysql_secure_installation

The further steps in this guide omit <code>-u -p</code> arguments to the MySQL client. Rather than passing them on the command line [https://dev.mysql.com/doc/refman/5.7/en/password-security-user.html] it is recommended to place the credentials in a file like <code>/root/.my.cnf</code> like

[client]
user=root
password=wip9Phae3Beijeed

We need a Galera replication user:

CREATE USER 'sstuser'@'localhost' IDENTIFIED BY 'OpIdjijwef0';
-- percona 5.6, mariadb 10.0
GRANT RELOAD, LOCK TABLES, REPLICATION CLIENT ON *.* TO 'sstuser'@'localhost';
-- percona 5.7, mariadb 10.1
GRANT PROCESS, RELOAD, LOCK TABLES, REPLICATION CLIENT ON *.* TO 'sstuser'@'localhost';
FLUSH PRIVILEGES;

(Debian specific note: MariaDB provided startup scripts use the distro's mechanism of verifying startup/shutdown using a system user, so we create that as well:

# mariadb 10.0, 10.1
GRANT ALL PRIVILEGES ON *.* TO "debian-sys-maint"@"localhost" IDENTIFIED BY "adBexthTsI5TaEps";

If you do this, yo need to synchronize the <code>/etc/mysql/debian.cnf</code> file from the first node to the other nodes as well.)

==== Other nodes ====

On the other nodes, we only need to restart the service now, to trigger a full state transfer from the first node to the other nodes.

We recommend to do this serially to let one state transfer complete before the second state transfer.

==== First node (continued) ====

Only applicable if you used <code>galera_new_cluster</code> before rather than the service script: In order to get the systemctl status consistent, restart the service on the first node:

# mariadb 10.1: restart the service so that the systemctl status is consistent
mysqladmin shutdown
service mysql bootstrap

=== Verify the replication ===

The key tool to verify replication status is

mysql> show status like "%wsrep%";

This will give a lot of output. You want to verify in particular

+------------------------------+--------------------------------------+
| Variable_name | Value |
+------------------------------+--------------------------------------+
| wsrep_cluster_size | 3 |
| wsrep_cluster_status | Primary |
| wsrep_local_state | 4 |
| wsrep_local_state_comment | Synced |
| wsrep_ready | ON |
+------------------------------+--------------------------------------+

You can also explicitly verify replication by creating / inserting DBs, tables, rows on one node and select on other nodes.

==== Troubleshooting ====

The logs are helpful. Always.

Common mistakes are listed below.

If the Galera module does not get loaded at all:
* Configuration settings in ''my.cnf'' which are incompatible to Galera
* Wrong path of the shared object providing the Galera plugin in wsrep.cnf (wsrep_provider)

If the first node starts, but the second / third nodes can not be added to the cluster:
* User for the replication not created correctly on the first Galera node
* SST fails due to missing / wrong version prerequisite packages (not everything is hardcoded in package dependencies -- make sure you got percona-xtrabackup installed in the correct version, and also socat). If SST fails, do not only look into mysqls primary error logs, but also into logfiles from the SST tool in /var/lib/mysql on the donor node.

=== Notes about configuring OX for use with Galera ===

==== Write requests ====

Open-Xchange supports Galera as database backend only in the configuration where all writes are directed to one Galera node. For availability, it makes sense to not configure one Galera node's IP address directly, but rather employ some HA solution which offers active-passive functionality. Options therefore are discussed below.

==== Read requests ====

Read requests can be directed to any node in the Galera cluster. Our standard approach is to recommend to use a loadbalancer to implement round-robin over all nodes in a Galera cluster for the read requests. But you can also chose to use a dedicated read node (the same node, or a different node, than the write node). Each of the approaches has its own advantages.

* Load balancer based setup: Read requests get distributed round-robin between the Galera nodes. Theoretically by distributing the load of the read requests, you benefit from lower latencies and more throughput. But this has never been benchmarked yet. For a discussion of available loadbalances, see next section. OX-wise, in this configuration, you have two alternatives:
** The Galera option wsrep_causal_reads=1 option enables you to configure OX with its replication monitor disabled (com.openexchange.database.replicationMonitor=false in configdb.properties). This is the setup which seems to perform best according to our experience as turning off the replication monitor reduces the commits on the DB and thus the write operations per second on the underlying storage significantly, which outweights the drawback from having higher commit latency due to fully synchronous mode.
** Alternatively, you can run Galera with wsrep_causal_reads=0 when switching on OX builtin replication monitor. This is also a valid setup.
* Use a designated floating IP for the read requests: This eliminates the need of a load balancer. With this option you will not gain any performance, but the quantitative benefit is unclear anyhow.
* Use the floating IP for the writes also for the reads: In this scenario, you direct all database queries only to one Galera node, and the other two nodes are only getting queries in case of a failure of that node. In this case, you can even use wsrep_causal_reads=0 while still having OX builtin replication monitor switched off. However we do not expect this option to be superior to the round-robin loadbalancer approach.

=== Loadbalancer options ===

While the JDBC driver has some round-robin load balancing capabilities built-in, we don't recommend it for production use since it lacks possibilities to check the Galera nodes health states.

Loadbalancers used for OX -> Galera loadbalancing should be able to implement active-passive instances for the write requests, and active-active (round-robin) instances for the read requests. (If they cannot implement active-passive, you can still take a floating IP therefore.) Furthermore it is required to configure node health checks not only on the TCP level (by a simple connect), but to query the Galera health status periodically, evaluating Galera WSREP status variables. Otherwise split-brain scenarios or other bad states cannot be detected. For an example of such an health check, see our [[Clustercheck]] page.

Some customers use loadbalancing appliances. It is important to check that if the (virtual) infrastructure offers "loadbalancer" instances that they satisfy the given requirements. Often this is not the case. In particular, a simple "DNS round robin" approach is not viable.

==== LVS/ipvsadm/keepalived ====

If you want to create your own loadbalancers based on Linux, we usually recommend LVS (Linux Virtual Servers) controlled by Keepalived. LVS is a set of kernel modules implementing a L4 loadbalancer which performs quite well. Keepalived is a userspace daemon to control LVS rules, using health checks to reconfigure LVS rules if required. Keepalived / LVS requires one (or, for availability, two) dedicated linux nodes to run on. This can be a disadvantage for some installations, but usually, it pays off. We provide some configuration information on Keepalived [[Keepalived|here]].

==== MariaDB Maxscale ====

Since Maxscale has become GA in 2015, it seems to have undergone significant stability, performance and functional improvements. We are currently experimenting with Maxscale and share our installation / configuration knowledge [[Maxscale|here]]. It looks quite promising and might become ''the standard replacement'' for HAproxy, while we still presume Keepalived offers superior robustness and performance, coming with the cost of the requirement for one (or more) dedicated loadbalancer nodes.

==== HAproxy ====

In case where the Keepalived based approach is not feasible due to its requirements on the infrastructure, it is also possible to use a HAproxy based solution where HAproxy processes run on each of the OX nodes, configured for one round-robin and one active/passive instance. OX is then connecting to the local HAproxy instances. It is vital to configure HAproxy timeouts different from the defaults, otherwise HAproxy will kill active DB connections, causing errors. Be aware that in large installations the number of (distributed) HAproxy instances can get quite large. Some configuration hints for HAproxy are available [[HAproxy|here]].

== Master/Slave database setup ==

While we also support also "legacy" (pre-GTID) Master/Slave replication, we recommend to use GTID based replication, for easier setup and failure recovery. Support for GTID based replication has been added with OX 7.8.0.

GTID has been available since MySQL 5.6, so no 5.5 installation instructions below, sorry. We try to be generic in this documentation (thus, applicable to Oracle Community Edition and MariaDB) and point out differences where needed. Note: Instructions below include information about Oracle Community MySQL 5.7 which is not yet formally supported.

=== Preparations ===

Our configuration process includes wiping and reinitializing the datadir. This is usually not a problem in a fresh installation. If you want to upgrade an existing database to GTID master-slave, please be prepared to wipe the datadir, i.e. take a <code>mysqldump</code> for later restoration into the properly configured master.

Depeding on the flavor of the current database, this can be something like

# mariadb or oracle mysql without GTIDs
mysqldump --databases configdb oxdb_{5..14} > backup.sql

# mysql 5.6 with GTIDs... we dont want GTIDs here
mysqldump --databases --set-gtid-purged=OFF configdb oxdb_{5..14} > backup.sql

Be sure to verify the list of databases.

=== Installation ===

Software installation is identical for master and slave.

Please follow the instructions for installing from The vendors.

* Oracle Community Edition: https://dev.mysql.com/doc/mysql-apt-repo-quick-guide/en/
* MariaDB (10.0, 10.1): https://downloads.mariadb.org/mariadb/repositories/

Stop the service (if it is running):

service mysql stop

=== Configuration ===

Configuration as per configuration files is also identical for master and slave.

Consult [[My.cnf]] for general recommendations how to configure databases for usage with OX.

For GTID based replication, make sure you add some configurables to a new <code>/etc/mysql/ox.conf.d/gtid.cnf</code> file (assuming you are following our proposed schema of adding a <code>!includedir /etc/mysql/ox.conf.d/</code>" directive to <code>/etc/mysql/my.cnf</code>):

# GTID
log-bin=mysql-bin
server-id=...
log_slave_updates = ON

Oracle Community Edition: we need to add also

enforce_gtid_consistency = ON
gtid_mode = ON

(GTID mode is on by default on MariaDB.)

Use unique a <code>server-id</code> for each server; like <code>1</code> for the master, <code>2</code> for slave. For more complicated setups (like multiple slaves), adjust accordingly.

Since applying our configuration / sizing requires reinitialization of the MySQL datadir, we wipe/recreate it. Caution: this assumes we are running an empty database. If there is data in the database you want to keep, use mysqldump. See Preparation section above.

So, to initialize the datadir:

cd /var/lib/
mv mysql mysql.old.datadir
mkdir mysql
chown mysql.mysql mysql

(When coming from an existing installation, be sure to wipe also old binlogs. They can confuse the server on startup. Their location varies by configuration.)

The step to initialize the datadir is different for the different DBs:

# MariaDB 10.0, 10.1
mysql_install_db

# Oracle 5.6
mysql_install_db -u mysql

# Oracle 5.7
mysqld --initialize-insecure --user=mysql

(Don't be worried about the <code>insecure</code>, it just means we set the db root pw in the next steps.)

Then:

service mysql restart
mysql_secure_installation

We want to emphasize the last step to run "secure".

Steps up to here apply to both the designated master and slave. The next steps will apply to the master.

=== Replication Setup ===

==== Master Setup ====

Create a replication user on the master (but, as always, pick your own password, and use the same password in the slave setup below):

mysql -e "CREATE USER 'repl'@'gtid-slave.localdomain' IDENTIFIED BY 'IvIjyoffod2'; GRANT REPLICATION SLAVE ON *.* TO 'repl'@'gtid-slave.localdomain';"

Now would also be the time to restore a previously created mysqldump, or add other users you need for adminstration, monitoring etc (like <code>debian-sys-maint@localhost</code>, for example). Adding the OX users is explained below ("Creating Open-Xchange user").

# If you took a dump for restore before
mysql < backup.sql

To prepare for the initial sync of the slave, set the master read-only:

mysql -e "SET @@global.read_only = ON;"

Create a dump to initialize the slave:

# MariaDB
mysqldump --all-databases --triggers --routines --events --master-data --gtid > master.sql

# Oracle
mysqldump --all-databases --triggers --routines --events --set-gtid-purged=ON > master.sql

Transfer to the slave:

scp master.sql gtid-slave:

==== Slave Setup ====

Configure the replication master settings. Note we don't need complicated binlog position settings etc with GTID.

Yet again DB-specific (use the repl user password from above):

# MariaDB
mysql -e 'CHANGE MASTER TO MASTER_HOST="gtid-master.localdomain", MASTER_USER="repl", MASTER_PASSWORD="IvIjyoffod2";'

# Oracle
mysql -e "CHANGE MASTER TO MASTER_HOST='gtid-master.localdomain', MASTER_USER='repl', MASTER_PASSWORD='IvIjyoffod2', MASTER_AUTO_POSITION=1;"
# https://www.percona.com/blog/2013/02/08/how-to-createrestore-a-slave-using-gtid-replication-in-mysql-5-6/
mysql -e "RESET MASTER;"

Read the master dump:

mysql < master.sql

Start replication on the slave:

mysql -e 'START SLAVE;'
mysql -e 'SHOW SLAVE STATUS\G'

==== Master Setup (continued) ====

Finally, unset read-only on the master:

# on the master
mysql -e "SET @@global.read_only = OFF;"

=== Configure OX to use with Master/Slave replication ===

Not much special wisdom here. OX was designed to be used with master/slave databases. For the ConfigDB, <code>configdb.properties</code> allows configuration of a <code>readUrl</code> and <code>writeUrl</code> (both of which are set to the correct values if you use <code>oxinstaller</code> with the correct arguments <code>--configdb-readhost</code>, <code>--configdb-writehost</code>).

(Obviously, the master is for writing and the slave is for reading.)

For the individiual UserDBs, use <code>registerdatabase -m true</code> for the masters and <code>registerdatabase -m false -M ...</code> for the respective slaves.

Be sure to have enabled the replication monitor in <code>configdb.properties</code>: <code>com.openexchange.database.replicationMonitor=true</code> (which it is by default); while GTID can show synchronous semantics, it is specified to silently fall back to asynchronous in certain circumstances, so synchronity is not guaranteed.

We recommend, though, to not register the databases directly by their native hostname or IP, but rather use some kind of HA system in order to be able to easily move a floating/failover IP from the master to the slave in case of master failure. Configuring and running such systems (like, corosync/pacemaker, keepalived, or whatever) is out of scope of this documentation, however.

== Creating Open-Xchange user ==

Now setup access for the Open-Xchange Server database user 'openexchange' to configdb and the oxdb for both groupware server addresses. These databases do not exist yet, but will be created during the Open-Xchange Server installation.

Note: The IPs in this example belong to the two different Open-Xchange Servers, please adjust them accordingly. And use a real password.

mysql> GRANT ALL PRIVILEGES ON *.* TO 'openexchange'@'10.20.30.213' IDENTIFIED BY 'IntyoyntOat1';
mysql> GRANT ALL PRIVILEGES ON *.* TO 'openexchange'@'10.20.30.215' IDENTIFIED BY 'IntyoyntOat1';

AppSuite:Running a cluster

2018-04-24T09:06:31Z

Dominik.epple: /* Rolling Upgrade without breaking Hazelcast upgrade */

<div class="title">Running a cluster</div>

__TOC__

= Concepts =

For inter-OX-communication over the network, multiple Open-Xchange servers can form a cluster. This brings different advantages regarding distribution and caching of volatile data, load balancing, scalability, fail-safety and robustness. Additionally, it provides the infrastructure for upcoming features of the Open-Xchange server.
The clustering capabilities of the Open-Xchange server are mainly built up on [http://hazelcast.com Hazelcast], an open source clustering and highly scalable data distribution platform for Java. The following article provides an overview about the current featureset and configuration options.

= Requirements =

== Synchronized system clock times ==
It is crucial that all involved members in a cluster do have their system clock times in sync with each other; e.g. by using an NTP service.

== HTTP routing ==
An OX cluster is always part of a larger picture. Usually there is front level loadbalancer as central HTTPS entry point to the platform. This loadbalancer optionally performs HTTPS termination and forwards HTTP(S) requests to webservers (the usual and only supported choice as of now is Apache). These webservers are performing HTTPS termination (if this is not happening on the loadbalancer) and serve static content, and (which is what is relevant for our discussion here) they forward dynamic requests to the OX backends.

A central requirement for the interaction of these components (loadbalancer, webservers, OX nodes) is that we have session stability based on the JSESSIONID cookie / jsessionid path component suffix. This means that our application sets a cookie named JSESSIONID which has a value like <large decimal number>.<route identifier>, e.g. "5661584529655240315.OX1". The route identifier here ("OX1" in this example) is taken by the OX node from a configuration setting from a config file and is specific to one OX node. HTTP routing must happen such that HTTP requests with a cookie with such a suffix always end up the corresponding OX node. There are furthermore specific cirumstances when passing this information via cookie is not possible. Then the JSESSIONID is transferred in a path component as "jsessionid=..." in the HTTP request. The routing mechanism needs to take that into account also.

There are mainly two options to implement this. If the Apache processes are running co-located on the same machines running the OX groupware processes, it is often desired to have the front level loadbalancer perform HTTP routing to the correct machines. If dedicated Apache nodes are employed, is is usually sufficient to have the front-level loadbalancer do HTTP routing to the Apache nodes in a round-robin fashion and perform routing to the correct OX nodes in the Apache nodes.

We provide sample configuration files to configure Apache (with mod_proxy_http) to perform HTTP routing correctly in our guides on OXpedia, e.g. [[AppSuite:Main_Page_AppSuite#quickinstall]]. Central elements are the directives "ProxySet stickysession=JSESSIONID|jsessionid scolonpathdelim=On" in conjunction with the "route=OX1" parameters to the BalancerMember lines in the Proxy definition. This is valid for Apache 2.2 as of Sep-2014.

How to configure a front level loadbalancer to perform HTTP equivalent HTTP routing is dependent on the specific loadbalancer implementation. If Apache is used as front level loadbalancer, the same configuration as discussed in the previous section can be employed. As of time of writing this text (Sep 2014), the alternative choices are thin. F5 BigIP is reported to be able to implement "jsessionid based persistence using iRules". nginx has the functionality in their commercial "nginx plus" product. (Both of these options have not been tested by OX.) Other loadbalancers with this functionality are not known to us.

If the front level loadbalancer is not capable of performing correct HTTP routing, is is required to configure correct HTTP routing on Apache level, even if Apache runs co-located on the OX nodes and thus cross-routing happens.

There are several reasons why we require session stability in exactly this way. We require session stabilty for horizontal scale-out; while we support transparent resuming / migration of user sessions in the OX cluster without need for users to re-authenticate, sessions wandering around randomly will consume a fixed amount resources corresponding to a running session on each OX node in the cluster, while a session sticky to one OX node will consume this fixed amount of resources only on one OX node. Furthermore there are mechanisms in OX like TokenLogin which work only of all requests beloning to one sequence get routed to the same OX node even if they stem from different machines with different IPs. Only the JSESSIONID (which in this case is transferred as jsessionid path component, as cookies do not work during a 302 redirect, which is part of this sequence) carries the required information where the request must be routed to.

Usual "routing based on cookie hash" is not sufficient here since it disregards the information which machine originally issued the cookie. It only ensures that the session will be sticky to any target, which statistically will not be the same machine that issued the cookie. OX will then set a new JSESSIONID cookie, assuming the session had been migrated. The loadbalancer will then route the session to a different target, as the hash of the cookie will differ. This procedure then happens iteratively until by chance the routing based on cookie hash will route the session to the correct target. By then, a lot of resources will have been wasted, by creating full (short-term) sessions on all OX nodes. Furthermore, processes like TokenLogin will not work this way.

= Configuration =

All settings regarding cluster setup are located in the configuration file ''hazelcast.properties''. The former used additional files ''cluster.properties'', ''mdns.properties'' and ''static-cluster-discovery.properties'' are no longer needed. The following gives an overview about the most important settings - please refer to the inline documentation of the configuration file for more advanced options.

Note: The configuration guide targets v7.4.0 of the OX server (and above). For older versions, please consult the history of this page. A full list of Hazelcast-related properties is available at https://documentation.open-xchange.com/components/middleware/config/7.8.4/#mode=features&feature=Hazelcast .

== General ==

To restrict access to the cluster and to separate the cluster from others in the local network, a name and password needs to be defined. Only backend nodes having the same values for those properties are able to join and form a cluster.

# Configures the name of the cluster. Only nodes using the same group name
# will join each other and form the cluster. Required if
# "com.openexchange.hazelcast.network.join" is not "empty" (see below).
com.openexchange.hazelcast.group.name=

# The password used when joining the cluster. Defaults to "wtV6$VQk8#+3ds!a".
# Please change this value, and ensure it's equal on all nodes in the cluster.
com.openexchange.hazelcast.group.password=wtV6$VQk8#+3ds!a

== Network ==

It's required to define the network interface that is used for cluster communication via ''com.openexchange.hazelcast.network.interfaces''. By default, the interface is restricted to the local loopback address only. To allow the same configuration amongst all nodes in the cluster, it's recommended to define the value using a wildcard matching the IP addresses of all nodes participating in the cluster, e.g. ''192.168.0.*''

# Comma-separated list of interface addresses hazelcast should use. Wildcards
# (*) and ranges (-) can be used. Leave blank to listen on all interfaces
# Especially in server environments with multiple network interfaces, it's
# recommended to specify the IP-address of the network interface to bind to
# explicitly. Defaults to "127.0.0.1" (local loopback only), needs to be
# adjusted when building a cluster of multiple backend nodes.
com.openexchange.hazelcast.network.interfaces=127.0.0.1

To form a cluster of multiple OX server nodes, different discovery mechanisms can be used. The discovery mechanism is specified via the property ''com.openexchange.hazelcast.network.join'':

# Specifies which mechanism is used to discover other backend nodes in the
# cluster. Possible values are "empty" (no discovery for single-node setups),
# "static" (fixed set of cluster member nodes) or "multicast" (automatic
# discovery of other nodes via multicast). Defaults to "empty". Depending on
# the specified value, further configuration might be needed, see "Networking"
# section below.
com.openexchange.hazelcast.network.join=empty

Generally, it's advised to use the same network join mechanism for all nodes in the cluster, and, in most cases, it's strongly recommended to use a ''static'' network join configuration. This will allow the nodes to join the cluster directly upon startup. With a ''multicast'' based setup, nodes will merge to an existing cluster possibly at some later time, thus not being able to access the distributed data until they've joined.

Depending on the network join setting, further configuration may be necessary, as decribed in the following paragraphs.

=== empty ===

When using the default value ''empty'', no other nodes are discovered in the cluster. This value is suitable for single-node installations. Note that other nodes that are configured to use other network join mechanisms may be still able to still to connect to this node, e.g. using a ''static'' network join, having the IP address of this host in the list of potential cluster members (see below).

=== static ===

The most common setting for ''com.openexchange.hazelcast.network.join'' is ''static''. A static cluster discovery uses a fixed list of IP addresses of the nodes in the cluster. During startup and after a specific interval, the underlying Hazelcast library probes for not yet joined nodes from this list and adds them to the cluster automatically. The address list is configured via ''com.openexchange.hazelcast.network.join.static.nodes'':

# Configures a comma-separated list of IP addresses / hostnames of possible
# nodes in the cluster, e.g. "10.20.30.12, 10.20.30.13:5701, 192.178.168.110".
# Only used if "com.openexchange.hazelcast.network.join" is set to "static".
# It doesn't hurt if the address of the local host appears in the list, so
# that it's still possible to use the same list throughout all nodes in the
# cluster.
com.openexchange.hazelcast.network.join.static.nodes=

For a fixed set of backend nodes, it's recommended to simply include the IP addresses of all nodes in the list, and use the same configuration for each node. However, it's only required to add the address of at least one other node in the cluster to allow the node to join the cluster. Also, when adding a new node to the cluster and this list is extended accordingly, existing nodes don't need to be shut down to recognize the new node, as long as the new node's address list contains at least one of the already running nodes.

=== multicast ===

For highly dynamic setups where nodes are added and removed from the cluster quite often and/or the host's IP addresses are not fixed, it's also possible to configure the network join via multicast. During startup and after a specific interval, the backend nodes initiate the multicast join process automatically, and discovered nodes form or join the cluster afterwards. The multicast group and port can be configured as follows:

# Configures the multicast address used to discover other nodes in the cluster
# dynamically. Only used if "com.openexchange.hazelcast.network.join" is set
# to "multicast". If the nodes reside in different subnets, please ensure that
# multicast is enabled between the subnets. Defaults to "224.2.2.3".
com.openexchange.hazelcast.network.join.multicast.group=224.2.2.3

# Configures the multicast port used to discover other nodes in the cluster
# dynamically. Only used if "com.openexchange.hazelcast.network.join" is set
# to "multicast". Defaults to "54327".
com.openexchange.hazelcast.network.join.multicast.port=54327

== Example ==

The following example shows how a simple cluster named ''MyCluster'' consisting of 4 backend nodes can be configured using ''static'' cluster discovery. The node's IP addresses are 10.0.0.15, 10.0.0.16, 10.0.0.17 and 10.0.0.18. Note that the same ''hazelcast.properties'' is used by all nodes.

com.openexchange.hazelcast.group.name=MyCluster
com.openexchange.hazelcast.group.password=secret
com.openexchange.hazelcast.network.join=static
com.openexchange.hazelcast.network.join.static.nodes=10.0.0.15,10.0.0.16,10.0.0.17,10.0.0.18
com.openexchange.hazelcast.network.interfaces=10.0.0.*

== Advanced Configuration ==

=== Lite Members (available since v7.8.4) ===

Lite members in a Hazelcast cluster are members that do not hold any data partitions, i.e. all read- and write operations to distributed maps are delegated to non-lite ("full") members. Apart from not having data partitions, lite members participate in the same way as other members: they can register listeners for distributed topics (e.g. cache invalidation events) or can be addressed for task execution (e.g. during realtime communication).

Similar to using a custom partitioning scheme, separating the nodes of a large cluster into few "full" members and many "lite" members helps to minimize the impact of JVM activities from a single node (mainly the garbage collector) on the whole cluster communication. Additionally, when starting or stopping lite members, no repartitioning of the distributed cluster data needs to be performed, which significantly decreases the node's startup- and shutdown time and reduces the necessary network communication to a minimum.

In medium or larger sized clusters, it is sufficient to have roughly 10 to 20 percent of the nodes configured as "full" members, while all other ones can be started as "lite" member nodes. Additionally, please note that the configured backup count in the map configurations should always be smaller than the total number of "full" members, otherwise, there may be problems if one of those data nodes is shut down temporarily for maintenance. So, the minimum number of "full" members is implicitly bound to the sum of a map's ''backupCount'' and ''asyncBackupCount'' properties, plus ''1'' for the original data partition.

The configured "full" members should preferrably not be used to serve client requests (by not adding them as endpoint in the loadbalancer), to ensure they are always responsive. Also, shutdown and startups of those "full" members should be reduced to a minimum to avoid repartitioning operations.

More general information regarding lite members is available at http://docs.hazelcast.org/docs/latest/manual/html-single/index.html#enabling-lite-members .

To configure a node as "lite" member, the following configuration should be applied in the node's ''hazelcast.properties'' file:

com.openexchange.hazelcast.liteMember=true

It's also recommended to use a "static" cluster discovery for the network join, and list all "full" member nodes here, so that join requests are handled by those nodes, too (and not the other nodes that are potentially prone to garbage collection delays.

=== Custom Partitioning ===

Note: Starting with v7.8.4, "Lite Members" should be used in favor of applying a custom partitioning scheme.

While originally being designed to separate the nodes holding distributed data into different risk groups for increased fail safety, a custom partitioning strategy may also be used to distinguish between nodes holding distributed data from those who should not.

This approach of custom partitioning may be used in a OX cluster, where usually different backend nodes serve different purposes. A common scenario is that there are nodes handling requests from the web interfaces, and others being responsible for USM/EAS traffic. Due to their nature of processing large chunks of synchronization data in memory, the USM/EAS nodes may encounter small delays when the Java garbage collector kicks in and suspends the Java Virtual Machine. Since those delays may also have an influence on hazelcast-based communication in the cluster, the idea is to instruct hazelcast to not store distributed data on that nodes. This is where a custom partitioning scheme comes into play.

To setup a custom partitioning scheme in the cluster, an additional ''hazelcast.xml'' configuration file is used, which should be placed into the ''hazelcast'' subdirectory of the OX configuration folder, usually at ''/opt/openexchange/etc/hazelcast''. Please note that it's vital that each node in the cluster is configured equally here, so the same ''hazelcast.xml'' file should be copied to each server. The configuration read from there is used as basis for all further settings that are taken from the ordinary ''hazelcast.properties'' config file.

To setup a custom partitioning scheme, the partition groups must be defined in the ''hazelcast.xml'' file. See the following file for an example configuration, where the three nodes ''10.10.10.60'', ''10.10.10.61'' and ''10.10.10.62'' are defined to form an own partitioning group each. Doing so, all distributed data will be stored at one of those nodes physically, while the corresponding backup data (if configured) at one of the other two nodes. All other nodes in the cluster will not be used to store distributed data, but will still be "full" hazelcast members, which is necessary for other cluster-wide operations the OX backends use.

Please note that the configured backup count in the map configurations should be smaller than the number of nodes here, otherwise, there may be problems if one of those data nodes is shut down temporarily for maintenance. So, the minimum number of nodes to define in the partition group sections is implicitly bound to the sum of a map's ''backupCount'' and ''asyncBackupCount'' properties, plus ''1'' for the original data partition.

<?xml version="1.0" encoding="UTF-8"?>


<hazelcast xsi:schemaLocation="http://www.hazelcast.com/schema/config hazelcast-config-3.1.xsd"
xmlns="http://www.hazelcast.com/schema/config"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
<partition-group enabled="true" group-type="CUSTOM">
<member-group>
<interface>10.10.10.60</interface>
</member-group>
<member-group>
<interface>10.10.10.61</interface>
</member-group>
<member-group>
<interface>10.10.10.62</interface>
</member-group>
</partition-group>
</hazelcast>

More general information regarding custom partioning is available at http://hazelcast.org/docs/latest/manual/html/partitiongroupconfig.html .

It's also recommended to use a "static" cluster discovery for the network join, and list same the nodes that are also configured in the parition groups here, so that join requests are handled by those nodes, too (and not the other nodes that are potentially prone to garbage collection delays.

After configuring a custom partitioning scheme, the data distribution may be verified, e.g. by inspecting the MBeans of the distributed maps via JMX.

= Features =

The following list gives an overview about different features that were implemented using the new cluster capabilities.

== Distributed Session Storage ==

Previously, when an Open-Xchange server was shutdown for maintenance, all user sessions that were bound to that machine were lost, i.e. the users needed to login again. With the distributed session storage, all sessions are backed by a distributed map in the cluster, so that they are no longer bound to a specific node in the cluster. When a node is shut down, the session data is still available in the cluster and can be accessed from the remaining nodes. The load-balancing techniques of the webserver then seamlessly routes the user session to another node, with no ''session expired'' errors. The distributed session storage comes with the package ''open-xchange-sessionstorage-hazelcast''. It's recommended to install this optional package in all clustered environments with multiple groupware server nodes.

'''Notes:'''
* While there's some kind of built-in session distribution among the nodes in the cluster, this should not be seen as a replacement for session-stickiness between the loadbalancer and groupware nodes, i.e. one should still configure the webserver to use sticky sessions for performance reasons.
* The distributed session storage is still an in-memory storage. While the session data is distributed and backed up on multiple nodes in the cluster, shutting down multiple or all nodes at the same time will lead to loss of the the distributed data. To avoid such data loss when shutting down a node, please follow the guidelines at [[ Updating_a_Cluster ]].

Depending on the cluster infrastructure, different backup-count configuration options might be set for the distributed session storage in the map configuration file ''sessions.properties'' in the ''hazelcast'' subdirectory:

com.openexchange.hazelcast.configuration.map.backupCount=1

The ''backupcount'' property configures the number of nodes with synchronized backups. Synchronized backups block operations until backups are successfully copied and acknowledgements are received. If 1 is set as the backup-count for example, then all entries of the map will be copied to another JVM for fail-safety. 0 means no backup. Any integer between 0 and 6. Default is 1, setting bigger than 6 has no effect.

com.openexchange.hazelcast.configuration.map.asyncBackupCount=0

The ''asyncbackup'' property configures the number of nodes with async backups. Async backups do not block operations and do not require acknowledgements. 0 means no backup. Any integer between 0 and 6. Default is 0, setting bigger than 6 has no effect.

Since session data is backed up by default continuously by multiple nodes in the cluster, the steps described in [[ Session_Migration ]] to trigger session migration to other nodes explicitly is obsolete and no longer needed with the distributed session storage.

Normally, sessions in the distributed storages are not evicted automatically, but are only removed when they're also removed from the session handler, either due to a logout operation or when exceeding the long-term session lifetime as configured by ''com.openexchange.sessiond.sessionLongLifeTime'' in ''sessiond.properties''. Under certain circumstances, i.e. the session is no longer accessed by the client and the OX node hosting the session in it's long-life container being shutdown, the remove operation from the distributed storage might not be triggered. Therefore, additionaly a maximum idle time of map-entries can be configured for the distributed sessions map via

com.openexchange.hazelcast.configuration.map.maxIdleSeconds=640000

To avoid unnecessary eviction, the value should be higher than the configured ''com.openexchange.sessiond.sessionLongLifeTime'' in ''sessiond.properties''.

== Remote Cache Invalidation ==

For faster access, groupware data is held in different caches by the server. Formerly, the caches utilized the TCP Lateral Auxiliary Cache plug in (LTCP) for the underlying JCS caches to broadcast updates and removals to caches on other OX nodes in the cluster. This could potentially lead to problems when remote invalidation was not working reliably due to network discovery problems. As an alternative, remote cache invalidation can also be performed using reliable publish/subscribe events built up on Hazelcast topics. This can be configured in the ''cache.properties'' configuration file, where the 'eventInvalidation' property can either be set to 'false' for the legacy behavior or 'true' for the new mechanism:

com.openexchange.caching.jcs.eventInvalidation=true

All nodes participating in the cluster should be configured equally.

Internally, if ''com.openexchange.caching.jcs.eventInvalidation'' is set to ''true'', LTCP is disabled in JCS caches. Instead, an internal mechanism based on distributed Hazelcast event topics is used to invalidate data throughout all nodes in the cluster after local update- and remove-operations. Put-operations aren't propagated (and haven't been with LTCP either), since all data put into caches can be locally loaded/evaluated at each node from the persistent storage layer.

Using Hazelcast-based cache invalidation also makes further configuration of the JCS auxiliaries obsolete in the ''cache.ccf'' configuration file. In that case, all ''jcs.auxiliary.LTCP.*'' configuration settings are virtually ignored. However, it's still required to mark caches that require cluster-wide invalidation via ''jcs.region.<cache_name>=LTCP'', just as before. So basically, when using the new default setting ''com.openexchange.caching.jcs.eventInvalidation=true'', it's recommended to just use the stock ''cache.ccf'' file, since no further LTCP configuration is required.

= Adminstration / Troubleshooting =

== Hazelcast Configuration ==

The underlying Hazelcast library can be configured using the file ''hazelcast.properties''.

'''Important''':<br>
By default property ''com.openexchange.hazelcast.network.interfaces'' is set to ''127.0.0.1''; meaning Hazelcast listens only to loop-back device. To build a cluster among remote nodes the appropriate network interface needs to be configured there. Leaving that property empty lets Hazelcast listen to all available network interfaces.

The Hazelcast JMX MBean can be enabled or disabled with the property ''com.openexchange.hazelcast.jmx''. The properties ''com.openexchange.hazelcast.mergeFirstRunDelay'' and ''com.openexchange.hazelcast.mergeRunDelay'' control the run intervals of the so-called ''Split Brain Handler'' of Hazelcast that initiates the cluster join process when a new node is started. More details can be found at http://www.hazelcast.com/docs/2.5/manual/single_html/#NetworkPartitioning.

The port ranges used by Hazelcast for incoming and outgoing connections can be controlled via the configuration parameters ''com.openexchange.hazelcast.networkConfig.port'', ''com.openexchange.hazelcast.networkConfig.portAutoIncrement'' and ''com.openexchange.hazelcast.networkConfig.outboundPortDefinitions''.

== Commandline Tool ==

To print out statistics about the cluster and the distributed data, the ''showruntimestats'' commandline tool can be executed witht the ''clusterstats'' ('c') argument. This provides an overview about the runtime cluster configuration of the node, other members in the cluster and distributed data structures.

== JMX ==

In the Open-Xchange server Java process, the MBean ''com.hazelcast'' can be used to monitor and manage different aspects of the underlying Hazelcast cluster. The ''com.hazelcast'' MBean provides detailed information about the cluster configuration and distributed data structures.

== Hazelcast Errors ==

When experiencing hazelcast related errors in the logfiles, most likely different versions of the packages are installed, leading to different message formats that can't be understood by nodes using another version. Examples for such errors are exceptions in hazelcast components regarding (de)serialization or other message processing.
This may happen when performing a consecutive update of all nodes in the cluster, where temporarily nodes with a heterogeneous setup try to communicate with each other. If the errors don't disappear after all nodes in the cluster have been update to the same package versions, it might be necessary to shutdown the cluster completely, so that all distributed data is cleared.

== Cluster Discovery Errors ==

* If the started OX nodes don't form a cluster, please double-check your configuration in ''hazelcast.properties''
* It's important to have the same cluster name defined in ''hazelcast.properties'' throughout all nodes in the cluster
* Especially when using multicast cluster discovery, it might take some time until the cluster is formed
* When using ''static'' cluster discovery, at least one other node in the cluster has to be configured in ''com.openexchange.hazelcast.network.join.static.nodes'' to allow joining, however, it's recommended to list all nodes in the cluster here

== Disable Cluster Features ==

The Hazelcast based clustering features can be disabled with the following property changes:
* Disable cluster discovery by setting ''com.openexchange.hazelcast.network.join'' to ''empty'' in ''hazelcast.properties''
* Disable Hazelcast by setting ''com.openexchange.hazelcast.enabled'' to false in ''hazelcast.properties''
* Disable message based cache event invalidation by setting ''com.openexchange.caching.jcs.eventInvalidation'' to ''false'' in ''cache.properties''

== Update from 6.22.1 to version 6.22.2 and above ==

As hazelcast will be used by default for the distribution of sessions starting 6.22.2 you have to adjust hazelcast according to our old cache configuration. First of all it's important that you install the open-xchange-sessionstorage-hazelcast package. This package will add the binding between hazelcast and the internal session management. Next you have to set a cluster name to the cluster.properties file (see [[#Cluster Discovery Errors]]). Furthermore you will have to add one of the two discovery modes mentioned in [[#Cluster Discovery]].

= Updating a Cluster =

Running a cluster means built-in failover on the one hand, but might require some attention when it comes to the point of upgrading the services on all nodes in the cluster. This chapter gives an overview about general concepts and hints for silent updates of the cluster.

== The Big Picture ==

Updating an OX App Suite cluster is possible in several ways. The involved steps always include

* Update the software by updating the packages through the distro's repository / software update tool
* Update the database schemas (so-called update tasks)

There are some precautions required, though.

=== Update Tasks Management ===

It is a feature of the OX App Suite middleware to automatically start update tasks on a database schema when a user tries to login whose context lives on that schema. For installations beyond a certain size, if you just update the OX App Suite software without special handling of the update tasks, user logins will trigger an uncontrolled storm of update tasks on the databases, potentially leading to resource contention, unnecessary long update tasks runtimes, excessive load on the database server, maybe even service outages.

So one key element of every update strategy is to avoid user logins on nodes which have already been updated to the new software version, while the database schemas are still on the old version. There are two fundamentally different approaches to this goal: use either a full downtime, or use a rolling update strategy.

We describe the update strategy in more detail in the next section. Note that these are still high-level outlines of the actual procedure, which requires additional details with regards to Hazelcast, given further down below.

==== Full downtime approach ====

The full downtime approach is quite straightforward and involves

* shutdown of all OX middleware nodes
* update the software on all OX App Suite (middleware and frontend) nodes
* execute the update tasks in a controlled way from one OX node
* restore the service

This is the most general approach and always available, even if the rolling approach is not available due to Hazelcast constraints.

==== Rolling strategy ====

It is possible to execute the update tasks decoupled from the real update of the rest from the cluster, days or even weeks ahead of time, with the following approach:

* If the load situation allows for it, take one node out of the loadbalancer (we call it the upgrade node). Otherwise, add a dedicated upgrade node to your cluster, identically configured to the other middleware nodes.
* Make sure there are no user sessions left on the upgrade node, and that no new sessions will be routed to that node
* update the software on the upgrade node
* execute all update tasks from the update node.

In the last step, users from affected schemas will be logged out and denied service while the update tasks are running on their database schema. This is typically a short unavailability (some minutes) for a small part (1000...7000 depending on the installation) of the user base. This unavailability is of much lower impact than the unavailability of a full downtime, but you still might want to do this in the off-business hours.

This way you end up with the production cluster running on the old version of OX App Suite, with the database already being upgraded to the next version. This is explicitly a valid and supported configuration. This approach offers the advantage that update tasks can be executed in advance, instead of doing them while the whole system is in a full maintenance downtime. Since update tasks can take some time, this is a considerable advantage.

For the actual upgrade of the production cluster, the remaining steps are:

* Upgrade and restart the OX App Suite software on one middleware node after another, one by one
* Upgrade the software on the OX App Suite frontend nodes (if these are separate nodes from the middleware nodes)

Hazelcast will ensure that sessions from nodes which you restart are taken over by other nodes in the cluster, so ideally this step works without losing user sessions.

For the rolling strategy to work as described, it is required that the old and new version of OX App Suite use compatible versions of the Hazelcast library. This is the case for most upgrades. However some upgrades must handle the situation that the new version of OX App Suite ships with a new version of Hazelcast incompatible to the version of Hazelcast shipped with the old version of OX App Suite. It will be stated in the release notes if this is the case for a given release. If so, then some additional steps are required during a rolling update to ensure session handling / invalidating during update tasks works properly. See below.

== HOWTO / step-by-step instructions ==

* Take backups of as much as possible (databases, OX config files, etc).
* Announce the maintenance to the users. The communication depends on which approach you chose: the full downtime approach will come with a full downtime for all users, while the rolling upgrade approach will result in some users will have a short loss of service while their schema upgrades.

=== Full downtime approach ===

* Initiate maintenance: Block HTTP sessions to the service. Put a reasonable maintenance page in place, probably some HTTP error 503 with a reasonable Retry-After header.
* Shutdown the service on all middleware nodes. Upgrade the software on all middleware and frontend nodes using the disto's package manager. See [[AppSuite:UpdatingOXPackages]] for details on how to do that. Don't forget the <code>touch-appsuite</code> step if required ("If you update only UI plugins without simultaneously upgrading the core UI packages to a new version").
* Start the <code>open-xchange</code> service on one node
* Execute update tasks from that node. See [[UpdateTasks]] for an explanation how to do that, in particular the [[UpdateTasks#How_to_see_all_schemas.3F|section]] about limited parallel execution.
* Start the <code>open-xchange</code> services on the middleware nodes.
* Perform some crosschecks like
** all middleware nodes joined the Hazelcast cluster
** all OSGI bundles (which are expected to be running) are running
** WebUI login is possible
** Some central functionality tests like sending mails, accessing drive, etc
* Restore service: allow HTTP sessions, remove the maintenance page.

=== Rolling Upgrade without breaking Hazelcast upgrade ===

Remember: as stated above, this is viable only if the release notes for the new version do not state that there are breaking Hazelcast changes. For example, with v7.8.4 there were breaking Hazelcast changes and in the Release Notes it was stated as follows.

https://software.open-xchange.com/products/appsuite/doc/Release_Notes_for_Release_7.8.4_2017-05-23.pdf
<blockquote>
Important - Please Note

There is a major Hazelcast library update to OX App Suite v7.8.4. This means that when updating from an earlier backend version, due to the upgraded library, it is not possible to form a cluster of nodes that run previous version of Hazelcast (i.e. exiting volatile data in the cluster will be lost during the update). A consistent Hazelcast cluster is needed for cluster-wide cache invalidation. To circumvent problems with database update tasks that need to perform cache invalidation, please follow the steps described here: http://oxpedia.org/wiki/index.php?title=AppSuite:Running_a_cluster#Upgrades_of_the_Hazelcast_library. Please also note that session migration is not possible between versions. This usually affects all user sessions that are stored in a distributed map, and will require the users to re-login after the update. Running incompatible versions of Hazelcast within a cluster will result in logentries showing the conflicting node and version information.
</blockquote>

If you find you are upgrading to a version with breaking Hazelcast changes, please consult the next section [[#Rolling_Upgrade_with_breaking_Hazelcast_upgrade]].

==== Description of the upgrade process ====

The procedure consists of a '''pre-update''' where one update node will be taken out of the HTTP traffic, to execute database update tasks from that node, and a '''real update''', where all of the cluster nodes will get updated to the new version of the software.

The pre-update will not make the new version of the software available to the users. It will run as kind of "background task", mostly invisible for the users (but see below for a description of the impact of the update tasks on user experience).

==== Pre-update ====

The following steps all refer to one special middleware node, the so-called ''upgrade node''. The other cluster nodes are not affected by this step.

* Take one middleware node (the upgrade node) out of the HTTP traffic by adjusting the apache mod_proxy tables. We propose a combination of the balancer_manager to do this during runtime without restart, but also update the config files to prevent service restarts of apache to accidentally route sessions to the upgrade node.
* Make sure there are no user sessions left on the upgrade node, and that no new sessions will be routed to that node
* Update packages on the upgrade node and restart the middleware service there. See [[AppSuite:UpdatingOXPackages]] for details on how to do that.
* Execute update tasks from that node. See [[UpdateTasks]] for an explanation how to do that.
** Note that executing update tasks on database schemas will result in users from the given database schema to be logged out and locked out during the update tasks.
** You might want to keep the load low on the DBs, to affect production operations as low as possible, and because with this decoupled update tasks approach there is no immediate time pressure. If you want to follow the [[UpdateTasks#How_to_see_all_schemas.3F|limited parallel]] approach, use a small, mild parallelity factor (e.g. 2 or maybe 4 if you know this by far does not saturate your DB platform).

==== Real Update ====

The following steps refer to all cluster nodes (but the upgrade node, which had been updated before).

* For one middleware cluster node after each nother:
** Update packages on that middleware node and restart the middleware service there. See [[AppSuite:UpdatingOXPackages]] for details on how to do that.
** Verify the node starts its bundles, joins the Hazelcast cluster, log files are clean, the node handles sessions
* For one frontend node after each other (if you've got separate frontend nodes):
** Update packages on that frontend node. See [[AppSuite:UpdatingOXPackages]] for details on how to do that.
* Finally, if required ("If you update only UI plugins without simultaneously upgrading the core UI packages to a new version"), execute <code>touch-appsuite</code> with a <code>--timestamp</code> argument as described on the page [[AppSuite:UpdatingOXPackages]]
* Perform final crosschecks like
** all middleware nodes joined the Hazelcast cluster
** all OSGI bundles (which are expected to be running) are running
** WebUI login is possible
** Some central functionality tests like sending mails, accessing drive, etc

=== Rolling Upgrade with breaking Hazelcast upgrade ===

Cf [[#Upgrades_of_the_Hazelcast_library]] below.

In principle the steps given in the previous section apply. However the upgrade needs to get the special Hazelcast Upgrade Package installed (e.g. one from <code>open-xchange-cluster-upgrade-from-76x</code>, <code>open-xchange-cluster-upgrade-from-780-782</code>, <code>open-xchange-cluster-upgrade-from-783</code>, <code>open-xchange-cluster-upgrade-from-784</code>, ...) during execution of the update tasks.

So the pre-update steps look like:

* Take one middleware node (the upgrade node) out of the HTTP traffic by adjusting the apache mod_proxy tables. We propose a combination of the balancer_manager to do this during runtime without restart, but also update the config files to prevent service restarts of apache to accidentally route sessions to the upgrade node.
* Make sure there are no user sessions left on the upgrade node, and that no new sessions will be routed to that node
* Update packages on the upgrade node and restart the middleware service there. See [[AppSuite:UpdatingOXPackages]] for details on how to do that.
* Install the special Hazelcast Upgrade Package on the upgrade node (e.g. one from <code>open-xchange-cluster-upgrade-from-76x</code>, <code>open-xchange-cluster-upgrade-from-780-782</code>, <code>open-xchange-cluster-upgrade-from-783</code>, <code>open-xchange-cluster-upgrade-from-784</code>, ...). Restart the service again.
* Execute update tasks from that node. See [[UpdateTasks]] for an explanation how to do that. You might want to keep the load low on the DBs, to affect production operations as low as possible, and because with this decoupled update tasks approach there is no immediate time pressure. If you want to follow the [[UpdateTasks#How_to_see_all_schemas.3F|limited parallel]] approach, use a small, mild parallelity factor (e.g. 2 or maybe 4 if you know this by far does not saturate your DB platform).

Note: don't worry if you don't see the upgrade node joining the legacy cluster: the upgrade node will not join the legacy cluster / not be visisble there since the upgrade node will be a so-called "native client" to the legacy cluster, and it will be created on the fly (and subsequently disposed again) for propagating an event. So also on <code>netstat</code> level the upgrade node will not have visible connections to the legacy cluster (unless for the very short timeframe when an actual even is sent). You can verify the functionality of that package by log lines like

<nowiki> Successfully initialzed Hazelcast client: <client-id>
Successfully got reference to cache event topic: cacheEvents-3
Publishing legacy cache event: <cache-event>

Successfully published legacy cache event, shutting down client after 546ms...</nowiki>

For the overly prudent it might be an idea to prepare a special test context with a test user living in its dedicated (test) schema, so you can test the functionality of this mechanis during upgrade first.

After the DB update tasks you can remove the special upgrade package again from the upgrade node.

The "Real Upgrade" procedure then looks like [[#Rolling_Upgrade_without_breaking_Hazelcast_upgrade|above]].

== Reference Documentation ==

=== Limitations ===

While in most cases a seamless, rolling upgrade of all nodes in the cluster is possible, there may be situations where nodes running a newer version of the Open-Xchange Server are not able to communicate with older nodes in the cluster, i.e. can't access distributed data or consume incompatible event notifications - especially, when the underlying Hazelcast library is part of the update, which does not support this scenario at the moment. In such cases, the release notes will contain corresponding information, so please have a look there before applying an update.

Additionally, there may always be some kind of race conditions during an update, i.e. client requests that can't be completed successfully or internal events not being deliverd to all nodes in the cluster. That's why the following information should only serve as a best-practices guide to minimize the impact of upgrades to the user experience.

=== Upgrading a single Node ===

Upgrading all nodes in the cluster should usually be done sequentially, i.o.w. one node after the other. This means that during the upgrade of one node, the node is temporarily disconnected from the other nodes in the cluster, and will join the cluster again after the update is completed. From the backend perspective, this is as easy as stopping the open-xchange service. other nodes in the cluster will recognize the disconnected node and start to repartition the shared cluster data automatically. But wait a minute - doing so would potentially lead to the webserver not registering the node being stopped immediately, resulting in temporary errors for currently logged in users until they are routed to another machine in the cluster. That's why it's good practice to tell the webserver's load balancer that the node should no longer fulfill incoming requests. The Apache Balancer Manager is an excellent tool for this ([http://httpd.apache.org/docs/2.2/mod/mod_status.html module ''mod_status'']). Look at the screen shot. Every node can be put into a disabled mode. Further requests will the redirected to other nodes in the cluster:

[[Image:balancer_manager.jpg]]

Afterwards, the open-xchange service on the disabled node can be stopped by executing:

$ /etc/init.d/open-xchange stop

or

$ service open-xchange stop

Now, the node is effectively in maintenance mode and any updates can take place. One could now verify the changed cluster infrastructure by accessing the Hazelcast MBeans either via JMX or the ''showruntimestats -c'' commandline tool (see above for details). There, the shut down node should no longer appear in the 'Member' section (com.hazelcast:type=Member).

When all upgrades are processed, the node open-xchange service can be started again by executing:

$ /etc/init.d/open-xchange start

or

$ service open-xchange start

As stated above, depending on the chosen cluster discovery mechanism, it might take some time until the node joins the cluster again. When using static cluster discovery, it will join the existing cluster usually directly during serivce startup, i.o.w. before other depending OSGi services are started. Otherwise, there might also be situations where the node cannot join the cluster directly, for example when there were no mDNS advertisments for other nodes in the cluster received yet. Then, it can take some additional time until the node finally joins the cluster. During startup of the node, you can observe the JMX console or the output of ''showruntimestats -c'' (com.hazelcast:type=Member) of another node in the cluster to verify when the node has joined.

After the node has joined, distributed data is re-partioned automatically, and the node is ready to server incoming requests again - so now the node can finally be enabled again in the load balancer configuration of the webserver. Afterwards, the next node in the cluster can be upgraded using the same procedure, until all nodes were processed.

=== Upgrades of the Hazelcast library ===

In case an upgrade includes a major update of the Hazelcast library, a newly upgraded node will usually not be able to connect to the nodes running the previous version. In this case, volatile cluster data is lost after all nodes in the cluster have been updated, including sessions held in the distributed session storage. As outlined above, the release notes will contain a corresponding warning in such cases.

Besides upgraded nodes not being able to access distributed data of the legacy cluster, this also affects new data not being available in the legacy cluster, which may cause troubles if the updated backend version needs to perform database update tasks. Database update tasks usually operate in a "blocking" way and all contexts associated with the schema being upgraded are disabled temporarily. Since context data itself is being held in caches on potentially each node in the cluster, the affected cache entries are invalidated during the database update. And, since cluster-wide cache invalidations again utilize Hazelcast functionality ([[#Remote Cache Invalidation]]), such invalidations normally won't be propagated to nodes running a previous version of the Hazelcast library.

To work around this specific scenario where an incompatible upgrade of the Hazelcast library needs to be performed along with blocking database update tasks, starting with v7.8.0, a supplementary package is available that explicitly enables the context cache invalidation of nodes running the previous Hazelcast library. This package follows the naming scheme ''open-xchange-cluster-upgrade-from-XXX'' (where XXX representing the version of the legacy version of the Open-Xchange server), and is available in the repositories for the updated server packages. This package should only be installed on the first node of the cluster that is going to be upgraded to the new version, and can be deactivated once the database upgrade tasks were executed successfully.

Once installed, a legacy cluster is discovered based on the available information in the ''hazelcast.properties'' configuration file in case cluster discovery is set to ''static''. If ''multicast'' is used, there's an alternative option to configure at least one of the addresses of the legacy cluster via ''com.openexchange.hazelcast.network.client.nodes''.

As an example, along with the server v7.8.0, a new package named ''open-xchange-cluster-upgrade-from-76x'' can be installed that aids in invalidating cluster server nodes running v7.6.x (which includes the Hazelcast library in version 3.2.4). Using this package, the recommended steps to update an OX cluster from version 7.6.x to version 7.8.0 would be:
# Pick a node from your cluster that you want to use for executing the database update tasks shipped with the new release
# Disable this node for incoming HTTP requests in your webserver configuration as described at [[#Upgrading a single Node]]
# Update the OX packages on this node, additionally install the package ''open-xchange-cluster-upgrade-from-76x''
# Restart the open-xchange services on this node
# Trigger the update task executions using the ''runUpdate'' commandline utitlty as described at [[UpdateTasks]]
# Once they are finished, uninstall the package ''open-xchange-cluster-upgrade-from-76x'' again
# Restart the open-xchange services on this node
# Re-enable the node for incoming HTTP requests in your webserver configuration as described at [[#Upgrading a single Node]]
# Upgrade all other nodes in the cluster as described at [[#Upgrading a single Node]]

Same steps apply to upgrading from v7.8.0 through v7.8.2 (incl.) to v7.8.3 using the package named ''open-xchange-cluster-upgrade-from-780-782'', since v7.8.0 through v7.8.2 (incl.) utilize Hazelcast v3.5.x, while v7.8.3 uses Hazelcast v3.6.4

Same steps apply to upgrading from v7.8.3 to v7.8.4 using the package named ''open-xchange-cluster-upgrade-from-783'', since v7.8.3 utilizes Hazelcast v3.7.1

Same steps apply to upgrading from v7.8.4 to v7.10.0 using the package named ''open-xchange-cluster-upgrade-from-784'', since v7.8.4 utilizes Hazelcast v3.8.1

'''Operations Note:''' The upgraded node will be added as so-called [http://docs.hazelcast.org/docs/2.3/manual/html/ch15.html Native Client] to the legacy Hazelcast Cluster.

<blockquote>
Native Client enables you to do all Hazelcast operations without being a member of the cluster.
[...]

However Native client is not member and relies on one of the cluster members.
</blockquote>

This means, the upgraded node will not be visible in the members list of the legacy Hazelcast cluster (<code>showruntimestats -c</code>). Furthermore, the native client will created and destructed on single context events, with the effect that connections will only be visible in the very moment of such an event. This means effectively that verification of the invalidation mechanis is only possible by actually executing the <code>runupdate</code> CLT. This should produce log lines like

Successfully initialzed Hazelcast client: <client-id>
Successfully got reference to cache event topic: cacheEvents-3
Publishing legacy cache event: <cache-event>
Successfully published legacy cache event, shutting down client after 546ms...

Most importantly, you should be able to observe correct functionality (users of affected contexts being logged out). It may be handy to prepare a dedicated schema with just test contexts inside. (How to create this is out of scope here, but hint: use <code>createschema</code> and <code>createcontext --schema-name</code>.)

=== Other Considerations ===

* It's always recommended to only upgrade one node after the other, always ensuring that the cluster has formed correctly between each shutdown/startup of a node.
* Do not stop a node while running the runUpdate script or the associated update task.
* During the time of such a rolling upgrade of all nodes, we have effectively heterogeneous software versions in the cluster, which potentially might lead to temporary inconsistencies. Therefore, all nodes in the cluster should be updated in one cycle (but still one after the other).
* Following the above guideline, it's also possible to add or remove nodes dynamically to the cluster, not only when disconnecting a node temporary for updates.
* In case of trouble, i.e. a node refuses to join the cluster again after restart, consult the logfiles first for any hints about what is causing the problem - both on the disconnected node, and also on other nodes in the network
* If there are general incompatibilities between two revisions of the Open-Xchange Server that prevent an operation in a cluster (release notes), it's recommended to choose another name for the cluster in ''cluster.properties'' for the nodes with the new version. This will temporary lead to two separate clusters during the rolling upgrade, and finally the old cluster being shut down completely after the last node was updated to the new version. While distributed data can't be migrated from one server version to another in this scenario due to incompatibilities, the uptime of the system itself is not affected, since the nodes in the new cluster are able to serve new incoming requests directly.
* When updating only UI plugins without also updating to a new version of the core UI, you also need to perform the additional step from [[AppSuite:UpdatingOXPackages#Updating_UI_plugins|Updating UI plugins]].

[[Category: AppSuite]] [[Category: Administration]] [[Category: Cluster]]

User:Dominik.epple/DocumentConverterInstall

2018-02-16T15:01:54Z

Dominik.epple: /* Final setup */

<div class="title">DocumentConverter Quickinstall Guide / Cheatsheet</div>

__TOC__

= Introduction =

This document aims to be a condensed "HOWTO" like installation walkthrough.

The other relevant documentation is to be considered as a reference.

This document describes the fully clustered setup. Simpler setups can be deduced as special case.

This document has been created and verified on CentOS7 and Debian Jessie, using OX App Suite 7.8.4.

= Design description =

We consider a clustered setup with multiple middleware nodes and multiple converter nodes.

We consider a high available connection from the middleware nodes to the converter nodes realized using HAproxy instances running locally on the middleware nodes which do simple round-robin to the Apache instances of the Fronted nodes, which will do ''correct'' session stickyness aware routing to the converter nodes.

We need the Apache instances to get correct ''session sticky'' routing behavior.

We need the HAproxy instances to connect to the Apache instances in a high available fashion. If there are other means in the infrastructure offering that functionality, this is also okay. We present the HAproxy based setup to give an example of a fully working setup.

= Software installation and configuration =

Note: this guide is written distro-agnostic, so please use the distro's appropriate command for installing packages.

Repos to be used are the standard repo definitions from https://software.open-xchange.com/products/.

== Middleware nodes ==

Prerequisites: Drive is installed and working. We need Drive to upload test documents and verify their conversion. This also assumes (since we are discussing a clustered setup) a clustered filestore is available and working. If you are only going for a single node proof-of-concept setup, that is not relevant of course.

Packages to be installed:

open-xchange-documentconverter-api open-xchange-documentconverter-client

Configuration:

<nowiki>/opt/open-xchange/etc/permissions.properties:
# assume a global switch on for testing. further config cascade stuff etc is out of scope
# of this document.
com.openexchange.capability.document_preview=true</nowiki>

<nowiki>documentconverter-client.properties:
# this needs to be adjusted. will be discussed below.
com.openexchange.documentconverter.client.remoteDocumentConverterUrl=http://host[:port]/documentconverterws</nowiki>

== Frontend nodes ==

None

== Converter nodes ==

Packages to be installed:

open-xchange-documentconverter-server open-xchange-documentconverter-api readerengine

(Note: the official documentation also mentions "pdf2svg" which at least on CentOS7 does not exist, but rather a package named "readerengine-pdf2svg" is pulled as dependency. So for the moment let's assume we don't need to install that explicitly, but if you are following this guide on Debian you should double-check.)

Configuration:

/opt/open-xchange/etc/server.properties:
# Pick a unique route, which will be configured consistently in apache
com.openexchange.server.backendRoute=DC1
# Clustered setups need to listen not only on localhost
com.openexchange.connector.networkListenerHost=*

/opt/open-xchange/documentconverter/etc/documentconverter.properties
# TODO figure out the recommended Cache setup for a clustered scenario
com.openexchange.documentconverter.RemoteCacheUrls = ?
# Default is 3. Adjust for your sizing.
com.openexchange.documentconverter.jobProcessorCount=3

Services configuration:

systemctl start open-xchange-documentconverter-server
systemctl enable open-xchange-documentconverter-server

Note: you don't need ConfigDB / any DB access. Don't connect the converter nodes to any database. Don't make it join any Hazelcast cluster. No further configuration is required.

If you are annoyed by the "unable to connect to ConfigDB" (false) error messages in the logs, you can configure logback to suppress them (TODO: add information how to do that).

= Configure the middleware to converter connectivity =

So that was the trivial part. Now it gets interesting.

== First test: direct connectivity ==

Pick a middleware node. Configure direct connection to one converter service which is called <code>dc1</code>. By default the service listens on port <code>8008</code> and listens on the path <code>/documentconverterws</code>.

<nowiki># curl http://dc1:8008/documentconverterws/
<html>
<head><meta charset="UTF-8"><title>Open-Xchange DC</title></head>
<body><h1 align="center">OX Software GmbH DC</h1>
<p>WebService is running...</p>
<p>Error Code: 0</p>
<p>API: v5</p></body>
</html></nowiki>

In particular, you should see

* HTTP status code 200 (not shown for clarity, but you can verify with <code>curl -v</code>)
* <code>WebService is running...</code>
* <code>Error Code: 0".</code>

If that is successful, go ahead and configure that URL in the middleware node

<nowiki>documentconverter-client.properties:
# this needs to be adjusted. will be discussed below.
com.openexchange.documentconverter.client.remoteDocumentConverterUrl=http://dc1:8008/documentconverterws</nowiki>

Do a testing cycle as described below.

If everything works, you confirmed that the middleware node and the converter node are configured correctly.

Now ensure all your middleware nodes and all your converter nodes are configured likewise. Make sure the converter nodes get unique <code>com.openexchange.server.backendRoute</code> values (see above).

== Second test: use Apache for loadbalancing ==

Pick one frontend node to configure its Apache for loadbalancing for the converter nodes.

There are sample configuration stanzas in our default configuration. They are just fine. Just make sure the <code>route</code> parameters match the ones from the converter nodes.

I want to emphasize to configure the <code>Allow</code> line correctly. The <code>/documentconverterws</code> endpoint must not be made available publicly!

A sample Apache configuration looks like

<nowiki><Proxy balancer://oxcluster_docs>
Order Deny,Allow
Deny from all
# configure the allowed IPs such that only the middleware nodes are able to access
# the /documentconverterws endpoint must not be made available publicly!
Allow from 10.0.1
BalancerMember http://dc1:8008 timeout=100 smax=0 ttl=60 retry=60 loadfactor=50 keepalive=On route=DC1
BalancerMember http://dc2:8008 timeout=100 smax=0 ttl=60 retry=60 loadfactor=50 keepalive=On route=DC2
# add further converter nodes, as many as you have
ProxySet stickysession=JSESSIONID|jsessionid scolonpathdelim=On
SetEnv proxy-initial-not-pooled
SetEnv proxy-sendchunked
</Proxy>

ProxyPass /documentconverterws balancer://oxcluster_docs/documentconverterws</nowiki>

With that configuration in place, test the connectivity from the middleware node to that endpoint, e.g. (assuming the frontend node is called <code>frontend1</code>):

<nowiki># curl http://frontend1/documentconverterws
<html>
<head><meta charset="UTF-8"><title>Open-Xchange DC</title></head>
<body><h1 align="center">OX Software GmbH DC</h1>
<p>WebService is running...</p>
<p>Error Code: 0</p>
<p>API: v5</p></body>
</html></nowiki>

The response looks exactly like before. The difference is just that we access the service now via Apache.

Execute the test multiple times. It must answer fast every time. If some answers are slow (maybe only once or twice) that is an indication that some route configuration was incorrect and Apache disabled some converter targets.

Make sure you are actually getting responses from the different converter nodes by whatever means suit you (e.g. tcpdump on the converter nodes, looking at Apache's <code>balancer-manager</code> to verify all nodes show up as <code>Ok</code>, the <code>Elected</code> number increases equally for all converter nodes, etc.)

Configure that endpoint in your test middleware node (<code>com.openexchange.documentconverter.client.remoteDocumentConverterUrl</code>). Restart the middleware service.

Do a full test cycle as described below and in particular make sure that in pop-out view all preview images are rendered correctly. That is the testcase for correct session stickyness.

If everything works correctly, make sure the Apache configuration is adjusted accordingly on all your Apache / Fronted nodes.

Now we have high availability for the converter nodes, but the Apache service is still a SPOF.

== Third test: adding local HAproxy instances ==

We add local HAproxy instances to eliminate the Apache SPOF.

Assuming HAproxy is already running locally for other services. If not, install it and do a basic configuration according to [HAproxy].

Add a stanza like

<nowiki>listen converter
mode http
option dontlognull
option redispatch
no option httpclose
timeout connect 10s
timeout client 2m
timeout server 2m
bind 127.0.0.1:8008
balance roundrobin
option httpchk GET /documentconverterws
server frontend1 frontend1:80 check port 80 inter 6000 rise 3 fall 1
server frontend2 frontend2:80 check port 80 inter 6000 rise 3 fall 1
# add more frontend nodes if you have them</nowiki>

Re-test that also this listener works correctly.

<nowiki># curl http://127.0.0.1:8008/documentconverterws
<html>
<head><meta charset="UTF-8"><title>Open-Xchange DC</title></head>
<body><h1 align="center">OX Software GmbH DC</h1>
<p>WebService is running...</p>
<p>Error Code: 0</p>
<p>API: v5</p></body>
</html></nowiki>

Same as before, execute the test multiple times, and verify in the Apache's <code>balancer-manager</code>s that everything works as expected.

Look at the HAproxy stats endpoint to verify all frontend nodes show as OK and get roughly the same number of sessions.

If that looks good, configure the HAproxy endpoint <code>http://127.0.0.1:8008/documentconverterws</code> in your test middleware node (<code>com.openexchange.documentconverter.client.remoteDocumentConverterUrl</code>). Restart the middleware service.

Do a full test cycle (as described below).

== Final setup ==

Assuming the test middleware node works as described in the previous section, do the same HAproxy configuration on all middleware nodes and configure the HAproxy endpoint <code>http://127.0.0.1:8008/documentconverterws</code> in all middleware nodes' documenconverter-client configuration.

The fully clustered setup should be working now.

= Testing =

It is highly recommended to make sure you always end up on the same middleware node for testing, by whatever means is required in your infrastructure to do so. For this document we assume you can access them directly (internal testing). Otherwise you'd need to expose special entry points to your setup.

If you can't ensure that, you need to configure all middleware nodes absolutely identical also during setup and testing, which is very cumbersome and error-prone.

If you reconfigured something on the middleware node(s), restart the service there.

<nowiki>service open-xchange restart</nowiki>

Wipe out caches on the converter node(s) and restart the service there:

<nowiki>service open-xchange-documentconverter-server stop
rm -rf /var/spool/open-xchange/documentconverter/readerengine.*/*
service open-xchange-documentconverter-server start</nowiki>

(Careful about your paths and when copy-pasting; I don't take responsibility of you remove something wrong.)

If your testuser is logged in from a previous test, log out.

Login your testuser.

Access Drive. Upload test documents if not done before. Preferably some small trivial docs, also some large multi-page docs with embedded diagrams etc.

Switch to "icons" or "tiles" view. You should get nice preview thumbnails for each document. (This is only a relevant test for the first time since the previews are stored in the database. TODO: find out how to wipe / re-test that.)

Click the "eye" icon to start the Document Viewer. You should be able to view documents with reasonable speed. Scrolling through the pages should be possible and fast.

Click the "pop out" icon to the top right to use pop-out view. Verify all pages are rendered correctly in the full-page view and in the thumbnail previews.

= Troubleshooting =

In most of the times in our experiments if something was not working, it was a misconfiguration on network level. (Wrong hostnames, IPs, ports, missing firewall adjustments, etc).

User:Dominik.epple/DocumentConverterInstall

2018-02-16T14:52:38Z

Dominik.epple:

<div class="title">DocumentConverter Quickinstall Guide / Cheatsheet</div>

__TOC__

= Introduction =

This document aims to be a condensed "HOWTO" like installation walkthrough.

The other relevant documentation is to be considered as a reference.

This document describes the fully clustered setup. Simpler setups can be deduced as special case.

This document has been created and verified on CentOS7 and Debian Jessie, using OX App Suite 7.8.4.

= Design description =

We consider a clustered setup with multiple middleware nodes and multiple converter nodes.

We consider a high available connection from the middleware nodes to the converter nodes realized using HAproxy instances running locally on the middleware nodes which do simple round-robin to the Apache instances of the Fronted nodes, which will do ''correct'' session stickyness aware routing to the converter nodes.

We need the Apache instances to get correct ''session sticky'' routing behavior.

We need the HAproxy instances to connect to the Apache instances in a high available fashion. If there are other means in the infrastructure offering that functionality, this is also okay. We present the HAproxy based setup to give an example of a fully working setup.

= Software installation and configuration =

Note: this guide is written distro-agnostic, so please use the distro's appropriate command for installing packages.

Repos to be used are the standard repo definitions from https://software.open-xchange.com/products/.

== Middleware nodes ==

Prerequisites: Drive is installed and working. We need Drive to upload test documents and verify their conversion. This also assumes (since we are discussing a clustered setup) a clustered filestore is available and working. If you are only going for a single node proof-of-concept setup, that is not relevant of course.

Packages to be installed:

open-xchange-documentconverter-api open-xchange-documentconverter-client

Configuration:

<nowiki>/opt/open-xchange/etc/permissions.properties:
# assume a global switch on for testing. further config cascade stuff etc is out of scope
# of this document.
com.openexchange.capability.document_preview=true</nowiki>

<nowiki>documentconverter-client.properties:
# this needs to be adjusted. will be discussed below.
com.openexchange.documentconverter.client.remoteDocumentConverterUrl=http://host[:port]/documentconverterws</nowiki>

== Frontend nodes ==

None

== Converter nodes ==

Packages to be installed:

open-xchange-documentconverter-server open-xchange-documentconverter-api readerengine

(Note: the official documentation also mentions "pdf2svg" which at least on CentOS7 does not exist, but rather a package named "readerengine-pdf2svg" is pulled as dependency. So for the moment let's assume we don't need to install that explicitly, but if you are following this guide on Debian you should double-check.)

Configuration:

/opt/open-xchange/etc/server.properties:
# Pick a unique route, which will be configured consistently in apache
com.openexchange.server.backendRoute=DC1
# Clustered setups need to listen not only on localhost
com.openexchange.connector.networkListenerHost=*

/opt/open-xchange/documentconverter/etc/documentconverter.properties
# TODO figure out the recommended Cache setup for a clustered scenario
com.openexchange.documentconverter.RemoteCacheUrls = ?
# Default is 3. Adjust for your sizing.
com.openexchange.documentconverter.jobProcessorCount=3

Services configuration:

systemctl start open-xchange-documentconverter-server
systemctl enable open-xchange-documentconverter-server

Note: you don't need ConfigDB / any DB access. Don't connect the converter nodes to any database. Don't make it join any Hazelcast cluster. No further configuration is required.

If you are annoyed by the "unable to connect to ConfigDB" (false) error messages in the logs, you can configure logback to suppress them (TODO: add information how to do that).

= Configure the middleware to converter connectivity =

So that was the trivial part. Now it gets interesting.

== First test: direct connectivity ==

Pick a middleware node. Configure direct connection to one converter service which is called <code>dc1</code>. By default the service listens on port <code>8008</code> and listens on the path <code>/documentconverterws</code>.

<nowiki># curl http://dc1:8008/documentconverterws/
<html>
<head><meta charset="UTF-8"><title>Open-Xchange DC</title></head>
<body><h1 align="center">OX Software GmbH DC</h1>
<p>WebService is running...</p>
<p>Error Code: 0</p>
<p>API: v5</p></body>
</html></nowiki>

In particular, you should see

* HTTP status code 200 (not shown for clarity, but you can verify with <code>curl -v</code>)
* <code>WebService is running...</code>
* <code>Error Code: 0".</code>

If that is successful, go ahead and configure that URL in the middleware node

<nowiki>documentconverter-client.properties:
# this needs to be adjusted. will be discussed below.
com.openexchange.documentconverter.client.remoteDocumentConverterUrl=http://dc1:8008/documentconverterws</nowiki>

Do a testing cycle as described below.

If everything works, you confirmed that the middleware node and the converter node are configured correctly.

Now ensure all your middleware nodes and all your converter nodes are configured likewise. Make sure the converter nodes get unique <code>com.openexchange.server.backendRoute</code> values (see above).

== Second test: use Apache for loadbalancing ==

Pick one frontend node to configure its Apache for loadbalancing for the converter nodes.

There are sample configuration stanzas in our default configuration. They are just fine. Just make sure the <code>route</code> parameters match the ones from the converter nodes.

I want to emphasize to configure the <code>Allow</code> line correctly. The <code>/documentconverterws</code> endpoint must not be made available publicly!

A sample Apache configuration looks like

<nowiki><Proxy balancer://oxcluster_docs>
Order Deny,Allow
Deny from all
# configure the allowed IPs such that only the middleware nodes are able to access
# the /documentconverterws endpoint must not be made available publicly!
Allow from 10.0.1
BalancerMember http://dc1:8008 timeout=100 smax=0 ttl=60 retry=60 loadfactor=50 keepalive=On route=DC1
BalancerMember http://dc2:8008 timeout=100 smax=0 ttl=60 retry=60 loadfactor=50 keepalive=On route=DC2
# add further converter nodes, as many as you have
ProxySet stickysession=JSESSIONID|jsessionid scolonpathdelim=On
SetEnv proxy-initial-not-pooled
SetEnv proxy-sendchunked
</Proxy>

ProxyPass /documentconverterws balancer://oxcluster_docs/documentconverterws</nowiki>

With that configuration in place, test the connectivity from the middleware node to that endpoint, e.g. (assuming the frontend node is called <code>frontend1</code>):

<nowiki># curl http://frontend1/documentconverterws
<html>
<head><meta charset="UTF-8"><title>Open-Xchange DC</title></head>
<body><h1 align="center">OX Software GmbH DC</h1>
<p>WebService is running...</p>
<p>Error Code: 0</p>
<p>API: v5</p></body>
</html></nowiki>

The response looks exactly like before. The difference is just that we access the service now via Apache.

Execute the test multiple times. It must answer fast every time. If some answers are slow (maybe only once or twice) that is an indication that some route configuration was incorrect and Apache disabled some converter targets.

Make sure you are actually getting responses from the different converter nodes by whatever means suit you (e.g. tcpdump on the converter nodes, looking at Apache's <code>balancer-manager</code> to verify all nodes show up as <code>Ok</code>, the <code>Elected</code> number increases equally for all converter nodes, etc.)

Configure that endpoint in your test middleware node (<code>com.openexchange.documentconverter.client.remoteDocumentConverterUrl</code>). Restart the middleware service.

Do a full test cycle as described below and in particular make sure that in pop-out view all preview images are rendered correctly. That is the testcase for correct session stickyness.

If everything works correctly, make sure the Apache configuration is adjusted accordingly on all your Apache / Fronted nodes.

Now we have high availability for the converter nodes, but the Apache service is still a SPOF.

== Third test: adding local HAproxy instances ==

We add local HAproxy instances to eliminate the Apache SPOF.

Assuming HAproxy is already running locally for other services. If not, install it and do a basic configuration according to [HAproxy].

Add a stanza like

<nowiki>listen converter
mode http
option dontlognull
option redispatch
no option httpclose
timeout connect 10s
timeout client 2m
timeout server 2m
bind 127.0.0.1:8008
balance roundrobin
option httpchk GET /documentconverterws
server frontend1 frontend1:80 check port 80 inter 6000 rise 3 fall 1
server frontend2 frontend2:80 check port 80 inter 6000 rise 3 fall 1
# add more frontend nodes if you have them</nowiki>

Re-test that also this listener works correctly.

<nowiki># curl http://127.0.0.1:8008/documentconverterws
<html>
<head><meta charset="UTF-8"><title>Open-Xchange DC</title></head>
<body><h1 align="center">OX Software GmbH DC</h1>
<p>WebService is running...</p>
<p>Error Code: 0</p>
<p>API: v5</p></body>
</html></nowiki>

Same as before, execute the test multiple times, and verify in the Apache's <code>balancer-manager</code>s that everything works as expected.

Look at the HAproxy stats endpoint to verify all frontend nodes show as OK and get roughly the same number of sessions.

If that looks good, configure the HAproxy endpoint <code>http://127.0.0.1:8008/documentconverterws</code> in your test middleware node (<code>com.openexchange.documentconverter.client.remoteDocumentConverterUrl</code>). Restart the middleware service.

Do a full test cycle (as described below).

== Final setup ==

Assuming the test middleware node works as described in the previous section, do the same HAproxy configuration on all middleware nodes and configure the HAproxy endpoint <code>http://127.0.0.1:8008/documentconverterws</code> in all middleware nodes' middleware configuration.

The fully clustered setup should be working now.

= Testing =

It is highly recommended to make sure you always end up on the same middleware node for testing, by whatever means is required in your infrastructure to do so. For this document we assume you can access them directly (internal testing). Otherwise you'd need to expose special entry points to your setup.

If you can't ensure that, you need to configure all middleware nodes absolutely identical also during setup and testing, which is very cumbersome and error-prone.

If you reconfigured something on the middleware node(s), restart the service there.

<nowiki>service open-xchange restart</nowiki>

Wipe out caches on the converter node(s) and restart the service there:

<nowiki>service open-xchange-documentconverter-server stop
rm -rf /var/spool/open-xchange/documentconverter/readerengine.*/*
service open-xchange-documentconverter-server start</nowiki>

(Careful about your paths and when copy-pasting; I don't take responsibility of you remove something wrong.)

If your testuser is logged in from a previous test, log out.

Login your testuser.

Access Drive. Upload test documents if not done before. Preferably some small trivial docs, also some large multi-page docs with embedded diagrams etc.

Switch to "icons" or "tiles" view. You should get nice preview thumbnails for each document. (This is only a relevant test for the first time since the previews are stored in the database. TODO: find out how to wipe / re-test that.)

Click the "eye" icon to start the Document Viewer. You should be able to view documents with reasonable speed. Scrolling through the pages should be possible and fast.

Click the "pop out" icon to the top right to use pop-out view. Verify all pages are rendered correctly in the full-page view and in the thumbnail previews.

= Troubleshooting =

In most of the times in our experiments if something was not working, it was a misconfiguration on network level. (Wrong hostnames, IPs, ports, missing firewall adjustments, etc).

User:Dominik.epple/NFS4

2018-02-16T11:49:51Z

Dominik.epple: Created page with "<div class="title">NFS4</div> __TOC__ = Overview = While for production we nowadays nearly exclusively see Object Storage based setup, for POC setups it is still convenient..."

<div class="title">NFS4</div>

__TOC__

= Overview =

While for production we nowadays nearly exclusively see Object Storage based setup, for POC setups it is still convenient to use NFS as filestore. Smaller footprint, less efforts, functional and fast enough for some functional testing.

However one disadvantage is that all NFS client machines need to have synchronized system userIDs in order for access to work, which is often not accounted for during initial setup and cumbersome to fix later.

One possible approach is to use NFS4 with Kerberos as in that flavor access is not granted based on numerical userIDs, but rather symbolic usernames.

The drawback of this solution is a slightly more complicated setup, but, once figured out how it works, it is surprisingly robust in first tests.

== KDC setup ==

Standard stuff, no special things.

Initialize the stash.

Create host keys for every host with host/hostname.fqdn and nfs/hostname.fqdn. Export to a keytab and transfer to NFS client as /etc/krb5.keytab.

Create a kerberos principal for the open-xchange user. Probably a high (unlimited) ticket lifetime will be helpful.

== NFS Server Setup ==

Decide for a directory to export. In a standard setup with exported user homedirectories you can use the filestore/ subdirectory of the open-xchange users home directory.

Standard linux practice is to bindmount that under some /export toplevel export point.

/etc/exports file:

/export kclient.lan(rw,fsid=0,sec=krb5i)

CentOS 7:

yum install nfs-utils

systemctl restart nfs
systemctl enable nfs

systemctl restart nfs-secure
systemctl enable nfs-secure

== NFS Client Setup ==

CentOS7:

systemctl restart nfs-secure
systemctl enable nfs-secure

mount -t nfs4 -o sec=krb5i knfs:/ /mnt

== Service Setup ==

For accessing data, the OX service needs a kerberos ticket.

# su - open-xchange
$ kinit

That is somewhat clumsy (TODO: find out how to automate that).

User:Dominik.epple/DocumentConverterInstall

2018-02-16T09:46:30Z

Dominik.epple: /* Troubleshooting */

<div class="title">DocumentConverter Quickinstall Guide / Cheatsheet</div>

__TOC__

= Introduction =

This document aims to be a condensed "HOWTO" like installation walkthrough.

The other relevant documentation is to be considered as a reference.

This document describes the fully clustered setup. Simpler setups can be deduced as special case.

This document has been created and verified on CentOS7 and OX App Suite 7.8.4.

= Design description =

We consider a clustered setup with multiple middleware nodes and multiple converter nodes.

We consider a high available connection from the middleware nodes to the converter nodes realized using HAproxy instances running locally on the middleware nodes which do simple round-robin to the Apache instances of the Fronted nodes, which will do ''correct'' session stickyness aware routing to the converter nodes.

We need the Apache instances to get correct ''session sticky'' routing behavior.

We need the HAproxy instances to connect to the Apache instances in a high available fashion. If there are other means in the infrastructure offering that functionality, this is also okay. We present the HAproxy based setup to give an example of a fully working setup.

= Software installation and configuration =

== Middleware nodes ==

Prerequisites: Drive is installed and working. We need Drive to upload test documents and verify their conversion. This also assumes (since we a discussing a clustered setup) a clustered filestore is available and working. If you are only going for a single node proof-of-concept setup, that is not relevant of course.

Packages to be installed:

open-xchange-documentconverter-api open-xchange-documentconverter-client

Configuration:

<nowiki>/opt/open-xchange/etc/permissions.properties:
# assume a global switch on for testing. further config cascade stuff etc is out of scope
# of this document.
com.openexchange.capability.document_preview=true</nowiki>

<nowiki>documentconverter-client.properties:
# this needs to be adjusted. will be discussed below.
com.openexchange.documentconverter.client.remoteDocumentConverterUrl=http://host[:port]/documentconverterws</nowiki>

== Frontend nodes ==

None

== Converter nodes ==

Packages to be installed:

open-xchange-documentconverter-server open-xchange-documentconverter-api readerengine

(Note: the official documentation also mentions "pdf2svg" which at least on CentOS7 does not exist, but rather a package named "readerengine-pdf2svg" is pulled as dependency. So for the moment let's assume we don't need to install that explicitly, but if you are following this guide on Debian you should double-check.)

Configuration:

/opt/open-xchange/etc/server.properties:
# Pick a unique route, which will be configured consistently in apache
com.openexchange.server.backendRoute=DC1
# Clustered setups need to listen not only on localhost
com.openexchange.connector.networkListenerHost=*

/opt/open-xchange/documentconverter/etc/documentconverter.properties
# TODO figure out the recommended Cache setup for a clustered scenario
com.openexchange.documentconverter.RemoteCacheUrls = ?
# Default is 3. Adjust for your sizing.
com.openexchange.documentconverter.jobProcessorCount=3

Services configuration:

systemctl start open-xchange-documentconverter-server
systemctl enable open-xchange-documentconverter-server

Note: you don't need ConfigDB / any DB access. Don't connect the converter nodes to any database. Don't make it join any Hazelcast cluster. No further configuration is required.

If you are annoyed by the "unable to connect to ConfigDB" (false) error messages in the logs, you can configure logback to suppress them (TODO: add information how to do that).

= Configure the middleware to converter connectivity =

So that was the trivial part. Now it gets interesting.

== First test: direct connectivity ==

Pick a middleware node. Configure direct connection to one converter service which is called <code>dc1</code>. By default the service listens on port <code>8008</code> and listens on the path <code>/documentconverterws</code>.

<nowiki># curl http://dc1:8008/documentconverterws/
<html>
<head><meta charset="UTF-8"><title>Open-Xchange DC</title></head>
<body><h1 align="center">OX Software GmbH DC</h1>
<p>WebService is running...</p>
<p>Error Code: 0</p>
<p>API: v5</p></body>
</html></nowiki>

That's how it should look like. HTTP status code 200 (not shown for clarity, but you can verify with curl -v, "WebService is running...", "Error Code: 0".

If that is successful, go ahead and configure that URL in the middleware node

<nowiki>documentconverter-client.properties:
# this needs to be adjusted. will be discussed below.
com.openexchange.documentconverter.client.remoteDocumentConverterUrl=http://dc1:8008/documentconverterws</nowiki>

Do a testing cycle as described below.

If everything works, you confirmed that the middleware node and the converter node are configured correctly.

Now ensure all your middleware nodes and all your converter nodes are configured likewise. Make sure the converter nodes get unique <code>com.openexchange.server.backendRoute</code> values (see above).

== Second test: use Apache for loadbalancing ==

Pick one frontend node to configure its Apache for loadbalancing for the converter nodes.

There are sample configuration stanzas in our default configuration. They are just fine. Just make sure the <code>route</code> parameters match the ones from the converter nodes.

I want to emphasize to configure the <code>Allow</code> line correctly. The <code>/documentconverterws</code> endpoint must not be made available publicly!

A sample Apache configuration looks like

<nowiki><Proxy balancer://oxcluster_docs>
Order Deny,Allow
Deny from all
# configure the allowed IPs such that only the middleware nodes are able to access
# the /documentconverterws endpoint must not be made available publicly!
Allow from 10.0.1
BalancerMember http://dc1:8008 timeout=100 smax=0 ttl=60 retry=60 loadfactor=50 keepalive=On route=DC1
BalancerMember http://dc2:8008 timeout=100 smax=0 ttl=60 retry=60 loadfactor=50 keepalive=On route=DC2
# add further converter nodes, as many as you have
ProxySet stickysession=JSESSIONID|jsessionid scolonpathdelim=On
SetEnv proxy-initial-not-pooled
SetEnv proxy-sendchunked
</Proxy>

ProxyPass /documentconverterws balancer://oxcluster_docs/documentconverterws</nowiki>

With that configuration in place, test the connectivity from the middleware node to that endpoint, e.g. (assuming the frontend node is called <code>frontend1</code>):

<nowiki># curl http://frontend1/documentconverterws
<html>
<head><meta charset="UTF-8"><title>Open-Xchange DC</title></head>
<body><h1 align="center">OX Software GmbH DC</h1>
<p>WebService is running...</p>
<p>Error Code: 0</p>
<p>API: v5</p></body>
</html></nowiki>

The response looks exactly like before. The difference is just that we access the service now via Apache.

Execute the test multiple times. It must answer fast every time. If some answers are slow (maybe only once or twice) that is an indication that some route configuration was incorrect and Apache disabled some converter targets.

Make sure you are actually getting responses from the different converter nodes by whatever means suit you (e.g. tcpdump on the converter nodes, looking at Apache's <code>balancer-manager</code> to verify all nodes show up as <code>Ok</code>, the <code>Elected</code> number increases equally for all converter nodes, etc.)

Configure that endpoint in your test middleware node (<code>com.openexchange.documentconverter.client.remoteDocumentConverterUrl</code>). Restart the middleware service.

Do a full test cycle as described below and in particular make sure that in pop-out view all preview images are rendered correctly. That is the testcase for correct session stickyness.

If everything works correctly, make sure the Apache configuration is adjusted accordingly on all your Apache / Fronted nodes.

Now we have high availability for the converter nodes, but the Apache service is still a SPOF.

== Third test: adding local HAproxy instances ==

We add local HAproxy instances to eliminate the Apache SPOF.

Assuming HAproxy is already running locally for other services. If not, install it and do a basic configuration according to [HAproxy].

Add a stanza like

<nowiki>listen converter
mode http
option dontlognull
option redispatch
no option httpclose
timeout connect 10s
timeout client 2m
timeout server 2m
bind 127.0.0.1:8008
balance roundrobin
option httpchk GET /documentconverterws
server frontend1 frontend1:80 check port 80 inter 6000 rise 3 fall 1
server frontend2 frontend2:80 check port 80 inter 6000 rise 3 fall 1
# add more frontend nodes if you have them</nowiki>

Re-test that also this listener works correctly.

<nowiki># curl http://127.0.0.1:8008/documentconverterws
<html>
<head><meta charset="UTF-8"><title>Open-Xchange DC</title></head>
<body><h1 align="center">OX Software GmbH DC</h1>
<p>WebService is running...</p>
<p>Error Code: 0</p>
<p>API: v5</p></body>
</html></nowiki>

Same as before, execute the test multiple times, and verify in the Apache's <code>balancer-manager</code>s that everything works as expected.

Look at the HAproxy stats endpoint to verify all frontend nodes show as OK and get roughly the same number of sessions.

If that looks good, configure the HAproxy endpoint <code>http://127.0.0.1:8008/documentconverterws</code> in your test middleware node (<code>com.openexchange.documentconverter.client.remoteDocumentConverterUrl</code>). Restart the middleware service.

Do a full test cycle (as described below).

== Final setup ==

Assuming the test middleware node works as described in the previous section, do the same HAproxy configuration on all middleware nodes and configure the HAproxy endpoint <code>http://127.0.0.1:8008/documentconverterws</code> in all middleware nodes' middleware configuration.

The fully clustered setup should be working now.

= Testing =

It is highly recommended to make sure you always end up on the same middleware node for testing, by whatever means is required in your infrastructure to do so. For this document we assume you can access them directly (internal testing). Otherwise you'd need to expose special entry points to your setup.

If you can't ensure that, you need to configure all middleware nodes absolutely identical also during setup and testing, which is very cumbersome and error-prone.

If you reconfigured something on the middleware node(s), restart the service there.

<nowiki>service open-xchange restart</nowiki>

Wipe out caches on the converter node(s) and restart the service there:

<nowiki>service open-xchange-documentconverter-server stop
rm -rf /var/spool/open-xchange/documentconverter/readerengine.*/*
service open-xchange-documentconverter-server start</nowiki>

(Careful about your paths and when copy-pasting; I don't take responsibility of you remove something wrong.)

If your testuser is logged in from a previous test, log out.

Login your testuser.

Access Drive. Upload test documents if not done before. Preferably some small trivial docs, also some large multi-page docs with embedded diagrams etc.

Switch to "icons" or "tiles" view. You should get nice preview thumbnails for each document. (This is only a relevant test for the first time since the previews are stored in the database. TODO: find out how to wipe / re-test that.)

Click the "eye" icon to start the Document Viewer. You should be able to view documents with reasonable speed. Scrolling through the pages should be possible and fast.

Click the "pop out" icon to the top right to use pop-out view. Verify all pages are rendered correctly in the full-page view and in the thumbnail previews.

= Troubleshooting =

In most of the times in our experiments if something was not working, it was a misconfiguration on network level. (Wrong hostnames, IPs, ports, missing firewall adjustments, etc).

User:Dominik.epple/DocumentConverterInstall

2018-02-16T09:37:25Z

Dominik.epple:

<div class="title">DocumentConverter Quickinstall Guide / Cheatsheet</div>

__TOC__

= Introduction =

This document aims to be a condensed "HOWTO" like installation walkthrough.

The other relevant documentation is to be considered as a reference.

This document describes the fully clustered setup. Simpler setups can be deduced as special case.

This document has been created and verified on CentOS7 and OX App Suite 7.8.4.

= Design description =

We consider a clustered setup with multiple middleware nodes and multiple converter nodes.

We consider a high available connection from the middleware nodes to the converter nodes realized using HAproxy instances running locally on the middleware nodes which do simple round-robin to the Apache instances of the Fronted nodes, which will do ''correct'' session stickyness aware routing to the converter nodes.

We need the Apache instances to get correct ''session sticky'' routing behavior.

We need the HAproxy instances to connect to the Apache instances in a high available fashion. If there are other means in the infrastructure offering that functionality, this is also okay. We present the HAproxy based setup to give an example of a fully working setup.

= Software installation and configuration =

== Middleware nodes ==

Prerequisites: Drive is installed and working. We need Drive to upload test documents and verify their conversion. This also assumes (since we a discussing a clustered setup) a clustered filestore is available and working. If you are only going for a single node proof-of-concept setup, that is not relevant of course.

Packages to be installed:

open-xchange-documentconverter-api open-xchange-documentconverter-client

Configuration:

<nowiki>/opt/open-xchange/etc/permissions.properties:
# assume a global switch on for testing. further config cascade stuff etc is out of scope
# of this document.
com.openexchange.capability.document_preview=true</nowiki>

<nowiki>documentconverter-client.properties:
# this needs to be adjusted. will be discussed below.
com.openexchange.documentconverter.client.remoteDocumentConverterUrl=http://host[:port]/documentconverterws</nowiki>

== Frontend nodes ==

None

== Converter nodes ==

Packages to be installed:

open-xchange-documentconverter-server open-xchange-documentconverter-api readerengine

(Note: the official documentation also mentions "pdf2svg" which at least on CentOS7 does not exist, but rather a package named "readerengine-pdf2svg" is pulled as dependency. So for the moment let's assume we don't need to install that explicitly, but if you are following this guide on Debian you should double-check.)

Configuration:

/opt/open-xchange/etc/server.properties:
# Pick a unique route, which will be configured consistently in apache
com.openexchange.server.backendRoute=DC1
# Clustered setups need to listen not only on localhost
com.openexchange.connector.networkListenerHost=*

/opt/open-xchange/documentconverter/etc/documentconverter.properties
# TODO figure out the recommended Cache setup for a clustered scenario
com.openexchange.documentconverter.RemoteCacheUrls = ?
# Default is 3. Adjust for your sizing.
com.openexchange.documentconverter.jobProcessorCount=3

Services configuration:

systemctl start open-xchange-documentconverter-server
systemctl enable open-xchange-documentconverter-server

Note: you don't need ConfigDB / any DB access. Don't connect the converter nodes to any database. Don't make it join any Hazelcast cluster. No further configuration is required.

If you are annoyed by the "unable to connect to ConfigDB" (false) error messages in the logs, you can configure logback to suppress them (TODO: add information how to do that).

= Configure the middleware to converter connectivity =

So that was the trivial part. Now it gets interesting.

== First test: direct connectivity ==

Pick a middleware node. Configure direct connection to one converter service which is called <code>dc1</code>. By default the service listens on port <code>8008</code> and listens on the path <code>/documentconverterws</code>.

<nowiki># curl http://dc1:8008/documentconverterws/
<html>
<head><meta charset="UTF-8"><title>Open-Xchange DC</title></head>
<body><h1 align="center">OX Software GmbH DC</h1>
<p>WebService is running...</p>
<p>Error Code: 0</p>
<p>API: v5</p></body>
</html></nowiki>

That's how it should look like. HTTP status code 200 (not shown for clarity, but you can verify with curl -v, "WebService is running...", "Error Code: 0".

If that is successful, go ahead and configure that URL in the middleware node

<nowiki>documentconverter-client.properties:
# this needs to be adjusted. will be discussed below.
com.openexchange.documentconverter.client.remoteDocumentConverterUrl=http://dc1:8008/documentconverterws</nowiki>

Do a testing cycle as described below.

If everything works, you confirmed that the middleware node and the converter node are configured correctly.

Now ensure all your middleware nodes and all your converter nodes are configured likewise. Make sure the converter nodes get unique <code>com.openexchange.server.backendRoute</code> values (see above).

== Second test: use Apache for loadbalancing ==

Pick one frontend node to configure its Apache for loadbalancing for the converter nodes.

There are sample configuration stanzas in our default configuration. They are just fine. Just make sure the <code>route</code> parameters match the ones from the converter nodes.

I want to emphasize to configure the <code>Allow</code> line correctly. The <code>/documentconverterws</code> endpoint must not be made available publicly!

A sample Apache configuration looks like

<nowiki><Proxy balancer://oxcluster_docs>
Order Deny,Allow
Deny from all
# configure the allowed IPs such that only the middleware nodes are able to access
# the /documentconverterws endpoint must not be made available publicly!
Allow from 10.0.1
BalancerMember http://dc1:8008 timeout=100 smax=0 ttl=60 retry=60 loadfactor=50 keepalive=On route=DC1
BalancerMember http://dc2:8008 timeout=100 smax=0 ttl=60 retry=60 loadfactor=50 keepalive=On route=DC2
# add further converter nodes, as many as you have
ProxySet stickysession=JSESSIONID|jsessionid scolonpathdelim=On
SetEnv proxy-initial-not-pooled
SetEnv proxy-sendchunked
</Proxy>

ProxyPass /documentconverterws balancer://oxcluster_docs/documentconverterws</nowiki>

With that configuration in place, test the connectivity from the middleware node to that endpoint, e.g. (assuming the frontend node is called <code>frontend1</code>):

<nowiki># curl http://frontend1/documentconverterws
<html>
<head><meta charset="UTF-8"><title>Open-Xchange DC</title></head>
<body><h1 align="center">OX Software GmbH DC</h1>
<p>WebService is running...</p>
<p>Error Code: 0</p>
<p>API: v5</p></body>
</html></nowiki>

The response looks exactly like before. The difference is just that we access the service now via Apache.

Execute the test multiple times. It must answer fast every time. If some answers are slow (maybe only once or twice) that is an indication that some route configuration was incorrect and Apache disabled some converter targets.

Make sure you are actually getting responses from the different converter nodes by whatever means suit you (e.g. tcpdump on the converter nodes, looking at Apache's <code>balancer-manager</code> to verify all nodes show up as <code>Ok</code>, the <code>Elected</code> number increases equally for all converter nodes, etc.)

Configure that endpoint in your test middleware node (<code>com.openexchange.documentconverter.client.remoteDocumentConverterUrl</code>). Restart the middleware service.

Do a full test cycle as described below and in particular make sure that in pop-out view all preview images are rendered correctly. That is the testcase for correct session stickyness.

If everything works correctly, make sure the Apache configuration is adjusted accordingly on all your Apache / Fronted nodes.

Now we have high availability for the converter nodes, but the Apache service is still a SPOF.

== Third test: adding local HAproxy instances ==

We add local HAproxy instances to eliminate the Apache SPOF.

Assuming HAproxy is already running locally for other services. If not, install it and do a basic configuration according to [HAproxy].

Add a stanza like

<nowiki>listen converter
mode http
option dontlognull
option redispatch
no option httpclose
timeout connect 10s
timeout client 2m
timeout server 2m
bind 127.0.0.1:8008
balance roundrobin
option httpchk GET /documentconverterws
server frontend1 frontend1:80 check port 80 inter 6000 rise 3 fall 1
server frontend2 frontend2:80 check port 80 inter 6000 rise 3 fall 1
# add more frontend nodes if you have them</nowiki>

Re-test that also this listener works correctly.

<nowiki># curl http://127.0.0.1:8008/documentconverterws
<html>
<head><meta charset="UTF-8"><title>Open-Xchange DC</title></head>
<body><h1 align="center">OX Software GmbH DC</h1>
<p>WebService is running...</p>
<p>Error Code: 0</p>
<p>API: v5</p></body>
</html></nowiki>

Same as before, execute the test multiple times, and verify in the Apache's <code>balancer-manager</code>s that everything works as expected.

Look at the HAproxy stats endpoint to verify all frontend nodes show as OK and get roughly the same number of sessions.

If that looks good, configure the HAproxy endpoint <code>http://127.0.0.1:8008/documentconverterws</code> in your test middleware node (<code>com.openexchange.documentconverter.client.remoteDocumentConverterUrl</code>). Restart the middleware service.

Do a full test cycle (as described below).

== Final setup ==

Assuming the test middleware node works as described in the previous section, do the same HAproxy configuration on all middleware nodes and configure the HAproxy endpoint <code>http://127.0.0.1:8008/documentconverterws</code> in all middleware nodes' middleware configuration.

The fully clustered setup should be working now.

= Testing =

It is highly recommended to make sure you always end up on the same middleware node for testing, by whatever means is required in your infrastructure to do so. For this document we assume you can access them directly (internal testing). Otherwise you'd need to expose special entry points to your setup.

If you can't ensure that, you need to configure all middleware nodes absolutely identical also during setup and testing, which is very cumbersome and error-prone.

If you reconfigured something on the middleware node(s), restart the service there.

<nowiki>service open-xchange restart</nowiki>

Wipe out caches on the converter node(s) and restart the service there:

<nowiki>service open-xchange-documentconverter-server stop
rm -rf /var/spool/open-xchange/documentconverter/readerengine.*/*
service open-xchange-documentconverter-server start</nowiki>

(Careful about your paths and when copy-pasting; I don't take responsibility of you remove something wrong.)

If your testuser is logged in from a previous test, log out.

Login your testuser.

Access Drive. Upload test documents if not done before. Preferably some small trivial docs, also some large multi-page docs with embedded diagrams etc.

Switch to "icons" or "tiles" view. You should get nice preview thumbnails for each document. (This is only a relevant test for the first time since the previews are stored in the database. TODO: find out how to wipe / re-test that.)

Click the "eye" icon to start the Document Viewer. You should be able to view documents with reasonable speed. Scrolling through the pages should be possible and fast.

Click the "pop out" icon to the top right to use pop-out view. Verify all pages are rendered correctly in the full-page view and in the thumbnail previews.

= Troubleshooting =

In most of the times in our experiments if something was not working, it was a misconfiguration on network level. (Wrong hostnames, IPs, ports, missing firewall adjustments, etc).

The documentconverter service

User:Dominik.epple/DocumentConverterInstall

2018-02-16T09:37:08Z

Dominik.epple:

<div class="title">DocumentConverter Quickinstall Guide / Cheatsheet</div>

__TOC__

= Introduction =

This document aims to be a condensed "HOWTO" like installation walkthrough.

The other relevant documentation is to be considered as a reference.

This document describes the fully clustered setup. Simpler setups can be deduced as special case.

This document has been created and verified on CentOS7 and OX App Suite 7.8.4.

= Design description =

We consider a clustered setup with multiple middleware nodes and multiple converter nodes.

We consider a high available connection from the middleware nodes to the converter nodes realized using HAproxy instances running locally on the middleware nodes which do simple round-robin to the Apache instances of the Fronted nodes, which will do ''correct'' session stickyness aware routing to the converter nodes.

We need the Apache instances to get correct ''session sticky'' routing behavior.

We need the HAproxy instances to connect to the Apache instances in a high available fashion. If there are other means in the infrastructure offering that functionality, this is also okay. We present the HAproxy based setup to give an example of a fully working setup.

= Software installation and configuration =

== Middleware nodes ==

Prerequisites: Drive is installed and working. We need Drive to upload test documents and verify their conversion. This also assumes (since we a discussing a clustered setup) a clustered filestore is available and working. If you are only going for a single node proof-of-concept setup, that is not relevant of course.

Packages to be installed:

open-xchange-documentconverter-api open-xchange-documentconverter-client

Configuration:

<nowiki>/opt/open-xchange/etc/permissions.properties:
# assume a global switch on for testing. further config cascade stuff etc is out of scope
# of this document.
com.openexchange.capability.document_preview=true</nowiki>

<nowiki>documentconverter-client.properties:
# this needs to be adjusted. will be discussed below.
com.openexchange.documentconverter.client.remoteDocumentConverterUrl=http://host[:port]/documentconverterws</nowiki>

== Frontend nodes ==

None

== Converter nodes ==

Packages to be installed:

open-xchange-documentconverter-server open-xchange-documentconverter-api readerengine

(Note: the official documentation also mentions "pdf2svg" which at least on CentOS7 does not exist, but rather a package named "readerengine-pdf2svg" is pulled as dependency. So for the moment let's assume we don't need to install that explicitly, but if you are following this guide on Debian you should double-check.)

Configuration:

/opt/open-xchange/etc/server.properties:
# Pick a unique route, which will be configured consistently in apache
com.openexchange.server.backendRoute=DC1
# Clustered setups need to listen not only on localhost
com.openexchange.connector.networkListenerHost=*

/opt/open-xchange/documentconverter/etc/documentconverter.properties
# TODO figure out the recommended Cache setup for a clustered scenario
com.openexchange.documentconverter.RemoteCacheUrls = ?
# Default is 3. Adjust for your sizing.
com.openexchange.documentconverter.jobProcessorCount=3

Services configuration:

systemctl start open-xchange-documentconverter-server
systemctl enable open-xchange-documentconverter-server

Note: you don't need ConfigDB / any DB access. Don't connect the converter nodes to any database. Don't make it join any Hazelcast cluster. No further configuration is required.

If you are annoyed by the "unable to connect to ConfigDB" (false) error messages in the logs, you can configure logback to suppress them (TODO: add information how to do that).

= Configure the middleware to converter connectivity =

So that was the trivial part. Now it gets interesting.

== First test: direct connectivity ==

Pick a middleware node. Configure direct connection to one converter service which is called <code>dc1</code>. By default the service listens on port <code>8008</code> and listens on the path <code>/documentconverterws</code>.

<nowiki># curl http://dc1:8008/documentconverterws/
<html>
<head><meta charset="UTF-8"><title>Open-Xchange DC</title></head>
<body><h1 align="center">OX Software GmbH DC</h1>
<p>WebService is running...</p>
<p>Error Code: 0</p>
<p>API: v5</p></body>
</html></nowiki>

That's how it should look like. HTTP status code 200 (not shown for clarity, but you can verify with curl -v, "WebService is running...", "Error Code: 0".

If that is successful, go ahead and configure that URL in the middleware node

<nowiki>documentconverter-client.properties:
# this needs to be adjusted. will be discussed below.
com.openexchange.documentconverter.client.remoteDocumentConverterUrl=http://dc1:8008/documentconverterws</nowiki>

Do a testing cycle as described below.

If everything works, you confirmed that the middleware node and the converter node are configured correctly.

Now ensure all your middleware nodes and all your converter nodes are configured likewise. Make sure the converter nodes get unique <code>com.openexchange.server.backendRoute</code> values (see above).

= Second test: use Apache for loadbalancing =

Pick one frontend node to configure its Apache for loadbalancing for the converter nodes.

There are sample configuration stanzas in our default configuration. They are just fine. Just make sure the <code>route</code> parameters match the ones from the converter nodes.

I want to emphasize to configure the <code>Allow</code> line correctly. The <code>/documentconverterws</code> endpoint must not be made available publicly!

A sample Apache configuration looks like

<nowiki><Proxy balancer://oxcluster_docs>
Order Deny,Allow
Deny from all
# configure the allowed IPs such that only the middleware nodes are able to access
# the /documentconverterws endpoint must not be made available publicly!
Allow from 10.0.1
BalancerMember http://dc1:8008 timeout=100 smax=0 ttl=60 retry=60 loadfactor=50 keepalive=On route=DC1
BalancerMember http://dc2:8008 timeout=100 smax=0 ttl=60 retry=60 loadfactor=50 keepalive=On route=DC2
# add further converter nodes, as many as you have
ProxySet stickysession=JSESSIONID|jsessionid scolonpathdelim=On
SetEnv proxy-initial-not-pooled
SetEnv proxy-sendchunked
</Proxy>

ProxyPass /documentconverterws balancer://oxcluster_docs/documentconverterws</nowiki>

With that configuration in place, test the connectivity from the middleware node to that endpoint, e.g. (assuming the frontend node is called <code>frontend1</code>):

<nowiki># curl http://frontend1/documentconverterws
<html>
<head><meta charset="UTF-8"><title>Open-Xchange DC</title></head>
<body><h1 align="center">OX Software GmbH DC</h1>
<p>WebService is running...</p>
<p>Error Code: 0</p>
<p>API: v5</p></body>
</html></nowiki>

The response looks exactly like before. The difference is just that we access the service now via Apache.

Execute the test multiple times. It must answer fast every time. If some answers are slow (maybe only once or twice) that is an indication that some route configuration was incorrect and Apache disabled some converter targets.

Make sure you are actually getting responses from the different converter nodes by whatever means suit you (e.g. tcpdump on the converter nodes, looking at Apache's <code>balancer-manager</code> to verify all nodes show up as <code>Ok</code>, the <code>Elected</code> number increases equally for all converter nodes, etc.)

Configure that endpoint in your test middleware node (<code>com.openexchange.documentconverter.client.remoteDocumentConverterUrl</code>). Restart the middleware service.

Do a full test cycle as described below and in particular make sure that in pop-out view all preview images are rendered correctly. That is the testcase for correct session stickyness.

If everything works correctly, make sure the Apache configuration is adjusted accordingly on all your Apache / Fronted nodes.

Now we have high availability for the converter nodes, but the Apache service is still a SPOF.

== Third test: adding local HAproxy instances ==

We add local HAproxy instances to eliminate the Apache SPOF.

Assuming HAproxy is already running locally for other services. If not, install it and do a basic configuration according to [HAproxy].

Add a stanza like

<nowiki>listen converter
mode http
option dontlognull
option redispatch
no option httpclose
timeout connect 10s
timeout client 2m
timeout server 2m
bind 127.0.0.1:8008
balance roundrobin
option httpchk GET /documentconverterws
server frontend1 frontend1:80 check port 80 inter 6000 rise 3 fall 1
server frontend2 frontend2:80 check port 80 inter 6000 rise 3 fall 1
# add more frontend nodes if you have them</nowiki>

Re-test that also this listener works correctly.

<nowiki># curl http://127.0.0.1:8008/documentconverterws
<html>
<head><meta charset="UTF-8"><title>Open-Xchange DC</title></head>
<body><h1 align="center">OX Software GmbH DC</h1>
<p>WebService is running...</p>
<p>Error Code: 0</p>
<p>API: v5</p></body>
</html></nowiki>

Same as before, execute the test multiple times, and verify in the Apache's <code>balancer-manager</code>s that everything works as expected.

Look at the HAproxy stats endpoint to verify all frontend nodes show as OK and get roughly the same number of sessions.

If that looks good, configure the HAproxy endpoint <code>http://127.0.0.1:8008/documentconverterws</code> in your test middleware node (<code>com.openexchange.documentconverter.client.remoteDocumentConverterUrl</code>). Restart the middleware service.

Do a full test cycle (as described below).

== Final setup ==

Assuming the test middleware node works as described in the previous section, do the same HAproxy configuration on all middleware nodes and configure the HAproxy endpoint <code>http://127.0.0.1:8008/documentconverterws</code> in all middleware nodes' middleware configuration.

The fully clustered setup should be working now.

= Testing =

It is highly recommended to make sure you always end up on the same middleware node for testing, by whatever means is required in your infrastructure to do so. For this document we assume you can access them directly (internal testing). Otherwise you'd need to expose special entry points to your setup.

If you can't ensure that, you need to configure all middleware nodes absolutely identical also during setup and testing, which is very cumbersome and error-prone.

If you reconfigured something on the middleware node(s), restart the service there.

<nowiki>service open-xchange restart</nowiki>

Wipe out caches on the converter node(s) and restart the service there:

<nowiki>service open-xchange-documentconverter-server stop
rm -rf /var/spool/open-xchange/documentconverter/readerengine.*/*
service open-xchange-documentconverter-server start</nowiki>

(Careful about your paths and when copy-pasting; I don't take responsibility of you remove something wrong.)

If your testuser is logged in from a previous test, log out.

Login your testuser.

Access Drive. Upload test documents if not done before. Preferably some small trivial docs, also some large multi-page docs with embedded diagrams etc.

Switch to "icons" or "tiles" view. You should get nice preview thumbnails for each document. (This is only a relevant test for the first time since the previews are stored in the database. TODO: find out how to wipe / re-test that.)

Click the "eye" icon to start the Document Viewer. You should be able to view documents with reasonable speed. Scrolling through the pages should be possible and fast.

Click the "pop out" icon to the top right to use pop-out view. Verify all pages are rendered correctly in the full-page view and in the thumbnail previews.

= Troubleshooting =

In most of the times in our experiments if something was not working, it was a misconfiguration on network level. (Wrong hostnames, IPs, ports, missing firewall adjustments, etc).

The documentconverter service

User:Dominik.epple/DocumentConverterInstall

2018-02-16T09:28:41Z

Dominik.epple:

<div class="title">DocumentConverter Quickinstall Guide / Cheatsheet</div>

__TOC__

= Introduction =

This document aims to be a condensed "HOWTO" like installation walkthrough.

The other relevant documentation is to be considered as a reference.

This document describes the fully clustered setup. Simpler setups can be deduced as special case.

This document has been created and verified on CentOS7 and OX App Suite 7.8.4.

= Design description =

We consider a clustered setup with multiple middleware nodes and multiple converter nodes.

We consider a high available connection from the middleware nodes to the converter nodes realized using HAproxy instances running locally on the middleware nodes which do simple round-robin to the Apache instances of the Fronted nodes, which will do ''correct'' session stickyness aware routing to the converter nodes.

We need the Apache instances to get correct ''session sticky'' routing behavior.

We need the HAproxy instances to connect to the Apache instances in a high available fashion. If there are other means in the infrastructure offering that functionality, this is also okay. We present the HAproxy based setup to give an example of a fully working setup.

= Software installation and configuration =

== Middleware nodes ==

Prerequisites: Drive is installed and working. We need Drive to upload test documents and verify their conversion. This also assumes (since we a discussing a clustered setup) a clustered filestore is available and working. If you are only going for a single node proof-of-concept setup, that is not relevant of course.

Packages to be installed:

open-xchange-documentconverter-api open-xchange-documentconverter-client

Configuration:

<nowiki>/opt/open-xchange/etc/permissions.properties:
# assume a global switch on for testing. further config cascade stuff etc is out of scope
# of this document.
com.openexchange.capability.document_preview=true</nowiki>

<nowiki>documentconverter-client.properties:
# this needs to be adjusted. will be discussed below.
com.openexchange.documentconverter.client.remoteDocumentConverterUrl=http://host[:port]/documentconverterws</nowiki>

== Frontend nodes ==

None

== Converter nodes ==

Packages to be installed:

open-xchange-documentconverter-server open-xchange-documentconverter-api readerengine

(Note: the official documentation also mentions "pdf2svg" which at least on CentOS7 does not exist, but rather a package named "readerengine-pdf2svg" is pulled as dependency. So for the moment let's assume we don't need to install that explicitly, but if you are following this guide on Debian you should double-check.)

Configuration:

/opt/open-xchange/etc/server.properties:
# Pick a unique route, which will be configured consistently in apache
com.openexchange.server.backendRoute=DC1
# Clustered setups need to listen not only on localhost
com.openexchange.connector.networkListenerHost=*

/opt/open-xchange/documentconverter/etc/documentconverter.properties
# TODO figure out the recommended Cache setup for a clustered scenario
com.openexchange.documentconverter.RemoteCacheUrls = ?
# Default is 3. Adjust for your sizing.
com.openexchange.documentconverter.jobProcessorCount=3

Services configuration:

systemctl start open-xchange-documentconverter-server
systemctl enable open-xchange-documentconverter-server

= Configure the middleware to converter connectivity =

So that was the trivial part. Now it gets interesting.

== First test: direct connectivity ==

Pick a middleware node. Configure direct connection to one converter service which is called <code>dc1</code>. By default the service listens on port <code>8008</code> and listens on the path <code>/documentconverterws</code>.

<nowiki># curl http://dc1:8008/documentconverterws/
<html>
<head><meta charset="UTF-8"><title>Open-Xchange DC</title></head>
<body><h1 align="center">OX Software GmbH DC</h1>
<p>WebService is running...</p>
<p>Error Code: 0</p>
<p>API: v5</p></body>
</html></nowiki>

That's how it should look like. HTTP status code 200 (not shown for clarity, but you can verify with curl -v, "WebService is running...", "Error Code: 0".

If that is successful, go ahead and configure that URL in the middleware node

<nowiki>documentconverter-client.properties:
# this needs to be adjusted. will be discussed below.
com.openexchange.documentconverter.client.remoteDocumentConverterUrl=http://dc1:8008/documentconverterws</nowiki>

Do a testing cycle as described below.

If everything works, you confirmed that the middleware node and the converter node are configured correctly.

Now ensure all your middleware nodes and all your converter nodes are configured likewise. Make sure the converter nodes get unique <code>com.openexchange.server.backendRoute</code> values (see above).

= Second test: use Apache for loadbalancing =

Pick one frontend node to configure its Apache for loadbalancing for the converter nodes.

There are sample configuration stanzas in our default configuration. They are just fine. Just make sure the <code>route</code> parameters match the ones from the converter nodes.

I want to emphasize to configure the <code>Allow</code> line correctly. The <code>/documentconverterws</code> endpoint must not be made available publicly!

A sample Apache configuration looks like

<nowiki><Proxy balancer://oxcluster_docs>
Order Deny,Allow
Deny from all
# configure the allowed IPs such that only the middleware nodes are able to access
# the /documentconverterws endpoint must not be made available publicly!
Allow from 10.0.1
BalancerMember http://dc1:8008 timeout=100 smax=0 ttl=60 retry=60 loadfactor=50 keepalive=On route=DC1
BalancerMember http://dc2:8008 timeout=100 smax=0 ttl=60 retry=60 loadfactor=50 keepalive=On route=DC2
# add further converter nodes, as many as you have
ProxySet stickysession=JSESSIONID|jsessionid scolonpathdelim=On
SetEnv proxy-initial-not-pooled
SetEnv proxy-sendchunked
</Proxy>

ProxyPass /documentconverterws balancer://oxcluster_docs/documentconverterws</nowiki>

With that configuration in place, test the connectivity from the middleware node to that endpoint, e.g. (assuming the frontend node is called <code>frontend1</code>):

<nowiki># curl http://frontend1/documentconverterws
<html>
<head><meta charset="UTF-8"><title>Open-Xchange DC</title></head>
<body><h1 align="center">OX Software GmbH DC</h1>
<p>WebService is running...</p>
<p>Error Code: 0</p>
<p>API: v5</p></body>
</html></nowiki>

The response looks exactly like before. The difference is just that we access the service now via Apache.

Execute the test multiple times. It must answer fast every time. If some answers are slow (maybe only once or twice) that is an indication that some route configuration was incorrect and Apache disabled some converter targets.

Make sure you are actually getting responses from the different converter nodes by whatever means suit you (e.g. tcpdump on the converter nodes, looking at Apache's <code>balancer-manager</code> to verify all nodes show up as <code>Ok</code>, the <code>Elected</code> number increases equally for all converter nodes, etc.)

Configure that endpoint in your test middleware node. Restart the middleware service.

Do a full test cycle as described below and in particular make sure that in pop-out view all preview images are rendered correctly. That is the testcase for correct session stickyness.

If everything works correctly, make sure the Apache configuration is adjusted accordingly on all your Apache / Fronted nodes.

Now we have high availability for the converter nodes, but the Apache service is still a SPOF.

== Third test: adding local HAproxy instances ==

We add local HAproxy instances to eliminate the Apache SPOF.

Assuming HAproxy is already running locally for other services. If not, install it and do a basic configuration according to [HAproxy].

Add a stanza like

<nowiki>listen converter
mode http
option dontlognull
option redispatch
no option httpclose
timeout connect 10s
timeout client 2m
timeout server 2m
bind 127.0.0.1:8008
balance roundrobin
option httpchk GET /documentconverterws
server frontend1 frontend1:80 check port 80 inter 6000 rise 3 fall 1
server frontend2 frontend2:80 check port 80 inter 6000 rise 3 fall 1
# add more frontend nodes if you have them</nowiki>

Re-test that also this listener works correctly.

<nowiki># curl http://127.0.0.1:8008/documentconverterws
<html>
<head><meta charset="UTF-8"><title>Open-Xchange DC</title></head>
<body><h1 align="center">OX Software GmbH DC</h1>
<p>WebService is running...</p>
<p>Error Code: 0</p>
<p>API: v5</p></body>
</html>

Same as before, execute the test multiple times, and verify in the Apache's <code>balancer-manager</code>s that everything works as expected.

Look at the HAproxy stats endpoint to verify all frontend nodes show as OK and get roughly the same number of sessions.

= Testing =

If you reconfigured something on the middleware node(s), restart the service there.

<nowiki>service open-xchange restart</nowiki>

Wipe out caches on the converter node(s) and restart the service there:

<nowiki>service open-xchange-documentconverter-server stop
rm -rf /var/spool/open-xchange/documentconverter/readerengine.*/*
service open-xchange-documentconverter-server start</nowiki>

(Careful about your paths and when copy-pasting; I don't take responsibility of you remove something wrong.)

If your testuser is logged in from a previous test, log out.

Login your testuser.

Access Drive. Upload test documents if not done before. Preferably some small trivial docs, also some large multi-page docs with embedded diagrams etc.

Switch to "icons" or "tiles" view. You should get nice preview thumbnails for each document. (This is only a relevant test for the first time since the previews are stored in the database. TODO: find out how to wipe / re-test that.)

Click the "eye" icon to start the Document Viewer. You should be able to view documents with reasonable speed. Scrolling through the pages should be possible and fast.

Click the "pop out" icon to the top right to use pop-out view. Verify all pages are rendered correctly in the full-page view and in the thumbnail previews.

User:Dominik.epple/DocumentConverterInstall

2018-02-16T09:19:39Z

Dominik.epple:

User:Dominik.epple/DocumentConverterInstall

2018-02-16T09:05:08Z

Dominik.epple:

User:Dominik.epple/DocumentConverterInstall

2018-02-16T08:50:48Z

Dominik.epple: Created page with "<div class="title">DocumentConverter Quickinstall Guide / Cheatsheet</div> __TOC__ = Introduction = This document aims to be a condensed "HOWTO" like installation walkthrou..."

Template:OXLoadBalancingClustering Database

2018-01-26T16:47:17Z

Dominik.epple: /* Configuration */

== Overview ==

You can choose between Galera or Master/Slave replication. We like to recommend to use Galera for higher redudancy, easier operations, und synchronous semantics (so you can run OX without our "replication monitor").

== Galera database setup ==

=== Preparations ===

Our configuration process includes wiping and reinitializing the datadir. This is usually not a problem in a fresh installation. If you want to upgrade an existing database to Galera cluster, please be prepared to wipe the datadir, i.e. take a <code>mysqldump</code> for later restoration into the properly configured master.

Depeding on the flavor of the current database, this can be something like

# mariadb or oracle mysql without GTIDs
mysqldump --databases configdb oxdb_{5..14} > backup.sql

# mysql 5.6 with GTIDs... we dont want GTIDs here
mysqldump --databases --set-gtid-purged=OFF configdb oxdb_{5..14} > backup.sql

Be sure to verify the list of databases.

=== Installation ===

Please follow the upstream docs for your preferred flavor to get the software installed on your system.

* Percona XtraDB Cluster (5.6, 5.7): https://www.percona.com/doc/percona-xtradb-cluster/LATEST/install/index.html
* MariaDB Galera Cluster (10.0, 10.1): https://mariadb.com/kb/en/library/getting-started-with-mariadb-galera-cluster/ (Note: with 10.0, socat is required, but not a package dependency, so you need to explicitly install also socat)

Make sure to doublecheck the service is not running (or stop it) after installation as we need to perform some reconfigurations.

service mysql stop

=== Configuration ===

Galera-specific MySQL configuration advise is included in our main [[My.cnf|MySQL configuration article]]. Please consult that page for configuration information.

That page suggests a setup were we add three custom config files to <code>/etc/mysql/ox.conf.d/</pre>: <code>ox.cnf</code> for general tuning/sizing, <code>wsrep.cnf</code> for clusterwide galera configuration, and <code>host.cnf</code> for host-specific settings.

Adjust the general settings and tunings in <code>ox.cnf</code> according to your sizing etc.

Adjust <code>wsrep.cnf</code> to reflect local paths, cluster member addresses, etc.

Adjust <code>host.cnf</code> to give node-local IPs, etc.

Version-specific hints:

# percona 5.6: unknown variable 'pxc_strict_mode=ENFORCING' ... unset that one
# mariadb 10.1: add wsrep_on=ON
# mariadb 10.0 and 10.1: set wsrep_node_incoming_address=192.168.1.22:3306 in host.cnf, otherwise the status wsrep_incoming_addresses might not be shown correctly(?!)

Some settings we recommend to change require that the database gets re-initialized. We assume you don't have data there (since we are covering a fresh install) or you have taken a backup for later restore as explained above in the Preparations section.

cd /var/lib/
mv mysql mysql.old.datadir
mkdir mysql
chown mysql.mysql mysql

# mariadb 10.0 and 10.1
mysql_install_db
# percona 5.6
mysqld --user=mysql
# percona 5.7
mysqld --initialize-insecure --user=mysql

(Don't be worried about the <code>insecure</code>, it just means we set the db root pw in the next steps.)

=== Cluster startup ===

Typically on startup a Galera node tries to join a cluster, and if it fails, it will exit. Thus, when no cluster nodes are running, the first cluster node to be started needs to be told to not try to join a cluster, but rather bootstrap a new cluster. The exact arguments vary from version to version and from flavor to flavor.

==== First node ====

So we initialize the cluster bootstrap on the first node:

# percona 5.6, 5.7
service mysql bootstrap-pxc
# mariadb 10.0
service mysql bootstrap
# mariadb 10.1: service mysql bootstrap seems to be broken, does not pass the necessary options to mysqld
galera_new_cluster

Run <code>mysql_secure_installation</code> for a "secure by default" installation:

mysql_secure_installation

The further steps in this guide omit <code>-u -p</code> arguments to the MySQL client. Rather than passing them on the command line [https://dev.mysql.com/doc/refman/5.7/en/password-security-user.html] it is recommended to place the credentials in a file like <code>/root/.my.cnf</code> like

[client]
user=root
password=wip9Phae3Beijeed

We need a Galera replication user:

CREATE USER 'sstuser'@'localhost' IDENTIFIED BY 'OpIdjijwef0';
-- percona 5.6, mariadb 10.0
GRANT RELOAD, LOCK TABLES, REPLICATION CLIENT ON *.* TO 'sstuser'@'localhost';
-- percona 5.7, mariadb 10.1
GRANT PROCESS, RELOAD, LOCK TABLES, REPLICATION CLIENT ON *.* TO 'sstuser'@'localhost';
FLUSH PRIVILEGES;

(Debian specific note: MariaDB provided startup scripts use the distro's mechanism of verifying startup/shutdown using a system user, so we create that as well:

# mariadb 10.0, 10.1
GRANT ALL PRIVILEGES ON *.* TO "debian-sys-maint"@"localhost" IDENTIFIED BY "adBexthTsI5TaEps";

If you do this, yo need to synchronize the <code>/etc/mysql/debian.cnf</code> file from the first node to the other nodes as well.)

==== Other nodes ====

On the other nodes, we only need to restart the service now, to trigger a full state transfer from the first node to the other nodes.

We recommend to do this serially to let one state transfer complete before the second state transfer.

==== First node (continued) ====

Only applicable if you used <code>galera_new_cluster</code> before rather than the service script: In order to get the systemctl status consistent, restart the service on the first node:

# mariadb 10.1: restart the service so that the systemctl status is consistent
mysqladmin shutdown
service mysql bootstrap

=== Verify the replication ===

The key tool to verify replication status is

mysql> show status like "%wsrep%";

This will give a lot of output. You want to verify in particular

+------------------------------+--------------------------------------+
| Variable_name | Value |
+------------------------------+--------------------------------------+
| wsrep_cluster_size | 3 |
| wsrep_cluster_status | Primary |
| wsrep_local_state | 4 |
| wsrep_local_state_comment | Synced |
| wsrep_ready | ON |
+------------------------------+--------------------------------------+

You can also explicitly verify replication by creating / inserting DBs, tables, rows on one node and select on other nodes.

==== Troubleshooting ====

The logs are helpful. Always.

Common mistakes are listed below.

If the Galera module does not get loaded at all:
* Configuration settings in ''my.cnf'' which are incompatible to Galera
* Wrong path of the shared object providing the Galera plugin in wsrep.cnf (wsrep_provider)

If the first node starts, but the second / third nodes can not be added to the cluster:
* User for the replication not created correctly on the first Galera node
* SST fails due to missing / wrong version prerequisite packages (not everything is hardcoded in package dependencies -- make sure you got percona-xtrabackup installed in the correct version, and also socat). If SST fails, do not only look into mysqls primary error logs, but also into logfiles from the SST tool in /var/lib/mysql on the donor node.

=== Notes about configuring OX for use with Galera ===

==== Write requests ====

Open-Xchange supports Galera as database backend only in the configuration where all writes are directed to one Galera node. For availability, it makes sense to not configure one Galera node's IP address directly, but rather employ some HA solution which offers active-passive functionality. Options therefore are discussed below.

==== Read requests ====

Read requests can be directed to any node in the Galera cluster. Our standard approach is to recommend to use a loadbalancer to implement round-robin over all nodes in a Galera cluster for the read requests. But you can also chose to use a dedicated read node (the same node, or a different node, than the write node). Each of the approaches has its own advantages.

* Load balancer based setup: Read requests get distributed round-robin between the Galera nodes. Theoretically by distributing the load of the read requests, you benefit from lower latencies and more throughput. But this has never been benchmarked yet. For a discussion of available loadbalances, see next section. OX-wise, in this configuration, you have two alternatives:
** The Galera option wsrep_causal_reads=1 option enables you to configure OX with its replication monitor disabled (com.openexchange.database.replicationMonitor=false in configdb.properties). This is the setup which seems to perform best according to our experience as turning off the replication monitor reduces the commits on the DB and thus the write operations per second on the underlying storage significantly, which outweights the drawback from having higher commit latency due to fully synchronous mode.
** Alternatively, you can run Galera with wsrep_causal_reads=0 when switching on OX builtin replication monitor. This is also a valid setup.
* Use a designated floating IP for the read requests: This eliminates the need of a load balancer. With this option you will not gain any performance, but the quantitative benefit is unclear anyhow.
* Use the floating IP for the writes also for the reads: In this scenario, you direct all database queries only to one Galera node, and the other two nodes are only getting queries in case of a failure of that node. In this case, you can even use wsrep_causal_reads=0 while still having OX builtin replication monitor switched off. However we do not expect this option to be superior to the round-robin loadbalancer approach.

=== Loadbalancer options ===

While the JDBC driver has some round-robin load balancing capabilities built-in, we don't recommend it for production use since it lacks possibilities to check the Galera nodes health states.

Loadbalancers used for OX -> Galera loadbalancing should be able to implement active-passive instances for the write requests, and active-active (round-robin) instances for the read requests. (If they cannot implement active-passive, you can still take a floating IP therefore.) Furthermore it is required to configure node health checks not only on the TCP level (by a simple connect), but to query the Galera health status periodically, evaluating Galera WSREP status variables. Otherwise split-brain scenarios or other bad states cannot be detected. For an example of such an health check, see our [[Clustercheck]] page.

Some customers use loadbalancing appliances. It is important to check that if the (virtual) infrastructure offers "loadbalancer" instances that they satisfy the given requirements. Often this is not the case. In particular, a simple "DNS round robin" approach is not viable.

==== LVS/ipvsadm/keepalived ====

If you want to create your own loadbalancers based on Linux, we usually recommend LVS (Linux Virtual Servers) controlled by Keepalived. LVS is a set of kernel modules implementing a L4 loadbalancer which performs quite well. Keepalived is a userspace daemon to control LVS rules, using health checks to reconfigure LVS rules if required. Keepalived / LVS requires one (or, for availability, two) dedicated linux nodes to run on. This can be a disadvantage for some installations, but usually, it pays off. We provide some configuration information on Keepalived [[Keepalived|here]].

==== MariaDB Maxscale ====

Since Maxscale has become GA in 2015, it seems to have undergone significant stability, performance and functional improvements. We are currently experimenting with Maxscale and share our installation / configuration knowledge [[Maxscale|here]]. It looks quite promising and might become ''the standard replacement'' for HAproxy, while we still presume Keepalived offers superior robustness and performance, coming with the cost of the requirement for one (or more) dedicated loadbalancer nodes.

==== HAproxy ====

In case where the Keepalived based approach is not feasible due to its requirements on the infrastructure, it is also possible to use a HAproxy based solution where HAproxy processes run on each of the OX nodes, configured for one round-robin and one active/passive instance. OX is then connecting to the local HAproxy instances. It is vital to configure HAproxy timeouts different from the defaults, otherwise HAproxy will kill active DB connections, causing errors. Be aware that in large installations the number of (distributed) HAproxy instances can get quite large. Some configuration hints for HAproxy are available [[HAproxy|here]].

== Master/Slave database setup ==

While we also support also "legacy" (pre-GTID) Master/Slave replication, we recommend to use GTID based replication, for easier setup and failure recovery. Support for GTID based replication has been added with OX 7.8.0.

GTID has been available since MySQL 5.6, so no 5.5 installation instructions below, sorry. We try to be generic in this documentation (thus, applicable to Oracle Community Edition and MariaDB) and point out differences where needed. Note: Instructions below include information about Oracle Community MySQL 5.7 which is not yet formally supported.

=== Preparations ===

Our configuration process includes wiping and reinitializing the datadir. This is usually not a problem in a fresh installation. If you want to upgrade an existing database to GTID master-slave, please be prepared to wipe the datadir, i.e. take a <code>mysqldump</code> for later restoration into the properly configured master.

Depeding on the flavor of the current database, this can be something like

# mariadb or oracle mysql without GTIDs
mysqldump --databases configdb oxdb_{5..14} > backup.sql

# mysql 5.6 with GTIDs... we dont want GTIDs here
mysqldump --databases --set-gtid-purged=OFF configdb oxdb_{5..14} > backup.sql

Be sure to verify the list of databases.

=== Installation ===

Software installation is identical for master and slave.

Please follow the instructions for installing from The vendors.

* Oracle Community Edition: https://dev.mysql.com/doc/mysql-apt-repo-quick-guide/en/
* MariaDB (10.0, 10.1): https://downloads.mariadb.org/mariadb/repositories/

Stop the service (if it is running):

service mysql stop

=== Configuration ===

Configuration as per configuration files is also identical for master and slave.

Consult [[My.cnf]] for general recommendations how to configure databases for usage with OX.

For GTID based replication, make sure you add some configurables to a new <code>/etc/mysql/ox.conf.d/gtid.cnf</code> file (assuming you are following our proposed schema of adding a <code>!includedir /etc/mysql/ox.conf.d/</code>" directive to <code>/etc/mysql/my.cnf</code>):

# GTID
log-bin=mysql-bin
server-id=...
log_slave_updates = ON

Oracle Community Edition: we need to add also

enforce_gtid_consistency = ON
gtid_mode = ON

(GTID mode is on by default on MariaDB.)

Use unique a <code>server-id</code> for each server; like <code>1</code> for the master, <code>2</code> for slave. For more complicated setups (like multiple slaves), adjust accordingly.

Since applying our configuration / sizing requires reinitialization of the MySQL datadir, we wipe/recreate it. Caution: this assumes we are running an empty database. If there is data in the database you want to keep, use mysqldump. See Preparation section above.

So, to initialize the datadir:

cd /var/lib/
mv mysql mysql.old.datadir
mkdir mysql
chown mysql.mysql mysql

(When coming from an existing installation, be sure to wipe also old binlogs. They can confuse the server on startup. Their location varies by configuration.)

The step to initialize the datadir is different for the different DBs:

# MariaDB 10.0, 10.1
mysql_install_db

# Oracle 5.6
mysql_install_db -u mysql

# Oracle 5.7
mysqld --initialize-insecure --user=mysql

(Don't be worried about the <code>insecure</code>, it just means we set the db root pw in the next steps.)

Then:

service mysql restart
mysql_secure_installation

We want to emphasize the last step to run "secure".

Steps up to here apply to both the designated master and slave. The next steps will apply to the master.

=== Replication Setup ===

==== Master Setup ====

Create a replication user on the master (but, as always, pick your own password, and use the same password in the slave setup below):

mysql -e "CREATE USER 'repl'@'gtid-slave.localdomain' IDENTIFIED BY 'IvIjyoffod2'; GRANT REPLICATION SLAVE ON *.* TO 'repl'@'gtid-slave.localdomain';"

Now would also be the time to restore a previously created mysqldump, or add other users you need for adminstration, monitoring etc (like <code>debian-sys-maint@localhost</code>, for example). Adding the OX users is explained below ("Creating Open-Xchange user").

# If you took a dump for restore before
mysql < backup.sql

To prepare for the initial sync of the slave, set the master read-only:

mysql -e "SET @@global.read_only = ON;"

Create a dump to initialize the slave:

# MariaDB
mysqldump --all-databases --triggers --routines --events --master-data --gtid > master.sql

# Oracle
mysqldump --all-databases --triggers --routines --events --set-gtid-purged=ON > master.sql

Transfer to the slave:

scp master.sql gtid-slave:

==== Slave Setup ====

Configure the replication master settings. Note we don't need complicated binlog position settings etc with GTID.

Yet again DB-specific (use the repl user password from above):

# MariaDB
mysql -e 'CHANGE MASTER TO MASTER_HOST="gtid-master.localdomain", MASTER_USER="repl", MASTER_PASSWORD="IvIjyoffod2";'

# Oracle
mysql -e "CHANGE MASTER TO MASTER_HOST='gtid-master.localdomain', MASTER_USER='repl', MASTER_PASSWORD='IvIjyoffod2', MASTER_AUTO_POSITION=1;"
# https://www.percona.com/blog/2013/02/08/how-to-createrestore-a-slave-using-gtid-replication-in-mysql-5-6/
mysql -e "RESET MASTER;"

Read the master dump:

mysql < master.sql

Start replication on the slave:

mysql -e 'START SLAVE;'
mysql -e 'SHOW SLAVE STATUS\G'

==== Master Setup (continued) ====

Finally, unset read-only on the master:

# on the master
mysql -e "SET @@global.read_only = OFF;"

=== Configure OX to use with Master/Slave replication ===

Not much special wisdom here. OX was designed to be used with master/slave databases. For the ConfigDB, <code>configdb.properties</code> allows configuration of a <code>readUrl</code> and <code>writeUrl</code> (both of which are set to the correct values if you use <code>oxinstaller</code> with the correct arguments <code>--configdb-readhost</code>, <code>--configdb-writehost</code>).

(Obviously, the master is for writing and the slave is for reading.)

For the individiual UserDBs, use <code>registerdatabase -m true</code> for the masters and <code>registerdatabase -m false -M ...</code> for the respective slaves.

Be sure to have enabled the replication monitor in <code>configdb.properties</code>: <code>com.openexchange.database.replicationMonitor=true</code> (which it is by default); while GTID can show synchronous semantics, it is specified to silently fall back to asynchronous in certain circumstances, so synchronity is not guaranteed.

We recommend, though, to not register the databases directly by their native hostname or IP, but rather use some kind of HA system in order to be able to easily move a floating/failover IP from the master to the slave in case of master failure. Configuring and running such systems (like, corosync/pacemaker, keepalived, or whatever) is out of scope of this documentation, however.

== Creating Open-Xchange user ==

Now setup access for the Open-Xchange Server database user 'openexchange' to configdb and the oxdb for both groupware server addresses. These databases do not exist yet, but will be created during the Open-Xchange Server installation.

Note: The IPs in this example belong to the two different Open-Xchange Servers, please adjust them accordingly. And use a real password.

mysql> GRANT ALL PRIVILEGES ON *.* TO 'openexchange'@'10.20.30.213' IDENTIFIED BY 'IntyoyntOat1';
mysql> GRANT ALL PRIVILEGES ON *.* TO 'openexchange'@'10.20.30.215' IDENTIFIED BY 'IntyoyntOat1';

User:Dominik.epple

2018-01-05T08:21:40Z

Dominik.epple: /* Prepare database */

== Install Guide ==

=== About this document ===

The aim of this document is to improve on the existing quickinstall guides to be more structured, provide a more extensive view on "single node and beyond" topics, follow closer to existing "best practices" (also, but not only security-wise), and point out what needs to be changed in clustered installations.

Most of the commands given in this document thus assume a high level design of "single-node, all-in-one".

This document was created on Debian Stretch (which, as of time of writing, is not even supported yet), but it should work as-is also for jessie. Porting to RHEL/SLES/... is TODO.

=== Preparations ===

==== System update ====

You want to start on latest patchlevel of your OS:

apt-get update
apt-get dist-upgrade
apt-get install less vim pwgen apt-transport-https

reboot

==== Pregenerate passwords ====

This guide shall feature copy-paste ready commands which create installations with no default passwords.

We will pre-generate some passwords which will live as dotfiles in /root.

pwgen -c -n 16 1 > /root/.oxpw
pwgen -c -n 16 1 > /root/.dbpw
pwgen -c -n 16 1 > /root/.dbrootpw

==== Prepare database ====

In real-world installations this will probably be multiple galera clusters of a supported flavor and version. For educational purposes a standalone DB on our single-node machine is sufficient.

Even for single-node, don't forget to apply database tuning. See [[My.cnf|our oxpedia article]] for default tunings. Note that typically you need to re-initialize the MySQL datadir after changing InnoDB sizing values, and subsequently start the service:

mysql_install_db
service mysql restart

We aim to create secure-by-default documentation, so here we go: Run mysql_secure_installation, and chose every security relevant option, but let the root password empty in this step, as we set it in the next step:

# leave the root password empty in mysql_secure_installation as we set it in the subsequent step
mysql_secure_installation
# now, configure the password from /root/.dbrootpw
mysql -e "UPDATE mysql.user SET Password=PASSWORD('$(cat /root/.dbrootpw)') WHERE User='root'; FLUSH PRIVILEGES;"
cat >/root/.my.cnf <<EOF
[client]
user=root
password=$(cat /root/.dbrootpw)
EOF

MySQL 5.7: the aforementioned must be adjusted using

ALTER USER USER() IDENTIFIED BY 'tiez7EiNgaish0ee';

These credentials also needs to be put in /etc/mysql/debian.cnf.

<div style="padding: 20px; border-style: solid; border-width: thin;">
Cluster note

On multiple db clusters, do it per node analogously. Just be aware the copy-paste command above expects the /root/.dbrootpw file.
</div>

==== Prepare OX user ====

While the packages will create the user automatically if it does not exist, we want to prepare the filestore now, and we need the user therefore.

useradd -r open-xchange

<div style="padding: 20px; border-style: solid; border-width: thin;">
Cluster Note

You should hard-wire the userid and groupid to the same fixed value. Otherwise, if you want to use a NFS filestore, you'll run into permissions problems, unless you use nfs4/kerberos/idmapd.

groupadd -r -g 999 open-xchange
useradd -r -g 999 -u 999 open-xchange
</div>

==== Prepare filestore ====

There are several options here.

===== Single-Node: local directory =====

For a single-node installation, you can just prepare a local directory:

mkdir /var/opt/filestore
chown open-xchange:open-xchange /var/opt/filestore

===== NFS =====

If using NFS:

Setup on the NFS server:

apt-get install nfs-kernel-server
service nfs-kernel-server restart

Configure /etc/exports. This is for traditional ip based access control; krb5 or other security configuration is out of scope of this document.

mkdir /var/opt/filestore
chown open-xchange:open-xchange /var/opt/filestore
echo "/var/opt/filestore 192.168.1.0/24(rw,sync,fsid=0,no_subtree_check)" >> /etc/exports
exportfs -a

Clients can then mount using

mkdir /var/opt/filestore
mount -t nfs -o vers=4 nfs-server:/filestore /var/opt/filestore

Or using fstab entries like

nfs-server:/filestore /var/opt/filestore nfs4 defaults 0 0

===== Object Store =====

You can use an object store. For lab environments Ceph is a convenient option. For demo / educational purpuses a "single node Ceph cluster" even co-located on your "single-node machine" is reasonble, but its setup is out of scope of this document. If you want to use this, be prepared to provide information about endpoint, bucket name, access key, secret key.

===== No filestore =====

If you dont want to provide a filestore, you can configure OX later to run without filestore. (Q: do we still need a dummy registerfilestore on a local directory in that event?)

==== Prepare mail system ====

Formally out of scope of this document.

If you need to create a testing dovecot/postfix setup, you can use our [https://gitlab.open-xchange.com/qa/performance/tree/develop/com.openexchange.test.performance.gatling/src/site-preprocess/dovecot performance testing sample config].

=== Install OX software ===

You need an ldb user and password for updates and proprietary repos. If you dont have such a user, you can still install the free components. You'll get a lot of authentication failed warnings however from apt tools unless you deconfigure the closed repos.

wget http://software.open-xchange.com/oxbuildkey.pub -O - | apt-key add -
wget -O/etc/apt/sources.list.d/ox.list http://software.open-xchange.com/products/DebianJessie.list
ldbuser=...
ldbpassword=...
sed -i -e "s/LDBUSER:LDBPASSWORD/$ldbuser:$ldbpassword/" /etc/apt/sources.list.d/ox.list
apt-get update
apt-get install open-xchange open-xchange-authentication-database open-xchange-grizzly open-xchange-admin open-xchange-appsuite-backend open-xchange-appsuite-manifest open-xchange-appsuite

<div style="padding: 20px; border-style: solid; border-width: thin;">
Cluster note

*if you want to have separate frontend (apache) and middleware (open-xchange) systems, make sure to install packages which require apache as dependency on the frontend nodes, and packages which require java as a dependency on the middleware nodes. Currently this results in the split
** Frontend nodes: open-xchange-appsuite
** Middleware nodes: everything else
* If you want to use an object store, install the corresponding open-xchange-filestore-xyz package, like open-xchange-filestore-s3
* For hazelcast session storage, install also open-xchange-sessionstorage-hazelcast
</div>

=== Install database schemas ===

If the DB runs on localhost and you have root access, you can use

/opt/open-xchange/sbin/initconfigdb --configdb-pass="$(cat /root/.dbpw)" -a

<div style="padding: 20px; border-style: solid; border-width: thin;">

Cluster note

Create the DB users on all write instances manually.

mysql -e "grant all privileges on *.* to 'openexchange'@'%' identified by '$(cat /root/.dbpw)';"

Run initconfigdb with some more options:

/opt/open-xchange/sbin/initconfigdb --configdb-user=openexchange --configdb-pass="$(cat /root/.dbpw)" --configdb-host=configdb-writehost

(initconfigdb needs to be run only once on one cluster node)
</div>

=== Initial configuration ===

/opt/open-xchange/sbin/oxinstaller --add-license=YOUR-OX-LICENSE-CODE --servername=oxserver --configdb-pass="$(cat /root/.dbpw)" --master-pass="$(cat /root/.oxpw)" --network-listener-host=localhost --servermemory 1024

'''servername''' is more like a ''clustername'' and needs to be the same for all nodes.

'''servermemory''' should be adjusted to reflect the expected number of concurrent active sessions; sizing assumption is 4MB per session.

<div style="padding: 20px; border-style: solid; border-width: thin;">
Cluster Note

--configdb-readhost=...
--configdb-writehost=...
--imapserver=...
--smtpserver=...
--mail-login-src=<login|mail|name>
--mail-server-src=<user|global>
--transport-server-src=<user|global>
--jkroute=APP1
--object-link-hostname=[service DNS name like ox.example.com]
--extras-link=[https://ox.example.com/]
--name-of-oxcluster=[something unique per cluster, like business-staging; see --servername]
--network-listener-host=<localhost|*>

oxinstaller needs to be run on each cluster node with identical options besides the jkroute, which must be unique per cluster node and match the corresponding apache option, see below.

In a cluster you also want to configure hazelcast; see [[AppSuite:Running_a_cluster#Configuration]]. Most prominent options are

com.openexchange.hazelcast.enabled=true
com.openexchange.hazelcast.group.name=<reasonable unique group name>
com.openexchange.hazelcast.group.password=<unique password, CHANGE THE SHIPPED DEFAULT PASSWORD!>
com.openexchange.hazelcast.network.join=static # static is recommended over multicast for robustness
com.openexchange.hazelcast.network.join.static.nodes=... # configure your nodes as a comma-separated list; a bootstrapping subset is acceptable
com.openexchange.hazelcast.network.interfaces=... # pick your subnet; Wildcards (*) and ranges (-) can be used
</div>

Start the service:

systemctl restart open-xchange

<div style="padding: 20px; border-style: solid; border-width: thin;">
Cluster Note

Start the service on every cluster node.
</div>

=== Registering stuff ===

Register the "server":

/opt/open-xchange/sbin/registerserver -n oxserver -A oxadminmaster -P "$(cat /root/.oxpw)"

<div style="padding: 20px; border-style: solid; border-width: thin;">
Cluster Note

All the register* commands need to be issued only once per cluster, as the effect is to enter corresponding lines in the configdb.
</div>

And the filestore:

/opt/open-xchange/sbin/registerfilestore -A oxadminmaster -P "$(cat /root/.oxpw)" -t file:/var/opt/filestore -s 1000000 -x 1000000

<div style="padding: 20px; border-style: solid; border-width: thin;">
Cluster Note

If you chose an object store, the corresponding registerfilestore line reads as follows (Ceph radosgw example):

/opt/open-xchange/sbin/registerfilestore -A oxadminmaster -P "$(cat /root/.oxpw)" -t s3://radosgw -s 1000000 -x 1000000

It requires configuration of the object store in filestore-s3.properties:

com.openexchange.filestore.s3.radosgw.endpoint=http://localhost:7480
com.openexchange.filestore.s3.radosgw.bucketName=oxbucket
com.openexchange.filestore.s3.radosgw.region=eu-west-1
com.openexchange.filestore.s3.radosgw.pathStyleAccess=true
com.openexchange.filestore.s3.radosgw.accessKey=...
com.openexchange.filestore.s3.radosgw.secretKey=...
com.openexchange.filestore.s3.radosgw.encryption=none
com.openexchange.filestore.s3.radosgw.signerOverride=S3SignerType
com.openexchange.filestore.s3.radosgw.chunkSize=5MB

Changing this file needs another

service open-xchange restart
</div>

And the database:

/opt/open-xchange/sbin/registerdatabase -A oxadminmaster -P "$(cat /root/.oxpw)" -n oxdb -p "$(cat /root/.dbpw)" -m true

<div style="padding: 20px; border-style: solid; border-width: thin;">
Cluster Note

You probably have multiple clusters with read and write URLs. Register them each like first registering the master, subsequently registering the slave URL with the corresponding master ID:

/opt/open-xchange/sbin/registerdatabase -A oxadminmaster -P "$(cat /root/.oxpw)" -n oxdb -p "$(cat /root/.dbpw)" -m true

This command gives as output the id of the registered db master, like

database 3 registered

Here, the id is 3. Use this id as argument of the -M switch of the next command:

/opt/open-xchange/sbin/registerdatabase -A oxadminmaster -P "$(cat /root/.oxpw)" -n oxdbr -p "$(cat /root/.dbpw)" -m false -M 3
</div>

=== Configure Apache ===

Create config files /etc/apache2/conf-enabled/proxy_http.conf, /etc/apache2/sites-enabled/000-default.conf by copy-pasting as explained in [[AppSuite:Open-Xchange_Installation_Guide_for_Debian_8.0#Configure_services]]

Make sure you are using mpm_event. Apply concurrent connections tuning as described in [[Tune_apache2_for_more_concurrent_connections]].

Configure modules and restart:

a2enmod proxy proxy_http proxy_balancer expires deflate headers rewrite mime setenvif lbmethod_byrequests
systemctl restart apache2

<div style="padding: 20px; border-style: solid; border-width: thin;">
Cluster Note

Make sure that each middleware node got its unique server.properties:com.openexchange.server.backendRoute, e.g. APP1, APP2, APP3, etc.

Configure them in the http proxy definitions like

BalancerMember http://ox1:8009 timeout=100 smax=0 ttl=60 retry=60 loadfactor=50 route=APP1
BalancerMember http://ox2:8009 timeout=100 smax=0 ttl=60 retry=60 loadfactor=50 route=APP2
BalancerMember http://ox3:8009 timeout=100 smax=0 ttl=60 retry=60 loadfactor=50 route=APP3

If you colocate apache on middleware nodes, you might want to minimize cross-node routing, by setting on each node for the local node a loadfactor=50 and for the other nodes a status=+H instead of the loadfactor. E.g. on node ox1:

BalancerMember http://ox1:8009 timeout=100 smax=0 ttl=60 retry=60 loadfactor=50 route=APP1
BalancerMember http://ox2:8009 timeout=100 smax=0 ttl=60 retry=60 status=+H route=APP2
BalancerMember http://ox3:8009 timeout=100 smax=0 ttl=60 retry=60 status=+H route=APP3

However this requires host-specific config files for apache http proxy.

Use touch-appsuite with one identical timestamp on each frontend node.

timestamp=$(date -u +%Y%m%d.%H%M%S)
for node in $all_frontend_nodes; do
ssh $node /opt/open-xchange/sbin/touch-appsuite --timestamp=$timestamp
done

</div>

=== Provision a Test User ===

Provision a sample context and user:

/opt/open-xchange/sbin/createcontext -c 1 -A oxadminmaster -P secret -N localdomain -u oxadmin -d "Admin User" -g Admin -s User -p secret -e oxadmin@localdomain -q 100 --access-combination-name groupware_premium
/opt/open-xchange/sbin/createuser -c 1 -A oxadmin -P secret -u testuser -d "Test User" -g Test -s User -p secret -e testuser@localdomain --access-combination-name groupware_premium

[[Context_Preprovisioning#Sample_Script]] provides an example how to fast-mode provision a huge number of contexts quickly.

AppSuite:Running a cluster

2017-12-13T08:10:58Z

Dominik.epple: /* Rolling Upgrade without breaking Hazelcast upgrade */

<div class="title">Running a cluster</div>

__TOC__

= Concepts =

For inter-OX-communication over the network, multiple Open-Xchange servers can form a cluster. This brings different advantages regarding distribution and caching of volatile data, load balancing, scalability, fail-safety and robustness. Additionally, it provides the infrastructure for upcoming features of the Open-Xchange server.
The clustering capabilities of the Open-Xchange server are mainly built up on [http://hazelcast.com Hazelcast], an open source clustering and highly scalable data distribution platform for Java. The following article provides an overview about the current featureset and configuration options.

= Requirements =

== Synchronized system clock times ==
It is crucial that all involved members in a cluster do have their system clock times in sync with each other; e.g. by using an NTP service.

== HTTP routing ==
An OX cluster is always part of a larger picture. Usually there is front level loadbalancer as central HTTPS entry point to the platform. This loadbalancer optionally performs HTTPS termination and forwards HTTP(S) requests to webservers (the usual and only supported choice as of now is Apache). These webservers are performing HTTPS termination (if this is not happening on the loadbalancer) and serve static content, and (which is what is relevant for our discussion here) they forward dynamic requests to the OX backends.

A central requirement for the interaction of these components (loadbalancer, webservers, OX nodes) is that we have session stability based on the JSESSIONID cookie / jsessionid path component suffix. This means that our application sets a cookie named JSESSIONID which has a value like <large decimal number>.<route identifier>, e.g. "5661584529655240315.OX1". The route identifier here ("OX1" in this example) is taken by the OX node from a configuration setting from a config file and is specific to one OX node. HTTP routing must happen such that HTTP requests with a cookie with such a suffix always end up the corresponding OX node. There are furthermore specific cirumstances when passing this information via cookie is not possible. Then the JSESSIONID is transferred in a path component as "jsessionid=..." in the HTTP request. The routing mechanism needs to take that into account also.

There are mainly two options to implement this. If the Apache processes are running co-located on the same machines running the OX groupware processes, it is often desired to have the front level loadbalancer perform HTTP routing to the correct machines. If dedicated Apache nodes are employed, is is usually sufficient to have the front-level loadbalancer do HTTP routing to the Apache nodes in a round-robin fashion and perform routing to the correct OX nodes in the Apache nodes.

We provide sample configuration files to configure Apache (with mod_proxy_http) to perform HTTP routing correctly in our guides on OXpedia, e.g. [[AppSuite:Main_Page_AppSuite#quickinstall]]. Central elements are the directives "ProxySet stickysession=JSESSIONID|jsessionid scolonpathdelim=On" in conjunction with the "route=OX1" parameters to the BalancerMember lines in the Proxy definition. This is valid for Apache 2.2 as of Sep-2014.

How to configure a front level loadbalancer to perform HTTP equivalent HTTP routing is dependent on the specific loadbalancer implementation. If Apache is used as front level loadbalancer, the same configuration as discussed in the previous section can be employed. As of time of writing this text (Sep 2014), the alternative choices are thin. F5 BigIP is reported to be able to implement "jsessionid based persistence using iRules". nginx has the functionality in their commercial "nginx plus" product. (Both of these options have not been tested by OX.) Other loadbalancers with this functionality are not known to us.

If the front level loadbalancer is not capable of performing correct HTTP routing, is is required to configure correct HTTP routing on Apache level, even if Apache runs co-located on the OX nodes and thus cross-routing happens.

There are several reasons why we require session stability in exactly this way. We require session stabilty for horizontal scale-out; while we support transparent resuming / migration of user sessions in the OX cluster without need for users to re-authenticate, sessions wandering around randomly will consume a fixed amount resources corresponding to a running session on each OX node in the cluster, while a session sticky to one OX node will consume this fixed amount of resources only on one OX node. Furthermore there are mechanisms in OX like TokenLogin which work only of all requests beloning to one sequence get routed to the same OX node even if they stem from different machines with different IPs. Only the JSESSIONID (which in this case is transferred as jsessionid path component, as cookies do not work during a 302 redirect, which is part of this sequence) carries the required information where the request must be routed to.

Usual "routing based on cookie hash" is not sufficient here since it disregards the information which machine originally issued the cookie. It only ensures that the session will be sticky to any target, which statistically will not be the same machine that issued the cookie. OX will then set a new JSESSIONID cookie, assuming the session had been migrated. The loadbalancer will then route the session to a different target, as the hash of the cookie will differ. This procedure then happens iteratively until by chance the routing based on cookie hash will route the session to the correct target. By then, a lot of resources will have been wasted, by creating full (short-term) sessions on all OX nodes. Furthermore, processes like TokenLogin will not work this way.

= Configuration =

All settings regarding cluster setup are located in the configuration file ''hazelcast.properties''. The former used additional files ''cluster.properties'', ''mdns.properties'' and ''static-cluster-discovery.properties'' are no longer needed. The following gives an overview about the most important settings - please refer to the inline documentation of the configuration file for more advanced options.

Note: The configuration guide targets v7.4.0 of the OX server (and above). For older versions, please consult the history of this page. A full list of Hazelcast-related properties is available at https://documentation.open-xchange.com/components/middleware/config/7.8.4/#mode=features&feature=Hazelcast .

== General ==

To restrict access to the cluster and to separate the cluster from others in the local network, a name and password needs to be defined. Only backend nodes having the same values for those properties are able to join and form a cluster.

# Configures the name of the cluster. Only nodes using the same group name
# will join each other and form the cluster. Required if
# "com.openexchange.hazelcast.network.join" is not "empty" (see below).
com.openexchange.hazelcast.group.name=

# The password used when joining the cluster. Defaults to "wtV6$VQk8#+3ds!a".
# Please change this value, and ensure it's equal on all nodes in the cluster.
com.openexchange.hazelcast.group.password=wtV6$VQk8#+3ds!a

== Network ==

It's required to define the network interface that is used for cluster communication via ''com.openexchange.hazelcast.network.interfaces''. By default, the interface is restricted to the local loopback address only. To allow the same configuration amongst all nodes in the cluster, it's recommended to define the value using a wildcard matching the IP addresses of all nodes participating in the cluster, e.g. ''192.168.0.*''

# Comma-separated list of interface addresses hazelcast should use. Wildcards
# (*) and ranges (-) can be used. Leave blank to listen on all interfaces
# Especially in server environments with multiple network interfaces, it's
# recommended to specify the IP-address of the network interface to bind to
# explicitly. Defaults to "127.0.0.1" (local loopback only), needs to be
# adjusted when building a cluster of multiple backend nodes.
com.openexchange.hazelcast.network.interfaces=127.0.0.1

To form a cluster of multiple OX server nodes, different discovery mechanisms can be used. The discovery mechanism is specified via the property ''com.openexchange.hazelcast.network.join'':

# Specifies which mechanism is used to discover other backend nodes in the
# cluster. Possible values are "empty" (no discovery for single-node setups),
# "static" (fixed set of cluster member nodes) or "multicast" (automatic
# discovery of other nodes via multicast). Defaults to "empty". Depending on
# the specified value, further configuration might be needed, see "Networking"
# section below.
com.openexchange.hazelcast.network.join=empty

Generally, it's advised to use the same network join mechanism for all nodes in the cluster, and, in most cases, it's strongly recommended to use a ''static'' network join configuration. This will allow the nodes to join the cluster directly upon startup. With a ''multicast'' based setup, nodes will merge to an existing cluster possibly at some later time, thus not being able to access the distributed data until they've joined.

Depending on the network join setting, further configuration may be necessary, as decribed in the following paragraphs.

=== empty ===

When using the default value ''empty'', no other nodes are discovered in the cluster. This value is suitable for single-node installations. Note that other nodes that are configured to use other network join mechanisms may be still able to still to connect to this node, e.g. using a ''static'' network join, having the IP address of this host in the list of potential cluster members (see below).

=== static ===

The most common setting for ''com.openexchange.hazelcast.network.join'' is ''static''. A static cluster discovery uses a fixed list of IP addresses of the nodes in the cluster. During startup and after a specific interval, the underlying Hazelcast library probes for not yet joined nodes from this list and adds them to the cluster automatically. The address list is configured via ''com.openexchange.hazelcast.network.join.static.nodes'':

# Configures a comma-separated list of IP addresses / hostnames of possible
# nodes in the cluster, e.g. "10.20.30.12, 10.20.30.13:5701, 192.178.168.110".
# Only used if "com.openexchange.hazelcast.network.join" is set to "static".
# It doesn't hurt if the address of the local host appears in the list, so
# that it's still possible to use the same list throughout all nodes in the
# cluster.
com.openexchange.hazelcast.network.join.static.nodes=

For a fixed set of backend nodes, it's recommended to simply include the IP addresses of all nodes in the list, and use the same configuration for each node. However, it's only required to add the address of at least one other node in the cluster to allow the node to join the cluster. Also, when adding a new node to the cluster and this list is extended accordingly, existing nodes don't need to be shut down to recognize the new node, as long as the new node's address list contains at least one of the already running nodes.

=== multicast ===

For highly dynamic setups where nodes are added and removed from the cluster quite often and/or the host's IP addresses are not fixed, it's also possible to configure the network join via multicast. During startup and after a specific interval, the backend nodes initiate the multicast join process automatically, and discovered nodes form or join the cluster afterwards. The multicast group and port can be configured as follows:

# Configures the multicast address used to discover other nodes in the cluster
# dynamically. Only used if "com.openexchange.hazelcast.network.join" is set
# to "multicast". If the nodes reside in different subnets, please ensure that
# multicast is enabled between the subnets. Defaults to "224.2.2.3".
com.openexchange.hazelcast.network.join.multicast.group=224.2.2.3

# Configures the multicast port used to discover other nodes in the cluster
# dynamically. Only used if "com.openexchange.hazelcast.network.join" is set
# to "multicast". Defaults to "54327".
com.openexchange.hazelcast.network.join.multicast.port=54327

== Example ==

The following example shows how a simple cluster named ''MyCluster'' consisting of 4 backend nodes can be configured using ''static'' cluster discovery. The node's IP addresses are 10.0.0.15, 10.0.0.16, 10.0.0.17 and 10.0.0.18. Note that the same ''hazelcast.properties'' is used by all nodes.

com.openexchange.hazelcast.group.name=MyCluster
com.openexchange.hazelcast.group.password=secret
com.openexchange.hazelcast.network.join=static
com.openexchange.hazelcast.network.join.static.nodes=10.0.0.15,10.0.0.16,10.0.0.17,10.0.0.18
com.openexchange.hazelcast.network.interfaces=10.0.0.*

== Advanced Configuration ==

=== Lite Members (available since v7.8.4) ===

Lite members in a Hazelcast cluster are members that do not hold any data partitions, i.e. all read- and write operations to distributed maps are delegated to non-lite ("full") members. Apart from not having data partitions, lite members participate in the same way as other members: they can register listeners for distributed topics (e.g. cache invalidation events) or can be addressed for task execution (e.g. during realtime communication).

Similar to using a custom partitioning scheme, separating the nodes of a large cluster into few "full" members and many "lite" members helps to minimize the impact of JVM activities from a single node (mainly the garbage collector) on the whole cluster communication. Additionally, when starting or stopping lite members, no repartitioning of the distributed cluster data needs to be performed, which significantly decreases the node's startup- and shutdown time and reduces the necessary network communication to a minimum.

In medium or larger sized clusters, it is sufficient to have roughly 10 to 20 percent of the nodes configured as "full" members, while all other ones can be started as "lite" member nodes. Additionally, please note that the configured backup count in the map configurations should always be smaller than the total number of "full" members, otherwise, there may be problems if one of those data nodes is shut down temporarily for maintenance. So, the minimum number of "full" members is implicitly bound to the sum of a map's ''backupCount'' and ''asyncBackupCount'' properties, plus ''1'' for the original data partition.

The configured "full" members should preferrably not be used to serve client requests (by not adding them as endpoint in the loadbalancer), to ensure they are always responsive. Also, shutdown and startups of those "full" members should be reduced to a minimum to avoid repartitioning operations.

More general information regarding lite members is available at http://docs.hazelcast.org/docs/latest/manual/html-single/index.html#enabling-lite-members .

To configure a node as "lite" member, the following configuration should be applied in the node's ''hazelcast.properties'' file:

com.openexchange.hazelcast.liteMember=true

It's also recommended to use a "static" cluster discovery for the network join, and list all "full" member nodes here, so that join requests are handled by those nodes, too (and not the other nodes that are potentially prone to garbage collection delays.

=== Custom Partitioning ===

Note: Starting with v7.8.4, "Lite Members" should be used in favor of applying a custom partitioning scheme.

While originally being designed to separate the nodes holding distributed data into different risk groups for increased fail safety, a custom partitioning strategy may also be used to distinguish between nodes holding distributed data from those who should not.

This approach of custom partitioning may be used in a OX cluster, where usually different backend nodes serve different purposes. A common scenario is that there are nodes handling requests from the web interfaces, and others being responsible for USM/EAS traffic. Due to their nature of processing large chunks of synchronization data in memory, the USM/EAS nodes may encounter small delays when the Java garbage collector kicks in and suspends the Java Virtual Machine. Since those delays may also have an influence on hazelcast-based communication in the cluster, the idea is to instruct hazelcast to not store distributed data on that nodes. This is where a custom partitioning scheme comes into play.

To setup a custom partitioning scheme in the cluster, an additional ''hazelcast.xml'' configuration file is used, which should be placed into the ''hazelcast'' subdirectory of the OX configuration folder, usually at ''/opt/openexchange/etc/hazelcast''. Please note that it's vital that each node in the cluster is configured equally here, so the same ''hazelcast.xml'' file should be copied to each server. The configuration read from there is used as basis for all further settings that are taken from the ordinary ''hazelcast.properties'' config file.

To setup a custom partitioning scheme, the partition groups must be defined in the ''hazelcast.xml'' file. See the following file for an example configuration, where the three nodes ''10.10.10.60'', ''10.10.10.61'' and ''10.10.10.62'' are defined to form an own partitioning group each. Doing so, all distributed data will be stored at one of those nodes physically, while the corresponding backup data (if configured) at one of the other two nodes. All other nodes in the cluster will not be used to store distributed data, but will still be "full" hazelcast members, which is necessary for other cluster-wide operations the OX backends use.

Please note that the configured backup count in the map configurations should be smaller than the number of nodes here, otherwise, there may be problems if one of those data nodes is shut down temporarily for maintenance. So, the minimum number of nodes to define in the partition group sections is implicitly bound to the sum of a map's ''backupCount'' and ''asyncBackupCount'' properties, plus ''1'' for the original data partition.

<?xml version="1.0" encoding="UTF-8"?>


<hazelcast xsi:schemaLocation="http://www.hazelcast.com/schema/config hazelcast-config-3.1.xsd"
xmlns="http://www.hazelcast.com/schema/config"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
<partition-group enabled="true" group-type="CUSTOM">
<member-group>
<interface>10.10.10.60</interface>
</member-group>
<member-group>
<interface>10.10.10.61</interface>
</member-group>
<member-group>
<interface>10.10.10.62</interface>
</member-group>
</partition-group>
</hazelcast>

More general information regarding custom partioning is available at http://hazelcast.org/docs/latest/manual/html/partitiongroupconfig.html .

It's also recommended to use a "static" cluster discovery for the network join, and list same the nodes that are also configured in the parition groups here, so that join requests are handled by those nodes, too (and not the other nodes that are potentially prone to garbage collection delays.

After configuring a custom partitioning scheme, the data distribution may be verified, e.g. by inspecting the MBeans of the distributed maps via JMX.

= Features =

The following list gives an overview about different features that were implemented using the new cluster capabilities.

== Distributed Session Storage ==

Previously, when an Open-Xchange server was shutdown for maintenance, all user sessions that were bound to that machine were lost, i.e. the users needed to login again. With the distributed session storage, all sessions are backed by a distributed map in the cluster, so that they are no longer bound to a specific node in the cluster. When a node is shut down, the session data is still available in the cluster and can be accessed from the remaining nodes. The load-balancing techniques of the webserver then seamlessly routes the user session to another node, with no ''session expired'' errors. The distributed session storage comes with the package ''open-xchange-sessionstorage-hazelcast''. It's recommended to install this optional package in all clustered environments with multiple groupware server nodes.

'''Notes:'''
* While there's some kind of built-in session distribution among the nodes in the cluster, this should not be seen as a replacement for session-stickiness between the loadbalancer and groupware nodes, i.e. one should still configure the webserver to use sticky sessions for performance reasons.
* The distributed session storage is still an in-memory storage. While the session data is distributed and backed up on multiple nodes in the cluster, shutting down multiple or all nodes at the same time will lead to loss of the the distributed data. To avoid such data loss when shutting down a node, please follow the guidelines at [[ Updating_a_Cluster ]].

Depending on the cluster infrastructure, different backup-count configuration options might be set for the distributed session storage in the map configuration file ''sessions.properties'' in the ''hazelcast'' subdirectory:

com.openexchange.hazelcast.configuration.map.backupCount=1

The ''backupcount'' property configures the number of nodes with synchronized backups. Synchronized backups block operations until backups are successfully copied and acknowledgements are received. If 1 is set as the backup-count for example, then all entries of the map will be copied to another JVM for fail-safety. 0 means no backup. Any integer between 0 and 6. Default is 1, setting bigger than 6 has no effect.

com.openexchange.hazelcast.configuration.map.asyncBackupCount=0

The ''asyncbackup'' property configures the number of nodes with async backups. Async backups do not block operations and do not require acknowledgements. 0 means no backup. Any integer between 0 and 6. Default is 0, setting bigger than 6 has no effect.

Since session data is backed up by default continuously by multiple nodes in the cluster, the steps described in [[ Session_Migration ]] to trigger session migration to other nodes explicitly is obsolete and no longer needed with the distributed session storage.

Normally, sessions in the distributed storages are not evicted automatically, but are only removed when they're also removed from the session handler, either due to a logout operation or when exceeding the long-term session lifetime as configured by ''com.openexchange.sessiond.sessionLongLifeTime'' in ''sessiond.properties''. Under certain circumstances, i.e. the session is no longer accessed by the client and the OX node hosting the session in it's long-life container being shutdown, the remove operation from the distributed storage might not be triggered. Therefore, additionaly a maximum idle time of map-entries can be configured for the distributed sessions map via

com.openexchange.hazelcast.configuration.map.maxIdleSeconds=640000

To avoid unnecessary eviction, the value should be higher than the configured ''com.openexchange.sessiond.sessionLongLifeTime'' in ''sessiond.properties''.

== Remote Cache Invalidation ==

For faster access, groupware data is held in different caches by the server. Formerly, the caches utilized the TCP Lateral Auxiliary Cache plug in (LTCP) for the underlying JCS caches to broadcast updates and removals to caches on other OX nodes in the cluster. This could potentially lead to problems when remote invalidation was not working reliably due to network discovery problems. As an alternative, remote cache invalidation can also be performed using reliable publish/subscribe events built up on Hazelcast topics. This can be configured in the ''cache.properties'' configuration file, where the 'eventInvalidation' property can either be set to 'false' for the legacy behavior or 'true' for the new mechanism:

com.openexchange.caching.jcs.eventInvalidation=true

All nodes participating in the cluster should be configured equally.

Internally, if ''com.openexchange.caching.jcs.eventInvalidation'' is set to ''true'', LTCP is disabled in JCS caches. Instead, an internal mechanism based on distributed Hazelcast event topics is used to invalidate data throughout all nodes in the cluster after local update- and remove-operations. Put-operations aren't propagated (and haven't been with LTCP either), since all data put into caches can be locally loaded/evaluated at each node from the persistent storage layer.

Using Hazelcast-based cache invalidation also makes further configuration of the JCS auxiliaries obsolete in the ''cache.ccf'' configuration file. In that case, all ''jcs.auxiliary.LTCP.*'' configuration settings are virtually ignored. However, it's still required to mark caches that require cluster-wide invalidation via ''jcs.region.<cache_name>=LTCP'', just as before. So basically, when using the new default setting ''com.openexchange.caching.jcs.eventInvalidation=true'', it's recommended to just use the stock ''cache.ccf'' file, since no further LTCP configuration is required.

= Adminstration / Troubleshooting =

== Hazelcast Configuration ==

The underlying Hazelcast library can be configured using the file ''hazelcast.properties''.

'''Important''':<br>
By default property ''com.openexchange.hazelcast.network.interfaces'' is set to ''127.0.0.1''; meaning Hazelcast listens only to loop-back device. To build a cluster among remote nodes the appropriate network interface needs to be configured there. Leaving that property empty lets Hazelcast listen to all available network interfaces.

The Hazelcast JMX MBean can be enabled or disabled with the property ''com.openexchange.hazelcast.jmx''. The properties ''com.openexchange.hazelcast.mergeFirstRunDelay'' and ''com.openexchange.hazelcast.mergeRunDelay'' control the run intervals of the so-called ''Split Brain Handler'' of Hazelcast that initiates the cluster join process when a new node is started. More details can be found at http://www.hazelcast.com/docs/2.5/manual/single_html/#NetworkPartitioning.

The port ranges used by Hazelcast for incoming and outgoing connections can be controlled via the configuration parameters ''com.openexchange.hazelcast.networkConfig.port'', ''com.openexchange.hazelcast.networkConfig.portAutoIncrement'' and ''com.openexchange.hazelcast.networkConfig.outboundPortDefinitions''.

== Commandline Tool ==

To print out statistics about the cluster and the distributed data, the ''showruntimestats'' commandline tool can be executed witht the ''clusterstats'' ('c') argument. This provides an overview about the runtime cluster configuration of the node, other members in the cluster and distributed data structures.

== JMX ==

In the Open-Xchange server Java process, the MBean ''com.hazelcast'' can be used to monitor and manage different aspects of the underlying Hazelcast cluster. The ''com.hazelcast'' MBean provides detailed information about the cluster configuration and distributed data structures.

== Hazelcast Errors ==

When experiencing hazelcast related errors in the logfiles, most likely different versions of the packages are installed, leading to different message formats that can't be understood by nodes using another version. Examples for such errors are exceptions in hazelcast components regarding (de)serialization or other message processing.
This may happen when performing a consecutive update of all nodes in the cluster, where temporarily nodes with a heterogeneous setup try to communicate with each other. If the errors don't disappear after all nodes in the cluster have been update to the same package versions, it might be necessary to shutdown the cluster completely, so that all distributed data is cleared.

== Cluster Discovery Errors ==

* If the started OX nodes don't form a cluster, please double-check your configuration in ''hazelcast.properties''
* It's important to have the same cluster name defined in ''hazelcast.properties'' throughout all nodes in the cluster
* Especially when using multicast cluster discovery, it might take some time until the cluster is formed
* When using ''static'' cluster discovery, at least one other node in the cluster has to be configured in ''com.openexchange.hazelcast.network.join.static.nodes'' to allow joining, however, it's recommended to list all nodes in the cluster here

== Disable Cluster Features ==

The Hazelcast based clustering features can be disabled with the following property changes:
* Disable cluster discovery by setting ''com.openexchange.hazelcast.network.join'' to ''empty'' in ''hazelcast.properties''
* Disable Hazelcast by setting ''com.openexchange.hazelcast.enabled'' to false in ''hazelcast.properties''
* Disable message based cache event invalidation by setting ''com.openexchange.caching.jcs.eventInvalidation'' to ''false'' in ''cache.properties''

== Update from 6.22.1 to version 6.22.2 and above ==

As hazelcast will be used by default for the distribution of sessions starting 6.22.2 you have to adjust hazelcast according to our old cache configuration. First of all it's important that you install the open-xchange-sessionstorage-hazelcast package. This package will add the binding between hazelcast and the internal session management. Next you have to set a cluster name to the cluster.properties file (see [[#Cluster Discovery Errors]]). Furthermore you will have to add one of the two discovery modes mentioned in [[#Cluster Discovery]].

= Updating a Cluster =

Running a cluster means built-in failover on the one hand, but might require some attention when it comes to the point of upgrading the services on all nodes in the cluster. This chapter gives an overview about general concepts and hints for silent updates of the cluster.

== The Big Picture ==

Updating an OX App Suite cluster is possible in several ways. The involved steps always include

* Update the software by updating the packages through the distro's repository / software update tool
* Update the database schemas (so-called update tasks)

There are some precautions required, though.

=== Update Tasks Management ===

It is a feature of the OX App Suite middleware to automatically start update tasks on a database schema when a user tries to login whose context lives on that schema. For installations beyond a certain size, if you just update the OX App Suite software without special handling of the update tasks, user logins will trigger an uncontrolled storm of update tasks on the databases, potentially leading to resource contention, unnecessary long update tasks runtimes, excessive load on the database server, maybe even service outages.

So one key element of every update strategy is to avoid user logins on nodes which have already been updated to the new software version, while the database schemas are still on the old version. There are two fundamentally different approaches to this goal: use either a full downtime, or use a rolling update strategy.

We describe the update strategy in more detail in the next section. Note that these are still high-level outlines of the actual procedure, which requires additional details with regards to Hazelcast, given further down below.

==== Full downtime approach ====

The full downtime approach is quite straightforward and involves

* shutdown of all OX middleware nodes
* update the software on all OX App Suite (middleware and frontend) nodes
* execute the update tasks in a controlled way from one OX node
* restore the service

This is the most general approach and always available, even if the rolling approach is not available due to Hazelcast constraints.

==== Rolling strategy ====

It is possible to execute the update tasks decoupled from the real update of the rest from the cluster, days or even weeks ahead of time, with the following approach:

* If the load situation allows for it, take one node out of the loadbalancer (we call it the upgrade node). Otherwise, add a dedicated upgrade node to your cluster, identically configured to the other middleware nodes.
* Make sure there are no user sessions left on the upgrade node, and that no new sessions will be routed to that node
* update the software on the upgrade node
* execute all update tasks from the update node.

In the last step, users from affected schemas will be logged out and denied service while the update tasks are running on their database schema. This is typically a short unavailability (some minutes) for a small part (1000...7000 depending on the installation) of the user base. This unavailability is of much lower impact than the unavailability of a full downtime, but you still might want to do this in the off-business hours.

This way you end up with the production cluster running on the old version of OX App Suite, with the database already being upgraded to the next version. This is explicitly a valid and supported configuration. This approach offers the advantage that update tasks can be executed in advance, instead of doing them while the whole system is in a full maintenance downtime. Since update tasks can take some time, this is a considerable advantage.

For the actual upgrade of the production cluster, the remaining steps are:

* Upgrade and restart the OX App Suite software on one middleware node after another, one by one
* Upgrade the software on the OX App Suite frontend nodes (if these are separate nodes from the middleware nodes)

Hazelcast will ensure that sessions from nodes which you restart are taken over by other nodes in the cluster, so ideally this step works without losing user sessions.

For the rolling strategy to work as described, it is required that the old and new version of OX App Suite use compatible versions of the Hazelcast library. This is the case for most upgrades. However some upgrades must handle the situation that the new version of OX App Suite ships with a new version of Hazelcast incompatible to the version of Hazelcast shipped with the old version of OX App Suite. It will be stated in the release notes if this is the case for a given release. If so, then some additional steps are required during a rolling update to ensure session handling / invalidating during update tasks works properly. See below.

== HOWTO / step-by-step instructions ==

* Take backups of as much as possible (databases, OX config files, etc).
* Announce the maintenance to the users. The communication depends on which approach you chose: the full downtime approach will come with a full downtime for all users, while the rolling upgrade approach will result in some users will have a short loss of service while their schema upgrades.

=== Full downtime approach ===

* Initiate maintenance: Block HTTP sessions to the service. Put a reasonable maintenance page in place, probably some HTTP error 503 with a reasonable Retry-After header.
* Shutdown the service on all middleware nodes. Upgrade the software on all middleware and frontend nodes using the disto's package manager. See [[AppSuite:UpdatingOXPackages]] for details on how to do that. Don't forget the <code>touch-appsuite</code> step if required ("If you update only UI plugins without simultaneously upgrading the core UI packages to a new version").
* Start the <code>open-xchange</code> service on one node
* Execute update tasks from that node. See [[UpdateTasks]] for an explanation how to do that, in particular the [[UpdateTasks#How_to_see_all_schemas.3F|section]] about limited parallel execution.
* Start the <code>open-xchange</code> services on the middleware nodes.
* Perform some crosschecks like
** all middleware nodes joined the Hazelcast cluster
** all OSGI bundles (which are expected to be running) are running
** WebUI login is possible
** Some central functionality tests like sending mails, accessing drive, etc
* Restore service: allow HTTP sessions, remove the maintenance page.

=== Rolling Upgrade without breaking Hazelcast upgrade ===

Remember: as stated above, this is viable only if the release notes for the new version do not state that there are breaking Hazelcast changes. For example, with v7.8.4 there were breaking Hazelcast changes and in the Release Notes it was stated as follows.

https://software.open-xchange.com/products/appsuite/doc/Release_Notes_for_Release_7.8.4_2017-05-23.pdf
<blockquote>
Important - Please Note

There is a major Hazelcast library update to OX App Suite v7.8.4. This means that when updating from an earlier backend version, due to the upgraded library, it is not possible to form a cluster of nodes that run previous version of Hazelcast (i.e. exiting volatile data in the cluster will be lost during the update). A consistent Hazelcast cluster is needed for cluster-wide cache invalidation. To circumvent problems with database update tasks that need to perform cache invalidation, please follow the steps described here: http://oxpedia.org/wiki/index.php?title=AppSuite:Running_a_cluster#Upgrades_of_the_Hazelcast_library. Please also note that session migration is not possible between versions. This usually affects all user sessions that are stored in a distributed map, and will require the users to re-login after the update. Running incompatible versions of Hazelcast within a cluster will result in logentries showing the conflicting node and version information.
</blockquote>

If you find you are upgrading to a version with breaking Hazelcast changes, please consult the next section [[#Rolling_Upgrade_with_breaking_Hazelcast_upgrade]].

Pre-update:

* Take one middleware node (the upgrade node) out of the HTTP traffic by adjusting the apache mod_proxy tables. We propose a combination of the balancer_manager to do this during runtime without restart, but also update the config files to prevent service restarts of apache to accidentally route sessions to the upgrade node.
* Make sure there are no user sessions left on the upgrade node, and that no new sessions will be routed to that node
* Update packages on the upgrade node and restart the middleware service there. See [[AppSuite:UpdatingOXPackages]] for details on how to do that.
* Execute update tasks from that node. See [[UpdateTasks]] for an explanation how to do that. You might want to keep the load low on the DBs, to affect production operations as low as possible, and because with this decoupled update tasks approach there is no immediate time pressure. If you want to follow the [[UpdateTasks#How_to_see_all_schemas.3F|limited parallel]] approach, use a small, mild parallelity factor (e.g. 2 or maybe 4 if you know this by far does not saturate your DB platform).

Real Update:

* For one middleware cluster node after each nother:
** Update packages on that middleware node and restart the middleware service there. See [[AppSuite:UpdatingOXPackages]] for details on how to do that.
** Verify the node starts its bundles, joins the Hazelcast cluster, log files are clean, the node handles sessions
* For one frontend node after each other (if you've got separate frontend nodes):
** Update packages on that frontend node. See [[AppSuite:UpdatingOXPackages]] for details on how to do that.
* Finally, if required ("If you update only UI plugins without simultaneously upgrading the core UI packages to a new version"), execute <code>touch-appsuite</code> with a <code>--timestamp</code> argument as described on the page [[AppSuite:UpdatingOXPackages]]
* Perform final crosschecks like
** all middleware nodes joined the Hazelcast cluster
** all OSGI bundles (which are expected to be running) are running
** WebUI login is possible
** Some central functionality tests like sending mails, accessing drive, etc

=== Rolling Upgrade with breaking Hazelcast upgrade ===

Cf [[#Upgrades_of_the_Hazelcast_library]] below.

In principle the steps given in the previous section apply. However the upgrade needs to get the special Hazelcast Upgrade Package installed (e.g. one from <code>open-xchange-cluster-upgrade-from-76x</code>, <code>open-xchange-cluster-upgrade-from-780-782</code>, <code>open-xchange-cluster-upgrade-from-783</code>, <code>open-xchange-cluster-upgrade-from-784</code>, ...) during execution of the update tasks.

So the pre-update steps look like:

* Take one middleware node (the upgrade node) out of the HTTP traffic by adjusting the apache mod_proxy tables. We propose a combination of the balancer_manager to do this during runtime without restart, but also update the config files to prevent service restarts of apache to accidentally route sessions to the upgrade node.
* Make sure there are no user sessions left on the upgrade node, and that no new sessions will be routed to that node
* Update packages on the upgrade node and restart the middleware service there. See [[AppSuite:UpdatingOXPackages]] for details on how to do that.
* Install the special Hazelcast Upgrade Package on the upgrade node (e.g. one from <code>open-xchange-cluster-upgrade-from-76x</code>, <code>open-xchange-cluster-upgrade-from-780-782</code>, <code>open-xchange-cluster-upgrade-from-783</code>, <code>open-xchange-cluster-upgrade-from-784</code>, ...). Restart the service again.
* Execute update tasks from that node. See [[UpdateTasks]] for an explanation how to do that. You might want to keep the load low on the DBs, to affect production operations as low as possible, and because with this decoupled update tasks approach there is no immediate time pressure. If you want to follow the [[UpdateTasks#How_to_see_all_schemas.3F|limited parallel]] approach, use a small, mild parallelity factor (e.g. 2 or maybe 4 if you know this by far does not saturate your DB platform).

Note: don't worry if you don't see the upgrade node joining the legacy cluster: the upgrade node will not join the legacy cluster / not be visisble there since the upgrade node will be a so-called "native client" to the legacy cluster, and it will be created on the fly (and subsequently disposed again) for propagating an event. So also on <code>netstat</code> level the upgrade node will not have visible connections to the legacy cluster (unless for the very short timeframe when an actual even is sent). You can verify the functionality of that package by log lines like

<nowiki> Successfully initialzed Hazelcast client: <client-id>
Successfully got reference to cache event topic: cacheEvents-3
Publishing legacy cache event: <cache-event>

Successfully published legacy cache event, shutting down client after 546ms...</nowiki>

For the overly prudent it might be an idea to prepare a special test context with a test user living in its dedicated (test) schema, so you can test the functionality of this mechanis during upgrade first.

After the DB update tasks you can remove the special upgrade package again from the upgrade node.

The "Real Upgrade" procedure then looks like [[#Rolling_Upgrade_without_breaking_Hazelcast_upgrade|above]].

== Reference Documentation ==

=== Limitations ===

While in most cases a seamless, rolling upgrade of all nodes in the cluster is possible, there may be situations where nodes running a newer version of the Open-Xchange Server are not able to communicate with older nodes in the cluster, i.e. can't access distributed data or consume incompatible event notifications - especially, when the underlying Hazelcast library is part of the update, which does not support this scenario at the moment. In such cases, the release notes will contain corresponding information, so please have a look there before applying an update.

Additionally, there may always be some kind of race conditions during an update, i.e. client requests that can't be completed successfully or internal events not being deliverd to all nodes in the cluster. That's why the following information should only serve as a best-practices guide to minimize the impact of upgrades to the user experience.

=== Upgrading a single Node ===

Upgrading all nodes in the cluster should usually be done sequentially, i.o.w. one node after the other. This means that during the upgrade of one node, the node is temporarily disconnected from the other nodes in the cluster, and will join the cluster again after the update is completed. From the backend perspective, this is as easy as stopping the open-xchange service. other nodes in the cluster will recognize the disconnected node and start to repartition the shared cluster data automatically. But wait a minute - doing so would potentially lead to the webserver not registering the node being stopped immediately, resulting in temporary errors for currently logged in users until they are routed to another machine in the cluster. That's why it's good practice to tell the webserver's load balancer that the node should no longer fulfill incoming requests. The Apache Balancer Manager is an excellent tool for this ([http://httpd.apache.org/docs/2.2/mod/mod_status.html module ''mod_status'']). Look at the screen shot. Every node can be put into a disabled mode. Further requests will the redirected to other nodes in the cluster:

[[Image:balancer_manager.jpg]]

Afterwards, the open-xchange service on the disabled node can be stopped by executing:

$ /etc/init.d/open-xchange stop

or

$ service open-xchange stop

Now, the node is effectively in maintenance mode and any updates can take place. One could now verify the changed cluster infrastructure by accessing the Hazelcast MBeans either via JMX or the ''showruntimestats -c'' commandline tool (see above for details). There, the shut down node should no longer appear in the 'Member' section (com.hazelcast:type=Member).

When all upgrades are processed, the node open-xchange service can be started again by executing:

$ /etc/init.d/open-xchange start

or

$ service open-xchange start

As stated above, depending on the chosen cluster discovery mechanism, it might take some time until the node joins the cluster again. When using static cluster discovery, it will join the existing cluster usually directly during serivce startup, i.o.w. before other depending OSGi services are started. Otherwise, there might also be situations where the node cannot join the cluster directly, for example when there were no mDNS advertisments for other nodes in the cluster received yet. Then, it can take some additional time until the node finally joins the cluster. During startup of the node, you can observe the JMX console or the output of ''showruntimestats -c'' (com.hazelcast:type=Member) of another node in the cluster to verify when the node has joined.

After the node has joined, distributed data is re-partioned automatically, and the node is ready to server incoming requests again - so now the node can finally be enabled again in the load balancer configuration of the webserver. Afterwards, the next node in the cluster can be upgraded using the same procedure, until all nodes were processed.

=== Upgrades of the Hazelcast library ===

In case an upgrade includes a major update of the Hazelcast library, a newly upgraded node will usually not be able to connect to the nodes running the previous version. In this case, volatile cluster data is lost after all nodes in the cluster have been updated, including sessions held in the distributed session storage. As outlined above, the release notes will contain a corresponding warning in such cases.

Besides upgraded nodes not being able to access distributed data of the legacy cluster, this also affects new data not being available in the legacy cluster, which may cause troubles if the updated backend version needs to perform database update tasks. Database update tasks usually operate in a "blocking" way and all contexts associated with the schema being upgraded are disabled temporarily. Since context data itself is being held in caches on potentially each node in the cluster, the affected cache entries are invalidated during the database update. And, since cluster-wide cache invalidations again utilize Hazelcast functionality ([[#Remote Cache Invalidation]]), such invalidations normally won't be propagated to nodes running a previous version of the Hazelcast library.

To work around this specific scenario where an incompatible upgrade of the Hazelcast library needs to be performed along with blocking database update tasks, starting with v7.8.0, a supplementary package is available that explicitly enables the context cache invalidation of nodes running the previous Hazelcast library. This package follows the naming scheme ''open-xchange-cluster-upgrade-from-XXX'' (where XXX representing the version of the legacy version of the Open-Xchange server), and is available in the repositories for the updated server packages. This package should only be installed on the first node of the cluster that is going to be upgraded to the new version, and can be deactivated once the database upgrade tasks were executed successfully.

Once installed, a legacy cluster is discovered based on the available information in the ''hazelcast.properties'' configuration file in case cluster discovery is set to ''static''. If ''multicast'' is used, there's an alternative option to configure at least one of the addresses of the legacy cluster via ''com.openexchange.hazelcast.network.client.nodes''.

As an example, along with the server v7.8.0, a new package named ''open-xchange-cluster-upgrade-from-76x'' can be installed that aids in invalidating cluster server nodes running v7.6.x (which includes the Hazelcast library in version 3.2.4). Using this package, the recommended steps to update an OX cluster from version 7.6.x to version 7.8.0 would be:
# Pick a node from your cluster that you want to use for executing the database update tasks shipped with the new release
# Disable this node for incoming HTTP requests in your webserver configuration as described at [[#Upgrading a single Node]]
# Update the OX packages on this node, additionally install the package ''open-xchange-cluster-upgrade-from-76x''
# Restart the open-xchange services on this node
# Trigger the update task executions using the ''runUpdate'' commandline utitlty as described at [[UpdateTasks]]
# Once they are finished, uninstall the package ''open-xchange-cluster-upgrade-from-76x'' again
# Restart the open-xchange services on this node
# Re-enable the node for incoming HTTP requests in your webserver configuration as described at [[#Upgrading a single Node]]
# Upgrade all other nodes in the cluster as described at [[#Upgrading a single Node]]

Same steps apply to upgrading from v7.8.0 through v7.8.2 (incl.) to v7.8.3 using the package named ''open-xchange-cluster-upgrade-from-780-782'', since v7.8.0 through v7.8.2 (incl.) utilize Hazelcast v3.5.x, while v7.8.3 uses Hazelcast v3.6.4

Same steps apply to upgrading from v7.8.3 to v7.8.4 using the package named ''open-xchange-cluster-upgrade-from-783'', since v7.8.3 utilizes Hazelcast v3.7.1

Same steps apply to upgrading from v7.8.4 to v7.10.0 using the package named ''open-xchange-cluster-upgrade-from-784'', since v7.8.4 utilizes Hazelcast v3.8.1

'''Operations Note:''' The upgraded node will be added as so-called [http://docs.hazelcast.org/docs/2.3/manual/html/ch15.html Native Client] to the legacy Hazelcast Cluster.

<blockquote>
Native Client enables you to do all Hazelcast operations without being a member of the cluster.
[...]

However Native client is not member and relies on one of the cluster members.
</blockquote>

This means, the upgraded node will not be visible in the members list of the legacy Hazelcast cluster (<code>showruntimestats -c</code>). Furthermore, the native client will created and destructed on single context events, with the effect that connections will only be visible in the very moment of such an event. This means effectively that verification of the invalidation mechanis is only possible by actually executing the <code>runupdate</code> CLT. This should produce log lines like

Successfully initialzed Hazelcast client: <client-id>
Successfully got reference to cache event topic: cacheEvents-3
Publishing legacy cache event: <cache-event>
Successfully published legacy cache event, shutting down client after 546ms...

Most importantly, you should be able to observe correct functionality (users of affected contexts being logged out). It may be handy to prepare a dedicated schema with just test contexts inside. (How to create this is out of scope here, but hint: use <code>createschema</code> and <code>createcontext --schema-name</code>.)

=== Other Considerations ===

* It's always recommended to only upgrade one node after the other, always ensuring that the cluster has formed correctly between each shutdown/startup of a node.
* Do not stop a node while running the runUpdate script or the associated update task.
* During the time of such a rolling upgrade of all nodes, we have effectively heterogeneous software versions in the cluster, which potentially might lead to temporary inconsistencies. Therefore, all nodes in the cluster should be updated in one cycle (but still one after the other).
* Following the above guideline, it's also possible to add or remove nodes dynamically to the cluster, not only when disconnecting a node temporary for updates.
* In case of trouble, i.e. a node refuses to join the cluster again after restart, consult the logfiles first for any hints about what is causing the problem - both on the disconnected node, and also on other nodes in the network
* If there are general incompatibilities between two revisions of the Open-Xchange Server that prevent an operation in a cluster (release notes), it's recommended to choose another name for the cluster in ''cluster.properties'' for the nodes with the new version. This will temporary lead to two separate clusters during the rolling upgrade, and finally the old cluster being shut down completely after the last node was updated to the new version. While distributed data can't be migrated from one server version to another in this scenario due to incompatibilities, the uptime of the system itself is not affected, since the nodes in the new cluster are able to serve new incoming requests directly.
* When updating only UI plugins without also updating to a new version of the core UI, you also need to perform the additional step from [[AppSuite:UpdatingOXPackages#Updating_UI_plugins|Updating UI plugins]].

[[Category: AppSuite]] [[Category: Administration]] [[Category: Cluster]]

AppSuite:Running a cluster

2017-12-13T08:10:24Z

Dominik.epple: /* Full downtime approach */

<div class="title">Running a cluster</div>

__TOC__

= Concepts =

For inter-OX-communication over the network, multiple Open-Xchange servers can form a cluster. This brings different advantages regarding distribution and caching of volatile data, load balancing, scalability, fail-safety and robustness. Additionally, it provides the infrastructure for upcoming features of the Open-Xchange server.
The clustering capabilities of the Open-Xchange server are mainly built up on [http://hazelcast.com Hazelcast], an open source clustering and highly scalable data distribution platform for Java. The following article provides an overview about the current featureset and configuration options.

= Requirements =

== Synchronized system clock times ==
It is crucial that all involved members in a cluster do have their system clock times in sync with each other; e.g. by using an NTP service.

== HTTP routing ==
An OX cluster is always part of a larger picture. Usually there is front level loadbalancer as central HTTPS entry point to the platform. This loadbalancer optionally performs HTTPS termination and forwards HTTP(S) requests to webservers (the usual and only supported choice as of now is Apache). These webservers are performing HTTPS termination (if this is not happening on the loadbalancer) and serve static content, and (which is what is relevant for our discussion here) they forward dynamic requests to the OX backends.

A central requirement for the interaction of these components (loadbalancer, webservers, OX nodes) is that we have session stability based on the JSESSIONID cookie / jsessionid path component suffix. This means that our application sets a cookie named JSESSIONID which has a value like <large decimal number>.<route identifier>, e.g. "5661584529655240315.OX1". The route identifier here ("OX1" in this example) is taken by the OX node from a configuration setting from a config file and is specific to one OX node. HTTP routing must happen such that HTTP requests with a cookie with such a suffix always end up the corresponding OX node. There are furthermore specific cirumstances when passing this information via cookie is not possible. Then the JSESSIONID is transferred in a path component as "jsessionid=..." in the HTTP request. The routing mechanism needs to take that into account also.

There are mainly two options to implement this. If the Apache processes are running co-located on the same machines running the OX groupware processes, it is often desired to have the front level loadbalancer perform HTTP routing to the correct machines. If dedicated Apache nodes are employed, is is usually sufficient to have the front-level loadbalancer do HTTP routing to the Apache nodes in a round-robin fashion and perform routing to the correct OX nodes in the Apache nodes.

We provide sample configuration files to configure Apache (with mod_proxy_http) to perform HTTP routing correctly in our guides on OXpedia, e.g. [[AppSuite:Main_Page_AppSuite#quickinstall]]. Central elements are the directives "ProxySet stickysession=JSESSIONID|jsessionid scolonpathdelim=On" in conjunction with the "route=OX1" parameters to the BalancerMember lines in the Proxy definition. This is valid for Apache 2.2 as of Sep-2014.

How to configure a front level loadbalancer to perform HTTP equivalent HTTP routing is dependent on the specific loadbalancer implementation. If Apache is used as front level loadbalancer, the same configuration as discussed in the previous section can be employed. As of time of writing this text (Sep 2014), the alternative choices are thin. F5 BigIP is reported to be able to implement "jsessionid based persistence using iRules". nginx has the functionality in their commercial "nginx plus" product. (Both of these options have not been tested by OX.) Other loadbalancers with this functionality are not known to us.

If the front level loadbalancer is not capable of performing correct HTTP routing, is is required to configure correct HTTP routing on Apache level, even if Apache runs co-located on the OX nodes and thus cross-routing happens.

There are several reasons why we require session stability in exactly this way. We require session stabilty for horizontal scale-out; while we support transparent resuming / migration of user sessions in the OX cluster without need for users to re-authenticate, sessions wandering around randomly will consume a fixed amount resources corresponding to a running session on each OX node in the cluster, while a session sticky to one OX node will consume this fixed amount of resources only on one OX node. Furthermore there are mechanisms in OX like TokenLogin which work only of all requests beloning to one sequence get routed to the same OX node even if they stem from different machines with different IPs. Only the JSESSIONID (which in this case is transferred as jsessionid path component, as cookies do not work during a 302 redirect, which is part of this sequence) carries the required information where the request must be routed to.

Usual "routing based on cookie hash" is not sufficient here since it disregards the information which machine originally issued the cookie. It only ensures that the session will be sticky to any target, which statistically will not be the same machine that issued the cookie. OX will then set a new JSESSIONID cookie, assuming the session had been migrated. The loadbalancer will then route the session to a different target, as the hash of the cookie will differ. This procedure then happens iteratively until by chance the routing based on cookie hash will route the session to the correct target. By then, a lot of resources will have been wasted, by creating full (short-term) sessions on all OX nodes. Furthermore, processes like TokenLogin will not work this way.

= Configuration =

All settings regarding cluster setup are located in the configuration file ''hazelcast.properties''. The former used additional files ''cluster.properties'', ''mdns.properties'' and ''static-cluster-discovery.properties'' are no longer needed. The following gives an overview about the most important settings - please refer to the inline documentation of the configuration file for more advanced options.

Note: The configuration guide targets v7.4.0 of the OX server (and above). For older versions, please consult the history of this page. A full list of Hazelcast-related properties is available at https://documentation.open-xchange.com/components/middleware/config/7.8.4/#mode=features&feature=Hazelcast .

== General ==

To restrict access to the cluster and to separate the cluster from others in the local network, a name and password needs to be defined. Only backend nodes having the same values for those properties are able to join and form a cluster.

# Configures the name of the cluster. Only nodes using the same group name
# will join each other and form the cluster. Required if
# "com.openexchange.hazelcast.network.join" is not "empty" (see below).
com.openexchange.hazelcast.group.name=

# The password used when joining the cluster. Defaults to "wtV6$VQk8#+3ds!a".
# Please change this value, and ensure it's equal on all nodes in the cluster.
com.openexchange.hazelcast.group.password=wtV6$VQk8#+3ds!a

== Network ==

It's required to define the network interface that is used for cluster communication via ''com.openexchange.hazelcast.network.interfaces''. By default, the interface is restricted to the local loopback address only. To allow the same configuration amongst all nodes in the cluster, it's recommended to define the value using a wildcard matching the IP addresses of all nodes participating in the cluster, e.g. ''192.168.0.*''

# Comma-separated list of interface addresses hazelcast should use. Wildcards
# (*) and ranges (-) can be used. Leave blank to listen on all interfaces
# Especially in server environments with multiple network interfaces, it's
# recommended to specify the IP-address of the network interface to bind to
# explicitly. Defaults to "127.0.0.1" (local loopback only), needs to be
# adjusted when building a cluster of multiple backend nodes.
com.openexchange.hazelcast.network.interfaces=127.0.0.1

To form a cluster of multiple OX server nodes, different discovery mechanisms can be used. The discovery mechanism is specified via the property ''com.openexchange.hazelcast.network.join'':

# Specifies which mechanism is used to discover other backend nodes in the
# cluster. Possible values are "empty" (no discovery for single-node setups),
# "static" (fixed set of cluster member nodes) or "multicast" (automatic
# discovery of other nodes via multicast). Defaults to "empty". Depending on
# the specified value, further configuration might be needed, see "Networking"
# section below.
com.openexchange.hazelcast.network.join=empty

Generally, it's advised to use the same network join mechanism for all nodes in the cluster, and, in most cases, it's strongly recommended to use a ''static'' network join configuration. This will allow the nodes to join the cluster directly upon startup. With a ''multicast'' based setup, nodes will merge to an existing cluster possibly at some later time, thus not being able to access the distributed data until they've joined.

Depending on the network join setting, further configuration may be necessary, as decribed in the following paragraphs.

=== empty ===

When using the default value ''empty'', no other nodes are discovered in the cluster. This value is suitable for single-node installations. Note that other nodes that are configured to use other network join mechanisms may be still able to still to connect to this node, e.g. using a ''static'' network join, having the IP address of this host in the list of potential cluster members (see below).

=== static ===

The most common setting for ''com.openexchange.hazelcast.network.join'' is ''static''. A static cluster discovery uses a fixed list of IP addresses of the nodes in the cluster. During startup and after a specific interval, the underlying Hazelcast library probes for not yet joined nodes from this list and adds them to the cluster automatically. The address list is configured via ''com.openexchange.hazelcast.network.join.static.nodes'':

# Configures a comma-separated list of IP addresses / hostnames of possible
# nodes in the cluster, e.g. "10.20.30.12, 10.20.30.13:5701, 192.178.168.110".
# Only used if "com.openexchange.hazelcast.network.join" is set to "static".
# It doesn't hurt if the address of the local host appears in the list, so
# that it's still possible to use the same list throughout all nodes in the
# cluster.
com.openexchange.hazelcast.network.join.static.nodes=

For a fixed set of backend nodes, it's recommended to simply include the IP addresses of all nodes in the list, and use the same configuration for each node. However, it's only required to add the address of at least one other node in the cluster to allow the node to join the cluster. Also, when adding a new node to the cluster and this list is extended accordingly, existing nodes don't need to be shut down to recognize the new node, as long as the new node's address list contains at least one of the already running nodes.

=== multicast ===

For highly dynamic setups where nodes are added and removed from the cluster quite often and/or the host's IP addresses are not fixed, it's also possible to configure the network join via multicast. During startup and after a specific interval, the backend nodes initiate the multicast join process automatically, and discovered nodes form or join the cluster afterwards. The multicast group and port can be configured as follows:

# Configures the multicast address used to discover other nodes in the cluster
# dynamically. Only used if "com.openexchange.hazelcast.network.join" is set
# to "multicast". If the nodes reside in different subnets, please ensure that
# multicast is enabled between the subnets. Defaults to "224.2.2.3".
com.openexchange.hazelcast.network.join.multicast.group=224.2.2.3

# Configures the multicast port used to discover other nodes in the cluster
# dynamically. Only used if "com.openexchange.hazelcast.network.join" is set
# to "multicast". Defaults to "54327".
com.openexchange.hazelcast.network.join.multicast.port=54327

== Example ==

The following example shows how a simple cluster named ''MyCluster'' consisting of 4 backend nodes can be configured using ''static'' cluster discovery. The node's IP addresses are 10.0.0.15, 10.0.0.16, 10.0.0.17 and 10.0.0.18. Note that the same ''hazelcast.properties'' is used by all nodes.

com.openexchange.hazelcast.group.name=MyCluster
com.openexchange.hazelcast.group.password=secret
com.openexchange.hazelcast.network.join=static
com.openexchange.hazelcast.network.join.static.nodes=10.0.0.15,10.0.0.16,10.0.0.17,10.0.0.18
com.openexchange.hazelcast.network.interfaces=10.0.0.*

== Advanced Configuration ==

=== Lite Members (available since v7.8.4) ===

Lite members in a Hazelcast cluster are members that do not hold any data partitions, i.e. all read- and write operations to distributed maps are delegated to non-lite ("full") members. Apart from not having data partitions, lite members participate in the same way as other members: they can register listeners for distributed topics (e.g. cache invalidation events) or can be addressed for task execution (e.g. during realtime communication).

Similar to using a custom partitioning scheme, separating the nodes of a large cluster into few "full" members and many "lite" members helps to minimize the impact of JVM activities from a single node (mainly the garbage collector) on the whole cluster communication. Additionally, when starting or stopping lite members, no repartitioning of the distributed cluster data needs to be performed, which significantly decreases the node's startup- and shutdown time and reduces the necessary network communication to a minimum.

In medium or larger sized clusters, it is sufficient to have roughly 10 to 20 percent of the nodes configured as "full" members, while all other ones can be started as "lite" member nodes. Additionally, please note that the configured backup count in the map configurations should always be smaller than the total number of "full" members, otherwise, there may be problems if one of those data nodes is shut down temporarily for maintenance. So, the minimum number of "full" members is implicitly bound to the sum of a map's ''backupCount'' and ''asyncBackupCount'' properties, plus ''1'' for the original data partition.

The configured "full" members should preferrably not be used to serve client requests (by not adding them as endpoint in the loadbalancer), to ensure they are always responsive. Also, shutdown and startups of those "full" members should be reduced to a minimum to avoid repartitioning operations.

More general information regarding lite members is available at http://docs.hazelcast.org/docs/latest/manual/html-single/index.html#enabling-lite-members .

To configure a node as "lite" member, the following configuration should be applied in the node's ''hazelcast.properties'' file:

com.openexchange.hazelcast.liteMember=true

It's also recommended to use a "static" cluster discovery for the network join, and list all "full" member nodes here, so that join requests are handled by those nodes, too (and not the other nodes that are potentially prone to garbage collection delays.

=== Custom Partitioning ===

Note: Starting with v7.8.4, "Lite Members" should be used in favor of applying a custom partitioning scheme.

While originally being designed to separate the nodes holding distributed data into different risk groups for increased fail safety, a custom partitioning strategy may also be used to distinguish between nodes holding distributed data from those who should not.

This approach of custom partitioning may be used in a OX cluster, where usually different backend nodes serve different purposes. A common scenario is that there are nodes handling requests from the web interfaces, and others being responsible for USM/EAS traffic. Due to their nature of processing large chunks of synchronization data in memory, the USM/EAS nodes may encounter small delays when the Java garbage collector kicks in and suspends the Java Virtual Machine. Since those delays may also have an influence on hazelcast-based communication in the cluster, the idea is to instruct hazelcast to not store distributed data on that nodes. This is where a custom partitioning scheme comes into play.

To setup a custom partitioning scheme in the cluster, an additional ''hazelcast.xml'' configuration file is used, which should be placed into the ''hazelcast'' subdirectory of the OX configuration folder, usually at ''/opt/openexchange/etc/hazelcast''. Please note that it's vital that each node in the cluster is configured equally here, so the same ''hazelcast.xml'' file should be copied to each server. The configuration read from there is used as basis for all further settings that are taken from the ordinary ''hazelcast.properties'' config file.

To setup a custom partitioning scheme, the partition groups must be defined in the ''hazelcast.xml'' file. See the following file for an example configuration, where the three nodes ''10.10.10.60'', ''10.10.10.61'' and ''10.10.10.62'' are defined to form an own partitioning group each. Doing so, all distributed data will be stored at one of those nodes physically, while the corresponding backup data (if configured) at one of the other two nodes. All other nodes in the cluster will not be used to store distributed data, but will still be "full" hazelcast members, which is necessary for other cluster-wide operations the OX backends use.

Please note that the configured backup count in the map configurations should be smaller than the number of nodes here, otherwise, there may be problems if one of those data nodes is shut down temporarily for maintenance. So, the minimum number of nodes to define in the partition group sections is implicitly bound to the sum of a map's ''backupCount'' and ''asyncBackupCount'' properties, plus ''1'' for the original data partition.

<?xml version="1.0" encoding="UTF-8"?>


<hazelcast xsi:schemaLocation="http://www.hazelcast.com/schema/config hazelcast-config-3.1.xsd"
xmlns="http://www.hazelcast.com/schema/config"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
<partition-group enabled="true" group-type="CUSTOM">
<member-group>
<interface>10.10.10.60</interface>
</member-group>
<member-group>
<interface>10.10.10.61</interface>
</member-group>
<member-group>
<interface>10.10.10.62</interface>
</member-group>
</partition-group>
</hazelcast>

More general information regarding custom partioning is available at http://hazelcast.org/docs/latest/manual/html/partitiongroupconfig.html .

It's also recommended to use a "static" cluster discovery for the network join, and list same the nodes that are also configured in the parition groups here, so that join requests are handled by those nodes, too (and not the other nodes that are potentially prone to garbage collection delays.

After configuring a custom partitioning scheme, the data distribution may be verified, e.g. by inspecting the MBeans of the distributed maps via JMX.

= Features =

The following list gives an overview about different features that were implemented using the new cluster capabilities.

== Distributed Session Storage ==

Previously, when an Open-Xchange server was shutdown for maintenance, all user sessions that were bound to that machine were lost, i.e. the users needed to login again. With the distributed session storage, all sessions are backed by a distributed map in the cluster, so that they are no longer bound to a specific node in the cluster. When a node is shut down, the session data is still available in the cluster and can be accessed from the remaining nodes. The load-balancing techniques of the webserver then seamlessly routes the user session to another node, with no ''session expired'' errors. The distributed session storage comes with the package ''open-xchange-sessionstorage-hazelcast''. It's recommended to install this optional package in all clustered environments with multiple groupware server nodes.

'''Notes:'''
* While there's some kind of built-in session distribution among the nodes in the cluster, this should not be seen as a replacement for session-stickiness between the loadbalancer and groupware nodes, i.e. one should still configure the webserver to use sticky sessions for performance reasons.
* The distributed session storage is still an in-memory storage. While the session data is distributed and backed up on multiple nodes in the cluster, shutting down multiple or all nodes at the same time will lead to loss of the the distributed data. To avoid such data loss when shutting down a node, please follow the guidelines at [[ Updating_a_Cluster ]].

Depending on the cluster infrastructure, different backup-count configuration options might be set for the distributed session storage in the map configuration file ''sessions.properties'' in the ''hazelcast'' subdirectory:

com.openexchange.hazelcast.configuration.map.backupCount=1

The ''backupcount'' property configures the number of nodes with synchronized backups. Synchronized backups block operations until backups are successfully copied and acknowledgements are received. If 1 is set as the backup-count for example, then all entries of the map will be copied to another JVM for fail-safety. 0 means no backup. Any integer between 0 and 6. Default is 1, setting bigger than 6 has no effect.

com.openexchange.hazelcast.configuration.map.asyncBackupCount=0

The ''asyncbackup'' property configures the number of nodes with async backups. Async backups do not block operations and do not require acknowledgements. 0 means no backup. Any integer between 0 and 6. Default is 0, setting bigger than 6 has no effect.

Since session data is backed up by default continuously by multiple nodes in the cluster, the steps described in [[ Session_Migration ]] to trigger session migration to other nodes explicitly is obsolete and no longer needed with the distributed session storage.

Normally, sessions in the distributed storages are not evicted automatically, but are only removed when they're also removed from the session handler, either due to a logout operation or when exceeding the long-term session lifetime as configured by ''com.openexchange.sessiond.sessionLongLifeTime'' in ''sessiond.properties''. Under certain circumstances, i.e. the session is no longer accessed by the client and the OX node hosting the session in it's long-life container being shutdown, the remove operation from the distributed storage might not be triggered. Therefore, additionaly a maximum idle time of map-entries can be configured for the distributed sessions map via

com.openexchange.hazelcast.configuration.map.maxIdleSeconds=640000

To avoid unnecessary eviction, the value should be higher than the configured ''com.openexchange.sessiond.sessionLongLifeTime'' in ''sessiond.properties''.

== Remote Cache Invalidation ==

For faster access, groupware data is held in different caches by the server. Formerly, the caches utilized the TCP Lateral Auxiliary Cache plug in (LTCP) for the underlying JCS caches to broadcast updates and removals to caches on other OX nodes in the cluster. This could potentially lead to problems when remote invalidation was not working reliably due to network discovery problems. As an alternative, remote cache invalidation can also be performed using reliable publish/subscribe events built up on Hazelcast topics. This can be configured in the ''cache.properties'' configuration file, where the 'eventInvalidation' property can either be set to 'false' for the legacy behavior or 'true' for the new mechanism:

com.openexchange.caching.jcs.eventInvalidation=true

All nodes participating in the cluster should be configured equally.

Internally, if ''com.openexchange.caching.jcs.eventInvalidation'' is set to ''true'', LTCP is disabled in JCS caches. Instead, an internal mechanism based on distributed Hazelcast event topics is used to invalidate data throughout all nodes in the cluster after local update- and remove-operations. Put-operations aren't propagated (and haven't been with LTCP either), since all data put into caches can be locally loaded/evaluated at each node from the persistent storage layer.

Using Hazelcast-based cache invalidation also makes further configuration of the JCS auxiliaries obsolete in the ''cache.ccf'' configuration file. In that case, all ''jcs.auxiliary.LTCP.*'' configuration settings are virtually ignored. However, it's still required to mark caches that require cluster-wide invalidation via ''jcs.region.<cache_name>=LTCP'', just as before. So basically, when using the new default setting ''com.openexchange.caching.jcs.eventInvalidation=true'', it's recommended to just use the stock ''cache.ccf'' file, since no further LTCP configuration is required.

= Adminstration / Troubleshooting =

== Hazelcast Configuration ==

The underlying Hazelcast library can be configured using the file ''hazelcast.properties''.

'''Important''':<br>
By default property ''com.openexchange.hazelcast.network.interfaces'' is set to ''127.0.0.1''; meaning Hazelcast listens only to loop-back device. To build a cluster among remote nodes the appropriate network interface needs to be configured there. Leaving that property empty lets Hazelcast listen to all available network interfaces.

The Hazelcast JMX MBean can be enabled or disabled with the property ''com.openexchange.hazelcast.jmx''. The properties ''com.openexchange.hazelcast.mergeFirstRunDelay'' and ''com.openexchange.hazelcast.mergeRunDelay'' control the run intervals of the so-called ''Split Brain Handler'' of Hazelcast that initiates the cluster join process when a new node is started. More details can be found at http://www.hazelcast.com/docs/2.5/manual/single_html/#NetworkPartitioning.

The port ranges used by Hazelcast for incoming and outgoing connections can be controlled via the configuration parameters ''com.openexchange.hazelcast.networkConfig.port'', ''com.openexchange.hazelcast.networkConfig.portAutoIncrement'' and ''com.openexchange.hazelcast.networkConfig.outboundPortDefinitions''.

== Commandline Tool ==

To print out statistics about the cluster and the distributed data, the ''showruntimestats'' commandline tool can be executed witht the ''clusterstats'' ('c') argument. This provides an overview about the runtime cluster configuration of the node, other members in the cluster and distributed data structures.

== JMX ==

In the Open-Xchange server Java process, the MBean ''com.hazelcast'' can be used to monitor and manage different aspects of the underlying Hazelcast cluster. The ''com.hazelcast'' MBean provides detailed information about the cluster configuration and distributed data structures.

== Hazelcast Errors ==

When experiencing hazelcast related errors in the logfiles, most likely different versions of the packages are installed, leading to different message formats that can't be understood by nodes using another version. Examples for such errors are exceptions in hazelcast components regarding (de)serialization or other message processing.
This may happen when performing a consecutive update of all nodes in the cluster, where temporarily nodes with a heterogeneous setup try to communicate with each other. If the errors don't disappear after all nodes in the cluster have been update to the same package versions, it might be necessary to shutdown the cluster completely, so that all distributed data is cleared.

== Cluster Discovery Errors ==

* If the started OX nodes don't form a cluster, please double-check your configuration in ''hazelcast.properties''
* It's important to have the same cluster name defined in ''hazelcast.properties'' throughout all nodes in the cluster
* Especially when using multicast cluster discovery, it might take some time until the cluster is formed
* When using ''static'' cluster discovery, at least one other node in the cluster has to be configured in ''com.openexchange.hazelcast.network.join.static.nodes'' to allow joining, however, it's recommended to list all nodes in the cluster here

== Disable Cluster Features ==

The Hazelcast based clustering features can be disabled with the following property changes:
* Disable cluster discovery by setting ''com.openexchange.hazelcast.network.join'' to ''empty'' in ''hazelcast.properties''
* Disable Hazelcast by setting ''com.openexchange.hazelcast.enabled'' to false in ''hazelcast.properties''
* Disable message based cache event invalidation by setting ''com.openexchange.caching.jcs.eventInvalidation'' to ''false'' in ''cache.properties''

== Update from 6.22.1 to version 6.22.2 and above ==

As hazelcast will be used by default for the distribution of sessions starting 6.22.2 you have to adjust hazelcast according to our old cache configuration. First of all it's important that you install the open-xchange-sessionstorage-hazelcast package. This package will add the binding between hazelcast and the internal session management. Next you have to set a cluster name to the cluster.properties file (see [[#Cluster Discovery Errors]]). Furthermore you will have to add one of the two discovery modes mentioned in [[#Cluster Discovery]].

= Updating a Cluster =

Running a cluster means built-in failover on the one hand, but might require some attention when it comes to the point of upgrading the services on all nodes in the cluster. This chapter gives an overview about general concepts and hints for silent updates of the cluster.

== The Big Picture ==

Updating an OX App Suite cluster is possible in several ways. The involved steps always include

* Update the software by updating the packages through the distro's repository / software update tool
* Update the database schemas (so-called update tasks)

There are some precautions required, though.

=== Update Tasks Management ===

It is a feature of the OX App Suite middleware to automatically start update tasks on a database schema when a user tries to login whose context lives on that schema. For installations beyond a certain size, if you just update the OX App Suite software without special handling of the update tasks, user logins will trigger an uncontrolled storm of update tasks on the databases, potentially leading to resource contention, unnecessary long update tasks runtimes, excessive load on the database server, maybe even service outages.

So one key element of every update strategy is to avoid user logins on nodes which have already been updated to the new software version, while the database schemas are still on the old version. There are two fundamentally different approaches to this goal: use either a full downtime, or use a rolling update strategy.

We describe the update strategy in more detail in the next section. Note that these are still high-level outlines of the actual procedure, which requires additional details with regards to Hazelcast, given further down below.

==== Full downtime approach ====

The full downtime approach is quite straightforward and involves

* shutdown of all OX middleware nodes
* update the software on all OX App Suite (middleware and frontend) nodes
* execute the update tasks in a controlled way from one OX node
* restore the service

This is the most general approach and always available, even if the rolling approach is not available due to Hazelcast constraints.

==== Rolling strategy ====

It is possible to execute the update tasks decoupled from the real update of the rest from the cluster, days or even weeks ahead of time, with the following approach:

* If the load situation allows for it, take one node out of the loadbalancer (we call it the upgrade node). Otherwise, add a dedicated upgrade node to your cluster, identically configured to the other middleware nodes.
* Make sure there are no user sessions left on the upgrade node, and that no new sessions will be routed to that node
* update the software on the upgrade node
* execute all update tasks from the update node.

In the last step, users from affected schemas will be logged out and denied service while the update tasks are running on their database schema. This is typically a short unavailability (some minutes) for a small part (1000...7000 depending on the installation) of the user base. This unavailability is of much lower impact than the unavailability of a full downtime, but you still might want to do this in the off-business hours.

This way you end up with the production cluster running on the old version of OX App Suite, with the database already being upgraded to the next version. This is explicitly a valid and supported configuration. This approach offers the advantage that update tasks can be executed in advance, instead of doing them while the whole system is in a full maintenance downtime. Since update tasks can take some time, this is a considerable advantage.

For the actual upgrade of the production cluster, the remaining steps are:

* Upgrade and restart the OX App Suite software on one middleware node after another, one by one
* Upgrade the software on the OX App Suite frontend nodes (if these are separate nodes from the middleware nodes)

Hazelcast will ensure that sessions from nodes which you restart are taken over by other nodes in the cluster, so ideally this step works without losing user sessions.

For the rolling strategy to work as described, it is required that the old and new version of OX App Suite use compatible versions of the Hazelcast library. This is the case for most upgrades. However some upgrades must handle the situation that the new version of OX App Suite ships with a new version of Hazelcast incompatible to the version of Hazelcast shipped with the old version of OX App Suite. It will be stated in the release notes if this is the case for a given release. If so, then some additional steps are required during a rolling update to ensure session handling / invalidating during update tasks works properly. See below.

== HOWTO / step-by-step instructions ==

* Take backups of as much as possible (databases, OX config files, etc).
* Announce the maintenance to the users. The communication depends on which approach you chose: the full downtime approach will come with a full downtime for all users, while the rolling upgrade approach will result in some users will have a short loss of service while their schema upgrades.

=== Full downtime approach ===

* Initiate maintenance: Block HTTP sessions to the service. Put a reasonable maintenance page in place, probably some HTTP error 503 with a reasonable Retry-After header.
* Shutdown the service on all middleware nodes. Upgrade the software on all middleware and frontend nodes using the disto's package manager. See [[AppSuite:UpdatingOXPackages]] for details on how to do that. Don't forget the <code>touch-appsuite</code> step if required ("If you update only UI plugins without simultaneously upgrading the core UI packages to a new version").
* Start the <code>open-xchange</code> service on one node
* Execute update tasks from that node. See [[UpdateTasks]] for an explanation how to do that, in particular the [[UpdateTasks#How_to_see_all_schemas.3F|section]] about limited parallel execution.
* Start the <code>open-xchange</code> services on the middleware nodes.
* Perform some crosschecks like
** all middleware nodes joined the Hazelcast cluster
** all OSGI bundles (which are expected to be running) are running
** WebUI login is possible
** Some central functionality tests like sending mails, accessing drive, etc
* Restore service: allow HTTP sessions, remove the maintenance page.

=== Rolling Upgrade without breaking Hazelcast upgrade ===

Remember: as stated above, this is viable only if the release notes for the new version do not state that there are breaking Hazelcast changes. For example, with v7.8.4 there were breaking Hazelcast changes and in the Release Notes it was stated as follows.

https://software.open-xchange.com/products/appsuite/doc/Release_Notes_for_Release_7.8.4_2017-05-23.pdf
<blockquote>
Important - Please Note

There is a major Hazelcast library update to OX App Suite v7.8.4. This means that when updating from an earlier backend version, due to the upgraded library, it is not possible to form a cluster of nodes that run previous version of Hazelcast (i.e. exiting volatile data in the cluster will be lost during the update). A consistent Hazelcast cluster is needed for cluster-wide cache invalidation. To circumvent problems with database update tasks that need to perform cache invalidation, please follow the steps described here: http://oxpedia.org/wiki/index.php?title=AppSuite:Running_a_cluster#Upgrades_of_the_Hazelcast_library. Please also note that session migration is not possible between versions. This usually affects all user sessions that are stored in a distributed map, and will require the users to re-login after the update. Running incompatible versions of Hazelcast within a cluster will result in logentries showing the conflicting node and version information.
</blockquote>

If you find you are upgrading to a version with breaking Hazelcast changes, please consult the next section [[#Rolling_Upgrade_with_breaking_Hazelcast_upgrade]].

Pre-update:

* Take one middleware node (the upgrade node) out of the HTTP traffic by adjusting the apache mod_proxy tables. We propose a combination of the balancer_manager to do this during runtime without restart, but also update the config files to prevent service restarts of apache to accidentally route sessions to the upgrade node.
* Make sure there are no user sessions left on the upgrade node, and that no new sessions will be routed to that node
* Update packages on the upgrade node and restart the middleware service there. See [[AppSuite:UpdatingOXPackages]] for details on how to do that.
* Execute update tasks from that node. See [[UpdateTasks]] for an explanation how to do that. You might want to keep the load low on the DBs, to affect production operations as low as possible, and because with this decoupled update tasks approach there is no immediate time pressure. If you want to follow the [[UpdateTasks#How_to_see_all_schemas.3F|limited parallel]] approach, use a small, mild parallelity factor (e.g. 2 or maybe 4 if you know this by far does not saturate your DB platform).

Real Update:

* For one middleware cluster node after each nother:
** Update packages on that middleware node and restart the middleware service there. See [[AppSuite:UpdatingOXPackages]] for details on how to do that.
** Verify the node starts its bundles, joins the Hazelcast cluster, log files are clean, the node handles sessions
* For one frontend node after each other (if you've got separate frontend nodes):
** Update packages on that frontend node. See [[AppSuite:UpdatingOXPackages]] for details on how to do that.
* Finally, execute <code>touch-appsuite</code> with a <code>--timestamp</code> argument as described on the page [[AppSuite:UpdatingOXPackages]]
* Perform final crosschecks like
** all middleware nodes joined the Hazelcast cluster
** all OSGI bundles (which are expected to be running) are running
** WebUI login is possible
** Some central functionality tests like sending mails, accessing drive, etc

=== Rolling Upgrade with breaking Hazelcast upgrade ===

Cf [[#Upgrades_of_the_Hazelcast_library]] below.

In principle the steps given in the previous section apply. However the upgrade needs to get the special Hazelcast Upgrade Package installed (e.g. one from <code>open-xchange-cluster-upgrade-from-76x</code>, <code>open-xchange-cluster-upgrade-from-780-782</code>, <code>open-xchange-cluster-upgrade-from-783</code>, <code>open-xchange-cluster-upgrade-from-784</code>, ...) during execution of the update tasks.

So the pre-update steps look like:

* Take one middleware node (the upgrade node) out of the HTTP traffic by adjusting the apache mod_proxy tables. We propose a combination of the balancer_manager to do this during runtime without restart, but also update the config files to prevent service restarts of apache to accidentally route sessions to the upgrade node.
* Make sure there are no user sessions left on the upgrade node, and that no new sessions will be routed to that node
* Update packages on the upgrade node and restart the middleware service there. See [[AppSuite:UpdatingOXPackages]] for details on how to do that.
* Install the special Hazelcast Upgrade Package on the upgrade node (e.g. one from <code>open-xchange-cluster-upgrade-from-76x</code>, <code>open-xchange-cluster-upgrade-from-780-782</code>, <code>open-xchange-cluster-upgrade-from-783</code>, <code>open-xchange-cluster-upgrade-from-784</code>, ...). Restart the service again.
* Execute update tasks from that node. See [[UpdateTasks]] for an explanation how to do that. You might want to keep the load low on the DBs, to affect production operations as low as possible, and because with this decoupled update tasks approach there is no immediate time pressure. If you want to follow the [[UpdateTasks#How_to_see_all_schemas.3F|limited parallel]] approach, use a small, mild parallelity factor (e.g. 2 or maybe 4 if you know this by far does not saturate your DB platform).

Note: don't worry if you don't see the upgrade node joining the legacy cluster: the upgrade node will not join the legacy cluster / not be visisble there since the upgrade node will be a so-called "native client" to the legacy cluster, and it will be created on the fly (and subsequently disposed again) for propagating an event. So also on <code>netstat</code> level the upgrade node will not have visible connections to the legacy cluster (unless for the very short timeframe when an actual even is sent). You can verify the functionality of that package by log lines like

<nowiki> Successfully initialzed Hazelcast client: <client-id>
Successfully got reference to cache event topic: cacheEvents-3
Publishing legacy cache event: <cache-event>

Successfully published legacy cache event, shutting down client after 546ms...</nowiki>

For the overly prudent it might be an idea to prepare a special test context with a test user living in its dedicated (test) schema, so you can test the functionality of this mechanis during upgrade first.

After the DB update tasks you can remove the special upgrade package again from the upgrade node.

The "Real Upgrade" procedure then looks like [[#Rolling_Upgrade_without_breaking_Hazelcast_upgrade|above]].

== Reference Documentation ==

=== Limitations ===

While in most cases a seamless, rolling upgrade of all nodes in the cluster is possible, there may be situations where nodes running a newer version of the Open-Xchange Server are not able to communicate with older nodes in the cluster, i.e. can't access distributed data or consume incompatible event notifications - especially, when the underlying Hazelcast library is part of the update, which does not support this scenario at the moment. In such cases, the release notes will contain corresponding information, so please have a look there before applying an update.

Additionally, there may always be some kind of race conditions during an update, i.e. client requests that can't be completed successfully or internal events not being deliverd to all nodes in the cluster. That's why the following information should only serve as a best-practices guide to minimize the impact of upgrades to the user experience.

=== Upgrading a single Node ===

Upgrading all nodes in the cluster should usually be done sequentially, i.o.w. one node after the other. This means that during the upgrade of one node, the node is temporarily disconnected from the other nodes in the cluster, and will join the cluster again after the update is completed. From the backend perspective, this is as easy as stopping the open-xchange service. other nodes in the cluster will recognize the disconnected node and start to repartition the shared cluster data automatically. But wait a minute - doing so would potentially lead to the webserver not registering the node being stopped immediately, resulting in temporary errors for currently logged in users until they are routed to another machine in the cluster. That's why it's good practice to tell the webserver's load balancer that the node should no longer fulfill incoming requests. The Apache Balancer Manager is an excellent tool for this ([http://httpd.apache.org/docs/2.2/mod/mod_status.html module ''mod_status'']). Look at the screen shot. Every node can be put into a disabled mode. Further requests will the redirected to other nodes in the cluster:

[[Image:balancer_manager.jpg]]

Afterwards, the open-xchange service on the disabled node can be stopped by executing:

$ /etc/init.d/open-xchange stop

or

$ service open-xchange stop

Now, the node is effectively in maintenance mode and any updates can take place. One could now verify the changed cluster infrastructure by accessing the Hazelcast MBeans either via JMX or the ''showruntimestats -c'' commandline tool (see above for details). There, the shut down node should no longer appear in the 'Member' section (com.hazelcast:type=Member).

When all upgrades are processed, the node open-xchange service can be started again by executing:

$ /etc/init.d/open-xchange start

or

$ service open-xchange start

As stated above, depending on the chosen cluster discovery mechanism, it might take some time until the node joins the cluster again. When using static cluster discovery, it will join the existing cluster usually directly during serivce startup, i.o.w. before other depending OSGi services are started. Otherwise, there might also be situations where the node cannot join the cluster directly, for example when there were no mDNS advertisments for other nodes in the cluster received yet. Then, it can take some additional time until the node finally joins the cluster. During startup of the node, you can observe the JMX console or the output of ''showruntimestats -c'' (com.hazelcast:type=Member) of another node in the cluster to verify when the node has joined.

After the node has joined, distributed data is re-partioned automatically, and the node is ready to server incoming requests again - so now the node can finally be enabled again in the load balancer configuration of the webserver. Afterwards, the next node in the cluster can be upgraded using the same procedure, until all nodes were processed.

=== Upgrades of the Hazelcast library ===

In case an upgrade includes a major update of the Hazelcast library, a newly upgraded node will usually not be able to connect to the nodes running the previous version. In this case, volatile cluster data is lost after all nodes in the cluster have been updated, including sessions held in the distributed session storage. As outlined above, the release notes will contain a corresponding warning in such cases.

Besides upgraded nodes not being able to access distributed data of the legacy cluster, this also affects new data not being available in the legacy cluster, which may cause troubles if the updated backend version needs to perform database update tasks. Database update tasks usually operate in a "blocking" way and all contexts associated with the schema being upgraded are disabled temporarily. Since context data itself is being held in caches on potentially each node in the cluster, the affected cache entries are invalidated during the database update. And, since cluster-wide cache invalidations again utilize Hazelcast functionality ([[#Remote Cache Invalidation]]), such invalidations normally won't be propagated to nodes running a previous version of the Hazelcast library.

To work around this specific scenario where an incompatible upgrade of the Hazelcast library needs to be performed along with blocking database update tasks, starting with v7.8.0, a supplementary package is available that explicitly enables the context cache invalidation of nodes running the previous Hazelcast library. This package follows the naming scheme ''open-xchange-cluster-upgrade-from-XXX'' (where XXX representing the version of the legacy version of the Open-Xchange server), and is available in the repositories for the updated server packages. This package should only be installed on the first node of the cluster that is going to be upgraded to the new version, and can be deactivated once the database upgrade tasks were executed successfully.

Once installed, a legacy cluster is discovered based on the available information in the ''hazelcast.properties'' configuration file in case cluster discovery is set to ''static''. If ''multicast'' is used, there's an alternative option to configure at least one of the addresses of the legacy cluster via ''com.openexchange.hazelcast.network.client.nodes''.

As an example, along with the server v7.8.0, a new package named ''open-xchange-cluster-upgrade-from-76x'' can be installed that aids in invalidating cluster server nodes running v7.6.x (which includes the Hazelcast library in version 3.2.4). Using this package, the recommended steps to update an OX cluster from version 7.6.x to version 7.8.0 would be:
# Pick a node from your cluster that you want to use for executing the database update tasks shipped with the new release
# Disable this node for incoming HTTP requests in your webserver configuration as described at [[#Upgrading a single Node]]
# Update the OX packages on this node, additionally install the package ''open-xchange-cluster-upgrade-from-76x''
# Restart the open-xchange services on this node
# Trigger the update task executions using the ''runUpdate'' commandline utitlty as described at [[UpdateTasks]]
# Once they are finished, uninstall the package ''open-xchange-cluster-upgrade-from-76x'' again
# Restart the open-xchange services on this node
# Re-enable the node for incoming HTTP requests in your webserver configuration as described at [[#Upgrading a single Node]]
# Upgrade all other nodes in the cluster as described at [[#Upgrading a single Node]]

Same steps apply to upgrading from v7.8.0 through v7.8.2 (incl.) to v7.8.3 using the package named ''open-xchange-cluster-upgrade-from-780-782'', since v7.8.0 through v7.8.2 (incl.) utilize Hazelcast v3.5.x, while v7.8.3 uses Hazelcast v3.6.4

Same steps apply to upgrading from v7.8.3 to v7.8.4 using the package named ''open-xchange-cluster-upgrade-from-783'', since v7.8.3 utilizes Hazelcast v3.7.1

Same steps apply to upgrading from v7.8.4 to v7.10.0 using the package named ''open-xchange-cluster-upgrade-from-784'', since v7.8.4 utilizes Hazelcast v3.8.1

'''Operations Note:''' The upgraded node will be added as so-called [http://docs.hazelcast.org/docs/2.3/manual/html/ch15.html Native Client] to the legacy Hazelcast Cluster.

<blockquote>
Native Client enables you to do all Hazelcast operations without being a member of the cluster.
[...]

However Native client is not member and relies on one of the cluster members.
</blockquote>

This means, the upgraded node will not be visible in the members list of the legacy Hazelcast cluster (<code>showruntimestats -c</code>). Furthermore, the native client will created and destructed on single context events, with the effect that connections will only be visible in the very moment of such an event. This means effectively that verification of the invalidation mechanis is only possible by actually executing the <code>runupdate</code> CLT. This should produce log lines like

Successfully initialzed Hazelcast client: <client-id>
Successfully got reference to cache event topic: cacheEvents-3
Publishing legacy cache event: <cache-event>
Successfully published legacy cache event, shutting down client after 546ms...

Most importantly, you should be able to observe correct functionality (users of affected contexts being logged out). It may be handy to prepare a dedicated schema with just test contexts inside. (How to create this is out of scope here, but hint: use <code>createschema</code> and <code>createcontext --schema-name</code>.)

=== Other Considerations ===

* It's always recommended to only upgrade one node after the other, always ensuring that the cluster has formed correctly between each shutdown/startup of a node.
* Do not stop a node while running the runUpdate script or the associated update task.
* During the time of such a rolling upgrade of all nodes, we have effectively heterogeneous software versions in the cluster, which potentially might lead to temporary inconsistencies. Therefore, all nodes in the cluster should be updated in one cycle (but still one after the other).
* Following the above guideline, it's also possible to add or remove nodes dynamically to the cluster, not only when disconnecting a node temporary for updates.
* In case of trouble, i.e. a node refuses to join the cluster again after restart, consult the logfiles first for any hints about what is causing the problem - both on the disconnected node, and also on other nodes in the network
* If there are general incompatibilities between two revisions of the Open-Xchange Server that prevent an operation in a cluster (release notes), it's recommended to choose another name for the cluster in ''cluster.properties'' for the nodes with the new version. This will temporary lead to two separate clusters during the rolling upgrade, and finally the old cluster being shut down completely after the last node was updated to the new version. While distributed data can't be migrated from one server version to another in this scenario due to incompatibilities, the uptime of the system itself is not affected, since the nodes in the new cluster are able to serve new incoming requests directly.
* When updating only UI plugins without also updating to a new version of the core UI, you also need to perform the additional step from [[AppSuite:UpdatingOXPackages#Updating_UI_plugins|Updating UI plugins]].

[[Category: AppSuite]] [[Category: Administration]] [[Category: Cluster]]

AppSuite:Running a cluster

2017-12-13T08:08:46Z

Dominik.epple: /* Rolling Upgrade without breaking Hazelcast upgrade */

<div class="title">Running a cluster</div>

__TOC__

= Concepts =

For inter-OX-communication over the network, multiple Open-Xchange servers can form a cluster. This brings different advantages regarding distribution and caching of volatile data, load balancing, scalability, fail-safety and robustness. Additionally, it provides the infrastructure for upcoming features of the Open-Xchange server.
The clustering capabilities of the Open-Xchange server are mainly built up on [http://hazelcast.com Hazelcast], an open source clustering and highly scalable data distribution platform for Java. The following article provides an overview about the current featureset and configuration options.

= Requirements =

== Synchronized system clock times ==
It is crucial that all involved members in a cluster do have their system clock times in sync with each other; e.g. by using an NTP service.

== HTTP routing ==
An OX cluster is always part of a larger picture. Usually there is front level loadbalancer as central HTTPS entry point to the platform. This loadbalancer optionally performs HTTPS termination and forwards HTTP(S) requests to webservers (the usual and only supported choice as of now is Apache). These webservers are performing HTTPS termination (if this is not happening on the loadbalancer) and serve static content, and (which is what is relevant for our discussion here) they forward dynamic requests to the OX backends.

A central requirement for the interaction of these components (loadbalancer, webservers, OX nodes) is that we have session stability based on the JSESSIONID cookie / jsessionid path component suffix. This means that our application sets a cookie named JSESSIONID which has a value like <large decimal number>.<route identifier>, e.g. "5661584529655240315.OX1". The route identifier here ("OX1" in this example) is taken by the OX node from a configuration setting from a config file and is specific to one OX node. HTTP routing must happen such that HTTP requests with a cookie with such a suffix always end up the corresponding OX node. There are furthermore specific cirumstances when passing this information via cookie is not possible. Then the JSESSIONID is transferred in a path component as "jsessionid=..." in the HTTP request. The routing mechanism needs to take that into account also.

There are mainly two options to implement this. If the Apache processes are running co-located on the same machines running the OX groupware processes, it is often desired to have the front level loadbalancer perform HTTP routing to the correct machines. If dedicated Apache nodes are employed, is is usually sufficient to have the front-level loadbalancer do HTTP routing to the Apache nodes in a round-robin fashion and perform routing to the correct OX nodes in the Apache nodes.

We provide sample configuration files to configure Apache (with mod_proxy_http) to perform HTTP routing correctly in our guides on OXpedia, e.g. [[AppSuite:Main_Page_AppSuite#quickinstall]]. Central elements are the directives "ProxySet stickysession=JSESSIONID|jsessionid scolonpathdelim=On" in conjunction with the "route=OX1" parameters to the BalancerMember lines in the Proxy definition. This is valid for Apache 2.2 as of Sep-2014.

How to configure a front level loadbalancer to perform HTTP equivalent HTTP routing is dependent on the specific loadbalancer implementation. If Apache is used as front level loadbalancer, the same configuration as discussed in the previous section can be employed. As of time of writing this text (Sep 2014), the alternative choices are thin. F5 BigIP is reported to be able to implement "jsessionid based persistence using iRules". nginx has the functionality in their commercial "nginx plus" product. (Both of these options have not been tested by OX.) Other loadbalancers with this functionality are not known to us.

If the front level loadbalancer is not capable of performing correct HTTP routing, is is required to configure correct HTTP routing on Apache level, even if Apache runs co-located on the OX nodes and thus cross-routing happens.

There are several reasons why we require session stability in exactly this way. We require session stabilty for horizontal scale-out; while we support transparent resuming / migration of user sessions in the OX cluster without need for users to re-authenticate, sessions wandering around randomly will consume a fixed amount resources corresponding to a running session on each OX node in the cluster, while a session sticky to one OX node will consume this fixed amount of resources only on one OX node. Furthermore there are mechanisms in OX like TokenLogin which work only of all requests beloning to one sequence get routed to the same OX node even if they stem from different machines with different IPs. Only the JSESSIONID (which in this case is transferred as jsessionid path component, as cookies do not work during a 302 redirect, which is part of this sequence) carries the required information where the request must be routed to.

Usual "routing based on cookie hash" is not sufficient here since it disregards the information which machine originally issued the cookie. It only ensures that the session will be sticky to any target, which statistically will not be the same machine that issued the cookie. OX will then set a new JSESSIONID cookie, assuming the session had been migrated. The loadbalancer will then route the session to a different target, as the hash of the cookie will differ. This procedure then happens iteratively until by chance the routing based on cookie hash will route the session to the correct target. By then, a lot of resources will have been wasted, by creating full (short-term) sessions on all OX nodes. Furthermore, processes like TokenLogin will not work this way.

= Configuration =

All settings regarding cluster setup are located in the configuration file ''hazelcast.properties''. The former used additional files ''cluster.properties'', ''mdns.properties'' and ''static-cluster-discovery.properties'' are no longer needed. The following gives an overview about the most important settings - please refer to the inline documentation of the configuration file for more advanced options.

Note: The configuration guide targets v7.4.0 of the OX server (and above). For older versions, please consult the history of this page. A full list of Hazelcast-related properties is available at https://documentation.open-xchange.com/components/middleware/config/7.8.4/#mode=features&feature=Hazelcast .

== General ==

To restrict access to the cluster and to separate the cluster from others in the local network, a name and password needs to be defined. Only backend nodes having the same values for those properties are able to join and form a cluster.

# Configures the name of the cluster. Only nodes using the same group name
# will join each other and form the cluster. Required if
# "com.openexchange.hazelcast.network.join" is not "empty" (see below).
com.openexchange.hazelcast.group.name=

# The password used when joining the cluster. Defaults to "wtV6$VQk8#+3ds!a".
# Please change this value, and ensure it's equal on all nodes in the cluster.
com.openexchange.hazelcast.group.password=wtV6$VQk8#+3ds!a

== Network ==

It's required to define the network interface that is used for cluster communication via ''com.openexchange.hazelcast.network.interfaces''. By default, the interface is restricted to the local loopback address only. To allow the same configuration amongst all nodes in the cluster, it's recommended to define the value using a wildcard matching the IP addresses of all nodes participating in the cluster, e.g. ''192.168.0.*''

# Comma-separated list of interface addresses hazelcast should use. Wildcards
# (*) and ranges (-) can be used. Leave blank to listen on all interfaces
# Especially in server environments with multiple network interfaces, it's
# recommended to specify the IP-address of the network interface to bind to
# explicitly. Defaults to "127.0.0.1" (local loopback only), needs to be
# adjusted when building a cluster of multiple backend nodes.
com.openexchange.hazelcast.network.interfaces=127.0.0.1

To form a cluster of multiple OX server nodes, different discovery mechanisms can be used. The discovery mechanism is specified via the property ''com.openexchange.hazelcast.network.join'':

# Specifies which mechanism is used to discover other backend nodes in the
# cluster. Possible values are "empty" (no discovery for single-node setups),
# "static" (fixed set of cluster member nodes) or "multicast" (automatic
# discovery of other nodes via multicast). Defaults to "empty". Depending on
# the specified value, further configuration might be needed, see "Networking"
# section below.
com.openexchange.hazelcast.network.join=empty

Generally, it's advised to use the same network join mechanism for all nodes in the cluster, and, in most cases, it's strongly recommended to use a ''static'' network join configuration. This will allow the nodes to join the cluster directly upon startup. With a ''multicast'' based setup, nodes will merge to an existing cluster possibly at some later time, thus not being able to access the distributed data until they've joined.

Depending on the network join setting, further configuration may be necessary, as decribed in the following paragraphs.

=== empty ===

When using the default value ''empty'', no other nodes are discovered in the cluster. This value is suitable for single-node installations. Note that other nodes that are configured to use other network join mechanisms may be still able to still to connect to this node, e.g. using a ''static'' network join, having the IP address of this host in the list of potential cluster members (see below).

=== static ===

The most common setting for ''com.openexchange.hazelcast.network.join'' is ''static''. A static cluster discovery uses a fixed list of IP addresses of the nodes in the cluster. During startup and after a specific interval, the underlying Hazelcast library probes for not yet joined nodes from this list and adds them to the cluster automatically. The address list is configured via ''com.openexchange.hazelcast.network.join.static.nodes'':

# Configures a comma-separated list of IP addresses / hostnames of possible
# nodes in the cluster, e.g. "10.20.30.12, 10.20.30.13:5701, 192.178.168.110".
# Only used if "com.openexchange.hazelcast.network.join" is set to "static".
# It doesn't hurt if the address of the local host appears in the list, so
# that it's still possible to use the same list throughout all nodes in the
# cluster.
com.openexchange.hazelcast.network.join.static.nodes=

For a fixed set of backend nodes, it's recommended to simply include the IP addresses of all nodes in the list, and use the same configuration for each node. However, it's only required to add the address of at least one other node in the cluster to allow the node to join the cluster. Also, when adding a new node to the cluster and this list is extended accordingly, existing nodes don't need to be shut down to recognize the new node, as long as the new node's address list contains at least one of the already running nodes.

=== multicast ===

For highly dynamic setups where nodes are added and removed from the cluster quite often and/or the host's IP addresses are not fixed, it's also possible to configure the network join via multicast. During startup and after a specific interval, the backend nodes initiate the multicast join process automatically, and discovered nodes form or join the cluster afterwards. The multicast group and port can be configured as follows:

# Configures the multicast address used to discover other nodes in the cluster
# dynamically. Only used if "com.openexchange.hazelcast.network.join" is set
# to "multicast". If the nodes reside in different subnets, please ensure that
# multicast is enabled between the subnets. Defaults to "224.2.2.3".
com.openexchange.hazelcast.network.join.multicast.group=224.2.2.3

# Configures the multicast port used to discover other nodes in the cluster
# dynamically. Only used if "com.openexchange.hazelcast.network.join" is set
# to "multicast". Defaults to "54327".
com.openexchange.hazelcast.network.join.multicast.port=54327

== Example ==

The following example shows how a simple cluster named ''MyCluster'' consisting of 4 backend nodes can be configured using ''static'' cluster discovery. The node's IP addresses are 10.0.0.15, 10.0.0.16, 10.0.0.17 and 10.0.0.18. Note that the same ''hazelcast.properties'' is used by all nodes.

com.openexchange.hazelcast.group.name=MyCluster
com.openexchange.hazelcast.group.password=secret
com.openexchange.hazelcast.network.join=static
com.openexchange.hazelcast.network.join.static.nodes=10.0.0.15,10.0.0.16,10.0.0.17,10.0.0.18
com.openexchange.hazelcast.network.interfaces=10.0.0.*

== Advanced Configuration ==

=== Lite Members (available since v7.8.4) ===

Lite members in a Hazelcast cluster are members that do not hold any data partitions, i.e. all read- and write operations to distributed maps are delegated to non-lite ("full") members. Apart from not having data partitions, lite members participate in the same way as other members: they can register listeners for distributed topics (e.g. cache invalidation events) or can be addressed for task execution (e.g. during realtime communication).

Similar to using a custom partitioning scheme, separating the nodes of a large cluster into few "full" members and many "lite" members helps to minimize the impact of JVM activities from a single node (mainly the garbage collector) on the whole cluster communication. Additionally, when starting or stopping lite members, no repartitioning of the distributed cluster data needs to be performed, which significantly decreases the node's startup- and shutdown time and reduces the necessary network communication to a minimum.

In medium or larger sized clusters, it is sufficient to have roughly 10 to 20 percent of the nodes configured as "full" members, while all other ones can be started as "lite" member nodes. Additionally, please note that the configured backup count in the map configurations should always be smaller than the total number of "full" members, otherwise, there may be problems if one of those data nodes is shut down temporarily for maintenance. So, the minimum number of "full" members is implicitly bound to the sum of a map's ''backupCount'' and ''asyncBackupCount'' properties, plus ''1'' for the original data partition.

The configured "full" members should preferrably not be used to serve client requests (by not adding them as endpoint in the loadbalancer), to ensure they are always responsive. Also, shutdown and startups of those "full" members should be reduced to a minimum to avoid repartitioning operations.

More general information regarding lite members is available at http://docs.hazelcast.org/docs/latest/manual/html-single/index.html#enabling-lite-members .

To configure a node as "lite" member, the following configuration should be applied in the node's ''hazelcast.properties'' file:

com.openexchange.hazelcast.liteMember=true

It's also recommended to use a "static" cluster discovery for the network join, and list all "full" member nodes here, so that join requests are handled by those nodes, too (and not the other nodes that are potentially prone to garbage collection delays.

=== Custom Partitioning ===

Note: Starting with v7.8.4, "Lite Members" should be used in favor of applying a custom partitioning scheme.

While originally being designed to separate the nodes holding distributed data into different risk groups for increased fail safety, a custom partitioning strategy may also be used to distinguish between nodes holding distributed data from those who should not.

This approach of custom partitioning may be used in a OX cluster, where usually different backend nodes serve different purposes. A common scenario is that there are nodes handling requests from the web interfaces, and others being responsible for USM/EAS traffic. Due to their nature of processing large chunks of synchronization data in memory, the USM/EAS nodes may encounter small delays when the Java garbage collector kicks in and suspends the Java Virtual Machine. Since those delays may also have an influence on hazelcast-based communication in the cluster, the idea is to instruct hazelcast to not store distributed data on that nodes. This is where a custom partitioning scheme comes into play.

To setup a custom partitioning scheme in the cluster, an additional ''hazelcast.xml'' configuration file is used, which should be placed into the ''hazelcast'' subdirectory of the OX configuration folder, usually at ''/opt/openexchange/etc/hazelcast''. Please note that it's vital that each node in the cluster is configured equally here, so the same ''hazelcast.xml'' file should be copied to each server. The configuration read from there is used as basis for all further settings that are taken from the ordinary ''hazelcast.properties'' config file.

To setup a custom partitioning scheme, the partition groups must be defined in the ''hazelcast.xml'' file. See the following file for an example configuration, where the three nodes ''10.10.10.60'', ''10.10.10.61'' and ''10.10.10.62'' are defined to form an own partitioning group each. Doing so, all distributed data will be stored at one of those nodes physically, while the corresponding backup data (if configured) at one of the other two nodes. All other nodes in the cluster will not be used to store distributed data, but will still be "full" hazelcast members, which is necessary for other cluster-wide operations the OX backends use.

Please note that the configured backup count in the map configurations should be smaller than the number of nodes here, otherwise, there may be problems if one of those data nodes is shut down temporarily for maintenance. So, the minimum number of nodes to define in the partition group sections is implicitly bound to the sum of a map's ''backupCount'' and ''asyncBackupCount'' properties, plus ''1'' for the original data partition.

<?xml version="1.0" encoding="UTF-8"?>


<hazelcast xsi:schemaLocation="http://www.hazelcast.com/schema/config hazelcast-config-3.1.xsd"
xmlns="http://www.hazelcast.com/schema/config"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
<partition-group enabled="true" group-type="CUSTOM">
<member-group>
<interface>10.10.10.60</interface>
</member-group>
<member-group>
<interface>10.10.10.61</interface>
</member-group>
<member-group>
<interface>10.10.10.62</interface>
</member-group>
</partition-group>
</hazelcast>

More general information regarding custom partioning is available at http://hazelcast.org/docs/latest/manual/html/partitiongroupconfig.html .

It's also recommended to use a "static" cluster discovery for the network join, and list same the nodes that are also configured in the parition groups here, so that join requests are handled by those nodes, too (and not the other nodes that are potentially prone to garbage collection delays.

After configuring a custom partitioning scheme, the data distribution may be verified, e.g. by inspecting the MBeans of the distributed maps via JMX.

= Features =

The following list gives an overview about different features that were implemented using the new cluster capabilities.

== Distributed Session Storage ==

Previously, when an Open-Xchange server was shutdown for maintenance, all user sessions that were bound to that machine were lost, i.e. the users needed to login again. With the distributed session storage, all sessions are backed by a distributed map in the cluster, so that they are no longer bound to a specific node in the cluster. When a node is shut down, the session data is still available in the cluster and can be accessed from the remaining nodes. The load-balancing techniques of the webserver then seamlessly routes the user session to another node, with no ''session expired'' errors. The distributed session storage comes with the package ''open-xchange-sessionstorage-hazelcast''. It's recommended to install this optional package in all clustered environments with multiple groupware server nodes.

'''Notes:'''
* While there's some kind of built-in session distribution among the nodes in the cluster, this should not be seen as a replacement for session-stickiness between the loadbalancer and groupware nodes, i.e. one should still configure the webserver to use sticky sessions for performance reasons.
* The distributed session storage is still an in-memory storage. While the session data is distributed and backed up on multiple nodes in the cluster, shutting down multiple or all nodes at the same time will lead to loss of the the distributed data. To avoid such data loss when shutting down a node, please follow the guidelines at [[ Updating_a_Cluster ]].

Depending on the cluster infrastructure, different backup-count configuration options might be set for the distributed session storage in the map configuration file ''sessions.properties'' in the ''hazelcast'' subdirectory:

com.openexchange.hazelcast.configuration.map.backupCount=1

The ''backupcount'' property configures the number of nodes with synchronized backups. Synchronized backups block operations until backups are successfully copied and acknowledgements are received. If 1 is set as the backup-count for example, then all entries of the map will be copied to another JVM for fail-safety. 0 means no backup. Any integer between 0 and 6. Default is 1, setting bigger than 6 has no effect.

com.openexchange.hazelcast.configuration.map.asyncBackupCount=0

The ''asyncbackup'' property configures the number of nodes with async backups. Async backups do not block operations and do not require acknowledgements. 0 means no backup. Any integer between 0 and 6. Default is 0, setting bigger than 6 has no effect.

Since session data is backed up by default continuously by multiple nodes in the cluster, the steps described in [[ Session_Migration ]] to trigger session migration to other nodes explicitly is obsolete and no longer needed with the distributed session storage.

Normally, sessions in the distributed storages are not evicted automatically, but are only removed when they're also removed from the session handler, either due to a logout operation or when exceeding the long-term session lifetime as configured by ''com.openexchange.sessiond.sessionLongLifeTime'' in ''sessiond.properties''. Under certain circumstances, i.e. the session is no longer accessed by the client and the OX node hosting the session in it's long-life container being shutdown, the remove operation from the distributed storage might not be triggered. Therefore, additionaly a maximum idle time of map-entries can be configured for the distributed sessions map via

com.openexchange.hazelcast.configuration.map.maxIdleSeconds=640000

To avoid unnecessary eviction, the value should be higher than the configured ''com.openexchange.sessiond.sessionLongLifeTime'' in ''sessiond.properties''.

== Remote Cache Invalidation ==

For faster access, groupware data is held in different caches by the server. Formerly, the caches utilized the TCP Lateral Auxiliary Cache plug in (LTCP) for the underlying JCS caches to broadcast updates and removals to caches on other OX nodes in the cluster. This could potentially lead to problems when remote invalidation was not working reliably due to network discovery problems. As an alternative, remote cache invalidation can also be performed using reliable publish/subscribe events built up on Hazelcast topics. This can be configured in the ''cache.properties'' configuration file, where the 'eventInvalidation' property can either be set to 'false' for the legacy behavior or 'true' for the new mechanism:

com.openexchange.caching.jcs.eventInvalidation=true

All nodes participating in the cluster should be configured equally.

Internally, if ''com.openexchange.caching.jcs.eventInvalidation'' is set to ''true'', LTCP is disabled in JCS caches. Instead, an internal mechanism based on distributed Hazelcast event topics is used to invalidate data throughout all nodes in the cluster after local update- and remove-operations. Put-operations aren't propagated (and haven't been with LTCP either), since all data put into caches can be locally loaded/evaluated at each node from the persistent storage layer.

Using Hazelcast-based cache invalidation also makes further configuration of the JCS auxiliaries obsolete in the ''cache.ccf'' configuration file. In that case, all ''jcs.auxiliary.LTCP.*'' configuration settings are virtually ignored. However, it's still required to mark caches that require cluster-wide invalidation via ''jcs.region.<cache_name>=LTCP'', just as before. So basically, when using the new default setting ''com.openexchange.caching.jcs.eventInvalidation=true'', it's recommended to just use the stock ''cache.ccf'' file, since no further LTCP configuration is required.

= Adminstration / Troubleshooting =

== Hazelcast Configuration ==

The underlying Hazelcast library can be configured using the file ''hazelcast.properties''.

'''Important''':<br>
By default property ''com.openexchange.hazelcast.network.interfaces'' is set to ''127.0.0.1''; meaning Hazelcast listens only to loop-back device. To build a cluster among remote nodes the appropriate network interface needs to be configured there. Leaving that property empty lets Hazelcast listen to all available network interfaces.

The Hazelcast JMX MBean can be enabled or disabled with the property ''com.openexchange.hazelcast.jmx''. The properties ''com.openexchange.hazelcast.mergeFirstRunDelay'' and ''com.openexchange.hazelcast.mergeRunDelay'' control the run intervals of the so-called ''Split Brain Handler'' of Hazelcast that initiates the cluster join process when a new node is started. More details can be found at http://www.hazelcast.com/docs/2.5/manual/single_html/#NetworkPartitioning.

The port ranges used by Hazelcast for incoming and outgoing connections can be controlled via the configuration parameters ''com.openexchange.hazelcast.networkConfig.port'', ''com.openexchange.hazelcast.networkConfig.portAutoIncrement'' and ''com.openexchange.hazelcast.networkConfig.outboundPortDefinitions''.

== Commandline Tool ==

To print out statistics about the cluster and the distributed data, the ''showruntimestats'' commandline tool can be executed witht the ''clusterstats'' ('c') argument. This provides an overview about the runtime cluster configuration of the node, other members in the cluster and distributed data structures.

== JMX ==

In the Open-Xchange server Java process, the MBean ''com.hazelcast'' can be used to monitor and manage different aspects of the underlying Hazelcast cluster. The ''com.hazelcast'' MBean provides detailed information about the cluster configuration and distributed data structures.

== Hazelcast Errors ==

When experiencing hazelcast related errors in the logfiles, most likely different versions of the packages are installed, leading to different message formats that can't be understood by nodes using another version. Examples for such errors are exceptions in hazelcast components regarding (de)serialization or other message processing.
This may happen when performing a consecutive update of all nodes in the cluster, where temporarily nodes with a heterogeneous setup try to communicate with each other. If the errors don't disappear after all nodes in the cluster have been update to the same package versions, it might be necessary to shutdown the cluster completely, so that all distributed data is cleared.

== Cluster Discovery Errors ==

* If the started OX nodes don't form a cluster, please double-check your configuration in ''hazelcast.properties''
* It's important to have the same cluster name defined in ''hazelcast.properties'' throughout all nodes in the cluster
* Especially when using multicast cluster discovery, it might take some time until the cluster is formed
* When using ''static'' cluster discovery, at least one other node in the cluster has to be configured in ''com.openexchange.hazelcast.network.join.static.nodes'' to allow joining, however, it's recommended to list all nodes in the cluster here

== Disable Cluster Features ==

The Hazelcast based clustering features can be disabled with the following property changes:
* Disable cluster discovery by setting ''com.openexchange.hazelcast.network.join'' to ''empty'' in ''hazelcast.properties''
* Disable Hazelcast by setting ''com.openexchange.hazelcast.enabled'' to false in ''hazelcast.properties''
* Disable message based cache event invalidation by setting ''com.openexchange.caching.jcs.eventInvalidation'' to ''false'' in ''cache.properties''

== Update from 6.22.1 to version 6.22.2 and above ==

As hazelcast will be used by default for the distribution of sessions starting 6.22.2 you have to adjust hazelcast according to our old cache configuration. First of all it's important that you install the open-xchange-sessionstorage-hazelcast package. This package will add the binding between hazelcast and the internal session management. Next you have to set a cluster name to the cluster.properties file (see [[#Cluster Discovery Errors]]). Furthermore you will have to add one of the two discovery modes mentioned in [[#Cluster Discovery]].

= Updating a Cluster =

Running a cluster means built-in failover on the one hand, but might require some attention when it comes to the point of upgrading the services on all nodes in the cluster. This chapter gives an overview about general concepts and hints for silent updates of the cluster.

== The Big Picture ==

Updating an OX App Suite cluster is possible in several ways. The involved steps always include

* Update the software by updating the packages through the distro's repository / software update tool
* Update the database schemas (so-called update tasks)

There are some precautions required, though.

=== Update Tasks Management ===

It is a feature of the OX App Suite middleware to automatically start update tasks on a database schema when a user tries to login whose context lives on that schema. For installations beyond a certain size, if you just update the OX App Suite software without special handling of the update tasks, user logins will trigger an uncontrolled storm of update tasks on the databases, potentially leading to resource contention, unnecessary long update tasks runtimes, excessive load on the database server, maybe even service outages.

So one key element of every update strategy is to avoid user logins on nodes which have already been updated to the new software version, while the database schemas are still on the old version. There are two fundamentally different approaches to this goal: use either a full downtime, or use a rolling update strategy.

We describe the update strategy in more detail in the next section. Note that these are still high-level outlines of the actual procedure, which requires additional details with regards to Hazelcast, given further down below.

==== Full downtime approach ====

The full downtime approach is quite straightforward and involves

* shutdown of all OX middleware nodes
* update the software on all OX App Suite (middleware and frontend) nodes
* execute the update tasks in a controlled way from one OX node
* restore the service

This is the most general approach and always available, even if the rolling approach is not available due to Hazelcast constraints.

==== Rolling strategy ====

It is possible to execute the update tasks decoupled from the real update of the rest from the cluster, days or even weeks ahead of time, with the following approach:

* If the load situation allows for it, take one node out of the loadbalancer (we call it the upgrade node). Otherwise, add a dedicated upgrade node to your cluster, identically configured to the other middleware nodes.
* Make sure there are no user sessions left on the upgrade node, and that no new sessions will be routed to that node
* update the software on the upgrade node
* execute all update tasks from the update node.

In the last step, users from affected schemas will be logged out and denied service while the update tasks are running on their database schema. This is typically a short unavailability (some minutes) for a small part (1000...7000 depending on the installation) of the user base. This unavailability is of much lower impact than the unavailability of a full downtime, but you still might want to do this in the off-business hours.

This way you end up with the production cluster running on the old version of OX App Suite, with the database already being upgraded to the next version. This is explicitly a valid and supported configuration. This approach offers the advantage that update tasks can be executed in advance, instead of doing them while the whole system is in a full maintenance downtime. Since update tasks can take some time, this is a considerable advantage.

For the actual upgrade of the production cluster, the remaining steps are:

* Upgrade and restart the OX App Suite software on one middleware node after another, one by one
* Upgrade the software on the OX App Suite frontend nodes (if these are separate nodes from the middleware nodes)

Hazelcast will ensure that sessions from nodes which you restart are taken over by other nodes in the cluster, so ideally this step works without losing user sessions.

For the rolling strategy to work as described, it is required that the old and new version of OX App Suite use compatible versions of the Hazelcast library. This is the case for most upgrades. However some upgrades must handle the situation that the new version of OX App Suite ships with a new version of Hazelcast incompatible to the version of Hazelcast shipped with the old version of OX App Suite. It will be stated in the release notes if this is the case for a given release. If so, then some additional steps are required during a rolling update to ensure session handling / invalidating during update tasks works properly. See below.

== HOWTO / step-by-step instructions ==

* Take backups of as much as possible (databases, OX config files, etc).
* Announce the maintenance to the users. The communication depends on which approach you chose: the full downtime approach will come with a full downtime for all users, while the rolling upgrade approach will result in some users will have a short loss of service while their schema upgrades.

=== Full downtime approach ===

* Initiate maintenance: Block HTTP sessions to the service. Put a reasonable maintenance page in place, probably some HTTP error 503 with a reasonable Retry-After header.
* Shutdown the service on all middleware nodes. Upgrade the software on all middleware and frontend nodes using the disto's package manager. See [[AppSuite:UpdatingOXPackages]] for details on how to do that. Don't forget the <code>touch-appsuite</code> step.
* Start the <code>open-xchange</code> service on one node
* Execute update tasks from that node. See [[UpdateTasks]] for an explanation how to do that, in particular the [[UpdateTasks#How_to_see_all_schemas.3F|section]] about limited parallel execution.
* Start the <code>open-xchange</code> services on the middleware nodes.
* Perform some crosschecks like
** all middleware nodes joined the Hazelcast cluster
** all OSGI bundles (which are expected to be running) are running
** WebUI login is possible
** Some central functionality tests like sending mails, accessing drive, etc
* Restore service: allow HTTP sessions, remove the maintenance page.

=== Rolling Upgrade without breaking Hazelcast upgrade ===

Remember: as stated above, this is viable only if the release notes for the new version do not state that there are breaking Hazelcast changes. For example, with v7.8.4 there were breaking Hazelcast changes and in the Release Notes it was stated as follows.

https://software.open-xchange.com/products/appsuite/doc/Release_Notes_for_Release_7.8.4_2017-05-23.pdf
<blockquote>
Important - Please Note

There is a major Hazelcast library update to OX App Suite v7.8.4. This means that when updating from an earlier backend version, due to the upgraded library, it is not possible to form a cluster of nodes that run previous version of Hazelcast (i.e. exiting volatile data in the cluster will be lost during the update). A consistent Hazelcast cluster is needed for cluster-wide cache invalidation. To circumvent problems with database update tasks that need to perform cache invalidation, please follow the steps described here: http://oxpedia.org/wiki/index.php?title=AppSuite:Running_a_cluster#Upgrades_of_the_Hazelcast_library. Please also note that session migration is not possible between versions. This usually affects all user sessions that are stored in a distributed map, and will require the users to re-login after the update. Running incompatible versions of Hazelcast within a cluster will result in logentries showing the conflicting node and version information.
</blockquote>

If you find you are upgrading to a version with breaking Hazelcast changes, please consult the next section [[#Rolling_Upgrade_with_breaking_Hazelcast_upgrade]].

Pre-update:

* Take one middleware node (the upgrade node) out of the HTTP traffic by adjusting the apache mod_proxy tables. We propose a combination of the balancer_manager to do this during runtime without restart, but also update the config files to prevent service restarts of apache to accidentally route sessions to the upgrade node.
* Make sure there are no user sessions left on the upgrade node, and that no new sessions will be routed to that node
* Update packages on the upgrade node and restart the middleware service there. See [[AppSuite:UpdatingOXPackages]] for details on how to do that.
* Execute update tasks from that node. See [[UpdateTasks]] for an explanation how to do that. You might want to keep the load low on the DBs, to affect production operations as low as possible, and because with this decoupled update tasks approach there is no immediate time pressure. If you want to follow the [[UpdateTasks#How_to_see_all_schemas.3F|limited parallel]] approach, use a small, mild parallelity factor (e.g. 2 or maybe 4 if you know this by far does not saturate your DB platform).

Real Update:

* For one middleware cluster node after each nother:
** Update packages on that middleware node and restart the middleware service there. See [[AppSuite:UpdatingOXPackages]] for details on how to do that.
** Verify the node starts its bundles, joins the Hazelcast cluster, log files are clean, the node handles sessions
* For one frontend node after each other (if you've got separate frontend nodes):
** Update packages on that frontend node. See [[AppSuite:UpdatingOXPackages]] for details on how to do that.
* Finally, execute <code>touch-appsuite</code> with a <code>--timestamp</code> argument as described on the page [[AppSuite:UpdatingOXPackages]]
* Perform final crosschecks like
** all middleware nodes joined the Hazelcast cluster
** all OSGI bundles (which are expected to be running) are running
** WebUI login is possible
** Some central functionality tests like sending mails, accessing drive, etc

=== Rolling Upgrade with breaking Hazelcast upgrade ===

Cf [[#Upgrades_of_the_Hazelcast_library]] below.

In principle the steps given in the previous section apply. However the upgrade needs to get the special Hazelcast Upgrade Package installed (e.g. one from <code>open-xchange-cluster-upgrade-from-76x</code>, <code>open-xchange-cluster-upgrade-from-780-782</code>, <code>open-xchange-cluster-upgrade-from-783</code>, <code>open-xchange-cluster-upgrade-from-784</code>, ...) during execution of the update tasks.

So the pre-update steps look like:

* Take one middleware node (the upgrade node) out of the HTTP traffic by adjusting the apache mod_proxy tables. We propose a combination of the balancer_manager to do this during runtime without restart, but also update the config files to prevent service restarts of apache to accidentally route sessions to the upgrade node.
* Make sure there are no user sessions left on the upgrade node, and that no new sessions will be routed to that node
* Update packages on the upgrade node and restart the middleware service there. See [[AppSuite:UpdatingOXPackages]] for details on how to do that.
* Install the special Hazelcast Upgrade Package on the upgrade node (e.g. one from <code>open-xchange-cluster-upgrade-from-76x</code>, <code>open-xchange-cluster-upgrade-from-780-782</code>, <code>open-xchange-cluster-upgrade-from-783</code>, <code>open-xchange-cluster-upgrade-from-784</code>, ...). Restart the service again.
* Execute update tasks from that node. See [[UpdateTasks]] for an explanation how to do that. You might want to keep the load low on the DBs, to affect production operations as low as possible, and because with this decoupled update tasks approach there is no immediate time pressure. If you want to follow the [[UpdateTasks#How_to_see_all_schemas.3F|limited parallel]] approach, use a small, mild parallelity factor (e.g. 2 or maybe 4 if you know this by far does not saturate your DB platform).

Note: don't worry if you don't see the upgrade node joining the legacy cluster: the upgrade node will not join the legacy cluster / not be visisble there since the upgrade node will be a so-called "native client" to the legacy cluster, and it will be created on the fly (and subsequently disposed again) for propagating an event. So also on <code>netstat</code> level the upgrade node will not have visible connections to the legacy cluster (unless for the very short timeframe when an actual even is sent). You can verify the functionality of that package by log lines like

<nowiki> Successfully initialzed Hazelcast client: <client-id>
Successfully got reference to cache event topic: cacheEvents-3
Publishing legacy cache event: <cache-event>

Successfully published legacy cache event, shutting down client after 546ms...</nowiki>

For the overly prudent it might be an idea to prepare a special test context with a test user living in its dedicated (test) schema, so you can test the functionality of this mechanis during upgrade first.

After the DB update tasks you can remove the special upgrade package again from the upgrade node.

The "Real Upgrade" procedure then looks like [[#Rolling_Upgrade_without_breaking_Hazelcast_upgrade|above]].

== Reference Documentation ==

=== Limitations ===

While in most cases a seamless, rolling upgrade of all nodes in the cluster is possible, there may be situations where nodes running a newer version of the Open-Xchange Server are not able to communicate with older nodes in the cluster, i.e. can't access distributed data or consume incompatible event notifications - especially, when the underlying Hazelcast library is part of the update, which does not support this scenario at the moment. In such cases, the release notes will contain corresponding information, so please have a look there before applying an update.

Additionally, there may always be some kind of race conditions during an update, i.e. client requests that can't be completed successfully or internal events not being deliverd to all nodes in the cluster. That's why the following information should only serve as a best-practices guide to minimize the impact of upgrades to the user experience.

=== Upgrading a single Node ===

Upgrading all nodes in the cluster should usually be done sequentially, i.o.w. one node after the other. This means that during the upgrade of one node, the node is temporarily disconnected from the other nodes in the cluster, and will join the cluster again after the update is completed. From the backend perspective, this is as easy as stopping the open-xchange service. other nodes in the cluster will recognize the disconnected node and start to repartition the shared cluster data automatically. But wait a minute - doing so would potentially lead to the webserver not registering the node being stopped immediately, resulting in temporary errors for currently logged in users until they are routed to another machine in the cluster. That's why it's good practice to tell the webserver's load balancer that the node should no longer fulfill incoming requests. The Apache Balancer Manager is an excellent tool for this ([http://httpd.apache.org/docs/2.2/mod/mod_status.html module ''mod_status'']). Look at the screen shot. Every node can be put into a disabled mode. Further requests will the redirected to other nodes in the cluster:

[[Image:balancer_manager.jpg]]

Afterwards, the open-xchange service on the disabled node can be stopped by executing:

$ /etc/init.d/open-xchange stop

or

$ service open-xchange stop

Now, the node is effectively in maintenance mode and any updates can take place. One could now verify the changed cluster infrastructure by accessing the Hazelcast MBeans either via JMX or the ''showruntimestats -c'' commandline tool (see above for details). There, the shut down node should no longer appear in the 'Member' section (com.hazelcast:type=Member).

When all upgrades are processed, the node open-xchange service can be started again by executing:

$ /etc/init.d/open-xchange start

or

$ service open-xchange start

As stated above, depending on the chosen cluster discovery mechanism, it might take some time until the node joins the cluster again. When using static cluster discovery, it will join the existing cluster usually directly during serivce startup, i.o.w. before other depending OSGi services are started. Otherwise, there might also be situations where the node cannot join the cluster directly, for example when there were no mDNS advertisments for other nodes in the cluster received yet. Then, it can take some additional time until the node finally joins the cluster. During startup of the node, you can observe the JMX console or the output of ''showruntimestats -c'' (com.hazelcast:type=Member) of another node in the cluster to verify when the node has joined.

After the node has joined, distributed data is re-partioned automatically, and the node is ready to server incoming requests again - so now the node can finally be enabled again in the load balancer configuration of the webserver. Afterwards, the next node in the cluster can be upgraded using the same procedure, until all nodes were processed.

=== Upgrades of the Hazelcast library ===

In case an upgrade includes a major update of the Hazelcast library, a newly upgraded node will usually not be able to connect to the nodes running the previous version. In this case, volatile cluster data is lost after all nodes in the cluster have been updated, including sessions held in the distributed session storage. As outlined above, the release notes will contain a corresponding warning in such cases.

Besides upgraded nodes not being able to access distributed data of the legacy cluster, this also affects new data not being available in the legacy cluster, which may cause troubles if the updated backend version needs to perform database update tasks. Database update tasks usually operate in a "blocking" way and all contexts associated with the schema being upgraded are disabled temporarily. Since context data itself is being held in caches on potentially each node in the cluster, the affected cache entries are invalidated during the database update. And, since cluster-wide cache invalidations again utilize Hazelcast functionality ([[#Remote Cache Invalidation]]), such invalidations normally won't be propagated to nodes running a previous version of the Hazelcast library.

To work around this specific scenario where an incompatible upgrade of the Hazelcast library needs to be performed along with blocking database update tasks, starting with v7.8.0, a supplementary package is available that explicitly enables the context cache invalidation of nodes running the previous Hazelcast library. This package follows the naming scheme ''open-xchange-cluster-upgrade-from-XXX'' (where XXX representing the version of the legacy version of the Open-Xchange server), and is available in the repositories for the updated server packages. This package should only be installed on the first node of the cluster that is going to be upgraded to the new version, and can be deactivated once the database upgrade tasks were executed successfully.

Once installed, a legacy cluster is discovered based on the available information in the ''hazelcast.properties'' configuration file in case cluster discovery is set to ''static''. If ''multicast'' is used, there's an alternative option to configure at least one of the addresses of the legacy cluster via ''com.openexchange.hazelcast.network.client.nodes''.

As an example, along with the server v7.8.0, a new package named ''open-xchange-cluster-upgrade-from-76x'' can be installed that aids in invalidating cluster server nodes running v7.6.x (which includes the Hazelcast library in version 3.2.4). Using this package, the recommended steps to update an OX cluster from version 7.6.x to version 7.8.0 would be:
# Pick a node from your cluster that you want to use for executing the database update tasks shipped with the new release
# Disable this node for incoming HTTP requests in your webserver configuration as described at [[#Upgrading a single Node]]
# Update the OX packages on this node, additionally install the package ''open-xchange-cluster-upgrade-from-76x''
# Restart the open-xchange services on this node
# Trigger the update task executions using the ''runUpdate'' commandline utitlty as described at [[UpdateTasks]]
# Once they are finished, uninstall the package ''open-xchange-cluster-upgrade-from-76x'' again
# Restart the open-xchange services on this node
# Re-enable the node for incoming HTTP requests in your webserver configuration as described at [[#Upgrading a single Node]]
# Upgrade all other nodes in the cluster as described at [[#Upgrading a single Node]]

Same steps apply to upgrading from v7.8.0 through v7.8.2 (incl.) to v7.8.3 using the package named ''open-xchange-cluster-upgrade-from-780-782'', since v7.8.0 through v7.8.2 (incl.) utilize Hazelcast v3.5.x, while v7.8.3 uses Hazelcast v3.6.4

Same steps apply to upgrading from v7.8.3 to v7.8.4 using the package named ''open-xchange-cluster-upgrade-from-783'', since v7.8.3 utilizes Hazelcast v3.7.1

Same steps apply to upgrading from v7.8.4 to v7.10.0 using the package named ''open-xchange-cluster-upgrade-from-784'', since v7.8.4 utilizes Hazelcast v3.8.1

'''Operations Note:''' The upgraded node will be added as so-called [http://docs.hazelcast.org/docs/2.3/manual/html/ch15.html Native Client] to the legacy Hazelcast Cluster.

<blockquote>
Native Client enables you to do all Hazelcast operations without being a member of the cluster.
[...]

However Native client is not member and relies on one of the cluster members.
</blockquote>

This means, the upgraded node will not be visible in the members list of the legacy Hazelcast cluster (<code>showruntimestats -c</code>). Furthermore, the native client will created and destructed on single context events, with the effect that connections will only be visible in the very moment of such an event. This means effectively that verification of the invalidation mechanis is only possible by actually executing the <code>runupdate</code> CLT. This should produce log lines like

Successfully initialzed Hazelcast client: <client-id>
Successfully got reference to cache event topic: cacheEvents-3
Publishing legacy cache event: <cache-event>
Successfully published legacy cache event, shutting down client after 546ms...

Most importantly, you should be able to observe correct functionality (users of affected contexts being logged out). It may be handy to prepare a dedicated schema with just test contexts inside. (How to create this is out of scope here, but hint: use <code>createschema</code> and <code>createcontext --schema-name</code>.)

=== Other Considerations ===

* It's always recommended to only upgrade one node after the other, always ensuring that the cluster has formed correctly between each shutdown/startup of a node.
* Do not stop a node while running the runUpdate script or the associated update task.
* During the time of such a rolling upgrade of all nodes, we have effectively heterogeneous software versions in the cluster, which potentially might lead to temporary inconsistencies. Therefore, all nodes in the cluster should be updated in one cycle (but still one after the other).
* Following the above guideline, it's also possible to add or remove nodes dynamically to the cluster, not only when disconnecting a node temporary for updates.
* In case of trouble, i.e. a node refuses to join the cluster again after restart, consult the logfiles first for any hints about what is causing the problem - both on the disconnected node, and also on other nodes in the network
* If there are general incompatibilities between two revisions of the Open-Xchange Server that prevent an operation in a cluster (release notes), it's recommended to choose another name for the cluster in ''cluster.properties'' for the nodes with the new version. This will temporary lead to two separate clusters during the rolling upgrade, and finally the old cluster being shut down completely after the last node was updated to the new version. While distributed data can't be migrated from one server version to another in this scenario due to incompatibilities, the uptime of the system itself is not affected, since the nodes in the new cluster are able to serve new incoming requests directly.
* When updating only UI plugins without also updating to a new version of the core UI, you also need to perform the additional step from [[AppSuite:UpdatingOXPackages#Updating_UI_plugins|Updating UI plugins]].

[[Category: AppSuite]] [[Category: Administration]] [[Category: Cluster]]