Open-Xchange - User contributions [en]

AppSuite:Open-Xchange Installation Guide for Debian 12.0

2024-09-06T16:22:55Z

Marens:

{{Version|7.10.6}}

= OX App Suite on Debian GNU/Linux 12 =

This article will guide you through the installation of OX App Suite, it describes the basic configuration and software requirements. As it is intended as a quick walk-through it assumes an existing installation of the operating system and requires average system administration skills. More, this guide will show you how to setup a basic installation with none of the typically used distributed environment settings. The objective of this guide is:

* To setup a single server installation
* To setup a database for a single database service, no replication
* To setup a single Open-Xchange instance, no cluster
* To provide a basic configuration setup, no mailserver configuration

= Requirements =

* Plain installed Debian GNU/Linux 12, no graphical tools required
* A supported Java Virtual Machine ([https://oxpedia.org/wiki/index.php?title=AppSuite:OX_System_Requirements#Server_Platforms learn more])
* A working internet connection
* vim is not installed by default on Debian. If you want to copy & paste the commands from this article into a shell window, you need to apt-get install vim first.

= Database installation =

Please consult our [[OXLoadBalancingClustering_Database#Standalone_database_setup|database installation instructions]] for information on how to install a database on the local system.

Before proceeding, make sure the local machine has got a working MySQL service in one of the supported versions / flavors with the configuration / tunings applied as mentioned on our [[My.cnf|corresponding page]].

= JRE Installation =

Debian Bookworm ships with OpenJDK 17, which is not suitable for OX App Suite. It is therefore required to install Eclipse Temurin 8 JRE with HotSpot VM. A comprehensive installation guide can be found at https://adoptium.net/de/installation/linux/#_deb_installation_on_debian_or_ubuntu. Quick instructions are:

# install repo key
$ apt install -y wget apt-transport-https gpg
$ wget -qO - https://packages.adoptium.net/artifactory/api/gpg/key/public | gpg --dearmor | tee /etc/apt/trusted.gpg.d/adoptium.gpg > /dev/null

# add repository
$ echo "deb https://packages.adoptium.net/artifactory/deb $(awk -F= '/^VERSION_CODENAME/{print$2}' /etc/os-release) main" | tee /etc/apt/sources.list.d/adoptium.list

# install jdk
$ apt update # update if you haven't already
$ apt install temurin-8-jdk

= Add Open-Xchange Repository =

Open-Xchange maintains public available software repositories for different platforms, such as Debian. This repository should be added to the Debian installation to enable simple installation and updates.

Start a console and modify the Debian repository information file. Also add the Open-Xchange software repository:

$ cat << EOF >> /etc/apt/sources.list.d/open-xchange.list

deb https://software.open-xchange.com/products/appsuite/stable/appsuiteui/DebianBookworm/ /
deb https://software.open-xchange.com/products/appsuite/stable/backend/DebianBookworm/ /

# if you have a valid maintenance subscription, please uncomment the
# following and add the ldb account data to the url so that the most recent
# packages get installed
# deb https://[CUSTOMERID:PASSWORD]@software.open-xchange.com/products/appsuite/stable/appsuiteui/updates/DebianBookworm /
# deb https://[CUSTOMERID:PASSWORD]@software.open-xchange.com/products/appsuite/stable/backend/updates/DebianBookworm /
EOF

= Updating repositories and install packages =

It is highly recommended to import the Open-Xchange build key to your package systems trusted keyring in order to make sure only Open-Xchange packages with valid signing are installed on the system. Otherwise you'll encounter warnings about untrusted package sources. To import the Open-Xchange buildkey, please refer to this quick guide: [[Importing_OX_Buildkey#Importing_key_into_apt_based_systems|Importing OX Buildkey]].

Reload the package index. This will download the package descriptions available at the software repositories and will enable the Open-Xchange repository as a valid source for signed packages:

$ apt update

The following command starts the download and installation process of all required package for Open-Xchange deployment:

If you want to install everything on a single server, just run

$ apt install open-xchange open-xchange-authentication-database open-xchange-grizzly \
open-xchange-admin open-xchange-appsuite \
open-xchange-appsuite-backend open-xchange-appsuite-manifest

'''Note 1:''' You have to choose between one of the available authentication packages depending on your requirements.
Open-Xchange configuration

To avoid confusion right at the start notice that Open-Xchange uses multiple administration levels and requires different credentials at some stages at the installation and server management. Note that the passwords chosen at this guide are weak and should be replaced by stronger passwords.

The MySQL database user
* Username: openexchange
* Password used at this guide: db_password
* Responsibility: Execute all kinds of database operations

The Open-Xchange Admin Master
* Username: oxadminmaster
* Password used at this guide: admin_master_password
* Responsibility: Manage contexts, manage all kinds of low level server configuration

The Context Admin
* Username: oxadmin
* Password used at this guide: admin_password
* Responsibility: Manage users/groups/resources inside a context

As stated above we assume the MySQL service has been installed previously, and it is running and available.

A good idea is to add the Open-Xchange binaries to PATH:

$ echo PATH=$PATH:/opt/open-xchange/sbin/ >> ~/.bashrc && . ~/.bashrc

Now we have to initialize the Open-Xchange configdb database. This can all be done by executing the initconfigdb script.

$ /opt/open-xchange/sbin/initconfigdb --configdb-pass=db_password -a --mysql-root-passwd=root_password

Use the --mysql-root-passwd option to supply the MySQL root password as configured during database installation.

Add the -i option if you want to remove an already existing open-xchange configdb.

'''Note:''' The -a parameter adds an openexchange account to MySQL. This account will be used for database connections from the OX App Suite middleware and requires [https://documentation.open-xchange.com/7.10.6/middleware/administration/db_user_privileges.html some privileges]. You can also create that account manually [https://oxpedia.org/wiki/index.php?title=OXLoadBalancingClustering_Database#Creating_Open-Xchange_user during database installation / configuration], in which case you can (should) skip the -a parameter here.

Before starting any service, all basic configuration files need to be set up correctly. The --configdb-pass option indicates the password of the openexchange database user previously created, the --master-pass options specifies the password of the Open-Xchange adminmaster user that will be created when executing the oxinstaller script.

'''Important:''' You should have your Open-Xchange license code at hand. If you do not plan to license Open-Xchange, you can use the option --no-license instead. Please also check [https://documentation.open-xchange.com/7.10.6/middleware/miscellaneous/report_client.html OXReportClient] documentation for more information about configuring a supported and maintained Open-Xchange server.

'''Important:''' For MAX_MEMORY_FOR_JAVAVM a rule of thumb for simple installations is half available system memory. The value must be in MB. For example "1024" for 1GB .

$ /opt/open-xchange/sbin/oxinstaller --add-license=YOUR-OX-LICENSE-CODE \
--servername=oxserver --configdb-pass=db_password \
--master-pass=admin_master_password --network-listener-host=localhost --servermemory MAX_MEMORY_FOR_JAVAVM

'''Note:''' In a clustered setup, --network-listener-host must be set to *

Now is a good time to configure the way OX will authenticate to your mail server. Edit the file /opt/open-xchange/etc/mail.properties and change the com.openexchange.mail.loginSource to use. This is very important for servers that require your full email address to log in with.

# adjust com.openexchange.mail.loginSource
$ vim /opt/open-xchange/etc/mail.properties

After initializing the configuration, restart the Open-Xchange Administration service by executing:

$ systemctl restart open-xchange

Next we have to register the local server at the Open-Xchange configdb database:

$ /opt/open-xchange/sbin/registerserver -n oxserver -A oxadminmaster -P admin_master_password

Now we have to create a local directory that should be used as Open-Xchange filestore. This directory will contain all Infostore content and files attached to groupware objects. To maintain access by the Open-Xchange Groupware service, it is required to grant permissions to the open-xchange system user.

$ mkdir /var/opt/filestore
$ chown open-xchange:open-xchange /var/opt/filestore

Now register the directory as a filestore at the Open-Xchange server:

$ /opt/open-xchange/sbin/registerfilestore -A oxadminmaster -P admin_master_password \
-t file:/var/opt/filestore -s 1000000

'''Note:''' You might want to adapt the value provided with -s, the "The maximum size of the filestore in MB", see registerfilestore --help.

'''Note 2:''' If you are setting up OX App Suite, you need a shared filestore accross your OX servers even though you do not plan to have the OX Files feature enabled for your customers.

Finally register the groupware database, this is a separated database where all groupware specific data is stored:

$ /opt/open-xchange/sbin/registerdatabase -A oxadminmaster -P admin_master_password \
-n oxdatabase -p db_password -m true

'''Note 3:''' Take into account that a global database is needed in order to store data across context boundaries. Please see this [https://oxpedia.org/wiki/index.php?title=AppSuite:CrossContextDatabase documentation] on how to register it.

= Configure services =

Now as the Open-Xchange Server has been set up and the database is running, we have to configure the Apache webserver and the mod_proxy_http module to access the groupware frontend. 

$ a2enmod proxy proxy_http proxy_balancer expires deflate headers rewrite mime setenvif lbmethod_byrequests

Configure the mod_proxy_http module by creating a new Apache configuration file.

$ vim /etc/apache2/conf-available/proxy_http.conf

<IfModule mod_proxy_http.c>
ProxyRequests Off
ProxyStatus On
# When enabled, this option will pass the Host: line from the incoming request to the proxied host.
ProxyPreserveHost On
# Please note that the servlet path to the soap API has changed:
<Location /webservices>
# restrict access to the soap provisioning API
Order Deny,Allow
Deny from all
Allow from 127.0.0.1
# you might add more ip addresses / networks here
# Allow from 192.168 10 172.16
</Location>

# The old path is kept for compatibility reasons
<Location /servlet/axis2/services>
Order Deny,Allow
Deny from all
Allow from 127.0.0.1
</Location>

# Enable the balancer manager mentioned in
# https://oxpedia.org/wiki/index.php?title=AppSuite:Running_a_cluster#Updating_a_Cluster
<IfModule mod_status.c>
<Location /balancer-manager>
SetHandler balancer-manager
Order Deny,Allow
Deny from all
Allow from 127.0.0.1
</Location>
</IfModule>

<Proxy balancer://oxcluster>
Order deny,allow
Allow from all
# multiple server setups need to have the hostname inserted instead localhost
BalancerMember http://localhost:8009 timeout=100 smax=0 ttl=60 retry=60 loadfactor=50 route=APP1
# Enable and maybe add additional hosts running OX here
# BalancerMember http://oxhost2:8009 timeout=100 smax=0 ttl=60 retry=60 loadfactor=50 route=APP2
ProxySet stickysession=JSESSIONID|jsessionid scolonpathdelim=On
SetEnv proxy-initial-not-pooled
SetEnv proxy-sendchunked
</Proxy>

# The standalone documentconverter(s) within your setup (if installed)
# Make sure to restrict access to backends only
# See: https://httpd.apache.org/docs/$YOUR_VERSION/mod/mod_authz_host.html#allow for more infos
#<Proxy balancer://oxcluster_docs>
# Order Deny,Allow
# Deny from all
# Allow from backend1IP
# BalancerMember http://converter_host:8009 timeout=100 smax=0 ttl=60 retry=60 loadfactor=50 keepalive=On route=APP3
# ProxySet stickysession=JSESSIONID|jsessionid scolonpathdelim=On
# SetEnv proxy-initial-not-pooled
# SetEnv proxy-sendchunked
#</Proxy>
# Define another Proxy Container with different timeout for the sync clients. Microsoft recommends a minimum value of 15 minutes.
# Setting the value lower than the one defined as com.openexchange.usm.eas.ping.max_heartbeat in eas.properties will lead to connection
# timeouts for clients. See https://support.microsoft.com/?kbid=905013 for additional information.
#
# NOTE for Apache versions < 2.4:
# When using a single node system or using BalancerMembers that are assigned to other balancers please add a second hostname for that
# BalancerMember's IP so Apache can treat it as additional BalancerMember with a different timeout.
#
# Example from /etc/hosts: 127.0.0.1 localhost localhost_sync
#
# Alternatively select one or more hosts of your cluster to be restricted to handle only eas/usm requests
<Proxy balancer://eas_oxcluster>
Order deny,allow
Allow from all
# multiple server setups need to have the hostname inserted instead localhost
BalancerMember http://localhost_sync:8009 timeout=1900 smax=0 ttl=60 retry=60 loadfactor=50 route=APP1
# Enable and maybe add additional hosts running OX here
# BalancerMember http://oxhost2:8009 timeout=1900 smax=0 ttl=60 retry=60 loadfactor=50 route=APP2
ProxySet stickysession=JSESSIONID|jsessionid scolonpathdelim=On
SetEnv proxy-initial-not-pooled
SetEnv proxy-sendchunked
</Proxy>

# When specifying additional mappings via the ProxyPass directive be aware that the first matching rule wins. Overlapping urls of
# mappings have to be ordered from longest URL to shortest URL.
#
# Example:
# ProxyPass /ajax balancer://oxcluster_with_100s_timeout/ajax
# ProxyPass /ajax/test balancer://oxcluster_with_200s_timeout/ajax/test
#
# Requests to /ajax/test would have a timeout of 100s instead of 200s
#
# See:
# - https://httpd.apache.org/docs/current/mod/mod_proxy.html#proxypass Ordering ProxyPass Directives
# - https://httpd.apache.org/docs/current/mod/mod_proxy.html#workers Worker Sharing
ProxyPass /ajax balancer://oxcluster/ajax
ProxyPass /appsuite/api balancer://oxcluster/ajax
ProxyPass /drive balancer://oxcluster/drive
ProxyPass /infostore balancer://oxcluster/infostore
ProxyPass /realtime balancer://oxcluster/realtime
ProxyPass /servlet balancer://oxcluster/servlet
ProxyPass /webservices balancer://oxcluster/webservices

#ProxyPass /documentconverterws balancer://oxcluster_docs/documentconverterws

ProxyPass /usm-json balancer://eas_oxcluster/usm-json
ProxyPass /Microsoft-Server-ActiveSync balancer://eas_oxcluster/Microsoft-Server-ActiveSync

</IfModule>

Modify the default website settings to display the Open-Xchange GUI

$ vim /etc/apache2/sites-enabled/000-default.conf

<VirtualHost *:80>
ServerAdmin webmaster@localhost

DocumentRoot /var/www/html
<Directory /var/www/html>
Options -Indexes +FollowSymLinks +MultiViews
AllowOverride None
Order allow,deny
allow from all
RedirectMatch ^/$ /appsuite/
</Directory>

<Directory /var/www/html/appsuite>
Options None +SymLinksIfOwnerMatch
AllowOverride Indexes FileInfo
</Directory>
</VirtualHost>

If you want to secure your Apache setup via HTTPS (which is highly recommended) or if you have proxies in front of your Apache please follow the instructions at:

* [https://oxpedia.org/wiki/index.php?title=AppSuite:Grizzly#.2Fopt.2Fopen-xchange.2Fetc.2Fserver.conf Grizzly configuration] in general, and specifically:
* [https://oxpedia.org/wiki/index.php?title=AppSuite:Grizzly#X-FORWARDED-PROTO_Header X-FORWARDED-PROTO Header]
* [https://oxpedia.org/wiki/index.php?title=AppSuite:Grizzly#X-FORWARDED-FOR_Header X-FORWARDED-FOR Header]

to properly instruct the backend about the security status of the connection and the remote IP used to contact the backend.

Enable the proxy configuration

$ a2enconf proxy_http.conf

After the configuration is done, restart the Apache webserver

$ systemctl restart apache2

== Apache Setting for more concurrent Connections ==

By default apache2 is configured to support 150 concurrent connections. This forces all parallel requests beyond that limit to wait. Especially if, for example, active sync clients maintain a permanent connection for push events to arrive. The following article explains how that can be done

[https://oxpedia.org/wiki/index.php?title=Tune_apache2_for_more_concurrent_connections Apache Setting for more concurrent Connections]

= Creating contexts and users =

Now as the whole setup is complete and you already should get a login screen when accessing the server with a webbrowser, we have to setup a context and a default user as the last step of this tutorial.

The mapping defaultcontext will allow you to set this context as the default one of the entire system so that users which will be created within this context can login into Open-Xchange Server without specifying their domain at the login screen. Only one context can be specified as defaultcontext. The oxadmin user that will be created by this command is the default admin of the created context. This account will gather additional functions that are also described in the administration manual. The context id parameter must to be unique and numeric, otherwise the server will complain when you try to create a context. New contexts must be created by the oxadminmaster user, user accounts inside a context are created with the credentials of the contexts oxadmin account. The access-combination-name property defines the set of available modules and functions for users of the context.

$ /opt/open-xchange/sbin/createcontext -A oxadminmaster -P admin_master_password -c 1 \
-u oxadmin -d "Context Admin" -g Admin -s User -p admin_password -L defaultcontext \
-e oxadmin@example.com -q 1024 --access-combination-name=groupware_standard

To create a user for testing purposes (Make sure the password you use here for the user is the same password as your email account or you will not be able to use the email module until it is set right):

$ /opt/open-xchange/sbin/createuser -c 1 -A oxadmin -P admin_password -u testuser \
-d "Test User" -g Test -s User -p secret -e testuser@example.com \
--imaplogin testuser --imapserver 127.0.0.1 --smtpserver 127.0.0.1

Now connect to the server with a webbrowser and login using the credentials testuser / secret.

A complete overview about the different parameter is provided at the [https://oxpedia.org/wiki/index.php?title=OX_Permission_Level permission matrix].

If you need to migrate a batch of users and contexts at once, check the CSV Batch Import documentation [https://oxpedia.org/wiki/index.php?title=Csv_import page].

= Log files and issue tracking =

== Default logging mechanism ==

Whenever unexpected or erroneous behavior takes place, it will be logged depending on the configured loglevel. All logfiles are stored at the operating systems default location. Events triggered by the Open-Xchange Groupware services are logged to a rotating file open-xchange.log.0. Those files are the very first place to monitor.

$ tail -f -n200 /var/log/open-xchange/open-xchange.log.0

== Alternative logging mechanisms ==

Apart from the default file logging mechanism, Open-Xchange supports logging via logback framework and therefore via syslog and/or logstash. This makes it possible to directly log to a local or remote syslog daemon or other services. Logback is highly customizable, please see the documentation below.

* [https://oxpedia.org/wiki/index.php?title=AppSuite:OX_Logging Logback configuration Guide OX App Suite]

[[Category: AppSuite]]

Importing OX Buildkey

2024-09-06T16:12:09Z

Marens: Debian 12 deprecated apt-key usage and can't work with pub files

__FORCETOC__

== Importing the Open-Xchange Buildkey ==

Open-Xchange RPM packages are signed with the Open-Xchange build key and the APT repository is signed according to http://wiki.debian.org/SecureApt.

Key ID: DFD4BCF6
Key fingerprint: 307D BCBD 75D2 B00E F81C 1E4C 1211 1537 DFD4 BCF6

The public key can be downloaded from public PGP keyservers or from https://software.open-xchange.com/0xDFD4BCF6-oxbuildkey.pub

Note that ''older'' releases have been signed using a different build key. You can download the public key to validate those packages here: https://software.open-xchange.com/0xEED949F0-oxbuildkey.pub

=== Importing key into APT based systems ===

==== From Debian 12 on ====

To add the keyring file to your trusted keys directory

$ wget -qO - https://software.open-xchange.com/0xDFD4BCF6-oxbuildkey.pub | gpg --dearmor > /etc/apt/trusted.gpg.d/0xDFD4BCF6-oxbuildkey.gpg

==== Up to Debian 11 ====

To import the key into the APT keyring, run

$ wget https://software.open-xchange.com/0xDFD4BCF6-oxbuildkey.pub -O - | apt-key add -

If this command returns "OK", you're done importing the buildkey and can go on with the installation. In case that does not work, split it into two steps:

$ wget https://software.open-xchange.com/0xDFD4BCF6-oxbuildkey.pub
$ apt-key add - < 0xDFD4BCF6-oxbuildkey.pub

To verify the success, run

$ apt-key list

The following or similar output should appear:

/etc/apt/trusted.gpg
--------------------
pub rsa4096 2020-05-25 [SC]
307D BCBD 75D2 B00E F81C 1E4C 1211 1537 DFD4 BCF6
uid [ unknown] Open-Xchange Release Team <release-team@open-xchange.com>
sub rsa4096 2020-05-25 [E]

=== Importing Key into RPM based systems ===

To import that key into the RPM database, use the following command (as root):

$ rpm --import https://software.open-xchange.com/0xDFD4BCF6-oxbuildkey.pub

to verify the success, use

$ rpm -qi gpg-pubkey

that should give an output similar to this one:

Name : gpg-pubkey
Version : dfd4bcf6
Release : 5ecb8c8f
Architecture: (none)
Install Date: Tue Feb 2 09:47:31 2021
Group : Public Keys
Size : 0
License : pubkey
Signature : (none)
Source RPM : (none)
Build Date : Mon May 25 11:14:55 2020
Build Host : localhost
Relocations : (not relocatable)
Packager : Open-Xchange Release Team <release-team@open-xchange.com>
Summary : gpg(Open-Xchange Release Team <release-team@open-xchange.com>)
Description :

-----BEGIN PGP PUBLIC KEY BLOCK-----
Version: rpm-4.14.2 (NSS-3)

xsFNBF7LjI8BEACi0I7wWTWa3mZUD28D3W+Gcb2xboEojBkLQXs3o9AELi2iz1Oz
gmdWtmoNl83BUtMazKD3MJsPJ+uHCpoCGD10rSIMoMw5plLJ+zCfoPhDVAfulDVj
3nIJnQXplZaPvOq1Z4ulWydfXOt3/X5M+bmFyDUfllIbWK07OEENR/IW4D6TPiNZ
DtiZLnXWKQy3UJMOWTtzDyXJKCB7dp4pfMjtDnbHYClBrh0q1skXMWpKQBLC239U
7ZAgs4UdyvABfGwLGfrPyz3H9nQGl/Olx6h+HOwvqTT/ucQwMgi7tpzlmAchlSd4
fWQyXZO79RRJIyV3pD+lKr/1r0CZy6GqfslzikYJFNi+iZY0ru4QsqRmLUtrjIZ7
al5zzJDYRpzd65J8G0gSWazz8C6QzNPw+36wDIF0jidWC3XQbkhsK6s1XlIbcwdS
+5sYsR0ShWeRIvPNB4yeZJU+WqK9YHYvqRn19lCzH/36DiNVbcMDViyl864eHPcs
zbIGAoPugEq6S0Ji/K+wlRFRdTATuwLt+WEaiOejynZ7+X9T1cptpOLV7ubz7SrX
/JpBRy2tzwFaLZH2vsiWvinfb7/x0FVvhdjuEryou7plZSlQL8FI0TCg9VUYX/DE
AHmx33hiDhVRuv3wSxxENvg3VbJf1PbQBlf7XP76MeFRhBlTXz3TsD1TlQARAQAB
zTlPcGVuLVhjaGFuZ2UgUmVsZWFzZSBUZWFtIDxyZWxlYXNlLXRlYW1Ab3Blbi14
Y2hhbmdlLmNvbT7CwY4EEwEIADgWIQQwfby9ddKwDvgcHkwSERU339S89gUCXsuM
jwIbAwULCQgHAgYVCgkICwIEFgIDAQIeAQIXgAAKCRASERU339S89lwgD/9kaiGD
crOjnBQd6OzaL0oSlePP6VWHxhSOoK3okHS6G3DJwdLxluhbCraYjFcquVE3QIS5
krYusQ8VoQ8Sb+TQAE4ew8ILNgGMkV9veZ3EtvcwDOM5G3gEVzReJdBZP/bxxe25
3Yq6V9b29ZuC5HEAzfcUspyMX3Uvwfiwub7uWxmmAKta6sH2aKZNNadcAAqwDnTQ
YnBlq34UJFIc1YlHQ74cCRWsletfceqRgksh2Fa26B4+bnAFe+SySJ/QE4Xj8vH4
+fht5VWyXFtHcIU6c/+TgdH5FTGze2Xv20drzMWPgcXBMEq2CTMw/G8y+BSTnuWm
Vi02o5GLVfVcpOqNnDdpqWH562UaS75uKgjE1sv4VK/NqP+5jJkAvP2xmhjiqTEo
Qidhz6/X/AYpJGer8JYtyepir/sRvTYOE9HmffV4ffXIJZ0FjNDR3HEeHjcxoPvh
GpXG9khpFv9C7DjiCiahu79zYYp544mPF9mmU5onVYNTnDyGJjxDQqO0B51mzYZU
qMMm98K6ZOpcpAwC/UV6ocMBH3/QD/gfpmnk4EfSPDUBMVbp/R1a4PsiKrRh2GQ6
JS5EYRzdM/UfJuDy4wIov3dxJEjcx34ipiFTuYBGCFAnFcMD6uzcqKRytg8Tign9
gMt8fJDawO2v9r+L/7WECfTXKdu4OAdOKM0eeM7BTQRey4yPARAAsR8AJ7EP7tnt
j+t+LMJPfR3Q0WAv8kwE4jQsuxrGHAIqnlW6hFqFtahBxLFAmb6eaxiGQbeC2wDl
noYYJh6gri6CkRUlW2QROIDmW8uoTMo6yjrO8l0TOlgI8m6vYcphPyueK6V2bx5O
yI+YN4Np8qbcxtdYUQ4OH2lHcsWetUkL++NU1hmKTpcRGETCdCLy9AWare6x5m4p
embPXYI43/9cyOO3jWSzLYRwFuoQ6dSxI1lwTCpHfsHuPesGSF//DzBBf5+maLO8
786hVKrU1x7Ei2nq2HH5be9dNHOnIXQE/VweNEpoSQf7YMB1QouxTm4XqzLO/TEh
/I/UELfNqCXxVAl5YDIehXpUj0dCT0NSRkYxbKCUkXmqxUo26cpXkdxSQyCykst/
RR7wlhqUUM3cdQVsKSbm7ZD3s06bjCF3BPRCCCgSuBEcz/RHi9eLTXzV4VCmQZsD
RU5pFVIG6JNgveZ6s2hlww0NwTj6RQL/dJrLV2egTAAg7KPWR+OI7UQO4cmgcOeU
PQ6i8zcPeD7MX7cdo6P6xd5ohwaWvhd5reJvODMWVfchthELySq4rZpxph2dXMoU
eNFgbO3iGqB4sT2+2UQV1JzVatBjfZpk2e4Kgl73vi7IPogN3ttqTc0m2gaCTRcp
nS5Q6l548f9i3s8eFX5t911CxJHlBs8AEQEAAcLBdgQYAQgAIBYhBDB9vL110rAO
+BweTBIRFTff1Lz2BQJey4yPAhsMAAoJEBIRFTff1Lz2NMAP/1CoAv6gDxhTdHtb
56n5PsHrZ0hJ0ngWfXVe610GTasNZ3i2E4JRyIkpQjIJPdre3bP0g9fL7dvB7ByS
JfEshLUFBYFxm3ds+q2aV7aOx6+9+7C3RLv+KVofB5V3qUsxXH3nhJWtL/JGhl/I
1uQ7qFW5DKA3tI33Fxy+fLgAXM7/s/AXmCQBAFx/87g2Qqy/ZyxZfcTigyWEjRQG
xtpQsWMZ5/67OmslnQojs4iYLVDN2RXx85Q2O/6rq2OgKZPjx/aCmcR6U5nwjIsq
ZF+r8eNAkbyPNIqpWVE8jYpk3hn8iue6huK+KzqrHJckJyp+UiTSSj/3bDp/+lcO
JfuSCG80W+05SCw5/TqLBVaFZCsxYZVU/ik9WwW7a5yTgE8cvXcoLbJ0dMqSGx0c
wcdJydHZuOKP8t5EJICbyKfeKnZmFFCLhn15uqIOkJjUQ4dYeYoYQb08NKh67+Uk
V4F2MGf47aQ4skORCvA3vaO5rUGdsyvvfJ+F7DQ1KYinvBTRfJ/hSjli+E20leOH
yjwcYaRXpN2pKM34eGZFaKiYssSWcaTQngkyGgZDbesT93G/bnrma+1yG57CjX/9
Qz5Iy+6e/6tqGTOvvIaQUTs+QdfUAys//usHefyliY6BkxESLZiejB6vKm22UfQG
4pjAF2BUyDHxqHCdOxgUroDcxspM
=OOBX
-----END PGP PUBLIC KEY BLOCK-----

[[Category: OX6]]
[[Category: AppSuite]]

AppSuite:Open-Xchange Installation Guide for Debian 12.0

2024-09-06T15:19:26Z

Marens: Distro and JDK specific updates

{{Version|7.10.6}}

= OX App Suite on Debian GNU/Linux 12 =

This article will guide you through the installation of OX App Suite, it describes the basic configuration and software requirements. As it is intended as a quick walk-through it assumes an existing installation of the operating system and requires average system administration skills. More, this guide will show you how to setup a basic installation with none of the typically used distributed environment settings. The objective of this guide is:

* To setup a single server installation
* To setup a database for a single database service, no replication
* To setup a single Open-Xchange instance, no cluster
* To provide a basic configuration setup, no mailserver configuration

= Requirements =

* Plain installed Debian GNU/Linux 12, no graphical tools required
* A supported Java Virtual Machine ([https://oxpedia.org/wiki/index.php?title=AppSuite:OX_System_Requirements#Server_Platforms learn more])
* A working internet connection
* vim is not installed by default on Debian. If you want to copy & paste the commands from this article into a shell window, you need to apt-get install vim first.

= Database installation =

Please consult our [[OXLoadBalancingClustering_Database#Standalone_database_setup|database installation instructions]] for information on how to install a database on the local system.

Before proceeding, make sure the local machine has got a working MySQL service in one of the supported versions / flavors with the configuration / tunings applied as mentioned on our [[My.cnf|corresponding page]].

= JRE Installation =

Debian Bookworm ships with OpenJDK 17, which is not suitable for OX App Suite. It is therefore required to install Eclipse Temurin 8 JRE with HotSpot VM. A comprehensive installation guide can be found at https://adoptium.net/de/installation/linux/#_deb_installation_on_debian_or_ubuntu. Quick instructions are:

# install repo key
$ apt install -y wget apt-transport-https gpg
$ wget -qO - https://packages.adoptium.net/artifactory/api/gpg/key/public | gpg --dearmor | tee /etc/apt/trusted.gpg.d/adoptium.gpg > /dev/null

# add repository
$ echo "deb https://packages.adoptium.net/artifactory/deb $(awk -F= '/^VERSION_CODENAME/{print$2}' /etc/os-release) main" | tee /etc/apt/sources.list.d/adoptium.list

# install jdk
$ apt update # update if you haven't already
$ apt install temurin-8-jdk

= Add Open-Xchange Repository =

Open-Xchange maintains public available software repositories for different platforms, such as Debian. This repository should be added to the Debian installation to enable simple installation and updates.

Start a console and modify the Debian repository information file. Also add the Open-Xchange software repository:

$ cat << EOF >> /etc/apt/sources.list.d/open-xchange.list

deb https://software.open-xchange.com/products/appsuite/stable/appsuiteui/DebianBookworm/ /
deb https://software.open-xchange.com/products/appsuite/stable/backend/DebianBookworm/ /

# if you have a valid maintenance subscription, please uncomment the
# following and add the ldb account data to the url so that the most recent
# packages get installed
# deb https://[CUSTOMERID:PASSWORD]@software.open-xchange.com/products/appsuite/stable/appsuiteui/updates/DebianBookworm /
# deb https://[CUSTOMERID:PASSWORD]@software.open-xchange.com/products/appsuite/stable/backend/updates/DebianBookworm /
EOF

= Updating repositories and install packages =

It is highly recommended to import the Open-Xchange build key to your package systems trusted keyring in order to make sure only Open-Xchange packages with valid signing are installed on the system. Otherwise you'll encounter warnings about untrusted package sources. To import the Open-Xchange buildkey, please refer to this quick guide: [[Importing_OX_Buildkey#Importing_key_into_apt_based_systems|Importing OX Buildkey]].

Reload the package index. This will download the package descriptions available at the software repositories and will enable the Open-Xchange repository as a valid source for signed packages:

$ apt-get update

The following command starts the download and installation process of all required package for Open-Xchange deployment:

If you want to install everything on a single server, just run

$ apt-get install open-xchange open-xchange-authentication-database open-xchange-grizzly \
open-xchange-admin open-xchange-appsuite \
open-xchange-appsuite-backend open-xchange-appsuite-manifest

'''Note 1:''' You have to choose between one of the available authentication packages depending on your requirements.
Open-Xchange configuration

To avoid confusion right at the start notice that Open-Xchange uses multiple administration levels and requires different credentials at some stages at the installation and server management. Note that the passwords chosen at this guide are weak and should be replaced by stronger passwords.

The MySQL database user
* Username: openexchange
* Password used at this guide: db_password
* Responsibility: Execute all kinds of database operations

The Open-Xchange Admin Master
* Username: oxadminmaster
* Password used at this guide: admin_master_password
* Responsibility: Manage contexts, manage all kinds of low level server configuration

The Context Admin
* Username: oxadmin
* Password used at this guide: admin_password
* Responsibility: Manage users/groups/resources inside a context

As stated above we assume the MySQL service has been installed previously, and it is running and available.

A good idea is to add the Open-Xchange binaries to PATH:

$ echo PATH=$PATH:/opt/open-xchange/sbin/ >> ~/.bashrc && . ~/.bashrc

Now we have to initialize the Open-Xchange configdb database. This can all be done by executing the initconfigdb script.

$ /opt/open-xchange/sbin/initconfigdb --configdb-pass=db_password -a --mysql-root-passwd=root_password

Use the --mysql-root-passwd option to supply the MySQL root password as configured during database installation.

Add the -i option if you want to remove an already existing open-xchange configdb.

'''Note:''' The -a parameter adds an openexchange account to MySQL. This account will be used for database connections from the OX App Suite middleware and requires [https://documentation.open-xchange.com/7.10.6/middleware/administration/db_user_privileges.html some privileges]. You can also create that account manually [https://oxpedia.org/wiki/index.php?title=OXLoadBalancingClustering_Database#Creating_Open-Xchange_user during database installation / configuration], in which case you can (should) skip the -a parameter here.

Before starting any service, all basic configuration files need to be set up correctly. The --configdb-pass option indicates the password of the openexchange database user previously created, the --master-pass options specifies the password of the Open-Xchange adminmaster user that will be created when executing the oxinstaller script.

'''Important:''' You should have your Open-Xchange license code at hand. If you do not plan to license Open-Xchange, you can use the option --no-license instead. Please also check [https://documentation.open-xchange.com/7.10.6/middleware/miscellaneous/report_client.html OXReportClient] documentation for more information about configuring a supported and maintained Open-Xchange server.

'''Important:''' For MAX_MEMORY_FOR_JAVAVM a rule of thumb for simple installations is half available system memory. The value must be in MB. For example "1024" for 1GB .

$ /opt/open-xchange/sbin/oxinstaller --add-license=YOUR-OX-LICENSE-CODE \
--servername=oxserver --configdb-pass=db_password \
--master-pass=admin_master_password --network-listener-host=localhost --servermemory MAX_MEMORY_FOR_JAVAVM

'''Note:''' In a clustered setup, --network-listener-host must be set to *

Now is a good time to configure the way OX will authenticate to your mail server. Edit the file /opt/open-xchange/etc/mail.properties and change the com.openexchange.mail.loginSource to use. This is very important for servers that require your full email address to log in with.

# adjust com.openexchange.mail.loginSource
$ vim /opt/open-xchange/etc/mail.properties

After initializing the configuration, restart the Open-Xchange Administration service by executing:

$ systemctl restart open-xchange

Next we have to register the local server at the Open-Xchange configdb database:

$ /opt/open-xchange/sbin/registerserver -n oxserver -A oxadminmaster -P admin_master_password

Now we have to create a local directory that should be used as Open-Xchange filestore. This directory will contain all Infostore content and files attached to groupware objects. To maintain access by the Open-Xchange Groupware service, it is required to grant permissions to the open-xchange system user.

$ mkdir /var/opt/filestore
$ chown open-xchange:open-xchange /var/opt/filestore

Now register the directory as a filestore at the Open-Xchange server:

$ /opt/open-xchange/sbin/registerfilestore -A oxadminmaster -P admin_master_password \
-t file:/var/opt/filestore -s 1000000

'''Note:''' You might want to adapt the value provided with -s, the "The maximum size of the filestore in MB", see registerfilestore --help.

'''Note 2:''' If you are setting up OX App Suite, you need a shared filestore accross your OX servers even though you do not plan to have the OX Files feature enabled for your customers.

Finally register the groupware database, this is a separated database where all groupware specific data is stored:

$ /opt/open-xchange/sbin/registerdatabase -A oxadminmaster -P admin_master_password \
-n oxdatabase -p db_password -m true

'''Note 3:''' Take into account that a global database is needed in order to store data across context boundaries. Please see this [https://oxpedia.org/wiki/index.php?title=AppSuite:CrossContextDatabase documentation] on how to register it.

= Configure services =

Now as the Open-Xchange Server has been set up and the database is running, we have to configure the Apache webserver and the mod_proxy_http module to access the groupware frontend. 

$ a2enmod proxy proxy_http proxy_balancer expires deflate headers rewrite mime setenvif lbmethod_byrequests

Configure the mod_proxy_http module by creating a new Apache configuration file.

$ vim /etc/apache2/conf-available/proxy_http.conf

<IfModule mod_proxy_http.c>
ProxyRequests Off
ProxyStatus On
# When enabled, this option will pass the Host: line from the incoming request to the proxied host.
ProxyPreserveHost On
# Please note that the servlet path to the soap API has changed:
<Location /webservices>
# restrict access to the soap provisioning API
Order Deny,Allow
Deny from all
Allow from 127.0.0.1
# you might add more ip addresses / networks here
# Allow from 192.168 10 172.16
</Location>

# The old path is kept for compatibility reasons
<Location /servlet/axis2/services>
Order Deny,Allow
Deny from all
Allow from 127.0.0.1
</Location>

# Enable the balancer manager mentioned in
# https://oxpedia.org/wiki/index.php?title=AppSuite:Running_a_cluster#Updating_a_Cluster
<IfModule mod_status.c>
<Location /balancer-manager>
SetHandler balancer-manager
Order Deny,Allow
Deny from all
Allow from 127.0.0.1
</Location>
</IfModule>

<Proxy balancer://oxcluster>
Order deny,allow
Allow from all
# multiple server setups need to have the hostname inserted instead localhost
BalancerMember http://localhost:8009 timeout=100 smax=0 ttl=60 retry=60 loadfactor=50 route=APP1
# Enable and maybe add additional hosts running OX here
# BalancerMember http://oxhost2:8009 timeout=100 smax=0 ttl=60 retry=60 loadfactor=50 route=APP2
ProxySet stickysession=JSESSIONID|jsessionid scolonpathdelim=On
SetEnv proxy-initial-not-pooled
SetEnv proxy-sendchunked
</Proxy>

# The standalone documentconverter(s) within your setup (if installed)
# Make sure to restrict access to backends only
# See: https://httpd.apache.org/docs/$YOUR_VERSION/mod/mod_authz_host.html#allow for more infos
#<Proxy balancer://oxcluster_docs>
# Order Deny,Allow
# Deny from all
# Allow from backend1IP
# BalancerMember http://converter_host:8009 timeout=100 smax=0 ttl=60 retry=60 loadfactor=50 keepalive=On route=APP3
# ProxySet stickysession=JSESSIONID|jsessionid scolonpathdelim=On
# SetEnv proxy-initial-not-pooled
# SetEnv proxy-sendchunked
#</Proxy>
# Define another Proxy Container with different timeout for the sync clients. Microsoft recommends a minimum value of 15 minutes.
# Setting the value lower than the one defined as com.openexchange.usm.eas.ping.max_heartbeat in eas.properties will lead to connection
# timeouts for clients. See https://support.microsoft.com/?kbid=905013 for additional information.
#
# NOTE for Apache versions < 2.4:
# When using a single node system or using BalancerMembers that are assigned to other balancers please add a second hostname for that
# BalancerMember's IP so Apache can treat it as additional BalancerMember with a different timeout.
#
# Example from /etc/hosts: 127.0.0.1 localhost localhost_sync
#
# Alternatively select one or more hosts of your cluster to be restricted to handle only eas/usm requests
<Proxy balancer://eas_oxcluster>
Order deny,allow
Allow from all
# multiple server setups need to have the hostname inserted instead localhost
BalancerMember http://localhost_sync:8009 timeout=1900 smax=0 ttl=60 retry=60 loadfactor=50 route=APP1
# Enable and maybe add additional hosts running OX here
# BalancerMember http://oxhost2:8009 timeout=1900 smax=0 ttl=60 retry=60 loadfactor=50 route=APP2
ProxySet stickysession=JSESSIONID|jsessionid scolonpathdelim=On
SetEnv proxy-initial-not-pooled
SetEnv proxy-sendchunked
</Proxy>

# When specifying additional mappings via the ProxyPass directive be aware that the first matching rule wins. Overlapping urls of
# mappings have to be ordered from longest URL to shortest URL.
#
# Example:
# ProxyPass /ajax balancer://oxcluster_with_100s_timeout/ajax
# ProxyPass /ajax/test balancer://oxcluster_with_200s_timeout/ajax/test
#
# Requests to /ajax/test would have a timeout of 100s instead of 200s
#
# See:
# - https://httpd.apache.org/docs/current/mod/mod_proxy.html#proxypass Ordering ProxyPass Directives
# - https://httpd.apache.org/docs/current/mod/mod_proxy.html#workers Worker Sharing
ProxyPass /ajax balancer://oxcluster/ajax
ProxyPass /appsuite/api balancer://oxcluster/ajax
ProxyPass /drive balancer://oxcluster/drive
ProxyPass /infostore balancer://oxcluster/infostore
ProxyPass /realtime balancer://oxcluster/realtime
ProxyPass /servlet balancer://oxcluster/servlet
ProxyPass /webservices balancer://oxcluster/webservices

#ProxyPass /documentconverterws balancer://oxcluster_docs/documentconverterws

ProxyPass /usm-json balancer://eas_oxcluster/usm-json
ProxyPass /Microsoft-Server-ActiveSync balancer://eas_oxcluster/Microsoft-Server-ActiveSync

</IfModule>

Modify the default website settings to display the Open-Xchange GUI

$ vim /etc/apache2/sites-enabled/000-default.conf

<VirtualHost *:80>
ServerAdmin webmaster@localhost

DocumentRoot /var/www/html
<Directory /var/www/html>
Options -Indexes +FollowSymLinks +MultiViews
AllowOverride None
Order allow,deny
allow from all
RedirectMatch ^/$ /appsuite/
</Directory>

<Directory /var/www/html/appsuite>
Options None +SymLinksIfOwnerMatch
AllowOverride Indexes FileInfo
</Directory>
</VirtualHost>

If you want to secure your Apache setup via HTTPS (which is highly recommended) or if you have proxies in front of your Apache please follow the instructions at:

* [https://oxpedia.org/wiki/index.php?title=AppSuite:Grizzly#.2Fopt.2Fopen-xchange.2Fetc.2Fserver.conf Grizzly configuration] in general, and specifically:
* [https://oxpedia.org/wiki/index.php?title=AppSuite:Grizzly#X-FORWARDED-PROTO_Header X-FORWARDED-PROTO Header]
* [https://oxpedia.org/wiki/index.php?title=AppSuite:Grizzly#X-FORWARDED-FOR_Header X-FORWARDED-FOR Header]

to properly instruct the backend about the security status of the connection and the remote IP used to contact the backend.

Enable the proxy configuration

$ a2enconf proxy_http.conf

After the configuration is done, restart the Apache webserver

$ systemctl restart apache2

== Apache Setting for more concurrent Connections ==

By default apache2 is configured to support 150 concurrent connections. This forces all parallel requests beyond that limit to wait. Especially if, for example, active sync clients maintain a permanent connection for push events to arrive. The following article explains how that can be done

[https://oxpedia.org/wiki/index.php?title=Tune_apache2_for_more_concurrent_connections Apache Setting for more concurrent Connections]

= Creating contexts and users =

Now as the whole setup is complete and you already should get a login screen when accessing the server with a webbrowser, we have to setup a context and a default user as the last step of this tutorial.

The mapping defaultcontext will allow you to set this context as the default one of the entire system so that users which will be created within this context can login into Open-Xchange Server without specifying their domain at the login screen. Only one context can be specified as defaultcontext. The oxadmin user that will be created by this command is the default admin of the created context. This account will gather additional functions that are also described in the administration manual. The context id parameter must to be unique and numeric, otherwise the server will complain when you try to create a context. New contexts must be created by the oxadminmaster user, user accounts inside a context are created with the credentials of the contexts oxadmin account. The access-combination-name property defines the set of available modules and functions for users of the context.

$ /opt/open-xchange/sbin/createcontext -A oxadminmaster -P admin_master_password -c 1 \
-u oxadmin -d "Context Admin" -g Admin -s User -p admin_password -L defaultcontext \
-e oxadmin@example.com -q 1024 --access-combination-name=groupware_standard

To create a user for testing purposes (Make sure the password you use here for the user is the same password as your email account or you will not be able to use the email module until it is set right):

$ /opt/open-xchange/sbin/createuser -c 1 -A oxadmin -P admin_password -u testuser \
-d "Test User" -g Test -s User -p secret -e testuser@example.com \
--imaplogin testuser --imapserver 127.0.0.1 --smtpserver 127.0.0.1

Now connect to the server with a webbrowser and login using the credentials testuser / secret.

A complete overview about the different parameter is provided at the [https://oxpedia.org/wiki/index.php?title=OX_Permission_Level permission matrix].

If you need to migrate a batch of users and contexts at once, check the CSV Batch Import documentation [https://oxpedia.org/wiki/index.php?title=Csv_import page].

= Log files and issue tracking =

== Default logging mechanism ==

Whenever unexpected or erroneous behavior takes place, it will be logged depending on the configured loglevel. All logfiles are stored at the operating systems default location. Events triggered by the Open-Xchange Groupware services are logged to a rotating file open-xchange.log.0. Those files are the very first place to monitor.

$ tail -f -n200 /var/log/open-xchange/open-xchange.log.0

== Alternative logging mechanisms ==

Apart from the default file logging mechanism, Open-Xchange supports logging via logback framework and therefore via syslog and/or logstash. This makes it possible to directly log to a local or remote syslog daemon or other services. Logback is highly customizable, please see the documentation below.

* [https://oxpedia.org/wiki/index.php?title=AppSuite:OX_Logging Logback configuration Guide OX App Suite]

[[Category: AppSuite]]

AppSuite:Open-Xchange Installation Guide for RHEL8

2023-10-09T16:05:04Z

Marens: /* Apache and SELinux */

{{Version|7.10.6}}

= Open-Xchange App Suite on RedHat Enterprise Linux 8 =

{{QuickInstIntro|release=appsuite}}

= Requirements =

* Plain installed RedHat Enterprise Linux 8 with latest updates
* Valid access to the RedHat Network
* A configured internet connection
* A supported Java Virtual Machine ([http://oxpedia.org/wiki/index.php?title=AppSuite:OX_System_Requirements#Server_Platforms learn more])
* <code>vim</code> is not installed by default on RHEL8. If you want to copy & paste the commands from this article into a shell window, you need to <code>dnf install vim</code> first.

= Database installation =

Please consult our [[OXLoadBalancingClustering_Database#Standalone_database_setup|database installation instructions]] for information on how to install a database on the local system.

Before proceeding, make sure the local machine has got a working MySQL service in one of the supported versions / flavors with the configuration / tunings applied as mentioned on our [[My.cnf|corresponding page]].

= Enabling required RedHat Repositories =

Ensure that your Redhat system is subscribed by e.g. following this article https://access.redhat.com/solutions/253273

{{AddReposRHEL|rhelname=RHEL8|release=appsuite}}

= Updating repositories and installing packages =

Reload the package index. This will download the package descriptions available at the software repositories:

$ dnf update

The following command starts the download and installation process of all required package for Open-Xchange deployment:

{{OXPackageInstallation|installer=dnf|release=appsuite}}

{{OXConfiguration}}

{{oxinstaller|connector=http}}

After initializing the configuration, start the Open-Xchange service by executing:

$ systemctl start open-xchange

{{OXRegister|globaldb=true}}

= Configure services =

{{ApacheOXModsIntro|connector=http}}

The default installation of the Apache webserver on RHEL provides a welcome screen which is not necessary for server operation, it can be removed by deleting the corresponding configuration file:

$ rm /etc/httpd/conf.d/welcome.conf

{{Template:ApacheAppSuiteConf|connector=http|connectorConf=/etc/httpd/conf.d/proxy_http.conf|apacheconf=/etc/httpd/conf.d/ox.conf|docroot=/var/www/html/|loadmodule=LoadModule proxy_http_module modules/mod_proxy_http.so|syncProxyName=eas_oxcluster}}

== Apache and SELinux ==

When using SELinux make sure to allow Apache to connect to the middleware process on port 8009. One way to achieve this could be:

$ setsebool -P httpd_can_network_connect 1

The following links contain more infos:
* [https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/html/selinux_users_and_administrators_guide/chap-managing_confined_services-the_apache_http_server SELinux: The Apache HTTP Server]
* [https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/html/selinux_users_and_administrators_guide/sect-managing_confined_services-the_apache_http_server-booleans SELinux: The Apache HTTP Server - Booleans]

After the configuration is done, restart the Apache webserver

$ systemctl start httpd

== Apache Setting for more concurrent Connections ==

By default apache2 is configured to support 150 concurrent connections. This forces all parallel requests beyond that limit to wait. Especially if, for example, active sync clients maintain a permanent connection for push events to arrive. The following article explains how that can be done

[[Tune_apache2_for_more_concurrent_connections|Apache Setting for more concurrent Connections]]

= Adding services to runlevels =

The new services are now installed and configured, but to make them start up on a server boot, they need to be added to some runlevels:

$ systemctl enable mariadb
$ systemctl enable httpd
$ systemctl enable open-xchange

{{ContextUserAndLogs}}

= Installing Open-Xchange Update packages =

Please read [[UpdatingOXPackages]] on how to get access to the latest Open-Xchange packages.

[[Category: AppSuite]]

AppSuite:Open-Xchange Installation Guide for RHEL8

2023-10-09T16:04:19Z

Marens:

{{Version|7.10.6}}

= Open-Xchange App Suite on RedHat Enterprise Linux 8 =

{{QuickInstIntro|release=appsuite}}

= Requirements =

* Plain installed RedHat Enterprise Linux 8 with latest updates
* Valid access to the RedHat Network
* A configured internet connection
* A supported Java Virtual Machine ([http://oxpedia.org/wiki/index.php?title=AppSuite:OX_System_Requirements#Server_Platforms learn more])
* <code>vim</code> is not installed by default on RHEL8. If you want to copy & paste the commands from this article into a shell window, you need to <code>dnf install vim</code> first.

= Database installation =

Please consult our [[OXLoadBalancingClustering_Database#Standalone_database_setup|database installation instructions]] for information on how to install a database on the local system.

Before proceeding, make sure the local machine has got a working MySQL service in one of the supported versions / flavors with the configuration / tunings applied as mentioned on our [[My.cnf|corresponding page]].

= Enabling required RedHat Repositories =

Ensure that your Redhat system is subscribed by e.g. following this article https://access.redhat.com/solutions/253273

{{AddReposRHEL|rhelname=RHEL8|release=appsuite}}

= Updating repositories and installing packages =

Reload the package index. This will download the package descriptions available at the software repositories:

$ dnf update

The following command starts the download and installation process of all required package for Open-Xchange deployment:

{{OXPackageInstallation|installer=dnf|release=appsuite}}

{{OXConfiguration}}

{{oxinstaller|connector=http}}

After initializing the configuration, start the Open-Xchange service by executing:

$ systemctl start open-xchange

{{OXRegister|globaldb=true}}

= Configure services =

{{ApacheOXModsIntro|connector=http}}

The default installation of the Apache webserver on RHEL provides a welcome screen which is not necessary for server operation, it can be removed by deleting the corresponding configuration file:

$ rm /etc/httpd/conf.d/welcome.conf

{{Template:ApacheAppSuiteConf|connector=http|connectorConf=/etc/httpd/conf.d/proxy_http.conf|apacheconf=/etc/httpd/conf.d/ox.conf|docroot=/var/www/html/|loadmodule=LoadModule proxy_http_module modules/mod_proxy_http.so|syncProxyName=eas_oxcluster}}

== Apache and SELinux ==

When using selinux make sure to allow Apache to connect to the middleware process on port 8009. One way to achieve this could be:

$ setsebool -P httpd_can_network_connect 1

The following links contain more infos:
* [https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/html/selinux_users_and_administrators_guide/chap-managing_confined_services-the_apache_http_server SELinux: The Apache HTTP Server]
* [https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/html/selinux_users_and_administrators_guide/sect-managing_confined_services-the_apache_http_server-booleans SELinux: The Apache HTTP Server - Booleans]

After the configuration is done, restart the Apache webserver

$ systemctl start httpd

== Apache Setting for more concurrent Connections ==

By default apache2 is configured to support 150 concurrent connections. This forces all parallel requests beyond that limit to wait. Especially if, for example, active sync clients maintain a permanent connection for push events to arrive. The following article explains how that can be done

[[Tune_apache2_for_more_concurrent_connections|Apache Setting for more concurrent Connections]]

= Adding services to runlevels =

The new services are now installed and configured, but to make them start up on a server boot, they need to be added to some runlevels:

$ systemctl enable mariadb
$ systemctl enable httpd
$ systemctl enable open-xchange

{{ContextUserAndLogs}}

= Installing Open-Xchange Update packages =

Please read [[UpdatingOXPackages]] on how to get access to the latest Open-Xchange packages.

[[Category: AppSuite]]

AppSuite:Open-Xchange Installation Guide for RHEL8

2023-10-09T11:57:23Z

Marens: dnf is default since rhel8

{{Version|7.10.6}}

= Open-Xchange App Suite on RedHat Enterprise Linux 8 =

{{QuickInstIntro|release=appsuite}}

= Requirements =

* Plain installed RedHat Enterprise Linux 8 with latest updates
* Valid access to the RedHat Network
* A configured internet connection
* A supported Java Virtual Machine ([http://oxpedia.org/wiki/index.php?title=AppSuite:OX_System_Requirements#Server_Platforms learn more])
* <code>vim</code> is not installed by default on RHEL8. If you want to copy & paste the commands from this article into a shell window, you need to <code>dnf install vim</code> first.

= Database installation =

Please consult our [[OXLoadBalancingClustering_Database#Standalone_database_setup|database installation instructions]] for information on how to install a database on the local system.

Before proceeding, make sure the local machine has got a working MySQL service in one of the supported versions / flavors with the configuration / tunings applied as mentioned on our [[My.cnf|corresponding page]].

= Enabling required RedHat Repositories =

Ensure that your Redhat system is subscribed by e.g. following this article https://access.redhat.com/solutions/253273

{{AddReposRHEL|rhelname=RHEL8|release=appsuite}}

= Updating repositories and installing packages =

Reload the package index. This will download the package descriptions available at the software repositories:

$ dnf update

The following command starts the download and installation process of all required package for Open-Xchange deployment:

{{OXPackageInstallation|installer=dnf|release=appsuite}}

{{OXConfiguration}}

{{oxinstaller|connector=http}}

After initializing the configuration, start the Open-Xchange service by executing:

$ systemctl start open-xchange

{{OXRegister|globaldb=true}}

= Configure services =

{{ApacheOXModsIntro|connector=http}}

The default installation of the Apache webserver on RHEL provides a welcome screen which is not necessary for server operation, it can be removed by deleting the corresponding configuration file:

$ rm /etc/httpd/conf.d/welcome.conf

{{Template:ApacheAppSuiteConf|connector=http|connectorConf=/etc/httpd/conf.d/proxy_http.conf|apacheconf=/etc/httpd/conf.d/ox.conf|docroot=/var/www/html/|loadmodule=LoadModule proxy_http_module modules/mod_proxy_http.so|syncProxyName=eas_oxcluster}}

After the configuration is done, restart the Apache webserver

$ systemctl start httpd

== Apache Setting for more concurrent Connections ==

By default apache2 is configured to support 150 concurrent connections. This forces all parallel requests beyond that limit to wait. Especially if, for example, active sync clients maintain a permanent connection for push events to arrive. The following article explains how that can be done

[[Tune_apache2_for_more_concurrent_connections|Apache Setting for more concurrent Connections]]

= Adding services to runlevels =

The new services are now installed and configured, but to make them start up on a server boot, they need to be added to some runlevels:

$ systemctl enable mariadb
$ systemctl enable httpd
$ systemctl enable open-xchange

{{ContextUserAndLogs}}

= Installing Open-Xchange Update packages =

Please read [[UpdatingOXPackages]] on how to get access to the latest Open-Xchange packages.

[[Category: AppSuite]]

AppSuite:Documents Installation Guide 7103

2019-06-13T17:25:32Z

Marens: typo

{{Version|7.10.3}}

= Download & Installation OX Documents =

=== General Information ===

OX Documents are browser based, cloud ready, text, spreadsheet and presentation products that can work with Microsoft Office and OpenOffice documents in a lossless way. And you can also collaborate with other people to edit shared documents on various devices.

== Requirements ==

See the [[AppSuite:OX_System_Requirements|Open-Xchange software requirements page]] for details.

OX Documents needs to be installed in an OX App Suite configuration with a [[AppSuite:Grizzly|backend based on Grizzly]] (including hazelcast as installed and activated by default).

For compatibility with the legacy MS binary format files (.DOC, .XLS, .PPT) and for printing functionality in OX Text the document converter components (incl. readerengine) are required. These are not available in the Community Version of OX AppSuite.

== Mandatory Modules ==

Additional functional modules need to be installed separately:

* The Document Converter API: See [[AppSuite:DocumentConverterAPIInstall|Document converter API installation instructions]]

== Installation ==

The OX Documents deployment consists of the packages <tt>open-xchange-documents-backend</tt> and <tt>open-xchange-documents-ui</tt>. The <tt>open-xchange-documents-backend</tt> package also requires and installs these packages:

* <tt>open-xchange-file-distribution</tt>
* <tt>open-xchange-sessionstorage-hazelcast</tt>

The following Packages are no longer needed (since 7.10.3)
* <tt>open-xchange-realtime-core</tt>
* <tt>open-xchange-realtime-json</tt>

=== Redhat Enterprise Linux 6 or CentOS 6 ===

Add the following repositories to your Open-Xchange yum configuration:

{{for loop||call=YUMRepo|pv=reponame|pc1n=path|pc1v=products/appsuite/stable|pc2n=rhelname|pc2v=RHEL6|office|office-web|documentconverter-api}}

if you have a valid maintenance subscription, please add also following repositories by replacing the credentials.

{{for loop||call=YUMRepo|pv=reponame|pc1n=path|pc1v=products/appsuite/stable|pc2n=rhelname|pc2v=RHEL6|pc3n=ldbaccount|pc3v=LDBUSER:LDBPASSWORD|office/updates|office-web/updates|documentconverter-api/updates}}
$ yum install open-xchange-documents-backend open-xchange-documents-ui open-xchange-documents-ui-static

=== Redhat Enterprise Linux 7 or CentOS 7 ===

Add the following repositories to your Open-Xchange yum configuration:

{{for loop||call=YUMRepo|pv=reponame|pc1n=path|pc1v=products/appsuite/stable|pc2n=rhelname|pc2v=RHEL7|office|office-web|documentconverter-api}}

if you have a valid maintenance subscription, please add also following repositories by replacing the credentials.

{{for loop||call=YUMRepo|pv=reponame|pc1n=path|pc1v=products/appsuite/stable|pc2n=rhelname|pc2v=RHEL7|pc3n=ldbaccount|pc3v=LDBUSER:LDBPASSWORD|office/updates|office-web/updates|documentconverter-api/updates}}

$ yum install open-xchange-documents-backend open-xchange-documents-ui open-xchange-documents-ui-static

=== Debian GNU/Linux 8.0 ===

Add the following repositories to your Open-Xchange apt configuration:

{{for loop||call=APTRepo|pv=reponame|pc1n=path|pc1v=products/appsuite/stable|pc2n=debianname|pc2v=DebianJessie|office|office-web|documentconverter-api}}

if you have a valid maintenance subscription, please add also following repositories by replacing the credentials.

{{for loop||call=APTRepo|pv=reponame|pc1n=path|pc1v=products/appsuite/stable|pc2n=debianname|pc2v=DebianJessie|pc3n=ldbaccount|pc3v=LDBUSER:LDBPASSWORD|office/updates|office-web/updates|documentconverter-api/updates}}

$ apt-get update
$ apt-get install open-xchange-documents-backend open-xchange-documents-ui open-xchange-documents-ui-static

=== Debian GNU/Linux 9.0 (valid from v7.10) ===

Add the following repositories to your Open-Xchange apt configuration:

{{for loop||call=APTRepo|pv=reponame|pc1n=path|pc1v=products/appsuite/stable|pc2n=debianname|pc2v=DebianStretch|office|office-web|documentconverter-api}}

if you have a valid maintenance subscription, please add also following repositories by replacing the credentials.

{{for loop||call=APTRepo|pv=reponame|pc1n=path|pc1v=products/appsuite/stable|pc2n=debianname|pc2v=DebianStretch|pc3n=ldbaccount|pc3v=LDBUSER:LDBPASSWORD|office/updates|office-web/updates|documentconverter-api/updates}}

$ apt-get update
$ apt-get install open-xchange-documents-backend open-xchange-documents-ui open-xchange-documents-ui-static

=== SUSE Linux Enterprise Server 12 ===

{{for loop||call=SUSERepo|pv=reponame|pc1n=path|pc1v=products/appsuite/stable|pc2n=susename|pc2v=SLE_12|office|office-web|documentconverter-api}}

if you have a valid maintenance subscription, please add also following repositories by replacing the credentials.

{{for loop||call=SUSERepo|pv=reponame|pc1n=path|pc1v=products/appsuite/stable|pc2n=susename|pc2v=SLE_12|pc3n=ldbaccount|pc3v=LDBUSER:LDBPASSWORD|office/updates|office-web/updates|documentconverter-api/updates}}

$ zypper ref
$ zypper install open-xchange-documents-backend open-xchange-documents-ui open-xchange-documents-ui-static

=== Univention Corporate Server ===

OX Text for OX App Suite for UCS is available in the Univention App Center. Please check the UMC module App Center for the installation of the OX Documents at your already available environment.

== Documents Collaboration Service ==

With 7.10.3 the Documents Collaboration Service has been introduced.

The Documents Collaboration Service needs to be installed, configured and run. It supports collaboration across Middleware nodes but is also mandatory for a small (even single node) environments.
The server can be run as one additional service (JVM) or for larger deployments several instances can be clustered.
The implementation is based on Spring Boot and uses the Java-based messaging server Apache ActiveMQ.

Note:
For upgrades from earlier versions than 7.10.2 the hazelcast cluster requires a complete shutdown, since the internal OX Documents data has changed. A rolling upgrade is not possible.

=== Documents Collaboration Server (DCS) ===

prepare documents-collaboration repo as described above.

The DCS will be installed via the package:
<pre>
open-xchange-documents-collaboration-server
</pre>

Target directory is /usr/share/open-xchange-documents-collaboration

Configuration file is /etc/documents-collaboration/dcs.properties. The following entries need to be reviewed and values adjusted to the setup. 
Configure bind host IP and JMS Port to listen to
<pre>
dcs.host = 192.168.10.2 # Interface of the DCS node
dcs.port = 61616
</pre>

A DCS writes its connection data to a database for discovery by middleware nodes
<pre>
db.host=192.168.10.172
db.port=3306
db.schema=dcsdb
db.username=<db_user>
db.password=<db_password>
</pre>
For management of DCS hosts the database used by the middleware can be configured.

The default logging path is set in //etc//documents-collaboration/logback-spring.xml and points to //var//log//open-xchange//documents-collaboration//documents-collaboration.log

The DCS Service will be started with
systemctl start open-xchange-documents-collaboration.service

A command line tool allows to check the values in the database
<pre>
# /usr/share/open-xchange-documents-collaboration/bin/documents-collaboration-admin.sh -l
+---------------------+---------------+---------------+---------+
| ID | Host | Interface | JMSPort |
+---------------------+---------------+---------------+---------+
| 192.168.10.25:61616 | 192.168.10.25 | 192.168.10.25 | 61616 |
+---------------------+---------------+---------------+---------+
</pre>

===Middleware node===

no new pkg repo, no new pkgs required for middleware

Discovery of the DCS service is configured in documents-collaboration-client.properties.
Add the appropriate values to access the dcsdb Database as set for the DCS above:
<pre>
com.openexchange.dcs.client.database.connectionURL=jdbc:mysql://databasehost:3306/dcsdb
com.openexchange.dcs.client.database.userName=db_user
com.openexchange.dcs.client.database.password=db_password
</pre>

A command line tool allows to check the values in the database
<pre>
$ /opt/open-xchange/sbin/documents-collaboration-admin -l
+---------------------+---------------+---------------+---------+
| ID | Host | Interface | JMSPort |
+---------------------+---------------+---------------+---------+
| 192.168.10.25:61616 | 192.168.10.25 | 192.168.10.25 | 61616 |
+---------------------+---------------+---------------+---------+
</pre>

== Printing and legacy MS binary formats Editing ==

For .DOC/.XLS/.PPT and/or printing support from OX Documents the Document Converter and readerengine components have to be installed separately (license key required)

* See [[AppSuite:DocumentConverterInstall|Document converter / readerengine installation instructions]]

== Monitoring ==

OX Documents offers runtime information via JMX about the document editors as well as the OX Documentconverter. This [[AppSuite:DocumentsMonitoring|article]] guides to the information how to access JMX data, configure and use Jolokia and integrate with munin.

== Configuration ==

=== Permissions ===

To enable the OX Documents Applications for OX Drive the associated permission has to be set.

The default setting for all users is changed in the file '''documents.properties''' in the directory ''/opt/open-xchange/etc''.

After installation the functionality is disabled:
<pre>
# Enables or disables the "text" module capability globally.
# com.openexchange.capability.text=true

# Enables or disables the "spreadsheet" module capability globally.
# com.openexchange.capability.spreadsheet=true

# Enables or disables the "presentation" module capability globally.
# com.openexchange.capability.presentation=true

</pre>

The following line enables the functionality:
<pre>
# Enables or disables the "text" module capability globally.
com.openexchange.capability.text=true
</pre>

Starting with 7.6.2, you can disable the text portal app. In '''permissions.properties''':
<pre>
permissions=textportaldisabled
</pre>

=== Collaboration on shared documents ===

If OX Documents functionality is used for guest users in a cluster of application servers, this setting needs to be adjusted to false in order to also have guest sessions available in the distributed storage.

in ''/opt/open-xchange/etc/sharing.properties''

<pre>
# Specifies whether guest sessions are treated as transient or not. Transient
# sessions are only held in the short-term session containers, and are not put
# into the distributed session storage. Defaults to "true".
com.openexchange.share.transientSessions=false
</pre>

com.openechange.share.transientSessions has to be set to false.

=== Spell Checking===

OX Text and OX Presentation use ''hunspell'' for spell checking functionality.
By default Spellchecking is enabled in ''/opt/open-xchange/etc/settings/office.properties''
with the following entry
<pre>
# Determines whether online spelling is enabled for office documents.
# Possible values: true|false
# If this property is missing true is taken.
io.ox/office//module/spellingEnabled=true
</pre>

After installation of the ''hunspell'' package for your platform the shared library and the US English dictionaries are available. Please check the file location the pkg files were installed into. (For instance on CentOS, use rpm -q –filesbypkg hunspell and verify the path of the lib files to be configured below)
Also verify that the Hunspell dictonary is present and not renamed to "myspell" etc.. if the dictionary is renamed, you will have to use that in the config below.

Additional dictionaries are published for your platform as packages hunspell-de-de, hunspell-fr, ...

Spell checking has to be configured in the file ''/opt/open-xchange/etc/hunspell.properties''.
The values for the shared library file and dictionary path have to be set according to the platform.

This example shows the values for Debian:

<pre>
# The name and location of the hunspell library
# Default value: "/usr/lib/x86_64-linux-gnu/libhunspell-1.3.so.0"
com.openexchange.spellchecker.hunspell.library=/usr/lib/x86_64-linux-gnu/libhunspell-1.4.so.0
# The location of the dictionaries
# Default value: "/usr/share/hunspell"
com.openexchange.spellchecker.hunspell.dictionaries=/usr/share/hunspell
</pre>

If spellchecking is enabled and the library path or dictionary path is wrong a warning is logged.

If a language is not supported by a hunspell dictionary (or produces error messages in the console log) myspell dictionaries are a working alternative.

=== Templates ===

OX Documents support different levels to provide document templates for users.

A number of ready-to-use templates for letters, memos, resumes, home budget, presentations, etc is included with the system. These templates are made available in a directory at each system node by installation of the package open-xchange-documents-templates. The path to this directory is defined in the office.properties file

<pre>
# Defines the absolute path where document templates are located inside the local (!) file system.
# The templates are displayed in the documents applications.
# If this property is missing a default '/opt/open-xchange/templates/documents' is taken.
io.ox/office//module/templatePath = /opt/open-xchange/templates/documents
</pre>

The content in the directory specified by the path must comply to the following rules:

<pre>
/opt/open-xchange/templates/documents
|
----> text [application type]
| |
| ----> en-US [language-region]
| |
| ----> en [language fallback, en is also fallback for all other non-convered languages]
| |
| ----> de-DE
| |
| ----> de
| |
| ----> common [language independent, used in addition to the language specific ones]
|
----> spreadsheet
|
----> presentation
</pre>

A context administrator can additionally provide global templates for all users in the same context.
If logged in with this role folders can be added to or removed from the list of global template folders. For regular users the list is displayed without the permission to modify it.

[[File:template-folder.png|border|600px|caption]]

Users can create and manage their own templates. These user templates are stored and retrieved from the user’s root folder (“My files”). In addition to the root folder the user can define more template folders, sub-folders or even external folders in the “Documents” section of the “Settings” menu. This helps to organize templates and allows cleanup of the root folder by removal of templates.

For installations of OX Spreadsheet based on Appsuite 7.8.0 and 7.8.1 please see [http://oxpedia.org/wiki/index.php?title=AppSuite:Spreadsheet_Installation_Guide_7_8_1 here] for different installation modes.

AppSuite:Mobile API Facade

2017-07-10T08:22:48Z

Marens: /* Proxy configuration */

= OX Facade for Mobile =

== General Information ==

The Mobile API Facade is a server component that brings the new native mobile mail apps together with the OX App Suite. We’ve built the façade based on the technology used and proven in the OX App Suite middleware. The facade is developed in Java, utilizing the OSGI Framework.

The Facade provides offline friendly HTTP interface by doing the work, the connections through the HTTP API, to the OX App Suite and providing only the data the offline capable clients need. In some cases multiple requests to the OX App Suite are combined into one for its clients.The facade also offers a method to tell the clients that information on the server haven’t changed since the last time the client asked for it. This reduces the amount of data to transmit to the clients under certain circumstances, especially important in mobile networks were bandwidth (and overall traffic) is limited. Thanks to facade some functionality can be shared among all clients and need only get implemented once. One example for this is the teaser text extraction, and HTML mail handling in general.The facade also provides a pluggable authentication system. In the default case, login request are just forwarded to the middleware. In more advanced use-cases, the login request is forwarded to IDM of the customer and an OX session is created from the access token from the IDM. For clients this is pretty straightforward.

== Requirements ==

The OX Facade has to be installed alongside an OX App Suite installation. It requires at least OX App Suite v7.8.4.

== OX Facade API ==

Further information about the OX Facade API can be found at:
https://documentation.open-xchange.com/components/facade/1.0.0/

= OX Mail Server-side Installation and Configuration on OX App Suite v7.8.x =

This chapter describes how the backend components of OX Mail are installed and configured on the server.

== Available packages ==

OX Facade is available with the following backend packages:

* ''open-xchange-mobile-api-facade''

Installation on the server varies depending on the underlying distribution, details are available in the following chapters.

=== Redhat Enterprise Linux 6 or CentOS 6 ===

Add the following repositories to your Open-Xchange yum configuration:

{{for loop||call=YUMRepo|pv=reponame|pc1n=path|pc1v=products/mobile-api-facade/stable|pc2n=rhelname|pc2v=RHEL6|mobile-api-facade}}

$ yum install open-xchange-mobile-api-facade

=== Redhat Enterprise Linux 7 or CentOS 7 ===

Add the following repositories to your Open-Xchange yum configuration:

{{for loop||call=YUMRepo|pv=reponame|pc1n=path|pc1v=products/mobile-api-facade/stable|pc2n=rhelname|pc2v=RHEL7|mobile-api-facade}}

$ yum install open-xchange-mobile-api-facade

=== Debian GNU/Linux 8.0 ===

Add the following repositories to your Open-Xchange apt configuration:

{{for loop||call=APTRepo|pv=reponame|pc1n=path|pc1v=products/mobile-api-facade/stable|pc2n=debianname|pc2v=DebianJessie|mobile-api-facade}}

$ apt-get update
$ apt-get install open-xchange-mobile-api-facade

=== SUSE Linux Enterprise Server 12 ===

{{for loop||call=SUSERepo|pv=reponame|pc1n=path|pc1v=products/mobile-api-facade/stable|pc2n=susename|pc2v=SLE_12|mobile-api-facade}}

$ zypper ref
$ zypper install open-xchange-mobile-api-facade

= Configuration OX Facade for Mobile =

== Introduction ==

To be able to use the native mail apps the Mobile API Facade needs to be installed in front of the OX App Suite middleware. This document describes how to configure the Mobile API Facade.

The Mobile API Facade stores its configuration in the files <code>/opt/open-xchange/mobile-api-facade/etc/facade.properties</code> (the global configuration) and in <code>/opt/open-xchange/mobile-api-facade/etc/mobile-api-facade-config.yml</code> (hostname specific configuration). Both files support the same configuration properties as can be seen on https://documentation.open-xchange.com/components/facade/config/1.0.0/.

== Connection to the OX App Suite Middleware ==

After installation of the facade package ("open-xchange-mobile-api-facade") the property <code>com.openexchange.mobile.api.facade.MiddlewareBaseUrl</code> needs to get set to the correct URL. This property needs to be explicitly configured by the administrator. It has no default value. Its possible to connect the Mobile API Facade directly to a middleware process, but this is highly discouraged. The Mobile API Facade should always connect to a middleware process through a load balancer.

An example:

com.openexchange.mobile.api.facade.MiddlewareBaseUrl=https://appsuite.example.com/appsuite/api

After this configuration the open-xchange-mobile-api-facade needs to get restarted.

== Proxy configuration ==

<Proxy balancer://oxcluster_facade>
Order Allow,Deny
Allow from all
BalancerMember http://appsuite-middleware1.example.com:8007 timeout=100 smax=0 ttl=60 retry=60
loadfactor=50 keepalive=On route=FOX1
BalancerMember http://appsuite-middleware2.example.com:8007 timeout=100 smax=0 ttl=60 retry=60
loadfactor=50 keepalive=On route=FOX2

ProxySet stickysession=JSESSIONID|jsessionidscolonpathdelim=On
SetEnv proxy-initial-not-pooled
SetEnv proxy-sendchunked
</Proxy>

ProxyPass /services/api-facade balancer://oxcluster_facade/services/api-facade

== Traffic compression ==

The clients need to exchange a lot of data with the facade to accomplish the task of a mail client. This traffic can be compressed. Clients add the header "Accept-Encoding: gzip,deflate" by default. For this to work the Apache web server in front of the facade needs to handle this as the facade itself is not returning compressed response bodies.

For Apache HTTPD <code>{mod_deflate}</code> needs to be enabled and the following line needs to be added to your virtual host:

AddOutputFilterByType DEFLATE text/html text/plain text/javascript application/javascript text/css
text/xml application/xml text/x-js application/x-javascript application/json

== Starting/Stopping the Facade Service ==

The facade runs as its own service independent of the normal OX App Suite middleware. For this on Debian-based system it can be started with

service open-xchange-mobile-api-facade start

It can be stopped with

service open-xchange-mobile-api-facade stop

== Ports ==

As the Mobile API Facade is its own process it also has its own JMX port and its own RMI port. The default JMX port is 9995 and the default RMI port is 1100. These ports needs to be explicitly specified to the command line tools using either JMX or RMI.

AppSuite:Mobile API Facade

2017-07-10T08:21:41Z

Marens: /* Proxy configuration */

= OX Facade for Mobile =

== General Information ==

The Mobile API Facade is a server component that brings the new native mobile mail apps together with the OX App Suite. We’ve built the façade based on the technology used and proven in the OX App Suite middleware. The facade is developed in Java, utilizing the OSGI Framework.

The Facade provides offline friendly HTTP interface by doing the work, the connections through the HTTP API, to the OX App Suite and providing only the data the offline capable clients need. In some cases multiple requests to the OX App Suite are combined into one for its clients.The facade also offers a method to tell the clients that information on the server haven’t changed since the last time the client asked for it. This reduces the amount of data to transmit to the clients under certain circumstances, especially important in mobile networks were bandwidth (and overall traffic) is limited. Thanks to facade some functionality can be shared among all clients and need only get implemented once. One example for this is the teaser text extraction, and HTML mail handling in general.The facade also provides a pluggable authentication system. In the default case, login request are just forwarded to the middleware. In more advanced use-cases, the login request is forwarded to IDM of the customer and an OX session is created from the access token from the IDM. For clients this is pretty straightforward.

== Requirements ==

The OX Facade has to be installed alongside an OX App Suite installation. It requires at least OX App Suite v7.8.4.

== OX Facade API ==

Further information about the OX Facade API can be found at:
https://documentation.open-xchange.com/components/facade/1.0.0/

= OX Mail Server-side Installation and Configuration on OX App Suite v7.8.x =

This chapter describes how the backend components of OX Mail are installed and configured on the server.

== Available packages ==

OX Facade is available with the following backend packages:

* ''open-xchange-mobile-api-facade''

Installation on the server varies depending on the underlying distribution, details are available in the following chapters.

=== Redhat Enterprise Linux 6 or CentOS 6 ===

Add the following repositories to your Open-Xchange yum configuration:

{{for loop||call=YUMRepo|pv=reponame|pc1n=path|pc1v=products/mobile-api-facade/stable|pc2n=rhelname|pc2v=RHEL6|mobile-api-facade}}

$ yum install open-xchange-mobile-api-facade

=== Redhat Enterprise Linux 7 or CentOS 7 ===

Add the following repositories to your Open-Xchange yum configuration:

{{for loop||call=YUMRepo|pv=reponame|pc1n=path|pc1v=products/mobile-api-facade/stable|pc2n=rhelname|pc2v=RHEL7|mobile-api-facade}}

$ yum install open-xchange-mobile-api-facade

=== Debian GNU/Linux 8.0 ===

Add the following repositories to your Open-Xchange apt configuration:

{{for loop||call=APTRepo|pv=reponame|pc1n=path|pc1v=products/mobile-api-facade/stable|pc2n=debianname|pc2v=DebianJessie|mobile-api-facade}}

$ apt-get update
$ apt-get install open-xchange-mobile-api-facade

=== SUSE Linux Enterprise Server 12 ===

{{for loop||call=SUSERepo|pv=reponame|pc1n=path|pc1v=products/mobile-api-facade/stable|pc2n=susename|pc2v=SLE_12|mobile-api-facade}}

$ zypper ref
$ zypper install open-xchange-mobile-api-facade

= Configuration OX Facade for Mobile =

== Introduction ==

To be able to use the native mail apps the Mobile API Facade needs to be installed in front of the OX App Suite middleware. This document describes how to configure the Mobile API Facade.

The Mobile API Facade stores its configuration in the files <code>/opt/open-xchange/mobile-api-facade/etc/facade.properties</code> (the global configuration) and in <code>/opt/open-xchange/mobile-api-facade/etc/mobile-api-facade-config.yml</code> (hostname specific configuration). Both files support the same configuration properties as can be seen on https://documentation.open-xchange.com/components/facade/config/1.0.0/.

== Connection to the OX App Suite Middleware ==

After installation of the facade package ("open-xchange-mobile-api-facade") the property <code>com.openexchange.mobile.api.facade.MiddlewareBaseUrl</code> needs to get set to the correct URL. This property needs to be explicitly configured by the administrator. It has no default value. Its possible to connect the Mobile API Facade directly to a middleware process, but this is highly discouraged. The Mobile API Facade should always connect to a middleware process through a load balancer.

An example:

com.openexchange.mobile.api.facade.MiddlewareBaseUrl=https://appsuite.example.com/appsuite/api

After this configuration the open-xchange-mobile-api-facade needs to get restarted.

== Proxy configuration ==

<Proxy balancer://oxcluster_facade>
Order Allow,Deny
Allow from all
BalancerMember http://appsuite-middleware1.somedomain:8007 timeout=100 smax=0 ttl=60 retry=60
loadfactor=50 keepalive=On route=FOX1
BalancerMember http://appsuite-middleware2.somedomain:8007 timeout=100 smax=0 ttl=60 retry=60
loadfactor=50 keepalive=On route=FOX2

ProxySet stickysession=JSESSIONID|jsessionidscolonpathdelim=On
SetEnv proxy-initial-not-pooled
SetEnv proxy-sendchunked
</Proxy>

ProxyPass /services/api-facade balancer://oxcluster_facade/services/api-facade

== Traffic compression ==

The clients need to exchange a lot of data with the facade to accomplish the task of a mail client. This traffic can be compressed. Clients add the header "Accept-Encoding: gzip,deflate" by default. For this to work the Apache web server in front of the facade needs to handle this as the facade itself is not returning compressed response bodies.

For Apache HTTPD <code>{mod_deflate}</code> needs to be enabled and the following line needs to be added to your virtual host:

AddOutputFilterByType DEFLATE text/html text/plain text/javascript application/javascript text/css
text/xml application/xml text/x-js application/x-javascript application/json

== Starting/Stopping the Facade Service ==

The facade runs as its own service independent of the normal OX App Suite middleware. For this on Debian-based system it can be started with

service open-xchange-mobile-api-facade start

It can be stopped with

service open-xchange-mobile-api-facade stop

== Ports ==

As the Mobile API Facade is its own process it also has its own JMX port and its own RMI port. The default JMX port is 9995 and the default RMI port is 1100. These ports needs to be explicitly specified to the command line tools using either JMX or RMI.

AppSuite:UpdatingOXPackages

2017-05-04T14:13:31Z

Marens: /* Warnings */

<div class="title">Updating OX App Suite packages</div>

''Synopsys'': This article describes how to update OX App Suite packages from one service pack to another.

== Warnings ==

Before we begin, here are several things that you should keep in mind:
* If you are '''updating a cluster, please do it one node after the other'''. Turn the node off, update, turn it on again, make sure it works in the cluster. Have a look here, too: [[Running_a_cluster#Upgrading_a_single_Node|Upgrading a single node]]
* If you don't have a cluster of OX nodes, then your complete installation needs to be taken down temporarily.
* Please '''update the frontend before the backend'''. Otherwise the old frontend won't work with the new backend and it's manifests.
* Please '''do not restart Open-Xchange during the database update'''. A database update can happen after installing minor or major updates. As soon as the first user tries to log in to the system or if any provisioning action is done, this update starts.

=== Upgrading to 7.8.4 ===
* During the upgrade to 7.8.4 the existing ''JAVA_XTRAOPTS'' configuration property of the middleware stack in the file ''ox-scriptconf.sh'' will automatically be split up and migrated to a new categories approach. Please have a look at [[AppSuite:MiddlewareStartup#Evaluate_JVM_commandline_options|JVM commandline options]] and review the automatically upgraded configuration file.

== How to get updates? ==

OX App Suite updates can be accessed by customers with a valid license.

In addition, you need to configure the [[OXReportClient]].

= Updating OX App Suite Packages =

== Installing Updates ==

A new service pack usually introduces new packages and requires configuration changes. To get all required new packages and configuration changes, the following '''must''' be done when installing updates.

(Please, keep the [[#Warning|warning about frontend updates before backend updates]] in mind)

=== On Debian based distributions ===

Add the following entry to <tt>/etc/apt/sources.list.d/open-xchange.list</tt>

deb http://CUSTOMERID:PASSWORD@software.open-xchange.com/products/appsuite/stable/appsuiteui/updates/DebianJessie/ /
deb http://CUSTOMERID:PASSWORD@software.open-xchange.com/products/appsuite/stable/backend/updates/DebianJessie /
Then run

$ apt-get update
$ apt-get dist-upgrade

If you want to see, what apt-get is going to do without actually doing it, you can run:

$ apt-get dist-upgrade -s

After the new packages are installed, the open-xchange process needs a restart:

<pre>$ /etc/init.d/open-xchange restart</pre>

=== On RPM based distributions ===

==== RHEL6/CentOS6 ====

Add the following entries to <tt>/etc/yum.repos.d/ox.repo</tt>:

[ox-updates-appsuiteui]
name=Open-Xchange Updates
baseurl=http://CUSTOMERID:PASSWORD@software.open-xchange.com/products/appsuite/stable/appsuiteui/updates/RHEL6/
gpgkey=http://software.open-xchange.com/oxbuildkey.pub
enabled=1
gpgcheck=1
metadata_expire=0m

[ox-updates-backend]
name=Open-Xchange Updates
baseurl=http://CUSTOMERID:PASSWORD@software.open-xchange.com/products/appsuite/stable/backend/updates/RHEL6/
gpgkey=http://software.open-xchange.com/oxbuildkey.pub
enabled=1
gpgcheck=1
metadata_expire=0m

and run

$ yum update

$ yum upgrade

After the new packages are installed, the open-xchange process needs a restart:

<pre>$ /etc/init.d/open-xchange restart</pre>
(Please, keep the [[#Warnings|warning about database updates]] in mind)

==== RHEL7/CentOS7 ====

Add the following entries to <tt>/etc/yum.repos.d/ox.repo</tt>:

[ox-updates-appsuiteui]
name=Open-Xchange Updates
baseurl=http://CUSTOMERID:PASSWORD@software.open-xchange.com/products/appsuite/stable/appsuiteui/updates/RHEL7/
gpgkey=http://software.open-xchange.com/oxbuildkey.pub
enabled=1
gpgcheck=1
metadata_expire=0m

[ox-updates-backend]
name=Open-Xchange Updates
baseurl=http://CUSTOMERID:PASSWORD@software.open-xchange.com/products/appsuite/stable/backend/updates/RHEL7/
gpgkey=http://software.open-xchange.com/oxbuildkey.pub
enabled=1
gpgcheck=1
metadata_expire=0m

and run

$ yum update

$ yum upgrade

After the new packages are installed, the open-xchange process needs a restart:

<pre>$ /etc/init.d/open-xchange restart</pre>
(Please, keep the [[#Warnings|warning about database updates]] in mind)

==== SLES12 ====

Add the updates repository to the repository list:

$ zypper ar http://CUSTOMERID:PASSWORD@software.open-xchange.com/products/appsuite/stable/appsuiteui/updates/SLE_12/ OXAPPSUITEUIUPDATES
$ zypper ar http://CUSTOMERID:PASSWORD@software.open-xchange.com/products/appsuite/stable/backend/updates/SLE_12/ OXBACKENDUPDATES

and run

$ zypper dup -r OXAPPSUITEUIUPDATES
$ zypper dup -r OXBACKENDUPDATES

You might need to run

$ zypper ref

to update the repository metadata before running ''zypper up''.

After the new packages are installed, the open-xchange process needs a restart:

<pre>$ rcopen-xchange restart</pre>
(Please, keep the [[#Warnings|warning about database updates]] in mind.

= Updating OX App Suite Packages for older versions =

== Installing Updates ==

A new service pack usually introduces new packages and requires configuration changes. To get all required new packages and configuration changes, the following '''must''' be done when installing updates.

(Please, keep the [[#Warning|warning about frontend updates before backend updates]] in mind)

=== On Debian based distributions ===

If you want to update an older version of Open-Xchange App Suite to the latest maintenance release, add the following entry to <tt>/etc/apt/sources.list.d/open-xchange.list</tt>. Replace VERSION with the version you are using (e.g. 7.6.2). See [[AppSuite:Version_Support_Committment]] for the currently supported versions. Also, make sure that your repository configuration points at the versioned base installation repositories (instead of using the <tt>stable</tt> symlink.)

deb http://CUSTOMERID:PASSWORD@software.open-xchange.com/products/appsuite/VERSION/appsuiteui/updates/DebianWheezy/ /
deb http://CUSTOMERID:PASSWORD@software.open-xchange.com/products/appsuite/VERSION/backend/updates/DebianWheezy/ /

deb http://CUSTOMERID:PASSWORD@software.open-xchange.com/products/appsuite/VERSION/appsuiteui/updates/DebianJessie/ /
deb http://CUSTOMERID:PASSWORD@software.open-xchange.com/products/appsuite/VERSION/backend/updates/DebianJessie /

Then run

$ apt-get update
$ apt-get dist-upgrade

If you want to see, what apt-get is going to do without actually doing it, you can run:

$ apt-get dist-upgrade -s

After the new packages are installed, the open-xchange process needs a restart:

<pre>$ /etc/init.d/open-xchange restart</pre>

=== On RPM based distributions ===

==== RHEL6/CentOS6 ====

If you want to update an older version of Open-Xchange App Suite to the latest maintenance release, add the following entry to <tt>/etc/yum.repos.d/ox.repo</tt>. Replace VERSION with the version you are using (e.g. 7.4.2, 7.6.0). See [[AppSuite:Version_Support_Committment]] for the currently supported versions. Also, make sure that your repository configuration points at the versioned base installation repositories (instead of using the <tt>stable</tt> symlink.)

[ox-updates-appsuiteui]
name=Open-Xchange Updates
baseurl=http://CUSTOMERID:PASSWORD@software.open-xchange.com/products/appsuite/VERSION/appsuiteui/updates/RHEL6/
gpgkey=http://software.open-xchange.com/oxbuildkey.pub
enabled=1
gpgcheck=1
metadata_expire=0m

[ox-updates-backend]
name=Open-Xchange Updates
baseurl=http://CUSTOMERID:PASSWORD@software.open-xchange.com/products/appsuite/VERSION/backend/updates/RHEL6/
gpgkey=http://software.open-xchange.com/oxbuildkey.pub
enabled=1
gpgcheck=1
metadata_expire=0m

and run

$ yum update

$ yum upgrade

After the new packages are installed, the open-xchange process needs a restart:

<pre>$ /etc/init.d/open-xchange restart</pre>
(Please, keep the [[#Warnings|warning about database updates]] in mind)

==== RHEL7/CentOS7 ====

If you want to update an older version of Open-Xchange App Suite to the latest maintenance release, add the following entry to <tt>/etc/yum.repos.d/ox.repo</tt>. Replace VERSION with the version you are using (e.g. 7.4.2, 7.6.0). See [[AppSuite:Version_Support_Committment]] for the currently supported versions. Also, make sure that your repository configuration points at the versioned base installation repositories (instead of using the <tt>stable</tt> symlink.)

[ox-updates-appsuiteui]
name=Open-Xchange Updates
baseurl=http://CUSTOMERID:PASSWORD@software.open-xchange.com/products/appsuite/VERSION/appsuiteui/updates/RHEL7/
gpgkey=http://software.open-xchange.com/oxbuildkey.pub
enabled=1
gpgcheck=1
metadata_expire=0m

[ox-updates-backend]
name=Open-Xchange Updates
baseurl=http://CUSTOMERID:PASSWORD@software.open-xchange.com/products/appsuite/VERSION/backend/updates/RHEL7/
gpgkey=http://software.open-xchange.com/oxbuildkey.pub
enabled=1
gpgcheck=1
metadata_expire=0m

and run

$ yum update

$ yum upgrade

After the new packages are installed, the open-xchange process needs a restart:

<pre>$ /etc/init.d/open-xchange restart</pre>
(Please, keep the [[#Warnings|warning about database updates]] in mind)

==== SLES 11 ====

If you want to update an older version of Open-Xchange App Suite to the latest maintenance release, add the following entry to the repository list. Replace VERSION with the version you are using (e.g. 7.4.2, 7.6.0). See [[AppSuite:Version_Support_Committment]] for the currently supported versions. Also, make sure that your repository configuration points at the versioned base installation repositories (instead of using the <tt>stable</tt> symlink.)

$ zypper ar http://CUSTOMERID:PASSWORD@software.open-xchange.com/products/appsuite/VERSION/appsuiteui/updates/SLES11/ OXAPPSUITEUIUPDATES
$ zypper ar http://CUSTOMERID:PASSWORD@software.open-xchange.com/products/appsuite/VERSION/backend/updates/SLES11/ OXBACKENDUPDATES

and run

$ zypper dup -r OXAPPSUITEUIUPDATES
$ zypper dup -r OXBACKENDUPDATES

You might need to run

$ zypper ref

to update the repository metadata before running ''zypper up''.

After the new packages are installed, the open-xchange process needs a restart:

<pre>$ rcopen-xchange restart</pre>
(Please, keep the [[#Warnings|warning about database updates]] in mind.

==== SLES 12 ====

If you want to update an older version of Open-Xchange App Suite to the latest maintenance release, add the following entry to the repository list. Replace VERSION with the version you are using (e.g. 7.4.2, 7.6.0). See [[AppSuite:Version_Support_Committment]] for the currently supported versions. Also, make sure that your repository configuration points at the versioned base installation repositories (instead of using the <tt>stable</tt> symlink.)

$ zypper ar http://CUSTOMERID:PASSWORD@software.open-xchange.com/products/appsuite/VERSION/appsuiteui/updates/SLE_12/ OXAPPSUITEUIUPDATES
$ zypper ar http://CUSTOMERID:PASSWORD@software.open-xchange.com/products/appsuite/VERSION/backend/updates/SLE_12/ OXBACKENDUPDATES

and run

$ zypper dup -r OXAPPSUITEUIUPDATES
$ zypper dup -r OXBACKENDUPDATES

You might need to run

$ zypper ref

to update the repository metadata before running ''zypper up''.

After the new packages are installed, the open-xchange process needs a restart:

<pre>$ rcopen-xchange restart</pre>
(Please, keep the [[#Warnings|warning about database updates]] in mind.

= Updating UI plugins =

If you update only UI plugins without simultaneously upgrading the core UI packages to a new version, you need to perform an additional step after the update to make the changes visible to users.

The package <tt>open-xchange-appsuite</tt> contains a timestamp which is different in each version. The JavaScript UI in every user's browser will reload all cached code and data whenever this value changes. Since this value does not change when updating only plugin packages, the value must be changed manually with the following command:

$ /opt/open-xchange/sbin/touch-appsuite --timestamp=20170101.123400

If you only have one OX node, the parameter <code>--timestamp</code> can be omitted. It defaults to the current UTC time anyway.

When updating a cluster, the command must use the same timestamp value on every node. Otherwise, browsers will clear their cache and re-download the entire UI code every time load balancer sends them to a different node (i.e. on almost every login). This is also why the timestamps can't be updated automatically: a node doesn't know which value other nodes will pick, which of them are part of the same update, etc.

The quickest way to obtain a value for the timestamp parameter is to run the command with the parameter <code>-h</code>. It will then print a help message with the current time as an example. For automation, you can use the output of

date -u +%Y%m%d.%H%M%S

Again: this value must be computed once, and then the same value must be used on every node in the cluster.

This step can be performed on each node separately after each node has been updated, or on all nodes at once after all nodes have been updated. If Apache has its own set of dedicated nodes, the <tt>touch-appsuite</tt> call must be performed on the web server nodes, after all corresponding <tt><plugin>-static</tt> packages have been updated.

[[Category: OX7]]
[[Category: AppSuite]]

[[Category: Administrator]]

AppSuite:MiddlewareStartup

2017-04-26T16:52:24Z

Marens: /* Format >= 7.8.4 */

== Systemd ==

==== Service definition ====
The default service definition as shipped by Open-Xchange is rather concise and easy to read. You can check by looking at the service file that is installed to either /usr/lib/systemd or /lib/systemd depending on your distribution of choice.

singlenode:~ # cat /usr/lib/systemd/system/open-xchange.service
[Unit]
After=remote-fs.target
After=time-sync.target ypbind.service sendmail.service cyrus.service

[Service]
User=open-xchange
PermissionsStartOnly=true
TimeoutStartSec=0
ExecStartPre=/opt/open-xchange/sbin/triggerupdatethemes -u
ExecStart=/opt/open-xchange/sbin/open-xchange
ExecStop=/opt/open-xchange/sbin/shutdown -w
ExecReload=/opt/open-xchange/sbin/triggerreloadconfiguration -d
KillMode=process
LimitNOFILE=65536
LimitNPROC=65536

[Install]
WantedBy=multi-user.target

==== Drop-in configs ====
Drop in configs allow administrators to easily adapt and override default service unit files. So if you want to change the default limits or add additional limits have a look at our default example drop-in config which is located at /etc/systemd/system/open-xchange.service.d/limits.conf

singlenode:~ # cat /etc/systemd/system/open-xchange.service.d/limits.conf
# Override and add options in this file
# See systemd.exec(5) for other limits

[Service]
#LimitNPROC=65536
#LimitNOFILE=65536

The [https://www.freedesktop.org/software/systemd/man/systemd.exec.html systemd.exec] documentation shows a whole lot of options that can be used by admins to adapt the default service to their specific needs.

 
[[File:mw_start_sysd.png|thumb|Service startup using systemd|left|1000px]]
 

''Note:'' The technical implementation of systemd differs as service definition and config aren't read every time during service start but the changes need to be reread with ''systemctl daemon-reload''. The above is just a simplyfied version to explain the general concept.

== System V / Upstart ==

==== Service definition ====

The default service definition as shipped by Open-Xchange can be found at /etc/init.d/open-xchange and will differ slightly depending on the distro in use.

 
[[File:mw_start_sysv.png|thumb|Service startup using upstart|left|1000px]]
 
== Apply Resource limits ==

As you can see in the diagrams above one of the differences is that:

* systemd reads all the infos wrt. resource limitations from the original service file and then applies possible further restrictions from the drop-in configs (steps 2 and 8 in the diagram ''Service startup using systemd'') and limits configured in /opt/open-xchange/etc/ox-scriptconf.sh aren't considered anylonger
* the older System V style init has to source /opt/open-xchange/etc/ox-scriptconf.sh to be able to read the configs for the maximum number of processes (NPROC) and files(NRFILES) that the middleware shall be allowed to create/open (steps 3,4 and 15 in the diagram ''Service startup using upstart'').

=== Background Infos ===

Several ways exist to restrict resources on a linux system from a global level down to user/groups or even shells and the processes started by them.

==== Sysctl ====
Sysctl is used to modify kernel parameters at runtime. E.g. to set the maximum number of files

$ sysctl -w fs.file-max=100000

To permanently set them append to the main configuration file and reload the settings

$ echo fs.file-max=100000 >> /etc/sysctl.conf
$ sysctl -p

More infos can be found via '''man sysctl'''

==== Limits.conf ====

Allows to restrict resources an a global, group or user level. E.g:

$ cat /etc/security/limits.d/90-nproc.conf
# Default limit for number of user's processes to prevent
# accidental fork bombs.
# See rhbz #432903 for reasoning.

* soft nproc 1024

From '''man limits.conf''':
<blockquote>
''Also, please note that all limit settings are set per login. They are not global, nor are they permanent; existing only for the duration of the session.''
</blockquote>

The limits per login are applied via the ''pam'' stack. See '''man pam''' and '''man pam_limits''' for more details. As those limits are bound to sessions they don't affect most daemons started by our supported init systems or init utils. Most state that they are ignored by design, see [https://bugs.launchpad.net/ubuntu/+source/upstart/+bug/938669/comments/4 upstart], [https://bugzilla.redhat.com/show_bug.cgi?id=754285#c1 systemd] and [https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=302079 start-stop-daemon]

===== Ulimit =====
From '''man bash'''
<blockquote>
ulimit [-HSTabcdefilmnpqrstuvx [limit]]
Provides control over the resources available to the shell and to processes started by it, on systems that allow such control.
</blockquote>
This is what we use in our System V compatible init scripts to increase resources for the open-xchange process across multiple distros. Currently only ''the maximum number of processes'' and ''the maximum number of open file descriptors available to a single user'' are increased via ulimit. The values are specified in '''/opt/open-xchange/ox-scriptconf.sh'''

==== Systemd ====

===== Control Groups=====

Control groups should only affect the OX middleware if you create/manage them yourself of if you are using a ''modern'' distribution that already uses systemd as init.

Citing from the kernel cgroup [https://www.kernel.org/doc/Documentation/cgroup-v2.txt documentation]:

<blockquote>
1-2. What is cgroup?

cgroup is a mechanism to organize processes hierarchically and
distribute system resources along the hierarchy in a controlled and
configurable manner.

cgroup is largely composed of two parts - the core and controllers.
cgroup core is primarily responsible for hierarchically organizing
processes. A cgroup controller is usually responsible for
distributing a specific type of system resource along the hierarchy
although there are utility controllers which serve purposes other than
resource distribution.

cgroups form a tree structure and every process in the system belongs
to one and only one cgroup. All threads of a process belong to the
same cgroup. On creation, all processes are put in the cgroup that
the parent process belongs to at the time. A process can be migrated
to another cgroup. Migration of a process doesn't affect already
existing descendant processes.

Following certain structural constraints, controllers may be enabled or
disabled selectively on a cgroup. All controller behaviors are
hierarchical - if a controller is enabled on a cgroup, it affects all
processes which belong to the cgroups consisting the inclusive
sub-hierarchy of the cgroup. When a controller is enabled on a nested
cgroup, it always restricts the resource distribution further. The
restrictions set closer to the root in the hierarchy can not be
overridden from further away.
</blockquote>

So processes are organized into a tree structure of control groups and controllers are responsible for the distribution of resources. So what kind of controllers exist?

<blockquote>
5. Controllers

5-1. CPU

The "cpu" controllers regulates distribution of CPU cycles. This
controller implements weight and absolute bandwidth limit models for
normal scheduling policy and absolute bandwidth allocation model for
realtime scheduling policy.

5-2. Memory

The "memory" controller regulates distribution of memory.
...
While not completely water-tight, all major memory usages by a given
cgroup are tracked so that the total memory consumption can be
accounted and controlled to a reasonable extent.

5-3. IO

The "io" controller regulates the distribution of IO resources. This
controller implements both weight based and absolute bandwidth or IOPS
limit distribution; however, weight based distribution is available
only if cfq-iosched is in use and neither scheme is available for
blk-mq devices.
</blockquote>

The open-xchange service is simply put into the default ''system.slice'' without applying further limits.

singlenode$ systemd-cgls
├─1 /sbin/init
├─system.slice
│ ├─avahi-daemon.service
│ │ ├─501 avahi-daemon: running [singlenode]
│ │ └─514 avahi-daemon: chroot helper
│ ├─console-kit-daemon.service
│ │ └─16164 /usr/sbin/console-kit-daemon --no-daemon
│ ├─dbus.service
│ │ └─508 /usr/bin/dbus-daemon --system --address=systemd: --nofork --nopidfile --systemd-activation
│ ├─munin-node.service
│ │ └─4290 /usr/bin/perl -wT /usr/sbin/munin-node
│ ├─open-xchange.service
│ │ └─6037 /usr/bin/java -Dsun.net.inetaddr.ttl=3600 -Dnetworkaddress.cache.ttl=3600 -Dnetworkaddress.cache.negative.ttl=10 ...

To check all the details use

singlenode:~ # systemctl show system.slice

==== Limits besides control groups ====
Besides control groups systemd allows you to apply other limits to the execution environment of your service. Here we can apply the limits that would normally be applied via ''limits.conf'' or ''ulimit''. Systemd uses [http://man7.org/linux/man-pages/man2/setrlimit.2.html setrlimit] for this. The options that we set by default are:

* LimitNOFILE
* LimitNPROC

which you have already seen when we took a look at the [[#Service_definiton|service definition]] and [[#Drop-in_config|drop-in configs]]. ''Again'': The complete documentation wrt. resource limit configurations in drop-in configs can be found at [https://www.freedesktop.org/software/systemd/man/systemd.exec.html systemd.exec]

== Verify limits ==

=== System V ===

Get the proper pid of the java process started for the middleware by either checking '''ps faux'''
root 4103 0.0 0.0 66528 1868 pts/0 S 22:05 0:00 su -s /bin/bash open-xchange -c /opt/open-xchange/sbin/open-xchange
494 4109 9.5 19.8 2224644 381312 ? Ssl 22:05 1:12 \_ /usr/bin/java -Dsun.net.inetaddr.ttl=3600 -Dnetworkaddress.cache.ttl=3600

or programmatically
singlenode:~ # pid=$(pgrep -P $(</var/run/open-xchange.pid) java)

and check the limits applied for that process
singlenode:~ # cat /proc/$pid/limits
Limit Soft Limit Hard Limit Units
Max cpu time unlimited unlimited seconds
Max file size unlimited unlimited bytes
Max data size unlimited unlimited bytes
Max stack size 8388608 unlimited bytes
Max core file size 0 unlimited bytes
Max resident set unlimited unlimited bytes
Max processes 65536 65536 processes
Max open files 65536 65536 files
Max locked memory 65536 65536 bytes
Max address space unlimited unlimited bytes
Max file locks unlimited unlimited locks
Max pending signals 24254 24254 signals
Max msgqueue size 819200 819200 bytes
Max nice priority 0 0
Max realtime priority 0 0
Max realtime timeout unlimited unlimited us

=== Systemd ===
singlenode:~ # systemctl show open-xchange | grep Limit
StartLimitInterval=10000000
StartLimitBurst=5
StartLimitAction=none
MemoryLimit=18446744073709551615
LimitCPU=18446744073709551615
LimitFSIZE=18446744073709551615
LimitDATA=18446744073709551615
LimitSTACK=18446744073709551615
LimitCORE=18446744073709551615
LimitRSS=18446744073709551615
LimitNOFILE=65536
LimitAS=18446744073709551615
LimitNPROC=65536
LimitMEMLOCK=65536
LimitLOCKS=18446744073709551615
LimitSIGPENDING=19827
LimitMSGQUEUE=819200
LimitNICE=0
LimitRTPRIO=0
LimitRTTIME=18446744073709551615

== Evaluate JVM commandline options ==

Before starting Java and the OSGi framework with all the Open-Xchange middleware bundles we have to evaluate several important JVM commandline options that influence the performance and stability of the Open-Xchange middleware stack. These commandline options are specified in '''/opt/open-xchange/etc/ox-scriptconf.sh'''

=== Format < 7.8.4 ===
For version up to 7.8.3 the JVM configuration options were all specified via a single property '''JAVA_XTRAOPTS''' which resulted in a rather long configuration string that was hard to read and maintain.

JAVA_XTRAOPTS="-Dsun.net.inetaddr.ttl=3600 -Dnetworkaddress.cache.ttl=3600 -Dnetworkaddress.cache.negative.ttl=10 \
-Dlogback.threadlocal.put.duplicate=false -server -Djava.awt.headless=true -Xmx512M -XX:MaxPermSize=256m -XX:+UseConcMarkSweepGC \
-XX:+UseParNewGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:NewRatio=3 -XX:+UseTLAB \
-Dosgi.compatibility.bootdelegation=false -XX:-OmitStackTraceInFastThrow"

During the startup of the open-xchange service (step 11 of diagram Service startup using systemd and step 10 of diagram Service startup using upstart) this JAVA_XTRAOPTS string goes through a basic validity check and is then passed to the JVM.

=== Format >= 7.8.4 ===
Beginning from version 7.8.4 the options are split up and sorted into different categories like:

JAVA_OPTS_GC="-XX:+UseConcMarkSweepGC -XX:+UseParNewGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:NewRatio=3 -XX:+DisableExplicitGC"
JAVA_OPTS_LOG="-Dlogback.threadlocal.put.duplicate=false -XX:-OmitStackTraceInFastThrow"
JAVA_OPTS_MEM="-XX:MaxHeapSize=512M -XX:MaxPermSize=256M -XX:+UseTLAB"
JAVA_OPTS_NET="-Dsun.net.inetaddr.ttl=3600 -Dnetworkaddress.cache.ttl=3600 -Dnetworkaddress.cache.negative.ttl=10"
JAVA_OPTS_OSGI="-Dosgi.compatibility.bootdelegation=false"
JAVA_OPTS_SERVER="-server -Djava.awt.headless=true"

JAVA_OPTS_OTHER=""

#JAVA_OPTS_DEBUG="-XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/mnt/heapdump -Xloggc:/var/log/open-xchange/gc.log -verbose:gc -XX:+PrintGCDateStamps \
-XX:+PrintHeapAtGC -XX:+PrintGCApplicationStoppedTime -XX:+PrintTenuringDistribution"

which is much easier to read and maintain. Furthermore we added a useful '''JAVA_OPTS_DEBUG''' configuration parameter that comes preconfigured but unused/commented so that customers can quickly enable the most commonly used debugging options in case they have to provide detailed system information to Open-Xchange.

During the startup of the open-xchange service (step 11 of diagram ''Service startup using systemd'' and step 10 of diagram ''Service startup using upstart'') all uncommented JAVA_OPTS_* categories will then be evaluated and combined again into a single string of commandline options that can then be passed to the JVM.

=== Upgrade procedure ===
During the upgrade '''7.8.3-revX -> 7.8.4-revY''' existing JAVA_XTRAOPTS will automatically be split up and migrated to the new categories approach. A backup of the file (ox-scriptconf.sh.timestamp) will be created and admins are urged to check for proper migration before deleting the backup file like it's already done with *.dpkg-dist or *.rpmnew files on Debian or RHEL/SLES distributions.

== Encoding detection during JVM startup ==

=== Background Infos ===

From ''man 7 locale''

DESCRIPTION
A locale is a set of language and cultural rules. These cover aspects such as lan‐
guage for messages, different character sets, lexicographic conventions, and so on.
A program needs to be able to determine its locale and act accordingly to be portable
to different cultures.

The format to specify a system locale is: '''''language[_territory][.codeset][@modifier]'''''.

Where all of

* en_US.UTF-8
* en_US.utf8
* en_US.utf-8
* en_US.UTF_8

will result in a valid UTF-8 charmap for the english language with locale categories configured as

LANG=en_US.UTF-8
LC_CTYPE="en_US.UTF-8"
LC_NUMERIC="en_US.UTF-8"
LC_TIME="en_US.UTF-8"
LC_COLLATE="en_US.UTF-8"
LC_MONETARY="en_US.UTF-8"
LC_MESSAGES="en_US.UTF-8"
LC_PAPER="en_US.UTF-8"
LC_NAME="en_US.UTF-8"
LC_ADDRESS="en_US.UTF-8"
LC_TELEPHONE="en_US.UTF-8"
LC_MEASUREMENT="en_US.UTF-8"
LC_IDENTIFICATION="en_US.UTF-8"
LC_ALL=

See for more infos: locale(1), locale(7), [https://en.wikipedia.org/wiki/List_of_ISO_639-1_codes language codes], [https://en.wikipedia.org/wiki/ISO_3166-1#Current_codes country codes] and [https://en.wikipedia.org/wiki/Character_encoding character encodings]

==== JVM startup ====

The default file.encoding detection is done at JVM startup based upon LC_CTYPE which makes sense as it's documented as:

CODESET (LC_CTYPE)
Return a string with the name of the character encoding used in the selected locale, such as "UTF-8", "ISO-8859-1",
or "ANSI_X3.4-1968" (better known as US-ASCII). This is the same string that you get with "locale charmap".
For a list of character encoding names, try "locale -m", cf. locale(1).

Furthermore this influences the defaultCharset selection of the JVM.

/**
* Returns the default charset of this Java virtual machine.
*
* The default charset is determined during virtual-machine startup and
* typically depends upon the locale and charset of the underlying
* operating system.
*
* @return A charset object for the default charset
*
* @since 1.5
*/
public static Charset defaultCharset() {
if (defaultCharset == null) {
synchronized (Charset.class) {
String csn = AccessController.doPrivileged(
new GetPropertyAction("file.encoding"));
Charset cs = lookup(csn);
if (cs != null)
defaultCharset = cs;
else
defaultCharset = forName("UTF-8");
}
}
return defaultCharset;
}

Although we try to specify valid unicode encodings for every IO operation the Open-Xchange middleware does, some parts of the JVM don't allow to specify the encoding to use and simply falls back to the detected default encoding. That's why '''admins have to make sure to provide a proper unicode environment for the middleware service during every service start''' or use documented workarounds provided by Open-Xchange (e.g. [http://documentation.open-xchange.com/components/middleware/config/develop/index.html#mode=features&feature=Password%20Change%20Script com.openexchange.passwordchange.script.base64]: Encoded strings as Base64 to circumvent character encoding issues on improperly configured distributions).

See the documentation of your linux distribution on how to configure a proper unicode locale.

=== Query which file encoding was detected by the JVM during startup ===
[[File:visualvm-file.encoding.png|thumb|Querying the JVM via visualvm|right|300px]]

The file encoding that was detected by the JVM during startup can be queried via JMX. You can use graphical management tools like [https://visualvm.github.io/ visualvm], query our [[Jolokia]] JMX bridge via http or just use the commandline like

/opt/open-xchange/sbin/showruntimestats -r | sed -ne 's/.*${key=file.encoding, value=[^}]*}$.*/\1/p'
{key=file.encoding, value='''UTF-8'''}

 

=== Locale passing from client to server via SSH ===

The following setting of sshd allows clients to specify which locale to use on the server and might lead to some confusion or unexpected behaviour when checking the system locale or even to the usage of broken locales during middleware startup if your client sends misconfigured or missing locale configuration to the server.

grep AcceptEnv /etc/ssh/sshd_config
AcceptEnv LANG LC_*

For example when i connect to the machine my local german locale is sent to the server

[vagrant@showcase-community-centos-7 ~]$ locale -v
LANG=de_DE.UTF-8
LC_CTYPE="de_DE.UTF-8"
LC_NUMERIC=de_DE.UTF-8
LC_TIME=de_DE.UTF-8
LC_COLLATE="de_DE.UTF-8"
LC_MONETARY=de_DE.UTF-8
LC_MESSAGES="de_DE.UTF-8"
LC_PAPER=de_DE.UTF-8
LC_NAME=de_DE.UTF-8
LC_ADDRESS=de_DE.UTF-8
LC_TELEPHONE=de_DE.UTF-8
LC_MEASUREMENT=de_DE.UTF-8
LC_IDENTIFICATION=de_DE.UTF-8
LC_ALL=

But after specifying that i want a new login shell the default locale configured for that user/system will be used instead!

[root@showcase-community-centos-7 vagrant]# su -s /bin/bash open-xchange -l
-bash-4.2$ locale -v
LANG=en_US.UTF-8
LC_CTYPE="en_US.UTF-8"
LC_NUMERIC="en_US.UTF-8"
LC_TIME="en_US.UTF-8"
LC_COLLATE="en_US.UTF-8"
LC_MONETARY="en_US.UTF-8"
LC_MESSAGES="en_US.UTF-8"
LC_PAPER="en_US.UTF-8"
LC_NAME="en_US.UTF-8"
LC_ADDRESS="en_US.UTF-8"
LC_TELEPHONE="en_US.UTF-8"
LC_MEASUREMENT="en_US.UTF-8"
LC_IDENTIFICATION="en_US.UTF-8"
LC_ALL=

=== Verify that your system is configured with a proper unicode locale for OX ===

Assuming you haven't reconfigured the locale of your root user

[root@showcase-community-centos-7 vagrant]# locale charmap
UTF-8
[root@showcase-community-centos-7 vagrant]#

== Warnings during early start ==

Early warnings before the start of the JVM and related logging libraries like logback and logstash forwarding are redirected to '''/var/log/open-xchange/open-xchange-console.log'''.

=== Limits ===
If the open-xchange startup script isn't able to set the proper limits on CentOS 6 due to e.g. hard limits being enforced via pam_limits you'll see the following warning in the console log during startup

singlenode open-xchange # cat open-xchange-console.log
/opt/open-xchange/sbin/open-xchange: line 115: ulimit: max user processes: cannot modify limit: Operation not permitted

=== Locale ===
If the open-xchange startup script isn't able to detect a unicode locale you'll see a warning like the following (the current charmap might differ)

WARNING: Couldn't detect proper unicode charmap in locale category LC_CTYPE.
WARNING: Current charmap: "ANSI_X3.4-1968" might cause encoding problems.

== Summary: Open-Xchange middleware on specific distros ==
The support for the mentioned mechanism of resource control differ depending on the distribution and the init system in use.

=== Debian 7 ===
;Init
: System V style
;OX Configurable Limits/Defaults
: nofile, nproc

The mentioned limits can be configured via '''/opt/open-xchange/etc/ox-scriptconf.sh'''. The limits are applied via ulimit in the service's init script. The open-xchange service is finally started via start-stop-daemon which doesn't doesn't consider '''/etc/security/limits.*'''

=== RHEL 6 / CentOS 6 ===
;Init
: Upstart, System V compatible
;OX Configurable Limits/Defaults
: nofile, nproc

The mentioned limits can be configured via '''/opt/open-xchange/etc/ox-scriptconf.sh'''. The limits are applied via ulimit in the service's init script. Furthermore as the open-xchange service is finally started via '''su ... open-xchange''' on this distro a user session is opened via su/pam and the default CentOS pam config reads the '''/etc/security/limits.*''' configuration by loading the pam stack like:

: '''/etc/pam.d/su'''
:: '''-> /etc/pam.d/system-auth'''
::: '''-> pam_limits.so'''

If NPROC isn't configured for the open-xchange-server it's restricted to '''1024''' globally by default to prevent accidental fork bombs, see '''/etc/security/limits.d/90-nproc.conf''' which can result in severe problems modern multithreaded applications.

=== RHEL 7 / CentOS 7 / Debian 8 / SLE 12 ===
;Init
: Systemd
;OX Configurable Limits/Defaults
: nofile, nproc

For systemd the default limits are configured directly in the service's unit file that is shipped by OX and located at '''/usr/lib/systemd/system/open-xchange.service'''. The drop-in config to override or extend the default unit file is located at '''/etc/systemd/system/open-xchange.service.d/limits.conf'''. [https://www.freedesktop.org/software/systemd/man/systemd.exec.html Systemd.exec] shows a whole lot of options that can be used by admins to adapt the default service to their specific needs.

----

[[File:diagram_sources.zip]]

AppSuite:MiddlewareStartup

2017-04-26T08:20:05Z

Marens: /* Verify that your system is configured with a proper unicode locale for OX */

== Systemd ==

==== Service definition ====
The default service definition as shipped by Open-Xchange is rather concise and easy to read. You can check by looking at the service file that is installed to either /usr/lib/systemd or /lib/systemd depending on your distribution of choice.

singlenode:~ # cat /usr/lib/systemd/system/open-xchange.service
[Unit]
After=remote-fs.target
After=time-sync.target ypbind.service sendmail.service cyrus.service

[Service]
User=open-xchange
PermissionsStartOnly=true
TimeoutStartSec=0
ExecStartPre=/opt/open-xchange/sbin/triggerupdatethemes -u
ExecStart=/opt/open-xchange/sbin/open-xchange
ExecStop=/opt/open-xchange/sbin/shutdown -w
ExecReload=/opt/open-xchange/sbin/triggerreloadconfiguration -d
KillMode=process
LimitNOFILE=65536
LimitNPROC=65536

[Install]
WantedBy=multi-user.target

==== Drop-in configs ====
Drop in configs allow administrators to easily adapt and override default service unit files. So if you want to change the default limits or add additional limits have a look at our default example drop-in config which is located at /etc/systemd/system/open-xchange.service.d/limits.conf

singlenode:~ # cat /etc/systemd/system/open-xchange.service.d/limits.conf
# Override and add options in this file
# See systemd.exec(5) for other limits

[Service]
#LimitNPROC=65536
#LimitNOFILE=65536

The [https://www.freedesktop.org/software/systemd/man/systemd.exec.html systemd.exec] documentation shows a whole lot of options that can be used by admins to adapt the default service to their specific needs.

 
[[File:mw_start_sysd.png|thumb|Service startup using systemd|left|1000px]]
 

''Note:'' The technical implementation of systemd differs as service definition and config aren't read every time during service start but the changes need to be reread with ''systemctl daemon-reload''. The above is just a simplyfied version to explain the general concept.

== System V / Upstart ==

==== Service definition ====

The default service definition as shipped by Open-Xchange can be found at /etc/init.d/open-xchange and will differ slightly depending on the distro in use.

 
[[File:mw_start_sysv.png|thumb|Service startup using upstart|left|1000px]]
 
== Apply Resource limits ==

As you can see in the diagrams above one of the differences is that:

* systemd reads all the infos wrt. resource limitations from the original service file and then applies possible further restrictions from the drop-in configs (steps 2 and 8 in the diagram ''Service startup using systemd'') and limits configured in /opt/open-xchange/etc/ox-scriptconf.sh aren't considered anylonger
* the older System V style init has to source /opt/open-xchange/etc/ox-scriptconf.sh to be able to read the configs for the maximum number of processes (NPROC) and files(NRFILES) that the middleware shall be allowed to create/open (steps 3,4 and 15 in the diagram ''Service startup using upstart'').

=== Background Infos ===

Several ways exist to restrict resources on a linux system from a global level down to user/groups or even shells and the processes started by them.

==== Sysctl ====
Sysctl is used to modify kernel parameters at runtime. E.g. to set the maximum number of files

$ sysctl -w fs.file-max=100000

To permanently set them append to the main configuration file and reload the settings

$ echo fs.file-max=100000 >> /etc/sysctl.conf
$ sysctl -p

More infos can be found via '''man sysctl'''

==== Limits.conf ====

Allows to restrict resources an a global, group or user level. E.g:

$ cat /etc/security/limits.d/90-nproc.conf
# Default limit for number of user's processes to prevent
# accidental fork bombs.
# See rhbz #432903 for reasoning.

* soft nproc 1024

From '''man limits.conf''':
<blockquote>
''Also, please note that all limit settings are set per login. They are not global, nor are they permanent; existing only for the duration of the session.''
</blockquote>

The limits per login are applied via the ''pam'' stack. See '''man pam''' and '''man pam_limits''' for more details. As those limits are bound to sessions they don't affect most daemons started by our supported init systems or init utils. Most state that they are ignored by design, see [https://bugs.launchpad.net/ubuntu/+source/upstart/+bug/938669/comments/4 upstart], [https://bugzilla.redhat.com/show_bug.cgi?id=754285#c1 systemd] and [https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=302079 start-stop-daemon]

===== Ulimit =====
From '''man bash'''
<blockquote>
ulimit [-HSTabcdefilmnpqrstuvx [limit]]
Provides control over the resources available to the shell and to processes started by it, on systems that allow such control.
</blockquote>
This is what we use in our System V compatible init scripts to increase resources for the open-xchange process across multiple distros. Currently only ''the maximum number of processes'' and ''the maximum number of open file descriptors available to a single user'' are increased via ulimit. The values are specified in '''/opt/open-xchange/ox-scriptconf.sh'''

==== Systemd ====

===== Control Groups=====

Control groups should only affect the OX middleware if you create/manage them yourself of if you are using a ''modern'' distribution that already uses systemd as init.

Citing from the kernel cgroup [https://www.kernel.org/doc/Documentation/cgroup-v2.txt documentation]:

<blockquote>
1-2. What is cgroup?

cgroup is a mechanism to organize processes hierarchically and
distribute system resources along the hierarchy in a controlled and
configurable manner.

cgroup is largely composed of two parts - the core and controllers.
cgroup core is primarily responsible for hierarchically organizing
processes. A cgroup controller is usually responsible for
distributing a specific type of system resource along the hierarchy
although there are utility controllers which serve purposes other than
resource distribution.

cgroups form a tree structure and every process in the system belongs
to one and only one cgroup. All threads of a process belong to the
same cgroup. On creation, all processes are put in the cgroup that
the parent process belongs to at the time. A process can be migrated
to another cgroup. Migration of a process doesn't affect already
existing descendant processes.

Following certain structural constraints, controllers may be enabled or
disabled selectively on a cgroup. All controller behaviors are
hierarchical - if a controller is enabled on a cgroup, it affects all
processes which belong to the cgroups consisting the inclusive
sub-hierarchy of the cgroup. When a controller is enabled on a nested
cgroup, it always restricts the resource distribution further. The
restrictions set closer to the root in the hierarchy can not be
overridden from further away.
</blockquote>

So processes are organized into a tree structure of control groups and controllers are responsible for the distribution of resources. So what kind of controllers exist?

<blockquote>
5. Controllers

5-1. CPU

The "cpu" controllers regulates distribution of CPU cycles. This
controller implements weight and absolute bandwidth limit models for
normal scheduling policy and absolute bandwidth allocation model for
realtime scheduling policy.

5-2. Memory

The "memory" controller regulates distribution of memory.
...
While not completely water-tight, all major memory usages by a given
cgroup are tracked so that the total memory consumption can be
accounted and controlled to a reasonable extent.

5-3. IO

The "io" controller regulates the distribution of IO resources. This
controller implements both weight based and absolute bandwidth or IOPS
limit distribution; however, weight based distribution is available
only if cfq-iosched is in use and neither scheme is available for
blk-mq devices.
</blockquote>

The open-xchange service is simply put into the default ''system.slice'' without applying further limits.

singlenode$ systemd-cgls
├─1 /sbin/init
├─system.slice
│ ├─avahi-daemon.service
│ │ ├─501 avahi-daemon: running [singlenode]
│ │ └─514 avahi-daemon: chroot helper
│ ├─console-kit-daemon.service
│ │ └─16164 /usr/sbin/console-kit-daemon --no-daemon
│ ├─dbus.service
│ │ └─508 /usr/bin/dbus-daemon --system --address=systemd: --nofork --nopidfile --systemd-activation
│ ├─munin-node.service
│ │ └─4290 /usr/bin/perl -wT /usr/sbin/munin-node
│ ├─open-xchange.service
│ │ └─6037 /usr/bin/java -Dsun.net.inetaddr.ttl=3600 -Dnetworkaddress.cache.ttl=3600 -Dnetworkaddress.cache.negative.ttl=10 ...

To check all the details use

singlenode:~ # systemctl show system.slice

==== Limits besides control groups ====
Besides control groups systemd allows you to apply other limits to the execution environment of your service. Here we can apply the limits that would normally be applied via ''limits.conf'' or ''ulimit''. Systemd uses [http://man7.org/linux/man-pages/man2/setrlimit.2.html setrlimit] for this. The options that we set by default are:

* LimitNOFILE
* LimitNPROC

which you have already seen when we took a look at the [[#Service_definiton|service definition]] and [[#Drop-in_config|drop-in configs]]. ''Again'': The complete documentation wrt. resource limit configurations in drop-in configs can be found at [https://www.freedesktop.org/software/systemd/man/systemd.exec.html systemd.exec]

== Verify limits ==

=== System V ===

Get the proper pid of the java process started for the middleware by either checking '''ps faux'''
root 4103 0.0 0.0 66528 1868 pts/0 S 22:05 0:00 su -s /bin/bash open-xchange -c /opt/open-xchange/sbin/open-xchange
494 4109 9.5 19.8 2224644 381312 ? Ssl 22:05 1:12 \_ /usr/bin/java -Dsun.net.inetaddr.ttl=3600 -Dnetworkaddress.cache.ttl=3600

or programmatically
singlenode:~ # pid=$(pgrep -P $(</var/run/open-xchange.pid) java)

and check the limits applied for that process
singlenode:~ # cat /proc/$pid/limits
Limit Soft Limit Hard Limit Units
Max cpu time unlimited unlimited seconds
Max file size unlimited unlimited bytes
Max data size unlimited unlimited bytes
Max stack size 8388608 unlimited bytes
Max core file size 0 unlimited bytes
Max resident set unlimited unlimited bytes
Max processes 65536 65536 processes
Max open files 65536 65536 files
Max locked memory 65536 65536 bytes
Max address space unlimited unlimited bytes
Max file locks unlimited unlimited locks
Max pending signals 24254 24254 signals
Max msgqueue size 819200 819200 bytes
Max nice priority 0 0
Max realtime priority 0 0
Max realtime timeout unlimited unlimited us

=== Systemd ===
singlenode:~ # systemctl show open-xchange | grep Limit
StartLimitInterval=10000000
StartLimitBurst=5
StartLimitAction=none
MemoryLimit=18446744073709551615
LimitCPU=18446744073709551615
LimitFSIZE=18446744073709551615
LimitDATA=18446744073709551615
LimitSTACK=18446744073709551615
LimitCORE=18446744073709551615
LimitRSS=18446744073709551615
LimitNOFILE=65536
LimitAS=18446744073709551615
LimitNPROC=65536
LimitMEMLOCK=65536
LimitLOCKS=18446744073709551615
LimitSIGPENDING=19827
LimitMSGQUEUE=819200
LimitNICE=0
LimitRTPRIO=0
LimitRTTIME=18446744073709551615

== Evaluate JVM commandline options ==

Before starting Java and the OSGi framework with all the Open-Xchange middleware bundles we have to evaluate several important JVM commandline options that influence the performance and stability of the Open-Xchange middleware stack. These commandline options are specified in '''/opt/open-xchange/etc/ox-scriptconf.sh'''

=== Format < 7.8.4 ===
For version up to 7.8.3 the JVM configuration options were all specified via a single property '''JAVA_XTRAOPTS''' which resulted in a rather long configuration string that was hard to read and maintain.

JAVA_XTRAOPTS="-Dsun.net.inetaddr.ttl=3600 -Dnetworkaddress.cache.ttl=3600 -Dnetworkaddress.cache.negative.ttl=10 \
-Dlogback.threadlocal.put.duplicate=false -server -Djava.awt.headless=true -Xmx512M -XX:MaxPermSize=256m -XX:+UseConcMarkSweepGC \
-XX:+UseParNewGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:NewRatio=3 -XX:+UseTLAB \
-Dosgi.compatibility.bootdelegation=false -XX:-OmitStackTraceInFastThrow"

During the startup of the open-xchange service (step 11 of diagram Service startup using systemd and step 10 of diagram Service startup using upstart) this JAVA_XTRAOPTS string goes through a basic validity check and is then passed to the JVM.

=== Format >= 7.8.4 ===
Beginning from version 7.8.4 the options are split up and sorted into different categories like:

JAVA_OPTS_GC="-XX:+UseConcMarkSweepGC -XX:+UseParNewGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:NewRatio=3 -XX:+DisableExplicitGC"
JAVA_OPTS_LOG="-Dlogback.threadlocal.put.duplicate=false -XX:-OmitStackTraceInFastThrow"
JAVA_OPTS_MEM="-XX:MaxHeapSize=512M -XX:MaxPermSize=256M -XX:+UseTLAB"
JAVA_OPTS_NET="-Dsun.net.inetaddr.ttl=3600 -Dnetworkaddress.cache.ttl=3600 -Dnetworkaddress.cache.negative.ttl=10"
JAVA_OPTS_OSGI="-Dosgi.compatibility.bootdelegation=false"
JAVA_OPTS_SERVER="-server -Djava.awt.headless=true"

JAVA_OPTS_OTHER=""

#JAVA_OPTS_DEBUG="-XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/mnt/heapdump -Xloggc:/var/log/open-xchange/gc.log -verbose:gc -XX:+PrintGCDateStamps \
-XX:+PrintHeapAtGC -XX:+PrintGCApplicationStoppedTime -XX:+PrintTenuringDistribution"

which is much easier to read an maintain. Furthermore we added a useful '''JAVA_OPTS_DEBUG''' configuration parameter that comes preconfigured but unused/commented so that customers can quickly enable the most commonly used debugging options in case they have to provide detailed system information to Open-Xchange.

During the startup of the open-xchange service (step 11 of diagram ''Service startup using systemd'' and step 10 of diagram ''Service startup using upstart'') all uncommented JAVA_OPTS_* categories will then be evaluated and combined again into a single string of commandline options that can then be passed to the JVM.

=== Upgrade procedure ===
During the upgrade '''7.8.3-revX -> 7.8.4-revY''' existing JAVA_XTRAOPTS will automatically be split up and migrated to the new categories approach. A backup of the file (ox-scriptconf.sh.timestamp) will be created and admins are urged to check for proper migration before deleting the backup file like it's already done with *.dpkg-dist or *.rpmnew files on Debian or RHEL/SLES distributions.

== Encoding detection during JVM startup ==

=== Background Infos ===

From ''man 7 locale''

DESCRIPTION
A locale is a set of language and cultural rules. These cover aspects such as lan‐
guage for messages, different character sets, lexicographic conventions, and so on.
A program needs to be able to determine its locale and act accordingly to be portable
to different cultures.

The format to specify a system locale is: '''''language[_territory][.codeset][@modifier]'''''.

Where all of

* en_US.UTF-8
* en_US.utf8
* en_US.utf-8
* en_US.UTF_8

will result in a valid UTF-8 charmap for the english language with locale categories configured as

LANG=en_US.UTF-8
LC_CTYPE="en_US.UTF-8"
LC_NUMERIC="en_US.UTF-8"
LC_TIME="en_US.UTF-8"
LC_COLLATE="en_US.UTF-8"
LC_MONETARY="en_US.UTF-8"
LC_MESSAGES="en_US.UTF-8"
LC_PAPER="en_US.UTF-8"
LC_NAME="en_US.UTF-8"
LC_ADDRESS="en_US.UTF-8"
LC_TELEPHONE="en_US.UTF-8"
LC_MEASUREMENT="en_US.UTF-8"
LC_IDENTIFICATION="en_US.UTF-8"
LC_ALL=

See for more infos: locale(1), locale(7), [https://en.wikipedia.org/wiki/List_of_ISO_639-1_codes language codes], [https://en.wikipedia.org/wiki/ISO_3166-1#Current_codes country codes] and [https://en.wikipedia.org/wiki/Character_encoding character encodings]

==== JVM startup ====

The default file.encoding detection is done at JVM startup based upon LC_CTYPE which makes sense as it's documented as:

CODESET (LC_CTYPE)
Return a string with the name of the character encoding used in the selected locale, such as "UTF-8", "ISO-8859-1",
or "ANSI_X3.4-1968" (better known as US-ASCII). This is the same string that you get with "locale charmap".
For a list of character encoding names, try "locale -m", cf. locale(1).

Furthermore this influences the defaultCharset selection of the JVM.

/**
* Returns the default charset of this Java virtual machine.
*
* The default charset is determined during virtual-machine startup and
* typically depends upon the locale and charset of the underlying
* operating system.
*
* @return A charset object for the default charset
*
* @since 1.5
*/
public static Charset defaultCharset() {
if (defaultCharset == null) {
synchronized (Charset.class) {
String csn = AccessController.doPrivileged(
new GetPropertyAction("file.encoding"));
Charset cs = lookup(csn);
if (cs != null)
defaultCharset = cs;
else
defaultCharset = forName("UTF-8");
}
}
return defaultCharset;
}

Although we try to specify valid unicode encodings for every IO operation the Open-Xchange middleware does, some parts of the JVM don't allow to specify the encoding to use and simply falls back to the detected default encoding. That's why '''admins have to make sure to provide a proper unicode environment for the middleware service during every service start''' or use documented workarounds provided by Open-Xchange (e.g. [http://documentation.open-xchange.com/components/middleware/config/develop/index.html#mode=features&feature=Password%20Change%20Script com.openexchange.passwordchange.script.base64]: Encoded strings as Base64 to circumvent character encoding issues on improperly configured distributions).

See the documentation of your linux distribution on how to configure a proper unicode locale.

=== Query which file encoding was detected by the JVM during startup ===
[[File:visualvm-file.encoding.png|thumb|Querying the JVM via visualvm|right|300px]]

The file encoding that was detected by the JVM during startup can be queried via JMX. You can use graphical management tools like [https://visualvm.github.io/ visualvm], query our [[Jolokia]] JMX bridge via http or just use the commandline like

/opt/open-xchange/sbin/showruntimestats -r | sed -ne 's/.*${key=file.encoding, value=[^}]*}$.*/\1/p'
{key=file.encoding, value='''UTF-8'''}

 

=== Locale passing from client to server via SSH ===

The following setting of sshd allows clients to specify which locale to use on the server and might lead to some confusion or unexpected behaviour when checking the system locale or even to the usage of broken locales during middleware startup if your client sends misconfigured or missing locale configuration to the server.

grep AcceptEnv /etc/ssh/sshd_config
AcceptEnv LANG LC_*

For example when i connect to the machine my local german locale is sent to the server

[vagrant@showcase-community-centos-7 ~]$ locale -v
LANG=de_DE.UTF-8
LC_CTYPE="de_DE.UTF-8"
LC_NUMERIC=de_DE.UTF-8
LC_TIME=de_DE.UTF-8
LC_COLLATE="de_DE.UTF-8"
LC_MONETARY=de_DE.UTF-8
LC_MESSAGES="de_DE.UTF-8"
LC_PAPER=de_DE.UTF-8
LC_NAME=de_DE.UTF-8
LC_ADDRESS=de_DE.UTF-8
LC_TELEPHONE=de_DE.UTF-8
LC_MEASUREMENT=de_DE.UTF-8
LC_IDENTIFICATION=de_DE.UTF-8
LC_ALL=

But after specifying that i want a new login shell the default locale configured for that user/system will be used instead!

[root@showcase-community-centos-7 vagrant]# su -s /bin/bash open-xchange -l
-bash-4.2$ locale -v
LANG=en_US.UTF-8
LC_CTYPE="en_US.UTF-8"
LC_NUMERIC="en_US.UTF-8"
LC_TIME="en_US.UTF-8"
LC_COLLATE="en_US.UTF-8"
LC_MONETARY="en_US.UTF-8"
LC_MESSAGES="en_US.UTF-8"
LC_PAPER="en_US.UTF-8"
LC_NAME="en_US.UTF-8"
LC_ADDRESS="en_US.UTF-8"
LC_TELEPHONE="en_US.UTF-8"
LC_MEASUREMENT="en_US.UTF-8"
LC_IDENTIFICATION="en_US.UTF-8"
LC_ALL=

=== Verify that your system is configured with a proper unicode locale for OX ===

Assuming you haven't reconfigured the locale of your root user

[root@showcase-community-centos-7 vagrant]# locale charmap
UTF-8
[root@showcase-community-centos-7 vagrant]#

== Warnings during early start ==

Early warnings before the start of the JVM and related logging libraries like logback and logstash forwarding are redirected to '''/var/log/open-xchange/open-xchange-console.log'''.

=== Limits ===
If the open-xchange startup script isn't able to set the proper limits on CentOS 6 due to e.g. hard limits being enforced via pam_limits you'll see the following warning in the console log during startup

singlenode open-xchange # cat open-xchange-console.log
/opt/open-xchange/sbin/open-xchange: line 115: ulimit: max user processes: cannot modify limit: Operation not permitted

=== Locale ===
If the open-xchange startup script isn't able to detect a unicode locale you'll see a warning like the following (the current charmap might differ)

WARNING: Couldn't detect proper unicode charmap in locale category LC_CTYPE.
WARNING: Current charmap: "ANSI_X3.4-1968" might cause encoding problems.

== Summary: Open-Xchange middleware on specific distros ==
The support for the mentioned mechanism of resource control differ depending on the distribution and the init system in use.

=== Debian 7 ===
;Init
: System V style
;OX Configurable Limits/Defaults
: nofile, nproc

The mentioned limits can be configured via '''/opt/open-xchange/etc/ox-scriptconf.sh'''. The limits are applied via ulimit in the service's init script. The open-xchange service is finally started via start-stop-daemon which doesn't doesn't consider '''/etc/security/limits.*'''

=== RHEL 6 / CentOS 6 ===
;Init
: Upstart, System V compatible
;OX Configurable Limits/Defaults
: nofile, nproc

The mentioned limits can be configured via '''/opt/open-xchange/etc/ox-scriptconf.sh'''. The limits are applied via ulimit in the service's init script. Furthermore as the open-xchange service is finally started via '''su ... open-xchange''' on this distro a user session is opened via su/pam and the default CentOS pam config reads the '''/etc/security/limits.*''' configuration by loading the pam stack like:

: '''/etc/pam.d/su'''
:: '''-> /etc/pam.d/system-auth'''
::: '''-> pam_limits.so'''

If NPROC isn't configured for the open-xchange-server it's restricted to '''1024''' globally by default to prevent accidental fork bombs, see '''/etc/security/limits.d/90-nproc.conf''' which can result in severe problems modern multithreaded applications.

=== RHEL 7 / CentOS 7 / Debian 8 / SLE 12 ===
;Init
: Systemd
;OX Configurable Limits/Defaults
: nofile, nproc

For systemd the default limits are configured directly in the service's unit file that is shipped by OX and located at '''/usr/lib/systemd/system/open-xchange.service'''. The drop-in config to override or extend the default unit file is located at '''/etc/systemd/system/open-xchange.service.d/limits.conf'''. [https://www.freedesktop.org/software/systemd/man/systemd.exec.html Systemd.exec] shows a whole lot of options that can be used by admins to adapt the default service to their specific needs.

----

[[File:diagram_sources.zip]]

AppSuite:MiddlewareStartup

2017-04-26T08:19:41Z

Marens: /* Verify that your system is configured with a proper unicode locale for OX */

== Systemd ==

==== Service definition ====
The default service definition as shipped by Open-Xchange is rather concise and easy to read. You can check by looking at the service file that is installed to either /usr/lib/systemd or /lib/systemd depending on your distribution of choice.

singlenode:~ # cat /usr/lib/systemd/system/open-xchange.service
[Unit]
After=remote-fs.target
After=time-sync.target ypbind.service sendmail.service cyrus.service

[Service]
User=open-xchange
PermissionsStartOnly=true
TimeoutStartSec=0
ExecStartPre=/opt/open-xchange/sbin/triggerupdatethemes -u
ExecStart=/opt/open-xchange/sbin/open-xchange
ExecStop=/opt/open-xchange/sbin/shutdown -w
ExecReload=/opt/open-xchange/sbin/triggerreloadconfiguration -d
KillMode=process
LimitNOFILE=65536
LimitNPROC=65536

[Install]
WantedBy=multi-user.target

==== Drop-in configs ====
Drop in configs allow administrators to easily adapt and override default service unit files. So if you want to change the default limits or add additional limits have a look at our default example drop-in config which is located at /etc/systemd/system/open-xchange.service.d/limits.conf

singlenode:~ # cat /etc/systemd/system/open-xchange.service.d/limits.conf
# Override and add options in this file
# See systemd.exec(5) for other limits

[Service]
#LimitNPROC=65536
#LimitNOFILE=65536

The [https://www.freedesktop.org/software/systemd/man/systemd.exec.html systemd.exec] documentation shows a whole lot of options that can be used by admins to adapt the default service to their specific needs.

 
[[File:mw_start_sysd.png|thumb|Service startup using systemd|left|1000px]]
 

''Note:'' The technical implementation of systemd differs as service definition and config aren't read every time during service start but the changes need to be reread with ''systemctl daemon-reload''. The above is just a simplyfied version to explain the general concept.

== System V / Upstart ==

==== Service definition ====

The default service definition as shipped by Open-Xchange can be found at /etc/init.d/open-xchange and will differ slightly depending on the distro in use.

 
[[File:mw_start_sysv.png|thumb|Service startup using upstart|left|1000px]]
 
== Apply Resource limits ==

As you can see in the diagrams above one of the differences is that:

* systemd reads all the infos wrt. resource limitations from the original service file and then applies possible further restrictions from the drop-in configs (steps 2 and 8 in the diagram ''Service startup using systemd'') and limits configured in /opt/open-xchange/etc/ox-scriptconf.sh aren't considered anylonger
* the older System V style init has to source /opt/open-xchange/etc/ox-scriptconf.sh to be able to read the configs for the maximum number of processes (NPROC) and files(NRFILES) that the middleware shall be allowed to create/open (steps 3,4 and 15 in the diagram ''Service startup using upstart'').

=== Background Infos ===

Several ways exist to restrict resources on a linux system from a global level down to user/groups or even shells and the processes started by them.

==== Sysctl ====
Sysctl is used to modify kernel parameters at runtime. E.g. to set the maximum number of files

$ sysctl -w fs.file-max=100000

To permanently set them append to the main configuration file and reload the settings

$ echo fs.file-max=100000 >> /etc/sysctl.conf
$ sysctl -p

More infos can be found via '''man sysctl'''

==== Limits.conf ====

Allows to restrict resources an a global, group or user level. E.g:

$ cat /etc/security/limits.d/90-nproc.conf
# Default limit for number of user's processes to prevent
# accidental fork bombs.
# See rhbz #432903 for reasoning.

* soft nproc 1024

From '''man limits.conf''':
<blockquote>
''Also, please note that all limit settings are set per login. They are not global, nor are they permanent; existing only for the duration of the session.''
</blockquote>

The limits per login are applied via the ''pam'' stack. See '''man pam''' and '''man pam_limits''' for more details. As those limits are bound to sessions they don't affect most daemons started by our supported init systems or init utils. Most state that they are ignored by design, see [https://bugs.launchpad.net/ubuntu/+source/upstart/+bug/938669/comments/4 upstart], [https://bugzilla.redhat.com/show_bug.cgi?id=754285#c1 systemd] and [https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=302079 start-stop-daemon]

===== Ulimit =====
From '''man bash'''
<blockquote>
ulimit [-HSTabcdefilmnpqrstuvx [limit]]
Provides control over the resources available to the shell and to processes started by it, on systems that allow such control.
</blockquote>
This is what we use in our System V compatible init scripts to increase resources for the open-xchange process across multiple distros. Currently only ''the maximum number of processes'' and ''the maximum number of open file descriptors available to a single user'' are increased via ulimit. The values are specified in '''/opt/open-xchange/ox-scriptconf.sh'''

==== Systemd ====

===== Control Groups=====

Control groups should only affect the OX middleware if you create/manage them yourself of if you are using a ''modern'' distribution that already uses systemd as init.

Citing from the kernel cgroup [https://www.kernel.org/doc/Documentation/cgroup-v2.txt documentation]:

<blockquote>
1-2. What is cgroup?

cgroup is a mechanism to organize processes hierarchically and
distribute system resources along the hierarchy in a controlled and
configurable manner.

cgroup is largely composed of two parts - the core and controllers.
cgroup core is primarily responsible for hierarchically organizing
processes. A cgroup controller is usually responsible for
distributing a specific type of system resource along the hierarchy
although there are utility controllers which serve purposes other than
resource distribution.

cgroups form a tree structure and every process in the system belongs
to one and only one cgroup. All threads of a process belong to the
same cgroup. On creation, all processes are put in the cgroup that
the parent process belongs to at the time. A process can be migrated
to another cgroup. Migration of a process doesn't affect already
existing descendant processes.

Following certain structural constraints, controllers may be enabled or
disabled selectively on a cgroup. All controller behaviors are
hierarchical - if a controller is enabled on a cgroup, it affects all
processes which belong to the cgroups consisting the inclusive
sub-hierarchy of the cgroup. When a controller is enabled on a nested
cgroup, it always restricts the resource distribution further. The
restrictions set closer to the root in the hierarchy can not be
overridden from further away.
</blockquote>

So processes are organized into a tree structure of control groups and controllers are responsible for the distribution of resources. So what kind of controllers exist?

<blockquote>
5. Controllers

5-1. CPU

The "cpu" controllers regulates distribution of CPU cycles. This
controller implements weight and absolute bandwidth limit models for
normal scheduling policy and absolute bandwidth allocation model for
realtime scheduling policy.

5-2. Memory

The "memory" controller regulates distribution of memory.
...
While not completely water-tight, all major memory usages by a given
cgroup are tracked so that the total memory consumption can be
accounted and controlled to a reasonable extent.

5-3. IO

The "io" controller regulates the distribution of IO resources. This
controller implements both weight based and absolute bandwidth or IOPS
limit distribution; however, weight based distribution is available
only if cfq-iosched is in use and neither scheme is available for
blk-mq devices.
</blockquote>

The open-xchange service is simply put into the default ''system.slice'' without applying further limits.

singlenode$ systemd-cgls
├─1 /sbin/init
├─system.slice
│ ├─avahi-daemon.service
│ │ ├─501 avahi-daemon: running [singlenode]
│ │ └─514 avahi-daemon: chroot helper
│ ├─console-kit-daemon.service
│ │ └─16164 /usr/sbin/console-kit-daemon --no-daemon
│ ├─dbus.service
│ │ └─508 /usr/bin/dbus-daemon --system --address=systemd: --nofork --nopidfile --systemd-activation
│ ├─munin-node.service
│ │ └─4290 /usr/bin/perl -wT /usr/sbin/munin-node
│ ├─open-xchange.service
│ │ └─6037 /usr/bin/java -Dsun.net.inetaddr.ttl=3600 -Dnetworkaddress.cache.ttl=3600 -Dnetworkaddress.cache.negative.ttl=10 ...

To check all the details use

singlenode:~ # systemctl show system.slice

==== Limits besides control groups ====
Besides control groups systemd allows you to apply other limits to the execution environment of your service. Here we can apply the limits that would normally be applied via ''limits.conf'' or ''ulimit''. Systemd uses [http://man7.org/linux/man-pages/man2/setrlimit.2.html setrlimit] for this. The options that we set by default are:

* LimitNOFILE
* LimitNPROC

which you have already seen when we took a look at the [[#Service_definiton|service definition]] and [[#Drop-in_config|drop-in configs]]. ''Again'': The complete documentation wrt. resource limit configurations in drop-in configs can be found at [https://www.freedesktop.org/software/systemd/man/systemd.exec.html systemd.exec]

== Verify limits ==

=== System V ===

Get the proper pid of the java process started for the middleware by either checking '''ps faux'''
root 4103 0.0 0.0 66528 1868 pts/0 S 22:05 0:00 su -s /bin/bash open-xchange -c /opt/open-xchange/sbin/open-xchange
494 4109 9.5 19.8 2224644 381312 ? Ssl 22:05 1:12 \_ /usr/bin/java -Dsun.net.inetaddr.ttl=3600 -Dnetworkaddress.cache.ttl=3600

or programmatically
singlenode:~ # pid=$(pgrep -P $(</var/run/open-xchange.pid) java)

and check the limits applied for that process
singlenode:~ # cat /proc/$pid/limits
Limit Soft Limit Hard Limit Units
Max cpu time unlimited unlimited seconds
Max file size unlimited unlimited bytes
Max data size unlimited unlimited bytes
Max stack size 8388608 unlimited bytes
Max core file size 0 unlimited bytes
Max resident set unlimited unlimited bytes
Max processes 65536 65536 processes
Max open files 65536 65536 files
Max locked memory 65536 65536 bytes
Max address space unlimited unlimited bytes
Max file locks unlimited unlimited locks
Max pending signals 24254 24254 signals
Max msgqueue size 819200 819200 bytes
Max nice priority 0 0
Max realtime priority 0 0
Max realtime timeout unlimited unlimited us

=== Systemd ===
singlenode:~ # systemctl show open-xchange | grep Limit
StartLimitInterval=10000000
StartLimitBurst=5
StartLimitAction=none
MemoryLimit=18446744073709551615
LimitCPU=18446744073709551615
LimitFSIZE=18446744073709551615
LimitDATA=18446744073709551615
LimitSTACK=18446744073709551615
LimitCORE=18446744073709551615
LimitRSS=18446744073709551615
LimitNOFILE=65536
LimitAS=18446744073709551615
LimitNPROC=65536
LimitMEMLOCK=65536
LimitLOCKS=18446744073709551615
LimitSIGPENDING=19827
LimitMSGQUEUE=819200
LimitNICE=0
LimitRTPRIO=0
LimitRTTIME=18446744073709551615

== Evaluate JVM commandline options ==

Before starting Java and the OSGi framework with all the Open-Xchange middleware bundles we have to evaluate several important JVM commandline options that influence the performance and stability of the Open-Xchange middleware stack. These commandline options are specified in '''/opt/open-xchange/etc/ox-scriptconf.sh'''

=== Format < 7.8.4 ===
For version up to 7.8.3 the JVM configuration options were all specified via a single property '''JAVA_XTRAOPTS''' which resulted in a rather long configuration string that was hard to read and maintain.

JAVA_XTRAOPTS="-Dsun.net.inetaddr.ttl=3600 -Dnetworkaddress.cache.ttl=3600 -Dnetworkaddress.cache.negative.ttl=10 \
-Dlogback.threadlocal.put.duplicate=false -server -Djava.awt.headless=true -Xmx512M -XX:MaxPermSize=256m -XX:+UseConcMarkSweepGC \
-XX:+UseParNewGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:NewRatio=3 -XX:+UseTLAB \
-Dosgi.compatibility.bootdelegation=false -XX:-OmitStackTraceInFastThrow"

During the startup of the open-xchange service (step 11 of diagram Service startup using systemd and step 10 of diagram Service startup using upstart) this JAVA_XTRAOPTS string goes through a basic validity check and is then passed to the JVM.

=== Format >= 7.8.4 ===
Beginning from version 7.8.4 the options are split up and sorted into different categories like:

JAVA_OPTS_GC="-XX:+UseConcMarkSweepGC -XX:+UseParNewGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:NewRatio=3 -XX:+DisableExplicitGC"
JAVA_OPTS_LOG="-Dlogback.threadlocal.put.duplicate=false -XX:-OmitStackTraceInFastThrow"
JAVA_OPTS_MEM="-XX:MaxHeapSize=512M -XX:MaxPermSize=256M -XX:+UseTLAB"
JAVA_OPTS_NET="-Dsun.net.inetaddr.ttl=3600 -Dnetworkaddress.cache.ttl=3600 -Dnetworkaddress.cache.negative.ttl=10"
JAVA_OPTS_OSGI="-Dosgi.compatibility.bootdelegation=false"
JAVA_OPTS_SERVER="-server -Djava.awt.headless=true"

JAVA_OPTS_OTHER=""

#JAVA_OPTS_DEBUG="-XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/mnt/heapdump -Xloggc:/var/log/open-xchange/gc.log -verbose:gc -XX:+PrintGCDateStamps \
-XX:+PrintHeapAtGC -XX:+PrintGCApplicationStoppedTime -XX:+PrintTenuringDistribution"

which is much easier to read an maintain. Furthermore we added a useful '''JAVA_OPTS_DEBUG''' configuration parameter that comes preconfigured but unused/commented so that customers can quickly enable the most commonly used debugging options in case they have to provide detailed system information to Open-Xchange.

During the startup of the open-xchange service (step 11 of diagram ''Service startup using systemd'' and step 10 of diagram ''Service startup using upstart'') all uncommented JAVA_OPTS_* categories will then be evaluated and combined again into a single string of commandline options that can then be passed to the JVM.

=== Upgrade procedure ===
During the upgrade '''7.8.3-revX -> 7.8.4-revY''' existing JAVA_XTRAOPTS will automatically be split up and migrated to the new categories approach. A backup of the file (ox-scriptconf.sh.timestamp) will be created and admins are urged to check for proper migration before deleting the backup file like it's already done with *.dpkg-dist or *.rpmnew files on Debian or RHEL/SLES distributions.

== Encoding detection during JVM startup ==

=== Background Infos ===

From ''man 7 locale''

DESCRIPTION
A locale is a set of language and cultural rules. These cover aspects such as lan‐
guage for messages, different character sets, lexicographic conventions, and so on.
A program needs to be able to determine its locale and act accordingly to be portable
to different cultures.

The format to specify a system locale is: '''''language[_territory][.codeset][@modifier]'''''.

Where all of

* en_US.UTF-8
* en_US.utf8
* en_US.utf-8
* en_US.UTF_8

will result in a valid UTF-8 charmap for the english language with locale categories configured as

LANG=en_US.UTF-8
LC_CTYPE="en_US.UTF-8"
LC_NUMERIC="en_US.UTF-8"
LC_TIME="en_US.UTF-8"
LC_COLLATE="en_US.UTF-8"
LC_MONETARY="en_US.UTF-8"
LC_MESSAGES="en_US.UTF-8"
LC_PAPER="en_US.UTF-8"
LC_NAME="en_US.UTF-8"
LC_ADDRESS="en_US.UTF-8"
LC_TELEPHONE="en_US.UTF-8"
LC_MEASUREMENT="en_US.UTF-8"
LC_IDENTIFICATION="en_US.UTF-8"
LC_ALL=

See for more infos: locale(1), locale(7), [https://en.wikipedia.org/wiki/List_of_ISO_639-1_codes language codes], [https://en.wikipedia.org/wiki/ISO_3166-1#Current_codes country codes] and [https://en.wikipedia.org/wiki/Character_encoding character encodings]

==== JVM startup ====

The default file.encoding detection is done at JVM startup based upon LC_CTYPE which makes sense as it's documented as:

CODESET (LC_CTYPE)
Return a string with the name of the character encoding used in the selected locale, such as "UTF-8", "ISO-8859-1",
or "ANSI_X3.4-1968" (better known as US-ASCII). This is the same string that you get with "locale charmap".
For a list of character encoding names, try "locale -m", cf. locale(1).

Furthermore this influences the defaultCharset selection of the JVM.

/**
* Returns the default charset of this Java virtual machine.
*
* The default charset is determined during virtual-machine startup and
* typically depends upon the locale and charset of the underlying
* operating system.
*
* @return A charset object for the default charset
*
* @since 1.5
*/
public static Charset defaultCharset() {
if (defaultCharset == null) {
synchronized (Charset.class) {
String csn = AccessController.doPrivileged(
new GetPropertyAction("file.encoding"));
Charset cs = lookup(csn);
if (cs != null)
defaultCharset = cs;
else
defaultCharset = forName("UTF-8");
}
}
return defaultCharset;
}

Although we try to specify valid unicode encodings for every IO operation the Open-Xchange middleware does, some parts of the JVM don't allow to specify the encoding to use and simply falls back to the detected default encoding. That's why '''admins have to make sure to provide a proper unicode environment for the middleware service during every service start''' or use documented workarounds provided by Open-Xchange (e.g. [http://documentation.open-xchange.com/components/middleware/config/develop/index.html#mode=features&feature=Password%20Change%20Script com.openexchange.passwordchange.script.base64]: Encoded strings as Base64 to circumvent character encoding issues on improperly configured distributions).

See the documentation of your linux distribution on how to configure a proper unicode locale.

=== Query which file encoding was detected by the JVM during startup ===
[[File:visualvm-file.encoding.png|thumb|Querying the JVM via visualvm|right|300px]]

The file encoding that was detected by the JVM during startup can be queried via JMX. You can use graphical management tools like [https://visualvm.github.io/ visualvm], query our [[Jolokia]] JMX bridge via http or just use the commandline like

/opt/open-xchange/sbin/showruntimestats -r | sed -ne 's/.*${key=file.encoding, value=[^}]*}$.*/\1/p'
{key=file.encoding, value='''UTF-8'''}

 

=== Locale passing from client to server via SSH ===

The following setting of sshd allows clients to specify which locale to use on the server and might lead to some confusion or unexpected behaviour when checking the system locale or even to the usage of broken locales during middleware startup if your client sends misconfigured or missing locale configuration to the server.

grep AcceptEnv /etc/ssh/sshd_config
AcceptEnv LANG LC_*

For example when i connect to the machine my local german locale is sent to the server

[vagrant@showcase-community-centos-7 ~]$ locale -v
LANG=de_DE.UTF-8
LC_CTYPE="de_DE.UTF-8"
LC_NUMERIC=de_DE.UTF-8
LC_TIME=de_DE.UTF-8
LC_COLLATE="de_DE.UTF-8"
LC_MONETARY=de_DE.UTF-8
LC_MESSAGES="de_DE.UTF-8"
LC_PAPER=de_DE.UTF-8
LC_NAME=de_DE.UTF-8
LC_ADDRESS=de_DE.UTF-8
LC_TELEPHONE=de_DE.UTF-8
LC_MEASUREMENT=de_DE.UTF-8
LC_IDENTIFICATION=de_DE.UTF-8
LC_ALL=

But after specifying that i want a new login shell the default locale configured for that user/system will be used instead!

[root@showcase-community-centos-7 vagrant]# su -s /bin/bash open-xchange -l
-bash-4.2$ locale -v
LANG=en_US.UTF-8
LC_CTYPE="en_US.UTF-8"
LC_NUMERIC="en_US.UTF-8"
LC_TIME="en_US.UTF-8"
LC_COLLATE="en_US.UTF-8"
LC_MONETARY="en_US.UTF-8"
LC_MESSAGES="en_US.UTF-8"
LC_PAPER="en_US.UTF-8"
LC_NAME="en_US.UTF-8"
LC_ADDRESS="en_US.UTF-8"
LC_TELEPHONE="en_US.UTF-8"
LC_MEASUREMENT="en_US.UTF-8"
LC_IDENTIFICATION="en_US.UTF-8"
LC_ALL=

=== Verify that your system is configured with a proper unicode locale for OX ===

Assuming you haven't reconfigured the locale of your root user

[root@showcase-community-centos-7 vagrant] locale charmap
UTF-8
[root@showcase-community-centos-7 vagrant]

== Warnings during early start ==

Early warnings before the start of the JVM and related logging libraries like logback and logstash forwarding are redirected to '''/var/log/open-xchange/open-xchange-console.log'''.

=== Limits ===
If the open-xchange startup script isn't able to set the proper limits on CentOS 6 due to e.g. hard limits being enforced via pam_limits you'll see the following warning in the console log during startup

singlenode open-xchange # cat open-xchange-console.log
/opt/open-xchange/sbin/open-xchange: line 115: ulimit: max user processes: cannot modify limit: Operation not permitted

=== Locale ===
If the open-xchange startup script isn't able to detect a unicode locale you'll see a warning like the following (the current charmap might differ)

WARNING: Couldn't detect proper unicode charmap in locale category LC_CTYPE.
WARNING: Current charmap: "ANSI_X3.4-1968" might cause encoding problems.

== Summary: Open-Xchange middleware on specific distros ==
The support for the mentioned mechanism of resource control differ depending on the distribution and the init system in use.

=== Debian 7 ===
;Init
: System V style
;OX Configurable Limits/Defaults
: nofile, nproc

The mentioned limits can be configured via '''/opt/open-xchange/etc/ox-scriptconf.sh'''. The limits are applied via ulimit in the service's init script. The open-xchange service is finally started via start-stop-daemon which doesn't doesn't consider '''/etc/security/limits.*'''

=== RHEL 6 / CentOS 6 ===
;Init
: Upstart, System V compatible
;OX Configurable Limits/Defaults
: nofile, nproc

The mentioned limits can be configured via '''/opt/open-xchange/etc/ox-scriptconf.sh'''. The limits are applied via ulimit in the service's init script. Furthermore as the open-xchange service is finally started via '''su ... open-xchange''' on this distro a user session is opened via su/pam and the default CentOS pam config reads the '''/etc/security/limits.*''' configuration by loading the pam stack like:

: '''/etc/pam.d/su'''
:: '''-> /etc/pam.d/system-auth'''
::: '''-> pam_limits.so'''

If NPROC isn't configured for the open-xchange-server it's restricted to '''1024''' globally by default to prevent accidental fork bombs, see '''/etc/security/limits.d/90-nproc.conf''' which can result in severe problems modern multithreaded applications.

=== RHEL 7 / CentOS 7 / Debian 8 / SLE 12 ===
;Init
: Systemd
;OX Configurable Limits/Defaults
: nofile, nproc

For systemd the default limits are configured directly in the service's unit file that is shipped by OX and located at '''/usr/lib/systemd/system/open-xchange.service'''. The drop-in config to override or extend the default unit file is located at '''/etc/systemd/system/open-xchange.service.d/limits.conf'''. [https://www.freedesktop.org/software/systemd/man/systemd.exec.html Systemd.exec] shows a whole lot of options that can be used by admins to adapt the default service to their specific needs.

----

[[File:diagram_sources.zip]]

AppSuite:MiddlewareStartup

2017-04-26T08:06:13Z

Marens: /* JVM startup */

== Systemd ==

==== Service definition ====
The default service definition as shipped by Open-Xchange is rather concise and easy to read. You can check by looking at the service file that is installed to either /usr/lib/systemd or /lib/systemd depending on your distribution of choice.

singlenode:~ # cat /usr/lib/systemd/system/open-xchange.service
[Unit]
After=remote-fs.target
After=time-sync.target ypbind.service sendmail.service cyrus.service

[Service]
User=open-xchange
PermissionsStartOnly=true
TimeoutStartSec=0
ExecStartPre=/opt/open-xchange/sbin/triggerupdatethemes -u
ExecStart=/opt/open-xchange/sbin/open-xchange
ExecStop=/opt/open-xchange/sbin/shutdown -w
ExecReload=/opt/open-xchange/sbin/triggerreloadconfiguration -d
KillMode=process
LimitNOFILE=65536
LimitNPROC=65536

[Install]
WantedBy=multi-user.target

==== Drop-in configs ====
Drop in configs allow administrators to easily adapt and override default service unit files. So if you want to change the default limits or add additional limits have a look at our default example drop-in config which is located at /etc/systemd/system/open-xchange.service.d/limits.conf

singlenode:~ # cat /etc/systemd/system/open-xchange.service.d/limits.conf
# Override and add options in this file
# See systemd.exec(5) for other limits

[Service]
#LimitNPROC=65536
#LimitNOFILE=65536

The [https://www.freedesktop.org/software/systemd/man/systemd.exec.html systemd.exec] documentation shows a whole lot of options that can be used by admins to adapt the default service to their specific needs.

 
[[File:mw_start_sysd.png|thumb|Service startup using systemd|left|1000px]]
 

''Note:'' The technical implementation of systemd differs as service definition and config aren't read every time during service start but the changes need to be reread with ''systemctl daemon-reload''. The above is just a simplyfied version to explain the general concept.

== System V / Upstart ==

==== Service definition ====

The default service definition as shipped by Open-Xchange can be found at /etc/init.d/open-xchange and will differ slightly depending on the distro in use.

 
[[File:mw_start_sysv.png|thumb|Service startup using upstart|left|1000px]]
 
== Apply Resource limits ==

As you can see in the diagrams above one of the differences is that:

* systemd reads all the infos wrt. resource limitations from the original service file and then applies possible further restrictions from the drop-in configs (steps 2 and 8 in the diagram ''Service startup using systemd'') and limits configured in /opt/open-xchange/etc/ox-scriptconf.sh aren't considered anylonger
* the older System V style init has to source /opt/open-xchange/etc/ox-scriptconf.sh to be able to read the configs for the maximum number of processes (NPROC) and files(NRFILES) that the middleware shall be allowed to create/open (steps 3,4 and 15 in the diagram ''Service startup using upstart'').

=== Background Infos ===

Several ways exist to restrict resources on a linux system from a global level down to user/groups or even shells and the processes started by them.

==== Sysctl ====
Sysctl is used to modify kernel parameters at runtime. E.g. to set the maximum number of files

$ sysctl -w fs.file-max=100000

To permanently set them append to the main configuration file and reload the settings

$ echo fs.file-max=100000 >> /etc/sysctl.conf
$ sysctl -p

More infos can be found via '''man sysctl'''

==== Limits.conf ====

Allows to restrict resources an a global, group or user level. E.g:

$ cat /etc/security/limits.d/90-nproc.conf
# Default limit for number of user's processes to prevent
# accidental fork bombs.
# See rhbz #432903 for reasoning.

* soft nproc 1024

From '''man limits.conf''':
<blockquote>
''Also, please note that all limit settings are set per login. They are not global, nor are they permanent; existing only for the duration of the session.''
</blockquote>

The limits per login are applied via the ''pam'' stack. See '''man pam''' and '''man pam_limits''' for more details. As those limits are bound to sessions they don't affect most daemons started by our supported init systems or init utils. Most state that they are ignored by design, see [https://bugs.launchpad.net/ubuntu/+source/upstart/+bug/938669/comments/4 upstart], [https://bugzilla.redhat.com/show_bug.cgi?id=754285#c1 systemd] and [https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=302079 start-stop-daemon]

===== Ulimit =====
From '''man bash'''
<blockquote>
ulimit [-HSTabcdefilmnpqrstuvx [limit]]
Provides control over the resources available to the shell and to processes started by it, on systems that allow such control.
</blockquote>
This is what we use in our System V compatible init scripts to increase resources for the open-xchange process across multiple distros. Currently only ''the maximum number of processes'' and ''the maximum number of open file descriptors available to a single user'' are increased via ulimit. The values are specified in '''/opt/open-xchange/ox-scriptconf.sh'''

==== Systemd ====

===== Control Groups=====

Control groups should only affect the OX middleware if you create/manage them yourself of if you are using a ''modern'' distribution that already uses systemd as init.

Citing from the kernel cgroup [https://www.kernel.org/doc/Documentation/cgroup-v2.txt documentation]:

<blockquote>
1-2. What is cgroup?

cgroup is a mechanism to organize processes hierarchically and
distribute system resources along the hierarchy in a controlled and
configurable manner.

cgroup is largely composed of two parts - the core and controllers.
cgroup core is primarily responsible for hierarchically organizing
processes. A cgroup controller is usually responsible for
distributing a specific type of system resource along the hierarchy
although there are utility controllers which serve purposes other than
resource distribution.

cgroups form a tree structure and every process in the system belongs
to one and only one cgroup. All threads of a process belong to the
same cgroup. On creation, all processes are put in the cgroup that
the parent process belongs to at the time. A process can be migrated
to another cgroup. Migration of a process doesn't affect already
existing descendant processes.

Following certain structural constraints, controllers may be enabled or
disabled selectively on a cgroup. All controller behaviors are
hierarchical - if a controller is enabled on a cgroup, it affects all
processes which belong to the cgroups consisting the inclusive
sub-hierarchy of the cgroup. When a controller is enabled on a nested
cgroup, it always restricts the resource distribution further. The
restrictions set closer to the root in the hierarchy can not be
overridden from further away.
</blockquote>

So processes are organized into a tree structure of control groups and controllers are responsible for the distribution of resources. So what kind of controllers exist?

<blockquote>
5. Controllers

5-1. CPU

The "cpu" controllers regulates distribution of CPU cycles. This
controller implements weight and absolute bandwidth limit models for
normal scheduling policy and absolute bandwidth allocation model for
realtime scheduling policy.

5-2. Memory

The "memory" controller regulates distribution of memory.
...
While not completely water-tight, all major memory usages by a given
cgroup are tracked so that the total memory consumption can be
accounted and controlled to a reasonable extent.

5-3. IO

The "io" controller regulates the distribution of IO resources. This
controller implements both weight based and absolute bandwidth or IOPS
limit distribution; however, weight based distribution is available
only if cfq-iosched is in use and neither scheme is available for
blk-mq devices.
</blockquote>

The open-xchange service is simply put into the default ''system.slice'' without applying further limits.

singlenode$ systemd-cgls
├─1 /sbin/init
├─system.slice
│ ├─avahi-daemon.service
│ │ ├─501 avahi-daemon: running [singlenode]
│ │ └─514 avahi-daemon: chroot helper
│ ├─console-kit-daemon.service
│ │ └─16164 /usr/sbin/console-kit-daemon --no-daemon
│ ├─dbus.service
│ │ └─508 /usr/bin/dbus-daemon --system --address=systemd: --nofork --nopidfile --systemd-activation
│ ├─munin-node.service
│ │ └─4290 /usr/bin/perl -wT /usr/sbin/munin-node
│ ├─open-xchange.service
│ │ └─6037 /usr/bin/java -Dsun.net.inetaddr.ttl=3600 -Dnetworkaddress.cache.ttl=3600 -Dnetworkaddress.cache.negative.ttl=10 ...

To check all the details use

singlenode:~ # systemctl show system.slice

==== Limits besides control groups ====
Besides control groups systemd allows you to apply other limits to the execution environment of your service. Here we can apply the limits that would normally be applied via ''limits.conf'' or ''ulimit''. Systemd uses [http://man7.org/linux/man-pages/man2/setrlimit.2.html setrlimit] for this. The options that we set by default are:

* LimitNOFILE
* LimitNPROC

which you have already seen when we took a look at the [[#Service_definiton|service definition]] and [[#Drop-in_config|drop-in configs]]. ''Again'': The complete documentation wrt. resource limit configurations in drop-in configs can be found at [https://www.freedesktop.org/software/systemd/man/systemd.exec.html systemd.exec]

== Verify limits ==

=== System V ===

Get the proper pid of the java process started for the middleware by either checking '''ps faux'''
root 4103 0.0 0.0 66528 1868 pts/0 S 22:05 0:00 su -s /bin/bash open-xchange -c /opt/open-xchange/sbin/open-xchange
494 4109 9.5 19.8 2224644 381312 ? Ssl 22:05 1:12 \_ /usr/bin/java -Dsun.net.inetaddr.ttl=3600 -Dnetworkaddress.cache.ttl=3600

or programmatically
singlenode:~ # pid=$(pgrep -P $(</var/run/open-xchange.pid) java)

and check the limits applied for that process
singlenode:~ # cat /proc/$pid/limits
Limit Soft Limit Hard Limit Units
Max cpu time unlimited unlimited seconds
Max file size unlimited unlimited bytes
Max data size unlimited unlimited bytes
Max stack size 8388608 unlimited bytes
Max core file size 0 unlimited bytes
Max resident set unlimited unlimited bytes
Max processes 65536 65536 processes
Max open files 65536 65536 files
Max locked memory 65536 65536 bytes
Max address space unlimited unlimited bytes
Max file locks unlimited unlimited locks
Max pending signals 24254 24254 signals
Max msgqueue size 819200 819200 bytes
Max nice priority 0 0
Max realtime priority 0 0
Max realtime timeout unlimited unlimited us

=== Systemd ===
singlenode:~ # systemctl show open-xchange | grep Limit
StartLimitInterval=10000000
StartLimitBurst=5
StartLimitAction=none
MemoryLimit=18446744073709551615
LimitCPU=18446744073709551615
LimitFSIZE=18446744073709551615
LimitDATA=18446744073709551615
LimitSTACK=18446744073709551615
LimitCORE=18446744073709551615
LimitRSS=18446744073709551615
LimitNOFILE=65536
LimitAS=18446744073709551615
LimitNPROC=65536
LimitMEMLOCK=65536
LimitLOCKS=18446744073709551615
LimitSIGPENDING=19827
LimitMSGQUEUE=819200
LimitNICE=0
LimitRTPRIO=0
LimitRTTIME=18446744073709551615

== Evaluate JVM commandline options ==

Before starting Java and the OSGi framework with all the Open-Xchange middleware bundles we have to evaluate several important JVM commandline options that influence the performance and stability of the Open-Xchange middleware stack. These commandline options are specified in '''/opt/open-xchange/etc/ox-scriptconf.sh'''

=== Format < 7.8.4 ===
For version up to 7.8.3 the JVM configuration options were all specified via a single property '''JAVA_XTRAOPTS''' which resulted in a rather long configuration string that was hard to read and maintain.

JAVA_XTRAOPTS="-Dsun.net.inetaddr.ttl=3600 -Dnetworkaddress.cache.ttl=3600 -Dnetworkaddress.cache.negative.ttl=10 \
-Dlogback.threadlocal.put.duplicate=false -server -Djava.awt.headless=true -Xmx512M -XX:MaxPermSize=256m -XX:+UseConcMarkSweepGC \
-XX:+UseParNewGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:NewRatio=3 -XX:+UseTLAB \
-Dosgi.compatibility.bootdelegation=false -XX:-OmitStackTraceInFastThrow"

During the startup of the open-xchange service (step 11 of diagram Service startup using systemd and step 10 of diagram Service startup using upstart) this JAVA_XTRAOPTS string goes through a basic validity check and is then passed to the JVM.

=== Format >= 7.8.4 ===
Beginning from version 7.8.4 the options are split up and sorted into different categories like:

JAVA_OPTS_GC="-XX:+UseConcMarkSweepGC -XX:+UseParNewGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:NewRatio=3 -XX:+DisableExplicitGC"
JAVA_OPTS_LOG="-Dlogback.threadlocal.put.duplicate=false -XX:-OmitStackTraceInFastThrow"
JAVA_OPTS_MEM="-XX:MaxHeapSize=512M -XX:MaxPermSize=256M -XX:+UseTLAB"
JAVA_OPTS_NET="-Dsun.net.inetaddr.ttl=3600 -Dnetworkaddress.cache.ttl=3600 -Dnetworkaddress.cache.negative.ttl=10"
JAVA_OPTS_OSGI="-Dosgi.compatibility.bootdelegation=false"
JAVA_OPTS_SERVER="-server -Djava.awt.headless=true"

JAVA_OPTS_OTHER=""

#JAVA_OPTS_DEBUG="-XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/mnt/heapdump -Xloggc:/var/log/open-xchange/gc.log -verbose:gc -XX:+PrintGCDateStamps \
-XX:+PrintHeapAtGC -XX:+PrintGCApplicationStoppedTime -XX:+PrintTenuringDistribution"

which is much easier to read an maintain. Furthermore we added a useful '''JAVA_OPTS_DEBUG''' configuration parameter that comes preconfigured but unused/commented so that customers can quickly enable the most commonly used debugging options in case they have to provide detailed system information to Open-Xchange.

During the startup of the open-xchange service (step 11 of diagram ''Service startup using systemd'' and step 10 of diagram ''Service startup using upstart'') all uncommented JAVA_OPTS_* categories will then be evaluated and combined again into a single string of commandline options that can then be passed to the JVM.

=== Upgrade procedure ===
During the upgrade '''7.8.3-revX -> 7.8.4-revY''' existing JAVA_XTRAOPTS will automatically be split up and migrated to the new categories approach. A backup of the file (ox-scriptconf.sh.timestamp) will be created and admins are urged to check for proper migration before deleting the backup file like it's already done with *.dpkg-dist or *.rpmnew files on Debian or RHEL/SLES distributions.

== Encoding detection during JVM startup ==

=== Background Infos ===

From ''man 7 locale''

DESCRIPTION
A locale is a set of language and cultural rules. These cover aspects such as lan‐
guage for messages, different character sets, lexicographic conventions, and so on.
A program needs to be able to determine its locale and act accordingly to be portable
to different cultures.

The format to specify a system locale is: '''''language[_territory][.codeset][@modifier]'''''.

Where all of

* en_US.UTF-8
* en_US.utf8
* en_US.utf-8
* en_US.UTF_8

will result in a valid UTF-8 charmap for the english language with locale categories configured as

LANG=en_US.UTF-8
LC_CTYPE="en_US.UTF-8"
LC_NUMERIC="en_US.UTF-8"
LC_TIME="en_US.UTF-8"
LC_COLLATE="en_US.UTF-8"
LC_MONETARY="en_US.UTF-8"
LC_MESSAGES="en_US.UTF-8"
LC_PAPER="en_US.UTF-8"
LC_NAME="en_US.UTF-8"
LC_ADDRESS="en_US.UTF-8"
LC_TELEPHONE="en_US.UTF-8"
LC_MEASUREMENT="en_US.UTF-8"
LC_IDENTIFICATION="en_US.UTF-8"
LC_ALL=

See for more infos: locale(1), locale(7), [https://en.wikipedia.org/wiki/List_of_ISO_639-1_codes language codes], [https://en.wikipedia.org/wiki/ISO_3166-1#Current_codes country codes] and [https://en.wikipedia.org/wiki/Character_encoding character encodings]

==== JVM startup ====

The default file.encoding detection is done at JVM startup based upon LC_CTYPE which makes sense as it's documented as:

CODESET (LC_CTYPE)
Return a string with the name of the character encoding used in the selected locale, such as "UTF-8", "ISO-8859-1",
or "ANSI_X3.4-1968" (better known as US-ASCII). This is the same string that you get with "locale charmap".
For a list of character encoding names, try "locale -m", cf. locale(1).

Furthermore this influences the defaultCharset selection of the JVM.

/**
* Returns the default charset of this Java virtual machine.
*
* The default charset is determined during virtual-machine startup and
* typically depends upon the locale and charset of the underlying
* operating system.
*
* @return A charset object for the default charset
*
* @since 1.5
*/
public static Charset defaultCharset() {
if (defaultCharset == null) {
synchronized (Charset.class) {
String csn = AccessController.doPrivileged(
new GetPropertyAction("file.encoding"));
Charset cs = lookup(csn);
if (cs != null)
defaultCharset = cs;
else
defaultCharset = forName("UTF-8");
}
}
return defaultCharset;
}

Although we try to specify valid unicode encodings for every IO operation the Open-Xchange middleware does, some parts of the JVM don't allow to specify the encoding to use and simply falls back to the detected default encoding. That's why '''admins have to make sure to provide a proper unicode environment for the middleware service during every service start''' or use documented workarounds provided by Open-Xchange (e.g. [http://documentation.open-xchange.com/components/middleware/config/develop/index.html#mode=features&feature=Password%20Change%20Script com.openexchange.passwordchange.script.base64]: Encoded strings as Base64 to circumvent character encoding issues on improperly configured distributions).

See the documentation of your linux distribution on how to configure a proper unicode locale.

=== Query which file encoding was detected by the JVM during startup ===
[[File:visualvm-file.encoding.png|thumb|Querying the JVM via visualvm|right|300px]]

The file encoding that was detected by the JVM during startup can be queried via JMX. You can use graphical management tools like [https://visualvm.github.io/ visualvm], query our [[Jolokia]] JMX bridge via http or just use the commandline like

/opt/open-xchange/sbin/showruntimestats -r | sed -ne 's/.*${key=file.encoding, value=[^}]*}$.*/\1/p'
{key=file.encoding, value='''UTF-8'''}

 

=== Locale passing from client to server via SSH ===

The following setting of sshd allows clients to specify which locale to use on the server and might lead to some confusion or unexpected behaviour when checking the system locale or even to the usage of broken locales during middleware startup if your client sends misconfigured or missing locale configuration to the server.

grep AcceptEnv /etc/ssh/sshd_config
AcceptEnv LANG LC_*

For example when i connect to the machine my local german locale is sent to the server

[vagrant@showcase-community-centos-7 ~]$ locale -v
LANG=de_DE.UTF-8
LC_CTYPE="de_DE.UTF-8"
LC_NUMERIC=de_DE.UTF-8
LC_TIME=de_DE.UTF-8
LC_COLLATE="de_DE.UTF-8"
LC_MONETARY=de_DE.UTF-8
LC_MESSAGES="de_DE.UTF-8"
LC_PAPER=de_DE.UTF-8
LC_NAME=de_DE.UTF-8
LC_ADDRESS=de_DE.UTF-8
LC_TELEPHONE=de_DE.UTF-8
LC_MEASUREMENT=de_DE.UTF-8
LC_IDENTIFICATION=de_DE.UTF-8
LC_ALL=

But after specifying that i want a new login shell the default locale configured for that user/system will be used instead!

[root@showcase-community-centos-7 vagrant]# su -s /bin/bash open-xchange -l
-bash-4.2$ locale -v
LANG=en_US.UTF-8
LC_CTYPE="en_US.UTF-8"
LC_NUMERIC="en_US.UTF-8"
LC_TIME="en_US.UTF-8"
LC_COLLATE="en_US.UTF-8"
LC_MONETARY="en_US.UTF-8"
LC_MESSAGES="en_US.UTF-8"
LC_PAPER="en_US.UTF-8"
LC_NAME="en_US.UTF-8"
LC_ADDRESS="en_US.UTF-8"
LC_TELEPHONE="en_US.UTF-8"
LC_MEASUREMENT="en_US.UTF-8"
LC_IDENTIFICATION="en_US.UTF-8"
LC_ALL=

=== Verify that your system is configured with a proper unicode locale for OX ===

Assuming you haven't reconfigured the locale of your root user

i|marens@karotte ~ $ su -
Password:
i|karotte ~ # locale charmap
UTF-8
i|karotte ~ #

== Warnings during early start ==

Early warnings before the start of the JVM and related logging libraries like logback and logstash forwarding are redirected to '''/var/log/open-xchange/open-xchange-console.log'''.

=== Limits ===
If the open-xchange startup script isn't able to set the proper limits on CentOS 6 due to e.g. hard limits being enforced via pam_limits you'll see the following warning in the console log during startup

singlenode open-xchange # cat open-xchange-console.log
/opt/open-xchange/sbin/open-xchange: line 115: ulimit: max user processes: cannot modify limit: Operation not permitted

=== Locale ===
If the open-xchange startup script isn't able to detect a unicode locale you'll see a warning like the following (the current charmap might differ)

WARNING: Couldn't detect proper unicode charmap in locale category LC_CTYPE.
WARNING: Current charmap: "ANSI_X3.4-1968" might cause encoding problems.

== Summary: Open-Xchange middleware on specific distros ==
The support for the mentioned mechanism of resource control differ depending on the distribution and the init system in use.

=== Debian 7 ===
;Init
: System V style
;OX Configurable Limits/Defaults
: nofile, nproc

The mentioned limits can be configured via '''/opt/open-xchange/etc/ox-scriptconf.sh'''. The limits are applied via ulimit in the service's init script. The open-xchange service is finally started via start-stop-daemon which doesn't doesn't consider '''/etc/security/limits.*'''

=== RHEL 6 / CentOS 6 ===
;Init
: Upstart, System V compatible
;OX Configurable Limits/Defaults
: nofile, nproc

The mentioned limits can be configured via '''/opt/open-xchange/etc/ox-scriptconf.sh'''. The limits are applied via ulimit in the service's init script. Furthermore as the open-xchange service is finally started via '''su ... open-xchange''' on this distro a user session is opened via su/pam and the default CentOS pam config reads the '''/etc/security/limits.*''' configuration by loading the pam stack like:

: '''/etc/pam.d/su'''
:: '''-> /etc/pam.d/system-auth'''
::: '''-> pam_limits.so'''

If NPROC isn't configured for the open-xchange-server it's restricted to '''1024''' globally by default to prevent accidental fork bombs, see '''/etc/security/limits.d/90-nproc.conf''' which can result in severe problems modern multithreaded applications.

=== RHEL 7 / CentOS 7 / Debian 8 / SLE 12 ===
;Init
: Systemd
;OX Configurable Limits/Defaults
: nofile, nproc

For systemd the default limits are configured directly in the service's unit file that is shipped by OX and located at '''/usr/lib/systemd/system/open-xchange.service'''. The drop-in config to override or extend the default unit file is located at '''/etc/systemd/system/open-xchange.service.d/limits.conf'''. [https://www.freedesktop.org/software/systemd/man/systemd.exec.html Systemd.exec] shows a whole lot of options that can be used by admins to adapt the default service to their specific needs.

----

[[File:diagram_sources.zip]]

AppSuite:MiddlewareStartup

2017-04-26T08:04:05Z

Marens: /* JVM startup */

== Systemd ==

==== Service definition ====
The default service definition as shipped by Open-Xchange is rather concise and easy to read. You can check by looking at the service file that is installed to either /usr/lib/systemd or /lib/systemd depending on your distribution of choice.

singlenode:~ # cat /usr/lib/systemd/system/open-xchange.service
[Unit]
After=remote-fs.target
After=time-sync.target ypbind.service sendmail.service cyrus.service

[Service]
User=open-xchange
PermissionsStartOnly=true
TimeoutStartSec=0
ExecStartPre=/opt/open-xchange/sbin/triggerupdatethemes -u
ExecStart=/opt/open-xchange/sbin/open-xchange
ExecStop=/opt/open-xchange/sbin/shutdown -w
ExecReload=/opt/open-xchange/sbin/triggerreloadconfiguration -d
KillMode=process
LimitNOFILE=65536
LimitNPROC=65536

[Install]
WantedBy=multi-user.target

==== Drop-in configs ====
Drop in configs allow administrators to easily adapt and override default service unit files. So if you want to change the default limits or add additional limits have a look at our default example drop-in config which is located at /etc/systemd/system/open-xchange.service.d/limits.conf

singlenode:~ # cat /etc/systemd/system/open-xchange.service.d/limits.conf
# Override and add options in this file
# See systemd.exec(5) for other limits

[Service]
#LimitNPROC=65536
#LimitNOFILE=65536

The [https://www.freedesktop.org/software/systemd/man/systemd.exec.html systemd.exec] documentation shows a whole lot of options that can be used by admins to adapt the default service to their specific needs.

 
[[File:mw_start_sysd.png|thumb|Service startup using systemd|left|1000px]]
 

''Note:'' The technical implementation of systemd differs as service definition and config aren't read every time during service start but the changes need to be reread with ''systemctl daemon-reload''. The above is just a simplyfied version to explain the general concept.

== System V / Upstart ==

==== Service definition ====

The default service definition as shipped by Open-Xchange can be found at /etc/init.d/open-xchange and will differ slightly depending on the distro in use.

 
[[File:mw_start_sysv.png|thumb|Service startup using upstart|left|1000px]]
 
== Apply Resource limits ==

As you can see in the diagrams above one of the differences is that:

* systemd reads all the infos wrt. resource limitations from the original service file and then applies possible further restrictions from the drop-in configs (steps 2 and 8 in the diagram ''Service startup using systemd'') and limits configured in /opt/open-xchange/etc/ox-scriptconf.sh aren't considered anylonger
* the older System V style init has to source /opt/open-xchange/etc/ox-scriptconf.sh to be able to read the configs for the maximum number of processes (NPROC) and files(NRFILES) that the middleware shall be allowed to create/open (steps 3,4 and 15 in the diagram ''Service startup using upstart'').

=== Background Infos ===

Several ways exist to restrict resources on a linux system from a global level down to user/groups or even shells and the processes started by them.

==== Sysctl ====
Sysctl is used to modify kernel parameters at runtime. E.g. to set the maximum number of files

$ sysctl -w fs.file-max=100000

To permanently set them append to the main configuration file and reload the settings

$ echo fs.file-max=100000 >> /etc/sysctl.conf
$ sysctl -p

More infos can be found via '''man sysctl'''

==== Limits.conf ====

Allows to restrict resources an a global, group or user level. E.g:

$ cat /etc/security/limits.d/90-nproc.conf
# Default limit for number of user's processes to prevent
# accidental fork bombs.
# See rhbz #432903 for reasoning.

* soft nproc 1024

From '''man limits.conf''':
<blockquote>
''Also, please note that all limit settings are set per login. They are not global, nor are they permanent; existing only for the duration of the session.''
</blockquote>

The limits per login are applied via the ''pam'' stack. See '''man pam''' and '''man pam_limits''' for more details. As those limits are bound to sessions they don't affect most daemons started by our supported init systems or init utils. Most state that they are ignored by design, see [https://bugs.launchpad.net/ubuntu/+source/upstart/+bug/938669/comments/4 upstart], [https://bugzilla.redhat.com/show_bug.cgi?id=754285#c1 systemd] and [https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=302079 start-stop-daemon]

===== Ulimit =====
From '''man bash'''
<blockquote>
ulimit [-HSTabcdefilmnpqrstuvx [limit]]
Provides control over the resources available to the shell and to processes started by it, on systems that allow such control.
</blockquote>
This is what we use in our System V compatible init scripts to increase resources for the open-xchange process across multiple distros. Currently only ''the maximum number of processes'' and ''the maximum number of open file descriptors available to a single user'' are increased via ulimit. The values are specified in '''/opt/open-xchange/ox-scriptconf.sh'''

==== Systemd ====

===== Control Groups=====

Control groups should only affect the OX middleware if you create/manage them yourself of if you are using a ''modern'' distribution that already uses systemd as init.

Citing from the kernel cgroup [https://www.kernel.org/doc/Documentation/cgroup-v2.txt documentation]:

<blockquote>
1-2. What is cgroup?

cgroup is a mechanism to organize processes hierarchically and
distribute system resources along the hierarchy in a controlled and
configurable manner.

cgroup is largely composed of two parts - the core and controllers.
cgroup core is primarily responsible for hierarchically organizing
processes. A cgroup controller is usually responsible for
distributing a specific type of system resource along the hierarchy
although there are utility controllers which serve purposes other than
resource distribution.

cgroups form a tree structure and every process in the system belongs
to one and only one cgroup. All threads of a process belong to the
same cgroup. On creation, all processes are put in the cgroup that
the parent process belongs to at the time. A process can be migrated
to another cgroup. Migration of a process doesn't affect already
existing descendant processes.

Following certain structural constraints, controllers may be enabled or
disabled selectively on a cgroup. All controller behaviors are
hierarchical - if a controller is enabled on a cgroup, it affects all
processes which belong to the cgroups consisting the inclusive
sub-hierarchy of the cgroup. When a controller is enabled on a nested
cgroup, it always restricts the resource distribution further. The
restrictions set closer to the root in the hierarchy can not be
overridden from further away.
</blockquote>

So processes are organized into a tree structure of control groups and controllers are responsible for the distribution of resources. So what kind of controllers exist?

<blockquote>
5. Controllers

5-1. CPU

The "cpu" controllers regulates distribution of CPU cycles. This
controller implements weight and absolute bandwidth limit models for
normal scheduling policy and absolute bandwidth allocation model for
realtime scheduling policy.

5-2. Memory

The "memory" controller regulates distribution of memory.
...
While not completely water-tight, all major memory usages by a given
cgroup are tracked so that the total memory consumption can be
accounted and controlled to a reasonable extent.

5-3. IO

The "io" controller regulates the distribution of IO resources. This
controller implements both weight based and absolute bandwidth or IOPS
limit distribution; however, weight based distribution is available
only if cfq-iosched is in use and neither scheme is available for
blk-mq devices.
</blockquote>

The open-xchange service is simply put into the default ''system.slice'' without applying further limits.

singlenode$ systemd-cgls
├─1 /sbin/init
├─system.slice
│ ├─avahi-daemon.service
│ │ ├─501 avahi-daemon: running [singlenode]
│ │ └─514 avahi-daemon: chroot helper
│ ├─console-kit-daemon.service
│ │ └─16164 /usr/sbin/console-kit-daemon --no-daemon
│ ├─dbus.service
│ │ └─508 /usr/bin/dbus-daemon --system --address=systemd: --nofork --nopidfile --systemd-activation
│ ├─munin-node.service
│ │ └─4290 /usr/bin/perl -wT /usr/sbin/munin-node
│ ├─open-xchange.service
│ │ └─6037 /usr/bin/java -Dsun.net.inetaddr.ttl=3600 -Dnetworkaddress.cache.ttl=3600 -Dnetworkaddress.cache.negative.ttl=10 ...

To check all the details use

singlenode:~ # systemctl show system.slice

==== Limits besides control groups ====
Besides control groups systemd allows you to apply other limits to the execution environment of your service. Here we can apply the limits that would normally be applied via ''limits.conf'' or ''ulimit''. Systemd uses [http://man7.org/linux/man-pages/man2/setrlimit.2.html setrlimit] for this. The options that we set by default are:

* LimitNOFILE
* LimitNPROC

which you have already seen when we took a look at the [[#Service_definiton|service definition]] and [[#Drop-in_config|drop-in configs]]. ''Again'': The complete documentation wrt. resource limit configurations in drop-in configs can be found at [https://www.freedesktop.org/software/systemd/man/systemd.exec.html systemd.exec]

== Verify limits ==

=== System V ===

Get the proper pid of the java process started for the middleware by either checking '''ps faux'''
root 4103 0.0 0.0 66528 1868 pts/0 S 22:05 0:00 su -s /bin/bash open-xchange -c /opt/open-xchange/sbin/open-xchange
494 4109 9.5 19.8 2224644 381312 ? Ssl 22:05 1:12 \_ /usr/bin/java -Dsun.net.inetaddr.ttl=3600 -Dnetworkaddress.cache.ttl=3600

or programmatically
singlenode:~ # pid=$(pgrep -P $(</var/run/open-xchange.pid) java)

and check the limits applied for that process
singlenode:~ # cat /proc/$pid/limits
Limit Soft Limit Hard Limit Units
Max cpu time unlimited unlimited seconds
Max file size unlimited unlimited bytes
Max data size unlimited unlimited bytes
Max stack size 8388608 unlimited bytes
Max core file size 0 unlimited bytes
Max resident set unlimited unlimited bytes
Max processes 65536 65536 processes
Max open files 65536 65536 files
Max locked memory 65536 65536 bytes
Max address space unlimited unlimited bytes
Max file locks unlimited unlimited locks
Max pending signals 24254 24254 signals
Max msgqueue size 819200 819200 bytes
Max nice priority 0 0
Max realtime priority 0 0
Max realtime timeout unlimited unlimited us

=== Systemd ===
singlenode:~ # systemctl show open-xchange | grep Limit
StartLimitInterval=10000000
StartLimitBurst=5
StartLimitAction=none
MemoryLimit=18446744073709551615
LimitCPU=18446744073709551615
LimitFSIZE=18446744073709551615
LimitDATA=18446744073709551615
LimitSTACK=18446744073709551615
LimitCORE=18446744073709551615
LimitRSS=18446744073709551615
LimitNOFILE=65536
LimitAS=18446744073709551615
LimitNPROC=65536
LimitMEMLOCK=65536
LimitLOCKS=18446744073709551615
LimitSIGPENDING=19827
LimitMSGQUEUE=819200
LimitNICE=0
LimitRTPRIO=0
LimitRTTIME=18446744073709551615

== Evaluate JVM commandline options ==

Before starting Java and the OSGi framework with all the Open-Xchange middleware bundles we have to evaluate several important JVM commandline options that influence the performance and stability of the Open-Xchange middleware stack. These commandline options are specified in '''/opt/open-xchange/etc/ox-scriptconf.sh'''

=== Format < 7.8.4 ===
For version up to 7.8.3 the JVM configuration options were all specified via a single property '''JAVA_XTRAOPTS''' which resulted in a rather long configuration string that was hard to read and maintain.

JAVA_XTRAOPTS="-Dsun.net.inetaddr.ttl=3600 -Dnetworkaddress.cache.ttl=3600 -Dnetworkaddress.cache.negative.ttl=10 \
-Dlogback.threadlocal.put.duplicate=false -server -Djava.awt.headless=true -Xmx512M -XX:MaxPermSize=256m -XX:+UseConcMarkSweepGC \
-XX:+UseParNewGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:NewRatio=3 -XX:+UseTLAB \
-Dosgi.compatibility.bootdelegation=false -XX:-OmitStackTraceInFastThrow"

During the startup of the open-xchange service (step 11 of diagram Service startup using systemd and step 10 of diagram Service startup using upstart) this JAVA_XTRAOPTS string goes through a basic validity check and is then passed to the JVM.

=== Format >= 7.8.4 ===
Beginning from version 7.8.4 the options are split up and sorted into different categories like:

JAVA_OPTS_GC="-XX:+UseConcMarkSweepGC -XX:+UseParNewGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:NewRatio=3 -XX:+DisableExplicitGC"
JAVA_OPTS_LOG="-Dlogback.threadlocal.put.duplicate=false -XX:-OmitStackTraceInFastThrow"
JAVA_OPTS_MEM="-XX:MaxHeapSize=512M -XX:MaxPermSize=256M -XX:+UseTLAB"
JAVA_OPTS_NET="-Dsun.net.inetaddr.ttl=3600 -Dnetworkaddress.cache.ttl=3600 -Dnetworkaddress.cache.negative.ttl=10"
JAVA_OPTS_OSGI="-Dosgi.compatibility.bootdelegation=false"
JAVA_OPTS_SERVER="-server -Djava.awt.headless=true"

JAVA_OPTS_OTHER=""

#JAVA_OPTS_DEBUG="-XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/mnt/heapdump -Xloggc:/var/log/open-xchange/gc.log -verbose:gc -XX:+PrintGCDateStamps \
-XX:+PrintHeapAtGC -XX:+PrintGCApplicationStoppedTime -XX:+PrintTenuringDistribution"

which is much easier to read an maintain. Furthermore we added a useful '''JAVA_OPTS_DEBUG''' configuration parameter that comes preconfigured but unused/commented so that customers can quickly enable the most commonly used debugging options in case they have to provide detailed system information to Open-Xchange.

During the startup of the open-xchange service (step 11 of diagram ''Service startup using systemd'' and step 10 of diagram ''Service startup using upstart'') all uncommented JAVA_OPTS_* categories will then be evaluated and combined again into a single string of commandline options that can then be passed to the JVM.

=== Upgrade procedure ===
During the upgrade '''7.8.3-revX -> 7.8.4-revY''' existing JAVA_XTRAOPTS will automatically be split up and migrated to the new categories approach. A backup of the file (ox-scriptconf.sh.timestamp) will be created and admins are urged to check for proper migration before deleting the backup file like it's already done with *.dpkg-dist or *.rpmnew files on Debian or RHEL/SLES distributions.

== Encoding detection during JVM startup ==

=== Background Infos ===

From ''man 7 locale''

DESCRIPTION
A locale is a set of language and cultural rules. These cover aspects such as lan‐
guage for messages, different character sets, lexicographic conventions, and so on.
A program needs to be able to determine its locale and act accordingly to be portable
to different cultures.

The format to specify a system locale is: '''''language[_territory][.codeset][@modifier]'''''.

Where all of

* en_US.UTF-8
* en_US.utf8
* en_US.utf-8
* en_US.UTF_8

will result in a valid UTF-8 charmap for the english language with locale categories configured as

LANG=en_US.UTF-8
LC_CTYPE="en_US.UTF-8"
LC_NUMERIC="en_US.UTF-8"
LC_TIME="en_US.UTF-8"
LC_COLLATE="en_US.UTF-8"
LC_MONETARY="en_US.UTF-8"
LC_MESSAGES="en_US.UTF-8"
LC_PAPER="en_US.UTF-8"
LC_NAME="en_US.UTF-8"
LC_ADDRESS="en_US.UTF-8"
LC_TELEPHONE="en_US.UTF-8"
LC_MEASUREMENT="en_US.UTF-8"
LC_IDENTIFICATION="en_US.UTF-8"
LC_ALL=

See for more infos: locale(1), locale(7), [https://en.wikipedia.org/wiki/List_of_ISO_639-1_codes language codes], [https://en.wikipedia.org/wiki/ISO_3166-1#Current_codes country codes] and [https://en.wikipedia.org/wiki/Character_encoding character encodings]

==== JVM startup ====

The default file.encoding detection is done at JVM startup based upon LC_CTYPE which makes sense as it's documented as:

CODESET (LC_CTYPE)
Return a string with the name of the character encoding used in the selected locale, such as "UTF-8", "ISO-8859-1",
or "ANSI_X3.4-1968" (better known as US-ASCII). This is the same string that you get with "locale charmap".
For a list of character encoding names, try "locale -m", cf. locale(1).

Furthermore this influences the defaultCharset selection of the JVM.

/**
* Returns the default charset of this Java virtual machine.
*
* The default charset is determined during virtual-machine startup and
* typically depends upon the locale and charset of the underlying
* operating system.
*
* @return A charset object for the default charset
*
* @since 1.5
*/
public static Charset defaultCharset() {
if (defaultCharset == null) {
synchronized (Charset.class) {
String csn = AccessController.doPrivileged(
new GetPropertyAction("file.encoding"));
Charset cs = lookup(csn);
if (cs != null)
defaultCharset = cs;
else
defaultCharset = forName("UTF-8");
}
}
return defaultCharset;
}

Although we try to specify valid unicode encodings for every IO operations the Open-Xchange middleware does some parts of the JVM don't allow to specify the encoding to use and falls back to the detected default encoding. That's why admins have to make sure to provide a proper unicode environment for the middleware service during every start or use documented workarounds provided by Open-Xchange (e.g. [http://documentation.open-xchange.com/components/middleware/config/develop/index.html#mode=features&feature=Password%20Change%20Script com.openexchange.passwordchange.script.base64]: Encoded strings as Base64 to circumvent character encoding issues on improperly configured distributions).

See the documentation of your linux distribution on how to configure a proper unicode locale.

=== Query which file encoding was detected by the JVM during startup ===
[[File:visualvm-file.encoding.png|thumb|Querying the JVM via visualvm|right|300px]]

The file encoding that was detected by the JVM during startup can be queried via JMX. You can use graphical management tools like [https://visualvm.github.io/ visualvm], query our [[Jolokia]] JMX bridge via http or just use the commandline like

/opt/open-xchange/sbin/showruntimestats -r | sed -ne 's/.*${key=file.encoding, value=[^}]*}$.*/\1/p'
{key=file.encoding, value='''UTF-8'''}

 

=== Locale passing from client to server via SSH ===

The following setting of sshd allows clients to specify which locale to use on the server and might lead to some confusion or unexpected behaviour when checking the system locale or even to the usage of broken locales during middleware startup if your client sends misconfigured or missing locale configuration to the server.

grep AcceptEnv /etc/ssh/sshd_config
AcceptEnv LANG LC_*

For example when i connect to the machine my local german locale is sent to the server

[vagrant@showcase-community-centos-7 ~]$ locale -v
LANG=de_DE.UTF-8
LC_CTYPE="de_DE.UTF-8"
LC_NUMERIC=de_DE.UTF-8
LC_TIME=de_DE.UTF-8
LC_COLLATE="de_DE.UTF-8"
LC_MONETARY=de_DE.UTF-8
LC_MESSAGES="de_DE.UTF-8"
LC_PAPER=de_DE.UTF-8
LC_NAME=de_DE.UTF-8
LC_ADDRESS=de_DE.UTF-8
LC_TELEPHONE=de_DE.UTF-8
LC_MEASUREMENT=de_DE.UTF-8
LC_IDENTIFICATION=de_DE.UTF-8
LC_ALL=

But after specifying that i want a new login shell the default locale configured for that user/system will be used instead!

[root@showcase-community-centos-7 vagrant]# su -s /bin/bash open-xchange -l
-bash-4.2$ locale -v
LANG=en_US.UTF-8
LC_CTYPE="en_US.UTF-8"
LC_NUMERIC="en_US.UTF-8"
LC_TIME="en_US.UTF-8"
LC_COLLATE="en_US.UTF-8"
LC_MONETARY="en_US.UTF-8"
LC_MESSAGES="en_US.UTF-8"
LC_PAPER="en_US.UTF-8"
LC_NAME="en_US.UTF-8"
LC_ADDRESS="en_US.UTF-8"
LC_TELEPHONE="en_US.UTF-8"
LC_MEASUREMENT="en_US.UTF-8"
LC_IDENTIFICATION="en_US.UTF-8"
LC_ALL=

=== Verify that your system is configured with a proper unicode locale for OX ===

Assuming you haven't reconfigured the locale of your root user

i|marens@karotte ~ $ su -
Password:
i|karotte ~ # locale charmap
UTF-8
i|karotte ~ #

== Warnings during early start ==

Early warnings before the start of the JVM and related logging libraries like logback and logstash forwarding are redirected to '''/var/log/open-xchange/open-xchange-console.log'''.

=== Limits ===
If the open-xchange startup script isn't able to set the proper limits on CentOS 6 due to e.g. hard limits being enforced via pam_limits you'll see the following warning in the console log during startup

singlenode open-xchange # cat open-xchange-console.log
/opt/open-xchange/sbin/open-xchange: line 115: ulimit: max user processes: cannot modify limit: Operation not permitted

=== Locale ===
If the open-xchange startup script isn't able to detect a unicode locale you'll see a warning like the following (the current charmap might differ)

WARNING: Couldn't detect proper unicode charmap in locale category LC_CTYPE.
WARNING: Current charmap: "ANSI_X3.4-1968" might cause encoding problems.

== Summary: Open-Xchange middleware on specific distros ==
The support for the mentioned mechanism of resource control differ depending on the distribution and the init system in use.

=== Debian 7 ===
;Init
: System V style
;OX Configurable Limits/Defaults
: nofile, nproc

The mentioned limits can be configured via '''/opt/open-xchange/etc/ox-scriptconf.sh'''. The limits are applied via ulimit in the service's init script. The open-xchange service is finally started via start-stop-daemon which doesn't doesn't consider '''/etc/security/limits.*'''

=== RHEL 6 / CentOS 6 ===
;Init
: Upstart, System V compatible
;OX Configurable Limits/Defaults
: nofile, nproc

The mentioned limits can be configured via '''/opt/open-xchange/etc/ox-scriptconf.sh'''. The limits are applied via ulimit in the service's init script. Furthermore as the open-xchange service is finally started via '''su ... open-xchange''' on this distro a user session is opened via su/pam and the default CentOS pam config reads the '''/etc/security/limits.*''' configuration by loading the pam stack like:

: '''/etc/pam.d/su'''
:: '''-> /etc/pam.d/system-auth'''
::: '''-> pam_limits.so'''

If NPROC isn't configured for the open-xchange-server it's restricted to '''1024''' globally by default to prevent accidental fork bombs, see '''/etc/security/limits.d/90-nproc.conf''' which can result in severe problems modern multithreaded applications.

=== RHEL 7 / CentOS 7 / Debian 8 / SLE 12 ===
;Init
: Systemd
;OX Configurable Limits/Defaults
: nofile, nproc

For systemd the default limits are configured directly in the service's unit file that is shipped by OX and located at '''/usr/lib/systemd/system/open-xchange.service'''. The drop-in config to override or extend the default unit file is located at '''/etc/systemd/system/open-xchange.service.d/limits.conf'''. [https://www.freedesktop.org/software/systemd/man/systemd.exec.html Systemd.exec] shows a whole lot of options that can be used by admins to adapt the default service to their specific needs.

----

[[File:diagram_sources.zip]]

AppSuite:MiddlewareStartup

2017-04-25T12:31:37Z

Marens: /* Query which file encoding was detected by the JVM during startup */

== Systemd ==

==== Service definition ====
The default service definition as shipped by Open-Xchange is rather concise and easy to read. You can check by looking at the service file that is installed to either /usr/lib/systemd or /lib/systemd depending on your distribution of choice.

singlenode:~ # cat /usr/lib/systemd/system/open-xchange.service
[Unit]
After=remote-fs.target
After=time-sync.target ypbind.service sendmail.service cyrus.service

[Service]
User=open-xchange
PermissionsStartOnly=true
TimeoutStartSec=0
ExecStartPre=/opt/open-xchange/sbin/triggerupdatethemes -u
ExecStart=/opt/open-xchange/sbin/open-xchange
ExecStop=/opt/open-xchange/sbin/shutdown -w
ExecReload=/opt/open-xchange/sbin/triggerreloadconfiguration -d
KillMode=process
LimitNOFILE=65536
LimitNPROC=65536

[Install]
WantedBy=multi-user.target

==== Drop-in configs ====
Drop in configs allow administrators to easily adapt and override default service unit files. So if you want to change the default limits or add additional limits have a look at our default example drop-in config which is located at /etc/systemd/system/open-xchange.service.d/limits.conf

singlenode:~ # cat /etc/systemd/system/open-xchange.service.d/limits.conf
# Override and add options in this file
# See systemd.exec(5) for other limits

[Service]
#LimitNPROC=65536
#LimitNOFILE=65536

The [https://www.freedesktop.org/software/systemd/man/systemd.exec.html systemd.exec] documentation shows a whole lot of options that can be used by admins to adapt the default service to their specific needs.

 
[[File:mw_start_sysd.png|thumb|Service startup using systemd|left|1000px]]
 

''Note:'' The technical implementation of systemd differs as service definition and config aren't read every time during service start but the changes need to be reread with ''systemctl daemon-reload''. The above is just a simplyfied version to explain the general concept.

== System V / Upstart ==

==== Service definition ====

The default service definition as shipped by Open-Xchange can be found at /etc/init.d/open-xchange and will differ slightly depending on the distro in use.

 
[[File:mw_start_sysv.png|thumb|Service startup using upstart|left|1000px]]
 
== Apply Resource limits ==

As you can see in the diagrams above one of the differences is that:

* systemd reads all the infos wrt. resource limitations from the original service file and then applies possible further restrictions from the drop-in configs (steps 2 and 8 in the diagram ''Service startup using systemd'') and limits configured in /opt/open-xchange/etc/ox-scriptconf.sh aren't considered anylonger
* the older System V style init has to source /opt/open-xchange/etc/ox-scriptconf.sh to be able to read the configs for the maximum number of processes (NPROC) and files(NRFILES) that the middleware shall be allowed to create/open (steps 3,4 and 15 in the diagram ''Service startup using upstart'').

=== Background Infos ===

Several ways exist to restrict resources on a linux system from a global level down to user/groups or even shells and the processes started by them.

==== Sysctl ====
Sysctl is used to modify kernel parameters at runtime. E.g. to set the maximum number of files

$ sysctl -w fs.file-max=100000

To permanently set them append to the main configuration file and reload the settings

$ echo fs.file-max=100000 >> /etc/sysctl.conf
$ sysctl -p

More infos can be found via '''man sysctl'''

==== Limits.conf ====

Allows to restrict resources an a global, group or user level. E.g:

$ cat /etc/security/limits.d/90-nproc.conf
# Default limit for number of user's processes to prevent
# accidental fork bombs.
# See rhbz #432903 for reasoning.

* soft nproc 1024

From '''man limits.conf''':
<blockquote>
''Also, please note that all limit settings are set per login. They are not global, nor are they permanent; existing only for the duration of the session.''
</blockquote>

The limits per login are applied via the ''pam'' stack. See '''man pam''' and '''man pam_limits''' for more details. As those limits are bound to sessions they don't affect most daemons started by our supported init systems or init utils. Most state that they are ignored by design, see [https://bugs.launchpad.net/ubuntu/+source/upstart/+bug/938669/comments/4 upstart], [https://bugzilla.redhat.com/show_bug.cgi?id=754285#c1 systemd] and [https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=302079 start-stop-daemon]

===== Ulimit =====
From '''man bash'''
<blockquote>
ulimit [-HSTabcdefilmnpqrstuvx [limit]]
Provides control over the resources available to the shell and to processes started by it, on systems that allow such control.
</blockquote>
This is what we use in our System V compatible init scripts to increase resources for the open-xchange process across multiple distros. Currently only ''the maximum number of processes'' and ''the maximum number of open file descriptors available to a single user'' are increased via ulimit. The values are specified in '''/opt/open-xchange/ox-scriptconf.sh'''

==== Systemd ====

===== Control Groups=====

Control groups should only affect the OX middleware if you create/manage them yourself of if you are using a ''modern'' distribution that already uses systemd as init.

Citing from the kernel cgroup [https://www.kernel.org/doc/Documentation/cgroup-v2.txt documentation]:

<blockquote>
1-2. What is cgroup?

cgroup is a mechanism to organize processes hierarchically and
distribute system resources along the hierarchy in a controlled and
configurable manner.

cgroup is largely composed of two parts - the core and controllers.
cgroup core is primarily responsible for hierarchically organizing
processes. A cgroup controller is usually responsible for
distributing a specific type of system resource along the hierarchy
although there are utility controllers which serve purposes other than
resource distribution.

cgroups form a tree structure and every process in the system belongs
to one and only one cgroup. All threads of a process belong to the
same cgroup. On creation, all processes are put in the cgroup that
the parent process belongs to at the time. A process can be migrated
to another cgroup. Migration of a process doesn't affect already
existing descendant processes.

Following certain structural constraints, controllers may be enabled or
disabled selectively on a cgroup. All controller behaviors are
hierarchical - if a controller is enabled on a cgroup, it affects all
processes which belong to the cgroups consisting the inclusive
sub-hierarchy of the cgroup. When a controller is enabled on a nested
cgroup, it always restricts the resource distribution further. The
restrictions set closer to the root in the hierarchy can not be
overridden from further away.
</blockquote>

So processes are organized into a tree structure of control groups and controllers are responsible for the distribution of resources. So what kind of controllers exist?

<blockquote>
5. Controllers

5-1. CPU

The "cpu" controllers regulates distribution of CPU cycles. This
controller implements weight and absolute bandwidth limit models for
normal scheduling policy and absolute bandwidth allocation model for
realtime scheduling policy.

5-2. Memory

The "memory" controller regulates distribution of memory.
...
While not completely water-tight, all major memory usages by a given
cgroup are tracked so that the total memory consumption can be
accounted and controlled to a reasonable extent.

5-3. IO

The "io" controller regulates the distribution of IO resources. This
controller implements both weight based and absolute bandwidth or IOPS
limit distribution; however, weight based distribution is available
only if cfq-iosched is in use and neither scheme is available for
blk-mq devices.
</blockquote>

The open-xchange service is simply put into the default ''system.slice'' without applying further limits.

singlenode$ systemd-cgls
├─1 /sbin/init
├─system.slice
│ ├─avahi-daemon.service
│ │ ├─501 avahi-daemon: running [singlenode]
│ │ └─514 avahi-daemon: chroot helper
│ ├─console-kit-daemon.service
│ │ └─16164 /usr/sbin/console-kit-daemon --no-daemon
│ ├─dbus.service
│ │ └─508 /usr/bin/dbus-daemon --system --address=systemd: --nofork --nopidfile --systemd-activation
│ ├─munin-node.service
│ │ └─4290 /usr/bin/perl -wT /usr/sbin/munin-node
│ ├─open-xchange.service
│ │ └─6037 /usr/bin/java -Dsun.net.inetaddr.ttl=3600 -Dnetworkaddress.cache.ttl=3600 -Dnetworkaddress.cache.negative.ttl=10 ...

To check all the details use

singlenode:~ # systemctl show system.slice

==== Limits besides control groups ====
Besides control groups systemd allows you to apply other limits to the execution environment of your service. Here we can apply the limits that would normally be applied via ''limits.conf'' or ''ulimit''. Systemd uses [http://man7.org/linux/man-pages/man2/setrlimit.2.html setrlimit] for this. The options that we set by default are:

* LimitNOFILE
* LimitNPROC

which you have already seen when we took a look at the [[#Service_definiton|service definition]] and [[#Drop-in_config|drop-in configs]]. ''Again'': The complete documentation wrt. resource limit configurations in drop-in configs can be found at [https://www.freedesktop.org/software/systemd/man/systemd.exec.html systemd.exec]

== Verify limits ==

=== System V ===

Get the proper pid of the java process started for the middleware by either checking '''ps faux'''
root 4103 0.0 0.0 66528 1868 pts/0 S 22:05 0:00 su -s /bin/bash open-xchange -c /opt/open-xchange/sbin/open-xchange
494 4109 9.5 19.8 2224644 381312 ? Ssl 22:05 1:12 \_ /usr/bin/java -Dsun.net.inetaddr.ttl=3600 -Dnetworkaddress.cache.ttl=3600

or programmatically
singlenode:~ # pid=$(pgrep -P $(</var/run/open-xchange.pid) java)

and check the limits applied for that process
singlenode:~ # cat /proc/$pid/limits
Limit Soft Limit Hard Limit Units
Max cpu time unlimited unlimited seconds
Max file size unlimited unlimited bytes
Max data size unlimited unlimited bytes
Max stack size 8388608 unlimited bytes
Max core file size 0 unlimited bytes
Max resident set unlimited unlimited bytes
Max processes 65536 65536 processes
Max open files 65536 65536 files
Max locked memory 65536 65536 bytes
Max address space unlimited unlimited bytes
Max file locks unlimited unlimited locks
Max pending signals 24254 24254 signals
Max msgqueue size 819200 819200 bytes
Max nice priority 0 0
Max realtime priority 0 0
Max realtime timeout unlimited unlimited us

=== Systemd ===
singlenode:~ # systemctl show open-xchange | grep Limit
StartLimitInterval=10000000
StartLimitBurst=5
StartLimitAction=none
MemoryLimit=18446744073709551615
LimitCPU=18446744073709551615
LimitFSIZE=18446744073709551615
LimitDATA=18446744073709551615
LimitSTACK=18446744073709551615
LimitCORE=18446744073709551615
LimitRSS=18446744073709551615
LimitNOFILE=65536
LimitAS=18446744073709551615
LimitNPROC=65536
LimitMEMLOCK=65536
LimitLOCKS=18446744073709551615
LimitSIGPENDING=19827
LimitMSGQUEUE=819200
LimitNICE=0
LimitRTPRIO=0
LimitRTTIME=18446744073709551615

== Evaluate JVM commandline options ==

Before starting Java and the OSGi framework with all the Open-Xchange middleware bundles we have to evaluate several important JVM commandline options that influence the performance and stability of the Open-Xchange middleware stack. These commandline options are specified in '''/opt/open-xchange/etc/ox-scriptconf.sh'''

=== Format < 7.8.4 ===
For version up to 7.8.3 the JVM configuration options were all specified via a single property '''JAVA_XTRAOPTS''' which resulted in a rather long configuration string that was hard to read and maintain.

JAVA_XTRAOPTS="-Dsun.net.inetaddr.ttl=3600 -Dnetworkaddress.cache.ttl=3600 -Dnetworkaddress.cache.negative.ttl=10 \
-Dlogback.threadlocal.put.duplicate=false -server -Djava.awt.headless=true -Xmx512M -XX:MaxPermSize=256m -XX:+UseConcMarkSweepGC \
-XX:+UseParNewGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:NewRatio=3 -XX:+UseTLAB \
-Dosgi.compatibility.bootdelegation=false -XX:-OmitStackTraceInFastThrow"

During the startup of the open-xchange service (step 11 of diagram Service startup using systemd and step 10 of diagram Service startup using upstart) this JAVA_XTRAOPTS string goes through a basic validity check and is then passed to the JVM.

=== Format >= 7.8.4 ===
Beginning from version 7.8.4 the options are split up and sorted into different categories like:

JAVA_OPTS_GC="-XX:+UseConcMarkSweepGC -XX:+UseParNewGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:NewRatio=3 -XX:+DisableExplicitGC"
JAVA_OPTS_LOG="-Dlogback.threadlocal.put.duplicate=false -XX:-OmitStackTraceInFastThrow"
JAVA_OPTS_MEM="-XX:MaxHeapSize=512M -XX:MaxPermSize=256M -XX:+UseTLAB"
JAVA_OPTS_NET="-Dsun.net.inetaddr.ttl=3600 -Dnetworkaddress.cache.ttl=3600 -Dnetworkaddress.cache.negative.ttl=10"
JAVA_OPTS_OSGI="-Dosgi.compatibility.bootdelegation=false"
JAVA_OPTS_SERVER="-server -Djava.awt.headless=true"

JAVA_OPTS_OTHER=""

#JAVA_OPTS_DEBUG="-XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/mnt/heapdump -Xloggc:/var/log/open-xchange/gc.log -verbose:gc -XX:+PrintGCDateStamps \
-XX:+PrintHeapAtGC -XX:+PrintGCApplicationStoppedTime -XX:+PrintTenuringDistribution"

which is much easier to read an maintain. Furthermore we added a useful '''JAVA_OPTS_DEBUG''' configuration parameter that comes preconfigured but unused/commented so that customers can quickly enable the most commonly used debugging options in case they have to provide detailed system information to Open-Xchange.

During the startup of the open-xchange service (step 11 of diagram ''Service startup using systemd'' and step 10 of diagram ''Service startup using upstart'') all uncommented JAVA_OPTS_* categories will then be evaluated and combined again into a single string of commandline options that can then be passed to the JVM.

=== Upgrade procedure ===
During the upgrade '''7.8.3-revX -> 7.8.4-revY''' existing JAVA_XTRAOPTS will automatically be split up and migrated to the new categories approach. A backup of the file (ox-scriptconf.sh.timestamp) will be created and admins are urged to check for proper migration before deleting the backup file like it's already done with *.dpkg-dist or *.rpmnew files on Debian or RHEL/SLES distributions.

== Encoding detection during JVM startup ==

=== Background Infos ===

From ''man 7 locale''

DESCRIPTION
A locale is a set of language and cultural rules. These cover aspects such as lan‐
guage for messages, different character sets, lexicographic conventions, and so on.
A program needs to be able to determine its locale and act accordingly to be portable
to different cultures.

The format to specify a system locale is: '''''language[_territory][.codeset][@modifier]'''''.

Where all of

* en_US.UTF-8
* en_US.utf8
* en_US.utf-8
* en_US.UTF_8

will result in a valid UTF-8 charmap for the english language with locale categories configured as

LANG=en_US.UTF-8
LC_CTYPE="en_US.UTF-8"
LC_NUMERIC="en_US.UTF-8"
LC_TIME="en_US.UTF-8"
LC_COLLATE="en_US.UTF-8"
LC_MONETARY="en_US.UTF-8"
LC_MESSAGES="en_US.UTF-8"
LC_PAPER="en_US.UTF-8"
LC_NAME="en_US.UTF-8"
LC_ADDRESS="en_US.UTF-8"
LC_TELEPHONE="en_US.UTF-8"
LC_MEASUREMENT="en_US.UTF-8"
LC_IDENTIFICATION="en_US.UTF-8"
LC_ALL=

See for more infos: locale(1), locale(7), [https://en.wikipedia.org/wiki/List_of_ISO_639-1_codes language codes], [https://en.wikipedia.org/wiki/ISO_3166-1#Current_codes country codes] and [https://en.wikipedia.org/wiki/Character_encoding character encodings]

==== JVM startup ====

The default file.encoding detection is done at JVM startup based upon LC_CTYPE which makes sense as it's documented as:

CODESET (LC_CTYPE)
Return a string with the name of the character encoding used in the selected locale, such as "UTF-8", "ISO-8859-1",
or "ANSI_X3.4-1968" (better known as US-ASCII). This is the same string that you get with "locale charmap".
For a list of character encoding names, try "locale -m", cf. locale(1).

Furthermore this influences the defaultCharset selection of the JVM.

/**
* Returns the default charset of this Java virtual machine.
*
* The default charset is determined during virtual-machine startup and
* typically depends upon the locale and charset of the underlying
* operating system.
*
* @return A charset object for the default charset
*
* @since 1.5
*/
public static Charset defaultCharset() {
if (defaultCharset == null) {
synchronized (Charset.class) {
String csn = AccessController.doPrivileged(
new GetPropertyAction("file.encoding"));
Charset cs = lookup(csn);
if (cs != null)
defaultCharset = cs;
else
defaultCharset = forName("UTF-8");
}
}
return defaultCharset;
}

Although we try to specify valid unicode encodings for every IO operations the Open-Xchange middleware does some parts of the JVM don't allow to specify the encoding to use and falls back to the detected default encoding. That's why admins have to make sure to provide a proper unicode environment for the middleware service during every start or use documented workarounds provided by Open-Xchange (e.g. com.openexchange.passwordchange.script.base64: Encoded strings as Base64 to circumvent character encoding issues on improperly configured distributions see [https://documentation.open-xchange.com/7.8.3/middleware/configuration/properties.html property documentation])

See the documentation of your linux distribution on how to configure a proper unicode locale.

=== Query which file encoding was detected by the JVM during startup ===
[[File:visualvm-file.encoding.png|thumb|Querying the JVM via visualvm|right|300px]]

The file encoding that was detected by the JVM during startup can be queried via JMX. You can use graphical management tools like [https://visualvm.github.io/ visualvm], query our [[Jolokia]] JMX bridge via http or just use the commandline like

/opt/open-xchange/sbin/showruntimestats -r | sed -ne 's/.*${key=file.encoding, value=[^}]*}$.*/\1/p'
{key=file.encoding, value='''UTF-8'''}

 

=== Locale passing from client to server via SSH ===

The following setting of sshd allows clients to specify which locale to use on the server and might lead to some confusion or unexpected behaviour when checking the system locale or even to the usage of broken locales during middleware startup if your client sends misconfigured or missing locale configuration to the server.

grep AcceptEnv /etc/ssh/sshd_config
AcceptEnv LANG LC_*

For example when i connect to the machine my local german locale is sent to the server

[vagrant@showcase-community-centos-7 ~]$ locale -v
LANG=de_DE.UTF-8
LC_CTYPE="de_DE.UTF-8"
LC_NUMERIC=de_DE.UTF-8
LC_TIME=de_DE.UTF-8
LC_COLLATE="de_DE.UTF-8"
LC_MONETARY=de_DE.UTF-8
LC_MESSAGES="de_DE.UTF-8"
LC_PAPER=de_DE.UTF-8
LC_NAME=de_DE.UTF-8
LC_ADDRESS=de_DE.UTF-8
LC_TELEPHONE=de_DE.UTF-8
LC_MEASUREMENT=de_DE.UTF-8
LC_IDENTIFICATION=de_DE.UTF-8
LC_ALL=

But after specifying that i want a new login shell the default locale configured for that user/system will be used instead!

[root@showcase-community-centos-7 vagrant]# su -s /bin/bash open-xchange -l
-bash-4.2$ locale -v
LANG=en_US.UTF-8
LC_CTYPE="en_US.UTF-8"
LC_NUMERIC="en_US.UTF-8"
LC_TIME="en_US.UTF-8"
LC_COLLATE="en_US.UTF-8"
LC_MONETARY="en_US.UTF-8"
LC_MESSAGES="en_US.UTF-8"
LC_PAPER="en_US.UTF-8"
LC_NAME="en_US.UTF-8"
LC_ADDRESS="en_US.UTF-8"
LC_TELEPHONE="en_US.UTF-8"
LC_MEASUREMENT="en_US.UTF-8"
LC_IDENTIFICATION="en_US.UTF-8"
LC_ALL=

=== Verify that your system is configured with a proper unicode locale for OX ===

Assuming you haven't reconfigured the locale of your root user

i|marens@karotte ~ $ su -
Password:
i|karotte ~ # locale charmap
UTF-8
i|karotte ~ #

== Warnings during early start ==

Early warnings before the start of the JVM and related logging libraries like logback and logstash forwarding are redirected to '''/var/log/open-xchange/open-xchange-console.log'''.

=== Limits ===
If the open-xchange startup script isn't able to set the proper limits on CentOS 6 due to e.g. hard limits being enforced via pam_limits you'll see the following warning in the console log during startup

singlenode open-xchange # cat open-xchange-console.log
/opt/open-xchange/sbin/open-xchange: line 115: ulimit: max user processes: cannot modify limit: Operation not permitted

=== Locale ===
If the open-xchange startup script isn't able to detect a unicode locale you'll see a warning like the following (the current charmap might differ)

WARNING: Couldn't detect proper unicode charmap in locale category LC_CTYPE.
WARNING: Current charmap: "ANSI_X3.4-1968" might cause encoding problems.

== Summary: Open-Xchange middleware on specific distros ==
The support for the mentioned mechanism of resource control differ depending on the distribution and the init system in use.

=== Debian 7 ===
;Init
: System V style
;OX Configurable Limits/Defaults
: nofile, nproc

The mentioned limits can be configured via '''/opt/open-xchange/etc/ox-scriptconf.sh'''. The limits are applied via ulimit in the service's init script. The open-xchange service is finally started via start-stop-daemon which doesn't doesn't consider '''/etc/security/limits.*'''

=== RHEL 6 / CentOS 6 ===
;Init
: Upstart, System V compatible
;OX Configurable Limits/Defaults
: nofile, nproc

The mentioned limits can be configured via '''/opt/open-xchange/etc/ox-scriptconf.sh'''. The limits are applied via ulimit in the service's init script. Furthermore as the open-xchange service is finally started via '''su ... open-xchange''' on this distro a user session is opened via su/pam and the default CentOS pam config reads the '''/etc/security/limits.*''' configuration by loading the pam stack like:

: '''/etc/pam.d/su'''
:: '''-> /etc/pam.d/system-auth'''
::: '''-> pam_limits.so'''

If NPROC isn't configured for the open-xchange-server it's restricted to '''1024''' globally by default to prevent accidental fork bombs, see '''/etc/security/limits.d/90-nproc.conf''' which can result in severe problems modern multithreaded applications.

=== RHEL 7 / CentOS 7 / Debian 8 / SLE 12 ===
;Init
: Systemd
;OX Configurable Limits/Defaults
: nofile, nproc

For systemd the default limits are configured directly in the service's unit file that is shipped by OX and located at '''/usr/lib/systemd/system/open-xchange.service'''. The drop-in config to override or extend the default unit file is located at '''/etc/systemd/system/open-xchange.service.d/limits.conf'''. [https://www.freedesktop.org/software/systemd/man/systemd.exec.html Systemd.exec] shows a whole lot of options that can be used by admins to adapt the default service to their specific needs.

----

[[File:diagram_sources.zip]]

AppSuite:MiddlewareStartup

2017-04-25T08:59:50Z

Marens: /* Limits besides control groups */

== Systemd ==

==== Service definition ====
The default service definition as shipped by Open-Xchange is rather concise and easy to read. You can check by looking at the service file that is installed to either /usr/lib/systemd or /lib/systemd depending on your distribution of choice.

singlenode:~ # cat /usr/lib/systemd/system/open-xchange.service
[Unit]
After=remote-fs.target
After=time-sync.target ypbind.service sendmail.service cyrus.service

[Service]
User=open-xchange
PermissionsStartOnly=true
TimeoutStartSec=0
ExecStartPre=/opt/open-xchange/sbin/triggerupdatethemes -u
ExecStart=/opt/open-xchange/sbin/open-xchange
ExecStop=/opt/open-xchange/sbin/shutdown -w
ExecReload=/opt/open-xchange/sbin/triggerreloadconfiguration -d
KillMode=process
LimitNOFILE=65536
LimitNPROC=65536

[Install]
WantedBy=multi-user.target

==== Drop-in configs ====
Drop in configs allow administrators to easily adapt and override default service unit files. So if you want to change the default limits or add additional limits have a look at our default example drop-in config which is located at /etc/systemd/system/open-xchange.service.d/limits.conf

singlenode:~ # cat /etc/systemd/system/open-xchange.service.d/limits.conf
# Override and add options in this file
# See systemd.exec(5) for other limits

[Service]
#LimitNPROC=65536
#LimitNOFILE=65536

The [https://www.freedesktop.org/software/systemd/man/systemd.exec.html systemd.exec] documentation shows a whole lot of options that can be used by admins to adapt the default service to their specific needs.

 
[[File:mw_start_sysd.png|thumb|Service startup using systemd|left|1000px]]
 

''Note:'' The technical implementation of systemd differs as service definition and config aren't read every time during service start but the changes need to be reread with ''systemctl daemon-reload''. The above is just a simplyfied version to explain the general concept.

== System V / Upstart ==

==== Service definition ====

The default service definition as shipped by Open-Xchange can be found at /etc/init.d/open-xchange and will differ slightly depending on the distro in use.

 
[[File:mw_start_sysv.png|thumb|Service startup using upstart|left|1000px]]
 
== Apply Resource limits ==

As you can see in the diagrams above one of the differences is that:

* systemd reads all the infos wrt. resource limitations from the original service file and then applies possible further restrictions from the drop-in configs (steps 2 and 8 in the diagram ''Service startup using systemd'') and limits configured in /opt/open-xchange/etc/ox-scriptconf.sh aren't considered anylonger
* the older System V style init has to source /opt/open-xchange/etc/ox-scriptconf.sh to be able to read the configs for the maximum number of processes (NPROC) and files(NRFILES) that the middleware shall be allowed to create/open (steps 3,4 and 15 in the diagram ''Service startup using upstart'').

=== Background Infos ===

Several ways exist to restrict resources on a linux system from a global level down to user/groups or even shells and the processes started by them.

==== Sysctl ====
Sysctl is used to modify kernel parameters at runtime. E.g. to set the maximum number of files

$ sysctl -w fs.file-max=100000

To permanently set them append to the main configuration file and reload the settings

$ echo fs.file-max=100000 >> /etc/sysctl.conf
$ sysctl -p

More infos can be found via '''man sysctl'''

==== Limits.conf ====

Allows to restrict resources an a global, group or user level. E.g:

$ cat /etc/security/limits.d/90-nproc.conf
# Default limit for number of user's processes to prevent
# accidental fork bombs.
# See rhbz #432903 for reasoning.

* soft nproc 1024

From '''man limits.conf''':
<blockquote>
''Also, please note that all limit settings are set per login. They are not global, nor are they permanent; existing only for the duration of the session.''
</blockquote>

The limits per login are applied via the ''pam'' stack. See '''man pam''' and '''man pam_limits''' for more details. As those limits are bound to sessions they don't affect most daemons started by our supported init systems or init utils. Most state that they are ignored by design, see [https://bugs.launchpad.net/ubuntu/+source/upstart/+bug/938669/comments/4 upstart], [https://bugzilla.redhat.com/show_bug.cgi?id=754285#c1 systemd] and [https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=302079 start-stop-daemon]

===== Ulimit =====
From '''man bash'''
<blockquote>
ulimit [-HSTabcdefilmnpqrstuvx [limit]]
Provides control over the resources available to the shell and to processes started by it, on systems that allow such control.
</blockquote>
This is what we use in our System V compatible init scripts to increase resources for the open-xchange process across multiple distros. Currently only ''the maximum number of processes'' and ''the maximum number of open file descriptors available to a single user'' are increased via ulimit. The values are specified in '''/opt/open-xchange/ox-scriptconf.sh'''

==== Systemd ====

===== Control Groups=====

Control groups should only affect the OX middleware if you create/manage them yourself of if you are using a ''modern'' distribution that already uses systemd as init.

Citing from the kernel cgroup [https://www.kernel.org/doc/Documentation/cgroup-v2.txt documentation]:

<blockquote>
1-2. What is cgroup?

cgroup is a mechanism to organize processes hierarchically and
distribute system resources along the hierarchy in a controlled and
configurable manner.

cgroup is largely composed of two parts - the core and controllers.
cgroup core is primarily responsible for hierarchically organizing
processes. A cgroup controller is usually responsible for
distributing a specific type of system resource along the hierarchy
although there are utility controllers which serve purposes other than
resource distribution.

cgroups form a tree structure and every process in the system belongs
to one and only one cgroup. All threads of a process belong to the
same cgroup. On creation, all processes are put in the cgroup that
the parent process belongs to at the time. A process can be migrated
to another cgroup. Migration of a process doesn't affect already
existing descendant processes.

Following certain structural constraints, controllers may be enabled or
disabled selectively on a cgroup. All controller behaviors are
hierarchical - if a controller is enabled on a cgroup, it affects all
processes which belong to the cgroups consisting the inclusive
sub-hierarchy of the cgroup. When a controller is enabled on a nested
cgroup, it always restricts the resource distribution further. The
restrictions set closer to the root in the hierarchy can not be
overridden from further away.
</blockquote>

So processes are organized into a tree structure of control groups and controllers are responsible for the distribution of resources. So what kind of controllers exist?

<blockquote>
5. Controllers

5-1. CPU

The "cpu" controllers regulates distribution of CPU cycles. This
controller implements weight and absolute bandwidth limit models for
normal scheduling policy and absolute bandwidth allocation model for
realtime scheduling policy.

5-2. Memory

The "memory" controller regulates distribution of memory.
...
While not completely water-tight, all major memory usages by a given
cgroup are tracked so that the total memory consumption can be
accounted and controlled to a reasonable extent.

5-3. IO

The "io" controller regulates the distribution of IO resources. This
controller implements both weight based and absolute bandwidth or IOPS
limit distribution; however, weight based distribution is available
only if cfq-iosched is in use and neither scheme is available for
blk-mq devices.
</blockquote>

The open-xchange service is simply put into the default ''system.slice'' without applying further limits.

singlenode$ systemd-cgls
├─1 /sbin/init
├─system.slice
│ ├─avahi-daemon.service
│ │ ├─501 avahi-daemon: running [singlenode]
│ │ └─514 avahi-daemon: chroot helper
│ ├─console-kit-daemon.service
│ │ └─16164 /usr/sbin/console-kit-daemon --no-daemon
│ ├─dbus.service
│ │ └─508 /usr/bin/dbus-daemon --system --address=systemd: --nofork --nopidfile --systemd-activation
│ ├─munin-node.service
│ │ └─4290 /usr/bin/perl -wT /usr/sbin/munin-node
│ ├─open-xchange.service
│ │ └─6037 /usr/bin/java -Dsun.net.inetaddr.ttl=3600 -Dnetworkaddress.cache.ttl=3600 -Dnetworkaddress.cache.negative.ttl=10 ...

To check all the details use

singlenode:~ # systemctl show system.slice

==== Limits besides control groups ====
Besides control groups systemd allows you to apply other limits to the execution environment of your service. Here we can apply the limits that would normally be applied via ''limits.conf'' or ''ulimit''. Systemd uses [http://man7.org/linux/man-pages/man2/setrlimit.2.html setrlimit] for this. The options that we set by default are:

* LimitNOFILE
* LimitNPROC

which you have already seen when we took a look at the [[#Service_definiton|service definition]] and [[#Drop-in_config|drop-in configs]]. ''Again'': The complete documentation wrt. resource limit configurations in drop-in configs can be found at [https://www.freedesktop.org/software/systemd/man/systemd.exec.html systemd.exec]

== Verify limits ==

=== System V ===

Get the proper pid of the java process started for the middleware by either checking '''ps faux'''
root 4103 0.0 0.0 66528 1868 pts/0 S 22:05 0:00 su -s /bin/bash open-xchange -c /opt/open-xchange/sbin/open-xchange
494 4109 9.5 19.8 2224644 381312 ? Ssl 22:05 1:12 \_ /usr/bin/java -Dsun.net.inetaddr.ttl=3600 -Dnetworkaddress.cache.ttl=3600

or programmatically
singlenode:~ # pid=$(pgrep -P $(</var/run/open-xchange.pid) java)

and check the limits applied for that process
singlenode:~ # cat /proc/$pid/limits
Limit Soft Limit Hard Limit Units
Max cpu time unlimited unlimited seconds
Max file size unlimited unlimited bytes
Max data size unlimited unlimited bytes
Max stack size 8388608 unlimited bytes
Max core file size 0 unlimited bytes
Max resident set unlimited unlimited bytes
Max processes 65536 65536 processes
Max open files 65536 65536 files
Max locked memory 65536 65536 bytes
Max address space unlimited unlimited bytes
Max file locks unlimited unlimited locks
Max pending signals 24254 24254 signals
Max msgqueue size 819200 819200 bytes
Max nice priority 0 0
Max realtime priority 0 0
Max realtime timeout unlimited unlimited us

=== Systemd ===
singlenode:~ # systemctl show open-xchange | grep Limit
StartLimitInterval=10000000
StartLimitBurst=5
StartLimitAction=none
MemoryLimit=18446744073709551615
LimitCPU=18446744073709551615
LimitFSIZE=18446744073709551615
LimitDATA=18446744073709551615
LimitSTACK=18446744073709551615
LimitCORE=18446744073709551615
LimitRSS=18446744073709551615
LimitNOFILE=65536
LimitAS=18446744073709551615
LimitNPROC=65536
LimitMEMLOCK=65536
LimitLOCKS=18446744073709551615
LimitSIGPENDING=19827
LimitMSGQUEUE=819200
LimitNICE=0
LimitRTPRIO=0
LimitRTTIME=18446744073709551615

== Evaluate JVM commandline options ==

Before starting Java and the OSGi framework with all the Open-Xchange middleware bundles we have to evaluate several important JVM commandline options that influence the performance and stability of the Open-Xchange middleware stack. These commandline options are specified in '''/opt/open-xchange/etc/ox-scriptconf.sh'''

=== Format < 7.8.4 ===
For version up to 7.8.3 the JVM configuration options were all specified via a single property '''JAVA_XTRAOPTS''' which resulted in a rather long configuration string that was hard to read and maintain.

JAVA_XTRAOPTS="-Dsun.net.inetaddr.ttl=3600 -Dnetworkaddress.cache.ttl=3600 -Dnetworkaddress.cache.negative.ttl=10 \
-Dlogback.threadlocal.put.duplicate=false -server -Djava.awt.headless=true -Xmx512M -XX:MaxPermSize=256m -XX:+UseConcMarkSweepGC \
-XX:+UseParNewGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:NewRatio=3 -XX:+UseTLAB \
-Dosgi.compatibility.bootdelegation=false -XX:-OmitStackTraceInFastThrow"

During the startup of the open-xchange service (step 11 of diagram Service startup using systemd and step 10 of diagram Service startup using upstart) this JAVA_XTRAOPTS string goes through a basic validity check and is then passed to the JVM.

=== Format >= 7.8.4 ===
Beginning from version 7.8.4 the options are split up and sorted into different categories like:

JAVA_OPTS_GC="-XX:+UseConcMarkSweepGC -XX:+UseParNewGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:NewRatio=3 -XX:+DisableExplicitGC"
JAVA_OPTS_LOG="-Dlogback.threadlocal.put.duplicate=false -XX:-OmitStackTraceInFastThrow"
JAVA_OPTS_MEM="-XX:MaxHeapSize=512M -XX:MaxPermSize=256M -XX:+UseTLAB"
JAVA_OPTS_NET="-Dsun.net.inetaddr.ttl=3600 -Dnetworkaddress.cache.ttl=3600 -Dnetworkaddress.cache.negative.ttl=10"
JAVA_OPTS_OSGI="-Dosgi.compatibility.bootdelegation=false"
JAVA_OPTS_SERVER="-server -Djava.awt.headless=true"

JAVA_OPTS_OTHER=""

#JAVA_OPTS_DEBUG="-XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/mnt/heapdump -Xloggc:/var/log/open-xchange/gc.log -verbose:gc -XX:+PrintGCDateStamps \
-XX:+PrintHeapAtGC -XX:+PrintGCApplicationStoppedTime -XX:+PrintTenuringDistribution"

which is much easier to read an maintain. Furthermore we added a useful '''JAVA_OPTS_DEBUG''' configuration parameter that comes preconfigured but unused/commented so that customers can quickly enable the most commonly used debugging options in case they have to provide detailed system information to Open-Xchange.

During the startup of the open-xchange service (step 11 of diagram ''Service startup using systemd'' and step 10 of diagram ''Service startup using upstart'') all uncommented JAVA_OPTS_* categories will then be evaluated and combined again into a single string of commandline options that can then be passed to the JVM.

=== Upgrade procedure ===
During the upgrade '''7.8.3-revX -> 7.8.4-revY''' existing JAVA_XTRAOPTS will automatically be split up and migrated to the new categories approach. A backup of the file (ox-scriptconf.sh.timestamp) will be created and admins are urged to check for proper migration before deleting the backup file like it's already done with *.dpkg-dist or *.rpmnew files on Debian or RHEL/SLES distributions.

== Encoding detection during JVM startup ==

=== Background Infos ===

From ''man 7 locale''

DESCRIPTION
A locale is a set of language and cultural rules. These cover aspects such as lan‐
guage for messages, different character sets, lexicographic conventions, and so on.
A program needs to be able to determine its locale and act accordingly to be portable
to different cultures.

The format to specify a system locale is: '''''language[_territory][.codeset][@modifier]'''''.

Where all of

* en_US.UTF-8
* en_US.utf8
* en_US.utf-8
* en_US.UTF_8

will result in a valid UTF-8 charmap for the english language with locale categories configured as

LANG=en_US.UTF-8
LC_CTYPE="en_US.UTF-8"
LC_NUMERIC="en_US.UTF-8"
LC_TIME="en_US.UTF-8"
LC_COLLATE="en_US.UTF-8"
LC_MONETARY="en_US.UTF-8"
LC_MESSAGES="en_US.UTF-8"
LC_PAPER="en_US.UTF-8"
LC_NAME="en_US.UTF-8"
LC_ADDRESS="en_US.UTF-8"
LC_TELEPHONE="en_US.UTF-8"
LC_MEASUREMENT="en_US.UTF-8"
LC_IDENTIFICATION="en_US.UTF-8"
LC_ALL=

See for more infos: locale(1), locale(7), [https://en.wikipedia.org/wiki/List_of_ISO_639-1_codes language codes], [https://en.wikipedia.org/wiki/ISO_3166-1#Current_codes country codes] and [https://en.wikipedia.org/wiki/Character_encoding character encodings]

==== JVM startup ====

The default file.encoding detection is done at JVM startup based upon LC_CTYPE which makes sense as it's documented as:

CODESET (LC_CTYPE)
Return a string with the name of the character encoding used in the selected locale, such as "UTF-8", "ISO-8859-1",
or "ANSI_X3.4-1968" (better known as US-ASCII). This is the same string that you get with "locale charmap".
For a list of character encoding names, try "locale -m", cf. locale(1).

Furthermore this influences the defaultCharset selection of the JVM.

/**
* Returns the default charset of this Java virtual machine.
*
* The default charset is determined during virtual-machine startup and
* typically depends upon the locale and charset of the underlying
* operating system.
*
* @return A charset object for the default charset
*
* @since 1.5
*/
public static Charset defaultCharset() {
if (defaultCharset == null) {
synchronized (Charset.class) {
String csn = AccessController.doPrivileged(
new GetPropertyAction("file.encoding"));
Charset cs = lookup(csn);
if (cs != null)
defaultCharset = cs;
else
defaultCharset = forName("UTF-8");
}
}
return defaultCharset;
}

Although we try to specify valid unicode encodings for every IO operations the Open-Xchange middleware does some parts of the JVM don't allow to specify the encoding to use and falls back to the detected default encoding. That's why admins have to make sure to provide a proper unicode environment for the middleware service during every start or use documented workarounds provided by Open-Xchange (e.g. com.openexchange.passwordchange.script.base64: Encoded strings as Base64 to circumvent character encoding issues on improperly configured distributions see [https://documentation.open-xchange.com/7.8.3/middleware/configuration/properties.html property documentation])

See the documentation of your linux distribution on how to configure a proper unicode locale.

=== Query which file encoding was detected by the JVM during startup ===
[[File:visualvm-file.encoding.png|thumb|Querying the JVM via visualvm|right|300px]]

The file encoding that was detected by the JVM during startup can be queried via JMX. You can use graphical management tools like [https://visualvm.github.io/ visualvm], query our [[Jolokia]] JMX bridge via http or just use the commandline like

/opt/open-xchange/sbin/showruntimestats -r | sed -ne 's/.*${key=file.encoding, value=[^}]*}$.*/\1/p'
{key=file.encoding, value=UTF-8}

 

=== Locale passing from client to server via SSH ===

The following setting of sshd allows clients to specify which locale to use on the server and might lead to some confusion or unexpected behaviour when checking the system locale or even to the usage of broken locales during middleware startup if your client sends misconfigured or missing locale configuration to the server.

grep AcceptEnv /etc/ssh/sshd_config
AcceptEnv LANG LC_*

For example when i connect to the machine my local german locale is sent to the server

[vagrant@showcase-community-centos-7 ~]$ locale -v
LANG=de_DE.UTF-8
LC_CTYPE="de_DE.UTF-8"
LC_NUMERIC=de_DE.UTF-8
LC_TIME=de_DE.UTF-8
LC_COLLATE="de_DE.UTF-8"
LC_MONETARY=de_DE.UTF-8
LC_MESSAGES="de_DE.UTF-8"
LC_PAPER=de_DE.UTF-8
LC_NAME=de_DE.UTF-8
LC_ADDRESS=de_DE.UTF-8
LC_TELEPHONE=de_DE.UTF-8
LC_MEASUREMENT=de_DE.UTF-8
LC_IDENTIFICATION=de_DE.UTF-8
LC_ALL=

But after specifying that i want a new login shell the default locale configured for that user/system will be used instead!

[root@showcase-community-centos-7 vagrant]# su -s /bin/bash open-xchange -l
-bash-4.2$ locale -v
LANG=en_US.UTF-8
LC_CTYPE="en_US.UTF-8"
LC_NUMERIC="en_US.UTF-8"
LC_TIME="en_US.UTF-8"
LC_COLLATE="en_US.UTF-8"
LC_MONETARY="en_US.UTF-8"
LC_MESSAGES="en_US.UTF-8"
LC_PAPER="en_US.UTF-8"
LC_NAME="en_US.UTF-8"
LC_ADDRESS="en_US.UTF-8"
LC_TELEPHONE="en_US.UTF-8"
LC_MEASUREMENT="en_US.UTF-8"
LC_IDENTIFICATION="en_US.UTF-8"
LC_ALL=

=== Verify that your system is configured with a proper unicode locale for OX ===

Assuming you haven't reconfigured the locale of your root user

i|marens@karotte ~ $ su -
Password:
i|karotte ~ # locale charmap
UTF-8
i|karotte ~ #

== Warnings during early start ==

Early warnings before the start of the JVM and related logging libraries like logback and logstash forwarding are redirected to '''/var/log/open-xchange/open-xchange-console.log'''.

=== Limits ===
If the open-xchange startup script isn't able to set the proper limits on CentOS 6 due to e.g. hard limits being enforced via pam_limits you'll see the following warning in the console log during startup

singlenode open-xchange # cat open-xchange-console.log
/opt/open-xchange/sbin/open-xchange: line 115: ulimit: max user processes: cannot modify limit: Operation not permitted

=== Locale ===
If the open-xchange startup script isn't able to detect a unicode locale you'll see a warning like the following (the current charmap might differ)

WARNING: Couldn't detect proper unicode charmap in locale category LC_CTYPE.
WARNING: Current charmap: "ANSI_X3.4-1968" might cause encoding problems.

== Summary: Open-Xchange middleware on specific distros ==
The support for the mentioned mechanism of resource control differ depending on the distribution and the init system in use.

=== Debian 7 ===
;Init
: System V style
;OX Configurable Limits/Defaults
: nofile, nproc

The mentioned limits can be configured via '''/opt/open-xchange/etc/ox-scriptconf.sh'''. The limits are applied via ulimit in the service's init script. The open-xchange service is finally started via start-stop-daemon which doesn't doesn't consider '''/etc/security/limits.*'''

=== RHEL 6 / CentOS 6 ===
;Init
: Upstart, System V compatible
;OX Configurable Limits/Defaults
: nofile, nproc

The mentioned limits can be configured via '''/opt/open-xchange/etc/ox-scriptconf.sh'''. The limits are applied via ulimit in the service's init script. Furthermore as the open-xchange service is finally started via '''su ... open-xchange''' on this distro a user session is opened via su/pam and the default CentOS pam config reads the '''/etc/security/limits.*''' configuration by loading the pam stack like:

: '''/etc/pam.d/su'''
:: '''-> /etc/pam.d/system-auth'''
::: '''-> pam_limits.so'''

If NPROC isn't configured for the open-xchange-server it's restricted to '''1024''' globally by default to prevent accidental fork bombs, see '''/etc/security/limits.d/90-nproc.conf''' which can result in severe problems modern multithreaded applications.

=== RHEL 7 / CentOS 7 / Debian 8 / SLE 12 ===
;Init
: Systemd
;OX Configurable Limits/Defaults
: nofile, nproc

For systemd the default limits are configured directly in the service's unit file that is shipped by OX and located at '''/usr/lib/systemd/system/open-xchange.service'''. The drop-in config to override or extend the default unit file is located at '''/etc/systemd/system/open-xchange.service.d/limits.conf'''. [https://www.freedesktop.org/software/systemd/man/systemd.exec.html Systemd.exec] shows a whole lot of options that can be used by admins to adapt the default service to their specific needs.

----

[[File:diagram_sources.zip]]

AppSuite:MiddlewareStartup

2017-04-25T08:57:43Z

Marens: /* Drop-in configs */

== Systemd ==

==== Service definition ====
The default service definition as shipped by Open-Xchange is rather concise and easy to read. You can check by looking at the service file that is installed to either /usr/lib/systemd or /lib/systemd depending on your distribution of choice.

singlenode:~ # cat /usr/lib/systemd/system/open-xchange.service
[Unit]
After=remote-fs.target
After=time-sync.target ypbind.service sendmail.service cyrus.service

[Service]
User=open-xchange
PermissionsStartOnly=true
TimeoutStartSec=0
ExecStartPre=/opt/open-xchange/sbin/triggerupdatethemes -u
ExecStart=/opt/open-xchange/sbin/open-xchange
ExecStop=/opt/open-xchange/sbin/shutdown -w
ExecReload=/opt/open-xchange/sbin/triggerreloadconfiguration -d
KillMode=process
LimitNOFILE=65536
LimitNPROC=65536

[Install]
WantedBy=multi-user.target

==== Drop-in configs ====
Drop in configs allow administrators to easily adapt and override default service unit files. So if you want to change the default limits or add additional limits have a look at our default example drop-in config which is located at /etc/systemd/system/open-xchange.service.d/limits.conf

singlenode:~ # cat /etc/systemd/system/open-xchange.service.d/limits.conf
# Override and add options in this file
# See systemd.exec(5) for other limits

[Service]
#LimitNPROC=65536
#LimitNOFILE=65536

The [https://www.freedesktop.org/software/systemd/man/systemd.exec.html systemd.exec] documentation shows a whole lot of options that can be used by admins to adapt the default service to their specific needs.

 
[[File:mw_start_sysd.png|thumb|Service startup using systemd|left|1000px]]
 

''Note:'' The technical implementation of systemd differs as service definition and config aren't read every time during service start but the changes need to be reread with ''systemctl daemon-reload''. The above is just a simplyfied version to explain the general concept.

== System V / Upstart ==

==== Service definition ====

The default service definition as shipped by Open-Xchange can be found at /etc/init.d/open-xchange and will differ slightly depending on the distro in use.

 
[[File:mw_start_sysv.png|thumb|Service startup using upstart|left|1000px]]
 
== Apply Resource limits ==

As you can see in the diagrams above one of the differences is that:

* systemd reads all the infos wrt. resource limitations from the original service file and then applies possible further restrictions from the drop-in configs (steps 2 and 8 in the diagram ''Service startup using systemd'') and limits configured in /opt/open-xchange/etc/ox-scriptconf.sh aren't considered anylonger
* the older System V style init has to source /opt/open-xchange/etc/ox-scriptconf.sh to be able to read the configs for the maximum number of processes (NPROC) and files(NRFILES) that the middleware shall be allowed to create/open (steps 3,4 and 15 in the diagram ''Service startup using upstart'').

=== Background Infos ===

Several ways exist to restrict resources on a linux system from a global level down to user/groups or even shells and the processes started by them.

==== Sysctl ====
Sysctl is used to modify kernel parameters at runtime. E.g. to set the maximum number of files

$ sysctl -w fs.file-max=100000

To permanently set them append to the main configuration file and reload the settings

$ echo fs.file-max=100000 >> /etc/sysctl.conf
$ sysctl -p

More infos can be found via '''man sysctl'''

==== Limits.conf ====

Allows to restrict resources an a global, group or user level. E.g:

$ cat /etc/security/limits.d/90-nproc.conf
# Default limit for number of user's processes to prevent
# accidental fork bombs.
# See rhbz #432903 for reasoning.

* soft nproc 1024

From '''man limits.conf''':
<blockquote>
''Also, please note that all limit settings are set per login. They are not global, nor are they permanent; existing only for the duration of the session.''
</blockquote>

The limits per login are applied via the ''pam'' stack. See '''man pam''' and '''man pam_limits''' for more details. As those limits are bound to sessions they don't affect most daemons started by our supported init systems or init utils. Most state that they are ignored by design, see [https://bugs.launchpad.net/ubuntu/+source/upstart/+bug/938669/comments/4 upstart], [https://bugzilla.redhat.com/show_bug.cgi?id=754285#c1 systemd] and [https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=302079 start-stop-daemon]

===== Ulimit =====
From '''man bash'''
<blockquote>
ulimit [-HSTabcdefilmnpqrstuvx [limit]]
Provides control over the resources available to the shell and to processes started by it, on systems that allow such control.
</blockquote>
This is what we use in our System V compatible init scripts to increase resources for the open-xchange process across multiple distros. Currently only ''the maximum number of processes'' and ''the maximum number of open file descriptors available to a single user'' are increased via ulimit. The values are specified in '''/opt/open-xchange/ox-scriptconf.sh'''

==== Systemd ====

===== Control Groups=====

Control groups should only affect the OX middleware if you create/manage them yourself of if you are using a ''modern'' distribution that already uses systemd as init.

Citing from the kernel cgroup [https://www.kernel.org/doc/Documentation/cgroup-v2.txt documentation]:

<blockquote>
1-2. What is cgroup?

cgroup is a mechanism to organize processes hierarchically and
distribute system resources along the hierarchy in a controlled and
configurable manner.

cgroup is largely composed of two parts - the core and controllers.
cgroup core is primarily responsible for hierarchically organizing
processes. A cgroup controller is usually responsible for
distributing a specific type of system resource along the hierarchy
although there are utility controllers which serve purposes other than
resource distribution.

cgroups form a tree structure and every process in the system belongs
to one and only one cgroup. All threads of a process belong to the
same cgroup. On creation, all processes are put in the cgroup that
the parent process belongs to at the time. A process can be migrated
to another cgroup. Migration of a process doesn't affect already
existing descendant processes.

Following certain structural constraints, controllers may be enabled or
disabled selectively on a cgroup. All controller behaviors are
hierarchical - if a controller is enabled on a cgroup, it affects all
processes which belong to the cgroups consisting the inclusive
sub-hierarchy of the cgroup. When a controller is enabled on a nested
cgroup, it always restricts the resource distribution further. The
restrictions set closer to the root in the hierarchy can not be
overridden from further away.
</blockquote>

So processes are organized into a tree structure of control groups and controllers are responsible for the distribution of resources. So what kind of controllers exist?

<blockquote>
5. Controllers

5-1. CPU

The "cpu" controllers regulates distribution of CPU cycles. This
controller implements weight and absolute bandwidth limit models for
normal scheduling policy and absolute bandwidth allocation model for
realtime scheduling policy.

5-2. Memory

The "memory" controller regulates distribution of memory.
...
While not completely water-tight, all major memory usages by a given
cgroup are tracked so that the total memory consumption can be
accounted and controlled to a reasonable extent.

5-3. IO

The "io" controller regulates the distribution of IO resources. This
controller implements both weight based and absolute bandwidth or IOPS
limit distribution; however, weight based distribution is available
only if cfq-iosched is in use and neither scheme is available for
blk-mq devices.
</blockquote>

The open-xchange service is simply put into the default ''system.slice'' without applying further limits.

singlenode$ systemd-cgls
├─1 /sbin/init
├─system.slice
│ ├─avahi-daemon.service
│ │ ├─501 avahi-daemon: running [singlenode]
│ │ └─514 avahi-daemon: chroot helper
│ ├─console-kit-daemon.service
│ │ └─16164 /usr/sbin/console-kit-daemon --no-daemon
│ ├─dbus.service
│ │ └─508 /usr/bin/dbus-daemon --system --address=systemd: --nofork --nopidfile --systemd-activation
│ ├─munin-node.service
│ │ └─4290 /usr/bin/perl -wT /usr/sbin/munin-node
│ ├─open-xchange.service
│ │ └─6037 /usr/bin/java -Dsun.net.inetaddr.ttl=3600 -Dnetworkaddress.cache.ttl=3600 -Dnetworkaddress.cache.negative.ttl=10 ...

To check all the details use

singlenode:~ # systemctl show system.slice

==== Limits besides control groups ====
Besides control groups systemd allows you to apply other limits to the execution environment of your service. Here we can apply the limits that would normally be applied via ''limits.conf'' or ''ulimit''. Systemd uses [http://man7.org/linux/man-pages/man2/setrlimit.2.html setrlimit] for this. The options that we set by default are:

* LimitNOFILE
* LimitNPROC

which you have already seen when we took a look at the [[#Service_definiton|service definition]] and [[#Drop-in_config|drop-in configs]]

== Verify limits ==

=== System V ===

Get the proper pid of the java process started for the middleware by either checking '''ps faux'''
root 4103 0.0 0.0 66528 1868 pts/0 S 22:05 0:00 su -s /bin/bash open-xchange -c /opt/open-xchange/sbin/open-xchange
494 4109 9.5 19.8 2224644 381312 ? Ssl 22:05 1:12 \_ /usr/bin/java -Dsun.net.inetaddr.ttl=3600 -Dnetworkaddress.cache.ttl=3600

or programmatically
singlenode:~ # pid=$(pgrep -P $(</var/run/open-xchange.pid) java)

and check the limits applied for that process
singlenode:~ # cat /proc/$pid/limits
Limit Soft Limit Hard Limit Units
Max cpu time unlimited unlimited seconds
Max file size unlimited unlimited bytes
Max data size unlimited unlimited bytes
Max stack size 8388608 unlimited bytes
Max core file size 0 unlimited bytes
Max resident set unlimited unlimited bytes
Max processes 65536 65536 processes
Max open files 65536 65536 files
Max locked memory 65536 65536 bytes
Max address space unlimited unlimited bytes
Max file locks unlimited unlimited locks
Max pending signals 24254 24254 signals
Max msgqueue size 819200 819200 bytes
Max nice priority 0 0
Max realtime priority 0 0
Max realtime timeout unlimited unlimited us

=== Systemd ===
singlenode:~ # systemctl show open-xchange | grep Limit
StartLimitInterval=10000000
StartLimitBurst=5
StartLimitAction=none
MemoryLimit=18446744073709551615
LimitCPU=18446744073709551615
LimitFSIZE=18446744073709551615
LimitDATA=18446744073709551615
LimitSTACK=18446744073709551615
LimitCORE=18446744073709551615
LimitRSS=18446744073709551615
LimitNOFILE=65536
LimitAS=18446744073709551615
LimitNPROC=65536
LimitMEMLOCK=65536
LimitLOCKS=18446744073709551615
LimitSIGPENDING=19827
LimitMSGQUEUE=819200
LimitNICE=0
LimitRTPRIO=0
LimitRTTIME=18446744073709551615

== Evaluate JVM commandline options ==

Before starting Java and the OSGi framework with all the Open-Xchange middleware bundles we have to evaluate several important JVM commandline options that influence the performance and stability of the Open-Xchange middleware stack. These commandline options are specified in '''/opt/open-xchange/etc/ox-scriptconf.sh'''

=== Format < 7.8.4 ===
For version up to 7.8.3 the JVM configuration options were all specified via a single property '''JAVA_XTRAOPTS''' which resulted in a rather long configuration string that was hard to read and maintain.

JAVA_XTRAOPTS="-Dsun.net.inetaddr.ttl=3600 -Dnetworkaddress.cache.ttl=3600 -Dnetworkaddress.cache.negative.ttl=10 \
-Dlogback.threadlocal.put.duplicate=false -server -Djava.awt.headless=true -Xmx512M -XX:MaxPermSize=256m -XX:+UseConcMarkSweepGC \
-XX:+UseParNewGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:NewRatio=3 -XX:+UseTLAB \
-Dosgi.compatibility.bootdelegation=false -XX:-OmitStackTraceInFastThrow"

During the startup of the open-xchange service (step 11 of diagram Service startup using systemd and step 10 of diagram Service startup using upstart) this JAVA_XTRAOPTS string goes through a basic validity check and is then passed to the JVM.

=== Format >= 7.8.4 ===
Beginning from version 7.8.4 the options are split up and sorted into different categories like:

JAVA_OPTS_GC="-XX:+UseConcMarkSweepGC -XX:+UseParNewGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:NewRatio=3 -XX:+DisableExplicitGC"
JAVA_OPTS_LOG="-Dlogback.threadlocal.put.duplicate=false -XX:-OmitStackTraceInFastThrow"
JAVA_OPTS_MEM="-XX:MaxHeapSize=512M -XX:MaxPermSize=256M -XX:+UseTLAB"
JAVA_OPTS_NET="-Dsun.net.inetaddr.ttl=3600 -Dnetworkaddress.cache.ttl=3600 -Dnetworkaddress.cache.negative.ttl=10"
JAVA_OPTS_OSGI="-Dosgi.compatibility.bootdelegation=false"
JAVA_OPTS_SERVER="-server -Djava.awt.headless=true"

JAVA_OPTS_OTHER=""

#JAVA_OPTS_DEBUG="-XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/mnt/heapdump -Xloggc:/var/log/open-xchange/gc.log -verbose:gc -XX:+PrintGCDateStamps \
-XX:+PrintHeapAtGC -XX:+PrintGCApplicationStoppedTime -XX:+PrintTenuringDistribution"

which is much easier to read an maintain. Furthermore we added a useful '''JAVA_OPTS_DEBUG''' configuration parameter that comes preconfigured but unused/commented so that customers can quickly enable the most commonly used debugging options in case they have to provide detailed system information to Open-Xchange.

During the startup of the open-xchange service (step 11 of diagram ''Service startup using systemd'' and step 10 of diagram ''Service startup using upstart'') all uncommented JAVA_OPTS_* categories will then be evaluated and combined again into a single string of commandline options that can then be passed to the JVM.

=== Upgrade procedure ===
During the upgrade '''7.8.3-revX -> 7.8.4-revY''' existing JAVA_XTRAOPTS will automatically be split up and migrated to the new categories approach. A backup of the file (ox-scriptconf.sh.timestamp) will be created and admins are urged to check for proper migration before deleting the backup file like it's already done with *.dpkg-dist or *.rpmnew files on Debian or RHEL/SLES distributions.

== Encoding detection during JVM startup ==

=== Background Infos ===

From ''man 7 locale''

DESCRIPTION
A locale is a set of language and cultural rules. These cover aspects such as lan‐
guage for messages, different character sets, lexicographic conventions, and so on.
A program needs to be able to determine its locale and act accordingly to be portable
to different cultures.

The format to specify a system locale is: '''''language[_territory][.codeset][@modifier]'''''.

Where all of

* en_US.UTF-8
* en_US.utf8
* en_US.utf-8
* en_US.UTF_8

will result in a valid UTF-8 charmap for the english language with locale categories configured as

LANG=en_US.UTF-8
LC_CTYPE="en_US.UTF-8"
LC_NUMERIC="en_US.UTF-8"
LC_TIME="en_US.UTF-8"
LC_COLLATE="en_US.UTF-8"
LC_MONETARY="en_US.UTF-8"
LC_MESSAGES="en_US.UTF-8"
LC_PAPER="en_US.UTF-8"
LC_NAME="en_US.UTF-8"
LC_ADDRESS="en_US.UTF-8"
LC_TELEPHONE="en_US.UTF-8"
LC_MEASUREMENT="en_US.UTF-8"
LC_IDENTIFICATION="en_US.UTF-8"
LC_ALL=

See for more infos: locale(1), locale(7), [https://en.wikipedia.org/wiki/List_of_ISO_639-1_codes language codes], [https://en.wikipedia.org/wiki/ISO_3166-1#Current_codes country codes] and [https://en.wikipedia.org/wiki/Character_encoding character encodings]

==== JVM startup ====

The default file.encoding detection is done at JVM startup based upon LC_CTYPE which makes sense as it's documented as:

CODESET (LC_CTYPE)
Return a string with the name of the character encoding used in the selected locale, such as "UTF-8", "ISO-8859-1",
or "ANSI_X3.4-1968" (better known as US-ASCII). This is the same string that you get with "locale charmap".
For a list of character encoding names, try "locale -m", cf. locale(1).

Furthermore this influences the defaultCharset selection of the JVM.

/**
* Returns the default charset of this Java virtual machine.
*
* The default charset is determined during virtual-machine startup and
* typically depends upon the locale and charset of the underlying
* operating system.
*
* @return A charset object for the default charset
*
* @since 1.5
*/
public static Charset defaultCharset() {
if (defaultCharset == null) {
synchronized (Charset.class) {
String csn = AccessController.doPrivileged(
new GetPropertyAction("file.encoding"));
Charset cs = lookup(csn);
if (cs != null)
defaultCharset = cs;
else
defaultCharset = forName("UTF-8");
}
}
return defaultCharset;
}

Although we try to specify valid unicode encodings for every IO operations the Open-Xchange middleware does some parts of the JVM don't allow to specify the encoding to use and falls back to the detected default encoding. That's why admins have to make sure to provide a proper unicode environment for the middleware service during every start or use documented workarounds provided by Open-Xchange (e.g. com.openexchange.passwordchange.script.base64: Encoded strings as Base64 to circumvent character encoding issues on improperly configured distributions see [https://documentation.open-xchange.com/7.8.3/middleware/configuration/properties.html property documentation])

See the documentation of your linux distribution on how to configure a proper unicode locale.

=== Query which file encoding was detected by the JVM during startup ===
[[File:visualvm-file.encoding.png|thumb|Querying the JVM via visualvm|right|300px]]

The file encoding that was detected by the JVM during startup can be queried via JMX. You can use graphical management tools like [https://visualvm.github.io/ visualvm], query our [[Jolokia]] JMX bridge via http or just use the commandline like

/opt/open-xchange/sbin/showruntimestats -r | sed -ne 's/.*${key=file.encoding, value=[^}]*}$.*/\1/p'
{key=file.encoding, value=UTF-8}

 

=== Locale passing from client to server via SSH ===

The following setting of sshd allows clients to specify which locale to use on the server and might lead to some confusion or unexpected behaviour when checking the system locale or even to the usage of broken locales during middleware startup if your client sends misconfigured or missing locale configuration to the server.

grep AcceptEnv /etc/ssh/sshd_config
AcceptEnv LANG LC_*

For example when i connect to the machine my local german locale is sent to the server

[vagrant@showcase-community-centos-7 ~]$ locale -v
LANG=de_DE.UTF-8
LC_CTYPE="de_DE.UTF-8"
LC_NUMERIC=de_DE.UTF-8
LC_TIME=de_DE.UTF-8
LC_COLLATE="de_DE.UTF-8"
LC_MONETARY=de_DE.UTF-8
LC_MESSAGES="de_DE.UTF-8"
LC_PAPER=de_DE.UTF-8
LC_NAME=de_DE.UTF-8
LC_ADDRESS=de_DE.UTF-8
LC_TELEPHONE=de_DE.UTF-8
LC_MEASUREMENT=de_DE.UTF-8
LC_IDENTIFICATION=de_DE.UTF-8
LC_ALL=

But after specifying that i want a new login shell the default locale configured for that user/system will be used instead!

[root@showcase-community-centos-7 vagrant]# su -s /bin/bash open-xchange -l
-bash-4.2$ locale -v
LANG=en_US.UTF-8
LC_CTYPE="en_US.UTF-8"
LC_NUMERIC="en_US.UTF-8"
LC_TIME="en_US.UTF-8"
LC_COLLATE="en_US.UTF-8"
LC_MONETARY="en_US.UTF-8"
LC_MESSAGES="en_US.UTF-8"
LC_PAPER="en_US.UTF-8"
LC_NAME="en_US.UTF-8"
LC_ADDRESS="en_US.UTF-8"
LC_TELEPHONE="en_US.UTF-8"
LC_MEASUREMENT="en_US.UTF-8"
LC_IDENTIFICATION="en_US.UTF-8"
LC_ALL=

=== Verify that your system is configured with a proper unicode locale for OX ===

Assuming you haven't reconfigured the locale of your root user

i|marens@karotte ~ $ su -
Password:
i|karotte ~ # locale charmap
UTF-8
i|karotte ~ #

== Warnings during early start ==

Early warnings before the start of the JVM and related logging libraries like logback and logstash forwarding are redirected to '''/var/log/open-xchange/open-xchange-console.log'''.

=== Limits ===
If the open-xchange startup script isn't able to set the proper limits on CentOS 6 due to e.g. hard limits being enforced via pam_limits you'll see the following warning in the console log during startup

singlenode open-xchange # cat open-xchange-console.log
/opt/open-xchange/sbin/open-xchange: line 115: ulimit: max user processes: cannot modify limit: Operation not permitted

=== Locale ===
If the open-xchange startup script isn't able to detect a unicode locale you'll see a warning like the following (the current charmap might differ)

WARNING: Couldn't detect proper unicode charmap in locale category LC_CTYPE.
WARNING: Current charmap: "ANSI_X3.4-1968" might cause encoding problems.

== Summary: Open-Xchange middleware on specific distros ==
The support for the mentioned mechanism of resource control differ depending on the distribution and the init system in use.

=== Debian 7 ===
;Init
: System V style
;OX Configurable Limits/Defaults
: nofile, nproc

The mentioned limits can be configured via '''/opt/open-xchange/etc/ox-scriptconf.sh'''. The limits are applied via ulimit in the service's init script. The open-xchange service is finally started via start-stop-daemon which doesn't doesn't consider '''/etc/security/limits.*'''

=== RHEL 6 / CentOS 6 ===
;Init
: Upstart, System V compatible
;OX Configurable Limits/Defaults
: nofile, nproc

The mentioned limits can be configured via '''/opt/open-xchange/etc/ox-scriptconf.sh'''. The limits are applied via ulimit in the service's init script. Furthermore as the open-xchange service is finally started via '''su ... open-xchange''' on this distro a user session is opened via su/pam and the default CentOS pam config reads the '''/etc/security/limits.*''' configuration by loading the pam stack like:

: '''/etc/pam.d/su'''
:: '''-> /etc/pam.d/system-auth'''
::: '''-> pam_limits.so'''

If NPROC isn't configured for the open-xchange-server it's restricted to '''1024''' globally by default to prevent accidental fork bombs, see '''/etc/security/limits.d/90-nproc.conf''' which can result in severe problems modern multithreaded applications.

=== RHEL 7 / CentOS 7 / Debian 8 / SLE 12 ===
;Init
: Systemd
;OX Configurable Limits/Defaults
: nofile, nproc

For systemd the default limits are configured directly in the service's unit file that is shipped by OX and located at '''/usr/lib/systemd/system/open-xchange.service'''. The drop-in config to override or extend the default unit file is located at '''/etc/systemd/system/open-xchange.service.d/limits.conf'''. [https://www.freedesktop.org/software/systemd/man/systemd.exec.html Systemd.exec] shows a whole lot of options that can be used by admins to adapt the default service to their specific needs.

----

[[File:diagram_sources.zip]]

AppSuite:MiddlewareStartup

2017-04-25T08:51:36Z

Marens: /* Drop-in configs */

== Systemd ==

==== Service definition ====
The default service definition as shipped by Open-Xchange is rather concise and easy to read. You can check by looking at the service file that is installed to either /usr/lib/systemd or /lib/systemd depending on your distribution of choice.

singlenode:~ # cat /usr/lib/systemd/system/open-xchange.service
[Unit]
After=remote-fs.target
After=time-sync.target ypbind.service sendmail.service cyrus.service

[Service]
User=open-xchange
PermissionsStartOnly=true
TimeoutStartSec=0
ExecStartPre=/opt/open-xchange/sbin/triggerupdatethemes -u
ExecStart=/opt/open-xchange/sbin/open-xchange
ExecStop=/opt/open-xchange/sbin/shutdown -w
ExecReload=/opt/open-xchange/sbin/triggerreloadconfiguration -d
KillMode=process
LimitNOFILE=65536
LimitNPROC=65536

[Install]
WantedBy=multi-user.target

==== Drop-in configs ====
Drop in configs allow administrators to easily adapt and override default service unit files. So if you want to change the default limits or add additional limits have a look at our default example drop-in config which is located at /etc/systemd/system/open-xchange.service.d/limits.conf

singlenode:~ # cat /etc/systemd/system/open-xchange.service.d/limits.conf
# Override and add options in this file
# See systemd.exec(5) for other limits

[Service]
#LimitNPROC=65536
#LimitNOFILE=65536

The systemd.exec documentation shows a whole lot of options that can be used by admins to adapt the default service to their specific needs.

 
[[File:mw_start_sysd.png|thumb|Service startup using systemd|left|1000px]]
 

''Note:'' The technical implementation of systemd differs as service definition and config aren't read every time during service start but the changes need to be reread with ''systemctl daemon-reload''. The above is just a simplyfied version to explain the general concept.

== System V / Upstart ==

==== Service definition ====

The default service definition as shipped by Open-Xchange can be found at /etc/init.d/open-xchange and will differ slightly depending on the distro in use.

 
[[File:mw_start_sysv.png|thumb|Service startup using upstart|left|1000px]]
 
== Apply Resource limits ==

As you can see in the diagrams above one of the differences is that:

* systemd reads all the infos wrt. resource limitations from the original service file and then applies possible further restrictions from the drop-in configs (steps 2 and 8 in the diagram ''Service startup using systemd'') and limits configured in /opt/open-xchange/etc/ox-scriptconf.sh aren't considered anylonger
* the older System V style init has to source /opt/open-xchange/etc/ox-scriptconf.sh to be able to read the configs for the maximum number of processes (NPROC) and files(NRFILES) that the middleware shall be allowed to create/open (steps 3,4 and 15 in the diagram ''Service startup using upstart'').

=== Background Infos ===

Several ways exist to restrict resources on a linux system from a global level down to user/groups or even shells and the processes started by them.

==== Sysctl ====
Sysctl is used to modify kernel parameters at runtime. E.g. to set the maximum number of files

$ sysctl -w fs.file-max=100000

To permanently set them append to the main configuration file and reload the settings

$ echo fs.file-max=100000 >> /etc/sysctl.conf
$ sysctl -p

More infos can be found via '''man sysctl'''

==== Limits.conf ====

Allows to restrict resources an a global, group or user level. E.g:

$ cat /etc/security/limits.d/90-nproc.conf
# Default limit for number of user's processes to prevent
# accidental fork bombs.
# See rhbz #432903 for reasoning.

* soft nproc 1024

From '''man limits.conf''':
<blockquote>
''Also, please note that all limit settings are set per login. They are not global, nor are they permanent; existing only for the duration of the session.''
</blockquote>

The limits per login are applied via the ''pam'' stack. See '''man pam''' and '''man pam_limits''' for more details. As those limits are bound to sessions they don't affect most daemons started by our supported init systems or init utils. Most state that they are ignored by design, see [https://bugs.launchpad.net/ubuntu/+source/upstart/+bug/938669/comments/4 upstart], [https://bugzilla.redhat.com/show_bug.cgi?id=754285#c1 systemd] and [https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=302079 start-stop-daemon]

===== Ulimit =====
From '''man bash'''
<blockquote>
ulimit [-HSTabcdefilmnpqrstuvx [limit]]
Provides control over the resources available to the shell and to processes started by it, on systems that allow such control.
</blockquote>
This is what we use in our System V compatible init scripts to increase resources for the open-xchange process across multiple distros. Currently only ''the maximum number of processes'' and ''the maximum number of open file descriptors available to a single user'' are increased via ulimit. The values are specified in '''/opt/open-xchange/ox-scriptconf.sh'''

==== Systemd ====

===== Control Groups=====

Control groups should only affect the OX middleware if you create/manage them yourself of if you are using a ''modern'' distribution that already uses systemd as init.

Citing from the kernel cgroup [https://www.kernel.org/doc/Documentation/cgroup-v2.txt documentation]:

<blockquote>
1-2. What is cgroup?

cgroup is a mechanism to organize processes hierarchically and
distribute system resources along the hierarchy in a controlled and
configurable manner.

cgroup is largely composed of two parts - the core and controllers.
cgroup core is primarily responsible for hierarchically organizing
processes. A cgroup controller is usually responsible for
distributing a specific type of system resource along the hierarchy
although there are utility controllers which serve purposes other than
resource distribution.

cgroups form a tree structure and every process in the system belongs
to one and only one cgroup. All threads of a process belong to the
same cgroup. On creation, all processes are put in the cgroup that
the parent process belongs to at the time. A process can be migrated
to another cgroup. Migration of a process doesn't affect already
existing descendant processes.

Following certain structural constraints, controllers may be enabled or
disabled selectively on a cgroup. All controller behaviors are
hierarchical - if a controller is enabled on a cgroup, it affects all
processes which belong to the cgroups consisting the inclusive
sub-hierarchy of the cgroup. When a controller is enabled on a nested
cgroup, it always restricts the resource distribution further. The
restrictions set closer to the root in the hierarchy can not be
overridden from further away.
</blockquote>

So processes are organized into a tree structure of control groups and controllers are responsible for the distribution of resources. So what kind of controllers exist?

<blockquote>
5. Controllers

5-1. CPU

The "cpu" controllers regulates distribution of CPU cycles. This
controller implements weight and absolute bandwidth limit models for
normal scheduling policy and absolute bandwidth allocation model for
realtime scheduling policy.

5-2. Memory

The "memory" controller regulates distribution of memory.
...
While not completely water-tight, all major memory usages by a given
cgroup are tracked so that the total memory consumption can be
accounted and controlled to a reasonable extent.

5-3. IO

The "io" controller regulates the distribution of IO resources. This
controller implements both weight based and absolute bandwidth or IOPS
limit distribution; however, weight based distribution is available
only if cfq-iosched is in use and neither scheme is available for
blk-mq devices.
</blockquote>

The open-xchange service is simply put into the default ''system.slice'' without applying further limits.

singlenode$ systemd-cgls
├─1 /sbin/init
├─system.slice
│ ├─avahi-daemon.service
│ │ ├─501 avahi-daemon: running [singlenode]
│ │ └─514 avahi-daemon: chroot helper
│ ├─console-kit-daemon.service
│ │ └─16164 /usr/sbin/console-kit-daemon --no-daemon
│ ├─dbus.service
│ │ └─508 /usr/bin/dbus-daemon --system --address=systemd: --nofork --nopidfile --systemd-activation
│ ├─munin-node.service
│ │ └─4290 /usr/bin/perl -wT /usr/sbin/munin-node
│ ├─open-xchange.service
│ │ └─6037 /usr/bin/java -Dsun.net.inetaddr.ttl=3600 -Dnetworkaddress.cache.ttl=3600 -Dnetworkaddress.cache.negative.ttl=10 ...

To check all the details use

singlenode:~ # systemctl show system.slice

==== Limits besides control groups ====
Besides control groups systemd allows you to apply other limits to the execution environment of your service. Here we can apply the limits that would normally be applied via ''limits.conf'' or ''ulimit''. Systemd uses [http://man7.org/linux/man-pages/man2/setrlimit.2.html setrlimit] for this. The options that we set by default are:

* LimitNOFILE
* LimitNPROC

which you have already seen when we took a look at the [[#Service_definiton|service definition]] and [[#Drop-in_config|drop-in configs]]

== Verify limits ==

=== System V ===

Get the proper pid of the java process started for the middleware by either checking '''ps faux'''
root 4103 0.0 0.0 66528 1868 pts/0 S 22:05 0:00 su -s /bin/bash open-xchange -c /opt/open-xchange/sbin/open-xchange
494 4109 9.5 19.8 2224644 381312 ? Ssl 22:05 1:12 \_ /usr/bin/java -Dsun.net.inetaddr.ttl=3600 -Dnetworkaddress.cache.ttl=3600

or programmatically
singlenode:~ # pid=$(pgrep -P $(</var/run/open-xchange.pid) java)

and check the limits applied for that process
singlenode:~ # cat /proc/$pid/limits
Limit Soft Limit Hard Limit Units
Max cpu time unlimited unlimited seconds
Max file size unlimited unlimited bytes
Max data size unlimited unlimited bytes
Max stack size 8388608 unlimited bytes
Max core file size 0 unlimited bytes
Max resident set unlimited unlimited bytes
Max processes 65536 65536 processes
Max open files 65536 65536 files
Max locked memory 65536 65536 bytes
Max address space unlimited unlimited bytes
Max file locks unlimited unlimited locks
Max pending signals 24254 24254 signals
Max msgqueue size 819200 819200 bytes
Max nice priority 0 0
Max realtime priority 0 0
Max realtime timeout unlimited unlimited us

=== Systemd ===
singlenode:~ # systemctl show open-xchange | grep Limit
StartLimitInterval=10000000
StartLimitBurst=5
StartLimitAction=none
MemoryLimit=18446744073709551615
LimitCPU=18446744073709551615
LimitFSIZE=18446744073709551615
LimitDATA=18446744073709551615
LimitSTACK=18446744073709551615
LimitCORE=18446744073709551615
LimitRSS=18446744073709551615
LimitNOFILE=65536
LimitAS=18446744073709551615
LimitNPROC=65536
LimitMEMLOCK=65536
LimitLOCKS=18446744073709551615
LimitSIGPENDING=19827
LimitMSGQUEUE=819200
LimitNICE=0
LimitRTPRIO=0
LimitRTTIME=18446744073709551615

== Evaluate JVM commandline options ==

Before starting Java and the OSGi framework with all the Open-Xchange middleware bundles we have to evaluate several important JVM commandline options that influence the performance and stability of the Open-Xchange middleware stack. These commandline options are specified in '''/opt/open-xchange/etc/ox-scriptconf.sh'''

=== Format < 7.8.4 ===
For version up to 7.8.3 the JVM configuration options were all specified via a single property '''JAVA_XTRAOPTS''' which resulted in a rather long configuration string that was hard to read and maintain.

JAVA_XTRAOPTS="-Dsun.net.inetaddr.ttl=3600 -Dnetworkaddress.cache.ttl=3600 -Dnetworkaddress.cache.negative.ttl=10 \
-Dlogback.threadlocal.put.duplicate=false -server -Djava.awt.headless=true -Xmx512M -XX:MaxPermSize=256m -XX:+UseConcMarkSweepGC \
-XX:+UseParNewGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:NewRatio=3 -XX:+UseTLAB \
-Dosgi.compatibility.bootdelegation=false -XX:-OmitStackTraceInFastThrow"

During the startup of the open-xchange service (step 11 of diagram Service startup using systemd and step 10 of diagram Service startup using upstart) this JAVA_XTRAOPTS string goes through a basic validity check and is then passed to the JVM.

=== Format >= 7.8.4 ===
Beginning from version 7.8.4 the options are split up and sorted into different categories like:

JAVA_OPTS_GC="-XX:+UseConcMarkSweepGC -XX:+UseParNewGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:NewRatio=3 -XX:+DisableExplicitGC"
JAVA_OPTS_LOG="-Dlogback.threadlocal.put.duplicate=false -XX:-OmitStackTraceInFastThrow"
JAVA_OPTS_MEM="-XX:MaxHeapSize=512M -XX:MaxPermSize=256M -XX:+UseTLAB"
JAVA_OPTS_NET="-Dsun.net.inetaddr.ttl=3600 -Dnetworkaddress.cache.ttl=3600 -Dnetworkaddress.cache.negative.ttl=10"
JAVA_OPTS_OSGI="-Dosgi.compatibility.bootdelegation=false"
JAVA_OPTS_SERVER="-server -Djava.awt.headless=true"

JAVA_OPTS_OTHER=""

#JAVA_OPTS_DEBUG="-XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/mnt/heapdump -Xloggc:/var/log/open-xchange/gc.log -verbose:gc -XX:+PrintGCDateStamps \
-XX:+PrintHeapAtGC -XX:+PrintGCApplicationStoppedTime -XX:+PrintTenuringDistribution"

which is much easier to read an maintain. Furthermore we added a useful '''JAVA_OPTS_DEBUG''' configuration parameter that comes preconfigured but unused/commented so that customers can quickly enable the most commonly used debugging options in case they have to provide detailed system information to Open-Xchange.

During the startup of the open-xchange service (step 11 of diagram ''Service startup using systemd'' and step 10 of diagram ''Service startup using upstart'') all uncommented JAVA_OPTS_* categories will then be evaluated and combined again into a single string of commandline options that can then be passed to the JVM.

=== Upgrade procedure ===
During the upgrade '''7.8.3-revX -> 7.8.4-revY''' existing JAVA_XTRAOPTS will automatically be split up and migrated to the new categories approach. A backup of the file (ox-scriptconf.sh.timestamp) will be created and admins are urged to check for proper migration before deleting the backup file like it's already done with *.dpkg-dist or *.rpmnew files on Debian or RHEL/SLES distributions.

== Encoding detection during JVM startup ==

=== Background Infos ===

From ''man 7 locale''

DESCRIPTION
A locale is a set of language and cultural rules. These cover aspects such as lan‐
guage for messages, different character sets, lexicographic conventions, and so on.
A program needs to be able to determine its locale and act accordingly to be portable
to different cultures.

The format to specify a system locale is: '''''language[_territory][.codeset][@modifier]'''''.

Where all of

* en_US.UTF-8
* en_US.utf8
* en_US.utf-8
* en_US.UTF_8

will result in a valid UTF-8 charmap for the english language with locale categories configured as

LANG=en_US.UTF-8
LC_CTYPE="en_US.UTF-8"
LC_NUMERIC="en_US.UTF-8"
LC_TIME="en_US.UTF-8"
LC_COLLATE="en_US.UTF-8"
LC_MONETARY="en_US.UTF-8"
LC_MESSAGES="en_US.UTF-8"
LC_PAPER="en_US.UTF-8"
LC_NAME="en_US.UTF-8"
LC_ADDRESS="en_US.UTF-8"
LC_TELEPHONE="en_US.UTF-8"
LC_MEASUREMENT="en_US.UTF-8"
LC_IDENTIFICATION="en_US.UTF-8"
LC_ALL=

See for more infos: locale(1), locale(7), [https://en.wikipedia.org/wiki/List_of_ISO_639-1_codes language codes], [https://en.wikipedia.org/wiki/ISO_3166-1#Current_codes country codes] and [https://en.wikipedia.org/wiki/Character_encoding character encodings]

==== JVM startup ====

The default file.encoding detection is done at JVM startup based upon LC_CTYPE which makes sense as it's documented as:

CODESET (LC_CTYPE)
Return a string with the name of the character encoding used in the selected locale, such as "UTF-8", "ISO-8859-1",
or "ANSI_X3.4-1968" (better known as US-ASCII). This is the same string that you get with "locale charmap".
For a list of character encoding names, try "locale -m", cf. locale(1).

Furthermore this influences the defaultCharset selection of the JVM.

/**
* Returns the default charset of this Java virtual machine.
*
* The default charset is determined during virtual-machine startup and
* typically depends upon the locale and charset of the underlying
* operating system.
*
* @return A charset object for the default charset
*
* @since 1.5
*/
public static Charset defaultCharset() {
if (defaultCharset == null) {
synchronized (Charset.class) {
String csn = AccessController.doPrivileged(
new GetPropertyAction("file.encoding"));
Charset cs = lookup(csn);
if (cs != null)
defaultCharset = cs;
else
defaultCharset = forName("UTF-8");
}
}
return defaultCharset;
}

Although we try to specify valid unicode encodings for every IO operations the Open-Xchange middleware does some parts of the JVM don't allow to specify the encoding to use and falls back to the detected default encoding. That's why admins have to make sure to provide a proper unicode environment for the middleware service during every start or use documented workarounds provided by Open-Xchange (e.g. com.openexchange.passwordchange.script.base64: Encoded strings as Base64 to circumvent character encoding issues on improperly configured distributions see [https://documentation.open-xchange.com/7.8.3/middleware/configuration/properties.html property documentation])

See the documentation of your linux distribution on how to configure a proper unicode locale.

=== Query which file encoding was detected by the JVM during startup ===
[[File:visualvm-file.encoding.png|thumb|Querying the JVM via visualvm|right|300px]]

The file encoding that was detected by the JVM during startup can be queried via JMX. You can use graphical management tools like [https://visualvm.github.io/ visualvm], query our [[Jolokia]] JMX bridge via http or just use the commandline like

/opt/open-xchange/sbin/showruntimestats -r | sed -ne 's/.*${key=file.encoding, value=[^}]*}$.*/\1/p'
{key=file.encoding, value=UTF-8}

 

=== Locale passing from client to server via SSH ===

The following setting of sshd allows clients to specify which locale to use on the server and might lead to some confusion or unexpected behaviour when checking the system locale or even to the usage of broken locales during middleware startup if your client sends misconfigured or missing locale configuration to the server.

grep AcceptEnv /etc/ssh/sshd_config
AcceptEnv LANG LC_*

For example when i connect to the machine my local german locale is sent to the server

[vagrant@showcase-community-centos-7 ~]$ locale -v
LANG=de_DE.UTF-8
LC_CTYPE="de_DE.UTF-8"
LC_NUMERIC=de_DE.UTF-8
LC_TIME=de_DE.UTF-8
LC_COLLATE="de_DE.UTF-8"
LC_MONETARY=de_DE.UTF-8
LC_MESSAGES="de_DE.UTF-8"
LC_PAPER=de_DE.UTF-8
LC_NAME=de_DE.UTF-8
LC_ADDRESS=de_DE.UTF-8
LC_TELEPHONE=de_DE.UTF-8
LC_MEASUREMENT=de_DE.UTF-8
LC_IDENTIFICATION=de_DE.UTF-8
LC_ALL=

But after specifying that i want a new login shell the default locale configured for that user/system will be used instead!

[root@showcase-community-centos-7 vagrant]# su -s /bin/bash open-xchange -l
-bash-4.2$ locale -v
LANG=en_US.UTF-8
LC_CTYPE="en_US.UTF-8"
LC_NUMERIC="en_US.UTF-8"
LC_TIME="en_US.UTF-8"
LC_COLLATE="en_US.UTF-8"
LC_MONETARY="en_US.UTF-8"
LC_MESSAGES="en_US.UTF-8"
LC_PAPER="en_US.UTF-8"
LC_NAME="en_US.UTF-8"
LC_ADDRESS="en_US.UTF-8"
LC_TELEPHONE="en_US.UTF-8"
LC_MEASUREMENT="en_US.UTF-8"
LC_IDENTIFICATION="en_US.UTF-8"
LC_ALL=

=== Verify that your system is configured with a proper unicode locale for OX ===

Assuming you haven't reconfigured the locale of your root user

i|marens@karotte ~ $ su -
Password:
i|karotte ~ # locale charmap
UTF-8
i|karotte ~ #

== Warnings during early start ==

Early warnings before the start of the JVM and related logging libraries like logback and logstash forwarding are redirected to '''/var/log/open-xchange/open-xchange-console.log'''.

=== Limits ===
If the open-xchange startup script isn't able to set the proper limits on CentOS 6 due to e.g. hard limits being enforced via pam_limits you'll see the following warning in the console log during startup

singlenode open-xchange # cat open-xchange-console.log
/opt/open-xchange/sbin/open-xchange: line 115: ulimit: max user processes: cannot modify limit: Operation not permitted

=== Locale ===
If the open-xchange startup script isn't able to detect a unicode locale you'll see a warning like the following (the current charmap might differ)

WARNING: Couldn't detect proper unicode charmap in locale category LC_CTYPE.
WARNING: Current charmap: "ANSI_X3.4-1968" might cause encoding problems.

== Summary: Open-Xchange middleware on specific distros ==
The support for the mentioned mechanism of resource control differ depending on the distribution and the init system in use.

=== Debian 7 ===
;Init
: System V style
;OX Configurable Limits/Defaults
: nofile, nproc

The mentioned limits can be configured via '''/opt/open-xchange/etc/ox-scriptconf.sh'''. The limits are applied via ulimit in the service's init script. The open-xchange service is finally started via start-stop-daemon which doesn't doesn't consider '''/etc/security/limits.*'''

=== RHEL 6 / CentOS 6 ===
;Init
: Upstart, System V compatible
;OX Configurable Limits/Defaults
: nofile, nproc

The mentioned limits can be configured via '''/opt/open-xchange/etc/ox-scriptconf.sh'''. The limits are applied via ulimit in the service's init script. Furthermore as the open-xchange service is finally started via '''su ... open-xchange''' on this distro a user session is opened via su/pam and the default CentOS pam config reads the '''/etc/security/limits.*''' configuration by loading the pam stack like:

: '''/etc/pam.d/su'''
:: '''-> /etc/pam.d/system-auth'''
::: '''-> pam_limits.so'''

If NPROC isn't configured for the open-xchange-server it's restricted to '''1024''' globally by default to prevent accidental fork bombs, see '''/etc/security/limits.d/90-nproc.conf''' which can result in severe problems modern multithreaded applications.

=== RHEL 7 / CentOS 7 / Debian 8 / SLE 12 ===
;Init
: Systemd
;OX Configurable Limits/Defaults
: nofile, nproc

For systemd the default limits are configured directly in the service's unit file that is shipped by OX and located at '''/usr/lib/systemd/system/open-xchange.service'''. The drop-in config to override or extend the default unit file is located at '''/etc/systemd/system/open-xchange.service.d/limits.conf'''. [https://www.freedesktop.org/software/systemd/man/systemd.exec.html Systemd.exec] shows a whole lot of options that can be used by admins to adapt the default service to their specific needs.

----

[[File:diagram_sources.zip]]

AppSuite:MiddlewareStartup

2017-04-24T21:46:51Z

Marens: /* JVM startup */

== Systemd ==

==== Service definition ====
The default service definition as shipped by Open-Xchange is rather concise and easy to read. You can check by looking at the service file that is installed to either /usr/lib/systemd or /lib/systemd depending on your distribution of choice.

singlenode:~ # cat /usr/lib/systemd/system/open-xchange.service
[Unit]
After=remote-fs.target
After=time-sync.target ypbind.service sendmail.service cyrus.service

[Service]
User=open-xchange
PermissionsStartOnly=true
TimeoutStartSec=0
ExecStartPre=/opt/open-xchange/sbin/triggerupdatethemes -u
ExecStart=/opt/open-xchange/sbin/open-xchange
ExecStop=/opt/open-xchange/sbin/shutdown -w
ExecReload=/opt/open-xchange/sbin/triggerreloadconfiguration -d
KillMode=process
LimitNOFILE=65536
LimitNPROC=65536

[Install]
WantedBy=multi-user.target

==== Drop-in configs ====
Drop in configs allow administrators to easily adapt and override default service unit files. So if you want to change the default limits or add additional limits have a look at our default example drop-in config which is located at /etc/systemd/system/open-xchange.service.d/limits.conf

singlenode:~ # cat /etc/systemd/system/open-xchange.service.d/limits.conf
# Override and add options in this file
# See systemd.exec(5) for other limits

[Service]
#LimitNPROC=65536
#LimitNOFILE=65536

The systemd.exec documentation shows a whole lot of options that can be used by admins to adapt the default service to their specific needs.

 
[[File:mw_start_sysd.png|thumb|Service startup using systemd|left|1000px]]
 

== System V / Upstart ==

==== Service definition ====

The default service definition as shipped by Open-Xchange can be found at /etc/init.d/open-xchange and will differ slightly depending on the distro in use.

 
[[File:mw_start_sysv.png|thumb|Service startup using upstart|left|1000px]]
 
== Apply Resource limits ==

As you can see in the diagrams above one of the differences is that:

* systemd reads all the infos wrt. resource limitations from the original service file and then applies possible further restrictions from the drop-in configs (steps 2 and 8 in the diagram ''Service startup using systemd'') and limits configured in /opt/open-xchange/etc/ox-scriptconf.sh aren't considered anylonger
* the older System V style init has to source /opt/open-xchange/etc/ox-scriptconf.sh to be able to read the configs for the maximum number of processes (NPROC) and files(NRFILES) that the middleware shall be allowed to create/open (steps 3,4 and 15 in the diagram ''Service startup using upstart'').

=== Background Infos ===

Several ways exist to restrict resources on a linux system from a global level down to user/groups or even shells and the processes started by them.

==== Sysctl ====
Sysctl is used to modify kernel parameters at runtime. E.g. to set the maximum number of files

$ sysctl -w fs.file-max=100000

To permanently set them append to the main configuration file and reload the settings

$ echo fs.file-max=100000 >> /etc/sysctl.conf
$ sysctl -p

More infos can be found via '''man sysctl'''

==== Limits.conf ====

Allows to restrict resources an a global, group or user level. E.g:

$ cat /etc/security/limits.d/90-nproc.conf
# Default limit for number of user's processes to prevent
# accidental fork bombs.
# See rhbz #432903 for reasoning.

* soft nproc 1024

From '''man limits.conf''':
<blockquote>
''Also, please note that all limit settings are set per login. They are not global, nor are they permanent; existing only for the duration of the session.''
</blockquote>

The limits per login are applied via the ''pam'' stack. See '''man pam''' and '''man pam_limits''' for more details. As those limits are bound to sessions they don't affect most daemons started by our supported init systems or init utils. Most state that they are ignored by design, see [https://bugs.launchpad.net/ubuntu/+source/upstart/+bug/938669/comments/4 upstart], [https://bugzilla.redhat.com/show_bug.cgi?id=754285#c1 systemd] and [https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=302079 start-stop-daemon]

===== Ulimit =====
From '''man bash'''
<blockquote>
ulimit [-HSTabcdefilmnpqrstuvx [limit]]
Provides control over the resources available to the shell and to processes started by it, on systems that allow such control.
</blockquote>
This is what we use in our System V compatible init scripts to increase resources for the open-xchange process across multiple distros. Currently only ''the maximum number of processes'' and ''the maximum number of open file descriptors available to a single user'' are increased via ulimit. The values are specified in '''/opt/open-xchange/ox-scriptconf.sh'''

==== Systemd ====

===== Control Groups=====

Control groups should only affect the OX middleware if you create/manage them yourself of if you are using a ''modern'' distribution that already uses systemd as init.

Citing from the kernel cgroup [https://www.kernel.org/doc/Documentation/cgroup-v2.txt documentation]:

<blockquote>
1-2. What is cgroup?

cgroup is a mechanism to organize processes hierarchically and
distribute system resources along the hierarchy in a controlled and
configurable manner.

cgroup is largely composed of two parts - the core and controllers.
cgroup core is primarily responsible for hierarchically organizing
processes. A cgroup controller is usually responsible for
distributing a specific type of system resource along the hierarchy
although there are utility controllers which serve purposes other than
resource distribution.

cgroups form a tree structure and every process in the system belongs
to one and only one cgroup. All threads of a process belong to the
same cgroup. On creation, all processes are put in the cgroup that
the parent process belongs to at the time. A process can be migrated
to another cgroup. Migration of a process doesn't affect already
existing descendant processes.

Following certain structural constraints, controllers may be enabled or
disabled selectively on a cgroup. All controller behaviors are
hierarchical - if a controller is enabled on a cgroup, it affects all
processes which belong to the cgroups consisting the inclusive
sub-hierarchy of the cgroup. When a controller is enabled on a nested
cgroup, it always restricts the resource distribution further. The
restrictions set closer to the root in the hierarchy can not be
overridden from further away.
</blockquote>

So processes are organized into a tree structure of control groups and controllers are responsible for the distribution of resources. So what kind of controllers exist?

<blockquote>
5. Controllers

5-1. CPU

The "cpu" controllers regulates distribution of CPU cycles. This
controller implements weight and absolute bandwidth limit models for
normal scheduling policy and absolute bandwidth allocation model for
realtime scheduling policy.

5-2. Memory

The "memory" controller regulates distribution of memory.
...
While not completely water-tight, all major memory usages by a given
cgroup are tracked so that the total memory consumption can be
accounted and controlled to a reasonable extent.

5-3. IO

The "io" controller regulates the distribution of IO resources. This
controller implements both weight based and absolute bandwidth or IOPS
limit distribution; however, weight based distribution is available
only if cfq-iosched is in use and neither scheme is available for
blk-mq devices.
</blockquote>

The open-xchange service is simply put into the default ''system.slice'' without applying further limits.

singlenode$ systemd-cgls
├─1 /sbin/init
├─system.slice
│ ├─avahi-daemon.service
│ │ ├─501 avahi-daemon: running [singlenode]
│ │ └─514 avahi-daemon: chroot helper
│ ├─console-kit-daemon.service
│ │ └─16164 /usr/sbin/console-kit-daemon --no-daemon
│ ├─dbus.service
│ │ └─508 /usr/bin/dbus-daemon --system --address=systemd: --nofork --nopidfile --systemd-activation
│ ├─munin-node.service
│ │ └─4290 /usr/bin/perl -wT /usr/sbin/munin-node
│ ├─open-xchange.service
│ │ └─6037 /usr/bin/java -Dsun.net.inetaddr.ttl=3600 -Dnetworkaddress.cache.ttl=3600 -Dnetworkaddress.cache.negative.ttl=10 ...

To check all the details use

singlenode:~ # systemctl show system.slice

==== Limits besides control groups ====
Besides control groups systemd allows you to apply other limits to the execution environment of your service. Here we can apply the limits that would normally be applied via ''limits.conf'' or ''ulimit''. Systemd uses [http://man7.org/linux/man-pages/man2/setrlimit.2.html setrlimit] for this. The options that we set by default are:

* LimitNOFILE
* LimitNPROC

which you have already seen when we took a look at the [[#Service_definiton|service definition]] and [[#Drop-in_config|drop-in configs]]

== Verify limits ==

=== System V ===

Get the proper pid of the java process started for the middleware by either checking '''ps faux'''
root 4103 0.0 0.0 66528 1868 pts/0 S 22:05 0:00 su -s /bin/bash open-xchange -c /opt/open-xchange/sbin/open-xchange
494 4109 9.5 19.8 2224644 381312 ? Ssl 22:05 1:12 \_ /usr/bin/java -Dsun.net.inetaddr.ttl=3600 -Dnetworkaddress.cache.ttl=3600

or programmatically
singlenode:~ # pid=$(pgrep -P $(</var/run/open-xchange.pid) java)

and check the limits applied for that process
singlenode:~ # cat /proc/$pid/limits
Limit Soft Limit Hard Limit Units
Max cpu time unlimited unlimited seconds
Max file size unlimited unlimited bytes
Max data size unlimited unlimited bytes
Max stack size 8388608 unlimited bytes
Max core file size 0 unlimited bytes
Max resident set unlimited unlimited bytes
Max processes 65536 65536 processes
Max open files 65536 65536 files
Max locked memory 65536 65536 bytes
Max address space unlimited unlimited bytes
Max file locks unlimited unlimited locks
Max pending signals 24254 24254 signals
Max msgqueue size 819200 819200 bytes
Max nice priority 0 0
Max realtime priority 0 0
Max realtime timeout unlimited unlimited us

=== Systemd ===
singlenode:~ # systemctl show open-xchange | grep Limit
StartLimitInterval=10000000
StartLimitBurst=5
StartLimitAction=none
MemoryLimit=18446744073709551615
LimitCPU=18446744073709551615
LimitFSIZE=18446744073709551615
LimitDATA=18446744073709551615
LimitSTACK=18446744073709551615
LimitCORE=18446744073709551615
LimitRSS=18446744073709551615
LimitNOFILE=65536
LimitAS=18446744073709551615
LimitNPROC=65536
LimitMEMLOCK=65536
LimitLOCKS=18446744073709551615
LimitSIGPENDING=19827
LimitMSGQUEUE=819200
LimitNICE=0
LimitRTPRIO=0
LimitRTTIME=18446744073709551615

== Evaluate JVM commandline options ==

Before starting Java and the OSGi framework with all the Open-Xchange middleware bundles we have to evaluate several important JVM commandline options that influence the performance and stability of the Open-Xchange middleware stack. These commandline options are specified in '''/opt/open-xchange/etc/ox-scriptconf.sh'''

=== Format < 7.8.4 ===
For version up to 7.8.3 the JVM configuration options were all specified via a single property '''JAVA_XTRAOPTS''' which resulted in a rather long configuration string that was hard to read and maintain.

JAVA_XTRAOPTS="-Dsun.net.inetaddr.ttl=3600 -Dnetworkaddress.cache.ttl=3600 -Dnetworkaddress.cache.negative.ttl=10 \
-Dlogback.threadlocal.put.duplicate=false -server -Djava.awt.headless=true -Xmx512M -XX:MaxPermSize=256m -XX:+UseConcMarkSweepGC \
-XX:+UseParNewGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:NewRatio=3 -XX:+UseTLAB \
-Dosgi.compatibility.bootdelegation=false -XX:-OmitStackTraceInFastThrow"

During the startup of the open-xchange service (step 11 of diagram Service startup using systemd and step 10 of diagram Service startup using upstart) this JAVA_XTRAOPTS string goes through a basic validity check and is then passed to the JVM.

=== Format >= 7.8.4 ===
Beginning from version 7.8.4 the options are split up and sorted into different categories like:

JAVA_OPTS_GC="-XX:+UseConcMarkSweepGC -XX:+UseParNewGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:NewRatio=3 -XX:+DisableExplicitGC"
JAVA_OPTS_LOG="-Dlogback.threadlocal.put.duplicate=false -XX:-OmitStackTraceInFastThrow"
JAVA_OPTS_MEM="-XX:MaxHeapSize=512M -XX:MaxPermSize=256M -XX:+UseTLAB"
JAVA_OPTS_NET="-Dsun.net.inetaddr.ttl=3600 -Dnetworkaddress.cache.ttl=3600 -Dnetworkaddress.cache.negative.ttl=10"
JAVA_OPTS_OSGI="-Dosgi.compatibility.bootdelegation=false"
JAVA_OPTS_SERVER="-server -Djava.awt.headless=true"

JAVA_OPTS_OTHER=""

#JAVA_OPTS_DEBUG="-XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/mnt/heapdump -Xloggc:/var/log/open-xchange/gc.log -verbose:gc -XX:+PrintGCDateStamps \
-XX:+PrintHeapAtGC -XX:+PrintGCApplicationStoppedTime -XX:+PrintTenuringDistribution"

which is much easier to read an maintain. Furthermore we added a useful '''JAVA_OPTS_DEBUG''' configuration parameter that comes preconfigured but unused/commented so that customers can quickly enable the most commonly used debugging options in case they have to provide detailed system information to Open-Xchange.

During the startup of the open-xchange service (step 11 of diagram ''Service startup using systemd'' and step 10 of diagram ''Service startup using upstart'') all uncommented JAVA_OPTS_* categories will then be evaluated and combined again into a single string of commandline options that can then be passed to the JVM.

=== Upgrade procedure ===
During the upgrade '''7.8.3-revX -> 7.8.4-revY''' existing JAVA_XTRAOPTS will automatically be split up and migrated to the new categories approach. A backup of the file (ox-scriptconf.sh.timestamp) will be created and admins are urged to check for proper migration before deleting the backup file like it's already done with *.dpkg-dist or *.rpmnew files on Debian or RHEL/SLES distributions.

== Encoding detection during JVM startup ==

=== Background Infos ===

From ''man 7 locale''

DESCRIPTION
A locale is a set of language and cultural rules. These cover aspects such as lan‐
guage for messages, different character sets, lexicographic conventions, and so on.
A program needs to be able to determine its locale and act accordingly to be portable
to different cultures.

The format to specify a system locale is: '''''language[_territory][.codeset][@modifier]'''''.

Where all of

* en_US.UTF-8
* en_US.utf8
* en_US.utf-8
* en_US.UTF_8

will result in a valid UTF-8 charmap for the english language with locale categories configured as

LANG=en_US.UTF-8
LC_CTYPE="en_US.UTF-8"
LC_NUMERIC="en_US.UTF-8"
LC_TIME="en_US.UTF-8"
LC_COLLATE="en_US.UTF-8"
LC_MONETARY="en_US.UTF-8"
LC_MESSAGES="en_US.UTF-8"
LC_PAPER="en_US.UTF-8"
LC_NAME="en_US.UTF-8"
LC_ADDRESS="en_US.UTF-8"
LC_TELEPHONE="en_US.UTF-8"
LC_MEASUREMENT="en_US.UTF-8"
LC_IDENTIFICATION="en_US.UTF-8"
LC_ALL=

See for more infos: locale(1), locale(7), [https://en.wikipedia.org/wiki/List_of_ISO_639-1_codes language codes], [https://en.wikipedia.org/wiki/ISO_3166-1#Current_codes country codes] and [https://en.wikipedia.org/wiki/Character_encoding character encodings]

==== JVM startup ====

The default file.encoding detection is done at JVM startup based upon LC_CTYPE which makes sense as it's documented as:

CODESET (LC_CTYPE)
Return a string with the name of the character encoding used in the selected locale, such as "UTF-8", "ISO-8859-1",
or "ANSI_X3.4-1968" (better known as US-ASCII). This is the same string that you get with "locale charmap".
For a list of character encoding names, try "locale -m", cf. locale(1).

Furthermore this influences the defaultCharset selection of the JVM.

/**
* Returns the default charset of this Java virtual machine.
*
* The default charset is determined during virtual-machine startup and
* typically depends upon the locale and charset of the underlying
* operating system.
*
* @return A charset object for the default charset
*
* @since 1.5
*/
public static Charset defaultCharset() {
if (defaultCharset == null) {
synchronized (Charset.class) {
String csn = AccessController.doPrivileged(
new GetPropertyAction("file.encoding"));
Charset cs = lookup(csn);
if (cs != null)
defaultCharset = cs;
else
defaultCharset = forName("UTF-8");
}
}
return defaultCharset;
}

Although we try to specify valid unicode encodings for every IO operations the Open-Xchange middleware does some parts of the JVM don't allow to specify the encoding to use and falls back to the detected default encoding. That's why admins have to make sure to provide a proper unicode environment for the middleware service during every start or use documented workarounds provided by Open-Xchange (e.g. com.openexchange.passwordchange.script.base64: Encoded strings as Base64 to circumvent character encoding issues on improperly configured distributions see [https://documentation.open-xchange.com/7.8.3/middleware/configuration/properties.html property documentation])

See the documentation of your linux distribution on how to configure a proper unicode locale.

=== Query which file encoding was detected by the JVM during startup ===
[[File:visualvm-file.encoding.png|thumb|Querying the JVM via visualvm|right|300px]]

The file encoding that was detected by the JVM during startup can be queried via JMX. You can use graphical management tools like [https://visualvm.github.io/ visualvm], query our [[Jolokia]] JMX bridge via http or just use the commandline like

/opt/open-xchange/sbin/showruntimestats -r | sed -ne 's/.*${key=file.encoding, value=[^}]*}$.*/\1/p'
{key=file.encoding, value=UTF-8}

 

=== Locale passing from client to server via SSH ===

The following setting of sshd allows clients to specify which locale to use on the server and might lead to some confusion or unexpected behaviour when checking the system locale or even to the usage of broken locales during middleware startup if your client sends misconfigured or missing locale configuration to the server.

grep AcceptEnv /etc/ssh/sshd_config
AcceptEnv LANG LC_*

For example when i connect to the machine my local german locale is sent to the server

[vagrant@showcase-community-centos-7 ~]$ locale -v
LANG=de_DE.UTF-8
LC_CTYPE="de_DE.UTF-8"
LC_NUMERIC=de_DE.UTF-8
LC_TIME=de_DE.UTF-8
LC_COLLATE="de_DE.UTF-8"
LC_MONETARY=de_DE.UTF-8
LC_MESSAGES="de_DE.UTF-8"
LC_PAPER=de_DE.UTF-8
LC_NAME=de_DE.UTF-8
LC_ADDRESS=de_DE.UTF-8
LC_TELEPHONE=de_DE.UTF-8
LC_MEASUREMENT=de_DE.UTF-8
LC_IDENTIFICATION=de_DE.UTF-8
LC_ALL=

But after specifying that i want a new login shell the default locale configured for that user/system will be used instead!

[root@showcase-community-centos-7 vagrant]# su -s /bin/bash open-xchange -l
-bash-4.2$ locale -v
LANG=en_US.UTF-8
LC_CTYPE="en_US.UTF-8"
LC_NUMERIC="en_US.UTF-8"
LC_TIME="en_US.UTF-8"
LC_COLLATE="en_US.UTF-8"
LC_MONETARY="en_US.UTF-8"
LC_MESSAGES="en_US.UTF-8"
LC_PAPER="en_US.UTF-8"
LC_NAME="en_US.UTF-8"
LC_ADDRESS="en_US.UTF-8"
LC_TELEPHONE="en_US.UTF-8"
LC_MEASUREMENT="en_US.UTF-8"
LC_IDENTIFICATION="en_US.UTF-8"
LC_ALL=

=== Verify that your system is configured with a proper unicode locale for OX ===

Assuming you haven't reconfigured the locale of your root user

i|marens@karotte ~ $ su -
Password:
i|karotte ~ # locale charmap
UTF-8
i|karotte ~ #

== Warnings during early start ==

Early warnings before the start of the JVM and related logging libraries like logback and logstash forwarding are redirected to '''/var/log/open-xchange/open-xchange-console.log'''.

=== Limits ===
If the open-xchange startup script isn't able to set the proper limits on CentOS 6 due to e.g. hard limits being enforced via pam_limits you'll see the following warning in the console log during startup

singlenode open-xchange # cat open-xchange-console.log
/opt/open-xchange/sbin/open-xchange: line 115: ulimit: max user processes: cannot modify limit: Operation not permitted

=== Locale ===
If the open-xchange startup script isn't able to detect a unicode locale you'll see a warning like the following (the current charmap might differ)

WARNING: Couldn't detect proper unicode charmap in locale category LC_CTYPE.
WARNING: Current charmap: "ANSI_X3.4-1968" might cause encoding problems.

== Summary: Open-Xchange middleware on specific distros ==
The support for the mentioned mechanism of resource control differ depending on the distribution and the init system in use.

=== Debian 7 ===
;Init
: System V style
;OX Configurable Limits/Defaults
: nofile, nproc

The mentioned limits can be configured via '''/opt/open-xchange/etc/ox-scriptconf.sh'''. The limits are applied via ulimit in the service's init script. The open-xchange service is finally started via start-stop-daemon which doesn't doesn't consider '''/etc/security/limits.*'''

=== RHEL 6 / CentOS 6 ===
;Init
: Upstart, System V compatible
;OX Configurable Limits/Defaults
: nofile, nproc

The mentioned limits can be configured via '''/opt/open-xchange/etc/ox-scriptconf.sh'''. The limits are applied via ulimit in the service's init script. Furthermore as the open-xchange service is finally started via '''su ... open-xchange''' on this distro a user session is opened via su/pam and the default CentOS pam config reads the '''/etc/security/limits.*''' configuration by loading the pam stack like:

: '''/etc/pam.d/su'''
:: '''-> /etc/pam.d/system-auth'''
::: '''-> pam_limits.so'''

If NPROC isn't configured for the open-xchange-server it's restricted to '''1024''' globally by default to prevent accidental fork bombs, see '''/etc/security/limits.d/90-nproc.conf''' which can result in severe problems modern multithreaded applications.

=== RHEL 7 / CentOS 7 / Debian 8 / SLE 12 ===
;Init
: Systemd
;OX Configurable Limits/Defaults
: nofile, nproc

For systemd the default limits are configured directly in the service's unit file that is shipped by OX and located at '''/usr/lib/systemd/system/open-xchange.service'''. The drop-in config to override or extend the default unit file is located at '''/etc/systemd/system/open-xchange.service.d/limits.conf'''. [https://www.freedesktop.org/software/systemd/man/systemd.exec.html Systemd.exec] shows a whole lot of options that can be used by admins to adapt the default service to their specific needs.

----

[[File:diagram_sources.zip]]

AppSuite:MiddlewareStartup

2017-04-24T21:20:42Z

Marens:

== Systemd ==

==== Service definition ====
The default service definition as shipped by Open-Xchange is rather concise and easy to read. You can check by looking at the service file that is installed to either /usr/lib/systemd or /lib/systemd depending on your distribution of choice.

singlenode:~ # cat /usr/lib/systemd/system/open-xchange.service
[Unit]
After=remote-fs.target
After=time-sync.target ypbind.service sendmail.service cyrus.service

[Service]
User=open-xchange
PermissionsStartOnly=true
TimeoutStartSec=0
ExecStartPre=/opt/open-xchange/sbin/triggerupdatethemes -u
ExecStart=/opt/open-xchange/sbin/open-xchange
ExecStop=/opt/open-xchange/sbin/shutdown -w
ExecReload=/opt/open-xchange/sbin/triggerreloadconfiguration -d
KillMode=process
LimitNOFILE=65536
LimitNPROC=65536

[Install]
WantedBy=multi-user.target

==== Drop-in configs ====
Drop in configs allow administrators to easily adapt and override default service unit files. So if you want to change the default limits or add additional limits have a look at our default example drop-in config which is located at /etc/systemd/system/open-xchange.service.d/limits.conf

singlenode:~ # cat /etc/systemd/system/open-xchange.service.d/limits.conf
# Override and add options in this file
# See systemd.exec(5) for other limits

[Service]
#LimitNPROC=65536
#LimitNOFILE=65536

The systemd.exec documentation shows a whole lot of options that can be used by admins to adapt the default service to their specific needs.

 
[[File:mw_start_sysd.png|thumb|Service startup using systemd|left|1000px]]
 

== System V / Upstart ==

==== Service definition ====

The default service definition as shipped by Open-Xchange can be found at /etc/init.d/open-xchange and will differ slightly depending on the distro in use.

 
[[File:mw_start_sysv.png|thumb|Service startup using upstart|left|1000px]]
 
== Apply Resource limits ==

As you can see in the diagrams above one of the differences is that:

* systemd reads all the infos wrt. resource limitations from the original service file and then applies possible further restrictions from the drop-in configs (steps 2 and 8 in the diagram ''Service startup using systemd'') and limits configured in /opt/open-xchange/etc/ox-scriptconf.sh aren't considered anylonger
* the older System V style init has to source /opt/open-xchange/etc/ox-scriptconf.sh to be able to read the configs for the maximum number of processes (NPROC) and files(NRFILES) that the middleware shall be allowed to create/open (steps 3,4 and 15 in the diagram ''Service startup using upstart'').

=== Background Infos ===

Several ways exist to restrict resources on a linux system from a global level down to user/groups or even shells and the processes started by them.

==== Sysctl ====
Sysctl is used to modify kernel parameters at runtime. E.g. to set the maximum number of files

$ sysctl -w fs.file-max=100000

To permanently set them append to the main configuration file and reload the settings

$ echo fs.file-max=100000 >> /etc/sysctl.conf
$ sysctl -p

More infos can be found via '''man sysctl'''

==== Limits.conf ====

Allows to restrict resources an a global, group or user level. E.g:

$ cat /etc/security/limits.d/90-nproc.conf
# Default limit for number of user's processes to prevent
# accidental fork bombs.
# See rhbz #432903 for reasoning.

* soft nproc 1024

From '''man limits.conf''':
<blockquote>
''Also, please note that all limit settings are set per login. They are not global, nor are they permanent; existing only for the duration of the session.''
</blockquote>

The limits per login are applied via the ''pam'' stack. See '''man pam''' and '''man pam_limits''' for more details. As those limits are bound to sessions they don't affect most daemons started by our supported init systems or init utils. Most state that they are ignored by design, see [https://bugs.launchpad.net/ubuntu/+source/upstart/+bug/938669/comments/4 upstart], [https://bugzilla.redhat.com/show_bug.cgi?id=754285#c1 systemd] and [https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=302079 start-stop-daemon]

===== Ulimit =====
From '''man bash'''
<blockquote>
ulimit [-HSTabcdefilmnpqrstuvx [limit]]
Provides control over the resources available to the shell and to processes started by it, on systems that allow such control.
</blockquote>
This is what we use in our System V compatible init scripts to increase resources for the open-xchange process across multiple distros. Currently only ''the maximum number of processes'' and ''the maximum number of open file descriptors available to a single user'' are increased via ulimit. The values are specified in '''/opt/open-xchange/ox-scriptconf.sh'''

==== Systemd ====

===== Control Groups=====

Control groups should only affect the OX middleware if you create/manage them yourself of if you are using a ''modern'' distribution that already uses systemd as init.

Citing from the kernel cgroup [https://www.kernel.org/doc/Documentation/cgroup-v2.txt documentation]:

<blockquote>
1-2. What is cgroup?

cgroup is a mechanism to organize processes hierarchically and
distribute system resources along the hierarchy in a controlled and
configurable manner.

cgroup is largely composed of two parts - the core and controllers.
cgroup core is primarily responsible for hierarchically organizing
processes. A cgroup controller is usually responsible for
distributing a specific type of system resource along the hierarchy
although there are utility controllers which serve purposes other than
resource distribution.

cgroups form a tree structure and every process in the system belongs
to one and only one cgroup. All threads of a process belong to the
same cgroup. On creation, all processes are put in the cgroup that
the parent process belongs to at the time. A process can be migrated
to another cgroup. Migration of a process doesn't affect already
existing descendant processes.

Following certain structural constraints, controllers may be enabled or
disabled selectively on a cgroup. All controller behaviors are
hierarchical - if a controller is enabled on a cgroup, it affects all
processes which belong to the cgroups consisting the inclusive
sub-hierarchy of the cgroup. When a controller is enabled on a nested
cgroup, it always restricts the resource distribution further. The
restrictions set closer to the root in the hierarchy can not be
overridden from further away.
</blockquote>

So processes are organized into a tree structure of control groups and controllers are responsible for the distribution of resources. So what kind of controllers exist?

<blockquote>
5. Controllers

5-1. CPU

The "cpu" controllers regulates distribution of CPU cycles. This
controller implements weight and absolute bandwidth limit models for
normal scheduling policy and absolute bandwidth allocation model for
realtime scheduling policy.

5-2. Memory

The "memory" controller regulates distribution of memory.
...
While not completely water-tight, all major memory usages by a given
cgroup are tracked so that the total memory consumption can be
accounted and controlled to a reasonable extent.

5-3. IO

The "io" controller regulates the distribution of IO resources. This
controller implements both weight based and absolute bandwidth or IOPS
limit distribution; however, weight based distribution is available
only if cfq-iosched is in use and neither scheme is available for
blk-mq devices.
</blockquote>

The open-xchange service is simply put into the default ''system.slice'' without applying further limits.

singlenode$ systemd-cgls
├─1 /sbin/init
├─system.slice
│ ├─avahi-daemon.service
│ │ ├─501 avahi-daemon: running [singlenode]
│ │ └─514 avahi-daemon: chroot helper
│ ├─console-kit-daemon.service
│ │ └─16164 /usr/sbin/console-kit-daemon --no-daemon
│ ├─dbus.service
│ │ └─508 /usr/bin/dbus-daemon --system --address=systemd: --nofork --nopidfile --systemd-activation
│ ├─munin-node.service
│ │ └─4290 /usr/bin/perl -wT /usr/sbin/munin-node
│ ├─open-xchange.service
│ │ └─6037 /usr/bin/java -Dsun.net.inetaddr.ttl=3600 -Dnetworkaddress.cache.ttl=3600 -Dnetworkaddress.cache.negative.ttl=10 ...

To check all the details use

singlenode:~ # systemctl show system.slice

==== Limits besides control groups ====
Besides control groups systemd allows you to apply other limits to the execution environment of your service. Here we can apply the limits that would normally be applied via ''limits.conf'' or ''ulimit''. Systemd uses [http://man7.org/linux/man-pages/man2/setrlimit.2.html setrlimit] for this. The options that we set by default are:

* LimitNOFILE
* LimitNPROC

which you have already seen when we took a look at the [[#Service_definiton|service definition]] and [[#Drop-in_config|drop-in configs]]

== Verify limits ==

=== System V ===

Get the proper pid of the java process started for the middleware by either checking '''ps faux'''
root 4103 0.0 0.0 66528 1868 pts/0 S 22:05 0:00 su -s /bin/bash open-xchange -c /opt/open-xchange/sbin/open-xchange
494 4109 9.5 19.8 2224644 381312 ? Ssl 22:05 1:12 \_ /usr/bin/java -Dsun.net.inetaddr.ttl=3600 -Dnetworkaddress.cache.ttl=3600

or programmatically
singlenode:~ # pid=$(pgrep -P $(</var/run/open-xchange.pid) java)

and check the limits applied for that process
singlenode:~ # cat /proc/$pid/limits
Limit Soft Limit Hard Limit Units
Max cpu time unlimited unlimited seconds
Max file size unlimited unlimited bytes
Max data size unlimited unlimited bytes
Max stack size 8388608 unlimited bytes
Max core file size 0 unlimited bytes
Max resident set unlimited unlimited bytes
Max processes 65536 65536 processes
Max open files 65536 65536 files
Max locked memory 65536 65536 bytes
Max address space unlimited unlimited bytes
Max file locks unlimited unlimited locks
Max pending signals 24254 24254 signals
Max msgqueue size 819200 819200 bytes
Max nice priority 0 0
Max realtime priority 0 0
Max realtime timeout unlimited unlimited us

=== Systemd ===
singlenode:~ # systemctl show open-xchange | grep Limit
StartLimitInterval=10000000
StartLimitBurst=5
StartLimitAction=none
MemoryLimit=18446744073709551615
LimitCPU=18446744073709551615
LimitFSIZE=18446744073709551615
LimitDATA=18446744073709551615
LimitSTACK=18446744073709551615
LimitCORE=18446744073709551615
LimitRSS=18446744073709551615
LimitNOFILE=65536
LimitAS=18446744073709551615
LimitNPROC=65536
LimitMEMLOCK=65536
LimitLOCKS=18446744073709551615
LimitSIGPENDING=19827
LimitMSGQUEUE=819200
LimitNICE=0
LimitRTPRIO=0
LimitRTTIME=18446744073709551615

== Evaluate JVM commandline options ==

Before starting Java and the OSGi framework with all the Open-Xchange middleware bundles we have to evaluate several important JVM commandline options that influence the performance and stability of the Open-Xchange middleware stack. These commandline options are specified in '''/opt/open-xchange/etc/ox-scriptconf.sh'''

=== Format < 7.8.4 ===
For version up to 7.8.3 the JVM configuration options were all specified via a single property '''JAVA_XTRAOPTS''' which resulted in a rather long configuration string that was hard to read and maintain.

JAVA_XTRAOPTS="-Dsun.net.inetaddr.ttl=3600 -Dnetworkaddress.cache.ttl=3600 -Dnetworkaddress.cache.negative.ttl=10 \
-Dlogback.threadlocal.put.duplicate=false -server -Djava.awt.headless=true -Xmx512M -XX:MaxPermSize=256m -XX:+UseConcMarkSweepGC \
-XX:+UseParNewGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:NewRatio=3 -XX:+UseTLAB \
-Dosgi.compatibility.bootdelegation=false -XX:-OmitStackTraceInFastThrow"

During the startup of the open-xchange service (step 11 of diagram Service startup using systemd and step 10 of diagram Service startup using upstart) this JAVA_XTRAOPTS string goes through a basic validity check and is then passed to the JVM.

=== Format >= 7.8.4 ===
Beginning from version 7.8.4 the options are split up and sorted into different categories like:

JAVA_OPTS_GC="-XX:+UseConcMarkSweepGC -XX:+UseParNewGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:NewRatio=3 -XX:+DisableExplicitGC"
JAVA_OPTS_LOG="-Dlogback.threadlocal.put.duplicate=false -XX:-OmitStackTraceInFastThrow"
JAVA_OPTS_MEM="-XX:MaxHeapSize=512M -XX:MaxPermSize=256M -XX:+UseTLAB"
JAVA_OPTS_NET="-Dsun.net.inetaddr.ttl=3600 -Dnetworkaddress.cache.ttl=3600 -Dnetworkaddress.cache.negative.ttl=10"
JAVA_OPTS_OSGI="-Dosgi.compatibility.bootdelegation=false"
JAVA_OPTS_SERVER="-server -Djava.awt.headless=true"

JAVA_OPTS_OTHER=""

#JAVA_OPTS_DEBUG="-XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/mnt/heapdump -Xloggc:/var/log/open-xchange/gc.log -verbose:gc -XX:+PrintGCDateStamps \
-XX:+PrintHeapAtGC -XX:+PrintGCApplicationStoppedTime -XX:+PrintTenuringDistribution"

which is much easier to read an maintain. Furthermore we added a useful '''JAVA_OPTS_DEBUG''' configuration parameter that comes preconfigured but unused/commented so that customers can quickly enable the most commonly used debugging options in case they have to provide detailed system information to Open-Xchange.

During the startup of the open-xchange service (step 11 of diagram ''Service startup using systemd'' and step 10 of diagram ''Service startup using upstart'') all uncommented JAVA_OPTS_* categories will then be evaluated and combined again into a single string of commandline options that can then be passed to the JVM.

=== Upgrade procedure ===
During the upgrade '''7.8.3-revX -> 7.8.4-revY''' existing JAVA_XTRAOPTS will automatically be split up and migrated to the new categories approach. A backup of the file (ox-scriptconf.sh.timestamp) will be created and admins are urged to check for proper migration before deleting the backup file like it's already done with *.dpkg-dist or *.rpmnew files on Debian or RHEL/SLES distributions.

== Encoding detection during JVM startup ==

=== Background Infos ===

From ''man 7 locale''

DESCRIPTION
A locale is a set of language and cultural rules. These cover aspects such as lan‐
guage for messages, different character sets, lexicographic conventions, and so on.
A program needs to be able to determine its locale and act accordingly to be portable
to different cultures.

The format to specify a system locale is: '''''language[_territory][.codeset][@modifier]'''''.

Where all of

* en_US.UTF-8
* en_US.utf8
* en_US.utf-8
* en_US.UTF_8

will result in a valid UTF-8 charmap for the english language with locale categories configured as

LANG=en_US.UTF-8
LC_CTYPE="en_US.UTF-8"
LC_NUMERIC="en_US.UTF-8"
LC_TIME="en_US.UTF-8"
LC_COLLATE="en_US.UTF-8"
LC_MONETARY="en_US.UTF-8"
LC_MESSAGES="en_US.UTF-8"
LC_PAPER="en_US.UTF-8"
LC_NAME="en_US.UTF-8"
LC_ADDRESS="en_US.UTF-8"
LC_TELEPHONE="en_US.UTF-8"
LC_MEASUREMENT="en_US.UTF-8"
LC_IDENTIFICATION="en_US.UTF-8"
LC_ALL=

See for more infos: locale(1), locale(7), [https://en.wikipedia.org/wiki/List_of_ISO_639-1_codes language codes], [https://en.wikipedia.org/wiki/ISO_3166-1#Current_codes country codes] and [https://en.wikipedia.org/wiki/Character_encoding character encodings]

==== JVM startup ====

The default file.encoding detection is done at JVM startup based upon LC_CTYPE which makes sense as it's documented as:

CODESET (LC_CTYPE)
Return a string with the name of the character encoding used in the selected locale, such as "UTF-8", "ISO-8859-1",
or "ANSI_X3.4-1968" (better known as US-ASCII). This is the same string that you get with "locale charmap".
For a list of character encoding names, try "locale -m", cf. locale(1).

Furthermore this influences the defaultCharset selection of the JVM.

/**
* Returns the default charset of this Java virtual machine.
*
* The default charset is determined during virtual-machine startup and
* typically depends upon the locale and charset of the underlying
* operating system.
*
* @return A charset object for the default charset
*
* @since 1.5
*/
public static Charset defaultCharset() {
if (defaultCharset == null) {
synchronized (Charset.class) {
String csn = AccessController.doPrivileged(
new GetPropertyAction("file.encoding"));
Charset cs = lookup(csn);
if (cs != null)
defaultCharset = cs;
else
defaultCharset = forName("UTF-8");
}
}
return defaultCharset;
}

Although we try to specify valid unicode encodings for every IO operations the Open-Xchange middleware does some parts of the JVM don't allow to specify the encoding to use and falls back to the detected default encoding. That's why admins have to make sure to provide a proper unicode environment for the middleware service during every start or use documented workarounds provided by Open-Xchange (e.g. )

=== Query which file encoding was detected by the JVM during startup ===
[[File:visualvm-file.encoding.png|thumb|Querying the JVM via visualvm|right|300px]]

The file encoding that was detected by the JVM during startup can be queried via JMX. You can use graphical management tools like [https://visualvm.github.io/ visualvm], query our [[Jolokia]] JMX bridge via http or just use the commandline like

/opt/open-xchange/sbin/showruntimestats -r | sed -ne 's/.*${key=file.encoding, value=[^}]*}$.*/\1/p'
{key=file.encoding, value=UTF-8}

 

=== Locale passing from client to server via SSH ===

The following setting of sshd allows clients to specify which locale to use on the server and might lead to some confusion or unexpected behaviour when checking the system locale or even to the usage of broken locales during middleware startup if your client sends misconfigured or missing locale configuration to the server.

grep AcceptEnv /etc/ssh/sshd_config
AcceptEnv LANG LC_*

For example when i connect to the machine my local german locale is sent to the server

[vagrant@showcase-community-centos-7 ~]$ locale -v
LANG=de_DE.UTF-8
LC_CTYPE="de_DE.UTF-8"
LC_NUMERIC=de_DE.UTF-8
LC_TIME=de_DE.UTF-8
LC_COLLATE="de_DE.UTF-8"
LC_MONETARY=de_DE.UTF-8
LC_MESSAGES="de_DE.UTF-8"
LC_PAPER=de_DE.UTF-8
LC_NAME=de_DE.UTF-8
LC_ADDRESS=de_DE.UTF-8
LC_TELEPHONE=de_DE.UTF-8
LC_MEASUREMENT=de_DE.UTF-8
LC_IDENTIFICATION=de_DE.UTF-8
LC_ALL=

But after specifying that i want a new login shell the default locale configured for that user/system will be used instead!

[root@showcase-community-centos-7 vagrant]# su -s /bin/bash open-xchange -l
-bash-4.2$ locale -v
LANG=en_US.UTF-8
LC_CTYPE="en_US.UTF-8"
LC_NUMERIC="en_US.UTF-8"
LC_TIME="en_US.UTF-8"
LC_COLLATE="en_US.UTF-8"
LC_MONETARY="en_US.UTF-8"
LC_MESSAGES="en_US.UTF-8"
LC_PAPER="en_US.UTF-8"
LC_NAME="en_US.UTF-8"
LC_ADDRESS="en_US.UTF-8"
LC_TELEPHONE="en_US.UTF-8"
LC_MEASUREMENT="en_US.UTF-8"
LC_IDENTIFICATION="en_US.UTF-8"
LC_ALL=

=== Verify that your system is configured with a proper unicode locale for OX ===

Assuming you haven't reconfigured the locale of your root user

i|marens@karotte ~ $ su -
Password:
i|karotte ~ # locale charmap
UTF-8
i|karotte ~ #

== Warnings during early start ==

Early warnings before the start of the JVM and related logging libraries like logback and logstash forwarding are redirected to '''/var/log/open-xchange/open-xchange-console.log'''.

=== Limits ===
If the open-xchange startup script isn't able to set the proper limits on CentOS 6 due to e.g. hard limits being enforced via pam_limits you'll see the following warning in the console log during startup

singlenode open-xchange # cat open-xchange-console.log
/opt/open-xchange/sbin/open-xchange: line 115: ulimit: max user processes: cannot modify limit: Operation not permitted

=== Locale ===
If the open-xchange startup script isn't able to detect a unicode locale you'll see a warning like the following (the current charmap might differ)

WARNING: Couldn't detect proper unicode charmap in locale category LC_CTYPE.
WARNING: Current charmap: "ANSI_X3.4-1968" might cause encoding problems.

== Summary: Open-Xchange middleware on specific distros ==
The support for the mentioned mechanism of resource control differ depending on the distribution and the init system in use.

=== Debian 7 ===
;Init
: System V style
;OX Configurable Limits/Defaults
: nofile, nproc

The mentioned limits can be configured via '''/opt/open-xchange/etc/ox-scriptconf.sh'''. The limits are applied via ulimit in the service's init script. The open-xchange service is finally started via start-stop-daemon which doesn't doesn't consider '''/etc/security/limits.*'''

=== RHEL 6 / CentOS 6 ===
;Init
: Upstart, System V compatible
;OX Configurable Limits/Defaults
: nofile, nproc

The mentioned limits can be configured via '''/opt/open-xchange/etc/ox-scriptconf.sh'''. The limits are applied via ulimit in the service's init script. Furthermore as the open-xchange service is finally started via '''su ... open-xchange''' on this distro a user session is opened via su/pam and the default CentOS pam config reads the '''/etc/security/limits.*''' configuration by loading the pam stack like:

: '''/etc/pam.d/su'''
:: '''-> /etc/pam.d/system-auth'''
::: '''-> pam_limits.so'''

If NPROC isn't configured for the open-xchange-server it's restricted to '''1024''' globally by default to prevent accidental fork bombs, see '''/etc/security/limits.d/90-nproc.conf''' which can result in severe problems modern multithreaded applications.

=== RHEL 7 / CentOS 7 / Debian 8 / SLE 12 ===
;Init
: Systemd
;OX Configurable Limits/Defaults
: nofile, nproc

For systemd the default limits are configured directly in the service's unit file that is shipped by OX and located at '''/usr/lib/systemd/system/open-xchange.service'''. The drop-in config to override or extend the default unit file is located at '''/etc/systemd/system/open-xchange.service.d/limits.conf'''. [https://www.freedesktop.org/software/systemd/man/systemd.exec.html Systemd.exec] shows a whole lot of options that can be used by admins to adapt the default service to their specific needs.

----

[[File:diagram_sources.zip]]

File:Diagram sources.zip

2017-04-24T21:20:10Z

Marens:

AppSuite:Main Page Advanced

2017-04-24T21:06:02Z

Marens: /* Overview */

__NOTOC__
= Overview =
*[[AppSuite:Architecture_Overview|Architecture Overview]]
*[[Appsuite:MiddlewareStartup|Middleware Startup]]

= OX App Suite UI Development =
This page contains all the information you need to get started with UI development.

This covers how-to articles including server communication, extension points and how to write widgets, applications and plugins:

[[AppSuite:UI | Frontend development articles]]

= Groupware Server customization =
* [[Automatic_Delete_OXObjects_Per_User|Automatically delete users OX-Objects and OX-Folders]]
* [[Login_variations|Background Information about Login Variation]]
* Auto login, session handling, single sign on
** [http://www.open-xchange.com/fileadmin/user_upload/open-xchange/document/presentation/OX-HE-Authentication-Sessionhandling-en-6.18.pdf Authentication and Sessionhandling white paper (EasyLogin)]
** [https://documentation.open-xchange.com/7.8.2/middleware/components/saml.html SAML SSO Integration]
** Custom login masks
*** [[Open-Xchange servlet for external login masks]]

= Advanced Configuration =
* [[AppSuite:CalDAVClients | Configuration CalDAV with Open-Xchange]]
* [[AppSuite:CardDAVClients | Configuration CardDAV with Open-Xchange]]
* [[AppSuite:Caldav_carddav_Bundles| Installation CardDAV/CalDAV with Open-Xchange]]
* [[AppSuite:Running_a_cluster| Configuration Cluster-Setup]]
* [[AppSuite:Grizzly| Configuration Connector based on Grizzly]]
* [[ContactStorageLDAP| LDAP Contact Storage]]
* [[AppSuite:Sharing_and_Guest_Mode |Sharing and Guest Mode]]
* [https://documentation.open-xchange.com/latest/ui/features/metrics.html Metrics]
* [https://documentation.open-xchange.com/latest/ui/features/upsell.html Upsell]
* [https://documentation.open-xchange.com/latest/middleware/components/oauth_provider.html OAuth 2.0 - Provider/Operator Guide and Client Developer Guide]
* [[AppSuite:File_Storages_per_User |File Storage per User ]]
* [[AppSuite:Paste_inline_images |Pasting External Images into Mail Compose ]]
* [[AppSuite:Create_custom_folderview_entries_in_settings_app |Custom Folder View Entries in Settings ]]
* [[AppSuite:DB_user_privileges | Database User Privileges]]
* [[AppSuite:CrossContextDatabase |GlobalDB/Cross-Context Database ]]
* [[AppSuite:Filestorages |Setup external File Stores ]]
* [[AppSuite:Client_Onboarding|Configuration guide for Open-Xchange Client Onboarding]]
* [https://documentation.open-xchange.com/latest/middleware/components/auditlogging.html Audit logging]
* [https://documentation.open-xchange.com/latest/middleware/components/drivemail.html Drive Mail]
* [https://documentation.open-xchange.com/latest/middleware/components/mail_categories.html Mail Categories]
* [https://documentation.open-xchange.com/latest/middleware/components/virtual_mail_attachments.html Virtual Mail Attachments Connector]
* [https://documentation.open-xchange.com/7.8.3/middleware/components/trusted_connections.html Configure trusted TLS certificates]
* [https://documentation.open-xchange.com/7.8.3/middleware/components/websockets.html Configure Web Sockets]
* [[AppSuite:ResourceLimits | Configure Middleware Resource Limits]]

= Programming Interfaces =
* The [https://documentation.open-xchange.com/latest/middleware/http_api.html HTTP API] is used by the Open-Xchange GUI and various 3rd party applications. It consists mainly of messages in JavaScript Object Notation ([http://json.org JSON]) sent over HTTP. Here is a general [https://documentation.open-xchange.com/latest/middleware/http_api/1_introduction.html Introduction to the HTTP API].
* Provisioning API to access the Open-Xchange Admin Daemon
** The [http://java.sun.com/javase/technologies/core/basic/rmi/index.jsp RMI] API is used for data provisioning of Contexts, Users, Groups and Resources as well as for configuring Databases, Filestores and OX Servers: [http://software.open-xchange.com/products/appsuite/doc/RMI/javadoc/ OX RMI API]
** The [http://oxpedia.org/wiki/index.php?title=AppSuite:AdminGuide_7.8.2#OX_App_Suite_Management_.28CLT.29 CLT ] are shell scripts that simplify groupware and service administration
** Create contexts/users with with [[Csv_import]]
** [[Open-Xchange-SOAP|Provisioning using SOAP]]
* The [[Oxmapi|Oxmapi]] is a windows library for programmers needed to communicate with the OX server
* {{DocLink|docpath=mal/|name=Open-Xchange Mail Abstraction Layer}}
* [[UDPPush]] Open-Xchange PUSH Interface for Groupware Objects

= Testing and QA =
* [[AppSuite:Web Tests|Web Tests]]
* [[AppSuite:Load Tests|Load Tests]]
* [[AppSuite:CLT Tests|CLT Tests]]
* [[AppSuite:SOAP Tests|SOAP Tests]]

= Statistics =

* [[AppSuite:Logincounter|logincounter]]

= Translations =
* [[AppSuite:Available_Translations| Available Language Translations]]
* [[GUI Translation|Translate Open-Xchange to your community language]]
* [[AppSuite:Translate_Open-Xchange_to_supported_language|Translate Open-Xchange to supported language]]

= Installation based on source code =
* [[SourceCodeAccess|Download the sourcecode]]

= Integration =
* {{DocLink|docpath=whitepaper/OX-Directory-Integration-Whitepaper-English.pdf|name=Directory Integration Whitepaper}}

= Additional =
* [[AppSuite:ISV_Mockups_Wireframes|Create Open-Xchange AppSuite Mockups / Wireframes]]
* [https://documentation.open-xchange.com/7.8.2/middleware/components/search/crossfolder_fts_in_mail.html Cross folder fulltext search with Dovecot]

AppSuite:ResourceLimits

2017-04-24T20:58:44Z

Marens: Add redirect to middleware startup

#REDIRECT [[:Appsuite:MiddlewareStartup]]

AppSuite:MiddlewareStartup

2017-04-24T20:57:25Z

Marens: Created page with " == Systemd == ==== Service definition ==== The default service definition as shipped by Open-Xchange is rather concise and easy to read. You can check by looking at the serv..."

== Systemd ==

==== Service definition ====
The default service definition as shipped by Open-Xchange is rather concise and easy to read. You can check by looking at the service file that is installed to either /usr/lib/systemd or /lib/systemd depending on your distribution of choice.

singlenode:~ # cat /usr/lib/systemd/system/open-xchange.service
[Unit]
After=remote-fs.target
After=time-sync.target ypbind.service sendmail.service cyrus.service

[Service]
User=open-xchange
PermissionsStartOnly=true
TimeoutStartSec=0
ExecStartPre=/opt/open-xchange/sbin/triggerupdatethemes -u
ExecStart=/opt/open-xchange/sbin/open-xchange
ExecStop=/opt/open-xchange/sbin/shutdown -w
ExecReload=/opt/open-xchange/sbin/triggerreloadconfiguration -d
KillMode=process
LimitNOFILE=65536
LimitNPROC=65536

[Install]
WantedBy=multi-user.target

==== Drop-in configs ====
Drop in configs allow administrators to easily adapt and override default service unit files. So if you want to change the default limits or add additional limits have a look at our default example drop-in config which is located at /etc/systemd/system/open-xchange.service.d/limits.conf

singlenode:~ # cat /etc/systemd/system/open-xchange.service.d/limits.conf
# Override and add options in this file
# See systemd.exec(5) for other limits

[Service]
#LimitNPROC=65536
#LimitNOFILE=65536

The systemd.exec documentation shows a whole lot of options that can be used by admins to adapt the default service to their specific needs.

 
[[File:mw_start_sysd.png|thumb|Service startup using systemd|left|1000px]]
 

== System V / Upstart ==

==== Service definition ====

The default service definition as shipped by Open-Xchange can be found at /etc/init.d/open-xchange and will differ slightly depending on the distro in use.

 
[[File:mw_start_sysv.png|thumb|Service startup using upstart|left|1000px]]
 
== Apply Resource limits ==

As you can see in the diagrams above one of the differences is that:

* systemd reads all the infos wrt. resource limitations from the original service file and then applies possible further restrictions from the drop-in configs (steps 2 and 8 in the diagram ''Service startup using systemd'') and limits configured in /opt/open-xchange/etc/ox-scriptconf.sh aren't considered anylonger
* the older System V style init has to source /opt/open-xchange/etc/ox-scriptconf.sh to be able to read the configs for the maximum number of processes (NPROC) and files(NRFILES) that the middleware shall be allowed to create/open (steps 3,4 and 15 in the diagram ''Service startup using upstart'').

=== Background Infos ===

Several ways exist to restrict resources on a linux system from a global level down to user/groups or even shells and the processes started by them.

==== Sysctl ====
Sysctl is used to modify kernel parameters at runtime. E.g. to set the maximum number of files

$ sysctl -w fs.file-max=100000

To permanently set them append to the main configuration file and reload the settings

$ echo fs.file-max=100000 >> /etc/sysctl.conf
$ sysctl -p

More infos can be found via '''man sysctl'''

==== Limits.conf ====

Allows to restrict resources an a global, group or user level. E.g:

$ cat /etc/security/limits.d/90-nproc.conf
# Default limit for number of user's processes to prevent
# accidental fork bombs.
# See rhbz #432903 for reasoning.

* soft nproc 1024

From '''man limits.conf''':
<blockquote>
''Also, please note that all limit settings are set per login. They are not global, nor are they permanent; existing only for the duration of the session.''
</blockquote>

The limits per login are applied via the ''pam'' stack. See '''man pam''' and '''man pam_limits''' for more details. As those limits are bound to sessions they don't affect most daemons started by our supported init systems or init utils. Most state that they are ignored by design, see [https://bugs.launchpad.net/ubuntu/+source/upstart/+bug/938669/comments/4 upstart], [https://bugzilla.redhat.com/show_bug.cgi?id=754285#c1 systemd] and [https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=302079 start-stop-daemon]

===== Ulimit =====
From '''man bash'''
<blockquote>
ulimit [-HSTabcdefilmnpqrstuvx [limit]]
Provides control over the resources available to the shell and to processes started by it, on systems that allow such control.
</blockquote>
This is what we use in our System V compatible init scripts to increase resources for the open-xchange process across multiple distros. Currently only ''the maximum number of processes'' and ''the maximum number of open file descriptors available to a single user'' are increased via ulimit. The values are specified in '''/opt/open-xchange/ox-scriptconf.sh'''

==== Systemd ====

===== Control Groups=====

Control groups should only affect the OX middleware if you create/manage them yourself of if you are using a ''modern'' distribution that already uses systemd as init.

Citing from the kernel cgroup [https://www.kernel.org/doc/Documentation/cgroup-v2.txt documentation]:

<blockquote>
1-2. What is cgroup?

cgroup is a mechanism to organize processes hierarchically and
distribute system resources along the hierarchy in a controlled and
configurable manner.

cgroup is largely composed of two parts - the core and controllers.
cgroup core is primarily responsible for hierarchically organizing
processes. A cgroup controller is usually responsible for
distributing a specific type of system resource along the hierarchy
although there are utility controllers which serve purposes other than
resource distribution.

cgroups form a tree structure and every process in the system belongs
to one and only one cgroup. All threads of a process belong to the
same cgroup. On creation, all processes are put in the cgroup that
the parent process belongs to at the time. A process can be migrated
to another cgroup. Migration of a process doesn't affect already
existing descendant processes.

Following certain structural constraints, controllers may be enabled or
disabled selectively on a cgroup. All controller behaviors are
hierarchical - if a controller is enabled on a cgroup, it affects all
processes which belong to the cgroups consisting the inclusive
sub-hierarchy of the cgroup. When a controller is enabled on a nested
cgroup, it always restricts the resource distribution further. The
restrictions set closer to the root in the hierarchy can not be
overridden from further away.
</blockquote>

So processes are organized into a tree structure of control groups and controllers are responsible for the distribution of resources. So what kind of controllers exist?

<blockquote>
5. Controllers

5-1. CPU

The "cpu" controllers regulates distribution of CPU cycles. This
controller implements weight and absolute bandwidth limit models for
normal scheduling policy and absolute bandwidth allocation model for
realtime scheduling policy.

5-2. Memory

The "memory" controller regulates distribution of memory.
...
While not completely water-tight, all major memory usages by a given
cgroup are tracked so that the total memory consumption can be
accounted and controlled to a reasonable extent.

5-3. IO

The "io" controller regulates the distribution of IO resources. This
controller implements both weight based and absolute bandwidth or IOPS
limit distribution; however, weight based distribution is available
only if cfq-iosched is in use and neither scheme is available for
blk-mq devices.
</blockquote>

The open-xchange service is simply put into the default ''system.slice'' without applying further limits.

singlenode$ systemd-cgls
├─1 /sbin/init
├─system.slice
│ ├─avahi-daemon.service
│ │ ├─501 avahi-daemon: running [singlenode]
│ │ └─514 avahi-daemon: chroot helper
│ ├─console-kit-daemon.service
│ │ └─16164 /usr/sbin/console-kit-daemon --no-daemon
│ ├─dbus.service
│ │ └─508 /usr/bin/dbus-daemon --system --address=systemd: --nofork --nopidfile --systemd-activation
│ ├─munin-node.service
│ │ └─4290 /usr/bin/perl -wT /usr/sbin/munin-node
│ ├─open-xchange.service
│ │ └─6037 /usr/bin/java -Dsun.net.inetaddr.ttl=3600 -Dnetworkaddress.cache.ttl=3600 -Dnetworkaddress.cache.negative.ttl=10 ...

To check all the details use

singlenode:~ # systemctl show system.slice

==== Limits besides control groups ====
Besides control groups systemd allows you to apply other limits to the execution environment of your service. Here we can apply the limits that would normally be applied via ''limits.conf'' or ''ulimit''. Systemd uses [http://man7.org/linux/man-pages/man2/setrlimit.2.html setrlimit] for this. The options that we set by default are:

* LimitNOFILE
* LimitNPROC

which you have already seen when we took a look at the [[#Service_definiton|service definition]] and [[#Drop-in_config|drop-in configs]]

== Verify limits ==

=== System V ===

Get the proper pid of the java process started for the middleware by either checking '''ps faux'''
root 4103 0.0 0.0 66528 1868 pts/0 S 22:05 0:00 su -s /bin/bash open-xchange -c /opt/open-xchange/sbin/open-xchange
494 4109 9.5 19.8 2224644 381312 ? Ssl 22:05 1:12 \_ /usr/bin/java -Dsun.net.inetaddr.ttl=3600 -Dnetworkaddress.cache.ttl=3600

or programmatically
singlenode:~ # pid=$(pgrep -P $(</var/run/open-xchange.pid) java)

and check the limits applied for that process
singlenode:~ # cat /proc/$pid/limits
Limit Soft Limit Hard Limit Units
Max cpu time unlimited unlimited seconds
Max file size unlimited unlimited bytes
Max data size unlimited unlimited bytes
Max stack size 8388608 unlimited bytes
Max core file size 0 unlimited bytes
Max resident set unlimited unlimited bytes
Max processes 65536 65536 processes
Max open files 65536 65536 files
Max locked memory 65536 65536 bytes
Max address space unlimited unlimited bytes
Max file locks unlimited unlimited locks
Max pending signals 24254 24254 signals
Max msgqueue size 819200 819200 bytes
Max nice priority 0 0
Max realtime priority 0 0
Max realtime timeout unlimited unlimited us

=== Systemd ===
singlenode:~ # systemctl show open-xchange | grep Limit
StartLimitInterval=10000000
StartLimitBurst=5
StartLimitAction=none
MemoryLimit=18446744073709551615
LimitCPU=18446744073709551615
LimitFSIZE=18446744073709551615
LimitDATA=18446744073709551615
LimitSTACK=18446744073709551615
LimitCORE=18446744073709551615
LimitRSS=18446744073709551615
LimitNOFILE=65536
LimitAS=18446744073709551615
LimitNPROC=65536
LimitMEMLOCK=65536
LimitLOCKS=18446744073709551615
LimitSIGPENDING=19827
LimitMSGQUEUE=819200
LimitNICE=0
LimitRTPRIO=0
LimitRTTIME=18446744073709551615

== Evaluate JVM commandline options ==

Before starting Java and the OSGi framework with all the Open-Xchange middleware bundles we have to evaluate several important JVM commandline options that influence the performance and stability of the Open-Xchange middleware stack. These commandline options are specified in '''/opt/open-xchange/etc/ox-scriptconf.sh'''

=== Format < 7.8.4 ===
For version up to 7.8.3 the JVM configuration options were all specified via a single property '''JAVA_XTRAOPTS''' which resulted in a rather long configuration string that was hard to read and maintain.

JAVA_XTRAOPTS="-Dsun.net.inetaddr.ttl=3600 -Dnetworkaddress.cache.ttl=3600 -Dnetworkaddress.cache.negative.ttl=10 \
-Dlogback.threadlocal.put.duplicate=false -server -Djava.awt.headless=true -Xmx512M -XX:MaxPermSize=256m -XX:+UseConcMarkSweepGC \
-XX:+UseParNewGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:NewRatio=3 -XX:+UseTLAB \
-Dosgi.compatibility.bootdelegation=false -XX:-OmitStackTraceInFastThrow"

During the startup of the open-xchange service (step 11 of diagram Service startup using systemd and step 10 of diagram Service startup using upstart) this JAVA_XTRAOPTS string goes through a basic validity check and is then passed to the JVM.

=== Format >= 7.8.4 ===
Beginning from version 7.8.4 the options are split up and sorted into different categories like:

JAVA_OPTS_GC="-XX:+UseConcMarkSweepGC -XX:+UseParNewGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:NewRatio=3 -XX:+DisableExplicitGC"
JAVA_OPTS_LOG="-Dlogback.threadlocal.put.duplicate=false -XX:-OmitStackTraceInFastThrow"
JAVA_OPTS_MEM="-XX:MaxHeapSize=512M -XX:MaxPermSize=256M -XX:+UseTLAB"
JAVA_OPTS_NET="-Dsun.net.inetaddr.ttl=3600 -Dnetworkaddress.cache.ttl=3600 -Dnetworkaddress.cache.negative.ttl=10"
JAVA_OPTS_OSGI="-Dosgi.compatibility.bootdelegation=false"
JAVA_OPTS_SERVER="-server -Djava.awt.headless=true"

JAVA_OPTS_OTHER=""

#JAVA_OPTS_DEBUG="-XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/mnt/heapdump -Xloggc:/var/log/open-xchange/gc.log -verbose:gc -XX:+PrintGCDateStamps \
-XX:+PrintHeapAtGC -XX:+PrintGCApplicationStoppedTime -XX:+PrintTenuringDistribution"

which is much easier to read an maintain. Furthermore we added a useful '''JAVA_OPTS_DEBUG''' configuration parameter that comes preconfigured but unused/commented so that customers can quickly enable the most commonly used debugging options in case they have to provide detailed system information to Open-Xchange.

During the startup of the open-xchange service (step 11 of diagram ''Service startup using systemd'' and step 10 of diagram ''Service startup using upstart'') all uncommented JAVA_OPTS_* categories will then be evaluated and combined again into a single string of commandline options that can then be passed to the JVM.

=== Upgrade procedure ===
During the upgrade '''7.8.3-revX -> 7.8.4-revY''' existing JAVA_XTRAOPTS will automatically be split up and migrated to the new categories approach. A backup of the file (ox-scriptconf.sh.timestamp) will be created and admins are urged to check for proper migration before deleting the backup file like it's already done with *.dpkg-dist or *.rpmnew files on Debian or RHEL/SLES distributions.

== Encoding detection during JVM startup ==

=== Background Infos ===

From ''man 7 locale''

DESCRIPTION
A locale is a set of language and cultural rules. These cover aspects such as lan‐
guage for messages, different character sets, lexicographic conventions, and so on.
A program needs to be able to determine its locale and act accordingly to be portable
to different cultures.

The format to specify a system locale is: '''''language[_territory][.codeset][@modifier]'''''.

Where all of

* en_US.UTF-8
* en_US.utf8
* en_US.utf-8
* en_US.UTF_8

will result in a valid UTF-8 charmap for the english language with locale categories configured as

LANG=en_US.UTF-8
LC_CTYPE="en_US.UTF-8"
LC_NUMERIC="en_US.UTF-8"
LC_TIME="en_US.UTF-8"
LC_COLLATE="en_US.UTF-8"
LC_MONETARY="en_US.UTF-8"
LC_MESSAGES="en_US.UTF-8"
LC_PAPER="en_US.UTF-8"
LC_NAME="en_US.UTF-8"
LC_ADDRESS="en_US.UTF-8"
LC_TELEPHONE="en_US.UTF-8"
LC_MEASUREMENT="en_US.UTF-8"
LC_IDENTIFICATION="en_US.UTF-8"
LC_ALL=

See for more infos: locale(1), locale(7), [https://en.wikipedia.org/wiki/List_of_ISO_639-1_codes language codes], [https://en.wikipedia.org/wiki/ISO_3166-1#Current_codes country codes] and [https://en.wikipedia.org/wiki/Character_encoding character encodings]

==== JVM startup ====

The default file.encoding detection is done at JVM startup based upon LC_CTYPE which makes sense as it's documented as:

CODESET (LC_CTYPE)
Return a string with the name of the character encoding used in the selected locale, such as "UTF-8", "ISO-8859-1",
or "ANSI_X3.4-1968" (better known as US-ASCII). This is the same string that you get with "locale charmap".
For a list of character encoding names, try "locale -m", cf. locale(1).

Furthermore this influences the defaultCharset selection of the JVM.

/**
* Returns the default charset of this Java virtual machine.
*
* The default charset is determined during virtual-machine startup and
* typically depends upon the locale and charset of the underlying
* operating system.
*
* @return A charset object for the default charset
*
* @since 1.5
*/
public static Charset defaultCharset() {
if (defaultCharset == null) {
synchronized (Charset.class) {
String csn = AccessController.doPrivileged(
new GetPropertyAction("file.encoding"));
Charset cs = lookup(csn);
if (cs != null)
defaultCharset = cs;
else
defaultCharset = forName("UTF-8");
}
}
return defaultCharset;
}

Although we try to specify valid unicode encodings for every IO operations the Open-Xchange middleware does some parts of the JVM don't allow to specify the encoding to use and falls back to the detected default encoding. That's why admins have to make sure to provide a proper unicode environment for the middleware service during every start or use documented workarounds provided by Open-Xchange (e.g. )

=== Query which file encoding was detected by the JVM during startup ===
[[File:visualvm-file.encoding.png|thumb|Querying the JVM via visualvm|right|300px]]

The file encoding that was detected by the JVM during startup can be queried via JMX. You can use graphical management tools like [https://visualvm.github.io/ visualvm], query our [[Jolokia]] JMX bridge via http or just use the commandline like

/opt/open-xchange/sbin/showruntimestats -r | sed -ne 's/.*${key=file.encoding, value=[^}]*}$.*/\1/p'
{key=file.encoding, value=UTF-8}

 

=== Locale passing from client to server via SSH ===

The following setting of sshd allows clients to specify which locale to use on the server and might lead to some confusion or unexpected behaviour when checking the system locale or even to the usage of broken locales during middleware startup if your client sends misconfigured or missing locale configuration to the server.

grep AcceptEnv /etc/ssh/sshd_config
AcceptEnv LANG LC_*

For example when i connect to the machine my local german locale is sent to the server

[vagrant@showcase-community-centos-7 ~]$ locale -v
LANG=de_DE.UTF-8
LC_CTYPE="de_DE.UTF-8"
LC_NUMERIC=de_DE.UTF-8
LC_TIME=de_DE.UTF-8
LC_COLLATE="de_DE.UTF-8"
LC_MONETARY=de_DE.UTF-8
LC_MESSAGES="de_DE.UTF-8"
LC_PAPER=de_DE.UTF-8
LC_NAME=de_DE.UTF-8
LC_ADDRESS=de_DE.UTF-8
LC_TELEPHONE=de_DE.UTF-8
LC_MEASUREMENT=de_DE.UTF-8
LC_IDENTIFICATION=de_DE.UTF-8
LC_ALL=

But after specifying that i want a new login shell the default locale configured for that user/system will be used instead!

[root@showcase-community-centos-7 vagrant]# su -s /bin/bash open-xchange -l
-bash-4.2$ locale -v
LANG=en_US.UTF-8
LC_CTYPE="en_US.UTF-8"
LC_NUMERIC="en_US.UTF-8"
LC_TIME="en_US.UTF-8"
LC_COLLATE="en_US.UTF-8"
LC_MONETARY="en_US.UTF-8"
LC_MESSAGES="en_US.UTF-8"
LC_PAPER="en_US.UTF-8"
LC_NAME="en_US.UTF-8"
LC_ADDRESS="en_US.UTF-8"
LC_TELEPHONE="en_US.UTF-8"
LC_MEASUREMENT="en_US.UTF-8"
LC_IDENTIFICATION="en_US.UTF-8"
LC_ALL=

=== Verify that your system is configured with a proper unicode locale for OX ===

Assuming you haven't reconfigured the locale of your root user

i|marens@karotte ~ $ su -
Password:
i|karotte ~ # locale charmap
UTF-8
i|karotte ~ #

== Warnings during early start ==

Early warnings before the start of the JVM and related logging libraries like logback and logstash forwarding are redirected to '''/var/log/open-xchange/open-xchange-console.log'''.

=== Limits ===
If the open-xchange startup script isn't able to set the proper limits on CentOS 6 due to e.g. hard limits being enforced via pam_limits you'll see the following warning in the console log during startup

singlenode open-xchange # cat open-xchange-console.log
/opt/open-xchange/sbin/open-xchange: line 115: ulimit: max user processes: cannot modify limit: Operation not permitted

=== Locale ===
If the open-xchange startup script isn't able to detect a unicode locale you'll see a warning like the following (the current charmap might differ)

WARNING: Couldn't detect proper unicode charmap in locale category LC_CTYPE.
WARNING: Current charmap: "ANSI_X3.4-1968" might cause encoding problems.

== Summary: Open-Xchange middleware on specific distros ==
The support for the mentioned mechanism of resource control differ depending on the distribution and the init system in use.

=== Debian 7 ===
;Init
: System V style
;OX Configurable Limits/Defaults
: nofile, nproc

The mentioned limits can be configured via '''/opt/open-xchange/etc/ox-scriptconf.sh'''. The limits are applied via ulimit in the service's init script. The open-xchange service is finally started via start-stop-daemon which doesn't doesn't consider '''/etc/security/limits.*'''

=== RHEL 6 / CentOS 6 ===
;Init
: Upstart, System V compatible
;OX Configurable Limits/Defaults
: nofile, nproc

The mentioned limits can be configured via '''/opt/open-xchange/etc/ox-scriptconf.sh'''. The limits are applied via ulimit in the service's init script. Furthermore as the open-xchange service is finally started via '''su ... open-xchange''' on this distro a user session is opened via su/pam and the default CentOS pam config reads the '''/etc/security/limits.*''' configuration by loading the pam stack like:

: '''/etc/pam.d/su'''
:: '''-> /etc/pam.d/system-auth'''
::: '''-> pam_limits.so'''

If NPROC isn't configured for the open-xchange-server it's restricted to '''1024''' globally by default to prevent accidental fork bombs, see '''/etc/security/limits.d/90-nproc.conf''' which can result in severe problems modern multithreaded applications.

=== RHEL 7 / CentOS 7 / Debian 8 / SLE 12 ===
;Init
: Systemd
;OX Configurable Limits/Defaults
: nofile, nproc

For systemd the default limits are configured directly in the service's unit file that is shipped by OX and located at '''/usr/lib/systemd/system/open-xchange.service'''. The drop-in config to override or extend the default unit file is located at '''/etc/systemd/system/open-xchange.service.d/limits.conf'''. [https://www.freedesktop.org/software/systemd/man/systemd.exec.html Systemd.exec] shows a whole lot of options that can be used by admins to adapt the default service to their specific needs.

User:Marens

2017-04-24T20:56:53Z

Marens:

[[Grizzly]]

[[AppSuite:Open-Xchange_Installation_Guide_for_Debian_6.0/sandbox]]

[[AppSuite:OX_Tutorial_10K/sandbox]]

[[AppSuite:OX_Tutorial_100K/sandbox]]

[[AppSuite:OX_Tutorial_1M/sandbox]]

[[Template:OX_HE_Tutorial_Design]]

[[Template:Oxinstaller/sandbox]]

[[Appsuite:ResourceLimits/sandbox]]

[[Appsuite:ResourceLimits]]

[[Appsuite:ResourceLimits]]

[[Sandbox:Middleware_Startup/sandbox]]

[[Appsuite:MiddlewareStartup]]

Sandbox:Middleware Startup/sandbox

2017-04-24T20:50:52Z

Marens: /* Encoding detection during JVM startup */

== Systemd ==

==== Service definition ====
The default service definition as shipped by Open-Xchange is rather concise and easy to read. You can check by looking at the service file that is installed to either /usr/lib/systemd or /lib/systemd depending on your distribution of choice.

singlenode:~ # cat /usr/lib/systemd/system/open-xchange.service
[Unit]
After=remote-fs.target
After=time-sync.target ypbind.service sendmail.service cyrus.service

[Service]
User=open-xchange
PermissionsStartOnly=true
TimeoutStartSec=0
ExecStartPre=/opt/open-xchange/sbin/triggerupdatethemes -u
ExecStart=/opt/open-xchange/sbin/open-xchange
ExecStop=/opt/open-xchange/sbin/shutdown -w
ExecReload=/opt/open-xchange/sbin/triggerreloadconfiguration -d
KillMode=process
LimitNOFILE=65536
LimitNPROC=65536

[Install]
WantedBy=multi-user.target

==== Drop-in configs ====
Drop in configs allow administrators to easily adapt and override default service unit files. So if you want to change the default limits or add additional limits have a look at our default example drop-in config which is located at /etc/systemd/system/open-xchange.service.d/limits.conf

singlenode:~ # cat /etc/systemd/system/open-xchange.service.d/limits.conf
# Override and add options in this file
# See systemd.exec(5) for other limits

[Service]
#LimitNPROC=65536
#LimitNOFILE=65536

The systemd.exec documentation shows a whole lot of options that can be used by admins to adapt the default service to their specific needs.

 
[[File:mw_start_sysd.png|thumb|Service startup using systemd|left|1000px]]
 

== System V / Upstart ==

==== Service definition ====

The default service definition as shipped by Open-Xchange can be found at /etc/init.d/open-xchange and will differ slightly depending on the distro in use.

 
[[File:mw_start_sysv.png|thumb|Service startup using upstart|left|1000px]]
 
== Apply Resource limits ==

As you can see in the diagrams above one of the differences is that:

* systemd reads all the infos wrt. resource limitations from the original service file and then applies possible further restrictions from the drop-in configs (steps 2 and 8 in the diagram ''Service startup using systemd'') and limits configured in /opt/open-xchange/etc/ox-scriptconf.sh aren't considered anylonger
* the older System V style init has to source /opt/open-xchange/etc/ox-scriptconf.sh to be able to read the configs for the maximum number of processes (NPROC) and files(NRFILES) that the middleware shall be allowed to create/open (steps 3,4 and 15 in the diagram ''Service startup using upstart'').

=== Background Infos ===

Several ways exist to restrict resources on a linux system from a global level down to user/groups or even shells and the processes started by them.

==== Sysctl ====
Sysctl is used to modify kernel parameters at runtime. E.g. to set the maximum number of files

$ sysctl -w fs.file-max=100000

To permanently set them append to the main configuration file and reload the settings

$ echo fs.file-max=100000 >> /etc/sysctl.conf
$ sysctl -p

More infos can be found via '''man sysctl'''

==== Limits.conf ====

Allows to restrict resources an a global, group or user level. E.g:

$ cat /etc/security/limits.d/90-nproc.conf
# Default limit for number of user's processes to prevent
# accidental fork bombs.
# See rhbz #432903 for reasoning.

* soft nproc 1024

From '''man limits.conf''':
<blockquote>
''Also, please note that all limit settings are set per login. They are not global, nor are they permanent; existing only for the duration of the session.''
</blockquote>

The limits per login are applied via the ''pam'' stack. See '''man pam''' and '''man pam_limits''' for more details. As those limits are bound to sessions they don't affect most daemons started by our supported init systems or init utils. Most state that they are ignored by design, see [https://bugs.launchpad.net/ubuntu/+source/upstart/+bug/938669/comments/4 upstart], [https://bugzilla.redhat.com/show_bug.cgi?id=754285#c1 systemd] and [https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=302079 start-stop-daemon]

===== Ulimit =====
From '''man bash'''
<blockquote>
ulimit [-HSTabcdefilmnpqrstuvx [limit]]
Provides control over the resources available to the shell and to processes started by it, on systems that allow such control.
</blockquote>
This is what we use in our System V compatible init scripts to increase resources for the open-xchange process across multiple distros. Currently only ''the maximum number of processes'' and ''the maximum number of open file descriptors available to a single user'' are increased via ulimit. The values are specified in '''/opt/open-xchange/ox-scriptconf.sh'''

==== Systemd ====

===== Control Groups=====

Control groups should only affect the OX middleware if you create/manage them yourself of if you are using a ''modern'' distribution that already uses systemd as init.

Citing from the kernel cgroup [https://www.kernel.org/doc/Documentation/cgroup-v2.txt documentation]:

<blockquote>
1-2. What is cgroup?

cgroup is a mechanism to organize processes hierarchically and
distribute system resources along the hierarchy in a controlled and
configurable manner.

cgroup is largely composed of two parts - the core and controllers.
cgroup core is primarily responsible for hierarchically organizing
processes. A cgroup controller is usually responsible for
distributing a specific type of system resource along the hierarchy
although there are utility controllers which serve purposes other than
resource distribution.

cgroups form a tree structure and every process in the system belongs
to one and only one cgroup. All threads of a process belong to the
same cgroup. On creation, all processes are put in the cgroup that
the parent process belongs to at the time. A process can be migrated
to another cgroup. Migration of a process doesn't affect already
existing descendant processes.

Following certain structural constraints, controllers may be enabled or
disabled selectively on a cgroup. All controller behaviors are
hierarchical - if a controller is enabled on a cgroup, it affects all
processes which belong to the cgroups consisting the inclusive
sub-hierarchy of the cgroup. When a controller is enabled on a nested
cgroup, it always restricts the resource distribution further. The
restrictions set closer to the root in the hierarchy can not be
overridden from further away.
</blockquote>

So processes are organized into a tree structure of control groups and controllers are responsible for the distribution of resources. So what kind of controllers exist?

<blockquote>
5. Controllers

5-1. CPU

The "cpu" controllers regulates distribution of CPU cycles. This
controller implements weight and absolute bandwidth limit models for
normal scheduling policy and absolute bandwidth allocation model for
realtime scheduling policy.

5-2. Memory

The "memory" controller regulates distribution of memory.
...
While not completely water-tight, all major memory usages by a given
cgroup are tracked so that the total memory consumption can be
accounted and controlled to a reasonable extent.

5-3. IO

The "io" controller regulates the distribution of IO resources. This
controller implements both weight based and absolute bandwidth or IOPS
limit distribution; however, weight based distribution is available
only if cfq-iosched is in use and neither scheme is available for
blk-mq devices.
</blockquote>

The open-xchange service is simply put into the default ''system.slice'' without applying further limits.

singlenode$ systemd-cgls
├─1 /sbin/init
├─system.slice
│ ├─avahi-daemon.service
│ │ ├─501 avahi-daemon: running [singlenode]
│ │ └─514 avahi-daemon: chroot helper
│ ├─console-kit-daemon.service
│ │ └─16164 /usr/sbin/console-kit-daemon --no-daemon
│ ├─dbus.service
│ │ └─508 /usr/bin/dbus-daemon --system --address=systemd: --nofork --nopidfile --systemd-activation
│ ├─munin-node.service
│ │ └─4290 /usr/bin/perl -wT /usr/sbin/munin-node
│ ├─open-xchange.service
│ │ └─6037 /usr/bin/java -Dsun.net.inetaddr.ttl=3600 -Dnetworkaddress.cache.ttl=3600 -Dnetworkaddress.cache.negative.ttl=10 ...

To check all the details use

singlenode:~ # systemctl show system.slice

==== Limits besides control groups ====
Besides control groups systemd allows you to apply other limits to the execution environment of your service. Here we can apply the limits that would normally be applied via ''limits.conf'' or ''ulimit''. Systemd uses [http://man7.org/linux/man-pages/man2/setrlimit.2.html setrlimit] for this. The options that we set by default are:

* LimitNOFILE
* LimitNPROC

which you have already seen when we took a look at the [[#Service_definiton|service definition]] and [[#Drop-in_config|drop-in configs]]

== Verify limits ==

=== System V ===

Get the proper pid of the java process started for the middleware by either checking '''ps faux'''
root 4103 0.0 0.0 66528 1868 pts/0 S 22:05 0:00 su -s /bin/bash open-xchange -c /opt/open-xchange/sbin/open-xchange
494 4109 9.5 19.8 2224644 381312 ? Ssl 22:05 1:12 \_ /usr/bin/java -Dsun.net.inetaddr.ttl=3600 -Dnetworkaddress.cache.ttl=3600

or programmatically
singlenode:~ # pid=$(pgrep -P $(</var/run/open-xchange.pid) java)

and check the limits applied for that process
singlenode:~ # cat /proc/$pid/limits
Limit Soft Limit Hard Limit Units
Max cpu time unlimited unlimited seconds
Max file size unlimited unlimited bytes
Max data size unlimited unlimited bytes
Max stack size 8388608 unlimited bytes
Max core file size 0 unlimited bytes
Max resident set unlimited unlimited bytes
Max processes 65536 65536 processes
Max open files 65536 65536 files
Max locked memory 65536 65536 bytes
Max address space unlimited unlimited bytes
Max file locks unlimited unlimited locks
Max pending signals 24254 24254 signals
Max msgqueue size 819200 819200 bytes
Max nice priority 0 0
Max realtime priority 0 0
Max realtime timeout unlimited unlimited us

=== Systemd ===
singlenode:~ # systemctl show open-xchange | grep Limit
StartLimitInterval=10000000
StartLimitBurst=5
StartLimitAction=none
MemoryLimit=18446744073709551615
LimitCPU=18446744073709551615
LimitFSIZE=18446744073709551615
LimitDATA=18446744073709551615
LimitSTACK=18446744073709551615
LimitCORE=18446744073709551615
LimitRSS=18446744073709551615
LimitNOFILE=65536
LimitAS=18446744073709551615
LimitNPROC=65536
LimitMEMLOCK=65536
LimitLOCKS=18446744073709551615
LimitSIGPENDING=19827
LimitMSGQUEUE=819200
LimitNICE=0
LimitRTPRIO=0
LimitRTTIME=18446744073709551615

== Evaluate JVM commandline options ==

Before starting Java and the OSGi framework with all the Open-Xchange middleware bundles we have to evaluate several important JVM commandline options that influence the performance and stability of the Open-Xchange middleware stack. These commandline options are specified in '''/opt/open-xchange/etc/ox-scriptconf.sh'''

=== Format < 7.8.4 ===
For version up to 7.8.3 the JVM configuration options were all specified via a single property '''JAVA_XTRAOPTS''' which resulted in a rather long configuration string that was hard to read and maintain.

JAVA_XTRAOPTS="-Dsun.net.inetaddr.ttl=3600 -Dnetworkaddress.cache.ttl=3600 -Dnetworkaddress.cache.negative.ttl=10 \
-Dlogback.threadlocal.put.duplicate=false -server -Djava.awt.headless=true -Xmx512M -XX:MaxPermSize=256m -XX:+UseConcMarkSweepGC \
-XX:+UseParNewGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:NewRatio=3 -XX:+UseTLAB \
-Dosgi.compatibility.bootdelegation=false -XX:-OmitStackTraceInFastThrow"

During the startup of the open-xchange service (step 11 of diagram Service startup using systemd and step 10 of diagram Service startup using upstart) this JAVA_XTRAOPTS string goes through a basic validity check and is then passed to the JVM.

=== Format >= 7.8.4 ===
Beginning from version 7.8.4 the options are split up and sorted into different categories like:

JAVA_OPTS_GC="-XX:+UseConcMarkSweepGC -XX:+UseParNewGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:NewRatio=3 -XX:+DisableExplicitGC"
JAVA_OPTS_LOG="-Dlogback.threadlocal.put.duplicate=false -XX:-OmitStackTraceInFastThrow"
JAVA_OPTS_MEM="-XX:MaxHeapSize=512M -XX:MaxPermSize=256M -XX:+UseTLAB"
JAVA_OPTS_NET="-Dsun.net.inetaddr.ttl=3600 -Dnetworkaddress.cache.ttl=3600 -Dnetworkaddress.cache.negative.ttl=10"
JAVA_OPTS_OSGI="-Dosgi.compatibility.bootdelegation=false"
JAVA_OPTS_SERVER="-server -Djava.awt.headless=true"

JAVA_OPTS_OTHER=""

#JAVA_OPTS_DEBUG="-XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/mnt/heapdump -Xloggc:/var/log/open-xchange/gc.log -verbose:gc -XX:+PrintGCDateStamps \
-XX:+PrintHeapAtGC -XX:+PrintGCApplicationStoppedTime -XX:+PrintTenuringDistribution"

which is much easier to read an maintain. Furthermore we added a useful '''JAVA_OPTS_DEBUG''' configuration parameter that comes preconfigured but unused/commented so that customers can quickly enable the most commonly used debugging options in case they have to provide detailed system information to Open-Xchange.

During the startup of the open-xchange service (step 11 of diagram ''Service startup using systemd'' and step 10 of diagram ''Service startup using upstart'') all uncommented JAVA_OPTS_* categories will then be evaluated and combined again into a single string of commandline options that can then be passed to the JVM.

=== Upgrade procedure ===
During the upgrade '''7.8.3-revX -> 7.8.4-revY''' existing JAVA_XTRAOPTS will automatically be split up and migrated to the new categories approach. A backup of the file (ox-scriptconf.sh.timestamp) will be created and admins are urged to check for proper migration before deleting the backup file like it's already done with *.dpkg-dist or *.rpmnew files on Debian or RHEL/SLES distributions.

== Encoding detection during JVM startup ==

=== Background Infos ===

From ''man 7 locale''

DESCRIPTION
A locale is a set of language and cultural rules. These cover aspects such as lan‐
guage for messages, different character sets, lexicographic conventions, and so on.
A program needs to be able to determine its locale and act accordingly to be portable
to different cultures.

The format to specify a system locale is: '''''language[_territory][.codeset][@modifier]'''''.

Where all of

* en_US.UTF-8
* en_US.utf8
* en_US.utf-8
* en_US.UTF_8

will result in a valid UTF-8 charmap for the english language with locale categories configured as

LANG=en_US.UTF-8
LC_CTYPE="en_US.UTF-8"
LC_NUMERIC="en_US.UTF-8"
LC_TIME="en_US.UTF-8"
LC_COLLATE="en_US.UTF-8"
LC_MONETARY="en_US.UTF-8"
LC_MESSAGES="en_US.UTF-8"
LC_PAPER="en_US.UTF-8"
LC_NAME="en_US.UTF-8"
LC_ADDRESS="en_US.UTF-8"
LC_TELEPHONE="en_US.UTF-8"
LC_MEASUREMENT="en_US.UTF-8"
LC_IDENTIFICATION="en_US.UTF-8"
LC_ALL=

See for more infos: locale(1), locale(7), [https://en.wikipedia.org/wiki/List_of_ISO_639-1_codes language codes], [https://en.wikipedia.org/wiki/ISO_3166-1#Current_codes country codes] and [https://en.wikipedia.org/wiki/Character_encoding character encodings]

==== JVM startup ====

The default file.encoding detection is done at JVM startup based upon LC_CTYPE which makes sense as it's documented as:

CODESET (LC_CTYPE)
Return a string with the name of the character encoding used in the selected locale, such as "UTF-8", "ISO-8859-1",
or "ANSI_X3.4-1968" (better known as US-ASCII). This is the same string that you get with "locale charmap".
For a list of character encoding names, try "locale -m", cf. locale(1).

Furthermore this influences the defaultCharset selection of the JVM.

/**
* Returns the default charset of this Java virtual machine.
*
* The default charset is determined during virtual-machine startup and
* typically depends upon the locale and charset of the underlying
* operating system.
*
* @return A charset object for the default charset
*
* @since 1.5
*/
public static Charset defaultCharset() {
if (defaultCharset == null) {
synchronized (Charset.class) {
String csn = AccessController.doPrivileged(
new GetPropertyAction("file.encoding"));
Charset cs = lookup(csn);
if (cs != null)
defaultCharset = cs;
else
defaultCharset = forName("UTF-8");
}
}
return defaultCharset;
}

Although we try to specify valid unicode encodings for every IO operations the Open-Xchange middleware does some parts of the JVM don't allow to specify the encoding to use and falls back to the detected default encoding. That's why admins have to make sure to provide a proper unicode environment for the middleware service during every start or use documented workarounds provided by Open-Xchange (e.g. )

=== Query which file encoding was detected by the JVM during startup ===
[[File:visualvm-file.encoding.png|thumb|Querying the JVM via visualvm|right|300px]]

The file encoding that was detected by the JVM during startup can be queried via JMX. You can use graphical management tools like [https://visualvm.github.io/ visualvm], query our [[Jolokia]] JMX bridge via http or just use the commandline like

/opt/open-xchange/sbin/showruntimestats -r | sed -ne 's/.*${key=file.encoding, value=[^}]*}$.*/\1/p'
{key=file.encoding, value=UTF-8}

 

=== Locale passing from client to server via SSH ===

The following setting of sshd allows clients to specify which locale to use on the server and might lead to some confusion or unexpected behaviour when checking the system locale or even to the usage of broken locales during middleware startup if your client sends misconfigured or missing locale configuration to the server.

grep AcceptEnv /etc/ssh/sshd_config
AcceptEnv LANG LC_*

For example when i connect to the machine my local german locale is sent to the server

[vagrant@showcase-community-centos-7 ~]$ locale -v
LANG=de_DE.UTF-8
LC_CTYPE="de_DE.UTF-8"
LC_NUMERIC=de_DE.UTF-8
LC_TIME=de_DE.UTF-8
LC_COLLATE="de_DE.UTF-8"
LC_MONETARY=de_DE.UTF-8
LC_MESSAGES="de_DE.UTF-8"
LC_PAPER=de_DE.UTF-8
LC_NAME=de_DE.UTF-8
LC_ADDRESS=de_DE.UTF-8
LC_TELEPHONE=de_DE.UTF-8
LC_MEASUREMENT=de_DE.UTF-8
LC_IDENTIFICATION=de_DE.UTF-8
LC_ALL=

But after specifying that i want a new login shell the default locale configured for that user/system will be used instead!

[root@showcase-community-centos-7 vagrant]# su -s /bin/bash open-xchange -l
-bash-4.2$ locale -v
LANG=en_US.UTF-8
LC_CTYPE="en_US.UTF-8"
LC_NUMERIC="en_US.UTF-8"
LC_TIME="en_US.UTF-8"
LC_COLLATE="en_US.UTF-8"
LC_MONETARY="en_US.UTF-8"
LC_MESSAGES="en_US.UTF-8"
LC_PAPER="en_US.UTF-8"
LC_NAME="en_US.UTF-8"
LC_ADDRESS="en_US.UTF-8"
LC_TELEPHONE="en_US.UTF-8"
LC_MEASUREMENT="en_US.UTF-8"
LC_IDENTIFICATION="en_US.UTF-8"
LC_ALL=

=== Verify that your system is configured with a proper unicode locale for OX ===

Assuming you haven't reconfigured the locale of your root user

i|marens@karotte ~ $ su -
Password:
i|karotte ~ # locale charmap
UTF-8
i|karotte ~ #

== Warnings during early start ==

Early warnings before the start of the JVM and related logging libraries like logback and logstash forwarding are redirected to '''/var/log/open-xchange/open-xchange-console.log'''.

=== Limits ===
If the open-xchange startup script isn't able to set the proper limits on CentOS 6 due to e.g. hard limits being enforced via pam_limits you'll see the following warning in the console log during startup

singlenode open-xchange # cat open-xchange-console.log
/opt/open-xchange/sbin/open-xchange: line 115: ulimit: max user processes: cannot modify limit: Operation not permitted

=== Locale ===
If the open-xchange startup script isn't able to detect a unicode locale you'll see a warning like the following (the current charmap might differ)

WARNING: Couldn't detect proper unicode charmap in locale category LC_CTYPE.
WARNING: Current charmap: "ANSI_X3.4-1968" might cause encoding problems.

== Summary: Open-Xchange middleware on specific distros ==
The support for the mentioned mechanism of resource control differ depending on the distribution and the init system in use.

=== Debian 7 ===
;Init
: System V style
;OX Configurable Limits/Defaults
: nofile, nproc

The mentioned limits can be configured via '''/opt/open-xchange/etc/ox-scriptconf.sh'''. The limits are applied via ulimit in the service's init script. The open-xchange service is finally started via start-stop-daemon which doesn't doesn't consider '''/etc/security/limits.*'''

=== RHEL 6 / CentOS 6 ===
;Init
: Upstart, System V compatible
;OX Configurable Limits/Defaults
: nofile, nproc

The mentioned limits can be configured via '''/opt/open-xchange/etc/ox-scriptconf.sh'''. The limits are applied via ulimit in the service's init script. Furthermore as the open-xchange service is finally started via '''su ... open-xchange''' on this distro a user session is opened via su/pam and the default CentOS pam config reads the '''/etc/security/limits.*''' configuration by loading the pam stack like:

: '''/etc/pam.d/su'''
:: '''-> /etc/pam.d/system-auth'''
::: '''-> pam_limits.so'''

If NPROC isn't configured for the open-xchange-server it's restricted to '''1024''' globally by default to prevent accidental fork bombs, see '''/etc/security/limits.d/90-nproc.conf''' which can result in severe problems modern multithreaded applications.

=== RHEL 7 / CentOS 7 / Debian 8 / SLE 12 ===
;Init
: Systemd
;OX Configurable Limits/Defaults
: nofile, nproc

For systemd the default limits are configured directly in the service's unit file that is shipped by OX and located at '''/usr/lib/systemd/system/open-xchange.service'''. The drop-in config to override or extend the default unit file is located at '''/etc/systemd/system/open-xchange.service.d/limits.conf'''. [https://www.freedesktop.org/software/systemd/man/systemd.exec.html Systemd.exec] shows a whole lot of options that can be used by admins to adapt the default service to their specific needs.

Sandbox:Middleware Startup/sandbox

2017-04-24T20:11:02Z

Marens: /* Locale passing from client to server via SSH */

== Systemd ==

==== Service definition ====
The default service definition as shipped by Open-Xchange is rather concise and easy to read. You can check by looking at the service file that is installed to either /usr/lib/systemd or /lib/systemd depending on your distribution of choice.

singlenode:~ # cat /usr/lib/systemd/system/open-xchange.service
[Unit]
After=remote-fs.target
After=time-sync.target ypbind.service sendmail.service cyrus.service

[Service]
User=open-xchange
PermissionsStartOnly=true
TimeoutStartSec=0
ExecStartPre=/opt/open-xchange/sbin/triggerupdatethemes -u
ExecStart=/opt/open-xchange/sbin/open-xchange
ExecStop=/opt/open-xchange/sbin/shutdown -w
ExecReload=/opt/open-xchange/sbin/triggerreloadconfiguration -d
KillMode=process
LimitNOFILE=65536
LimitNPROC=65536

[Install]
WantedBy=multi-user.target

==== Drop-in configs ====
Drop in configs allow administrators to easily adapt and override default service unit files. So if you want to change the default limits or add additional limits have a look at our default example drop-in config which is located at /etc/systemd/system/open-xchange.service.d/limits.conf

singlenode:~ # cat /etc/systemd/system/open-xchange.service.d/limits.conf
# Override and add options in this file
# See systemd.exec(5) for other limits

[Service]
#LimitNPROC=65536
#LimitNOFILE=65536

The systemd.exec documentation shows a whole lot of options that can be used by admins to adapt the default service to their specific needs.

 
[[File:mw_start_sysd.png|thumb|Service startup using systemd|left|1000px]]
 

== System V / Upstart ==

==== Service definition ====

The default service definition as shipped by Open-Xchange can be found at /etc/init.d/open-xchange and will differ slightly depending on the distro in use.

 
[[File:mw_start_sysv.png|thumb|Service startup using upstart|left|1000px]]
 
== Apply Resource limits ==

As you can see in the diagrams above one of the differences is that:

* systemd reads all the infos wrt. resource limitations from the original service file and then applies possible further restrictions from the drop-in configs (steps 2 and 8 in the diagram ''Service startup using systemd'') and limits configured in /opt/open-xchange/etc/ox-scriptconf.sh aren't considered anylonger
* the older System V style init has to source /opt/open-xchange/etc/ox-scriptconf.sh to be able to read the configs for the maximum number of processes (NPROC) and files(NRFILES) that the middleware shall be allowed to create/open (steps 3,4 and 15 in the diagram ''Service startup using upstart'').

=== Background Infos ===

Several ways exist to restrict resources on a linux system from a global level down to user/groups or even shells and the processes started by them.

==== Sysctl ====
Sysctl is used to modify kernel parameters at runtime. E.g. to set the maximum number of files

$ sysctl -w fs.file-max=100000

To permanently set them append to the main configuration file and reload the settings

$ echo fs.file-max=100000 >> /etc/sysctl.conf
$ sysctl -p

More infos can be found via '''man sysctl'''

==== Limits.conf ====

Allows to restrict resources an a global, group or user level. E.g:

$ cat /etc/security/limits.d/90-nproc.conf
# Default limit for number of user's processes to prevent
# accidental fork bombs.
# See rhbz #432903 for reasoning.

* soft nproc 1024

From '''man limits.conf''':
<blockquote>
''Also, please note that all limit settings are set per login. They are not global, nor are they permanent; existing only for the duration of the session.''
</blockquote>

The limits per login are applied via the ''pam'' stack. See '''man pam''' and '''man pam_limits''' for more details. As those limits are bound to sessions they don't affect most daemons started by our supported init systems or init utils. Most state that they are ignored by design, see [https://bugs.launchpad.net/ubuntu/+source/upstart/+bug/938669/comments/4 upstart], [https://bugzilla.redhat.com/show_bug.cgi?id=754285#c1 systemd] and [https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=302079 start-stop-daemon]

===== Ulimit =====
From '''man bash'''
<blockquote>
ulimit [-HSTabcdefilmnpqrstuvx [limit]]
Provides control over the resources available to the shell and to processes started by it, on systems that allow such control.
</blockquote>
This is what we use in our System V compatible init scripts to increase resources for the open-xchange process across multiple distros. Currently only ''the maximum number of processes'' and ''the maximum number of open file descriptors available to a single user'' are increased via ulimit. The values are specified in '''/opt/open-xchange/ox-scriptconf.sh'''

==== Systemd ====

===== Control Groups=====

Control groups should only affect the OX middleware if you create/manage them yourself of if you are using a ''modern'' distribution that already uses systemd as init.

Citing from the kernel cgroup [https://www.kernel.org/doc/Documentation/cgroup-v2.txt documentation]:

<blockquote>
1-2. What is cgroup?

cgroup is a mechanism to organize processes hierarchically and
distribute system resources along the hierarchy in a controlled and
configurable manner.

cgroup is largely composed of two parts - the core and controllers.
cgroup core is primarily responsible for hierarchically organizing
processes. A cgroup controller is usually responsible for
distributing a specific type of system resource along the hierarchy
although there are utility controllers which serve purposes other than
resource distribution.

cgroups form a tree structure and every process in the system belongs
to one and only one cgroup. All threads of a process belong to the
same cgroup. On creation, all processes are put in the cgroup that
the parent process belongs to at the time. A process can be migrated
to another cgroup. Migration of a process doesn't affect already
existing descendant processes.

Following certain structural constraints, controllers may be enabled or
disabled selectively on a cgroup. All controller behaviors are
hierarchical - if a controller is enabled on a cgroup, it affects all
processes which belong to the cgroups consisting the inclusive
sub-hierarchy of the cgroup. When a controller is enabled on a nested
cgroup, it always restricts the resource distribution further. The
restrictions set closer to the root in the hierarchy can not be
overridden from further away.
</blockquote>

So processes are organized into a tree structure of control groups and controllers are responsible for the distribution of resources. So what kind of controllers exist?

<blockquote>
5. Controllers

5-1. CPU

The "cpu" controllers regulates distribution of CPU cycles. This
controller implements weight and absolute bandwidth limit models for
normal scheduling policy and absolute bandwidth allocation model for
realtime scheduling policy.

5-2. Memory

The "memory" controller regulates distribution of memory.
...
While not completely water-tight, all major memory usages by a given
cgroup are tracked so that the total memory consumption can be
accounted and controlled to a reasonable extent.

5-3. IO

The "io" controller regulates the distribution of IO resources. This
controller implements both weight based and absolute bandwidth or IOPS
limit distribution; however, weight based distribution is available
only if cfq-iosched is in use and neither scheme is available for
blk-mq devices.
</blockquote>

The open-xchange service is simply put into the default ''system.slice'' without applying further limits.

singlenode$ systemd-cgls
├─1 /sbin/init
├─system.slice
│ ├─avahi-daemon.service
│ │ ├─501 avahi-daemon: running [singlenode]
│ │ └─514 avahi-daemon: chroot helper
│ ├─console-kit-daemon.service
│ │ └─16164 /usr/sbin/console-kit-daemon --no-daemon
│ ├─dbus.service
│ │ └─508 /usr/bin/dbus-daemon --system --address=systemd: --nofork --nopidfile --systemd-activation
│ ├─munin-node.service
│ │ └─4290 /usr/bin/perl -wT /usr/sbin/munin-node
│ ├─open-xchange.service
│ │ └─6037 /usr/bin/java -Dsun.net.inetaddr.ttl=3600 -Dnetworkaddress.cache.ttl=3600 -Dnetworkaddress.cache.negative.ttl=10 ...

To check all the details use

singlenode:~ # systemctl show system.slice

==== Limits besides control groups ====
Besides control groups systemd allows you to apply other limits to the execution environment of your service. Here we can apply the limits that would normally be applied via ''limits.conf'' or ''ulimit''. Systemd uses [http://man7.org/linux/man-pages/man2/setrlimit.2.html setrlimit] for this. The options that we set by default are:

* LimitNOFILE
* LimitNPROC

which you have already seen when we took a look at the [[#Service_definiton|service definition]] and [[#Drop-in_config|drop-in configs]]

== Verify limits ==

=== System V ===

Get the proper pid of the java process started for the middleware by either checking '''ps faux'''
root 4103 0.0 0.0 66528 1868 pts/0 S 22:05 0:00 su -s /bin/bash open-xchange -c /opt/open-xchange/sbin/open-xchange
494 4109 9.5 19.8 2224644 381312 ? Ssl 22:05 1:12 \_ /usr/bin/java -Dsun.net.inetaddr.ttl=3600 -Dnetworkaddress.cache.ttl=3600

or programmatically
singlenode:~ # pid=$(pgrep -P $(</var/run/open-xchange.pid) java)

and check the limits applied for that process
singlenode:~ # cat /proc/$pid/limits
Limit Soft Limit Hard Limit Units
Max cpu time unlimited unlimited seconds
Max file size unlimited unlimited bytes
Max data size unlimited unlimited bytes
Max stack size 8388608 unlimited bytes
Max core file size 0 unlimited bytes
Max resident set unlimited unlimited bytes
Max processes 65536 65536 processes
Max open files 65536 65536 files
Max locked memory 65536 65536 bytes
Max address space unlimited unlimited bytes
Max file locks unlimited unlimited locks
Max pending signals 24254 24254 signals
Max msgqueue size 819200 819200 bytes
Max nice priority 0 0
Max realtime priority 0 0
Max realtime timeout unlimited unlimited us

=== Systemd ===
singlenode:~ # systemctl show open-xchange | grep Limit
StartLimitInterval=10000000
StartLimitBurst=5
StartLimitAction=none
MemoryLimit=18446744073709551615
LimitCPU=18446744073709551615
LimitFSIZE=18446744073709551615
LimitDATA=18446744073709551615
LimitSTACK=18446744073709551615
LimitCORE=18446744073709551615
LimitRSS=18446744073709551615
LimitNOFILE=65536
LimitAS=18446744073709551615
LimitNPROC=65536
LimitMEMLOCK=65536
LimitLOCKS=18446744073709551615
LimitSIGPENDING=19827
LimitMSGQUEUE=819200
LimitNICE=0
LimitRTPRIO=0
LimitRTTIME=18446744073709551615

== Evaluate JVM commandline options ==

Before starting Java and the OSGi framework with all the Open-Xchange middleware bundles we have to evaluate several important JVM commandline options that influence the performance and stability of the Open-Xchange middleware stack. These commandline options are specified in '''/opt/open-xchange/etc/ox-scriptconf.sh'''

=== Format < 7.8.4 ===
For version up to 7.8.3 the JVM configuration options were all specified via a single property '''JAVA_XTRAOPTS''' which resulted in a rather long configuration string that was hard to read and maintain.

JAVA_XTRAOPTS="-Dsun.net.inetaddr.ttl=3600 -Dnetworkaddress.cache.ttl=3600 -Dnetworkaddress.cache.negative.ttl=10 \
-Dlogback.threadlocal.put.duplicate=false -server -Djava.awt.headless=true -Xmx512M -XX:MaxPermSize=256m -XX:+UseConcMarkSweepGC \
-XX:+UseParNewGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:NewRatio=3 -XX:+UseTLAB \
-Dosgi.compatibility.bootdelegation=false -XX:-OmitStackTraceInFastThrow"

During the startup of the open-xchange service (step 11 of diagram Service startup using systemd and step 10 of diagram Service startup using upstart) this JAVA_XTRAOPTS string goes through a basic validity check and is then passed to the JVM.

=== Format >= 7.8.4 ===
Beginning from version 7.8.4 the options are split up and sorted into different categories like:

JAVA_OPTS_GC="-XX:+UseConcMarkSweepGC -XX:+UseParNewGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:NewRatio=3 -XX:+DisableExplicitGC"
JAVA_OPTS_LOG="-Dlogback.threadlocal.put.duplicate=false -XX:-OmitStackTraceInFastThrow"
JAVA_OPTS_MEM="-XX:MaxHeapSize=512M -XX:MaxPermSize=256M -XX:+UseTLAB"
JAVA_OPTS_NET="-Dsun.net.inetaddr.ttl=3600 -Dnetworkaddress.cache.ttl=3600 -Dnetworkaddress.cache.negative.ttl=10"
JAVA_OPTS_OSGI="-Dosgi.compatibility.bootdelegation=false"
JAVA_OPTS_SERVER="-server -Djava.awt.headless=true"

JAVA_OPTS_OTHER=""

#JAVA_OPTS_DEBUG="-XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/mnt/heapdump -Xloggc:/var/log/open-xchange/gc.log -verbose:gc -XX:+PrintGCDateStamps \
-XX:+PrintHeapAtGC -XX:+PrintGCApplicationStoppedTime -XX:+PrintTenuringDistribution"

which is much easier to read an maintain. Furthermore we added a useful '''JAVA_OPTS_DEBUG''' configuration parameter that comes preconfigured but unused/commented so that customers can quickly enable the most commonly used debugging options in case they have to provide detailed system information to Open-Xchange.

During the startup of the open-xchange service (step 11 of diagram ''Service startup using systemd'' and step 10 of diagram ''Service startup using upstart'') all uncommented JAVA_OPTS_* categories will then be evaluated and combined again into a single string of commandline options that can then be passed to the JVM.

=== Upgrade procedure ===
During the upgrade '''7.8.3-revX -> 7.8.4-revY''' existing JAVA_XTRAOPTS will automatically be split up and migrated to the new categories approach. A backup of the file (ox-scriptconf.sh.timestamp) will be created and admins are urged to check for proper migration before deleting the backup file like it's already done with *.dpkg-dist or *.rpmnew files on Debian or RHEL/SLES distributions.

== Encoding detection during JVM startup ==

=== Background Infos ===
All of en_US.UTF-8, .utf8, .utf-8, .UTF_8 will result in a valid UTF-8 charmap

==== JVM startup ====

The default file.encoding detection is done at JVM startup based upon LC_CTYPE which makes sense as it's documented as:

CODESET (LC_CTYPE)
Return a string with the name of the character encoding used in the selected locale, such as "UTF-8", "ISO-8859-1",
or "ANSI_X3.4-1968" (better known as US-ASCII). This is the same string that you get with "locale charmap".
For a list of character encoding names, try "locale -m", cf. locale(1).

Furthermore this influences the defaultCharset selection of the JVM.

/**
* Returns the default charset of this Java virtual machine.
*
* The default charset is determined during virtual-machine startup and
* typically depends upon the locale and charset of the underlying
* operating system.
*
* @return A charset object for the default charset
*
* @since 1.5
*/
public static Charset defaultCharset() {
if (defaultCharset == null) {
synchronized (Charset.class) {
String csn = AccessController.doPrivileged(
new GetPropertyAction("file.encoding"));
Charset cs = lookup(csn);
if (cs != null)
defaultCharset = cs;
else
defaultCharset = forName("UTF-8");
}
}
return defaultCharset;
}

Although we try to specify valid unicode encodings for every IO operations the Open-Xchange middleware does some parts of the JVM don't allow to specify the encoding to use and falls back to the detected default encoding. That's why admins have to make sure to provide a proper unicode environment for the middleware service during every start or use documented workarounds provided by Open-Xchange (e.g. )

=== Query which file encoding was detected by the JVM during startup ===
[[File:visualvm-file.encoding.png|thumb|Querying the JVM via visualvm|right|300px]]

The file encoding that was detected by the JVM during startup can be queried via JMX. You can use graphical management tools like [https://visualvm.github.io/ visualvm], query our [[Jolokia]] JMX bridge via http or just use the commandline like

/opt/open-xchange/sbin/showruntimestats -r | sed -ne 's/.*${key=file.encoding, value=[^}]*}$.*/\1/p'
{key=file.encoding, value=UTF-8}

 

=== Locale passing from client to server via SSH ===

The following setting of sshd allows clients to specify which locale to use on the server and might lead to some confusion or unexpected behaviour when checking the system locale or even to the usage of broken locales during middleware startup if your client sends misconfigured or missing locale configuration to the server.

grep AcceptEnv /etc/ssh/sshd_config
AcceptEnv LANG LC_*

For example when i connect to the machine my local german locale is sent to the server

[vagrant@showcase-community-centos-7 ~]$ locale -v
LANG=de_DE.UTF-8
LC_CTYPE="de_DE.UTF-8"
LC_NUMERIC=de_DE.UTF-8
LC_TIME=de_DE.UTF-8
LC_COLLATE="de_DE.UTF-8"
LC_MONETARY=de_DE.UTF-8
LC_MESSAGES="de_DE.UTF-8"
LC_PAPER=de_DE.UTF-8
LC_NAME=de_DE.UTF-8
LC_ADDRESS=de_DE.UTF-8
LC_TELEPHONE=de_DE.UTF-8
LC_MEASUREMENT=de_DE.UTF-8
LC_IDENTIFICATION=de_DE.UTF-8
LC_ALL=

But after specifying that i want a new login shell the default locale configured for that user/system will be used instead!

[root@showcase-community-centos-7 vagrant]# su -s /bin/bash open-xchange -l
-bash-4.2$ locale -v
LANG=en_US.UTF-8
LC_CTYPE="en_US.UTF-8"
LC_NUMERIC="en_US.UTF-8"
LC_TIME="en_US.UTF-8"
LC_COLLATE="en_US.UTF-8"
LC_MONETARY="en_US.UTF-8"
LC_MESSAGES="en_US.UTF-8"
LC_PAPER="en_US.UTF-8"
LC_NAME="en_US.UTF-8"
LC_ADDRESS="en_US.UTF-8"
LC_TELEPHONE="en_US.UTF-8"
LC_MEASUREMENT="en_US.UTF-8"
LC_IDENTIFICATION="en_US.UTF-8"
LC_ALL=

== Warnings during early start ==

Early warnings before the start of the JVM and related logging libraries like logback and logstash forwarding are redirected to '''/var/log/open-xchange/open-xchange-console.log'''.

=== Limits ===
If the open-xchange startup script isn't able to set the proper limits on CentOS 6 due to e.g. hard limits being enforced via pam_limits you'll see the following warning in the console log during startup

singlenode open-xchange # cat open-xchange-console.log
/opt/open-xchange/sbin/open-xchange: line 115: ulimit: max user processes: cannot modify limit: Operation not permitted

=== Locale ===
If the open-xchange startup script isn't able to detect a unicode locale you'll see a warning like the following (the current charmap might differ)

WARNING: Couldn't detect proper unicode charmap in locale category LC_CTYPE.
WARNING: Current charmap: "ANSI_X3.4-1968" might cause encoding problems.

== Summary: Open-Xchange middleware on specific distros ==
The support for the mentioned mechanism of resource control differ depending on the distribution and the init system in use.

=== Debian 7 ===
;Init
: System V style
;OX Configurable Limits/Defaults
: nofile, nproc

The mentioned limits can be configured via '''/opt/open-xchange/etc/ox-scriptconf.sh'''. The limits are applied via ulimit in the service's init script. The open-xchange service is finally started via start-stop-daemon which doesn't doesn't consider '''/etc/security/limits.*'''

=== RHEL 6 / CentOS 6 ===
;Init
: Upstart, System V compatible
;OX Configurable Limits/Defaults
: nofile, nproc

The mentioned limits can be configured via '''/opt/open-xchange/etc/ox-scriptconf.sh'''. The limits are applied via ulimit in the service's init script. Furthermore as the open-xchange service is finally started via '''su ... open-xchange''' on this distro a user session is opened via su/pam and the default CentOS pam config reads the '''/etc/security/limits.*''' configuration by loading the pam stack like:

: '''/etc/pam.d/su'''
:: '''-> /etc/pam.d/system-auth'''
::: '''-> pam_limits.so'''

If NPROC isn't configured for the open-xchange-server it's restricted to '''1024''' globally by default to prevent accidental fork bombs, see '''/etc/security/limits.d/90-nproc.conf''' which can result in severe problems modern multithreaded applications.

=== RHEL 7 / CentOS 7 / Debian 8 / SLE 12 ===
;Init
: Systemd
;OX Configurable Limits/Defaults
: nofile, nproc

For systemd the default limits are configured directly in the service's unit file that is shipped by OX and located at '''/usr/lib/systemd/system/open-xchange.service'''. The drop-in config to override or extend the default unit file is located at '''/etc/systemd/system/open-xchange.service.d/limits.conf'''. [https://www.freedesktop.org/software/systemd/man/systemd.exec.html Systemd.exec] shows a whole lot of options that can be used by admins to adapt the default service to their specific needs.

Sandbox:Middleware Startup/sandbox

2017-04-24T20:10:14Z

Marens:

== Systemd ==

==== Service definition ====
The default service definition as shipped by Open-Xchange is rather concise and easy to read. You can check by looking at the service file that is installed to either /usr/lib/systemd or /lib/systemd depending on your distribution of choice.

singlenode:~ # cat /usr/lib/systemd/system/open-xchange.service
[Unit]
After=remote-fs.target
After=time-sync.target ypbind.service sendmail.service cyrus.service

[Service]
User=open-xchange
PermissionsStartOnly=true
TimeoutStartSec=0
ExecStartPre=/opt/open-xchange/sbin/triggerupdatethemes -u
ExecStart=/opt/open-xchange/sbin/open-xchange
ExecStop=/opt/open-xchange/sbin/shutdown -w
ExecReload=/opt/open-xchange/sbin/triggerreloadconfiguration -d
KillMode=process
LimitNOFILE=65536
LimitNPROC=65536

[Install]
WantedBy=multi-user.target

==== Drop-in configs ====
Drop in configs allow administrators to easily adapt and override default service unit files. So if you want to change the default limits or add additional limits have a look at our default example drop-in config which is located at /etc/systemd/system/open-xchange.service.d/limits.conf

singlenode:~ # cat /etc/systemd/system/open-xchange.service.d/limits.conf
# Override and add options in this file
# See systemd.exec(5) for other limits

[Service]
#LimitNPROC=65536
#LimitNOFILE=65536

The systemd.exec documentation shows a whole lot of options that can be used by admins to adapt the default service to their specific needs.

 
[[File:mw_start_sysd.png|thumb|Service startup using systemd|left|1000px]]
 

== System V / Upstart ==

==== Service definition ====

The default service definition as shipped by Open-Xchange can be found at /etc/init.d/open-xchange and will differ slightly depending on the distro in use.

 
[[File:mw_start_sysv.png|thumb|Service startup using upstart|left|1000px]]
 
== Apply Resource limits ==

As you can see in the diagrams above one of the differences is that:

* systemd reads all the infos wrt. resource limitations from the original service file and then applies possible further restrictions from the drop-in configs (steps 2 and 8 in the diagram ''Service startup using systemd'') and limits configured in /opt/open-xchange/etc/ox-scriptconf.sh aren't considered anylonger
* the older System V style init has to source /opt/open-xchange/etc/ox-scriptconf.sh to be able to read the configs for the maximum number of processes (NPROC) and files(NRFILES) that the middleware shall be allowed to create/open (steps 3,4 and 15 in the diagram ''Service startup using upstart'').

=== Background Infos ===

Several ways exist to restrict resources on a linux system from a global level down to user/groups or even shells and the processes started by them.

==== Sysctl ====
Sysctl is used to modify kernel parameters at runtime. E.g. to set the maximum number of files

$ sysctl -w fs.file-max=100000

To permanently set them append to the main configuration file and reload the settings

$ echo fs.file-max=100000 >> /etc/sysctl.conf
$ sysctl -p

More infos can be found via '''man sysctl'''

==== Limits.conf ====

Allows to restrict resources an a global, group or user level. E.g:

$ cat /etc/security/limits.d/90-nproc.conf
# Default limit for number of user's processes to prevent
# accidental fork bombs.
# See rhbz #432903 for reasoning.

* soft nproc 1024

From '''man limits.conf''':
<blockquote>
''Also, please note that all limit settings are set per login. They are not global, nor are they permanent; existing only for the duration of the session.''
</blockquote>

The limits per login are applied via the ''pam'' stack. See '''man pam''' and '''man pam_limits''' for more details. As those limits are bound to sessions they don't affect most daemons started by our supported init systems or init utils. Most state that they are ignored by design, see [https://bugs.launchpad.net/ubuntu/+source/upstart/+bug/938669/comments/4 upstart], [https://bugzilla.redhat.com/show_bug.cgi?id=754285#c1 systemd] and [https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=302079 start-stop-daemon]

===== Ulimit =====
From '''man bash'''
<blockquote>
ulimit [-HSTabcdefilmnpqrstuvx [limit]]
Provides control over the resources available to the shell and to processes started by it, on systems that allow such control.
</blockquote>
This is what we use in our System V compatible init scripts to increase resources for the open-xchange process across multiple distros. Currently only ''the maximum number of processes'' and ''the maximum number of open file descriptors available to a single user'' are increased via ulimit. The values are specified in '''/opt/open-xchange/ox-scriptconf.sh'''

==== Systemd ====

===== Control Groups=====

Control groups should only affect the OX middleware if you create/manage them yourself of if you are using a ''modern'' distribution that already uses systemd as init.

Citing from the kernel cgroup [https://www.kernel.org/doc/Documentation/cgroup-v2.txt documentation]:

<blockquote>
1-2. What is cgroup?

cgroup is a mechanism to organize processes hierarchically and
distribute system resources along the hierarchy in a controlled and
configurable manner.

cgroup is largely composed of two parts - the core and controllers.
cgroup core is primarily responsible for hierarchically organizing
processes. A cgroup controller is usually responsible for
distributing a specific type of system resource along the hierarchy
although there are utility controllers which serve purposes other than
resource distribution.

cgroups form a tree structure and every process in the system belongs
to one and only one cgroup. All threads of a process belong to the
same cgroup. On creation, all processes are put in the cgroup that
the parent process belongs to at the time. A process can be migrated
to another cgroup. Migration of a process doesn't affect already
existing descendant processes.

Following certain structural constraints, controllers may be enabled or
disabled selectively on a cgroup. All controller behaviors are
hierarchical - if a controller is enabled on a cgroup, it affects all
processes which belong to the cgroups consisting the inclusive
sub-hierarchy of the cgroup. When a controller is enabled on a nested
cgroup, it always restricts the resource distribution further. The
restrictions set closer to the root in the hierarchy can not be
overridden from further away.
</blockquote>

So processes are organized into a tree structure of control groups and controllers are responsible for the distribution of resources. So what kind of controllers exist?

<blockquote>
5. Controllers

5-1. CPU

The "cpu" controllers regulates distribution of CPU cycles. This
controller implements weight and absolute bandwidth limit models for
normal scheduling policy and absolute bandwidth allocation model for
realtime scheduling policy.

5-2. Memory

The "memory" controller regulates distribution of memory.
...
While not completely water-tight, all major memory usages by a given
cgroup are tracked so that the total memory consumption can be
accounted and controlled to a reasonable extent.

5-3. IO

The "io" controller regulates the distribution of IO resources. This
controller implements both weight based and absolute bandwidth or IOPS
limit distribution; however, weight based distribution is available
only if cfq-iosched is in use and neither scheme is available for
blk-mq devices.
</blockquote>

The open-xchange service is simply put into the default ''system.slice'' without applying further limits.

singlenode$ systemd-cgls
├─1 /sbin/init
├─system.slice
│ ├─avahi-daemon.service
│ │ ├─501 avahi-daemon: running [singlenode]
│ │ └─514 avahi-daemon: chroot helper
│ ├─console-kit-daemon.service
│ │ └─16164 /usr/sbin/console-kit-daemon --no-daemon
│ ├─dbus.service
│ │ └─508 /usr/bin/dbus-daemon --system --address=systemd: --nofork --nopidfile --systemd-activation
│ ├─munin-node.service
│ │ └─4290 /usr/bin/perl -wT /usr/sbin/munin-node
│ ├─open-xchange.service
│ │ └─6037 /usr/bin/java -Dsun.net.inetaddr.ttl=3600 -Dnetworkaddress.cache.ttl=3600 -Dnetworkaddress.cache.negative.ttl=10 ...

To check all the details use

singlenode:~ # systemctl show system.slice

==== Limits besides control groups ====
Besides control groups systemd allows you to apply other limits to the execution environment of your service. Here we can apply the limits that would normally be applied via ''limits.conf'' or ''ulimit''. Systemd uses [http://man7.org/linux/man-pages/man2/setrlimit.2.html setrlimit] for this. The options that we set by default are:

* LimitNOFILE
* LimitNPROC

which you have already seen when we took a look at the [[#Service_definiton|service definition]] and [[#Drop-in_config|drop-in configs]]

== Verify limits ==

=== System V ===

Get the proper pid of the java process started for the middleware by either checking '''ps faux'''
root 4103 0.0 0.0 66528 1868 pts/0 S 22:05 0:00 su -s /bin/bash open-xchange -c /opt/open-xchange/sbin/open-xchange
494 4109 9.5 19.8 2224644 381312 ? Ssl 22:05 1:12 \_ /usr/bin/java -Dsun.net.inetaddr.ttl=3600 -Dnetworkaddress.cache.ttl=3600

or programmatically
singlenode:~ # pid=$(pgrep -P $(</var/run/open-xchange.pid) java)

and check the limits applied for that process
singlenode:~ # cat /proc/$pid/limits
Limit Soft Limit Hard Limit Units
Max cpu time unlimited unlimited seconds
Max file size unlimited unlimited bytes
Max data size unlimited unlimited bytes
Max stack size 8388608 unlimited bytes
Max core file size 0 unlimited bytes
Max resident set unlimited unlimited bytes
Max processes 65536 65536 processes
Max open files 65536 65536 files
Max locked memory 65536 65536 bytes
Max address space unlimited unlimited bytes
Max file locks unlimited unlimited locks
Max pending signals 24254 24254 signals
Max msgqueue size 819200 819200 bytes
Max nice priority 0 0
Max realtime priority 0 0
Max realtime timeout unlimited unlimited us

=== Systemd ===
singlenode:~ # systemctl show open-xchange | grep Limit
StartLimitInterval=10000000
StartLimitBurst=5
StartLimitAction=none
MemoryLimit=18446744073709551615
LimitCPU=18446744073709551615
LimitFSIZE=18446744073709551615
LimitDATA=18446744073709551615
LimitSTACK=18446744073709551615
LimitCORE=18446744073709551615
LimitRSS=18446744073709551615
LimitNOFILE=65536
LimitAS=18446744073709551615
LimitNPROC=65536
LimitMEMLOCK=65536
LimitLOCKS=18446744073709551615
LimitSIGPENDING=19827
LimitMSGQUEUE=819200
LimitNICE=0
LimitRTPRIO=0
LimitRTTIME=18446744073709551615

== Evaluate JVM commandline options ==

Before starting Java and the OSGi framework with all the Open-Xchange middleware bundles we have to evaluate several important JVM commandline options that influence the performance and stability of the Open-Xchange middleware stack. These commandline options are specified in '''/opt/open-xchange/etc/ox-scriptconf.sh'''

=== Format < 7.8.4 ===
For version up to 7.8.3 the JVM configuration options were all specified via a single property '''JAVA_XTRAOPTS''' which resulted in a rather long configuration string that was hard to read and maintain.

JAVA_XTRAOPTS="-Dsun.net.inetaddr.ttl=3600 -Dnetworkaddress.cache.ttl=3600 -Dnetworkaddress.cache.negative.ttl=10 \
-Dlogback.threadlocal.put.duplicate=false -server -Djava.awt.headless=true -Xmx512M -XX:MaxPermSize=256m -XX:+UseConcMarkSweepGC \
-XX:+UseParNewGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:NewRatio=3 -XX:+UseTLAB \
-Dosgi.compatibility.bootdelegation=false -XX:-OmitStackTraceInFastThrow"

During the startup of the open-xchange service (step 11 of diagram Service startup using systemd and step 10 of diagram Service startup using upstart) this JAVA_XTRAOPTS string goes through a basic validity check and is then passed to the JVM.

=== Format >= 7.8.4 ===
Beginning from version 7.8.4 the options are split up and sorted into different categories like:

JAVA_OPTS_GC="-XX:+UseConcMarkSweepGC -XX:+UseParNewGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:NewRatio=3 -XX:+DisableExplicitGC"
JAVA_OPTS_LOG="-Dlogback.threadlocal.put.duplicate=false -XX:-OmitStackTraceInFastThrow"
JAVA_OPTS_MEM="-XX:MaxHeapSize=512M -XX:MaxPermSize=256M -XX:+UseTLAB"
JAVA_OPTS_NET="-Dsun.net.inetaddr.ttl=3600 -Dnetworkaddress.cache.ttl=3600 -Dnetworkaddress.cache.negative.ttl=10"
JAVA_OPTS_OSGI="-Dosgi.compatibility.bootdelegation=false"
JAVA_OPTS_SERVER="-server -Djava.awt.headless=true"

JAVA_OPTS_OTHER=""

#JAVA_OPTS_DEBUG="-XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/mnt/heapdump -Xloggc:/var/log/open-xchange/gc.log -verbose:gc -XX:+PrintGCDateStamps \
-XX:+PrintHeapAtGC -XX:+PrintGCApplicationStoppedTime -XX:+PrintTenuringDistribution"

which is much easier to read an maintain. Furthermore we added a useful '''JAVA_OPTS_DEBUG''' configuration parameter that comes preconfigured but unused/commented so that customers can quickly enable the most commonly used debugging options in case they have to provide detailed system information to Open-Xchange.

During the startup of the open-xchange service (step 11 of diagram ''Service startup using systemd'' and step 10 of diagram ''Service startup using upstart'') all uncommented JAVA_OPTS_* categories will then be evaluated and combined again into a single string of commandline options that can then be passed to the JVM.

=== Upgrade procedure ===
During the upgrade '''7.8.3-revX -> 7.8.4-revY''' existing JAVA_XTRAOPTS will automatically be split up and migrated to the new categories approach. A backup of the file (ox-scriptconf.sh.timestamp) will be created and admins are urged to check for proper migration before deleting the backup file like it's already done with *.dpkg-dist or *.rpmnew files on Debian or RHEL/SLES distributions.

== Encoding detection during JVM startup ==

=== Background Infos ===
All of en_US.UTF-8, .utf8, .utf-8, .UTF_8 will result in a valid UTF-8 charmap

==== JVM startup ====

The default file.encoding detection is done at JVM startup based upon LC_CTYPE which makes sense as it's documented as:

CODESET (LC_CTYPE)
Return a string with the name of the character encoding used in the selected locale, such as "UTF-8", "ISO-8859-1",
or "ANSI_X3.4-1968" (better known as US-ASCII). This is the same string that you get with "locale charmap".
For a list of character encoding names, try "locale -m", cf. locale(1).

Furthermore this influences the defaultCharset selection of the JVM.

/**
* Returns the default charset of this Java virtual machine.
*
* The default charset is determined during virtual-machine startup and
* typically depends upon the locale and charset of the underlying
* operating system.
*
* @return A charset object for the default charset
*
* @since 1.5
*/
public static Charset defaultCharset() {
if (defaultCharset == null) {
synchronized (Charset.class) {
String csn = AccessController.doPrivileged(
new GetPropertyAction("file.encoding"));
Charset cs = lookup(csn);
if (cs != null)
defaultCharset = cs;
else
defaultCharset = forName("UTF-8");
}
}
return defaultCharset;
}

Although we try to specify valid unicode encodings for every IO operations the Open-Xchange middleware does some parts of the JVM don't allow to specify the encoding to use and falls back to the detected default encoding. That's why admins have to make sure to provide a proper unicode environment for the middleware service during every start or use documented workarounds provided by Open-Xchange (e.g. )

=== Query which file encoding was detected by the JVM during startup ===
[[File:visualvm-file.encoding.png|thumb|Querying the JVM via visualvm|right|300px]]

The file encoding that was detected by the JVM during startup can be queried via JMX. You can use graphical management tools like [https://visualvm.github.io/ visualvm], query our [[Jolokia]] JMX bridge via http or just use the commandline like

/opt/open-xchange/sbin/showruntimestats -r | sed -ne 's/.*${key=file.encoding, value=[^}]*}$.*/\1/p'
{key=file.encoding, value=UTF-8}

 

=== Locale passing from client to server via SSH ===

The following setting of sshd allows clients to specify which locale to use on the server and might lead to some confusion or unexpected behaviour when checking the system locale or even to the usage of broken locales during middleware startup if your client sends misconfigured or missing locale configuration to the server.

grep AcceptEnv /etc/ssh/sshd_config
AcceptEnv LANG LC_*

For example when i connect to the machine my local german locale is sent to the server

[vagrant@showcase-community-centos-7 ~]$ locale -v
LANG=de_DE.UTF-8
LC_CTYPE="de_DE.UTF-8"
LC_NUMERIC=de_DE.UTF-8
LC_TIME=de_DE.UTF-8
LC_COLLATE="de_DE.UTF-8"
LC_MONETARY=de_DE.UTF-8
LC_MESSAGES="de_DE.UTF-8"
LC_PAPER=de_DE.UTF-8
LC_NAME=de_DE.UTF-8
LC_ADDRESS=de_DE.UTF-8
LC_TELEPHONE=de_DE.UTF-8
LC_MEASUREMENT=de_DE.UTF-8
LC_IDENTIFICATION=de_DE.UTF-8
LC_ALL=

But after specifying that i want a new login shell the default locale configured for that user/system will be used instead!

[root@showcase-community-centos-7 vagrant]# su -s /bin/bash open-xchange -l
-bash-4.2$ locale -v
LANG=en_US.UTF-8
LC_CTYPE="en_US.UTF-8"
LC_NUMERIC="en_US.UTF-8"
LC_TIME="en_US.UTF-8"
LC_COLLATE="en_US.UTF-8"
LC_MONETARY="en_US.UTF-8"
LC_MESSAGES="en_US.UTF-8"
LC_PAPER="en_US.UTF-8"
LC_NAME="en_US.UTF-8"
LC_ADDRESS="en_US.UTF-8"
LC_TELEPHONE="en_US.UTF-8"
LC_MEASUREMENT="en_US.UTF-8"
LC_IDENTIFICATION="en_US.UTF-8"
LC_ALL=

Add details from https://bugs.open-xchange.com/show_bug.cgi?id=51074

== Warnings during early start ==

Early warnings before the start of the JVM and related logging libraries like logback and logstash forwarding are redirected to '''/var/log/open-xchange/open-xchange-console.log'''.

=== Limits ===
If the open-xchange startup script isn't able to set the proper limits on CentOS 6 due to e.g. hard limits being enforced via pam_limits you'll see the following warning in the console log during startup

singlenode open-xchange # cat open-xchange-console.log
/opt/open-xchange/sbin/open-xchange: line 115: ulimit: max user processes: cannot modify limit: Operation not permitted

=== Locale ===
If the open-xchange startup script isn't able to detect a unicode locale you'll see a warning like the following (the current charmap might differ)

WARNING: Couldn't detect proper unicode charmap in locale category LC_CTYPE.
WARNING: Current charmap: "ANSI_X3.4-1968" might cause encoding problems.

== Summary: Open-Xchange middleware on specific distros ==
The support for the mentioned mechanism of resource control differ depending on the distribution and the init system in use.

=== Debian 7 ===
;Init
: System V style
;OX Configurable Limits/Defaults
: nofile, nproc

The mentioned limits can be configured via '''/opt/open-xchange/etc/ox-scriptconf.sh'''. The limits are applied via ulimit in the service's init script. The open-xchange service is finally started via start-stop-daemon which doesn't doesn't consider '''/etc/security/limits.*'''

=== RHEL 6 / CentOS 6 ===
;Init
: Upstart, System V compatible
;OX Configurable Limits/Defaults
: nofile, nproc

The mentioned limits can be configured via '''/opt/open-xchange/etc/ox-scriptconf.sh'''. The limits are applied via ulimit in the service's init script. Furthermore as the open-xchange service is finally started via '''su ... open-xchange''' on this distro a user session is opened via su/pam and the default CentOS pam config reads the '''/etc/security/limits.*''' configuration by loading the pam stack like:

: '''/etc/pam.d/su'''
:: '''-> /etc/pam.d/system-auth'''
::: '''-> pam_limits.so'''

If NPROC isn't configured for the open-xchange-server it's restricted to '''1024''' globally by default to prevent accidental fork bombs, see '''/etc/security/limits.d/90-nproc.conf''' which can result in severe problems modern multithreaded applications.

=== RHEL 7 / CentOS 7 / Debian 8 / SLE 12 ===
;Init
: Systemd
;OX Configurable Limits/Defaults
: nofile, nproc

For systemd the default limits are configured directly in the service's unit file that is shipped by OX and located at '''/usr/lib/systemd/system/open-xchange.service'''. The drop-in config to override or extend the default unit file is located at '''/etc/systemd/system/open-xchange.service.d/limits.conf'''. [https://www.freedesktop.org/software/systemd/man/systemd.exec.html Systemd.exec] shows a whole lot of options that can be used by admins to adapt the default service to their specific needs.

Sandbox:Middleware Startup/sandbox

2017-04-24T20:06:45Z

Marens:

== Systemd ==

==== Service definition ====
The default service definition as shipped by Open-Xchange is rather concise and easy to read. You can check by looking at the service file that is installed to either /usr/lib/systemd or /lib/systemd depending on your distribution of choice.

singlenode:~ # cat /usr/lib/systemd/system/open-xchange.service
[Unit]
After=remote-fs.target
After=time-sync.target ypbind.service sendmail.service cyrus.service

[Service]
User=open-xchange
PermissionsStartOnly=true
TimeoutStartSec=0
ExecStartPre=/opt/open-xchange/sbin/triggerupdatethemes -u
ExecStart=/opt/open-xchange/sbin/open-xchange
ExecStop=/opt/open-xchange/sbin/shutdown -w
ExecReload=/opt/open-xchange/sbin/triggerreloadconfiguration -d
KillMode=process
LimitNOFILE=65536
LimitNPROC=65536

[Install]
WantedBy=multi-user.target

==== Drop-in configs ====
Drop in configs allow administrators to easily adapt and override default service unit files. So if you want to change the default limits or add additional limits have a look at our default example drop-in config which is located at /etc/systemd/system/open-xchange.service.d/limits.conf

singlenode:~ # cat /etc/systemd/system/open-xchange.service.d/limits.conf
# Override and add options in this file
# See systemd.exec(5) for other limits

[Service]
#LimitNPROC=65536
#LimitNOFILE=65536

The systemd.exec documentation shows a whole lot of options that can be used by admins to adapt the default service to their specific needs.

 
[[File:mw_start_sysd.png|thumb|Service startup using systemd|left|1000px]]
 

== System V / Upstart ==

==== Service definition ====

The default service definition as shipped by Open-Xchange can be found at /etc/init.d/open-xchange and will differ slightly depending on the distro in use.

 
[[File:mw_start_sysv.png|thumb|Service startup using upstart|left|1000px]]
 
== Apply Resource limits ==

As you can see in the diagrams above one of the differences is that:

* systemd reads all the infos wrt. resource limitations from the original service file and then applies possible further restrictions from the drop-in configs (steps 2 and 8 in the diagram ''Service startup using systemd'') and limits configured in /opt/open-xchange/etc/ox-scriptconf.sh aren't considered anylonger
* the older System V style init has to source /opt/open-xchange/etc/ox-scriptconf.sh to be able to read the configs for the maximum number of processes (NPROC) and files(NRFILES) that the middleware shall be allowed to create/open (steps 3,4 and 15 in the diagram ''Service startup using upstart'').

=== Background Infos ===

Several ways exist to restrict resources on a linux system from a global level down to user/groups or even shells and the processes started by them.

==== Sysctl ====
Sysctl is used to modify kernel parameters at runtime. E.g. to set the maximum number of files

$ sysctl -w fs.file-max=100000

To permanently set them append to the main configuration file and reload the settings

$ echo fs.file-max=100000 >> /etc/sysctl.conf
$ sysctl -p

More infos can be found via '''man sysctl'''

==== Limits.conf ====

Allows to restrict resources an a global, group or user level. E.g:

$ cat /etc/security/limits.d/90-nproc.conf
# Default limit for number of user's processes to prevent
# accidental fork bombs.
# See rhbz #432903 for reasoning.

* soft nproc 1024

From '''man limits.conf''':
<blockquote>
''Also, please note that all limit settings are set per login. They are not global, nor are they permanent; existing only for the duration of the session.''
</blockquote>

The limits per login are applied via the ''pam'' stack. See '''man pam''' and '''man pam_limits''' for more details. As those limits are bound to sessions they don't affect most daemons started by our supported init systems or init utils. Most state that they are ignored by design, see [https://bugs.launchpad.net/ubuntu/+source/upstart/+bug/938669/comments/4 upstart], [https://bugzilla.redhat.com/show_bug.cgi?id=754285#c1 systemd] and [https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=302079 start-stop-daemon]

===== Ulimit =====
From '''man bash'''
<blockquote>
ulimit [-HSTabcdefilmnpqrstuvx [limit]]
Provides control over the resources available to the shell and to processes started by it, on systems that allow such control.
</blockquote>
This is what we use in our System V compatible init scripts to increase resources for the open-xchange process across multiple distros. Currently only ''the maximum number of processes'' and ''the maximum number of open file descriptors available to a single user'' are increased via ulimit. The values are specified in '''/opt/open-xchange/ox-scriptconf.sh'''

==== Systemd ====

===== Control Groups=====

Control groups should only affect the OX middleware if you create/manage them yourself of if you are using a ''modern'' distribution that already uses systemd as init.

Citing from the kernel cgroup [https://www.kernel.org/doc/Documentation/cgroup-v2.txt documentation]:

<blockquote>
1-2. What is cgroup?

cgroup is a mechanism to organize processes hierarchically and
distribute system resources along the hierarchy in a controlled and
configurable manner.

cgroup is largely composed of two parts - the core and controllers.
cgroup core is primarily responsible for hierarchically organizing
processes. A cgroup controller is usually responsible for
distributing a specific type of system resource along the hierarchy
although there are utility controllers which serve purposes other than
resource distribution.

cgroups form a tree structure and every process in the system belongs
to one and only one cgroup. All threads of a process belong to the
same cgroup. On creation, all processes are put in the cgroup that
the parent process belongs to at the time. A process can be migrated
to another cgroup. Migration of a process doesn't affect already
existing descendant processes.

Following certain structural constraints, controllers may be enabled or
disabled selectively on a cgroup. All controller behaviors are
hierarchical - if a controller is enabled on a cgroup, it affects all
processes which belong to the cgroups consisting the inclusive
sub-hierarchy of the cgroup. When a controller is enabled on a nested
cgroup, it always restricts the resource distribution further. The
restrictions set closer to the root in the hierarchy can not be
overridden from further away.
</blockquote>

So processes are organized into a tree structure of control groups and controllers are responsible for the distribution of resources. So what kind of controllers exist?

<blockquote>
5. Controllers

5-1. CPU

The "cpu" controllers regulates distribution of CPU cycles. This
controller implements weight and absolute bandwidth limit models for
normal scheduling policy and absolute bandwidth allocation model for
realtime scheduling policy.

5-2. Memory

The "memory" controller regulates distribution of memory.
...
While not completely water-tight, all major memory usages by a given
cgroup are tracked so that the total memory consumption can be
accounted and controlled to a reasonable extent.

5-3. IO

The "io" controller regulates the distribution of IO resources. This
controller implements both weight based and absolute bandwidth or IOPS
limit distribution; however, weight based distribution is available
only if cfq-iosched is in use and neither scheme is available for
blk-mq devices.
</blockquote>

The open-xchange service is simply put into the default ''system.slice'' without applying further limits.

singlenode$ systemd-cgls
├─1 /sbin/init
├─system.slice
│ ├─avahi-daemon.service
│ │ ├─501 avahi-daemon: running [singlenode]
│ │ └─514 avahi-daemon: chroot helper
│ ├─console-kit-daemon.service
│ │ └─16164 /usr/sbin/console-kit-daemon --no-daemon
│ ├─dbus.service
│ │ └─508 /usr/bin/dbus-daemon --system --address=systemd: --nofork --nopidfile --systemd-activation
│ ├─munin-node.service
│ │ └─4290 /usr/bin/perl -wT /usr/sbin/munin-node
│ ├─open-xchange.service
│ │ └─6037 /usr/bin/java -Dsun.net.inetaddr.ttl=3600 -Dnetworkaddress.cache.ttl=3600 -Dnetworkaddress.cache.negative.ttl=10 ...

To check all the details use

singlenode:~ # systemctl show system.slice

==== Limits besides control groups ====
Besides control groups systemd allows you to apply other limits to the execution environment of your service. Here we can apply the limits that would normally be applied via ''limits.conf'' or ''ulimit''. Systemd uses [http://man7.org/linux/man-pages/man2/setrlimit.2.html setrlimit] for this. The options that we set by default are:

* LimitNOFILE
* LimitNPROC

which you have already seen when we took a look at the [[#Service_definiton|service definition]] and [[#Drop-in_config|drop-in configs]]

== Verify limits ==

=== System V ===

Get the proper pid of the java process started for the middleware by either checking '''ps faux'''
root 4103 0.0 0.0 66528 1868 pts/0 S 22:05 0:00 su -s /bin/bash open-xchange -c /opt/open-xchange/sbin/open-xchange
494 4109 9.5 19.8 2224644 381312 ? Ssl 22:05 1:12 \_ /usr/bin/java -Dsun.net.inetaddr.ttl=3600 -Dnetworkaddress.cache.ttl=3600

or programmatically
singlenode:~ # pid=$(pgrep -P $(</var/run/open-xchange.pid) java)

and check the limits applied for that process
singlenode:~ # cat /proc/$pid/limits
Limit Soft Limit Hard Limit Units
Max cpu time unlimited unlimited seconds
Max file size unlimited unlimited bytes
Max data size unlimited unlimited bytes
Max stack size 8388608 unlimited bytes
Max core file size 0 unlimited bytes
Max resident set unlimited unlimited bytes
Max processes 65536 65536 processes
Max open files 65536 65536 files
Max locked memory 65536 65536 bytes
Max address space unlimited unlimited bytes
Max file locks unlimited unlimited locks
Max pending signals 24254 24254 signals
Max msgqueue size 819200 819200 bytes
Max nice priority 0 0
Max realtime priority 0 0
Max realtime timeout unlimited unlimited us

=== Systemd ===
singlenode:~ # systemctl show open-xchange | grep Limit
StartLimitInterval=10000000
StartLimitBurst=5
StartLimitAction=none
MemoryLimit=18446744073709551615
LimitCPU=18446744073709551615
LimitFSIZE=18446744073709551615
LimitDATA=18446744073709551615
LimitSTACK=18446744073709551615
LimitCORE=18446744073709551615
LimitRSS=18446744073709551615
LimitNOFILE=65536
LimitAS=18446744073709551615
LimitNPROC=65536
LimitMEMLOCK=65536
LimitLOCKS=18446744073709551615
LimitSIGPENDING=19827
LimitMSGQUEUE=819200
LimitNICE=0
LimitRTPRIO=0
LimitRTTIME=18446744073709551615

== Evaluate JVM commandline options ==

Before starting Java and the OSGi framework with all the Open-Xchange middleware bundles we have to evaluate several important JVM commandline options that influence the performance and stability of the Open-Xchange middleware stack. These commandline options are specified in '''/opt/open-xchange/etc/ox-scriptconf.sh'''

=== Format < 7.8.4 ===
For version up to 7.8.3 the JVM configuration options were all specified via a single property '''JAVA_XTRAOPTS''' which resulted in a rather long configuration string that was hard to read and maintain.

JAVA_XTRAOPTS="-Dsun.net.inetaddr.ttl=3600 -Dnetworkaddress.cache.ttl=3600 -Dnetworkaddress.cache.negative.ttl=10 \
-Dlogback.threadlocal.put.duplicate=false -server -Djava.awt.headless=true -Xmx512M -XX:MaxPermSize=256m -XX:+UseConcMarkSweepGC \
-XX:+UseParNewGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:NewRatio=3 -XX:+UseTLAB \
-Dosgi.compatibility.bootdelegation=false -XX:-OmitStackTraceInFastThrow"

During the startup of the open-xchange service (step 11 of diagram Service startup using systemd and step 10 of diagram Service startup using upstart) this JAVA_XTRAOPTS string goes through a basic validity check and is then passed to the JVM.

=== Format >= 7.8.4 ===
Beginning from version 7.8.4 the options are split up and sorted into different categories like:

JAVA_OPTS_GC="-XX:+UseConcMarkSweepGC -XX:+UseParNewGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:NewRatio=3 -XX:+DisableExplicitGC"
JAVA_OPTS_LOG="-Dlogback.threadlocal.put.duplicate=false -XX:-OmitStackTraceInFastThrow"
JAVA_OPTS_MEM="-XX:MaxHeapSize=512M -XX:MaxPermSize=256M -XX:+UseTLAB"
JAVA_OPTS_NET="-Dsun.net.inetaddr.ttl=3600 -Dnetworkaddress.cache.ttl=3600 -Dnetworkaddress.cache.negative.ttl=10"
JAVA_OPTS_OSGI="-Dosgi.compatibility.bootdelegation=false"
JAVA_OPTS_SERVER="-server -Djava.awt.headless=true"

JAVA_OPTS_OTHER=""

#JAVA_OPTS_DEBUG="-XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/mnt/heapdump -Xloggc:/var/log/open-xchange/gc.log -verbose:gc -XX:+PrintGCDateStamps \
-XX:+PrintHeapAtGC -XX:+PrintGCApplicationStoppedTime -XX:+PrintTenuringDistribution"

which is much easier to read an maintain. Furthermore we added a useful '''JAVA_OPTS_DEBUG''' configuration parameter that comes preconfigured but unused/commented so that customers can quickly enable the most commonly used debugging options in case they have to provide detailed system information to Open-Xchange.

During the startup of the open-xchange service (step 11 of diagram ''Service startup using systemd'' and step 10 of diagram ''Service startup using upstart'') all uncommented JAVA_OPTS_* categories will then be evaluated and combined again into a single string of commandline options that can then be passed to the JVM.

=== Upgrade procedure ===
During the upgrade '''7.8.3-revX -> 7.8.4-revY''' existing JAVA_XTRAOPTS will automatically be split up and migrated to the new categories approach. A backup of the file (ox-scriptconf.sh.timestamp) will be created and admins are urged to check for proper migration before deleting the backup file like it's already done with *.dpkg-dist or *.rpmnew files on Debian or RHEL/SLES distributions.

== Encoding detection during JVM startup ==

=== Background Infos ===
All of en_US.UTF-8, .utf8, .utf-8, .UTF_8 will result in a valid UTF-8 charmap

==== JVM startup ====

The default file.encoding detection is done at JVM startup based upon LC_CTYPE which makes sense as it's documented as:

CODESET (LC_CTYPE)
Return a string with the name of the character encoding used in the selected locale, such as "UTF-8", "ISO-8859-1",
or "ANSI_X3.4-1968" (better known as US-ASCII). This is the same string that you get with "locale charmap".
For a list of character encoding names, try "locale -m", cf. locale(1).

Furthermore this influences the defaultCharset selection of the JVM.

/**
* Returns the default charset of this Java virtual machine.
*
* The default charset is determined during virtual-machine startup and
* typically depends upon the locale and charset of the underlying
* operating system.
*
* @return A charset object for the default charset
*
* @since 1.5
*/
public static Charset defaultCharset() {
if (defaultCharset == null) {
synchronized (Charset.class) {
String csn = AccessController.doPrivileged(
new GetPropertyAction("file.encoding"));
Charset cs = lookup(csn);
if (cs != null)
defaultCharset = cs;
else
defaultCharset = forName("UTF-8");
}
}
return defaultCharset;
}

Although we try to specify valid unicode encodings for every IO operations the Open-Xchange middleware does some parts of the JVM don't allow to specify the encoding to use and falls back to the detected default encoding. That's why admins have to make sure to provide a proper unicode environment for the middleware service during every start or use documented workarounds provided by Open-Xchange (e.g. )

=== Query which file encoding was detected by the JVM during startup ===
[[File:visualvm-file.encoding.png|thumb|Querying the JVM via visualvm|right|300px]]

The file encoding that was detected by the JVM during startup can be queried via JMX. You can use graphical management tools like [https://visualvm.github.io/ visualvm], query our [[Jolokia]] JMX bridge via http or just use the commandline like

/opt/open-xchange/sbin/showruntimestats -r | sed -ne 's/.*${key=file.encoding, value=[^}]*}$.*/\1/p'
{key=file.encoding, value=UTF-8}

=== Locale passing from client to server via SSH ===

The following setting of sshd allows clients to specify which locale to use on the server and might lead to some confusion or unexpected behaviour when checking the system locale or even to the usage of broken locales during middleware startup if your client sends misconfigured or missing locale configuration to the server.

grep AcceptEnv /etc/ssh/sshd_config
AcceptEnv LANG LC_*

For example when i connect to the machine my local german locale is sent to the server

[vagrant@showcase-community-centos-7 ~]$ locale -v
LANG=de_DE.UTF-8
LC_CTYPE="de_DE.UTF-8"
LC_NUMERIC=de_DE.UTF-8
LC_TIME=de_DE.UTF-8
LC_COLLATE="de_DE.UTF-8"
LC_MONETARY=de_DE.UTF-8
LC_MESSAGES="de_DE.UTF-8"
LC_PAPER=de_DE.UTF-8
LC_NAME=de_DE.UTF-8
LC_ADDRESS=de_DE.UTF-8
LC_TELEPHONE=de_DE.UTF-8
LC_MEASUREMENT=de_DE.UTF-8
LC_IDENTIFICATION=de_DE.UTF-8
LC_ALL=

But after specifying that i want a new login shell the default locale configured for that user/system will be used instead!

[root@showcase-community-centos-7 vagrant]# su -s /bin/bash open-xchange -l
-bash-4.2$ locale -v
LANG=en_US.UTF-8
LC_CTYPE="en_US.UTF-8"
LC_NUMERIC="en_US.UTF-8"
LC_TIME="en_US.UTF-8"
LC_COLLATE="en_US.UTF-8"
LC_MONETARY="en_US.UTF-8"
LC_MESSAGES="en_US.UTF-8"
LC_PAPER="en_US.UTF-8"
LC_NAME="en_US.UTF-8"
LC_ADDRESS="en_US.UTF-8"
LC_TELEPHONE="en_US.UTF-8"
LC_MEASUREMENT="en_US.UTF-8"
LC_IDENTIFICATION="en_US.UTF-8"
LC_ALL=

Add details from https://bugs.open-xchange.com/show_bug.cgi?id=51074

== Warnings during early start ==

Early warnings before the start of the JVM and related logging libraries like logback and logstash forwarding are redirected to '''/var/log/open-xchange/open-xchange-console.log'''.

=== Limits ===
If the open-xchange startup script isn't able to set the proper limits on CentOS 6 due to e.g. hard limits being enforced via pam_limits you'll see the following warning in the console log during startup

singlenode open-xchange # cat open-xchange-console.log
/opt/open-xchange/sbin/open-xchange: line 115: ulimit: max user processes: cannot modify limit: Operation not permitted

=== Locale ===
If the open-xchange startup script isn't able to detect a unicode locale you'll see a warning like the following (the current charmap might differ)

WARNING: Couldn't detect proper unicode charmap in locale category LC_CTYPE.
WARNING: Current charmap: "ANSI_X3.4-1968" might cause encoding problems.

== Summary: Open-Xchange middleware on specific distros ==
The support for the mentioned mechanism of resource control differ depending on the distribution and the init system in use.

=== Debian 7 ===
;Init
: System V style
;OX Configurable Limits/Defaults
: nofile, nproc

The mentioned limits can be configured via '''/opt/open-xchange/etc/ox-scriptconf.sh'''. The limits are applied via ulimit in the service's init script. The open-xchange service is finally started via start-stop-daemon which doesn't doesn't consider '''/etc/security/limits.*'''

=== RHEL 6 / CentOS 6 ===
;Init
: Upstart, System V compatible
;OX Configurable Limits/Defaults
: nofile, nproc

The mentioned limits can be configured via '''/opt/open-xchange/etc/ox-scriptconf.sh'''. The limits are applied via ulimit in the service's init script. Furthermore as the open-xchange service is finally started via '''su ... open-xchange''' on this distro a user session is opened via su/pam and the default CentOS pam config reads the '''/etc/security/limits.*''' configuration by loading the pam stack like:

: '''/etc/pam.d/su'''
:: '''-> /etc/pam.d/system-auth'''
::: '''-> pam_limits.so'''

If NPROC isn't configured for the open-xchange-server it's restricted to '''1024''' globally by default to prevent accidental fork bombs, see '''/etc/security/limits.d/90-nproc.conf''' which can result in severe problems modern multithreaded applications.

=== RHEL 7 / CentOS 7 / Debian 8 / SLE 12 ===
;Init
: Systemd
;OX Configurable Limits/Defaults
: nofile, nproc

For systemd the default limits are configured directly in the service's unit file that is shipped by OX and located at '''/usr/lib/systemd/system/open-xchange.service'''. The drop-in config to override or extend the default unit file is located at '''/etc/systemd/system/open-xchange.service.d/limits.conf'''. [https://www.freedesktop.org/software/systemd/man/systemd.exec.html Systemd.exec] shows a whole lot of options that can be used by admins to adapt the default service to their specific needs.

File:Visualvm-file.encoding.png

2017-04-24T20:01:55Z

Marens:

Sandbox:Middleware Startup/sandbox

2017-04-24T20:01:02Z

Marens: /* Query which file encoding was detected by the JVM during startup */

== Systemd ==

==== Service definition ====
The default service definition as shipped by Open-Xchange is rather concise and easy to read. You can check by looking at the service file that is installed to either /usr/lib/systemd or /lib/systemd depending on your distribution of choice.

singlenode:~ # cat /usr/lib/systemd/system/open-xchange.service
[Unit]
After=remote-fs.target
After=time-sync.target ypbind.service sendmail.service cyrus.service

[Service]
User=open-xchange
PermissionsStartOnly=true
TimeoutStartSec=0
ExecStartPre=/opt/open-xchange/sbin/triggerupdatethemes -u
ExecStart=/opt/open-xchange/sbin/open-xchange
ExecStop=/opt/open-xchange/sbin/shutdown -w
ExecReload=/opt/open-xchange/sbin/triggerreloadconfiguration -d
KillMode=process
LimitNOFILE=65536
LimitNPROC=65536

[Install]
WantedBy=multi-user.target

==== Drop-in configs ====
Drop in configs allow administrators to easily adapt and override default service unit files. So if you want to change the default limits or add additional limits have a look at our default example drop-in config which is located at /etc/systemd/system/open-xchange.service.d/limits.conf

singlenode:~ # cat /etc/systemd/system/open-xchange.service.d/limits.conf
# Override and add options in this file
# See systemd.exec(5) for other limits

[Service]
#LimitNPROC=65536
#LimitNOFILE=65536

The systemd.exec documentation shows a whole lot of options that can be used by admins to adapt the default service to their specific needs.

 
[[File:mw_start_sysd.png|thumb|Service startup using systemd|left|1000px]]
 

== System V / Upstart ==

==== Service definition ====

The default service definition as shipped by Open-Xchange can be found at /etc/init.d/open-xchange and will differ slightly depending on the distro in use.

 
[[File:mw_start_sysv.png|thumb|Service startup using upstart|left|1000px]]
 
== Apply Resource limits ==

As you can see in the diagrams above one of the differences is that:

* systemd reads all the infos wrt. resource limitations from the original service file and then applies possible further restrictions from the drop-in configs (steps 2 and 8 in the diagram ''Service startup using systemd'') and limits configured in /opt/open-xchange/etc/ox-scriptconf.sh aren't considered anylonger
* the older System V style init has to source /opt/open-xchange/etc/ox-scriptconf.sh to be able to read the configs for the maximum number of processes (NPROC) and files(NRFILES) that the middleware shall be allowed to create/open (steps 3,4 and 15 in the diagram ''Service startup using upstart'').

=== Background Infos ===

Several ways exist to restrict resources on a linux system from a global level down to user/groups or even shells and the processes started by them.

==== Sysctl ====
Sysctl is used to modify kernel parameters at runtime. E.g. to set the maximum number of files

$ sysctl -w fs.file-max=100000

To permanently set them append to the main configuration file and reload the settings

$ echo fs.file-max=100000 >> /etc/sysctl.conf
$ sysctl -p

More infos can be found via '''man sysctl'''

==== Limits.conf ====

Allows to restrict resources an a global, group or user level. E.g:

$ cat /etc/security/limits.d/90-nproc.conf
# Default limit for number of user's processes to prevent
# accidental fork bombs.
# See rhbz #432903 for reasoning.

* soft nproc 1024

From '''man limits.conf''':
<blockquote>
''Also, please note that all limit settings are set per login. They are not global, nor are they permanent; existing only for the duration of the session.''
</blockquote>

The limits per login are applied via the ''pam'' stack. See '''man pam''' and '''man pam_limits''' for more details. As those limits are bound to sessions they don't affect most daemons started by our supported init systems or init utils. Most state that they are ignored by design, see [https://bugs.launchpad.net/ubuntu/+source/upstart/+bug/938669/comments/4 upstart], [https://bugzilla.redhat.com/show_bug.cgi?id=754285#c1 systemd] and [https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=302079 start-stop-daemon]

===== Ulimit =====
From '''man bash'''
<blockquote>
ulimit [-HSTabcdefilmnpqrstuvx [limit]]
Provides control over the resources available to the shell and to processes started by it, on systems that allow such control.
</blockquote>
This is what we use in our System V compatible init scripts to increase resources for the open-xchange process across multiple distros. Currently only ''the maximum number of processes'' and ''the maximum number of open file descriptors available to a single user'' are increased via ulimit. The values are specified in '''/opt/open-xchange/ox-scriptconf.sh'''

==== Systemd ====

===== Control Groups=====

Control groups should only affect the OX middleware if you create/manage them yourself of if you are using a ''modern'' distribution that already uses systemd as init.

Citing from the kernel cgroup [https://www.kernel.org/doc/Documentation/cgroup-v2.txt documentation]:

<blockquote>
1-2. What is cgroup?

cgroup is a mechanism to organize processes hierarchically and
distribute system resources along the hierarchy in a controlled and
configurable manner.

cgroup is largely composed of two parts - the core and controllers.
cgroup core is primarily responsible for hierarchically organizing
processes. A cgroup controller is usually responsible for
distributing a specific type of system resource along the hierarchy
although there are utility controllers which serve purposes other than
resource distribution.

cgroups form a tree structure and every process in the system belongs
to one and only one cgroup. All threads of a process belong to the
same cgroup. On creation, all processes are put in the cgroup that
the parent process belongs to at the time. A process can be migrated
to another cgroup. Migration of a process doesn't affect already
existing descendant processes.

Following certain structural constraints, controllers may be enabled or
disabled selectively on a cgroup. All controller behaviors are
hierarchical - if a controller is enabled on a cgroup, it affects all
processes which belong to the cgroups consisting the inclusive
sub-hierarchy of the cgroup. When a controller is enabled on a nested
cgroup, it always restricts the resource distribution further. The
restrictions set closer to the root in the hierarchy can not be
overridden from further away.
</blockquote>

So processes are organized into a tree structure of control groups and controllers are responsible for the distribution of resources. So what kind of controllers exist?

<blockquote>
5. Controllers

5-1. CPU

The "cpu" controllers regulates distribution of CPU cycles. This
controller implements weight and absolute bandwidth limit models for
normal scheduling policy and absolute bandwidth allocation model for
realtime scheduling policy.

5-2. Memory

The "memory" controller regulates distribution of memory.
...
While not completely water-tight, all major memory usages by a given
cgroup are tracked so that the total memory consumption can be
accounted and controlled to a reasonable extent.

5-3. IO

The "io" controller regulates the distribution of IO resources. This
controller implements both weight based and absolute bandwidth or IOPS
limit distribution; however, weight based distribution is available
only if cfq-iosched is in use and neither scheme is available for
blk-mq devices.
</blockquote>

The open-xchange service is simply put into the default ''system.slice'' without applying further limits.

singlenode$ systemd-cgls
├─1 /sbin/init
├─system.slice
│ ├─avahi-daemon.service
│ │ ├─501 avahi-daemon: running [singlenode]
│ │ └─514 avahi-daemon: chroot helper
│ ├─console-kit-daemon.service
│ │ └─16164 /usr/sbin/console-kit-daemon --no-daemon
│ ├─dbus.service
│ │ └─508 /usr/bin/dbus-daemon --system --address=systemd: --nofork --nopidfile --systemd-activation
│ ├─munin-node.service
│ │ └─4290 /usr/bin/perl -wT /usr/sbin/munin-node
│ ├─open-xchange.service
│ │ └─6037 /usr/bin/java -Dsun.net.inetaddr.ttl=3600 -Dnetworkaddress.cache.ttl=3600 -Dnetworkaddress.cache.negative.ttl=10 ...

To check all the details use

singlenode:~ # systemctl show system.slice

==== Limits besides control groups ====
Besides control groups systemd allows you to apply other limits to the execution environment of your service. Here we can apply the limits that would normally be applied via ''limits.conf'' or ''ulimit''. Systemd uses [http://man7.org/linux/man-pages/man2/setrlimit.2.html setrlimit] for this. The options that we set by default are:

* LimitNOFILE
* LimitNPROC

which you have already seen when we took a look at the [[#Service_definiton|service definition]] and [[#Drop-in_config|drop-in configs]]

== Verify limits ==

=== System V ===

Get the proper pid of the java process started for the middleware by either checking '''ps faux'''
root 4103 0.0 0.0 66528 1868 pts/0 S 22:05 0:00 su -s /bin/bash open-xchange -c /opt/open-xchange/sbin/open-xchange
494 4109 9.5 19.8 2224644 381312 ? Ssl 22:05 1:12 \_ /usr/bin/java -Dsun.net.inetaddr.ttl=3600 -Dnetworkaddress.cache.ttl=3600

or programmatically
singlenode:~ # pid=$(pgrep -P $(</var/run/open-xchange.pid) java)

and check the limits applied for that process
singlenode:~ # cat /proc/$pid/limits
Limit Soft Limit Hard Limit Units
Max cpu time unlimited unlimited seconds
Max file size unlimited unlimited bytes
Max data size unlimited unlimited bytes
Max stack size 8388608 unlimited bytes
Max core file size 0 unlimited bytes
Max resident set unlimited unlimited bytes
Max processes 65536 65536 processes
Max open files 65536 65536 files
Max locked memory 65536 65536 bytes
Max address space unlimited unlimited bytes
Max file locks unlimited unlimited locks
Max pending signals 24254 24254 signals
Max msgqueue size 819200 819200 bytes
Max nice priority 0 0
Max realtime priority 0 0
Max realtime timeout unlimited unlimited us

=== Systemd ===
singlenode:~ # systemctl show open-xchange | grep Limit
StartLimitInterval=10000000
StartLimitBurst=5
StartLimitAction=none
MemoryLimit=18446744073709551615
LimitCPU=18446744073709551615
LimitFSIZE=18446744073709551615
LimitDATA=18446744073709551615
LimitSTACK=18446744073709551615
LimitCORE=18446744073709551615
LimitRSS=18446744073709551615
LimitNOFILE=65536
LimitAS=18446744073709551615
LimitNPROC=65536
LimitMEMLOCK=65536
LimitLOCKS=18446744073709551615
LimitSIGPENDING=19827
LimitMSGQUEUE=819200
LimitNICE=0
LimitRTPRIO=0
LimitRTTIME=18446744073709551615

== Evaluate JVM commandline options ==

Before starting Java and the OSGi framework with all the Open-Xchange middleware bundles we have to evaluate several important JVM commandline options that influence the performance and stability of the Open-Xchange middleware stack. These commandline options are specified in '''/opt/open-xchange/etc/ox-scriptconf.sh'''

=== Format < 7.8.4 ===
For version up to 7.8.3 the JVM configuration options were all specified via a single property '''JAVA_XTRAOPTS''' which resulted in a rather long configuration string that was hard to read and maintain.

JAVA_XTRAOPTS="-Dsun.net.inetaddr.ttl=3600 -Dnetworkaddress.cache.ttl=3600 -Dnetworkaddress.cache.negative.ttl=10 \
-Dlogback.threadlocal.put.duplicate=false -server -Djava.awt.headless=true -Xmx512M -XX:MaxPermSize=256m -XX:+UseConcMarkSweepGC \
-XX:+UseParNewGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:NewRatio=3 -XX:+UseTLAB \
-Dosgi.compatibility.bootdelegation=false -XX:-OmitStackTraceInFastThrow"

During the startup of the open-xchange service (step 11 of diagram Service startup using systemd and step 10 of diagram Service startup using upstart) this JAVA_XTRAOPTS string goes through a basic validity check and is then passed to the JVM.

=== Format >= 7.8.4 ===
Beginning from version 7.8.4 the options are split up and sorted into different categories like:

JAVA_OPTS_GC="-XX:+UseConcMarkSweepGC -XX:+UseParNewGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:NewRatio=3 -XX:+DisableExplicitGC"
JAVA_OPTS_LOG="-Dlogback.threadlocal.put.duplicate=false -XX:-OmitStackTraceInFastThrow"
JAVA_OPTS_MEM="-XX:MaxHeapSize=512M -XX:MaxPermSize=256M -XX:+UseTLAB"
JAVA_OPTS_NET="-Dsun.net.inetaddr.ttl=3600 -Dnetworkaddress.cache.ttl=3600 -Dnetworkaddress.cache.negative.ttl=10"
JAVA_OPTS_OSGI="-Dosgi.compatibility.bootdelegation=false"
JAVA_OPTS_SERVER="-server -Djava.awt.headless=true"

JAVA_OPTS_OTHER=""

#JAVA_OPTS_DEBUG="-XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/mnt/heapdump -Xloggc:/var/log/open-xchange/gc.log -verbose:gc -XX:+PrintGCDateStamps \
-XX:+PrintHeapAtGC -XX:+PrintGCApplicationStoppedTime -XX:+PrintTenuringDistribution"

which is much easier to read an maintain. Furthermore we added a useful '''JAVA_OPTS_DEBUG''' configuration parameter that comes preconfigured but unused/commented so that customers can quickly enable the most commonly used debugging options in case they have to provide detailed system information to Open-Xchange.

During the startup of the open-xchange service (step 11 of diagram ''Service startup using systemd'' and step 10 of diagram ''Service startup using upstart'') all uncommented JAVA_OPTS_* categories will then be evaluated and combined again into a single string of commandline options that can then be passed to the JVM.

=== Upgrade procedure ===
During the upgrade '''7.8.3-revX -> 7.8.4-revY''' existing JAVA_XTRAOPTS will automatically be split up and migrated to the new categories approach. A backup of the file (ox-scriptconf.sh.timestamp) will be created and admins are urged to check for proper migration before deleting the backup file like it's already done with *.dpkg-dist or *.rpmnew files on Debian or RHEL/SLES distributions.

== Encoding detection during JVM startup ==

=== Background Infos ===
All of en_US.UTF-8, .utf8, .utf-8, .UTF_8 will result in a valid UTF-8 charmap

==== JVM startup ====

The default file.encoding detection is done at JVM startup based upon LC_CTYPE which makes sense as it's documented as:

CODESET (LC_CTYPE)
Return a string with the name of the character encoding used in the selected locale, such as "UTF-8", "ISO-8859-1",
or "ANSI_X3.4-1968" (better known as US-ASCII). This is the same string that you get with "locale charmap".
For a list of character encoding names, try "locale -m", cf. locale(1).

Furthermore this influences the defaultCharset selection of the JVM.

/**
* Returns the default charset of this Java virtual machine.
*
* The default charset is determined during virtual-machine startup and
* typically depends upon the locale and charset of the underlying
* operating system.
*
* @return A charset object for the default charset
*
* @since 1.5
*/
public static Charset defaultCharset() {
if (defaultCharset == null) {
synchronized (Charset.class) {
String csn = AccessController.doPrivileged(
new GetPropertyAction("file.encoding"));
Charset cs = lookup(csn);
if (cs != null)
defaultCharset = cs;
else
defaultCharset = forName("UTF-8");
}
}
return defaultCharset;
}

Although we try to specify valid unicode encodings for every IO operations the Open-Xchange middleware does some parts of the JVM don't allow to specify the encoding to use and falls back to the detected default encoding. That's why admins have to make sure to provide a proper unicode environment for the middleware service during every start or use documented workarounds provided by Open-Xchange (e.g. )

=== Query which file encoding was detected by the JVM during startup ===
[[File:visualvm-file.encoding.png]]

The file encoding that was detected by the JVM during startup can be queried via JMX. You can use graphical management tools like [[https://visualvm.github.io/|visualvm]], query our [[https://oxpedia.org/wiki/index.php?title=Jolokia|Jolokia]] JMX bridge via http or just

/opt/open-xchange/sbin/showruntimestats -r | sed -ne 's/.*${key=file.encoding, value=[^}]*}$.*/\1/p'
{key=file.encoding, value=UTF-8}

=== Locale passing from client to server via SSH ===

The following setting of sshd allows clients to specify which locale to use on the server and might lead to some confusion or unexpected behaviour when checking the system locale or even to the usage of broken locales during middleware startup if your client sends misconfigured or missing locale configuration to the server.

grep AcceptEnv /etc/ssh/sshd_config
AcceptEnv LANG LC_*

For example when i connect to the machine my local german locale is sent to the server

[vagrant@showcase-community-centos-7 ~]$ locale -v
LANG=de_DE.UTF-8
LC_CTYPE="de_DE.UTF-8"
LC_NUMERIC=de_DE.UTF-8
LC_TIME=de_DE.UTF-8
LC_COLLATE="de_DE.UTF-8"
LC_MONETARY=de_DE.UTF-8
LC_MESSAGES="de_DE.UTF-8"
LC_PAPER=de_DE.UTF-8
LC_NAME=de_DE.UTF-8
LC_ADDRESS=de_DE.UTF-8
LC_TELEPHONE=de_DE.UTF-8
LC_MEASUREMENT=de_DE.UTF-8
LC_IDENTIFICATION=de_DE.UTF-8
LC_ALL=

But after specifying that i want a new login shell the default locale configured for that user/system will be used instead!

[root@showcase-community-centos-7 vagrant]# su -s /bin/bash open-xchange -l
-bash-4.2$ locale -v
LANG=en_US.UTF-8
LC_CTYPE="en_US.UTF-8"
LC_NUMERIC="en_US.UTF-8"
LC_TIME="en_US.UTF-8"
LC_COLLATE="en_US.UTF-8"
LC_MONETARY="en_US.UTF-8"
LC_MESSAGES="en_US.UTF-8"
LC_PAPER="en_US.UTF-8"
LC_NAME="en_US.UTF-8"
LC_ADDRESS="en_US.UTF-8"
LC_TELEPHONE="en_US.UTF-8"
LC_MEASUREMENT="en_US.UTF-8"
LC_IDENTIFICATION="en_US.UTF-8"
LC_ALL=

Add details from https://bugs.open-xchange.com/show_bug.cgi?id=51074

== Warnings during early start ==

Early warnings before the start of the JVM and related logging libraries like logback and logstash forwarding are redirected to '''/var/log/open-xchange/open-xchange-console.log'''.

=== Limits ===
If the open-xchange startup script isn't able to set the proper limits on CentOS 6 due to e.g. hard limits being enforced via pam_limits you'll see the following warning in the console log during startup

singlenode open-xchange # cat open-xchange-console.log
/opt/open-xchange/sbin/open-xchange: line 115: ulimit: max user processes: cannot modify limit: Operation not permitted

=== Locale ===
If the open-xchange startup script isn't able to detect a unicode locale you'll see a warning like the following (the current charmap might differ)

WARNING: Couldn't detect proper unicode charmap in locale category LC_CTYPE.
WARNING: Current charmap: "ANSI_X3.4-1968" might cause encoding problems.

== Summary: Open-Xchange middleware on specific distros ==
The support for the mentioned mechanism of resource control differ depending on the distribution and the init system in use.

=== Debian 7 ===
;Init
: System V style
;OX Configurable Limits/Defaults
: nofile, nproc

The mentioned limits can be configured via '''/opt/open-xchange/etc/ox-scriptconf.sh'''. The limits are applied via ulimit in the service's init script. The open-xchange service is finally started via start-stop-daemon which doesn't doesn't consider '''/etc/security/limits.*'''

=== RHEL 6 / CentOS 6 ===
;Init
: Upstart, System V compatible
;OX Configurable Limits/Defaults
: nofile, nproc

The mentioned limits can be configured via '''/opt/open-xchange/etc/ox-scriptconf.sh'''. The limits are applied via ulimit in the service's init script. Furthermore as the open-xchange service is finally started via '''su ... open-xchange''' on this distro a user session is opened via su/pam and the default CentOS pam config reads the '''/etc/security/limits.*''' configuration by loading the pam stack like:

: '''/etc/pam.d/su'''
:: '''-> /etc/pam.d/system-auth'''
::: '''-> pam_limits.so'''

If NPROC isn't configured for the open-xchange-server it's restricted to '''1024''' globally by default to prevent accidental fork bombs, see '''/etc/security/limits.d/90-nproc.conf''' which can result in severe problems modern multithreaded applications.

=== RHEL 7 / CentOS 7 / Debian 8 / SLE 12 ===
;Init
: Systemd
;OX Configurable Limits/Defaults
: nofile, nproc

For systemd the default limits are configured directly in the service's unit file that is shipped by OX and located at '''/usr/lib/systemd/system/open-xchange.service'''. The drop-in config to override or extend the default unit file is located at '''/etc/systemd/system/open-xchange.service.d/limits.conf'''. [https://www.freedesktop.org/software/systemd/man/systemd.exec.html Systemd.exec] shows a whole lot of options that can be used by admins to adapt the default service to their specific needs.

Sandbox:Middleware Startup/sandbox

2017-04-24T19:48:01Z

Marens: /* Format >= 7.8.4 */

== Systemd ==

==== Service definition ====
The default service definition as shipped by Open-Xchange is rather concise and easy to read. You can check by looking at the service file that is installed to either /usr/lib/systemd or /lib/systemd depending on your distribution of choice.

singlenode:~ # cat /usr/lib/systemd/system/open-xchange.service
[Unit]
After=remote-fs.target
After=time-sync.target ypbind.service sendmail.service cyrus.service

[Service]
User=open-xchange
PermissionsStartOnly=true
TimeoutStartSec=0
ExecStartPre=/opt/open-xchange/sbin/triggerupdatethemes -u
ExecStart=/opt/open-xchange/sbin/open-xchange
ExecStop=/opt/open-xchange/sbin/shutdown -w
ExecReload=/opt/open-xchange/sbin/triggerreloadconfiguration -d
KillMode=process
LimitNOFILE=65536
LimitNPROC=65536

[Install]
WantedBy=multi-user.target

==== Drop-in configs ====
Drop in configs allow administrators to easily adapt and override default service unit files. So if you want to change the default limits or add additional limits have a look at our default example drop-in config which is located at /etc/systemd/system/open-xchange.service.d/limits.conf

singlenode:~ # cat /etc/systemd/system/open-xchange.service.d/limits.conf
# Override and add options in this file
# See systemd.exec(5) for other limits

[Service]
#LimitNPROC=65536
#LimitNOFILE=65536

The systemd.exec documentation shows a whole lot of options that can be used by admins to adapt the default service to their specific needs.

 
[[File:mw_start_sysd.png|thumb|Service startup using systemd|left|1000px]]
 

== System V / Upstart ==

==== Service definition ====

The default service definition as shipped by Open-Xchange can be found at /etc/init.d/open-xchange and will differ slightly depending on the distro in use.

 
[[File:mw_start_sysv.png|thumb|Service startup using upstart|left|1000px]]
 
== Apply Resource limits ==

As you can see in the diagrams above one of the differences is that:

* systemd reads all the infos wrt. resource limitations from the original service file and then applies possible further restrictions from the drop-in configs (steps 2 and 8 in the diagram ''Service startup using systemd'') and limits configured in /opt/open-xchange/etc/ox-scriptconf.sh aren't considered anylonger
* the older System V style init has to source /opt/open-xchange/etc/ox-scriptconf.sh to be able to read the configs for the maximum number of processes (NPROC) and files(NRFILES) that the middleware shall be allowed to create/open (steps 3,4 and 15 in the diagram ''Service startup using upstart'').

=== Background Infos ===

Several ways exist to restrict resources on a linux system from a global level down to user/groups or even shells and the processes started by them.

==== Sysctl ====
Sysctl is used to modify kernel parameters at runtime. E.g. to set the maximum number of files

$ sysctl -w fs.file-max=100000

To permanently set them append to the main configuration file and reload the settings

$ echo fs.file-max=100000 >> /etc/sysctl.conf
$ sysctl -p

More infos can be found via '''man sysctl'''

==== Limits.conf ====

Allows to restrict resources an a global, group or user level. E.g:

$ cat /etc/security/limits.d/90-nproc.conf
# Default limit for number of user's processes to prevent
# accidental fork bombs.
# See rhbz #432903 for reasoning.

* soft nproc 1024

From '''man limits.conf''':
<blockquote>
''Also, please note that all limit settings are set per login. They are not global, nor are they permanent; existing only for the duration of the session.''
</blockquote>

The limits per login are applied via the ''pam'' stack. See '''man pam''' and '''man pam_limits''' for more details. As those limits are bound to sessions they don't affect most daemons started by our supported init systems or init utils. Most state that they are ignored by design, see [https://bugs.launchpad.net/ubuntu/+source/upstart/+bug/938669/comments/4 upstart], [https://bugzilla.redhat.com/show_bug.cgi?id=754285#c1 systemd] and [https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=302079 start-stop-daemon]

===== Ulimit =====
From '''man bash'''
<blockquote>
ulimit [-HSTabcdefilmnpqrstuvx [limit]]
Provides control over the resources available to the shell and to processes started by it, on systems that allow such control.
</blockquote>
This is what we use in our System V compatible init scripts to increase resources for the open-xchange process across multiple distros. Currently only ''the maximum number of processes'' and ''the maximum number of open file descriptors available to a single user'' are increased via ulimit. The values are specified in '''/opt/open-xchange/ox-scriptconf.sh'''

==== Systemd ====

===== Control Groups=====

Control groups should only affect the OX middleware if you create/manage them yourself of if you are using a ''modern'' distribution that already uses systemd as init.

Citing from the kernel cgroup [https://www.kernel.org/doc/Documentation/cgroup-v2.txt documentation]:

<blockquote>
1-2. What is cgroup?

cgroup is a mechanism to organize processes hierarchically and
distribute system resources along the hierarchy in a controlled and
configurable manner.

cgroup is largely composed of two parts - the core and controllers.
cgroup core is primarily responsible for hierarchically organizing
processes. A cgroup controller is usually responsible for
distributing a specific type of system resource along the hierarchy
although there are utility controllers which serve purposes other than
resource distribution.

cgroups form a tree structure and every process in the system belongs
to one and only one cgroup. All threads of a process belong to the
same cgroup. On creation, all processes are put in the cgroup that
the parent process belongs to at the time. A process can be migrated
to another cgroup. Migration of a process doesn't affect already
existing descendant processes.

Following certain structural constraints, controllers may be enabled or
disabled selectively on a cgroup. All controller behaviors are
hierarchical - if a controller is enabled on a cgroup, it affects all
processes which belong to the cgroups consisting the inclusive
sub-hierarchy of the cgroup. When a controller is enabled on a nested
cgroup, it always restricts the resource distribution further. The
restrictions set closer to the root in the hierarchy can not be
overridden from further away.
</blockquote>

So processes are organized into a tree structure of control groups and controllers are responsible for the distribution of resources. So what kind of controllers exist?

<blockquote>
5. Controllers

5-1. CPU

The "cpu" controllers regulates distribution of CPU cycles. This
controller implements weight and absolute bandwidth limit models for
normal scheduling policy and absolute bandwidth allocation model for
realtime scheduling policy.

5-2. Memory

The "memory" controller regulates distribution of memory.
...
While not completely water-tight, all major memory usages by a given
cgroup are tracked so that the total memory consumption can be
accounted and controlled to a reasonable extent.

5-3. IO

The "io" controller regulates the distribution of IO resources. This
controller implements both weight based and absolute bandwidth or IOPS
limit distribution; however, weight based distribution is available
only if cfq-iosched is in use and neither scheme is available for
blk-mq devices.
</blockquote>

The open-xchange service is simply put into the default ''system.slice'' without applying further limits.

singlenode$ systemd-cgls
├─1 /sbin/init
├─system.slice
│ ├─avahi-daemon.service
│ │ ├─501 avahi-daemon: running [singlenode]
│ │ └─514 avahi-daemon: chroot helper
│ ├─console-kit-daemon.service
│ │ └─16164 /usr/sbin/console-kit-daemon --no-daemon
│ ├─dbus.service
│ │ └─508 /usr/bin/dbus-daemon --system --address=systemd: --nofork --nopidfile --systemd-activation
│ ├─munin-node.service
│ │ └─4290 /usr/bin/perl -wT /usr/sbin/munin-node
│ ├─open-xchange.service
│ │ └─6037 /usr/bin/java -Dsun.net.inetaddr.ttl=3600 -Dnetworkaddress.cache.ttl=3600 -Dnetworkaddress.cache.negative.ttl=10 ...

To check all the details use

singlenode:~ # systemctl show system.slice

==== Limits besides control groups ====
Besides control groups systemd allows you to apply other limits to the execution environment of your service. Here we can apply the limits that would normally be applied via ''limits.conf'' or ''ulimit''. Systemd uses [http://man7.org/linux/man-pages/man2/setrlimit.2.html setrlimit] for this. The options that we set by default are:

* LimitNOFILE
* LimitNPROC

which you have already seen when we took a look at the [[#Service_definiton|service definition]] and [[#Drop-in_config|drop-in configs]]

== Verify limits ==

=== System V ===

Get the proper pid of the java process started for the middleware by either checking '''ps faux'''
root 4103 0.0 0.0 66528 1868 pts/0 S 22:05 0:00 su -s /bin/bash open-xchange -c /opt/open-xchange/sbin/open-xchange
494 4109 9.5 19.8 2224644 381312 ? Ssl 22:05 1:12 \_ /usr/bin/java -Dsun.net.inetaddr.ttl=3600 -Dnetworkaddress.cache.ttl=3600

or programmatically
singlenode:~ # pid=$(pgrep -P $(</var/run/open-xchange.pid) java)

and check the limits applied for that process
singlenode:~ # cat /proc/$pid/limits
Limit Soft Limit Hard Limit Units
Max cpu time unlimited unlimited seconds
Max file size unlimited unlimited bytes
Max data size unlimited unlimited bytes
Max stack size 8388608 unlimited bytes
Max core file size 0 unlimited bytes
Max resident set unlimited unlimited bytes
Max processes 65536 65536 processes
Max open files 65536 65536 files
Max locked memory 65536 65536 bytes
Max address space unlimited unlimited bytes
Max file locks unlimited unlimited locks
Max pending signals 24254 24254 signals
Max msgqueue size 819200 819200 bytes
Max nice priority 0 0
Max realtime priority 0 0
Max realtime timeout unlimited unlimited us

=== Systemd ===
singlenode:~ # systemctl show open-xchange | grep Limit
StartLimitInterval=10000000
StartLimitBurst=5
StartLimitAction=none
MemoryLimit=18446744073709551615
LimitCPU=18446744073709551615
LimitFSIZE=18446744073709551615
LimitDATA=18446744073709551615
LimitSTACK=18446744073709551615
LimitCORE=18446744073709551615
LimitRSS=18446744073709551615
LimitNOFILE=65536
LimitAS=18446744073709551615
LimitNPROC=65536
LimitMEMLOCK=65536
LimitLOCKS=18446744073709551615
LimitSIGPENDING=19827
LimitMSGQUEUE=819200
LimitNICE=0
LimitRTPRIO=0
LimitRTTIME=18446744073709551615

== Evaluate JVM commandline options ==

Before starting Java and the OSGi framework with all the Open-Xchange middleware bundles we have to evaluate several important JVM commandline options that influence the performance and stability of the Open-Xchange middleware stack. These commandline options are specified in '''/opt/open-xchange/etc/ox-scriptconf.sh'''

=== Format < 7.8.4 ===
For version up to 7.8.3 the JVM configuration options were all specified via a single property '''JAVA_XTRAOPTS''' which resulted in a rather long configuration string that was hard to read and maintain.

JAVA_XTRAOPTS="-Dsun.net.inetaddr.ttl=3600 -Dnetworkaddress.cache.ttl=3600 -Dnetworkaddress.cache.negative.ttl=10 \
-Dlogback.threadlocal.put.duplicate=false -server -Djava.awt.headless=true -Xmx512M -XX:MaxPermSize=256m -XX:+UseConcMarkSweepGC \
-XX:+UseParNewGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:NewRatio=3 -XX:+UseTLAB \
-Dosgi.compatibility.bootdelegation=false -XX:-OmitStackTraceInFastThrow"

During the startup of the open-xchange service (step 11 of diagram Service startup using systemd and step 10 of diagram Service startup using upstart) this JAVA_XTRAOPTS string goes through a basic validity check and is then passed to the JVM.

=== Format >= 7.8.4 ===
Beginning from version 7.8.4 the options are split up and sorted into different categories like:

JAVA_OPTS_GC="-XX:+UseConcMarkSweepGC -XX:+UseParNewGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:NewRatio=3 -XX:+DisableExplicitGC"
JAVA_OPTS_LOG="-Dlogback.threadlocal.put.duplicate=false -XX:-OmitStackTraceInFastThrow"
JAVA_OPTS_MEM="-XX:MaxHeapSize=512M -XX:MaxPermSize=256M -XX:+UseTLAB"
JAVA_OPTS_NET="-Dsun.net.inetaddr.ttl=3600 -Dnetworkaddress.cache.ttl=3600 -Dnetworkaddress.cache.negative.ttl=10"
JAVA_OPTS_OSGI="-Dosgi.compatibility.bootdelegation=false"
JAVA_OPTS_SERVER="-server -Djava.awt.headless=true"

JAVA_OPTS_OTHER=""

#JAVA_OPTS_DEBUG="-XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/mnt/heapdump -Xloggc:/var/log/open-xchange/gc.log -verbose:gc -XX:+PrintGCDateStamps \
-XX:+PrintHeapAtGC -XX:+PrintGCApplicationStoppedTime -XX:+PrintTenuringDistribution"

which is much easier to read an maintain. Furthermore we added a useful '''JAVA_OPTS_DEBUG''' configuration parameter that comes preconfigured but unused/commented so that customers can quickly enable the most commonly used debugging options in case they have to provide detailed system information to Open-Xchange.

During the startup of the open-xchange service (step 11 of diagram ''Service startup using systemd'' and step 10 of diagram ''Service startup using upstart'') all uncommented JAVA_OPTS_* categories will then be evaluated and combined again into a single string of commandline options that can then be passed to the JVM.

=== Upgrade procedure ===
During the upgrade '''7.8.3-revX -> 7.8.4-revY''' existing JAVA_XTRAOPTS will automatically be split up and migrated to the new categories approach. A backup of the file (ox-scriptconf.sh.timestamp) will be created and admins are urged to check for proper migration before deleting the backup file like it's already done with *.dpkg-dist or *.rpmnew files on Debian or RHEL/SLES distributions.

== Encoding detection during JVM startup ==

=== Background Infos ===
All of en_US.UTF-8, .utf8, .utf-8, .UTF_8 will result in a valid UTF-8 charmap

==== JVM startup ====

The default file.encoding detection is done at JVM startup based upon LC_CTYPE which makes sense as it's documented as:

CODESET (LC_CTYPE)
Return a string with the name of the character encoding used in the selected locale, such as "UTF-8", "ISO-8859-1",
or "ANSI_X3.4-1968" (better known as US-ASCII). This is the same string that you get with "locale charmap".
For a list of character encoding names, try "locale -m", cf. locale(1).

Furthermore this influences the defaultCharset selection of the JVM.

/**
* Returns the default charset of this Java virtual machine.
*
* The default charset is determined during virtual-machine startup and
* typically depends upon the locale and charset of the underlying
* operating system.
*
* @return A charset object for the default charset
*
* @since 1.5
*/
public static Charset defaultCharset() {
if (defaultCharset == null) {
synchronized (Charset.class) {
String csn = AccessController.doPrivileged(
new GetPropertyAction("file.encoding"));
Charset cs = lookup(csn);
if (cs != null)
defaultCharset = cs;
else
defaultCharset = forName("UTF-8");
}
}
return defaultCharset;
}

Although we try to specify valid unicode encodings for every IO operations the Open-Xchange middleware does some parts of the JVM don't allow to specify the encoding to use and falls back to the detected default encoding. That's why admins have to make sure to provide a proper unicode environment for the middleware service during every start or use documented workarounds provided by Open-Xchange (e.g. )

=== Query which file encoding was detected by the JVM during startup ===

/opt/open-xchange/sbin/showruntimestats -r | sed -ne 's/.*${key=file.encoding, value=[^}]*}$.*/\1/p'
{key=file.encoding, value=UTF-8}

=== Locale passing from client to server via SSH ===

The following setting of sshd allows clients to specify which locale to use on the server and might lead to some confusion or unexpected behaviour when checking the system locale or even to the usage of broken locales during middleware startup if your client sends misconfigured or missing locale configuration to the server.

grep AcceptEnv /etc/ssh/sshd_config
AcceptEnv LANG LC_*

For example when i connect to the machine my local german locale is sent to the server

[vagrant@showcase-community-centos-7 ~]$ locale -v
LANG=de_DE.UTF-8
LC_CTYPE="de_DE.UTF-8"
LC_NUMERIC=de_DE.UTF-8
LC_TIME=de_DE.UTF-8
LC_COLLATE="de_DE.UTF-8"
LC_MONETARY=de_DE.UTF-8
LC_MESSAGES="de_DE.UTF-8"
LC_PAPER=de_DE.UTF-8
LC_NAME=de_DE.UTF-8
LC_ADDRESS=de_DE.UTF-8
LC_TELEPHONE=de_DE.UTF-8
LC_MEASUREMENT=de_DE.UTF-8
LC_IDENTIFICATION=de_DE.UTF-8
LC_ALL=

But after specifying that i want a new login shell the default locale configured for that user/system will be used instead!

[root@showcase-community-centos-7 vagrant]# su -s /bin/bash open-xchange -l
-bash-4.2$ locale -v
LANG=en_US.UTF-8
LC_CTYPE="en_US.UTF-8"
LC_NUMERIC="en_US.UTF-8"
LC_TIME="en_US.UTF-8"
LC_COLLATE="en_US.UTF-8"
LC_MONETARY="en_US.UTF-8"
LC_MESSAGES="en_US.UTF-8"
LC_PAPER="en_US.UTF-8"
LC_NAME="en_US.UTF-8"
LC_ADDRESS="en_US.UTF-8"
LC_TELEPHONE="en_US.UTF-8"
LC_MEASUREMENT="en_US.UTF-8"
LC_IDENTIFICATION="en_US.UTF-8"
LC_ALL=

Add details from https://bugs.open-xchange.com/show_bug.cgi?id=51074

== Warnings during early start ==

Early warnings before the start of the JVM and related logging libraries like logback and logstash forwarding are redirected to '''/var/log/open-xchange/open-xchange-console.log'''.

=== Limits ===
If the open-xchange startup script isn't able to set the proper limits on CentOS 6 due to e.g. hard limits being enforced via pam_limits you'll see the following warning in the console log during startup

singlenode open-xchange # cat open-xchange-console.log
/opt/open-xchange/sbin/open-xchange: line 115: ulimit: max user processes: cannot modify limit: Operation not permitted

=== Locale ===
If the open-xchange startup script isn't able to detect a unicode locale you'll see a warning like the following (the current charmap might differ)

WARNING: Couldn't detect proper unicode charmap in locale category LC_CTYPE.
WARNING: Current charmap: "ANSI_X3.4-1968" might cause encoding problems.

== Summary: Open-Xchange middleware on specific distros ==
The support for the mentioned mechanism of resource control differ depending on the distribution and the init system in use.

=== Debian 7 ===
;Init
: System V style
;OX Configurable Limits/Defaults
: nofile, nproc

The mentioned limits can be configured via '''/opt/open-xchange/etc/ox-scriptconf.sh'''. The limits are applied via ulimit in the service's init script. The open-xchange service is finally started via start-stop-daemon which doesn't doesn't consider '''/etc/security/limits.*'''

=== RHEL 6 / CentOS 6 ===
;Init
: Upstart, System V compatible
;OX Configurable Limits/Defaults
: nofile, nproc

The mentioned limits can be configured via '''/opt/open-xchange/etc/ox-scriptconf.sh'''. The limits are applied via ulimit in the service's init script. Furthermore as the open-xchange service is finally started via '''su ... open-xchange''' on this distro a user session is opened via su/pam and the default CentOS pam config reads the '''/etc/security/limits.*''' configuration by loading the pam stack like:

: '''/etc/pam.d/su'''
:: '''-> /etc/pam.d/system-auth'''
::: '''-> pam_limits.so'''

If NPROC isn't configured for the open-xchange-server it's restricted to '''1024''' globally by default to prevent accidental fork bombs, see '''/etc/security/limits.d/90-nproc.conf''' which can result in severe problems modern multithreaded applications.

=== RHEL 7 / CentOS 7 / Debian 8 / SLE 12 ===
;Init
: Systemd
;OX Configurable Limits/Defaults
: nofile, nproc

For systemd the default limits are configured directly in the service's unit file that is shipped by OX and located at '''/usr/lib/systemd/system/open-xchange.service'''. The drop-in config to override or extend the default unit file is located at '''/etc/systemd/system/open-xchange.service.d/limits.conf'''. [https://www.freedesktop.org/software/systemd/man/systemd.exec.html Systemd.exec] shows a whole lot of options that can be used by admins to adapt the default service to their specific needs.

Sandbox:Middleware Startup/sandbox

2017-04-24T19:47:06Z

Marens: /* Format < 7.8.4 */

== Systemd ==

==== Service definition ====
The default service definition as shipped by Open-Xchange is rather concise and easy to read. You can check by looking at the service file that is installed to either /usr/lib/systemd or /lib/systemd depending on your distribution of choice.

singlenode:~ # cat /usr/lib/systemd/system/open-xchange.service
[Unit]
After=remote-fs.target
After=time-sync.target ypbind.service sendmail.service cyrus.service

[Service]
User=open-xchange
PermissionsStartOnly=true
TimeoutStartSec=0
ExecStartPre=/opt/open-xchange/sbin/triggerupdatethemes -u
ExecStart=/opt/open-xchange/sbin/open-xchange
ExecStop=/opt/open-xchange/sbin/shutdown -w
ExecReload=/opt/open-xchange/sbin/triggerreloadconfiguration -d
KillMode=process
LimitNOFILE=65536
LimitNPROC=65536

[Install]
WantedBy=multi-user.target

==== Drop-in configs ====
Drop in configs allow administrators to easily adapt and override default service unit files. So if you want to change the default limits or add additional limits have a look at our default example drop-in config which is located at /etc/systemd/system/open-xchange.service.d/limits.conf

singlenode:~ # cat /etc/systemd/system/open-xchange.service.d/limits.conf
# Override and add options in this file
# See systemd.exec(5) for other limits

[Service]
#LimitNPROC=65536
#LimitNOFILE=65536

The systemd.exec documentation shows a whole lot of options that can be used by admins to adapt the default service to their specific needs.

 
[[File:mw_start_sysd.png|thumb|Service startup using systemd|left|1000px]]
 

== System V / Upstart ==

==== Service definition ====

The default service definition as shipped by Open-Xchange can be found at /etc/init.d/open-xchange and will differ slightly depending on the distro in use.

 
[[File:mw_start_sysv.png|thumb|Service startup using upstart|left|1000px]]
 
== Apply Resource limits ==

As you can see in the diagrams above one of the differences is that:

* systemd reads all the infos wrt. resource limitations from the original service file and then applies possible further restrictions from the drop-in configs (steps 2 and 8 in the diagram ''Service startup using systemd'') and limits configured in /opt/open-xchange/etc/ox-scriptconf.sh aren't considered anylonger
* the older System V style init has to source /opt/open-xchange/etc/ox-scriptconf.sh to be able to read the configs for the maximum number of processes (NPROC) and files(NRFILES) that the middleware shall be allowed to create/open (steps 3,4 and 15 in the diagram ''Service startup using upstart'').

=== Background Infos ===

Several ways exist to restrict resources on a linux system from a global level down to user/groups or even shells and the processes started by them.

==== Sysctl ====
Sysctl is used to modify kernel parameters at runtime. E.g. to set the maximum number of files

$ sysctl -w fs.file-max=100000

To permanently set them append to the main configuration file and reload the settings

$ echo fs.file-max=100000 >> /etc/sysctl.conf
$ sysctl -p

More infos can be found via '''man sysctl'''

==== Limits.conf ====

Allows to restrict resources an a global, group or user level. E.g:

$ cat /etc/security/limits.d/90-nproc.conf
# Default limit for number of user's processes to prevent
# accidental fork bombs.
# See rhbz #432903 for reasoning.

* soft nproc 1024

From '''man limits.conf''':
<blockquote>
''Also, please note that all limit settings are set per login. They are not global, nor are they permanent; existing only for the duration of the session.''
</blockquote>

The limits per login are applied via the ''pam'' stack. See '''man pam''' and '''man pam_limits''' for more details. As those limits are bound to sessions they don't affect most daemons started by our supported init systems or init utils. Most state that they are ignored by design, see [https://bugs.launchpad.net/ubuntu/+source/upstart/+bug/938669/comments/4 upstart], [https://bugzilla.redhat.com/show_bug.cgi?id=754285#c1 systemd] and [https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=302079 start-stop-daemon]

===== Ulimit =====
From '''man bash'''
<blockquote>
ulimit [-HSTabcdefilmnpqrstuvx [limit]]
Provides control over the resources available to the shell and to processes started by it, on systems that allow such control.
</blockquote>
This is what we use in our System V compatible init scripts to increase resources for the open-xchange process across multiple distros. Currently only ''the maximum number of processes'' and ''the maximum number of open file descriptors available to a single user'' are increased via ulimit. The values are specified in '''/opt/open-xchange/ox-scriptconf.sh'''

==== Systemd ====

===== Control Groups=====

Control groups should only affect the OX middleware if you create/manage them yourself of if you are using a ''modern'' distribution that already uses systemd as init.

Citing from the kernel cgroup [https://www.kernel.org/doc/Documentation/cgroup-v2.txt documentation]:

<blockquote>
1-2. What is cgroup?

cgroup is a mechanism to organize processes hierarchically and
distribute system resources along the hierarchy in a controlled and
configurable manner.

cgroup is largely composed of two parts - the core and controllers.
cgroup core is primarily responsible for hierarchically organizing
processes. A cgroup controller is usually responsible for
distributing a specific type of system resource along the hierarchy
although there are utility controllers which serve purposes other than
resource distribution.

cgroups form a tree structure and every process in the system belongs
to one and only one cgroup. All threads of a process belong to the
same cgroup. On creation, all processes are put in the cgroup that
the parent process belongs to at the time. A process can be migrated
to another cgroup. Migration of a process doesn't affect already
existing descendant processes.

Following certain structural constraints, controllers may be enabled or
disabled selectively on a cgroup. All controller behaviors are
hierarchical - if a controller is enabled on a cgroup, it affects all
processes which belong to the cgroups consisting the inclusive
sub-hierarchy of the cgroup. When a controller is enabled on a nested
cgroup, it always restricts the resource distribution further. The
restrictions set closer to the root in the hierarchy can not be
overridden from further away.
</blockquote>

So processes are organized into a tree structure of control groups and controllers are responsible for the distribution of resources. So what kind of controllers exist?

<blockquote>
5. Controllers

5-1. CPU

The "cpu" controllers regulates distribution of CPU cycles. This
controller implements weight and absolute bandwidth limit models for
normal scheduling policy and absolute bandwidth allocation model for
realtime scheduling policy.

5-2. Memory

The "memory" controller regulates distribution of memory.
...
While not completely water-tight, all major memory usages by a given
cgroup are tracked so that the total memory consumption can be
accounted and controlled to a reasonable extent.

5-3. IO

The "io" controller regulates the distribution of IO resources. This
controller implements both weight based and absolute bandwidth or IOPS
limit distribution; however, weight based distribution is available
only if cfq-iosched is in use and neither scheme is available for
blk-mq devices.
</blockquote>

The open-xchange service is simply put into the default ''system.slice'' without applying further limits.

singlenode$ systemd-cgls
├─1 /sbin/init
├─system.slice
│ ├─avahi-daemon.service
│ │ ├─501 avahi-daemon: running [singlenode]
│ │ └─514 avahi-daemon: chroot helper
│ ├─console-kit-daemon.service
│ │ └─16164 /usr/sbin/console-kit-daemon --no-daemon
│ ├─dbus.service
│ │ └─508 /usr/bin/dbus-daemon --system --address=systemd: --nofork --nopidfile --systemd-activation
│ ├─munin-node.service
│ │ └─4290 /usr/bin/perl -wT /usr/sbin/munin-node
│ ├─open-xchange.service
│ │ └─6037 /usr/bin/java -Dsun.net.inetaddr.ttl=3600 -Dnetworkaddress.cache.ttl=3600 -Dnetworkaddress.cache.negative.ttl=10 ...

To check all the details use

singlenode:~ # systemctl show system.slice

==== Limits besides control groups ====
Besides control groups systemd allows you to apply other limits to the execution environment of your service. Here we can apply the limits that would normally be applied via ''limits.conf'' or ''ulimit''. Systemd uses [http://man7.org/linux/man-pages/man2/setrlimit.2.html setrlimit] for this. The options that we set by default are:

* LimitNOFILE
* LimitNPROC

which you have already seen when we took a look at the [[#Service_definiton|service definition]] and [[#Drop-in_config|drop-in configs]]

== Verify limits ==

=== System V ===

Get the proper pid of the java process started for the middleware by either checking '''ps faux'''
root 4103 0.0 0.0 66528 1868 pts/0 S 22:05 0:00 su -s /bin/bash open-xchange -c /opt/open-xchange/sbin/open-xchange
494 4109 9.5 19.8 2224644 381312 ? Ssl 22:05 1:12 \_ /usr/bin/java -Dsun.net.inetaddr.ttl=3600 -Dnetworkaddress.cache.ttl=3600

or programmatically
singlenode:~ # pid=$(pgrep -P $(</var/run/open-xchange.pid) java)

and check the limits applied for that process
singlenode:~ # cat /proc/$pid/limits
Limit Soft Limit Hard Limit Units
Max cpu time unlimited unlimited seconds
Max file size unlimited unlimited bytes
Max data size unlimited unlimited bytes
Max stack size 8388608 unlimited bytes
Max core file size 0 unlimited bytes
Max resident set unlimited unlimited bytes
Max processes 65536 65536 processes
Max open files 65536 65536 files
Max locked memory 65536 65536 bytes
Max address space unlimited unlimited bytes
Max file locks unlimited unlimited locks
Max pending signals 24254 24254 signals
Max msgqueue size 819200 819200 bytes
Max nice priority 0 0
Max realtime priority 0 0
Max realtime timeout unlimited unlimited us

=== Systemd ===
singlenode:~ # systemctl show open-xchange | grep Limit
StartLimitInterval=10000000
StartLimitBurst=5
StartLimitAction=none
MemoryLimit=18446744073709551615
LimitCPU=18446744073709551615
LimitFSIZE=18446744073709551615
LimitDATA=18446744073709551615
LimitSTACK=18446744073709551615
LimitCORE=18446744073709551615
LimitRSS=18446744073709551615
LimitNOFILE=65536
LimitAS=18446744073709551615
LimitNPROC=65536
LimitMEMLOCK=65536
LimitLOCKS=18446744073709551615
LimitSIGPENDING=19827
LimitMSGQUEUE=819200
LimitNICE=0
LimitRTPRIO=0
LimitRTTIME=18446744073709551615

== Evaluate JVM commandline options ==

Before starting Java and the OSGi framework with all the Open-Xchange middleware bundles we have to evaluate several important JVM commandline options that influence the performance and stability of the Open-Xchange middleware stack. These commandline options are specified in '''/opt/open-xchange/etc/ox-scriptconf.sh'''

=== Format < 7.8.4 ===
For version up to 7.8.3 the JVM configuration options were all specified via a single property '''JAVA_XTRAOPTS''' which resulted in a rather long configuration string that was hard to read and maintain.

JAVA_XTRAOPTS="-Dsun.net.inetaddr.ttl=3600 -Dnetworkaddress.cache.ttl=3600 -Dnetworkaddress.cache.negative.ttl=10 \
-Dlogback.threadlocal.put.duplicate=false -server -Djava.awt.headless=true -Xmx512M -XX:MaxPermSize=256m -XX:+UseConcMarkSweepGC \
-XX:+UseParNewGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:NewRatio=3 -XX:+UseTLAB \
-Dosgi.compatibility.bootdelegation=false -XX:-OmitStackTraceInFastThrow"

During the startup of the open-xchange service (step 11 of diagram Service startup using systemd and step 10 of diagram Service startup using upstart) this JAVA_XTRAOPTS string goes through a basic validity check and is then passed to the JVM.

=== Format >= 7.8.4 ===
Beginning from version 7.8.4 the options are split up and sorted into different categories like:

JAVA_OPTS_GC="-XX:+UseConcMarkSweepGC -XX:+UseParNewGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:NewRatio=3 -XX:+DisableExplicitGC"
JAVA_OPTS_LOG="-Dlogback.threadlocal.put.duplicate=false -XX:-OmitStackTraceInFastThrow"
JAVA_OPTS_MEM="-XX:MaxHeapSize=512M -XX:MaxPermSize=256M -XX:+UseTLAB"
JAVA_OPTS_NET="-Dsun.net.inetaddr.ttl=3600 -Dnetworkaddress.cache.ttl=3600 -Dnetworkaddress.cache.negative.ttl=10"
JAVA_OPTS_OSGI="-Dosgi.compatibility.bootdelegation=false"
JAVA_OPTS_SERVER="-server -Djava.awt.headless=true"

JAVA_OPTS_OTHER=""

#JAVA_OPTS_DEBUG="-XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/mnt/heapdump -Xloggc:/var/log/open-xchange/gc.log -verbose:gc -XX:+PrintGCDateStamps \
-XX:+PrintHeapAtGC -XX:+PrintGCApplicationStoppedTime -XX:+PrintTenuringDistribution"

which is much easier to read an maintain. Furthermore we added a useful '''JAVA_OPTS_DEBUG''' configuration parameter that comes preconfigured but unused/commented so that customers can quickly enable the most commonly used debugging options in case they have to provide detailed system information to Open-Xchange.

During the startup of the open-xchange service (step 11 of diagram ''Service startup using systemd'' and step 10 of diagram ''Service startup using upstart''.) all uncommented JAVA_OPTS_* categories will then be evaluated and combined again into a single string of commandline options that can then be passed to the JVM.

=== Upgrade procedure ===
During the upgrade '''7.8.3-revX -> 7.8.4-revY''' existing JAVA_XTRAOPTS will automatically be split up and migrated to the new categories approach. A backup of the file (ox-scriptconf.sh.timestamp) will be created and admins are urged to check for proper migration before deleting the backup file like it's already done with *.dpkg-dist or *.rpmnew files on Debian or RHEL/SLES distributions.

== Encoding detection during JVM startup ==

=== Background Infos ===
All of en_US.UTF-8, .utf8, .utf-8, .UTF_8 will result in a valid UTF-8 charmap

==== JVM startup ====

The default file.encoding detection is done at JVM startup based upon LC_CTYPE which makes sense as it's documented as:

CODESET (LC_CTYPE)
Return a string with the name of the character encoding used in the selected locale, such as "UTF-8", "ISO-8859-1",
or "ANSI_X3.4-1968" (better known as US-ASCII). This is the same string that you get with "locale charmap".
For a list of character encoding names, try "locale -m", cf. locale(1).

Furthermore this influences the defaultCharset selection of the JVM.

/**
* Returns the default charset of this Java virtual machine.
*
* The default charset is determined during virtual-machine startup and
* typically depends upon the locale and charset of the underlying
* operating system.
*
* @return A charset object for the default charset
*
* @since 1.5
*/
public static Charset defaultCharset() {
if (defaultCharset == null) {
synchronized (Charset.class) {
String csn = AccessController.doPrivileged(
new GetPropertyAction("file.encoding"));
Charset cs = lookup(csn);
if (cs != null)
defaultCharset = cs;
else
defaultCharset = forName("UTF-8");
}
}
return defaultCharset;
}

Although we try to specify valid unicode encodings for every IO operations the Open-Xchange middleware does some parts of the JVM don't allow to specify the encoding to use and falls back to the detected default encoding. That's why admins have to make sure to provide a proper unicode environment for the middleware service during every start or use documented workarounds provided by Open-Xchange (e.g. )

=== Query which file encoding was detected by the JVM during startup ===

/opt/open-xchange/sbin/showruntimestats -r | sed -ne 's/.*${key=file.encoding, value=[^}]*}$.*/\1/p'
{key=file.encoding, value=UTF-8}

=== Locale passing from client to server via SSH ===

The following setting of sshd allows clients to specify which locale to use on the server and might lead to some confusion or unexpected behaviour when checking the system locale or even to the usage of broken locales during middleware startup if your client sends misconfigured or missing locale configuration to the server.

grep AcceptEnv /etc/ssh/sshd_config
AcceptEnv LANG LC_*

For example when i connect to the machine my local german locale is sent to the server

[vagrant@showcase-community-centos-7 ~]$ locale -v
LANG=de_DE.UTF-8
LC_CTYPE="de_DE.UTF-8"
LC_NUMERIC=de_DE.UTF-8
LC_TIME=de_DE.UTF-8
LC_COLLATE="de_DE.UTF-8"
LC_MONETARY=de_DE.UTF-8
LC_MESSAGES="de_DE.UTF-8"
LC_PAPER=de_DE.UTF-8
LC_NAME=de_DE.UTF-8
LC_ADDRESS=de_DE.UTF-8
LC_TELEPHONE=de_DE.UTF-8
LC_MEASUREMENT=de_DE.UTF-8
LC_IDENTIFICATION=de_DE.UTF-8
LC_ALL=

But after specifying that i want a new login shell the default locale configured for that user/system will be used instead!

[root@showcase-community-centos-7 vagrant]# su -s /bin/bash open-xchange -l
-bash-4.2$ locale -v
LANG=en_US.UTF-8
LC_CTYPE="en_US.UTF-8"
LC_NUMERIC="en_US.UTF-8"
LC_TIME="en_US.UTF-8"
LC_COLLATE="en_US.UTF-8"
LC_MONETARY="en_US.UTF-8"
LC_MESSAGES="en_US.UTF-8"
LC_PAPER="en_US.UTF-8"
LC_NAME="en_US.UTF-8"
LC_ADDRESS="en_US.UTF-8"
LC_TELEPHONE="en_US.UTF-8"
LC_MEASUREMENT="en_US.UTF-8"
LC_IDENTIFICATION="en_US.UTF-8"
LC_ALL=

Add details from https://bugs.open-xchange.com/show_bug.cgi?id=51074

== Warnings during early start ==

Early warnings before the start of the JVM and related logging libraries like logback and logstash forwarding are redirected to '''/var/log/open-xchange/open-xchange-console.log'''.

=== Limits ===
If the open-xchange startup script isn't able to set the proper limits on CentOS 6 due to e.g. hard limits being enforced via pam_limits you'll see the following warning in the console log during startup

singlenode open-xchange # cat open-xchange-console.log
/opt/open-xchange/sbin/open-xchange: line 115: ulimit: max user processes: cannot modify limit: Operation not permitted

=== Locale ===
If the open-xchange startup script isn't able to detect a unicode locale you'll see a warning like the following (the current charmap might differ)

WARNING: Couldn't detect proper unicode charmap in locale category LC_CTYPE.
WARNING: Current charmap: "ANSI_X3.4-1968" might cause encoding problems.

== Summary: Open-Xchange middleware on specific distros ==
The support for the mentioned mechanism of resource control differ depending on the distribution and the init system in use.

=== Debian 7 ===
;Init
: System V style
;OX Configurable Limits/Defaults
: nofile, nproc

The mentioned limits can be configured via '''/opt/open-xchange/etc/ox-scriptconf.sh'''. The limits are applied via ulimit in the service's init script. The open-xchange service is finally started via start-stop-daemon which doesn't doesn't consider '''/etc/security/limits.*'''

=== RHEL 6 / CentOS 6 ===
;Init
: Upstart, System V compatible
;OX Configurable Limits/Defaults
: nofile, nproc

The mentioned limits can be configured via '''/opt/open-xchange/etc/ox-scriptconf.sh'''. The limits are applied via ulimit in the service's init script. Furthermore as the open-xchange service is finally started via '''su ... open-xchange''' on this distro a user session is opened via su/pam and the default CentOS pam config reads the '''/etc/security/limits.*''' configuration by loading the pam stack like:

: '''/etc/pam.d/su'''
:: '''-> /etc/pam.d/system-auth'''
::: '''-> pam_limits.so'''

If NPROC isn't configured for the open-xchange-server it's restricted to '''1024''' globally by default to prevent accidental fork bombs, see '''/etc/security/limits.d/90-nproc.conf''' which can result in severe problems modern multithreaded applications.

=== RHEL 7 / CentOS 7 / Debian 8 / SLE 12 ===
;Init
: Systemd
;OX Configurable Limits/Defaults
: nofile, nproc

For systemd the default limits are configured directly in the service's unit file that is shipped by OX and located at '''/usr/lib/systemd/system/open-xchange.service'''. The drop-in config to override or extend the default unit file is located at '''/etc/systemd/system/open-xchange.service.d/limits.conf'''. [https://www.freedesktop.org/software/systemd/man/systemd.exec.html Systemd.exec] shows a whole lot of options that can be used by admins to adapt the default service to their specific needs.

Sandbox:Middleware Startup/sandbox

2017-04-24T19:45:29Z

Marens: /* Apply Resource limits */

== Systemd ==

==== Service definition ====
The default service definition as shipped by Open-Xchange is rather concise and easy to read. You can check by looking at the service file that is installed to either /usr/lib/systemd or /lib/systemd depending on your distribution of choice.

singlenode:~ # cat /usr/lib/systemd/system/open-xchange.service
[Unit]
After=remote-fs.target
After=time-sync.target ypbind.service sendmail.service cyrus.service

[Service]
User=open-xchange
PermissionsStartOnly=true
TimeoutStartSec=0
ExecStartPre=/opt/open-xchange/sbin/triggerupdatethemes -u
ExecStart=/opt/open-xchange/sbin/open-xchange
ExecStop=/opt/open-xchange/sbin/shutdown -w
ExecReload=/opt/open-xchange/sbin/triggerreloadconfiguration -d
KillMode=process
LimitNOFILE=65536
LimitNPROC=65536

[Install]
WantedBy=multi-user.target

==== Drop-in configs ====
Drop in configs allow administrators to easily adapt and override default service unit files. So if you want to change the default limits or add additional limits have a look at our default example drop-in config which is located at /etc/systemd/system/open-xchange.service.d/limits.conf

singlenode:~ # cat /etc/systemd/system/open-xchange.service.d/limits.conf
# Override and add options in this file
# See systemd.exec(5) for other limits

[Service]
#LimitNPROC=65536
#LimitNOFILE=65536

The systemd.exec documentation shows a whole lot of options that can be used by admins to adapt the default service to their specific needs.

 
[[File:mw_start_sysd.png|thumb|Service startup using systemd|left|1000px]]
 

== System V / Upstart ==

==== Service definition ====

The default service definition as shipped by Open-Xchange can be found at /etc/init.d/open-xchange and will differ slightly depending on the distro in use.

 
[[File:mw_start_sysv.png|thumb|Service startup using upstart|left|1000px]]
 
== Apply Resource limits ==

As you can see in the diagrams above one of the differences is that:

* systemd reads all the infos wrt. resource limitations from the original service file and then applies possible further restrictions from the drop-in configs (steps 2 and 8 in the diagram ''Service startup using systemd'') and limits configured in /opt/open-xchange/etc/ox-scriptconf.sh aren't considered anylonger
* the older System V style init has to source /opt/open-xchange/etc/ox-scriptconf.sh to be able to read the configs for the maximum number of processes (NPROC) and files(NRFILES) that the middleware shall be allowed to create/open (steps 3,4 and 15 in the diagram ''Service startup using upstart'').

=== Background Infos ===

Several ways exist to restrict resources on a linux system from a global level down to user/groups or even shells and the processes started by them.

==== Sysctl ====
Sysctl is used to modify kernel parameters at runtime. E.g. to set the maximum number of files

$ sysctl -w fs.file-max=100000

To permanently set them append to the main configuration file and reload the settings

$ echo fs.file-max=100000 >> /etc/sysctl.conf
$ sysctl -p

More infos can be found via '''man sysctl'''

==== Limits.conf ====

Allows to restrict resources an a global, group or user level. E.g:

$ cat /etc/security/limits.d/90-nproc.conf
# Default limit for number of user's processes to prevent
# accidental fork bombs.
# See rhbz #432903 for reasoning.

* soft nproc 1024

From '''man limits.conf''':
<blockquote>
''Also, please note that all limit settings are set per login. They are not global, nor are they permanent; existing only for the duration of the session.''
</blockquote>

The limits per login are applied via the ''pam'' stack. See '''man pam''' and '''man pam_limits''' for more details. As those limits are bound to sessions they don't affect most daemons started by our supported init systems or init utils. Most state that they are ignored by design, see [https://bugs.launchpad.net/ubuntu/+source/upstart/+bug/938669/comments/4 upstart], [https://bugzilla.redhat.com/show_bug.cgi?id=754285#c1 systemd] and [https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=302079 start-stop-daemon]

===== Ulimit =====
From '''man bash'''
<blockquote>
ulimit [-HSTabcdefilmnpqrstuvx [limit]]
Provides control over the resources available to the shell and to processes started by it, on systems that allow such control.
</blockquote>
This is what we use in our System V compatible init scripts to increase resources for the open-xchange process across multiple distros. Currently only ''the maximum number of processes'' and ''the maximum number of open file descriptors available to a single user'' are increased via ulimit. The values are specified in '''/opt/open-xchange/ox-scriptconf.sh'''

==== Systemd ====

===== Control Groups=====

Control groups should only affect the OX middleware if you create/manage them yourself of if you are using a ''modern'' distribution that already uses systemd as init.

Citing from the kernel cgroup [https://www.kernel.org/doc/Documentation/cgroup-v2.txt documentation]:

<blockquote>
1-2. What is cgroup?

cgroup is a mechanism to organize processes hierarchically and
distribute system resources along the hierarchy in a controlled and
configurable manner.

cgroup is largely composed of two parts - the core and controllers.
cgroup core is primarily responsible for hierarchically organizing
processes. A cgroup controller is usually responsible for
distributing a specific type of system resource along the hierarchy
although there are utility controllers which serve purposes other than
resource distribution.

cgroups form a tree structure and every process in the system belongs
to one and only one cgroup. All threads of a process belong to the
same cgroup. On creation, all processes are put in the cgroup that
the parent process belongs to at the time. A process can be migrated
to another cgroup. Migration of a process doesn't affect already
existing descendant processes.

Following certain structural constraints, controllers may be enabled or
disabled selectively on a cgroup. All controller behaviors are
hierarchical - if a controller is enabled on a cgroup, it affects all
processes which belong to the cgroups consisting the inclusive
sub-hierarchy of the cgroup. When a controller is enabled on a nested
cgroup, it always restricts the resource distribution further. The
restrictions set closer to the root in the hierarchy can not be
overridden from further away.
</blockquote>

So processes are organized into a tree structure of control groups and controllers are responsible for the distribution of resources. So what kind of controllers exist?

<blockquote>
5. Controllers

5-1. CPU

The "cpu" controllers regulates distribution of CPU cycles. This
controller implements weight and absolute bandwidth limit models for
normal scheduling policy and absolute bandwidth allocation model for
realtime scheduling policy.

5-2. Memory

The "memory" controller regulates distribution of memory.
...
While not completely water-tight, all major memory usages by a given
cgroup are tracked so that the total memory consumption can be
accounted and controlled to a reasonable extent.

5-3. IO

The "io" controller regulates the distribution of IO resources. This
controller implements both weight based and absolute bandwidth or IOPS
limit distribution; however, weight based distribution is available
only if cfq-iosched is in use and neither scheme is available for
blk-mq devices.
</blockquote>

The open-xchange service is simply put into the default ''system.slice'' without applying further limits.

singlenode$ systemd-cgls
├─1 /sbin/init
├─system.slice
│ ├─avahi-daemon.service
│ │ ├─501 avahi-daemon: running [singlenode]
│ │ └─514 avahi-daemon: chroot helper
│ ├─console-kit-daemon.service
│ │ └─16164 /usr/sbin/console-kit-daemon --no-daemon
│ ├─dbus.service
│ │ └─508 /usr/bin/dbus-daemon --system --address=systemd: --nofork --nopidfile --systemd-activation
│ ├─munin-node.service
│ │ └─4290 /usr/bin/perl -wT /usr/sbin/munin-node
│ ├─open-xchange.service
│ │ └─6037 /usr/bin/java -Dsun.net.inetaddr.ttl=3600 -Dnetworkaddress.cache.ttl=3600 -Dnetworkaddress.cache.negative.ttl=10 ...

To check all the details use

singlenode:~ # systemctl show system.slice

==== Limits besides control groups ====
Besides control groups systemd allows you to apply other limits to the execution environment of your service. Here we can apply the limits that would normally be applied via ''limits.conf'' or ''ulimit''. Systemd uses [http://man7.org/linux/man-pages/man2/setrlimit.2.html setrlimit] for this. The options that we set by default are:

* LimitNOFILE
* LimitNPROC

which you have already seen when we took a look at the [[#Service_definiton|service definition]] and [[#Drop-in_config|drop-in configs]]

== Verify limits ==

=== System V ===

Get the proper pid of the java process started for the middleware by either checking '''ps faux'''
root 4103 0.0 0.0 66528 1868 pts/0 S 22:05 0:00 su -s /bin/bash open-xchange -c /opt/open-xchange/sbin/open-xchange
494 4109 9.5 19.8 2224644 381312 ? Ssl 22:05 1:12 \_ /usr/bin/java -Dsun.net.inetaddr.ttl=3600 -Dnetworkaddress.cache.ttl=3600

or programmatically
singlenode:~ # pid=$(pgrep -P $(</var/run/open-xchange.pid) java)

and check the limits applied for that process
singlenode:~ # cat /proc/$pid/limits
Limit Soft Limit Hard Limit Units
Max cpu time unlimited unlimited seconds
Max file size unlimited unlimited bytes
Max data size unlimited unlimited bytes
Max stack size 8388608 unlimited bytes
Max core file size 0 unlimited bytes
Max resident set unlimited unlimited bytes
Max processes 65536 65536 processes
Max open files 65536 65536 files
Max locked memory 65536 65536 bytes
Max address space unlimited unlimited bytes
Max file locks unlimited unlimited locks
Max pending signals 24254 24254 signals
Max msgqueue size 819200 819200 bytes
Max nice priority 0 0
Max realtime priority 0 0
Max realtime timeout unlimited unlimited us

=== Systemd ===
singlenode:~ # systemctl show open-xchange | grep Limit
StartLimitInterval=10000000
StartLimitBurst=5
StartLimitAction=none
MemoryLimit=18446744073709551615
LimitCPU=18446744073709551615
LimitFSIZE=18446744073709551615
LimitDATA=18446744073709551615
LimitSTACK=18446744073709551615
LimitCORE=18446744073709551615
LimitRSS=18446744073709551615
LimitNOFILE=65536
LimitAS=18446744073709551615
LimitNPROC=65536
LimitMEMLOCK=65536
LimitLOCKS=18446744073709551615
LimitSIGPENDING=19827
LimitMSGQUEUE=819200
LimitNICE=0
LimitRTPRIO=0
LimitRTTIME=18446744073709551615

== Evaluate JVM commandline options ==

Before starting Java and the OSGi framework with all the Open-Xchange middleware bundles we have to evaluate several important JVM commandline options that influence the performance and stability of the Open-Xchange middleware stack. These commandline options are specified in '''/opt/open-xchange/etc/ox-scriptconf.sh'''

=== Format < 7.8.4 ===
For version up to 7.8.3 the JVM configuration options were all specified via a single property '''JAVA_XTRAOPTS''' which resulted in a rather long configuration string that was hard to read and maintain.

JAVA_XTRAOPTS="-Dsun.net.inetaddr.ttl=3600 -Dnetworkaddress.cache.ttl=3600 -Dnetworkaddress.cache.negative.ttl=10 \
-Dlogback.threadlocal.put.duplicate=false -server -Djava.awt.headless=true -Xmx512M -XX:MaxPermSize=256m -XX:+UseConcMarkSweepGC \
-XX:+UseParNewGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:NewRatio=3 -XX:+UseTLAB \
-Dosgi.compatibility.bootdelegation=false -XX:-OmitStackTraceInFastThrow"

During the startup of the open-xchange service (step 11 of diagram Service startup using systemd and step 10 of diagram Service startup using upstart.) this JAVA_XTRAOPTS string goes through a basic validity check and is then passed to the JVM.

=== Format >= 7.8.4 ===
Beginning from version 7.8.4 the options are split up and sorted into different categories like:

JAVA_OPTS_GC="-XX:+UseConcMarkSweepGC -XX:+UseParNewGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:NewRatio=3 -XX:+DisableExplicitGC"
JAVA_OPTS_LOG="-Dlogback.threadlocal.put.duplicate=false -XX:-OmitStackTraceInFastThrow"
JAVA_OPTS_MEM="-XX:MaxHeapSize=512M -XX:MaxPermSize=256M -XX:+UseTLAB"
JAVA_OPTS_NET="-Dsun.net.inetaddr.ttl=3600 -Dnetworkaddress.cache.ttl=3600 -Dnetworkaddress.cache.negative.ttl=10"
JAVA_OPTS_OSGI="-Dosgi.compatibility.bootdelegation=false"
JAVA_OPTS_SERVER="-server -Djava.awt.headless=true"

JAVA_OPTS_OTHER=""

#JAVA_OPTS_DEBUG="-XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/mnt/heapdump -Xloggc:/var/log/open-xchange/gc.log -verbose:gc -XX:+PrintGCDateStamps \
-XX:+PrintHeapAtGC -XX:+PrintGCApplicationStoppedTime -XX:+PrintTenuringDistribution"

which is much easier to read an maintain. Furthermore we added a useful '''JAVA_OPTS_DEBUG''' configuration parameter that comes preconfigured but unused/commented so that customers can quickly enable the most commonly used debugging options in case they have to provide detailed system information to Open-Xchange.

During the startup of the open-xchange service (step 11 of diagram ''Service startup using systemd'' and step 10 of diagram ''Service startup using upstart''.) all uncommented JAVA_OPTS_* categories will then be evaluated and combined again into a single string of commandline options that can then be passed to the JVM.

=== Upgrade procedure ===
During the upgrade '''7.8.3-revX -> 7.8.4-revY''' existing JAVA_XTRAOPTS will automatically be split up and migrated to the new categories approach. A backup of the file (ox-scriptconf.sh.timestamp) will be created and admins are urged to check for proper migration before deleting the backup file like it's already done with *.dpkg-dist or *.rpmnew files on Debian or RHEL/SLES distributions.

== Encoding detection during JVM startup ==

=== Background Infos ===
All of en_US.UTF-8, .utf8, .utf-8, .UTF_8 will result in a valid UTF-8 charmap

==== JVM startup ====

The default file.encoding detection is done at JVM startup based upon LC_CTYPE which makes sense as it's documented as:

CODESET (LC_CTYPE)
Return a string with the name of the character encoding used in the selected locale, such as "UTF-8", "ISO-8859-1",
or "ANSI_X3.4-1968" (better known as US-ASCII). This is the same string that you get with "locale charmap".
For a list of character encoding names, try "locale -m", cf. locale(1).

Furthermore this influences the defaultCharset selection of the JVM.

/**
* Returns the default charset of this Java virtual machine.
*
* The default charset is determined during virtual-machine startup and
* typically depends upon the locale and charset of the underlying
* operating system.
*
* @return A charset object for the default charset
*
* @since 1.5
*/
public static Charset defaultCharset() {
if (defaultCharset == null) {
synchronized (Charset.class) {
String csn = AccessController.doPrivileged(
new GetPropertyAction("file.encoding"));
Charset cs = lookup(csn);
if (cs != null)
defaultCharset = cs;
else
defaultCharset = forName("UTF-8");
}
}
return defaultCharset;
}

Although we try to specify valid unicode encodings for every IO operations the Open-Xchange middleware does some parts of the JVM don't allow to specify the encoding to use and falls back to the detected default encoding. That's why admins have to make sure to provide a proper unicode environment for the middleware service during every start or use documented workarounds provided by Open-Xchange (e.g. )

=== Query which file encoding was detected by the JVM during startup ===

/opt/open-xchange/sbin/showruntimestats -r | sed -ne 's/.*${key=file.encoding, value=[^}]*}$.*/\1/p'
{key=file.encoding, value=UTF-8}

=== Locale passing from client to server via SSH ===

The following setting of sshd allows clients to specify which locale to use on the server and might lead to some confusion or unexpected behaviour when checking the system locale or even to the usage of broken locales during middleware startup if your client sends misconfigured or missing locale configuration to the server.

grep AcceptEnv /etc/ssh/sshd_config
AcceptEnv LANG LC_*

For example when i connect to the machine my local german locale is sent to the server

[vagrant@showcase-community-centos-7 ~]$ locale -v
LANG=de_DE.UTF-8
LC_CTYPE="de_DE.UTF-8"
LC_NUMERIC=de_DE.UTF-8
LC_TIME=de_DE.UTF-8
LC_COLLATE="de_DE.UTF-8"
LC_MONETARY=de_DE.UTF-8
LC_MESSAGES="de_DE.UTF-8"
LC_PAPER=de_DE.UTF-8
LC_NAME=de_DE.UTF-8
LC_ADDRESS=de_DE.UTF-8
LC_TELEPHONE=de_DE.UTF-8
LC_MEASUREMENT=de_DE.UTF-8
LC_IDENTIFICATION=de_DE.UTF-8
LC_ALL=

But after specifying that i want a new login shell the default locale configured for that user/system will be used instead!

[root@showcase-community-centos-7 vagrant]# su -s /bin/bash open-xchange -l
-bash-4.2$ locale -v
LANG=en_US.UTF-8
LC_CTYPE="en_US.UTF-8"
LC_NUMERIC="en_US.UTF-8"
LC_TIME="en_US.UTF-8"
LC_COLLATE="en_US.UTF-8"
LC_MONETARY="en_US.UTF-8"
LC_MESSAGES="en_US.UTF-8"
LC_PAPER="en_US.UTF-8"
LC_NAME="en_US.UTF-8"
LC_ADDRESS="en_US.UTF-8"
LC_TELEPHONE="en_US.UTF-8"
LC_MEASUREMENT="en_US.UTF-8"
LC_IDENTIFICATION="en_US.UTF-8"
LC_ALL=

Add details from https://bugs.open-xchange.com/show_bug.cgi?id=51074

== Warnings during early start ==

Early warnings before the start of the JVM and related logging libraries like logback and logstash forwarding are redirected to '''/var/log/open-xchange/open-xchange-console.log'''.

=== Limits ===
If the open-xchange startup script isn't able to set the proper limits on CentOS 6 due to e.g. hard limits being enforced via pam_limits you'll see the following warning in the console log during startup

singlenode open-xchange # cat open-xchange-console.log
/opt/open-xchange/sbin/open-xchange: line 115: ulimit: max user processes: cannot modify limit: Operation not permitted

=== Locale ===
If the open-xchange startup script isn't able to detect a unicode locale you'll see a warning like the following (the current charmap might differ)

WARNING: Couldn't detect proper unicode charmap in locale category LC_CTYPE.
WARNING: Current charmap: "ANSI_X3.4-1968" might cause encoding problems.

== Summary: Open-Xchange middleware on specific distros ==
The support for the mentioned mechanism of resource control differ depending on the distribution and the init system in use.

=== Debian 7 ===
;Init
: System V style
;OX Configurable Limits/Defaults
: nofile, nproc

The mentioned limits can be configured via '''/opt/open-xchange/etc/ox-scriptconf.sh'''. The limits are applied via ulimit in the service's init script. The open-xchange service is finally started via start-stop-daemon which doesn't doesn't consider '''/etc/security/limits.*'''

=== RHEL 6 / CentOS 6 ===
;Init
: Upstart, System V compatible
;OX Configurable Limits/Defaults
: nofile, nproc

The mentioned limits can be configured via '''/opt/open-xchange/etc/ox-scriptconf.sh'''. The limits are applied via ulimit in the service's init script. Furthermore as the open-xchange service is finally started via '''su ... open-xchange''' on this distro a user session is opened via su/pam and the default CentOS pam config reads the '''/etc/security/limits.*''' configuration by loading the pam stack like:

: '''/etc/pam.d/su'''
:: '''-> /etc/pam.d/system-auth'''
::: '''-> pam_limits.so'''

If NPROC isn't configured for the open-xchange-server it's restricted to '''1024''' globally by default to prevent accidental fork bombs, see '''/etc/security/limits.d/90-nproc.conf''' which can result in severe problems modern multithreaded applications.

=== RHEL 7 / CentOS 7 / Debian 8 / SLE 12 ===
;Init
: Systemd
;OX Configurable Limits/Defaults
: nofile, nproc

For systemd the default limits are configured directly in the service's unit file that is shipped by OX and located at '''/usr/lib/systemd/system/open-xchange.service'''. The drop-in config to override or extend the default unit file is located at '''/etc/systemd/system/open-xchange.service.d/limits.conf'''. [https://www.freedesktop.org/software/systemd/man/systemd.exec.html Systemd.exec] shows a whole lot of options that can be used by admins to adapt the default service to their specific needs.

Sandbox:Middleware Startup/sandbox

2017-04-24T19:44:39Z

Marens: /* Apply Resource limits */

== Systemd ==

==== Service definition ====
The default service definition as shipped by Open-Xchange is rather concise and easy to read. You can check by looking at the service file that is installed to either /usr/lib/systemd or /lib/systemd depending on your distribution of choice.

singlenode:~ # cat /usr/lib/systemd/system/open-xchange.service
[Unit]
After=remote-fs.target
After=time-sync.target ypbind.service sendmail.service cyrus.service

[Service]
User=open-xchange
PermissionsStartOnly=true
TimeoutStartSec=0
ExecStartPre=/opt/open-xchange/sbin/triggerupdatethemes -u
ExecStart=/opt/open-xchange/sbin/open-xchange
ExecStop=/opt/open-xchange/sbin/shutdown -w
ExecReload=/opt/open-xchange/sbin/triggerreloadconfiguration -d
KillMode=process
LimitNOFILE=65536
LimitNPROC=65536

[Install]
WantedBy=multi-user.target

==== Drop-in configs ====
Drop in configs allow administrators to easily adapt and override default service unit files. So if you want to change the default limits or add additional limits have a look at our default example drop-in config which is located at /etc/systemd/system/open-xchange.service.d/limits.conf

singlenode:~ # cat /etc/systemd/system/open-xchange.service.d/limits.conf
# Override and add options in this file
# See systemd.exec(5) for other limits

[Service]
#LimitNPROC=65536
#LimitNOFILE=65536

The systemd.exec documentation shows a whole lot of options that can be used by admins to adapt the default service to their specific needs.

 
[[File:mw_start_sysd.png|thumb|Service startup using systemd|left|1000px]]
 

== System V / Upstart ==

==== Service definition ====

The default service definition as shipped by Open-Xchange can be found at /etc/init.d/open-xchange and will differ slightly depending on the distro in use.

 
[[File:mw_start_sysv.png|thumb|Service startup using upstart|left|1000px]]
 
== Apply Resource limits ==

As you can see in the diagrams above one of the differences is that:

* systemd reads all the infos wrt. resource limitations from the original service file and then applies possible further restrictions from the drop-in configs (steps 2 and 8 in the diagram ''Service startup using systemd'') and limits configured in /opt/open-xchange/etc/ox-scriptconf.sh aren't considered anylonger when using systemd
* the older System V style init has to source /opt/open-xchange/etc/ox-scriptconf.sh to be able to read the configs for the maximum number of processes (NPROC) and files(NRFILES) that the middleware shall be allowed to create/open (steps 3,4 and 15 in the diagram ''Service startup using upstart'').

=== Background Infos ===

Several ways exist to restrict resources on a linux system from a global level down to user/groups or even shells and the processes started by them.

==== Sysctl ====
Sysctl is used to modify kernel parameters at runtime. E.g. to set the maximum number of files

$ sysctl -w fs.file-max=100000

To permanently set them append to the main configuration file and reload the settings

$ echo fs.file-max=100000 >> /etc/sysctl.conf
$ sysctl -p

More infos can be found via '''man sysctl'''

==== Limits.conf ====

Allows to restrict resources an a global, group or user level. E.g:

$ cat /etc/security/limits.d/90-nproc.conf
# Default limit for number of user's processes to prevent
# accidental fork bombs.
# See rhbz #432903 for reasoning.

* soft nproc 1024

From '''man limits.conf''':
<blockquote>
''Also, please note that all limit settings are set per login. They are not global, nor are they permanent; existing only for the duration of the session.''
</blockquote>

The limits per login are applied via the ''pam'' stack. See '''man pam''' and '''man pam_limits''' for more details. As those limits are bound to sessions they don't affect most daemons started by our supported init systems or init utils. Most state that they are ignored by design, see [https://bugs.launchpad.net/ubuntu/+source/upstart/+bug/938669/comments/4 upstart], [https://bugzilla.redhat.com/show_bug.cgi?id=754285#c1 systemd] and [https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=302079 start-stop-daemon]

===== Ulimit =====
From '''man bash'''
<blockquote>
ulimit [-HSTabcdefilmnpqrstuvx [limit]]
Provides control over the resources available to the shell and to processes started by it, on systems that allow such control.
</blockquote>
This is what we use in our System V compatible init scripts to increase resources for the open-xchange process across multiple distros. Currently only ''the maximum number of processes'' and ''the maximum number of open file descriptors available to a single user'' are increased via ulimit. The values are specified in '''/opt/open-xchange/ox-scriptconf.sh'''

==== Systemd ====

===== Control Groups=====

Control groups should only affect the OX middleware if you create/manage them yourself of if you are using a ''modern'' distribution that already uses systemd as init.

Citing from the kernel cgroup [https://www.kernel.org/doc/Documentation/cgroup-v2.txt documentation]:

<blockquote>
1-2. What is cgroup?

cgroup is a mechanism to organize processes hierarchically and
distribute system resources along the hierarchy in a controlled and
configurable manner.

cgroup is largely composed of two parts - the core and controllers.
cgroup core is primarily responsible for hierarchically organizing
processes. A cgroup controller is usually responsible for
distributing a specific type of system resource along the hierarchy
although there are utility controllers which serve purposes other than
resource distribution.

cgroups form a tree structure and every process in the system belongs
to one and only one cgroup. All threads of a process belong to the
same cgroup. On creation, all processes are put in the cgroup that
the parent process belongs to at the time. A process can be migrated
to another cgroup. Migration of a process doesn't affect already
existing descendant processes.

Following certain structural constraints, controllers may be enabled or
disabled selectively on a cgroup. All controller behaviors are
hierarchical - if a controller is enabled on a cgroup, it affects all
processes which belong to the cgroups consisting the inclusive
sub-hierarchy of the cgroup. When a controller is enabled on a nested
cgroup, it always restricts the resource distribution further. The
restrictions set closer to the root in the hierarchy can not be
overridden from further away.
</blockquote>

So processes are organized into a tree structure of control groups and controllers are responsible for the distribution of resources. So what kind of controllers exist?

<blockquote>
5. Controllers

5-1. CPU

The "cpu" controllers regulates distribution of CPU cycles. This
controller implements weight and absolute bandwidth limit models for
normal scheduling policy and absolute bandwidth allocation model for
realtime scheduling policy.

5-2. Memory

The "memory" controller regulates distribution of memory.
...
While not completely water-tight, all major memory usages by a given
cgroup are tracked so that the total memory consumption can be
accounted and controlled to a reasonable extent.

5-3. IO

The "io" controller regulates the distribution of IO resources. This
controller implements both weight based and absolute bandwidth or IOPS
limit distribution; however, weight based distribution is available
only if cfq-iosched is in use and neither scheme is available for
blk-mq devices.
</blockquote>

The open-xchange service is simply put into the default ''system.slice'' without applying further limits.

singlenode$ systemd-cgls
├─1 /sbin/init
├─system.slice
│ ├─avahi-daemon.service
│ │ ├─501 avahi-daemon: running [singlenode]
│ │ └─514 avahi-daemon: chroot helper
│ ├─console-kit-daemon.service
│ │ └─16164 /usr/sbin/console-kit-daemon --no-daemon
│ ├─dbus.service
│ │ └─508 /usr/bin/dbus-daemon --system --address=systemd: --nofork --nopidfile --systemd-activation
│ ├─munin-node.service
│ │ └─4290 /usr/bin/perl -wT /usr/sbin/munin-node
│ ├─open-xchange.service
│ │ └─6037 /usr/bin/java -Dsun.net.inetaddr.ttl=3600 -Dnetworkaddress.cache.ttl=3600 -Dnetworkaddress.cache.negative.ttl=10 ...

To check all the details use

singlenode:~ # systemctl show system.slice

==== Limits besides control groups ====
Besides control groups systemd allows you to apply other limits to the execution environment of your service. Here we can apply the limits that would normally be applied via ''limits.conf'' or ''ulimit''. Systemd uses [http://man7.org/linux/man-pages/man2/setrlimit.2.html setrlimit] for this. The options that we set by default are:

* LimitNOFILE
* LimitNPROC

which you have already seen when we took a look at the [[#Service_definiton|service definition]] and [[#Drop-in_config|drop-in configs]]

== Verify limits ==

=== System V ===

Get the proper pid of the java process started for the middleware by either checking '''ps faux'''
root 4103 0.0 0.0 66528 1868 pts/0 S 22:05 0:00 su -s /bin/bash open-xchange -c /opt/open-xchange/sbin/open-xchange
494 4109 9.5 19.8 2224644 381312 ? Ssl 22:05 1:12 \_ /usr/bin/java -Dsun.net.inetaddr.ttl=3600 -Dnetworkaddress.cache.ttl=3600

or programmatically
singlenode:~ # pid=$(pgrep -P $(</var/run/open-xchange.pid) java)

and check the limits applied for that process
singlenode:~ # cat /proc/$pid/limits
Limit Soft Limit Hard Limit Units
Max cpu time unlimited unlimited seconds
Max file size unlimited unlimited bytes
Max data size unlimited unlimited bytes
Max stack size 8388608 unlimited bytes
Max core file size 0 unlimited bytes
Max resident set unlimited unlimited bytes
Max processes 65536 65536 processes
Max open files 65536 65536 files
Max locked memory 65536 65536 bytes
Max address space unlimited unlimited bytes
Max file locks unlimited unlimited locks
Max pending signals 24254 24254 signals
Max msgqueue size 819200 819200 bytes
Max nice priority 0 0
Max realtime priority 0 0
Max realtime timeout unlimited unlimited us

=== Systemd ===
singlenode:~ # systemctl show open-xchange | grep Limit
StartLimitInterval=10000000
StartLimitBurst=5
StartLimitAction=none
MemoryLimit=18446744073709551615
LimitCPU=18446744073709551615
LimitFSIZE=18446744073709551615
LimitDATA=18446744073709551615
LimitSTACK=18446744073709551615
LimitCORE=18446744073709551615
LimitRSS=18446744073709551615
LimitNOFILE=65536
LimitAS=18446744073709551615
LimitNPROC=65536
LimitMEMLOCK=65536
LimitLOCKS=18446744073709551615
LimitSIGPENDING=19827
LimitMSGQUEUE=819200
LimitNICE=0
LimitRTPRIO=0
LimitRTTIME=18446744073709551615

== Evaluate JVM commandline options ==

Before starting Java and the OSGi framework with all the Open-Xchange middleware bundles we have to evaluate several important JVM commandline options that influence the performance and stability of the Open-Xchange middleware stack. These commandline options are specified in '''/opt/open-xchange/etc/ox-scriptconf.sh'''

=== Format < 7.8.4 ===
For version up to 7.8.3 the JVM configuration options were all specified via a single property '''JAVA_XTRAOPTS''' which resulted in a rather long configuration string that was hard to read and maintain.

JAVA_XTRAOPTS="-Dsun.net.inetaddr.ttl=3600 -Dnetworkaddress.cache.ttl=3600 -Dnetworkaddress.cache.negative.ttl=10 \
-Dlogback.threadlocal.put.duplicate=false -server -Djava.awt.headless=true -Xmx512M -XX:MaxPermSize=256m -XX:+UseConcMarkSweepGC \
-XX:+UseParNewGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:NewRatio=3 -XX:+UseTLAB \
-Dosgi.compatibility.bootdelegation=false -XX:-OmitStackTraceInFastThrow"

During the startup of the open-xchange service (step 11 of diagram Service startup using systemd and step 10 of diagram Service startup using upstart.) this JAVA_XTRAOPTS string goes through a basic validity check and is then passed to the JVM.

=== Format >= 7.8.4 ===
Beginning from version 7.8.4 the options are split up and sorted into different categories like:

JAVA_OPTS_GC="-XX:+UseConcMarkSweepGC -XX:+UseParNewGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:NewRatio=3 -XX:+DisableExplicitGC"
JAVA_OPTS_LOG="-Dlogback.threadlocal.put.duplicate=false -XX:-OmitStackTraceInFastThrow"
JAVA_OPTS_MEM="-XX:MaxHeapSize=512M -XX:MaxPermSize=256M -XX:+UseTLAB"
JAVA_OPTS_NET="-Dsun.net.inetaddr.ttl=3600 -Dnetworkaddress.cache.ttl=3600 -Dnetworkaddress.cache.negative.ttl=10"
JAVA_OPTS_OSGI="-Dosgi.compatibility.bootdelegation=false"
JAVA_OPTS_SERVER="-server -Djava.awt.headless=true"

JAVA_OPTS_OTHER=""

#JAVA_OPTS_DEBUG="-XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/mnt/heapdump -Xloggc:/var/log/open-xchange/gc.log -verbose:gc -XX:+PrintGCDateStamps \
-XX:+PrintHeapAtGC -XX:+PrintGCApplicationStoppedTime -XX:+PrintTenuringDistribution"

which is much easier to read an maintain. Furthermore we added a useful '''JAVA_OPTS_DEBUG''' configuration parameter that comes preconfigured but unused/commented so that customers can quickly enable the most commonly used debugging options in case they have to provide detailed system information to Open-Xchange.

During the startup of the open-xchange service (step 11 of diagram ''Service startup using systemd'' and step 10 of diagram ''Service startup using upstart''.) all uncommented JAVA_OPTS_* categories will then be evaluated and combined again into a single string of commandline options that can then be passed to the JVM.

=== Upgrade procedure ===
During the upgrade '''7.8.3-revX -> 7.8.4-revY''' existing JAVA_XTRAOPTS will automatically be split up and migrated to the new categories approach. A backup of the file (ox-scriptconf.sh.timestamp) will be created and admins are urged to check for proper migration before deleting the backup file like it's already done with *.dpkg-dist or *.rpmnew files on Debian or RHEL/SLES distributions.

== Encoding detection during JVM startup ==

=== Background Infos ===
All of en_US.UTF-8, .utf8, .utf-8, .UTF_8 will result in a valid UTF-8 charmap

==== JVM startup ====

The default file.encoding detection is done at JVM startup based upon LC_CTYPE which makes sense as it's documented as:

CODESET (LC_CTYPE)
Return a string with the name of the character encoding used in the selected locale, such as "UTF-8", "ISO-8859-1",
or "ANSI_X3.4-1968" (better known as US-ASCII). This is the same string that you get with "locale charmap".
For a list of character encoding names, try "locale -m", cf. locale(1).

Furthermore this influences the defaultCharset selection of the JVM.

/**
* Returns the default charset of this Java virtual machine.
*
* The default charset is determined during virtual-machine startup and
* typically depends upon the locale and charset of the underlying
* operating system.
*
* @return A charset object for the default charset
*
* @since 1.5
*/
public static Charset defaultCharset() {
if (defaultCharset == null) {
synchronized (Charset.class) {
String csn = AccessController.doPrivileged(
new GetPropertyAction("file.encoding"));
Charset cs = lookup(csn);
if (cs != null)
defaultCharset = cs;
else
defaultCharset = forName("UTF-8");
}
}
return defaultCharset;
}

Although we try to specify valid unicode encodings for every IO operations the Open-Xchange middleware does some parts of the JVM don't allow to specify the encoding to use and falls back to the detected default encoding. That's why admins have to make sure to provide a proper unicode environment for the middleware service during every start or use documented workarounds provided by Open-Xchange (e.g. )

=== Query which file encoding was detected by the JVM during startup ===

/opt/open-xchange/sbin/showruntimestats -r | sed -ne 's/.*${key=file.encoding, value=[^}]*}$.*/\1/p'
{key=file.encoding, value=UTF-8}

=== Locale passing from client to server via SSH ===

The following setting of sshd allows clients to specify which locale to use on the server and might lead to some confusion or unexpected behaviour when checking the system locale or even to the usage of broken locales during middleware startup if your client sends misconfigured or missing locale configuration to the server.

grep AcceptEnv /etc/ssh/sshd_config
AcceptEnv LANG LC_*

For example when i connect to the machine my local german locale is sent to the server

[vagrant@showcase-community-centos-7 ~]$ locale -v
LANG=de_DE.UTF-8
LC_CTYPE="de_DE.UTF-8"
LC_NUMERIC=de_DE.UTF-8
LC_TIME=de_DE.UTF-8
LC_COLLATE="de_DE.UTF-8"
LC_MONETARY=de_DE.UTF-8
LC_MESSAGES="de_DE.UTF-8"
LC_PAPER=de_DE.UTF-8
LC_NAME=de_DE.UTF-8
LC_ADDRESS=de_DE.UTF-8
LC_TELEPHONE=de_DE.UTF-8
LC_MEASUREMENT=de_DE.UTF-8
LC_IDENTIFICATION=de_DE.UTF-8
LC_ALL=

But after specifying that i want a new login shell the default locale configured for that user/system will be used instead!

[root@showcase-community-centos-7 vagrant]# su -s /bin/bash open-xchange -l
-bash-4.2$ locale -v
LANG=en_US.UTF-8
LC_CTYPE="en_US.UTF-8"
LC_NUMERIC="en_US.UTF-8"
LC_TIME="en_US.UTF-8"
LC_COLLATE="en_US.UTF-8"
LC_MONETARY="en_US.UTF-8"
LC_MESSAGES="en_US.UTF-8"
LC_PAPER="en_US.UTF-8"
LC_NAME="en_US.UTF-8"
LC_ADDRESS="en_US.UTF-8"
LC_TELEPHONE="en_US.UTF-8"
LC_MEASUREMENT="en_US.UTF-8"
LC_IDENTIFICATION="en_US.UTF-8"
LC_ALL=

Add details from https://bugs.open-xchange.com/show_bug.cgi?id=51074

== Warnings during early start ==

Early warnings before the start of the JVM and related logging libraries like logback and logstash forwarding are redirected to '''/var/log/open-xchange/open-xchange-console.log'''.

=== Limits ===
If the open-xchange startup script isn't able to set the proper limits on CentOS 6 due to e.g. hard limits being enforced via pam_limits you'll see the following warning in the console log during startup

singlenode open-xchange # cat open-xchange-console.log
/opt/open-xchange/sbin/open-xchange: line 115: ulimit: max user processes: cannot modify limit: Operation not permitted

=== Locale ===
If the open-xchange startup script isn't able to detect a unicode locale you'll see a warning like the following (the current charmap might differ)

WARNING: Couldn't detect proper unicode charmap in locale category LC_CTYPE.
WARNING: Current charmap: "ANSI_X3.4-1968" might cause encoding problems.

== Summary: Open-Xchange middleware on specific distros ==
The support for the mentioned mechanism of resource control differ depending on the distribution and the init system in use.

=== Debian 7 ===
;Init
: System V style
;OX Configurable Limits/Defaults
: nofile, nproc

The mentioned limits can be configured via '''/opt/open-xchange/etc/ox-scriptconf.sh'''. The limits are applied via ulimit in the service's init script. The open-xchange service is finally started via start-stop-daemon which doesn't doesn't consider '''/etc/security/limits.*'''

=== RHEL 6 / CentOS 6 ===
;Init
: Upstart, System V compatible
;OX Configurable Limits/Defaults
: nofile, nproc

The mentioned limits can be configured via '''/opt/open-xchange/etc/ox-scriptconf.sh'''. The limits are applied via ulimit in the service's init script. Furthermore as the open-xchange service is finally started via '''su ... open-xchange''' on this distro a user session is opened via su/pam and the default CentOS pam config reads the '''/etc/security/limits.*''' configuration by loading the pam stack like:

: '''/etc/pam.d/su'''
:: '''-> /etc/pam.d/system-auth'''
::: '''-> pam_limits.so'''

If NPROC isn't configured for the open-xchange-server it's restricted to '''1024''' globally by default to prevent accidental fork bombs, see '''/etc/security/limits.d/90-nproc.conf''' which can result in severe problems modern multithreaded applications.

=== RHEL 7 / CentOS 7 / Debian 8 / SLE 12 ===
;Init
: Systemd
;OX Configurable Limits/Defaults
: nofile, nproc

For systemd the default limits are configured directly in the service's unit file that is shipped by OX and located at '''/usr/lib/systemd/system/open-xchange.service'''. The drop-in config to override or extend the default unit file is located at '''/etc/systemd/system/open-xchange.service.d/limits.conf'''. [https://www.freedesktop.org/software/systemd/man/systemd.exec.html Systemd.exec] shows a whole lot of options that can be used by admins to adapt the default service to their specific needs.

Sandbox:Middleware Startup/sandbox

2017-04-24T19:41:02Z

Marens:

== Systemd ==

==== Service definition ====
The default service definition as shipped by Open-Xchange is rather concise and easy to read. You can check by looking at the service file that is installed to either /usr/lib/systemd or /lib/systemd depending on your distribution of choice.

singlenode:~ # cat /usr/lib/systemd/system/open-xchange.service
[Unit]
After=remote-fs.target
After=time-sync.target ypbind.service sendmail.service cyrus.service

[Service]
User=open-xchange
PermissionsStartOnly=true
TimeoutStartSec=0
ExecStartPre=/opt/open-xchange/sbin/triggerupdatethemes -u
ExecStart=/opt/open-xchange/sbin/open-xchange
ExecStop=/opt/open-xchange/sbin/shutdown -w
ExecReload=/opt/open-xchange/sbin/triggerreloadconfiguration -d
KillMode=process
LimitNOFILE=65536
LimitNPROC=65536

[Install]
WantedBy=multi-user.target

==== Drop-in configs ====
Drop in configs allow administrators to easily adapt and override default service unit files. So if you want to change the default limits or add additional limits have a look at our default example drop-in config which is located at /etc/systemd/system/open-xchange.service.d/limits.conf

singlenode:~ # cat /etc/systemd/system/open-xchange.service.d/limits.conf
# Override and add options in this file
# See systemd.exec(5) for other limits

[Service]
#LimitNPROC=65536
#LimitNOFILE=65536

The systemd.exec documentation shows a whole lot of options that can be used by admins to adapt the default service to their specific needs.

 
[[File:mw_start_sysd.png|thumb|Service startup using systemd|left|1000px]]
 

== System V / Upstart ==

==== Service definition ====

The default service definition as shipped by Open-Xchange can be found at /etc/init.d/open-xchange and will differ slightly depending on the distro in use.

 
[[File:mw_start_sysv.png|thumb|Service startup using upstart|left|1000px]]
 
== Apply Resource limits ==

As you can see in the two diagrams above one of the differences is that systemd reads all the infos wrt. resource limitations from the original service file and then applies possible further restrictions from the drop-in configs (steps 2 and 8 in the diagram). Limits configured in /opt/open-xchange/etc/ox-scriptconf.sh aren't considered anylonger when using systemd while the older System V style init has to source /opt/open-xchange/etc/ox-scriptconf.sh to be able to read the configs for the maximum number of processes (NPROC) and files(NRFILES) that the middleware shall be allowed to create/open (steps 3,4 and 15 in the diagram).

=== Background Infos ===

Several ways exist to restrict resources on a linux system from a global level down to user/groups or even shells and the processes started by them.

==== Sysctl ====
Sysctl is used to modify kernel parameters at runtime. E.g. to set the maximum number of files

$ sysctl -w fs.file-max=100000

To permanently set them append to the main configuration file and reload the settings

$ echo fs.file-max=100000 >> /etc/sysctl.conf
$ sysctl -p

More infos can be found via '''man sysctl'''

==== Limits.conf ====

Allows to restrict resources an a global, group or user level. E.g:

$ cat /etc/security/limits.d/90-nproc.conf
# Default limit for number of user's processes to prevent
# accidental fork bombs.
# See rhbz #432903 for reasoning.

* soft nproc 1024

From '''man limits.conf''':
<blockquote>
''Also, please note that all limit settings are set per login. They are not global, nor are they permanent; existing only for the duration of the session.''
</blockquote>

The limits per login are applied via the ''pam'' stack. See '''man pam''' and '''man pam_limits''' for more details. As those limits are bound to sessions they don't affect most daemons started by our supported init systems or init utils. Most state that they are ignored by design, see [https://bugs.launchpad.net/ubuntu/+source/upstart/+bug/938669/comments/4 upstart], [https://bugzilla.redhat.com/show_bug.cgi?id=754285#c1 systemd] and [https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=302079 start-stop-daemon]

===== Ulimit =====
From '''man bash'''
<blockquote>
ulimit [-HSTabcdefilmnpqrstuvx [limit]]
Provides control over the resources available to the shell and to processes started by it, on systems that allow such control.
</blockquote>
This is what we use in our System V compatible init scripts to increase resources for the open-xchange process across multiple distros. Currently only ''the maximum number of processes'' and ''the maximum number of open file descriptors available to a single user'' are increased via ulimit. The values are specified in '''/opt/open-xchange/ox-scriptconf.sh'''

==== Systemd ====

===== Control Groups=====

Control groups should only affect the OX middleware if you create/manage them yourself of if you are using a ''modern'' distribution that already uses systemd as init.

Citing from the kernel cgroup [https://www.kernel.org/doc/Documentation/cgroup-v2.txt documentation]:

<blockquote>
1-2. What is cgroup?

cgroup is a mechanism to organize processes hierarchically and
distribute system resources along the hierarchy in a controlled and
configurable manner.

cgroup is largely composed of two parts - the core and controllers.
cgroup core is primarily responsible for hierarchically organizing
processes. A cgroup controller is usually responsible for
distributing a specific type of system resource along the hierarchy
although there are utility controllers which serve purposes other than
resource distribution.

cgroups form a tree structure and every process in the system belongs
to one and only one cgroup. All threads of a process belong to the
same cgroup. On creation, all processes are put in the cgroup that
the parent process belongs to at the time. A process can be migrated
to another cgroup. Migration of a process doesn't affect already
existing descendant processes.

Following certain structural constraints, controllers may be enabled or
disabled selectively on a cgroup. All controller behaviors are
hierarchical - if a controller is enabled on a cgroup, it affects all
processes which belong to the cgroups consisting the inclusive
sub-hierarchy of the cgroup. When a controller is enabled on a nested
cgroup, it always restricts the resource distribution further. The
restrictions set closer to the root in the hierarchy can not be
overridden from further away.
</blockquote>

So processes are organized into a tree structure of control groups and controllers are responsible for the distribution of resources. So what kind of controllers exist?

<blockquote>
5. Controllers

5-1. CPU

The "cpu" controllers regulates distribution of CPU cycles. This
controller implements weight and absolute bandwidth limit models for
normal scheduling policy and absolute bandwidth allocation model for
realtime scheduling policy.

5-2. Memory

The "memory" controller regulates distribution of memory.
...
While not completely water-tight, all major memory usages by a given
cgroup are tracked so that the total memory consumption can be
accounted and controlled to a reasonable extent.

5-3. IO

The "io" controller regulates the distribution of IO resources. This
controller implements both weight based and absolute bandwidth or IOPS
limit distribution; however, weight based distribution is available
only if cfq-iosched is in use and neither scheme is available for
blk-mq devices.
</blockquote>

The open-xchange service is simply put into the default ''system.slice'' without applying further limits.

singlenode$ systemd-cgls
├─1 /sbin/init
├─system.slice
│ ├─avahi-daemon.service
│ │ ├─501 avahi-daemon: running [singlenode]
│ │ └─514 avahi-daemon: chroot helper
│ ├─console-kit-daemon.service
│ │ └─16164 /usr/sbin/console-kit-daemon --no-daemon
│ ├─dbus.service
│ │ └─508 /usr/bin/dbus-daemon --system --address=systemd: --nofork --nopidfile --systemd-activation
│ ├─munin-node.service
│ │ └─4290 /usr/bin/perl -wT /usr/sbin/munin-node
│ ├─open-xchange.service
│ │ └─6037 /usr/bin/java -Dsun.net.inetaddr.ttl=3600 -Dnetworkaddress.cache.ttl=3600 -Dnetworkaddress.cache.negative.ttl=10 ...

To check all the details use

singlenode:~ # systemctl show system.slice

==== Limits besides control groups ====
Besides control groups systemd allows you to apply other limits to the execution environment of your service. Here we can apply the limits that would normally be applied via ''limits.conf'' or ''ulimit''. Systemd uses [http://man7.org/linux/man-pages/man2/setrlimit.2.html setrlimit] for this. The options that we set by default are:

* LimitNOFILE
* LimitNPROC

which you have already seen when we took a look at the [[#Service_definiton|service definition]] and [[#Drop-in_config|drop-in configs]]

== Verify limits ==

=== System V ===

Get the proper pid of the java process started for the middleware by either checking '''ps faux'''
root 4103 0.0 0.0 66528 1868 pts/0 S 22:05 0:00 su -s /bin/bash open-xchange -c /opt/open-xchange/sbin/open-xchange
494 4109 9.5 19.8 2224644 381312 ? Ssl 22:05 1:12 \_ /usr/bin/java -Dsun.net.inetaddr.ttl=3600 -Dnetworkaddress.cache.ttl=3600

or programmatically
singlenode:~ # pid=$(pgrep -P $(</var/run/open-xchange.pid) java)

and check the limits applied for that process
singlenode:~ # cat /proc/$pid/limits
Limit Soft Limit Hard Limit Units
Max cpu time unlimited unlimited seconds
Max file size unlimited unlimited bytes
Max data size unlimited unlimited bytes
Max stack size 8388608 unlimited bytes
Max core file size 0 unlimited bytes
Max resident set unlimited unlimited bytes
Max processes 65536 65536 processes
Max open files 65536 65536 files
Max locked memory 65536 65536 bytes
Max address space unlimited unlimited bytes
Max file locks unlimited unlimited locks
Max pending signals 24254 24254 signals
Max msgqueue size 819200 819200 bytes
Max nice priority 0 0
Max realtime priority 0 0
Max realtime timeout unlimited unlimited us

=== Systemd ===
singlenode:~ # systemctl show open-xchange | grep Limit
StartLimitInterval=10000000
StartLimitBurst=5
StartLimitAction=none
MemoryLimit=18446744073709551615
LimitCPU=18446744073709551615
LimitFSIZE=18446744073709551615
LimitDATA=18446744073709551615
LimitSTACK=18446744073709551615
LimitCORE=18446744073709551615
LimitRSS=18446744073709551615
LimitNOFILE=65536
LimitAS=18446744073709551615
LimitNPROC=65536
LimitMEMLOCK=65536
LimitLOCKS=18446744073709551615
LimitSIGPENDING=19827
LimitMSGQUEUE=819200
LimitNICE=0
LimitRTPRIO=0
LimitRTTIME=18446744073709551615

== Evaluate JVM commandline options ==

Before starting Java and the OSGi framework with all the Open-Xchange middleware bundles we have to evaluate several important JVM commandline options that influence the performance and stability of the Open-Xchange middleware stack. These commandline options are specified in '''/opt/open-xchange/etc/ox-scriptconf.sh'''

=== Format < 7.8.4 ===
For version up to 7.8.3 the JVM configuration options were all specified via a single property '''JAVA_XTRAOPTS''' which resulted in a rather long configuration string that was hard to read and maintain.

JAVA_XTRAOPTS="-Dsun.net.inetaddr.ttl=3600 -Dnetworkaddress.cache.ttl=3600 -Dnetworkaddress.cache.negative.ttl=10 \
-Dlogback.threadlocal.put.duplicate=false -server -Djava.awt.headless=true -Xmx512M -XX:MaxPermSize=256m -XX:+UseConcMarkSweepGC \
-XX:+UseParNewGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:NewRatio=3 -XX:+UseTLAB \
-Dosgi.compatibility.bootdelegation=false -XX:-OmitStackTraceInFastThrow"

During the startup of the open-xchange service (step 11 of diagram Service startup using systemd and step 10 of diagram Service startup using upstart.) this JAVA_XTRAOPTS string goes through a basic validity check and is then passed to the JVM.

=== Format >= 7.8.4 ===
Beginning from version 7.8.4 the options are split up and sorted into different categories like:

JAVA_OPTS_GC="-XX:+UseConcMarkSweepGC -XX:+UseParNewGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:NewRatio=3 -XX:+DisableExplicitGC"
JAVA_OPTS_LOG="-Dlogback.threadlocal.put.duplicate=false -XX:-OmitStackTraceInFastThrow"
JAVA_OPTS_MEM="-XX:MaxHeapSize=512M -XX:MaxPermSize=256M -XX:+UseTLAB"
JAVA_OPTS_NET="-Dsun.net.inetaddr.ttl=3600 -Dnetworkaddress.cache.ttl=3600 -Dnetworkaddress.cache.negative.ttl=10"
JAVA_OPTS_OSGI="-Dosgi.compatibility.bootdelegation=false"
JAVA_OPTS_SERVER="-server -Djava.awt.headless=true"

JAVA_OPTS_OTHER=""

#JAVA_OPTS_DEBUG="-XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/mnt/heapdump -Xloggc:/var/log/open-xchange/gc.log -verbose:gc -XX:+PrintGCDateStamps \
-XX:+PrintHeapAtGC -XX:+PrintGCApplicationStoppedTime -XX:+PrintTenuringDistribution"

which is much easier to read an maintain. Furthermore we added a useful '''JAVA_OPTS_DEBUG''' configuration parameter that comes preconfigured but unused/commented so that customers can quickly enable the most commonly used debugging options in case they have to provide detailed system information to Open-Xchange.

During the startup of the open-xchange service (step 11 of diagram ''Service startup using systemd'' and step 10 of diagram ''Service startup using upstart''.) all uncommented JAVA_OPTS_* categories will then be evaluated and combined again into a single string of commandline options that can then be passed to the JVM.

=== Upgrade procedure ===
During the upgrade '''7.8.3-revX -> 7.8.4-revY''' existing JAVA_XTRAOPTS will automatically be split up and migrated to the new categories approach. A backup of the file (ox-scriptconf.sh.timestamp) will be created and admins are urged to check for proper migration before deleting the backup file like it's already done with *.dpkg-dist or *.rpmnew files on Debian or RHEL/SLES distributions.

== Encoding detection during JVM startup ==

=== Background Infos ===
All of en_US.UTF-8, .utf8, .utf-8, .UTF_8 will result in a valid UTF-8 charmap

==== JVM startup ====

The default file.encoding detection is done at JVM startup based upon LC_CTYPE which makes sense as it's documented as:

CODESET (LC_CTYPE)
Return a string with the name of the character encoding used in the selected locale, such as "UTF-8", "ISO-8859-1",
or "ANSI_X3.4-1968" (better known as US-ASCII). This is the same string that you get with "locale charmap".
For a list of character encoding names, try "locale -m", cf. locale(1).

Furthermore this influences the defaultCharset selection of the JVM.

/**
* Returns the default charset of this Java virtual machine.
*
* The default charset is determined during virtual-machine startup and
* typically depends upon the locale and charset of the underlying
* operating system.
*
* @return A charset object for the default charset
*
* @since 1.5
*/
public static Charset defaultCharset() {
if (defaultCharset == null) {
synchronized (Charset.class) {
String csn = AccessController.doPrivileged(
new GetPropertyAction("file.encoding"));
Charset cs = lookup(csn);
if (cs != null)
defaultCharset = cs;
else
defaultCharset = forName("UTF-8");
}
}
return defaultCharset;
}

Although we try to specify valid unicode encodings for every IO operations the Open-Xchange middleware does some parts of the JVM don't allow to specify the encoding to use and falls back to the detected default encoding. That's why admins have to make sure to provide a proper unicode environment for the middleware service during every start or use documented workarounds provided by Open-Xchange (e.g. )

=== Query which file encoding was detected by the JVM during startup ===

/opt/open-xchange/sbin/showruntimestats -r | sed -ne 's/.*${key=file.encoding, value=[^}]*}$.*/\1/p'
{key=file.encoding, value=UTF-8}

=== Locale passing from client to server via SSH ===

The following setting of sshd allows clients to specify which locale to use on the server and might lead to some confusion or unexpected behaviour when checking the system locale or even to the usage of broken locales during middleware startup if your client sends misconfigured or missing locale configuration to the server.

grep AcceptEnv /etc/ssh/sshd_config
AcceptEnv LANG LC_*

For example when i connect to the machine my local german locale is sent to the server

[vagrant@showcase-community-centos-7 ~]$ locale -v
LANG=de_DE.UTF-8
LC_CTYPE="de_DE.UTF-8"
LC_NUMERIC=de_DE.UTF-8
LC_TIME=de_DE.UTF-8
LC_COLLATE="de_DE.UTF-8"
LC_MONETARY=de_DE.UTF-8
LC_MESSAGES="de_DE.UTF-8"
LC_PAPER=de_DE.UTF-8
LC_NAME=de_DE.UTF-8
LC_ADDRESS=de_DE.UTF-8
LC_TELEPHONE=de_DE.UTF-8
LC_MEASUREMENT=de_DE.UTF-8
LC_IDENTIFICATION=de_DE.UTF-8
LC_ALL=

But after specifying that i want a new login shell the default locale configured for that user/system will be used instead!

[root@showcase-community-centos-7 vagrant]# su -s /bin/bash open-xchange -l
-bash-4.2$ locale -v
LANG=en_US.UTF-8
LC_CTYPE="en_US.UTF-8"
LC_NUMERIC="en_US.UTF-8"
LC_TIME="en_US.UTF-8"
LC_COLLATE="en_US.UTF-8"
LC_MONETARY="en_US.UTF-8"
LC_MESSAGES="en_US.UTF-8"
LC_PAPER="en_US.UTF-8"
LC_NAME="en_US.UTF-8"
LC_ADDRESS="en_US.UTF-8"
LC_TELEPHONE="en_US.UTF-8"
LC_MEASUREMENT="en_US.UTF-8"
LC_IDENTIFICATION="en_US.UTF-8"
LC_ALL=

Add details from https://bugs.open-xchange.com/show_bug.cgi?id=51074

== Warnings during early start ==

Early warnings before the start of the JVM and related logging libraries like logback and logstash forwarding are redirected to '''/var/log/open-xchange/open-xchange-console.log'''.

=== Limits ===
If the open-xchange startup script isn't able to set the proper limits on CentOS 6 due to e.g. hard limits being enforced via pam_limits you'll see the following warning in the console log during startup

singlenode open-xchange # cat open-xchange-console.log
/opt/open-xchange/sbin/open-xchange: line 115: ulimit: max user processes: cannot modify limit: Operation not permitted

=== Locale ===
If the open-xchange startup script isn't able to detect a unicode locale you'll see a warning like the following (the current charmap might differ)

WARNING: Couldn't detect proper unicode charmap in locale category LC_CTYPE.
WARNING: Current charmap: "ANSI_X3.4-1968" might cause encoding problems.

== Summary: Open-Xchange middleware on specific distros ==
The support for the mentioned mechanism of resource control differ depending on the distribution and the init system in use.

=== Debian 7 ===
;Init
: System V style
;OX Configurable Limits/Defaults
: nofile, nproc

The mentioned limits can be configured via '''/opt/open-xchange/etc/ox-scriptconf.sh'''. The limits are applied via ulimit in the service's init script. The open-xchange service is finally started via start-stop-daemon which doesn't doesn't consider '''/etc/security/limits.*'''

=== RHEL 6 / CentOS 6 ===
;Init
: Upstart, System V compatible
;OX Configurable Limits/Defaults
: nofile, nproc

The mentioned limits can be configured via '''/opt/open-xchange/etc/ox-scriptconf.sh'''. The limits are applied via ulimit in the service's init script. Furthermore as the open-xchange service is finally started via '''su ... open-xchange''' on this distro a user session is opened via su/pam and the default CentOS pam config reads the '''/etc/security/limits.*''' configuration by loading the pam stack like:

: '''/etc/pam.d/su'''
:: '''-> /etc/pam.d/system-auth'''
::: '''-> pam_limits.so'''

If NPROC isn't configured for the open-xchange-server it's restricted to '''1024''' globally by default to prevent accidental fork bombs, see '''/etc/security/limits.d/90-nproc.conf''' which can result in severe problems modern multithreaded applications.

=== RHEL 7 / CentOS 7 / Debian 8 / SLE 12 ===
;Init
: Systemd
;OX Configurable Limits/Defaults
: nofile, nproc

For systemd the default limits are configured directly in the service's unit file that is shipped by OX and located at '''/usr/lib/systemd/system/open-xchange.service'''. The drop-in config to override or extend the default unit file is located at '''/etc/systemd/system/open-xchange.service.d/limits.conf'''. [https://www.freedesktop.org/software/systemd/man/systemd.exec.html Systemd.exec] shows a whole lot of options that can be used by admins to adapt the default service to their specific needs.

Sandbox:Middleware Startup/sandbox

2017-04-24T19:38:31Z

Marens: /* Warnings during early */

== Systemd ==

==== Service definition ====
The default service definition as shipped by Open-Xchange is rather concise and easy to read. You can check by looking at the service file that is installed to either /usr/lib/systemd or /lib/systemd depending on your distribution of choice.

singlenode:~ # cat /usr/lib/systemd/system/open-xchange.service
[Unit]
After=remote-fs.target
After=time-sync.target ypbind.service sendmail.service cyrus.service

[Service]
User=open-xchange
PermissionsStartOnly=true
TimeoutStartSec=0
ExecStartPre=/opt/open-xchange/sbin/triggerupdatethemes -u
ExecStart=/opt/open-xchange/sbin/open-xchange
ExecStop=/opt/open-xchange/sbin/shutdown -w
ExecReload=/opt/open-xchange/sbin/triggerreloadconfiguration -d
KillMode=process
LimitNOFILE=65536
LimitNPROC=65536

[Install]
WantedBy=multi-user.target

==== Drop-in configs ====
Drop in configs allow administrators to easily adapt and override default service unit files. So if you want to change the default limits or add additional limits have a look at our default example drop-in config which is located at /etc/systemd/system/open-xchange.service.d/limits.conf

singlenode:~ # cat /etc/systemd/system/open-xchange.service.d/limits.conf
# Override and add options in this file
# See systemd.exec(5) for other limits

[Service]
#LimitNPROC=65536
#LimitNOFILE=65536

The systemd.exec documentation shows a whole lot of options that can be used by admins to adapt the default service to their specific needs.

 
[[File:mw_start_sysd.png|thumb|Service startup using systemd|left|1000px]]
 

== System V / Upstart ==

==== Service definition ====

The default service definition as shipped by Open-Xchange can be found at /etc/init.d/open-xchange and will differ slightly depending on the distro in use.

 
[[File:mw_start_sysv.png|thumb|Service startup using upstart|left|1000px]]
 
== Apply Resource limits ==

As you can see in the two diagrams above one of the differences is that systemd reads all the infos wrt. resource limitations from the original service file and then applies possible further restrictions from the drop-in configs (steps 2 and 8 in the diagram). Limits configured in /opt/open-xchange/etc/ox-scriptconf.sh aren't considered anylonger when using systemd while the older System V style init has to source /opt/open-xchange/etc/ox-scriptconf.sh to be able to read the configs for the maximum number of processes (NPROC) and files(NRFILES) that the middleware shall be allowed to create/open (steps 3,4 and 15 in the diagram).

=== Background Infos ===

Several ways exist to restrict resources on a linux system from a global level down to user/groups or even shells and the processes started by them.

==== Sysctl ====
Sysctl is used to modify kernel parameters at runtime. E.g. to set the maximum number of files

$ sysctl -w fs.file-max=100000

To permanently set them append to the main configuration file and reload the settings

$ echo fs.file-max=100000 >> /etc/sysctl.conf
$ sysctl -p

More infos can be found via '''man sysctl'''

==== Limits.conf ====

Allows to restrict resources an a global, group or user level. E.g:

$ cat /etc/security/limits.d/90-nproc.conf
# Default limit for number of user's processes to prevent
# accidental fork bombs.
# See rhbz #432903 for reasoning.

* soft nproc 1024

From '''man limits.conf''':
<blockquote>
''Also, please note that all limit settings are set per login. They are not global, nor are they permanent; existing only for the duration of the session.''
</blockquote>

The limits per login are applied via the ''pam'' stack. See '''man pam''' and '''man pam_limits''' for more details. As those limits are bound to sessions they don't affect most daemons started by our supported init systems or init utils. Most state that they are ignored by design, see [https://bugs.launchpad.net/ubuntu/+source/upstart/+bug/938669/comments/4 upstart], [https://bugzilla.redhat.com/show_bug.cgi?id=754285#c1 systemd] and [https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=302079 start-stop-daemon]

===== Ulimit =====
From '''man bash'''
<blockquote>
ulimit [-HSTabcdefilmnpqrstuvx [limit]]
Provides control over the resources available to the shell and to processes started by it, on systems that allow such control.
</blockquote>
This is what we use in our System V compatible init scripts to increase resources for the open-xchange process across multiple distros. Currently only ''the maximum number of processes'' and ''the maximum number of open file descriptors available to a single user'' are increased via ulimit. The values are specified in '''/opt/open-xchange/ox-scriptconf.sh'''

==== Systemd ====

===== Control Groups=====

Control groups should only affect the OX middleware if you create/manage them yourself of if you are using a ''modern'' distribution that already uses systemd as init.

Citing from the kernel cgroup [https://www.kernel.org/doc/Documentation/cgroup-v2.txt documentation]:

<blockquote>
1-2. What is cgroup?

cgroup is a mechanism to organize processes hierarchically and
distribute system resources along the hierarchy in a controlled and
configurable manner.

cgroup is largely composed of two parts - the core and controllers.
cgroup core is primarily responsible for hierarchically organizing
processes. A cgroup controller is usually responsible for
distributing a specific type of system resource along the hierarchy
although there are utility controllers which serve purposes other than
resource distribution.

cgroups form a tree structure and every process in the system belongs
to one and only one cgroup. All threads of a process belong to the
same cgroup. On creation, all processes are put in the cgroup that
the parent process belongs to at the time. A process can be migrated
to another cgroup. Migration of a process doesn't affect already
existing descendant processes.

Following certain structural constraints, controllers may be enabled or
disabled selectively on a cgroup. All controller behaviors are
hierarchical - if a controller is enabled on a cgroup, it affects all
processes which belong to the cgroups consisting the inclusive
sub-hierarchy of the cgroup. When a controller is enabled on a nested
cgroup, it always restricts the resource distribution further. The
restrictions set closer to the root in the hierarchy can not be
overridden from further away.
</blockquote>

So processes are organized into a tree structure of control groups and controllers are responsible for the distribution of resources. So what kind of controllers exist?

<blockquote>
5. Controllers

5-1. CPU

The "cpu" controllers regulates distribution of CPU cycles. This
controller implements weight and absolute bandwidth limit models for
normal scheduling policy and absolute bandwidth allocation model for
realtime scheduling policy.

5-2. Memory

The "memory" controller regulates distribution of memory.
...
While not completely water-tight, all major memory usages by a given
cgroup are tracked so that the total memory consumption can be
accounted and controlled to a reasonable extent.

5-3. IO

The "io" controller regulates the distribution of IO resources. This
controller implements both weight based and absolute bandwidth or IOPS
limit distribution; however, weight based distribution is available
only if cfq-iosched is in use and neither scheme is available for
blk-mq devices.
</blockquote>

The open-xchange service is simply put into the default ''system.slice'' without applying further limits.

singlenode$ systemd-cgls
├─1 /sbin/init
├─system.slice
│ ├─avahi-daemon.service
│ │ ├─501 avahi-daemon: running [singlenode]
│ │ └─514 avahi-daemon: chroot helper
│ ├─console-kit-daemon.service
│ │ └─16164 /usr/sbin/console-kit-daemon --no-daemon
│ ├─dbus.service
│ │ └─508 /usr/bin/dbus-daemon --system --address=systemd: --nofork --nopidfile --systemd-activation
│ ├─munin-node.service
│ │ └─4290 /usr/bin/perl -wT /usr/sbin/munin-node
│ ├─open-xchange.service
│ │ └─6037 /usr/bin/java -Dsun.net.inetaddr.ttl=3600 -Dnetworkaddress.cache.ttl=3600 -Dnetworkaddress.cache.negative.ttl=10 ...

To check all the details use

singlenode:~ # systemctl show system.slice

==== Limits besides control groups ====
Besides control groups systemd allows you to apply other limits to the execution environment of your service. Here we can apply the limits that would normally be applied via ''limits.conf'' or ''ulimit''. Systemd uses [http://man7.org/linux/man-pages/man2/setrlimit.2.html setrlimit] for this. The options that we set by default are:

* LimitNOFILE
* LimitNPROC

which you have already seen when we took a look at the [[#Service_definiton|service definition]] and [[#Drop-in_config|drop-in configs]]

== Verify limits ==

=== System V ===

Get the proper pid of the java process started for the middleware by either checking '''ps faux'''
root 4103 0.0 0.0 66528 1868 pts/0 S 22:05 0:00 su -s /bin/bash open-xchange -c /opt/open-xchange/sbin/open-xchange
494 4109 9.5 19.8 2224644 381312 ? Ssl 22:05 1:12 \_ /usr/bin/java -Dsun.net.inetaddr.ttl=3600 -Dnetworkaddress.cache.ttl=3600

or programmatically
singlenode:~ # pid=$(pgrep -P $(</var/run/open-xchange.pid) java)

and check the limits applied for that process
singlenode:~ # cat /proc/$pid/limits
Limit Soft Limit Hard Limit Units
Max cpu time unlimited unlimited seconds
Max file size unlimited unlimited bytes
Max data size unlimited unlimited bytes
Max stack size 8388608 unlimited bytes
Max core file size 0 unlimited bytes
Max resident set unlimited unlimited bytes
Max processes 65536 65536 processes
Max open files 65536 65536 files
Max locked memory 65536 65536 bytes
Max address space unlimited unlimited bytes
Max file locks unlimited unlimited locks
Max pending signals 24254 24254 signals
Max msgqueue size 819200 819200 bytes
Max nice priority 0 0
Max realtime priority 0 0
Max realtime timeout unlimited unlimited us

=== Systemd ===
singlenode:~ # systemctl show open-xchange | grep Limit
StartLimitInterval=10000000
StartLimitBurst=5
StartLimitAction=none
MemoryLimit=18446744073709551615
LimitCPU=18446744073709551615
LimitFSIZE=18446744073709551615
LimitDATA=18446744073709551615
LimitSTACK=18446744073709551615
LimitCORE=18446744073709551615
LimitRSS=18446744073709551615
LimitNOFILE=65536
LimitAS=18446744073709551615
LimitNPROC=65536
LimitMEMLOCK=65536
LimitLOCKS=18446744073709551615
LimitSIGPENDING=19827
LimitMSGQUEUE=819200
LimitNICE=0
LimitRTPRIO=0
LimitRTTIME=18446744073709551615

== Open-Xchange middleware on specific distros ==
The support for the mentioned mechanism of resource control differ depending on the distribution and the init system in use.

=== Debian 7 ===
;Init
: System V style
;OX Configurable Limits/Defaults
: nofile, nproc

The mentioned limits can be configured via '''/opt/open-xchange/etc/ox-scriptconf.sh'''. The limits are applied via ulimit in the service's init script. The open-xchange service is finally started via start-stop-daemon which doesn't doesn't consider '''/etc/security/limits.*'''

=== RHEL 6 / CentOS 6 ===
;Init
: Upstart, System V compatible
;OX Configurable Limits/Defaults
: nofile, nproc

The mentioned limits can be configured via '''/opt/open-xchange/etc/ox-scriptconf.sh'''. The limits are applied via ulimit in the service's init script. Furthermore as the open-xchange service is finally started via '''su ... open-xchange''' on this distro a user session is opened via su/pam and the default CentOS pam config reads the '''/etc/security/limits.*''' configuration by loading the pam stack like:

: '''/etc/pam.d/su'''
:: '''-> /etc/pam.d/system-auth'''
::: '''-> pam_limits.so'''

If NPROC isn't configured for the open-xchange-server it's restricted to '''1024''' globally by default to prevent accidental fork bombs, see '''/etc/security/limits.d/90-nproc.conf''' which can result in severe problems modern multithreaded applications.

=== RHEL 7 / CentOS 7 / Debian 8 / SLE 12 ===
;Init
: Systemd
;OX Configurable Limits/Defaults
: nofile, nproc

For systemd the default limits are configured directly in the service's unit file that is shipped by OX and located at '''/usr/lib/systemd/system/open-xchange.service'''. The drop-in config to override or extend the default unit file is located at '''/etc/systemd/system/open-xchange.service.d/limits.conf'''. [https://www.freedesktop.org/software/systemd/man/systemd.exec.html Systemd.exec] shows a whole lot of options that can be used by admins to adapt the default service to their specific needs.

== Evaluate JVM commandline options ==

Before starting Java and the OSGi framework with all the Open-Xchange middleware bundles we have to evaluate several important JVM commandline options that influence the performance and stability of the Open-Xchange middleware stack. These commandline options are specified in '''/opt/open-xchange/etc/ox-scriptconf.sh'''

=== Format < 7.8.4 ===
For version up to 7.8.3 the JVM configuration options were all specified via a single property '''JAVA_XTRAOPTS''' which resulted in a rather long configuration string that was hard to read and maintain.

JAVA_XTRAOPTS="-Dsun.net.inetaddr.ttl=3600 -Dnetworkaddress.cache.ttl=3600 -Dnetworkaddress.cache.negative.ttl=10 \
-Dlogback.threadlocal.put.duplicate=false -server -Djava.awt.headless=true -Xmx512M -XX:MaxPermSize=256m -XX:+UseConcMarkSweepGC \
-XX:+UseParNewGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:NewRatio=3 -XX:+UseTLAB \
-Dosgi.compatibility.bootdelegation=false -XX:-OmitStackTraceInFastThrow"

During the startup of the open-xchange service (step 11 of diagram Service startup using systemd and step 10 of diagram Service startup using upstart.) this JAVA_XTRAOPTS string goes through a basic validity check and is then passed to the JVM.

=== Format >= 7.8.4 ===
Beginning from version 7.8.4 the options are split up and sorted into different categories like:

JAVA_OPTS_GC="-XX:+UseConcMarkSweepGC -XX:+UseParNewGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:NewRatio=3 -XX:+DisableExplicitGC"
JAVA_OPTS_LOG="-Dlogback.threadlocal.put.duplicate=false -XX:-OmitStackTraceInFastThrow"
JAVA_OPTS_MEM="-XX:MaxHeapSize=512M -XX:MaxPermSize=256M -XX:+UseTLAB"
JAVA_OPTS_NET="-Dsun.net.inetaddr.ttl=3600 -Dnetworkaddress.cache.ttl=3600 -Dnetworkaddress.cache.negative.ttl=10"
JAVA_OPTS_OSGI="-Dosgi.compatibility.bootdelegation=false"
JAVA_OPTS_SERVER="-server -Djava.awt.headless=true"

JAVA_OPTS_OTHER=""

#JAVA_OPTS_DEBUG="-XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/mnt/heapdump -Xloggc:/var/log/open-xchange/gc.log -verbose:gc -XX:+PrintGCDateStamps \
-XX:+PrintHeapAtGC -XX:+PrintGCApplicationStoppedTime -XX:+PrintTenuringDistribution"

which is much easier to read an maintain. Furthermore we added a useful '''JAVA_OPTS_DEBUG''' configuration parameter that comes preconfigured but unused/commented so that customers can quickly enable the most commonly used debugging options in case they have to provide detailed system information to Open-Xchange.

During the startup of the open-xchange service (step 11 of diagram ''Service startup using systemd'' and step 10 of diagram ''Service startup using upstart''.) all uncommented JAVA_OPTS_* categories will then be evaluated and combined again into a single string of commandline options that can then be passed to the JVM.

=== Upgrade procedure ===
During the upgrade '''7.8.3-revX -> 7.8.4-revY''' existing JAVA_XTRAOPTS will automatically be split up and migrated to the new categories approach. A backup of the file (ox-scriptconf.sh.timestamp) will be created and admins are urged to check for proper migration before deleting the backup file like it's already done with *.dpkg-dist or *.rpmnew files on Debian or RHEL/SLES distributions.

== Encoding detection during JVM startup ==

=== Background Infos ===
All of en_US.UTF-8, .utf8, .utf-8, .UTF_8 will result in a valid UTF-8 charmap

==== JVM startup ====

The default file.encoding detection is done at JVM startup based upon LC_CTYPE which makes sense as it's documented as:

CODESET (LC_CTYPE)
Return a string with the name of the character encoding used in the selected locale, such as "UTF-8", "ISO-8859-1",
or "ANSI_X3.4-1968" (better known as US-ASCII). This is the same string that you get with "locale charmap".
For a list of character encoding names, try "locale -m", cf. locale(1).

Furthermore this influences the defaultCharset selection of the JVM.

/**
* Returns the default charset of this Java virtual machine.
*
* The default charset is determined during virtual-machine startup and
* typically depends upon the locale and charset of the underlying
* operating system.
*
* @return A charset object for the default charset
*
* @since 1.5
*/
public static Charset defaultCharset() {
if (defaultCharset == null) {
synchronized (Charset.class) {
String csn = AccessController.doPrivileged(
new GetPropertyAction("file.encoding"));
Charset cs = lookup(csn);
if (cs != null)
defaultCharset = cs;
else
defaultCharset = forName("UTF-8");
}
}
return defaultCharset;
}

Although we try to specify valid unicode encodings for every IO operations the Open-Xchange middleware does some parts of the JVM don't allow to specify the encoding to use and falls back to the detected default encoding. That's why admins have to make sure to provide a proper unicode environment for the middleware service during every start or use documented workarounds provided by Open-Xchange (e.g. )

=== Query which file encoding was detected by the JVM during startup ===

/opt/open-xchange/sbin/showruntimestats -r | sed -ne 's/.*${key=file.encoding, value=[^}]*}$.*/\1/p'
{key=file.encoding, value=UTF-8}

=== Locale passing from client to server via SSH ===

The following setting of sshd allows clients to specify which locale to use on the server and might lead to some confusion or unexpected behaviour when checking the system locale or even to the usage of broken locales during middleware startup if your client sends misconfigured or missing locale configuration to the server.

grep AcceptEnv /etc/ssh/sshd_config
AcceptEnv LANG LC_*

For example when i connect to the machine my local german locale is sent to the server

[vagrant@showcase-community-centos-7 ~]$ locale -v
LANG=de_DE.UTF-8
LC_CTYPE="de_DE.UTF-8"
LC_NUMERIC=de_DE.UTF-8
LC_TIME=de_DE.UTF-8
LC_COLLATE="de_DE.UTF-8"
LC_MONETARY=de_DE.UTF-8
LC_MESSAGES="de_DE.UTF-8"
LC_PAPER=de_DE.UTF-8
LC_NAME=de_DE.UTF-8
LC_ADDRESS=de_DE.UTF-8
LC_TELEPHONE=de_DE.UTF-8
LC_MEASUREMENT=de_DE.UTF-8
LC_IDENTIFICATION=de_DE.UTF-8
LC_ALL=

But after specifying that i want a new login shell the default locale configured for that user/system will be used instead!

[root@showcase-community-centos-7 vagrant]# su -s /bin/bash open-xchange -l
-bash-4.2$ locale -v
LANG=en_US.UTF-8
LC_CTYPE="en_US.UTF-8"
LC_NUMERIC="en_US.UTF-8"
LC_TIME="en_US.UTF-8"
LC_COLLATE="en_US.UTF-8"
LC_MONETARY="en_US.UTF-8"
LC_MESSAGES="en_US.UTF-8"
LC_PAPER="en_US.UTF-8"
LC_NAME="en_US.UTF-8"
LC_ADDRESS="en_US.UTF-8"
LC_TELEPHONE="en_US.UTF-8"
LC_MEASUREMENT="en_US.UTF-8"
LC_IDENTIFICATION="en_US.UTF-8"
LC_ALL=

Add details from https://bugs.open-xchange.com/show_bug.cgi?id=51074

== Warnings during early start ==

Early warnings before the start of the JVM and related logging libraries like logback and logstash forwarding are redirected to '''/var/log/open-xchange/open-xchange-console.log'''.

=== Limits ===
If the open-xchange startup script isn't able to set the proper limits on CentOS 6 due to e.g. hard limits being enforced via pam_limits you'll see the following warning in the console log during startup

singlenode open-xchange # cat open-xchange-console.log
/opt/open-xchange/sbin/open-xchange: line 115: ulimit: max user processes: cannot modify limit: Operation not permitted

=== Locale ===
If the open-xchange startup script isn't able to detect a unicode locale you'll see a warning like the following (the current charmap might differ)

WARNING: Couldn't detect proper unicode charmap in locale category LC_CTYPE.
WARNING: Current charmap: "ANSI_X3.4-1968" might cause encoding problems.

Sandbox:Middleware Startup/sandbox

2017-04-24T19:20:08Z

Marens: /* Encoding detection during JVM startup */

== Systemd ==

==== Service definition ====
The default service definition as shipped by Open-Xchange is rather concise and easy to read. You can check by looking at the service file that is installed to either /usr/lib/systemd or /lib/systemd depending on your distribution of choice.

singlenode:~ # cat /usr/lib/systemd/system/open-xchange.service
[Unit]
After=remote-fs.target
After=time-sync.target ypbind.service sendmail.service cyrus.service

[Service]
User=open-xchange
PermissionsStartOnly=true
TimeoutStartSec=0
ExecStartPre=/opt/open-xchange/sbin/triggerupdatethemes -u
ExecStart=/opt/open-xchange/sbin/open-xchange
ExecStop=/opt/open-xchange/sbin/shutdown -w
ExecReload=/opt/open-xchange/sbin/triggerreloadconfiguration -d
KillMode=process
LimitNOFILE=65536
LimitNPROC=65536

[Install]
WantedBy=multi-user.target

==== Drop-in configs ====
Drop in configs allow administrators to easily adapt and override default service unit files. So if you want to change the default limits or add additional limits have a look at our default example drop-in config which is located at /etc/systemd/system/open-xchange.service.d/limits.conf

singlenode:~ # cat /etc/systemd/system/open-xchange.service.d/limits.conf
# Override and add options in this file
# See systemd.exec(5) for other limits

[Service]
#LimitNPROC=65536
#LimitNOFILE=65536

The systemd.exec documentation shows a whole lot of options that can be used by admins to adapt the default service to their specific needs.

 
[[File:mw_start_sysd.png|thumb|Service startup using systemd|left|1000px]]
 

== System V / Upstart ==

==== Service definition ====

The default service definition as shipped by Open-Xchange can be found at /etc/init.d/open-xchange and will differ slightly depending on the distro in use.

 
[[File:mw_start_sysv.png|thumb|Service startup using upstart|left|1000px]]
 
== Apply Resource limits ==

As you can see in the two diagrams above one of the differences is that systemd reads all the infos wrt. resource limitations from the original service file and then applies possible further restrictions from the drop-in configs (steps 2 and 8 in the diagram). Limits configured in /opt/open-xchange/etc/ox-scriptconf.sh aren't considered anylonger when using systemd while the older System V style init has to source /opt/open-xchange/etc/ox-scriptconf.sh to be able to read the configs for the maximum number of processes (NPROC) and files(NRFILES) that the middleware shall be allowed to create/open (steps 3,4 and 15 in the diagram).

=== Background Infos ===

Several ways exist to restrict resources on a linux system from a global level down to user/groups or even shells and the processes started by them.

==== Sysctl ====
Sysctl is used to modify kernel parameters at runtime. E.g. to set the maximum number of files

$ sysctl -w fs.file-max=100000

To permanently set them append to the main configuration file and reload the settings

$ echo fs.file-max=100000 >> /etc/sysctl.conf
$ sysctl -p

More infos can be found via '''man sysctl'''

==== Limits.conf ====

Allows to restrict resources an a global, group or user level. E.g:

$ cat /etc/security/limits.d/90-nproc.conf
# Default limit for number of user's processes to prevent
# accidental fork bombs.
# See rhbz #432903 for reasoning.

* soft nproc 1024

From '''man limits.conf''':
<blockquote>
''Also, please note that all limit settings are set per login. They are not global, nor are they permanent; existing only for the duration of the session.''
</blockquote>

The limits per login are applied via the ''pam'' stack. See '''man pam''' and '''man pam_limits''' for more details. As those limits are bound to sessions they don't affect most daemons started by our supported init systems or init utils. Most state that they are ignored by design, see [https://bugs.launchpad.net/ubuntu/+source/upstart/+bug/938669/comments/4 upstart], [https://bugzilla.redhat.com/show_bug.cgi?id=754285#c1 systemd] and [https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=302079 start-stop-daemon]

===== Ulimit =====
From '''man bash'''
<blockquote>
ulimit [-HSTabcdefilmnpqrstuvx [limit]]
Provides control over the resources available to the shell and to processes started by it, on systems that allow such control.
</blockquote>
This is what we use in our System V compatible init scripts to increase resources for the open-xchange process across multiple distros. Currently only ''the maximum number of processes'' and ''the maximum number of open file descriptors available to a single user'' are increased via ulimit. The values are specified in '''/opt/open-xchange/ox-scriptconf.sh'''

==== Systemd ====

===== Control Groups=====

Control groups should only affect the OX middleware if you create/manage them yourself of if you are using a ''modern'' distribution that already uses systemd as init.

Citing from the kernel cgroup [https://www.kernel.org/doc/Documentation/cgroup-v2.txt documentation]:

<blockquote>
1-2. What is cgroup?

cgroup is a mechanism to organize processes hierarchically and
distribute system resources along the hierarchy in a controlled and
configurable manner.

cgroup is largely composed of two parts - the core and controllers.
cgroup core is primarily responsible for hierarchically organizing
processes. A cgroup controller is usually responsible for
distributing a specific type of system resource along the hierarchy
although there are utility controllers which serve purposes other than
resource distribution.

cgroups form a tree structure and every process in the system belongs
to one and only one cgroup. All threads of a process belong to the
same cgroup. On creation, all processes are put in the cgroup that
the parent process belongs to at the time. A process can be migrated
to another cgroup. Migration of a process doesn't affect already
existing descendant processes.

Following certain structural constraints, controllers may be enabled or
disabled selectively on a cgroup. All controller behaviors are
hierarchical - if a controller is enabled on a cgroup, it affects all
processes which belong to the cgroups consisting the inclusive
sub-hierarchy of the cgroup. When a controller is enabled on a nested
cgroup, it always restricts the resource distribution further. The
restrictions set closer to the root in the hierarchy can not be
overridden from further away.
</blockquote>

So processes are organized into a tree structure of control groups and controllers are responsible for the distribution of resources. So what kind of controllers exist?

<blockquote>
5. Controllers

5-1. CPU

The "cpu" controllers regulates distribution of CPU cycles. This
controller implements weight and absolute bandwidth limit models for
normal scheduling policy and absolute bandwidth allocation model for
realtime scheduling policy.

5-2. Memory

The "memory" controller regulates distribution of memory.
...
While not completely water-tight, all major memory usages by a given
cgroup are tracked so that the total memory consumption can be
accounted and controlled to a reasonable extent.

5-3. IO

The "io" controller regulates the distribution of IO resources. This
controller implements both weight based and absolute bandwidth or IOPS
limit distribution; however, weight based distribution is available
only if cfq-iosched is in use and neither scheme is available for
blk-mq devices.
</blockquote>

The open-xchange service is simply put into the default ''system.slice'' without applying further limits.

singlenode$ systemd-cgls
├─1 /sbin/init
├─system.slice
│ ├─avahi-daemon.service
│ │ ├─501 avahi-daemon: running [singlenode]
│ │ └─514 avahi-daemon: chroot helper
│ ├─console-kit-daemon.service
│ │ └─16164 /usr/sbin/console-kit-daemon --no-daemon
│ ├─dbus.service
│ │ └─508 /usr/bin/dbus-daemon --system --address=systemd: --nofork --nopidfile --systemd-activation
│ ├─munin-node.service
│ │ └─4290 /usr/bin/perl -wT /usr/sbin/munin-node
│ ├─open-xchange.service
│ │ └─6037 /usr/bin/java -Dsun.net.inetaddr.ttl=3600 -Dnetworkaddress.cache.ttl=3600 -Dnetworkaddress.cache.negative.ttl=10 ...

To check all the details use

singlenode:~ # systemctl show system.slice

==== Limits besides control groups ====
Besides control groups systemd allows you to apply other limits to the execution environment of your service. Here we can apply the limits that would normally be applied via ''limits.conf'' or ''ulimit''. Systemd uses [http://man7.org/linux/man-pages/man2/setrlimit.2.html setrlimit] for this. The options that we set by default are:

* LimitNOFILE
* LimitNPROC

which you have already seen when we took a look at the [[#Service_definiton|service definition]] and [[#Drop-in_config|drop-in configs]]

== Verify limits ==

=== System V ===

Get the proper pid of the java process started for the middleware by either checking '''ps faux'''
root 4103 0.0 0.0 66528 1868 pts/0 S 22:05 0:00 su -s /bin/bash open-xchange -c /opt/open-xchange/sbin/open-xchange
494 4109 9.5 19.8 2224644 381312 ? Ssl 22:05 1:12 \_ /usr/bin/java -Dsun.net.inetaddr.ttl=3600 -Dnetworkaddress.cache.ttl=3600

or programmatically
singlenode:~ # pid=$(pgrep -P $(</var/run/open-xchange.pid) java)

and check the limits applied for that process
singlenode:~ # cat /proc/$pid/limits
Limit Soft Limit Hard Limit Units
Max cpu time unlimited unlimited seconds
Max file size unlimited unlimited bytes
Max data size unlimited unlimited bytes
Max stack size 8388608 unlimited bytes
Max core file size 0 unlimited bytes
Max resident set unlimited unlimited bytes
Max processes 65536 65536 processes
Max open files 65536 65536 files
Max locked memory 65536 65536 bytes
Max address space unlimited unlimited bytes
Max file locks unlimited unlimited locks
Max pending signals 24254 24254 signals
Max msgqueue size 819200 819200 bytes
Max nice priority 0 0
Max realtime priority 0 0
Max realtime timeout unlimited unlimited us

=== Systemd ===
singlenode:~ # systemctl show open-xchange | grep Limit
StartLimitInterval=10000000
StartLimitBurst=5
StartLimitAction=none
MemoryLimit=18446744073709551615
LimitCPU=18446744073709551615
LimitFSIZE=18446744073709551615
LimitDATA=18446744073709551615
LimitSTACK=18446744073709551615
LimitCORE=18446744073709551615
LimitRSS=18446744073709551615
LimitNOFILE=65536
LimitAS=18446744073709551615
LimitNPROC=65536
LimitMEMLOCK=65536
LimitLOCKS=18446744073709551615
LimitSIGPENDING=19827
LimitMSGQUEUE=819200
LimitNICE=0
LimitRTPRIO=0
LimitRTTIME=18446744073709551615

== Open-Xchange middleware on specific distros ==
The support for the mentioned mechanism of resource control differ depending on the distribution and the init system in use.

=== Debian 7 ===
;Init
: System V style
;OX Configurable Limits/Defaults
: nofile, nproc

The mentioned limits can be configured via '''/opt/open-xchange/etc/ox-scriptconf.sh'''. The limits are applied via ulimit in the service's init script. The open-xchange service is finally started via start-stop-daemon which doesn't doesn't consider '''/etc/security/limits.*'''

=== RHEL 6 / CentOS 6 ===
;Init
: Upstart, System V compatible
;OX Configurable Limits/Defaults
: nofile, nproc

The mentioned limits can be configured via '''/opt/open-xchange/etc/ox-scriptconf.sh'''. The limits are applied via ulimit in the service's init script. Furthermore as the open-xchange service is finally started via '''su ... open-xchange''' on this distro a user session is opened via su/pam and the default CentOS pam config reads the '''/etc/security/limits.*''' configuration by loading the pam stack like:

: '''/etc/pam.d/su'''
:: '''-> /etc/pam.d/system-auth'''
::: '''-> pam_limits.so'''

If NPROC isn't configured for the open-xchange-server it's restricted to '''1024''' globally by default to prevent accidental fork bombs, see '''/etc/security/limits.d/90-nproc.conf''' which can result in severe problems modern multithreaded applications.

=== RHEL 7 / CentOS 7 / Debian 8 / SLE 12 ===
;Init
: Systemd
;OX Configurable Limits/Defaults
: nofile, nproc

For systemd the default limits are configured directly in the service's unit file that is shipped by OX and located at '''/usr/lib/systemd/system/open-xchange.service'''. The drop-in config to override or extend the default unit file is located at '''/etc/systemd/system/open-xchange.service.d/limits.conf'''. [https://www.freedesktop.org/software/systemd/man/systemd.exec.html Systemd.exec] shows a whole lot of options that can be used by admins to adapt the default service to their specific needs.

== Evaluate JVM commandline options ==

Before starting Java and the OSGi framework with all the Open-Xchange middleware bundles we have to evaluate several important JVM commandline options that influence the performance and stability of the Open-Xchange middleware stack. These commandline options are specified in '''/opt/open-xchange/etc/ox-scriptconf.sh'''

=== Format < 7.8.4 ===
For version up to 7.8.3 the JVM configuration options were all specified via a single property '''JAVA_XTRAOPTS''' which resulted in a rather long configuration string that was hard to read and maintain.

JAVA_XTRAOPTS="-Dsun.net.inetaddr.ttl=3600 -Dnetworkaddress.cache.ttl=3600 -Dnetworkaddress.cache.negative.ttl=10 \
-Dlogback.threadlocal.put.duplicate=false -server -Djava.awt.headless=true -Xmx512M -XX:MaxPermSize=256m -XX:+UseConcMarkSweepGC \
-XX:+UseParNewGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:NewRatio=3 -XX:+UseTLAB \
-Dosgi.compatibility.bootdelegation=false -XX:-OmitStackTraceInFastThrow"

During the startup of the open-xchange service (step 11 of diagram Service startup using systemd and step 10 of diagram Service startup using upstart.) this JAVA_XTRAOPTS string goes through a basic validity check and is then passed to the JVM.

=== Format >= 7.8.4 ===
Beginning from version 7.8.4 the options are split up and sorted into different categories like:

JAVA_OPTS_GC="-XX:+UseConcMarkSweepGC -XX:+UseParNewGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:NewRatio=3 -XX:+DisableExplicitGC"
JAVA_OPTS_LOG="-Dlogback.threadlocal.put.duplicate=false -XX:-OmitStackTraceInFastThrow"
JAVA_OPTS_MEM="-XX:MaxHeapSize=512M -XX:MaxPermSize=256M -XX:+UseTLAB"
JAVA_OPTS_NET="-Dsun.net.inetaddr.ttl=3600 -Dnetworkaddress.cache.ttl=3600 -Dnetworkaddress.cache.negative.ttl=10"
JAVA_OPTS_OSGI="-Dosgi.compatibility.bootdelegation=false"
JAVA_OPTS_SERVER="-server -Djava.awt.headless=true"

JAVA_OPTS_OTHER=""

#JAVA_OPTS_DEBUG="-XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/mnt/heapdump -Xloggc:/var/log/open-xchange/gc.log -verbose:gc -XX:+PrintGCDateStamps \
-XX:+PrintHeapAtGC -XX:+PrintGCApplicationStoppedTime -XX:+PrintTenuringDistribution"

which is much easier to read an maintain. Furthermore we added a useful '''JAVA_OPTS_DEBUG''' configuration parameter that comes preconfigured but unused/commented so that customers can quickly enable the most commonly used debugging options in case they have to provide detailed system information to Open-Xchange.

During the startup of the open-xchange service (step 11 of diagram ''Service startup using systemd'' and step 10 of diagram ''Service startup using upstart''.) all uncommented JAVA_OPTS_* categories will then be evaluated and combined again into a single string of commandline options that can then be passed to the JVM.

=== Upgrade procedure ===
During the upgrade '''7.8.3-revX -> 7.8.4-revY''' existing JAVA_XTRAOPTS will automatically be split up and migrated to the new categories approach. A backup of the file (ox-scriptconf.sh.timestamp) will be created and admins are urged to check for proper migration before deleting the backup file like it's already done with *.dpkg-dist or *.rpmnew files on Debian or RHEL/SLES distributions.

== Encoding detection during JVM startup ==

=== Background Infos ===
All of en_US.UTF-8, .utf8, .utf-8, .UTF_8 will result in a valid UTF-8 charmap

==== JVM startup ====

The default file.encoding detection is done at JVM startup based upon LC_CTYPE which makes sense as it's documented as:

CODESET (LC_CTYPE)
Return a string with the name of the character encoding used in the selected locale, such as "UTF-8", "ISO-8859-1",
or "ANSI_X3.4-1968" (better known as US-ASCII). This is the same string that you get with "locale charmap".
For a list of character encoding names, try "locale -m", cf. locale(1).

Furthermore this influences the defaultCharset selection of the JVM.

/**
* Returns the default charset of this Java virtual machine.
*
* The default charset is determined during virtual-machine startup and
* typically depends upon the locale and charset of the underlying
* operating system.
*
* @return A charset object for the default charset
*
* @since 1.5
*/
public static Charset defaultCharset() {
if (defaultCharset == null) {
synchronized (Charset.class) {
String csn = AccessController.doPrivileged(
new GetPropertyAction("file.encoding"));
Charset cs = lookup(csn);
if (cs != null)
defaultCharset = cs;
else
defaultCharset = forName("UTF-8");
}
}
return defaultCharset;
}

Although we try to specify valid unicode encodings for every IO operations the Open-Xchange middleware does some parts of the JVM don't allow to specify the encoding to use and falls back to the detected default encoding. That's why admins have to make sure to provide a proper unicode environment for the middleware service during every start or use documented workarounds provided by Open-Xchange (e.g. )

=== Query which file encoding was detected by the JVM during startup ===

/opt/open-xchange/sbin/showruntimestats -r | sed -ne 's/.*${key=file.encoding, value=[^}]*}$.*/\1/p'
{key=file.encoding, value=UTF-8}

=== Locale passing from client to server via SSH ===

The following setting of sshd allows clients to specify which locale to use on the server and might lead to some confusion or unexpected behaviour when checking the system locale or even to the usage of broken locales during middleware startup if your client sends misconfigured or missing locale configuration to the server.

grep AcceptEnv /etc/ssh/sshd_config
AcceptEnv LANG LC_*

For example when i connect to the machine my local german locale is sent to the server

[vagrant@showcase-community-centos-7 ~]$ locale -v
LANG=de_DE.UTF-8
LC_CTYPE="de_DE.UTF-8"
LC_NUMERIC=de_DE.UTF-8
LC_TIME=de_DE.UTF-8
LC_COLLATE="de_DE.UTF-8"
LC_MONETARY=de_DE.UTF-8
LC_MESSAGES="de_DE.UTF-8"
LC_PAPER=de_DE.UTF-8
LC_NAME=de_DE.UTF-8
LC_ADDRESS=de_DE.UTF-8
LC_TELEPHONE=de_DE.UTF-8
LC_MEASUREMENT=de_DE.UTF-8
LC_IDENTIFICATION=de_DE.UTF-8
LC_ALL=

But after specifying that i want a new login shell the default locale configured for that user/system will be used instead!

[root@showcase-community-centos-7 vagrant]# su -s /bin/bash open-xchange -l
-bash-4.2$ locale -v
LANG=en_US.UTF-8
LC_CTYPE="en_US.UTF-8"
LC_NUMERIC="en_US.UTF-8"
LC_TIME="en_US.UTF-8"
LC_COLLATE="en_US.UTF-8"
LC_MONETARY="en_US.UTF-8"
LC_MESSAGES="en_US.UTF-8"
LC_PAPER="en_US.UTF-8"
LC_NAME="en_US.UTF-8"
LC_ADDRESS="en_US.UTF-8"
LC_TELEPHONE="en_US.UTF-8"
LC_MEASUREMENT="en_US.UTF-8"
LC_IDENTIFICATION="en_US.UTF-8"
LC_ALL=

Add details from https://bugs.open-xchange.com/show_bug.cgi?id=51074

== Warnings during early ==

Early warnings before the start of the JVM and related logging libraries like logback and logstash forwarding are redirected to /var/log/open-xchange/open-xchange-console.log.

=== Limits ===
If the open-xchange startup script isn't able to set the proper limits on CentOS 6 due to e.g. hard limits being enforced via pam_limits you'll see the following warning in the console log during startup

singlenode open-xchange # cat open-xchange-console.log
/opt/open-xchange/sbin/open-xchange: line 115: ulimit: max user processes: cannot modify limit: Operation not permitted

=== Locale ===
If the open-xchange startup script isn't able to detect a unicode locale you'll see a warning like the following (the current charmap might differ)

WARNING: Couldn't detect proper unicode charmap in locale category LC_CTYPE.
WARNING: Current charmap: "ANSI_X3.4-1968" might cause encoding problems.

Sandbox:Middleware Startup/sandbox

2017-04-24T17:58:11Z

Marens: /* Encoding detection during JVM startup */

== Systemd ==

==== Service definition ====
The default service definition as shipped by Open-Xchange is rather concise and easy to read. You can check by looking at the service file that is installed to either /usr/lib/systemd or /lib/systemd depending on your distribution of choice.

singlenode:~ # cat /usr/lib/systemd/system/open-xchange.service
[Unit]
After=remote-fs.target
After=time-sync.target ypbind.service sendmail.service cyrus.service

[Service]
User=open-xchange
PermissionsStartOnly=true
TimeoutStartSec=0
ExecStartPre=/opt/open-xchange/sbin/triggerupdatethemes -u
ExecStart=/opt/open-xchange/sbin/open-xchange
ExecStop=/opt/open-xchange/sbin/shutdown -w
ExecReload=/opt/open-xchange/sbin/triggerreloadconfiguration -d
KillMode=process
LimitNOFILE=65536
LimitNPROC=65536

[Install]
WantedBy=multi-user.target

==== Drop-in configs ====
Drop in configs allow administrators to easily adapt and override default service unit files. So if you want to change the default limits or add additional limits have a look at our default example drop-in config which is located at /etc/systemd/system/open-xchange.service.d/limits.conf

singlenode:~ # cat /etc/systemd/system/open-xchange.service.d/limits.conf
# Override and add options in this file
# See systemd.exec(5) for other limits

[Service]
#LimitNPROC=65536
#LimitNOFILE=65536

The systemd.exec documentation shows a whole lot of options that can be used by admins to adapt the default service to their specific needs.

 
[[File:mw_start_sysd.png|thumb|Service startup using systemd|left|1000px]]
 

== System V / Upstart ==

==== Service definition ====

The default service definition as shipped by Open-Xchange can be found at /etc/init.d/open-xchange and will differ slightly depending on the distro in use.

 
[[File:mw_start_sysv.png|thumb|Service startup using upstart|left|1000px]]
 
== Apply Resource limits ==

As you can see in the two diagrams above one of the differences is that systemd reads all the infos wrt. resource limitations from the original service file and then applies possible further restrictions from the drop-in configs (steps 2 and 8 in the diagram). Limits configured in /opt/open-xchange/etc/ox-scriptconf.sh aren't considered anylonger when using systemd while the older System V style init has to source /opt/open-xchange/etc/ox-scriptconf.sh to be able to read the configs for the maximum number of processes (NPROC) and files(NRFILES) that the middleware shall be allowed to create/open (steps 3,4 and 15 in the diagram).

=== Background Infos ===

Several ways exist to restrict resources on a linux system from a global level down to user/groups or even shells and the processes started by them.

==== Sysctl ====
Sysctl is used to modify kernel parameters at runtime. E.g. to set the maximum number of files

$ sysctl -w fs.file-max=100000

To permanently set them append to the main configuration file and reload the settings

$ echo fs.file-max=100000 >> /etc/sysctl.conf
$ sysctl -p

More infos can be found via '''man sysctl'''

==== Limits.conf ====

Allows to restrict resources an a global, group or user level. E.g:

$ cat /etc/security/limits.d/90-nproc.conf
# Default limit for number of user's processes to prevent
# accidental fork bombs.
# See rhbz #432903 for reasoning.

* soft nproc 1024

From '''man limits.conf''':
<blockquote>
''Also, please note that all limit settings are set per login. They are not global, nor are they permanent; existing only for the duration of the session.''
</blockquote>

The limits per login are applied via the ''pam'' stack. See '''man pam''' and '''man pam_limits''' for more details. As those limits are bound to sessions they don't affect most daemons started by our supported init systems or init utils. Most state that they are ignored by design, see [https://bugs.launchpad.net/ubuntu/+source/upstart/+bug/938669/comments/4 upstart], [https://bugzilla.redhat.com/show_bug.cgi?id=754285#c1 systemd] and [https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=302079 start-stop-daemon]

===== Ulimit =====
From '''man bash'''
<blockquote>
ulimit [-HSTabcdefilmnpqrstuvx [limit]]
Provides control over the resources available to the shell and to processes started by it, on systems that allow such control.
</blockquote>
This is what we use in our System V compatible init scripts to increase resources for the open-xchange process across multiple distros. Currently only ''the maximum number of processes'' and ''the maximum number of open file descriptors available to a single user'' are increased via ulimit. The values are specified in '''/opt/open-xchange/ox-scriptconf.sh'''

==== Systemd ====

===== Control Groups=====

Control groups should only affect the OX middleware if you create/manage them yourself of if you are using a ''modern'' distribution that already uses systemd as init.

Citing from the kernel cgroup [https://www.kernel.org/doc/Documentation/cgroup-v2.txt documentation]:

<blockquote>
1-2. What is cgroup?

cgroup is a mechanism to organize processes hierarchically and
distribute system resources along the hierarchy in a controlled and
configurable manner.

cgroup is largely composed of two parts - the core and controllers.
cgroup core is primarily responsible for hierarchically organizing
processes. A cgroup controller is usually responsible for
distributing a specific type of system resource along the hierarchy
although there are utility controllers which serve purposes other than
resource distribution.

cgroups form a tree structure and every process in the system belongs
to one and only one cgroup. All threads of a process belong to the
same cgroup. On creation, all processes are put in the cgroup that
the parent process belongs to at the time. A process can be migrated
to another cgroup. Migration of a process doesn't affect already
existing descendant processes.

Following certain structural constraints, controllers may be enabled or
disabled selectively on a cgroup. All controller behaviors are
hierarchical - if a controller is enabled on a cgroup, it affects all
processes which belong to the cgroups consisting the inclusive
sub-hierarchy of the cgroup. When a controller is enabled on a nested
cgroup, it always restricts the resource distribution further. The
restrictions set closer to the root in the hierarchy can not be
overridden from further away.
</blockquote>

So processes are organized into a tree structure of control groups and controllers are responsible for the distribution of resources. So what kind of controllers exist?

<blockquote>
5. Controllers

5-1. CPU

The "cpu" controllers regulates distribution of CPU cycles. This
controller implements weight and absolute bandwidth limit models for
normal scheduling policy and absolute bandwidth allocation model for
realtime scheduling policy.

5-2. Memory

The "memory" controller regulates distribution of memory.
...
While not completely water-tight, all major memory usages by a given
cgroup are tracked so that the total memory consumption can be
accounted and controlled to a reasonable extent.

5-3. IO

The "io" controller regulates the distribution of IO resources. This
controller implements both weight based and absolute bandwidth or IOPS
limit distribution; however, weight based distribution is available
only if cfq-iosched is in use and neither scheme is available for
blk-mq devices.
</blockquote>

The open-xchange service is simply put into the default ''system.slice'' without applying further limits.

singlenode$ systemd-cgls
├─1 /sbin/init
├─system.slice
│ ├─avahi-daemon.service
│ │ ├─501 avahi-daemon: running [singlenode]
│ │ └─514 avahi-daemon: chroot helper
│ ├─console-kit-daemon.service
│ │ └─16164 /usr/sbin/console-kit-daemon --no-daemon
│ ├─dbus.service
│ │ └─508 /usr/bin/dbus-daemon --system --address=systemd: --nofork --nopidfile --systemd-activation
│ ├─munin-node.service
│ │ └─4290 /usr/bin/perl -wT /usr/sbin/munin-node
│ ├─open-xchange.service
│ │ └─6037 /usr/bin/java -Dsun.net.inetaddr.ttl=3600 -Dnetworkaddress.cache.ttl=3600 -Dnetworkaddress.cache.negative.ttl=10 ...

To check all the details use

singlenode:~ # systemctl show system.slice

==== Limits besides control groups ====
Besides control groups systemd allows you to apply other limits to the execution environment of your service. Here we can apply the limits that would normally be applied via ''limits.conf'' or ''ulimit''. Systemd uses [http://man7.org/linux/man-pages/man2/setrlimit.2.html setrlimit] for this. The options that we set by default are:

* LimitNOFILE
* LimitNPROC

which you have already seen when we took a look at the [[#Service_definiton|service definition]] and [[#Drop-in_config|drop-in configs]]

== Verify limits ==

=== System V ===

Get the proper pid of the java process started for the middleware by either checking '''ps faux'''
root 4103 0.0 0.0 66528 1868 pts/0 S 22:05 0:00 su -s /bin/bash open-xchange -c /opt/open-xchange/sbin/open-xchange
494 4109 9.5 19.8 2224644 381312 ? Ssl 22:05 1:12 \_ /usr/bin/java -Dsun.net.inetaddr.ttl=3600 -Dnetworkaddress.cache.ttl=3600

or programmatically
singlenode:~ # pid=$(pgrep -P $(</var/run/open-xchange.pid) java)

and check the limits applied for that process
singlenode:~ # cat /proc/$pid/limits
Limit Soft Limit Hard Limit Units
Max cpu time unlimited unlimited seconds
Max file size unlimited unlimited bytes
Max data size unlimited unlimited bytes
Max stack size 8388608 unlimited bytes
Max core file size 0 unlimited bytes
Max resident set unlimited unlimited bytes
Max processes 65536 65536 processes
Max open files 65536 65536 files
Max locked memory 65536 65536 bytes
Max address space unlimited unlimited bytes
Max file locks unlimited unlimited locks
Max pending signals 24254 24254 signals
Max msgqueue size 819200 819200 bytes
Max nice priority 0 0
Max realtime priority 0 0
Max realtime timeout unlimited unlimited us

=== Systemd ===
singlenode:~ # systemctl show open-xchange | grep Limit
StartLimitInterval=10000000
StartLimitBurst=5
StartLimitAction=none
MemoryLimit=18446744073709551615
LimitCPU=18446744073709551615
LimitFSIZE=18446744073709551615
LimitDATA=18446744073709551615
LimitSTACK=18446744073709551615
LimitCORE=18446744073709551615
LimitRSS=18446744073709551615
LimitNOFILE=65536
LimitAS=18446744073709551615
LimitNPROC=65536
LimitMEMLOCK=65536
LimitLOCKS=18446744073709551615
LimitSIGPENDING=19827
LimitMSGQUEUE=819200
LimitNICE=0
LimitRTPRIO=0
LimitRTTIME=18446744073709551615

== Open-Xchange middleware on specific distros ==
The support for the mentioned mechanism of resource control differ depending on the distribution and the init system in use.

=== Debian 7 ===
;Init
: System V style
;OX Configurable Limits/Defaults
: nofile, nproc

The mentioned limits can be configured via '''/opt/open-xchange/etc/ox-scriptconf.sh'''. The limits are applied via ulimit in the service's init script. The open-xchange service is finally started via start-stop-daemon which doesn't doesn't consider '''/etc/security/limits.*'''

=== RHEL 6 / CentOS 6 ===
;Init
: Upstart, System V compatible
;OX Configurable Limits/Defaults
: nofile, nproc

The mentioned limits can be configured via '''/opt/open-xchange/etc/ox-scriptconf.sh'''. The limits are applied via ulimit in the service's init script. Furthermore as the open-xchange service is finally started via '''su ... open-xchange''' on this distro a user session is opened via su/pam and the default CentOS pam config reads the '''/etc/security/limits.*''' configuration by loading the pam stack like:

: '''/etc/pam.d/su'''
:: '''-> /etc/pam.d/system-auth'''
::: '''-> pam_limits.so'''

If NPROC isn't configured for the open-xchange-server it's restricted to '''1024''' globally by default to prevent accidental fork bombs, see '''/etc/security/limits.d/90-nproc.conf''' which can result in severe problems modern multithreaded applications.

=== RHEL 7 / CentOS 7 / Debian 8 / SLE 12 ===
;Init
: Systemd
;OX Configurable Limits/Defaults
: nofile, nproc

For systemd the default limits are configured directly in the service's unit file that is shipped by OX and located at '''/usr/lib/systemd/system/open-xchange.service'''. The drop-in config to override or extend the default unit file is located at '''/etc/systemd/system/open-xchange.service.d/limits.conf'''. [https://www.freedesktop.org/software/systemd/man/systemd.exec.html Systemd.exec] shows a whole lot of options that can be used by admins to adapt the default service to their specific needs.

== Evaluate JVM commandline options ==

Before starting Java and the OSGi framework with all the Open-Xchange middleware bundles we have to evaluate several important JVM commandline options that influence the performance and stability of the Open-Xchange middleware stack. These commandline options are specified in '''/opt/open-xchange/etc/ox-scriptconf.sh'''

=== Format < 7.8.4 ===
For version up to 7.8.3 the JVM configuration options were all specified via a single property '''JAVA_XTRAOPTS''' which resulted in a rather long configuration string that was hard to read and maintain.

JAVA_XTRAOPTS="-Dsun.net.inetaddr.ttl=3600 -Dnetworkaddress.cache.ttl=3600 -Dnetworkaddress.cache.negative.ttl=10 \
-Dlogback.threadlocal.put.duplicate=false -server -Djava.awt.headless=true -Xmx512M -XX:MaxPermSize=256m -XX:+UseConcMarkSweepGC \
-XX:+UseParNewGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:NewRatio=3 -XX:+UseTLAB \
-Dosgi.compatibility.bootdelegation=false -XX:-OmitStackTraceInFastThrow"

During the startup of the open-xchange service (step 11 of diagram Service startup using systemd and step 10 of diagram Service startup using upstart.) this JAVA_XTRAOPTS string goes through a basic validity check and is then passed to the JVM.

=== Format >= 7.8.4 ===
Beginning from version 7.8.4 the options are split up and sorted into different categories like:

JAVA_OPTS_GC="-XX:+UseConcMarkSweepGC -XX:+UseParNewGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:NewRatio=3 -XX:+DisableExplicitGC"
JAVA_OPTS_LOG="-Dlogback.threadlocal.put.duplicate=false -XX:-OmitStackTraceInFastThrow"
JAVA_OPTS_MEM="-XX:MaxHeapSize=512M -XX:MaxPermSize=256M -XX:+UseTLAB"
JAVA_OPTS_NET="-Dsun.net.inetaddr.ttl=3600 -Dnetworkaddress.cache.ttl=3600 -Dnetworkaddress.cache.negative.ttl=10"
JAVA_OPTS_OSGI="-Dosgi.compatibility.bootdelegation=false"
JAVA_OPTS_SERVER="-server -Djava.awt.headless=true"

JAVA_OPTS_OTHER=""

#JAVA_OPTS_DEBUG="-XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/mnt/heapdump -Xloggc:/var/log/open-xchange/gc.log -verbose:gc -XX:+PrintGCDateStamps \
-XX:+PrintHeapAtGC -XX:+PrintGCApplicationStoppedTime -XX:+PrintTenuringDistribution"

which is much easier to read an maintain. Furthermore we added a useful '''JAVA_OPTS_DEBUG''' configuration parameter that comes preconfigured but unused/commented so that customers can quickly enable the most commonly used debugging options in case they have to provide detailed system information to Open-Xchange.

During the startup of the open-xchange service (step 11 of diagram ''Service startup using systemd'' and step 10 of diagram ''Service startup using upstart''.) all uncommented JAVA_OPTS_* categories will then be evaluated and combined again into a single string of commandline options that can then be passed to the JVM.

=== Upgrade procedure ===
During the upgrade '''7.8.3-revX -> 7.8.4-revY''' existing JAVA_XTRAOPTS will automatically be split up and migrated to the new categories approach. A backup of the file (ox-scriptconf.sh.timestamp) will be created and admins are urged to check for proper migration before deleting the backup file like it's already done with *.dpkg-dist or *.rpmnew files on Debian or RHEL/SLES distributions.

== Encoding detection during JVM startup ==

=== Background Infos ===

=== Query which file encoding was detected by the JVM during startup ===

/opt/open-xchange/sbin/showruntimestats -r | sed -ne 's/.*${key=file.encoding, value=[^}]*}$.*/\1/p'
{key=file.encoding, value=UTF-8}

=== Locale passing from client to server via SSH ===

The following setting of sshd allows clients to specify which locale to use on the server and might lead to some confusion or unexpected behaviour when checking the system locale or even to the usage of broken locales during middleware startup if your client sends misconfigured or missing locale configuration to the server.

grep AcceptEnv /etc/ssh/sshd_config
AcceptEnv LANG LC_*

For example when i connect to the machine my local german locale is sent to the server

[vagrant@showcase-community-centos-7 ~]$ locale -v
LANG=de_DE.UTF-8
LC_CTYPE="de_DE.UTF-8"
LC_NUMERIC=de_DE.UTF-8
LC_TIME=de_DE.UTF-8
LC_COLLATE="de_DE.UTF-8"
LC_MONETARY=de_DE.UTF-8
LC_MESSAGES="de_DE.UTF-8"
LC_PAPER=de_DE.UTF-8
LC_NAME=de_DE.UTF-8
LC_ADDRESS=de_DE.UTF-8
LC_TELEPHONE=de_DE.UTF-8
LC_MEASUREMENT=de_DE.UTF-8
LC_IDENTIFICATION=de_DE.UTF-8
LC_ALL=

But after specifying that i want a new login shell the default locale configured for that user/system will be used instead!

[root@showcase-community-centos-7 vagrant]# su -s /bin/bash open-xchange -l
-bash-4.2$ locale -v
LANG=en_US.UTF-8
LC_CTYPE="en_US.UTF-8"
LC_NUMERIC="en_US.UTF-8"
LC_TIME="en_US.UTF-8"
LC_COLLATE="en_US.UTF-8"
LC_MONETARY="en_US.UTF-8"
LC_MESSAGES="en_US.UTF-8"
LC_PAPER="en_US.UTF-8"
LC_NAME="en_US.UTF-8"
LC_ADDRESS="en_US.UTF-8"
LC_TELEPHONE="en_US.UTF-8"
LC_MEASUREMENT="en_US.UTF-8"
LC_IDENTIFICATION="en_US.UTF-8"
LC_ALL=

Add details from https://bugs.open-xchange.com/show_bug.cgi?id=51074

== Warnings during early ==

Early warnings before the start of the JVM and related logging libraries like logback and logstash forwarding are redirected to /var/log/open-xchange/open-xchange-console.log.

=== Limits ===
If the open-xchange startup script isn't able to set the proper limits on CentOS 6 due to e.g. hard limits being enforced via pam_limits you'll see the following warning in the console log during startup

singlenode open-xchange # cat open-xchange-console.log
/opt/open-xchange/sbin/open-xchange: line 115: ulimit: max user processes: cannot modify limit: Operation not permitted

=== Locale ===
If the open-xchange startup script isn't able to detect a unicode locale you'll see a warning like the following (the current charmap might differ)

WARNING: Couldn't detect proper unicode charmap in locale category LC_CTYPE.
WARNING: Current charmap: "ANSI_X3.4-1968" might cause encoding problems.

Sandbox:Middleware Startup/sandbox

2017-04-24T17:38:01Z

Marens: /* Evaluate JVM commandline options */

== Systemd ==

==== Service definition ====
The default service definition as shipped by Open-Xchange is rather concise and easy to read. You can check by looking at the service file that is installed to either /usr/lib/systemd or /lib/systemd depending on your distribution of choice.

singlenode:~ # cat /usr/lib/systemd/system/open-xchange.service
[Unit]
After=remote-fs.target
After=time-sync.target ypbind.service sendmail.service cyrus.service

[Service]
User=open-xchange
PermissionsStartOnly=true
TimeoutStartSec=0
ExecStartPre=/opt/open-xchange/sbin/triggerupdatethemes -u
ExecStart=/opt/open-xchange/sbin/open-xchange
ExecStop=/opt/open-xchange/sbin/shutdown -w
ExecReload=/opt/open-xchange/sbin/triggerreloadconfiguration -d
KillMode=process
LimitNOFILE=65536
LimitNPROC=65536

[Install]
WantedBy=multi-user.target

==== Drop-in configs ====
Drop in configs allow administrators to easily adapt and override default service unit files. So if you want to change the default limits or add additional limits have a look at our default example drop-in config which is located at /etc/systemd/system/open-xchange.service.d/limits.conf

singlenode:~ # cat /etc/systemd/system/open-xchange.service.d/limits.conf
# Override and add options in this file
# See systemd.exec(5) for other limits

[Service]
#LimitNPROC=65536
#LimitNOFILE=65536

The systemd.exec documentation shows a whole lot of options that can be used by admins to adapt the default service to their specific needs.

 
[[File:mw_start_sysd.png|thumb|Service startup using systemd|left|1000px]]
 

== System V / Upstart ==

==== Service definition ====

The default service definition as shipped by Open-Xchange can be found at /etc/init.d/open-xchange and will differ slightly depending on the distro in use.

 
[[File:mw_start_sysv.png|thumb|Service startup using upstart|left|1000px]]
 
== Apply Resource limits ==

As you can see in the two diagrams above one of the differences is that systemd reads all the infos wrt. resource limitations from the original service file and then applies possible further restrictions from the drop-in configs (steps 2 and 8 in the diagram). Limits configured in /opt/open-xchange/etc/ox-scriptconf.sh aren't considered anylonger when using systemd while the older System V style init has to source /opt/open-xchange/etc/ox-scriptconf.sh to be able to read the configs for the maximum number of processes (NPROC) and files(NRFILES) that the middleware shall be allowed to create/open (steps 3,4 and 15 in the diagram).

=== Background Infos ===

Several ways exist to restrict resources on a linux system from a global level down to user/groups or even shells and the processes started by them.

==== Sysctl ====
Sysctl is used to modify kernel parameters at runtime. E.g. to set the maximum number of files

$ sysctl -w fs.file-max=100000

To permanently set them append to the main configuration file and reload the settings

$ echo fs.file-max=100000 >> /etc/sysctl.conf
$ sysctl -p

More infos can be found via '''man sysctl'''

==== Limits.conf ====

Allows to restrict resources an a global, group or user level. E.g:

$ cat /etc/security/limits.d/90-nproc.conf
# Default limit for number of user's processes to prevent
# accidental fork bombs.
# See rhbz #432903 for reasoning.

* soft nproc 1024

From '''man limits.conf''':
<blockquote>
''Also, please note that all limit settings are set per login. They are not global, nor are they permanent; existing only for the duration of the session.''
</blockquote>

The limits per login are applied via the ''pam'' stack. See '''man pam''' and '''man pam_limits''' for more details. As those limits are bound to sessions they don't affect most daemons started by our supported init systems or init utils. Most state that they are ignored by design, see [https://bugs.launchpad.net/ubuntu/+source/upstart/+bug/938669/comments/4 upstart], [https://bugzilla.redhat.com/show_bug.cgi?id=754285#c1 systemd] and [https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=302079 start-stop-daemon]

===== Ulimit =====
From '''man bash'''
<blockquote>
ulimit [-HSTabcdefilmnpqrstuvx [limit]]
Provides control over the resources available to the shell and to processes started by it, on systems that allow such control.
</blockquote>
This is what we use in our System V compatible init scripts to increase resources for the open-xchange process across multiple distros. Currently only ''the maximum number of processes'' and ''the maximum number of open file descriptors available to a single user'' are increased via ulimit. The values are specified in '''/opt/open-xchange/ox-scriptconf.sh'''

==== Systemd ====

===== Control Groups=====

Control groups should only affect the OX middleware if you create/manage them yourself of if you are using a ''modern'' distribution that already uses systemd as init.

Citing from the kernel cgroup [https://www.kernel.org/doc/Documentation/cgroup-v2.txt documentation]:

<blockquote>
1-2. What is cgroup?

cgroup is a mechanism to organize processes hierarchically and
distribute system resources along the hierarchy in a controlled and
configurable manner.

cgroup is largely composed of two parts - the core and controllers.
cgroup core is primarily responsible for hierarchically organizing
processes. A cgroup controller is usually responsible for
distributing a specific type of system resource along the hierarchy
although there are utility controllers which serve purposes other than
resource distribution.

cgroups form a tree structure and every process in the system belongs
to one and only one cgroup. All threads of a process belong to the
same cgroup. On creation, all processes are put in the cgroup that
the parent process belongs to at the time. A process can be migrated
to another cgroup. Migration of a process doesn't affect already
existing descendant processes.

Following certain structural constraints, controllers may be enabled or
disabled selectively on a cgroup. All controller behaviors are
hierarchical - if a controller is enabled on a cgroup, it affects all
processes which belong to the cgroups consisting the inclusive
sub-hierarchy of the cgroup. When a controller is enabled on a nested
cgroup, it always restricts the resource distribution further. The
restrictions set closer to the root in the hierarchy can not be
overridden from further away.
</blockquote>

So processes are organized into a tree structure of control groups and controllers are responsible for the distribution of resources. So what kind of controllers exist?

<blockquote>
5. Controllers

5-1. CPU

The "cpu" controllers regulates distribution of CPU cycles. This
controller implements weight and absolute bandwidth limit models for
normal scheduling policy and absolute bandwidth allocation model for
realtime scheduling policy.

5-2. Memory

The "memory" controller regulates distribution of memory.
...
While not completely water-tight, all major memory usages by a given
cgroup are tracked so that the total memory consumption can be
accounted and controlled to a reasonable extent.

5-3. IO

The "io" controller regulates the distribution of IO resources. This
controller implements both weight based and absolute bandwidth or IOPS
limit distribution; however, weight based distribution is available
only if cfq-iosched is in use and neither scheme is available for
blk-mq devices.
</blockquote>

The open-xchange service is simply put into the default ''system.slice'' without applying further limits.

singlenode$ systemd-cgls
├─1 /sbin/init
├─system.slice
│ ├─avahi-daemon.service
│ │ ├─501 avahi-daemon: running [singlenode]
│ │ └─514 avahi-daemon: chroot helper
│ ├─console-kit-daemon.service
│ │ └─16164 /usr/sbin/console-kit-daemon --no-daemon
│ ├─dbus.service
│ │ └─508 /usr/bin/dbus-daemon --system --address=systemd: --nofork --nopidfile --systemd-activation
│ ├─munin-node.service
│ │ └─4290 /usr/bin/perl -wT /usr/sbin/munin-node
│ ├─open-xchange.service
│ │ └─6037 /usr/bin/java -Dsun.net.inetaddr.ttl=3600 -Dnetworkaddress.cache.ttl=3600 -Dnetworkaddress.cache.negative.ttl=10 ...

To check all the details use

singlenode:~ # systemctl show system.slice

==== Limits besides control groups ====
Besides control groups systemd allows you to apply other limits to the execution environment of your service. Here we can apply the limits that would normally be applied via ''limits.conf'' or ''ulimit''. Systemd uses [http://man7.org/linux/man-pages/man2/setrlimit.2.html setrlimit] for this. The options that we set by default are:

* LimitNOFILE
* LimitNPROC

which you have already seen when we took a look at the [[#Service_definiton|service definition]] and [[#Drop-in_config|drop-in configs]]

== Verify limits ==

=== System V ===

Get the proper pid of the java process started for the middleware by either checking '''ps faux'''
root 4103 0.0 0.0 66528 1868 pts/0 S 22:05 0:00 su -s /bin/bash open-xchange -c /opt/open-xchange/sbin/open-xchange
494 4109 9.5 19.8 2224644 381312 ? Ssl 22:05 1:12 \_ /usr/bin/java -Dsun.net.inetaddr.ttl=3600 -Dnetworkaddress.cache.ttl=3600

or programmatically
singlenode:~ # pid=$(pgrep -P $(</var/run/open-xchange.pid) java)

and check the limits applied for that process
singlenode:~ # cat /proc/$pid/limits
Limit Soft Limit Hard Limit Units
Max cpu time unlimited unlimited seconds
Max file size unlimited unlimited bytes
Max data size unlimited unlimited bytes
Max stack size 8388608 unlimited bytes
Max core file size 0 unlimited bytes
Max resident set unlimited unlimited bytes
Max processes 65536 65536 processes
Max open files 65536 65536 files
Max locked memory 65536 65536 bytes
Max address space unlimited unlimited bytes
Max file locks unlimited unlimited locks
Max pending signals 24254 24254 signals
Max msgqueue size 819200 819200 bytes
Max nice priority 0 0
Max realtime priority 0 0
Max realtime timeout unlimited unlimited us

=== Systemd ===
singlenode:~ # systemctl show open-xchange | grep Limit
StartLimitInterval=10000000
StartLimitBurst=5
StartLimitAction=none
MemoryLimit=18446744073709551615
LimitCPU=18446744073709551615
LimitFSIZE=18446744073709551615
LimitDATA=18446744073709551615
LimitSTACK=18446744073709551615
LimitCORE=18446744073709551615
LimitRSS=18446744073709551615
LimitNOFILE=65536
LimitAS=18446744073709551615
LimitNPROC=65536
LimitMEMLOCK=65536
LimitLOCKS=18446744073709551615
LimitSIGPENDING=19827
LimitMSGQUEUE=819200
LimitNICE=0
LimitRTPRIO=0
LimitRTTIME=18446744073709551615

== Open-Xchange middleware on specific distros ==
The support for the mentioned mechanism of resource control differ depending on the distribution and the init system in use.

=== Debian 7 ===
;Init
: System V style
;OX Configurable Limits/Defaults
: nofile, nproc

The mentioned limits can be configured via '''/opt/open-xchange/etc/ox-scriptconf.sh'''. The limits are applied via ulimit in the service's init script. The open-xchange service is finally started via start-stop-daemon which doesn't doesn't consider '''/etc/security/limits.*'''

=== RHEL 6 / CentOS 6 ===
;Init
: Upstart, System V compatible
;OX Configurable Limits/Defaults
: nofile, nproc

The mentioned limits can be configured via '''/opt/open-xchange/etc/ox-scriptconf.sh'''. The limits are applied via ulimit in the service's init script. Furthermore as the open-xchange service is finally started via '''su ... open-xchange''' on this distro a user session is opened via su/pam and the default CentOS pam config reads the '''/etc/security/limits.*''' configuration by loading the pam stack like:

: '''/etc/pam.d/su'''
:: '''-> /etc/pam.d/system-auth'''
::: '''-> pam_limits.so'''

If NPROC isn't configured for the open-xchange-server it's restricted to '''1024''' globally by default to prevent accidental fork bombs, see '''/etc/security/limits.d/90-nproc.conf''' which can result in severe problems modern multithreaded applications.

=== RHEL 7 / CentOS 7 / Debian 8 / SLE 12 ===
;Init
: Systemd
;OX Configurable Limits/Defaults
: nofile, nproc

For systemd the default limits are configured directly in the service's unit file that is shipped by OX and located at '''/usr/lib/systemd/system/open-xchange.service'''. The drop-in config to override or extend the default unit file is located at '''/etc/systemd/system/open-xchange.service.d/limits.conf'''. [https://www.freedesktop.org/software/systemd/man/systemd.exec.html Systemd.exec] shows a whole lot of options that can be used by admins to adapt the default service to their specific needs.

== Evaluate JVM commandline options ==

Before starting Java and the OSGi framework with all the Open-Xchange middleware bundles we have to evaluate several important JVM commandline options that influence the performance and stability of the Open-Xchange middleware stack. These commandline options are specified in '''/opt/open-xchange/etc/ox-scriptconf.sh'''

=== Format < 7.8.4 ===
For version up to 7.8.3 the JVM configuration options were all specified via a single property '''JAVA_XTRAOPTS''' which resulted in a rather long configuration string that was hard to read and maintain.

JAVA_XTRAOPTS="-Dsun.net.inetaddr.ttl=3600 -Dnetworkaddress.cache.ttl=3600 -Dnetworkaddress.cache.negative.ttl=10 \
-Dlogback.threadlocal.put.duplicate=false -server -Djava.awt.headless=true -Xmx512M -XX:MaxPermSize=256m -XX:+UseConcMarkSweepGC \
-XX:+UseParNewGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:NewRatio=3 -XX:+UseTLAB \
-Dosgi.compatibility.bootdelegation=false -XX:-OmitStackTraceInFastThrow"

During the startup of the open-xchange service (step 11 of diagram Service startup using systemd and step 10 of diagram Service startup using upstart.) this JAVA_XTRAOPTS string goes through a basic validity check and is then passed to the JVM.

=== Format >= 7.8.4 ===
Beginning from version 7.8.4 the options are split up and sorted into different categories like:

JAVA_OPTS_GC="-XX:+UseConcMarkSweepGC -XX:+UseParNewGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:NewRatio=3 -XX:+DisableExplicitGC"
JAVA_OPTS_LOG="-Dlogback.threadlocal.put.duplicate=false -XX:-OmitStackTraceInFastThrow"
JAVA_OPTS_MEM="-XX:MaxHeapSize=512M -XX:MaxPermSize=256M -XX:+UseTLAB"
JAVA_OPTS_NET="-Dsun.net.inetaddr.ttl=3600 -Dnetworkaddress.cache.ttl=3600 -Dnetworkaddress.cache.negative.ttl=10"
JAVA_OPTS_OSGI="-Dosgi.compatibility.bootdelegation=false"
JAVA_OPTS_SERVER="-server -Djava.awt.headless=true"

JAVA_OPTS_OTHER=""

#JAVA_OPTS_DEBUG="-XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/mnt/heapdump -Xloggc:/var/log/open-xchange/gc.log -verbose:gc -XX:+PrintGCDateStamps \
-XX:+PrintHeapAtGC -XX:+PrintGCApplicationStoppedTime -XX:+PrintTenuringDistribution"

which is much easier to read an maintain. Furthermore we added a useful '''JAVA_OPTS_DEBUG''' configuration parameter that comes preconfigured but unused/commented so that customers can quickly enable the most commonly used debugging options in case they have to provide detailed system information to Open-Xchange.

During the startup of the open-xchange service (step 11 of diagram ''Service startup using systemd'' and step 10 of diagram ''Service startup using upstart''.) all uncommented JAVA_OPTS_* categories will then be evaluated and combined again into a single string of commandline options that can then be passed to the JVM.

=== Upgrade procedure ===
During the upgrade '''7.8.3-revX -> 7.8.4-revY''' existing JAVA_XTRAOPTS will automatically be split up and migrated to the new categories approach. A backup of the file (ox-scriptconf.sh.timestamp) will be created and admins are urged to check for proper migration before deleting the backup file like it's already done with *.dpkg-dist or *.rpmnew files on Debian or RHEL/SLES distributions.

== Encoding detection during JVM startup ==
Add details from https://bugs.open-xchange.com/show_bug.cgi?id=51074

== Warnings during early ==

Early warnings before the start of the JVM and related logging libraries like logback and logstash forwarding are redirected to /var/log/open-xchange/open-xchange-console.log.

=== Limits ===
If the open-xchange startup script isn't able to set the proper limits on CentOS 6 due to e.g. hard limits being enforced via pam_limits you'll see the following warning in the console log during startup

singlenode open-xchange # cat open-xchange-console.log
/opt/open-xchange/sbin/open-xchange: line 115: ulimit: max user processes: cannot modify limit: Operation not permitted

=== Locale ===
If the open-xchange startup script isn't able to detect a unicode locale you'll see a warning like the following (the current charmap might differ)

WARNING: Couldn't detect proper unicode charmap in locale category LC_CTYPE.
WARNING: Current charmap: "ANSI_X3.4-1968" might cause encoding problems.

Sandbox:Middleware Startup/sandbox

2017-04-24T14:38:00Z

Marens:

Sandbox:Middleware Startup/sandbox

2017-04-24T14:24:42Z

Marens:

Sandbox:Middleware Startup/sandbox

2017-04-24T13:49:54Z

Marens:

== Systemd ==

[[File:mw_start_sysd.png|1000px]]

==== Service definition ====
The default service definition as shipped by Open-Xchange is rather concise and easy to read. You can check by looking at the service file that is installed to either /usr/lib/systemd or /lib/systemd depending on your distribution of choice.

singlenode:~ # cat /usr/lib/systemd/system/open-xchange.service
[Unit]
After=remote-fs.target
After=time-sync.target ypbind.service sendmail.service cyrus.service

[Service]
User=open-xchange
PermissionsStartOnly=true
TimeoutStartSec=0
ExecStartPre=/opt/open-xchange/sbin/triggerupdatethemes -u
ExecStart=/opt/open-xchange/sbin/open-xchange
ExecStop=/opt/open-xchange/sbin/shutdown -w
ExecReload=/opt/open-xchange/sbin/triggerreloadconfiguration -d
KillMode=process
LimitNOFILE=65536
LimitNPROC=65536

[Install]
WantedBy=multi-user.target

==== Drop-in configs ====
Drop in configs allow administrators to easily adapt and override default service unit files. So if you want to change the default limits or add additional limits have a look at our default example drop-in config which is located at /etc/systemd/system/open-xchange.service.d/limits.conf

singlenode:~ # cat /etc/systemd/system/open-xchange.service.d/limits.conf
# Override and add options in this file
# See systemd.exec(5) for other limits

[Service]
#LimitNPROC=65536
#LimitNOFILE=65536

The systemd.exec documentation shows a whole lot of options that can be used by admins to adapt the default service to their specific needs.

== System V / Upstart ==

[[File:mw_start_sysv.png|1000px]]

==== Service definition ====

== Apply Resource limits ==

=== Background Infos ===

Several ways exist to restrict resources on a linux system from a global level down to user/groups or even shells and the processes started by them.

==== Sysctl ====
Sysctl is used to modify kernel parameters at runtime. E.g. to set the maximum number of files

$ sysctl -w fs.file-max=100000

To permanently set them append to the main configuration file and reload the settings

$ echo fs.file-max=100000 >> /etc/sysctl.conf
$ sysctl -p

More infos can be found via '''man sysctl'''

==== Limits.conf ====

Allows to restrict resources an a global, group or user level. E.g:

$ cat /etc/security/limits.d/90-nproc.conf
# Default limit for number of user's processes to prevent
# accidental fork bombs.
# See rhbz #432903 for reasoning.

* soft nproc 1024

From '''man limits.conf''':
<blockquote>
''Also, please note that all limit settings are set per login. They are not global, nor are they permanent; existing only for the duration of the session.''
</blockquote>

The limits per login are applied via the ''pam'' stack. See '''man pam''' and '''man pam_limits''' for more details. As those limits are bound to sessions they don't affect most daemons started by our supported init systems or init utils. Most state that they are ignored by design, see [https://bugs.launchpad.net/ubuntu/+source/upstart/+bug/938669/comments/4 upstart], [https://bugzilla.redhat.com/show_bug.cgi?id=754285#c1 systemd] and [https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=302079 start-stop-daemon]

===== Ulimit =====
From '''man bash'''
<blockquote>
ulimit [-HSTabcdefilmnpqrstuvx [limit]]
Provides control over the resources available to the shell and to processes started by it, on systems that allow such control.
</blockquote>
This is what we use in our System V compatible init scripts to increase resources for the open-xchange process across multiple distros. Currently only ''the maximum number of processes'' and ''the maximum number of open file descriptors available to a single user'' are increased via ulimit. The values are specified in '''/opt/open-xchange/ox-scriptconf.sh'''

==== Systemd ====

===== Control Groups=====

Control groups should only affect the OX middleware if you create/manage them yourself of if you are using a ''modern'' distribution that already uses systemd as init.

Citing from the kernel cgroup [https://www.kernel.org/doc/Documentation/cgroup-v2.txt documentation]:

<blockquote>
1-2. What is cgroup?

cgroup is a mechanism to organize processes hierarchically and
distribute system resources along the hierarchy in a controlled and
configurable manner.

cgroup is largely composed of two parts - the core and controllers.
cgroup core is primarily responsible for hierarchically organizing
processes. A cgroup controller is usually responsible for
distributing a specific type of system resource along the hierarchy
although there are utility controllers which serve purposes other than
resource distribution.

cgroups form a tree structure and every process in the system belongs
to one and only one cgroup. All threads of a process belong to the
same cgroup. On creation, all processes are put in the cgroup that
the parent process belongs to at the time. A process can be migrated
to another cgroup. Migration of a process doesn't affect already
existing descendant processes.

Following certain structural constraints, controllers may be enabled or
disabled selectively on a cgroup. All controller behaviors are
hierarchical - if a controller is enabled on a cgroup, it affects all
processes which belong to the cgroups consisting the inclusive
sub-hierarchy of the cgroup. When a controller is enabled on a nested
cgroup, it always restricts the resource distribution further. The
restrictions set closer to the root in the hierarchy can not be
overridden from further away.
</blockquote>

So processes are organized into a tree structure of control groups and controllers are responsible for the distribution of resources. So what kind of controllers exist?

<blockquote>
5. Controllers

5-1. CPU

The "cpu" controllers regulates distribution of CPU cycles. This
controller implements weight and absolute bandwidth limit models for
normal scheduling policy and absolute bandwidth allocation model for
realtime scheduling policy.

5-2. Memory

The "memory" controller regulates distribution of memory.
...
While not completely water-tight, all major memory usages by a given
cgroup are tracked so that the total memory consumption can be
accounted and controlled to a reasonable extent.

5-3. IO

The "io" controller regulates the distribution of IO resources. This
controller implements both weight based and absolute bandwidth or IOPS
limit distribution; however, weight based distribution is available
only if cfq-iosched is in use and neither scheme is available for
blk-mq devices.
</blockquote>

The open-xchange service is simply put into the default ''system.slice'' without applying further limits.

singlenode$ systemd-cgls
├─1 /sbin/init
├─system.slice
│ ├─avahi-daemon.service
│ │ ├─501 avahi-daemon: running [singlenode]
│ │ └─514 avahi-daemon: chroot helper
│ ├─console-kit-daemon.service
│ │ └─16164 /usr/sbin/console-kit-daemon --no-daemon
│ ├─dbus.service
│ │ └─508 /usr/bin/dbus-daemon --system --address=systemd: --nofork --nopidfile --systemd-activation
│ ├─munin-node.service
│ │ └─4290 /usr/bin/perl -wT /usr/sbin/munin-node
│ ├─open-xchange.service
│ │ └─6037 /usr/bin/java -Dsun.net.inetaddr.ttl=3600 -Dnetworkaddress.cache.ttl=3600 -Dnetworkaddress.cache.negative.ttl=10 ...

To check all the details use

singlenode:~ # systemctl show system.slice

==== Limits besides control groups ====
Besides control groups systemd allows you to apply other limits to the execution environment of your service. Here we can apply the limits that would normally be applied via ''limits.conf'' or ''ulimit''. Systemd uses [http://man7.org/linux/man-pages/man2/setrlimit.2.html setrlimit] for this. The options that we set by default are:

* LimitNOFILE
* LimitNPROC

You can check this by looking at the service file that is shipped by default ('''Note:''' Depending on your distro the service files may be located either in /usr/lib/systemd or /lib/systemd. We'll use /usr/lib in this oxpedia article)

singlenode:~ # cat /usr/lib/systemd/system/open-xchange.service
[Unit]
After=remote-fs.target
After=time-sync.target ypbind.service sendmail.service cyrus.service

[Service]
User=open-xchange
PermissionsStartOnly=true
TimeoutStartSec=0
ExecStartPre=/opt/open-xchange/sbin/triggerupdatethemes -u
ExecStart=/opt/open-xchange/sbin/open-xchange
ExecStop=/opt/open-xchange/sbin/shutdown -w
ExecReload=/opt/open-xchange/sbin/triggerreloadconfiguration -d
KillMode=process
LimitNOFILE=65536
LimitNPROC=65536

[Install]
WantedBy=multi-user.target

===== Drop-in configs =====
Drop in configs allow administrators to easily override the default service unit files. So if you want to change the default limits or add additional limits have a look at

singlenode:~ # cat /etc/systemd/system/open-xchange.service.d/limits.conf
# Override and add options in this file
# See systemd.exec(5) for other limits

[Service]
#LimitNPROC=65536
#LimitNOFILE=65536

== Verify limits ==

=== System V ===

Get the proper pid of the java process started for the middleware by either checking '''ps faux'''
root 4103 0.0 0.0 66528 1868 pts/0 S 22:05 0:00 su -s /bin/bash open-xchange -c /opt/open-xchange/sbin/open-xchange
494 4109 9.5 19.8 2224644 381312 ? Ssl 22:05 1:12 \_ /usr/bin/java -Dsun.net.inetaddr.ttl=3600 -Dnetworkaddress.cache.ttl=3600

or programmatically
singlenode:~ # pid=$(pgrep -P $(</var/run/open-xchange.pid) java)

and check the limits applied for that process
singlenode:~ # cat /proc/$pid/limits
Limit Soft Limit Hard Limit Units
Max cpu time unlimited unlimited seconds
Max file size unlimited unlimited bytes
Max data size unlimited unlimited bytes
Max stack size 8388608 unlimited bytes
Max core file size 0 unlimited bytes
Max resident set unlimited unlimited bytes
Max processes 65536 65536 processes
Max open files 65536 65536 files
Max locked memory 65536 65536 bytes
Max address space unlimited unlimited bytes
Max file locks unlimited unlimited locks
Max pending signals 24254 24254 signals
Max msgqueue size 819200 819200 bytes
Max nice priority 0 0
Max realtime priority 0 0
Max realtime timeout unlimited unlimited us

=== Systemd ===
singlenode:~ # systemctl show open-xchange | grep Limit
StartLimitInterval=10000000
StartLimitBurst=5
StartLimitAction=none
MemoryLimit=18446744073709551615
LimitCPU=18446744073709551615
LimitFSIZE=18446744073709551615
LimitDATA=18446744073709551615
LimitSTACK=18446744073709551615
LimitCORE=18446744073709551615
LimitRSS=18446744073709551615
LimitNOFILE=65536
LimitAS=18446744073709551615
LimitNPROC=65536
LimitMEMLOCK=65536
LimitLOCKS=18446744073709551615
LimitSIGPENDING=19827
LimitMSGQUEUE=819200
LimitNICE=0
LimitRTPRIO=0
LimitRTTIME=18446744073709551615

== Open-Xchange middleware on specific distros ==
The support for the mentioned mechanism of resource control differ depending on the distribution and the init system in use.

=== Debian 7 ===
;Init
: System V style
;OX Configurable Limits/Defaults
: nofile, nproc

The mentioned limits can be configured via '''/opt/open-xchange/etc/ox-scriptconf.sh'''. The limits are applied via ulimit in the service's init script. The open-xchange service is finally started via start-stop-daemon which doesn't doesn't consider '''/etc/security/limits.*'''

=== RHEL 6 / CentOS 6 ===
;Init
: Upstart, System V compatible
;OX Configurable Limits/Defaults
: nofile, nproc

The mentioned limits can be configured via '''/opt/open-xchange/etc/ox-scriptconf.sh'''. The limits are applied via ulimit in the service's init script. Furthermore as the open-xchange service is finally started via '''su ... open-xchange''' on this distro a user session is opened via su/pam and the default CentOS pam config reads the '''/etc/security/limits.*''' configuration by loading the pam stack like:

: '''/etc/pam.d/su'''
:: '''-> /etc/pam.d/system-auth'''
::: '''-> pam_limits.so'''

If NPROC isn't configured for the open-xchange-server it's restricted to '''1024''' globally by default to prevent accidental fork bombs, see '''/etc/security/limits.d/90-nproc.conf''' which can result in severe problems modern multithreaded applications.

=== RHEL 7 / CentOS 7 / Debian 8 / SLE 12 ===
;Init
: Systemd
;OX Configurable Limits/Defaults
: nofile, nproc

For systemd the default limits are configured directly in the service's unit file that is shipped by OX and located at '''/usr/lib/systemd/system/open-xchange.service'''. The drop-in config to override or extend the default unit file is located at '''/etc/systemd/system/open-xchange.service.d/limits.conf'''. [https://www.freedesktop.org/software/systemd/man/systemd.exec.html Systemd.exec] shows a whole lot of options that can be used by admins to adapt the default service to their specific needs.

== Warnings ==
If the open-xchange startup script isn't able to set the proper limits on CentOS 6 due to e.g. hard limits being enforced via pam_limits you'll see the following warning in the console log during startup

singlenode open-xchange # cat open-xchange-console.log
/opt/open-xchange/sbin/open-xchange: line 115: ulimit: max user processes: cannot modify limit: Operation not permitted

== Evaluate JVM commandline options ==

== Encoding detection during JVM startup ==
Add details from https://bugs.open-xchange.com/show_bug.cgi?id=51074

Sandbox:Middleware Startup/sandbox

2017-04-24T13:23:14Z

Marens:

== Systemd ==

[[File:mw_start_sysd.png|1000px]]

=== Systemd details explained ===

For systemd the default limits are configured directly in the service's unit file that is shipped by OX and located at /usr/lib/systemd/system/open-xchange.service. The drop-in config to override or extend the default unit file is located at /etc/systemd/system/open-xchange.service.d/limits.conf. Systemd.exec shows a whole lot of options that can be used by admins to adapt the default service to their specific needs.

==== 2 Drop-in Configs ====

== System V / Upstart ==

[[File:mw_start_sysv.png|1000px]]

== System V / Upstart details explained ==

=== Recheck resource limits ===

Explain why it's needed and linkg to resource limits

== Apply Resource limits ==

{{AppSuite:ResourceLimits}}

== Evaluate JVM commandline options ==

== Encoding detection during JVM startup ==
Add details from https://bugs.open-xchange.com/show_bug.cgi?id=51074

File:Mw start sysd.png

2017-04-24T13:07:21Z

Marens:

Sandbox:Middleware Startup/sandbox

2017-04-24T13:07:08Z

Marens: /* Systemd */

= Middleware start =

== Systemd ==

[[File:mw_start_sysd.png|1000px]]

== System V / Upstart ==

[[File:mw_start_sysv.png|1000px]]

== Details explained ==

=== Evaluate JVM commandline options ===

=== Apply Resource limits ===

{{AppSuite:ResourceLimits}}

=== Encoding detection during JVM startup ===
Add details from https://bugs.open-xchange.com/show_bug.cgi?id=51074

File:Mw start sysv.png

2017-04-24T13:06:39Z

Marens: Marens uploaded a new version of "File:Mw start sysv.png"

middleware start with system v / upstart

Sandbox:Middleware Startup/sandbox

2017-04-21T12:15:03Z

Marens: /* Encoding detection during JVM startup */

= Middleware start =

== Systemd ==

[[File:mw_start.png|1000px]]

== System V / Upstart ==

[[File:mw_start_sysv.png|1000px]]

== Details explained ==

=== Evaluate JVM commandline options ===

=== Apply Resource limits ===

{{AppSuite:ResourceLimits}}

=== Encoding detection during JVM startup ===
Add details from https://bugs.open-xchange.com/show_bug.cgi?id=51074

Sandbox:Middleware Startup/sandbox

2017-04-20T23:07:31Z

Marens: