https://oxpedia.org/wiki/api.php?action=feedcontributions&user=Marens&feedformat=atomOpen-Xchange - User contributions [en]2024-03-28T16:51:52ZUser contributionsMediaWiki 1.31.0https://oxpedia.org/wiki/index.php?title=AppSuite:Open-Xchange_Installation_Guide_for_RHEL8&diff=28418AppSuite:Open-Xchange Installation Guide for RHEL82023-10-09T16:05:04Z<p>Marens: /* Apache and SELinux */</p>
<hr />
<div>{{Version|7.10.6}}<br />
<br />
= Open-Xchange App Suite on RedHat Enterprise Linux 8 =<br />
<br />
{{QuickInstIntro|release=appsuite}}<br />
<br />
= Requirements =<br />
<br />
* Plain installed RedHat Enterprise Linux 8 with latest updates<br />
* Valid access to the RedHat Network<br />
* A configured internet connection<br />
* A supported Java Virtual Machine ([http://oxpedia.org/wiki/index.php?title=AppSuite:OX_System_Requirements#Server_Platforms learn more])<br />
* <code>vim</code> is not installed by default on RHEL8. If you want to copy & paste the commands from this article into a shell window, you need to <code>dnf install vim</code> first.<br />
<br />
= Database installation =<br />
<br />
Please consult our [[OXLoadBalancingClustering_Database#Standalone_database_setup|database installation instructions]] for information on how to install a database on the local system.<br />
<br />
Before proceeding, make sure the local machine has got a working MySQL service in one of the supported versions / flavors with the configuration / tunings applied as mentioned on our [[My.cnf|corresponding page]].<br />
<br />
= Enabling required RedHat Repositories =<br />
<br />
Ensure that your Redhat system is subscribed by e.g. following this article https://access.redhat.com/solutions/253273<br />
<br />
{{AddReposRHEL|rhelname=RHEL8|release=appsuite}}<br />
<br />
= Updating repositories and installing packages =<br />
<br />
Reload the package index. This will download the package descriptions available at the software repositories:<br />
<br />
$ dnf update<br />
<br />
The following command starts the download and installation process of all required package for Open-Xchange deployment:<br />
<br />
{{OXPackageInstallation|installer=dnf|release=appsuite}}<br />
<br />
{{OXConfiguration}}<br />
<br />
{{oxinstaller|connector=http}}<br />
<br />
After initializing the configuration, start the Open-Xchange service by executing:<br />
<br />
$ systemctl start open-xchange<br />
<br />
{{OXRegister|globaldb=true}}<br />
<br />
= Configure services =<br />
<br />
{{ApacheOXModsIntro|connector=http}}<br />
<br />
The default installation of the Apache webserver on RHEL provides a welcome screen which is not necessary for server operation, it can be removed by deleting the corresponding configuration file:<br />
<br />
$ rm /etc/httpd/conf.d/welcome.conf<br />
<br />
{{Template:ApacheAppSuiteConf|connector=http|connectorConf=/etc/httpd/conf.d/proxy_http.conf|apacheconf=/etc/httpd/conf.d/ox.conf|docroot=/var/www/html/|loadmodule=LoadModule proxy_http_module modules/mod_proxy_http.so|syncProxyName=eas_oxcluster}}<br />
<br />
== Apache and SELinux ==<br />
<br />
When using SELinux make sure to allow Apache to connect to the middleware process on port 8009. One way to achieve this could be:<br />
<br />
$ setsebool -P httpd_can_network_connect 1<br />
<br />
The following links contain more infos:<br />
* [https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/html/selinux_users_and_administrators_guide/chap-managing_confined_services-the_apache_http_server SELinux: The Apache HTTP Server]<br />
* [https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/html/selinux_users_and_administrators_guide/sect-managing_confined_services-the_apache_http_server-booleans SELinux: The Apache HTTP Server - Booleans]<br />
<br />
After the configuration is done, restart the Apache webserver<br />
<br />
$ systemctl start httpd<br />
<br />
== Apache Setting for more concurrent Connections ==<br />
<br />
By default apache2 is configured to support 150 concurrent connections. This forces all parallel requests beyond that limit to wait. Especially if, for example, active sync clients maintain a permanent connection for push events to arrive. The following article explains how that can be done<br />
<br />
[[Tune_apache2_for_more_concurrent_connections|Apache Setting for more concurrent Connections]]<br />
<br />
= Adding services to runlevels =<br />
<br />
The new services are now installed and configured, but to make them start up on a server boot, they need to be added to some runlevels:<br />
<br />
$ systemctl enable mariadb<br />
$ systemctl enable httpd<br />
$ systemctl enable open-xchange<br />
<br />
{{ContextUserAndLogs}}<br />
<br />
= Installing Open-Xchange Update packages =<br />
<br />
Please read [[UpdatingOXPackages]] on how to get access to the latest Open-Xchange packages.<br />
<br />
[[Category: AppSuite]]</div>Marenshttps://oxpedia.org/wiki/index.php?title=AppSuite:Open-Xchange_Installation_Guide_for_RHEL8&diff=28415AppSuite:Open-Xchange Installation Guide for RHEL82023-10-09T16:04:19Z<p>Marens: </p>
<hr />
<div>{{Version|7.10.6}}<br />
<br />
= Open-Xchange App Suite on RedHat Enterprise Linux 8 =<br />
<br />
{{QuickInstIntro|release=appsuite}}<br />
<br />
= Requirements =<br />
<br />
* Plain installed RedHat Enterprise Linux 8 with latest updates<br />
* Valid access to the RedHat Network<br />
* A configured internet connection<br />
* A supported Java Virtual Machine ([http://oxpedia.org/wiki/index.php?title=AppSuite:OX_System_Requirements#Server_Platforms learn more])<br />
* <code>vim</code> is not installed by default on RHEL8. If you want to copy & paste the commands from this article into a shell window, you need to <code>dnf install vim</code> first.<br />
<br />
= Database installation =<br />
<br />
Please consult our [[OXLoadBalancingClustering_Database#Standalone_database_setup|database installation instructions]] for information on how to install a database on the local system.<br />
<br />
Before proceeding, make sure the local machine has got a working MySQL service in one of the supported versions / flavors with the configuration / tunings applied as mentioned on our [[My.cnf|corresponding page]].<br />
<br />
= Enabling required RedHat Repositories =<br />
<br />
Ensure that your Redhat system is subscribed by e.g. following this article https://access.redhat.com/solutions/253273<br />
<br />
{{AddReposRHEL|rhelname=RHEL8|release=appsuite}}<br />
<br />
= Updating repositories and installing packages =<br />
<br />
Reload the package index. This will download the package descriptions available at the software repositories:<br />
<br />
$ dnf update<br />
<br />
The following command starts the download and installation process of all required package for Open-Xchange deployment:<br />
<br />
{{OXPackageInstallation|installer=dnf|release=appsuite}}<br />
<br />
{{OXConfiguration}}<br />
<br />
{{oxinstaller|connector=http}}<br />
<br />
After initializing the configuration, start the Open-Xchange service by executing:<br />
<br />
$ systemctl start open-xchange<br />
<br />
{{OXRegister|globaldb=true}}<br />
<br />
= Configure services =<br />
<br />
{{ApacheOXModsIntro|connector=http}}<br />
<br />
The default installation of the Apache webserver on RHEL provides a welcome screen which is not necessary for server operation, it can be removed by deleting the corresponding configuration file:<br />
<br />
$ rm /etc/httpd/conf.d/welcome.conf<br />
<br />
{{Template:ApacheAppSuiteConf|connector=http|connectorConf=/etc/httpd/conf.d/proxy_http.conf|apacheconf=/etc/httpd/conf.d/ox.conf|docroot=/var/www/html/|loadmodule=LoadModule proxy_http_module modules/mod_proxy_http.so|syncProxyName=eas_oxcluster}}<br />
<br />
== Apache and SELinux ==<br />
<br />
When using selinux make sure to allow Apache to connect to the middleware process on port 8009. One way to achieve this could be:<br />
<br />
$ setsebool -P httpd_can_network_connect 1<br />
<br />
The following links contain more infos:<br />
* [https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/html/selinux_users_and_administrators_guide/chap-managing_confined_services-the_apache_http_server SELinux: The Apache HTTP Server]<br />
* [https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/html/selinux_users_and_administrators_guide/sect-managing_confined_services-the_apache_http_server-booleans SELinux: The Apache HTTP Server - Booleans]<br />
<br />
After the configuration is done, restart the Apache webserver<br />
<br />
$ systemctl start httpd<br />
<br />
== Apache Setting for more concurrent Connections ==<br />
<br />
By default apache2 is configured to support 150 concurrent connections. This forces all parallel requests beyond that limit to wait. Especially if, for example, active sync clients maintain a permanent connection for push events to arrive. The following article explains how that can be done<br />
<br />
[[Tune_apache2_for_more_concurrent_connections|Apache Setting for more concurrent Connections]]<br />
<br />
= Adding services to runlevels =<br />
<br />
The new services are now installed and configured, but to make them start up on a server boot, they need to be added to some runlevels:<br />
<br />
$ systemctl enable mariadb<br />
$ systemctl enable httpd<br />
$ systemctl enable open-xchange<br />
<br />
{{ContextUserAndLogs}}<br />
<br />
= Installing Open-Xchange Update packages =<br />
<br />
Please read [[UpdatingOXPackages]] on how to get access to the latest Open-Xchange packages.<br />
<br />
[[Category: AppSuite]]</div>Marenshttps://oxpedia.org/wiki/index.php?title=AppSuite:Open-Xchange_Installation_Guide_for_RHEL8&diff=28385AppSuite:Open-Xchange Installation Guide for RHEL82023-10-09T11:57:23Z<p>Marens: dnf is default since rhel8</p>
<hr />
<div>{{Version|7.10.6}}<br />
<br />
= Open-Xchange App Suite on RedHat Enterprise Linux 8 =<br />
<br />
{{QuickInstIntro|release=appsuite}}<br />
<br />
= Requirements =<br />
<br />
* Plain installed RedHat Enterprise Linux 8 with latest updates<br />
* Valid access to the RedHat Network<br />
* A configured internet connection<br />
* A supported Java Virtual Machine ([http://oxpedia.org/wiki/index.php?title=AppSuite:OX_System_Requirements#Server_Platforms learn more])<br />
* <code>vim</code> is not installed by default on RHEL8. If you want to copy & paste the commands from this article into a shell window, you need to <code>dnf install vim</code> first.<br />
<br />
= Database installation =<br />
<br />
Please consult our [[OXLoadBalancingClustering_Database#Standalone_database_setup|database installation instructions]] for information on how to install a database on the local system.<br />
<br />
Before proceeding, make sure the local machine has got a working MySQL service in one of the supported versions / flavors with the configuration / tunings applied as mentioned on our [[My.cnf|corresponding page]].<br />
<br />
= Enabling required RedHat Repositories =<br />
<br />
Ensure that your Redhat system is subscribed by e.g. following this article https://access.redhat.com/solutions/253273<br />
<br />
{{AddReposRHEL|rhelname=RHEL8|release=appsuite}}<br />
<br />
= Updating repositories and installing packages =<br />
<br />
Reload the package index. This will download the package descriptions available at the software repositories:<br />
<br />
$ dnf update<br />
<br />
The following command starts the download and installation process of all required package for Open-Xchange deployment:<br />
<br />
{{OXPackageInstallation|installer=dnf|release=appsuite}}<br />
<br />
{{OXConfiguration}}<br />
<br />
{{oxinstaller|connector=http}}<br />
<br />
After initializing the configuration, start the Open-Xchange service by executing:<br />
<br />
$ systemctl start open-xchange<br />
<br />
{{OXRegister|globaldb=true}}<br />
<br />
= Configure services =<br />
<br />
{{ApacheOXModsIntro|connector=http}}<br />
<br />
The default installation of the Apache webserver on RHEL provides a welcome screen which is not necessary for server operation, it can be removed by deleting the corresponding configuration file:<br />
<br />
$ rm /etc/httpd/conf.d/welcome.conf<br />
<br />
{{Template:ApacheAppSuiteConf|connector=http|connectorConf=/etc/httpd/conf.d/proxy_http.conf|apacheconf=/etc/httpd/conf.d/ox.conf|docroot=/var/www/html/|loadmodule=LoadModule proxy_http_module modules/mod_proxy_http.so|syncProxyName=eas_oxcluster}}<br />
<br />
After the configuration is done, restart the Apache webserver<br />
<br />
$ systemctl start httpd<br />
<br />
== Apache Setting for more concurrent Connections ==<br />
<br />
By default apache2 is configured to support 150 concurrent connections. This forces all parallel requests beyond that limit to wait. Especially if, for example, active sync clients maintain a permanent connection for push events to arrive. The following article explains how that can be done<br />
<br />
[[Tune_apache2_for_more_concurrent_connections|Apache Setting for more concurrent Connections]]<br />
<br />
= Adding services to runlevels =<br />
<br />
The new services are now installed and configured, but to make them start up on a server boot, they need to be added to some runlevels:<br />
<br />
$ systemctl enable mariadb<br />
$ systemctl enable httpd<br />
$ systemctl enable open-xchange<br />
<br />
{{ContextUserAndLogs}}<br />
<br />
= Installing Open-Xchange Update packages =<br />
<br />
Please read [[UpdatingOXPackages]] on how to get access to the latest Open-Xchange packages.<br />
<br />
[[Category: AppSuite]]</div>Marenshttps://oxpedia.org/wiki/index.php?title=AppSuite:Documents_Installation_Guide_7103&diff=24729AppSuite:Documents Installation Guide 71032019-06-13T17:25:32Z<p>Marens: typo</p>
<hr />
<div>{{Version|7.10.3}}<br />
<br />
= Download & Installation OX Documents =<br />
<br />
=== General Information ===<br />
<br />
OX Documents are browser based, cloud ready, text, spreadsheet and presentation products that can work with Microsoft Office and OpenOffice documents in a lossless way. And you can also collaborate with other people to edit shared documents on various devices.<br />
<br />
== Requirements ==<br />
<br />
See the [[AppSuite:OX_System_Requirements|Open-Xchange software requirements page]] for details.<br />
<br />
OX Documents needs to be installed in an OX App Suite configuration with a [[AppSuite:Grizzly|backend based on Grizzly]] (including hazelcast as installed and activated by default).<br />
<br />
For compatibility with the legacy MS binary format files (.DOC, .XLS, .PPT) and for printing functionality in OX Text the document converter components (incl. readerengine) are required. These are not available in the Community Version of OX AppSuite.<br />
<br />
== Mandatory Modules ==<br />
<br />
Additional functional modules need to be installed separately: <br />
<br />
* The Document Converter API: See [[AppSuite:DocumentConverterAPIInstall|Document converter API installation instructions]]<br />
<br />
== Installation ==<br />
<br />
The OX Documents deployment consists of the packages <tt>open-xchange-documents-backend</tt> and <tt>open-xchange-documents-ui</tt>. The <tt>open-xchange-documents-backend</tt> package also requires and installs these packages:<br />
<br />
* <tt>open-xchange-file-distribution</tt><br />
* <tt>open-xchange-sessionstorage-hazelcast</tt><br />
<br />
The following Packages are no longer needed (since 7.10.3)<br />
* <tt>open-xchange-realtime-core</tt><br />
* <tt>open-xchange-realtime-json</tt><br />
<br />
<br />
=== Redhat Enterprise Linux 6 or CentOS 6 ===<br />
<br />
Add the following repositories to your Open-Xchange yum configuration:<br />
<br />
{{for loop||call=YUMRepo|pv=reponame|pc1n=path|pc1v=products/appsuite/stable|pc2n=rhelname|pc2v=RHEL6|office|office-web|documentconverter-api}}<br />
<br />
if you have a valid maintenance subscription, please add also following repositories by replacing the credentials.<br />
<br />
{{for loop||call=YUMRepo|pv=reponame|pc1n=path|pc1v=products/appsuite/stable|pc2n=rhelname|pc2v=RHEL6|pc3n=ldbaccount|pc3v=LDBUSER:LDBPASSWORD|office/updates|office-web/updates|documentconverter-api/updates}}<br />
$ yum install open-xchange-documents-backend open-xchange-documents-ui open-xchange-documents-ui-static<br />
<br />
=== Redhat Enterprise Linux 7 or CentOS 7 ===<br />
<br />
Add the following repositories to your Open-Xchange yum configuration:<br />
<br />
{{for loop||call=YUMRepo|pv=reponame|pc1n=path|pc1v=products/appsuite/stable|pc2n=rhelname|pc2v=RHEL7|office|office-web|documentconverter-api}}<br />
<br />
if you have a valid maintenance subscription, please add also following repositories by replacing the credentials.<br />
<br />
{{for loop||call=YUMRepo|pv=reponame|pc1n=path|pc1v=products/appsuite/stable|pc2n=rhelname|pc2v=RHEL7|pc3n=ldbaccount|pc3v=LDBUSER:LDBPASSWORD|office/updates|office-web/updates|documentconverter-api/updates}}<br />
<br />
$ yum install open-xchange-documents-backend open-xchange-documents-ui open-xchange-documents-ui-static<br />
<br />
=== Debian GNU/Linux 8.0 ===<br />
<br />
Add the following repositories to your Open-Xchange apt configuration:<br />
<br />
{{for loop||call=APTRepo|pv=reponame|pc1n=path|pc1v=products/appsuite/stable|pc2n=debianname|pc2v=DebianJessie|office|office-web|documentconverter-api}}<br />
<br />
if you have a valid maintenance subscription, please add also following repositories by replacing the credentials.<br />
<br />
{{for loop||call=APTRepo|pv=reponame|pc1n=path|pc1v=products/appsuite/stable|pc2n=debianname|pc2v=DebianJessie|pc3n=ldbaccount|pc3v=LDBUSER:LDBPASSWORD|office/updates|office-web/updates|documentconverter-api/updates}}<br />
<br />
$ apt-get update<br />
$ apt-get install open-xchange-documents-backend open-xchange-documents-ui open-xchange-documents-ui-static<br />
<br />
=== Debian GNU/Linux 9.0 (valid from v7.10) ===<br />
<br />
Add the following repositories to your Open-Xchange apt configuration:<br />
<br />
{{for loop||call=APTRepo|pv=reponame|pc1n=path|pc1v=products/appsuite/stable|pc2n=debianname|pc2v=DebianStretch|office|office-web|documentconverter-api}}<br />
<br />
if you have a valid maintenance subscription, please add also following repositories by replacing the credentials.<br />
<br />
{{for loop||call=APTRepo|pv=reponame|pc1n=path|pc1v=products/appsuite/stable|pc2n=debianname|pc2v=DebianStretch|pc3n=ldbaccount|pc3v=LDBUSER:LDBPASSWORD|office/updates|office-web/updates|documentconverter-api/updates}}<br />
<br />
$ apt-get update<br />
$ apt-get install open-xchange-documents-backend open-xchange-documents-ui open-xchange-documents-ui-static<br />
<br />
=== SUSE Linux Enterprise Server 12 === <br />
<br />
{{for loop||call=SUSERepo|pv=reponame|pc1n=path|pc1v=products/appsuite/stable|pc2n=susename|pc2v=SLE_12|office|office-web|documentconverter-api}}<br />
<br />
if you have a valid maintenance subscription, please add also following repositories by replacing the credentials.<br />
<br />
{{for loop||call=SUSERepo|pv=reponame|pc1n=path|pc1v=products/appsuite/stable|pc2n=susename|pc2v=SLE_12|pc3n=ldbaccount|pc3v=LDBUSER:LDBPASSWORD|office/updates|office-web/updates|documentconverter-api/updates}}<br />
<br />
$ zypper ref<br />
$ zypper install open-xchange-documents-backend open-xchange-documents-ui open-xchange-documents-ui-static<br />
<br />
=== Univention Corporate Server ===<br />
<br />
OX Text for OX App Suite for UCS is available in the Univention App Center. Please check the UMC module App Center for the installation of the OX Documents at your already available environment.<br />
<br />
<br />
== Documents Collaboration Service ==<br />
<br />
With 7.10.3 the Documents Collaboration Service has been introduced.<br />
<br />
The Documents Collaboration Service needs to be installed, configured and run. It supports collaboration across Middleware nodes but is also mandatory for a small (even single node) environments.<br />
The server can be run as one additional service (JVM) or for larger deployments several instances can be clustered.<br />
The implementation is based on Spring Boot and uses the Java-based messaging server Apache ActiveMQ.<br />
<br />
Note:</br><br />
For upgrades from earlier versions than 7.10.2 the hazelcast cluster requires a complete shutdown, since the internal OX Documents data has changed. A rolling upgrade is not possible.<br />
<br />
=== Documents Collaboration Server (DCS) ===<br />
<br />
prepare documents-collaboration repo as described above.<br />
<br />
The DCS will be installed via the package:<br />
<pre><br />
open-xchange-documents-collaboration-server<br />
</pre><br />
<br />
Target directory is <b>/usr/share/open-xchange-documents-collaboration</b><br />
<br />
Configuration file is <b>/etc/documents-collaboration/dcs.properties</b>. The following entries need to be reviewed and values adjusted to the setup.<br><br />
Configure bind host IP and JMS Port to listen to<br />
<pre><br />
dcs.host = 192.168.10.2 # Interface of the DCS node<br />
dcs.port = 61616<br />
</pre> <br />
<br />
A DCS writes its connection data to a database for discovery by middleware nodes<br />
<pre><br />
db.host=192.168.10.172<br />
db.port=3306<br />
db.schema=dcsdb<br />
db.username=<db_user><br />
db.password=<db_password><br />
</pre><br />
For management of DCS hosts the database used by the middleware can be configured.<br />
<br />
The default logging path is set in //etc//documents-collaboration/logback-spring.xml and points to //var//log//open-xchange//documents-collaboration//documents-collaboration.log<br />
<br />
The DCS Service will be started with <br />
systemctl start open-xchange-documents-collaboration.service<br />
<br />
A command line tool allows to check the values in the database<br />
<pre><br />
# /usr/share/open-xchange-documents-collaboration/bin/documents-collaboration-admin.sh -l<br />
+---------------------+---------------+---------------+---------+<br />
| ID | Host | Interface | JMSPort |<br />
+---------------------+---------------+---------------+---------+<br />
| 192.168.10.25:61616 | 192.168.10.25 | 192.168.10.25 | 61616 |<br />
+---------------------+---------------+---------------+---------+<br />
</pre><br />
<br />
<br />
===Middleware node===<br />
<br />
no new pkg repo, no new pkgs required for middleware<br />
<br />
Discovery of the DCS service is configured in <b>documents-collaboration-client.properties</b>.<br />
Add the appropriate values to access the dcsdb Database as set for the DCS above:<br />
<pre><br />
com.openexchange.dcs.client.database.connectionURL=jdbc:mysql://databasehost:3306/dcsdb<br />
com.openexchange.dcs.client.database.userName=db_user<br />
com.openexchange.dcs.client.database.password=db_password<br />
</pre><br />
<br />
A command line tool allows to check the values in the database<br />
<pre><br />
$ /opt/open-xchange/sbin/documents-collaboration-admin -l<br />
+---------------------+---------------+---------------+---------+<br />
| ID | Host | Interface | JMSPort |<br />
+---------------------+---------------+---------------+---------+<br />
| 192.168.10.25:61616 | 192.168.10.25 | 192.168.10.25 | 61616 |<br />
+---------------------+---------------+---------------+---------+<br />
</pre><br />
<br />
== Printing and legacy MS binary formats Editing ==<br />
<br />
For .DOC/.XLS/.PPT and/or printing support from OX Documents the Document Converter and readerengine components have to be installed separately (license key required)<br />
<br />
* See [[AppSuite:DocumentConverterInstall|Document converter / readerengine installation instructions]]<br />
<br />
== Monitoring ==<br />
<br />
OX Documents offers runtime information via JMX about the document editors as well as the OX Documentconverter. This [[AppSuite:DocumentsMonitoring|article]] guides to the information how to access JMX data, configure and use Jolokia and integrate with munin.<br />
<br />
== Configuration ==<br />
<br />
=== Permissions ===<br />
<br />
To enable the OX Documents Applications for OX Drive the associated permission has to be set.<br />
<br />
The default setting for all users is changed in the file '''documents.properties''' in the directory ''/opt/open-xchange/etc''.<br />
<br />
After installation the functionality is disabled:<br />
<pre><br />
# Enables or disables the "text" module capability globally.<br />
# com.openexchange.capability.text=true<br />
<br />
# Enables or disables the "spreadsheet" module capability globally.<br />
# com.openexchange.capability.spreadsheet=true<br />
<br />
# Enables or disables the "presentation" module capability globally.<br />
# com.openexchange.capability.presentation=true<br />
<br />
</pre><br />
<br />
The following line enables the functionality:<br />
<pre><br />
# Enables or disables the "text" module capability globally.<br />
com.openexchange.capability.text=true<br />
</pre><br />
<br />
Starting with 7.6.2, you can disable the text portal app. In '''permissions.properties''':<br />
<pre><br />
permissions=textportaldisabled<br />
</pre><br />
<br />
=== Collaboration on shared documents ===<br />
<br />
If OX Documents functionality is used for guest users in a cluster of application servers, this setting needs to be adjusted to false in order to also have guest sessions available in the distributed storage.<br />
<br />
in ''/opt/open-xchange/etc/sharing.properties'' <br />
<br />
<pre><br />
# Specifies whether guest sessions are treated as transient or not. Transient<br />
# sessions are only held in the short-term session containers, and are not put<br />
# into the distributed session storage. Defaults to "true".<br />
com.openexchange.share.transientSessions=false<br />
</pre><br />
<br />
com.openechange.share.transientSessions has to be set to false.<br />
<br />
=== Spell Checking===<br />
<br />
OX Text and OX Presentation use ''hunspell'' for spell checking functionality.<br />
By default Spellchecking is enabled in ''/opt/open-xchange/etc/settings/office.properties''<br />
with the following entry<br />
<pre><br />
# Determines whether online spelling is enabled for office documents.<br />
# Possible values: true|false<br />
# If this property is missing true is taken.<br />
io.ox/office//module/spellingEnabled=true<br />
</pre><br />
<br />
After installation of the ''hunspell'' package for your platform the shared library and the US English dictionaries are available. Please check the file location the pkg files were installed into. (For instance on CentOS, use rpm -q –filesbypkg hunspell and verify the path of the lib files to be configured below)<br />
Also verify that the Hunspell dictonary is present and not renamed to "myspell" etc.. if the dictionary is renamed, you will have to use that in the config below. <br />
<br />
Additional dictionaries are published for your platform as packages hunspell-de-de, hunspell-fr, ...<br />
<br />
Spell checking has to be configured in the file ''/opt/open-xchange/etc/hunspell.properties''.<br />
The values for the shared library file and dictionary path have to be set according to the platform.<br />
<br />
This example shows the values for Debian:<br />
<br />
<pre><br />
# The name and location of the hunspell library<br />
# Default value: "/usr/lib/x86_64-linux-gnu/libhunspell-1.3.so.0"<br />
com.openexchange.spellchecker.hunspell.library=/usr/lib/x86_64-linux-gnu/libhunspell-1.4.so.0<br />
# The location of the dictionaries<br />
# Default value: "/usr/share/hunspell"<br />
com.openexchange.spellchecker.hunspell.dictionaries=/usr/share/hunspell<br />
</pre><br />
<br />
If spellchecking is enabled and the library path or dictionary path is wrong a warning is logged.<br />
<br />
If a language is not supported by a hunspell dictionary (or produces error messages in the console log) myspell dictionaries are a working alternative.<br />
<br />
=== Templates ===<br />
<br />
OX Documents support different levels to provide document templates for users.<br />
<br />
A number of ready-to-use templates for letters, memos, resumes, home budget, presentations, etc is included with the system. These templates are made available in a directory at each system node by installation of the package open-xchange-documents-templates. The path to this directory is defined in the office.properties file <br />
<br />
<pre><br />
# Defines the absolute path where document templates are located inside the local (!) file system.<br />
# The templates are displayed in the documents applications.<br />
# If this property is missing a default '/opt/open-xchange/templates/documents' is taken.<br />
io.ox/office//module/templatePath = /opt/open-xchange/templates/documents<br />
</pre><br />
<br />
The content in the directory specified by the path must comply to the following rules:<br />
<br />
<pre><br />
/opt/open-xchange/templates/documents <br />
| <br />
----> text [application type]<br />
| | <br />
| ----> en-US [language-region]<br />
| | <br />
| ----> en [language fallback, en is also fallback for all other non-convered languages]<br />
| | <br />
| ----> de-DE<br />
| | <br />
| ----> de<br />
| | <br />
| ----> common [language independent, used in addition to the language specific ones]<br />
| <br />
----> spreadsheet<br />
| <br />
----> presentation<br />
</pre><br />
<br />
A context administrator can additionally provide global templates for all users in the same context.<br />
If logged in with this role folders can be added to or removed from the list of global template folders. For regular users the list is displayed without the permission to modify it.<br />
<br />
[[File:template-folder.png|border|600px|caption]]<br />
<br />
Users can create and manage their own templates. These user templates are stored and retrieved from the user’s root folder (“My files”). In addition to the root folder the user can define more template folders, sub-folders or even external folders in the “Documents” section of the “Settings” menu. This helps to organize templates and allows cleanup of the root folder by removal of templates.<br />
<br />
For installations of OX Spreadsheet based on Appsuite 7.8.0 and 7.8.1 please see [http://oxpedia.org/wiki/index.php?title=AppSuite:Spreadsheet_Installation_Guide_7_8_1 here] for different installation modes.</div>Marenshttps://oxpedia.org/wiki/index.php?title=AppSuite:Mobile_API_Facade&diff=23536AppSuite:Mobile API Facade2017-07-10T08:22:48Z<p>Marens: /* Proxy configuration */</p>
<hr />
<div>= OX Facade for Mobile =<br />
<br />
== General Information ==<br />
<br />
The Mobile API Facade is a server component that brings the new native mobile mail apps together with the OX App Suite. We’ve built the façade based on the technology used and proven in the OX App Suite middleware. The facade is developed in Java, utilizing the OSGI Framework.<br />
<br />
The Facade provides offline friendly HTTP interface by doing the work, the connections through the HTTP API, to the OX App Suite and providing only the data the offline capable clients need. In some cases multiple requests to the OX App Suite are combined into one for its clients.The facade also offers a method to tell the clients that information on the server haven’t changed since the last time the client asked for it. This reduces the amount of data to transmit to the clients under certain circumstances, especially important in mobile networks were bandwidth (and overall traffic) is limited. Thanks to facade some functionality can be shared among all clients and need only get implemented once. One example for this is the teaser text extraction, and HTML mail handling in general.The facade also provides a pluggable authentication system. In the default case, login request are just forwarded to the middleware. In more advanced use-cases, the login request is forwarded to IDM of the customer and an OX session is created from the access token from the IDM. For clients this is pretty straightforward.<br />
<br />
== Requirements ==<br />
<br />
The OX Facade has to be installed alongside an OX App Suite installation. It requires at least OX App Suite v7.8.4.<br />
<br />
== OX Facade API ==<br />
<br />
Further information about the OX Facade API can be found at:<br />
https://documentation.open-xchange.com/components/facade/1.0.0/<br />
<br />
= OX Mail Server-side Installation and Configuration on OX App Suite v7.8.x =<br />
<br />
This chapter describes how the backend components of OX Mail are installed and configured on the server.<br />
<br />
== Available packages ==<br />
<br />
OX Facade is available with the following backend packages:<br />
<br />
* ''open-xchange-mobile-api-facade''<br />
<br />
Installation on the server varies depending on the underlying distribution, details are available in the following chapters.<br />
<br />
=== Redhat Enterprise Linux 6 or CentOS 6 ===<br />
<br />
Add the following repositories to your Open-Xchange yum configuration:<br />
<br />
{{for loop||call=YUMRepo|pv=reponame|pc1n=path|pc1v=products/mobile-api-facade/stable|pc2n=rhelname|pc2v=RHEL6|mobile-api-facade}}<br />
<br />
$ yum install open-xchange-mobile-api-facade<br />
<br />
=== Redhat Enterprise Linux 7 or CentOS 7 ===<br />
<br />
Add the following repositories to your Open-Xchange yum configuration:<br />
<br />
{{for loop||call=YUMRepo|pv=reponame|pc1n=path|pc1v=products/mobile-api-facade/stable|pc2n=rhelname|pc2v=RHEL7|mobile-api-facade}}<br />
<br />
$ yum install open-xchange-mobile-api-facade<br />
<br />
=== Debian GNU/Linux 8.0 ===<br />
<br />
Add the following repositories to your Open-Xchange apt configuration:<br />
<br />
{{for loop||call=APTRepo|pv=reponame|pc1n=path|pc1v=products/mobile-api-facade/stable|pc2n=debianname|pc2v=DebianJessie|mobile-api-facade}}<br />
<br />
$ apt-get update<br />
$ apt-get install open-xchange-mobile-api-facade<br />
<br />
=== SUSE Linux Enterprise Server 12 === <br />
<br />
{{for loop||call=SUSERepo|pv=reponame|pc1n=path|pc1v=products/mobile-api-facade/stable|pc2n=susename|pc2v=SLE_12|mobile-api-facade}}<br />
<br />
$ zypper ref<br />
$ zypper install open-xchange-mobile-api-facade<br />
<br />
= Configuration OX Facade for Mobile =<br />
<br />
== Introduction ==<br />
<br />
To be able to use the native mail apps the Mobile API Facade needs to be installed in front of the OX App Suite middleware. This document describes how to configure the Mobile API Facade.<br />
<br />
The Mobile API Facade stores its configuration in the files <code>/opt/open-xchange/mobile-api-facade/etc/facade.properties</code> (the global configuration) and in <code>/opt/open-xchange/mobile-api-facade/etc/mobile-api-facade-config.yml</code> (hostname specific configuration). Both files support the same configuration properties as can be seen on https://documentation.open-xchange.com/components/facade/config/1.0.0/.<br />
<br />
== Connection to the OX App Suite Middleware ==<br />
<br />
After installation of the facade package ("open-xchange-mobile-api-facade") the property <code>com.openexchange.mobile.api.facade.MiddlewareBaseUrl</code> needs to get set to the correct URL. This property needs to be explicitly configured by the administrator. It has no default value. Its possible to connect the Mobile API Facade directly to a middleware process, but this is highly discouraged. The Mobile API Facade should always connect to a middleware process through a load balancer.<br />
<br />
An example:<br />
<br />
com.openexchange.mobile.api.facade.MiddlewareBaseUrl=https://appsuite.example.com/appsuite/api<br />
<br />
After this configuration the open-xchange-mobile-api-facade needs to get restarted.<br />
<br />
== Proxy configuration ==<br />
<br />
<Proxy balancer://oxcluster_facade><br />
Order Allow,Deny<br />
Allow from all<br />
BalancerMember http://appsuite-middleware1.example.com:8007 timeout=100 smax=0 ttl=60 retry=60 <br />
loadfactor=50 keepalive=On route=FOX1<br />
BalancerMember http://appsuite-middleware2.example.com:8007 timeout=100 smax=0 ttl=60 retry=60 <br />
loadfactor=50 keepalive=On route=FOX2<br />
<br />
ProxySet stickysession=JSESSIONID|jsessionidscolonpathdelim=On<br />
SetEnv proxy-initial-not-pooled<br />
SetEnv proxy-sendchunked<br />
</Proxy><br />
<br />
ProxyPass /services/api-facade balancer://oxcluster_facade/services/api-facade<br />
<br />
== Traffic compression ==<br />
<br />
The clients need to exchange a lot of data with the facade to accomplish the task of a mail client. This traffic can be compressed. Clients add the header "Accept-Encoding: gzip,deflate" by default. For this to work the Apache web server in front of the facade needs to handle this as the facade itself is not returning compressed response bodies.<br />
<br />
For Apache HTTPD <code>{mod_deflate}</code> needs to be enabled and the following line needs to be added to your virtual host:<br />
<br />
AddOutputFilterByType DEFLATE text/html text/plain text/javascript application/javascript text/css <br />
text/xml application/xml text/x-js application/x-javascript application/json<br />
<br />
== Starting/Stopping the Facade Service ==<br />
<br />
The facade runs as its own service independent of the normal OX App Suite middleware. For this on Debian-based system it can be started with<br />
<br />
service open-xchange-mobile-api-facade start<br />
<br />
It can be stopped with<br />
<br />
service open-xchange-mobile-api-facade stop<br />
<br />
== Ports ==<br />
<br />
As the Mobile API Facade is its own process it also has its own JMX port and its own RMI port. The default JMX port is 9995 and the default RMI port is 1100. These ports needs to be explicitly specified to the command line tools using either JMX or RMI.</div>Marenshttps://oxpedia.org/wiki/index.php?title=AppSuite:Mobile_API_Facade&diff=23535AppSuite:Mobile API Facade2017-07-10T08:21:41Z<p>Marens: /* Proxy configuration */</p>
<hr />
<div>= OX Facade for Mobile =<br />
<br />
== General Information ==<br />
<br />
The Mobile API Facade is a server component that brings the new native mobile mail apps together with the OX App Suite. We’ve built the façade based on the technology used and proven in the OX App Suite middleware. The facade is developed in Java, utilizing the OSGI Framework.<br />
<br />
The Facade provides offline friendly HTTP interface by doing the work, the connections through the HTTP API, to the OX App Suite and providing only the data the offline capable clients need. In some cases multiple requests to the OX App Suite are combined into one for its clients.The facade also offers a method to tell the clients that information on the server haven’t changed since the last time the client asked for it. This reduces the amount of data to transmit to the clients under certain circumstances, especially important in mobile networks were bandwidth (and overall traffic) is limited. Thanks to facade some functionality can be shared among all clients and need only get implemented once. One example for this is the teaser text extraction, and HTML mail handling in general.The facade also provides a pluggable authentication system. In the default case, login request are just forwarded to the middleware. In more advanced use-cases, the login request is forwarded to IDM of the customer and an OX session is created from the access token from the IDM. For clients this is pretty straightforward.<br />
<br />
== Requirements ==<br />
<br />
The OX Facade has to be installed alongside an OX App Suite installation. It requires at least OX App Suite v7.8.4.<br />
<br />
== OX Facade API ==<br />
<br />
Further information about the OX Facade API can be found at:<br />
https://documentation.open-xchange.com/components/facade/1.0.0/<br />
<br />
= OX Mail Server-side Installation and Configuration on OX App Suite v7.8.x =<br />
<br />
This chapter describes how the backend components of OX Mail are installed and configured on the server.<br />
<br />
== Available packages ==<br />
<br />
OX Facade is available with the following backend packages:<br />
<br />
* ''open-xchange-mobile-api-facade''<br />
<br />
Installation on the server varies depending on the underlying distribution, details are available in the following chapters.<br />
<br />
=== Redhat Enterprise Linux 6 or CentOS 6 ===<br />
<br />
Add the following repositories to your Open-Xchange yum configuration:<br />
<br />
{{for loop||call=YUMRepo|pv=reponame|pc1n=path|pc1v=products/mobile-api-facade/stable|pc2n=rhelname|pc2v=RHEL6|mobile-api-facade}}<br />
<br />
$ yum install open-xchange-mobile-api-facade<br />
<br />
=== Redhat Enterprise Linux 7 or CentOS 7 ===<br />
<br />
Add the following repositories to your Open-Xchange yum configuration:<br />
<br />
{{for loop||call=YUMRepo|pv=reponame|pc1n=path|pc1v=products/mobile-api-facade/stable|pc2n=rhelname|pc2v=RHEL7|mobile-api-facade}}<br />
<br />
$ yum install open-xchange-mobile-api-facade<br />
<br />
=== Debian GNU/Linux 8.0 ===<br />
<br />
Add the following repositories to your Open-Xchange apt configuration:<br />
<br />
{{for loop||call=APTRepo|pv=reponame|pc1n=path|pc1v=products/mobile-api-facade/stable|pc2n=debianname|pc2v=DebianJessie|mobile-api-facade}}<br />
<br />
$ apt-get update<br />
$ apt-get install open-xchange-mobile-api-facade<br />
<br />
=== SUSE Linux Enterprise Server 12 === <br />
<br />
{{for loop||call=SUSERepo|pv=reponame|pc1n=path|pc1v=products/mobile-api-facade/stable|pc2n=susename|pc2v=SLE_12|mobile-api-facade}}<br />
<br />
$ zypper ref<br />
$ zypper install open-xchange-mobile-api-facade<br />
<br />
= Configuration OX Facade for Mobile =<br />
<br />
== Introduction ==<br />
<br />
To be able to use the native mail apps the Mobile API Facade needs to be installed in front of the OX App Suite middleware. This document describes how to configure the Mobile API Facade.<br />
<br />
The Mobile API Facade stores its configuration in the files <code>/opt/open-xchange/mobile-api-facade/etc/facade.properties</code> (the global configuration) and in <code>/opt/open-xchange/mobile-api-facade/etc/mobile-api-facade-config.yml</code> (hostname specific configuration). Both files support the same configuration properties as can be seen on https://documentation.open-xchange.com/components/facade/config/1.0.0/.<br />
<br />
== Connection to the OX App Suite Middleware ==<br />
<br />
After installation of the facade package ("open-xchange-mobile-api-facade") the property <code>com.openexchange.mobile.api.facade.MiddlewareBaseUrl</code> needs to get set to the correct URL. This property needs to be explicitly configured by the administrator. It has no default value. Its possible to connect the Mobile API Facade directly to a middleware process, but this is highly discouraged. The Mobile API Facade should always connect to a middleware process through a load balancer.<br />
<br />
An example:<br />
<br />
com.openexchange.mobile.api.facade.MiddlewareBaseUrl=https://appsuite.example.com/appsuite/api<br />
<br />
After this configuration the open-xchange-mobile-api-facade needs to get restarted.<br />
<br />
== Proxy configuration ==<br />
<br />
<Proxy balancer://oxcluster_facade><br />
Order Allow,Deny<br />
Allow from all<br />
BalancerMember http://appsuite-middleware1.somedomain:8007 timeout=100 smax=0 ttl=60 retry=60 <br />
loadfactor=50 keepalive=On route=FOX1<br />
BalancerMember http://appsuite-middleware2.somedomain:8007 timeout=100 smax=0 ttl=60 retry=60 <br />
loadfactor=50 keepalive=On route=FOX2<br />
<br />
ProxySet stickysession=JSESSIONID|jsessionidscolonpathdelim=On<br />
SetEnv proxy-initial-not-pooled<br />
SetEnv proxy-sendchunked<br />
</Proxy><br />
<br />
ProxyPass /services/api-facade balancer://oxcluster_facade/services/api-facade<br />
<br />
== Traffic compression ==<br />
<br />
The clients need to exchange a lot of data with the facade to accomplish the task of a mail client. This traffic can be compressed. Clients add the header "Accept-Encoding: gzip,deflate" by default. For this to work the Apache web server in front of the facade needs to handle this as the facade itself is not returning compressed response bodies.<br />
<br />
For Apache HTTPD <code>{mod_deflate}</code> needs to be enabled and the following line needs to be added to your virtual host:<br />
<br />
AddOutputFilterByType DEFLATE text/html text/plain text/javascript application/javascript text/css <br />
text/xml application/xml text/x-js application/x-javascript application/json<br />
<br />
== Starting/Stopping the Facade Service ==<br />
<br />
The facade runs as its own service independent of the normal OX App Suite middleware. For this on Debian-based system it can be started with<br />
<br />
service open-xchange-mobile-api-facade start<br />
<br />
It can be stopped with<br />
<br />
service open-xchange-mobile-api-facade stop<br />
<br />
== Ports ==<br />
<br />
As the Mobile API Facade is its own process it also has its own JMX port and its own RMI port. The default JMX port is 9995 and the default RMI port is 1100. These ports needs to be explicitly specified to the command line tools using either JMX or RMI.</div>Marenshttps://oxpedia.org/wiki/index.php?title=AppSuite:UpdatingOXPackages&diff=23266AppSuite:UpdatingOXPackages2017-05-04T14:13:31Z<p>Marens: /* Warnings */</p>
<hr />
<div><div class="title">Updating OX App Suite packages</div><br />
<br />
''Synopsys'': This article describes how to update OX App Suite packages from one service pack to another.<br />
<br />
== Warnings ==<br />
<br />
Before we begin, here are several things that you should keep in mind:<br />
* If you are '''updating a cluster, please do it one node after the other'''. Turn the node off, update, turn it on again, make sure it works in the cluster. Have a look here, too: [[Running_a_cluster#Upgrading_a_single_Node|Upgrading a single node]]<br />
* If you don't have a cluster of OX nodes, then your complete installation needs to be taken down temporarily.<br />
* Please '''update the frontend before the backend'''. Otherwise the old frontend won't work with the new backend and it's manifests.<br />
* Please '''do not restart Open-Xchange during the database update'''. A database update can happen after installing minor or major updates. As soon as the first user tries to log in to the system or if any provisioning action is done, this update starts.<br />
<br />
=== Upgrading to 7.8.4 ===<br />
* During the upgrade to 7.8.4 the existing ''JAVA_XTRAOPTS'' configuration property of the middleware stack in the file ''ox-scriptconf.sh'' will automatically be split up and migrated to a new categories approach. Please have a look at [[AppSuite:MiddlewareStartup#Evaluate_JVM_commandline_options|JVM commandline options]] and review the automatically upgraded configuration file.<br />
<br />
== How to get updates? ==<br />
<br />
OX App Suite updates can be accessed by customers with a valid license.<br />
<br />
In addition, you need to configure the [[OXReportClient]].<br />
<br />
= Updating OX App Suite Packages =<br />
<br />
<br />
== Installing Updates ==<br />
<br />
A new service pack usually introduces new packages and requires configuration changes. To get all required new packages and configuration changes, the following '''must''' be done when installing updates.<br />
<br />
(Please, keep the [[#Warning|warning about frontend updates before backend updates]] in mind)<br />
<br />
=== On Debian based distributions ===<br />
<br />
Add the following entry to <tt>/etc/apt/sources.list.d/open-xchange.list</tt><br />
<br />
deb http://CUSTOMERID:PASSWORD@software.open-xchange.com/products/appsuite/stable/appsuiteui/updates/DebianJessie/ /<br />
deb http://CUSTOMERID:PASSWORD@software.open-xchange.com/products/appsuite/stable/backend/updates/DebianJessie /<br />
Then run<br />
<br />
$ apt-get update<br />
$ apt-get dist-upgrade<br />
<br />
If you want to see, what apt-get is going to do without actually doing it, you can run:<br />
<br />
$ apt-get dist-upgrade -s<br />
<br />
After the new packages are installed, the open-xchange process needs a restart:<br />
<br />
<pre>$ /etc/init.d/open-xchange restart</pre><br />
<br />
=== On RPM based distributions ===<br />
<br />
==== RHEL6/CentOS6 ====<br />
<br />
Add the following entries to <tt>/etc/yum.repos.d/ox.repo</tt>:<br />
<br />
[ox-updates-appsuiteui]<br />
name=Open-Xchange Updates<br />
baseurl=http://CUSTOMERID:PASSWORD@software.open-xchange.com/products/appsuite/stable/appsuiteui/updates/RHEL6/<br />
gpgkey=http://software.open-xchange.com/oxbuildkey.pub<br />
enabled=1<br />
gpgcheck=1<br />
metadata_expire=0m<br />
<br />
[ox-updates-backend]<br />
name=Open-Xchange Updates<br />
baseurl=http://CUSTOMERID:PASSWORD@software.open-xchange.com/products/appsuite/stable/backend/updates/RHEL6/<br />
gpgkey=http://software.open-xchange.com/oxbuildkey.pub<br />
enabled=1<br />
gpgcheck=1<br />
metadata_expire=0m<br />
<br />
and run<br />
<br />
$ yum update<br />
<br />
$ yum upgrade<br />
<br />
After the new packages are installed, the open-xchange process needs a restart:<br />
<br />
<pre>$ /etc/init.d/open-xchange restart</pre><br />
(Please, keep the [[#Warnings|warning about database updates]] in mind)<br />
<br />
==== RHEL7/CentOS7 ====<br />
<br />
Add the following entries to <tt>/etc/yum.repos.d/ox.repo</tt>:<br />
<br />
[ox-updates-appsuiteui]<br />
name=Open-Xchange Updates<br />
baseurl=http://CUSTOMERID:PASSWORD@software.open-xchange.com/products/appsuite/stable/appsuiteui/updates/RHEL7/<br />
gpgkey=http://software.open-xchange.com/oxbuildkey.pub<br />
enabled=1<br />
gpgcheck=1<br />
metadata_expire=0m<br />
<br />
[ox-updates-backend]<br />
name=Open-Xchange Updates<br />
baseurl=http://CUSTOMERID:PASSWORD@software.open-xchange.com/products/appsuite/stable/backend/updates/RHEL7/<br />
gpgkey=http://software.open-xchange.com/oxbuildkey.pub<br />
enabled=1<br />
gpgcheck=1<br />
metadata_expire=0m<br />
<br />
and run<br />
<br />
$ yum update<br />
<br />
$ yum upgrade<br />
<br />
After the new packages are installed, the open-xchange process needs a restart:<br />
<br />
<pre>$ /etc/init.d/open-xchange restart</pre><br />
(Please, keep the [[#Warnings|warning about database updates]] in mind)<br />
<br />
==== SLES12 ====<br />
<br />
Add the updates repository to the repository list:<br />
<br />
$ zypper ar http://CUSTOMERID:PASSWORD@software.open-xchange.com/products/appsuite/stable/appsuiteui/updates/SLE_12/ OXAPPSUITEUIUPDATES<br />
$ zypper ar http://CUSTOMERID:PASSWORD@software.open-xchange.com/products/appsuite/stable/backend/updates/SLE_12/ OXBACKENDUPDATES<br />
<br />
and run<br />
<br />
$ zypper dup -r OXAPPSUITEUIUPDATES<br />
$ zypper dup -r OXBACKENDUPDATES<br />
<br />
You might need to run<br />
<br />
$ zypper ref<br />
<br />
to update the repository metadata before running ''zypper up''.<br />
<br />
After the new packages are installed, the open-xchange process needs a restart:<br />
<br />
<pre>$ rcopen-xchange restart</pre><br />
(Please, keep the [[#Warnings|warning about database updates]] in mind.<br />
<br />
= Updating OX App Suite Packages for older versions =<br />
<br />
<br />
== Installing Updates ==<br />
<br />
A new service pack usually introduces new packages and requires configuration changes. To get all required new packages and configuration changes, the following '''must''' be done when installing updates.<br />
<br />
(Please, keep the [[#Warning|warning about frontend updates before backend updates]] in mind)<br />
<br />
=== On Debian based distributions ===<br />
<br />
If you want to update an older version of Open-Xchange App Suite to the latest maintenance release, add the following entry to <tt>/etc/apt/sources.list.d/open-xchange.list</tt>. Replace VERSION with the version you are using (e.g. 7.6.2). See [[AppSuite:Version_Support_Committment]] for the currently supported versions. Also, make sure that your repository configuration points at the versioned base installation repositories (instead of using the <tt>stable</tt> symlink.)<br />
<br />
deb http://CUSTOMERID:PASSWORD@software.open-xchange.com/products/appsuite/VERSION/appsuiteui/updates/DebianWheezy/ /<br />
deb http://CUSTOMERID:PASSWORD@software.open-xchange.com/products/appsuite/VERSION/backend/updates/DebianWheezy/ /<br />
<br />
deb http://CUSTOMERID:PASSWORD@software.open-xchange.com/products/appsuite/VERSION/appsuiteui/updates/DebianJessie/ /<br />
deb http://CUSTOMERID:PASSWORD@software.open-xchange.com/products/appsuite/VERSION/backend/updates/DebianJessie /<br />
<br />
Then run<br />
<br />
$ apt-get update<br />
$ apt-get dist-upgrade<br />
<br />
If you want to see, what apt-get is going to do without actually doing it, you can run:<br />
<br />
$ apt-get dist-upgrade -s<br />
<br />
After the new packages are installed, the open-xchange process needs a restart:<br />
<br />
<pre>$ /etc/init.d/open-xchange restart</pre><br />
<br />
=== On RPM based distributions ===<br />
<br />
==== RHEL6/CentOS6 ====<br />
<br />
If you want to update an older version of Open-Xchange App Suite to the latest maintenance release, add the following entry to <tt>/etc/yum.repos.d/ox.repo</tt>. Replace VERSION with the version you are using (e.g. 7.4.2, 7.6.0). See [[AppSuite:Version_Support_Committment]] for the currently supported versions. Also, make sure that your repository configuration points at the versioned base installation repositories (instead of using the <tt>stable</tt> symlink.)<br />
<br />
[ox-updates-appsuiteui]<br />
name=Open-Xchange Updates<br />
baseurl=http://CUSTOMERID:PASSWORD@software.open-xchange.com/products/appsuite/VERSION/appsuiteui/updates/RHEL6/<br />
gpgkey=http://software.open-xchange.com/oxbuildkey.pub<br />
enabled=1<br />
gpgcheck=1<br />
metadata_expire=0m<br />
<br />
[ox-updates-backend]<br />
name=Open-Xchange Updates<br />
baseurl=http://CUSTOMERID:PASSWORD@software.open-xchange.com/products/appsuite/VERSION/backend/updates/RHEL6/<br />
gpgkey=http://software.open-xchange.com/oxbuildkey.pub<br />
enabled=1<br />
gpgcheck=1<br />
metadata_expire=0m<br />
<br />
and run<br />
<br />
$ yum update<br />
<br />
$ yum upgrade<br />
<br />
After the new packages are installed, the open-xchange process needs a restart:<br />
<br />
<pre>$ /etc/init.d/open-xchange restart</pre><br />
(Please, keep the [[#Warnings|warning about database updates]] in mind)<br />
<br />
==== RHEL7/CentOS7 ====<br />
<br />
If you want to update an older version of Open-Xchange App Suite to the latest maintenance release, add the following entry to <tt>/etc/yum.repos.d/ox.repo</tt>. Replace VERSION with the version you are using (e.g. 7.4.2, 7.6.0). See [[AppSuite:Version_Support_Committment]] for the currently supported versions. Also, make sure that your repository configuration points at the versioned base installation repositories (instead of using the <tt>stable</tt> symlink.)<br />
<br />
[ox-updates-appsuiteui]<br />
name=Open-Xchange Updates<br />
baseurl=http://CUSTOMERID:PASSWORD@software.open-xchange.com/products/appsuite/VERSION/appsuiteui/updates/RHEL7/<br />
gpgkey=http://software.open-xchange.com/oxbuildkey.pub<br />
enabled=1<br />
gpgcheck=1<br />
metadata_expire=0m<br />
<br />
[ox-updates-backend]<br />
name=Open-Xchange Updates<br />
baseurl=http://CUSTOMERID:PASSWORD@software.open-xchange.com/products/appsuite/VERSION/backend/updates/RHEL7/<br />
gpgkey=http://software.open-xchange.com/oxbuildkey.pub<br />
enabled=1<br />
gpgcheck=1<br />
metadata_expire=0m<br />
<br />
and run<br />
<br />
$ yum update<br />
<br />
$ yum upgrade<br />
<br />
After the new packages are installed, the open-xchange process needs a restart:<br />
<br />
<pre>$ /etc/init.d/open-xchange restart</pre><br />
(Please, keep the [[#Warnings|warning about database updates]] in mind)<br />
<br />
==== SLES 11 ====<br />
<br />
If you want to update an older version of Open-Xchange App Suite to the latest maintenance release, add the following entry to the repository list. Replace VERSION with the version you are using (e.g. 7.4.2, 7.6.0). See [[AppSuite:Version_Support_Committment]] for the currently supported versions. Also, make sure that your repository configuration points at the versioned base installation repositories (instead of using the <tt>stable</tt> symlink.)<br />
<br />
$ zypper ar http://CUSTOMERID:PASSWORD@software.open-xchange.com/products/appsuite/VERSION/appsuiteui/updates/SLES11/ OXAPPSUITEUIUPDATES<br />
$ zypper ar http://CUSTOMERID:PASSWORD@software.open-xchange.com/products/appsuite/VERSION/backend/updates/SLES11/ OXBACKENDUPDATES<br />
<br />
and run<br />
<br />
$ zypper dup -r OXAPPSUITEUIUPDATES<br />
$ zypper dup -r OXBACKENDUPDATES<br />
<br />
You might need to run<br />
<br />
$ zypper ref<br />
<br />
to update the repository metadata before running ''zypper up''.<br />
<br />
After the new packages are installed, the open-xchange process needs a restart:<br />
<br />
<pre>$ rcopen-xchange restart</pre><br />
(Please, keep the [[#Warnings|warning about database updates]] in mind.<br />
<br />
==== SLES 12 ====<br />
<br />
If you want to update an older version of Open-Xchange App Suite to the latest maintenance release, add the following entry to the repository list. Replace VERSION with the version you are using (e.g. 7.4.2, 7.6.0). See [[AppSuite:Version_Support_Committment]] for the currently supported versions. Also, make sure that your repository configuration points at the versioned base installation repositories (instead of using the <tt>stable</tt> symlink.)<br />
<br />
$ zypper ar http://CUSTOMERID:PASSWORD@software.open-xchange.com/products/appsuite/VERSION/appsuiteui/updates/SLE_12/ OXAPPSUITEUIUPDATES<br />
$ zypper ar http://CUSTOMERID:PASSWORD@software.open-xchange.com/products/appsuite/VERSION/backend/updates/SLE_12/ OXBACKENDUPDATES<br />
<br />
and run<br />
<br />
$ zypper dup -r OXAPPSUITEUIUPDATES<br />
$ zypper dup -r OXBACKENDUPDATES<br />
<br />
You might need to run<br />
<br />
$ zypper ref<br />
<br />
to update the repository metadata before running ''zypper up''.<br />
<br />
After the new packages are installed, the open-xchange process needs a restart:<br />
<br />
<pre>$ rcopen-xchange restart</pre><br />
(Please, keep the [[#Warnings|warning about database updates]] in mind.<br />
<br />
= Updating UI plugins =<br />
<br />
If you update only UI plugins without simultaneously upgrading the core UI packages to a new version, you need to perform an additional step after the update to make the changes visible to users.<br />
<br />
The package <tt>open-xchange-appsuite</tt> contains a timestamp which is different in each version. The JavaScript UI in every user's browser will reload all cached code and data whenever this value changes. Since this value does not change when updating only plugin packages, the value must be changed manually with the following command:<br />
<br />
$ /opt/open-xchange/sbin/touch-appsuite --timestamp=20170101.123400<br />
<br />
If you only have one OX node, the parameter <code>--timestamp</code> can be omitted. It defaults to the current UTC time anyway.<br />
<br />
When updating a cluster, the command must use the same timestamp value on every node. Otherwise, browsers will clear their cache and re-download the entire UI code every time load balancer sends them to a different node (i.e. on almost every login). This is also why the timestamps can't be updated automatically: a node doesn't know which value other nodes will pick, which of them are part of the same update, etc.<br />
<br />
The quickest way to obtain a value for the timestamp parameter is to run the command with the parameter <code>-h</code>. It will then print a help message with the current time as an example. For automation, you can use the output of<br />
<br />
date -u +%Y%m%d.%H%M%S<br />
<br />
Again: this value must be computed once, and then the same value must be used on every node in the cluster.<br />
<br />
This step can be performed on each node separately after each node has been updated, or on all nodes at once after all nodes have been updated. If Apache has its own set of dedicated nodes, the <tt>touch-appsuite</tt> call must be performed on the web server nodes, after all corresponding <tt><plugin>-static</tt> packages have been updated.<br />
<br />
[[Category: OX7]]<br />
[[Category: AppSuite]]<br />
<br />
[[Category: Administrator]]</div>Marenshttps://oxpedia.org/wiki/index.php?title=AppSuite:MiddlewareStartup&diff=23247AppSuite:MiddlewareStartup2017-04-26T16:52:24Z<p>Marens: /* Format >= 7.8.4 */</p>
<hr />
<div><br />
== Systemd ==<br />
<br />
==== Service definition ====<br />
The default service definition as shipped by Open-Xchange is rather concise and easy to read. You can check by looking at the service file that is installed to either /usr/lib/systemd or /lib/systemd depending on your distribution of choice.<br />
<br />
singlenode:~ # cat /usr/lib/systemd/system/open-xchange.service <br />
[Unit]<br />
After=remote-fs.target<br />
After=time-sync.target ypbind.service sendmail.service cyrus.service <br />
<br />
[Service]<br />
User=open-xchange<br />
PermissionsStartOnly=true<br />
TimeoutStartSec=0<br />
ExecStartPre=/opt/open-xchange/sbin/triggerupdatethemes -u<br />
ExecStart=/opt/open-xchange/sbin/open-xchange<br />
ExecStop=/opt/open-xchange/sbin/shutdown -w<br />
ExecReload=/opt/open-xchange/sbin/triggerreloadconfiguration -d<br />
KillMode=process<br />
LimitNOFILE=65536<br />
LimitNPROC=65536<br />
<br />
[Install]<br />
WantedBy=multi-user.target<br />
<br />
==== Drop-in configs ====<br />
Drop in configs allow administrators to easily adapt and override default service unit files. So if you want to change the default limits or add additional limits have a look at our default example drop-in config which is located at /etc/systemd/system/open-xchange.service.d/limits.conf<br />
<br />
singlenode:~ # cat /etc/systemd/system/open-xchange.service.d/limits.conf <br />
# Override and add options in this file<br />
# See systemd.exec(5) for other limits<br />
<br />
[Service]<br />
#LimitNPROC=65536<br />
#LimitNOFILE=65536<br />
<br />
The [https://www.freedesktop.org/software/systemd/man/systemd.exec.html systemd.exec] documentation shows a whole lot of options that can be used by admins to adapt the default service to their specific needs.<br />
<br />
<br clear=all><br />
[[File:mw_start_sysd.png|thumb|Service startup using systemd|left|1000px]]<br />
<br clear=all><br />
<br />
''Note:'' The technical implementation of systemd differs as service definition and config aren't read every time during service start but the changes need to be reread with ''systemctl daemon-reload''. The above is just a simplyfied version to explain the general concept.<br />
<br />
== System V / Upstart ==<br />
<br />
==== Service definition ====<br />
<br />
The default service definition as shipped by Open-Xchange can be found at /etc/init.d/open-xchange and will differ slightly depending on the distro in use.<br />
<br />
<br clear=all><br />
[[File:mw_start_sysv.png|thumb|Service startup using upstart|left|1000px]]<br />
<br clear=all><br />
== Apply Resource limits ==<br />
<br />
As you can see in the diagrams above one of the differences is that:<br />
<br />
* systemd reads all the infos wrt. resource limitations from the original service file and then applies possible further restrictions from the drop-in configs (steps 2 and 8 in the diagram ''Service startup using systemd'') and limits configured in /opt/open-xchange/etc/ox-scriptconf.sh aren't considered anylonger<br />
* the older System V style init has to source /opt/open-xchange/etc/ox-scriptconf.sh to be able to read the configs for the maximum number of processes (NPROC) and files(NRFILES) that the middleware shall be allowed to create/open (steps 3,4 and 15 in the diagram ''Service startup using upstart'').<br />
<br />
=== Background Infos ===<br />
<br />
Several ways exist to restrict resources on a linux system from a global level down to user/groups or even shells and the processes started by them.<br />
<br />
==== Sysctl ====<br />
Sysctl is used to modify kernel parameters at runtime. E.g. to set the maximum number of files<br />
<br />
$ sysctl -w fs.file-max=100000<br />
<br />
To permanently set them append to the main configuration file and reload the settings<br />
<br />
$ echo fs.file-max=100000 >> /etc/sysctl.conf<br />
$ sysctl -p<br />
<br />
More infos can be found via '''man sysctl'''<br />
<br />
==== Limits.conf ====<br />
<br />
Allows to restrict resources an a global, group or user level. E.g:<br />
<br />
$ cat /etc/security/limits.d/90-nproc.conf <br />
# Default limit for number of user's processes to prevent<br />
# accidental fork bombs.<br />
# See rhbz #432903 for reasoning.<br />
<br />
* soft nproc 1024<br />
<br />
From '''man limits.conf''':<br />
<blockquote><br />
''Also, please note that all limit settings are set per login. They are not global, nor are they permanent; existing only for the duration of the session.''<br />
</blockquote><br />
<br />
The limits per login are applied via the ''pam'' stack. See '''man pam''' and '''man pam_limits''' for more details. As those limits are bound to sessions they don't affect most daemons started by our supported init systems or init utils. Most state that they are ignored by design, see [https://bugs.launchpad.net/ubuntu/+source/upstart/+bug/938669/comments/4 upstart], [https://bugzilla.redhat.com/show_bug.cgi?id=754285#c1 systemd] and [https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=302079 start-stop-daemon] <br />
<br />
===== Ulimit =====<br />
From '''man bash'''<br />
<blockquote><br />
ulimit [-HSTabcdefilmnpqrstuvx [limit]]<br />
Provides control over the resources available to the shell and to processes started by it, on systems that allow such control.<br />
</blockquote><br />
This is what we use in our System V compatible init scripts to increase resources for the open-xchange process across multiple distros. Currently only ''the maximum number of processes'' and ''the maximum number of open file descriptors available to a single user'' are increased via ulimit. The values are specified in '''/opt/open-xchange/ox-scriptconf.sh'''<br />
<br />
==== Systemd ====<br />
<br />
===== Control Groups=====<br />
<br />
Control groups should only affect the OX middleware if you create/manage them yourself of if you are using a ''modern'' distribution that already uses systemd as init.<br />
<br />
Citing from the kernel cgroup [https://www.kernel.org/doc/Documentation/cgroup-v2.txt documentation]:<br />
<br />
<blockquote><br />
1-2. What is cgroup?<br />
<br />
cgroup is a mechanism to organize processes hierarchically and<br />
distribute system resources along the hierarchy in a controlled and<br />
configurable manner.<br />
<br />
cgroup is largely composed of two parts - the core and controllers.<br />
cgroup core is primarily responsible for hierarchically organizing<br />
processes. A cgroup controller is usually responsible for<br />
distributing a specific type of system resource along the hierarchy<br />
although there are utility controllers which serve purposes other than<br />
resource distribution.<br />
<br />
cgroups form a tree structure and every process in the system belongs<br />
to one and only one cgroup. All threads of a process belong to the<br />
same cgroup. On creation, all processes are put in the cgroup that<br />
the parent process belongs to at the time. A process can be migrated<br />
to another cgroup. Migration of a process doesn't affect already<br />
existing descendant processes.<br />
<br />
Following certain structural constraints, controllers may be enabled or<br />
disabled selectively on a cgroup. All controller behaviors are<br />
hierarchical - if a controller is enabled on a cgroup, it affects all<br />
processes which belong to the cgroups consisting the inclusive<br />
sub-hierarchy of the cgroup. When a controller is enabled on a nested<br />
cgroup, it always restricts the resource distribution further. The<br />
restrictions set closer to the root in the hierarchy can not be<br />
overridden from further away.<br />
</blockquote><br />
<br />
So processes are organized into a tree structure of control groups and controllers are responsible for the distribution of resources. So what kind of controllers exist?<br />
<br />
<blockquote><br />
5. Controllers<br />
<br />
5-1. CPU<br />
<br />
The "cpu" controllers regulates distribution of CPU cycles. This<br />
controller implements weight and absolute bandwidth limit models for<br />
normal scheduling policy and absolute bandwidth allocation model for<br />
realtime scheduling policy.<br />
<br />
5-2. Memory<br />
<br />
The "memory" controller regulates distribution of memory.<br />
...<br />
While not completely water-tight, all major memory usages by a given<br />
cgroup are tracked so that the total memory consumption can be<br />
accounted and controlled to a reasonable extent.<br />
<br />
5-3. IO<br />
<br />
The "io" controller regulates the distribution of IO resources. This<br />
controller implements both weight based and absolute bandwidth or IOPS<br />
limit distribution; however, weight based distribution is available<br />
only if cfq-iosched is in use and neither scheme is available for<br />
blk-mq devices.<br />
</blockquote><br />
<br />
The open-xchange service is simply put into the default ''system.slice'' without applying further limits.<br />
<br />
singlenode$ systemd-cgls<br />
├─1 /sbin/init<br />
├─system.slice<br />
│ ├─avahi-daemon.service<br />
│ │ ├─501 avahi-daemon: running [singlenode]<br />
│ │ └─514 avahi-daemon: chroot helper<br />
│ ├─console-kit-daemon.service<br />
│ │ └─16164 /usr/sbin/console-kit-daemon --no-daemon<br />
│ ├─dbus.service<br />
│ │ └─508 /usr/bin/dbus-daemon --system --address=systemd: --nofork --nopidfile --systemd-activation<br />
│ ├─munin-node.service<br />
│ │ └─4290 /usr/bin/perl -wT /usr/sbin/munin-node<br />
│ ├─open-xchange.service<br />
│ │ └─6037 /usr/bin/java -Dsun.net.inetaddr.ttl=3600 -Dnetworkaddress.cache.ttl=3600 -Dnetworkaddress.cache.negative.ttl=10 ...<br />
<br />
To check all the details use<br />
<br />
singlenode:~ # systemctl show system.slice<br />
<br />
==== Limits besides control groups ====<br />
Besides control groups systemd allows you to apply other limits to the execution environment of your service. Here we can apply the limits that would normally be applied via ''limits.conf'' or ''ulimit''. Systemd uses [http://man7.org/linux/man-pages/man2/setrlimit.2.html setrlimit] for this. The options that we set by default are:<br />
<br />
* LimitNOFILE<br />
* LimitNPROC<br />
<br />
which you have already seen when we took a look at the [[#Service_definiton|service definition]] and [[#Drop-in_config|drop-in configs]]. ''Again'': The complete documentation wrt. resource limit configurations in drop-in configs can be found at [https://www.freedesktop.org/software/systemd/man/systemd.exec.html systemd.exec]<br />
<br />
== Verify limits ==<br />
<br />
=== System V ===<br />
<br />
Get the proper pid of the java process started for the middleware by either checking '''ps faux'''<br />
root 4103 0.0 0.0 66528 1868 pts/0 S 22:05 0:00 su -s /bin/bash open-xchange -c /opt/open-xchange/sbin/open-xchange<br />
494 4109 9.5 19.8 2224644 381312 ? Ssl 22:05 1:12 \_ /usr/bin/java -Dsun.net.inetaddr.ttl=3600 -Dnetworkaddress.cache.ttl=3600<br />
<br />
or programmatically<br />
singlenode:~ # pid=$(pgrep -P $(</var/run/open-xchange.pid) java)<br />
<br />
and check the limits applied for that process<br />
singlenode:~ # cat /proc/$pid/limits<br />
Limit Soft Limit Hard Limit Units<br />
Max cpu time unlimited unlimited seconds<br />
Max file size unlimited unlimited bytes<br />
Max data size unlimited unlimited bytes<br />
Max stack size 8388608 unlimited bytes<br />
Max core file size 0 unlimited bytes<br />
Max resident set unlimited unlimited bytes<br />
Max processes 65536 65536 processes<br />
Max open files 65536 65536 files<br />
Max locked memory 65536 65536 bytes<br />
Max address space unlimited unlimited bytes<br />
Max file locks unlimited unlimited locks<br />
Max pending signals 24254 24254 signals<br />
Max msgqueue size 819200 819200 bytes<br />
Max nice priority 0 0<br />
Max realtime priority 0 0<br />
Max realtime timeout unlimited unlimited us<br />
<br />
=== Systemd ===<br />
singlenode:~ # systemctl show open-xchange | grep Limit<br />
StartLimitInterval=10000000<br />
StartLimitBurst=5<br />
StartLimitAction=none<br />
MemoryLimit=18446744073709551615<br />
LimitCPU=18446744073709551615<br />
LimitFSIZE=18446744073709551615<br />
LimitDATA=18446744073709551615<br />
LimitSTACK=18446744073709551615<br />
LimitCORE=18446744073709551615<br />
LimitRSS=18446744073709551615<br />
LimitNOFILE=65536<br />
LimitAS=18446744073709551615<br />
LimitNPROC=65536<br />
LimitMEMLOCK=65536<br />
LimitLOCKS=18446744073709551615<br />
LimitSIGPENDING=19827<br />
LimitMSGQUEUE=819200<br />
LimitNICE=0<br />
LimitRTPRIO=0<br />
LimitRTTIME=18446744073709551615<br />
<br />
== Evaluate JVM commandline options ==<br />
<br />
Before starting Java and the OSGi framework with all the Open-Xchange middleware bundles we have to evaluate several important JVM commandline options that influence the performance and stability of the Open-Xchange middleware stack. These commandline options are specified in '''/opt/open-xchange/etc/ox-scriptconf.sh'''<br />
<br />
=== Format < 7.8.4 ===<br />
For version up to 7.8.3 the JVM configuration options were all specified via a single property '''JAVA_XTRAOPTS''' which resulted in a rather long configuration string that was hard to read and maintain.<br />
<br />
JAVA_XTRAOPTS="-Dsun.net.inetaddr.ttl=3600 -Dnetworkaddress.cache.ttl=3600 -Dnetworkaddress.cache.negative.ttl=10 \<br />
-Dlogback.threadlocal.put.duplicate=false -server -Djava.awt.headless=true -Xmx512M -XX:MaxPermSize=256m -XX:+UseConcMarkSweepGC \<br />
-XX:+UseParNewGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:NewRatio=3 -XX:+UseTLAB \<br />
-Dosgi.compatibility.bootdelegation=false -XX:-OmitStackTraceInFastThrow"<br />
<br />
During the startup of the open-xchange service (step 11 of diagram Service startup using systemd and step 10 of diagram Service startup using upstart) this JAVA_XTRAOPTS string goes through a basic validity check and is then passed to the JVM.<br />
<br />
=== Format >= 7.8.4 ===<br />
Beginning from version 7.8.4 the options are split up and sorted into different categories like:<br />
<br />
JAVA_OPTS_GC="-XX:+UseConcMarkSweepGC -XX:+UseParNewGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:NewRatio=3 -XX:+DisableExplicitGC"<br />
JAVA_OPTS_LOG="-Dlogback.threadlocal.put.duplicate=false -XX:-OmitStackTraceInFastThrow"<br />
JAVA_OPTS_MEM="-XX:MaxHeapSize=512M -XX:MaxPermSize=256M -XX:+UseTLAB"<br />
JAVA_OPTS_NET="-Dsun.net.inetaddr.ttl=3600 -Dnetworkaddress.cache.ttl=3600 -Dnetworkaddress.cache.negative.ttl=10"<br />
JAVA_OPTS_OSGI="-Dosgi.compatibility.bootdelegation=false"<br />
JAVA_OPTS_SERVER="-server -Djava.awt.headless=true"<br />
<br />
JAVA_OPTS_OTHER=""<br />
<br />
#JAVA_OPTS_DEBUG="-XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/mnt/heapdump -Xloggc:/var/log/open-xchange/gc.log -verbose:gc -XX:+PrintGCDateStamps \<br />
-XX:+PrintHeapAtGC -XX:+PrintGCApplicationStoppedTime -XX:+PrintTenuringDistribution"<br />
<br />
which is much easier to read and maintain. Furthermore we added a useful '''JAVA_OPTS_DEBUG''' configuration parameter that comes preconfigured but unused/commented so that customers can quickly enable the most commonly used debugging options in case they have to provide detailed system information to Open-Xchange.<br />
<br />
During the startup of the open-xchange service (step 11 of diagram ''Service startup using systemd'' and step 10 of diagram ''Service startup using upstart'') all uncommented JAVA_OPTS_* categories will then be evaluated and combined again into a single string of commandline options that can then be passed to the JVM.<br />
<br />
=== Upgrade procedure ===<br />
During the upgrade '''7.8.3-revX -> 7.8.4-revY''' existing JAVA_XTRAOPTS will automatically be split up and migrated to the new categories approach. A backup of the file (ox-scriptconf.sh.timestamp) will be created and admins are urged to check for proper migration before deleting the backup file like it's already done with *.dpkg-dist or *.rpmnew files on Debian or RHEL/SLES distributions.<br />
<br />
== Encoding detection during JVM startup ==<br />
<br />
=== Background Infos ===<br />
<br />
From ''man 7 locale''<br />
<br />
DESCRIPTION<br />
A locale is a set of language and cultural rules. These cover aspects such as lan‐<br />
guage for messages, different character sets, lexicographic conventions, and so on.<br />
A program needs to be able to determine its locale and act accordingly to be portable<br />
to different cultures.<br />
<br />
The format to specify a system locale is: '''''language[_territory][.codeset][@modifier]'''''.<br />
<br />
Where all of<br />
<br />
* en_US.UTF-8<br />
* en_US.utf8<br />
* en_US.utf-8<br />
* en_US.UTF_8 <br />
<br />
will result in a valid UTF-8 charmap for the english language with locale categories configured as<br />
<br />
LANG=en_US.UTF-8<br />
LC_CTYPE="en_US.UTF-8"<br />
LC_NUMERIC="en_US.UTF-8"<br />
LC_TIME="en_US.UTF-8"<br />
LC_COLLATE="en_US.UTF-8"<br />
LC_MONETARY="en_US.UTF-8"<br />
LC_MESSAGES="en_US.UTF-8"<br />
LC_PAPER="en_US.UTF-8"<br />
LC_NAME="en_US.UTF-8"<br />
LC_ADDRESS="en_US.UTF-8"<br />
LC_TELEPHONE="en_US.UTF-8"<br />
LC_MEASUREMENT="en_US.UTF-8"<br />
LC_IDENTIFICATION="en_US.UTF-8"<br />
LC_ALL=<br />
<br />
See for more infos: locale(1), locale(7), [https://en.wikipedia.org/wiki/List_of_ISO_639-1_codes language codes], [https://en.wikipedia.org/wiki/ISO_3166-1#Current_codes country codes] and [https://en.wikipedia.org/wiki/Character_encoding character encodings]<br />
<br />
==== JVM startup ====<br />
<br />
The default file.encoding detection is done at JVM startup based upon LC_CTYPE which makes sense as it's documented as:<br />
<br />
CODESET (LC_CTYPE)<br />
Return a string with the name of the character encoding used in the selected locale, such as "UTF-8", "ISO-8859-1",<br />
or "ANSI_X3.4-1968" (better known as US-ASCII). This is the same string that you get with "locale charmap".<br />
For a list of character encoding names, try "locale -m", cf. locale(1).<br />
<br />
Furthermore this influences the defaultCharset selection of the JVM.<br />
<br />
/**<br />
* Returns the default charset of this Java virtual machine.<br />
*<br />
* The default charset is determined during virtual-machine startup and<br />
* typically depends upon the locale and charset of the underlying<br />
* operating system.<br />
*<br />
* @return A charset object for the default charset<br />
*<br />
* @since 1.5<br />
*/<br />
public static Charset defaultCharset() {<br />
if (defaultCharset == null) {<br />
synchronized (Charset.class) {<br />
String csn = AccessController.doPrivileged(<br />
new GetPropertyAction("file.encoding"));<br />
Charset cs = lookup(csn);<br />
if (cs != null)<br />
defaultCharset = cs;<br />
else<br />
defaultCharset = forName("UTF-8");<br />
}<br />
}<br />
return defaultCharset;<br />
}<br />
<br />
Although we try to specify valid unicode encodings for every IO operation the Open-Xchange middleware does, some parts of the JVM don't allow to specify the encoding to use and simply falls back to the detected default encoding. That's why '''admins have to make sure to provide a proper unicode environment for the middleware service during every service start''' or use documented workarounds provided by Open-Xchange (e.g. [http://documentation.open-xchange.com/components/middleware/config/develop/index.html#mode=features&feature=Password%20Change%20Script com.openexchange.passwordchange.script.base64]: Encoded strings as Base64 to circumvent character encoding issues on improperly configured distributions).<br />
<br />
See the documentation of your linux distribution on how to configure a proper unicode locale.<br />
<br />
=== Query which file encoding was detected by the JVM during startup ===<br />
[[File:visualvm-file.encoding.png|thumb|Querying the JVM via visualvm|right|300px]]<br />
<br />
The file encoding that was detected by the JVM during startup can be queried via JMX. You can use graphical management tools like [https://visualvm.github.io/ visualvm], query our [[Jolokia]] JMX bridge via http or just use the commandline like<br />
<br />
/opt/open-xchange/sbin/showruntimestats -r | sed -ne 's/.*\({key=file.encoding, value=[^}]*}\).*/\1/p'<br />
{key=file.encoding, value='''UTF-8'''}<br />
<br />
<br clear=all><br />
<br />
=== Locale passing from client to server via SSH ===<br />
<br />
The following setting of sshd allows clients to specify which locale to use on the server and might lead to some confusion or unexpected behaviour when checking the system locale or even to the usage of broken locales during middleware startup if your client sends misconfigured or missing locale configuration to the server.<br />
<br />
grep AcceptEnv /etc/ssh/sshd_config <br />
AcceptEnv LANG LC_*<br />
<br />
For example when i connect to the machine my local german locale is sent to the server <br />
<br />
[vagrant@showcase-community-centos-7 ~]$ locale -v <br />
LANG=de_DE.UTF-8<br />
LC_CTYPE="de_DE.UTF-8"<br />
LC_NUMERIC=de_DE.UTF-8<br />
LC_TIME=de_DE.UTF-8<br />
LC_COLLATE="de_DE.UTF-8"<br />
LC_MONETARY=de_DE.UTF-8<br />
LC_MESSAGES="de_DE.UTF-8"<br />
LC_PAPER=de_DE.UTF-8<br />
LC_NAME=de_DE.UTF-8<br />
LC_ADDRESS=de_DE.UTF-8<br />
LC_TELEPHONE=de_DE.UTF-8<br />
LC_MEASUREMENT=de_DE.UTF-8<br />
LC_IDENTIFICATION=de_DE.UTF-8<br />
LC_ALL=<br />
<br />
But after specifying that i want a new login shell the default locale configured for that user/system will be used instead!<br />
<br />
[root@showcase-community-centos-7 vagrant]# su -s /bin/bash open-xchange -l<br />
-bash-4.2$ locale -v <br />
LANG=en_US.UTF-8<br />
LC_CTYPE="en_US.UTF-8"<br />
LC_NUMERIC="en_US.UTF-8"<br />
LC_TIME="en_US.UTF-8"<br />
LC_COLLATE="en_US.UTF-8"<br />
LC_MONETARY="en_US.UTF-8"<br />
LC_MESSAGES="en_US.UTF-8"<br />
LC_PAPER="en_US.UTF-8"<br />
LC_NAME="en_US.UTF-8"<br />
LC_ADDRESS="en_US.UTF-8"<br />
LC_TELEPHONE="en_US.UTF-8"<br />
LC_MEASUREMENT="en_US.UTF-8"<br />
LC_IDENTIFICATION="en_US.UTF-8"<br />
LC_ALL=<br />
<br />
=== Verify that your system is configured with a proper unicode locale for OX ===<br />
<br />
Assuming you haven't reconfigured the locale of your root user<br />
<br />
[root@showcase-community-centos-7 vagrant]# locale charmap<br />
UTF-8<br />
[root@showcase-community-centos-7 vagrant]#<br />
<br />
== Warnings during early start ==<br />
<br />
Early warnings before the start of the JVM and related logging libraries like logback and logstash forwarding are redirected to '''/var/log/open-xchange/open-xchange-console.log'''.<br />
<br />
=== Limits ===<br />
If the open-xchange startup script isn't able to set the proper limits on CentOS 6 due to e.g. hard limits being enforced via pam_limits you'll see the following warning in the console log during startup<br />
<br />
singlenode open-xchange # cat open-xchange-console.log<br />
/opt/open-xchange/sbin/open-xchange: line 115: ulimit: max user processes: cannot modify limit: Operation not permitted<br />
<br />
=== Locale ===<br />
If the open-xchange startup script isn't able to detect a unicode locale you'll see a warning like the following (the current charmap might differ)<br />
<br />
WARNING: Couldn't detect proper unicode charmap in locale category LC_CTYPE.<br />
WARNING: Current charmap: "ANSI_X3.4-1968" might cause encoding problems.<br />
<br />
== Summary: Open-Xchange middleware on specific distros ==<br />
The support for the mentioned mechanism of resource control differ depending on the distribution and the init system in use.<br />
<br />
=== Debian 7 ===<br />
;Init<br />
: System V style<br />
;OX Configurable Limits/Defaults<br />
: nofile, nproc<br />
<br />
The mentioned limits can be configured via '''/opt/open-xchange/etc/ox-scriptconf.sh'''. The limits are applied via ulimit in the service's init script. The open-xchange service is finally started via start-stop-daemon which doesn't doesn't consider '''/etc/security/limits.*'''<br />
<br />
=== RHEL 6 / CentOS 6 ===<br />
;Init<br />
: Upstart, System V compatible<br />
;OX Configurable Limits/Defaults<br />
: nofile, nproc<br />
<br />
The mentioned limits can be configured via '''/opt/open-xchange/etc/ox-scriptconf.sh'''. The limits are applied via ulimit in the service's init script. Furthermore as the open-xchange service is finally started via '''su ... open-xchange''' on this distro a user session is opened via su/pam and the default CentOS pam config reads the '''/etc/security/limits.*''' configuration by loading the pam stack like:<br />
<br />
: '''/etc/pam.d/su'''<br />
:: '''-> /etc/pam.d/system-auth'''<br />
::: '''-> pam_limits.so'''<br />
<br />
If NPROC isn't configured for the open-xchange-server it's restricted to '''1024''' globally by default to prevent accidental fork bombs, see '''/etc/security/limits.d/90-nproc.conf''' which can result in severe problems modern multithreaded applications.<br />
<br />
=== RHEL 7 / CentOS 7 / Debian 8 / SLE 12 ===<br />
;Init<br />
: Systemd<br />
;OX Configurable Limits/Defaults<br />
: nofile, nproc<br />
<br />
For systemd the default limits are configured directly in the service's unit file that is shipped by OX and located at '''/usr/lib/systemd/system/open-xchange.service'''. The drop-in config to override or extend the default unit file is located at '''/etc/systemd/system/open-xchange.service.d/limits.conf'''. [https://www.freedesktop.org/software/systemd/man/systemd.exec.html Systemd.exec] shows a whole lot of options that can be used by admins to adapt the default service to their specific needs.<br />
<br />
----<br />
<br />
[[File:diagram_sources.zip]]</div>Marenshttps://oxpedia.org/wiki/index.php?title=AppSuite:MiddlewareStartup&diff=23246AppSuite:MiddlewareStartup2017-04-26T08:20:05Z<p>Marens: /* Verify that your system is configured with a proper unicode locale for OX */</p>
<hr />
<div><br />
== Systemd ==<br />
<br />
==== Service definition ====<br />
The default service definition as shipped by Open-Xchange is rather concise and easy to read. You can check by looking at the service file that is installed to either /usr/lib/systemd or /lib/systemd depending on your distribution of choice.<br />
<br />
singlenode:~ # cat /usr/lib/systemd/system/open-xchange.service <br />
[Unit]<br />
After=remote-fs.target<br />
After=time-sync.target ypbind.service sendmail.service cyrus.service <br />
<br />
[Service]<br />
User=open-xchange<br />
PermissionsStartOnly=true<br />
TimeoutStartSec=0<br />
ExecStartPre=/opt/open-xchange/sbin/triggerupdatethemes -u<br />
ExecStart=/opt/open-xchange/sbin/open-xchange<br />
ExecStop=/opt/open-xchange/sbin/shutdown -w<br />
ExecReload=/opt/open-xchange/sbin/triggerreloadconfiguration -d<br />
KillMode=process<br />
LimitNOFILE=65536<br />
LimitNPROC=65536<br />
<br />
[Install]<br />
WantedBy=multi-user.target<br />
<br />
==== Drop-in configs ====<br />
Drop in configs allow administrators to easily adapt and override default service unit files. So if you want to change the default limits or add additional limits have a look at our default example drop-in config which is located at /etc/systemd/system/open-xchange.service.d/limits.conf<br />
<br />
singlenode:~ # cat /etc/systemd/system/open-xchange.service.d/limits.conf <br />
# Override and add options in this file<br />
# See systemd.exec(5) for other limits<br />
<br />
[Service]<br />
#LimitNPROC=65536<br />
#LimitNOFILE=65536<br />
<br />
The [https://www.freedesktop.org/software/systemd/man/systemd.exec.html systemd.exec] documentation shows a whole lot of options that can be used by admins to adapt the default service to their specific needs.<br />
<br />
<br clear=all><br />
[[File:mw_start_sysd.png|thumb|Service startup using systemd|left|1000px]]<br />
<br clear=all><br />
<br />
''Note:'' The technical implementation of systemd differs as service definition and config aren't read every time during service start but the changes need to be reread with ''systemctl daemon-reload''. The above is just a simplyfied version to explain the general concept.<br />
<br />
== System V / Upstart ==<br />
<br />
==== Service definition ====<br />
<br />
The default service definition as shipped by Open-Xchange can be found at /etc/init.d/open-xchange and will differ slightly depending on the distro in use.<br />
<br />
<br clear=all><br />
[[File:mw_start_sysv.png|thumb|Service startup using upstart|left|1000px]]<br />
<br clear=all><br />
== Apply Resource limits ==<br />
<br />
As you can see in the diagrams above one of the differences is that:<br />
<br />
* systemd reads all the infos wrt. resource limitations from the original service file and then applies possible further restrictions from the drop-in configs (steps 2 and 8 in the diagram ''Service startup using systemd'') and limits configured in /opt/open-xchange/etc/ox-scriptconf.sh aren't considered anylonger<br />
* the older System V style init has to source /opt/open-xchange/etc/ox-scriptconf.sh to be able to read the configs for the maximum number of processes (NPROC) and files(NRFILES) that the middleware shall be allowed to create/open (steps 3,4 and 15 in the diagram ''Service startup using upstart'').<br />
<br />
=== Background Infos ===<br />
<br />
Several ways exist to restrict resources on a linux system from a global level down to user/groups or even shells and the processes started by them.<br />
<br />
==== Sysctl ====<br />
Sysctl is used to modify kernel parameters at runtime. E.g. to set the maximum number of files<br />
<br />
$ sysctl -w fs.file-max=100000<br />
<br />
To permanently set them append to the main configuration file and reload the settings<br />
<br />
$ echo fs.file-max=100000 >> /etc/sysctl.conf<br />
$ sysctl -p<br />
<br />
More infos can be found via '''man sysctl'''<br />
<br />
==== Limits.conf ====<br />
<br />
Allows to restrict resources an a global, group or user level. E.g:<br />
<br />
$ cat /etc/security/limits.d/90-nproc.conf <br />
# Default limit for number of user's processes to prevent<br />
# accidental fork bombs.<br />
# See rhbz #432903 for reasoning.<br />
<br />
* soft nproc 1024<br />
<br />
From '''man limits.conf''':<br />
<blockquote><br />
''Also, please note that all limit settings are set per login. They are not global, nor are they permanent; existing only for the duration of the session.''<br />
</blockquote><br />
<br />
The limits per login are applied via the ''pam'' stack. See '''man pam''' and '''man pam_limits''' for more details. As those limits are bound to sessions they don't affect most daemons started by our supported init systems or init utils. Most state that they are ignored by design, see [https://bugs.launchpad.net/ubuntu/+source/upstart/+bug/938669/comments/4 upstart], [https://bugzilla.redhat.com/show_bug.cgi?id=754285#c1 systemd] and [https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=302079 start-stop-daemon] <br />
<br />
===== Ulimit =====<br />
From '''man bash'''<br />
<blockquote><br />
ulimit [-HSTabcdefilmnpqrstuvx [limit]]<br />
Provides control over the resources available to the shell and to processes started by it, on systems that allow such control.<br />
</blockquote><br />
This is what we use in our System V compatible init scripts to increase resources for the open-xchange process across multiple distros. Currently only ''the maximum number of processes'' and ''the maximum number of open file descriptors available to a single user'' are increased via ulimit. The values are specified in '''/opt/open-xchange/ox-scriptconf.sh'''<br />
<br />
==== Systemd ====<br />
<br />
===== Control Groups=====<br />
<br />
Control groups should only affect the OX middleware if you create/manage them yourself of if you are using a ''modern'' distribution that already uses systemd as init.<br />
<br />
Citing from the kernel cgroup [https://www.kernel.org/doc/Documentation/cgroup-v2.txt documentation]:<br />
<br />
<blockquote><br />
1-2. What is cgroup?<br />
<br />
cgroup is a mechanism to organize processes hierarchically and<br />
distribute system resources along the hierarchy in a controlled and<br />
configurable manner.<br />
<br />
cgroup is largely composed of two parts - the core and controllers.<br />
cgroup core is primarily responsible for hierarchically organizing<br />
processes. A cgroup controller is usually responsible for<br />
distributing a specific type of system resource along the hierarchy<br />
although there are utility controllers which serve purposes other than<br />
resource distribution.<br />
<br />
cgroups form a tree structure and every process in the system belongs<br />
to one and only one cgroup. All threads of a process belong to the<br />
same cgroup. On creation, all processes are put in the cgroup that<br />
the parent process belongs to at the time. A process can be migrated<br />
to another cgroup. Migration of a process doesn't affect already<br />
existing descendant processes.<br />
<br />
Following certain structural constraints, controllers may be enabled or<br />
disabled selectively on a cgroup. All controller behaviors are<br />
hierarchical - if a controller is enabled on a cgroup, it affects all<br />
processes which belong to the cgroups consisting the inclusive<br />
sub-hierarchy of the cgroup. When a controller is enabled on a nested<br />
cgroup, it always restricts the resource distribution further. The<br />
restrictions set closer to the root in the hierarchy can not be<br />
overridden from further away.<br />
</blockquote><br />
<br />
So processes are organized into a tree structure of control groups and controllers are responsible for the distribution of resources. So what kind of controllers exist?<br />
<br />
<blockquote><br />
5. Controllers<br />
<br />
5-1. CPU<br />
<br />
The "cpu" controllers regulates distribution of CPU cycles. This<br />
controller implements weight and absolute bandwidth limit models for<br />
normal scheduling policy and absolute bandwidth allocation model for<br />
realtime scheduling policy.<br />
<br />
5-2. Memory<br />
<br />
The "memory" controller regulates distribution of memory.<br />
...<br />
While not completely water-tight, all major memory usages by a given<br />
cgroup are tracked so that the total memory consumption can be<br />
accounted and controlled to a reasonable extent.<br />
<br />
5-3. IO<br />
<br />
The "io" controller regulates the distribution of IO resources. This<br />
controller implements both weight based and absolute bandwidth or IOPS<br />
limit distribution; however, weight based distribution is available<br />
only if cfq-iosched is in use and neither scheme is available for<br />
blk-mq devices.<br />
</blockquote><br />
<br />
The open-xchange service is simply put into the default ''system.slice'' without applying further limits.<br />
<br />
singlenode$ systemd-cgls<br />
├─1 /sbin/init<br />
├─system.slice<br />
│ ├─avahi-daemon.service<br />
│ │ ├─501 avahi-daemon: running [singlenode]<br />
│ │ └─514 avahi-daemon: chroot helper<br />
│ ├─console-kit-daemon.service<br />
│ │ └─16164 /usr/sbin/console-kit-daemon --no-daemon<br />
│ ├─dbus.service<br />
│ │ └─508 /usr/bin/dbus-daemon --system --address=systemd: --nofork --nopidfile --systemd-activation<br />
│ ├─munin-node.service<br />
│ │ └─4290 /usr/bin/perl -wT /usr/sbin/munin-node<br />
│ ├─open-xchange.service<br />
│ │ └─6037 /usr/bin/java -Dsun.net.inetaddr.ttl=3600 -Dnetworkaddress.cache.ttl=3600 -Dnetworkaddress.cache.negative.ttl=10 ...<br />
<br />
To check all the details use<br />
<br />
singlenode:~ # systemctl show system.slice<br />
<br />
==== Limits besides control groups ====<br />
Besides control groups systemd allows you to apply other limits to the execution environment of your service. Here we can apply the limits that would normally be applied via ''limits.conf'' or ''ulimit''. Systemd uses [http://man7.org/linux/man-pages/man2/setrlimit.2.html setrlimit] for this. The options that we set by default are:<br />
<br />
* LimitNOFILE<br />
* LimitNPROC<br />
<br />
which you have already seen when we took a look at the [[#Service_definiton|service definition]] and [[#Drop-in_config|drop-in configs]]. ''Again'': The complete documentation wrt. resource limit configurations in drop-in configs can be found at [https://www.freedesktop.org/software/systemd/man/systemd.exec.html systemd.exec]<br />
<br />
== Verify limits ==<br />
<br />
=== System V ===<br />
<br />
Get the proper pid of the java process started for the middleware by either checking '''ps faux'''<br />
root 4103 0.0 0.0 66528 1868 pts/0 S 22:05 0:00 su -s /bin/bash open-xchange -c /opt/open-xchange/sbin/open-xchange<br />
494 4109 9.5 19.8 2224644 381312 ? Ssl 22:05 1:12 \_ /usr/bin/java -Dsun.net.inetaddr.ttl=3600 -Dnetworkaddress.cache.ttl=3600<br />
<br />
or programmatically<br />
singlenode:~ # pid=$(pgrep -P $(</var/run/open-xchange.pid) java)<br />
<br />
and check the limits applied for that process<br />
singlenode:~ # cat /proc/$pid/limits<br />
Limit Soft Limit Hard Limit Units<br />
Max cpu time unlimited unlimited seconds<br />
Max file size unlimited unlimited bytes<br />
Max data size unlimited unlimited bytes<br />
Max stack size 8388608 unlimited bytes<br />
Max core file size 0 unlimited bytes<br />
Max resident set unlimited unlimited bytes<br />
Max processes 65536 65536 processes<br />
Max open files 65536 65536 files<br />
Max locked memory 65536 65536 bytes<br />
Max address space unlimited unlimited bytes<br />
Max file locks unlimited unlimited locks<br />
Max pending signals 24254 24254 signals<br />
Max msgqueue size 819200 819200 bytes<br />
Max nice priority 0 0<br />
Max realtime priority 0 0<br />
Max realtime timeout unlimited unlimited us<br />
<br />
=== Systemd ===<br />
singlenode:~ # systemctl show open-xchange | grep Limit<br />
StartLimitInterval=10000000<br />
StartLimitBurst=5<br />
StartLimitAction=none<br />
MemoryLimit=18446744073709551615<br />
LimitCPU=18446744073709551615<br />
LimitFSIZE=18446744073709551615<br />
LimitDATA=18446744073709551615<br />
LimitSTACK=18446744073709551615<br />
LimitCORE=18446744073709551615<br />
LimitRSS=18446744073709551615<br />
LimitNOFILE=65536<br />
LimitAS=18446744073709551615<br />
LimitNPROC=65536<br />
LimitMEMLOCK=65536<br />
LimitLOCKS=18446744073709551615<br />
LimitSIGPENDING=19827<br />
LimitMSGQUEUE=819200<br />
LimitNICE=0<br />
LimitRTPRIO=0<br />
LimitRTTIME=18446744073709551615<br />
<br />
== Evaluate JVM commandline options ==<br />
<br />
Before starting Java and the OSGi framework with all the Open-Xchange middleware bundles we have to evaluate several important JVM commandline options that influence the performance and stability of the Open-Xchange middleware stack. These commandline options are specified in '''/opt/open-xchange/etc/ox-scriptconf.sh'''<br />
<br />
=== Format < 7.8.4 ===<br />
For version up to 7.8.3 the JVM configuration options were all specified via a single property '''JAVA_XTRAOPTS''' which resulted in a rather long configuration string that was hard to read and maintain.<br />
<br />
JAVA_XTRAOPTS="-Dsun.net.inetaddr.ttl=3600 -Dnetworkaddress.cache.ttl=3600 -Dnetworkaddress.cache.negative.ttl=10 \<br />
-Dlogback.threadlocal.put.duplicate=false -server -Djava.awt.headless=true -Xmx512M -XX:MaxPermSize=256m -XX:+UseConcMarkSweepGC \<br />
-XX:+UseParNewGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:NewRatio=3 -XX:+UseTLAB \<br />
-Dosgi.compatibility.bootdelegation=false -XX:-OmitStackTraceInFastThrow"<br />
<br />
During the startup of the open-xchange service (step 11 of diagram Service startup using systemd and step 10 of diagram Service startup using upstart) this JAVA_XTRAOPTS string goes through a basic validity check and is then passed to the JVM.<br />
<br />
=== Format >= 7.8.4 ===<br />
Beginning from version 7.8.4 the options are split up and sorted into different categories like:<br />
<br />
JAVA_OPTS_GC="-XX:+UseConcMarkSweepGC -XX:+UseParNewGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:NewRatio=3 -XX:+DisableExplicitGC"<br />
JAVA_OPTS_LOG="-Dlogback.threadlocal.put.duplicate=false -XX:-OmitStackTraceInFastThrow"<br />
JAVA_OPTS_MEM="-XX:MaxHeapSize=512M -XX:MaxPermSize=256M -XX:+UseTLAB"<br />
JAVA_OPTS_NET="-Dsun.net.inetaddr.ttl=3600 -Dnetworkaddress.cache.ttl=3600 -Dnetworkaddress.cache.negative.ttl=10"<br />
JAVA_OPTS_OSGI="-Dosgi.compatibility.bootdelegation=false"<br />
JAVA_OPTS_SERVER="-server -Djava.awt.headless=true"<br />
<br />
JAVA_OPTS_OTHER=""<br />
<br />
#JAVA_OPTS_DEBUG="-XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/mnt/heapdump -Xloggc:/var/log/open-xchange/gc.log -verbose:gc -XX:+PrintGCDateStamps \<br />
-XX:+PrintHeapAtGC -XX:+PrintGCApplicationStoppedTime -XX:+PrintTenuringDistribution"<br />
<br />
which is much easier to read an maintain. Furthermore we added a useful '''JAVA_OPTS_DEBUG''' configuration parameter that comes preconfigured but unused/commented so that customers can quickly enable the most commonly used debugging options in case they have to provide detailed system information to Open-Xchange.<br />
<br />
During the startup of the open-xchange service (step 11 of diagram ''Service startup using systemd'' and step 10 of diagram ''Service startup using upstart'') all uncommented JAVA_OPTS_* categories will then be evaluated and combined again into a single string of commandline options that can then be passed to the JVM.<br />
<br />
=== Upgrade procedure ===<br />
During the upgrade '''7.8.3-revX -> 7.8.4-revY''' existing JAVA_XTRAOPTS will automatically be split up and migrated to the new categories approach. A backup of the file (ox-scriptconf.sh.timestamp) will be created and admins are urged to check for proper migration before deleting the backup file like it's already done with *.dpkg-dist or *.rpmnew files on Debian or RHEL/SLES distributions.<br />
<br />
== Encoding detection during JVM startup ==<br />
<br />
=== Background Infos ===<br />
<br />
From ''man 7 locale''<br />
<br />
DESCRIPTION<br />
A locale is a set of language and cultural rules. These cover aspects such as lan‐<br />
guage for messages, different character sets, lexicographic conventions, and so on.<br />
A program needs to be able to determine its locale and act accordingly to be portable<br />
to different cultures.<br />
<br />
The format to specify a system locale is: '''''language[_territory][.codeset][@modifier]'''''.<br />
<br />
Where all of<br />
<br />
* en_US.UTF-8<br />
* en_US.utf8<br />
* en_US.utf-8<br />
* en_US.UTF_8 <br />
<br />
will result in a valid UTF-8 charmap for the english language with locale categories configured as<br />
<br />
LANG=en_US.UTF-8<br />
LC_CTYPE="en_US.UTF-8"<br />
LC_NUMERIC="en_US.UTF-8"<br />
LC_TIME="en_US.UTF-8"<br />
LC_COLLATE="en_US.UTF-8"<br />
LC_MONETARY="en_US.UTF-8"<br />
LC_MESSAGES="en_US.UTF-8"<br />
LC_PAPER="en_US.UTF-8"<br />
LC_NAME="en_US.UTF-8"<br />
LC_ADDRESS="en_US.UTF-8"<br />
LC_TELEPHONE="en_US.UTF-8"<br />
LC_MEASUREMENT="en_US.UTF-8"<br />
LC_IDENTIFICATION="en_US.UTF-8"<br />
LC_ALL=<br />
<br />
See for more infos: locale(1), locale(7), [https://en.wikipedia.org/wiki/List_of_ISO_639-1_codes language codes], [https://en.wikipedia.org/wiki/ISO_3166-1#Current_codes country codes] and [https://en.wikipedia.org/wiki/Character_encoding character encodings]<br />
<br />
==== JVM startup ====<br />
<br />
The default file.encoding detection is done at JVM startup based upon LC_CTYPE which makes sense as it's documented as:<br />
<br />
CODESET (LC_CTYPE)<br />
Return a string with the name of the character encoding used in the selected locale, such as "UTF-8", "ISO-8859-1",<br />
or "ANSI_X3.4-1968" (better known as US-ASCII). This is the same string that you get with "locale charmap".<br />
For a list of character encoding names, try "locale -m", cf. locale(1).<br />
<br />
Furthermore this influences the defaultCharset selection of the JVM.<br />
<br />
/**<br />
* Returns the default charset of this Java virtual machine.<br />
*<br />
* The default charset is determined during virtual-machine startup and<br />
* typically depends upon the locale and charset of the underlying<br />
* operating system.<br />
*<br />
* @return A charset object for the default charset<br />
*<br />
* @since 1.5<br />
*/<br />
public static Charset defaultCharset() {<br />
if (defaultCharset == null) {<br />
synchronized (Charset.class) {<br />
String csn = AccessController.doPrivileged(<br />
new GetPropertyAction("file.encoding"));<br />
Charset cs = lookup(csn);<br />
if (cs != null)<br />
defaultCharset = cs;<br />
else<br />
defaultCharset = forName("UTF-8");<br />
}<br />
}<br />
return defaultCharset;<br />
}<br />
<br />
Although we try to specify valid unicode encodings for every IO operation the Open-Xchange middleware does, some parts of the JVM don't allow to specify the encoding to use and simply falls back to the detected default encoding. That's why '''admins have to make sure to provide a proper unicode environment for the middleware service during every service start''' or use documented workarounds provided by Open-Xchange (e.g. [http://documentation.open-xchange.com/components/middleware/config/develop/index.html#mode=features&feature=Password%20Change%20Script com.openexchange.passwordchange.script.base64]: Encoded strings as Base64 to circumvent character encoding issues on improperly configured distributions).<br />
<br />
See the documentation of your linux distribution on how to configure a proper unicode locale.<br />
<br />
=== Query which file encoding was detected by the JVM during startup ===<br />
[[File:visualvm-file.encoding.png|thumb|Querying the JVM via visualvm|right|300px]]<br />
<br />
The file encoding that was detected by the JVM during startup can be queried via JMX. You can use graphical management tools like [https://visualvm.github.io/ visualvm], query our [[Jolokia]] JMX bridge via http or just use the commandline like<br />
<br />
/opt/open-xchange/sbin/showruntimestats -r | sed -ne 's/.*\({key=file.encoding, value=[^}]*}\).*/\1/p'<br />
{key=file.encoding, value='''UTF-8'''}<br />
<br />
<br clear=all><br />
<br />
=== Locale passing from client to server via SSH ===<br />
<br />
The following setting of sshd allows clients to specify which locale to use on the server and might lead to some confusion or unexpected behaviour when checking the system locale or even to the usage of broken locales during middleware startup if your client sends misconfigured or missing locale configuration to the server.<br />
<br />
grep AcceptEnv /etc/ssh/sshd_config <br />
AcceptEnv LANG LC_*<br />
<br />
For example when i connect to the machine my local german locale is sent to the server <br />
<br />
[vagrant@showcase-community-centos-7 ~]$ locale -v <br />
LANG=de_DE.UTF-8<br />
LC_CTYPE="de_DE.UTF-8"<br />
LC_NUMERIC=de_DE.UTF-8<br />
LC_TIME=de_DE.UTF-8<br />
LC_COLLATE="de_DE.UTF-8"<br />
LC_MONETARY=de_DE.UTF-8<br />
LC_MESSAGES="de_DE.UTF-8"<br />
LC_PAPER=de_DE.UTF-8<br />
LC_NAME=de_DE.UTF-8<br />
LC_ADDRESS=de_DE.UTF-8<br />
LC_TELEPHONE=de_DE.UTF-8<br />
LC_MEASUREMENT=de_DE.UTF-8<br />
LC_IDENTIFICATION=de_DE.UTF-8<br />
LC_ALL=<br />
<br />
But after specifying that i want a new login shell the default locale configured for that user/system will be used instead!<br />
<br />
[root@showcase-community-centos-7 vagrant]# su -s /bin/bash open-xchange -l<br />
-bash-4.2$ locale -v <br />
LANG=en_US.UTF-8<br />
LC_CTYPE="en_US.UTF-8"<br />
LC_NUMERIC="en_US.UTF-8"<br />
LC_TIME="en_US.UTF-8"<br />
LC_COLLATE="en_US.UTF-8"<br />
LC_MONETARY="en_US.UTF-8"<br />
LC_MESSAGES="en_US.UTF-8"<br />
LC_PAPER="en_US.UTF-8"<br />
LC_NAME="en_US.UTF-8"<br />
LC_ADDRESS="en_US.UTF-8"<br />
LC_TELEPHONE="en_US.UTF-8"<br />
LC_MEASUREMENT="en_US.UTF-8"<br />
LC_IDENTIFICATION="en_US.UTF-8"<br />
LC_ALL=<br />
<br />
=== Verify that your system is configured with a proper unicode locale for OX ===<br />
<br />
Assuming you haven't reconfigured the locale of your root user<br />
<br />
[root@showcase-community-centos-7 vagrant]# locale charmap<br />
UTF-8<br />
[root@showcase-community-centos-7 vagrant]#<br />
<br />
== Warnings during early start ==<br />
<br />
Early warnings before the start of the JVM and related logging libraries like logback and logstash forwarding are redirected to '''/var/log/open-xchange/open-xchange-console.log'''.<br />
<br />
=== Limits ===<br />
If the open-xchange startup script isn't able to set the proper limits on CentOS 6 due to e.g. hard limits being enforced via pam_limits you'll see the following warning in the console log during startup<br />
<br />
singlenode open-xchange # cat open-xchange-console.log<br />
/opt/open-xchange/sbin/open-xchange: line 115: ulimit: max user processes: cannot modify limit: Operation not permitted<br />
<br />
=== Locale ===<br />
If the open-xchange startup script isn't able to detect a unicode locale you'll see a warning like the following (the current charmap might differ)<br />
<br />
WARNING: Couldn't detect proper unicode charmap in locale category LC_CTYPE.<br />
WARNING: Current charmap: "ANSI_X3.4-1968" might cause encoding problems.<br />
<br />
== Summary: Open-Xchange middleware on specific distros ==<br />
The support for the mentioned mechanism of resource control differ depending on the distribution and the init system in use.<br />
<br />
=== Debian 7 ===<br />
;Init<br />
: System V style<br />
;OX Configurable Limits/Defaults<br />
: nofile, nproc<br />
<br />
The mentioned limits can be configured via '''/opt/open-xchange/etc/ox-scriptconf.sh'''. The limits are applied via ulimit in the service's init script. The open-xchange service is finally started via start-stop-daemon which doesn't doesn't consider '''/etc/security/limits.*'''<br />
<br />
=== RHEL 6 / CentOS 6 ===<br />
;Init<br />
: Upstart, System V compatible<br />
;OX Configurable Limits/Defaults<br />
: nofile, nproc<br />
<br />
The mentioned limits can be configured via '''/opt/open-xchange/etc/ox-scriptconf.sh'''. The limits are applied via ulimit in the service's init script. Furthermore as the open-xchange service is finally started via '''su ... open-xchange''' on this distro a user session is opened via su/pam and the default CentOS pam config reads the '''/etc/security/limits.*''' configuration by loading the pam stack like:<br />
<br />
: '''/etc/pam.d/su'''<br />
:: '''-> /etc/pam.d/system-auth'''<br />
::: '''-> pam_limits.so'''<br />
<br />
If NPROC isn't configured for the open-xchange-server it's restricted to '''1024''' globally by default to prevent accidental fork bombs, see '''/etc/security/limits.d/90-nproc.conf''' which can result in severe problems modern multithreaded applications.<br />
<br />
=== RHEL 7 / CentOS 7 / Debian 8 / SLE 12 ===<br />
;Init<br />
: Systemd<br />
;OX Configurable Limits/Defaults<br />
: nofile, nproc<br />
<br />
For systemd the default limits are configured directly in the service's unit file that is shipped by OX and located at '''/usr/lib/systemd/system/open-xchange.service'''. The drop-in config to override or extend the default unit file is located at '''/etc/systemd/system/open-xchange.service.d/limits.conf'''. [https://www.freedesktop.org/software/systemd/man/systemd.exec.html Systemd.exec] shows a whole lot of options that can be used by admins to adapt the default service to their specific needs.<br />
<br />
----<br />
<br />
[[File:diagram_sources.zip]]</div>Marenshttps://oxpedia.org/wiki/index.php?title=AppSuite:MiddlewareStartup&diff=23245AppSuite:MiddlewareStartup2017-04-26T08:19:41Z<p>Marens: /* Verify that your system is configured with a proper unicode locale for OX */</p>
<hr />
<div><br />
== Systemd ==<br />
<br />
==== Service definition ====<br />
The default service definition as shipped by Open-Xchange is rather concise and easy to read. You can check by looking at the service file that is installed to either /usr/lib/systemd or /lib/systemd depending on your distribution of choice.<br />
<br />
singlenode:~ # cat /usr/lib/systemd/system/open-xchange.service <br />
[Unit]<br />
After=remote-fs.target<br />
After=time-sync.target ypbind.service sendmail.service cyrus.service <br />
<br />
[Service]<br />
User=open-xchange<br />
PermissionsStartOnly=true<br />
TimeoutStartSec=0<br />
ExecStartPre=/opt/open-xchange/sbin/triggerupdatethemes -u<br />
ExecStart=/opt/open-xchange/sbin/open-xchange<br />
ExecStop=/opt/open-xchange/sbin/shutdown -w<br />
ExecReload=/opt/open-xchange/sbin/triggerreloadconfiguration -d<br />
KillMode=process<br />
LimitNOFILE=65536<br />
LimitNPROC=65536<br />
<br />
[Install]<br />
WantedBy=multi-user.target<br />
<br />
==== Drop-in configs ====<br />
Drop in configs allow administrators to easily adapt and override default service unit files. So if you want to change the default limits or add additional limits have a look at our default example drop-in config which is located at /etc/systemd/system/open-xchange.service.d/limits.conf<br />
<br />
singlenode:~ # cat /etc/systemd/system/open-xchange.service.d/limits.conf <br />
# Override and add options in this file<br />
# See systemd.exec(5) for other limits<br />
<br />
[Service]<br />
#LimitNPROC=65536<br />
#LimitNOFILE=65536<br />
<br />
The [https://www.freedesktop.org/software/systemd/man/systemd.exec.html systemd.exec] documentation shows a whole lot of options that can be used by admins to adapt the default service to their specific needs.<br />
<br />
<br clear=all><br />
[[File:mw_start_sysd.png|thumb|Service startup using systemd|left|1000px]]<br />
<br clear=all><br />
<br />
''Note:'' The technical implementation of systemd differs as service definition and config aren't read every time during service start but the changes need to be reread with ''systemctl daemon-reload''. The above is just a simplyfied version to explain the general concept.<br />
<br />
== System V / Upstart ==<br />
<br />
==== Service definition ====<br />
<br />
The default service definition as shipped by Open-Xchange can be found at /etc/init.d/open-xchange and will differ slightly depending on the distro in use.<br />
<br />
<br clear=all><br />
[[File:mw_start_sysv.png|thumb|Service startup using upstart|left|1000px]]<br />
<br clear=all><br />
== Apply Resource limits ==<br />
<br />
As you can see in the diagrams above one of the differences is that:<br />
<br />
* systemd reads all the infos wrt. resource limitations from the original service file and then applies possible further restrictions from the drop-in configs (steps 2 and 8 in the diagram ''Service startup using systemd'') and limits configured in /opt/open-xchange/etc/ox-scriptconf.sh aren't considered anylonger<br />
* the older System V style init has to source /opt/open-xchange/etc/ox-scriptconf.sh to be able to read the configs for the maximum number of processes (NPROC) and files(NRFILES) that the middleware shall be allowed to create/open (steps 3,4 and 15 in the diagram ''Service startup using upstart'').<br />
<br />
=== Background Infos ===<br />
<br />
Several ways exist to restrict resources on a linux system from a global level down to user/groups or even shells and the processes started by them.<br />
<br />
==== Sysctl ====<br />
Sysctl is used to modify kernel parameters at runtime. E.g. to set the maximum number of files<br />
<br />
$ sysctl -w fs.file-max=100000<br />
<br />
To permanently set them append to the main configuration file and reload the settings<br />
<br />
$ echo fs.file-max=100000 >> /etc/sysctl.conf<br />
$ sysctl -p<br />
<br />
More infos can be found via '''man sysctl'''<br />
<br />
==== Limits.conf ====<br />
<br />
Allows to restrict resources an a global, group or user level. E.g:<br />
<br />
$ cat /etc/security/limits.d/90-nproc.conf <br />
# Default limit for number of user's processes to prevent<br />
# accidental fork bombs.<br />
# See rhbz #432903 for reasoning.<br />
<br />
* soft nproc 1024<br />
<br />
From '''man limits.conf''':<br />
<blockquote><br />
''Also, please note that all limit settings are set per login. They are not global, nor are they permanent; existing only for the duration of the session.''<br />
</blockquote><br />
<br />
The limits per login are applied via the ''pam'' stack. See '''man pam''' and '''man pam_limits''' for more details. As those limits are bound to sessions they don't affect most daemons started by our supported init systems or init utils. Most state that they are ignored by design, see [https://bugs.launchpad.net/ubuntu/+source/upstart/+bug/938669/comments/4 upstart], [https://bugzilla.redhat.com/show_bug.cgi?id=754285#c1 systemd] and [https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=302079 start-stop-daemon] <br />
<br />
===== Ulimit =====<br />
From '''man bash'''<br />
<blockquote><br />
ulimit [-HSTabcdefilmnpqrstuvx [limit]]<br />
Provides control over the resources available to the shell and to processes started by it, on systems that allow such control.<br />
</blockquote><br />
This is what we use in our System V compatible init scripts to increase resources for the open-xchange process across multiple distros. Currently only ''the maximum number of processes'' and ''the maximum number of open file descriptors available to a single user'' are increased via ulimit. The values are specified in '''/opt/open-xchange/ox-scriptconf.sh'''<br />
<br />
==== Systemd ====<br />
<br />
===== Control Groups=====<br />
<br />
Control groups should only affect the OX middleware if you create/manage them yourself of if you are using a ''modern'' distribution that already uses systemd as init.<br />
<br />
Citing from the kernel cgroup [https://www.kernel.org/doc/Documentation/cgroup-v2.txt documentation]:<br />
<br />
<blockquote><br />
1-2. What is cgroup?<br />
<br />
cgroup is a mechanism to organize processes hierarchically and<br />
distribute system resources along the hierarchy in a controlled and<br />
configurable manner.<br />
<br />
cgroup is largely composed of two parts - the core and controllers.<br />
cgroup core is primarily responsible for hierarchically organizing<br />
processes. A cgroup controller is usually responsible for<br />
distributing a specific type of system resource along the hierarchy<br />
although there are utility controllers which serve purposes other than<br />
resource distribution.<br />
<br />
cgroups form a tree structure and every process in the system belongs<br />
to one and only one cgroup. All threads of a process belong to the<br />
same cgroup. On creation, all processes are put in the cgroup that<br />
the parent process belongs to at the time. A process can be migrated<br />
to another cgroup. Migration of a process doesn't affect already<br />
existing descendant processes.<br />
<br />
Following certain structural constraints, controllers may be enabled or<br />
disabled selectively on a cgroup. All controller behaviors are<br />
hierarchical - if a controller is enabled on a cgroup, it affects all<br />
processes which belong to the cgroups consisting the inclusive<br />
sub-hierarchy of the cgroup. When a controller is enabled on a nested<br />
cgroup, it always restricts the resource distribution further. The<br />
restrictions set closer to the root in the hierarchy can not be<br />
overridden from further away.<br />
</blockquote><br />
<br />
So processes are organized into a tree structure of control groups and controllers are responsible for the distribution of resources. So what kind of controllers exist?<br />
<br />
<blockquote><br />
5. Controllers<br />
<br />
5-1. CPU<br />
<br />
The "cpu" controllers regulates distribution of CPU cycles. This<br />
controller implements weight and absolute bandwidth limit models for<br />
normal scheduling policy and absolute bandwidth allocation model for<br />
realtime scheduling policy.<br />
<br />
5-2. Memory<br />
<br />
The "memory" controller regulates distribution of memory.<br />
...<br />
While not completely water-tight, all major memory usages by a given<br />
cgroup are tracked so that the total memory consumption can be<br />
accounted and controlled to a reasonable extent.<br />
<br />
5-3. IO<br />
<br />
The "io" controller regulates the distribution of IO resources. This<br />
controller implements both weight based and absolute bandwidth or IOPS<br />
limit distribution; however, weight based distribution is available<br />
only if cfq-iosched is in use and neither scheme is available for<br />
blk-mq devices.<br />
</blockquote><br />
<br />
The open-xchange service is simply put into the default ''system.slice'' without applying further limits.<br />
<br />
singlenode$ systemd-cgls<br />
├─1 /sbin/init<br />
├─system.slice<br />
│ ├─avahi-daemon.service<br />
│ │ ├─501 avahi-daemon: running [singlenode]<br />
│ │ └─514 avahi-daemon: chroot helper<br />
│ ├─console-kit-daemon.service<br />
│ │ └─16164 /usr/sbin/console-kit-daemon --no-daemon<br />
│ ├─dbus.service<br />
│ │ └─508 /usr/bin/dbus-daemon --system --address=systemd: --nofork --nopidfile --systemd-activation<br />
│ ├─munin-node.service<br />
│ │ └─4290 /usr/bin/perl -wT /usr/sbin/munin-node<br />
│ ├─open-xchange.service<br />
│ │ └─6037 /usr/bin/java -Dsun.net.inetaddr.ttl=3600 -Dnetworkaddress.cache.ttl=3600 -Dnetworkaddress.cache.negative.ttl=10 ...<br />
<br />
To check all the details use<br />
<br />
singlenode:~ # systemctl show system.slice<br />
<br />
==== Limits besides control groups ====<br />
Besides control groups systemd allows you to apply other limits to the execution environment of your service. Here we can apply the limits that would normally be applied via ''limits.conf'' or ''ulimit''. Systemd uses [http://man7.org/linux/man-pages/man2/setrlimit.2.html setrlimit] for this. The options that we set by default are:<br />
<br />
* LimitNOFILE<br />
* LimitNPROC<br />
<br />
which you have already seen when we took a look at the [[#Service_definiton|service definition]] and [[#Drop-in_config|drop-in configs]]. ''Again'': The complete documentation wrt. resource limit configurations in drop-in configs can be found at [https://www.freedesktop.org/software/systemd/man/systemd.exec.html systemd.exec]<br />
<br />
== Verify limits ==<br />
<br />
=== System V ===<br />
<br />
Get the proper pid of the java process started for the middleware by either checking '''ps faux'''<br />
root 4103 0.0 0.0 66528 1868 pts/0 S 22:05 0:00 su -s /bin/bash open-xchange -c /opt/open-xchange/sbin/open-xchange<br />
494 4109 9.5 19.8 2224644 381312 ? Ssl 22:05 1:12 \_ /usr/bin/java -Dsun.net.inetaddr.ttl=3600 -Dnetworkaddress.cache.ttl=3600<br />
<br />
or programmatically<br />
singlenode:~ # pid=$(pgrep -P $(</var/run/open-xchange.pid) java)<br />
<br />
and check the limits applied for that process<br />
singlenode:~ # cat /proc/$pid/limits<br />
Limit Soft Limit Hard Limit Units<br />
Max cpu time unlimited unlimited seconds<br />
Max file size unlimited unlimited bytes<br />
Max data size unlimited unlimited bytes<br />
Max stack size 8388608 unlimited bytes<br />
Max core file size 0 unlimited bytes<br />
Max resident set unlimited unlimited bytes<br />
Max processes 65536 65536 processes<br />
Max open files 65536 65536 files<br />
Max locked memory 65536 65536 bytes<br />
Max address space unlimited unlimited bytes<br />
Max file locks unlimited unlimited locks<br />
Max pending signals 24254 24254 signals<br />
Max msgqueue size 819200 819200 bytes<br />
Max nice priority 0 0<br />
Max realtime priority 0 0<br />
Max realtime timeout unlimited unlimited us<br />
<br />
=== Systemd ===<br />
singlenode:~ # systemctl show open-xchange | grep Limit<br />
StartLimitInterval=10000000<br />
StartLimitBurst=5<br />
StartLimitAction=none<br />
MemoryLimit=18446744073709551615<br />
LimitCPU=18446744073709551615<br />
LimitFSIZE=18446744073709551615<br />
LimitDATA=18446744073709551615<br />
LimitSTACK=18446744073709551615<br />
LimitCORE=18446744073709551615<br />
LimitRSS=18446744073709551615<br />
LimitNOFILE=65536<br />
LimitAS=18446744073709551615<br />
LimitNPROC=65536<br />
LimitMEMLOCK=65536<br />
LimitLOCKS=18446744073709551615<br />
LimitSIGPENDING=19827<br />
LimitMSGQUEUE=819200<br />
LimitNICE=0<br />
LimitRTPRIO=0<br />
LimitRTTIME=18446744073709551615<br />
<br />
== Evaluate JVM commandline options ==<br />
<br />
Before starting Java and the OSGi framework with all the Open-Xchange middleware bundles we have to evaluate several important JVM commandline options that influence the performance and stability of the Open-Xchange middleware stack. These commandline options are specified in '''/opt/open-xchange/etc/ox-scriptconf.sh'''<br />
<br />
=== Format < 7.8.4 ===<br />
For version up to 7.8.3 the JVM configuration options were all specified via a single property '''JAVA_XTRAOPTS''' which resulted in a rather long configuration string that was hard to read and maintain.<br />
<br />
JAVA_XTRAOPTS="-Dsun.net.inetaddr.ttl=3600 -Dnetworkaddress.cache.ttl=3600 -Dnetworkaddress.cache.negative.ttl=10 \<br />
-Dlogback.threadlocal.put.duplicate=false -server -Djava.awt.headless=true -Xmx512M -XX:MaxPermSize=256m -XX:+UseConcMarkSweepGC \<br />
-XX:+UseParNewGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:NewRatio=3 -XX:+UseTLAB \<br />
-Dosgi.compatibility.bootdelegation=false -XX:-OmitStackTraceInFastThrow"<br />
<br />
During the startup of the open-xchange service (step 11 of diagram Service startup using systemd and step 10 of diagram Service startup using upstart) this JAVA_XTRAOPTS string goes through a basic validity check and is then passed to the JVM.<br />
<br />
=== Format >= 7.8.4 ===<br />
Beginning from version 7.8.4 the options are split up and sorted into different categories like:<br />
<br />
JAVA_OPTS_GC="-XX:+UseConcMarkSweepGC -XX:+UseParNewGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:NewRatio=3 -XX:+DisableExplicitGC"<br />
JAVA_OPTS_LOG="-Dlogback.threadlocal.put.duplicate=false -XX:-OmitStackTraceInFastThrow"<br />
JAVA_OPTS_MEM="-XX:MaxHeapSize=512M -XX:MaxPermSize=256M -XX:+UseTLAB"<br />
JAVA_OPTS_NET="-Dsun.net.inetaddr.ttl=3600 -Dnetworkaddress.cache.ttl=3600 -Dnetworkaddress.cache.negative.ttl=10"<br />
JAVA_OPTS_OSGI="-Dosgi.compatibility.bootdelegation=false"<br />
JAVA_OPTS_SERVER="-server -Djava.awt.headless=true"<br />
<br />
JAVA_OPTS_OTHER=""<br />
<br />
#JAVA_OPTS_DEBUG="-XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/mnt/heapdump -Xloggc:/var/log/open-xchange/gc.log -verbose:gc -XX:+PrintGCDateStamps \<br />
-XX:+PrintHeapAtGC -XX:+PrintGCApplicationStoppedTime -XX:+PrintTenuringDistribution"<br />
<br />
which is much easier to read an maintain. Furthermore we added a useful '''JAVA_OPTS_DEBUG''' configuration parameter that comes preconfigured but unused/commented so that customers can quickly enable the most commonly used debugging options in case they have to provide detailed system information to Open-Xchange.<br />
<br />
During the startup of the open-xchange service (step 11 of diagram ''Service startup using systemd'' and step 10 of diagram ''Service startup using upstart'') all uncommented JAVA_OPTS_* categories will then be evaluated and combined again into a single string of commandline options that can then be passed to the JVM.<br />
<br />
=== Upgrade procedure ===<br />
During the upgrade '''7.8.3-revX -> 7.8.4-revY''' existing JAVA_XTRAOPTS will automatically be split up and migrated to the new categories approach. A backup of the file (ox-scriptconf.sh.timestamp) will be created and admins are urged to check for proper migration before deleting the backup file like it's already done with *.dpkg-dist or *.rpmnew files on Debian or RHEL/SLES distributions.<br />
<br />
== Encoding detection during JVM startup ==<br />
<br />
=== Background Infos ===<br />
<br />
From ''man 7 locale''<br />
<br />
DESCRIPTION<br />
A locale is a set of language and cultural rules. These cover aspects such as lan‐<br />
guage for messages, different character sets, lexicographic conventions, and so on.<br />
A program needs to be able to determine its locale and act accordingly to be portable<br />
to different cultures.<br />
<br />
The format to specify a system locale is: '''''language[_territory][.codeset][@modifier]'''''.<br />
<br />
Where all of<br />
<br />
* en_US.UTF-8<br />
* en_US.utf8<br />
* en_US.utf-8<br />
* en_US.UTF_8 <br />
<br />
will result in a valid UTF-8 charmap for the english language with locale categories configured as<br />
<br />
LANG=en_US.UTF-8<br />
LC_CTYPE="en_US.UTF-8"<br />
LC_NUMERIC="en_US.UTF-8"<br />
LC_TIME="en_US.UTF-8"<br />
LC_COLLATE="en_US.UTF-8"<br />
LC_MONETARY="en_US.UTF-8"<br />
LC_MESSAGES="en_US.UTF-8"<br />
LC_PAPER="en_US.UTF-8"<br />
LC_NAME="en_US.UTF-8"<br />
LC_ADDRESS="en_US.UTF-8"<br />
LC_TELEPHONE="en_US.UTF-8"<br />
LC_MEASUREMENT="en_US.UTF-8"<br />
LC_IDENTIFICATION="en_US.UTF-8"<br />
LC_ALL=<br />
<br />
See for more infos: locale(1), locale(7), [https://en.wikipedia.org/wiki/List_of_ISO_639-1_codes language codes], [https://en.wikipedia.org/wiki/ISO_3166-1#Current_codes country codes] and [https://en.wikipedia.org/wiki/Character_encoding character encodings]<br />
<br />
==== JVM startup ====<br />
<br />
The default file.encoding detection is done at JVM startup based upon LC_CTYPE which makes sense as it's documented as:<br />
<br />
CODESET (LC_CTYPE)<br />
Return a string with the name of the character encoding used in the selected locale, such as "UTF-8", "ISO-8859-1",<br />
or "ANSI_X3.4-1968" (better known as US-ASCII). This is the same string that you get with "locale charmap".<br />
For a list of character encoding names, try "locale -m", cf. locale(1).<br />
<br />
Furthermore this influences the defaultCharset selection of the JVM.<br />
<br />
/**<br />
* Returns the default charset of this Java virtual machine.<br />
*<br />
* The default charset is determined during virtual-machine startup and<br />
* typically depends upon the locale and charset of the underlying<br />
* operating system.<br />
*<br />
* @return A charset object for the default charset<br />
*<br />
* @since 1.5<br />
*/<br />
public static Charset defaultCharset() {<br />
if (defaultCharset == null) {<br />
synchronized (Charset.class) {<br />
String csn = AccessController.doPrivileged(<br />
new GetPropertyAction("file.encoding"));<br />
Charset cs = lookup(csn);<br />
if (cs != null)<br />
defaultCharset = cs;<br />
else<br />
defaultCharset = forName("UTF-8");<br />
}<br />
}<br />
return defaultCharset;<br />
}<br />
<br />
Although we try to specify valid unicode encodings for every IO operation the Open-Xchange middleware does, some parts of the JVM don't allow to specify the encoding to use and simply falls back to the detected default encoding. That's why '''admins have to make sure to provide a proper unicode environment for the middleware service during every service start''' or use documented workarounds provided by Open-Xchange (e.g. [http://documentation.open-xchange.com/components/middleware/config/develop/index.html#mode=features&feature=Password%20Change%20Script com.openexchange.passwordchange.script.base64]: Encoded strings as Base64 to circumvent character encoding issues on improperly configured distributions).<br />
<br />
See the documentation of your linux distribution on how to configure a proper unicode locale.<br />
<br />
=== Query which file encoding was detected by the JVM during startup ===<br />
[[File:visualvm-file.encoding.png|thumb|Querying the JVM via visualvm|right|300px]]<br />
<br />
The file encoding that was detected by the JVM during startup can be queried via JMX. You can use graphical management tools like [https://visualvm.github.io/ visualvm], query our [[Jolokia]] JMX bridge via http or just use the commandline like<br />
<br />
/opt/open-xchange/sbin/showruntimestats -r | sed -ne 's/.*\({key=file.encoding, value=[^}]*}\).*/\1/p'<br />
{key=file.encoding, value='''UTF-8'''}<br />
<br />
<br clear=all><br />
<br />
=== Locale passing from client to server via SSH ===<br />
<br />
The following setting of sshd allows clients to specify which locale to use on the server and might lead to some confusion or unexpected behaviour when checking the system locale or even to the usage of broken locales during middleware startup if your client sends misconfigured or missing locale configuration to the server.<br />
<br />
grep AcceptEnv /etc/ssh/sshd_config <br />
AcceptEnv LANG LC_*<br />
<br />
For example when i connect to the machine my local german locale is sent to the server <br />
<br />
[vagrant@showcase-community-centos-7 ~]$ locale -v <br />
LANG=de_DE.UTF-8<br />
LC_CTYPE="de_DE.UTF-8"<br />
LC_NUMERIC=de_DE.UTF-8<br />
LC_TIME=de_DE.UTF-8<br />
LC_COLLATE="de_DE.UTF-8"<br />
LC_MONETARY=de_DE.UTF-8<br />
LC_MESSAGES="de_DE.UTF-8"<br />
LC_PAPER=de_DE.UTF-8<br />
LC_NAME=de_DE.UTF-8<br />
LC_ADDRESS=de_DE.UTF-8<br />
LC_TELEPHONE=de_DE.UTF-8<br />
LC_MEASUREMENT=de_DE.UTF-8<br />
LC_IDENTIFICATION=de_DE.UTF-8<br />
LC_ALL=<br />
<br />
But after specifying that i want a new login shell the default locale configured for that user/system will be used instead!<br />
<br />
[root@showcase-community-centos-7 vagrant]# su -s /bin/bash open-xchange -l<br />
-bash-4.2$ locale -v <br />
LANG=en_US.UTF-8<br />
LC_CTYPE="en_US.UTF-8"<br />
LC_NUMERIC="en_US.UTF-8"<br />
LC_TIME="en_US.UTF-8"<br />
LC_COLLATE="en_US.UTF-8"<br />
LC_MONETARY="en_US.UTF-8"<br />
LC_MESSAGES="en_US.UTF-8"<br />
LC_PAPER="en_US.UTF-8"<br />
LC_NAME="en_US.UTF-8"<br />
LC_ADDRESS="en_US.UTF-8"<br />
LC_TELEPHONE="en_US.UTF-8"<br />
LC_MEASUREMENT="en_US.UTF-8"<br />
LC_IDENTIFICATION="en_US.UTF-8"<br />
LC_ALL=<br />
<br />
=== Verify that your system is configured with a proper unicode locale for OX ===<br />
<br />
Assuming you haven't reconfigured the locale of your root user<br />
<br />
[root@showcase-community-centos-7 vagrant] locale charmap<br />
UTF-8<br />
[root@showcase-community-centos-7 vagrant]<br />
<br />
== Warnings during early start ==<br />
<br />
Early warnings before the start of the JVM and related logging libraries like logback and logstash forwarding are redirected to '''/var/log/open-xchange/open-xchange-console.log'''.<br />
<br />
=== Limits ===<br />
If the open-xchange startup script isn't able to set the proper limits on CentOS 6 due to e.g. hard limits being enforced via pam_limits you'll see the following warning in the console log during startup<br />
<br />
singlenode open-xchange # cat open-xchange-console.log<br />
/opt/open-xchange/sbin/open-xchange: line 115: ulimit: max user processes: cannot modify limit: Operation not permitted<br />
<br />
=== Locale ===<br />
If the open-xchange startup script isn't able to detect a unicode locale you'll see a warning like the following (the current charmap might differ)<br />
<br />
WARNING: Couldn't detect proper unicode charmap in locale category LC_CTYPE.<br />
WARNING: Current charmap: "ANSI_X3.4-1968" might cause encoding problems.<br />
<br />
== Summary: Open-Xchange middleware on specific distros ==<br />
The support for the mentioned mechanism of resource control differ depending on the distribution and the init system in use.<br />
<br />
=== Debian 7 ===<br />
;Init<br />
: System V style<br />
;OX Configurable Limits/Defaults<br />
: nofile, nproc<br />
<br />
The mentioned limits can be configured via '''/opt/open-xchange/etc/ox-scriptconf.sh'''. The limits are applied via ulimit in the service's init script. The open-xchange service is finally started via start-stop-daemon which doesn't doesn't consider '''/etc/security/limits.*'''<br />
<br />
=== RHEL 6 / CentOS 6 ===<br />
;Init<br />
: Upstart, System V compatible<br />
;OX Configurable Limits/Defaults<br />
: nofile, nproc<br />
<br />
The mentioned limits can be configured via '''/opt/open-xchange/etc/ox-scriptconf.sh'''. The limits are applied via ulimit in the service's init script. Furthermore as the open-xchange service is finally started via '''su ... open-xchange''' on this distro a user session is opened via su/pam and the default CentOS pam config reads the '''/etc/security/limits.*''' configuration by loading the pam stack like:<br />
<br />
: '''/etc/pam.d/su'''<br />
:: '''-> /etc/pam.d/system-auth'''<br />
::: '''-> pam_limits.so'''<br />
<br />
If NPROC isn't configured for the open-xchange-server it's restricted to '''1024''' globally by default to prevent accidental fork bombs, see '''/etc/security/limits.d/90-nproc.conf''' which can result in severe problems modern multithreaded applications.<br />
<br />
=== RHEL 7 / CentOS 7 / Debian 8 / SLE 12 ===<br />
;Init<br />
: Systemd<br />
;OX Configurable Limits/Defaults<br />
: nofile, nproc<br />
<br />
For systemd the default limits are configured directly in the service's unit file that is shipped by OX and located at '''/usr/lib/systemd/system/open-xchange.service'''. The drop-in config to override or extend the default unit file is located at '''/etc/systemd/system/open-xchange.service.d/limits.conf'''. [https://www.freedesktop.org/software/systemd/man/systemd.exec.html Systemd.exec] shows a whole lot of options that can be used by admins to adapt the default service to their specific needs.<br />
<br />
----<br />
<br />
[[File:diagram_sources.zip]]</div>Marenshttps://oxpedia.org/wiki/index.php?title=AppSuite:MiddlewareStartup&diff=23244AppSuite:MiddlewareStartup2017-04-26T08:06:13Z<p>Marens: /* JVM startup */</p>
<hr />
<div><br />
== Systemd ==<br />
<br />
==== Service definition ====<br />
The default service definition as shipped by Open-Xchange is rather concise and easy to read. You can check by looking at the service file that is installed to either /usr/lib/systemd or /lib/systemd depending on your distribution of choice.<br />
<br />
singlenode:~ # cat /usr/lib/systemd/system/open-xchange.service <br />
[Unit]<br />
After=remote-fs.target<br />
After=time-sync.target ypbind.service sendmail.service cyrus.service <br />
<br />
[Service]<br />
User=open-xchange<br />
PermissionsStartOnly=true<br />
TimeoutStartSec=0<br />
ExecStartPre=/opt/open-xchange/sbin/triggerupdatethemes -u<br />
ExecStart=/opt/open-xchange/sbin/open-xchange<br />
ExecStop=/opt/open-xchange/sbin/shutdown -w<br />
ExecReload=/opt/open-xchange/sbin/triggerreloadconfiguration -d<br />
KillMode=process<br />
LimitNOFILE=65536<br />
LimitNPROC=65536<br />
<br />
[Install]<br />
WantedBy=multi-user.target<br />
<br />
==== Drop-in configs ====<br />
Drop in configs allow administrators to easily adapt and override default service unit files. So if you want to change the default limits or add additional limits have a look at our default example drop-in config which is located at /etc/systemd/system/open-xchange.service.d/limits.conf<br />
<br />
singlenode:~ # cat /etc/systemd/system/open-xchange.service.d/limits.conf <br />
# Override and add options in this file<br />
# See systemd.exec(5) for other limits<br />
<br />
[Service]<br />
#LimitNPROC=65536<br />
#LimitNOFILE=65536<br />
<br />
The [https://www.freedesktop.org/software/systemd/man/systemd.exec.html systemd.exec] documentation shows a whole lot of options that can be used by admins to adapt the default service to their specific needs.<br />
<br />
<br clear=all><br />
[[File:mw_start_sysd.png|thumb|Service startup using systemd|left|1000px]]<br />
<br clear=all><br />
<br />
''Note:'' The technical implementation of systemd differs as service definition and config aren't read every time during service start but the changes need to be reread with ''systemctl daemon-reload''. The above is just a simplyfied version to explain the general concept.<br />
<br />
== System V / Upstart ==<br />
<br />
==== Service definition ====<br />
<br />
The default service definition as shipped by Open-Xchange can be found at /etc/init.d/open-xchange and will differ slightly depending on the distro in use.<br />
<br />
<br clear=all><br />
[[File:mw_start_sysv.png|thumb|Service startup using upstart|left|1000px]]<br />
<br clear=all><br />
== Apply Resource limits ==<br />
<br />
As you can see in the diagrams above one of the differences is that:<br />
<br />
* systemd reads all the infos wrt. resource limitations from the original service file and then applies possible further restrictions from the drop-in configs (steps 2 and 8 in the diagram ''Service startup using systemd'') and limits configured in /opt/open-xchange/etc/ox-scriptconf.sh aren't considered anylonger<br />
* the older System V style init has to source /opt/open-xchange/etc/ox-scriptconf.sh to be able to read the configs for the maximum number of processes (NPROC) and files(NRFILES) that the middleware shall be allowed to create/open (steps 3,4 and 15 in the diagram ''Service startup using upstart'').<br />
<br />
=== Background Infos ===<br />
<br />
Several ways exist to restrict resources on a linux system from a global level down to user/groups or even shells and the processes started by them.<br />
<br />
==== Sysctl ====<br />
Sysctl is used to modify kernel parameters at runtime. E.g. to set the maximum number of files<br />
<br />
$ sysctl -w fs.file-max=100000<br />
<br />
To permanently set them append to the main configuration file and reload the settings<br />
<br />
$ echo fs.file-max=100000 >> /etc/sysctl.conf<br />
$ sysctl -p<br />
<br />
More infos can be found via '''man sysctl'''<br />
<br />
==== Limits.conf ====<br />
<br />
Allows to restrict resources an a global, group or user level. E.g:<br />
<br />
$ cat /etc/security/limits.d/90-nproc.conf <br />
# Default limit for number of user's processes to prevent<br />
# accidental fork bombs.<br />
# See rhbz #432903 for reasoning.<br />
<br />
* soft nproc 1024<br />
<br />
From '''man limits.conf''':<br />
<blockquote><br />
''Also, please note that all limit settings are set per login. They are not global, nor are they permanent; existing only for the duration of the session.''<br />
</blockquote><br />
<br />
The limits per login are applied via the ''pam'' stack. See '''man pam''' and '''man pam_limits''' for more details. As those limits are bound to sessions they don't affect most daemons started by our supported init systems or init utils. Most state that they are ignored by design, see [https://bugs.launchpad.net/ubuntu/+source/upstart/+bug/938669/comments/4 upstart], [https://bugzilla.redhat.com/show_bug.cgi?id=754285#c1 systemd] and [https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=302079 start-stop-daemon] <br />
<br />
===== Ulimit =====<br />
From '''man bash'''<br />
<blockquote><br />
ulimit [-HSTabcdefilmnpqrstuvx [limit]]<br />
Provides control over the resources available to the shell and to processes started by it, on systems that allow such control.<br />
</blockquote><br />
This is what we use in our System V compatible init scripts to increase resources for the open-xchange process across multiple distros. Currently only ''the maximum number of processes'' and ''the maximum number of open file descriptors available to a single user'' are increased via ulimit. The values are specified in '''/opt/open-xchange/ox-scriptconf.sh'''<br />
<br />
==== Systemd ====<br />
<br />
===== Control Groups=====<br />
<br />
Control groups should only affect the OX middleware if you create/manage them yourself of if you are using a ''modern'' distribution that already uses systemd as init.<br />
<br />
Citing from the kernel cgroup [https://www.kernel.org/doc/Documentation/cgroup-v2.txt documentation]:<br />
<br />
<blockquote><br />
1-2. What is cgroup?<br />
<br />
cgroup is a mechanism to organize processes hierarchically and<br />
distribute system resources along the hierarchy in a controlled and<br />
configurable manner.<br />
<br />
cgroup is largely composed of two parts - the core and controllers.<br />
cgroup core is primarily responsible for hierarchically organizing<br />
processes. A cgroup controller is usually responsible for<br />
distributing a specific type of system resource along the hierarchy<br />
although there are utility controllers which serve purposes other than<br />
resource distribution.<br />
<br />
cgroups form a tree structure and every process in the system belongs<br />
to one and only one cgroup. All threads of a process belong to the<br />
same cgroup. On creation, all processes are put in the cgroup that<br />
the parent process belongs to at the time. A process can be migrated<br />
to another cgroup. Migration of a process doesn't affect already<br />
existing descendant processes.<br />
<br />
Following certain structural constraints, controllers may be enabled or<br />
disabled selectively on a cgroup. All controller behaviors are<br />
hierarchical - if a controller is enabled on a cgroup, it affects all<br />
processes which belong to the cgroups consisting the inclusive<br />
sub-hierarchy of the cgroup. When a controller is enabled on a nested<br />
cgroup, it always restricts the resource distribution further. The<br />
restrictions set closer to the root in the hierarchy can not be<br />
overridden from further away.<br />
</blockquote><br />
<br />
So processes are organized into a tree structure of control groups and controllers are responsible for the distribution of resources. So what kind of controllers exist?<br />
<br />
<blockquote><br />
5. Controllers<br />
<br />
5-1. CPU<br />
<br />
The "cpu" controllers regulates distribution of CPU cycles. This<br />
controller implements weight and absolute bandwidth limit models for<br />
normal scheduling policy and absolute bandwidth allocation model for<br />
realtime scheduling policy.<br />
<br />
5-2. Memory<br />
<br />
The "memory" controller regulates distribution of memory.<br />
...<br />
While not completely water-tight, all major memory usages by a given<br />
cgroup are tracked so that the total memory consumption can be<br />
accounted and controlled to a reasonable extent.<br />
<br />
5-3. IO<br />
<br />
The "io" controller regulates the distribution of IO resources. This<br />
controller implements both weight based and absolute bandwidth or IOPS<br />
limit distribution; however, weight based distribution is available<br />
only if cfq-iosched is in use and neither scheme is available for<br />
blk-mq devices.<br />
</blockquote><br />
<br />
The open-xchange service is simply put into the default ''system.slice'' without applying further limits.<br />
<br />
singlenode$ systemd-cgls<br />
├─1 /sbin/init<br />
├─system.slice<br />
│ ├─avahi-daemon.service<br />
│ │ ├─501 avahi-daemon: running [singlenode]<br />
│ │ └─514 avahi-daemon: chroot helper<br />
│ ├─console-kit-daemon.service<br />
│ │ └─16164 /usr/sbin/console-kit-daemon --no-daemon<br />
│ ├─dbus.service<br />
│ │ └─508 /usr/bin/dbus-daemon --system --address=systemd: --nofork --nopidfile --systemd-activation<br />
│ ├─munin-node.service<br />
│ │ └─4290 /usr/bin/perl -wT /usr/sbin/munin-node<br />
│ ├─open-xchange.service<br />
│ │ └─6037 /usr/bin/java -Dsun.net.inetaddr.ttl=3600 -Dnetworkaddress.cache.ttl=3600 -Dnetworkaddress.cache.negative.ttl=10 ...<br />
<br />
To check all the details use<br />
<br />
singlenode:~ # systemctl show system.slice<br />
<br />
==== Limits besides control groups ====<br />
Besides control groups systemd allows you to apply other limits to the execution environment of your service. Here we can apply the limits that would normally be applied via ''limits.conf'' or ''ulimit''. Systemd uses [http://man7.org/linux/man-pages/man2/setrlimit.2.html setrlimit] for this. The options that we set by default are:<br />
<br />
* LimitNOFILE<br />
* LimitNPROC<br />
<br />
which you have already seen when we took a look at the [[#Service_definiton|service definition]] and [[#Drop-in_config|drop-in configs]]. ''Again'': The complete documentation wrt. resource limit configurations in drop-in configs can be found at [https://www.freedesktop.org/software/systemd/man/systemd.exec.html systemd.exec]<br />
<br />
== Verify limits ==<br />
<br />
=== System V ===<br />
<br />
Get the proper pid of the java process started for the middleware by either checking '''ps faux'''<br />
root 4103 0.0 0.0 66528 1868 pts/0 S 22:05 0:00 su -s /bin/bash open-xchange -c /opt/open-xchange/sbin/open-xchange<br />
494 4109 9.5 19.8 2224644 381312 ? Ssl 22:05 1:12 \_ /usr/bin/java -Dsun.net.inetaddr.ttl=3600 -Dnetworkaddress.cache.ttl=3600<br />
<br />
or programmatically<br />
singlenode:~ # pid=$(pgrep -P $(</var/run/open-xchange.pid) java)<br />
<br />
and check the limits applied for that process<br />
singlenode:~ # cat /proc/$pid/limits<br />
Limit Soft Limit Hard Limit Units<br />
Max cpu time unlimited unlimited seconds<br />
Max file size unlimited unlimited bytes<br />
Max data size unlimited unlimited bytes<br />
Max stack size 8388608 unlimited bytes<br />
Max core file size 0 unlimited bytes<br />
Max resident set unlimited unlimited bytes<br />
Max processes 65536 65536 processes<br />
Max open files 65536 65536 files<br />
Max locked memory 65536 65536 bytes<br />
Max address space unlimited unlimited bytes<br />
Max file locks unlimited unlimited locks<br />
Max pending signals 24254 24254 signals<br />
Max msgqueue size 819200 819200 bytes<br />
Max nice priority 0 0<br />
Max realtime priority 0 0<br />
Max realtime timeout unlimited unlimited us<br />
<br />
=== Systemd ===<br />
singlenode:~ # systemctl show open-xchange | grep Limit<br />
StartLimitInterval=10000000<br />
StartLimitBurst=5<br />
StartLimitAction=none<br />
MemoryLimit=18446744073709551615<br />
LimitCPU=18446744073709551615<br />
LimitFSIZE=18446744073709551615<br />
LimitDATA=18446744073709551615<br />
LimitSTACK=18446744073709551615<br />
LimitCORE=18446744073709551615<br />
LimitRSS=18446744073709551615<br />
LimitNOFILE=65536<br />
LimitAS=18446744073709551615<br />
LimitNPROC=65536<br />
LimitMEMLOCK=65536<br />
LimitLOCKS=18446744073709551615<br />
LimitSIGPENDING=19827<br />
LimitMSGQUEUE=819200<br />
LimitNICE=0<br />
LimitRTPRIO=0<br />
LimitRTTIME=18446744073709551615<br />
<br />
== Evaluate JVM commandline options ==<br />
<br />
Before starting Java and the OSGi framework with all the Open-Xchange middleware bundles we have to evaluate several important JVM commandline options that influence the performance and stability of the Open-Xchange middleware stack. These commandline options are specified in '''/opt/open-xchange/etc/ox-scriptconf.sh'''<br />
<br />
=== Format < 7.8.4 ===<br />
For version up to 7.8.3 the JVM configuration options were all specified via a single property '''JAVA_XTRAOPTS''' which resulted in a rather long configuration string that was hard to read and maintain.<br />
<br />
JAVA_XTRAOPTS="-Dsun.net.inetaddr.ttl=3600 -Dnetworkaddress.cache.ttl=3600 -Dnetworkaddress.cache.negative.ttl=10 \<br />
-Dlogback.threadlocal.put.duplicate=false -server -Djava.awt.headless=true -Xmx512M -XX:MaxPermSize=256m -XX:+UseConcMarkSweepGC \<br />
-XX:+UseParNewGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:NewRatio=3 -XX:+UseTLAB \<br />
-Dosgi.compatibility.bootdelegation=false -XX:-OmitStackTraceInFastThrow"<br />
<br />
During the startup of the open-xchange service (step 11 of diagram Service startup using systemd and step 10 of diagram Service startup using upstart) this JAVA_XTRAOPTS string goes through a basic validity check and is then passed to the JVM.<br />
<br />
=== Format >= 7.8.4 ===<br />
Beginning from version 7.8.4 the options are split up and sorted into different categories like:<br />
<br />
JAVA_OPTS_GC="-XX:+UseConcMarkSweepGC -XX:+UseParNewGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:NewRatio=3 -XX:+DisableExplicitGC"<br />
JAVA_OPTS_LOG="-Dlogback.threadlocal.put.duplicate=false -XX:-OmitStackTraceInFastThrow"<br />
JAVA_OPTS_MEM="-XX:MaxHeapSize=512M -XX:MaxPermSize=256M -XX:+UseTLAB"<br />
JAVA_OPTS_NET="-Dsun.net.inetaddr.ttl=3600 -Dnetworkaddress.cache.ttl=3600 -Dnetworkaddress.cache.negative.ttl=10"<br />
JAVA_OPTS_OSGI="-Dosgi.compatibility.bootdelegation=false"<br />
JAVA_OPTS_SERVER="-server -Djava.awt.headless=true"<br />
<br />
JAVA_OPTS_OTHER=""<br />
<br />
#JAVA_OPTS_DEBUG="-XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/mnt/heapdump -Xloggc:/var/log/open-xchange/gc.log -verbose:gc -XX:+PrintGCDateStamps \<br />
-XX:+PrintHeapAtGC -XX:+PrintGCApplicationStoppedTime -XX:+PrintTenuringDistribution"<br />
<br />
which is much easier to read an maintain. Furthermore we added a useful '''JAVA_OPTS_DEBUG''' configuration parameter that comes preconfigured but unused/commented so that customers can quickly enable the most commonly used debugging options in case they have to provide detailed system information to Open-Xchange.<br />
<br />
During the startup of the open-xchange service (step 11 of diagram ''Service startup using systemd'' and step 10 of diagram ''Service startup using upstart'') all uncommented JAVA_OPTS_* categories will then be evaluated and combined again into a single string of commandline options that can then be passed to the JVM.<br />
<br />
=== Upgrade procedure ===<br />
During the upgrade '''7.8.3-revX -> 7.8.4-revY''' existing JAVA_XTRAOPTS will automatically be split up and migrated to the new categories approach. A backup of the file (ox-scriptconf.sh.timestamp) will be created and admins are urged to check for proper migration before deleting the backup file like it's already done with *.dpkg-dist or *.rpmnew files on Debian or RHEL/SLES distributions.<br />
<br />
== Encoding detection during JVM startup ==<br />
<br />
=== Background Infos ===<br />
<br />
From ''man 7 locale''<br />
<br />
DESCRIPTION<br />
A locale is a set of language and cultural rules. These cover aspects such as lan‐<br />
guage for messages, different character sets, lexicographic conventions, and so on.<br />
A program needs to be able to determine its locale and act accordingly to be portable<br />
to different cultures.<br />
<br />
The format to specify a system locale is: '''''language[_territory][.codeset][@modifier]'''''.<br />
<br />
Where all of<br />
<br />
* en_US.UTF-8<br />
* en_US.utf8<br />
* en_US.utf-8<br />
* en_US.UTF_8 <br />
<br />
will result in a valid UTF-8 charmap for the english language with locale categories configured as<br />
<br />
LANG=en_US.UTF-8<br />
LC_CTYPE="en_US.UTF-8"<br />
LC_NUMERIC="en_US.UTF-8"<br />
LC_TIME="en_US.UTF-8"<br />
LC_COLLATE="en_US.UTF-8"<br />
LC_MONETARY="en_US.UTF-8"<br />
LC_MESSAGES="en_US.UTF-8"<br />
LC_PAPER="en_US.UTF-8"<br />
LC_NAME="en_US.UTF-8"<br />
LC_ADDRESS="en_US.UTF-8"<br />
LC_TELEPHONE="en_US.UTF-8"<br />
LC_MEASUREMENT="en_US.UTF-8"<br />
LC_IDENTIFICATION="en_US.UTF-8"<br />
LC_ALL=<br />
<br />
See for more infos: locale(1), locale(7), [https://en.wikipedia.org/wiki/List_of_ISO_639-1_codes language codes], [https://en.wikipedia.org/wiki/ISO_3166-1#Current_codes country codes] and [https://en.wikipedia.org/wiki/Character_encoding character encodings]<br />
<br />
==== JVM startup ====<br />
<br />
The default file.encoding detection is done at JVM startup based upon LC_CTYPE which makes sense as it's documented as:<br />
<br />
CODESET (LC_CTYPE)<br />
Return a string with the name of the character encoding used in the selected locale, such as "UTF-8", "ISO-8859-1",<br />
or "ANSI_X3.4-1968" (better known as US-ASCII). This is the same string that you get with "locale charmap".<br />
For a list of character encoding names, try "locale -m", cf. locale(1).<br />
<br />
Furthermore this influences the defaultCharset selection of the JVM.<br />
<br />
/**<br />
* Returns the default charset of this Java virtual machine.<br />
*<br />
* The default charset is determined during virtual-machine startup and<br />
* typically depends upon the locale and charset of the underlying<br />
* operating system.<br />
*<br />
* @return A charset object for the default charset<br />
*<br />
* @since 1.5<br />
*/<br />
public static Charset defaultCharset() {<br />
if (defaultCharset == null) {<br />
synchronized (Charset.class) {<br />
String csn = AccessController.doPrivileged(<br />
new GetPropertyAction("file.encoding"));<br />
Charset cs = lookup(csn);<br />
if (cs != null)<br />
defaultCharset = cs;<br />
else<br />
defaultCharset = forName("UTF-8");<br />
}<br />
}<br />
return defaultCharset;<br />
}<br />
<br />
Although we try to specify valid unicode encodings for every IO operation the Open-Xchange middleware does, some parts of the JVM don't allow to specify the encoding to use and simply falls back to the detected default encoding. That's why '''admins have to make sure to provide a proper unicode environment for the middleware service during every service start''' or use documented workarounds provided by Open-Xchange (e.g. [http://documentation.open-xchange.com/components/middleware/config/develop/index.html#mode=features&feature=Password%20Change%20Script com.openexchange.passwordchange.script.base64]: Encoded strings as Base64 to circumvent character encoding issues on improperly configured distributions).<br />
<br />
See the documentation of your linux distribution on how to configure a proper unicode locale.<br />
<br />
=== Query which file encoding was detected by the JVM during startup ===<br />
[[File:visualvm-file.encoding.png|thumb|Querying the JVM via visualvm|right|300px]]<br />
<br />
The file encoding that was detected by the JVM during startup can be queried via JMX. You can use graphical management tools like [https://visualvm.github.io/ visualvm], query our [[Jolokia]] JMX bridge via http or just use the commandline like<br />
<br />
/opt/open-xchange/sbin/showruntimestats -r | sed -ne 's/.*\({key=file.encoding, value=[^}]*}\).*/\1/p'<br />
{key=file.encoding, value='''UTF-8'''}<br />
<br />
<br clear=all><br />
<br />
=== Locale passing from client to server via SSH ===<br />
<br />
The following setting of sshd allows clients to specify which locale to use on the server and might lead to some confusion or unexpected behaviour when checking the system locale or even to the usage of broken locales during middleware startup if your client sends misconfigured or missing locale configuration to the server.<br />
<br />
grep AcceptEnv /etc/ssh/sshd_config <br />
AcceptEnv LANG LC_*<br />
<br />
For example when i connect to the machine my local german locale is sent to the server <br />
<br />
[vagrant@showcase-community-centos-7 ~]$ locale -v <br />
LANG=de_DE.UTF-8<br />
LC_CTYPE="de_DE.UTF-8"<br />
LC_NUMERIC=de_DE.UTF-8<br />
LC_TIME=de_DE.UTF-8<br />
LC_COLLATE="de_DE.UTF-8"<br />
LC_MONETARY=de_DE.UTF-8<br />
LC_MESSAGES="de_DE.UTF-8"<br />
LC_PAPER=de_DE.UTF-8<br />
LC_NAME=de_DE.UTF-8<br />
LC_ADDRESS=de_DE.UTF-8<br />
LC_TELEPHONE=de_DE.UTF-8<br />
LC_MEASUREMENT=de_DE.UTF-8<br />
LC_IDENTIFICATION=de_DE.UTF-8<br />
LC_ALL=<br />
<br />
But after specifying that i want a new login shell the default locale configured for that user/system will be used instead!<br />
<br />
[root@showcase-community-centos-7 vagrant]# su -s /bin/bash open-xchange -l<br />
-bash-4.2$ locale -v <br />
LANG=en_US.UTF-8<br />
LC_CTYPE="en_US.UTF-8"<br />
LC_NUMERIC="en_US.UTF-8"<br />
LC_TIME="en_US.UTF-8"<br />
LC_COLLATE="en_US.UTF-8"<br />
LC_MONETARY="en_US.UTF-8"<br />
LC_MESSAGES="en_US.UTF-8"<br />
LC_PAPER="en_US.UTF-8"<br />
LC_NAME="en_US.UTF-8"<br />
LC_ADDRESS="en_US.UTF-8"<br />
LC_TELEPHONE="en_US.UTF-8"<br />
LC_MEASUREMENT="en_US.UTF-8"<br />
LC_IDENTIFICATION="en_US.UTF-8"<br />
LC_ALL=<br />
<br />
=== Verify that your system is configured with a proper unicode locale for OX ===<br />
<br />
Assuming you haven't reconfigured the locale of your root user<br />
<br />
i|marens@karotte ~ $ su -<br />
Password: <br />
i|karotte ~ # locale charmap<br />
UTF-8<br />
i|karotte ~ #<br />
<br />
== Warnings during early start ==<br />
<br />
Early warnings before the start of the JVM and related logging libraries like logback and logstash forwarding are redirected to '''/var/log/open-xchange/open-xchange-console.log'''.<br />
<br />
=== Limits ===<br />
If the open-xchange startup script isn't able to set the proper limits on CentOS 6 due to e.g. hard limits being enforced via pam_limits you'll see the following warning in the console log during startup<br />
<br />
singlenode open-xchange # cat open-xchange-console.log<br />
/opt/open-xchange/sbin/open-xchange: line 115: ulimit: max user processes: cannot modify limit: Operation not permitted<br />
<br />
=== Locale ===<br />
If the open-xchange startup script isn't able to detect a unicode locale you'll see a warning like the following (the current charmap might differ)<br />
<br />
WARNING: Couldn't detect proper unicode charmap in locale category LC_CTYPE.<br />
WARNING: Current charmap: "ANSI_X3.4-1968" might cause encoding problems.<br />
<br />
== Summary: Open-Xchange middleware on specific distros ==<br />
The support for the mentioned mechanism of resource control differ depending on the distribution and the init system in use.<br />
<br />
=== Debian 7 ===<br />
;Init<br />
: System V style<br />
;OX Configurable Limits/Defaults<br />
: nofile, nproc<br />
<br />
The mentioned limits can be configured via '''/opt/open-xchange/etc/ox-scriptconf.sh'''. The limits are applied via ulimit in the service's init script. The open-xchange service is finally started via start-stop-daemon which doesn't doesn't consider '''/etc/security/limits.*'''<br />
<br />
=== RHEL 6 / CentOS 6 ===<br />
;Init<br />
: Upstart, System V compatible<br />
;OX Configurable Limits/Defaults<br />
: nofile, nproc<br />
<br />
The mentioned limits can be configured via '''/opt/open-xchange/etc/ox-scriptconf.sh'''. The limits are applied via ulimit in the service's init script. Furthermore as the open-xchange service is finally started via '''su ... open-xchange''' on this distro a user session is opened via su/pam and the default CentOS pam config reads the '''/etc/security/limits.*''' configuration by loading the pam stack like:<br />
<br />
: '''/etc/pam.d/su'''<br />
:: '''-> /etc/pam.d/system-auth'''<br />
::: '''-> pam_limits.so'''<br />
<br />
If NPROC isn't configured for the open-xchange-server it's restricted to '''1024''' globally by default to prevent accidental fork bombs, see '''/etc/security/limits.d/90-nproc.conf''' which can result in severe problems modern multithreaded applications.<br />
<br />
=== RHEL 7 / CentOS 7 / Debian 8 / SLE 12 ===<br />
;Init<br />
: Systemd<br />
;OX Configurable Limits/Defaults<br />
: nofile, nproc<br />
<br />
For systemd the default limits are configured directly in the service's unit file that is shipped by OX and located at '''/usr/lib/systemd/system/open-xchange.service'''. The drop-in config to override or extend the default unit file is located at '''/etc/systemd/system/open-xchange.service.d/limits.conf'''. [https://www.freedesktop.org/software/systemd/man/systemd.exec.html Systemd.exec] shows a whole lot of options that can be used by admins to adapt the default service to their specific needs.<br />
<br />
----<br />
<br />
[[File:diagram_sources.zip]]</div>Marenshttps://oxpedia.org/wiki/index.php?title=AppSuite:MiddlewareStartup&diff=23243AppSuite:MiddlewareStartup2017-04-26T08:04:05Z<p>Marens: /* JVM startup */</p>
<hr />
<div><br />
== Systemd ==<br />
<br />
==== Service definition ====<br />
The default service definition as shipped by Open-Xchange is rather concise and easy to read. You can check by looking at the service file that is installed to either /usr/lib/systemd or /lib/systemd depending on your distribution of choice.<br />
<br />
singlenode:~ # cat /usr/lib/systemd/system/open-xchange.service <br />
[Unit]<br />
After=remote-fs.target<br />
After=time-sync.target ypbind.service sendmail.service cyrus.service <br />
<br />
[Service]<br />
User=open-xchange<br />
PermissionsStartOnly=true<br />
TimeoutStartSec=0<br />
ExecStartPre=/opt/open-xchange/sbin/triggerupdatethemes -u<br />
ExecStart=/opt/open-xchange/sbin/open-xchange<br />
ExecStop=/opt/open-xchange/sbin/shutdown -w<br />
ExecReload=/opt/open-xchange/sbin/triggerreloadconfiguration -d<br />
KillMode=process<br />
LimitNOFILE=65536<br />
LimitNPROC=65536<br />
<br />
[Install]<br />
WantedBy=multi-user.target<br />
<br />
==== Drop-in configs ====<br />
Drop in configs allow administrators to easily adapt and override default service unit files. So if you want to change the default limits or add additional limits have a look at our default example drop-in config which is located at /etc/systemd/system/open-xchange.service.d/limits.conf<br />
<br />
singlenode:~ # cat /etc/systemd/system/open-xchange.service.d/limits.conf <br />
# Override and add options in this file<br />
# See systemd.exec(5) for other limits<br />
<br />
[Service]<br />
#LimitNPROC=65536<br />
#LimitNOFILE=65536<br />
<br />
The [https://www.freedesktop.org/software/systemd/man/systemd.exec.html systemd.exec] documentation shows a whole lot of options that can be used by admins to adapt the default service to their specific needs.<br />
<br />
<br clear=all><br />
[[File:mw_start_sysd.png|thumb|Service startup using systemd|left|1000px]]<br />
<br clear=all><br />
<br />
''Note:'' The technical implementation of systemd differs as service definition and config aren't read every time during service start but the changes need to be reread with ''systemctl daemon-reload''. The above is just a simplyfied version to explain the general concept.<br />
<br />
== System V / Upstart ==<br />
<br />
==== Service definition ====<br />
<br />
The default service definition as shipped by Open-Xchange can be found at /etc/init.d/open-xchange and will differ slightly depending on the distro in use.<br />
<br />
<br clear=all><br />
[[File:mw_start_sysv.png|thumb|Service startup using upstart|left|1000px]]<br />
<br clear=all><br />
== Apply Resource limits ==<br />
<br />
As you can see in the diagrams above one of the differences is that:<br />
<br />
* systemd reads all the infos wrt. resource limitations from the original service file and then applies possible further restrictions from the drop-in configs (steps 2 and 8 in the diagram ''Service startup using systemd'') and limits configured in /opt/open-xchange/etc/ox-scriptconf.sh aren't considered anylonger<br />
* the older System V style init has to source /opt/open-xchange/etc/ox-scriptconf.sh to be able to read the configs for the maximum number of processes (NPROC) and files(NRFILES) that the middleware shall be allowed to create/open (steps 3,4 and 15 in the diagram ''Service startup using upstart'').<br />
<br />
=== Background Infos ===<br />
<br />
Several ways exist to restrict resources on a linux system from a global level down to user/groups or even shells and the processes started by them.<br />
<br />
==== Sysctl ====<br />
Sysctl is used to modify kernel parameters at runtime. E.g. to set the maximum number of files<br />
<br />
$ sysctl -w fs.file-max=100000<br />
<br />
To permanently set them append to the main configuration file and reload the settings<br />
<br />
$ echo fs.file-max=100000 >> /etc/sysctl.conf<br />
$ sysctl -p<br />
<br />
More infos can be found via '''man sysctl'''<br />
<br />
==== Limits.conf ====<br />
<br />
Allows to restrict resources an a global, group or user level. E.g:<br />
<br />
$ cat /etc/security/limits.d/90-nproc.conf <br />
# Default limit for number of user's processes to prevent<br />
# accidental fork bombs.<br />
# See rhbz #432903 for reasoning.<br />
<br />
* soft nproc 1024<br />
<br />
From '''man limits.conf''':<br />
<blockquote><br />
''Also, please note that all limit settings are set per login. They are not global, nor are they permanent; existing only for the duration of the session.''<br />
</blockquote><br />
<br />
The limits per login are applied via the ''pam'' stack. See '''man pam''' and '''man pam_limits''' for more details. As those limits are bound to sessions they don't affect most daemons started by our supported init systems or init utils. Most state that they are ignored by design, see [https://bugs.launchpad.net/ubuntu/+source/upstart/+bug/938669/comments/4 upstart], [https://bugzilla.redhat.com/show_bug.cgi?id=754285#c1 systemd] and [https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=302079 start-stop-daemon] <br />
<br />
===== Ulimit =====<br />
From '''man bash'''<br />
<blockquote><br />
ulimit [-HSTabcdefilmnpqrstuvx [limit]]<br />
Provides control over the resources available to the shell and to processes started by it, on systems that allow such control.<br />
</blockquote><br />
This is what we use in our System V compatible init scripts to increase resources for the open-xchange process across multiple distros. Currently only ''the maximum number of processes'' and ''the maximum number of open file descriptors available to a single user'' are increased via ulimit. The values are specified in '''/opt/open-xchange/ox-scriptconf.sh'''<br />
<br />
==== Systemd ====<br />
<br />
===== Control Groups=====<br />
<br />
Control groups should only affect the OX middleware if you create/manage them yourself of if you are using a ''modern'' distribution that already uses systemd as init.<br />
<br />
Citing from the kernel cgroup [https://www.kernel.org/doc/Documentation/cgroup-v2.txt documentation]:<br />
<br />
<blockquote><br />
1-2. What is cgroup?<br />
<br />
cgroup is a mechanism to organize processes hierarchically and<br />
distribute system resources along the hierarchy in a controlled and<br />
configurable manner.<br />
<br />
cgroup is largely composed of two parts - the core and controllers.<br />
cgroup core is primarily responsible for hierarchically organizing<br />
processes. A cgroup controller is usually responsible for<br />
distributing a specific type of system resource along the hierarchy<br />
although there are utility controllers which serve purposes other than<br />
resource distribution.<br />
<br />
cgroups form a tree structure and every process in the system belongs<br />
to one and only one cgroup. All threads of a process belong to the<br />
same cgroup. On creation, all processes are put in the cgroup that<br />
the parent process belongs to at the time. A process can be migrated<br />
to another cgroup. Migration of a process doesn't affect already<br />
existing descendant processes.<br />
<br />
Following certain structural constraints, controllers may be enabled or<br />
disabled selectively on a cgroup. All controller behaviors are<br />
hierarchical - if a controller is enabled on a cgroup, it affects all<br />
processes which belong to the cgroups consisting the inclusive<br />
sub-hierarchy of the cgroup. When a controller is enabled on a nested<br />
cgroup, it always restricts the resource distribution further. The<br />
restrictions set closer to the root in the hierarchy can not be<br />
overridden from further away.<br />
</blockquote><br />
<br />
So processes are organized into a tree structure of control groups and controllers are responsible for the distribution of resources. So what kind of controllers exist?<br />
<br />
<blockquote><br />
5. Controllers<br />
<br />
5-1. CPU<br />
<br />
The "cpu" controllers regulates distribution of CPU cycles. This<br />
controller implements weight and absolute bandwidth limit models for<br />
normal scheduling policy and absolute bandwidth allocation model for<br />
realtime scheduling policy.<br />
<br />
5-2. Memory<br />
<br />
The "memory" controller regulates distribution of memory.<br />
...<br />
While not completely water-tight, all major memory usages by a given<br />
cgroup are tracked so that the total memory consumption can be<br />
accounted and controlled to a reasonable extent.<br />
<br />
5-3. IO<br />
<br />
The "io" controller regulates the distribution of IO resources. This<br />
controller implements both weight based and absolute bandwidth or IOPS<br />
limit distribution; however, weight based distribution is available<br />
only if cfq-iosched is in use and neither scheme is available for<br />
blk-mq devices.<br />
</blockquote><br />
<br />
The open-xchange service is simply put into the default ''system.slice'' without applying further limits.<br />
<br />
singlenode$ systemd-cgls<br />
├─1 /sbin/init<br />
├─system.slice<br />
│ ├─avahi-daemon.service<br />
│ │ ├─501 avahi-daemon: running [singlenode]<br />
│ │ └─514 avahi-daemon: chroot helper<br />
│ ├─console-kit-daemon.service<br />
│ │ └─16164 /usr/sbin/console-kit-daemon --no-daemon<br />
│ ├─dbus.service<br />
│ │ └─508 /usr/bin/dbus-daemon --system --address=systemd: --nofork --nopidfile --systemd-activation<br />
│ ├─munin-node.service<br />
│ │ └─4290 /usr/bin/perl -wT /usr/sbin/munin-node<br />
│ ├─open-xchange.service<br />
│ │ └─6037 /usr/bin/java -Dsun.net.inetaddr.ttl=3600 -Dnetworkaddress.cache.ttl=3600 -Dnetworkaddress.cache.negative.ttl=10 ...<br />
<br />
To check all the details use<br />
<br />
singlenode:~ # systemctl show system.slice<br />
<br />
==== Limits besides control groups ====<br />
Besides control groups systemd allows you to apply other limits to the execution environment of your service. Here we can apply the limits that would normally be applied via ''limits.conf'' or ''ulimit''. Systemd uses [http://man7.org/linux/man-pages/man2/setrlimit.2.html setrlimit] for this. The options that we set by default are:<br />
<br />
* LimitNOFILE<br />
* LimitNPROC<br />
<br />
which you have already seen when we took a look at the [[#Service_definiton|service definition]] and [[#Drop-in_config|drop-in configs]]. ''Again'': The complete documentation wrt. resource limit configurations in drop-in configs can be found at [https://www.freedesktop.org/software/systemd/man/systemd.exec.html systemd.exec]<br />
<br />
== Verify limits ==<br />
<br />
=== System V ===<br />
<br />
Get the proper pid of the java process started for the middleware by either checking '''ps faux'''<br />
root 4103 0.0 0.0 66528 1868 pts/0 S 22:05 0:00 su -s /bin/bash open-xchange -c /opt/open-xchange/sbin/open-xchange<br />
494 4109 9.5 19.8 2224644 381312 ? Ssl 22:05 1:12 \_ /usr/bin/java -Dsun.net.inetaddr.ttl=3600 -Dnetworkaddress.cache.ttl=3600<br />
<br />
or programmatically<br />
singlenode:~ # pid=$(pgrep -P $(</var/run/open-xchange.pid) java)<br />
<br />
and check the limits applied for that process<br />
singlenode:~ # cat /proc/$pid/limits<br />
Limit Soft Limit Hard Limit Units<br />
Max cpu time unlimited unlimited seconds<br />
Max file size unlimited unlimited bytes<br />
Max data size unlimited unlimited bytes<br />
Max stack size 8388608 unlimited bytes<br />
Max core file size 0 unlimited bytes<br />
Max resident set unlimited unlimited bytes<br />
Max processes 65536 65536 processes<br />
Max open files 65536 65536 files<br />
Max locked memory 65536 65536 bytes<br />
Max address space unlimited unlimited bytes<br />
Max file locks unlimited unlimited locks<br />
Max pending signals 24254 24254 signals<br />
Max msgqueue size 819200 819200 bytes<br />
Max nice priority 0 0<br />
Max realtime priority 0 0<br />
Max realtime timeout unlimited unlimited us<br />
<br />
=== Systemd ===<br />
singlenode:~ # systemctl show open-xchange | grep Limit<br />
StartLimitInterval=10000000<br />
StartLimitBurst=5<br />
StartLimitAction=none<br />
MemoryLimit=18446744073709551615<br />
LimitCPU=18446744073709551615<br />
LimitFSIZE=18446744073709551615<br />
LimitDATA=18446744073709551615<br />
LimitSTACK=18446744073709551615<br />
LimitCORE=18446744073709551615<br />
LimitRSS=18446744073709551615<br />
LimitNOFILE=65536<br />
LimitAS=18446744073709551615<br />
LimitNPROC=65536<br />
LimitMEMLOCK=65536<br />
LimitLOCKS=18446744073709551615<br />
LimitSIGPENDING=19827<br />
LimitMSGQUEUE=819200<br />
LimitNICE=0<br />
LimitRTPRIO=0<br />
LimitRTTIME=18446744073709551615<br />
<br />
== Evaluate JVM commandline options ==<br />
<br />
Before starting Java and the OSGi framework with all the Open-Xchange middleware bundles we have to evaluate several important JVM commandline options that influence the performance and stability of the Open-Xchange middleware stack. These commandline options are specified in '''/opt/open-xchange/etc/ox-scriptconf.sh'''<br />
<br />
=== Format < 7.8.4 ===<br />
For version up to 7.8.3 the JVM configuration options were all specified via a single property '''JAVA_XTRAOPTS''' which resulted in a rather long configuration string that was hard to read and maintain.<br />
<br />
JAVA_XTRAOPTS="-Dsun.net.inetaddr.ttl=3600 -Dnetworkaddress.cache.ttl=3600 -Dnetworkaddress.cache.negative.ttl=10 \<br />
-Dlogback.threadlocal.put.duplicate=false -server -Djava.awt.headless=true -Xmx512M -XX:MaxPermSize=256m -XX:+UseConcMarkSweepGC \<br />
-XX:+UseParNewGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:NewRatio=3 -XX:+UseTLAB \<br />
-Dosgi.compatibility.bootdelegation=false -XX:-OmitStackTraceInFastThrow"<br />
<br />
During the startup of the open-xchange service (step 11 of diagram Service startup using systemd and step 10 of diagram Service startup using upstart) this JAVA_XTRAOPTS string goes through a basic validity check and is then passed to the JVM.<br />
<br />
=== Format >= 7.8.4 ===<br />
Beginning from version 7.8.4 the options are split up and sorted into different categories like:<br />
<br />
JAVA_OPTS_GC="-XX:+UseConcMarkSweepGC -XX:+UseParNewGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:NewRatio=3 -XX:+DisableExplicitGC"<br />
JAVA_OPTS_LOG="-Dlogback.threadlocal.put.duplicate=false -XX:-OmitStackTraceInFastThrow"<br />
JAVA_OPTS_MEM="-XX:MaxHeapSize=512M -XX:MaxPermSize=256M -XX:+UseTLAB"<br />
JAVA_OPTS_NET="-Dsun.net.inetaddr.ttl=3600 -Dnetworkaddress.cache.ttl=3600 -Dnetworkaddress.cache.negative.ttl=10"<br />
JAVA_OPTS_OSGI="-Dosgi.compatibility.bootdelegation=false"<br />
JAVA_OPTS_SERVER="-server -Djava.awt.headless=true"<br />
<br />
JAVA_OPTS_OTHER=""<br />
<br />
#JAVA_OPTS_DEBUG="-XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/mnt/heapdump -Xloggc:/var/log/open-xchange/gc.log -verbose:gc -XX:+PrintGCDateStamps \<br />
-XX:+PrintHeapAtGC -XX:+PrintGCApplicationStoppedTime -XX:+PrintTenuringDistribution"<br />
<br />
which is much easier to read an maintain. Furthermore we added a useful '''JAVA_OPTS_DEBUG''' configuration parameter that comes preconfigured but unused/commented so that customers can quickly enable the most commonly used debugging options in case they have to provide detailed system information to Open-Xchange.<br />
<br />
During the startup of the open-xchange service (step 11 of diagram ''Service startup using systemd'' and step 10 of diagram ''Service startup using upstart'') all uncommented JAVA_OPTS_* categories will then be evaluated and combined again into a single string of commandline options that can then be passed to the JVM.<br />
<br />
=== Upgrade procedure ===<br />
During the upgrade '''7.8.3-revX -> 7.8.4-revY''' existing JAVA_XTRAOPTS will automatically be split up and migrated to the new categories approach. A backup of the file (ox-scriptconf.sh.timestamp) will be created and admins are urged to check for proper migration before deleting the backup file like it's already done with *.dpkg-dist or *.rpmnew files on Debian or RHEL/SLES distributions.<br />
<br />
== Encoding detection during JVM startup ==<br />
<br />
=== Background Infos ===<br />
<br />
From ''man 7 locale''<br />
<br />
DESCRIPTION<br />
A locale is a set of language and cultural rules. These cover aspects such as lan‐<br />
guage for messages, different character sets, lexicographic conventions, and so on.<br />
A program needs to be able to determine its locale and act accordingly to be portable<br />
to different cultures.<br />
<br />
The format to specify a system locale is: '''''language[_territory][.codeset][@modifier]'''''.<br />
<br />
Where all of<br />
<br />
* en_US.UTF-8<br />
* en_US.utf8<br />
* en_US.utf-8<br />
* en_US.UTF_8 <br />
<br />
will result in a valid UTF-8 charmap for the english language with locale categories configured as<br />
<br />
LANG=en_US.UTF-8<br />
LC_CTYPE="en_US.UTF-8"<br />
LC_NUMERIC="en_US.UTF-8"<br />
LC_TIME="en_US.UTF-8"<br />
LC_COLLATE="en_US.UTF-8"<br />
LC_MONETARY="en_US.UTF-8"<br />
LC_MESSAGES="en_US.UTF-8"<br />
LC_PAPER="en_US.UTF-8"<br />
LC_NAME="en_US.UTF-8"<br />
LC_ADDRESS="en_US.UTF-8"<br />
LC_TELEPHONE="en_US.UTF-8"<br />
LC_MEASUREMENT="en_US.UTF-8"<br />
LC_IDENTIFICATION="en_US.UTF-8"<br />
LC_ALL=<br />
<br />
See for more infos: locale(1), locale(7), [https://en.wikipedia.org/wiki/List_of_ISO_639-1_codes language codes], [https://en.wikipedia.org/wiki/ISO_3166-1#Current_codes country codes] and [https://en.wikipedia.org/wiki/Character_encoding character encodings]<br />
<br />
==== JVM startup ====<br />
<br />
The default file.encoding detection is done at JVM startup based upon LC_CTYPE which makes sense as it's documented as:<br />
<br />
CODESET (LC_CTYPE)<br />
Return a string with the name of the character encoding used in the selected locale, such as "UTF-8", "ISO-8859-1",<br />
or "ANSI_X3.4-1968" (better known as US-ASCII). This is the same string that you get with "locale charmap".<br />
For a list of character encoding names, try "locale -m", cf. locale(1).<br />
<br />
Furthermore this influences the defaultCharset selection of the JVM.<br />
<br />
/**<br />
* Returns the default charset of this Java virtual machine.<br />
*<br />
* The default charset is determined during virtual-machine startup and<br />
* typically depends upon the locale and charset of the underlying<br />
* operating system.<br />
*<br />
* @return A charset object for the default charset<br />
*<br />
* @since 1.5<br />
*/<br />
public static Charset defaultCharset() {<br />
if (defaultCharset == null) {<br />
synchronized (Charset.class) {<br />
String csn = AccessController.doPrivileged(<br />
new GetPropertyAction("file.encoding"));<br />
Charset cs = lookup(csn);<br />
if (cs != null)<br />
defaultCharset = cs;<br />
else<br />
defaultCharset = forName("UTF-8");<br />
}<br />
}<br />
return defaultCharset;<br />
}<br />
<br />
Although we try to specify valid unicode encodings for every IO operations the Open-Xchange middleware does some parts of the JVM don't allow to specify the encoding to use and falls back to the detected default encoding. That's why admins have to make sure to provide a proper unicode environment for the middleware service during every start or use documented workarounds provided by Open-Xchange (e.g. [http://documentation.open-xchange.com/components/middleware/config/develop/index.html#mode=features&feature=Password%20Change%20Script com.openexchange.passwordchange.script.base64]: Encoded strings as Base64 to circumvent character encoding issues on improperly configured distributions).<br />
<br />
See the documentation of your linux distribution on how to configure a proper unicode locale.<br />
<br />
=== Query which file encoding was detected by the JVM during startup ===<br />
[[File:visualvm-file.encoding.png|thumb|Querying the JVM via visualvm|right|300px]]<br />
<br />
The file encoding that was detected by the JVM during startup can be queried via JMX. You can use graphical management tools like [https://visualvm.github.io/ visualvm], query our [[Jolokia]] JMX bridge via http or just use the commandline like<br />
<br />
/opt/open-xchange/sbin/showruntimestats -r | sed -ne 's/.*\({key=file.encoding, value=[^}]*}\).*/\1/p'<br />
{key=file.encoding, value='''UTF-8'''}<br />
<br />
<br clear=all><br />
<br />
=== Locale passing from client to server via SSH ===<br />
<br />
The following setting of sshd allows clients to specify which locale to use on the server and might lead to some confusion or unexpected behaviour when checking the system locale or even to the usage of broken locales during middleware startup if your client sends misconfigured or missing locale configuration to the server.<br />
<br />
grep AcceptEnv /etc/ssh/sshd_config <br />
AcceptEnv LANG LC_*<br />
<br />
For example when i connect to the machine my local german locale is sent to the server <br />
<br />
[vagrant@showcase-community-centos-7 ~]$ locale -v <br />
LANG=de_DE.UTF-8<br />
LC_CTYPE="de_DE.UTF-8"<br />
LC_NUMERIC=de_DE.UTF-8<br />
LC_TIME=de_DE.UTF-8<br />
LC_COLLATE="de_DE.UTF-8"<br />
LC_MONETARY=de_DE.UTF-8<br />
LC_MESSAGES="de_DE.UTF-8"<br />
LC_PAPER=de_DE.UTF-8<br />
LC_NAME=de_DE.UTF-8<br />
LC_ADDRESS=de_DE.UTF-8<br />
LC_TELEPHONE=de_DE.UTF-8<br />
LC_MEASUREMENT=de_DE.UTF-8<br />
LC_IDENTIFICATION=de_DE.UTF-8<br />
LC_ALL=<br />
<br />
But after specifying that i want a new login shell the default locale configured for that user/system will be used instead!<br />
<br />
[root@showcase-community-centos-7 vagrant]# su -s /bin/bash open-xchange -l<br />
-bash-4.2$ locale -v <br />
LANG=en_US.UTF-8<br />
LC_CTYPE="en_US.UTF-8"<br />
LC_NUMERIC="en_US.UTF-8"<br />
LC_TIME="en_US.UTF-8"<br />
LC_COLLATE="en_US.UTF-8"<br />
LC_MONETARY="en_US.UTF-8"<br />
LC_MESSAGES="en_US.UTF-8"<br />
LC_PAPER="en_US.UTF-8"<br />
LC_NAME="en_US.UTF-8"<br />
LC_ADDRESS="en_US.UTF-8"<br />
LC_TELEPHONE="en_US.UTF-8"<br />
LC_MEASUREMENT="en_US.UTF-8"<br />
LC_IDENTIFICATION="en_US.UTF-8"<br />
LC_ALL=<br />
<br />
=== Verify that your system is configured with a proper unicode locale for OX ===<br />
<br />
Assuming you haven't reconfigured the locale of your root user<br />
<br />
i|marens@karotte ~ $ su -<br />
Password: <br />
i|karotte ~ # locale charmap<br />
UTF-8<br />
i|karotte ~ #<br />
<br />
== Warnings during early start ==<br />
<br />
Early warnings before the start of the JVM and related logging libraries like logback and logstash forwarding are redirected to '''/var/log/open-xchange/open-xchange-console.log'''.<br />
<br />
=== Limits ===<br />
If the open-xchange startup script isn't able to set the proper limits on CentOS 6 due to e.g. hard limits being enforced via pam_limits you'll see the following warning in the console log during startup<br />
<br />
singlenode open-xchange # cat open-xchange-console.log<br />
/opt/open-xchange/sbin/open-xchange: line 115: ulimit: max user processes: cannot modify limit: Operation not permitted<br />
<br />
=== Locale ===<br />
If the open-xchange startup script isn't able to detect a unicode locale you'll see a warning like the following (the current charmap might differ)<br />
<br />
WARNING: Couldn't detect proper unicode charmap in locale category LC_CTYPE.<br />
WARNING: Current charmap: "ANSI_X3.4-1968" might cause encoding problems.<br />
<br />
== Summary: Open-Xchange middleware on specific distros ==<br />
The support for the mentioned mechanism of resource control differ depending on the distribution and the init system in use.<br />
<br />
=== Debian 7 ===<br />
;Init<br />
: System V style<br />
;OX Configurable Limits/Defaults<br />
: nofile, nproc<br />
<br />
The mentioned limits can be configured via '''/opt/open-xchange/etc/ox-scriptconf.sh'''. The limits are applied via ulimit in the service's init script. The open-xchange service is finally started via start-stop-daemon which doesn't doesn't consider '''/etc/security/limits.*'''<br />
<br />
=== RHEL 6 / CentOS 6 ===<br />
;Init<br />
: Upstart, System V compatible<br />
;OX Configurable Limits/Defaults<br />
: nofile, nproc<br />
<br />
The mentioned limits can be configured via '''/opt/open-xchange/etc/ox-scriptconf.sh'''. The limits are applied via ulimit in the service's init script. Furthermore as the open-xchange service is finally started via '''su ... open-xchange''' on this distro a user session is opened via su/pam and the default CentOS pam config reads the '''/etc/security/limits.*''' configuration by loading the pam stack like:<br />
<br />
: '''/etc/pam.d/su'''<br />
:: '''-> /etc/pam.d/system-auth'''<br />
::: '''-> pam_limits.so'''<br />
<br />
If NPROC isn't configured for the open-xchange-server it's restricted to '''1024''' globally by default to prevent accidental fork bombs, see '''/etc/security/limits.d/90-nproc.conf''' which can result in severe problems modern multithreaded applications.<br />
<br />
=== RHEL 7 / CentOS 7 / Debian 8 / SLE 12 ===<br />
;Init<br />
: Systemd<br />
;OX Configurable Limits/Defaults<br />
: nofile, nproc<br />
<br />
For systemd the default limits are configured directly in the service's unit file that is shipped by OX and located at '''/usr/lib/systemd/system/open-xchange.service'''. The drop-in config to override or extend the default unit file is located at '''/etc/systemd/system/open-xchange.service.d/limits.conf'''. [https://www.freedesktop.org/software/systemd/man/systemd.exec.html Systemd.exec] shows a whole lot of options that can be used by admins to adapt the default service to their specific needs.<br />
<br />
----<br />
<br />
[[File:diagram_sources.zip]]</div>Marenshttps://oxpedia.org/wiki/index.php?title=AppSuite:MiddlewareStartup&diff=23242AppSuite:MiddlewareStartup2017-04-25T12:31:37Z<p>Marens: /* Query which file encoding was detected by the JVM during startup */</p>
<hr />
<div><br />
== Systemd ==<br />
<br />
==== Service definition ====<br />
The default service definition as shipped by Open-Xchange is rather concise and easy to read. You can check by looking at the service file that is installed to either /usr/lib/systemd or /lib/systemd depending on your distribution of choice.<br />
<br />
singlenode:~ # cat /usr/lib/systemd/system/open-xchange.service <br />
[Unit]<br />
After=remote-fs.target<br />
After=time-sync.target ypbind.service sendmail.service cyrus.service <br />
<br />
[Service]<br />
User=open-xchange<br />
PermissionsStartOnly=true<br />
TimeoutStartSec=0<br />
ExecStartPre=/opt/open-xchange/sbin/triggerupdatethemes -u<br />
ExecStart=/opt/open-xchange/sbin/open-xchange<br />
ExecStop=/opt/open-xchange/sbin/shutdown -w<br />
ExecReload=/opt/open-xchange/sbin/triggerreloadconfiguration -d<br />
KillMode=process<br />
LimitNOFILE=65536<br />
LimitNPROC=65536<br />
<br />
[Install]<br />
WantedBy=multi-user.target<br />
<br />
==== Drop-in configs ====<br />
Drop in configs allow administrators to easily adapt and override default service unit files. So if you want to change the default limits or add additional limits have a look at our default example drop-in config which is located at /etc/systemd/system/open-xchange.service.d/limits.conf<br />
<br />
singlenode:~ # cat /etc/systemd/system/open-xchange.service.d/limits.conf <br />
# Override and add options in this file<br />
# See systemd.exec(5) for other limits<br />
<br />
[Service]<br />
#LimitNPROC=65536<br />
#LimitNOFILE=65536<br />
<br />
The [https://www.freedesktop.org/software/systemd/man/systemd.exec.html systemd.exec] documentation shows a whole lot of options that can be used by admins to adapt the default service to their specific needs.<br />
<br />
<br clear=all><br />
[[File:mw_start_sysd.png|thumb|Service startup using systemd|left|1000px]]<br />
<br clear=all><br />
<br />
''Note:'' The technical implementation of systemd differs as service definition and config aren't read every time during service start but the changes need to be reread with ''systemctl daemon-reload''. The above is just a simplyfied version to explain the general concept.<br />
<br />
== System V / Upstart ==<br />
<br />
==== Service definition ====<br />
<br />
The default service definition as shipped by Open-Xchange can be found at /etc/init.d/open-xchange and will differ slightly depending on the distro in use.<br />
<br />
<br clear=all><br />
[[File:mw_start_sysv.png|thumb|Service startup using upstart|left|1000px]]<br />
<br clear=all><br />
== Apply Resource limits ==<br />
<br />
As you can see in the diagrams above one of the differences is that:<br />
<br />
* systemd reads all the infos wrt. resource limitations from the original service file and then applies possible further restrictions from the drop-in configs (steps 2 and 8 in the diagram ''Service startup using systemd'') and limits configured in /opt/open-xchange/etc/ox-scriptconf.sh aren't considered anylonger<br />
* the older System V style init has to source /opt/open-xchange/etc/ox-scriptconf.sh to be able to read the configs for the maximum number of processes (NPROC) and files(NRFILES) that the middleware shall be allowed to create/open (steps 3,4 and 15 in the diagram ''Service startup using upstart'').<br />
<br />
=== Background Infos ===<br />
<br />
Several ways exist to restrict resources on a linux system from a global level down to user/groups or even shells and the processes started by them.<br />
<br />
==== Sysctl ====<br />
Sysctl is used to modify kernel parameters at runtime. E.g. to set the maximum number of files<br />
<br />
$ sysctl -w fs.file-max=100000<br />
<br />
To permanently set them append to the main configuration file and reload the settings<br />
<br />
$ echo fs.file-max=100000 >> /etc/sysctl.conf<br />
$ sysctl -p<br />
<br />
More infos can be found via '''man sysctl'''<br />
<br />
==== Limits.conf ====<br />
<br />
Allows to restrict resources an a global, group or user level. E.g:<br />
<br />
$ cat /etc/security/limits.d/90-nproc.conf <br />
# Default limit for number of user's processes to prevent<br />
# accidental fork bombs.<br />
# See rhbz #432903 for reasoning.<br />
<br />
* soft nproc 1024<br />
<br />
From '''man limits.conf''':<br />
<blockquote><br />
''Also, please note that all limit settings are set per login. They are not global, nor are they permanent; existing only for the duration of the session.''<br />
</blockquote><br />
<br />
The limits per login are applied via the ''pam'' stack. See '''man pam''' and '''man pam_limits''' for more details. As those limits are bound to sessions they don't affect most daemons started by our supported init systems or init utils. Most state that they are ignored by design, see [https://bugs.launchpad.net/ubuntu/+source/upstart/+bug/938669/comments/4 upstart], [https://bugzilla.redhat.com/show_bug.cgi?id=754285#c1 systemd] and [https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=302079 start-stop-daemon] <br />
<br />
===== Ulimit =====<br />
From '''man bash'''<br />
<blockquote><br />
ulimit [-HSTabcdefilmnpqrstuvx [limit]]<br />
Provides control over the resources available to the shell and to processes started by it, on systems that allow such control.<br />
</blockquote><br />
This is what we use in our System V compatible init scripts to increase resources for the open-xchange process across multiple distros. Currently only ''the maximum number of processes'' and ''the maximum number of open file descriptors available to a single user'' are increased via ulimit. The values are specified in '''/opt/open-xchange/ox-scriptconf.sh'''<br />
<br />
==== Systemd ====<br />
<br />
===== Control Groups=====<br />
<br />
Control groups should only affect the OX middleware if you create/manage them yourself of if you are using a ''modern'' distribution that already uses systemd as init.<br />
<br />
Citing from the kernel cgroup [https://www.kernel.org/doc/Documentation/cgroup-v2.txt documentation]:<br />
<br />
<blockquote><br />
1-2. What is cgroup?<br />
<br />
cgroup is a mechanism to organize processes hierarchically and<br />
distribute system resources along the hierarchy in a controlled and<br />
configurable manner.<br />
<br />
cgroup is largely composed of two parts - the core and controllers.<br />
cgroup core is primarily responsible for hierarchically organizing<br />
processes. A cgroup controller is usually responsible for<br />
distributing a specific type of system resource along the hierarchy<br />
although there are utility controllers which serve purposes other than<br />
resource distribution.<br />
<br />
cgroups form a tree structure and every process in the system belongs<br />
to one and only one cgroup. All threads of a process belong to the<br />
same cgroup. On creation, all processes are put in the cgroup that<br />
the parent process belongs to at the time. A process can be migrated<br />
to another cgroup. Migration of a process doesn't affect already<br />
existing descendant processes.<br />
<br />
Following certain structural constraints, controllers may be enabled or<br />
disabled selectively on a cgroup. All controller behaviors are<br />
hierarchical - if a controller is enabled on a cgroup, it affects all<br />
processes which belong to the cgroups consisting the inclusive<br />
sub-hierarchy of the cgroup. When a controller is enabled on a nested<br />
cgroup, it always restricts the resource distribution further. The<br />
restrictions set closer to the root in the hierarchy can not be<br />
overridden from further away.<br />
</blockquote><br />
<br />
So processes are organized into a tree structure of control groups and controllers are responsible for the distribution of resources. So what kind of controllers exist?<br />
<br />
<blockquote><br />
5. Controllers<br />
<br />
5-1. CPU<br />
<br />
The "cpu" controllers regulates distribution of CPU cycles. This<br />
controller implements weight and absolute bandwidth limit models for<br />
normal scheduling policy and absolute bandwidth allocation model for<br />
realtime scheduling policy.<br />
<br />
5-2. Memory<br />
<br />
The "memory" controller regulates distribution of memory.<br />
...<br />
While not completely water-tight, all major memory usages by a given<br />
cgroup are tracked so that the total memory consumption can be<br />
accounted and controlled to a reasonable extent.<br />
<br />
5-3. IO<br />
<br />
The "io" controller regulates the distribution of IO resources. This<br />
controller implements both weight based and absolute bandwidth or IOPS<br />
limit distribution; however, weight based distribution is available<br />
only if cfq-iosched is in use and neither scheme is available for<br />
blk-mq devices.<br />
</blockquote><br />
<br />
The open-xchange service is simply put into the default ''system.slice'' without applying further limits.<br />
<br />
singlenode$ systemd-cgls<br />
├─1 /sbin/init<br />
├─system.slice<br />
│ ├─avahi-daemon.service<br />
│ │ ├─501 avahi-daemon: running [singlenode]<br />
│ │ └─514 avahi-daemon: chroot helper<br />
│ ├─console-kit-daemon.service<br />
│ │ └─16164 /usr/sbin/console-kit-daemon --no-daemon<br />
│ ├─dbus.service<br />
│ │ └─508 /usr/bin/dbus-daemon --system --address=systemd: --nofork --nopidfile --systemd-activation<br />
│ ├─munin-node.service<br />
│ │ └─4290 /usr/bin/perl -wT /usr/sbin/munin-node<br />
│ ├─open-xchange.service<br />
│ │ └─6037 /usr/bin/java -Dsun.net.inetaddr.ttl=3600 -Dnetworkaddress.cache.ttl=3600 -Dnetworkaddress.cache.negative.ttl=10 ...<br />
<br />
To check all the details use<br />
<br />
singlenode:~ # systemctl show system.slice<br />
<br />
==== Limits besides control groups ====<br />
Besides control groups systemd allows you to apply other limits to the execution environment of your service. Here we can apply the limits that would normally be applied via ''limits.conf'' or ''ulimit''. Systemd uses [http://man7.org/linux/man-pages/man2/setrlimit.2.html setrlimit] for this. The options that we set by default are:<br />
<br />
* LimitNOFILE<br />
* LimitNPROC<br />
<br />
which you have already seen when we took a look at the [[#Service_definiton|service definition]] and [[#Drop-in_config|drop-in configs]]. ''Again'': The complete documentation wrt. resource limit configurations in drop-in configs can be found at [https://www.freedesktop.org/software/systemd/man/systemd.exec.html systemd.exec]<br />
<br />
== Verify limits ==<br />
<br />
=== System V ===<br />
<br />
Get the proper pid of the java process started for the middleware by either checking '''ps faux'''<br />
root 4103 0.0 0.0 66528 1868 pts/0 S 22:05 0:00 su -s /bin/bash open-xchange -c /opt/open-xchange/sbin/open-xchange<br />
494 4109 9.5 19.8 2224644 381312 ? Ssl 22:05 1:12 \_ /usr/bin/java -Dsun.net.inetaddr.ttl=3600 -Dnetworkaddress.cache.ttl=3600<br />
<br />
or programmatically<br />
singlenode:~ # pid=$(pgrep -P $(</var/run/open-xchange.pid) java)<br />
<br />
and check the limits applied for that process<br />
singlenode:~ # cat /proc/$pid/limits<br />
Limit Soft Limit Hard Limit Units<br />
Max cpu time unlimited unlimited seconds<br />
Max file size unlimited unlimited bytes<br />
Max data size unlimited unlimited bytes<br />
Max stack size 8388608 unlimited bytes<br />
Max core file size 0 unlimited bytes<br />
Max resident set unlimited unlimited bytes<br />
Max processes 65536 65536 processes<br />
Max open files 65536 65536 files<br />
Max locked memory 65536 65536 bytes<br />
Max address space unlimited unlimited bytes<br />
Max file locks unlimited unlimited locks<br />
Max pending signals 24254 24254 signals<br />
Max msgqueue size 819200 819200 bytes<br />
Max nice priority 0 0<br />
Max realtime priority 0 0<br />
Max realtime timeout unlimited unlimited us<br />
<br />
=== Systemd ===<br />
singlenode:~ # systemctl show open-xchange | grep Limit<br />
StartLimitInterval=10000000<br />
StartLimitBurst=5<br />
StartLimitAction=none<br />
MemoryLimit=18446744073709551615<br />
LimitCPU=18446744073709551615<br />
LimitFSIZE=18446744073709551615<br />
LimitDATA=18446744073709551615<br />
LimitSTACK=18446744073709551615<br />
LimitCORE=18446744073709551615<br />
LimitRSS=18446744073709551615<br />
LimitNOFILE=65536<br />
LimitAS=18446744073709551615<br />
LimitNPROC=65536<br />
LimitMEMLOCK=65536<br />
LimitLOCKS=18446744073709551615<br />
LimitSIGPENDING=19827<br />
LimitMSGQUEUE=819200<br />
LimitNICE=0<br />
LimitRTPRIO=0<br />
LimitRTTIME=18446744073709551615<br />
<br />
== Evaluate JVM commandline options ==<br />
<br />
Before starting Java and the OSGi framework with all the Open-Xchange middleware bundles we have to evaluate several important JVM commandline options that influence the performance and stability of the Open-Xchange middleware stack. These commandline options are specified in '''/opt/open-xchange/etc/ox-scriptconf.sh'''<br />
<br />
=== Format < 7.8.4 ===<br />
For version up to 7.8.3 the JVM configuration options were all specified via a single property '''JAVA_XTRAOPTS''' which resulted in a rather long configuration string that was hard to read and maintain.<br />
<br />
JAVA_XTRAOPTS="-Dsun.net.inetaddr.ttl=3600 -Dnetworkaddress.cache.ttl=3600 -Dnetworkaddress.cache.negative.ttl=10 \<br />
-Dlogback.threadlocal.put.duplicate=false -server -Djava.awt.headless=true -Xmx512M -XX:MaxPermSize=256m -XX:+UseConcMarkSweepGC \<br />
-XX:+UseParNewGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:NewRatio=3 -XX:+UseTLAB \<br />
-Dosgi.compatibility.bootdelegation=false -XX:-OmitStackTraceInFastThrow"<br />
<br />
During the startup of the open-xchange service (step 11 of diagram Service startup using systemd and step 10 of diagram Service startup using upstart) this JAVA_XTRAOPTS string goes through a basic validity check and is then passed to the JVM.<br />
<br />
=== Format >= 7.8.4 ===<br />
Beginning from version 7.8.4 the options are split up and sorted into different categories like:<br />
<br />
JAVA_OPTS_GC="-XX:+UseConcMarkSweepGC -XX:+UseParNewGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:NewRatio=3 -XX:+DisableExplicitGC"<br />
JAVA_OPTS_LOG="-Dlogback.threadlocal.put.duplicate=false -XX:-OmitStackTraceInFastThrow"<br />
JAVA_OPTS_MEM="-XX:MaxHeapSize=512M -XX:MaxPermSize=256M -XX:+UseTLAB"<br />
JAVA_OPTS_NET="-Dsun.net.inetaddr.ttl=3600 -Dnetworkaddress.cache.ttl=3600 -Dnetworkaddress.cache.negative.ttl=10"<br />
JAVA_OPTS_OSGI="-Dosgi.compatibility.bootdelegation=false"<br />
JAVA_OPTS_SERVER="-server -Djava.awt.headless=true"<br />
<br />
JAVA_OPTS_OTHER=""<br />
<br />
#JAVA_OPTS_DEBUG="-XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/mnt/heapdump -Xloggc:/var/log/open-xchange/gc.log -verbose:gc -XX:+PrintGCDateStamps \<br />
-XX:+PrintHeapAtGC -XX:+PrintGCApplicationStoppedTime -XX:+PrintTenuringDistribution"<br />
<br />
which is much easier to read an maintain. Furthermore we added a useful '''JAVA_OPTS_DEBUG''' configuration parameter that comes preconfigured but unused/commented so that customers can quickly enable the most commonly used debugging options in case they have to provide detailed system information to Open-Xchange.<br />
<br />
During the startup of the open-xchange service (step 11 of diagram ''Service startup using systemd'' and step 10 of diagram ''Service startup using upstart'') all uncommented JAVA_OPTS_* categories will then be evaluated and combined again into a single string of commandline options that can then be passed to the JVM.<br />
<br />
=== Upgrade procedure ===<br />
During the upgrade '''7.8.3-revX -> 7.8.4-revY''' existing JAVA_XTRAOPTS will automatically be split up and migrated to the new categories approach. A backup of the file (ox-scriptconf.sh.timestamp) will be created and admins are urged to check for proper migration before deleting the backup file like it's already done with *.dpkg-dist or *.rpmnew files on Debian or RHEL/SLES distributions.<br />
<br />
== Encoding detection during JVM startup ==<br />
<br />
=== Background Infos ===<br />
<br />
From ''man 7 locale''<br />
<br />
DESCRIPTION<br />
A locale is a set of language and cultural rules. These cover aspects such as lan‐<br />
guage for messages, different character sets, lexicographic conventions, and so on.<br />
A program needs to be able to determine its locale and act accordingly to be portable<br />
to different cultures.<br />
<br />
The format to specify a system locale is: '''''language[_territory][.codeset][@modifier]'''''.<br />
<br />
Where all of<br />
<br />
* en_US.UTF-8<br />
* en_US.utf8<br />
* en_US.utf-8<br />
* en_US.UTF_8 <br />
<br />
will result in a valid UTF-8 charmap for the english language with locale categories configured as<br />
<br />
LANG=en_US.UTF-8<br />
LC_CTYPE="en_US.UTF-8"<br />
LC_NUMERIC="en_US.UTF-8"<br />
LC_TIME="en_US.UTF-8"<br />
LC_COLLATE="en_US.UTF-8"<br />
LC_MONETARY="en_US.UTF-8"<br />
LC_MESSAGES="en_US.UTF-8"<br />
LC_PAPER="en_US.UTF-8"<br />
LC_NAME="en_US.UTF-8"<br />
LC_ADDRESS="en_US.UTF-8"<br />
LC_TELEPHONE="en_US.UTF-8"<br />
LC_MEASUREMENT="en_US.UTF-8"<br />
LC_IDENTIFICATION="en_US.UTF-8"<br />
LC_ALL=<br />
<br />
See for more infos: locale(1), locale(7), [https://en.wikipedia.org/wiki/List_of_ISO_639-1_codes language codes], [https://en.wikipedia.org/wiki/ISO_3166-1#Current_codes country codes] and [https://en.wikipedia.org/wiki/Character_encoding character encodings]<br />
<br />
==== JVM startup ====<br />
<br />
The default file.encoding detection is done at JVM startup based upon LC_CTYPE which makes sense as it's documented as:<br />
<br />
CODESET (LC_CTYPE)<br />
Return a string with the name of the character encoding used in the selected locale, such as "UTF-8", "ISO-8859-1",<br />
or "ANSI_X3.4-1968" (better known as US-ASCII). This is the same string that you get with "locale charmap".<br />
For a list of character encoding names, try "locale -m", cf. locale(1).<br />
<br />
Furthermore this influences the defaultCharset selection of the JVM.<br />
<br />
/**<br />
* Returns the default charset of this Java virtual machine.<br />
*<br />
* The default charset is determined during virtual-machine startup and<br />
* typically depends upon the locale and charset of the underlying<br />
* operating system.<br />
*<br />
* @return A charset object for the default charset<br />
*<br />
* @since 1.5<br />
*/<br />
public static Charset defaultCharset() {<br />
if (defaultCharset == null) {<br />
synchronized (Charset.class) {<br />
String csn = AccessController.doPrivileged(<br />
new GetPropertyAction("file.encoding"));<br />
Charset cs = lookup(csn);<br />
if (cs != null)<br />
defaultCharset = cs;<br />
else<br />
defaultCharset = forName("UTF-8");<br />
}<br />
}<br />
return defaultCharset;<br />
}<br />
<br />
Although we try to specify valid unicode encodings for every IO operations the Open-Xchange middleware does some parts of the JVM don't allow to specify the encoding to use and falls back to the detected default encoding. That's why admins have to make sure to provide a proper unicode environment for the middleware service during every start or use documented workarounds provided by Open-Xchange (e.g. com.openexchange.passwordchange.script.base64: Encoded strings as Base64 to circumvent character encoding issues on improperly configured distributions see [https://documentation.open-xchange.com/7.8.3/middleware/configuration/properties.html property documentation])<br />
<br />
See the documentation of your linux distribution on how to configure a proper unicode locale.<br />
<br />
=== Query which file encoding was detected by the JVM during startup ===<br />
[[File:visualvm-file.encoding.png|thumb|Querying the JVM via visualvm|right|300px]]<br />
<br />
The file encoding that was detected by the JVM during startup can be queried via JMX. You can use graphical management tools like [https://visualvm.github.io/ visualvm], query our [[Jolokia]] JMX bridge via http or just use the commandline like<br />
<br />
/opt/open-xchange/sbin/showruntimestats -r | sed -ne 's/.*\({key=file.encoding, value=[^}]*}\).*/\1/p'<br />
{key=file.encoding, value='''UTF-8'''}<br />
<br />
<br clear=all><br />
<br />
=== Locale passing from client to server via SSH ===<br />
<br />
The following setting of sshd allows clients to specify which locale to use on the server and might lead to some confusion or unexpected behaviour when checking the system locale or even to the usage of broken locales during middleware startup if your client sends misconfigured or missing locale configuration to the server.<br />
<br />
grep AcceptEnv /etc/ssh/sshd_config <br />
AcceptEnv LANG LC_*<br />
<br />
For example when i connect to the machine my local german locale is sent to the server <br />
<br />
[vagrant@showcase-community-centos-7 ~]$ locale -v <br />
LANG=de_DE.UTF-8<br />
LC_CTYPE="de_DE.UTF-8"<br />
LC_NUMERIC=de_DE.UTF-8<br />
LC_TIME=de_DE.UTF-8<br />
LC_COLLATE="de_DE.UTF-8"<br />
LC_MONETARY=de_DE.UTF-8<br />
LC_MESSAGES="de_DE.UTF-8"<br />
LC_PAPER=de_DE.UTF-8<br />
LC_NAME=de_DE.UTF-8<br />
LC_ADDRESS=de_DE.UTF-8<br />
LC_TELEPHONE=de_DE.UTF-8<br />
LC_MEASUREMENT=de_DE.UTF-8<br />
LC_IDENTIFICATION=de_DE.UTF-8<br />
LC_ALL=<br />
<br />
But after specifying that i want a new login shell the default locale configured for that user/system will be used instead!<br />
<br />
[root@showcase-community-centos-7 vagrant]# su -s /bin/bash open-xchange -l<br />
-bash-4.2$ locale -v <br />
LANG=en_US.UTF-8<br />
LC_CTYPE="en_US.UTF-8"<br />
LC_NUMERIC="en_US.UTF-8"<br />
LC_TIME="en_US.UTF-8"<br />
LC_COLLATE="en_US.UTF-8"<br />
LC_MONETARY="en_US.UTF-8"<br />
LC_MESSAGES="en_US.UTF-8"<br />
LC_PAPER="en_US.UTF-8"<br />
LC_NAME="en_US.UTF-8"<br />
LC_ADDRESS="en_US.UTF-8"<br />
LC_TELEPHONE="en_US.UTF-8"<br />
LC_MEASUREMENT="en_US.UTF-8"<br />
LC_IDENTIFICATION="en_US.UTF-8"<br />
LC_ALL=<br />
<br />
=== Verify that your system is configured with a proper unicode locale for OX ===<br />
<br />
Assuming you haven't reconfigured the locale of your root user<br />
<br />
i|marens@karotte ~ $ su -<br />
Password: <br />
i|karotte ~ # locale charmap<br />
UTF-8<br />
i|karotte ~ #<br />
<br />
== Warnings during early start ==<br />
<br />
Early warnings before the start of the JVM and related logging libraries like logback and logstash forwarding are redirected to '''/var/log/open-xchange/open-xchange-console.log'''.<br />
<br />
=== Limits ===<br />
If the open-xchange startup script isn't able to set the proper limits on CentOS 6 due to e.g. hard limits being enforced via pam_limits you'll see the following warning in the console log during startup<br />
<br />
singlenode open-xchange # cat open-xchange-console.log<br />
/opt/open-xchange/sbin/open-xchange: line 115: ulimit: max user processes: cannot modify limit: Operation not permitted<br />
<br />
=== Locale ===<br />
If the open-xchange startup script isn't able to detect a unicode locale you'll see a warning like the following (the current charmap might differ)<br />
<br />
WARNING: Couldn't detect proper unicode charmap in locale category LC_CTYPE.<br />
WARNING: Current charmap: "ANSI_X3.4-1968" might cause encoding problems.<br />
<br />
== Summary: Open-Xchange middleware on specific distros ==<br />
The support for the mentioned mechanism of resource control differ depending on the distribution and the init system in use.<br />
<br />
=== Debian 7 ===<br />
;Init<br />
: System V style<br />
;OX Configurable Limits/Defaults<br />
: nofile, nproc<br />
<br />
The mentioned limits can be configured via '''/opt/open-xchange/etc/ox-scriptconf.sh'''. The limits are applied via ulimit in the service's init script. The open-xchange service is finally started via start-stop-daemon which doesn't doesn't consider '''/etc/security/limits.*'''<br />
<br />
=== RHEL 6 / CentOS 6 ===<br />
;Init<br />
: Upstart, System V compatible<br />
;OX Configurable Limits/Defaults<br />
: nofile, nproc<br />
<br />
The mentioned limits can be configured via '''/opt/open-xchange/etc/ox-scriptconf.sh'''. The limits are applied via ulimit in the service's init script. Furthermore as the open-xchange service is finally started via '''su ... open-xchange''' on this distro a user session is opened via su/pam and the default CentOS pam config reads the '''/etc/security/limits.*''' configuration by loading the pam stack like:<br />
<br />
: '''/etc/pam.d/su'''<br />
:: '''-> /etc/pam.d/system-auth'''<br />
::: '''-> pam_limits.so'''<br />
<br />
If NPROC isn't configured for the open-xchange-server it's restricted to '''1024''' globally by default to prevent accidental fork bombs, see '''/etc/security/limits.d/90-nproc.conf''' which can result in severe problems modern multithreaded applications.<br />
<br />
=== RHEL 7 / CentOS 7 / Debian 8 / SLE 12 ===<br />
;Init<br />
: Systemd<br />
;OX Configurable Limits/Defaults<br />
: nofile, nproc<br />
<br />
For systemd the default limits are configured directly in the service's unit file that is shipped by OX and located at '''/usr/lib/systemd/system/open-xchange.service'''. The drop-in config to override or extend the default unit file is located at '''/etc/systemd/system/open-xchange.service.d/limits.conf'''. [https://www.freedesktop.org/software/systemd/man/systemd.exec.html Systemd.exec] shows a whole lot of options that can be used by admins to adapt the default service to their specific needs.<br />
<br />
----<br />
<br />
[[File:diagram_sources.zip]]</div>Marenshttps://oxpedia.org/wiki/index.php?title=AppSuite:MiddlewareStartup&diff=23241AppSuite:MiddlewareStartup2017-04-25T08:59:50Z<p>Marens: /* Limits besides control groups */</p>
<hr />
<div><br />
== Systemd ==<br />
<br />
==== Service definition ====<br />
The default service definition as shipped by Open-Xchange is rather concise and easy to read. You can check by looking at the service file that is installed to either /usr/lib/systemd or /lib/systemd depending on your distribution of choice.<br />
<br />
singlenode:~ # cat /usr/lib/systemd/system/open-xchange.service <br />
[Unit]<br />
After=remote-fs.target<br />
After=time-sync.target ypbind.service sendmail.service cyrus.service <br />
<br />
[Service]<br />
User=open-xchange<br />
PermissionsStartOnly=true<br />
TimeoutStartSec=0<br />
ExecStartPre=/opt/open-xchange/sbin/triggerupdatethemes -u<br />
ExecStart=/opt/open-xchange/sbin/open-xchange<br />
ExecStop=/opt/open-xchange/sbin/shutdown -w<br />
ExecReload=/opt/open-xchange/sbin/triggerreloadconfiguration -d<br />
KillMode=process<br />
LimitNOFILE=65536<br />
LimitNPROC=65536<br />
<br />
[Install]<br />
WantedBy=multi-user.target<br />
<br />
==== Drop-in configs ====<br />
Drop in configs allow administrators to easily adapt and override default service unit files. So if you want to change the default limits or add additional limits have a look at our default example drop-in config which is located at /etc/systemd/system/open-xchange.service.d/limits.conf<br />
<br />
singlenode:~ # cat /etc/systemd/system/open-xchange.service.d/limits.conf <br />
# Override and add options in this file<br />
# See systemd.exec(5) for other limits<br />
<br />
[Service]<br />
#LimitNPROC=65536<br />
#LimitNOFILE=65536<br />
<br />
The [https://www.freedesktop.org/software/systemd/man/systemd.exec.html systemd.exec] documentation shows a whole lot of options that can be used by admins to adapt the default service to their specific needs.<br />
<br />
<br clear=all><br />
[[File:mw_start_sysd.png|thumb|Service startup using systemd|left|1000px]]<br />
<br clear=all><br />
<br />
''Note:'' The technical implementation of systemd differs as service definition and config aren't read every time during service start but the changes need to be reread with ''systemctl daemon-reload''. The above is just a simplyfied version to explain the general concept.<br />
<br />
== System V / Upstart ==<br />
<br />
==== Service definition ====<br />
<br />
The default service definition as shipped by Open-Xchange can be found at /etc/init.d/open-xchange and will differ slightly depending on the distro in use.<br />
<br />
<br clear=all><br />
[[File:mw_start_sysv.png|thumb|Service startup using upstart|left|1000px]]<br />
<br clear=all><br />
== Apply Resource limits ==<br />
<br />
As you can see in the diagrams above one of the differences is that:<br />
<br />
* systemd reads all the infos wrt. resource limitations from the original service file and then applies possible further restrictions from the drop-in configs (steps 2 and 8 in the diagram ''Service startup using systemd'') and limits configured in /opt/open-xchange/etc/ox-scriptconf.sh aren't considered anylonger<br />
* the older System V style init has to source /opt/open-xchange/etc/ox-scriptconf.sh to be able to read the configs for the maximum number of processes (NPROC) and files(NRFILES) that the middleware shall be allowed to create/open (steps 3,4 and 15 in the diagram ''Service startup using upstart'').<br />
<br />
=== Background Infos ===<br />
<br />
Several ways exist to restrict resources on a linux system from a global level down to user/groups or even shells and the processes started by them.<br />
<br />
==== Sysctl ====<br />
Sysctl is used to modify kernel parameters at runtime. E.g. to set the maximum number of files<br />
<br />
$ sysctl -w fs.file-max=100000<br />
<br />
To permanently set them append to the main configuration file and reload the settings<br />
<br />
$ echo fs.file-max=100000 >> /etc/sysctl.conf<br />
$ sysctl -p<br />
<br />
More infos can be found via '''man sysctl'''<br />
<br />
==== Limits.conf ====<br />
<br />
Allows to restrict resources an a global, group or user level. E.g:<br />
<br />
$ cat /etc/security/limits.d/90-nproc.conf <br />
# Default limit for number of user's processes to prevent<br />
# accidental fork bombs.<br />
# See rhbz #432903 for reasoning.<br />
<br />
* soft nproc 1024<br />
<br />
From '''man limits.conf''':<br />
<blockquote><br />
''Also, please note that all limit settings are set per login. They are not global, nor are they permanent; existing only for the duration of the session.''<br />
</blockquote><br />
<br />
The limits per login are applied via the ''pam'' stack. See '''man pam''' and '''man pam_limits''' for more details. As those limits are bound to sessions they don't affect most daemons started by our supported init systems or init utils. Most state that they are ignored by design, see [https://bugs.launchpad.net/ubuntu/+source/upstart/+bug/938669/comments/4 upstart], [https://bugzilla.redhat.com/show_bug.cgi?id=754285#c1 systemd] and [https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=302079 start-stop-daemon] <br />
<br />
===== Ulimit =====<br />
From '''man bash'''<br />
<blockquote><br />
ulimit [-HSTabcdefilmnpqrstuvx [limit]]<br />
Provides control over the resources available to the shell and to processes started by it, on systems that allow such control.<br />
</blockquote><br />
This is what we use in our System V compatible init scripts to increase resources for the open-xchange process across multiple distros. Currently only ''the maximum number of processes'' and ''the maximum number of open file descriptors available to a single user'' are increased via ulimit. The values are specified in '''/opt/open-xchange/ox-scriptconf.sh'''<br />
<br />
==== Systemd ====<br />
<br />
===== Control Groups=====<br />
<br />
Control groups should only affect the OX middleware if you create/manage them yourself of if you are using a ''modern'' distribution that already uses systemd as init.<br />
<br />
Citing from the kernel cgroup [https://www.kernel.org/doc/Documentation/cgroup-v2.txt documentation]:<br />
<br />
<blockquote><br />
1-2. What is cgroup?<br />
<br />
cgroup is a mechanism to organize processes hierarchically and<br />
distribute system resources along the hierarchy in a controlled and<br />
configurable manner.<br />
<br />
cgroup is largely composed of two parts - the core and controllers.<br />
cgroup core is primarily responsible for hierarchically organizing<br />
processes. A cgroup controller is usually responsible for<br />
distributing a specific type of system resource along the hierarchy<br />
although there are utility controllers which serve purposes other than<br />
resource distribution.<br />
<br />
cgroups form a tree structure and every process in the system belongs<br />
to one and only one cgroup. All threads of a process belong to the<br />
same cgroup. On creation, all processes are put in the cgroup that<br />
the parent process belongs to at the time. A process can be migrated<br />
to another cgroup. Migration of a process doesn't affect already<br />
existing descendant processes.<br />
<br />
Following certain structural constraints, controllers may be enabled or<br />
disabled selectively on a cgroup. All controller behaviors are<br />
hierarchical - if a controller is enabled on a cgroup, it affects all<br />
processes which belong to the cgroups consisting the inclusive<br />
sub-hierarchy of the cgroup. When a controller is enabled on a nested<br />
cgroup, it always restricts the resource distribution further. The<br />
restrictions set closer to the root in the hierarchy can not be<br />
overridden from further away.<br />
</blockquote><br />
<br />
So processes are organized into a tree structure of control groups and controllers are responsible for the distribution of resources. So what kind of controllers exist?<br />
<br />
<blockquote><br />
5. Controllers<br />
<br />
5-1. CPU<br />
<br />
The "cpu" controllers regulates distribution of CPU cycles. This<br />
controller implements weight and absolute bandwidth limit models for<br />
normal scheduling policy and absolute bandwidth allocation model for<br />
realtime scheduling policy.<br />
<br />
5-2. Memory<br />
<br />
The "memory" controller regulates distribution of memory.<br />
...<br />
While not completely water-tight, all major memory usages by a given<br />
cgroup are tracked so that the total memory consumption can be<br />
accounted and controlled to a reasonable extent.<br />
<br />
5-3. IO<br />
<br />
The "io" controller regulates the distribution of IO resources. This<br />
controller implements both weight based and absolute bandwidth or IOPS<br />
limit distribution; however, weight based distribution is available<br />
only if cfq-iosched is in use and neither scheme is available for<br />
blk-mq devices.<br />
</blockquote><br />
<br />
The open-xchange service is simply put into the default ''system.slice'' without applying further limits.<br />
<br />
singlenode$ systemd-cgls<br />
├─1 /sbin/init<br />
├─system.slice<br />
│ ├─avahi-daemon.service<br />
│ │ ├─501 avahi-daemon: running [singlenode]<br />
│ │ └─514 avahi-daemon: chroot helper<br />
│ ├─console-kit-daemon.service<br />
│ │ └─16164 /usr/sbin/console-kit-daemon --no-daemon<br />
│ ├─dbus.service<br />
│ │ └─508 /usr/bin/dbus-daemon --system --address=systemd: --nofork --nopidfile --systemd-activation<br />
│ ├─munin-node.service<br />
│ │ └─4290 /usr/bin/perl -wT /usr/sbin/munin-node<br />
│ ├─open-xchange.service<br />
│ │ └─6037 /usr/bin/java -Dsun.net.inetaddr.ttl=3600 -Dnetworkaddress.cache.ttl=3600 -Dnetworkaddress.cache.negative.ttl=10 ...<br />
<br />
To check all the details use<br />
<br />
singlenode:~ # systemctl show system.slice<br />
<br />
==== Limits besides control groups ====<br />
Besides control groups systemd allows you to apply other limits to the execution environment of your service. Here we can apply the limits that would normally be applied via ''limits.conf'' or ''ulimit''. Systemd uses [http://man7.org/linux/man-pages/man2/setrlimit.2.html setrlimit] for this. The options that we set by default are:<br />
<br />
* LimitNOFILE<br />
* LimitNPROC<br />
<br />
which you have already seen when we took a look at the [[#Service_definiton|service definition]] and [[#Drop-in_config|drop-in configs]]. ''Again'': The complete documentation wrt. resource limit configurations in drop-in configs can be found at [https://www.freedesktop.org/software/systemd/man/systemd.exec.html systemd.exec]<br />
<br />
== Verify limits ==<br />
<br />
=== System V ===<br />
<br />
Get the proper pid of the java process started for the middleware by either checking '''ps faux'''<br />
root 4103 0.0 0.0 66528 1868 pts/0 S 22:05 0:00 su -s /bin/bash open-xchange -c /opt/open-xchange/sbin/open-xchange<br />
494 4109 9.5 19.8 2224644 381312 ? Ssl 22:05 1:12 \_ /usr/bin/java -Dsun.net.inetaddr.ttl=3600 -Dnetworkaddress.cache.ttl=3600<br />
<br />
or programmatically<br />
singlenode:~ # pid=$(pgrep -P $(</var/run/open-xchange.pid) java)<br />
<br />
and check the limits applied for that process<br />
singlenode:~ # cat /proc/$pid/limits<br />
Limit Soft Limit Hard Limit Units<br />
Max cpu time unlimited unlimited seconds<br />
Max file size unlimited unlimited bytes<br />
Max data size unlimited unlimited bytes<br />
Max stack size 8388608 unlimited bytes<br />
Max core file size 0 unlimited bytes<br />
Max resident set unlimited unlimited bytes<br />
Max processes 65536 65536 processes<br />
Max open files 65536 65536 files<br />
Max locked memory 65536 65536 bytes<br />
Max address space unlimited unlimited bytes<br />
Max file locks unlimited unlimited locks<br />
Max pending signals 24254 24254 signals<br />
Max msgqueue size 819200 819200 bytes<br />
Max nice priority 0 0<br />
Max realtime priority 0 0<br />
Max realtime timeout unlimited unlimited us<br />
<br />
=== Systemd ===<br />
singlenode:~ # systemctl show open-xchange | grep Limit<br />
StartLimitInterval=10000000<br />
StartLimitBurst=5<br />
StartLimitAction=none<br />
MemoryLimit=18446744073709551615<br />
LimitCPU=18446744073709551615<br />
LimitFSIZE=18446744073709551615<br />
LimitDATA=18446744073709551615<br />
LimitSTACK=18446744073709551615<br />
LimitCORE=18446744073709551615<br />
LimitRSS=18446744073709551615<br />
LimitNOFILE=65536<br />
LimitAS=18446744073709551615<br />
LimitNPROC=65536<br />
LimitMEMLOCK=65536<br />
LimitLOCKS=18446744073709551615<br />
LimitSIGPENDING=19827<br />
LimitMSGQUEUE=819200<br />
LimitNICE=0<br />
LimitRTPRIO=0<br />
LimitRTTIME=18446744073709551615<br />
<br />
== Evaluate JVM commandline options ==<br />
<br />
Before starting Java and the OSGi framework with all the Open-Xchange middleware bundles we have to evaluate several important JVM commandline options that influence the performance and stability of the Open-Xchange middleware stack. These commandline options are specified in '''/opt/open-xchange/etc/ox-scriptconf.sh'''<br />
<br />
=== Format < 7.8.4 ===<br />
For version up to 7.8.3 the JVM configuration options were all specified via a single property '''JAVA_XTRAOPTS''' which resulted in a rather long configuration string that was hard to read and maintain.<br />
<br />
JAVA_XTRAOPTS="-Dsun.net.inetaddr.ttl=3600 -Dnetworkaddress.cache.ttl=3600 -Dnetworkaddress.cache.negative.ttl=10 \<br />
-Dlogback.threadlocal.put.duplicate=false -server -Djava.awt.headless=true -Xmx512M -XX:MaxPermSize=256m -XX:+UseConcMarkSweepGC \<br />
-XX:+UseParNewGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:NewRatio=3 -XX:+UseTLAB \<br />
-Dosgi.compatibility.bootdelegation=false -XX:-OmitStackTraceInFastThrow"<br />
<br />
During the startup of the open-xchange service (step 11 of diagram Service startup using systemd and step 10 of diagram Service startup using upstart) this JAVA_XTRAOPTS string goes through a basic validity check and is then passed to the JVM.<br />
<br />
=== Format >= 7.8.4 ===<br />
Beginning from version 7.8.4 the options are split up and sorted into different categories like:<br />
<br />
JAVA_OPTS_GC="-XX:+UseConcMarkSweepGC -XX:+UseParNewGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:NewRatio=3 -XX:+DisableExplicitGC"<br />
JAVA_OPTS_LOG="-Dlogback.threadlocal.put.duplicate=false -XX:-OmitStackTraceInFastThrow"<br />
JAVA_OPTS_MEM="-XX:MaxHeapSize=512M -XX:MaxPermSize=256M -XX:+UseTLAB"<br />
JAVA_OPTS_NET="-Dsun.net.inetaddr.ttl=3600 -Dnetworkaddress.cache.ttl=3600 -Dnetworkaddress.cache.negative.ttl=10"<br />
JAVA_OPTS_OSGI="-Dosgi.compatibility.bootdelegation=false"<br />
JAVA_OPTS_SERVER="-server -Djava.awt.headless=true"<br />
<br />
JAVA_OPTS_OTHER=""<br />
<br />
#JAVA_OPTS_DEBUG="-XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/mnt/heapdump -Xloggc:/var/log/open-xchange/gc.log -verbose:gc -XX:+PrintGCDateStamps \<br />
-XX:+PrintHeapAtGC -XX:+PrintGCApplicationStoppedTime -XX:+PrintTenuringDistribution"<br />
<br />
which is much easier to read an maintain. Furthermore we added a useful '''JAVA_OPTS_DEBUG''' configuration parameter that comes preconfigured but unused/commented so that customers can quickly enable the most commonly used debugging options in case they have to provide detailed system information to Open-Xchange.<br />
<br />
During the startup of the open-xchange service (step 11 of diagram ''Service startup using systemd'' and step 10 of diagram ''Service startup using upstart'') all uncommented JAVA_OPTS_* categories will then be evaluated and combined again into a single string of commandline options that can then be passed to the JVM.<br />
<br />
=== Upgrade procedure ===<br />
During the upgrade '''7.8.3-revX -> 7.8.4-revY''' existing JAVA_XTRAOPTS will automatically be split up and migrated to the new categories approach. A backup of the file (ox-scriptconf.sh.timestamp) will be created and admins are urged to check for proper migration before deleting the backup file like it's already done with *.dpkg-dist or *.rpmnew files on Debian or RHEL/SLES distributions.<br />
<br />
== Encoding detection during JVM startup ==<br />
<br />
=== Background Infos ===<br />
<br />
From ''man 7 locale''<br />
<br />
DESCRIPTION<br />
A locale is a set of language and cultural rules. These cover aspects such as lan‐<br />
guage for messages, different character sets, lexicographic conventions, and so on.<br />
A program needs to be able to determine its locale and act accordingly to be portable<br />
to different cultures.<br />
<br />
The format to specify a system locale is: '''''language[_territory][.codeset][@modifier]'''''.<br />
<br />
Where all of<br />
<br />
* en_US.UTF-8<br />
* en_US.utf8<br />
* en_US.utf-8<br />
* en_US.UTF_8 <br />
<br />
will result in a valid UTF-8 charmap for the english language with locale categories configured as<br />
<br />
LANG=en_US.UTF-8<br />
LC_CTYPE="en_US.UTF-8"<br />
LC_NUMERIC="en_US.UTF-8"<br />
LC_TIME="en_US.UTF-8"<br />
LC_COLLATE="en_US.UTF-8"<br />
LC_MONETARY="en_US.UTF-8"<br />
LC_MESSAGES="en_US.UTF-8"<br />
LC_PAPER="en_US.UTF-8"<br />
LC_NAME="en_US.UTF-8"<br />
LC_ADDRESS="en_US.UTF-8"<br />
LC_TELEPHONE="en_US.UTF-8"<br />
LC_MEASUREMENT="en_US.UTF-8"<br />
LC_IDENTIFICATION="en_US.UTF-8"<br />
LC_ALL=<br />
<br />
See for more infos: locale(1), locale(7), [https://en.wikipedia.org/wiki/List_of_ISO_639-1_codes language codes], [https://en.wikipedia.org/wiki/ISO_3166-1#Current_codes country codes] and [https://en.wikipedia.org/wiki/Character_encoding character encodings]<br />
<br />
==== JVM startup ====<br />
<br />
The default file.encoding detection is done at JVM startup based upon LC_CTYPE which makes sense as it's documented as:<br />
<br />
CODESET (LC_CTYPE)<br />
Return a string with the name of the character encoding used in the selected locale, such as "UTF-8", "ISO-8859-1",<br />
or "ANSI_X3.4-1968" (better known as US-ASCII). This is the same string that you get with "locale charmap".<br />
For a list of character encoding names, try "locale -m", cf. locale(1).<br />
<br />
Furthermore this influences the defaultCharset selection of the JVM.<br />
<br />
/**<br />
* Returns the default charset of this Java virtual machine.<br />
*<br />
* The default charset is determined during virtual-machine startup and<br />
* typically depends upon the locale and charset of the underlying<br />
* operating system.<br />
*<br />
* @return A charset object for the default charset<br />
*<br />
* @since 1.5<br />
*/<br />
public static Charset defaultCharset() {<br />
if (defaultCharset == null) {<br />
synchronized (Charset.class) {<br />
String csn = AccessController.doPrivileged(<br />
new GetPropertyAction("file.encoding"));<br />
Charset cs = lookup(csn);<br />
if (cs != null)<br />
defaultCharset = cs;<br />
else<br />
defaultCharset = forName("UTF-8");<br />
}<br />
}<br />
return defaultCharset;<br />
}<br />
<br />
Although we try to specify valid unicode encodings for every IO operations the Open-Xchange middleware does some parts of the JVM don't allow to specify the encoding to use and falls back to the detected default encoding. That's why admins have to make sure to provide a proper unicode environment for the middleware service during every start or use documented workarounds provided by Open-Xchange (e.g. com.openexchange.passwordchange.script.base64: Encoded strings as Base64 to circumvent character encoding issues on improperly configured distributions see [https://documentation.open-xchange.com/7.8.3/middleware/configuration/properties.html property documentation])<br />
<br />
See the documentation of your linux distribution on how to configure a proper unicode locale.<br />
<br />
=== Query which file encoding was detected by the JVM during startup ===<br />
[[File:visualvm-file.encoding.png|thumb|Querying the JVM via visualvm|right|300px]]<br />
<br />
The file encoding that was detected by the JVM during startup can be queried via JMX. You can use graphical management tools like [https://visualvm.github.io/ visualvm], query our [[Jolokia]] JMX bridge via http or just use the commandline like<br />
<br />
/opt/open-xchange/sbin/showruntimestats -r | sed -ne 's/.*\({key=file.encoding, value=[^}]*}\).*/\1/p'<br />
{key=file.encoding, value=UTF-8}<br />
<br />
<br clear=all><br />
<br />
=== Locale passing from client to server via SSH ===<br />
<br />
The following setting of sshd allows clients to specify which locale to use on the server and might lead to some confusion or unexpected behaviour when checking the system locale or even to the usage of broken locales during middleware startup if your client sends misconfigured or missing locale configuration to the server.<br />
<br />
grep AcceptEnv /etc/ssh/sshd_config <br />
AcceptEnv LANG LC_*<br />
<br />
For example when i connect to the machine my local german locale is sent to the server <br />
<br />
[vagrant@showcase-community-centos-7 ~]$ locale -v <br />
LANG=de_DE.UTF-8<br />
LC_CTYPE="de_DE.UTF-8"<br />
LC_NUMERIC=de_DE.UTF-8<br />
LC_TIME=de_DE.UTF-8<br />
LC_COLLATE="de_DE.UTF-8"<br />
LC_MONETARY=de_DE.UTF-8<br />
LC_MESSAGES="de_DE.UTF-8"<br />
LC_PAPER=de_DE.UTF-8<br />
LC_NAME=de_DE.UTF-8<br />
LC_ADDRESS=de_DE.UTF-8<br />
LC_TELEPHONE=de_DE.UTF-8<br />
LC_MEASUREMENT=de_DE.UTF-8<br />
LC_IDENTIFICATION=de_DE.UTF-8<br />
LC_ALL=<br />
<br />
But after specifying that i want a new login shell the default locale configured for that user/system will be used instead!<br />
<br />
[root@showcase-community-centos-7 vagrant]# su -s /bin/bash open-xchange -l<br />
-bash-4.2$ locale -v <br />
LANG=en_US.UTF-8<br />
LC_CTYPE="en_US.UTF-8"<br />
LC_NUMERIC="en_US.UTF-8"<br />
LC_TIME="en_US.UTF-8"<br />
LC_COLLATE="en_US.UTF-8"<br />
LC_MONETARY="en_US.UTF-8"<br />
LC_MESSAGES="en_US.UTF-8"<br />
LC_PAPER="en_US.UTF-8"<br />
LC_NAME="en_US.UTF-8"<br />
LC_ADDRESS="en_US.UTF-8"<br />
LC_TELEPHONE="en_US.UTF-8"<br />
LC_MEASUREMENT="en_US.UTF-8"<br />
LC_IDENTIFICATION="en_US.UTF-8"<br />
LC_ALL=<br />
<br />
=== Verify that your system is configured with a proper unicode locale for OX ===<br />
<br />
Assuming you haven't reconfigured the locale of your root user<br />
<br />
i|marens@karotte ~ $ su -<br />
Password: <br />
i|karotte ~ # locale charmap<br />
UTF-8<br />
i|karotte ~ #<br />
<br />
== Warnings during early start ==<br />
<br />
Early warnings before the start of the JVM and related logging libraries like logback and logstash forwarding are redirected to '''/var/log/open-xchange/open-xchange-console.log'''.<br />
<br />
=== Limits ===<br />
If the open-xchange startup script isn't able to set the proper limits on CentOS 6 due to e.g. hard limits being enforced via pam_limits you'll see the following warning in the console log during startup<br />
<br />
singlenode open-xchange # cat open-xchange-console.log<br />
/opt/open-xchange/sbin/open-xchange: line 115: ulimit: max user processes: cannot modify limit: Operation not permitted<br />
<br />
=== Locale ===<br />
If the open-xchange startup script isn't able to detect a unicode locale you'll see a warning like the following (the current charmap might differ)<br />
<br />
WARNING: Couldn't detect proper unicode charmap in locale category LC_CTYPE.<br />
WARNING: Current charmap: "ANSI_X3.4-1968" might cause encoding problems.<br />
<br />
== Summary: Open-Xchange middleware on specific distros ==<br />
The support for the mentioned mechanism of resource control differ depending on the distribution and the init system in use.<br />
<br />
=== Debian 7 ===<br />
;Init<br />
: System V style<br />
;OX Configurable Limits/Defaults<br />
: nofile, nproc<br />
<br />
The mentioned limits can be configured via '''/opt/open-xchange/etc/ox-scriptconf.sh'''. The limits are applied via ulimit in the service's init script. The open-xchange service is finally started via start-stop-daemon which doesn't doesn't consider '''/etc/security/limits.*'''<br />
<br />
=== RHEL 6 / CentOS 6 ===<br />
;Init<br />
: Upstart, System V compatible<br />
;OX Configurable Limits/Defaults<br />
: nofile, nproc<br />
<br />
The mentioned limits can be configured via '''/opt/open-xchange/etc/ox-scriptconf.sh'''. The limits are applied via ulimit in the service's init script. Furthermore as the open-xchange service is finally started via '''su ... open-xchange''' on this distro a user session is opened via su/pam and the default CentOS pam config reads the '''/etc/security/limits.*''' configuration by loading the pam stack like:<br />
<br />
: '''/etc/pam.d/su'''<br />
:: '''-> /etc/pam.d/system-auth'''<br />
::: '''-> pam_limits.so'''<br />
<br />
If NPROC isn't configured for the open-xchange-server it's restricted to '''1024''' globally by default to prevent accidental fork bombs, see '''/etc/security/limits.d/90-nproc.conf''' which can result in severe problems modern multithreaded applications.<br />
<br />
=== RHEL 7 / CentOS 7 / Debian 8 / SLE 12 ===<br />
;Init<br />
: Systemd<br />
;OX Configurable Limits/Defaults<br />
: nofile, nproc<br />
<br />
For systemd the default limits are configured directly in the service's unit file that is shipped by OX and located at '''/usr/lib/systemd/system/open-xchange.service'''. The drop-in config to override or extend the default unit file is located at '''/etc/systemd/system/open-xchange.service.d/limits.conf'''. [https://www.freedesktop.org/software/systemd/man/systemd.exec.html Systemd.exec] shows a whole lot of options that can be used by admins to adapt the default service to their specific needs.<br />
<br />
----<br />
<br />
[[File:diagram_sources.zip]]</div>Marenshttps://oxpedia.org/wiki/index.php?title=AppSuite:MiddlewareStartup&diff=23240AppSuite:MiddlewareStartup2017-04-25T08:57:43Z<p>Marens: /* Drop-in configs */</p>
<hr />
<div><br />
== Systemd ==<br />
<br />
==== Service definition ====<br />
The default service definition as shipped by Open-Xchange is rather concise and easy to read. You can check by looking at the service file that is installed to either /usr/lib/systemd or /lib/systemd depending on your distribution of choice.<br />
<br />
singlenode:~ # cat /usr/lib/systemd/system/open-xchange.service <br />
[Unit]<br />
After=remote-fs.target<br />
After=time-sync.target ypbind.service sendmail.service cyrus.service <br />
<br />
[Service]<br />
User=open-xchange<br />
PermissionsStartOnly=true<br />
TimeoutStartSec=0<br />
ExecStartPre=/opt/open-xchange/sbin/triggerupdatethemes -u<br />
ExecStart=/opt/open-xchange/sbin/open-xchange<br />
ExecStop=/opt/open-xchange/sbin/shutdown -w<br />
ExecReload=/opt/open-xchange/sbin/triggerreloadconfiguration -d<br />
KillMode=process<br />
LimitNOFILE=65536<br />
LimitNPROC=65536<br />
<br />
[Install]<br />
WantedBy=multi-user.target<br />
<br />
==== Drop-in configs ====<br />
Drop in configs allow administrators to easily adapt and override default service unit files. So if you want to change the default limits or add additional limits have a look at our default example drop-in config which is located at /etc/systemd/system/open-xchange.service.d/limits.conf<br />
<br />
singlenode:~ # cat /etc/systemd/system/open-xchange.service.d/limits.conf <br />
# Override and add options in this file<br />
# See systemd.exec(5) for other limits<br />
<br />
[Service]<br />
#LimitNPROC=65536<br />
#LimitNOFILE=65536<br />
<br />
The [https://www.freedesktop.org/software/systemd/man/systemd.exec.html systemd.exec] documentation shows a whole lot of options that can be used by admins to adapt the default service to their specific needs.<br />
<br />
<br clear=all><br />
[[File:mw_start_sysd.png|thumb|Service startup using systemd|left|1000px]]<br />
<br clear=all><br />
<br />
''Note:'' The technical implementation of systemd differs as service definition and config aren't read every time during service start but the changes need to be reread with ''systemctl daemon-reload''. The above is just a simplyfied version to explain the general concept.<br />
<br />
== System V / Upstart ==<br />
<br />
==== Service definition ====<br />
<br />
The default service definition as shipped by Open-Xchange can be found at /etc/init.d/open-xchange and will differ slightly depending on the distro in use.<br />
<br />
<br clear=all><br />
[[File:mw_start_sysv.png|thumb|Service startup using upstart|left|1000px]]<br />
<br clear=all><br />
== Apply Resource limits ==<br />
<br />
As you can see in the diagrams above one of the differences is that:<br />
<br />
* systemd reads all the infos wrt. resource limitations from the original service file and then applies possible further restrictions from the drop-in configs (steps 2 and 8 in the diagram ''Service startup using systemd'') and limits configured in /opt/open-xchange/etc/ox-scriptconf.sh aren't considered anylonger<br />
* the older System V style init has to source /opt/open-xchange/etc/ox-scriptconf.sh to be able to read the configs for the maximum number of processes (NPROC) and files(NRFILES) that the middleware shall be allowed to create/open (steps 3,4 and 15 in the diagram ''Service startup using upstart'').<br />
<br />
=== Background Infos ===<br />
<br />
Several ways exist to restrict resources on a linux system from a global level down to user/groups or even shells and the processes started by them.<br />
<br />
==== Sysctl ====<br />
Sysctl is used to modify kernel parameters at runtime. E.g. to set the maximum number of files<br />
<br />
$ sysctl -w fs.file-max=100000<br />
<br />
To permanently set them append to the main configuration file and reload the settings<br />
<br />
$ echo fs.file-max=100000 >> /etc/sysctl.conf<br />
$ sysctl -p<br />
<br />
More infos can be found via '''man sysctl'''<br />
<br />
==== Limits.conf ====<br />
<br />
Allows to restrict resources an a global, group or user level. E.g:<br />
<br />
$ cat /etc/security/limits.d/90-nproc.conf <br />
# Default limit for number of user's processes to prevent<br />
# accidental fork bombs.<br />
# See rhbz #432903 for reasoning.<br />
<br />
* soft nproc 1024<br />
<br />
From '''man limits.conf''':<br />
<blockquote><br />
''Also, please note that all limit settings are set per login. They are not global, nor are they permanent; existing only for the duration of the session.''<br />
</blockquote><br />
<br />
The limits per login are applied via the ''pam'' stack. See '''man pam''' and '''man pam_limits''' for more details. As those limits are bound to sessions they don't affect most daemons started by our supported init systems or init utils. Most state that they are ignored by design, see [https://bugs.launchpad.net/ubuntu/+source/upstart/+bug/938669/comments/4 upstart], [https://bugzilla.redhat.com/show_bug.cgi?id=754285#c1 systemd] and [https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=302079 start-stop-daemon] <br />
<br />
===== Ulimit =====<br />
From '''man bash'''<br />
<blockquote><br />
ulimit [-HSTabcdefilmnpqrstuvx [limit]]<br />
Provides control over the resources available to the shell and to processes started by it, on systems that allow such control.<br />
</blockquote><br />
This is what we use in our System V compatible init scripts to increase resources for the open-xchange process across multiple distros. Currently only ''the maximum number of processes'' and ''the maximum number of open file descriptors available to a single user'' are increased via ulimit. The values are specified in '''/opt/open-xchange/ox-scriptconf.sh'''<br />
<br />
==== Systemd ====<br />
<br />
===== Control Groups=====<br />
<br />
Control groups should only affect the OX middleware if you create/manage them yourself of if you are using a ''modern'' distribution that already uses systemd as init.<br />
<br />
Citing from the kernel cgroup [https://www.kernel.org/doc/Documentation/cgroup-v2.txt documentation]:<br />
<br />
<blockquote><br />
1-2. What is cgroup?<br />
<br />
cgroup is a mechanism to organize processes hierarchically and<br />
distribute system resources along the hierarchy in a controlled and<br />
configurable manner.<br />
<br />
cgroup is largely composed of two parts - the core and controllers.<br />
cgroup core is primarily responsible for hierarchically organizing<br />
processes. A cgroup controller is usually responsible for<br />
distributing a specific type of system resource along the hierarchy<br />
although there are utility controllers which serve purposes other than<br />
resource distribution.<br />
<br />
cgroups form a tree structure and every process in the system belongs<br />
to one and only one cgroup. All threads of a process belong to the<br />
same cgroup. On creation, all processes are put in the cgroup that<br />
the parent process belongs to at the time. A process can be migrated<br />
to another cgroup. Migration of a process doesn't affect already<br />
existing descendant processes.<br />
<br />
Following certain structural constraints, controllers may be enabled or<br />
disabled selectively on a cgroup. All controller behaviors are<br />
hierarchical - if a controller is enabled on a cgroup, it affects all<br />
processes which belong to the cgroups consisting the inclusive<br />
sub-hierarchy of the cgroup. When a controller is enabled on a nested<br />
cgroup, it always restricts the resource distribution further. The<br />
restrictions set closer to the root in the hierarchy can not be<br />
overridden from further away.<br />
</blockquote><br />
<br />
So processes are organized into a tree structure of control groups and controllers are responsible for the distribution of resources. So what kind of controllers exist?<br />
<br />
<blockquote><br />
5. Controllers<br />
<br />
5-1. CPU<br />
<br />
The "cpu" controllers regulates distribution of CPU cycles. This<br />
controller implements weight and absolute bandwidth limit models for<br />
normal scheduling policy and absolute bandwidth allocation model for<br />
realtime scheduling policy.<br />
<br />
5-2. Memory<br />
<br />
The "memory" controller regulates distribution of memory.<br />
...<br />
While not completely water-tight, all major memory usages by a given<br />
cgroup are tracked so that the total memory consumption can be<br />
accounted and controlled to a reasonable extent.<br />
<br />
5-3. IO<br />
<br />
The "io" controller regulates the distribution of IO resources. This<br />
controller implements both weight based and absolute bandwidth or IOPS<br />
limit distribution; however, weight based distribution is available<br />
only if cfq-iosched is in use and neither scheme is available for<br />
blk-mq devices.<br />
</blockquote><br />
<br />
The open-xchange service is simply put into the default ''system.slice'' without applying further limits.<br />
<br />
singlenode$ systemd-cgls<br />
├─1 /sbin/init<br />
├─system.slice<br />
│ ├─avahi-daemon.service<br />
│ │ ├─501 avahi-daemon: running [singlenode]<br />
│ │ └─514 avahi-daemon: chroot helper<br />
│ ├─console-kit-daemon.service<br />
│ │ └─16164 /usr/sbin/console-kit-daemon --no-daemon<br />
│ ├─dbus.service<br />
│ │ └─508 /usr/bin/dbus-daemon --system --address=systemd: --nofork --nopidfile --systemd-activation<br />
│ ├─munin-node.service<br />
│ │ └─4290 /usr/bin/perl -wT /usr/sbin/munin-node<br />
│ ├─open-xchange.service<br />
│ │ └─6037 /usr/bin/java -Dsun.net.inetaddr.ttl=3600 -Dnetworkaddress.cache.ttl=3600 -Dnetworkaddress.cache.negative.ttl=10 ...<br />
<br />
To check all the details use<br />
<br />
singlenode:~ # systemctl show system.slice<br />
<br />
==== Limits besides control groups ====<br />
Besides control groups systemd allows you to apply other limits to the execution environment of your service. Here we can apply the limits that would normally be applied via ''limits.conf'' or ''ulimit''. Systemd uses [http://man7.org/linux/man-pages/man2/setrlimit.2.html setrlimit] for this. The options that we set by default are:<br />
<br />
* LimitNOFILE<br />
* LimitNPROC<br />
<br />
which you have already seen when we took a look at the [[#Service_definiton|service definition]] and [[#Drop-in_config|drop-in configs]]<br />
<br />
== Verify limits ==<br />
<br />
=== System V ===<br />
<br />
Get the proper pid of the java process started for the middleware by either checking '''ps faux'''<br />
root 4103 0.0 0.0 66528 1868 pts/0 S 22:05 0:00 su -s /bin/bash open-xchange -c /opt/open-xchange/sbin/open-xchange<br />
494 4109 9.5 19.8 2224644 381312 ? Ssl 22:05 1:12 \_ /usr/bin/java -Dsun.net.inetaddr.ttl=3600 -Dnetworkaddress.cache.ttl=3600<br />
<br />
or programmatically<br />
singlenode:~ # pid=$(pgrep -P $(</var/run/open-xchange.pid) java)<br />
<br />
and check the limits applied for that process<br />
singlenode:~ # cat /proc/$pid/limits<br />
Limit Soft Limit Hard Limit Units<br />
Max cpu time unlimited unlimited seconds<br />
Max file size unlimited unlimited bytes<br />
Max data size unlimited unlimited bytes<br />
Max stack size 8388608 unlimited bytes<br />
Max core file size 0 unlimited bytes<br />
Max resident set unlimited unlimited bytes<br />
Max processes 65536 65536 processes<br />
Max open files 65536 65536 files<br />
Max locked memory 65536 65536 bytes<br />
Max address space unlimited unlimited bytes<br />
Max file locks unlimited unlimited locks<br />
Max pending signals 24254 24254 signals<br />
Max msgqueue size 819200 819200 bytes<br />
Max nice priority 0 0<br />
Max realtime priority 0 0<br />
Max realtime timeout unlimited unlimited us<br />
<br />
=== Systemd ===<br />
singlenode:~ # systemctl show open-xchange | grep Limit<br />
StartLimitInterval=10000000<br />
StartLimitBurst=5<br />
StartLimitAction=none<br />
MemoryLimit=18446744073709551615<br />
LimitCPU=18446744073709551615<br />
LimitFSIZE=18446744073709551615<br />
LimitDATA=18446744073709551615<br />
LimitSTACK=18446744073709551615<br />
LimitCORE=18446744073709551615<br />
LimitRSS=18446744073709551615<br />
LimitNOFILE=65536<br />
LimitAS=18446744073709551615<br />
LimitNPROC=65536<br />
LimitMEMLOCK=65536<br />
LimitLOCKS=18446744073709551615<br />
LimitSIGPENDING=19827<br />
LimitMSGQUEUE=819200<br />
LimitNICE=0<br />
LimitRTPRIO=0<br />
LimitRTTIME=18446744073709551615<br />
<br />
== Evaluate JVM commandline options ==<br />
<br />
Before starting Java and the OSGi framework with all the Open-Xchange middleware bundles we have to evaluate several important JVM commandline options that influence the performance and stability of the Open-Xchange middleware stack. These commandline options are specified in '''/opt/open-xchange/etc/ox-scriptconf.sh'''<br />
<br />
=== Format < 7.8.4 ===<br />
For version up to 7.8.3 the JVM configuration options were all specified via a single property '''JAVA_XTRAOPTS''' which resulted in a rather long configuration string that was hard to read and maintain.<br />
<br />
JAVA_XTRAOPTS="-Dsun.net.inetaddr.ttl=3600 -Dnetworkaddress.cache.ttl=3600 -Dnetworkaddress.cache.negative.ttl=10 \<br />
-Dlogback.threadlocal.put.duplicate=false -server -Djava.awt.headless=true -Xmx512M -XX:MaxPermSize=256m -XX:+UseConcMarkSweepGC \<br />
-XX:+UseParNewGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:NewRatio=3 -XX:+UseTLAB \<br />
-Dosgi.compatibility.bootdelegation=false -XX:-OmitStackTraceInFastThrow"<br />
<br />
During the startup of the open-xchange service (step 11 of diagram Service startup using systemd and step 10 of diagram Service startup using upstart) this JAVA_XTRAOPTS string goes through a basic validity check and is then passed to the JVM.<br />
<br />
=== Format >= 7.8.4 ===<br />
Beginning from version 7.8.4 the options are split up and sorted into different categories like:<br />
<br />
JAVA_OPTS_GC="-XX:+UseConcMarkSweepGC -XX:+UseParNewGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:NewRatio=3 -XX:+DisableExplicitGC"<br />
JAVA_OPTS_LOG="-Dlogback.threadlocal.put.duplicate=false -XX:-OmitStackTraceInFastThrow"<br />
JAVA_OPTS_MEM="-XX:MaxHeapSize=512M -XX:MaxPermSize=256M -XX:+UseTLAB"<br />
JAVA_OPTS_NET="-Dsun.net.inetaddr.ttl=3600 -Dnetworkaddress.cache.ttl=3600 -Dnetworkaddress.cache.negative.ttl=10"<br />
JAVA_OPTS_OSGI="-Dosgi.compatibility.bootdelegation=false"<br />
JAVA_OPTS_SERVER="-server -Djava.awt.headless=true"<br />
<br />
JAVA_OPTS_OTHER=""<br />
<br />
#JAVA_OPTS_DEBUG="-XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/mnt/heapdump -Xloggc:/var/log/open-xchange/gc.log -verbose:gc -XX:+PrintGCDateStamps \<br />
-XX:+PrintHeapAtGC -XX:+PrintGCApplicationStoppedTime -XX:+PrintTenuringDistribution"<br />
<br />
which is much easier to read an maintain. Furthermore we added a useful '''JAVA_OPTS_DEBUG''' configuration parameter that comes preconfigured but unused/commented so that customers can quickly enable the most commonly used debugging options in case they have to provide detailed system information to Open-Xchange.<br />
<br />
During the startup of the open-xchange service (step 11 of diagram ''Service startup using systemd'' and step 10 of diagram ''Service startup using upstart'') all uncommented JAVA_OPTS_* categories will then be evaluated and combined again into a single string of commandline options that can then be passed to the JVM.<br />
<br />
=== Upgrade procedure ===<br />
During the upgrade '''7.8.3-revX -> 7.8.4-revY''' existing JAVA_XTRAOPTS will automatically be split up and migrated to the new categories approach. A backup of the file (ox-scriptconf.sh.timestamp) will be created and admins are urged to check for proper migration before deleting the backup file like it's already done with *.dpkg-dist or *.rpmnew files on Debian or RHEL/SLES distributions.<br />
<br />
== Encoding detection during JVM startup ==<br />
<br />
=== Background Infos ===<br />
<br />
From ''man 7 locale''<br />
<br />
DESCRIPTION<br />
A locale is a set of language and cultural rules. These cover aspects such as lan‐<br />
guage for messages, different character sets, lexicographic conventions, and so on.<br />
A program needs to be able to determine its locale and act accordingly to be portable<br />
to different cultures.<br />
<br />
The format to specify a system locale is: '''''language[_territory][.codeset][@modifier]'''''.<br />
<br />
Where all of<br />
<br />
* en_US.UTF-8<br />
* en_US.utf8<br />
* en_US.utf-8<br />
* en_US.UTF_8 <br />
<br />
will result in a valid UTF-8 charmap for the english language with locale categories configured as<br />
<br />
LANG=en_US.UTF-8<br />
LC_CTYPE="en_US.UTF-8"<br />
LC_NUMERIC="en_US.UTF-8"<br />
LC_TIME="en_US.UTF-8"<br />
LC_COLLATE="en_US.UTF-8"<br />
LC_MONETARY="en_US.UTF-8"<br />
LC_MESSAGES="en_US.UTF-8"<br />
LC_PAPER="en_US.UTF-8"<br />
LC_NAME="en_US.UTF-8"<br />
LC_ADDRESS="en_US.UTF-8"<br />
LC_TELEPHONE="en_US.UTF-8"<br />
LC_MEASUREMENT="en_US.UTF-8"<br />
LC_IDENTIFICATION="en_US.UTF-8"<br />
LC_ALL=<br />
<br />
See for more infos: locale(1), locale(7), [https://en.wikipedia.org/wiki/List_of_ISO_639-1_codes language codes], [https://en.wikipedia.org/wiki/ISO_3166-1#Current_codes country codes] and [https://en.wikipedia.org/wiki/Character_encoding character encodings]<br />
<br />
==== JVM startup ====<br />
<br />
The default file.encoding detection is done at JVM startup based upon LC_CTYPE which makes sense as it's documented as:<br />
<br />
CODESET (LC_CTYPE)<br />
Return a string with the name of the character encoding used in the selected locale, such as "UTF-8", "ISO-8859-1",<br />
or "ANSI_X3.4-1968" (better known as US-ASCII). This is the same string that you get with "locale charmap".<br />
For a list of character encoding names, try "locale -m", cf. locale(1).<br />
<br />
Furthermore this influences the defaultCharset selection of the JVM.<br />
<br />
/**<br />
* Returns the default charset of this Java virtual machine.<br />
*<br />
* The default charset is determined during virtual-machine startup and<br />
* typically depends upon the locale and charset of the underlying<br />
* operating system.<br />
*<br />
* @return A charset object for the default charset<br />
*<br />
* @since 1.5<br />
*/<br />
public static Charset defaultCharset() {<br />
if (defaultCharset == null) {<br />
synchronized (Charset.class) {<br />
String csn = AccessController.doPrivileged(<br />
new GetPropertyAction("file.encoding"));<br />
Charset cs = lookup(csn);<br />
if (cs != null)<br />
defaultCharset = cs;<br />
else<br />
defaultCharset = forName("UTF-8");<br />
}<br />
}<br />
return defaultCharset;<br />
}<br />
<br />
Although we try to specify valid unicode encodings for every IO operations the Open-Xchange middleware does some parts of the JVM don't allow to specify the encoding to use and falls back to the detected default encoding. That's why admins have to make sure to provide a proper unicode environment for the middleware service during every start or use documented workarounds provided by Open-Xchange (e.g. com.openexchange.passwordchange.script.base64: Encoded strings as Base64 to circumvent character encoding issues on improperly configured distributions see [https://documentation.open-xchange.com/7.8.3/middleware/configuration/properties.html property documentation])<br />
<br />
See the documentation of your linux distribution on how to configure a proper unicode locale.<br />
<br />
=== Query which file encoding was detected by the JVM during startup ===<br />
[[File:visualvm-file.encoding.png|thumb|Querying the JVM via visualvm|right|300px]]<br />
<br />
The file encoding that was detected by the JVM during startup can be queried via JMX. You can use graphical management tools like [https://visualvm.github.io/ visualvm], query our [[Jolokia]] JMX bridge via http or just use the commandline like<br />
<br />
/opt/open-xchange/sbin/showruntimestats -r | sed -ne 's/.*\({key=file.encoding, value=[^}]*}\).*/\1/p'<br />
{key=file.encoding, value=UTF-8}<br />
<br />
<br clear=all><br />
<br />
=== Locale passing from client to server via SSH ===<br />
<br />
The following setting of sshd allows clients to specify which locale to use on the server and might lead to some confusion or unexpected behaviour when checking the system locale or even to the usage of broken locales during middleware startup if your client sends misconfigured or missing locale configuration to the server.<br />
<br />
grep AcceptEnv /etc/ssh/sshd_config <br />
AcceptEnv LANG LC_*<br />
<br />
For example when i connect to the machine my local german locale is sent to the server <br />
<br />
[vagrant@showcase-community-centos-7 ~]$ locale -v <br />
LANG=de_DE.UTF-8<br />
LC_CTYPE="de_DE.UTF-8"<br />
LC_NUMERIC=de_DE.UTF-8<br />
LC_TIME=de_DE.UTF-8<br />
LC_COLLATE="de_DE.UTF-8"<br />
LC_MONETARY=de_DE.UTF-8<br />
LC_MESSAGES="de_DE.UTF-8"<br />
LC_PAPER=de_DE.UTF-8<br />
LC_NAME=de_DE.UTF-8<br />
LC_ADDRESS=de_DE.UTF-8<br />
LC_TELEPHONE=de_DE.UTF-8<br />
LC_MEASUREMENT=de_DE.UTF-8<br />
LC_IDENTIFICATION=de_DE.UTF-8<br />
LC_ALL=<br />
<br />
But after specifying that i want a new login shell the default locale configured for that user/system will be used instead!<br />
<br />
[root@showcase-community-centos-7 vagrant]# su -s /bin/bash open-xchange -l<br />
-bash-4.2$ locale -v <br />
LANG=en_US.UTF-8<br />
LC_CTYPE="en_US.UTF-8"<br />
LC_NUMERIC="en_US.UTF-8"<br />
LC_TIME="en_US.UTF-8"<br />
LC_COLLATE="en_US.UTF-8"<br />
LC_MONETARY="en_US.UTF-8"<br />
LC_MESSAGES="en_US.UTF-8"<br />
LC_PAPER="en_US.UTF-8"<br />
LC_NAME="en_US.UTF-8"<br />
LC_ADDRESS="en_US.UTF-8"<br />
LC_TELEPHONE="en_US.UTF-8"<br />
LC_MEASUREMENT="en_US.UTF-8"<br />
LC_IDENTIFICATION="en_US.UTF-8"<br />
LC_ALL=<br />
<br />
=== Verify that your system is configured with a proper unicode locale for OX ===<br />
<br />
Assuming you haven't reconfigured the locale of your root user<br />
<br />
i|marens@karotte ~ $ su -<br />
Password: <br />
i|karotte ~ # locale charmap<br />
UTF-8<br />
i|karotte ~ #<br />
<br />
== Warnings during early start ==<br />
<br />
Early warnings before the start of the JVM and related logging libraries like logback and logstash forwarding are redirected to '''/var/log/open-xchange/open-xchange-console.log'''.<br />
<br />
=== Limits ===<br />
If the open-xchange startup script isn't able to set the proper limits on CentOS 6 due to e.g. hard limits being enforced via pam_limits you'll see the following warning in the console log during startup<br />
<br />
singlenode open-xchange # cat open-xchange-console.log<br />
/opt/open-xchange/sbin/open-xchange: line 115: ulimit: max user processes: cannot modify limit: Operation not permitted<br />
<br />
=== Locale ===<br />
If the open-xchange startup script isn't able to detect a unicode locale you'll see a warning like the following (the current charmap might differ)<br />
<br />
WARNING: Couldn't detect proper unicode charmap in locale category LC_CTYPE.<br />
WARNING: Current charmap: "ANSI_X3.4-1968" might cause encoding problems.<br />
<br />
== Summary: Open-Xchange middleware on specific distros ==<br />
The support for the mentioned mechanism of resource control differ depending on the distribution and the init system in use.<br />
<br />
=== Debian 7 ===<br />
;Init<br />
: System V style<br />
;OX Configurable Limits/Defaults<br />
: nofile, nproc<br />
<br />
The mentioned limits can be configured via '''/opt/open-xchange/etc/ox-scriptconf.sh'''. The limits are applied via ulimit in the service's init script. The open-xchange service is finally started via start-stop-daemon which doesn't doesn't consider '''/etc/security/limits.*'''<br />
<br />
=== RHEL 6 / CentOS 6 ===<br />
;Init<br />
: Upstart, System V compatible<br />
;OX Configurable Limits/Defaults<br />
: nofile, nproc<br />
<br />
The mentioned limits can be configured via '''/opt/open-xchange/etc/ox-scriptconf.sh'''. The limits are applied via ulimit in the service's init script. Furthermore as the open-xchange service is finally started via '''su ... open-xchange''' on this distro a user session is opened via su/pam and the default CentOS pam config reads the '''/etc/security/limits.*''' configuration by loading the pam stack like:<br />
<br />
: '''/etc/pam.d/su'''<br />
:: '''-> /etc/pam.d/system-auth'''<br />
::: '''-> pam_limits.so'''<br />
<br />
If NPROC isn't configured for the open-xchange-server it's restricted to '''1024''' globally by default to prevent accidental fork bombs, see '''/etc/security/limits.d/90-nproc.conf''' which can result in severe problems modern multithreaded applications.<br />
<br />
=== RHEL 7 / CentOS 7 / Debian 8 / SLE 12 ===<br />
;Init<br />
: Systemd<br />
;OX Configurable Limits/Defaults<br />
: nofile, nproc<br />
<br />
For systemd the default limits are configured directly in the service's unit file that is shipped by OX and located at '''/usr/lib/systemd/system/open-xchange.service'''. The drop-in config to override or extend the default unit file is located at '''/etc/systemd/system/open-xchange.service.d/limits.conf'''. [https://www.freedesktop.org/software/systemd/man/systemd.exec.html Systemd.exec] shows a whole lot of options that can be used by admins to adapt the default service to their specific needs.<br />
<br />
----<br />
<br />
[[File:diagram_sources.zip]]</div>Marenshttps://oxpedia.org/wiki/index.php?title=AppSuite:MiddlewareStartup&diff=23239AppSuite:MiddlewareStartup2017-04-25T08:51:36Z<p>Marens: /* Drop-in configs */</p>
<hr />
<div><br />
== Systemd ==<br />
<br />
==== Service definition ====<br />
The default service definition as shipped by Open-Xchange is rather concise and easy to read. You can check by looking at the service file that is installed to either /usr/lib/systemd or /lib/systemd depending on your distribution of choice.<br />
<br />
singlenode:~ # cat /usr/lib/systemd/system/open-xchange.service <br />
[Unit]<br />
After=remote-fs.target<br />
After=time-sync.target ypbind.service sendmail.service cyrus.service <br />
<br />
[Service]<br />
User=open-xchange<br />
PermissionsStartOnly=true<br />
TimeoutStartSec=0<br />
ExecStartPre=/opt/open-xchange/sbin/triggerupdatethemes -u<br />
ExecStart=/opt/open-xchange/sbin/open-xchange<br />
ExecStop=/opt/open-xchange/sbin/shutdown -w<br />
ExecReload=/opt/open-xchange/sbin/triggerreloadconfiguration -d<br />
KillMode=process<br />
LimitNOFILE=65536<br />
LimitNPROC=65536<br />
<br />
[Install]<br />
WantedBy=multi-user.target<br />
<br />
==== Drop-in configs ====<br />
Drop in configs allow administrators to easily adapt and override default service unit files. So if you want to change the default limits or add additional limits have a look at our default example drop-in config which is located at /etc/systemd/system/open-xchange.service.d/limits.conf<br />
<br />
singlenode:~ # cat /etc/systemd/system/open-xchange.service.d/limits.conf <br />
# Override and add options in this file<br />
# See systemd.exec(5) for other limits<br />
<br />
[Service]<br />
#LimitNPROC=65536<br />
#LimitNOFILE=65536<br />
<br />
The systemd.exec documentation shows a whole lot of options that can be used by admins to adapt the default service to their specific needs.<br />
<br />
<br clear=all><br />
[[File:mw_start_sysd.png|thumb|Service startup using systemd|left|1000px]]<br />
<br clear=all><br />
<br />
''Note:'' The technical implementation of systemd differs as service definition and config aren't read every time during service start but the changes need to be reread with ''systemctl daemon-reload''. The above is just a simplyfied version to explain the general concept.<br />
<br />
== System V / Upstart ==<br />
<br />
==== Service definition ====<br />
<br />
The default service definition as shipped by Open-Xchange can be found at /etc/init.d/open-xchange and will differ slightly depending on the distro in use.<br />
<br />
<br clear=all><br />
[[File:mw_start_sysv.png|thumb|Service startup using upstart|left|1000px]]<br />
<br clear=all><br />
== Apply Resource limits ==<br />
<br />
As you can see in the diagrams above one of the differences is that:<br />
<br />
* systemd reads all the infos wrt. resource limitations from the original service file and then applies possible further restrictions from the drop-in configs (steps 2 and 8 in the diagram ''Service startup using systemd'') and limits configured in /opt/open-xchange/etc/ox-scriptconf.sh aren't considered anylonger<br />
* the older System V style init has to source /opt/open-xchange/etc/ox-scriptconf.sh to be able to read the configs for the maximum number of processes (NPROC) and files(NRFILES) that the middleware shall be allowed to create/open (steps 3,4 and 15 in the diagram ''Service startup using upstart'').<br />
<br />
=== Background Infos ===<br />
<br />
Several ways exist to restrict resources on a linux system from a global level down to user/groups or even shells and the processes started by them.<br />
<br />
==== Sysctl ====<br />
Sysctl is used to modify kernel parameters at runtime. E.g. to set the maximum number of files<br />
<br />
$ sysctl -w fs.file-max=100000<br />
<br />
To permanently set them append to the main configuration file and reload the settings<br />
<br />
$ echo fs.file-max=100000 >> /etc/sysctl.conf<br />
$ sysctl -p<br />
<br />
More infos can be found via '''man sysctl'''<br />
<br />
==== Limits.conf ====<br />
<br />
Allows to restrict resources an a global, group or user level. E.g:<br />
<br />
$ cat /etc/security/limits.d/90-nproc.conf <br />
# Default limit for number of user's processes to prevent<br />
# accidental fork bombs.<br />
# See rhbz #432903 for reasoning.<br />
<br />
* soft nproc 1024<br />
<br />
From '''man limits.conf''':<br />
<blockquote><br />
''Also, please note that all limit settings are set per login. They are not global, nor are they permanent; existing only for the duration of the session.''<br />
</blockquote><br />
<br />
The limits per login are applied via the ''pam'' stack. See '''man pam''' and '''man pam_limits''' for more details. As those limits are bound to sessions they don't affect most daemons started by our supported init systems or init utils. Most state that they are ignored by design, see [https://bugs.launchpad.net/ubuntu/+source/upstart/+bug/938669/comments/4 upstart], [https://bugzilla.redhat.com/show_bug.cgi?id=754285#c1 systemd] and [https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=302079 start-stop-daemon] <br />
<br />
===== Ulimit =====<br />
From '''man bash'''<br />
<blockquote><br />
ulimit [-HSTabcdefilmnpqrstuvx [limit]]<br />
Provides control over the resources available to the shell and to processes started by it, on systems that allow such control.<br />
</blockquote><br />
This is what we use in our System V compatible init scripts to increase resources for the open-xchange process across multiple distros. Currently only ''the maximum number of processes'' and ''the maximum number of open file descriptors available to a single user'' are increased via ulimit. The values are specified in '''/opt/open-xchange/ox-scriptconf.sh'''<br />
<br />
==== Systemd ====<br />
<br />
===== Control Groups=====<br />
<br />
Control groups should only affect the OX middleware if you create/manage them yourself of if you are using a ''modern'' distribution that already uses systemd as init.<br />
<br />
Citing from the kernel cgroup [https://www.kernel.org/doc/Documentation/cgroup-v2.txt documentation]:<br />
<br />
<blockquote><br />
1-2. What is cgroup?<br />
<br />
cgroup is a mechanism to organize processes hierarchically and<br />
distribute system resources along the hierarchy in a controlled and<br />
configurable manner.<br />
<br />
cgroup is largely composed of two parts - the core and controllers.<br />
cgroup core is primarily responsible for hierarchically organizing<br />
processes. A cgroup controller is usually responsible for<br />
distributing a specific type of system resource along the hierarchy<br />
although there are utility controllers which serve purposes other than<br />
resource distribution.<br />
<br />
cgroups form a tree structure and every process in the system belongs<br />
to one and only one cgroup. All threads of a process belong to the<br />
same cgroup. On creation, all processes are put in the cgroup that<br />
the parent process belongs to at the time. A process can be migrated<br />
to another cgroup. Migration of a process doesn't affect already<br />
existing descendant processes.<br />
<br />
Following certain structural constraints, controllers may be enabled or<br />
disabled selectively on a cgroup. All controller behaviors are<br />
hierarchical - if a controller is enabled on a cgroup, it affects all<br />
processes which belong to the cgroups consisting the inclusive<br />
sub-hierarchy of the cgroup. When a controller is enabled on a nested<br />
cgroup, it always restricts the resource distribution further. The<br />
restrictions set closer to the root in the hierarchy can not be<br />
overridden from further away.<br />
</blockquote><br />
<br />
So processes are organized into a tree structure of control groups and controllers are responsible for the distribution of resources. So what kind of controllers exist?<br />
<br />
<blockquote><br />
5. Controllers<br />
<br />
5-1. CPU<br />
<br />
The "cpu" controllers regulates distribution of CPU cycles. This<br />
controller implements weight and absolute bandwidth limit models for<br />
normal scheduling policy and absolute bandwidth allocation model for<br />
realtime scheduling policy.<br />
<br />
5-2. Memory<br />
<br />
The "memory" controller regulates distribution of memory.<br />
...<br />
While not completely water-tight, all major memory usages by a given<br />
cgroup are tracked so that the total memory consumption can be<br />
accounted and controlled to a reasonable extent.<br />
<br />
5-3. IO<br />
<br />
The "io" controller regulates the distribution of IO resources. This<br />
controller implements both weight based and absolute bandwidth or IOPS<br />
limit distribution; however, weight based distribution is available<br />
only if cfq-iosched is in use and neither scheme is available for<br />
blk-mq devices.<br />
</blockquote><br />
<br />
The open-xchange service is simply put into the default ''system.slice'' without applying further limits.<br />
<br />
singlenode$ systemd-cgls<br />
├─1 /sbin/init<br />
├─system.slice<br />
│ ├─avahi-daemon.service<br />
│ │ ├─501 avahi-daemon: running [singlenode]<br />
│ │ └─514 avahi-daemon: chroot helper<br />
│ ├─console-kit-daemon.service<br />
│ │ └─16164 /usr/sbin/console-kit-daemon --no-daemon<br />
│ ├─dbus.service<br />
│ │ └─508 /usr/bin/dbus-daemon --system --address=systemd: --nofork --nopidfile --systemd-activation<br />
│ ├─munin-node.service<br />
│ │ └─4290 /usr/bin/perl -wT /usr/sbin/munin-node<br />
│ ├─open-xchange.service<br />
│ │ └─6037 /usr/bin/java -Dsun.net.inetaddr.ttl=3600 -Dnetworkaddress.cache.ttl=3600 -Dnetworkaddress.cache.negative.ttl=10 ...<br />
<br />
To check all the details use<br />
<br />
singlenode:~ # systemctl show system.slice<br />
<br />
==== Limits besides control groups ====<br />
Besides control groups systemd allows you to apply other limits to the execution environment of your service. Here we can apply the limits that would normally be applied via ''limits.conf'' or ''ulimit''. Systemd uses [http://man7.org/linux/man-pages/man2/setrlimit.2.html setrlimit] for this. The options that we set by default are:<br />
<br />
* LimitNOFILE<br />
* LimitNPROC<br />
<br />
which you have already seen when we took a look at the [[#Service_definiton|service definition]] and [[#Drop-in_config|drop-in configs]]<br />
<br />
== Verify limits ==<br />
<br />
=== System V ===<br />
<br />
Get the proper pid of the java process started for the middleware by either checking '''ps faux'''<br />
root 4103 0.0 0.0 66528 1868 pts/0 S 22:05 0:00 su -s /bin/bash open-xchange -c /opt/open-xchange/sbin/open-xchange<br />
494 4109 9.5 19.8 2224644 381312 ? Ssl 22:05 1:12 \_ /usr/bin/java -Dsun.net.inetaddr.ttl=3600 -Dnetworkaddress.cache.ttl=3600<br />
<br />
or programmatically<br />
singlenode:~ # pid=$(pgrep -P $(</var/run/open-xchange.pid) java)<br />
<br />
and check the limits applied for that process<br />
singlenode:~ # cat /proc/$pid/limits<br />
Limit Soft Limit Hard Limit Units<br />
Max cpu time unlimited unlimited seconds<br />
Max file size unlimited unlimited bytes<br />
Max data size unlimited unlimited bytes<br />
Max stack size 8388608 unlimited bytes<br />
Max core file size 0 unlimited bytes<br />
Max resident set unlimited unlimited bytes<br />
Max processes 65536 65536 processes<br />
Max open files 65536 65536 files<br />
Max locked memory 65536 65536 bytes<br />
Max address space unlimited unlimited bytes<br />
Max file locks unlimited unlimited locks<br />
Max pending signals 24254 24254 signals<br />
Max msgqueue size 819200 819200 bytes<br />
Max nice priority 0 0<br />
Max realtime priority 0 0<br />
Max realtime timeout unlimited unlimited us<br />
<br />
=== Systemd ===<br />
singlenode:~ # systemctl show open-xchange | grep Limit<br />
StartLimitInterval=10000000<br />
StartLimitBurst=5<br />
StartLimitAction=none<br />
MemoryLimit=18446744073709551615<br />
LimitCPU=18446744073709551615<br />
LimitFSIZE=18446744073709551615<br />
LimitDATA=18446744073709551615<br />
LimitSTACK=18446744073709551615<br />
LimitCORE=18446744073709551615<br />
LimitRSS=18446744073709551615<br />
LimitNOFILE=65536<br />
LimitAS=18446744073709551615<br />
LimitNPROC=65536<br />
LimitMEMLOCK=65536<br />
LimitLOCKS=18446744073709551615<br />
LimitSIGPENDING=19827<br />
LimitMSGQUEUE=819200<br />
LimitNICE=0<br />
LimitRTPRIO=0<br />
LimitRTTIME=18446744073709551615<br />
<br />
== Evaluate JVM commandline options ==<br />
<br />
Before starting Java and the OSGi framework with all the Open-Xchange middleware bundles we have to evaluate several important JVM commandline options that influence the performance and stability of the Open-Xchange middleware stack. These commandline options are specified in '''/opt/open-xchange/etc/ox-scriptconf.sh'''<br />
<br />
=== Format < 7.8.4 ===<br />
For version up to 7.8.3 the JVM configuration options were all specified via a single property '''JAVA_XTRAOPTS''' which resulted in a rather long configuration string that was hard to read and maintain.<br />
<br />
JAVA_XTRAOPTS="-Dsun.net.inetaddr.ttl=3600 -Dnetworkaddress.cache.ttl=3600 -Dnetworkaddress.cache.negative.ttl=10 \<br />
-Dlogback.threadlocal.put.duplicate=false -server -Djava.awt.headless=true -Xmx512M -XX:MaxPermSize=256m -XX:+UseConcMarkSweepGC \<br />
-XX:+UseParNewGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:NewRatio=3 -XX:+UseTLAB \<br />
-Dosgi.compatibility.bootdelegation=false -XX:-OmitStackTraceInFastThrow"<br />
<br />
During the startup of the open-xchange service (step 11 of diagram Service startup using systemd and step 10 of diagram Service startup using upstart) this JAVA_XTRAOPTS string goes through a basic validity check and is then passed to the JVM.<br />
<br />
=== Format >= 7.8.4 ===<br />
Beginning from version 7.8.4 the options are split up and sorted into different categories like:<br />
<br />
JAVA_OPTS_GC="-XX:+UseConcMarkSweepGC -XX:+UseParNewGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:NewRatio=3 -XX:+DisableExplicitGC"<br />
JAVA_OPTS_LOG="-Dlogback.threadlocal.put.duplicate=false -XX:-OmitStackTraceInFastThrow"<br />
JAVA_OPTS_MEM="-XX:MaxHeapSize=512M -XX:MaxPermSize=256M -XX:+UseTLAB"<br />
JAVA_OPTS_NET="-Dsun.net.inetaddr.ttl=3600 -Dnetworkaddress.cache.ttl=3600 -Dnetworkaddress.cache.negative.ttl=10"<br />
JAVA_OPTS_OSGI="-Dosgi.compatibility.bootdelegation=false"<br />
JAVA_OPTS_SERVER="-server -Djava.awt.headless=true"<br />
<br />
JAVA_OPTS_OTHER=""<br />
<br />
#JAVA_OPTS_DEBUG="-XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/mnt/heapdump -Xloggc:/var/log/open-xchange/gc.log -verbose:gc -XX:+PrintGCDateStamps \<br />
-XX:+PrintHeapAtGC -XX:+PrintGCApplicationStoppedTime -XX:+PrintTenuringDistribution"<br />
<br />
which is much easier to read an maintain. Furthermore we added a useful '''JAVA_OPTS_DEBUG''' configuration parameter that comes preconfigured but unused/commented so that customers can quickly enable the most commonly used debugging options in case they have to provide detailed system information to Open-Xchange.<br />
<br />
During the startup of the open-xchange service (step 11 of diagram ''Service startup using systemd'' and step 10 of diagram ''Service startup using upstart'') all uncommented JAVA_OPTS_* categories will then be evaluated and combined again into a single string of commandline options that can then be passed to the JVM.<br />
<br />
=== Upgrade procedure ===<br />
During the upgrade '''7.8.3-revX -> 7.8.4-revY''' existing JAVA_XTRAOPTS will automatically be split up and migrated to the new categories approach. A backup of the file (ox-scriptconf.sh.timestamp) will be created and admins are urged to check for proper migration before deleting the backup file like it's already done with *.dpkg-dist or *.rpmnew files on Debian or RHEL/SLES distributions.<br />
<br />
== Encoding detection during JVM startup ==<br />
<br />
=== Background Infos ===<br />
<br />
From ''man 7 locale''<br />
<br />
DESCRIPTION<br />
A locale is a set of language and cultural rules. These cover aspects such as lan‐<br />
guage for messages, different character sets, lexicographic conventions, and so on.<br />
A program needs to be able to determine its locale and act accordingly to be portable<br />
to different cultures.<br />
<br />
The format to specify a system locale is: '''''language[_territory][.codeset][@modifier]'''''.<br />
<br />
Where all of<br />
<br />
* en_US.UTF-8<br />
* en_US.utf8<br />
* en_US.utf-8<br />
* en_US.UTF_8 <br />
<br />
will result in a valid UTF-8 charmap for the english language with locale categories configured as<br />
<br />
LANG=en_US.UTF-8<br />
LC_CTYPE="en_US.UTF-8"<br />
LC_NUMERIC="en_US.UTF-8"<br />
LC_TIME="en_US.UTF-8"<br />
LC_COLLATE="en_US.UTF-8"<br />
LC_MONETARY="en_US.UTF-8"<br />
LC_MESSAGES="en_US.UTF-8"<br />
LC_PAPER="en_US.UTF-8"<br />
LC_NAME="en_US.UTF-8"<br />
LC_ADDRESS="en_US.UTF-8"<br />
LC_TELEPHONE="en_US.UTF-8"<br />
LC_MEASUREMENT="en_US.UTF-8"<br />
LC_IDENTIFICATION="en_US.UTF-8"<br />
LC_ALL=<br />
<br />
See for more infos: locale(1), locale(7), [https://en.wikipedia.org/wiki/List_of_ISO_639-1_codes language codes], [https://en.wikipedia.org/wiki/ISO_3166-1#Current_codes country codes] and [https://en.wikipedia.org/wiki/Character_encoding character encodings]<br />
<br />
==== JVM startup ====<br />
<br />
The default file.encoding detection is done at JVM startup based upon LC_CTYPE which makes sense as it's documented as:<br />
<br />
CODESET (LC_CTYPE)<br />
Return a string with the name of the character encoding used in the selected locale, such as "UTF-8", "ISO-8859-1",<br />
or "ANSI_X3.4-1968" (better known as US-ASCII). This is the same string that you get with "locale charmap".<br />
For a list of character encoding names, try "locale -m", cf. locale(1).<br />
<br />
Furthermore this influences the defaultCharset selection of the JVM.<br />
<br />
/**<br />
* Returns the default charset of this Java virtual machine.<br />
*<br />
* The default charset is determined during virtual-machine startup and<br />
* typically depends upon the locale and charset of the underlying<br />
* operating system.<br />
*<br />
* @return A charset object for the default charset<br />
*<br />
* @since 1.5<br />
*/<br />
public static Charset defaultCharset() {<br />
if (defaultCharset == null) {<br />
synchronized (Charset.class) {<br />
String csn = AccessController.doPrivileged(<br />
new GetPropertyAction("file.encoding"));<br />
Charset cs = lookup(csn);<br />
if (cs != null)<br />
defaultCharset = cs;<br />
else<br />
defaultCharset = forName("UTF-8");<br />
}<br />
}<br />
return defaultCharset;<br />
}<br />
<br />
Although we try to specify valid unicode encodings for every IO operations the Open-Xchange middleware does some parts of the JVM don't allow to specify the encoding to use and falls back to the detected default encoding. That's why admins have to make sure to provide a proper unicode environment for the middleware service during every start or use documented workarounds provided by Open-Xchange (e.g. com.openexchange.passwordchange.script.base64: Encoded strings as Base64 to circumvent character encoding issues on improperly configured distributions see [https://documentation.open-xchange.com/7.8.3/middleware/configuration/properties.html property documentation])<br />
<br />
See the documentation of your linux distribution on how to configure a proper unicode locale.<br />
<br />
=== Query which file encoding was detected by the JVM during startup ===<br />
[[File:visualvm-file.encoding.png|thumb|Querying the JVM via visualvm|right|300px]]<br />
<br />
The file encoding that was detected by the JVM during startup can be queried via JMX. You can use graphical management tools like [https://visualvm.github.io/ visualvm], query our [[Jolokia]] JMX bridge via http or just use the commandline like<br />
<br />
/opt/open-xchange/sbin/showruntimestats -r | sed -ne 's/.*\({key=file.encoding, value=[^}]*}\).*/\1/p'<br />
{key=file.encoding, value=UTF-8}<br />
<br />
<br clear=all><br />
<br />
=== Locale passing from client to server via SSH ===<br />
<br />
The following setting of sshd allows clients to specify which locale to use on the server and might lead to some confusion or unexpected behaviour when checking the system locale or even to the usage of broken locales during middleware startup if your client sends misconfigured or missing locale configuration to the server.<br />
<br />
grep AcceptEnv /etc/ssh/sshd_config <br />
AcceptEnv LANG LC_*<br />
<br />
For example when i connect to the machine my local german locale is sent to the server <br />
<br />
[vagrant@showcase-community-centos-7 ~]$ locale -v <br />
LANG=de_DE.UTF-8<br />
LC_CTYPE="de_DE.UTF-8"<br />
LC_NUMERIC=de_DE.UTF-8<br />
LC_TIME=de_DE.UTF-8<br />
LC_COLLATE="de_DE.UTF-8"<br />
LC_MONETARY=de_DE.UTF-8<br />
LC_MESSAGES="de_DE.UTF-8"<br />
LC_PAPER=de_DE.UTF-8<br />
LC_NAME=de_DE.UTF-8<br />
LC_ADDRESS=de_DE.UTF-8<br />
LC_TELEPHONE=de_DE.UTF-8<br />
LC_MEASUREMENT=de_DE.UTF-8<br />
LC_IDENTIFICATION=de_DE.UTF-8<br />
LC_ALL=<br />
<br />
But after specifying that i want a new login shell the default locale configured for that user/system will be used instead!<br />
<br />
[root@showcase-community-centos-7 vagrant]# su -s /bin/bash open-xchange -l<br />
-bash-4.2$ locale -v <br />
LANG=en_US.UTF-8<br />
LC_CTYPE="en_US.UTF-8"<br />
LC_NUMERIC="en_US.UTF-8"<br />
LC_TIME="en_US.UTF-8"<br />
LC_COLLATE="en_US.UTF-8"<br />
LC_MONETARY="en_US.UTF-8"<br />
LC_MESSAGES="en_US.UTF-8"<br />
LC_PAPER="en_US.UTF-8"<br />
LC_NAME="en_US.UTF-8"<br />
LC_ADDRESS="en_US.UTF-8"<br />
LC_TELEPHONE="en_US.UTF-8"<br />
LC_MEASUREMENT="en_US.UTF-8"<br />
LC_IDENTIFICATION="en_US.UTF-8"<br />
LC_ALL=<br />
<br />
=== Verify that your system is configured with a proper unicode locale for OX ===<br />
<br />
Assuming you haven't reconfigured the locale of your root user<br />
<br />
i|marens@karotte ~ $ su -<br />
Password: <br />
i|karotte ~ # locale charmap<br />
UTF-8<br />
i|karotte ~ #<br />
<br />
== Warnings during early start ==<br />
<br />
Early warnings before the start of the JVM and related logging libraries like logback and logstash forwarding are redirected to '''/var/log/open-xchange/open-xchange-console.log'''.<br />
<br />
=== Limits ===<br />
If the open-xchange startup script isn't able to set the proper limits on CentOS 6 due to e.g. hard limits being enforced via pam_limits you'll see the following warning in the console log during startup<br />
<br />
singlenode open-xchange # cat open-xchange-console.log<br />
/opt/open-xchange/sbin/open-xchange: line 115: ulimit: max user processes: cannot modify limit: Operation not permitted<br />
<br />
=== Locale ===<br />
If the open-xchange startup script isn't able to detect a unicode locale you'll see a warning like the following (the current charmap might differ)<br />
<br />
WARNING: Couldn't detect proper unicode charmap in locale category LC_CTYPE.<br />
WARNING: Current charmap: "ANSI_X3.4-1968" might cause encoding problems.<br />
<br />
== Summary: Open-Xchange middleware on specific distros ==<br />
The support for the mentioned mechanism of resource control differ depending on the distribution and the init system in use.<br />
<br />
=== Debian 7 ===<br />
;Init<br />
: System V style<br />
;OX Configurable Limits/Defaults<br />
: nofile, nproc<br />
<br />
The mentioned limits can be configured via '''/opt/open-xchange/etc/ox-scriptconf.sh'''. The limits are applied via ulimit in the service's init script. The open-xchange service is finally started via start-stop-daemon which doesn't doesn't consider '''/etc/security/limits.*'''<br />
<br />
=== RHEL 6 / CentOS 6 ===<br />
;Init<br />
: Upstart, System V compatible<br />
;OX Configurable Limits/Defaults<br />
: nofile, nproc<br />
<br />
The mentioned limits can be configured via '''/opt/open-xchange/etc/ox-scriptconf.sh'''. The limits are applied via ulimit in the service's init script. Furthermore as the open-xchange service is finally started via '''su ... open-xchange''' on this distro a user session is opened via su/pam and the default CentOS pam config reads the '''/etc/security/limits.*''' configuration by loading the pam stack like:<br />
<br />
: '''/etc/pam.d/su'''<br />
:: '''-> /etc/pam.d/system-auth'''<br />
::: '''-> pam_limits.so'''<br />
<br />
If NPROC isn't configured for the open-xchange-server it's restricted to '''1024''' globally by default to prevent accidental fork bombs, see '''/etc/security/limits.d/90-nproc.conf''' which can result in severe problems modern multithreaded applications.<br />
<br />
=== RHEL 7 / CentOS 7 / Debian 8 / SLE 12 ===<br />
;Init<br />
: Systemd<br />
;OX Configurable Limits/Defaults<br />
: nofile, nproc<br />
<br />
For systemd the default limits are configured directly in the service's unit file that is shipped by OX and located at '''/usr/lib/systemd/system/open-xchange.service'''. The drop-in config to override or extend the default unit file is located at '''/etc/systemd/system/open-xchange.service.d/limits.conf'''. [https://www.freedesktop.org/software/systemd/man/systemd.exec.html Systemd.exec] shows a whole lot of options that can be used by admins to adapt the default service to their specific needs.<br />
<br />
----<br />
<br />
[[File:diagram_sources.zip]]</div>Marenshttps://oxpedia.org/wiki/index.php?title=AppSuite:MiddlewareStartup&diff=23238AppSuite:MiddlewareStartup2017-04-24T21:46:51Z<p>Marens: /* JVM startup */</p>
<hr />
<div><br />
== Systemd ==<br />
<br />
==== Service definition ====<br />
The default service definition as shipped by Open-Xchange is rather concise and easy to read. You can check by looking at the service file that is installed to either /usr/lib/systemd or /lib/systemd depending on your distribution of choice.<br />
<br />
singlenode:~ # cat /usr/lib/systemd/system/open-xchange.service <br />
[Unit]<br />
After=remote-fs.target<br />
After=time-sync.target ypbind.service sendmail.service cyrus.service <br />
<br />
[Service]<br />
User=open-xchange<br />
PermissionsStartOnly=true<br />
TimeoutStartSec=0<br />
ExecStartPre=/opt/open-xchange/sbin/triggerupdatethemes -u<br />
ExecStart=/opt/open-xchange/sbin/open-xchange<br />
ExecStop=/opt/open-xchange/sbin/shutdown -w<br />
ExecReload=/opt/open-xchange/sbin/triggerreloadconfiguration -d<br />
KillMode=process<br />
LimitNOFILE=65536<br />
LimitNPROC=65536<br />
<br />
[Install]<br />
WantedBy=multi-user.target<br />
<br />
==== Drop-in configs ====<br />
Drop in configs allow administrators to easily adapt and override default service unit files. So if you want to change the default limits or add additional limits have a look at our default example drop-in config which is located at /etc/systemd/system/open-xchange.service.d/limits.conf<br />
<br />
singlenode:~ # cat /etc/systemd/system/open-xchange.service.d/limits.conf <br />
# Override and add options in this file<br />
# See systemd.exec(5) for other limits<br />
<br />
[Service]<br />
#LimitNPROC=65536<br />
#LimitNOFILE=65536<br />
<br />
The systemd.exec documentation shows a whole lot of options that can be used by admins to adapt the default service to their specific needs.<br />
<br />
<br clear=all><br />
[[File:mw_start_sysd.png|thumb|Service startup using systemd|left|1000px]]<br />
<br clear=all><br />
<br />
== System V / Upstart ==<br />
<br />
==== Service definition ====<br />
<br />
The default service definition as shipped by Open-Xchange can be found at /etc/init.d/open-xchange and will differ slightly depending on the distro in use.<br />
<br />
<br clear=all><br />
[[File:mw_start_sysv.png|thumb|Service startup using upstart|left|1000px]]<br />
<br clear=all><br />
== Apply Resource limits ==<br />
<br />
As you can see in the diagrams above one of the differences is that:<br />
<br />
* systemd reads all the infos wrt. resource limitations from the original service file and then applies possible further restrictions from the drop-in configs (steps 2 and 8 in the diagram ''Service startup using systemd'') and limits configured in /opt/open-xchange/etc/ox-scriptconf.sh aren't considered anylonger<br />
* the older System V style init has to source /opt/open-xchange/etc/ox-scriptconf.sh to be able to read the configs for the maximum number of processes (NPROC) and files(NRFILES) that the middleware shall be allowed to create/open (steps 3,4 and 15 in the diagram ''Service startup using upstart'').<br />
<br />
=== Background Infos ===<br />
<br />
Several ways exist to restrict resources on a linux system from a global level down to user/groups or even shells and the processes started by them.<br />
<br />
==== Sysctl ====<br />
Sysctl is used to modify kernel parameters at runtime. E.g. to set the maximum number of files<br />
<br />
$ sysctl -w fs.file-max=100000<br />
<br />
To permanently set them append to the main configuration file and reload the settings<br />
<br />
$ echo fs.file-max=100000 >> /etc/sysctl.conf<br />
$ sysctl -p<br />
<br />
More infos can be found via '''man sysctl'''<br />
<br />
==== Limits.conf ====<br />
<br />
Allows to restrict resources an a global, group or user level. E.g:<br />
<br />
$ cat /etc/security/limits.d/90-nproc.conf <br />
# Default limit for number of user's processes to prevent<br />
# accidental fork bombs.<br />
# See rhbz #432903 for reasoning.<br />
<br />
* soft nproc 1024<br />
<br />
From '''man limits.conf''':<br />
<blockquote><br />
''Also, please note that all limit settings are set per login. They are not global, nor are they permanent; existing only for the duration of the session.''<br />
</blockquote><br />
<br />
The limits per login are applied via the ''pam'' stack. See '''man pam''' and '''man pam_limits''' for more details. As those limits are bound to sessions they don't affect most daemons started by our supported init systems or init utils. Most state that they are ignored by design, see [https://bugs.launchpad.net/ubuntu/+source/upstart/+bug/938669/comments/4 upstart], [https://bugzilla.redhat.com/show_bug.cgi?id=754285#c1 systemd] and [https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=302079 start-stop-daemon] <br />
<br />
===== Ulimit =====<br />
From '''man bash'''<br />
<blockquote><br />
ulimit [-HSTabcdefilmnpqrstuvx [limit]]<br />
Provides control over the resources available to the shell and to processes started by it, on systems that allow such control.<br />
</blockquote><br />
This is what we use in our System V compatible init scripts to increase resources for the open-xchange process across multiple distros. Currently only ''the maximum number of processes'' and ''the maximum number of open file descriptors available to a single user'' are increased via ulimit. The values are specified in '''/opt/open-xchange/ox-scriptconf.sh'''<br />
<br />
==== Systemd ====<br />
<br />
===== Control Groups=====<br />
<br />
Control groups should only affect the OX middleware if you create/manage them yourself of if you are using a ''modern'' distribution that already uses systemd as init.<br />
<br />
Citing from the kernel cgroup [https://www.kernel.org/doc/Documentation/cgroup-v2.txt documentation]:<br />
<br />
<blockquote><br />
1-2. What is cgroup?<br />
<br />
cgroup is a mechanism to organize processes hierarchically and<br />
distribute system resources along the hierarchy in a controlled and<br />
configurable manner.<br />
<br />
cgroup is largely composed of two parts - the core and controllers.<br />
cgroup core is primarily responsible for hierarchically organizing<br />
processes. A cgroup controller is usually responsible for<br />
distributing a specific type of system resource along the hierarchy<br />
although there are utility controllers which serve purposes other than<br />
resource distribution.<br />
<br />
cgroups form a tree structure and every process in the system belongs<br />
to one and only one cgroup. All threads of a process belong to the<br />
same cgroup. On creation, all processes are put in the cgroup that<br />
the parent process belongs to at the time. A process can be migrated<br />
to another cgroup. Migration of a process doesn't affect already<br />
existing descendant processes.<br />
<br />
Following certain structural constraints, controllers may be enabled or<br />
disabled selectively on a cgroup. All controller behaviors are<br />
hierarchical - if a controller is enabled on a cgroup, it affects all<br />
processes which belong to the cgroups consisting the inclusive<br />
sub-hierarchy of the cgroup. When a controller is enabled on a nested<br />
cgroup, it always restricts the resource distribution further. The<br />
restrictions set closer to the root in the hierarchy can not be<br />
overridden from further away.<br />
</blockquote><br />
<br />
So processes are organized into a tree structure of control groups and controllers are responsible for the distribution of resources. So what kind of controllers exist?<br />
<br />
<blockquote><br />
5. Controllers<br />
<br />
5-1. CPU<br />
<br />
The "cpu" controllers regulates distribution of CPU cycles. This<br />
controller implements weight and absolute bandwidth limit models for<br />
normal scheduling policy and absolute bandwidth allocation model for<br />
realtime scheduling policy.<br />
<br />
5-2. Memory<br />
<br />
The "memory" controller regulates distribution of memory.<br />
...<br />
While not completely water-tight, all major memory usages by a given<br />
cgroup are tracked so that the total memory consumption can be<br />
accounted and controlled to a reasonable extent.<br />
<br />
5-3. IO<br />
<br />
The "io" controller regulates the distribution of IO resources. This<br />
controller implements both weight based and absolute bandwidth or IOPS<br />
limit distribution; however, weight based distribution is available<br />
only if cfq-iosched is in use and neither scheme is available for<br />
blk-mq devices.<br />
</blockquote><br />
<br />
The open-xchange service is simply put into the default ''system.slice'' without applying further limits.<br />
<br />
singlenode$ systemd-cgls<br />
├─1 /sbin/init<br />
├─system.slice<br />
│ ├─avahi-daemon.service<br />
│ │ ├─501 avahi-daemon: running [singlenode]<br />
│ │ └─514 avahi-daemon: chroot helper<br />
│ ├─console-kit-daemon.service<br />
│ │ └─16164 /usr/sbin/console-kit-daemon --no-daemon<br />
│ ├─dbus.service<br />
│ │ └─508 /usr/bin/dbus-daemon --system --address=systemd: --nofork --nopidfile --systemd-activation<br />
│ ├─munin-node.service<br />
│ │ └─4290 /usr/bin/perl -wT /usr/sbin/munin-node<br />
│ ├─open-xchange.service<br />
│ │ └─6037 /usr/bin/java -Dsun.net.inetaddr.ttl=3600 -Dnetworkaddress.cache.ttl=3600 -Dnetworkaddress.cache.negative.ttl=10 ...<br />
<br />
To check all the details use<br />
<br />
singlenode:~ # systemctl show system.slice<br />
<br />
==== Limits besides control groups ====<br />
Besides control groups systemd allows you to apply other limits to the execution environment of your service. Here we can apply the limits that would normally be applied via ''limits.conf'' or ''ulimit''. Systemd uses [http://man7.org/linux/man-pages/man2/setrlimit.2.html setrlimit] for this. The options that we set by default are:<br />
<br />
* LimitNOFILE<br />
* LimitNPROC<br />
<br />
which you have already seen when we took a look at the [[#Service_definiton|service definition]] and [[#Drop-in_config|drop-in configs]]<br />
<br />
== Verify limits ==<br />
<br />
=== System V ===<br />
<br />
Get the proper pid of the java process started for the middleware by either checking '''ps faux'''<br />
root 4103 0.0 0.0 66528 1868 pts/0 S 22:05 0:00 su -s /bin/bash open-xchange -c /opt/open-xchange/sbin/open-xchange<br />
494 4109 9.5 19.8 2224644 381312 ? Ssl 22:05 1:12 \_ /usr/bin/java -Dsun.net.inetaddr.ttl=3600 -Dnetworkaddress.cache.ttl=3600<br />
<br />
or programmatically<br />
singlenode:~ # pid=$(pgrep -P $(</var/run/open-xchange.pid) java)<br />
<br />
and check the limits applied for that process<br />
singlenode:~ # cat /proc/$pid/limits<br />
Limit Soft Limit Hard Limit Units<br />
Max cpu time unlimited unlimited seconds<br />
Max file size unlimited unlimited bytes<br />
Max data size unlimited unlimited bytes<br />
Max stack size 8388608 unlimited bytes<br />
Max core file size 0 unlimited bytes<br />
Max resident set unlimited unlimited bytes<br />
Max processes 65536 65536 processes<br />
Max open files 65536 65536 files<br />
Max locked memory 65536 65536 bytes<br />
Max address space unlimited unlimited bytes<br />
Max file locks unlimited unlimited locks<br />
Max pending signals 24254 24254 signals<br />
Max msgqueue size 819200 819200 bytes<br />
Max nice priority 0 0<br />
Max realtime priority 0 0<br />
Max realtime timeout unlimited unlimited us<br />
<br />
=== Systemd ===<br />
singlenode:~ # systemctl show open-xchange | grep Limit<br />
StartLimitInterval=10000000<br />
StartLimitBurst=5<br />
StartLimitAction=none<br />
MemoryLimit=18446744073709551615<br />
LimitCPU=18446744073709551615<br />
LimitFSIZE=18446744073709551615<br />
LimitDATA=18446744073709551615<br />
LimitSTACK=18446744073709551615<br />
LimitCORE=18446744073709551615<br />
LimitRSS=18446744073709551615<br />
LimitNOFILE=65536<br />
LimitAS=18446744073709551615<br />
LimitNPROC=65536<br />
LimitMEMLOCK=65536<br />
LimitLOCKS=18446744073709551615<br />
LimitSIGPENDING=19827<br />
LimitMSGQUEUE=819200<br />
LimitNICE=0<br />
LimitRTPRIO=0<br />
LimitRTTIME=18446744073709551615<br />
<br />
== Evaluate JVM commandline options ==<br />
<br />
Before starting Java and the OSGi framework with all the Open-Xchange middleware bundles we have to evaluate several important JVM commandline options that influence the performance and stability of the Open-Xchange middleware stack. These commandline options are specified in '''/opt/open-xchange/etc/ox-scriptconf.sh'''<br />
<br />
=== Format < 7.8.4 ===<br />
For version up to 7.8.3 the JVM configuration options were all specified via a single property '''JAVA_XTRAOPTS''' which resulted in a rather long configuration string that was hard to read and maintain.<br />
<br />
JAVA_XTRAOPTS="-Dsun.net.inetaddr.ttl=3600 -Dnetworkaddress.cache.ttl=3600 -Dnetworkaddress.cache.negative.ttl=10 \<br />
-Dlogback.threadlocal.put.duplicate=false -server -Djava.awt.headless=true -Xmx512M -XX:MaxPermSize=256m -XX:+UseConcMarkSweepGC \<br />
-XX:+UseParNewGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:NewRatio=3 -XX:+UseTLAB \<br />
-Dosgi.compatibility.bootdelegation=false -XX:-OmitStackTraceInFastThrow"<br />
<br />
During the startup of the open-xchange service (step 11 of diagram Service startup using systemd and step 10 of diagram Service startup using upstart) this JAVA_XTRAOPTS string goes through a basic validity check and is then passed to the JVM.<br />
<br />
=== Format >= 7.8.4 ===<br />
Beginning from version 7.8.4 the options are split up and sorted into different categories like:<br />
<br />
JAVA_OPTS_GC="-XX:+UseConcMarkSweepGC -XX:+UseParNewGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:NewRatio=3 -XX:+DisableExplicitGC"<br />
JAVA_OPTS_LOG="-Dlogback.threadlocal.put.duplicate=false -XX:-OmitStackTraceInFastThrow"<br />
JAVA_OPTS_MEM="-XX:MaxHeapSize=512M -XX:MaxPermSize=256M -XX:+UseTLAB"<br />
JAVA_OPTS_NET="-Dsun.net.inetaddr.ttl=3600 -Dnetworkaddress.cache.ttl=3600 -Dnetworkaddress.cache.negative.ttl=10"<br />
JAVA_OPTS_OSGI="-Dosgi.compatibility.bootdelegation=false"<br />
JAVA_OPTS_SERVER="-server -Djava.awt.headless=true"<br />
<br />
JAVA_OPTS_OTHER=""<br />
<br />
#JAVA_OPTS_DEBUG="-XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/mnt/heapdump -Xloggc:/var/log/open-xchange/gc.log -verbose:gc -XX:+PrintGCDateStamps \<br />
-XX:+PrintHeapAtGC -XX:+PrintGCApplicationStoppedTime -XX:+PrintTenuringDistribution"<br />
<br />
which is much easier to read an maintain. Furthermore we added a useful '''JAVA_OPTS_DEBUG''' configuration parameter that comes preconfigured but unused/commented so that customers can quickly enable the most commonly used debugging options in case they have to provide detailed system information to Open-Xchange.<br />
<br />
During the startup of the open-xchange service (step 11 of diagram ''Service startup using systemd'' and step 10 of diagram ''Service startup using upstart'') all uncommented JAVA_OPTS_* categories will then be evaluated and combined again into a single string of commandline options that can then be passed to the JVM.<br />
<br />
=== Upgrade procedure ===<br />
During the upgrade '''7.8.3-revX -> 7.8.4-revY''' existing JAVA_XTRAOPTS will automatically be split up and migrated to the new categories approach. A backup of the file (ox-scriptconf.sh.timestamp) will be created and admins are urged to check for proper migration before deleting the backup file like it's already done with *.dpkg-dist or *.rpmnew files on Debian or RHEL/SLES distributions.<br />
<br />
== Encoding detection during JVM startup ==<br />
<br />
=== Background Infos ===<br />
<br />
From ''man 7 locale''<br />
<br />
DESCRIPTION<br />
A locale is a set of language and cultural rules. These cover aspects such as lan‐<br />
guage for messages, different character sets, lexicographic conventions, and so on.<br />
A program needs to be able to determine its locale and act accordingly to be portable<br />
to different cultures.<br />
<br />
The format to specify a system locale is: '''''language[_territory][.codeset][@modifier]'''''.<br />
<br />
Where all of<br />
<br />
* en_US.UTF-8<br />
* en_US.utf8<br />
* en_US.utf-8<br />
* en_US.UTF_8 <br />
<br />
will result in a valid UTF-8 charmap for the english language with locale categories configured as<br />
<br />
LANG=en_US.UTF-8<br />
LC_CTYPE="en_US.UTF-8"<br />
LC_NUMERIC="en_US.UTF-8"<br />
LC_TIME="en_US.UTF-8"<br />
LC_COLLATE="en_US.UTF-8"<br />
LC_MONETARY="en_US.UTF-8"<br />
LC_MESSAGES="en_US.UTF-8"<br />
LC_PAPER="en_US.UTF-8"<br />
LC_NAME="en_US.UTF-8"<br />
LC_ADDRESS="en_US.UTF-8"<br />
LC_TELEPHONE="en_US.UTF-8"<br />
LC_MEASUREMENT="en_US.UTF-8"<br />
LC_IDENTIFICATION="en_US.UTF-8"<br />
LC_ALL=<br />
<br />
See for more infos: locale(1), locale(7), [https://en.wikipedia.org/wiki/List_of_ISO_639-1_codes language codes], [https://en.wikipedia.org/wiki/ISO_3166-1#Current_codes country codes] and [https://en.wikipedia.org/wiki/Character_encoding character encodings]<br />
<br />
==== JVM startup ====<br />
<br />
The default file.encoding detection is done at JVM startup based upon LC_CTYPE which makes sense as it's documented as:<br />
<br />
CODESET (LC_CTYPE)<br />
Return a string with the name of the character encoding used in the selected locale, such as "UTF-8", "ISO-8859-1",<br />
or "ANSI_X3.4-1968" (better known as US-ASCII). This is the same string that you get with "locale charmap".<br />
For a list of character encoding names, try "locale -m", cf. locale(1).<br />
<br />
Furthermore this influences the defaultCharset selection of the JVM.<br />
<br />
/**<br />
* Returns the default charset of this Java virtual machine.<br />
*<br />
* The default charset is determined during virtual-machine startup and<br />
* typically depends upon the locale and charset of the underlying<br />
* operating system.<br />
*<br />
* @return A charset object for the default charset<br />
*<br />
* @since 1.5<br />
*/<br />
public static Charset defaultCharset() {<br />
if (defaultCharset == null) {<br />
synchronized (Charset.class) {<br />
String csn = AccessController.doPrivileged(<br />
new GetPropertyAction("file.encoding"));<br />
Charset cs = lookup(csn);<br />
if (cs != null)<br />
defaultCharset = cs;<br />
else<br />
defaultCharset = forName("UTF-8");<br />
}<br />
}<br />
return defaultCharset;<br />
}<br />
<br />
Although we try to specify valid unicode encodings for every IO operations the Open-Xchange middleware does some parts of the JVM don't allow to specify the encoding to use and falls back to the detected default encoding. That's why admins have to make sure to provide a proper unicode environment for the middleware service during every start or use documented workarounds provided by Open-Xchange (e.g. com.openexchange.passwordchange.script.base64: Encoded strings as Base64 to circumvent character encoding issues on improperly configured distributions see [https://documentation.open-xchange.com/7.8.3/middleware/configuration/properties.html property documentation])<br />
<br />
See the documentation of your linux distribution on how to configure a proper unicode locale.<br />
<br />
=== Query which file encoding was detected by the JVM during startup ===<br />
[[File:visualvm-file.encoding.png|thumb|Querying the JVM via visualvm|right|300px]]<br />
<br />
The file encoding that was detected by the JVM during startup can be queried via JMX. You can use graphical management tools like [https://visualvm.github.io/ visualvm], query our [[Jolokia]] JMX bridge via http or just use the commandline like<br />
<br />
/opt/open-xchange/sbin/showruntimestats -r | sed -ne 's/.*\({key=file.encoding, value=[^}]*}\).*/\1/p'<br />
{key=file.encoding, value=UTF-8}<br />
<br />
<br clear=all><br />
<br />
=== Locale passing from client to server via SSH ===<br />
<br />
The following setting of sshd allows clients to specify which locale to use on the server and might lead to some confusion or unexpected behaviour when checking the system locale or even to the usage of broken locales during middleware startup if your client sends misconfigured or missing locale configuration to the server.<br />
<br />
grep AcceptEnv /etc/ssh/sshd_config <br />
AcceptEnv LANG LC_*<br />
<br />
For example when i connect to the machine my local german locale is sent to the server <br />
<br />
[vagrant@showcase-community-centos-7 ~]$ locale -v <br />
LANG=de_DE.UTF-8<br />
LC_CTYPE="de_DE.UTF-8"<br />
LC_NUMERIC=de_DE.UTF-8<br />
LC_TIME=de_DE.UTF-8<br />
LC_COLLATE="de_DE.UTF-8"<br />
LC_MONETARY=de_DE.UTF-8<br />
LC_MESSAGES="de_DE.UTF-8"<br />
LC_PAPER=de_DE.UTF-8<br />
LC_NAME=de_DE.UTF-8<br />
LC_ADDRESS=de_DE.UTF-8<br />
LC_TELEPHONE=de_DE.UTF-8<br />
LC_MEASUREMENT=de_DE.UTF-8<br />
LC_IDENTIFICATION=de_DE.UTF-8<br />
LC_ALL=<br />
<br />
But after specifying that i want a new login shell the default locale configured for that user/system will be used instead!<br />
<br />
[root@showcase-community-centos-7 vagrant]# su -s /bin/bash open-xchange -l<br />
-bash-4.2$ locale -v <br />
LANG=en_US.UTF-8<br />
LC_CTYPE="en_US.UTF-8"<br />
LC_NUMERIC="en_US.UTF-8"<br />
LC_TIME="en_US.UTF-8"<br />
LC_COLLATE="en_US.UTF-8"<br />
LC_MONETARY="en_US.UTF-8"<br />
LC_MESSAGES="en_US.UTF-8"<br />
LC_PAPER="en_US.UTF-8"<br />
LC_NAME="en_US.UTF-8"<br />
LC_ADDRESS="en_US.UTF-8"<br />
LC_TELEPHONE="en_US.UTF-8"<br />
LC_MEASUREMENT="en_US.UTF-8"<br />
LC_IDENTIFICATION="en_US.UTF-8"<br />
LC_ALL=<br />
<br />
=== Verify that your system is configured with a proper unicode locale for OX ===<br />
<br />
Assuming you haven't reconfigured the locale of your root user<br />
<br />
i|marens@karotte ~ $ su -<br />
Password: <br />
i|karotte ~ # locale charmap<br />
UTF-8<br />
i|karotte ~ #<br />
<br />
== Warnings during early start ==<br />
<br />
Early warnings before the start of the JVM and related logging libraries like logback and logstash forwarding are redirected to '''/var/log/open-xchange/open-xchange-console.log'''.<br />
<br />
=== Limits ===<br />
If the open-xchange startup script isn't able to set the proper limits on CentOS 6 due to e.g. hard limits being enforced via pam_limits you'll see the following warning in the console log during startup<br />
<br />
singlenode open-xchange # cat open-xchange-console.log<br />
/opt/open-xchange/sbin/open-xchange: line 115: ulimit: max user processes: cannot modify limit: Operation not permitted<br />
<br />
=== Locale ===<br />
If the open-xchange startup script isn't able to detect a unicode locale you'll see a warning like the following (the current charmap might differ)<br />
<br />
WARNING: Couldn't detect proper unicode charmap in locale category LC_CTYPE.<br />
WARNING: Current charmap: "ANSI_X3.4-1968" might cause encoding problems.<br />
<br />
== Summary: Open-Xchange middleware on specific distros ==<br />
The support for the mentioned mechanism of resource control differ depending on the distribution and the init system in use.<br />
<br />
=== Debian 7 ===<br />
;Init<br />
: System V style<br />
;OX Configurable Limits/Defaults<br />
: nofile, nproc<br />
<br />
The mentioned limits can be configured via '''/opt/open-xchange/etc/ox-scriptconf.sh'''. The limits are applied via ulimit in the service's init script. The open-xchange service is finally started via start-stop-daemon which doesn't doesn't consider '''/etc/security/limits.*'''<br />
<br />
=== RHEL 6 / CentOS 6 ===<br />
;Init<br />
: Upstart, System V compatible<br />
;OX Configurable Limits/Defaults<br />
: nofile, nproc<br />
<br />
The mentioned limits can be configured via '''/opt/open-xchange/etc/ox-scriptconf.sh'''. The limits are applied via ulimit in the service's init script. Furthermore as the open-xchange service is finally started via '''su ... open-xchange''' on this distro a user session is opened via su/pam and the default CentOS pam config reads the '''/etc/security/limits.*''' configuration by loading the pam stack like:<br />
<br />
: '''/etc/pam.d/su'''<br />
:: '''-> /etc/pam.d/system-auth'''<br />
::: '''-> pam_limits.so'''<br />
<br />
If NPROC isn't configured for the open-xchange-server it's restricted to '''1024''' globally by default to prevent accidental fork bombs, see '''/etc/security/limits.d/90-nproc.conf''' which can result in severe problems modern multithreaded applications.<br />
<br />
=== RHEL 7 / CentOS 7 / Debian 8 / SLE 12 ===<br />
;Init<br />
: Systemd<br />
;OX Configurable Limits/Defaults<br />
: nofile, nproc<br />
<br />
For systemd the default limits are configured directly in the service's unit file that is shipped by OX and located at '''/usr/lib/systemd/system/open-xchange.service'''. The drop-in config to override or extend the default unit file is located at '''/etc/systemd/system/open-xchange.service.d/limits.conf'''. [https://www.freedesktop.org/software/systemd/man/systemd.exec.html Systemd.exec] shows a whole lot of options that can be used by admins to adapt the default service to their specific needs.<br />
<br />
----<br />
<br />
[[File:diagram_sources.zip]]</div>Marenshttps://oxpedia.org/wiki/index.php?title=AppSuite:MiddlewareStartup&diff=23237AppSuite:MiddlewareStartup2017-04-24T21:20:42Z<p>Marens: </p>
<hr />
<div><br />
== Systemd ==<br />
<br />
==== Service definition ====<br />
The default service definition as shipped by Open-Xchange is rather concise and easy to read. You can check by looking at the service file that is installed to either /usr/lib/systemd or /lib/systemd depending on your distribution of choice.<br />
<br />
singlenode:~ # cat /usr/lib/systemd/system/open-xchange.service <br />
[Unit]<br />
After=remote-fs.target<br />
After=time-sync.target ypbind.service sendmail.service cyrus.service <br />
<br />
[Service]<br />
User=open-xchange<br />
PermissionsStartOnly=true<br />
TimeoutStartSec=0<br />
ExecStartPre=/opt/open-xchange/sbin/triggerupdatethemes -u<br />
ExecStart=/opt/open-xchange/sbin/open-xchange<br />
ExecStop=/opt/open-xchange/sbin/shutdown -w<br />
ExecReload=/opt/open-xchange/sbin/triggerreloadconfiguration -d<br />
KillMode=process<br />
LimitNOFILE=65536<br />
LimitNPROC=65536<br />
<br />
[Install]<br />
WantedBy=multi-user.target<br />
<br />
==== Drop-in configs ====<br />
Drop in configs allow administrators to easily adapt and override default service unit files. So if you want to change the default limits or add additional limits have a look at our default example drop-in config which is located at /etc/systemd/system/open-xchange.service.d/limits.conf<br />
<br />
singlenode:~ # cat /etc/systemd/system/open-xchange.service.d/limits.conf <br />
# Override and add options in this file<br />
# See systemd.exec(5) for other limits<br />
<br />
[Service]<br />
#LimitNPROC=65536<br />
#LimitNOFILE=65536<br />
<br />
The systemd.exec documentation shows a whole lot of options that can be used by admins to adapt the default service to their specific needs.<br />
<br />
<br clear=all><br />
[[File:mw_start_sysd.png|thumb|Service startup using systemd|left|1000px]]<br />
<br clear=all><br />
<br />
== System V / Upstart ==<br />
<br />
==== Service definition ====<br />
<br />
The default service definition as shipped by Open-Xchange can be found at /etc/init.d/open-xchange and will differ slightly depending on the distro in use.<br />
<br />
<br clear=all><br />
[[File:mw_start_sysv.png|thumb|Service startup using upstart|left|1000px]]<br />
<br clear=all><br />
== Apply Resource limits ==<br />
<br />
As you can see in the diagrams above one of the differences is that:<br />
<br />
* systemd reads all the infos wrt. resource limitations from the original service file and then applies possible further restrictions from the drop-in configs (steps 2 and 8 in the diagram ''Service startup using systemd'') and limits configured in /opt/open-xchange/etc/ox-scriptconf.sh aren't considered anylonger<br />
* the older System V style init has to source /opt/open-xchange/etc/ox-scriptconf.sh to be able to read the configs for the maximum number of processes (NPROC) and files(NRFILES) that the middleware shall be allowed to create/open (steps 3,4 and 15 in the diagram ''Service startup using upstart'').<br />
<br />
=== Background Infos ===<br />
<br />
Several ways exist to restrict resources on a linux system from a global level down to user/groups or even shells and the processes started by them.<br />
<br />
==== Sysctl ====<br />
Sysctl is used to modify kernel parameters at runtime. E.g. to set the maximum number of files<br />
<br />
$ sysctl -w fs.file-max=100000<br />
<br />
To permanently set them append to the main configuration file and reload the settings<br />
<br />
$ echo fs.file-max=100000 >> /etc/sysctl.conf<br />
$ sysctl -p<br />
<br />
More infos can be found via '''man sysctl'''<br />
<br />
==== Limits.conf ====<br />
<br />
Allows to restrict resources an a global, group or user level. E.g:<br />
<br />
$ cat /etc/security/limits.d/90-nproc.conf <br />
# Default limit for number of user's processes to prevent<br />
# accidental fork bombs.<br />
# See rhbz #432903 for reasoning.<br />
<br />
* soft nproc 1024<br />
<br />
From '''man limits.conf''':<br />
<blockquote><br />
''Also, please note that all limit settings are set per login. They are not global, nor are they permanent; existing only for the duration of the session.''<br />
</blockquote><br />
<br />
The limits per login are applied via the ''pam'' stack. See '''man pam''' and '''man pam_limits''' for more details. As those limits are bound to sessions they don't affect most daemons started by our supported init systems or init utils. Most state that they are ignored by design, see [https://bugs.launchpad.net/ubuntu/+source/upstart/+bug/938669/comments/4 upstart], [https://bugzilla.redhat.com/show_bug.cgi?id=754285#c1 systemd] and [https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=302079 start-stop-daemon] <br />
<br />
===== Ulimit =====<br />
From '''man bash'''<br />
<blockquote><br />
ulimit [-HSTabcdefilmnpqrstuvx [limit]]<br />
Provides control over the resources available to the shell and to processes started by it, on systems that allow such control.<br />
</blockquote><br />
This is what we use in our System V compatible init scripts to increase resources for the open-xchange process across multiple distros. Currently only ''the maximum number of processes'' and ''the maximum number of open file descriptors available to a single user'' are increased via ulimit. The values are specified in '''/opt/open-xchange/ox-scriptconf.sh'''<br />
<br />
==== Systemd ====<br />
<br />
===== Control Groups=====<br />
<br />
Control groups should only affect the OX middleware if you create/manage them yourself of if you are using a ''modern'' distribution that already uses systemd as init.<br />
<br />
Citing from the kernel cgroup [https://www.kernel.org/doc/Documentation/cgroup-v2.txt documentation]:<br />
<br />
<blockquote><br />
1-2. What is cgroup?<br />
<br />
cgroup is a mechanism to organize processes hierarchically and<br />
distribute system resources along the hierarchy in a controlled and<br />
configurable manner.<br />
<br />
cgroup is largely composed of two parts - the core and controllers.<br />
cgroup core is primarily responsible for hierarchically organizing<br />
processes. A cgroup controller is usually responsible for<br />
distributing a specific type of system resource along the hierarchy<br />
although there are utility controllers which serve purposes other than<br />
resource distribution.<br />
<br />
cgroups form a tree structure and every process in the system belongs<br />
to one and only one cgroup. All threads of a process belong to the<br />
same cgroup. On creation, all processes are put in the cgroup that<br />
the parent process belongs to at the time. A process can be migrated<br />
to another cgroup. Migration of a process doesn't affect already<br />
existing descendant processes.<br />
<br />
Following certain structural constraints, controllers may be enabled or<br />
disabled selectively on a cgroup. All controller behaviors are<br />
hierarchical - if a controller is enabled on a cgroup, it affects all<br />
processes which belong to the cgroups consisting the inclusive<br />
sub-hierarchy of the cgroup. When a controller is enabled on a nested<br />
cgroup, it always restricts the resource distribution further. The<br />
restrictions set closer to the root in the hierarchy can not be<br />
overridden from further away.<br />
</blockquote><br />
<br />
So processes are organized into a tree structure of control groups and controllers are responsible for the distribution of resources. So what kind of controllers exist?<br />
<br />
<blockquote><br />
5. Controllers<br />
<br />
5-1. CPU<br />
<br />
The "cpu" controllers regulates distribution of CPU cycles. This<br />
controller implements weight and absolute bandwidth limit models for<br />
normal scheduling policy and absolute bandwidth allocation model for<br />
realtime scheduling policy.<br />
<br />
5-2. Memory<br />
<br />
The "memory" controller regulates distribution of memory.<br />
...<br />
While not completely water-tight, all major memory usages by a given<br />
cgroup are tracked so that the total memory consumption can be<br />
accounted and controlled to a reasonable extent.<br />
<br />
5-3. IO<br />
<br />
The "io" controller regulates the distribution of IO resources. This<br />
controller implements both weight based and absolute bandwidth or IOPS<br />
limit distribution; however, weight based distribution is available<br />
only if cfq-iosched is in use and neither scheme is available for<br />
blk-mq devices.<br />
</blockquote><br />
<br />
The open-xchange service is simply put into the default ''system.slice'' without applying further limits.<br />
<br />
singlenode$ systemd-cgls<br />
├─1 /sbin/init<br />
├─system.slice<br />
│ ├─avahi-daemon.service<br />
│ │ ├─501 avahi-daemon: running [singlenode]<br />
│ │ └─514 avahi-daemon: chroot helper<br />
│ ├─console-kit-daemon.service<br />
│ │ └─16164 /usr/sbin/console-kit-daemon --no-daemon<br />
│ ├─dbus.service<br />
│ │ └─508 /usr/bin/dbus-daemon --system --address=systemd: --nofork --nopidfile --systemd-activation<br />
│ ├─munin-node.service<br />
│ │ └─4290 /usr/bin/perl -wT /usr/sbin/munin-node<br />
│ ├─open-xchange.service<br />
│ │ └─6037 /usr/bin/java -Dsun.net.inetaddr.ttl=3600 -Dnetworkaddress.cache.ttl=3600 -Dnetworkaddress.cache.negative.ttl=10 ...<br />
<br />
To check all the details use<br />
<br />
singlenode:~ # systemctl show system.slice<br />
<br />
==== Limits besides control groups ====<br />
Besides control groups systemd allows you to apply other limits to the execution environment of your service. Here we can apply the limits that would normally be applied via ''limits.conf'' or ''ulimit''. Systemd uses [http://man7.org/linux/man-pages/man2/setrlimit.2.html setrlimit] for this. The options that we set by default are:<br />
<br />
* LimitNOFILE<br />
* LimitNPROC<br />
<br />
which you have already seen when we took a look at the [[#Service_definiton|service definition]] and [[#Drop-in_config|drop-in configs]]<br />
<br />
== Verify limits ==<br />
<br />
=== System V ===<br />
<br />
Get the proper pid of the java process started for the middleware by either checking '''ps faux'''<br />
root 4103 0.0 0.0 66528 1868 pts/0 S 22:05 0:00 su -s /bin/bash open-xchange -c /opt/open-xchange/sbin/open-xchange<br />
494 4109 9.5 19.8 2224644 381312 ? Ssl 22:05 1:12 \_ /usr/bin/java -Dsun.net.inetaddr.ttl=3600 -Dnetworkaddress.cache.ttl=3600<br />
<br />
or programmatically<br />
singlenode:~ # pid=$(pgrep -P $(</var/run/open-xchange.pid) java)<br />
<br />
and check the limits applied for that process<br />
singlenode:~ # cat /proc/$pid/limits<br />
Limit Soft Limit Hard Limit Units<br />
Max cpu time unlimited unlimited seconds<br />
Max file size unlimited unlimited bytes<br />
Max data size unlimited unlimited bytes<br />
Max stack size 8388608 unlimited bytes<br />
Max core file size 0 unlimited bytes<br />
Max resident set unlimited unlimited bytes<br />
Max processes 65536 65536 processes<br />
Max open files 65536 65536 files<br />
Max locked memory 65536 65536 bytes<br />
Max address space unlimited unlimited bytes<br />
Max file locks unlimited unlimited locks<br />
Max pending signals 24254 24254 signals<br />
Max msgqueue size 819200 819200 bytes<br />
Max nice priority 0 0<br />
Max realtime priority 0 0<br />
Max realtime timeout unlimited unlimited us<br />
<br />
=== Systemd ===<br />
singlenode:~ # systemctl show open-xchange | grep Limit<br />
StartLimitInterval=10000000<br />
StartLimitBurst=5<br />
StartLimitAction=none<br />
MemoryLimit=18446744073709551615<br />
LimitCPU=18446744073709551615<br />
LimitFSIZE=18446744073709551615<br />
LimitDATA=18446744073709551615<br />
LimitSTACK=18446744073709551615<br />
LimitCORE=18446744073709551615<br />
LimitRSS=18446744073709551615<br />
LimitNOFILE=65536<br />
LimitAS=18446744073709551615<br />
LimitNPROC=65536<br />
LimitMEMLOCK=65536<br />
LimitLOCKS=18446744073709551615<br />
LimitSIGPENDING=19827<br />
LimitMSGQUEUE=819200<br />
LimitNICE=0<br />
LimitRTPRIO=0<br />
LimitRTTIME=18446744073709551615<br />
<br />
== Evaluate JVM commandline options ==<br />
<br />
Before starting Java and the OSGi framework with all the Open-Xchange middleware bundles we have to evaluate several important JVM commandline options that influence the performance and stability of the Open-Xchange middleware stack. These commandline options are specified in '''/opt/open-xchange/etc/ox-scriptconf.sh'''<br />
<br />
=== Format < 7.8.4 ===<br />
For version up to 7.8.3 the JVM configuration options were all specified via a single property '''JAVA_XTRAOPTS''' which resulted in a rather long configuration string that was hard to read and maintain.<br />
<br />
JAVA_XTRAOPTS="-Dsun.net.inetaddr.ttl=3600 -Dnetworkaddress.cache.ttl=3600 -Dnetworkaddress.cache.negative.ttl=10 \<br />
-Dlogback.threadlocal.put.duplicate=false -server -Djava.awt.headless=true -Xmx512M -XX:MaxPermSize=256m -XX:+UseConcMarkSweepGC \<br />
-XX:+UseParNewGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:NewRatio=3 -XX:+UseTLAB \<br />
-Dosgi.compatibility.bootdelegation=false -XX:-OmitStackTraceInFastThrow"<br />
<br />
During the startup of the open-xchange service (step 11 of diagram Service startup using systemd and step 10 of diagram Service startup using upstart) this JAVA_XTRAOPTS string goes through a basic validity check and is then passed to the JVM.<br />
<br />
=== Format >= 7.8.4 ===<br />
Beginning from version 7.8.4 the options are split up and sorted into different categories like:<br />
<br />
JAVA_OPTS_GC="-XX:+UseConcMarkSweepGC -XX:+UseParNewGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:NewRatio=3 -XX:+DisableExplicitGC"<br />
JAVA_OPTS_LOG="-Dlogback.threadlocal.put.duplicate=false -XX:-OmitStackTraceInFastThrow"<br />
JAVA_OPTS_MEM="-XX:MaxHeapSize=512M -XX:MaxPermSize=256M -XX:+UseTLAB"<br />
JAVA_OPTS_NET="-Dsun.net.inetaddr.ttl=3600 -Dnetworkaddress.cache.ttl=3600 -Dnetworkaddress.cache.negative.ttl=10"<br />
JAVA_OPTS_OSGI="-Dosgi.compatibility.bootdelegation=false"<br />
JAVA_OPTS_SERVER="-server -Djava.awt.headless=true"<br />
<br />
JAVA_OPTS_OTHER=""<br />
<br />
#JAVA_OPTS_DEBUG="-XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/mnt/heapdump -Xloggc:/var/log/open-xchange/gc.log -verbose:gc -XX:+PrintGCDateStamps \<br />
-XX:+PrintHeapAtGC -XX:+PrintGCApplicationStoppedTime -XX:+PrintTenuringDistribution"<br />
<br />
which is much easier to read an maintain. Furthermore we added a useful '''JAVA_OPTS_DEBUG''' configuration parameter that comes preconfigured but unused/commented so that customers can quickly enable the most commonly used debugging options in case they have to provide detailed system information to Open-Xchange.<br />
<br />
During the startup of the open-xchange service (step 11 of diagram ''Service startup using systemd'' and step 10 of diagram ''Service startup using upstart'') all uncommented JAVA_OPTS_* categories will then be evaluated and combined again into a single string of commandline options that can then be passed to the JVM.<br />
<br />
=== Upgrade procedure ===<br />
During the upgrade '''7.8.3-revX -> 7.8.4-revY''' existing JAVA_XTRAOPTS will automatically be split up and migrated to the new categories approach. A backup of the file (ox-scriptconf.sh.timestamp) will be created and admins are urged to check for proper migration before deleting the backup file like it's already done with *.dpkg-dist or *.rpmnew files on Debian or RHEL/SLES distributions.<br />
<br />
== Encoding detection during JVM startup ==<br />
<br />
=== Background Infos ===<br />
<br />
From ''man 7 locale''<br />
<br />
DESCRIPTION<br />
A locale is a set of language and cultural rules. These cover aspects such as lan‐<br />
guage for messages, different character sets, lexicographic conventions, and so on.<br />
A program needs to be able to determine its locale and act accordingly to be portable<br />
to different cultures.<br />
<br />
The format to specify a system locale is: '''''language[_territory][.codeset][@modifier]'''''.<br />
<br />
Where all of<br />
<br />
* en_US.UTF-8<br />
* en_US.utf8<br />
* en_US.utf-8<br />
* en_US.UTF_8 <br />
<br />
will result in a valid UTF-8 charmap for the english language with locale categories configured as<br />
<br />
LANG=en_US.UTF-8<br />
LC_CTYPE="en_US.UTF-8"<br />
LC_NUMERIC="en_US.UTF-8"<br />
LC_TIME="en_US.UTF-8"<br />
LC_COLLATE="en_US.UTF-8"<br />
LC_MONETARY="en_US.UTF-8"<br />
LC_MESSAGES="en_US.UTF-8"<br />
LC_PAPER="en_US.UTF-8"<br />
LC_NAME="en_US.UTF-8"<br />
LC_ADDRESS="en_US.UTF-8"<br />
LC_TELEPHONE="en_US.UTF-8"<br />
LC_MEASUREMENT="en_US.UTF-8"<br />
LC_IDENTIFICATION="en_US.UTF-8"<br />
LC_ALL=<br />
<br />
See for more infos: locale(1), locale(7), [https://en.wikipedia.org/wiki/List_of_ISO_639-1_codes language codes], [https://en.wikipedia.org/wiki/ISO_3166-1#Current_codes country codes] and [https://en.wikipedia.org/wiki/Character_encoding character encodings]<br />
<br />
==== JVM startup ====<br />
<br />
The default file.encoding detection is done at JVM startup based upon LC_CTYPE which makes sense as it's documented as:<br />
<br />
CODESET (LC_CTYPE)<br />
Return a string with the name of the character encoding used in the selected locale, such as "UTF-8", "ISO-8859-1",<br />
or "ANSI_X3.4-1968" (better known as US-ASCII). This is the same string that you get with "locale charmap".<br />
For a list of character encoding names, try "locale -m", cf. locale(1).<br />
<br />
Furthermore this influences the defaultCharset selection of the JVM.<br />
<br />
/**<br />
* Returns the default charset of this Java virtual machine.<br />
*<br />
* The default charset is determined during virtual-machine startup and<br />
* typically depends upon the locale and charset of the underlying<br />
* operating system.<br />
*<br />
* @return A charset object for the default charset<br />
*<br />
* @since 1.5<br />
*/<br />
public static Charset defaultCharset() {<br />
if (defaultCharset == null) {<br />
synchronized (Charset.class) {<br />
String csn = AccessController.doPrivileged(<br />
new GetPropertyAction("file.encoding"));<br />
Charset cs = lookup(csn);<br />
if (cs != null)<br />
defaultCharset = cs;<br />
else<br />
defaultCharset = forName("UTF-8");<br />
}<br />
}<br />
return defaultCharset;<br />
}<br />
<br />
Although we try to specify valid unicode encodings for every IO operations the Open-Xchange middleware does some parts of the JVM don't allow to specify the encoding to use and falls back to the detected default encoding. That's why admins have to make sure to provide a proper unicode environment for the middleware service during every start or use documented workarounds provided by Open-Xchange (e.g. )<br />
<br />
=== Query which file encoding was detected by the JVM during startup ===<br />
[[File:visualvm-file.encoding.png|thumb|Querying the JVM via visualvm|right|300px]]<br />
<br />
The file encoding that was detected by the JVM during startup can be queried via JMX. You can use graphical management tools like [https://visualvm.github.io/ visualvm], query our [[Jolokia]] JMX bridge via http or just use the commandline like<br />
<br />
/opt/open-xchange/sbin/showruntimestats -r | sed -ne 's/.*\({key=file.encoding, value=[^}]*}\).*/\1/p'<br />
{key=file.encoding, value=UTF-8}<br />
<br />
<br clear=all><br />
<br />
=== Locale passing from client to server via SSH ===<br />
<br />
The following setting of sshd allows clients to specify which locale to use on the server and might lead to some confusion or unexpected behaviour when checking the system locale or even to the usage of broken locales during middleware startup if your client sends misconfigured or missing locale configuration to the server.<br />
<br />
grep AcceptEnv /etc/ssh/sshd_config <br />
AcceptEnv LANG LC_*<br />
<br />
For example when i connect to the machine my local german locale is sent to the server <br />
<br />
[vagrant@showcase-community-centos-7 ~]$ locale -v <br />
LANG=de_DE.UTF-8<br />
LC_CTYPE="de_DE.UTF-8"<br />
LC_NUMERIC=de_DE.UTF-8<br />
LC_TIME=de_DE.UTF-8<br />
LC_COLLATE="de_DE.UTF-8"<br />
LC_MONETARY=de_DE.UTF-8<br />
LC_MESSAGES="de_DE.UTF-8"<br />
LC_PAPER=de_DE.UTF-8<br />
LC_NAME=de_DE.UTF-8<br />
LC_ADDRESS=de_DE.UTF-8<br />
LC_TELEPHONE=de_DE.UTF-8<br />
LC_MEASUREMENT=de_DE.UTF-8<br />
LC_IDENTIFICATION=de_DE.UTF-8<br />
LC_ALL=<br />
<br />
But after specifying that i want a new login shell the default locale configured for that user/system will be used instead!<br />
<br />
[root@showcase-community-centos-7 vagrant]# su -s /bin/bash open-xchange -l<br />
-bash-4.2$ locale -v <br />
LANG=en_US.UTF-8<br />
LC_CTYPE="en_US.UTF-8"<br />
LC_NUMERIC="en_US.UTF-8"<br />
LC_TIME="en_US.UTF-8"<br />
LC_COLLATE="en_US.UTF-8"<br />
LC_MONETARY="en_US.UTF-8"<br />
LC_MESSAGES="en_US.UTF-8"<br />
LC_PAPER="en_US.UTF-8"<br />
LC_NAME="en_US.UTF-8"<br />
LC_ADDRESS="en_US.UTF-8"<br />
LC_TELEPHONE="en_US.UTF-8"<br />
LC_MEASUREMENT="en_US.UTF-8"<br />
LC_IDENTIFICATION="en_US.UTF-8"<br />
LC_ALL=<br />
<br />
=== Verify that your system is configured with a proper unicode locale for OX ===<br />
<br />
Assuming you haven't reconfigured the locale of your root user<br />
<br />
i|marens@karotte ~ $ su -<br />
Password: <br />
i|karotte ~ # locale charmap<br />
UTF-8<br />
i|karotte ~ #<br />
<br />
== Warnings during early start ==<br />
<br />
Early warnings before the start of the JVM and related logging libraries like logback and logstash forwarding are redirected to '''/var/log/open-xchange/open-xchange-console.log'''.<br />
<br />
=== Limits ===<br />
If the open-xchange startup script isn't able to set the proper limits on CentOS 6 due to e.g. hard limits being enforced via pam_limits you'll see the following warning in the console log during startup<br />
<br />
singlenode open-xchange # cat open-xchange-console.log<br />
/opt/open-xchange/sbin/open-xchange: line 115: ulimit: max user processes: cannot modify limit: Operation not permitted<br />
<br />
=== Locale ===<br />
If the open-xchange startup script isn't able to detect a unicode locale you'll see a warning like the following (the current charmap might differ)<br />
<br />
WARNING: Couldn't detect proper unicode charmap in locale category LC_CTYPE.<br />
WARNING: Current charmap: "ANSI_X3.4-1968" might cause encoding problems.<br />
<br />
== Summary: Open-Xchange middleware on specific distros ==<br />
The support for the mentioned mechanism of resource control differ depending on the distribution and the init system in use.<br />
<br />
=== Debian 7 ===<br />
;Init<br />
: System V style<br />
;OX Configurable Limits/Defaults<br />
: nofile, nproc<br />
<br />
The mentioned limits can be configured via '''/opt/open-xchange/etc/ox-scriptconf.sh'''. The limits are applied via ulimit in the service's init script. The open-xchange service is finally started via start-stop-daemon which doesn't doesn't consider '''/etc/security/limits.*'''<br />
<br />
=== RHEL 6 / CentOS 6 ===<br />
;Init<br />
: Upstart, System V compatible<br />
;OX Configurable Limits/Defaults<br />
: nofile, nproc<br />
<br />
The mentioned limits can be configured via '''/opt/open-xchange/etc/ox-scriptconf.sh'''. The limits are applied via ulimit in the service's init script. Furthermore as the open-xchange service is finally started via '''su ... open-xchange''' on this distro a user session is opened via su/pam and the default CentOS pam config reads the '''/etc/security/limits.*''' configuration by loading the pam stack like:<br />
<br />
: '''/etc/pam.d/su'''<br />
:: '''-> /etc/pam.d/system-auth'''<br />
::: '''-> pam_limits.so'''<br />
<br />
If NPROC isn't configured for the open-xchange-server it's restricted to '''1024''' globally by default to prevent accidental fork bombs, see '''/etc/security/limits.d/90-nproc.conf''' which can result in severe problems modern multithreaded applications.<br />
<br />
=== RHEL 7 / CentOS 7 / Debian 8 / SLE 12 ===<br />
;Init<br />
: Systemd<br />
;OX Configurable Limits/Defaults<br />
: nofile, nproc<br />
<br />
For systemd the default limits are configured directly in the service's unit file that is shipped by OX and located at '''/usr/lib/systemd/system/open-xchange.service'''. The drop-in config to override or extend the default unit file is located at '''/etc/systemd/system/open-xchange.service.d/limits.conf'''. [https://www.freedesktop.org/software/systemd/man/systemd.exec.html Systemd.exec] shows a whole lot of options that can be used by admins to adapt the default service to their specific needs.<br />
<br />
----<br />
<br />
[[File:diagram_sources.zip]]</div>Marenshttps://oxpedia.org/wiki/index.php?title=File:Diagram_sources.zip&diff=23236File:Diagram sources.zip2017-04-24T21:20:10Z<p>Marens: </p>
<hr />
<div></div>Marenshttps://oxpedia.org/wiki/index.php?title=AppSuite:Main_Page_Advanced&diff=23235AppSuite:Main Page Advanced2017-04-24T21:06:02Z<p>Marens: /* Overview */</p>
<hr />
<div>__NOTOC__<br />
= Overview =<br />
*[[AppSuite:Architecture_Overview|Architecture Overview]]<br />
*[[Appsuite:MiddlewareStartup|Middleware Startup]]<br />
<br />
= OX App Suite UI Development =<br />
This page contains all the information you need to get started with UI development.<br />
<br />
This covers how-to articles including server communication, extension points and how to write widgets, applications and plugins:<br />
<br />
[[AppSuite:UI | Frontend development articles]]<br />
<br />
= Groupware Server customization =<br />
* [[Automatic_Delete_OXObjects_Per_User|Automatically delete users OX-Objects and OX-Folders]]<br />
* [[Login_variations|Background Information about Login Variation]]<br />
* Auto login, session handling, single sign on<br />
** [http://www.open-xchange.com/fileadmin/user_upload/open-xchange/document/presentation/OX-HE-Authentication-Sessionhandling-en-6.18.pdf Authentication and Sessionhandling white paper (EasyLogin)]<br />
** [https://documentation.open-xchange.com/7.8.2/middleware/components/saml.html SAML SSO Integration]<br />
** Custom login masks<br />
*** [[Open-Xchange servlet for external login masks]]<br />
<br />
= Advanced Configuration =<br />
* [[AppSuite:CalDAVClients | Configuration CalDAV with Open-Xchange]]<br />
* [[AppSuite:CardDAVClients | Configuration CardDAV with Open-Xchange]]<br />
* [[AppSuite:Caldav_carddav_Bundles| Installation CardDAV/CalDAV with Open-Xchange]]<br />
* [[AppSuite:Running_a_cluster| Configuration Cluster-Setup]]<br />
* [[AppSuite:Grizzly| Configuration Connector based on Grizzly]]<br />
* [[ContactStorageLDAP| LDAP Contact Storage]]<br />
* [[AppSuite:Sharing_and_Guest_Mode |Sharing and Guest Mode]]<br />
* [https://documentation.open-xchange.com/latest/ui/features/metrics.html Metrics]<br />
* [https://documentation.open-xchange.com/latest/ui/features/upsell.html Upsell]<br />
* [https://documentation.open-xchange.com/latest/middleware/components/oauth_provider.html OAuth 2.0 - Provider/Operator Guide and Client Developer Guide]<br />
* [[AppSuite:File_Storages_per_User |File Storage per User ]]<br />
* [[AppSuite:Paste_inline_images |Pasting External Images into Mail Compose ]]<br />
* [[AppSuite:Create_custom_folderview_entries_in_settings_app |Custom Folder View Entries in Settings ]]<br />
* [[AppSuite:DB_user_privileges | Database User Privileges]]<br />
* [[AppSuite:CrossContextDatabase |GlobalDB/Cross-Context Database ]]<br />
* [[AppSuite:Filestorages |Setup external File Stores ]]<br />
* [[AppSuite:Client_Onboarding|Configuration guide for Open-Xchange Client Onboarding]]<br />
* [https://documentation.open-xchange.com/latest/middleware/components/auditlogging.html Audit logging]<br />
* [https://documentation.open-xchange.com/latest/middleware/components/drivemail.html Drive Mail]<br />
* [https://documentation.open-xchange.com/latest/middleware/components/mail_categories.html Mail Categories]<br />
* [https://documentation.open-xchange.com/latest/middleware/components/virtual_mail_attachments.html Virtual Mail Attachments Connector]<br />
* [https://documentation.open-xchange.com/7.8.3/middleware/components/trusted_connections.html Configure trusted TLS certificates]<br />
* [https://documentation.open-xchange.com/7.8.3/middleware/components/websockets.html Configure Web Sockets]<br />
* [[AppSuite:ResourceLimits | Configure Middleware Resource Limits]]<br />
<br />
= Programming Interfaces =<br />
* The [https://documentation.open-xchange.com/latest/middleware/http_api.html HTTP API] is used by the Open-Xchange GUI and various 3rd party applications. It consists mainly of messages in JavaScript Object Notation ([http://json.org JSON]) sent over HTTP. Here is a general [https://documentation.open-xchange.com/latest/middleware/http_api/1_introduction.html Introduction to the HTTP API].<br />
* Provisioning API to access the Open-Xchange Admin Daemon<br />
** The [http://java.sun.com/javase/technologies/core/basic/rmi/index.jsp RMI] API is used for data provisioning of Contexts, Users, Groups and Resources as well as for configuring Databases, Filestores and OX Servers: [http://software.open-xchange.com/products/appsuite/doc/RMI/javadoc/ OX RMI API]<br />
** The [http://oxpedia.org/wiki/index.php?title=AppSuite:AdminGuide_7.8.2#OX_App_Suite_Management_.28CLT.29 CLT ] are shell scripts that simplify groupware and service administration<br />
** Create contexts/users with with [[Csv_import]]<br />
** [[Open-Xchange-SOAP|Provisioning using SOAP]]<br />
* The [[Oxmapi|Oxmapi]] is a windows library for programmers needed to communicate with the OX server<br />
* {{DocLink|docpath=mal/|name=Open-Xchange Mail Abstraction Layer}}<br />
* [[UDPPush]] Open-Xchange PUSH Interface for Groupware Objects<br />
<br />
= Testing and QA =<br />
* [[AppSuite:Web Tests|Web Tests]]<br />
* [[AppSuite:Load Tests|Load Tests]]<br />
* [[AppSuite:CLT Tests|CLT Tests]]<br />
* [[AppSuite:SOAP Tests|SOAP Tests]]<br />
<br />
= Statistics =<br />
<br />
* [[AppSuite:Logincounter|logincounter]]<br />
<br />
= Translations =<br />
* [[AppSuite:Available_Translations| Available Language Translations]]<br />
* [[GUI Translation|Translate Open-Xchange to your community language]]<br />
* [[AppSuite:Translate_Open-Xchange_to_supported_language|Translate Open-Xchange to supported language]]<br />
<br />
= Installation based on source code =<br />
* [[SourceCodeAccess|Download the sourcecode]]<br />
<br />
= Integration =<br />
* {{DocLink|docpath=whitepaper/OX-Directory-Integration-Whitepaper-English.pdf|name=Directory Integration Whitepaper}}<br />
<br />
= Additional =<br />
* [[AppSuite:ISV_Mockups_Wireframes|Create Open-Xchange AppSuite Mockups / Wireframes]]<br />
* [https://documentation.open-xchange.com/7.8.2/middleware/components/search/crossfolder_fts_in_mail.html Cross folder fulltext search with Dovecot]</div>Marenshttps://oxpedia.org/wiki/index.php?title=AppSuite:ResourceLimits&diff=23234AppSuite:ResourceLimits2017-04-24T20:58:44Z<p>Marens: Add redirect to middleware startup</p>
<hr />
<div>#REDIRECT [[:Appsuite:MiddlewareStartup]]</div>Marenshttps://oxpedia.org/wiki/index.php?title=AppSuite:MiddlewareStartup&diff=23233AppSuite:MiddlewareStartup2017-04-24T20:57:25Z<p>Marens: Created page with " == Systemd == ==== Service definition ==== The default service definition as shipped by Open-Xchange is rather concise and easy to read. You can check by looking at the serv..."</p>
<hr />
<div><br />
== Systemd ==<br />
<br />
==== Service definition ====<br />
The default service definition as shipped by Open-Xchange is rather concise and easy to read. You can check by looking at the service file that is installed to either /usr/lib/systemd or /lib/systemd depending on your distribution of choice.<br />
<br />
singlenode:~ # cat /usr/lib/systemd/system/open-xchange.service <br />
[Unit]<br />
After=remote-fs.target<br />
After=time-sync.target ypbind.service sendmail.service cyrus.service <br />
<br />
[Service]<br />
User=open-xchange<br />
PermissionsStartOnly=true<br />
TimeoutStartSec=0<br />
ExecStartPre=/opt/open-xchange/sbin/triggerupdatethemes -u<br />
ExecStart=/opt/open-xchange/sbin/open-xchange<br />
ExecStop=/opt/open-xchange/sbin/shutdown -w<br />
ExecReload=/opt/open-xchange/sbin/triggerreloadconfiguration -d<br />
KillMode=process<br />
LimitNOFILE=65536<br />
LimitNPROC=65536<br />
<br />
[Install]<br />
WantedBy=multi-user.target<br />
<br />
==== Drop-in configs ====<br />
Drop in configs allow administrators to easily adapt and override default service unit files. So if you want to change the default limits or add additional limits have a look at our default example drop-in config which is located at /etc/systemd/system/open-xchange.service.d/limits.conf<br />
<br />
singlenode:~ # cat /etc/systemd/system/open-xchange.service.d/limits.conf <br />
# Override and add options in this file<br />
# See systemd.exec(5) for other limits<br />
<br />
[Service]<br />
#LimitNPROC=65536<br />
#LimitNOFILE=65536<br />
<br />
The systemd.exec documentation shows a whole lot of options that can be used by admins to adapt the default service to their specific needs.<br />
<br />
<br clear=all><br />
[[File:mw_start_sysd.png|thumb|Service startup using systemd|left|1000px]]<br />
<br clear=all><br />
<br />
== System V / Upstart ==<br />
<br />
==== Service definition ====<br />
<br />
The default service definition as shipped by Open-Xchange can be found at /etc/init.d/open-xchange and will differ slightly depending on the distro in use.<br />
<br />
<br clear=all><br />
[[File:mw_start_sysv.png|thumb|Service startup using upstart|left|1000px]]<br />
<br clear=all><br />
== Apply Resource limits ==<br />
<br />
As you can see in the diagrams above one of the differences is that:<br />
<br />
* systemd reads all the infos wrt. resource limitations from the original service file and then applies possible further restrictions from the drop-in configs (steps 2 and 8 in the diagram ''Service startup using systemd'') and limits configured in /opt/open-xchange/etc/ox-scriptconf.sh aren't considered anylonger<br />
* the older System V style init has to source /opt/open-xchange/etc/ox-scriptconf.sh to be able to read the configs for the maximum number of processes (NPROC) and files(NRFILES) that the middleware shall be allowed to create/open (steps 3,4 and 15 in the diagram ''Service startup using upstart'').<br />
<br />
=== Background Infos ===<br />
<br />
Several ways exist to restrict resources on a linux system from a global level down to user/groups or even shells and the processes started by them.<br />
<br />
==== Sysctl ====<br />
Sysctl is used to modify kernel parameters at runtime. E.g. to set the maximum number of files<br />
<br />
$ sysctl -w fs.file-max=100000<br />
<br />
To permanently set them append to the main configuration file and reload the settings<br />
<br />
$ echo fs.file-max=100000 >> /etc/sysctl.conf<br />
$ sysctl -p<br />
<br />
More infos can be found via '''man sysctl'''<br />
<br />
==== Limits.conf ====<br />
<br />
Allows to restrict resources an a global, group or user level. E.g:<br />
<br />
$ cat /etc/security/limits.d/90-nproc.conf <br />
# Default limit for number of user's processes to prevent<br />
# accidental fork bombs.<br />
# See rhbz #432903 for reasoning.<br />
<br />
* soft nproc 1024<br />
<br />
From '''man limits.conf''':<br />
<blockquote><br />
''Also, please note that all limit settings are set per login. They are not global, nor are they permanent; existing only for the duration of the session.''<br />
</blockquote><br />
<br />
The limits per login are applied via the ''pam'' stack. See '''man pam''' and '''man pam_limits''' for more details. As those limits are bound to sessions they don't affect most daemons started by our supported init systems or init utils. Most state that they are ignored by design, see [https://bugs.launchpad.net/ubuntu/+source/upstart/+bug/938669/comments/4 upstart], [https://bugzilla.redhat.com/show_bug.cgi?id=754285#c1 systemd] and [https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=302079 start-stop-daemon] <br />
<br />
===== Ulimit =====<br />
From '''man bash'''<br />
<blockquote><br />
ulimit [-HSTabcdefilmnpqrstuvx [limit]]<br />
Provides control over the resources available to the shell and to processes started by it, on systems that allow such control.<br />
</blockquote><br />
This is what we use in our System V compatible init scripts to increase resources for the open-xchange process across multiple distros. Currently only ''the maximum number of processes'' and ''the maximum number of open file descriptors available to a single user'' are increased via ulimit. The values are specified in '''/opt/open-xchange/ox-scriptconf.sh'''<br />
<br />
==== Systemd ====<br />
<br />
===== Control Groups=====<br />
<br />
Control groups should only affect the OX middleware if you create/manage them yourself of if you are using a ''modern'' distribution that already uses systemd as init.<br />
<br />
Citing from the kernel cgroup [https://www.kernel.org/doc/Documentation/cgroup-v2.txt documentation]:<br />
<br />
<blockquote><br />
1-2. What is cgroup?<br />
<br />
cgroup is a mechanism to organize processes hierarchically and<br />
distribute system resources along the hierarchy in a controlled and<br />
configurable manner.<br />
<br />
cgroup is largely composed of two parts - the core and controllers.<br />
cgroup core is primarily responsible for hierarchically organizing<br />
processes. A cgroup controller is usually responsible for<br />
distributing a specific type of system resource along the hierarchy<br />
although there are utility controllers which serve purposes other than<br />
resource distribution.<br />
<br />
cgroups form a tree structure and every process in the system belongs<br />
to one and only one cgroup. All threads of a process belong to the<br />
same cgroup. On creation, all processes are put in the cgroup that<br />
the parent process belongs to at the time. A process can be migrated<br />
to another cgroup. Migration of a process doesn't affect already<br />
existing descendant processes.<br />
<br />
Following certain structural constraints, controllers may be enabled or<br />
disabled selectively on a cgroup. All controller behaviors are<br />
hierarchical - if a controller is enabled on a cgroup, it affects all<br />
processes which belong to the cgroups consisting the inclusive<br />
sub-hierarchy of the cgroup. When a controller is enabled on a nested<br />
cgroup, it always restricts the resource distribution further. The<br />
restrictions set closer to the root in the hierarchy can not be<br />
overridden from further away.<br />
</blockquote><br />
<br />
So processes are organized into a tree structure of control groups and controllers are responsible for the distribution of resources. So what kind of controllers exist?<br />
<br />
<blockquote><br />
5. Controllers<br />
<br />
5-1. CPU<br />
<br />
The "cpu" controllers regulates distribution of CPU cycles. This<br />
controller implements weight and absolute bandwidth limit models for<br />
normal scheduling policy and absolute bandwidth allocation model for<br />
realtime scheduling policy.<br />
<br />
5-2. Memory<br />
<br />
The "memory" controller regulates distribution of memory.<br />
...<br />
While not completely water-tight, all major memory usages by a given<br />
cgroup are tracked so that the total memory consumption can be<br />
accounted and controlled to a reasonable extent.<br />
<br />
5-3. IO<br />
<br />
The "io" controller regulates the distribution of IO resources. This<br />
controller implements both weight based and absolute bandwidth or IOPS<br />
limit distribution; however, weight based distribution is available<br />
only if cfq-iosched is in use and neither scheme is available for<br />
blk-mq devices.<br />
</blockquote><br />
<br />
The open-xchange service is simply put into the default ''system.slice'' without applying further limits.<br />
<br />
singlenode$ systemd-cgls<br />
├─1 /sbin/init<br />
├─system.slice<br />
│ ├─avahi-daemon.service<br />
│ │ ├─501 avahi-daemon: running [singlenode]<br />
│ │ └─514 avahi-daemon: chroot helper<br />
│ ├─console-kit-daemon.service<br />
│ │ └─16164 /usr/sbin/console-kit-daemon --no-daemon<br />
│ ├─dbus.service<br />
│ │ └─508 /usr/bin/dbus-daemon --system --address=systemd: --nofork --nopidfile --systemd-activation<br />
│ ├─munin-node.service<br />
│ │ └─4290 /usr/bin/perl -wT /usr/sbin/munin-node<br />
│ ├─open-xchange.service<br />
│ │ └─6037 /usr/bin/java -Dsun.net.inetaddr.ttl=3600 -Dnetworkaddress.cache.ttl=3600 -Dnetworkaddress.cache.negative.ttl=10 ...<br />
<br />
To check all the details use<br />
<br />
singlenode:~ # systemctl show system.slice<br />
<br />
==== Limits besides control groups ====<br />
Besides control groups systemd allows you to apply other limits to the execution environment of your service. Here we can apply the limits that would normally be applied via ''limits.conf'' or ''ulimit''. Systemd uses [http://man7.org/linux/man-pages/man2/setrlimit.2.html setrlimit] for this. The options that we set by default are:<br />
<br />
* LimitNOFILE<br />
* LimitNPROC<br />
<br />
which you have already seen when we took a look at the [[#Service_definiton|service definition]] and [[#Drop-in_config|drop-in configs]]<br />
<br />
== Verify limits ==<br />
<br />
=== System V ===<br />
<br />
Get the proper pid of the java process started for the middleware by either checking '''ps faux'''<br />
root 4103 0.0 0.0 66528 1868 pts/0 S 22:05 0:00 su -s /bin/bash open-xchange -c /opt/open-xchange/sbin/open-xchange<br />
494 4109 9.5 19.8 2224644 381312 ? Ssl 22:05 1:12 \_ /usr/bin/java -Dsun.net.inetaddr.ttl=3600 -Dnetworkaddress.cache.ttl=3600<br />
<br />
or programmatically<br />
singlenode:~ # pid=$(pgrep -P $(</var/run/open-xchange.pid) java)<br />
<br />
and check the limits applied for that process<br />
singlenode:~ # cat /proc/$pid/limits<br />
Limit Soft Limit Hard Limit Units<br />
Max cpu time unlimited unlimited seconds<br />
Max file size unlimited unlimited bytes<br />
Max data size unlimited unlimited bytes<br />
Max stack size 8388608 unlimited bytes<br />
Max core file size 0 unlimited bytes<br />
Max resident set unlimited unlimited bytes<br />
Max processes 65536 65536 processes<br />
Max open files 65536 65536 files<br />
Max locked memory 65536 65536 bytes<br />
Max address space unlimited unlimited bytes<br />
Max file locks unlimited unlimited locks<br />
Max pending signals 24254 24254 signals<br />
Max msgqueue size 819200 819200 bytes<br />
Max nice priority 0 0<br />
Max realtime priority 0 0<br />
Max realtime timeout unlimited unlimited us<br />
<br />
=== Systemd ===<br />
singlenode:~ # systemctl show open-xchange | grep Limit<br />
StartLimitInterval=10000000<br />
StartLimitBurst=5<br />
StartLimitAction=none<br />
MemoryLimit=18446744073709551615<br />
LimitCPU=18446744073709551615<br />
LimitFSIZE=18446744073709551615<br />
LimitDATA=18446744073709551615<br />
LimitSTACK=18446744073709551615<br />
LimitCORE=18446744073709551615<br />
LimitRSS=18446744073709551615<br />
LimitNOFILE=65536<br />
LimitAS=18446744073709551615<br />
LimitNPROC=65536<br />
LimitMEMLOCK=65536<br />
LimitLOCKS=18446744073709551615<br />
LimitSIGPENDING=19827<br />
LimitMSGQUEUE=819200<br />
LimitNICE=0<br />
LimitRTPRIO=0<br />
LimitRTTIME=18446744073709551615<br />
<br />
== Evaluate JVM commandline options ==<br />
<br />
Before starting Java and the OSGi framework with all the Open-Xchange middleware bundles we have to evaluate several important JVM commandline options that influence the performance and stability of the Open-Xchange middleware stack. These commandline options are specified in '''/opt/open-xchange/etc/ox-scriptconf.sh'''<br />
<br />
=== Format < 7.8.4 ===<br />
For version up to 7.8.3 the JVM configuration options were all specified via a single property '''JAVA_XTRAOPTS''' which resulted in a rather long configuration string that was hard to read and maintain.<br />
<br />
JAVA_XTRAOPTS="-Dsun.net.inetaddr.ttl=3600 -Dnetworkaddress.cache.ttl=3600 -Dnetworkaddress.cache.negative.ttl=10 \<br />
-Dlogback.threadlocal.put.duplicate=false -server -Djava.awt.headless=true -Xmx512M -XX:MaxPermSize=256m -XX:+UseConcMarkSweepGC \<br />
-XX:+UseParNewGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:NewRatio=3 -XX:+UseTLAB \<br />
-Dosgi.compatibility.bootdelegation=false -XX:-OmitStackTraceInFastThrow"<br />
<br />
During the startup of the open-xchange service (step 11 of diagram Service startup using systemd and step 10 of diagram Service startup using upstart) this JAVA_XTRAOPTS string goes through a basic validity check and is then passed to the JVM.<br />
<br />
=== Format >= 7.8.4 ===<br />
Beginning from version 7.8.4 the options are split up and sorted into different categories like:<br />
<br />
JAVA_OPTS_GC="-XX:+UseConcMarkSweepGC -XX:+UseParNewGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:NewRatio=3 -XX:+DisableExplicitGC"<br />
JAVA_OPTS_LOG="-Dlogback.threadlocal.put.duplicate=false -XX:-OmitStackTraceInFastThrow"<br />
JAVA_OPTS_MEM="-XX:MaxHeapSize=512M -XX:MaxPermSize=256M -XX:+UseTLAB"<br />
JAVA_OPTS_NET="-Dsun.net.inetaddr.ttl=3600 -Dnetworkaddress.cache.ttl=3600 -Dnetworkaddress.cache.negative.ttl=10"<br />
JAVA_OPTS_OSGI="-Dosgi.compatibility.bootdelegation=false"<br />
JAVA_OPTS_SERVER="-server -Djava.awt.headless=true"<br />
<br />
JAVA_OPTS_OTHER=""<br />
<br />
#JAVA_OPTS_DEBUG="-XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/mnt/heapdump -Xloggc:/var/log/open-xchange/gc.log -verbose:gc -XX:+PrintGCDateStamps \<br />
-XX:+PrintHeapAtGC -XX:+PrintGCApplicationStoppedTime -XX:+PrintTenuringDistribution"<br />
<br />
which is much easier to read an maintain. Furthermore we added a useful '''JAVA_OPTS_DEBUG''' configuration parameter that comes preconfigured but unused/commented so that customers can quickly enable the most commonly used debugging options in case they have to provide detailed system information to Open-Xchange.<br />
<br />
During the startup of the open-xchange service (step 11 of diagram ''Service startup using systemd'' and step 10 of diagram ''Service startup using upstart'') all uncommented JAVA_OPTS_* categories will then be evaluated and combined again into a single string of commandline options that can then be passed to the JVM.<br />
<br />
=== Upgrade procedure ===<br />
During the upgrade '''7.8.3-revX -> 7.8.4-revY''' existing JAVA_XTRAOPTS will automatically be split up and migrated to the new categories approach. A backup of the file (ox-scriptconf.sh.timestamp) will be created and admins are urged to check for proper migration before deleting the backup file like it's already done with *.dpkg-dist or *.rpmnew files on Debian or RHEL/SLES distributions.<br />
<br />
== Encoding detection during JVM startup ==<br />
<br />
=== Background Infos ===<br />
<br />
From ''man 7 locale''<br />
<br />
DESCRIPTION<br />
A locale is a set of language and cultural rules. These cover aspects such as lan‐<br />
guage for messages, different character sets, lexicographic conventions, and so on.<br />
A program needs to be able to determine its locale and act accordingly to be portable<br />
to different cultures.<br />
<br />
The format to specify a system locale is: '''''language[_territory][.codeset][@modifier]'''''.<br />
<br />
Where all of<br />
<br />
* en_US.UTF-8<br />
* en_US.utf8<br />
* en_US.utf-8<br />
* en_US.UTF_8 <br />
<br />
will result in a valid UTF-8 charmap for the english language with locale categories configured as<br />
<br />
LANG=en_US.UTF-8<br />
LC_CTYPE="en_US.UTF-8"<br />
LC_NUMERIC="en_US.UTF-8"<br />
LC_TIME="en_US.UTF-8"<br />
LC_COLLATE="en_US.UTF-8"<br />
LC_MONETARY="en_US.UTF-8"<br />
LC_MESSAGES="en_US.UTF-8"<br />
LC_PAPER="en_US.UTF-8"<br />
LC_NAME="en_US.UTF-8"<br />
LC_ADDRESS="en_US.UTF-8"<br />
LC_TELEPHONE="en_US.UTF-8"<br />
LC_MEASUREMENT="en_US.UTF-8"<br />
LC_IDENTIFICATION="en_US.UTF-8"<br />
LC_ALL=<br />
<br />
See for more infos: locale(1), locale(7), [https://en.wikipedia.org/wiki/List_of_ISO_639-1_codes language codes], [https://en.wikipedia.org/wiki/ISO_3166-1#Current_codes country codes] and [https://en.wikipedia.org/wiki/Character_encoding character encodings]<br />
<br />
==== JVM startup ====<br />
<br />
The default file.encoding detection is done at JVM startup based upon LC_CTYPE which makes sense as it's documented as:<br />
<br />
CODESET (LC_CTYPE)<br />
Return a string with the name of the character encoding used in the selected locale, such as "UTF-8", "ISO-8859-1",<br />
or "ANSI_X3.4-1968" (better known as US-ASCII). This is the same string that you get with "locale charmap".<br />
For a list of character encoding names, try "locale -m", cf. locale(1).<br />
<br />
Furthermore this influences the defaultCharset selection of the JVM.<br />
<br />
/**<br />
* Returns the default charset of this Java virtual machine.<br />
*<br />
* The default charset is determined during virtual-machine startup and<br />
* typically depends upon the locale and charset of the underlying<br />
* operating system.<br />
*<br />
* @return A charset object for the default charset<br />
*<br />
* @since 1.5<br />
*/<br />
public static Charset defaultCharset() {<br />
if (defaultCharset == null) {<br />
synchronized (Charset.class) {<br />
String csn = AccessController.doPrivileged(<br />
new GetPropertyAction("file.encoding"));<br />
Charset cs = lookup(csn);<br />
if (cs != null)<br />
defaultCharset = cs;<br />
else<br />
defaultCharset = forName("UTF-8");<br />
}<br />
}<br />
return defaultCharset;<br />
}<br />
<br />
Although we try to specify valid unicode encodings for every IO operations the Open-Xchange middleware does some parts of the JVM don't allow to specify the encoding to use and falls back to the detected default encoding. That's why admins have to make sure to provide a proper unicode environment for the middleware service during every start or use documented workarounds provided by Open-Xchange (e.g. )<br />
<br />
=== Query which file encoding was detected by the JVM during startup ===<br />
[[File:visualvm-file.encoding.png|thumb|Querying the JVM via visualvm|right|300px]]<br />
<br />
The file encoding that was detected by the JVM during startup can be queried via JMX. You can use graphical management tools like [https://visualvm.github.io/ visualvm], query our [[Jolokia]] JMX bridge via http or just use the commandline like<br />
<br />
/opt/open-xchange/sbin/showruntimestats -r | sed -ne 's/.*\({key=file.encoding, value=[^}]*}\).*/\1/p'<br />
{key=file.encoding, value=UTF-8}<br />
<br />
<br clear=all><br />
<br />
=== Locale passing from client to server via SSH ===<br />
<br />
The following setting of sshd allows clients to specify which locale to use on the server and might lead to some confusion or unexpected behaviour when checking the system locale or even to the usage of broken locales during middleware startup if your client sends misconfigured or missing locale configuration to the server.<br />
<br />
grep AcceptEnv /etc/ssh/sshd_config <br />
AcceptEnv LANG LC_*<br />
<br />
For example when i connect to the machine my local german locale is sent to the server <br />
<br />
[vagrant@showcase-community-centos-7 ~]$ locale -v <br />
LANG=de_DE.UTF-8<br />
LC_CTYPE="de_DE.UTF-8"<br />
LC_NUMERIC=de_DE.UTF-8<br />
LC_TIME=de_DE.UTF-8<br />
LC_COLLATE="de_DE.UTF-8"<br />
LC_MONETARY=de_DE.UTF-8<br />
LC_MESSAGES="de_DE.UTF-8"<br />
LC_PAPER=de_DE.UTF-8<br />
LC_NAME=de_DE.UTF-8<br />
LC_ADDRESS=de_DE.UTF-8<br />
LC_TELEPHONE=de_DE.UTF-8<br />
LC_MEASUREMENT=de_DE.UTF-8<br />
LC_IDENTIFICATION=de_DE.UTF-8<br />
LC_ALL=<br />
<br />
But after specifying that i want a new login shell the default locale configured for that user/system will be used instead!<br />
<br />
[root@showcase-community-centos-7 vagrant]# su -s /bin/bash open-xchange -l<br />
-bash-4.2$ locale -v <br />
LANG=en_US.UTF-8<br />
LC_CTYPE="en_US.UTF-8"<br />
LC_NUMERIC="en_US.UTF-8"<br />
LC_TIME="en_US.UTF-8"<br />
LC_COLLATE="en_US.UTF-8"<br />
LC_MONETARY="en_US.UTF-8"<br />
LC_MESSAGES="en_US.UTF-8"<br />
LC_PAPER="en_US.UTF-8"<br />
LC_NAME="en_US.UTF-8"<br />
LC_ADDRESS="en_US.UTF-8"<br />
LC_TELEPHONE="en_US.UTF-8"<br />
LC_MEASUREMENT="en_US.UTF-8"<br />
LC_IDENTIFICATION="en_US.UTF-8"<br />
LC_ALL=<br />
<br />
=== Verify that your system is configured with a proper unicode locale for OX ===<br />
<br />
Assuming you haven't reconfigured the locale of your root user<br />
<br />
i|marens@karotte ~ $ su -<br />
Password: <br />
i|karotte ~ # locale charmap<br />
UTF-8<br />
i|karotte ~ #<br />
<br />
== Warnings during early start ==<br />
<br />
Early warnings before the start of the JVM and related logging libraries like logback and logstash forwarding are redirected to '''/var/log/open-xchange/open-xchange-console.log'''.<br />
<br />
=== Limits ===<br />
If the open-xchange startup script isn't able to set the proper limits on CentOS 6 due to e.g. hard limits being enforced via pam_limits you'll see the following warning in the console log during startup<br />
<br />
singlenode open-xchange # cat open-xchange-console.log<br />
/opt/open-xchange/sbin/open-xchange: line 115: ulimit: max user processes: cannot modify limit: Operation not permitted<br />
<br />
=== Locale ===<br />
If the open-xchange startup script isn't able to detect a unicode locale you'll see a warning like the following (the current charmap might differ)<br />
<br />
WARNING: Couldn't detect proper unicode charmap in locale category LC_CTYPE.<br />
WARNING: Current charmap: "ANSI_X3.4-1968" might cause encoding problems.<br />
<br />
== Summary: Open-Xchange middleware on specific distros ==<br />
The support for the mentioned mechanism of resource control differ depending on the distribution and the init system in use.<br />
<br />
=== Debian 7 ===<br />
;Init<br />
: System V style<br />
;OX Configurable Limits/Defaults<br />
: nofile, nproc<br />
<br />
The mentioned limits can be configured via '''/opt/open-xchange/etc/ox-scriptconf.sh'''. The limits are applied via ulimit in the service's init script. The open-xchange service is finally started via start-stop-daemon which doesn't doesn't consider '''/etc/security/limits.*'''<br />
<br />
=== RHEL 6 / CentOS 6 ===<br />
;Init<br />
: Upstart, System V compatible<br />
;OX Configurable Limits/Defaults<br />
: nofile, nproc<br />
<br />
The mentioned limits can be configured via '''/opt/open-xchange/etc/ox-scriptconf.sh'''. The limits are applied via ulimit in the service's init script. Furthermore as the open-xchange service is finally started via '''su ... open-xchange''' on this distro a user session is opened via su/pam and the default CentOS pam config reads the '''/etc/security/limits.*''' configuration by loading the pam stack like:<br />
<br />
: '''/etc/pam.d/su'''<br />
:: '''-> /etc/pam.d/system-auth'''<br />
::: '''-> pam_limits.so'''<br />
<br />
If NPROC isn't configured for the open-xchange-server it's restricted to '''1024''' globally by default to prevent accidental fork bombs, see '''/etc/security/limits.d/90-nproc.conf''' which can result in severe problems modern multithreaded applications.<br />
<br />
=== RHEL 7 / CentOS 7 / Debian 8 / SLE 12 ===<br />
;Init<br />
: Systemd<br />
;OX Configurable Limits/Defaults<br />
: nofile, nproc<br />
<br />
For systemd the default limits are configured directly in the service's unit file that is shipped by OX and located at '''/usr/lib/systemd/system/open-xchange.service'''. The drop-in config to override or extend the default unit file is located at '''/etc/systemd/system/open-xchange.service.d/limits.conf'''. [https://www.freedesktop.org/software/systemd/man/systemd.exec.html Systemd.exec] shows a whole lot of options that can be used by admins to adapt the default service to their specific needs.</div>Marenshttps://oxpedia.org/wiki/index.php?title=User:Marens&diff=23232User:Marens2017-04-24T20:56:53Z<p>Marens: </p>
<hr />
<div>[[Grizzly]]<br />
<br />
[[AppSuite:Open-Xchange_Installation_Guide_for_Debian_6.0/sandbox]]<br />
<br />
[[AppSuite:OX_Tutorial_10K/sandbox]]<br />
<br />
[[AppSuite:OX_Tutorial_100K/sandbox]]<br />
<br />
[[AppSuite:OX_Tutorial_1M/sandbox]]<br />
<br />
[[Template:OX_HE_Tutorial_Design]]<br />
<br />
[[Template:Oxinstaller/sandbox]]<br />
<br />
[[Appsuite:ResourceLimits/sandbox]]<br />
<br />
[[Appsuite:ResourceLimits]]<br />
<br />
[[Appsuite:ResourceLimits]]<br />
<br />
[[Sandbox:Middleware_Startup/sandbox]]<br />
<br />
[[Appsuite:MiddlewareStartup]]</div>Marenshttps://oxpedia.org/wiki/index.php?title=Sandbox:Middleware_Startup/sandbox&diff=23231Sandbox:Middleware Startup/sandbox2017-04-24T20:50:52Z<p>Marens: /* Encoding detection during JVM startup */</p>
<hr />
<div><br />
== Systemd ==<br />
<br />
==== Service definition ====<br />
The default service definition as shipped by Open-Xchange is rather concise and easy to read. You can check by looking at the service file that is installed to either /usr/lib/systemd or /lib/systemd depending on your distribution of choice.<br />
<br />
singlenode:~ # cat /usr/lib/systemd/system/open-xchange.service <br />
[Unit]<br />
After=remote-fs.target<br />
After=time-sync.target ypbind.service sendmail.service cyrus.service <br />
<br />
[Service]<br />
User=open-xchange<br />
PermissionsStartOnly=true<br />
TimeoutStartSec=0<br />
ExecStartPre=/opt/open-xchange/sbin/triggerupdatethemes -u<br />
ExecStart=/opt/open-xchange/sbin/open-xchange<br />
ExecStop=/opt/open-xchange/sbin/shutdown -w<br />
ExecReload=/opt/open-xchange/sbin/triggerreloadconfiguration -d<br />
KillMode=process<br />
LimitNOFILE=65536<br />
LimitNPROC=65536<br />
<br />
[Install]<br />
WantedBy=multi-user.target<br />
<br />
==== Drop-in configs ====<br />
Drop in configs allow administrators to easily adapt and override default service unit files. So if you want to change the default limits or add additional limits have a look at our default example drop-in config which is located at /etc/systemd/system/open-xchange.service.d/limits.conf<br />
<br />
singlenode:~ # cat /etc/systemd/system/open-xchange.service.d/limits.conf <br />
# Override and add options in this file<br />
# See systemd.exec(5) for other limits<br />
<br />
[Service]<br />
#LimitNPROC=65536<br />
#LimitNOFILE=65536<br />
<br />
The systemd.exec documentation shows a whole lot of options that can be used by admins to adapt the default service to their specific needs.<br />
<br />
<br clear=all><br />
[[File:mw_start_sysd.png|thumb|Service startup using systemd|left|1000px]]<br />
<br clear=all><br />
<br />
== System V / Upstart ==<br />
<br />
==== Service definition ====<br />
<br />
The default service definition as shipped by Open-Xchange can be found at /etc/init.d/open-xchange and will differ slightly depending on the distro in use.<br />
<br />
<br clear=all><br />
[[File:mw_start_sysv.png|thumb|Service startup using upstart|left|1000px]]<br />
<br clear=all><br />
== Apply Resource limits ==<br />
<br />
As you can see in the diagrams above one of the differences is that:<br />
<br />
* systemd reads all the infos wrt. resource limitations from the original service file and then applies possible further restrictions from the drop-in configs (steps 2 and 8 in the diagram ''Service startup using systemd'') and limits configured in /opt/open-xchange/etc/ox-scriptconf.sh aren't considered anylonger<br />
* the older System V style init has to source /opt/open-xchange/etc/ox-scriptconf.sh to be able to read the configs for the maximum number of processes (NPROC) and files(NRFILES) that the middleware shall be allowed to create/open (steps 3,4 and 15 in the diagram ''Service startup using upstart'').<br />
<br />
=== Background Infos ===<br />
<br />
Several ways exist to restrict resources on a linux system from a global level down to user/groups or even shells and the processes started by them.<br />
<br />
==== Sysctl ====<br />
Sysctl is used to modify kernel parameters at runtime. E.g. to set the maximum number of files<br />
<br />
$ sysctl -w fs.file-max=100000<br />
<br />
To permanently set them append to the main configuration file and reload the settings<br />
<br />
$ echo fs.file-max=100000 >> /etc/sysctl.conf<br />
$ sysctl -p<br />
<br />
More infos can be found via '''man sysctl'''<br />
<br />
==== Limits.conf ====<br />
<br />
Allows to restrict resources an a global, group or user level. E.g:<br />
<br />
$ cat /etc/security/limits.d/90-nproc.conf <br />
# Default limit for number of user's processes to prevent<br />
# accidental fork bombs.<br />
# See rhbz #432903 for reasoning.<br />
<br />
* soft nproc 1024<br />
<br />
From '''man limits.conf''':<br />
<blockquote><br />
''Also, please note that all limit settings are set per login. They are not global, nor are they permanent; existing only for the duration of the session.''<br />
</blockquote><br />
<br />
The limits per login are applied via the ''pam'' stack. See '''man pam''' and '''man pam_limits''' for more details. As those limits are bound to sessions they don't affect most daemons started by our supported init systems or init utils. Most state that they are ignored by design, see [https://bugs.launchpad.net/ubuntu/+source/upstart/+bug/938669/comments/4 upstart], [https://bugzilla.redhat.com/show_bug.cgi?id=754285#c1 systemd] and [https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=302079 start-stop-daemon] <br />
<br />
===== Ulimit =====<br />
From '''man bash'''<br />
<blockquote><br />
ulimit [-HSTabcdefilmnpqrstuvx [limit]]<br />
Provides control over the resources available to the shell and to processes started by it, on systems that allow such control.<br />
</blockquote><br />
This is what we use in our System V compatible init scripts to increase resources for the open-xchange process across multiple distros. Currently only ''the maximum number of processes'' and ''the maximum number of open file descriptors available to a single user'' are increased via ulimit. The values are specified in '''/opt/open-xchange/ox-scriptconf.sh'''<br />
<br />
==== Systemd ====<br />
<br />
===== Control Groups=====<br />
<br />
Control groups should only affect the OX middleware if you create/manage them yourself of if you are using a ''modern'' distribution that already uses systemd as init.<br />
<br />
Citing from the kernel cgroup [https://www.kernel.org/doc/Documentation/cgroup-v2.txt documentation]:<br />
<br />
<blockquote><br />
1-2. What is cgroup?<br />
<br />
cgroup is a mechanism to organize processes hierarchically and<br />
distribute system resources along the hierarchy in a controlled and<br />
configurable manner.<br />
<br />
cgroup is largely composed of two parts - the core and controllers.<br />
cgroup core is primarily responsible for hierarchically organizing<br />
processes. A cgroup controller is usually responsible for<br />
distributing a specific type of system resource along the hierarchy<br />
although there are utility controllers which serve purposes other than<br />
resource distribution.<br />
<br />
cgroups form a tree structure and every process in the system belongs<br />
to one and only one cgroup. All threads of a process belong to the<br />
same cgroup. On creation, all processes are put in the cgroup that<br />
the parent process belongs to at the time. A process can be migrated<br />
to another cgroup. Migration of a process doesn't affect already<br />
existing descendant processes.<br />
<br />
Following certain structural constraints, controllers may be enabled or<br />
disabled selectively on a cgroup. All controller behaviors are<br />
hierarchical - if a controller is enabled on a cgroup, it affects all<br />
processes which belong to the cgroups consisting the inclusive<br />
sub-hierarchy of the cgroup. When a controller is enabled on a nested<br />
cgroup, it always restricts the resource distribution further. The<br />
restrictions set closer to the root in the hierarchy can not be<br />
overridden from further away.<br />
</blockquote><br />
<br />
So processes are organized into a tree structure of control groups and controllers are responsible for the distribution of resources. So what kind of controllers exist?<br />
<br />
<blockquote><br />
5. Controllers<br />
<br />
5-1. CPU<br />
<br />
The "cpu" controllers regulates distribution of CPU cycles. This<br />
controller implements weight and absolute bandwidth limit models for<br />
normal scheduling policy and absolute bandwidth allocation model for<br />
realtime scheduling policy.<br />
<br />
5-2. Memory<br />
<br />
The "memory" controller regulates distribution of memory.<br />
...<br />
While not completely water-tight, all major memory usages by a given<br />
cgroup are tracked so that the total memory consumption can be<br />
accounted and controlled to a reasonable extent.<br />
<br />
5-3. IO<br />
<br />
The "io" controller regulates the distribution of IO resources. This<br />
controller implements both weight based and absolute bandwidth or IOPS<br />
limit distribution; however, weight based distribution is available<br />
only if cfq-iosched is in use and neither scheme is available for<br />
blk-mq devices.<br />
</blockquote><br />
<br />
The open-xchange service is simply put into the default ''system.slice'' without applying further limits.<br />
<br />
singlenode$ systemd-cgls<br />
├─1 /sbin/init<br />
├─system.slice<br />
│ ├─avahi-daemon.service<br />
│ │ ├─501 avahi-daemon: running [singlenode]<br />
│ │ └─514 avahi-daemon: chroot helper<br />
│ ├─console-kit-daemon.service<br />
│ │ └─16164 /usr/sbin/console-kit-daemon --no-daemon<br />
│ ├─dbus.service<br />
│ │ └─508 /usr/bin/dbus-daemon --system --address=systemd: --nofork --nopidfile --systemd-activation<br />
│ ├─munin-node.service<br />
│ │ └─4290 /usr/bin/perl -wT /usr/sbin/munin-node<br />
│ ├─open-xchange.service<br />
│ │ └─6037 /usr/bin/java -Dsun.net.inetaddr.ttl=3600 -Dnetworkaddress.cache.ttl=3600 -Dnetworkaddress.cache.negative.ttl=10 ...<br />
<br />
To check all the details use<br />
<br />
singlenode:~ # systemctl show system.slice<br />
<br />
==== Limits besides control groups ====<br />
Besides control groups systemd allows you to apply other limits to the execution environment of your service. Here we can apply the limits that would normally be applied via ''limits.conf'' or ''ulimit''. Systemd uses [http://man7.org/linux/man-pages/man2/setrlimit.2.html setrlimit] for this. The options that we set by default are:<br />
<br />
* LimitNOFILE<br />
* LimitNPROC<br />
<br />
which you have already seen when we took a look at the [[#Service_definiton|service definition]] and [[#Drop-in_config|drop-in configs]]<br />
<br />
== Verify limits ==<br />
<br />
=== System V ===<br />
<br />
Get the proper pid of the java process started for the middleware by either checking '''ps faux'''<br />
root 4103 0.0 0.0 66528 1868 pts/0 S 22:05 0:00 su -s /bin/bash open-xchange -c /opt/open-xchange/sbin/open-xchange<br />
494 4109 9.5 19.8 2224644 381312 ? Ssl 22:05 1:12 \_ /usr/bin/java -Dsun.net.inetaddr.ttl=3600 -Dnetworkaddress.cache.ttl=3600<br />
<br />
or programmatically<br />
singlenode:~ # pid=$(pgrep -P $(</var/run/open-xchange.pid) java)<br />
<br />
and check the limits applied for that process<br />
singlenode:~ # cat /proc/$pid/limits<br />
Limit Soft Limit Hard Limit Units<br />
Max cpu time unlimited unlimited seconds<br />
Max file size unlimited unlimited bytes<br />
Max data size unlimited unlimited bytes<br />
Max stack size 8388608 unlimited bytes<br />
Max core file size 0 unlimited bytes<br />
Max resident set unlimited unlimited bytes<br />
Max processes 65536 65536 processes<br />
Max open files 65536 65536 files<br />
Max locked memory 65536 65536 bytes<br />
Max address space unlimited unlimited bytes<br />
Max file locks unlimited unlimited locks<br />
Max pending signals 24254 24254 signals<br />
Max msgqueue size 819200 819200 bytes<br />
Max nice priority 0 0<br />
Max realtime priority 0 0<br />
Max realtime timeout unlimited unlimited us<br />
<br />
=== Systemd ===<br />
singlenode:~ # systemctl show open-xchange | grep Limit<br />
StartLimitInterval=10000000<br />
StartLimitBurst=5<br />
StartLimitAction=none<br />
MemoryLimit=18446744073709551615<br />
LimitCPU=18446744073709551615<br />
LimitFSIZE=18446744073709551615<br />
LimitDATA=18446744073709551615<br />
LimitSTACK=18446744073709551615<br />
LimitCORE=18446744073709551615<br />
LimitRSS=18446744073709551615<br />
LimitNOFILE=65536<br />
LimitAS=18446744073709551615<br />
LimitNPROC=65536<br />
LimitMEMLOCK=65536<br />
LimitLOCKS=18446744073709551615<br />
LimitSIGPENDING=19827<br />
LimitMSGQUEUE=819200<br />
LimitNICE=0<br />
LimitRTPRIO=0<br />
LimitRTTIME=18446744073709551615<br />
<br />
== Evaluate JVM commandline options ==<br />
<br />
Before starting Java and the OSGi framework with all the Open-Xchange middleware bundles we have to evaluate several important JVM commandline options that influence the performance and stability of the Open-Xchange middleware stack. These commandline options are specified in '''/opt/open-xchange/etc/ox-scriptconf.sh'''<br />
<br />
=== Format < 7.8.4 ===<br />
For version up to 7.8.3 the JVM configuration options were all specified via a single property '''JAVA_XTRAOPTS''' which resulted in a rather long configuration string that was hard to read and maintain.<br />
<br />
JAVA_XTRAOPTS="-Dsun.net.inetaddr.ttl=3600 -Dnetworkaddress.cache.ttl=3600 -Dnetworkaddress.cache.negative.ttl=10 \<br />
-Dlogback.threadlocal.put.duplicate=false -server -Djava.awt.headless=true -Xmx512M -XX:MaxPermSize=256m -XX:+UseConcMarkSweepGC \<br />
-XX:+UseParNewGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:NewRatio=3 -XX:+UseTLAB \<br />
-Dosgi.compatibility.bootdelegation=false -XX:-OmitStackTraceInFastThrow"<br />
<br />
During the startup of the open-xchange service (step 11 of diagram Service startup using systemd and step 10 of diagram Service startup using upstart) this JAVA_XTRAOPTS string goes through a basic validity check and is then passed to the JVM.<br />
<br />
=== Format >= 7.8.4 ===<br />
Beginning from version 7.8.4 the options are split up and sorted into different categories like:<br />
<br />
JAVA_OPTS_GC="-XX:+UseConcMarkSweepGC -XX:+UseParNewGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:NewRatio=3 -XX:+DisableExplicitGC"<br />
JAVA_OPTS_LOG="-Dlogback.threadlocal.put.duplicate=false -XX:-OmitStackTraceInFastThrow"<br />
JAVA_OPTS_MEM="-XX:MaxHeapSize=512M -XX:MaxPermSize=256M -XX:+UseTLAB"<br />
JAVA_OPTS_NET="-Dsun.net.inetaddr.ttl=3600 -Dnetworkaddress.cache.ttl=3600 -Dnetworkaddress.cache.negative.ttl=10"<br />
JAVA_OPTS_OSGI="-Dosgi.compatibility.bootdelegation=false"<br />
JAVA_OPTS_SERVER="-server -Djava.awt.headless=true"<br />
<br />
JAVA_OPTS_OTHER=""<br />
<br />
#JAVA_OPTS_DEBUG="-XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/mnt/heapdump -Xloggc:/var/log/open-xchange/gc.log -verbose:gc -XX:+PrintGCDateStamps \<br />
-XX:+PrintHeapAtGC -XX:+PrintGCApplicationStoppedTime -XX:+PrintTenuringDistribution"<br />
<br />
which is much easier to read an maintain. Furthermore we added a useful '''JAVA_OPTS_DEBUG''' configuration parameter that comes preconfigured but unused/commented so that customers can quickly enable the most commonly used debugging options in case they have to provide detailed system information to Open-Xchange.<br />
<br />
During the startup of the open-xchange service (step 11 of diagram ''Service startup using systemd'' and step 10 of diagram ''Service startup using upstart'') all uncommented JAVA_OPTS_* categories will then be evaluated and combined again into a single string of commandline options that can then be passed to the JVM.<br />
<br />
=== Upgrade procedure ===<br />
During the upgrade '''7.8.3-revX -> 7.8.4-revY''' existing JAVA_XTRAOPTS will automatically be split up and migrated to the new categories approach. A backup of the file (ox-scriptconf.sh.timestamp) will be created and admins are urged to check for proper migration before deleting the backup file like it's already done with *.dpkg-dist or *.rpmnew files on Debian or RHEL/SLES distributions.<br />
<br />
== Encoding detection during JVM startup ==<br />
<br />
=== Background Infos ===<br />
<br />
From ''man 7 locale''<br />
<br />
DESCRIPTION<br />
A locale is a set of language and cultural rules. These cover aspects such as lan‐<br />
guage for messages, different character sets, lexicographic conventions, and so on.<br />
A program needs to be able to determine its locale and act accordingly to be portable<br />
to different cultures.<br />
<br />
The format to specify a system locale is: '''''language[_territory][.codeset][@modifier]'''''.<br />
<br />
Where all of<br />
<br />
* en_US.UTF-8<br />
* en_US.utf8<br />
* en_US.utf-8<br />
* en_US.UTF_8 <br />
<br />
will result in a valid UTF-8 charmap for the english language with locale categories configured as<br />
<br />
LANG=en_US.UTF-8<br />
LC_CTYPE="en_US.UTF-8"<br />
LC_NUMERIC="en_US.UTF-8"<br />
LC_TIME="en_US.UTF-8"<br />
LC_COLLATE="en_US.UTF-8"<br />
LC_MONETARY="en_US.UTF-8"<br />
LC_MESSAGES="en_US.UTF-8"<br />
LC_PAPER="en_US.UTF-8"<br />
LC_NAME="en_US.UTF-8"<br />
LC_ADDRESS="en_US.UTF-8"<br />
LC_TELEPHONE="en_US.UTF-8"<br />
LC_MEASUREMENT="en_US.UTF-8"<br />
LC_IDENTIFICATION="en_US.UTF-8"<br />
LC_ALL=<br />
<br />
See for more infos: locale(1), locale(7), [https://en.wikipedia.org/wiki/List_of_ISO_639-1_codes language codes], [https://en.wikipedia.org/wiki/ISO_3166-1#Current_codes country codes] and [https://en.wikipedia.org/wiki/Character_encoding character encodings]<br />
<br />
==== JVM startup ====<br />
<br />
The default file.encoding detection is done at JVM startup based upon LC_CTYPE which makes sense as it's documented as:<br />
<br />
CODESET (LC_CTYPE)<br />
Return a string with the name of the character encoding used in the selected locale, such as "UTF-8", "ISO-8859-1",<br />
or "ANSI_X3.4-1968" (better known as US-ASCII). This is the same string that you get with "locale charmap".<br />
For a list of character encoding names, try "locale -m", cf. locale(1).<br />
<br />
Furthermore this influences the defaultCharset selection of the JVM.<br />
<br />
/**<br />
* Returns the default charset of this Java virtual machine.<br />
*<br />
* The default charset is determined during virtual-machine startup and<br />
* typically depends upon the locale and charset of the underlying<br />
* operating system.<br />
*<br />
* @return A charset object for the default charset<br />
*<br />
* @since 1.5<br />
*/<br />
public static Charset defaultCharset() {<br />
if (defaultCharset == null) {<br />
synchronized (Charset.class) {<br />
String csn = AccessController.doPrivileged(<br />
new GetPropertyAction("file.encoding"));<br />
Charset cs = lookup(csn);<br />
if (cs != null)<br />
defaultCharset = cs;<br />
else<br />
defaultCharset = forName("UTF-8");<br />
}<br />
}<br />
return defaultCharset;<br />
}<br />
<br />
Although we try to specify valid unicode encodings for every IO operations the Open-Xchange middleware does some parts of the JVM don't allow to specify the encoding to use and falls back to the detected default encoding. That's why admins have to make sure to provide a proper unicode environment for the middleware service during every start or use documented workarounds provided by Open-Xchange (e.g. )<br />
<br />
=== Query which file encoding was detected by the JVM during startup ===<br />
[[File:visualvm-file.encoding.png|thumb|Querying the JVM via visualvm|right|300px]]<br />
<br />
The file encoding that was detected by the JVM during startup can be queried via JMX. You can use graphical management tools like [https://visualvm.github.io/ visualvm], query our [[Jolokia]] JMX bridge via http or just use the commandline like<br />
<br />
/opt/open-xchange/sbin/showruntimestats -r | sed -ne 's/.*\({key=file.encoding, value=[^}]*}\).*/\1/p'<br />
{key=file.encoding, value=UTF-8}<br />
<br />
<br clear=all><br />
<br />
=== Locale passing from client to server via SSH ===<br />
<br />
The following setting of sshd allows clients to specify which locale to use on the server and might lead to some confusion or unexpected behaviour when checking the system locale or even to the usage of broken locales during middleware startup if your client sends misconfigured or missing locale configuration to the server.<br />
<br />
grep AcceptEnv /etc/ssh/sshd_config <br />
AcceptEnv LANG LC_*<br />
<br />
For example when i connect to the machine my local german locale is sent to the server <br />
<br />
[vagrant@showcase-community-centos-7 ~]$ locale -v <br />
LANG=de_DE.UTF-8<br />
LC_CTYPE="de_DE.UTF-8"<br />
LC_NUMERIC=de_DE.UTF-8<br />
LC_TIME=de_DE.UTF-8<br />
LC_COLLATE="de_DE.UTF-8"<br />
LC_MONETARY=de_DE.UTF-8<br />
LC_MESSAGES="de_DE.UTF-8"<br />
LC_PAPER=de_DE.UTF-8<br />
LC_NAME=de_DE.UTF-8<br />
LC_ADDRESS=de_DE.UTF-8<br />
LC_TELEPHONE=de_DE.UTF-8<br />
LC_MEASUREMENT=de_DE.UTF-8<br />
LC_IDENTIFICATION=de_DE.UTF-8<br />
LC_ALL=<br />
<br />
But after specifying that i want a new login shell the default locale configured for that user/system will be used instead!<br />
<br />
[root@showcase-community-centos-7 vagrant]# su -s /bin/bash open-xchange -l<br />
-bash-4.2$ locale -v <br />
LANG=en_US.UTF-8<br />
LC_CTYPE="en_US.UTF-8"<br />
LC_NUMERIC="en_US.UTF-8"<br />
LC_TIME="en_US.UTF-8"<br />
LC_COLLATE="en_US.UTF-8"<br />
LC_MONETARY="en_US.UTF-8"<br />
LC_MESSAGES="en_US.UTF-8"<br />
LC_PAPER="en_US.UTF-8"<br />
LC_NAME="en_US.UTF-8"<br />
LC_ADDRESS="en_US.UTF-8"<br />
LC_TELEPHONE="en_US.UTF-8"<br />
LC_MEASUREMENT="en_US.UTF-8"<br />
LC_IDENTIFICATION="en_US.UTF-8"<br />
LC_ALL=<br />
<br />
=== Verify that your system is configured with a proper unicode locale for OX ===<br />
<br />
Assuming you haven't reconfigured the locale of your root user<br />
<br />
i|marens@karotte ~ $ su -<br />
Password: <br />
i|karotte ~ # locale charmap<br />
UTF-8<br />
i|karotte ~ #<br />
<br />
== Warnings during early start ==<br />
<br />
Early warnings before the start of the JVM and related logging libraries like logback and logstash forwarding are redirected to '''/var/log/open-xchange/open-xchange-console.log'''.<br />
<br />
=== Limits ===<br />
If the open-xchange startup script isn't able to set the proper limits on CentOS 6 due to e.g. hard limits being enforced via pam_limits you'll see the following warning in the console log during startup<br />
<br />
singlenode open-xchange # cat open-xchange-console.log<br />
/opt/open-xchange/sbin/open-xchange: line 115: ulimit: max user processes: cannot modify limit: Operation not permitted<br />
<br />
=== Locale ===<br />
If the open-xchange startup script isn't able to detect a unicode locale you'll see a warning like the following (the current charmap might differ)<br />
<br />
WARNING: Couldn't detect proper unicode charmap in locale category LC_CTYPE.<br />
WARNING: Current charmap: "ANSI_X3.4-1968" might cause encoding problems.<br />
<br />
== Summary: Open-Xchange middleware on specific distros ==<br />
The support for the mentioned mechanism of resource control differ depending on the distribution and the init system in use.<br />
<br />
=== Debian 7 ===<br />
;Init<br />
: System V style<br />
;OX Configurable Limits/Defaults<br />
: nofile, nproc<br />
<br />
The mentioned limits can be configured via '''/opt/open-xchange/etc/ox-scriptconf.sh'''. The limits are applied via ulimit in the service's init script. The open-xchange service is finally started via start-stop-daemon which doesn't doesn't consider '''/etc/security/limits.*'''<br />
<br />
=== RHEL 6 / CentOS 6 ===<br />
;Init<br />
: Upstart, System V compatible<br />
;OX Configurable Limits/Defaults<br />
: nofile, nproc<br />
<br />
The mentioned limits can be configured via '''/opt/open-xchange/etc/ox-scriptconf.sh'''. The limits are applied via ulimit in the service's init script. Furthermore as the open-xchange service is finally started via '''su ... open-xchange''' on this distro a user session is opened via su/pam and the default CentOS pam config reads the '''/etc/security/limits.*''' configuration by loading the pam stack like:<br />
<br />
: '''/etc/pam.d/su'''<br />
:: '''-> /etc/pam.d/system-auth'''<br />
::: '''-> pam_limits.so'''<br />
<br />
If NPROC isn't configured for the open-xchange-server it's restricted to '''1024''' globally by default to prevent accidental fork bombs, see '''/etc/security/limits.d/90-nproc.conf''' which can result in severe problems modern multithreaded applications.<br />
<br />
=== RHEL 7 / CentOS 7 / Debian 8 / SLE 12 ===<br />
;Init<br />
: Systemd<br />
;OX Configurable Limits/Defaults<br />
: nofile, nproc<br />
<br />
For systemd the default limits are configured directly in the service's unit file that is shipped by OX and located at '''/usr/lib/systemd/system/open-xchange.service'''. The drop-in config to override or extend the default unit file is located at '''/etc/systemd/system/open-xchange.service.d/limits.conf'''. [https://www.freedesktop.org/software/systemd/man/systemd.exec.html Systemd.exec] shows a whole lot of options that can be used by admins to adapt the default service to their specific needs.</div>Marenshttps://oxpedia.org/wiki/index.php?title=Sandbox:Middleware_Startup/sandbox&diff=23230Sandbox:Middleware Startup/sandbox2017-04-24T20:11:02Z<p>Marens: /* Locale passing from client to server via SSH */</p>
<hr />
<div><br />
== Systemd ==<br />
<br />
==== Service definition ====<br />
The default service definition as shipped by Open-Xchange is rather concise and easy to read. You can check by looking at the service file that is installed to either /usr/lib/systemd or /lib/systemd depending on your distribution of choice.<br />
<br />
singlenode:~ # cat /usr/lib/systemd/system/open-xchange.service <br />
[Unit]<br />
After=remote-fs.target<br />
After=time-sync.target ypbind.service sendmail.service cyrus.service <br />
<br />
[Service]<br />
User=open-xchange<br />
PermissionsStartOnly=true<br />
TimeoutStartSec=0<br />
ExecStartPre=/opt/open-xchange/sbin/triggerupdatethemes -u<br />
ExecStart=/opt/open-xchange/sbin/open-xchange<br />
ExecStop=/opt/open-xchange/sbin/shutdown -w<br />
ExecReload=/opt/open-xchange/sbin/triggerreloadconfiguration -d<br />
KillMode=process<br />
LimitNOFILE=65536<br />
LimitNPROC=65536<br />
<br />
[Install]<br />
WantedBy=multi-user.target<br />
<br />
==== Drop-in configs ====<br />
Drop in configs allow administrators to easily adapt and override default service unit files. So if you want to change the default limits or add additional limits have a look at our default example drop-in config which is located at /etc/systemd/system/open-xchange.service.d/limits.conf<br />
<br />
singlenode:~ # cat /etc/systemd/system/open-xchange.service.d/limits.conf <br />
# Override and add options in this file<br />
# See systemd.exec(5) for other limits<br />
<br />
[Service]<br />
#LimitNPROC=65536<br />
#LimitNOFILE=65536<br />
<br />
The systemd.exec documentation shows a whole lot of options that can be used by admins to adapt the default service to their specific needs.<br />
<br />
<br clear=all><br />
[[File:mw_start_sysd.png|thumb|Service startup using systemd|left|1000px]]<br />
<br clear=all><br />
<br />
== System V / Upstart ==<br />
<br />
==== Service definition ====<br />
<br />
The default service definition as shipped by Open-Xchange can be found at /etc/init.d/open-xchange and will differ slightly depending on the distro in use.<br />
<br />
<br clear=all><br />
[[File:mw_start_sysv.png|thumb|Service startup using upstart|left|1000px]]<br />
<br clear=all><br />
== Apply Resource limits ==<br />
<br />
As you can see in the diagrams above one of the differences is that:<br />
<br />
* systemd reads all the infos wrt. resource limitations from the original service file and then applies possible further restrictions from the drop-in configs (steps 2 and 8 in the diagram ''Service startup using systemd'') and limits configured in /opt/open-xchange/etc/ox-scriptconf.sh aren't considered anylonger<br />
* the older System V style init has to source /opt/open-xchange/etc/ox-scriptconf.sh to be able to read the configs for the maximum number of processes (NPROC) and files(NRFILES) that the middleware shall be allowed to create/open (steps 3,4 and 15 in the diagram ''Service startup using upstart'').<br />
<br />
=== Background Infos ===<br />
<br />
Several ways exist to restrict resources on a linux system from a global level down to user/groups or even shells and the processes started by them.<br />
<br />
==== Sysctl ====<br />
Sysctl is used to modify kernel parameters at runtime. E.g. to set the maximum number of files<br />
<br />
$ sysctl -w fs.file-max=100000<br />
<br />
To permanently set them append to the main configuration file and reload the settings<br />
<br />
$ echo fs.file-max=100000 >> /etc/sysctl.conf<br />
$ sysctl -p<br />
<br />
More infos can be found via '''man sysctl'''<br />
<br />
==== Limits.conf ====<br />
<br />
Allows to restrict resources an a global, group or user level. E.g:<br />
<br />
$ cat /etc/security/limits.d/90-nproc.conf <br />
# Default limit for number of user's processes to prevent<br />
# accidental fork bombs.<br />
# See rhbz #432903 for reasoning.<br />
<br />
* soft nproc 1024<br />
<br />
From '''man limits.conf''':<br />
<blockquote><br />
''Also, please note that all limit settings are set per login. They are not global, nor are they permanent; existing only for the duration of the session.''<br />
</blockquote><br />
<br />
The limits per login are applied via the ''pam'' stack. See '''man pam''' and '''man pam_limits''' for more details. As those limits are bound to sessions they don't affect most daemons started by our supported init systems or init utils. Most state that they are ignored by design, see [https://bugs.launchpad.net/ubuntu/+source/upstart/+bug/938669/comments/4 upstart], [https://bugzilla.redhat.com/show_bug.cgi?id=754285#c1 systemd] and [https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=302079 start-stop-daemon] <br />
<br />
===== Ulimit =====<br />
From '''man bash'''<br />
<blockquote><br />
ulimit [-HSTabcdefilmnpqrstuvx [limit]]<br />
Provides control over the resources available to the shell and to processes started by it, on systems that allow such control.<br />
</blockquote><br />
This is what we use in our System V compatible init scripts to increase resources for the open-xchange process across multiple distros. Currently only ''the maximum number of processes'' and ''the maximum number of open file descriptors available to a single user'' are increased via ulimit. The values are specified in '''/opt/open-xchange/ox-scriptconf.sh'''<br />
<br />
==== Systemd ====<br />
<br />
===== Control Groups=====<br />
<br />
Control groups should only affect the OX middleware if you create/manage them yourself of if you are using a ''modern'' distribution that already uses systemd as init.<br />
<br />
Citing from the kernel cgroup [https://www.kernel.org/doc/Documentation/cgroup-v2.txt documentation]:<br />
<br />
<blockquote><br />
1-2. What is cgroup?<br />
<br />
cgroup is a mechanism to organize processes hierarchically and<br />
distribute system resources along the hierarchy in a controlled and<br />
configurable manner.<br />
<br />
cgroup is largely composed of two parts - the core and controllers.<br />
cgroup core is primarily responsible for hierarchically organizing<br />
processes. A cgroup controller is usually responsible for<br />
distributing a specific type of system resource along the hierarchy<br />
although there are utility controllers which serve purposes other than<br />
resource distribution.<br />
<br />
cgroups form a tree structure and every process in the system belongs<br />
to one and only one cgroup. All threads of a process belong to the<br />
same cgroup. On creation, all processes are put in the cgroup that<br />
the parent process belongs to at the time. A process can be migrated<br />
to another cgroup. Migration of a process doesn't affect already<br />
existing descendant processes.<br />
<br />
Following certain structural constraints, controllers may be enabled or<br />
disabled selectively on a cgroup. All controller behaviors are<br />
hierarchical - if a controller is enabled on a cgroup, it affects all<br />
processes which belong to the cgroups consisting the inclusive<br />
sub-hierarchy of the cgroup. When a controller is enabled on a nested<br />
cgroup, it always restricts the resource distribution further. The<br />
restrictions set closer to the root in the hierarchy can not be<br />
overridden from further away.<br />
</blockquote><br />
<br />
So processes are organized into a tree structure of control groups and controllers are responsible for the distribution of resources. So what kind of controllers exist?<br />
<br />
<blockquote><br />
5. Controllers<br />
<br />
5-1. CPU<br />
<br />
The "cpu" controllers regulates distribution of CPU cycles. This<br />
controller implements weight and absolute bandwidth limit models for<br />
normal scheduling policy and absolute bandwidth allocation model for<br />
realtime scheduling policy.<br />
<br />
5-2. Memory<br />
<br />
The "memory" controller regulates distribution of memory.<br />
...<br />
While not completely water-tight, all major memory usages by a given<br />
cgroup are tracked so that the total memory consumption can be<br />
accounted and controlled to a reasonable extent.<br />
<br />
5-3. IO<br />
<br />
The "io" controller regulates the distribution of IO resources. This<br />
controller implements both weight based and absolute bandwidth or IOPS<br />
limit distribution; however, weight based distribution is available<br />
only if cfq-iosched is in use and neither scheme is available for<br />
blk-mq devices.<br />
</blockquote><br />
<br />
The open-xchange service is simply put into the default ''system.slice'' without applying further limits.<br />
<br />
singlenode$ systemd-cgls<br />
├─1 /sbin/init<br />
├─system.slice<br />
│ ├─avahi-daemon.service<br />
│ │ ├─501 avahi-daemon: running [singlenode]<br />
│ │ └─514 avahi-daemon: chroot helper<br />
│ ├─console-kit-daemon.service<br />
│ │ └─16164 /usr/sbin/console-kit-daemon --no-daemon<br />
│ ├─dbus.service<br />
│ │ └─508 /usr/bin/dbus-daemon --system --address=systemd: --nofork --nopidfile --systemd-activation<br />
│ ├─munin-node.service<br />
│ │ └─4290 /usr/bin/perl -wT /usr/sbin/munin-node<br />
│ ├─open-xchange.service<br />
│ │ └─6037 /usr/bin/java -Dsun.net.inetaddr.ttl=3600 -Dnetworkaddress.cache.ttl=3600 -Dnetworkaddress.cache.negative.ttl=10 ...<br />
<br />
To check all the details use<br />
<br />
singlenode:~ # systemctl show system.slice<br />
<br />
==== Limits besides control groups ====<br />
Besides control groups systemd allows you to apply other limits to the execution environment of your service. Here we can apply the limits that would normally be applied via ''limits.conf'' or ''ulimit''. Systemd uses [http://man7.org/linux/man-pages/man2/setrlimit.2.html setrlimit] for this. The options that we set by default are:<br />
<br />
* LimitNOFILE<br />
* LimitNPROC<br />
<br />
which you have already seen when we took a look at the [[#Service_definiton|service definition]] and [[#Drop-in_config|drop-in configs]]<br />
<br />
== Verify limits ==<br />
<br />
=== System V ===<br />
<br />
Get the proper pid of the java process started for the middleware by either checking '''ps faux'''<br />
root 4103 0.0 0.0 66528 1868 pts/0 S 22:05 0:00 su -s /bin/bash open-xchange -c /opt/open-xchange/sbin/open-xchange<br />
494 4109 9.5 19.8 2224644 381312 ? Ssl 22:05 1:12 \_ /usr/bin/java -Dsun.net.inetaddr.ttl=3600 -Dnetworkaddress.cache.ttl=3600<br />
<br />
or programmatically<br />
singlenode:~ # pid=$(pgrep -P $(</var/run/open-xchange.pid) java)<br />
<br />
and check the limits applied for that process<br />
singlenode:~ # cat /proc/$pid/limits<br />
Limit Soft Limit Hard Limit Units<br />
Max cpu time unlimited unlimited seconds<br />
Max file size unlimited unlimited bytes<br />
Max data size unlimited unlimited bytes<br />
Max stack size 8388608 unlimited bytes<br />
Max core file size 0 unlimited bytes<br />
Max resident set unlimited unlimited bytes<br />
Max processes 65536 65536 processes<br />
Max open files 65536 65536 files<br />
Max locked memory 65536 65536 bytes<br />
Max address space unlimited unlimited bytes<br />
Max file locks unlimited unlimited locks<br />
Max pending signals 24254 24254 signals<br />
Max msgqueue size 819200 819200 bytes<br />
Max nice priority 0 0<br />
Max realtime priority 0 0<br />
Max realtime timeout unlimited unlimited us<br />
<br />
=== Systemd ===<br />
singlenode:~ # systemctl show open-xchange | grep Limit<br />
StartLimitInterval=10000000<br />
StartLimitBurst=5<br />
StartLimitAction=none<br />
MemoryLimit=18446744073709551615<br />
LimitCPU=18446744073709551615<br />
LimitFSIZE=18446744073709551615<br />
LimitDATA=18446744073709551615<br />
LimitSTACK=18446744073709551615<br />
LimitCORE=18446744073709551615<br />
LimitRSS=18446744073709551615<br />
LimitNOFILE=65536<br />
LimitAS=18446744073709551615<br />
LimitNPROC=65536<br />
LimitMEMLOCK=65536<br />
LimitLOCKS=18446744073709551615<br />
LimitSIGPENDING=19827<br />
LimitMSGQUEUE=819200<br />
LimitNICE=0<br />
LimitRTPRIO=0<br />
LimitRTTIME=18446744073709551615<br />
<br />
== Evaluate JVM commandline options ==<br />
<br />
Before starting Java and the OSGi framework with all the Open-Xchange middleware bundles we have to evaluate several important JVM commandline options that influence the performance and stability of the Open-Xchange middleware stack. These commandline options are specified in '''/opt/open-xchange/etc/ox-scriptconf.sh'''<br />
<br />
=== Format < 7.8.4 ===<br />
For version up to 7.8.3 the JVM configuration options were all specified via a single property '''JAVA_XTRAOPTS''' which resulted in a rather long configuration string that was hard to read and maintain.<br />
<br />
JAVA_XTRAOPTS="-Dsun.net.inetaddr.ttl=3600 -Dnetworkaddress.cache.ttl=3600 -Dnetworkaddress.cache.negative.ttl=10 \<br />
-Dlogback.threadlocal.put.duplicate=false -server -Djava.awt.headless=true -Xmx512M -XX:MaxPermSize=256m -XX:+UseConcMarkSweepGC \<br />
-XX:+UseParNewGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:NewRatio=3 -XX:+UseTLAB \<br />
-Dosgi.compatibility.bootdelegation=false -XX:-OmitStackTraceInFastThrow"<br />
<br />
During the startup of the open-xchange service (step 11 of diagram Service startup using systemd and step 10 of diagram Service startup using upstart) this JAVA_XTRAOPTS string goes through a basic validity check and is then passed to the JVM.<br />
<br />
=== Format >= 7.8.4 ===<br />
Beginning from version 7.8.4 the options are split up and sorted into different categories like:<br />
<br />
JAVA_OPTS_GC="-XX:+UseConcMarkSweepGC -XX:+UseParNewGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:NewRatio=3 -XX:+DisableExplicitGC"<br />
JAVA_OPTS_LOG="-Dlogback.threadlocal.put.duplicate=false -XX:-OmitStackTraceInFastThrow"<br />
JAVA_OPTS_MEM="-XX:MaxHeapSize=512M -XX:MaxPermSize=256M -XX:+UseTLAB"<br />
JAVA_OPTS_NET="-Dsun.net.inetaddr.ttl=3600 -Dnetworkaddress.cache.ttl=3600 -Dnetworkaddress.cache.negative.ttl=10"<br />
JAVA_OPTS_OSGI="-Dosgi.compatibility.bootdelegation=false"<br />
JAVA_OPTS_SERVER="-server -Djava.awt.headless=true"<br />
<br />
JAVA_OPTS_OTHER=""<br />
<br />
#JAVA_OPTS_DEBUG="-XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/mnt/heapdump -Xloggc:/var/log/open-xchange/gc.log -verbose:gc -XX:+PrintGCDateStamps \<br />
-XX:+PrintHeapAtGC -XX:+PrintGCApplicationStoppedTime -XX:+PrintTenuringDistribution"<br />
<br />
which is much easier to read an maintain. Furthermore we added a useful '''JAVA_OPTS_DEBUG''' configuration parameter that comes preconfigured but unused/commented so that customers can quickly enable the most commonly used debugging options in case they have to provide detailed system information to Open-Xchange.<br />
<br />
During the startup of the open-xchange service (step 11 of diagram ''Service startup using systemd'' and step 10 of diagram ''Service startup using upstart'') all uncommented JAVA_OPTS_* categories will then be evaluated and combined again into a single string of commandline options that can then be passed to the JVM.<br />
<br />
=== Upgrade procedure ===<br />
During the upgrade '''7.8.3-revX -> 7.8.4-revY''' existing JAVA_XTRAOPTS will automatically be split up and migrated to the new categories approach. A backup of the file (ox-scriptconf.sh.timestamp) will be created and admins are urged to check for proper migration before deleting the backup file like it's already done with *.dpkg-dist or *.rpmnew files on Debian or RHEL/SLES distributions.<br />
<br />
== Encoding detection during JVM startup ==<br />
<br />
=== Background Infos ===<br />
All of en_US.UTF-8, .utf8, .utf-8, .UTF_8 will result in a valid UTF-8 charmap<br />
<br />
==== JVM startup ====<br />
<br />
The default file.encoding detection is done at JVM startup based upon LC_CTYPE which makes sense as it's documented as:<br />
<br />
CODESET (LC_CTYPE)<br />
Return a string with the name of the character encoding used in the selected locale, such as "UTF-8", "ISO-8859-1",<br />
or "ANSI_X3.4-1968" (better known as US-ASCII). This is the same string that you get with "locale charmap".<br />
For a list of character encoding names, try "locale -m", cf. locale(1).<br />
<br />
Furthermore this influences the defaultCharset selection of the JVM.<br />
<br />
/**<br />
* Returns the default charset of this Java virtual machine.<br />
*<br />
* The default charset is determined during virtual-machine startup and<br />
* typically depends upon the locale and charset of the underlying<br />
* operating system.<br />
*<br />
* @return A charset object for the default charset<br />
*<br />
* @since 1.5<br />
*/<br />
public static Charset defaultCharset() {<br />
if (defaultCharset == null) {<br />
synchronized (Charset.class) {<br />
String csn = AccessController.doPrivileged(<br />
new GetPropertyAction("file.encoding"));<br />
Charset cs = lookup(csn);<br />
if (cs != null)<br />
defaultCharset = cs;<br />
else<br />
defaultCharset = forName("UTF-8");<br />
}<br />
}<br />
return defaultCharset;<br />
}<br />
<br />
Although we try to specify valid unicode encodings for every IO operations the Open-Xchange middleware does some parts of the JVM don't allow to specify the encoding to use and falls back to the detected default encoding. That's why admins have to make sure to provide a proper unicode environment for the middleware service during every start or use documented workarounds provided by Open-Xchange (e.g. )<br />
<br />
=== Query which file encoding was detected by the JVM during startup ===<br />
[[File:visualvm-file.encoding.png|thumb|Querying the JVM via visualvm|right|300px]]<br />
<br />
The file encoding that was detected by the JVM during startup can be queried via JMX. You can use graphical management tools like [https://visualvm.github.io/ visualvm], query our [[Jolokia]] JMX bridge via http or just use the commandline like<br />
<br />
/opt/open-xchange/sbin/showruntimestats -r | sed -ne 's/.*\({key=file.encoding, value=[^}]*}\).*/\1/p'<br />
{key=file.encoding, value=UTF-8}<br />
<br />
<br clear=all><br />
<br />
=== Locale passing from client to server via SSH ===<br />
<br />
The following setting of sshd allows clients to specify which locale to use on the server and might lead to some confusion or unexpected behaviour when checking the system locale or even to the usage of broken locales during middleware startup if your client sends misconfigured or missing locale configuration to the server.<br />
<br />
grep AcceptEnv /etc/ssh/sshd_config <br />
AcceptEnv LANG LC_*<br />
<br />
For example when i connect to the machine my local german locale is sent to the server <br />
<br />
[vagrant@showcase-community-centos-7 ~]$ locale -v <br />
LANG=de_DE.UTF-8<br />
LC_CTYPE="de_DE.UTF-8"<br />
LC_NUMERIC=de_DE.UTF-8<br />
LC_TIME=de_DE.UTF-8<br />
LC_COLLATE="de_DE.UTF-8"<br />
LC_MONETARY=de_DE.UTF-8<br />
LC_MESSAGES="de_DE.UTF-8"<br />
LC_PAPER=de_DE.UTF-8<br />
LC_NAME=de_DE.UTF-8<br />
LC_ADDRESS=de_DE.UTF-8<br />
LC_TELEPHONE=de_DE.UTF-8<br />
LC_MEASUREMENT=de_DE.UTF-8<br />
LC_IDENTIFICATION=de_DE.UTF-8<br />
LC_ALL=<br />
<br />
But after specifying that i want a new login shell the default locale configured for that user/system will be used instead!<br />
<br />
[root@showcase-community-centos-7 vagrant]# su -s /bin/bash open-xchange -l<br />
-bash-4.2$ locale -v <br />
LANG=en_US.UTF-8<br />
LC_CTYPE="en_US.UTF-8"<br />
LC_NUMERIC="en_US.UTF-8"<br />
LC_TIME="en_US.UTF-8"<br />
LC_COLLATE="en_US.UTF-8"<br />
LC_MONETARY="en_US.UTF-8"<br />
LC_MESSAGES="en_US.UTF-8"<br />
LC_PAPER="en_US.UTF-8"<br />
LC_NAME="en_US.UTF-8"<br />
LC_ADDRESS="en_US.UTF-8"<br />
LC_TELEPHONE="en_US.UTF-8"<br />
LC_MEASUREMENT="en_US.UTF-8"<br />
LC_IDENTIFICATION="en_US.UTF-8"<br />
LC_ALL=<br />
<br />
== Warnings during early start ==<br />
<br />
Early warnings before the start of the JVM and related logging libraries like logback and logstash forwarding are redirected to '''/var/log/open-xchange/open-xchange-console.log'''.<br />
<br />
=== Limits ===<br />
If the open-xchange startup script isn't able to set the proper limits on CentOS 6 due to e.g. hard limits being enforced via pam_limits you'll see the following warning in the console log during startup<br />
<br />
singlenode open-xchange # cat open-xchange-console.log<br />
/opt/open-xchange/sbin/open-xchange: line 115: ulimit: max user processes: cannot modify limit: Operation not permitted<br />
<br />
=== Locale ===<br />
If the open-xchange startup script isn't able to detect a unicode locale you'll see a warning like the following (the current charmap might differ)<br />
<br />
WARNING: Couldn't detect proper unicode charmap in locale category LC_CTYPE.<br />
WARNING: Current charmap: "ANSI_X3.4-1968" might cause encoding problems.<br />
<br />
== Summary: Open-Xchange middleware on specific distros ==<br />
The support for the mentioned mechanism of resource control differ depending on the distribution and the init system in use.<br />
<br />
=== Debian 7 ===<br />
;Init<br />
: System V style<br />
;OX Configurable Limits/Defaults<br />
: nofile, nproc<br />
<br />
The mentioned limits can be configured via '''/opt/open-xchange/etc/ox-scriptconf.sh'''. The limits are applied via ulimit in the service's init script. The open-xchange service is finally started via start-stop-daemon which doesn't doesn't consider '''/etc/security/limits.*'''<br />
<br />
=== RHEL 6 / CentOS 6 ===<br />
;Init<br />
: Upstart, System V compatible<br />
;OX Configurable Limits/Defaults<br />
: nofile, nproc<br />
<br />
The mentioned limits can be configured via '''/opt/open-xchange/etc/ox-scriptconf.sh'''. The limits are applied via ulimit in the service's init script. Furthermore as the open-xchange service is finally started via '''su ... open-xchange''' on this distro a user session is opened via su/pam and the default CentOS pam config reads the '''/etc/security/limits.*''' configuration by loading the pam stack like:<br />
<br />
: '''/etc/pam.d/su'''<br />
:: '''-> /etc/pam.d/system-auth'''<br />
::: '''-> pam_limits.so'''<br />
<br />
If NPROC isn't configured for the open-xchange-server it's restricted to '''1024''' globally by default to prevent accidental fork bombs, see '''/etc/security/limits.d/90-nproc.conf''' which can result in severe problems modern multithreaded applications.<br />
<br />
=== RHEL 7 / CentOS 7 / Debian 8 / SLE 12 ===<br />
;Init<br />
: Systemd<br />
;OX Configurable Limits/Defaults<br />
: nofile, nproc<br />
<br />
For systemd the default limits are configured directly in the service's unit file that is shipped by OX and located at '''/usr/lib/systemd/system/open-xchange.service'''. The drop-in config to override or extend the default unit file is located at '''/etc/systemd/system/open-xchange.service.d/limits.conf'''. [https://www.freedesktop.org/software/systemd/man/systemd.exec.html Systemd.exec] shows a whole lot of options that can be used by admins to adapt the default service to their specific needs.</div>Marenshttps://oxpedia.org/wiki/index.php?title=Sandbox:Middleware_Startup/sandbox&diff=23229Sandbox:Middleware Startup/sandbox2017-04-24T20:10:14Z<p>Marens: </p>
<hr />
<div><br />
== Systemd ==<br />
<br />
==== Service definition ====<br />
The default service definition as shipped by Open-Xchange is rather concise and easy to read. You can check by looking at the service file that is installed to either /usr/lib/systemd or /lib/systemd depending on your distribution of choice.<br />
<br />
singlenode:~ # cat /usr/lib/systemd/system/open-xchange.service <br />
[Unit]<br />
After=remote-fs.target<br />
After=time-sync.target ypbind.service sendmail.service cyrus.service <br />
<br />
[Service]<br />
User=open-xchange<br />
PermissionsStartOnly=true<br />
TimeoutStartSec=0<br />
ExecStartPre=/opt/open-xchange/sbin/triggerupdatethemes -u<br />
ExecStart=/opt/open-xchange/sbin/open-xchange<br />
ExecStop=/opt/open-xchange/sbin/shutdown -w<br />
ExecReload=/opt/open-xchange/sbin/triggerreloadconfiguration -d<br />
KillMode=process<br />
LimitNOFILE=65536<br />
LimitNPROC=65536<br />
<br />
[Install]<br />
WantedBy=multi-user.target<br />
<br />
==== Drop-in configs ====<br />
Drop in configs allow administrators to easily adapt and override default service unit files. So if you want to change the default limits or add additional limits have a look at our default example drop-in config which is located at /etc/systemd/system/open-xchange.service.d/limits.conf<br />
<br />
singlenode:~ # cat /etc/systemd/system/open-xchange.service.d/limits.conf <br />
# Override and add options in this file<br />
# See systemd.exec(5) for other limits<br />
<br />
[Service]<br />
#LimitNPROC=65536<br />
#LimitNOFILE=65536<br />
<br />
The systemd.exec documentation shows a whole lot of options that can be used by admins to adapt the default service to their specific needs.<br />
<br />
<br clear=all><br />
[[File:mw_start_sysd.png|thumb|Service startup using systemd|left|1000px]]<br />
<br clear=all><br />
<br />
== System V / Upstart ==<br />
<br />
==== Service definition ====<br />
<br />
The default service definition as shipped by Open-Xchange can be found at /etc/init.d/open-xchange and will differ slightly depending on the distro in use.<br />
<br />
<br clear=all><br />
[[File:mw_start_sysv.png|thumb|Service startup using upstart|left|1000px]]<br />
<br clear=all><br />
== Apply Resource limits ==<br />
<br />
As you can see in the diagrams above one of the differences is that:<br />
<br />
* systemd reads all the infos wrt. resource limitations from the original service file and then applies possible further restrictions from the drop-in configs (steps 2 and 8 in the diagram ''Service startup using systemd'') and limits configured in /opt/open-xchange/etc/ox-scriptconf.sh aren't considered anylonger<br />
* the older System V style init has to source /opt/open-xchange/etc/ox-scriptconf.sh to be able to read the configs for the maximum number of processes (NPROC) and files(NRFILES) that the middleware shall be allowed to create/open (steps 3,4 and 15 in the diagram ''Service startup using upstart'').<br />
<br />
=== Background Infos ===<br />
<br />
Several ways exist to restrict resources on a linux system from a global level down to user/groups or even shells and the processes started by them.<br />
<br />
==== Sysctl ====<br />
Sysctl is used to modify kernel parameters at runtime. E.g. to set the maximum number of files<br />
<br />
$ sysctl -w fs.file-max=100000<br />
<br />
To permanently set them append to the main configuration file and reload the settings<br />
<br />
$ echo fs.file-max=100000 >> /etc/sysctl.conf<br />
$ sysctl -p<br />
<br />
More infos can be found via '''man sysctl'''<br />
<br />
==== Limits.conf ====<br />
<br />
Allows to restrict resources an a global, group or user level. E.g:<br />
<br />
$ cat /etc/security/limits.d/90-nproc.conf <br />
# Default limit for number of user's processes to prevent<br />
# accidental fork bombs.<br />
# See rhbz #432903 for reasoning.<br />
<br />
* soft nproc 1024<br />
<br />
From '''man limits.conf''':<br />
<blockquote><br />
''Also, please note that all limit settings are set per login. They are not global, nor are they permanent; existing only for the duration of the session.''<br />
</blockquote><br />
<br />
The limits per login are applied via the ''pam'' stack. See '''man pam''' and '''man pam_limits''' for more details. As those limits are bound to sessions they don't affect most daemons started by our supported init systems or init utils. Most state that they are ignored by design, see [https://bugs.launchpad.net/ubuntu/+source/upstart/+bug/938669/comments/4 upstart], [https://bugzilla.redhat.com/show_bug.cgi?id=754285#c1 systemd] and [https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=302079 start-stop-daemon] <br />
<br />
===== Ulimit =====<br />
From '''man bash'''<br />
<blockquote><br />
ulimit [-HSTabcdefilmnpqrstuvx [limit]]<br />
Provides control over the resources available to the shell and to processes started by it, on systems that allow such control.<br />
</blockquote><br />
This is what we use in our System V compatible init scripts to increase resources for the open-xchange process across multiple distros. Currently only ''the maximum number of processes'' and ''the maximum number of open file descriptors available to a single user'' are increased via ulimit. The values are specified in '''/opt/open-xchange/ox-scriptconf.sh'''<br />
<br />
==== Systemd ====<br />
<br />
===== Control Groups=====<br />
<br />
Control groups should only affect the OX middleware if you create/manage them yourself of if you are using a ''modern'' distribution that already uses systemd as init.<br />
<br />
Citing from the kernel cgroup [https://www.kernel.org/doc/Documentation/cgroup-v2.txt documentation]:<br />
<br />
<blockquote><br />
1-2. What is cgroup?<br />
<br />
cgroup is a mechanism to organize processes hierarchically and<br />
distribute system resources along the hierarchy in a controlled and<br />
configurable manner.<br />
<br />
cgroup is largely composed of two parts - the core and controllers.<br />
cgroup core is primarily responsible for hierarchically organizing<br />
processes. A cgroup controller is usually responsible for<br />
distributing a specific type of system resource along the hierarchy<br />
although there are utility controllers which serve purposes other than<br />
resource distribution.<br />
<br />
cgroups form a tree structure and every process in the system belongs<br />
to one and only one cgroup. All threads of a process belong to the<br />
same cgroup. On creation, all processes are put in the cgroup that<br />
the parent process belongs to at the time. A process can be migrated<br />
to another cgroup. Migration of a process doesn't affect already<br />
existing descendant processes.<br />
<br />
Following certain structural constraints, controllers may be enabled or<br />
disabled selectively on a cgroup. All controller behaviors are<br />
hierarchical - if a controller is enabled on a cgroup, it affects all<br />
processes which belong to the cgroups consisting the inclusive<br />
sub-hierarchy of the cgroup. When a controller is enabled on a nested<br />
cgroup, it always restricts the resource distribution further. The<br />
restrictions set closer to the root in the hierarchy can not be<br />
overridden from further away.<br />
</blockquote><br />
<br />
So processes are organized into a tree structure of control groups and controllers are responsible for the distribution of resources. So what kind of controllers exist?<br />
<br />
<blockquote><br />
5. Controllers<br />
<br />
5-1. CPU<br />
<br />
The "cpu" controllers regulates distribution of CPU cycles. This<br />
controller implements weight and absolute bandwidth limit models for<br />
normal scheduling policy and absolute bandwidth allocation model for<br />
realtime scheduling policy.<br />
<br />
5-2. Memory<br />
<br />
The "memory" controller regulates distribution of memory.<br />
...<br />
While not completely water-tight, all major memory usages by a given<br />
cgroup are tracked so that the total memory consumption can be<br />
accounted and controlled to a reasonable extent.<br />
<br />
5-3. IO<br />
<br />
The "io" controller regulates the distribution of IO resources. This<br />
controller implements both weight based and absolute bandwidth or IOPS<br />
limit distribution; however, weight based distribution is available<br />
only if cfq-iosched is in use and neither scheme is available for<br />
blk-mq devices.<br />
</blockquote><br />
<br />
The open-xchange service is simply put into the default ''system.slice'' without applying further limits.<br />
<br />
singlenode$ systemd-cgls<br />
├─1 /sbin/init<br />
├─system.slice<br />
│ ├─avahi-daemon.service<br />
│ │ ├─501 avahi-daemon: running [singlenode]<br />
│ │ └─514 avahi-daemon: chroot helper<br />
│ ├─console-kit-daemon.service<br />
│ │ └─16164 /usr/sbin/console-kit-daemon --no-daemon<br />
│ ├─dbus.service<br />
│ │ └─508 /usr/bin/dbus-daemon --system --address=systemd: --nofork --nopidfile --systemd-activation<br />
│ ├─munin-node.service<br />
│ │ └─4290 /usr/bin/perl -wT /usr/sbin/munin-node<br />
│ ├─open-xchange.service<br />
│ │ └─6037 /usr/bin/java -Dsun.net.inetaddr.ttl=3600 -Dnetworkaddress.cache.ttl=3600 -Dnetworkaddress.cache.negative.ttl=10 ...<br />
<br />
To check all the details use<br />
<br />
singlenode:~ # systemctl show system.slice<br />
<br />
==== Limits besides control groups ====<br />
Besides control groups systemd allows you to apply other limits to the execution environment of your service. Here we can apply the limits that would normally be applied via ''limits.conf'' or ''ulimit''. Systemd uses [http://man7.org/linux/man-pages/man2/setrlimit.2.html setrlimit] for this. The options that we set by default are:<br />
<br />
* LimitNOFILE<br />
* LimitNPROC<br />
<br />
which you have already seen when we took a look at the [[#Service_definiton|service definition]] and [[#Drop-in_config|drop-in configs]]<br />
<br />
== Verify limits ==<br />
<br />
=== System V ===<br />
<br />
Get the proper pid of the java process started for the middleware by either checking '''ps faux'''<br />
root 4103 0.0 0.0 66528 1868 pts/0 S 22:05 0:00 su -s /bin/bash open-xchange -c /opt/open-xchange/sbin/open-xchange<br />
494 4109 9.5 19.8 2224644 381312 ? Ssl 22:05 1:12 \_ /usr/bin/java -Dsun.net.inetaddr.ttl=3600 -Dnetworkaddress.cache.ttl=3600<br />
<br />
or programmatically<br />
singlenode:~ # pid=$(pgrep -P $(</var/run/open-xchange.pid) java)<br />
<br />
and check the limits applied for that process<br />
singlenode:~ # cat /proc/$pid/limits<br />
Limit Soft Limit Hard Limit Units<br />
Max cpu time unlimited unlimited seconds<br />
Max file size unlimited unlimited bytes<br />
Max data size unlimited unlimited bytes<br />
Max stack size 8388608 unlimited bytes<br />
Max core file size 0 unlimited bytes<br />
Max resident set unlimited unlimited bytes<br />
Max processes 65536 65536 processes<br />
Max open files 65536 65536 files<br />
Max locked memory 65536 65536 bytes<br />
Max address space unlimited unlimited bytes<br />
Max file locks unlimited unlimited locks<br />
Max pending signals 24254 24254 signals<br />
Max msgqueue size 819200 819200 bytes<br />
Max nice priority 0 0<br />
Max realtime priority 0 0<br />
Max realtime timeout unlimited unlimited us<br />
<br />
=== Systemd ===<br />
singlenode:~ # systemctl show open-xchange | grep Limit<br />
StartLimitInterval=10000000<br />
StartLimitBurst=5<br />
StartLimitAction=none<br />
MemoryLimit=18446744073709551615<br />
LimitCPU=18446744073709551615<br />
LimitFSIZE=18446744073709551615<br />
LimitDATA=18446744073709551615<br />
LimitSTACK=18446744073709551615<br />
LimitCORE=18446744073709551615<br />
LimitRSS=18446744073709551615<br />
LimitNOFILE=65536<br />
LimitAS=18446744073709551615<br />
LimitNPROC=65536<br />
LimitMEMLOCK=65536<br />
LimitLOCKS=18446744073709551615<br />
LimitSIGPENDING=19827<br />
LimitMSGQUEUE=819200<br />
LimitNICE=0<br />
LimitRTPRIO=0<br />
LimitRTTIME=18446744073709551615<br />
<br />
== Evaluate JVM commandline options ==<br />
<br />
Before starting Java and the OSGi framework with all the Open-Xchange middleware bundles we have to evaluate several important JVM commandline options that influence the performance and stability of the Open-Xchange middleware stack. These commandline options are specified in '''/opt/open-xchange/etc/ox-scriptconf.sh'''<br />
<br />
=== Format < 7.8.4 ===<br />
For version up to 7.8.3 the JVM configuration options were all specified via a single property '''JAVA_XTRAOPTS''' which resulted in a rather long configuration string that was hard to read and maintain.<br />
<br />
JAVA_XTRAOPTS="-Dsun.net.inetaddr.ttl=3600 -Dnetworkaddress.cache.ttl=3600 -Dnetworkaddress.cache.negative.ttl=10 \<br />
-Dlogback.threadlocal.put.duplicate=false -server -Djava.awt.headless=true -Xmx512M -XX:MaxPermSize=256m -XX:+UseConcMarkSweepGC \<br />
-XX:+UseParNewGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:NewRatio=3 -XX:+UseTLAB \<br />
-Dosgi.compatibility.bootdelegation=false -XX:-OmitStackTraceInFastThrow"<br />
<br />
During the startup of the open-xchange service (step 11 of diagram Service startup using systemd and step 10 of diagram Service startup using upstart) this JAVA_XTRAOPTS string goes through a basic validity check and is then passed to the JVM.<br />
<br />
=== Format >= 7.8.4 ===<br />
Beginning from version 7.8.4 the options are split up and sorted into different categories like:<br />
<br />
JAVA_OPTS_GC="-XX:+UseConcMarkSweepGC -XX:+UseParNewGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:NewRatio=3 -XX:+DisableExplicitGC"<br />
JAVA_OPTS_LOG="-Dlogback.threadlocal.put.duplicate=false -XX:-OmitStackTraceInFastThrow"<br />
JAVA_OPTS_MEM="-XX:MaxHeapSize=512M -XX:MaxPermSize=256M -XX:+UseTLAB"<br />
JAVA_OPTS_NET="-Dsun.net.inetaddr.ttl=3600 -Dnetworkaddress.cache.ttl=3600 -Dnetworkaddress.cache.negative.ttl=10"<br />
JAVA_OPTS_OSGI="-Dosgi.compatibility.bootdelegation=false"<br />
JAVA_OPTS_SERVER="-server -Djava.awt.headless=true"<br />
<br />
JAVA_OPTS_OTHER=""<br />
<br />
#JAVA_OPTS_DEBUG="-XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/mnt/heapdump -Xloggc:/var/log/open-xchange/gc.log -verbose:gc -XX:+PrintGCDateStamps \<br />
-XX:+PrintHeapAtGC -XX:+PrintGCApplicationStoppedTime -XX:+PrintTenuringDistribution"<br />
<br />
which is much easier to read an maintain. Furthermore we added a useful '''JAVA_OPTS_DEBUG''' configuration parameter that comes preconfigured but unused/commented so that customers can quickly enable the most commonly used debugging options in case they have to provide detailed system information to Open-Xchange.<br />
<br />
During the startup of the open-xchange service (step 11 of diagram ''Service startup using systemd'' and step 10 of diagram ''Service startup using upstart'') all uncommented JAVA_OPTS_* categories will then be evaluated and combined again into a single string of commandline options that can then be passed to the JVM.<br />
<br />
=== Upgrade procedure ===<br />
During the upgrade '''7.8.3-revX -> 7.8.4-revY''' existing JAVA_XTRAOPTS will automatically be split up and migrated to the new categories approach. A backup of the file (ox-scriptconf.sh.timestamp) will be created and admins are urged to check for proper migration before deleting the backup file like it's already done with *.dpkg-dist or *.rpmnew files on Debian or RHEL/SLES distributions.<br />
<br />
== Encoding detection during JVM startup ==<br />
<br />
=== Background Infos ===<br />
All of en_US.UTF-8, .utf8, .utf-8, .UTF_8 will result in a valid UTF-8 charmap<br />
<br />
==== JVM startup ====<br />
<br />
The default file.encoding detection is done at JVM startup based upon LC_CTYPE which makes sense as it's documented as:<br />
<br />
CODESET (LC_CTYPE)<br />
Return a string with the name of the character encoding used in the selected locale, such as "UTF-8", "ISO-8859-1",<br />
or "ANSI_X3.4-1968" (better known as US-ASCII). This is the same string that you get with "locale charmap".<br />
For a list of character encoding names, try "locale -m", cf. locale(1).<br />
<br />
Furthermore this influences the defaultCharset selection of the JVM.<br />
<br />
/**<br />
* Returns the default charset of this Java virtual machine.<br />
*<br />
* The default charset is determined during virtual-machine startup and<br />
* typically depends upon the locale and charset of the underlying<br />
* operating system.<br />
*<br />
* @return A charset object for the default charset<br />
*<br />
* @since 1.5<br />
*/<br />
public static Charset defaultCharset() {<br />
if (defaultCharset == null) {<br />
synchronized (Charset.class) {<br />
String csn = AccessController.doPrivileged(<br />
new GetPropertyAction("file.encoding"));<br />
Charset cs = lookup(csn);<br />
if (cs != null)<br />
defaultCharset = cs;<br />
else<br />
defaultCharset = forName("UTF-8");<br />
}<br />
}<br />
return defaultCharset;<br />
}<br />
<br />
Although we try to specify valid unicode encodings for every IO operations the Open-Xchange middleware does some parts of the JVM don't allow to specify the encoding to use and falls back to the detected default encoding. That's why admins have to make sure to provide a proper unicode environment for the middleware service during every start or use documented workarounds provided by Open-Xchange (e.g. )<br />
<br />
=== Query which file encoding was detected by the JVM during startup ===<br />
[[File:visualvm-file.encoding.png|thumb|Querying the JVM via visualvm|right|300px]]<br />
<br />
The file encoding that was detected by the JVM during startup can be queried via JMX. You can use graphical management tools like [https://visualvm.github.io/ visualvm], query our [[Jolokia]] JMX bridge via http or just use the commandline like<br />
<br />
/opt/open-xchange/sbin/showruntimestats -r | sed -ne 's/.*\({key=file.encoding, value=[^}]*}\).*/\1/p'<br />
{key=file.encoding, value=UTF-8}<br />
<br />
<br clear=all><br />
<br />
=== Locale passing from client to server via SSH ===<br />
<br />
The following setting of sshd allows clients to specify which locale to use on the server and might lead to some confusion or unexpected behaviour when checking the system locale or even to the usage of broken locales during middleware startup if your client sends misconfigured or missing locale configuration to the server.<br />
<br />
grep AcceptEnv /etc/ssh/sshd_config <br />
AcceptEnv LANG LC_*<br />
<br />
For example when i connect to the machine my local german locale is sent to the server <br />
<br />
[vagrant@showcase-community-centos-7 ~]$ locale -v <br />
LANG=de_DE.UTF-8<br />
LC_CTYPE="de_DE.UTF-8"<br />
LC_NUMERIC=de_DE.UTF-8<br />
LC_TIME=de_DE.UTF-8<br />
LC_COLLATE="de_DE.UTF-8"<br />
LC_MONETARY=de_DE.UTF-8<br />
LC_MESSAGES="de_DE.UTF-8"<br />
LC_PAPER=de_DE.UTF-8<br />
LC_NAME=de_DE.UTF-8<br />
LC_ADDRESS=de_DE.UTF-8<br />
LC_TELEPHONE=de_DE.UTF-8<br />
LC_MEASUREMENT=de_DE.UTF-8<br />
LC_IDENTIFICATION=de_DE.UTF-8<br />
LC_ALL=<br />
<br />
But after specifying that i want a new login shell the default locale configured for that user/system will be used instead!<br />
<br />
[root@showcase-community-centos-7 vagrant]# su -s /bin/bash open-xchange -l<br />
-bash-4.2$ locale -v <br />
LANG=en_US.UTF-8<br />
LC_CTYPE="en_US.UTF-8"<br />
LC_NUMERIC="en_US.UTF-8"<br />
LC_TIME="en_US.UTF-8"<br />
LC_COLLATE="en_US.UTF-8"<br />
LC_MONETARY="en_US.UTF-8"<br />
LC_MESSAGES="en_US.UTF-8"<br />
LC_PAPER="en_US.UTF-8"<br />
LC_NAME="en_US.UTF-8"<br />
LC_ADDRESS="en_US.UTF-8"<br />
LC_TELEPHONE="en_US.UTF-8"<br />
LC_MEASUREMENT="en_US.UTF-8"<br />
LC_IDENTIFICATION="en_US.UTF-8"<br />
LC_ALL=<br />
<br />
Add details from https://bugs.open-xchange.com/show_bug.cgi?id=51074<br />
<br />
== Warnings during early start ==<br />
<br />
Early warnings before the start of the JVM and related logging libraries like logback and logstash forwarding are redirected to '''/var/log/open-xchange/open-xchange-console.log'''.<br />
<br />
=== Limits ===<br />
If the open-xchange startup script isn't able to set the proper limits on CentOS 6 due to e.g. hard limits being enforced via pam_limits you'll see the following warning in the console log during startup<br />
<br />
singlenode open-xchange # cat open-xchange-console.log<br />
/opt/open-xchange/sbin/open-xchange: line 115: ulimit: max user processes: cannot modify limit: Operation not permitted<br />
<br />
=== Locale ===<br />
If the open-xchange startup script isn't able to detect a unicode locale you'll see a warning like the following (the current charmap might differ)<br />
<br />
WARNING: Couldn't detect proper unicode charmap in locale category LC_CTYPE.<br />
WARNING: Current charmap: "ANSI_X3.4-1968" might cause encoding problems.<br />
<br />
== Summary: Open-Xchange middleware on specific distros ==<br />
The support for the mentioned mechanism of resource control differ depending on the distribution and the init system in use.<br />
<br />
=== Debian 7 ===<br />
;Init<br />
: System V style<br />
;OX Configurable Limits/Defaults<br />
: nofile, nproc<br />
<br />
The mentioned limits can be configured via '''/opt/open-xchange/etc/ox-scriptconf.sh'''. The limits are applied via ulimit in the service's init script. The open-xchange service is finally started via start-stop-daemon which doesn't doesn't consider '''/etc/security/limits.*'''<br />
<br />
=== RHEL 6 / CentOS 6 ===<br />
;Init<br />
: Upstart, System V compatible<br />
;OX Configurable Limits/Defaults<br />
: nofile, nproc<br />
<br />
The mentioned limits can be configured via '''/opt/open-xchange/etc/ox-scriptconf.sh'''. The limits are applied via ulimit in the service's init script. Furthermore as the open-xchange service is finally started via '''su ... open-xchange''' on this distro a user session is opened via su/pam and the default CentOS pam config reads the '''/etc/security/limits.*''' configuration by loading the pam stack like:<br />
<br />
: '''/etc/pam.d/su'''<br />
:: '''-> /etc/pam.d/system-auth'''<br />
::: '''-> pam_limits.so'''<br />
<br />
If NPROC isn't configured for the open-xchange-server it's restricted to '''1024''' globally by default to prevent accidental fork bombs, see '''/etc/security/limits.d/90-nproc.conf''' which can result in severe problems modern multithreaded applications.<br />
<br />
=== RHEL 7 / CentOS 7 / Debian 8 / SLE 12 ===<br />
;Init<br />
: Systemd<br />
;OX Configurable Limits/Defaults<br />
: nofile, nproc<br />
<br />
For systemd the default limits are configured directly in the service's unit file that is shipped by OX and located at '''/usr/lib/systemd/system/open-xchange.service'''. The drop-in config to override or extend the default unit file is located at '''/etc/systemd/system/open-xchange.service.d/limits.conf'''. [https://www.freedesktop.org/software/systemd/man/systemd.exec.html Systemd.exec] shows a whole lot of options that can be used by admins to adapt the default service to their specific needs.</div>Marenshttps://oxpedia.org/wiki/index.php?title=Sandbox:Middleware_Startup/sandbox&diff=23228Sandbox:Middleware Startup/sandbox2017-04-24T20:06:45Z<p>Marens: </p>
<hr />
<div><br />
== Systemd ==<br />
<br />
==== Service definition ====<br />
The default service definition as shipped by Open-Xchange is rather concise and easy to read. You can check by looking at the service file that is installed to either /usr/lib/systemd or /lib/systemd depending on your distribution of choice.<br />
<br />
singlenode:~ # cat /usr/lib/systemd/system/open-xchange.service <br />
[Unit]<br />
After=remote-fs.target<br />
After=time-sync.target ypbind.service sendmail.service cyrus.service <br />
<br />
[Service]<br />
User=open-xchange<br />
PermissionsStartOnly=true<br />
TimeoutStartSec=0<br />
ExecStartPre=/opt/open-xchange/sbin/triggerupdatethemes -u<br />
ExecStart=/opt/open-xchange/sbin/open-xchange<br />
ExecStop=/opt/open-xchange/sbin/shutdown -w<br />
ExecReload=/opt/open-xchange/sbin/triggerreloadconfiguration -d<br />
KillMode=process<br />
LimitNOFILE=65536<br />
LimitNPROC=65536<br />
<br />
[Install]<br />
WantedBy=multi-user.target<br />
<br />
==== Drop-in configs ====<br />
Drop in configs allow administrators to easily adapt and override default service unit files. So if you want to change the default limits or add additional limits have a look at our default example drop-in config which is located at /etc/systemd/system/open-xchange.service.d/limits.conf<br />
<br />
singlenode:~ # cat /etc/systemd/system/open-xchange.service.d/limits.conf <br />
# Override and add options in this file<br />
# See systemd.exec(5) for other limits<br />
<br />
[Service]<br />
#LimitNPROC=65536<br />
#LimitNOFILE=65536<br />
<br />
The systemd.exec documentation shows a whole lot of options that can be used by admins to adapt the default service to their specific needs.<br />
<br />
<br clear=all><br />
[[File:mw_start_sysd.png|thumb|Service startup using systemd|left|1000px]]<br />
<br clear=all><br />
<br />
== System V / Upstart ==<br />
<br />
==== Service definition ====<br />
<br />
The default service definition as shipped by Open-Xchange can be found at /etc/init.d/open-xchange and will differ slightly depending on the distro in use.<br />
<br />
<br clear=all><br />
[[File:mw_start_sysv.png|thumb|Service startup using upstart|left|1000px]]<br />
<br clear=all><br />
== Apply Resource limits ==<br />
<br />
As you can see in the diagrams above one of the differences is that:<br />
<br />
* systemd reads all the infos wrt. resource limitations from the original service file and then applies possible further restrictions from the drop-in configs (steps 2 and 8 in the diagram ''Service startup using systemd'') and limits configured in /opt/open-xchange/etc/ox-scriptconf.sh aren't considered anylonger<br />
* the older System V style init has to source /opt/open-xchange/etc/ox-scriptconf.sh to be able to read the configs for the maximum number of processes (NPROC) and files(NRFILES) that the middleware shall be allowed to create/open (steps 3,4 and 15 in the diagram ''Service startup using upstart'').<br />
<br />
=== Background Infos ===<br />
<br />
Several ways exist to restrict resources on a linux system from a global level down to user/groups or even shells and the processes started by them.<br />
<br />
==== Sysctl ====<br />
Sysctl is used to modify kernel parameters at runtime. E.g. to set the maximum number of files<br />
<br />
$ sysctl -w fs.file-max=100000<br />
<br />
To permanently set them append to the main configuration file and reload the settings<br />
<br />
$ echo fs.file-max=100000 >> /etc/sysctl.conf<br />
$ sysctl -p<br />
<br />
More infos can be found via '''man sysctl'''<br />
<br />
==== Limits.conf ====<br />
<br />
Allows to restrict resources an a global, group or user level. E.g:<br />
<br />
$ cat /etc/security/limits.d/90-nproc.conf <br />
# Default limit for number of user's processes to prevent<br />
# accidental fork bombs.<br />
# See rhbz #432903 for reasoning.<br />
<br />
* soft nproc 1024<br />
<br />
From '''man limits.conf''':<br />
<blockquote><br />
''Also, please note that all limit settings are set per login. They are not global, nor are they permanent; existing only for the duration of the session.''<br />
</blockquote><br />
<br />
The limits per login are applied via the ''pam'' stack. See '''man pam''' and '''man pam_limits''' for more details. As those limits are bound to sessions they don't affect most daemons started by our supported init systems or init utils. Most state that they are ignored by design, see [https://bugs.launchpad.net/ubuntu/+source/upstart/+bug/938669/comments/4 upstart], [https://bugzilla.redhat.com/show_bug.cgi?id=754285#c1 systemd] and [https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=302079 start-stop-daemon] <br />
<br />
===== Ulimit =====<br />
From '''man bash'''<br />
<blockquote><br />
ulimit [-HSTabcdefilmnpqrstuvx [limit]]<br />
Provides control over the resources available to the shell and to processes started by it, on systems that allow such control.<br />
</blockquote><br />
This is what we use in our System V compatible init scripts to increase resources for the open-xchange process across multiple distros. Currently only ''the maximum number of processes'' and ''the maximum number of open file descriptors available to a single user'' are increased via ulimit. The values are specified in '''/opt/open-xchange/ox-scriptconf.sh'''<br />
<br />
==== Systemd ====<br />
<br />
===== Control Groups=====<br />
<br />
Control groups should only affect the OX middleware if you create/manage them yourself of if you are using a ''modern'' distribution that already uses systemd as init.<br />
<br />
Citing from the kernel cgroup [https://www.kernel.org/doc/Documentation/cgroup-v2.txt documentation]:<br />
<br />
<blockquote><br />
1-2. What is cgroup?<br />
<br />
cgroup is a mechanism to organize processes hierarchically and<br />
distribute system resources along the hierarchy in a controlled and<br />
configurable manner.<br />
<br />
cgroup is largely composed of two parts - the core and controllers.<br />
cgroup core is primarily responsible for hierarchically organizing<br />
processes. A cgroup controller is usually responsible for<br />
distributing a specific type of system resource along the hierarchy<br />
although there are utility controllers which serve purposes other than<br />
resource distribution.<br />
<br />
cgroups form a tree structure and every process in the system belongs<br />
to one and only one cgroup. All threads of a process belong to the<br />
same cgroup. On creation, all processes are put in the cgroup that<br />
the parent process belongs to at the time. A process can be migrated<br />
to another cgroup. Migration of a process doesn't affect already<br />
existing descendant processes.<br />
<br />
Following certain structural constraints, controllers may be enabled or<br />
disabled selectively on a cgroup. All controller behaviors are<br />
hierarchical - if a controller is enabled on a cgroup, it affects all<br />
processes which belong to the cgroups consisting the inclusive<br />
sub-hierarchy of the cgroup. When a controller is enabled on a nested<br />
cgroup, it always restricts the resource distribution further. The<br />
restrictions set closer to the root in the hierarchy can not be<br />
overridden from further away.<br />
</blockquote><br />
<br />
So processes are organized into a tree structure of control groups and controllers are responsible for the distribution of resources. So what kind of controllers exist?<br />
<br />
<blockquote><br />
5. Controllers<br />
<br />
5-1. CPU<br />
<br />
The "cpu" controllers regulates distribution of CPU cycles. This<br />
controller implements weight and absolute bandwidth limit models for<br />
normal scheduling policy and absolute bandwidth allocation model for<br />
realtime scheduling policy.<br />
<br />
5-2. Memory<br />
<br />
The "memory" controller regulates distribution of memory.<br />
...<br />
While not completely water-tight, all major memory usages by a given<br />
cgroup are tracked so that the total memory consumption can be<br />
accounted and controlled to a reasonable extent.<br />
<br />
5-3. IO<br />
<br />
The "io" controller regulates the distribution of IO resources. This<br />
controller implements both weight based and absolute bandwidth or IOPS<br />
limit distribution; however, weight based distribution is available<br />
only if cfq-iosched is in use and neither scheme is available for<br />
blk-mq devices.<br />
</blockquote><br />
<br />
The open-xchange service is simply put into the default ''system.slice'' without applying further limits.<br />
<br />
singlenode$ systemd-cgls<br />
├─1 /sbin/init<br />
├─system.slice<br />
│ ├─avahi-daemon.service<br />
│ │ ├─501 avahi-daemon: running [singlenode]<br />
│ │ └─514 avahi-daemon: chroot helper<br />
│ ├─console-kit-daemon.service<br />
│ │ └─16164 /usr/sbin/console-kit-daemon --no-daemon<br />
│ ├─dbus.service<br />
│ │ └─508 /usr/bin/dbus-daemon --system --address=systemd: --nofork --nopidfile --systemd-activation<br />
│ ├─munin-node.service<br />
│ │ └─4290 /usr/bin/perl -wT /usr/sbin/munin-node<br />
│ ├─open-xchange.service<br />
│ │ └─6037 /usr/bin/java -Dsun.net.inetaddr.ttl=3600 -Dnetworkaddress.cache.ttl=3600 -Dnetworkaddress.cache.negative.ttl=10 ...<br />
<br />
To check all the details use<br />
<br />
singlenode:~ # systemctl show system.slice<br />
<br />
==== Limits besides control groups ====<br />
Besides control groups systemd allows you to apply other limits to the execution environment of your service. Here we can apply the limits that would normally be applied via ''limits.conf'' or ''ulimit''. Systemd uses [http://man7.org/linux/man-pages/man2/setrlimit.2.html setrlimit] for this. The options that we set by default are:<br />
<br />
* LimitNOFILE<br />
* LimitNPROC<br />
<br />
which you have already seen when we took a look at the [[#Service_definiton|service definition]] and [[#Drop-in_config|drop-in configs]]<br />
<br />
== Verify limits ==<br />
<br />
=== System V ===<br />
<br />
Get the proper pid of the java process started for the middleware by either checking '''ps faux'''<br />
root 4103 0.0 0.0 66528 1868 pts/0 S 22:05 0:00 su -s /bin/bash open-xchange -c /opt/open-xchange/sbin/open-xchange<br />
494 4109 9.5 19.8 2224644 381312 ? Ssl 22:05 1:12 \_ /usr/bin/java -Dsun.net.inetaddr.ttl=3600 -Dnetworkaddress.cache.ttl=3600<br />
<br />
or programmatically<br />
singlenode:~ # pid=$(pgrep -P $(</var/run/open-xchange.pid) java)<br />
<br />
and check the limits applied for that process<br />
singlenode:~ # cat /proc/$pid/limits<br />
Limit Soft Limit Hard Limit Units<br />
Max cpu time unlimited unlimited seconds<br />
Max file size unlimited unlimited bytes<br />
Max data size unlimited unlimited bytes<br />
Max stack size 8388608 unlimited bytes<br />
Max core file size 0 unlimited bytes<br />
Max resident set unlimited unlimited bytes<br />
Max processes 65536 65536 processes<br />
Max open files 65536 65536 files<br />
Max locked memory 65536 65536 bytes<br />
Max address space unlimited unlimited bytes<br />
Max file locks unlimited unlimited locks<br />
Max pending signals 24254 24254 signals<br />
Max msgqueue size 819200 819200 bytes<br />
Max nice priority 0 0<br />
Max realtime priority 0 0<br />
Max realtime timeout unlimited unlimited us<br />
<br />
=== Systemd ===<br />
singlenode:~ # systemctl show open-xchange | grep Limit<br />
StartLimitInterval=10000000<br />
StartLimitBurst=5<br />
StartLimitAction=none<br />
MemoryLimit=18446744073709551615<br />
LimitCPU=18446744073709551615<br />
LimitFSIZE=18446744073709551615<br />
LimitDATA=18446744073709551615<br />
LimitSTACK=18446744073709551615<br />
LimitCORE=18446744073709551615<br />
LimitRSS=18446744073709551615<br />
LimitNOFILE=65536<br />
LimitAS=18446744073709551615<br />
LimitNPROC=65536<br />
LimitMEMLOCK=65536<br />
LimitLOCKS=18446744073709551615<br />
LimitSIGPENDING=19827<br />
LimitMSGQUEUE=819200<br />
LimitNICE=0<br />
LimitRTPRIO=0<br />
LimitRTTIME=18446744073709551615<br />
<br />
== Evaluate JVM commandline options ==<br />
<br />
Before starting Java and the OSGi framework with all the Open-Xchange middleware bundles we have to evaluate several important JVM commandline options that influence the performance and stability of the Open-Xchange middleware stack. These commandline options are specified in '''/opt/open-xchange/etc/ox-scriptconf.sh'''<br />
<br />
=== Format < 7.8.4 ===<br />
For version up to 7.8.3 the JVM configuration options were all specified via a single property '''JAVA_XTRAOPTS''' which resulted in a rather long configuration string that was hard to read and maintain.<br />
<br />
JAVA_XTRAOPTS="-Dsun.net.inetaddr.ttl=3600 -Dnetworkaddress.cache.ttl=3600 -Dnetworkaddress.cache.negative.ttl=10 \<br />
-Dlogback.threadlocal.put.duplicate=false -server -Djava.awt.headless=true -Xmx512M -XX:MaxPermSize=256m -XX:+UseConcMarkSweepGC \<br />
-XX:+UseParNewGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:NewRatio=3 -XX:+UseTLAB \<br />
-Dosgi.compatibility.bootdelegation=false -XX:-OmitStackTraceInFastThrow"<br />
<br />
During the startup of the open-xchange service (step 11 of diagram Service startup using systemd and step 10 of diagram Service startup using upstart) this JAVA_XTRAOPTS string goes through a basic validity check and is then passed to the JVM.<br />
<br />
=== Format >= 7.8.4 ===<br />
Beginning from version 7.8.4 the options are split up and sorted into different categories like:<br />
<br />
JAVA_OPTS_GC="-XX:+UseConcMarkSweepGC -XX:+UseParNewGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:NewRatio=3 -XX:+DisableExplicitGC"<br />
JAVA_OPTS_LOG="-Dlogback.threadlocal.put.duplicate=false -XX:-OmitStackTraceInFastThrow"<br />
JAVA_OPTS_MEM="-XX:MaxHeapSize=512M -XX:MaxPermSize=256M -XX:+UseTLAB"<br />
JAVA_OPTS_NET="-Dsun.net.inetaddr.ttl=3600 -Dnetworkaddress.cache.ttl=3600 -Dnetworkaddress.cache.negative.ttl=10"<br />
JAVA_OPTS_OSGI="-Dosgi.compatibility.bootdelegation=false"<br />
JAVA_OPTS_SERVER="-server -Djava.awt.headless=true"<br />
<br />
JAVA_OPTS_OTHER=""<br />
<br />
#JAVA_OPTS_DEBUG="-XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/mnt/heapdump -Xloggc:/var/log/open-xchange/gc.log -verbose:gc -XX:+PrintGCDateStamps \<br />
-XX:+PrintHeapAtGC -XX:+PrintGCApplicationStoppedTime -XX:+PrintTenuringDistribution"<br />
<br />
which is much easier to read an maintain. Furthermore we added a useful '''JAVA_OPTS_DEBUG''' configuration parameter that comes preconfigured but unused/commented so that customers can quickly enable the most commonly used debugging options in case they have to provide detailed system information to Open-Xchange.<br />
<br />
During the startup of the open-xchange service (step 11 of diagram ''Service startup using systemd'' and step 10 of diagram ''Service startup using upstart'') all uncommented JAVA_OPTS_* categories will then be evaluated and combined again into a single string of commandline options that can then be passed to the JVM.<br />
<br />
=== Upgrade procedure ===<br />
During the upgrade '''7.8.3-revX -> 7.8.4-revY''' existing JAVA_XTRAOPTS will automatically be split up and migrated to the new categories approach. A backup of the file (ox-scriptconf.sh.timestamp) will be created and admins are urged to check for proper migration before deleting the backup file like it's already done with *.dpkg-dist or *.rpmnew files on Debian or RHEL/SLES distributions.<br />
<br />
== Encoding detection during JVM startup ==<br />
<br />
=== Background Infos ===<br />
All of en_US.UTF-8, .utf8, .utf-8, .UTF_8 will result in a valid UTF-8 charmap<br />
<br />
==== JVM startup ====<br />
<br />
The default file.encoding detection is done at JVM startup based upon LC_CTYPE which makes sense as it's documented as:<br />
<br />
CODESET (LC_CTYPE)<br />
Return a string with the name of the character encoding used in the selected locale, such as "UTF-8", "ISO-8859-1",<br />
or "ANSI_X3.4-1968" (better known as US-ASCII). This is the same string that you get with "locale charmap".<br />
For a list of character encoding names, try "locale -m", cf. locale(1).<br />
<br />
Furthermore this influences the defaultCharset selection of the JVM.<br />
<br />
/**<br />
* Returns the default charset of this Java virtual machine.<br />
*<br />
* The default charset is determined during virtual-machine startup and<br />
* typically depends upon the locale and charset of the underlying<br />
* operating system.<br />
*<br />
* @return A charset object for the default charset<br />
*<br />
* @since 1.5<br />
*/<br />
public static Charset defaultCharset() {<br />
if (defaultCharset == null) {<br />
synchronized (Charset.class) {<br />
String csn = AccessController.doPrivileged(<br />
new GetPropertyAction("file.encoding"));<br />
Charset cs = lookup(csn);<br />
if (cs != null)<br />
defaultCharset = cs;<br />
else<br />
defaultCharset = forName("UTF-8");<br />
}<br />
}<br />
return defaultCharset;<br />
}<br />
<br />
Although we try to specify valid unicode encodings for every IO operations the Open-Xchange middleware does some parts of the JVM don't allow to specify the encoding to use and falls back to the detected default encoding. That's why admins have to make sure to provide a proper unicode environment for the middleware service during every start or use documented workarounds provided by Open-Xchange (e.g. )<br />
<br />
=== Query which file encoding was detected by the JVM during startup ===<br />
[[File:visualvm-file.encoding.png|thumb|Querying the JVM via visualvm|right|300px]]<br />
<br />
<br />
The file encoding that was detected by the JVM during startup can be queried via JMX. You can use graphical management tools like [https://visualvm.github.io/ visualvm], query our [[Jolokia]] JMX bridge via http or just use the commandline like<br />
<br />
/opt/open-xchange/sbin/showruntimestats -r | sed -ne 's/.*\({key=file.encoding, value=[^}]*}\).*/\1/p'<br />
{key=file.encoding, value=UTF-8}<br />
<br />
=== Locale passing from client to server via SSH ===<br />
<br />
The following setting of sshd allows clients to specify which locale to use on the server and might lead to some confusion or unexpected behaviour when checking the system locale or even to the usage of broken locales during middleware startup if your client sends misconfigured or missing locale configuration to the server.<br />
<br />
grep AcceptEnv /etc/ssh/sshd_config <br />
AcceptEnv LANG LC_*<br />
<br />
For example when i connect to the machine my local german locale is sent to the server <br />
<br />
[vagrant@showcase-community-centos-7 ~]$ locale -v <br />
LANG=de_DE.UTF-8<br />
LC_CTYPE="de_DE.UTF-8"<br />
LC_NUMERIC=de_DE.UTF-8<br />
LC_TIME=de_DE.UTF-8<br />
LC_COLLATE="de_DE.UTF-8"<br />
LC_MONETARY=de_DE.UTF-8<br />
LC_MESSAGES="de_DE.UTF-8"<br />
LC_PAPER=de_DE.UTF-8<br />
LC_NAME=de_DE.UTF-8<br />
LC_ADDRESS=de_DE.UTF-8<br />
LC_TELEPHONE=de_DE.UTF-8<br />
LC_MEASUREMENT=de_DE.UTF-8<br />
LC_IDENTIFICATION=de_DE.UTF-8<br />
LC_ALL=<br />
<br />
But after specifying that i want a new login shell the default locale configured for that user/system will be used instead!<br />
<br />
[root@showcase-community-centos-7 vagrant]# su -s /bin/bash open-xchange -l<br />
-bash-4.2$ locale -v <br />
LANG=en_US.UTF-8<br />
LC_CTYPE="en_US.UTF-8"<br />
LC_NUMERIC="en_US.UTF-8"<br />
LC_TIME="en_US.UTF-8"<br />
LC_COLLATE="en_US.UTF-8"<br />
LC_MONETARY="en_US.UTF-8"<br />
LC_MESSAGES="en_US.UTF-8"<br />
LC_PAPER="en_US.UTF-8"<br />
LC_NAME="en_US.UTF-8"<br />
LC_ADDRESS="en_US.UTF-8"<br />
LC_TELEPHONE="en_US.UTF-8"<br />
LC_MEASUREMENT="en_US.UTF-8"<br />
LC_IDENTIFICATION="en_US.UTF-8"<br />
LC_ALL=<br />
<br />
Add details from https://bugs.open-xchange.com/show_bug.cgi?id=51074<br />
<br />
== Warnings during early start ==<br />
<br />
Early warnings before the start of the JVM and related logging libraries like logback and logstash forwarding are redirected to '''/var/log/open-xchange/open-xchange-console.log'''.<br />
<br />
=== Limits ===<br />
If the open-xchange startup script isn't able to set the proper limits on CentOS 6 due to e.g. hard limits being enforced via pam_limits you'll see the following warning in the console log during startup<br />
<br />
singlenode open-xchange # cat open-xchange-console.log<br />
/opt/open-xchange/sbin/open-xchange: line 115: ulimit: max user processes: cannot modify limit: Operation not permitted<br />
<br />
=== Locale ===<br />
If the open-xchange startup script isn't able to detect a unicode locale you'll see a warning like the following (the current charmap might differ)<br />
<br />
WARNING: Couldn't detect proper unicode charmap in locale category LC_CTYPE.<br />
WARNING: Current charmap: "ANSI_X3.4-1968" might cause encoding problems.<br />
<br />
== Summary: Open-Xchange middleware on specific distros ==<br />
The support for the mentioned mechanism of resource control differ depending on the distribution and the init system in use.<br />
<br />
=== Debian 7 ===<br />
;Init<br />
: System V style<br />
;OX Configurable Limits/Defaults<br />
: nofile, nproc<br />
<br />
The mentioned limits can be configured via '''/opt/open-xchange/etc/ox-scriptconf.sh'''. The limits are applied via ulimit in the service's init script. The open-xchange service is finally started via start-stop-daemon which doesn't doesn't consider '''/etc/security/limits.*'''<br />
<br />
=== RHEL 6 / CentOS 6 ===<br />
;Init<br />
: Upstart, System V compatible<br />
;OX Configurable Limits/Defaults<br />
: nofile, nproc<br />
<br />
The mentioned limits can be configured via '''/opt/open-xchange/etc/ox-scriptconf.sh'''. The limits are applied via ulimit in the service's init script. Furthermore as the open-xchange service is finally started via '''su ... open-xchange''' on this distro a user session is opened via su/pam and the default CentOS pam config reads the '''/etc/security/limits.*''' configuration by loading the pam stack like:<br />
<br />
: '''/etc/pam.d/su'''<br />
:: '''-> /etc/pam.d/system-auth'''<br />
::: '''-> pam_limits.so'''<br />
<br />
If NPROC isn't configured for the open-xchange-server it's restricted to '''1024''' globally by default to prevent accidental fork bombs, see '''/etc/security/limits.d/90-nproc.conf''' which can result in severe problems modern multithreaded applications.<br />
<br />
=== RHEL 7 / CentOS 7 / Debian 8 / SLE 12 ===<br />
;Init<br />
: Systemd<br />
;OX Configurable Limits/Defaults<br />
: nofile, nproc<br />
<br />
For systemd the default limits are configured directly in the service's unit file that is shipped by OX and located at '''/usr/lib/systemd/system/open-xchange.service'''. The drop-in config to override or extend the default unit file is located at '''/etc/systemd/system/open-xchange.service.d/limits.conf'''. [https://www.freedesktop.org/software/systemd/man/systemd.exec.html Systemd.exec] shows a whole lot of options that can be used by admins to adapt the default service to their specific needs.</div>Marenshttps://oxpedia.org/wiki/index.php?title=File:Visualvm-file.encoding.png&diff=23227File:Visualvm-file.encoding.png2017-04-24T20:01:55Z<p>Marens: </p>
<hr />
<div></div>Marenshttps://oxpedia.org/wiki/index.php?title=Sandbox:Middleware_Startup/sandbox&diff=23226Sandbox:Middleware Startup/sandbox2017-04-24T20:01:02Z<p>Marens: /* Query which file encoding was detected by the JVM during startup */</p>
<hr />
<div><br />
== Systemd ==<br />
<br />
==== Service definition ====<br />
The default service definition as shipped by Open-Xchange is rather concise and easy to read. You can check by looking at the service file that is installed to either /usr/lib/systemd or /lib/systemd depending on your distribution of choice.<br />
<br />
singlenode:~ # cat /usr/lib/systemd/system/open-xchange.service <br />
[Unit]<br />
After=remote-fs.target<br />
After=time-sync.target ypbind.service sendmail.service cyrus.service <br />
<br />
[Service]<br />
User=open-xchange<br />
PermissionsStartOnly=true<br />
TimeoutStartSec=0<br />
ExecStartPre=/opt/open-xchange/sbin/triggerupdatethemes -u<br />
ExecStart=/opt/open-xchange/sbin/open-xchange<br />
ExecStop=/opt/open-xchange/sbin/shutdown -w<br />
ExecReload=/opt/open-xchange/sbin/triggerreloadconfiguration -d<br />
KillMode=process<br />
LimitNOFILE=65536<br />
LimitNPROC=65536<br />
<br />
[Install]<br />
WantedBy=multi-user.target<br />
<br />
==== Drop-in configs ====<br />
Drop in configs allow administrators to easily adapt and override default service unit files. So if you want to change the default limits or add additional limits have a look at our default example drop-in config which is located at /etc/systemd/system/open-xchange.service.d/limits.conf<br />
<br />
singlenode:~ # cat /etc/systemd/system/open-xchange.service.d/limits.conf <br />
# Override and add options in this file<br />
# See systemd.exec(5) for other limits<br />
<br />
[Service]<br />
#LimitNPROC=65536<br />
#LimitNOFILE=65536<br />
<br />
The systemd.exec documentation shows a whole lot of options that can be used by admins to adapt the default service to their specific needs.<br />
<br />
<br clear=all><br />
[[File:mw_start_sysd.png|thumb|Service startup using systemd|left|1000px]]<br />
<br clear=all><br />
<br />
== System V / Upstart ==<br />
<br />
==== Service definition ====<br />
<br />
The default service definition as shipped by Open-Xchange can be found at /etc/init.d/open-xchange and will differ slightly depending on the distro in use.<br />
<br />
<br clear=all><br />
[[File:mw_start_sysv.png|thumb|Service startup using upstart|left|1000px]]<br />
<br clear=all><br />
== Apply Resource limits ==<br />
<br />
As you can see in the diagrams above one of the differences is that:<br />
<br />
* systemd reads all the infos wrt. resource limitations from the original service file and then applies possible further restrictions from the drop-in configs (steps 2 and 8 in the diagram ''Service startup using systemd'') and limits configured in /opt/open-xchange/etc/ox-scriptconf.sh aren't considered anylonger<br />
* the older System V style init has to source /opt/open-xchange/etc/ox-scriptconf.sh to be able to read the configs for the maximum number of processes (NPROC) and files(NRFILES) that the middleware shall be allowed to create/open (steps 3,4 and 15 in the diagram ''Service startup using upstart'').<br />
<br />
=== Background Infos ===<br />
<br />
Several ways exist to restrict resources on a linux system from a global level down to user/groups or even shells and the processes started by them.<br />
<br />
==== Sysctl ====<br />
Sysctl is used to modify kernel parameters at runtime. E.g. to set the maximum number of files<br />
<br />
$ sysctl -w fs.file-max=100000<br />
<br />
To permanently set them append to the main configuration file and reload the settings<br />
<br />
$ echo fs.file-max=100000 >> /etc/sysctl.conf<br />
$ sysctl -p<br />
<br />
More infos can be found via '''man sysctl'''<br />
<br />
==== Limits.conf ====<br />
<br />
Allows to restrict resources an a global, group or user level. E.g:<br />
<br />
$ cat /etc/security/limits.d/90-nproc.conf <br />
# Default limit for number of user's processes to prevent<br />
# accidental fork bombs.<br />
# See rhbz #432903 for reasoning.<br />
<br />
* soft nproc 1024<br />
<br />
From '''man limits.conf''':<br />
<blockquote><br />
''Also, please note that all limit settings are set per login. They are not global, nor are they permanent; existing only for the duration of the session.''<br />
</blockquote><br />
<br />
The limits per login are applied via the ''pam'' stack. See '''man pam''' and '''man pam_limits''' for more details. As those limits are bound to sessions they don't affect most daemons started by our supported init systems or init utils. Most state that they are ignored by design, see [https://bugs.launchpad.net/ubuntu/+source/upstart/+bug/938669/comments/4 upstart], [https://bugzilla.redhat.com/show_bug.cgi?id=754285#c1 systemd] and [https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=302079 start-stop-daemon] <br />
<br />
===== Ulimit =====<br />
From '''man bash'''<br />
<blockquote><br />
ulimit [-HSTabcdefilmnpqrstuvx [limit]]<br />
Provides control over the resources available to the shell and to processes started by it, on systems that allow such control.<br />
</blockquote><br />
This is what we use in our System V compatible init scripts to increase resources for the open-xchange process across multiple distros. Currently only ''the maximum number of processes'' and ''the maximum number of open file descriptors available to a single user'' are increased via ulimit. The values are specified in '''/opt/open-xchange/ox-scriptconf.sh'''<br />
<br />
==== Systemd ====<br />
<br />
===== Control Groups=====<br />
<br />
Control groups should only affect the OX middleware if you create/manage them yourself of if you are using a ''modern'' distribution that already uses systemd as init.<br />
<br />
Citing from the kernel cgroup [https://www.kernel.org/doc/Documentation/cgroup-v2.txt documentation]:<br />
<br />
<blockquote><br />
1-2. What is cgroup?<br />
<br />
cgroup is a mechanism to organize processes hierarchically and<br />
distribute system resources along the hierarchy in a controlled and<br />
configurable manner.<br />
<br />
cgroup is largely composed of two parts - the core and controllers.<br />
cgroup core is primarily responsible for hierarchically organizing<br />
processes. A cgroup controller is usually responsible for<br />
distributing a specific type of system resource along the hierarchy<br />
although there are utility controllers which serve purposes other than<br />
resource distribution.<br />
<br />
cgroups form a tree structure and every process in the system belongs<br />
to one and only one cgroup. All threads of a process belong to the<br />
same cgroup. On creation, all processes are put in the cgroup that<br />
the parent process belongs to at the time. A process can be migrated<br />
to another cgroup. Migration of a process doesn't affect already<br />
existing descendant processes.<br />
<br />
Following certain structural constraints, controllers may be enabled or<br />
disabled selectively on a cgroup. All controller behaviors are<br />
hierarchical - if a controller is enabled on a cgroup, it affects all<br />
processes which belong to the cgroups consisting the inclusive<br />
sub-hierarchy of the cgroup. When a controller is enabled on a nested<br />
cgroup, it always restricts the resource distribution further. The<br />
restrictions set closer to the root in the hierarchy can not be<br />
overridden from further away.<br />
</blockquote><br />
<br />
So processes are organized into a tree structure of control groups and controllers are responsible for the distribution of resources. So what kind of controllers exist?<br />
<br />
<blockquote><br />
5. Controllers<br />
<br />
5-1. CPU<br />
<br />
The "cpu" controllers regulates distribution of CPU cycles. This<br />
controller implements weight and absolute bandwidth limit models for<br />
normal scheduling policy and absolute bandwidth allocation model for<br />
realtime scheduling policy.<br />
<br />
5-2. Memory<br />
<br />
The "memory" controller regulates distribution of memory.<br />
...<br />
While not completely water-tight, all major memory usages by a given<br />
cgroup are tracked so that the total memory consumption can be<br />
accounted and controlled to a reasonable extent.<br />
<br />
5-3. IO<br />
<br />
The "io" controller regulates the distribution of IO resources. This<br />
controller implements both weight based and absolute bandwidth or IOPS<br />
limit distribution; however, weight based distribution is available<br />
only if cfq-iosched is in use and neither scheme is available for<br />
blk-mq devices.<br />
</blockquote><br />
<br />
The open-xchange service is simply put into the default ''system.slice'' without applying further limits.<br />
<br />
singlenode$ systemd-cgls<br />
├─1 /sbin/init<br />
├─system.slice<br />
│ ├─avahi-daemon.service<br />
│ │ ├─501 avahi-daemon: running [singlenode]<br />
│ │ └─514 avahi-daemon: chroot helper<br />
│ ├─console-kit-daemon.service<br />
│ │ └─16164 /usr/sbin/console-kit-daemon --no-daemon<br />
│ ├─dbus.service<br />
│ │ └─508 /usr/bin/dbus-daemon --system --address=systemd: --nofork --nopidfile --systemd-activation<br />
│ ├─munin-node.service<br />
│ │ └─4290 /usr/bin/perl -wT /usr/sbin/munin-node<br />
│ ├─open-xchange.service<br />
│ │ └─6037 /usr/bin/java -Dsun.net.inetaddr.ttl=3600 -Dnetworkaddress.cache.ttl=3600 -Dnetworkaddress.cache.negative.ttl=10 ...<br />
<br />
To check all the details use<br />
<br />
singlenode:~ # systemctl show system.slice<br />
<br />
==== Limits besides control groups ====<br />
Besides control groups systemd allows you to apply other limits to the execution environment of your service. Here we can apply the limits that would normally be applied via ''limits.conf'' or ''ulimit''. Systemd uses [http://man7.org/linux/man-pages/man2/setrlimit.2.html setrlimit] for this. The options that we set by default are:<br />
<br />
* LimitNOFILE<br />
* LimitNPROC<br />
<br />
which you have already seen when we took a look at the [[#Service_definiton|service definition]] and [[#Drop-in_config|drop-in configs]]<br />
<br />
== Verify limits ==<br />
<br />
=== System V ===<br />
<br />
Get the proper pid of the java process started for the middleware by either checking '''ps faux'''<br />
root 4103 0.0 0.0 66528 1868 pts/0 S 22:05 0:00 su -s /bin/bash open-xchange -c /opt/open-xchange/sbin/open-xchange<br />
494 4109 9.5 19.8 2224644 381312 ? Ssl 22:05 1:12 \_ /usr/bin/java -Dsun.net.inetaddr.ttl=3600 -Dnetworkaddress.cache.ttl=3600<br />
<br />
or programmatically<br />
singlenode:~ # pid=$(pgrep -P $(</var/run/open-xchange.pid) java)<br />
<br />
and check the limits applied for that process<br />
singlenode:~ # cat /proc/$pid/limits<br />
Limit Soft Limit Hard Limit Units<br />
Max cpu time unlimited unlimited seconds<br />
Max file size unlimited unlimited bytes<br />
Max data size unlimited unlimited bytes<br />
Max stack size 8388608 unlimited bytes<br />
Max core file size 0 unlimited bytes<br />
Max resident set unlimited unlimited bytes<br />
Max processes 65536 65536 processes<br />
Max open files 65536 65536 files<br />
Max locked memory 65536 65536 bytes<br />
Max address space unlimited unlimited bytes<br />
Max file locks unlimited unlimited locks<br />
Max pending signals 24254 24254 signals<br />
Max msgqueue size 819200 819200 bytes<br />
Max nice priority 0 0<br />
Max realtime priority 0 0<br />
Max realtime timeout unlimited unlimited us<br />
<br />
=== Systemd ===<br />
singlenode:~ # systemctl show open-xchange | grep Limit<br />
StartLimitInterval=10000000<br />
StartLimitBurst=5<br />
StartLimitAction=none<br />
MemoryLimit=18446744073709551615<br />
LimitCPU=18446744073709551615<br />
LimitFSIZE=18446744073709551615<br />
LimitDATA=18446744073709551615<br />
LimitSTACK=18446744073709551615<br />
LimitCORE=18446744073709551615<br />
LimitRSS=18446744073709551615<br />
LimitNOFILE=65536<br />
LimitAS=18446744073709551615<br />
LimitNPROC=65536<br />
LimitMEMLOCK=65536<br />
LimitLOCKS=18446744073709551615<br />
LimitSIGPENDING=19827<br />
LimitMSGQUEUE=819200<br />
LimitNICE=0<br />
LimitRTPRIO=0<br />
LimitRTTIME=18446744073709551615<br />
<br />
== Evaluate JVM commandline options ==<br />
<br />
Before starting Java and the OSGi framework with all the Open-Xchange middleware bundles we have to evaluate several important JVM commandline options that influence the performance and stability of the Open-Xchange middleware stack. These commandline options are specified in '''/opt/open-xchange/etc/ox-scriptconf.sh'''<br />
<br />
=== Format < 7.8.4 ===<br />
For version up to 7.8.3 the JVM configuration options were all specified via a single property '''JAVA_XTRAOPTS''' which resulted in a rather long configuration string that was hard to read and maintain.<br />
<br />
JAVA_XTRAOPTS="-Dsun.net.inetaddr.ttl=3600 -Dnetworkaddress.cache.ttl=3600 -Dnetworkaddress.cache.negative.ttl=10 \<br />
-Dlogback.threadlocal.put.duplicate=false -server -Djava.awt.headless=true -Xmx512M -XX:MaxPermSize=256m -XX:+UseConcMarkSweepGC \<br />
-XX:+UseParNewGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:NewRatio=3 -XX:+UseTLAB \<br />
-Dosgi.compatibility.bootdelegation=false -XX:-OmitStackTraceInFastThrow"<br />
<br />
During the startup of the open-xchange service (step 11 of diagram Service startup using systemd and step 10 of diagram Service startup using upstart) this JAVA_XTRAOPTS string goes through a basic validity check and is then passed to the JVM.<br />
<br />
=== Format >= 7.8.4 ===<br />
Beginning from version 7.8.4 the options are split up and sorted into different categories like:<br />
<br />
JAVA_OPTS_GC="-XX:+UseConcMarkSweepGC -XX:+UseParNewGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:NewRatio=3 -XX:+DisableExplicitGC"<br />
JAVA_OPTS_LOG="-Dlogback.threadlocal.put.duplicate=false -XX:-OmitStackTraceInFastThrow"<br />
JAVA_OPTS_MEM="-XX:MaxHeapSize=512M -XX:MaxPermSize=256M -XX:+UseTLAB"<br />
JAVA_OPTS_NET="-Dsun.net.inetaddr.ttl=3600 -Dnetworkaddress.cache.ttl=3600 -Dnetworkaddress.cache.negative.ttl=10"<br />
JAVA_OPTS_OSGI="-Dosgi.compatibility.bootdelegation=false"<br />
JAVA_OPTS_SERVER="-server -Djava.awt.headless=true"<br />
<br />
JAVA_OPTS_OTHER=""<br />
<br />
#JAVA_OPTS_DEBUG="-XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/mnt/heapdump -Xloggc:/var/log/open-xchange/gc.log -verbose:gc -XX:+PrintGCDateStamps \<br />
-XX:+PrintHeapAtGC -XX:+PrintGCApplicationStoppedTime -XX:+PrintTenuringDistribution"<br />
<br />
which is much easier to read an maintain. Furthermore we added a useful '''JAVA_OPTS_DEBUG''' configuration parameter that comes preconfigured but unused/commented so that customers can quickly enable the most commonly used debugging options in case they have to provide detailed system information to Open-Xchange.<br />
<br />
During the startup of the open-xchange service (step 11 of diagram ''Service startup using systemd'' and step 10 of diagram ''Service startup using upstart'') all uncommented JAVA_OPTS_* categories will then be evaluated and combined again into a single string of commandline options that can then be passed to the JVM.<br />
<br />
=== Upgrade procedure ===<br />
During the upgrade '''7.8.3-revX -> 7.8.4-revY''' existing JAVA_XTRAOPTS will automatically be split up and migrated to the new categories approach. A backup of the file (ox-scriptconf.sh.timestamp) will be created and admins are urged to check for proper migration before deleting the backup file like it's already done with *.dpkg-dist or *.rpmnew files on Debian or RHEL/SLES distributions.<br />
<br />
== Encoding detection during JVM startup ==<br />
<br />
=== Background Infos ===<br />
All of en_US.UTF-8, .utf8, .utf-8, .UTF_8 will result in a valid UTF-8 charmap<br />
<br />
==== JVM startup ====<br />
<br />
The default file.encoding detection is done at JVM startup based upon LC_CTYPE which makes sense as it's documented as:<br />
<br />
CODESET (LC_CTYPE)<br />
Return a string with the name of the character encoding used in the selected locale, such as "UTF-8", "ISO-8859-1",<br />
or "ANSI_X3.4-1968" (better known as US-ASCII). This is the same string that you get with "locale charmap".<br />
For a list of character encoding names, try "locale -m", cf. locale(1).<br />
<br />
Furthermore this influences the defaultCharset selection of the JVM.<br />
<br />
/**<br />
* Returns the default charset of this Java virtual machine.<br />
*<br />
* The default charset is determined during virtual-machine startup and<br />
* typically depends upon the locale and charset of the underlying<br />
* operating system.<br />
*<br />
* @return A charset object for the default charset<br />
*<br />
* @since 1.5<br />
*/<br />
public static Charset defaultCharset() {<br />
if (defaultCharset == null) {<br />
synchronized (Charset.class) {<br />
String csn = AccessController.doPrivileged(<br />
new GetPropertyAction("file.encoding"));<br />
Charset cs = lookup(csn);<br />
if (cs != null)<br />
defaultCharset = cs;<br />
else<br />
defaultCharset = forName("UTF-8");<br />
}<br />
}<br />
return defaultCharset;<br />
}<br />
<br />
Although we try to specify valid unicode encodings for every IO operations the Open-Xchange middleware does some parts of the JVM don't allow to specify the encoding to use and falls back to the detected default encoding. That's why admins have to make sure to provide a proper unicode environment for the middleware service during every start or use documented workarounds provided by Open-Xchange (e.g. )<br />
<br />
=== Query which file encoding was detected by the JVM during startup ===<br />
[[File:visualvm-file.encoding.png]]<br />
<br />
The file encoding that was detected by the JVM during startup can be queried via JMX. You can use graphical management tools like [[https://visualvm.github.io/|visualvm]], query our [[https://oxpedia.org/wiki/index.php?title=Jolokia|Jolokia]] JMX bridge via http or just <br />
<br />
/opt/open-xchange/sbin/showruntimestats -r | sed -ne 's/.*\({key=file.encoding, value=[^}]*}\).*/\1/p'<br />
{key=file.encoding, value=UTF-8}<br />
<br />
=== Locale passing from client to server via SSH ===<br />
<br />
The following setting of sshd allows clients to specify which locale to use on the server and might lead to some confusion or unexpected behaviour when checking the system locale or even to the usage of broken locales during middleware startup if your client sends misconfigured or missing locale configuration to the server.<br />
<br />
grep AcceptEnv /etc/ssh/sshd_config <br />
AcceptEnv LANG LC_*<br />
<br />
For example when i connect to the machine my local german locale is sent to the server <br />
<br />
[vagrant@showcase-community-centos-7 ~]$ locale -v <br />
LANG=de_DE.UTF-8<br />
LC_CTYPE="de_DE.UTF-8"<br />
LC_NUMERIC=de_DE.UTF-8<br />
LC_TIME=de_DE.UTF-8<br />
LC_COLLATE="de_DE.UTF-8"<br />
LC_MONETARY=de_DE.UTF-8<br />
LC_MESSAGES="de_DE.UTF-8"<br />
LC_PAPER=de_DE.UTF-8<br />
LC_NAME=de_DE.UTF-8<br />
LC_ADDRESS=de_DE.UTF-8<br />
LC_TELEPHONE=de_DE.UTF-8<br />
LC_MEASUREMENT=de_DE.UTF-8<br />
LC_IDENTIFICATION=de_DE.UTF-8<br />
LC_ALL=<br />
<br />
But after specifying that i want a new login shell the default locale configured for that user/system will be used instead!<br />
<br />
[root@showcase-community-centos-7 vagrant]# su -s /bin/bash open-xchange -l<br />
-bash-4.2$ locale -v <br />
LANG=en_US.UTF-8<br />
LC_CTYPE="en_US.UTF-8"<br />
LC_NUMERIC="en_US.UTF-8"<br />
LC_TIME="en_US.UTF-8"<br />
LC_COLLATE="en_US.UTF-8"<br />
LC_MONETARY="en_US.UTF-8"<br />
LC_MESSAGES="en_US.UTF-8"<br />
LC_PAPER="en_US.UTF-8"<br />
LC_NAME="en_US.UTF-8"<br />
LC_ADDRESS="en_US.UTF-8"<br />
LC_TELEPHONE="en_US.UTF-8"<br />
LC_MEASUREMENT="en_US.UTF-8"<br />
LC_IDENTIFICATION="en_US.UTF-8"<br />
LC_ALL=<br />
<br />
Add details from https://bugs.open-xchange.com/show_bug.cgi?id=51074<br />
<br />
== Warnings during early start ==<br />
<br />
Early warnings before the start of the JVM and related logging libraries like logback and logstash forwarding are redirected to '''/var/log/open-xchange/open-xchange-console.log'''.<br />
<br />
=== Limits ===<br />
If the open-xchange startup script isn't able to set the proper limits on CentOS 6 due to e.g. hard limits being enforced via pam_limits you'll see the following warning in the console log during startup<br />
<br />
singlenode open-xchange # cat open-xchange-console.log<br />
/opt/open-xchange/sbin/open-xchange: line 115: ulimit: max user processes: cannot modify limit: Operation not permitted<br />
<br />
=== Locale ===<br />
If the open-xchange startup script isn't able to detect a unicode locale you'll see a warning like the following (the current charmap might differ)<br />
<br />
WARNING: Couldn't detect proper unicode charmap in locale category LC_CTYPE.<br />
WARNING: Current charmap: "ANSI_X3.4-1968" might cause encoding problems.<br />
<br />
== Summary: Open-Xchange middleware on specific distros ==<br />
The support for the mentioned mechanism of resource control differ depending on the distribution and the init system in use.<br />
<br />
=== Debian 7 ===<br />
;Init<br />
: System V style<br />
;OX Configurable Limits/Defaults<br />
: nofile, nproc<br />
<br />
The mentioned limits can be configured via '''/opt/open-xchange/etc/ox-scriptconf.sh'''. The limits are applied via ulimit in the service's init script. The open-xchange service is finally started via start-stop-daemon which doesn't doesn't consider '''/etc/security/limits.*'''<br />
<br />
=== RHEL 6 / CentOS 6 ===<br />
;Init<br />
: Upstart, System V compatible<br />
;OX Configurable Limits/Defaults<br />
: nofile, nproc<br />
<br />
The mentioned limits can be configured via '''/opt/open-xchange/etc/ox-scriptconf.sh'''. The limits are applied via ulimit in the service's init script. Furthermore as the open-xchange service is finally started via '''su ... open-xchange''' on this distro a user session is opened via su/pam and the default CentOS pam config reads the '''/etc/security/limits.*''' configuration by loading the pam stack like:<br />
<br />
: '''/etc/pam.d/su'''<br />
:: '''-> /etc/pam.d/system-auth'''<br />
::: '''-> pam_limits.so'''<br />
<br />
If NPROC isn't configured for the open-xchange-server it's restricted to '''1024''' globally by default to prevent accidental fork bombs, see '''/etc/security/limits.d/90-nproc.conf''' which can result in severe problems modern multithreaded applications.<br />
<br />
=== RHEL 7 / CentOS 7 / Debian 8 / SLE 12 ===<br />
;Init<br />
: Systemd<br />
;OX Configurable Limits/Defaults<br />
: nofile, nproc<br />
<br />
For systemd the default limits are configured directly in the service's unit file that is shipped by OX and located at '''/usr/lib/systemd/system/open-xchange.service'''. The drop-in config to override or extend the default unit file is located at '''/etc/systemd/system/open-xchange.service.d/limits.conf'''. [https://www.freedesktop.org/software/systemd/man/systemd.exec.html Systemd.exec] shows a whole lot of options that can be used by admins to adapt the default service to their specific needs.</div>Marenshttps://oxpedia.org/wiki/index.php?title=Sandbox:Middleware_Startup/sandbox&diff=23225Sandbox:Middleware Startup/sandbox2017-04-24T19:48:01Z<p>Marens: /* Format >= 7.8.4 */</p>
<hr />
<div><br />
== Systemd ==<br />
<br />
==== Service definition ====<br />
The default service definition as shipped by Open-Xchange is rather concise and easy to read. You can check by looking at the service file that is installed to either /usr/lib/systemd or /lib/systemd depending on your distribution of choice.<br />
<br />
singlenode:~ # cat /usr/lib/systemd/system/open-xchange.service <br />
[Unit]<br />
After=remote-fs.target<br />
After=time-sync.target ypbind.service sendmail.service cyrus.service <br />
<br />
[Service]<br />
User=open-xchange<br />
PermissionsStartOnly=true<br />
TimeoutStartSec=0<br />
ExecStartPre=/opt/open-xchange/sbin/triggerupdatethemes -u<br />
ExecStart=/opt/open-xchange/sbin/open-xchange<br />
ExecStop=/opt/open-xchange/sbin/shutdown -w<br />
ExecReload=/opt/open-xchange/sbin/triggerreloadconfiguration -d<br />
KillMode=process<br />
LimitNOFILE=65536<br />
LimitNPROC=65536<br />
<br />
[Install]<br />
WantedBy=multi-user.target<br />
<br />
==== Drop-in configs ====<br />
Drop in configs allow administrators to easily adapt and override default service unit files. So if you want to change the default limits or add additional limits have a look at our default example drop-in config which is located at /etc/systemd/system/open-xchange.service.d/limits.conf<br />
<br />
singlenode:~ # cat /etc/systemd/system/open-xchange.service.d/limits.conf <br />
# Override and add options in this file<br />
# See systemd.exec(5) for other limits<br />
<br />
[Service]<br />
#LimitNPROC=65536<br />
#LimitNOFILE=65536<br />
<br />
The systemd.exec documentation shows a whole lot of options that can be used by admins to adapt the default service to their specific needs.<br />
<br />
<br clear=all><br />
[[File:mw_start_sysd.png|thumb|Service startup using systemd|left|1000px]]<br />
<br clear=all><br />
<br />
== System V / Upstart ==<br />
<br />
==== Service definition ====<br />
<br />
The default service definition as shipped by Open-Xchange can be found at /etc/init.d/open-xchange and will differ slightly depending on the distro in use.<br />
<br />
<br clear=all><br />
[[File:mw_start_sysv.png|thumb|Service startup using upstart|left|1000px]]<br />
<br clear=all><br />
== Apply Resource limits ==<br />
<br />
As you can see in the diagrams above one of the differences is that:<br />
<br />
* systemd reads all the infos wrt. resource limitations from the original service file and then applies possible further restrictions from the drop-in configs (steps 2 and 8 in the diagram ''Service startup using systemd'') and limits configured in /opt/open-xchange/etc/ox-scriptconf.sh aren't considered anylonger<br />
* the older System V style init has to source /opt/open-xchange/etc/ox-scriptconf.sh to be able to read the configs for the maximum number of processes (NPROC) and files(NRFILES) that the middleware shall be allowed to create/open (steps 3,4 and 15 in the diagram ''Service startup using upstart'').<br />
<br />
=== Background Infos ===<br />
<br />
Several ways exist to restrict resources on a linux system from a global level down to user/groups or even shells and the processes started by them.<br />
<br />
==== Sysctl ====<br />
Sysctl is used to modify kernel parameters at runtime. E.g. to set the maximum number of files<br />
<br />
$ sysctl -w fs.file-max=100000<br />
<br />
To permanently set them append to the main configuration file and reload the settings<br />
<br />
$ echo fs.file-max=100000 >> /etc/sysctl.conf<br />
$ sysctl -p<br />
<br />
More infos can be found via '''man sysctl'''<br />
<br />
==== Limits.conf ====<br />
<br />
Allows to restrict resources an a global, group or user level. E.g:<br />
<br />
$ cat /etc/security/limits.d/90-nproc.conf <br />
# Default limit for number of user's processes to prevent<br />
# accidental fork bombs.<br />
# See rhbz #432903 for reasoning.<br />
<br />
* soft nproc 1024<br />
<br />
From '''man limits.conf''':<br />
<blockquote><br />
''Also, please note that all limit settings are set per login. They are not global, nor are they permanent; existing only for the duration of the session.''<br />
</blockquote><br />
<br />
The limits per login are applied via the ''pam'' stack. See '''man pam''' and '''man pam_limits''' for more details. As those limits are bound to sessions they don't affect most daemons started by our supported init systems or init utils. Most state that they are ignored by design, see [https://bugs.launchpad.net/ubuntu/+source/upstart/+bug/938669/comments/4 upstart], [https://bugzilla.redhat.com/show_bug.cgi?id=754285#c1 systemd] and [https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=302079 start-stop-daemon] <br />
<br />
===== Ulimit =====<br />
From '''man bash'''<br />
<blockquote><br />
ulimit [-HSTabcdefilmnpqrstuvx [limit]]<br />
Provides control over the resources available to the shell and to processes started by it, on systems that allow such control.<br />
</blockquote><br />
This is what we use in our System V compatible init scripts to increase resources for the open-xchange process across multiple distros. Currently only ''the maximum number of processes'' and ''the maximum number of open file descriptors available to a single user'' are increased via ulimit. The values are specified in '''/opt/open-xchange/ox-scriptconf.sh'''<br />
<br />
==== Systemd ====<br />
<br />
===== Control Groups=====<br />
<br />
Control groups should only affect the OX middleware if you create/manage them yourself of if you are using a ''modern'' distribution that already uses systemd as init.<br />
<br />
Citing from the kernel cgroup [https://www.kernel.org/doc/Documentation/cgroup-v2.txt documentation]:<br />
<br />
<blockquote><br />
1-2. What is cgroup?<br />
<br />
cgroup is a mechanism to organize processes hierarchically and<br />
distribute system resources along the hierarchy in a controlled and<br />
configurable manner.<br />
<br />
cgroup is largely composed of two parts - the core and controllers.<br />
cgroup core is primarily responsible for hierarchically organizing<br />
processes. A cgroup controller is usually responsible for<br />
distributing a specific type of system resource along the hierarchy<br />
although there are utility controllers which serve purposes other than<br />
resource distribution.<br />
<br />
cgroups form a tree structure and every process in the system belongs<br />
to one and only one cgroup. All threads of a process belong to the<br />
same cgroup. On creation, all processes are put in the cgroup that<br />
the parent process belongs to at the time. A process can be migrated<br />
to another cgroup. Migration of a process doesn't affect already<br />
existing descendant processes.<br />
<br />
Following certain structural constraints, controllers may be enabled or<br />
disabled selectively on a cgroup. All controller behaviors are<br />
hierarchical - if a controller is enabled on a cgroup, it affects all<br />
processes which belong to the cgroups consisting the inclusive<br />
sub-hierarchy of the cgroup. When a controller is enabled on a nested<br />
cgroup, it always restricts the resource distribution further. The<br />
restrictions set closer to the root in the hierarchy can not be<br />
overridden from further away.<br />
</blockquote><br />
<br />
So processes are organized into a tree structure of control groups and controllers are responsible for the distribution of resources. So what kind of controllers exist?<br />
<br />
<blockquote><br />
5. Controllers<br />
<br />
5-1. CPU<br />
<br />
The "cpu" controllers regulates distribution of CPU cycles. This<br />
controller implements weight and absolute bandwidth limit models for<br />
normal scheduling policy and absolute bandwidth allocation model for<br />
realtime scheduling policy.<br />
<br />
5-2. Memory<br />
<br />
The "memory" controller regulates distribution of memory.<br />
...<br />
While not completely water-tight, all major memory usages by a given<br />
cgroup are tracked so that the total memory consumption can be<br />
accounted and controlled to a reasonable extent.<br />
<br />
5-3. IO<br />
<br />
The "io" controller regulates the distribution of IO resources. This<br />
controller implements both weight based and absolute bandwidth or IOPS<br />
limit distribution; however, weight based distribution is available<br />
only if cfq-iosched is in use and neither scheme is available for<br />
blk-mq devices.<br />
</blockquote><br />
<br />
The open-xchange service is simply put into the default ''system.slice'' without applying further limits.<br />
<br />
singlenode$ systemd-cgls<br />
├─1 /sbin/init<br />
├─system.slice<br />
│ ├─avahi-daemon.service<br />
│ │ ├─501 avahi-daemon: running [singlenode]<br />
│ │ └─514 avahi-daemon: chroot helper<br />
│ ├─console-kit-daemon.service<br />
│ │ └─16164 /usr/sbin/console-kit-daemon --no-daemon<br />
│ ├─dbus.service<br />
│ │ └─508 /usr/bin/dbus-daemon --system --address=systemd: --nofork --nopidfile --systemd-activation<br />
│ ├─munin-node.service<br />
│ │ └─4290 /usr/bin/perl -wT /usr/sbin/munin-node<br />
│ ├─open-xchange.service<br />
│ │ └─6037 /usr/bin/java -Dsun.net.inetaddr.ttl=3600 -Dnetworkaddress.cache.ttl=3600 -Dnetworkaddress.cache.negative.ttl=10 ...<br />
<br />
To check all the details use<br />
<br />
singlenode:~ # systemctl show system.slice<br />
<br />
==== Limits besides control groups ====<br />
Besides control groups systemd allows you to apply other limits to the execution environment of your service. Here we can apply the limits that would normally be applied via ''limits.conf'' or ''ulimit''. Systemd uses [http://man7.org/linux/man-pages/man2/setrlimit.2.html setrlimit] for this. The options that we set by default are:<br />
<br />
* LimitNOFILE<br />
* LimitNPROC<br />
<br />
which you have already seen when we took a look at the [[#Service_definiton|service definition]] and [[#Drop-in_config|drop-in configs]]<br />
<br />
== Verify limits ==<br />
<br />
=== System V ===<br />
<br />
Get the proper pid of the java process started for the middleware by either checking '''ps faux'''<br />
root 4103 0.0 0.0 66528 1868 pts/0 S 22:05 0:00 su -s /bin/bash open-xchange -c /opt/open-xchange/sbin/open-xchange<br />
494 4109 9.5 19.8 2224644 381312 ? Ssl 22:05 1:12 \_ /usr/bin/java -Dsun.net.inetaddr.ttl=3600 -Dnetworkaddress.cache.ttl=3600<br />
<br />
or programmatically<br />
singlenode:~ # pid=$(pgrep -P $(</var/run/open-xchange.pid) java)<br />
<br />
and check the limits applied for that process<br />
singlenode:~ # cat /proc/$pid/limits<br />
Limit Soft Limit Hard Limit Units<br />
Max cpu time unlimited unlimited seconds<br />
Max file size unlimited unlimited bytes<br />
Max data size unlimited unlimited bytes<br />
Max stack size 8388608 unlimited bytes<br />
Max core file size 0 unlimited bytes<br />
Max resident set unlimited unlimited bytes<br />
Max processes 65536 65536 processes<br />
Max open files 65536 65536 files<br />
Max locked memory 65536 65536 bytes<br />
Max address space unlimited unlimited bytes<br />
Max file locks unlimited unlimited locks<br />
Max pending signals 24254 24254 signals<br />
Max msgqueue size 819200 819200 bytes<br />
Max nice priority 0 0<br />
Max realtime priority 0 0<br />
Max realtime timeout unlimited unlimited us<br />
<br />
=== Systemd ===<br />
singlenode:~ # systemctl show open-xchange | grep Limit<br />
StartLimitInterval=10000000<br />
StartLimitBurst=5<br />
StartLimitAction=none<br />
MemoryLimit=18446744073709551615<br />
LimitCPU=18446744073709551615<br />
LimitFSIZE=18446744073709551615<br />
LimitDATA=18446744073709551615<br />
LimitSTACK=18446744073709551615<br />
LimitCORE=18446744073709551615<br />
LimitRSS=18446744073709551615<br />
LimitNOFILE=65536<br />
LimitAS=18446744073709551615<br />
LimitNPROC=65536<br />
LimitMEMLOCK=65536<br />
LimitLOCKS=18446744073709551615<br />
LimitSIGPENDING=19827<br />
LimitMSGQUEUE=819200<br />
LimitNICE=0<br />
LimitRTPRIO=0<br />
LimitRTTIME=18446744073709551615<br />
<br />
== Evaluate JVM commandline options ==<br />
<br />
Before starting Java and the OSGi framework with all the Open-Xchange middleware bundles we have to evaluate several important JVM commandline options that influence the performance and stability of the Open-Xchange middleware stack. These commandline options are specified in '''/opt/open-xchange/etc/ox-scriptconf.sh'''<br />
<br />
=== Format < 7.8.4 ===<br />
For version up to 7.8.3 the JVM configuration options were all specified via a single property '''JAVA_XTRAOPTS''' which resulted in a rather long configuration string that was hard to read and maintain.<br />
<br />
JAVA_XTRAOPTS="-Dsun.net.inetaddr.ttl=3600 -Dnetworkaddress.cache.ttl=3600 -Dnetworkaddress.cache.negative.ttl=10 \<br />
-Dlogback.threadlocal.put.duplicate=false -server -Djava.awt.headless=true -Xmx512M -XX:MaxPermSize=256m -XX:+UseConcMarkSweepGC \<br />
-XX:+UseParNewGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:NewRatio=3 -XX:+UseTLAB \<br />
-Dosgi.compatibility.bootdelegation=false -XX:-OmitStackTraceInFastThrow"<br />
<br />
During the startup of the open-xchange service (step 11 of diagram Service startup using systemd and step 10 of diagram Service startup using upstart) this JAVA_XTRAOPTS string goes through a basic validity check and is then passed to the JVM.<br />
<br />
=== Format >= 7.8.4 ===<br />
Beginning from version 7.8.4 the options are split up and sorted into different categories like:<br />
<br />
JAVA_OPTS_GC="-XX:+UseConcMarkSweepGC -XX:+UseParNewGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:NewRatio=3 -XX:+DisableExplicitGC"<br />
JAVA_OPTS_LOG="-Dlogback.threadlocal.put.duplicate=false -XX:-OmitStackTraceInFastThrow"<br />
JAVA_OPTS_MEM="-XX:MaxHeapSize=512M -XX:MaxPermSize=256M -XX:+UseTLAB"<br />
JAVA_OPTS_NET="-Dsun.net.inetaddr.ttl=3600 -Dnetworkaddress.cache.ttl=3600 -Dnetworkaddress.cache.negative.ttl=10"<br />
JAVA_OPTS_OSGI="-Dosgi.compatibility.bootdelegation=false"<br />
JAVA_OPTS_SERVER="-server -Djava.awt.headless=true"<br />
<br />
JAVA_OPTS_OTHER=""<br />
<br />
#JAVA_OPTS_DEBUG="-XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/mnt/heapdump -Xloggc:/var/log/open-xchange/gc.log -verbose:gc -XX:+PrintGCDateStamps \<br />
-XX:+PrintHeapAtGC -XX:+PrintGCApplicationStoppedTime -XX:+PrintTenuringDistribution"<br />
<br />
which is much easier to read an maintain. Furthermore we added a useful '''JAVA_OPTS_DEBUG''' configuration parameter that comes preconfigured but unused/commented so that customers can quickly enable the most commonly used debugging options in case they have to provide detailed system information to Open-Xchange.<br />
<br />
During the startup of the open-xchange service (step 11 of diagram ''Service startup using systemd'' and step 10 of diagram ''Service startup using upstart'') all uncommented JAVA_OPTS_* categories will then be evaluated and combined again into a single string of commandline options that can then be passed to the JVM.<br />
<br />
=== Upgrade procedure ===<br />
During the upgrade '''7.8.3-revX -> 7.8.4-revY''' existing JAVA_XTRAOPTS will automatically be split up and migrated to the new categories approach. A backup of the file (ox-scriptconf.sh.timestamp) will be created and admins are urged to check for proper migration before deleting the backup file like it's already done with *.dpkg-dist or *.rpmnew files on Debian or RHEL/SLES distributions.<br />
<br />
== Encoding detection during JVM startup ==<br />
<br />
=== Background Infos ===<br />
All of en_US.UTF-8, .utf8, .utf-8, .UTF_8 will result in a valid UTF-8 charmap<br />
<br />
==== JVM startup ====<br />
<br />
The default file.encoding detection is done at JVM startup based upon LC_CTYPE which makes sense as it's documented as:<br />
<br />
CODESET (LC_CTYPE)<br />
Return a string with the name of the character encoding used in the selected locale, such as "UTF-8", "ISO-8859-1",<br />
or "ANSI_X3.4-1968" (better known as US-ASCII). This is the same string that you get with "locale charmap".<br />
For a list of character encoding names, try "locale -m", cf. locale(1).<br />
<br />
Furthermore this influences the defaultCharset selection of the JVM.<br />
<br />
/**<br />
* Returns the default charset of this Java virtual machine.<br />
*<br />
* The default charset is determined during virtual-machine startup and<br />
* typically depends upon the locale and charset of the underlying<br />
* operating system.<br />
*<br />
* @return A charset object for the default charset<br />
*<br />
* @since 1.5<br />
*/<br />
public static Charset defaultCharset() {<br />
if (defaultCharset == null) {<br />
synchronized (Charset.class) {<br />
String csn = AccessController.doPrivileged(<br />
new GetPropertyAction("file.encoding"));<br />
Charset cs = lookup(csn);<br />
if (cs != null)<br />
defaultCharset = cs;<br />
else<br />
defaultCharset = forName("UTF-8");<br />
}<br />
}<br />
return defaultCharset;<br />
}<br />
<br />
Although we try to specify valid unicode encodings for every IO operations the Open-Xchange middleware does some parts of the JVM don't allow to specify the encoding to use and falls back to the detected default encoding. That's why admins have to make sure to provide a proper unicode environment for the middleware service during every start or use documented workarounds provided by Open-Xchange (e.g. )<br />
<br />
=== Query which file encoding was detected by the JVM during startup ===<br />
<br />
/opt/open-xchange/sbin/showruntimestats -r | sed -ne 's/.*\({key=file.encoding, value=[^}]*}\).*/\1/p'<br />
{key=file.encoding, value=UTF-8}<br />
<br />
=== Locale passing from client to server via SSH ===<br />
<br />
The following setting of sshd allows clients to specify which locale to use on the server and might lead to some confusion or unexpected behaviour when checking the system locale or even to the usage of broken locales during middleware startup if your client sends misconfigured or missing locale configuration to the server.<br />
<br />
grep AcceptEnv /etc/ssh/sshd_config <br />
AcceptEnv LANG LC_*<br />
<br />
For example when i connect to the machine my local german locale is sent to the server <br />
<br />
[vagrant@showcase-community-centos-7 ~]$ locale -v <br />
LANG=de_DE.UTF-8<br />
LC_CTYPE="de_DE.UTF-8"<br />
LC_NUMERIC=de_DE.UTF-8<br />
LC_TIME=de_DE.UTF-8<br />
LC_COLLATE="de_DE.UTF-8"<br />
LC_MONETARY=de_DE.UTF-8<br />
LC_MESSAGES="de_DE.UTF-8"<br />
LC_PAPER=de_DE.UTF-8<br />
LC_NAME=de_DE.UTF-8<br />
LC_ADDRESS=de_DE.UTF-8<br />
LC_TELEPHONE=de_DE.UTF-8<br />
LC_MEASUREMENT=de_DE.UTF-8<br />
LC_IDENTIFICATION=de_DE.UTF-8<br />
LC_ALL=<br />
<br />
But after specifying that i want a new login shell the default locale configured for that user/system will be used instead!<br />
<br />
[root@showcase-community-centos-7 vagrant]# su -s /bin/bash open-xchange -l<br />
-bash-4.2$ locale -v <br />
LANG=en_US.UTF-8<br />
LC_CTYPE="en_US.UTF-8"<br />
LC_NUMERIC="en_US.UTF-8"<br />
LC_TIME="en_US.UTF-8"<br />
LC_COLLATE="en_US.UTF-8"<br />
LC_MONETARY="en_US.UTF-8"<br />
LC_MESSAGES="en_US.UTF-8"<br />
LC_PAPER="en_US.UTF-8"<br />
LC_NAME="en_US.UTF-8"<br />
LC_ADDRESS="en_US.UTF-8"<br />
LC_TELEPHONE="en_US.UTF-8"<br />
LC_MEASUREMENT="en_US.UTF-8"<br />
LC_IDENTIFICATION="en_US.UTF-8"<br />
LC_ALL=<br />
<br />
Add details from https://bugs.open-xchange.com/show_bug.cgi?id=51074<br />
<br />
== Warnings during early start ==<br />
<br />
Early warnings before the start of the JVM and related logging libraries like logback and logstash forwarding are redirected to '''/var/log/open-xchange/open-xchange-console.log'''.<br />
<br />
=== Limits ===<br />
If the open-xchange startup script isn't able to set the proper limits on CentOS 6 due to e.g. hard limits being enforced via pam_limits you'll see the following warning in the console log during startup<br />
<br />
singlenode open-xchange # cat open-xchange-console.log<br />
/opt/open-xchange/sbin/open-xchange: line 115: ulimit: max user processes: cannot modify limit: Operation not permitted<br />
<br />
=== Locale ===<br />
If the open-xchange startup script isn't able to detect a unicode locale you'll see a warning like the following (the current charmap might differ)<br />
<br />
WARNING: Couldn't detect proper unicode charmap in locale category LC_CTYPE.<br />
WARNING: Current charmap: "ANSI_X3.4-1968" might cause encoding problems.<br />
<br />
== Summary: Open-Xchange middleware on specific distros ==<br />
The support for the mentioned mechanism of resource control differ depending on the distribution and the init system in use.<br />
<br />
=== Debian 7 ===<br />
;Init<br />
: System V style<br />
;OX Configurable Limits/Defaults<br />
: nofile, nproc<br />
<br />
The mentioned limits can be configured via '''/opt/open-xchange/etc/ox-scriptconf.sh'''. The limits are applied via ulimit in the service's init script. The open-xchange service is finally started via start-stop-daemon which doesn't doesn't consider '''/etc/security/limits.*'''<br />
<br />
=== RHEL 6 / CentOS 6 ===<br />
;Init<br />
: Upstart, System V compatible<br />
;OX Configurable Limits/Defaults<br />
: nofile, nproc<br />
<br />
The mentioned limits can be configured via '''/opt/open-xchange/etc/ox-scriptconf.sh'''. The limits are applied via ulimit in the service's init script. Furthermore as the open-xchange service is finally started via '''su ... open-xchange''' on this distro a user session is opened via su/pam and the default CentOS pam config reads the '''/etc/security/limits.*''' configuration by loading the pam stack like:<br />
<br />
: '''/etc/pam.d/su'''<br />
:: '''-> /etc/pam.d/system-auth'''<br />
::: '''-> pam_limits.so'''<br />
<br />
If NPROC isn't configured for the open-xchange-server it's restricted to '''1024''' globally by default to prevent accidental fork bombs, see '''/etc/security/limits.d/90-nproc.conf''' which can result in severe problems modern multithreaded applications.<br />
<br />
=== RHEL 7 / CentOS 7 / Debian 8 / SLE 12 ===<br />
;Init<br />
: Systemd<br />
;OX Configurable Limits/Defaults<br />
: nofile, nproc<br />
<br />
For systemd the default limits are configured directly in the service's unit file that is shipped by OX and located at '''/usr/lib/systemd/system/open-xchange.service'''. The drop-in config to override or extend the default unit file is located at '''/etc/systemd/system/open-xchange.service.d/limits.conf'''. [https://www.freedesktop.org/software/systemd/man/systemd.exec.html Systemd.exec] shows a whole lot of options that can be used by admins to adapt the default service to their specific needs.</div>Marenshttps://oxpedia.org/wiki/index.php?title=Sandbox:Middleware_Startup/sandbox&diff=23224Sandbox:Middleware Startup/sandbox2017-04-24T19:47:06Z<p>Marens: /* Format < 7.8.4 */</p>
<hr />
<div><br />
== Systemd ==<br />
<br />
==== Service definition ====<br />
The default service definition as shipped by Open-Xchange is rather concise and easy to read. You can check by looking at the service file that is installed to either /usr/lib/systemd or /lib/systemd depending on your distribution of choice.<br />
<br />
singlenode:~ # cat /usr/lib/systemd/system/open-xchange.service <br />
[Unit]<br />
After=remote-fs.target<br />
After=time-sync.target ypbind.service sendmail.service cyrus.service <br />
<br />
[Service]<br />
User=open-xchange<br />
PermissionsStartOnly=true<br />
TimeoutStartSec=0<br />
ExecStartPre=/opt/open-xchange/sbin/triggerupdatethemes -u<br />
ExecStart=/opt/open-xchange/sbin/open-xchange<br />
ExecStop=/opt/open-xchange/sbin/shutdown -w<br />
ExecReload=/opt/open-xchange/sbin/triggerreloadconfiguration -d<br />
KillMode=process<br />
LimitNOFILE=65536<br />
LimitNPROC=65536<br />
<br />
[Install]<br />
WantedBy=multi-user.target<br />
<br />
==== Drop-in configs ====<br />
Drop in configs allow administrators to easily adapt and override default service unit files. So if you want to change the default limits or add additional limits have a look at our default example drop-in config which is located at /etc/systemd/system/open-xchange.service.d/limits.conf<br />
<br />
singlenode:~ # cat /etc/systemd/system/open-xchange.service.d/limits.conf <br />
# Override and add options in this file<br />
# See systemd.exec(5) for other limits<br />
<br />
[Service]<br />
#LimitNPROC=65536<br />
#LimitNOFILE=65536<br />
<br />
The systemd.exec documentation shows a whole lot of options that can be used by admins to adapt the default service to their specific needs.<br />
<br />
<br clear=all><br />
[[File:mw_start_sysd.png|thumb|Service startup using systemd|left|1000px]]<br />
<br clear=all><br />
<br />
== System V / Upstart ==<br />
<br />
==== Service definition ====<br />
<br />
The default service definition as shipped by Open-Xchange can be found at /etc/init.d/open-xchange and will differ slightly depending on the distro in use.<br />
<br />
<br clear=all><br />
[[File:mw_start_sysv.png|thumb|Service startup using upstart|left|1000px]]<br />
<br clear=all><br />
== Apply Resource limits ==<br />
<br />
As you can see in the diagrams above one of the differences is that:<br />
<br />
* systemd reads all the infos wrt. resource limitations from the original service file and then applies possible further restrictions from the drop-in configs (steps 2 and 8 in the diagram ''Service startup using systemd'') and limits configured in /opt/open-xchange/etc/ox-scriptconf.sh aren't considered anylonger<br />
* the older System V style init has to source /opt/open-xchange/etc/ox-scriptconf.sh to be able to read the configs for the maximum number of processes (NPROC) and files(NRFILES) that the middleware shall be allowed to create/open (steps 3,4 and 15 in the diagram ''Service startup using upstart'').<br />
<br />
=== Background Infos ===<br />
<br />
Several ways exist to restrict resources on a linux system from a global level down to user/groups or even shells and the processes started by them.<br />
<br />
==== Sysctl ====<br />
Sysctl is used to modify kernel parameters at runtime. E.g. to set the maximum number of files<br />
<br />
$ sysctl -w fs.file-max=100000<br />
<br />
To permanently set them append to the main configuration file and reload the settings<br />
<br />
$ echo fs.file-max=100000 >> /etc/sysctl.conf<br />
$ sysctl -p<br />
<br />
More infos can be found via '''man sysctl'''<br />
<br />
==== Limits.conf ====<br />
<br />
Allows to restrict resources an a global, group or user level. E.g:<br />
<br />
$ cat /etc/security/limits.d/90-nproc.conf <br />
# Default limit for number of user's processes to prevent<br />
# accidental fork bombs.<br />
# See rhbz #432903 for reasoning.<br />
<br />
* soft nproc 1024<br />
<br />
From '''man limits.conf''':<br />
<blockquote><br />
''Also, please note that all limit settings are set per login. They are not global, nor are they permanent; existing only for the duration of the session.''<br />
</blockquote><br />
<br />
The limits per login are applied via the ''pam'' stack. See '''man pam''' and '''man pam_limits''' for more details. As those limits are bound to sessions they don't affect most daemons started by our supported init systems or init utils. Most state that they are ignored by design, see [https://bugs.launchpad.net/ubuntu/+source/upstart/+bug/938669/comments/4 upstart], [https://bugzilla.redhat.com/show_bug.cgi?id=754285#c1 systemd] and [https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=302079 start-stop-daemon] <br />
<br />
===== Ulimit =====<br />
From '''man bash'''<br />
<blockquote><br />
ulimit [-HSTabcdefilmnpqrstuvx [limit]]<br />
Provides control over the resources available to the shell and to processes started by it, on systems that allow such control.<br />
</blockquote><br />
This is what we use in our System V compatible init scripts to increase resources for the open-xchange process across multiple distros. Currently only ''the maximum number of processes'' and ''the maximum number of open file descriptors available to a single user'' are increased via ulimit. The values are specified in '''/opt/open-xchange/ox-scriptconf.sh'''<br />
<br />
==== Systemd ====<br />
<br />
===== Control Groups=====<br />
<br />
Control groups should only affect the OX middleware if you create/manage them yourself of if you are using a ''modern'' distribution that already uses systemd as init.<br />
<br />
Citing from the kernel cgroup [https://www.kernel.org/doc/Documentation/cgroup-v2.txt documentation]:<br />
<br />
<blockquote><br />
1-2. What is cgroup?<br />
<br />
cgroup is a mechanism to organize processes hierarchically and<br />
distribute system resources along the hierarchy in a controlled and<br />
configurable manner.<br />
<br />
cgroup is largely composed of two parts - the core and controllers.<br />
cgroup core is primarily responsible for hierarchically organizing<br />
processes. A cgroup controller is usually responsible for<br />
distributing a specific type of system resource along the hierarchy<br />
although there are utility controllers which serve purposes other than<br />
resource distribution.<br />
<br />
cgroups form a tree structure and every process in the system belongs<br />
to one and only one cgroup. All threads of a process belong to the<br />
same cgroup. On creation, all processes are put in the cgroup that<br />
the parent process belongs to at the time. A process can be migrated<br />
to another cgroup. Migration of a process doesn't affect already<br />
existing descendant processes.<br />
<br />
Following certain structural constraints, controllers may be enabled or<br />
disabled selectively on a cgroup. All controller behaviors are<br />
hierarchical - if a controller is enabled on a cgroup, it affects all<br />
processes which belong to the cgroups consisting the inclusive<br />
sub-hierarchy of the cgroup. When a controller is enabled on a nested<br />
cgroup, it always restricts the resource distribution further. The<br />
restrictions set closer to the root in the hierarchy can not be<br />
overridden from further away.<br />
</blockquote><br />
<br />
So processes are organized into a tree structure of control groups and controllers are responsible for the distribution of resources. So what kind of controllers exist?<br />
<br />
<blockquote><br />
5. Controllers<br />
<br />
5-1. CPU<br />
<br />
The "cpu" controllers regulates distribution of CPU cycles. This<br />
controller implements weight and absolute bandwidth limit models for<br />
normal scheduling policy and absolute bandwidth allocation model for<br />
realtime scheduling policy.<br />
<br />
5-2. Memory<br />
<br />
The "memory" controller regulates distribution of memory.<br />
...<br />
While not completely water-tight, all major memory usages by a given<br />
cgroup are tracked so that the total memory consumption can be<br />
accounted and controlled to a reasonable extent.<br />
<br />
5-3. IO<br />
<br />
The "io" controller regulates the distribution of IO resources. This<br />
controller implements both weight based and absolute bandwidth or IOPS<br />
limit distribution; however, weight based distribution is available<br />
only if cfq-iosched is in use and neither scheme is available for<br />
blk-mq devices.<br />
</blockquote><br />
<br />
The open-xchange service is simply put into the default ''system.slice'' without applying further limits.<br />
<br />
singlenode$ systemd-cgls<br />
├─1 /sbin/init<br />
├─system.slice<br />
│ ├─avahi-daemon.service<br />
│ │ ├─501 avahi-daemon: running [singlenode]<br />
│ │ └─514 avahi-daemon: chroot helper<br />
│ ├─console-kit-daemon.service<br />
│ │ └─16164 /usr/sbin/console-kit-daemon --no-daemon<br />
│ ├─dbus.service<br />
│ │ └─508 /usr/bin/dbus-daemon --system --address=systemd: --nofork --nopidfile --systemd-activation<br />
│ ├─munin-node.service<br />
│ │ └─4290 /usr/bin/perl -wT /usr/sbin/munin-node<br />
│ ├─open-xchange.service<br />
│ │ └─6037 /usr/bin/java -Dsun.net.inetaddr.ttl=3600 -Dnetworkaddress.cache.ttl=3600 -Dnetworkaddress.cache.negative.ttl=10 ...<br />
<br />
To check all the details use<br />
<br />
singlenode:~ # systemctl show system.slice<br />
<br />
==== Limits besides control groups ====<br />
Besides control groups systemd allows you to apply other limits to the execution environment of your service. Here we can apply the limits that would normally be applied via ''limits.conf'' or ''ulimit''. Systemd uses [http://man7.org/linux/man-pages/man2/setrlimit.2.html setrlimit] for this. The options that we set by default are:<br />
<br />
* LimitNOFILE<br />
* LimitNPROC<br />
<br />
which you have already seen when we took a look at the [[#Service_definiton|service definition]] and [[#Drop-in_config|drop-in configs]]<br />
<br />
== Verify limits ==<br />
<br />
=== System V ===<br />
<br />
Get the proper pid of the java process started for the middleware by either checking '''ps faux'''<br />
root 4103 0.0 0.0 66528 1868 pts/0 S 22:05 0:00 su -s /bin/bash open-xchange -c /opt/open-xchange/sbin/open-xchange<br />
494 4109 9.5 19.8 2224644 381312 ? Ssl 22:05 1:12 \_ /usr/bin/java -Dsun.net.inetaddr.ttl=3600 -Dnetworkaddress.cache.ttl=3600<br />
<br />
or programmatically<br />
singlenode:~ # pid=$(pgrep -P $(</var/run/open-xchange.pid) java)<br />
<br />
and check the limits applied for that process<br />
singlenode:~ # cat /proc/$pid/limits<br />
Limit Soft Limit Hard Limit Units<br />
Max cpu time unlimited unlimited seconds<br />
Max file size unlimited unlimited bytes<br />
Max data size unlimited unlimited bytes<br />
Max stack size 8388608 unlimited bytes<br />
Max core file size 0 unlimited bytes<br />
Max resident set unlimited unlimited bytes<br />
Max processes 65536 65536 processes<br />
Max open files 65536 65536 files<br />
Max locked memory 65536 65536 bytes<br />
Max address space unlimited unlimited bytes<br />
Max file locks unlimited unlimited locks<br />
Max pending signals 24254 24254 signals<br />
Max msgqueue size 819200 819200 bytes<br />
Max nice priority 0 0<br />
Max realtime priority 0 0<br />
Max realtime timeout unlimited unlimited us<br />
<br />
=== Systemd ===<br />
singlenode:~ # systemctl show open-xchange | grep Limit<br />
StartLimitInterval=10000000<br />
StartLimitBurst=5<br />
StartLimitAction=none<br />
MemoryLimit=18446744073709551615<br />
LimitCPU=18446744073709551615<br />
LimitFSIZE=18446744073709551615<br />
LimitDATA=18446744073709551615<br />
LimitSTACK=18446744073709551615<br />
LimitCORE=18446744073709551615<br />
LimitRSS=18446744073709551615<br />
LimitNOFILE=65536<br />
LimitAS=18446744073709551615<br />
LimitNPROC=65536<br />
LimitMEMLOCK=65536<br />
LimitLOCKS=18446744073709551615<br />
LimitSIGPENDING=19827<br />
LimitMSGQUEUE=819200<br />
LimitNICE=0<br />
LimitRTPRIO=0<br />
LimitRTTIME=18446744073709551615<br />
<br />
== Evaluate JVM commandline options ==<br />
<br />
Before starting Java and the OSGi framework with all the Open-Xchange middleware bundles we have to evaluate several important JVM commandline options that influence the performance and stability of the Open-Xchange middleware stack. These commandline options are specified in '''/opt/open-xchange/etc/ox-scriptconf.sh'''<br />
<br />
=== Format < 7.8.4 ===<br />
For version up to 7.8.3 the JVM configuration options were all specified via a single property '''JAVA_XTRAOPTS''' which resulted in a rather long configuration string that was hard to read and maintain.<br />
<br />
JAVA_XTRAOPTS="-Dsun.net.inetaddr.ttl=3600 -Dnetworkaddress.cache.ttl=3600 -Dnetworkaddress.cache.negative.ttl=10 \<br />
-Dlogback.threadlocal.put.duplicate=false -server -Djava.awt.headless=true -Xmx512M -XX:MaxPermSize=256m -XX:+UseConcMarkSweepGC \<br />
-XX:+UseParNewGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:NewRatio=3 -XX:+UseTLAB \<br />
-Dosgi.compatibility.bootdelegation=false -XX:-OmitStackTraceInFastThrow"<br />
<br />
During the startup of the open-xchange service (step 11 of diagram Service startup using systemd and step 10 of diagram Service startup using upstart) this JAVA_XTRAOPTS string goes through a basic validity check and is then passed to the JVM.<br />
<br />
=== Format >= 7.8.4 ===<br />
Beginning from version 7.8.4 the options are split up and sorted into different categories like:<br />
<br />
JAVA_OPTS_GC="-XX:+UseConcMarkSweepGC -XX:+UseParNewGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:NewRatio=3 -XX:+DisableExplicitGC"<br />
JAVA_OPTS_LOG="-Dlogback.threadlocal.put.duplicate=false -XX:-OmitStackTraceInFastThrow"<br />
JAVA_OPTS_MEM="-XX:MaxHeapSize=512M -XX:MaxPermSize=256M -XX:+UseTLAB"<br />
JAVA_OPTS_NET="-Dsun.net.inetaddr.ttl=3600 -Dnetworkaddress.cache.ttl=3600 -Dnetworkaddress.cache.negative.ttl=10"<br />
JAVA_OPTS_OSGI="-Dosgi.compatibility.bootdelegation=false"<br />
JAVA_OPTS_SERVER="-server -Djava.awt.headless=true"<br />
<br />
JAVA_OPTS_OTHER=""<br />
<br />
#JAVA_OPTS_DEBUG="-XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/mnt/heapdump -Xloggc:/var/log/open-xchange/gc.log -verbose:gc -XX:+PrintGCDateStamps \<br />
-XX:+PrintHeapAtGC -XX:+PrintGCApplicationStoppedTime -XX:+PrintTenuringDistribution"<br />
<br />
which is much easier to read an maintain. Furthermore we added a useful '''JAVA_OPTS_DEBUG''' configuration parameter that comes preconfigured but unused/commented so that customers can quickly enable the most commonly used debugging options in case they have to provide detailed system information to Open-Xchange.<br />
<br />
During the startup of the open-xchange service (step 11 of diagram ''Service startup using systemd'' and step 10 of diagram ''Service startup using upstart''.) all uncommented JAVA_OPTS_* categories will then be evaluated and combined again into a single string of commandline options that can then be passed to the JVM.<br />
<br />
=== Upgrade procedure ===<br />
During the upgrade '''7.8.3-revX -> 7.8.4-revY''' existing JAVA_XTRAOPTS will automatically be split up and migrated to the new categories approach. A backup of the file (ox-scriptconf.sh.timestamp) will be created and admins are urged to check for proper migration before deleting the backup file like it's already done with *.dpkg-dist or *.rpmnew files on Debian or RHEL/SLES distributions.<br />
<br />
== Encoding detection during JVM startup ==<br />
<br />
=== Background Infos ===<br />
All of en_US.UTF-8, .utf8, .utf-8, .UTF_8 will result in a valid UTF-8 charmap<br />
<br />
==== JVM startup ====<br />
<br />
The default file.encoding detection is done at JVM startup based upon LC_CTYPE which makes sense as it's documented as:<br />
<br />
CODESET (LC_CTYPE)<br />
Return a string with the name of the character encoding used in the selected locale, such as "UTF-8", "ISO-8859-1",<br />
or "ANSI_X3.4-1968" (better known as US-ASCII). This is the same string that you get with "locale charmap".<br />
For a list of character encoding names, try "locale -m", cf. locale(1).<br />
<br />
Furthermore this influences the defaultCharset selection of the JVM.<br />
<br />
/**<br />
* Returns the default charset of this Java virtual machine.<br />
*<br />
* The default charset is determined during virtual-machine startup and<br />
* typically depends upon the locale and charset of the underlying<br />
* operating system.<br />
*<br />
* @return A charset object for the default charset<br />
*<br />
* @since 1.5<br />
*/<br />
public static Charset defaultCharset() {<br />
if (defaultCharset == null) {<br />
synchronized (Charset.class) {<br />
String csn = AccessController.doPrivileged(<br />
new GetPropertyAction("file.encoding"));<br />
Charset cs = lookup(csn);<br />
if (cs != null)<br />
defaultCharset = cs;<br />
else<br />
defaultCharset = forName("UTF-8");<br />
}<br />
}<br />
return defaultCharset;<br />
}<br />
<br />
Although we try to specify valid unicode encodings for every IO operations the Open-Xchange middleware does some parts of the JVM don't allow to specify the encoding to use and falls back to the detected default encoding. That's why admins have to make sure to provide a proper unicode environment for the middleware service during every start or use documented workarounds provided by Open-Xchange (e.g. )<br />
<br />
=== Query which file encoding was detected by the JVM during startup ===<br />
<br />
/opt/open-xchange/sbin/showruntimestats -r | sed -ne 's/.*\({key=file.encoding, value=[^}]*}\).*/\1/p'<br />
{key=file.encoding, value=UTF-8}<br />
<br />
=== Locale passing from client to server via SSH ===<br />
<br />
The following setting of sshd allows clients to specify which locale to use on the server and might lead to some confusion or unexpected behaviour when checking the system locale or even to the usage of broken locales during middleware startup if your client sends misconfigured or missing locale configuration to the server.<br />
<br />
grep AcceptEnv /etc/ssh/sshd_config <br />
AcceptEnv LANG LC_*<br />
<br />
For example when i connect to the machine my local german locale is sent to the server <br />
<br />
[vagrant@showcase-community-centos-7 ~]$ locale -v <br />
LANG=de_DE.UTF-8<br />
LC_CTYPE="de_DE.UTF-8"<br />
LC_NUMERIC=de_DE.UTF-8<br />
LC_TIME=de_DE.UTF-8<br />
LC_COLLATE="de_DE.UTF-8"<br />
LC_MONETARY=de_DE.UTF-8<br />
LC_MESSAGES="de_DE.UTF-8"<br />
LC_PAPER=de_DE.UTF-8<br />
LC_NAME=de_DE.UTF-8<br />
LC_ADDRESS=de_DE.UTF-8<br />
LC_TELEPHONE=de_DE.UTF-8<br />
LC_MEASUREMENT=de_DE.UTF-8<br />
LC_IDENTIFICATION=de_DE.UTF-8<br />
LC_ALL=<br />
<br />
But after specifying that i want a new login shell the default locale configured for that user/system will be used instead!<br />
<br />
[root@showcase-community-centos-7 vagrant]# su -s /bin/bash open-xchange -l<br />
-bash-4.2$ locale -v <br />
LANG=en_US.UTF-8<br />
LC_CTYPE="en_US.UTF-8"<br />
LC_NUMERIC="en_US.UTF-8"<br />
LC_TIME="en_US.UTF-8"<br />
LC_COLLATE="en_US.UTF-8"<br />
LC_MONETARY="en_US.UTF-8"<br />
LC_MESSAGES="en_US.UTF-8"<br />
LC_PAPER="en_US.UTF-8"<br />
LC_NAME="en_US.UTF-8"<br />
LC_ADDRESS="en_US.UTF-8"<br />
LC_TELEPHONE="en_US.UTF-8"<br />
LC_MEASUREMENT="en_US.UTF-8"<br />
LC_IDENTIFICATION="en_US.UTF-8"<br />
LC_ALL=<br />
<br />
Add details from https://bugs.open-xchange.com/show_bug.cgi?id=51074<br />
<br />
== Warnings during early start ==<br />
<br />
Early warnings before the start of the JVM and related logging libraries like logback and logstash forwarding are redirected to '''/var/log/open-xchange/open-xchange-console.log'''.<br />
<br />
=== Limits ===<br />
If the open-xchange startup script isn't able to set the proper limits on CentOS 6 due to e.g. hard limits being enforced via pam_limits you'll see the following warning in the console log during startup<br />
<br />
singlenode open-xchange # cat open-xchange-console.log<br />
/opt/open-xchange/sbin/open-xchange: line 115: ulimit: max user processes: cannot modify limit: Operation not permitted<br />
<br />
=== Locale ===<br />
If the open-xchange startup script isn't able to detect a unicode locale you'll see a warning like the following (the current charmap might differ)<br />
<br />
WARNING: Couldn't detect proper unicode charmap in locale category LC_CTYPE.<br />
WARNING: Current charmap: "ANSI_X3.4-1968" might cause encoding problems.<br />
<br />
== Summary: Open-Xchange middleware on specific distros ==<br />
The support for the mentioned mechanism of resource control differ depending on the distribution and the init system in use.<br />
<br />
=== Debian 7 ===<br />
;Init<br />
: System V style<br />
;OX Configurable Limits/Defaults<br />
: nofile, nproc<br />
<br />
The mentioned limits can be configured via '''/opt/open-xchange/etc/ox-scriptconf.sh'''. The limits are applied via ulimit in the service's init script. The open-xchange service is finally started via start-stop-daemon which doesn't doesn't consider '''/etc/security/limits.*'''<br />
<br />
=== RHEL 6 / CentOS 6 ===<br />
;Init<br />
: Upstart, System V compatible<br />
;OX Configurable Limits/Defaults<br />
: nofile, nproc<br />
<br />
The mentioned limits can be configured via '''/opt/open-xchange/etc/ox-scriptconf.sh'''. The limits are applied via ulimit in the service's init script. Furthermore as the open-xchange service is finally started via '''su ... open-xchange''' on this distro a user session is opened via su/pam and the default CentOS pam config reads the '''/etc/security/limits.*''' configuration by loading the pam stack like:<br />
<br />
: '''/etc/pam.d/su'''<br />
:: '''-> /etc/pam.d/system-auth'''<br />
::: '''-> pam_limits.so'''<br />
<br />
If NPROC isn't configured for the open-xchange-server it's restricted to '''1024''' globally by default to prevent accidental fork bombs, see '''/etc/security/limits.d/90-nproc.conf''' which can result in severe problems modern multithreaded applications.<br />
<br />
=== RHEL 7 / CentOS 7 / Debian 8 / SLE 12 ===<br />
;Init<br />
: Systemd<br />
;OX Configurable Limits/Defaults<br />
: nofile, nproc<br />
<br />
For systemd the default limits are configured directly in the service's unit file that is shipped by OX and located at '''/usr/lib/systemd/system/open-xchange.service'''. The drop-in config to override or extend the default unit file is located at '''/etc/systemd/system/open-xchange.service.d/limits.conf'''. [https://www.freedesktop.org/software/systemd/man/systemd.exec.html Systemd.exec] shows a whole lot of options that can be used by admins to adapt the default service to their specific needs.</div>Marenshttps://oxpedia.org/wiki/index.php?title=Sandbox:Middleware_Startup/sandbox&diff=23223Sandbox:Middleware Startup/sandbox2017-04-24T19:45:29Z<p>Marens: /* Apply Resource limits */</p>
<hr />
<div><br />
== Systemd ==<br />
<br />
==== Service definition ====<br />
The default service definition as shipped by Open-Xchange is rather concise and easy to read. You can check by looking at the service file that is installed to either /usr/lib/systemd or /lib/systemd depending on your distribution of choice.<br />
<br />
singlenode:~ # cat /usr/lib/systemd/system/open-xchange.service <br />
[Unit]<br />
After=remote-fs.target<br />
After=time-sync.target ypbind.service sendmail.service cyrus.service <br />
<br />
[Service]<br />
User=open-xchange<br />
PermissionsStartOnly=true<br />
TimeoutStartSec=0<br />
ExecStartPre=/opt/open-xchange/sbin/triggerupdatethemes -u<br />
ExecStart=/opt/open-xchange/sbin/open-xchange<br />
ExecStop=/opt/open-xchange/sbin/shutdown -w<br />
ExecReload=/opt/open-xchange/sbin/triggerreloadconfiguration -d<br />
KillMode=process<br />
LimitNOFILE=65536<br />
LimitNPROC=65536<br />
<br />
[Install]<br />
WantedBy=multi-user.target<br />
<br />
==== Drop-in configs ====<br />
Drop in configs allow administrators to easily adapt and override default service unit files. So if you want to change the default limits or add additional limits have a look at our default example drop-in config which is located at /etc/systemd/system/open-xchange.service.d/limits.conf<br />
<br />
singlenode:~ # cat /etc/systemd/system/open-xchange.service.d/limits.conf <br />
# Override and add options in this file<br />
# See systemd.exec(5) for other limits<br />
<br />
[Service]<br />
#LimitNPROC=65536<br />
#LimitNOFILE=65536<br />
<br />
The systemd.exec documentation shows a whole lot of options that can be used by admins to adapt the default service to their specific needs.<br />
<br />
<br clear=all><br />
[[File:mw_start_sysd.png|thumb|Service startup using systemd|left|1000px]]<br />
<br clear=all><br />
<br />
== System V / Upstart ==<br />
<br />
==== Service definition ====<br />
<br />
The default service definition as shipped by Open-Xchange can be found at /etc/init.d/open-xchange and will differ slightly depending on the distro in use.<br />
<br />
<br clear=all><br />
[[File:mw_start_sysv.png|thumb|Service startup using upstart|left|1000px]]<br />
<br clear=all><br />
== Apply Resource limits ==<br />
<br />
As you can see in the diagrams above one of the differences is that:<br />
<br />
* systemd reads all the infos wrt. resource limitations from the original service file and then applies possible further restrictions from the drop-in configs (steps 2 and 8 in the diagram ''Service startup using systemd'') and limits configured in /opt/open-xchange/etc/ox-scriptconf.sh aren't considered anylonger<br />
* the older System V style init has to source /opt/open-xchange/etc/ox-scriptconf.sh to be able to read the configs for the maximum number of processes (NPROC) and files(NRFILES) that the middleware shall be allowed to create/open (steps 3,4 and 15 in the diagram ''Service startup using upstart'').<br />
<br />
=== Background Infos ===<br />
<br />
Several ways exist to restrict resources on a linux system from a global level down to user/groups or even shells and the processes started by them.<br />
<br />
==== Sysctl ====<br />
Sysctl is used to modify kernel parameters at runtime. E.g. to set the maximum number of files<br />
<br />
$ sysctl -w fs.file-max=100000<br />
<br />
To permanently set them append to the main configuration file and reload the settings<br />
<br />
$ echo fs.file-max=100000 >> /etc/sysctl.conf<br />
$ sysctl -p<br />
<br />
More infos can be found via '''man sysctl'''<br />
<br />
==== Limits.conf ====<br />
<br />
Allows to restrict resources an a global, group or user level. E.g:<br />
<br />
$ cat /etc/security/limits.d/90-nproc.conf <br />
# Default limit for number of user's processes to prevent<br />
# accidental fork bombs.<br />
# See rhbz #432903 for reasoning.<br />
<br />
* soft nproc 1024<br />
<br />
From '''man limits.conf''':<br />
<blockquote><br />
''Also, please note that all limit settings are set per login. They are not global, nor are they permanent; existing only for the duration of the session.''<br />
</blockquote><br />
<br />
The limits per login are applied via the ''pam'' stack. See '''man pam''' and '''man pam_limits''' for more details. As those limits are bound to sessions they don't affect most daemons started by our supported init systems or init utils. Most state that they are ignored by design, see [https://bugs.launchpad.net/ubuntu/+source/upstart/+bug/938669/comments/4 upstart], [https://bugzilla.redhat.com/show_bug.cgi?id=754285#c1 systemd] and [https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=302079 start-stop-daemon] <br />
<br />
===== Ulimit =====<br />
From '''man bash'''<br />
<blockquote><br />
ulimit [-HSTabcdefilmnpqrstuvx [limit]]<br />
Provides control over the resources available to the shell and to processes started by it, on systems that allow such control.<br />
</blockquote><br />
This is what we use in our System V compatible init scripts to increase resources for the open-xchange process across multiple distros. Currently only ''the maximum number of processes'' and ''the maximum number of open file descriptors available to a single user'' are increased via ulimit. The values are specified in '''/opt/open-xchange/ox-scriptconf.sh'''<br />
<br />
==== Systemd ====<br />
<br />
===== Control Groups=====<br />
<br />
Control groups should only affect the OX middleware if you create/manage them yourself of if you are using a ''modern'' distribution that already uses systemd as init.<br />
<br />
Citing from the kernel cgroup [https://www.kernel.org/doc/Documentation/cgroup-v2.txt documentation]:<br />
<br />
<blockquote><br />
1-2. What is cgroup?<br />
<br />
cgroup is a mechanism to organize processes hierarchically and<br />
distribute system resources along the hierarchy in a controlled and<br />
configurable manner.<br />
<br />
cgroup is largely composed of two parts - the core and controllers.<br />
cgroup core is primarily responsible for hierarchically organizing<br />
processes. A cgroup controller is usually responsible for<br />
distributing a specific type of system resource along the hierarchy<br />
although there are utility controllers which serve purposes other than<br />
resource distribution.<br />
<br />
cgroups form a tree structure and every process in the system belongs<br />
to one and only one cgroup. All threads of a process belong to the<br />
same cgroup. On creation, all processes are put in the cgroup that<br />
the parent process belongs to at the time. A process can be migrated<br />
to another cgroup. Migration of a process doesn't affect already<br />
existing descendant processes.<br />
<br />
Following certain structural constraints, controllers may be enabled or<br />
disabled selectively on a cgroup. All controller behaviors are<br />
hierarchical - if a controller is enabled on a cgroup, it affects all<br />
processes which belong to the cgroups consisting the inclusive<br />
sub-hierarchy of the cgroup. When a controller is enabled on a nested<br />
cgroup, it always restricts the resource distribution further. The<br />
restrictions set closer to the root in the hierarchy can not be<br />
overridden from further away.<br />
</blockquote><br />
<br />
So processes are organized into a tree structure of control groups and controllers are responsible for the distribution of resources. So what kind of controllers exist?<br />
<br />
<blockquote><br />
5. Controllers<br />
<br />
5-1. CPU<br />
<br />
The "cpu" controllers regulates distribution of CPU cycles. This<br />
controller implements weight and absolute bandwidth limit models for<br />
normal scheduling policy and absolute bandwidth allocation model for<br />
realtime scheduling policy.<br />
<br />
5-2. Memory<br />
<br />
The "memory" controller regulates distribution of memory.<br />
...<br />
While not completely water-tight, all major memory usages by a given<br />
cgroup are tracked so that the total memory consumption can be<br />
accounted and controlled to a reasonable extent.<br />
<br />
5-3. IO<br />
<br />
The "io" controller regulates the distribution of IO resources. This<br />
controller implements both weight based and absolute bandwidth or IOPS<br />
limit distribution; however, weight based distribution is available<br />
only if cfq-iosched is in use and neither scheme is available for<br />
blk-mq devices.<br />
</blockquote><br />
<br />
The open-xchange service is simply put into the default ''system.slice'' without applying further limits.<br />
<br />
singlenode$ systemd-cgls<br />
├─1 /sbin/init<br />
├─system.slice<br />
│ ├─avahi-daemon.service<br />
│ │ ├─501 avahi-daemon: running [singlenode]<br />
│ │ └─514 avahi-daemon: chroot helper<br />
│ ├─console-kit-daemon.service<br />
│ │ └─16164 /usr/sbin/console-kit-daemon --no-daemon<br />
│ ├─dbus.service<br />
│ │ └─508 /usr/bin/dbus-daemon --system --address=systemd: --nofork --nopidfile --systemd-activation<br />
│ ├─munin-node.service<br />
│ │ └─4290 /usr/bin/perl -wT /usr/sbin/munin-node<br />
│ ├─open-xchange.service<br />
│ │ └─6037 /usr/bin/java -Dsun.net.inetaddr.ttl=3600 -Dnetworkaddress.cache.ttl=3600 -Dnetworkaddress.cache.negative.ttl=10 ...<br />
<br />
To check all the details use<br />
<br />
singlenode:~ # systemctl show system.slice<br />
<br />
==== Limits besides control groups ====<br />
Besides control groups systemd allows you to apply other limits to the execution environment of your service. Here we can apply the limits that would normally be applied via ''limits.conf'' or ''ulimit''. Systemd uses [http://man7.org/linux/man-pages/man2/setrlimit.2.html setrlimit] for this. The options that we set by default are:<br />
<br />
* LimitNOFILE<br />
* LimitNPROC<br />
<br />
which you have already seen when we took a look at the [[#Service_definiton|service definition]] and [[#Drop-in_config|drop-in configs]]<br />
<br />
== Verify limits ==<br />
<br />
=== System V ===<br />
<br />
Get the proper pid of the java process started for the middleware by either checking '''ps faux'''<br />
root 4103 0.0 0.0 66528 1868 pts/0 S 22:05 0:00 su -s /bin/bash open-xchange -c /opt/open-xchange/sbin/open-xchange<br />
494 4109 9.5 19.8 2224644 381312 ? Ssl 22:05 1:12 \_ /usr/bin/java -Dsun.net.inetaddr.ttl=3600 -Dnetworkaddress.cache.ttl=3600<br />
<br />
or programmatically<br />
singlenode:~ # pid=$(pgrep -P $(</var/run/open-xchange.pid) java)<br />
<br />
and check the limits applied for that process<br />
singlenode:~ # cat /proc/$pid/limits<br />
Limit Soft Limit Hard Limit Units<br />
Max cpu time unlimited unlimited seconds<br />
Max file size unlimited unlimited bytes<br />
Max data size unlimited unlimited bytes<br />
Max stack size 8388608 unlimited bytes<br />
Max core file size 0 unlimited bytes<br />
Max resident set unlimited unlimited bytes<br />
Max processes 65536 65536 processes<br />
Max open files 65536 65536 files<br />
Max locked memory 65536 65536 bytes<br />
Max address space unlimited unlimited bytes<br />
Max file locks unlimited unlimited locks<br />
Max pending signals 24254 24254 signals<br />
Max msgqueue size 819200 819200 bytes<br />
Max nice priority 0 0<br />
Max realtime priority 0 0<br />
Max realtime timeout unlimited unlimited us<br />
<br />
=== Systemd ===<br />
singlenode:~ # systemctl show open-xchange | grep Limit<br />
StartLimitInterval=10000000<br />
StartLimitBurst=5<br />
StartLimitAction=none<br />
MemoryLimit=18446744073709551615<br />
LimitCPU=18446744073709551615<br />
LimitFSIZE=18446744073709551615<br />
LimitDATA=18446744073709551615<br />
LimitSTACK=18446744073709551615<br />
LimitCORE=18446744073709551615<br />
LimitRSS=18446744073709551615<br />
LimitNOFILE=65536<br />
LimitAS=18446744073709551615<br />
LimitNPROC=65536<br />
LimitMEMLOCK=65536<br />
LimitLOCKS=18446744073709551615<br />
LimitSIGPENDING=19827<br />
LimitMSGQUEUE=819200<br />
LimitNICE=0<br />
LimitRTPRIO=0<br />
LimitRTTIME=18446744073709551615<br />
<br />
== Evaluate JVM commandline options ==<br />
<br />
Before starting Java and the OSGi framework with all the Open-Xchange middleware bundles we have to evaluate several important JVM commandline options that influence the performance and stability of the Open-Xchange middleware stack. These commandline options are specified in '''/opt/open-xchange/etc/ox-scriptconf.sh'''<br />
<br />
=== Format < 7.8.4 ===<br />
For version up to 7.8.3 the JVM configuration options were all specified via a single property '''JAVA_XTRAOPTS''' which resulted in a rather long configuration string that was hard to read and maintain.<br />
<br />
JAVA_XTRAOPTS="-Dsun.net.inetaddr.ttl=3600 -Dnetworkaddress.cache.ttl=3600 -Dnetworkaddress.cache.negative.ttl=10 \<br />
-Dlogback.threadlocal.put.duplicate=false -server -Djava.awt.headless=true -Xmx512M -XX:MaxPermSize=256m -XX:+UseConcMarkSweepGC \<br />
-XX:+UseParNewGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:NewRatio=3 -XX:+UseTLAB \<br />
-Dosgi.compatibility.bootdelegation=false -XX:-OmitStackTraceInFastThrow"<br />
<br />
During the startup of the open-xchange service (step 11 of diagram Service startup using systemd and step 10 of diagram Service startup using upstart.) this JAVA_XTRAOPTS string goes through a basic validity check and is then passed to the JVM. <br />
<br />
=== Format >= 7.8.4 ===<br />
Beginning from version 7.8.4 the options are split up and sorted into different categories like:<br />
<br />
JAVA_OPTS_GC="-XX:+UseConcMarkSweepGC -XX:+UseParNewGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:NewRatio=3 -XX:+DisableExplicitGC"<br />
JAVA_OPTS_LOG="-Dlogback.threadlocal.put.duplicate=false -XX:-OmitStackTraceInFastThrow"<br />
JAVA_OPTS_MEM="-XX:MaxHeapSize=512M -XX:MaxPermSize=256M -XX:+UseTLAB"<br />
JAVA_OPTS_NET="-Dsun.net.inetaddr.ttl=3600 -Dnetworkaddress.cache.ttl=3600 -Dnetworkaddress.cache.negative.ttl=10"<br />
JAVA_OPTS_OSGI="-Dosgi.compatibility.bootdelegation=false"<br />
JAVA_OPTS_SERVER="-server -Djava.awt.headless=true"<br />
<br />
JAVA_OPTS_OTHER=""<br />
<br />
#JAVA_OPTS_DEBUG="-XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/mnt/heapdump -Xloggc:/var/log/open-xchange/gc.log -verbose:gc -XX:+PrintGCDateStamps \<br />
-XX:+PrintHeapAtGC -XX:+PrintGCApplicationStoppedTime -XX:+PrintTenuringDistribution"<br />
<br />
which is much easier to read an maintain. Furthermore we added a useful '''JAVA_OPTS_DEBUG''' configuration parameter that comes preconfigured but unused/commented so that customers can quickly enable the most commonly used debugging options in case they have to provide detailed system information to Open-Xchange.<br />
<br />
During the startup of the open-xchange service (step 11 of diagram ''Service startup using systemd'' and step 10 of diagram ''Service startup using upstart''.) all uncommented JAVA_OPTS_* categories will then be evaluated and combined again into a single string of commandline options that can then be passed to the JVM.<br />
<br />
=== Upgrade procedure ===<br />
During the upgrade '''7.8.3-revX -> 7.8.4-revY''' existing JAVA_XTRAOPTS will automatically be split up and migrated to the new categories approach. A backup of the file (ox-scriptconf.sh.timestamp) will be created and admins are urged to check for proper migration before deleting the backup file like it's already done with *.dpkg-dist or *.rpmnew files on Debian or RHEL/SLES distributions.<br />
<br />
== Encoding detection during JVM startup ==<br />
<br />
=== Background Infos ===<br />
All of en_US.UTF-8, .utf8, .utf-8, .UTF_8 will result in a valid UTF-8 charmap<br />
<br />
==== JVM startup ====<br />
<br />
The default file.encoding detection is done at JVM startup based upon LC_CTYPE which makes sense as it's documented as:<br />
<br />
CODESET (LC_CTYPE)<br />
Return a string with the name of the character encoding used in the selected locale, such as "UTF-8", "ISO-8859-1",<br />
or "ANSI_X3.4-1968" (better known as US-ASCII). This is the same string that you get with "locale charmap".<br />
For a list of character encoding names, try "locale -m", cf. locale(1).<br />
<br />
Furthermore this influences the defaultCharset selection of the JVM.<br />
<br />
/**<br />
* Returns the default charset of this Java virtual machine.<br />
*<br />
* The default charset is determined during virtual-machine startup and<br />
* typically depends upon the locale and charset of the underlying<br />
* operating system.<br />
*<br />
* @return A charset object for the default charset<br />
*<br />
* @since 1.5<br />
*/<br />
public static Charset defaultCharset() {<br />
if (defaultCharset == null) {<br />
synchronized (Charset.class) {<br />
String csn = AccessController.doPrivileged(<br />
new GetPropertyAction("file.encoding"));<br />
Charset cs = lookup(csn);<br />
if (cs != null)<br />
defaultCharset = cs;<br />
else<br />
defaultCharset = forName("UTF-8");<br />
}<br />
}<br />
return defaultCharset;<br />
}<br />
<br />
Although we try to specify valid unicode encodings for every IO operations the Open-Xchange middleware does some parts of the JVM don't allow to specify the encoding to use and falls back to the detected default encoding. That's why admins have to make sure to provide a proper unicode environment for the middleware service during every start or use documented workarounds provided by Open-Xchange (e.g. )<br />
<br />
=== Query which file encoding was detected by the JVM during startup ===<br />
<br />
/opt/open-xchange/sbin/showruntimestats -r | sed -ne 's/.*\({key=file.encoding, value=[^}]*}\).*/\1/p'<br />
{key=file.encoding, value=UTF-8}<br />
<br />
=== Locale passing from client to server via SSH ===<br />
<br />
The following setting of sshd allows clients to specify which locale to use on the server and might lead to some confusion or unexpected behaviour when checking the system locale or even to the usage of broken locales during middleware startup if your client sends misconfigured or missing locale configuration to the server.<br />
<br />
grep AcceptEnv /etc/ssh/sshd_config <br />
AcceptEnv LANG LC_*<br />
<br />
For example when i connect to the machine my local german locale is sent to the server <br />
<br />
[vagrant@showcase-community-centos-7 ~]$ locale -v <br />
LANG=de_DE.UTF-8<br />
LC_CTYPE="de_DE.UTF-8"<br />
LC_NUMERIC=de_DE.UTF-8<br />
LC_TIME=de_DE.UTF-8<br />
LC_COLLATE="de_DE.UTF-8"<br />
LC_MONETARY=de_DE.UTF-8<br />
LC_MESSAGES="de_DE.UTF-8"<br />
LC_PAPER=de_DE.UTF-8<br />
LC_NAME=de_DE.UTF-8<br />
LC_ADDRESS=de_DE.UTF-8<br />
LC_TELEPHONE=de_DE.UTF-8<br />
LC_MEASUREMENT=de_DE.UTF-8<br />
LC_IDENTIFICATION=de_DE.UTF-8<br />
LC_ALL=<br />
<br />
But after specifying that i want a new login shell the default locale configured for that user/system will be used instead!<br />
<br />
[root@showcase-community-centos-7 vagrant]# su -s /bin/bash open-xchange -l<br />
-bash-4.2$ locale -v <br />
LANG=en_US.UTF-8<br />
LC_CTYPE="en_US.UTF-8"<br />
LC_NUMERIC="en_US.UTF-8"<br />
LC_TIME="en_US.UTF-8"<br />
LC_COLLATE="en_US.UTF-8"<br />
LC_MONETARY="en_US.UTF-8"<br />
LC_MESSAGES="en_US.UTF-8"<br />
LC_PAPER="en_US.UTF-8"<br />
LC_NAME="en_US.UTF-8"<br />
LC_ADDRESS="en_US.UTF-8"<br />
LC_TELEPHONE="en_US.UTF-8"<br />
LC_MEASUREMENT="en_US.UTF-8"<br />
LC_IDENTIFICATION="en_US.UTF-8"<br />
LC_ALL=<br />
<br />
Add details from https://bugs.open-xchange.com/show_bug.cgi?id=51074<br />
<br />
== Warnings during early start ==<br />
<br />
Early warnings before the start of the JVM and related logging libraries like logback and logstash forwarding are redirected to '''/var/log/open-xchange/open-xchange-console.log'''.<br />
<br />
=== Limits ===<br />
If the open-xchange startup script isn't able to set the proper limits on CentOS 6 due to e.g. hard limits being enforced via pam_limits you'll see the following warning in the console log during startup<br />
<br />
singlenode open-xchange # cat open-xchange-console.log<br />
/opt/open-xchange/sbin/open-xchange: line 115: ulimit: max user processes: cannot modify limit: Operation not permitted<br />
<br />
=== Locale ===<br />
If the open-xchange startup script isn't able to detect a unicode locale you'll see a warning like the following (the current charmap might differ)<br />
<br />
WARNING: Couldn't detect proper unicode charmap in locale category LC_CTYPE.<br />
WARNING: Current charmap: "ANSI_X3.4-1968" might cause encoding problems.<br />
<br />
== Summary: Open-Xchange middleware on specific distros ==<br />
The support for the mentioned mechanism of resource control differ depending on the distribution and the init system in use.<br />
<br />
=== Debian 7 ===<br />
;Init<br />
: System V style<br />
;OX Configurable Limits/Defaults<br />
: nofile, nproc<br />
<br />
The mentioned limits can be configured via '''/opt/open-xchange/etc/ox-scriptconf.sh'''. The limits are applied via ulimit in the service's init script. The open-xchange service is finally started via start-stop-daemon which doesn't doesn't consider '''/etc/security/limits.*'''<br />
<br />
=== RHEL 6 / CentOS 6 ===<br />
;Init<br />
: Upstart, System V compatible<br />
;OX Configurable Limits/Defaults<br />
: nofile, nproc<br />
<br />
The mentioned limits can be configured via '''/opt/open-xchange/etc/ox-scriptconf.sh'''. The limits are applied via ulimit in the service's init script. Furthermore as the open-xchange service is finally started via '''su ... open-xchange''' on this distro a user session is opened via su/pam and the default CentOS pam config reads the '''/etc/security/limits.*''' configuration by loading the pam stack like:<br />
<br />
: '''/etc/pam.d/su'''<br />
:: '''-> /etc/pam.d/system-auth'''<br />
::: '''-> pam_limits.so'''<br />
<br />
If NPROC isn't configured for the open-xchange-server it's restricted to '''1024''' globally by default to prevent accidental fork bombs, see '''/etc/security/limits.d/90-nproc.conf''' which can result in severe problems modern multithreaded applications.<br />
<br />
=== RHEL 7 / CentOS 7 / Debian 8 / SLE 12 ===<br />
;Init<br />
: Systemd<br />
;OX Configurable Limits/Defaults<br />
: nofile, nproc<br />
<br />
For systemd the default limits are configured directly in the service's unit file that is shipped by OX and located at '''/usr/lib/systemd/system/open-xchange.service'''. The drop-in config to override or extend the default unit file is located at '''/etc/systemd/system/open-xchange.service.d/limits.conf'''. [https://www.freedesktop.org/software/systemd/man/systemd.exec.html Systemd.exec] shows a whole lot of options that can be used by admins to adapt the default service to their specific needs.</div>Marenshttps://oxpedia.org/wiki/index.php?title=Sandbox:Middleware_Startup/sandbox&diff=23222Sandbox:Middleware Startup/sandbox2017-04-24T19:44:39Z<p>Marens: /* Apply Resource limits */</p>
<hr />
<div><br />
== Systemd ==<br />
<br />
==== Service definition ====<br />
The default service definition as shipped by Open-Xchange is rather concise and easy to read. You can check by looking at the service file that is installed to either /usr/lib/systemd or /lib/systemd depending on your distribution of choice.<br />
<br />
singlenode:~ # cat /usr/lib/systemd/system/open-xchange.service <br />
[Unit]<br />
After=remote-fs.target<br />
After=time-sync.target ypbind.service sendmail.service cyrus.service <br />
<br />
[Service]<br />
User=open-xchange<br />
PermissionsStartOnly=true<br />
TimeoutStartSec=0<br />
ExecStartPre=/opt/open-xchange/sbin/triggerupdatethemes -u<br />
ExecStart=/opt/open-xchange/sbin/open-xchange<br />
ExecStop=/opt/open-xchange/sbin/shutdown -w<br />
ExecReload=/opt/open-xchange/sbin/triggerreloadconfiguration -d<br />
KillMode=process<br />
LimitNOFILE=65536<br />
LimitNPROC=65536<br />
<br />
[Install]<br />
WantedBy=multi-user.target<br />
<br />
==== Drop-in configs ====<br />
Drop in configs allow administrators to easily adapt and override default service unit files. So if you want to change the default limits or add additional limits have a look at our default example drop-in config which is located at /etc/systemd/system/open-xchange.service.d/limits.conf<br />
<br />
singlenode:~ # cat /etc/systemd/system/open-xchange.service.d/limits.conf <br />
# Override and add options in this file<br />
# See systemd.exec(5) for other limits<br />
<br />
[Service]<br />
#LimitNPROC=65536<br />
#LimitNOFILE=65536<br />
<br />
The systemd.exec documentation shows a whole lot of options that can be used by admins to adapt the default service to their specific needs.<br />
<br />
<br clear=all><br />
[[File:mw_start_sysd.png|thumb|Service startup using systemd|left|1000px]]<br />
<br clear=all><br />
<br />
== System V / Upstart ==<br />
<br />
==== Service definition ====<br />
<br />
The default service definition as shipped by Open-Xchange can be found at /etc/init.d/open-xchange and will differ slightly depending on the distro in use.<br />
<br />
<br clear=all><br />
[[File:mw_start_sysv.png|thumb|Service startup using upstart|left|1000px]]<br />
<br clear=all><br />
== Apply Resource limits ==<br />
<br />
As you can see in the diagrams above one of the differences is that:<br />
<br />
* systemd reads all the infos wrt. resource limitations from the original service file and then applies possible further restrictions from the drop-in configs (steps 2 and 8 in the diagram ''Service startup using systemd'') and limits configured in /opt/open-xchange/etc/ox-scriptconf.sh aren't considered anylonger when using systemd<br />
* the older System V style init has to source /opt/open-xchange/etc/ox-scriptconf.sh to be able to read the configs for the maximum number of processes (NPROC) and files(NRFILES) that the middleware shall be allowed to create/open (steps 3,4 and 15 in the diagram ''Service startup using upstart'').<br />
<br />
=== Background Infos ===<br />
<br />
Several ways exist to restrict resources on a linux system from a global level down to user/groups or even shells and the processes started by them.<br />
<br />
==== Sysctl ====<br />
Sysctl is used to modify kernel parameters at runtime. E.g. to set the maximum number of files<br />
<br />
$ sysctl -w fs.file-max=100000<br />
<br />
To permanently set them append to the main configuration file and reload the settings<br />
<br />
$ echo fs.file-max=100000 >> /etc/sysctl.conf<br />
$ sysctl -p<br />
<br />
More infos can be found via '''man sysctl'''<br />
<br />
==== Limits.conf ====<br />
<br />
Allows to restrict resources an a global, group or user level. E.g:<br />
<br />
$ cat /etc/security/limits.d/90-nproc.conf <br />
# Default limit for number of user's processes to prevent<br />
# accidental fork bombs.<br />
# See rhbz #432903 for reasoning.<br />
<br />
* soft nproc 1024<br />
<br />
From '''man limits.conf''':<br />
<blockquote><br />
''Also, please note that all limit settings are set per login. They are not global, nor are they permanent; existing only for the duration of the session.''<br />
</blockquote><br />
<br />
The limits per login are applied via the ''pam'' stack. See '''man pam''' and '''man pam_limits''' for more details. As those limits are bound to sessions they don't affect most daemons started by our supported init systems or init utils. Most state that they are ignored by design, see [https://bugs.launchpad.net/ubuntu/+source/upstart/+bug/938669/comments/4 upstart], [https://bugzilla.redhat.com/show_bug.cgi?id=754285#c1 systemd] and [https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=302079 start-stop-daemon] <br />
<br />
===== Ulimit =====<br />
From '''man bash'''<br />
<blockquote><br />
ulimit [-HSTabcdefilmnpqrstuvx [limit]]<br />
Provides control over the resources available to the shell and to processes started by it, on systems that allow such control.<br />
</blockquote><br />
This is what we use in our System V compatible init scripts to increase resources for the open-xchange process across multiple distros. Currently only ''the maximum number of processes'' and ''the maximum number of open file descriptors available to a single user'' are increased via ulimit. The values are specified in '''/opt/open-xchange/ox-scriptconf.sh'''<br />
<br />
==== Systemd ====<br />
<br />
===== Control Groups=====<br />
<br />
Control groups should only affect the OX middleware if you create/manage them yourself of if you are using a ''modern'' distribution that already uses systemd as init.<br />
<br />
Citing from the kernel cgroup [https://www.kernel.org/doc/Documentation/cgroup-v2.txt documentation]:<br />
<br />
<blockquote><br />
1-2. What is cgroup?<br />
<br />
cgroup is a mechanism to organize processes hierarchically and<br />
distribute system resources along the hierarchy in a controlled and<br />
configurable manner.<br />
<br />
cgroup is largely composed of two parts - the core and controllers.<br />
cgroup core is primarily responsible for hierarchically organizing<br />
processes. A cgroup controller is usually responsible for<br />
distributing a specific type of system resource along the hierarchy<br />
although there are utility controllers which serve purposes other than<br />
resource distribution.<br />
<br />
cgroups form a tree structure and every process in the system belongs<br />
to one and only one cgroup. All threads of a process belong to the<br />
same cgroup. On creation, all processes are put in the cgroup that<br />
the parent process belongs to at the time. A process can be migrated<br />
to another cgroup. Migration of a process doesn't affect already<br />
existing descendant processes.<br />
<br />
Following certain structural constraints, controllers may be enabled or<br />
disabled selectively on a cgroup. All controller behaviors are<br />
hierarchical - if a controller is enabled on a cgroup, it affects all<br />
processes which belong to the cgroups consisting the inclusive<br />
sub-hierarchy of the cgroup. When a controller is enabled on a nested<br />
cgroup, it always restricts the resource distribution further. The<br />
restrictions set closer to the root in the hierarchy can not be<br />
overridden from further away.<br />
</blockquote><br />
<br />
So processes are organized into a tree structure of control groups and controllers are responsible for the distribution of resources. So what kind of controllers exist?<br />
<br />
<blockquote><br />
5. Controllers<br />
<br />
5-1. CPU<br />
<br />
The "cpu" controllers regulates distribution of CPU cycles. This<br />
controller implements weight and absolute bandwidth limit models for<br />
normal scheduling policy and absolute bandwidth allocation model for<br />
realtime scheduling policy.<br />
<br />
5-2. Memory<br />
<br />
The "memory" controller regulates distribution of memory.<br />
...<br />
While not completely water-tight, all major memory usages by a given<br />
cgroup are tracked so that the total memory consumption can be<br />
accounted and controlled to a reasonable extent.<br />
<br />
5-3. IO<br />
<br />
The "io" controller regulates the distribution of IO resources. This<br />
controller implements both weight based and absolute bandwidth or IOPS<br />
limit distribution; however, weight based distribution is available<br />
only if cfq-iosched is in use and neither scheme is available for<br />
blk-mq devices.<br />
</blockquote><br />
<br />
The open-xchange service is simply put into the default ''system.slice'' without applying further limits.<br />
<br />
singlenode$ systemd-cgls<br />
├─1 /sbin/init<br />
├─system.slice<br />
│ ├─avahi-daemon.service<br />
│ │ ├─501 avahi-daemon: running [singlenode]<br />
│ │ └─514 avahi-daemon: chroot helper<br />
│ ├─console-kit-daemon.service<br />
│ │ └─16164 /usr/sbin/console-kit-daemon --no-daemon<br />
│ ├─dbus.service<br />
│ │ └─508 /usr/bin/dbus-daemon --system --address=systemd: --nofork --nopidfile --systemd-activation<br />
│ ├─munin-node.service<br />
│ │ └─4290 /usr/bin/perl -wT /usr/sbin/munin-node<br />
│ ├─open-xchange.service<br />
│ │ └─6037 /usr/bin/java -Dsun.net.inetaddr.ttl=3600 -Dnetworkaddress.cache.ttl=3600 -Dnetworkaddress.cache.negative.ttl=10 ...<br />
<br />
To check all the details use<br />
<br />
singlenode:~ # systemctl show system.slice<br />
<br />
==== Limits besides control groups ====<br />
Besides control groups systemd allows you to apply other limits to the execution environment of your service. Here we can apply the limits that would normally be applied via ''limits.conf'' or ''ulimit''. Systemd uses [http://man7.org/linux/man-pages/man2/setrlimit.2.html setrlimit] for this. The options that we set by default are:<br />
<br />
* LimitNOFILE<br />
* LimitNPROC<br />
<br />
which you have already seen when we took a look at the [[#Service_definiton|service definition]] and [[#Drop-in_config|drop-in configs]]<br />
<br />
== Verify limits ==<br />
<br />
=== System V ===<br />
<br />
Get the proper pid of the java process started for the middleware by either checking '''ps faux'''<br />
root 4103 0.0 0.0 66528 1868 pts/0 S 22:05 0:00 su -s /bin/bash open-xchange -c /opt/open-xchange/sbin/open-xchange<br />
494 4109 9.5 19.8 2224644 381312 ? Ssl 22:05 1:12 \_ /usr/bin/java -Dsun.net.inetaddr.ttl=3600 -Dnetworkaddress.cache.ttl=3600<br />
<br />
or programmatically<br />
singlenode:~ # pid=$(pgrep -P $(</var/run/open-xchange.pid) java)<br />
<br />
and check the limits applied for that process<br />
singlenode:~ # cat /proc/$pid/limits<br />
Limit Soft Limit Hard Limit Units<br />
Max cpu time unlimited unlimited seconds<br />
Max file size unlimited unlimited bytes<br />
Max data size unlimited unlimited bytes<br />
Max stack size 8388608 unlimited bytes<br />
Max core file size 0 unlimited bytes<br />
Max resident set unlimited unlimited bytes<br />
Max processes 65536 65536 processes<br />
Max open files 65536 65536 files<br />
Max locked memory 65536 65536 bytes<br />
Max address space unlimited unlimited bytes<br />
Max file locks unlimited unlimited locks<br />
Max pending signals 24254 24254 signals<br />
Max msgqueue size 819200 819200 bytes<br />
Max nice priority 0 0<br />
Max realtime priority 0 0<br />
Max realtime timeout unlimited unlimited us<br />
<br />
=== Systemd ===<br />
singlenode:~ # systemctl show open-xchange | grep Limit<br />
StartLimitInterval=10000000<br />
StartLimitBurst=5<br />
StartLimitAction=none<br />
MemoryLimit=18446744073709551615<br />
LimitCPU=18446744073709551615<br />
LimitFSIZE=18446744073709551615<br />
LimitDATA=18446744073709551615<br />
LimitSTACK=18446744073709551615<br />
LimitCORE=18446744073709551615<br />
LimitRSS=18446744073709551615<br />
LimitNOFILE=65536<br />
LimitAS=18446744073709551615<br />
LimitNPROC=65536<br />
LimitMEMLOCK=65536<br />
LimitLOCKS=18446744073709551615<br />
LimitSIGPENDING=19827<br />
LimitMSGQUEUE=819200<br />
LimitNICE=0<br />
LimitRTPRIO=0<br />
LimitRTTIME=18446744073709551615<br />
<br />
== Evaluate JVM commandline options ==<br />
<br />
Before starting Java and the OSGi framework with all the Open-Xchange middleware bundles we have to evaluate several important JVM commandline options that influence the performance and stability of the Open-Xchange middleware stack. These commandline options are specified in '''/opt/open-xchange/etc/ox-scriptconf.sh'''<br />
<br />
=== Format < 7.8.4 ===<br />
For version up to 7.8.3 the JVM configuration options were all specified via a single property '''JAVA_XTRAOPTS''' which resulted in a rather long configuration string that was hard to read and maintain.<br />
<br />
JAVA_XTRAOPTS="-Dsun.net.inetaddr.ttl=3600 -Dnetworkaddress.cache.ttl=3600 -Dnetworkaddress.cache.negative.ttl=10 \<br />
-Dlogback.threadlocal.put.duplicate=false -server -Djava.awt.headless=true -Xmx512M -XX:MaxPermSize=256m -XX:+UseConcMarkSweepGC \<br />
-XX:+UseParNewGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:NewRatio=3 -XX:+UseTLAB \<br />
-Dosgi.compatibility.bootdelegation=false -XX:-OmitStackTraceInFastThrow"<br />
<br />
During the startup of the open-xchange service (step 11 of diagram Service startup using systemd and step 10 of diagram Service startup using upstart.) this JAVA_XTRAOPTS string goes through a basic validity check and is then passed to the JVM. <br />
<br />
=== Format >= 7.8.4 ===<br />
Beginning from version 7.8.4 the options are split up and sorted into different categories like:<br />
<br />
JAVA_OPTS_GC="-XX:+UseConcMarkSweepGC -XX:+UseParNewGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:NewRatio=3 -XX:+DisableExplicitGC"<br />
JAVA_OPTS_LOG="-Dlogback.threadlocal.put.duplicate=false -XX:-OmitStackTraceInFastThrow"<br />
JAVA_OPTS_MEM="-XX:MaxHeapSize=512M -XX:MaxPermSize=256M -XX:+UseTLAB"<br />
JAVA_OPTS_NET="-Dsun.net.inetaddr.ttl=3600 -Dnetworkaddress.cache.ttl=3600 -Dnetworkaddress.cache.negative.ttl=10"<br />
JAVA_OPTS_OSGI="-Dosgi.compatibility.bootdelegation=false"<br />
JAVA_OPTS_SERVER="-server -Djava.awt.headless=true"<br />
<br />
JAVA_OPTS_OTHER=""<br />
<br />
#JAVA_OPTS_DEBUG="-XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/mnt/heapdump -Xloggc:/var/log/open-xchange/gc.log -verbose:gc -XX:+PrintGCDateStamps \<br />
-XX:+PrintHeapAtGC -XX:+PrintGCApplicationStoppedTime -XX:+PrintTenuringDistribution"<br />
<br />
which is much easier to read an maintain. Furthermore we added a useful '''JAVA_OPTS_DEBUG''' configuration parameter that comes preconfigured but unused/commented so that customers can quickly enable the most commonly used debugging options in case they have to provide detailed system information to Open-Xchange.<br />
<br />
During the startup of the open-xchange service (step 11 of diagram ''Service startup using systemd'' and step 10 of diagram ''Service startup using upstart''.) all uncommented JAVA_OPTS_* categories will then be evaluated and combined again into a single string of commandline options that can then be passed to the JVM.<br />
<br />
=== Upgrade procedure ===<br />
During the upgrade '''7.8.3-revX -> 7.8.4-revY''' existing JAVA_XTRAOPTS will automatically be split up and migrated to the new categories approach. A backup of the file (ox-scriptconf.sh.timestamp) will be created and admins are urged to check for proper migration before deleting the backup file like it's already done with *.dpkg-dist or *.rpmnew files on Debian or RHEL/SLES distributions.<br />
<br />
== Encoding detection during JVM startup ==<br />
<br />
=== Background Infos ===<br />
All of en_US.UTF-8, .utf8, .utf-8, .UTF_8 will result in a valid UTF-8 charmap<br />
<br />
==== JVM startup ====<br />
<br />
The default file.encoding detection is done at JVM startup based upon LC_CTYPE which makes sense as it's documented as:<br />
<br />
CODESET (LC_CTYPE)<br />
Return a string with the name of the character encoding used in the selected locale, such as "UTF-8", "ISO-8859-1",<br />
or "ANSI_X3.4-1968" (better known as US-ASCII). This is the same string that you get with "locale charmap".<br />
For a list of character encoding names, try "locale -m", cf. locale(1).<br />
<br />
Furthermore this influences the defaultCharset selection of the JVM.<br />
<br />
/**<br />
* Returns the default charset of this Java virtual machine.<br />
*<br />
* The default charset is determined during virtual-machine startup and<br />
* typically depends upon the locale and charset of the underlying<br />
* operating system.<br />
*<br />
* @return A charset object for the default charset<br />
*<br />
* @since 1.5<br />
*/<br />
public static Charset defaultCharset() {<br />
if (defaultCharset == null) {<br />
synchronized (Charset.class) {<br />
String csn = AccessController.doPrivileged(<br />
new GetPropertyAction("file.encoding"));<br />
Charset cs = lookup(csn);<br />
if (cs != null)<br />
defaultCharset = cs;<br />
else<br />
defaultCharset = forName("UTF-8");<br />
}<br />
}<br />
return defaultCharset;<br />
}<br />
<br />
Although we try to specify valid unicode encodings for every IO operations the Open-Xchange middleware does some parts of the JVM don't allow to specify the encoding to use and falls back to the detected default encoding. That's why admins have to make sure to provide a proper unicode environment for the middleware service during every start or use documented workarounds provided by Open-Xchange (e.g. )<br />
<br />
=== Query which file encoding was detected by the JVM during startup ===<br />
<br />
/opt/open-xchange/sbin/showruntimestats -r | sed -ne 's/.*\({key=file.encoding, value=[^}]*}\).*/\1/p'<br />
{key=file.encoding, value=UTF-8}<br />
<br />
=== Locale passing from client to server via SSH ===<br />
<br />
The following setting of sshd allows clients to specify which locale to use on the server and might lead to some confusion or unexpected behaviour when checking the system locale or even to the usage of broken locales during middleware startup if your client sends misconfigured or missing locale configuration to the server.<br />
<br />
grep AcceptEnv /etc/ssh/sshd_config <br />
AcceptEnv LANG LC_*<br />
<br />
For example when i connect to the machine my local german locale is sent to the server <br />
<br />
[vagrant@showcase-community-centos-7 ~]$ locale -v <br />
LANG=de_DE.UTF-8<br />
LC_CTYPE="de_DE.UTF-8"<br />
LC_NUMERIC=de_DE.UTF-8<br />
LC_TIME=de_DE.UTF-8<br />
LC_COLLATE="de_DE.UTF-8"<br />
LC_MONETARY=de_DE.UTF-8<br />
LC_MESSAGES="de_DE.UTF-8"<br />
LC_PAPER=de_DE.UTF-8<br />
LC_NAME=de_DE.UTF-8<br />
LC_ADDRESS=de_DE.UTF-8<br />
LC_TELEPHONE=de_DE.UTF-8<br />
LC_MEASUREMENT=de_DE.UTF-8<br />
LC_IDENTIFICATION=de_DE.UTF-8<br />
LC_ALL=<br />
<br />
But after specifying that i want a new login shell the default locale configured for that user/system will be used instead!<br />
<br />
[root@showcase-community-centos-7 vagrant]# su -s /bin/bash open-xchange -l<br />
-bash-4.2$ locale -v <br />
LANG=en_US.UTF-8<br />
LC_CTYPE="en_US.UTF-8"<br />
LC_NUMERIC="en_US.UTF-8"<br />
LC_TIME="en_US.UTF-8"<br />
LC_COLLATE="en_US.UTF-8"<br />
LC_MONETARY="en_US.UTF-8"<br />
LC_MESSAGES="en_US.UTF-8"<br />
LC_PAPER="en_US.UTF-8"<br />
LC_NAME="en_US.UTF-8"<br />
LC_ADDRESS="en_US.UTF-8"<br />
LC_TELEPHONE="en_US.UTF-8"<br />
LC_MEASUREMENT="en_US.UTF-8"<br />
LC_IDENTIFICATION="en_US.UTF-8"<br />
LC_ALL=<br />
<br />
Add details from https://bugs.open-xchange.com/show_bug.cgi?id=51074<br />
<br />
== Warnings during early start ==<br />
<br />
Early warnings before the start of the JVM and related logging libraries like logback and logstash forwarding are redirected to '''/var/log/open-xchange/open-xchange-console.log'''.<br />
<br />
=== Limits ===<br />
If the open-xchange startup script isn't able to set the proper limits on CentOS 6 due to e.g. hard limits being enforced via pam_limits you'll see the following warning in the console log during startup<br />
<br />
singlenode open-xchange # cat open-xchange-console.log<br />
/opt/open-xchange/sbin/open-xchange: line 115: ulimit: max user processes: cannot modify limit: Operation not permitted<br />
<br />
=== Locale ===<br />
If the open-xchange startup script isn't able to detect a unicode locale you'll see a warning like the following (the current charmap might differ)<br />
<br />
WARNING: Couldn't detect proper unicode charmap in locale category LC_CTYPE.<br />
WARNING: Current charmap: "ANSI_X3.4-1968" might cause encoding problems.<br />
<br />
== Summary: Open-Xchange middleware on specific distros ==<br />
The support for the mentioned mechanism of resource control differ depending on the distribution and the init system in use.<br />
<br />
=== Debian 7 ===<br />
;Init<br />
: System V style<br />
;OX Configurable Limits/Defaults<br />
: nofile, nproc<br />
<br />
The mentioned limits can be configured via '''/opt/open-xchange/etc/ox-scriptconf.sh'''. The limits are applied via ulimit in the service's init script. The open-xchange service is finally started via start-stop-daemon which doesn't doesn't consider '''/etc/security/limits.*'''<br />
<br />
=== RHEL 6 / CentOS 6 ===<br />
;Init<br />
: Upstart, System V compatible<br />
;OX Configurable Limits/Defaults<br />
: nofile, nproc<br />
<br />
The mentioned limits can be configured via '''/opt/open-xchange/etc/ox-scriptconf.sh'''. The limits are applied via ulimit in the service's init script. Furthermore as the open-xchange service is finally started via '''su ... open-xchange''' on this distro a user session is opened via su/pam and the default CentOS pam config reads the '''/etc/security/limits.*''' configuration by loading the pam stack like:<br />
<br />
: '''/etc/pam.d/su'''<br />
:: '''-> /etc/pam.d/system-auth'''<br />
::: '''-> pam_limits.so'''<br />
<br />
If NPROC isn't configured for the open-xchange-server it's restricted to '''1024''' globally by default to prevent accidental fork bombs, see '''/etc/security/limits.d/90-nproc.conf''' which can result in severe problems modern multithreaded applications.<br />
<br />
=== RHEL 7 / CentOS 7 / Debian 8 / SLE 12 ===<br />
;Init<br />
: Systemd<br />
;OX Configurable Limits/Defaults<br />
: nofile, nproc<br />
<br />
For systemd the default limits are configured directly in the service's unit file that is shipped by OX and located at '''/usr/lib/systemd/system/open-xchange.service'''. The drop-in config to override or extend the default unit file is located at '''/etc/systemd/system/open-xchange.service.d/limits.conf'''. [https://www.freedesktop.org/software/systemd/man/systemd.exec.html Systemd.exec] shows a whole lot of options that can be used by admins to adapt the default service to their specific needs.</div>Marenshttps://oxpedia.org/wiki/index.php?title=Sandbox:Middleware_Startup/sandbox&diff=23221Sandbox:Middleware Startup/sandbox2017-04-24T19:41:02Z<p>Marens: </p>
<hr />
<div><br />
== Systemd ==<br />
<br />
==== Service definition ====<br />
The default service definition as shipped by Open-Xchange is rather concise and easy to read. You can check by looking at the service file that is installed to either /usr/lib/systemd or /lib/systemd depending on your distribution of choice.<br />
<br />
singlenode:~ # cat /usr/lib/systemd/system/open-xchange.service <br />
[Unit]<br />
After=remote-fs.target<br />
After=time-sync.target ypbind.service sendmail.service cyrus.service <br />
<br />
[Service]<br />
User=open-xchange<br />
PermissionsStartOnly=true<br />
TimeoutStartSec=0<br />
ExecStartPre=/opt/open-xchange/sbin/triggerupdatethemes -u<br />
ExecStart=/opt/open-xchange/sbin/open-xchange<br />
ExecStop=/opt/open-xchange/sbin/shutdown -w<br />
ExecReload=/opt/open-xchange/sbin/triggerreloadconfiguration -d<br />
KillMode=process<br />
LimitNOFILE=65536<br />
LimitNPROC=65536<br />
<br />
[Install]<br />
WantedBy=multi-user.target<br />
<br />
==== Drop-in configs ====<br />
Drop in configs allow administrators to easily adapt and override default service unit files. So if you want to change the default limits or add additional limits have a look at our default example drop-in config which is located at /etc/systemd/system/open-xchange.service.d/limits.conf<br />
<br />
singlenode:~ # cat /etc/systemd/system/open-xchange.service.d/limits.conf <br />
# Override and add options in this file<br />
# See systemd.exec(5) for other limits<br />
<br />
[Service]<br />
#LimitNPROC=65536<br />
#LimitNOFILE=65536<br />
<br />
The systemd.exec documentation shows a whole lot of options that can be used by admins to adapt the default service to their specific needs.<br />
<br />
<br clear=all><br />
[[File:mw_start_sysd.png|thumb|Service startup using systemd|left|1000px]]<br />
<br clear=all><br />
<br />
== System V / Upstart ==<br />
<br />
==== Service definition ====<br />
<br />
The default service definition as shipped by Open-Xchange can be found at /etc/init.d/open-xchange and will differ slightly depending on the distro in use.<br />
<br />
<br clear=all><br />
[[File:mw_start_sysv.png|thumb|Service startup using upstart|left|1000px]]<br />
<br clear=all><br />
== Apply Resource limits ==<br />
<br />
As you can see in the two diagrams above one of the differences is that systemd reads all the infos wrt. resource limitations from the original service file and then applies possible further restrictions from the drop-in configs (steps 2 and 8 in the diagram). Limits configured in /opt/open-xchange/etc/ox-scriptconf.sh aren't considered anylonger when using systemd while the older System V style init has to source /opt/open-xchange/etc/ox-scriptconf.sh to be able to read the configs for the maximum number of processes (NPROC) and files(NRFILES) that the middleware shall be allowed to create/open (steps 3,4 and 15 in the diagram).<br />
<br />
=== Background Infos ===<br />
<br />
Several ways exist to restrict resources on a linux system from a global level down to user/groups or even shells and the processes started by them.<br />
<br />
==== Sysctl ====<br />
Sysctl is used to modify kernel parameters at runtime. E.g. to set the maximum number of files<br />
<br />
$ sysctl -w fs.file-max=100000<br />
<br />
To permanently set them append to the main configuration file and reload the settings<br />
<br />
$ echo fs.file-max=100000 >> /etc/sysctl.conf<br />
$ sysctl -p<br />
<br />
More infos can be found via '''man sysctl'''<br />
<br />
==== Limits.conf ====<br />
<br />
Allows to restrict resources an a global, group or user level. E.g:<br />
<br />
$ cat /etc/security/limits.d/90-nproc.conf <br />
# Default limit for number of user's processes to prevent<br />
# accidental fork bombs.<br />
# See rhbz #432903 for reasoning.<br />
<br />
* soft nproc 1024<br />
<br />
From '''man limits.conf''':<br />
<blockquote><br />
''Also, please note that all limit settings are set per login. They are not global, nor are they permanent; existing only for the duration of the session.''<br />
</blockquote><br />
<br />
The limits per login are applied via the ''pam'' stack. See '''man pam''' and '''man pam_limits''' for more details. As those limits are bound to sessions they don't affect most daemons started by our supported init systems or init utils. Most state that they are ignored by design, see [https://bugs.launchpad.net/ubuntu/+source/upstart/+bug/938669/comments/4 upstart], [https://bugzilla.redhat.com/show_bug.cgi?id=754285#c1 systemd] and [https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=302079 start-stop-daemon] <br />
<br />
===== Ulimit =====<br />
From '''man bash'''<br />
<blockquote><br />
ulimit [-HSTabcdefilmnpqrstuvx [limit]]<br />
Provides control over the resources available to the shell and to processes started by it, on systems that allow such control.<br />
</blockquote><br />
This is what we use in our System V compatible init scripts to increase resources for the open-xchange process across multiple distros. Currently only ''the maximum number of processes'' and ''the maximum number of open file descriptors available to a single user'' are increased via ulimit. The values are specified in '''/opt/open-xchange/ox-scriptconf.sh'''<br />
<br />
==== Systemd ====<br />
<br />
===== Control Groups=====<br />
<br />
Control groups should only affect the OX middleware if you create/manage them yourself of if you are using a ''modern'' distribution that already uses systemd as init.<br />
<br />
Citing from the kernel cgroup [https://www.kernel.org/doc/Documentation/cgroup-v2.txt documentation]:<br />
<br />
<blockquote><br />
1-2. What is cgroup?<br />
<br />
cgroup is a mechanism to organize processes hierarchically and<br />
distribute system resources along the hierarchy in a controlled and<br />
configurable manner.<br />
<br />
cgroup is largely composed of two parts - the core and controllers.<br />
cgroup core is primarily responsible for hierarchically organizing<br />
processes. A cgroup controller is usually responsible for<br />
distributing a specific type of system resource along the hierarchy<br />
although there are utility controllers which serve purposes other than<br />
resource distribution.<br />
<br />
cgroups form a tree structure and every process in the system belongs<br />
to one and only one cgroup. All threads of a process belong to the<br />
same cgroup. On creation, all processes are put in the cgroup that<br />
the parent process belongs to at the time. A process can be migrated<br />
to another cgroup. Migration of a process doesn't affect already<br />
existing descendant processes.<br />
<br />
Following certain structural constraints, controllers may be enabled or<br />
disabled selectively on a cgroup. All controller behaviors are<br />
hierarchical - if a controller is enabled on a cgroup, it affects all<br />
processes which belong to the cgroups consisting the inclusive<br />
sub-hierarchy of the cgroup. When a controller is enabled on a nested<br />
cgroup, it always restricts the resource distribution further. The<br />
restrictions set closer to the root in the hierarchy can not be<br />
overridden from further away.<br />
</blockquote><br />
<br />
So processes are organized into a tree structure of control groups and controllers are responsible for the distribution of resources. So what kind of controllers exist?<br />
<br />
<blockquote><br />
5. Controllers<br />
<br />
5-1. CPU<br />
<br />
The "cpu" controllers regulates distribution of CPU cycles. This<br />
controller implements weight and absolute bandwidth limit models for<br />
normal scheduling policy and absolute bandwidth allocation model for<br />
realtime scheduling policy.<br />
<br />
5-2. Memory<br />
<br />
The "memory" controller regulates distribution of memory.<br />
...<br />
While not completely water-tight, all major memory usages by a given<br />
cgroup are tracked so that the total memory consumption can be<br />
accounted and controlled to a reasonable extent.<br />
<br />
5-3. IO<br />
<br />
The "io" controller regulates the distribution of IO resources. This<br />
controller implements both weight based and absolute bandwidth or IOPS<br />
limit distribution; however, weight based distribution is available<br />
only if cfq-iosched is in use and neither scheme is available for<br />
blk-mq devices.<br />
</blockquote><br />
<br />
The open-xchange service is simply put into the default ''system.slice'' without applying further limits.<br />
<br />
singlenode$ systemd-cgls<br />
├─1 /sbin/init<br />
├─system.slice<br />
│ ├─avahi-daemon.service<br />
│ │ ├─501 avahi-daemon: running [singlenode]<br />
│ │ └─514 avahi-daemon: chroot helper<br />
│ ├─console-kit-daemon.service<br />
│ │ └─16164 /usr/sbin/console-kit-daemon --no-daemon<br />
│ ├─dbus.service<br />
│ │ └─508 /usr/bin/dbus-daemon --system --address=systemd: --nofork --nopidfile --systemd-activation<br />
│ ├─munin-node.service<br />
│ │ └─4290 /usr/bin/perl -wT /usr/sbin/munin-node<br />
│ ├─open-xchange.service<br />
│ │ └─6037 /usr/bin/java -Dsun.net.inetaddr.ttl=3600 -Dnetworkaddress.cache.ttl=3600 -Dnetworkaddress.cache.negative.ttl=10 ...<br />
<br />
To check all the details use<br />
<br />
singlenode:~ # systemctl show system.slice<br />
<br />
==== Limits besides control groups ====<br />
Besides control groups systemd allows you to apply other limits to the execution environment of your service. Here we can apply the limits that would normally be applied via ''limits.conf'' or ''ulimit''. Systemd uses [http://man7.org/linux/man-pages/man2/setrlimit.2.html setrlimit] for this. The options that we set by default are:<br />
<br />
* LimitNOFILE<br />
* LimitNPROC<br />
<br />
which you have already seen when we took a look at the [[#Service_definiton|service definition]] and [[#Drop-in_config|drop-in configs]]<br />
<br />
<br />
== Verify limits ==<br />
<br />
=== System V ===<br />
<br />
Get the proper pid of the java process started for the middleware by either checking '''ps faux'''<br />
root 4103 0.0 0.0 66528 1868 pts/0 S 22:05 0:00 su -s /bin/bash open-xchange -c /opt/open-xchange/sbin/open-xchange<br />
494 4109 9.5 19.8 2224644 381312 ? Ssl 22:05 1:12 \_ /usr/bin/java -Dsun.net.inetaddr.ttl=3600 -Dnetworkaddress.cache.ttl=3600<br />
<br />
or programmatically<br />
singlenode:~ # pid=$(pgrep -P $(</var/run/open-xchange.pid) java)<br />
<br />
and check the limits applied for that process<br />
singlenode:~ # cat /proc/$pid/limits<br />
Limit Soft Limit Hard Limit Units<br />
Max cpu time unlimited unlimited seconds<br />
Max file size unlimited unlimited bytes<br />
Max data size unlimited unlimited bytes<br />
Max stack size 8388608 unlimited bytes<br />
Max core file size 0 unlimited bytes<br />
Max resident set unlimited unlimited bytes<br />
Max processes 65536 65536 processes<br />
Max open files 65536 65536 files<br />
Max locked memory 65536 65536 bytes<br />
Max address space unlimited unlimited bytes<br />
Max file locks unlimited unlimited locks<br />
Max pending signals 24254 24254 signals<br />
Max msgqueue size 819200 819200 bytes<br />
Max nice priority 0 0<br />
Max realtime priority 0 0<br />
Max realtime timeout unlimited unlimited us<br />
<br />
=== Systemd ===<br />
singlenode:~ # systemctl show open-xchange | grep Limit<br />
StartLimitInterval=10000000<br />
StartLimitBurst=5<br />
StartLimitAction=none<br />
MemoryLimit=18446744073709551615<br />
LimitCPU=18446744073709551615<br />
LimitFSIZE=18446744073709551615<br />
LimitDATA=18446744073709551615<br />
LimitSTACK=18446744073709551615<br />
LimitCORE=18446744073709551615<br />
LimitRSS=18446744073709551615<br />
LimitNOFILE=65536<br />
LimitAS=18446744073709551615<br />
LimitNPROC=65536<br />
LimitMEMLOCK=65536<br />
LimitLOCKS=18446744073709551615<br />
LimitSIGPENDING=19827<br />
LimitMSGQUEUE=819200<br />
LimitNICE=0<br />
LimitRTPRIO=0<br />
LimitRTTIME=18446744073709551615<br />
<br />
== Evaluate JVM commandline options ==<br />
<br />
Before starting Java and the OSGi framework with all the Open-Xchange middleware bundles we have to evaluate several important JVM commandline options that influence the performance and stability of the Open-Xchange middleware stack. These commandline options are specified in '''/opt/open-xchange/etc/ox-scriptconf.sh'''<br />
<br />
=== Format < 7.8.4 ===<br />
For version up to 7.8.3 the JVM configuration options were all specified via a single property '''JAVA_XTRAOPTS''' which resulted in a rather long configuration string that was hard to read and maintain.<br />
<br />
JAVA_XTRAOPTS="-Dsun.net.inetaddr.ttl=3600 -Dnetworkaddress.cache.ttl=3600 -Dnetworkaddress.cache.negative.ttl=10 \<br />
-Dlogback.threadlocal.put.duplicate=false -server -Djava.awt.headless=true -Xmx512M -XX:MaxPermSize=256m -XX:+UseConcMarkSweepGC \<br />
-XX:+UseParNewGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:NewRatio=3 -XX:+UseTLAB \<br />
-Dosgi.compatibility.bootdelegation=false -XX:-OmitStackTraceInFastThrow"<br />
<br />
During the startup of the open-xchange service (step 11 of diagram Service startup using systemd and step 10 of diagram Service startup using upstart.) this JAVA_XTRAOPTS string goes through a basic validity check and is then passed to the JVM. <br />
<br />
=== Format >= 7.8.4 ===<br />
Beginning from version 7.8.4 the options are split up and sorted into different categories like:<br />
<br />
JAVA_OPTS_GC="-XX:+UseConcMarkSweepGC -XX:+UseParNewGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:NewRatio=3 -XX:+DisableExplicitGC"<br />
JAVA_OPTS_LOG="-Dlogback.threadlocal.put.duplicate=false -XX:-OmitStackTraceInFastThrow"<br />
JAVA_OPTS_MEM="-XX:MaxHeapSize=512M -XX:MaxPermSize=256M -XX:+UseTLAB"<br />
JAVA_OPTS_NET="-Dsun.net.inetaddr.ttl=3600 -Dnetworkaddress.cache.ttl=3600 -Dnetworkaddress.cache.negative.ttl=10"<br />
JAVA_OPTS_OSGI="-Dosgi.compatibility.bootdelegation=false"<br />
JAVA_OPTS_SERVER="-server -Djava.awt.headless=true"<br />
<br />
JAVA_OPTS_OTHER=""<br />
<br />
#JAVA_OPTS_DEBUG="-XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/mnt/heapdump -Xloggc:/var/log/open-xchange/gc.log -verbose:gc -XX:+PrintGCDateStamps \<br />
-XX:+PrintHeapAtGC -XX:+PrintGCApplicationStoppedTime -XX:+PrintTenuringDistribution"<br />
<br />
which is much easier to read an maintain. Furthermore we added a useful '''JAVA_OPTS_DEBUG''' configuration parameter that comes preconfigured but unused/commented so that customers can quickly enable the most commonly used debugging options in case they have to provide detailed system information to Open-Xchange.<br />
<br />
During the startup of the open-xchange service (step 11 of diagram ''Service startup using systemd'' and step 10 of diagram ''Service startup using upstart''.) all uncommented JAVA_OPTS_* categories will then be evaluated and combined again into a single string of commandline options that can then be passed to the JVM.<br />
<br />
=== Upgrade procedure ===<br />
During the upgrade '''7.8.3-revX -> 7.8.4-revY''' existing JAVA_XTRAOPTS will automatically be split up and migrated to the new categories approach. A backup of the file (ox-scriptconf.sh.timestamp) will be created and admins are urged to check for proper migration before deleting the backup file like it's already done with *.dpkg-dist or *.rpmnew files on Debian or RHEL/SLES distributions.<br />
<br />
== Encoding detection during JVM startup ==<br />
<br />
=== Background Infos ===<br />
All of en_US.UTF-8, .utf8, .utf-8, .UTF_8 will result in a valid UTF-8 charmap<br />
<br />
==== JVM startup ====<br />
<br />
The default file.encoding detection is done at JVM startup based upon LC_CTYPE which makes sense as it's documented as:<br />
<br />
CODESET (LC_CTYPE)<br />
Return a string with the name of the character encoding used in the selected locale, such as "UTF-8", "ISO-8859-1",<br />
or "ANSI_X3.4-1968" (better known as US-ASCII). This is the same string that you get with "locale charmap".<br />
For a list of character encoding names, try "locale -m", cf. locale(1).<br />
<br />
Furthermore this influences the defaultCharset selection of the JVM.<br />
<br />
/**<br />
* Returns the default charset of this Java virtual machine.<br />
*<br />
* The default charset is determined during virtual-machine startup and<br />
* typically depends upon the locale and charset of the underlying<br />
* operating system.<br />
*<br />
* @return A charset object for the default charset<br />
*<br />
* @since 1.5<br />
*/<br />
public static Charset defaultCharset() {<br />
if (defaultCharset == null) {<br />
synchronized (Charset.class) {<br />
String csn = AccessController.doPrivileged(<br />
new GetPropertyAction("file.encoding"));<br />
Charset cs = lookup(csn);<br />
if (cs != null)<br />
defaultCharset = cs;<br />
else<br />
defaultCharset = forName("UTF-8");<br />
}<br />
}<br />
return defaultCharset;<br />
}<br />
<br />
Although we try to specify valid unicode encodings for every IO operations the Open-Xchange middleware does some parts of the JVM don't allow to specify the encoding to use and falls back to the detected default encoding. That's why admins have to make sure to provide a proper unicode environment for the middleware service during every start or use documented workarounds provided by Open-Xchange (e.g. )<br />
<br />
=== Query which file encoding was detected by the JVM during startup ===<br />
<br />
/opt/open-xchange/sbin/showruntimestats -r | sed -ne 's/.*\({key=file.encoding, value=[^}]*}\).*/\1/p'<br />
{key=file.encoding, value=UTF-8}<br />
<br />
=== Locale passing from client to server via SSH ===<br />
<br />
The following setting of sshd allows clients to specify which locale to use on the server and might lead to some confusion or unexpected behaviour when checking the system locale or even to the usage of broken locales during middleware startup if your client sends misconfigured or missing locale configuration to the server.<br />
<br />
grep AcceptEnv /etc/ssh/sshd_config <br />
AcceptEnv LANG LC_*<br />
<br />
For example when i connect to the machine my local german locale is sent to the server <br />
<br />
[vagrant@showcase-community-centos-7 ~]$ locale -v <br />
LANG=de_DE.UTF-8<br />
LC_CTYPE="de_DE.UTF-8"<br />
LC_NUMERIC=de_DE.UTF-8<br />
LC_TIME=de_DE.UTF-8<br />
LC_COLLATE="de_DE.UTF-8"<br />
LC_MONETARY=de_DE.UTF-8<br />
LC_MESSAGES="de_DE.UTF-8"<br />
LC_PAPER=de_DE.UTF-8<br />
LC_NAME=de_DE.UTF-8<br />
LC_ADDRESS=de_DE.UTF-8<br />
LC_TELEPHONE=de_DE.UTF-8<br />
LC_MEASUREMENT=de_DE.UTF-8<br />
LC_IDENTIFICATION=de_DE.UTF-8<br />
LC_ALL=<br />
<br />
But after specifying that i want a new login shell the default locale configured for that user/system will be used instead!<br />
<br />
[root@showcase-community-centos-7 vagrant]# su -s /bin/bash open-xchange -l<br />
-bash-4.2$ locale -v <br />
LANG=en_US.UTF-8<br />
LC_CTYPE="en_US.UTF-8"<br />
LC_NUMERIC="en_US.UTF-8"<br />
LC_TIME="en_US.UTF-8"<br />
LC_COLLATE="en_US.UTF-8"<br />
LC_MONETARY="en_US.UTF-8"<br />
LC_MESSAGES="en_US.UTF-8"<br />
LC_PAPER="en_US.UTF-8"<br />
LC_NAME="en_US.UTF-8"<br />
LC_ADDRESS="en_US.UTF-8"<br />
LC_TELEPHONE="en_US.UTF-8"<br />
LC_MEASUREMENT="en_US.UTF-8"<br />
LC_IDENTIFICATION="en_US.UTF-8"<br />
LC_ALL=<br />
<br />
Add details from https://bugs.open-xchange.com/show_bug.cgi?id=51074<br />
<br />
== Warnings during early start ==<br />
<br />
Early warnings before the start of the JVM and related logging libraries like logback and logstash forwarding are redirected to '''/var/log/open-xchange/open-xchange-console.log'''.<br />
<br />
=== Limits ===<br />
If the open-xchange startup script isn't able to set the proper limits on CentOS 6 due to e.g. hard limits being enforced via pam_limits you'll see the following warning in the console log during startup<br />
<br />
singlenode open-xchange # cat open-xchange-console.log<br />
/opt/open-xchange/sbin/open-xchange: line 115: ulimit: max user processes: cannot modify limit: Operation not permitted<br />
<br />
=== Locale ===<br />
If the open-xchange startup script isn't able to detect a unicode locale you'll see a warning like the following (the current charmap might differ)<br />
<br />
WARNING: Couldn't detect proper unicode charmap in locale category LC_CTYPE.<br />
WARNING: Current charmap: "ANSI_X3.4-1968" might cause encoding problems.<br />
<br />
== Summary: Open-Xchange middleware on specific distros ==<br />
The support for the mentioned mechanism of resource control differ depending on the distribution and the init system in use.<br />
<br />
=== Debian 7 ===<br />
;Init<br />
: System V style<br />
;OX Configurable Limits/Defaults<br />
: nofile, nproc<br />
<br />
The mentioned limits can be configured via '''/opt/open-xchange/etc/ox-scriptconf.sh'''. The limits are applied via ulimit in the service's init script. The open-xchange service is finally started via start-stop-daemon which doesn't doesn't consider '''/etc/security/limits.*'''<br />
<br />
=== RHEL 6 / CentOS 6 ===<br />
;Init<br />
: Upstart, System V compatible<br />
;OX Configurable Limits/Defaults<br />
: nofile, nproc<br />
<br />
The mentioned limits can be configured via '''/opt/open-xchange/etc/ox-scriptconf.sh'''. The limits are applied via ulimit in the service's init script. Furthermore as the open-xchange service is finally started via '''su ... open-xchange''' on this distro a user session is opened via su/pam and the default CentOS pam config reads the '''/etc/security/limits.*''' configuration by loading the pam stack like:<br />
<br />
: '''/etc/pam.d/su'''<br />
:: '''-> /etc/pam.d/system-auth'''<br />
::: '''-> pam_limits.so'''<br />
<br />
If NPROC isn't configured for the open-xchange-server it's restricted to '''1024''' globally by default to prevent accidental fork bombs, see '''/etc/security/limits.d/90-nproc.conf''' which can result in severe problems modern multithreaded applications.<br />
<br />
=== RHEL 7 / CentOS 7 / Debian 8 / SLE 12 ===<br />
;Init<br />
: Systemd<br />
;OX Configurable Limits/Defaults<br />
: nofile, nproc<br />
<br />
For systemd the default limits are configured directly in the service's unit file that is shipped by OX and located at '''/usr/lib/systemd/system/open-xchange.service'''. The drop-in config to override or extend the default unit file is located at '''/etc/systemd/system/open-xchange.service.d/limits.conf'''. [https://www.freedesktop.org/software/systemd/man/systemd.exec.html Systemd.exec] shows a whole lot of options that can be used by admins to adapt the default service to their specific needs.</div>Marenshttps://oxpedia.org/wiki/index.php?title=Sandbox:Middleware_Startup/sandbox&diff=23220Sandbox:Middleware Startup/sandbox2017-04-24T19:38:31Z<p>Marens: /* Warnings during early */</p>
<hr />
<div><br />
== Systemd ==<br />
<br />
==== Service definition ====<br />
The default service definition as shipped by Open-Xchange is rather concise and easy to read. You can check by looking at the service file that is installed to either /usr/lib/systemd or /lib/systemd depending on your distribution of choice.<br />
<br />
singlenode:~ # cat /usr/lib/systemd/system/open-xchange.service <br />
[Unit]<br />
After=remote-fs.target<br />
After=time-sync.target ypbind.service sendmail.service cyrus.service <br />
<br />
[Service]<br />
User=open-xchange<br />
PermissionsStartOnly=true<br />
TimeoutStartSec=0<br />
ExecStartPre=/opt/open-xchange/sbin/triggerupdatethemes -u<br />
ExecStart=/opt/open-xchange/sbin/open-xchange<br />
ExecStop=/opt/open-xchange/sbin/shutdown -w<br />
ExecReload=/opt/open-xchange/sbin/triggerreloadconfiguration -d<br />
KillMode=process<br />
LimitNOFILE=65536<br />
LimitNPROC=65536<br />
<br />
[Install]<br />
WantedBy=multi-user.target<br />
<br />
==== Drop-in configs ====<br />
Drop in configs allow administrators to easily adapt and override default service unit files. So if you want to change the default limits or add additional limits have a look at our default example drop-in config which is located at /etc/systemd/system/open-xchange.service.d/limits.conf<br />
<br />
singlenode:~ # cat /etc/systemd/system/open-xchange.service.d/limits.conf <br />
# Override and add options in this file<br />
# See systemd.exec(5) for other limits<br />
<br />
[Service]<br />
#LimitNPROC=65536<br />
#LimitNOFILE=65536<br />
<br />
The systemd.exec documentation shows a whole lot of options that can be used by admins to adapt the default service to their specific needs.<br />
<br />
<br clear=all><br />
[[File:mw_start_sysd.png|thumb|Service startup using systemd|left|1000px]]<br />
<br clear=all><br />
<br />
== System V / Upstart ==<br />
<br />
==== Service definition ====<br />
<br />
The default service definition as shipped by Open-Xchange can be found at /etc/init.d/open-xchange and will differ slightly depending on the distro in use.<br />
<br />
<br clear=all><br />
[[File:mw_start_sysv.png|thumb|Service startup using upstart|left|1000px]]<br />
<br clear=all><br />
== Apply Resource limits ==<br />
<br />
As you can see in the two diagrams above one of the differences is that systemd reads all the infos wrt. resource limitations from the original service file and then applies possible further restrictions from the drop-in configs (steps 2 and 8 in the diagram). Limits configured in /opt/open-xchange/etc/ox-scriptconf.sh aren't considered anylonger when using systemd while the older System V style init has to source /opt/open-xchange/etc/ox-scriptconf.sh to be able to read the configs for the maximum number of processes (NPROC) and files(NRFILES) that the middleware shall be allowed to create/open (steps 3,4 and 15 in the diagram).<br />
<br />
=== Background Infos ===<br />
<br />
Several ways exist to restrict resources on a linux system from a global level down to user/groups or even shells and the processes started by them.<br />
<br />
==== Sysctl ====<br />
Sysctl is used to modify kernel parameters at runtime. E.g. to set the maximum number of files<br />
<br />
$ sysctl -w fs.file-max=100000<br />
<br />
To permanently set them append to the main configuration file and reload the settings<br />
<br />
$ echo fs.file-max=100000 >> /etc/sysctl.conf<br />
$ sysctl -p<br />
<br />
More infos can be found via '''man sysctl'''<br />
<br />
==== Limits.conf ====<br />
<br />
Allows to restrict resources an a global, group or user level. E.g:<br />
<br />
$ cat /etc/security/limits.d/90-nproc.conf <br />
# Default limit for number of user's processes to prevent<br />
# accidental fork bombs.<br />
# See rhbz #432903 for reasoning.<br />
<br />
* soft nproc 1024<br />
<br />
From '''man limits.conf''':<br />
<blockquote><br />
''Also, please note that all limit settings are set per login. They are not global, nor are they permanent; existing only for the duration of the session.''<br />
</blockquote><br />
<br />
The limits per login are applied via the ''pam'' stack. See '''man pam''' and '''man pam_limits''' for more details. As those limits are bound to sessions they don't affect most daemons started by our supported init systems or init utils. Most state that they are ignored by design, see [https://bugs.launchpad.net/ubuntu/+source/upstart/+bug/938669/comments/4 upstart], [https://bugzilla.redhat.com/show_bug.cgi?id=754285#c1 systemd] and [https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=302079 start-stop-daemon] <br />
<br />
===== Ulimit =====<br />
From '''man bash'''<br />
<blockquote><br />
ulimit [-HSTabcdefilmnpqrstuvx [limit]]<br />
Provides control over the resources available to the shell and to processes started by it, on systems that allow such control.<br />
</blockquote><br />
This is what we use in our System V compatible init scripts to increase resources for the open-xchange process across multiple distros. Currently only ''the maximum number of processes'' and ''the maximum number of open file descriptors available to a single user'' are increased via ulimit. The values are specified in '''/opt/open-xchange/ox-scriptconf.sh'''<br />
<br />
==== Systemd ====<br />
<br />
===== Control Groups=====<br />
<br />
Control groups should only affect the OX middleware if you create/manage them yourself of if you are using a ''modern'' distribution that already uses systemd as init.<br />
<br />
Citing from the kernel cgroup [https://www.kernel.org/doc/Documentation/cgroup-v2.txt documentation]:<br />
<br />
<blockquote><br />
1-2. What is cgroup?<br />
<br />
cgroup is a mechanism to organize processes hierarchically and<br />
distribute system resources along the hierarchy in a controlled and<br />
configurable manner.<br />
<br />
cgroup is largely composed of two parts - the core and controllers.<br />
cgroup core is primarily responsible for hierarchically organizing<br />
processes. A cgroup controller is usually responsible for<br />
distributing a specific type of system resource along the hierarchy<br />
although there are utility controllers which serve purposes other than<br />
resource distribution.<br />
<br />
cgroups form a tree structure and every process in the system belongs<br />
to one and only one cgroup. All threads of a process belong to the<br />
same cgroup. On creation, all processes are put in the cgroup that<br />
the parent process belongs to at the time. A process can be migrated<br />
to another cgroup. Migration of a process doesn't affect already<br />
existing descendant processes.<br />
<br />
Following certain structural constraints, controllers may be enabled or<br />
disabled selectively on a cgroup. All controller behaviors are<br />
hierarchical - if a controller is enabled on a cgroup, it affects all<br />
processes which belong to the cgroups consisting the inclusive<br />
sub-hierarchy of the cgroup. When a controller is enabled on a nested<br />
cgroup, it always restricts the resource distribution further. The<br />
restrictions set closer to the root in the hierarchy can not be<br />
overridden from further away.<br />
</blockquote><br />
<br />
So processes are organized into a tree structure of control groups and controllers are responsible for the distribution of resources. So what kind of controllers exist?<br />
<br />
<blockquote><br />
5. Controllers<br />
<br />
5-1. CPU<br />
<br />
The "cpu" controllers regulates distribution of CPU cycles. This<br />
controller implements weight and absolute bandwidth limit models for<br />
normal scheduling policy and absolute bandwidth allocation model for<br />
realtime scheduling policy.<br />
<br />
5-2. Memory<br />
<br />
The "memory" controller regulates distribution of memory.<br />
...<br />
While not completely water-tight, all major memory usages by a given<br />
cgroup are tracked so that the total memory consumption can be<br />
accounted and controlled to a reasonable extent.<br />
<br />
5-3. IO<br />
<br />
The "io" controller regulates the distribution of IO resources. This<br />
controller implements both weight based and absolute bandwidth or IOPS<br />
limit distribution; however, weight based distribution is available<br />
only if cfq-iosched is in use and neither scheme is available for<br />
blk-mq devices.<br />
</blockquote><br />
<br />
The open-xchange service is simply put into the default ''system.slice'' without applying further limits.<br />
<br />
singlenode$ systemd-cgls<br />
├─1 /sbin/init<br />
├─system.slice<br />
│ ├─avahi-daemon.service<br />
│ │ ├─501 avahi-daemon: running [singlenode]<br />
│ │ └─514 avahi-daemon: chroot helper<br />
│ ├─console-kit-daemon.service<br />
│ │ └─16164 /usr/sbin/console-kit-daemon --no-daemon<br />
│ ├─dbus.service<br />
│ │ └─508 /usr/bin/dbus-daemon --system --address=systemd: --nofork --nopidfile --systemd-activation<br />
│ ├─munin-node.service<br />
│ │ └─4290 /usr/bin/perl -wT /usr/sbin/munin-node<br />
│ ├─open-xchange.service<br />
│ │ └─6037 /usr/bin/java -Dsun.net.inetaddr.ttl=3600 -Dnetworkaddress.cache.ttl=3600 -Dnetworkaddress.cache.negative.ttl=10 ...<br />
<br />
To check all the details use<br />
<br />
singlenode:~ # systemctl show system.slice<br />
<br />
==== Limits besides control groups ====<br />
Besides control groups systemd allows you to apply other limits to the execution environment of your service. Here we can apply the limits that would normally be applied via ''limits.conf'' or ''ulimit''. Systemd uses [http://man7.org/linux/man-pages/man2/setrlimit.2.html setrlimit] for this. The options that we set by default are:<br />
<br />
* LimitNOFILE<br />
* LimitNPROC<br />
<br />
which you have already seen when we took a look at the [[#Service_definiton|service definition]] and [[#Drop-in_config|drop-in configs]]<br />
<br />
<br />
== Verify limits ==<br />
<br />
=== System V ===<br />
<br />
Get the proper pid of the java process started for the middleware by either checking '''ps faux'''<br />
root 4103 0.0 0.0 66528 1868 pts/0 S 22:05 0:00 su -s /bin/bash open-xchange -c /opt/open-xchange/sbin/open-xchange<br />
494 4109 9.5 19.8 2224644 381312 ? Ssl 22:05 1:12 \_ /usr/bin/java -Dsun.net.inetaddr.ttl=3600 -Dnetworkaddress.cache.ttl=3600<br />
<br />
or programmatically<br />
singlenode:~ # pid=$(pgrep -P $(</var/run/open-xchange.pid) java)<br />
<br />
and check the limits applied for that process<br />
singlenode:~ # cat /proc/$pid/limits<br />
Limit Soft Limit Hard Limit Units<br />
Max cpu time unlimited unlimited seconds<br />
Max file size unlimited unlimited bytes<br />
Max data size unlimited unlimited bytes<br />
Max stack size 8388608 unlimited bytes<br />
Max core file size 0 unlimited bytes<br />
Max resident set unlimited unlimited bytes<br />
Max processes 65536 65536 processes<br />
Max open files 65536 65536 files<br />
Max locked memory 65536 65536 bytes<br />
Max address space unlimited unlimited bytes<br />
Max file locks unlimited unlimited locks<br />
Max pending signals 24254 24254 signals<br />
Max msgqueue size 819200 819200 bytes<br />
Max nice priority 0 0<br />
Max realtime priority 0 0<br />
Max realtime timeout unlimited unlimited us<br />
<br />
=== Systemd ===<br />
singlenode:~ # systemctl show open-xchange | grep Limit<br />
StartLimitInterval=10000000<br />
StartLimitBurst=5<br />
StartLimitAction=none<br />
MemoryLimit=18446744073709551615<br />
LimitCPU=18446744073709551615<br />
LimitFSIZE=18446744073709551615<br />
LimitDATA=18446744073709551615<br />
LimitSTACK=18446744073709551615<br />
LimitCORE=18446744073709551615<br />
LimitRSS=18446744073709551615<br />
LimitNOFILE=65536<br />
LimitAS=18446744073709551615<br />
LimitNPROC=65536<br />
LimitMEMLOCK=65536<br />
LimitLOCKS=18446744073709551615<br />
LimitSIGPENDING=19827<br />
LimitMSGQUEUE=819200<br />
LimitNICE=0<br />
LimitRTPRIO=0<br />
LimitRTTIME=18446744073709551615<br />
<br />
<br />
== Open-Xchange middleware on specific distros ==<br />
The support for the mentioned mechanism of resource control differ depending on the distribution and the init system in use.<br />
<br />
=== Debian 7 ===<br />
;Init<br />
: System V style<br />
;OX Configurable Limits/Defaults<br />
: nofile, nproc<br />
<br />
The mentioned limits can be configured via '''/opt/open-xchange/etc/ox-scriptconf.sh'''. The limits are applied via ulimit in the service's init script. The open-xchange service is finally started via start-stop-daemon which doesn't doesn't consider '''/etc/security/limits.*'''<br />
<br />
=== RHEL 6 / CentOS 6 ===<br />
;Init<br />
: Upstart, System V compatible<br />
;OX Configurable Limits/Defaults<br />
: nofile, nproc<br />
<br />
The mentioned limits can be configured via '''/opt/open-xchange/etc/ox-scriptconf.sh'''. The limits are applied via ulimit in the service's init script. Furthermore as the open-xchange service is finally started via '''su ... open-xchange''' on this distro a user session is opened via su/pam and the default CentOS pam config reads the '''/etc/security/limits.*''' configuration by loading the pam stack like:<br />
<br />
: '''/etc/pam.d/su'''<br />
:: '''-> /etc/pam.d/system-auth'''<br />
::: '''-> pam_limits.so'''<br />
<br />
If NPROC isn't configured for the open-xchange-server it's restricted to '''1024''' globally by default to prevent accidental fork bombs, see '''/etc/security/limits.d/90-nproc.conf''' which can result in severe problems modern multithreaded applications.<br />
<br />
=== RHEL 7 / CentOS 7 / Debian 8 / SLE 12 ===<br />
;Init<br />
: Systemd<br />
;OX Configurable Limits/Defaults<br />
: nofile, nproc<br />
<br />
For systemd the default limits are configured directly in the service's unit file that is shipped by OX and located at '''/usr/lib/systemd/system/open-xchange.service'''. The drop-in config to override or extend the default unit file is located at '''/etc/systemd/system/open-xchange.service.d/limits.conf'''. [https://www.freedesktop.org/software/systemd/man/systemd.exec.html Systemd.exec] shows a whole lot of options that can be used by admins to adapt the default service to their specific needs.<br />
<br />
== Evaluate JVM commandline options ==<br />
<br />
Before starting Java and the OSGi framework with all the Open-Xchange middleware bundles we have to evaluate several important JVM commandline options that influence the performance and stability of the Open-Xchange middleware stack. These commandline options are specified in '''/opt/open-xchange/etc/ox-scriptconf.sh'''<br />
<br />
=== Format < 7.8.4 ===<br />
For version up to 7.8.3 the JVM configuration options were all specified via a single property '''JAVA_XTRAOPTS''' which resulted in a rather long configuration string that was hard to read and maintain.<br />
<br />
JAVA_XTRAOPTS="-Dsun.net.inetaddr.ttl=3600 -Dnetworkaddress.cache.ttl=3600 -Dnetworkaddress.cache.negative.ttl=10 \<br />
-Dlogback.threadlocal.put.duplicate=false -server -Djava.awt.headless=true -Xmx512M -XX:MaxPermSize=256m -XX:+UseConcMarkSweepGC \<br />
-XX:+UseParNewGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:NewRatio=3 -XX:+UseTLAB \<br />
-Dosgi.compatibility.bootdelegation=false -XX:-OmitStackTraceInFastThrow"<br />
<br />
During the startup of the open-xchange service (step 11 of diagram Service startup using systemd and step 10 of diagram Service startup using upstart.) this JAVA_XTRAOPTS string goes through a basic validity check and is then passed to the JVM. <br />
<br />
=== Format >= 7.8.4 ===<br />
Beginning from version 7.8.4 the options are split up and sorted into different categories like:<br />
<br />
JAVA_OPTS_GC="-XX:+UseConcMarkSweepGC -XX:+UseParNewGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:NewRatio=3 -XX:+DisableExplicitGC"<br />
JAVA_OPTS_LOG="-Dlogback.threadlocal.put.duplicate=false -XX:-OmitStackTraceInFastThrow"<br />
JAVA_OPTS_MEM="-XX:MaxHeapSize=512M -XX:MaxPermSize=256M -XX:+UseTLAB"<br />
JAVA_OPTS_NET="-Dsun.net.inetaddr.ttl=3600 -Dnetworkaddress.cache.ttl=3600 -Dnetworkaddress.cache.negative.ttl=10"<br />
JAVA_OPTS_OSGI="-Dosgi.compatibility.bootdelegation=false"<br />
JAVA_OPTS_SERVER="-server -Djava.awt.headless=true"<br />
<br />
JAVA_OPTS_OTHER=""<br />
<br />
#JAVA_OPTS_DEBUG="-XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/mnt/heapdump -Xloggc:/var/log/open-xchange/gc.log -verbose:gc -XX:+PrintGCDateStamps \<br />
-XX:+PrintHeapAtGC -XX:+PrintGCApplicationStoppedTime -XX:+PrintTenuringDistribution"<br />
<br />
which is much easier to read an maintain. Furthermore we added a useful '''JAVA_OPTS_DEBUG''' configuration parameter that comes preconfigured but unused/commented so that customers can quickly enable the most commonly used debugging options in case they have to provide detailed system information to Open-Xchange.<br />
<br />
During the startup of the open-xchange service (step 11 of diagram ''Service startup using systemd'' and step 10 of diagram ''Service startup using upstart''.) all uncommented JAVA_OPTS_* categories will then be evaluated and combined again into a single string of commandline options that can then be passed to the JVM.<br />
<br />
=== Upgrade procedure ===<br />
During the upgrade '''7.8.3-revX -> 7.8.4-revY''' existing JAVA_XTRAOPTS will automatically be split up and migrated to the new categories approach. A backup of the file (ox-scriptconf.sh.timestamp) will be created and admins are urged to check for proper migration before deleting the backup file like it's already done with *.dpkg-dist or *.rpmnew files on Debian or RHEL/SLES distributions.<br />
<br />
== Encoding detection during JVM startup ==<br />
<br />
=== Background Infos ===<br />
All of en_US.UTF-8, .utf8, .utf-8, .UTF_8 will result in a valid UTF-8 charmap<br />
<br />
==== JVM startup ====<br />
<br />
The default file.encoding detection is done at JVM startup based upon LC_CTYPE which makes sense as it's documented as:<br />
<br />
CODESET (LC_CTYPE)<br />
Return a string with the name of the character encoding used in the selected locale, such as "UTF-8", "ISO-8859-1",<br />
or "ANSI_X3.4-1968" (better known as US-ASCII). This is the same string that you get with "locale charmap".<br />
For a list of character encoding names, try "locale -m", cf. locale(1).<br />
<br />
Furthermore this influences the defaultCharset selection of the JVM.<br />
<br />
/**<br />
* Returns the default charset of this Java virtual machine.<br />
*<br />
* The default charset is determined during virtual-machine startup and<br />
* typically depends upon the locale and charset of the underlying<br />
* operating system.<br />
*<br />
* @return A charset object for the default charset<br />
*<br />
* @since 1.5<br />
*/<br />
public static Charset defaultCharset() {<br />
if (defaultCharset == null) {<br />
synchronized (Charset.class) {<br />
String csn = AccessController.doPrivileged(<br />
new GetPropertyAction("file.encoding"));<br />
Charset cs = lookup(csn);<br />
if (cs != null)<br />
defaultCharset = cs;<br />
else<br />
defaultCharset = forName("UTF-8");<br />
}<br />
}<br />
return defaultCharset;<br />
}<br />
<br />
Although we try to specify valid unicode encodings for every IO operations the Open-Xchange middleware does some parts of the JVM don't allow to specify the encoding to use and falls back to the detected default encoding. That's why admins have to make sure to provide a proper unicode environment for the middleware service during every start or use documented workarounds provided by Open-Xchange (e.g. )<br />
<br />
=== Query which file encoding was detected by the JVM during startup ===<br />
<br />
/opt/open-xchange/sbin/showruntimestats -r | sed -ne 's/.*\({key=file.encoding, value=[^}]*}\).*/\1/p'<br />
{key=file.encoding, value=UTF-8}<br />
<br />
=== Locale passing from client to server via SSH ===<br />
<br />
The following setting of sshd allows clients to specify which locale to use on the server and might lead to some confusion or unexpected behaviour when checking the system locale or even to the usage of broken locales during middleware startup if your client sends misconfigured or missing locale configuration to the server.<br />
<br />
grep AcceptEnv /etc/ssh/sshd_config <br />
AcceptEnv LANG LC_*<br />
<br />
For example when i connect to the machine my local german locale is sent to the server <br />
<br />
[vagrant@showcase-community-centos-7 ~]$ locale -v <br />
LANG=de_DE.UTF-8<br />
LC_CTYPE="de_DE.UTF-8"<br />
LC_NUMERIC=de_DE.UTF-8<br />
LC_TIME=de_DE.UTF-8<br />
LC_COLLATE="de_DE.UTF-8"<br />
LC_MONETARY=de_DE.UTF-8<br />
LC_MESSAGES="de_DE.UTF-8"<br />
LC_PAPER=de_DE.UTF-8<br />
LC_NAME=de_DE.UTF-8<br />
LC_ADDRESS=de_DE.UTF-8<br />
LC_TELEPHONE=de_DE.UTF-8<br />
LC_MEASUREMENT=de_DE.UTF-8<br />
LC_IDENTIFICATION=de_DE.UTF-8<br />
LC_ALL=<br />
<br />
But after specifying that i want a new login shell the default locale configured for that user/system will be used instead!<br />
<br />
[root@showcase-community-centos-7 vagrant]# su -s /bin/bash open-xchange -l<br />
-bash-4.2$ locale -v <br />
LANG=en_US.UTF-8<br />
LC_CTYPE="en_US.UTF-8"<br />
LC_NUMERIC="en_US.UTF-8"<br />
LC_TIME="en_US.UTF-8"<br />
LC_COLLATE="en_US.UTF-8"<br />
LC_MONETARY="en_US.UTF-8"<br />
LC_MESSAGES="en_US.UTF-8"<br />
LC_PAPER="en_US.UTF-8"<br />
LC_NAME="en_US.UTF-8"<br />
LC_ADDRESS="en_US.UTF-8"<br />
LC_TELEPHONE="en_US.UTF-8"<br />
LC_MEASUREMENT="en_US.UTF-8"<br />
LC_IDENTIFICATION="en_US.UTF-8"<br />
LC_ALL=<br />
<br />
Add details from https://bugs.open-xchange.com/show_bug.cgi?id=51074<br />
<br />
== Warnings during early start ==<br />
<br />
Early warnings before the start of the JVM and related logging libraries like logback and logstash forwarding are redirected to '''/var/log/open-xchange/open-xchange-console.log'''.<br />
<br />
=== Limits ===<br />
If the open-xchange startup script isn't able to set the proper limits on CentOS 6 due to e.g. hard limits being enforced via pam_limits you'll see the following warning in the console log during startup<br />
<br />
singlenode open-xchange # cat open-xchange-console.log<br />
/opt/open-xchange/sbin/open-xchange: line 115: ulimit: max user processes: cannot modify limit: Operation not permitted<br />
<br />
=== Locale ===<br />
If the open-xchange startup script isn't able to detect a unicode locale you'll see a warning like the following (the current charmap might differ)<br />
<br />
WARNING: Couldn't detect proper unicode charmap in locale category LC_CTYPE.<br />
WARNING: Current charmap: "ANSI_X3.4-1968" might cause encoding problems.</div>Marenshttps://oxpedia.org/wiki/index.php?title=Sandbox:Middleware_Startup/sandbox&diff=23219Sandbox:Middleware Startup/sandbox2017-04-24T19:20:08Z<p>Marens: /* Encoding detection during JVM startup */</p>
<hr />
<div><br />
== Systemd ==<br />
<br />
==== Service definition ====<br />
The default service definition as shipped by Open-Xchange is rather concise and easy to read. You can check by looking at the service file that is installed to either /usr/lib/systemd or /lib/systemd depending on your distribution of choice.<br />
<br />
singlenode:~ # cat /usr/lib/systemd/system/open-xchange.service <br />
[Unit]<br />
After=remote-fs.target<br />
After=time-sync.target ypbind.service sendmail.service cyrus.service <br />
<br />
[Service]<br />
User=open-xchange<br />
PermissionsStartOnly=true<br />
TimeoutStartSec=0<br />
ExecStartPre=/opt/open-xchange/sbin/triggerupdatethemes -u<br />
ExecStart=/opt/open-xchange/sbin/open-xchange<br />
ExecStop=/opt/open-xchange/sbin/shutdown -w<br />
ExecReload=/opt/open-xchange/sbin/triggerreloadconfiguration -d<br />
KillMode=process<br />
LimitNOFILE=65536<br />
LimitNPROC=65536<br />
<br />
[Install]<br />
WantedBy=multi-user.target<br />
<br />
==== Drop-in configs ====<br />
Drop in configs allow administrators to easily adapt and override default service unit files. So if you want to change the default limits or add additional limits have a look at our default example drop-in config which is located at /etc/systemd/system/open-xchange.service.d/limits.conf<br />
<br />
singlenode:~ # cat /etc/systemd/system/open-xchange.service.d/limits.conf <br />
# Override and add options in this file<br />
# See systemd.exec(5) for other limits<br />
<br />
[Service]<br />
#LimitNPROC=65536<br />
#LimitNOFILE=65536<br />
<br />
The systemd.exec documentation shows a whole lot of options that can be used by admins to adapt the default service to their specific needs.<br />
<br />
<br clear=all><br />
[[File:mw_start_sysd.png|thumb|Service startup using systemd|left|1000px]]<br />
<br clear=all><br />
<br />
== System V / Upstart ==<br />
<br />
==== Service definition ====<br />
<br />
The default service definition as shipped by Open-Xchange can be found at /etc/init.d/open-xchange and will differ slightly depending on the distro in use.<br />
<br />
<br clear=all><br />
[[File:mw_start_sysv.png|thumb|Service startup using upstart|left|1000px]]<br />
<br clear=all><br />
== Apply Resource limits ==<br />
<br />
As you can see in the two diagrams above one of the differences is that systemd reads all the infos wrt. resource limitations from the original service file and then applies possible further restrictions from the drop-in configs (steps 2 and 8 in the diagram). Limits configured in /opt/open-xchange/etc/ox-scriptconf.sh aren't considered anylonger when using systemd while the older System V style init has to source /opt/open-xchange/etc/ox-scriptconf.sh to be able to read the configs for the maximum number of processes (NPROC) and files(NRFILES) that the middleware shall be allowed to create/open (steps 3,4 and 15 in the diagram).<br />
<br />
=== Background Infos ===<br />
<br />
Several ways exist to restrict resources on a linux system from a global level down to user/groups or even shells and the processes started by them.<br />
<br />
==== Sysctl ====<br />
Sysctl is used to modify kernel parameters at runtime. E.g. to set the maximum number of files<br />
<br />
$ sysctl -w fs.file-max=100000<br />
<br />
To permanently set them append to the main configuration file and reload the settings<br />
<br />
$ echo fs.file-max=100000 >> /etc/sysctl.conf<br />
$ sysctl -p<br />
<br />
More infos can be found via '''man sysctl'''<br />
<br />
==== Limits.conf ====<br />
<br />
Allows to restrict resources an a global, group or user level. E.g:<br />
<br />
$ cat /etc/security/limits.d/90-nproc.conf <br />
# Default limit for number of user's processes to prevent<br />
# accidental fork bombs.<br />
# See rhbz #432903 for reasoning.<br />
<br />
* soft nproc 1024<br />
<br />
From '''man limits.conf''':<br />
<blockquote><br />
''Also, please note that all limit settings are set per login. They are not global, nor are they permanent; existing only for the duration of the session.''<br />
</blockquote><br />
<br />
The limits per login are applied via the ''pam'' stack. See '''man pam''' and '''man pam_limits''' for more details. As those limits are bound to sessions they don't affect most daemons started by our supported init systems or init utils. Most state that they are ignored by design, see [https://bugs.launchpad.net/ubuntu/+source/upstart/+bug/938669/comments/4 upstart], [https://bugzilla.redhat.com/show_bug.cgi?id=754285#c1 systemd] and [https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=302079 start-stop-daemon] <br />
<br />
===== Ulimit =====<br />
From '''man bash'''<br />
<blockquote><br />
ulimit [-HSTabcdefilmnpqrstuvx [limit]]<br />
Provides control over the resources available to the shell and to processes started by it, on systems that allow such control.<br />
</blockquote><br />
This is what we use in our System V compatible init scripts to increase resources for the open-xchange process across multiple distros. Currently only ''the maximum number of processes'' and ''the maximum number of open file descriptors available to a single user'' are increased via ulimit. The values are specified in '''/opt/open-xchange/ox-scriptconf.sh'''<br />
<br />
==== Systemd ====<br />
<br />
===== Control Groups=====<br />
<br />
Control groups should only affect the OX middleware if you create/manage them yourself of if you are using a ''modern'' distribution that already uses systemd as init.<br />
<br />
Citing from the kernel cgroup [https://www.kernel.org/doc/Documentation/cgroup-v2.txt documentation]:<br />
<br />
<blockquote><br />
1-2. What is cgroup?<br />
<br />
cgroup is a mechanism to organize processes hierarchically and<br />
distribute system resources along the hierarchy in a controlled and<br />
configurable manner.<br />
<br />
cgroup is largely composed of two parts - the core and controllers.<br />
cgroup core is primarily responsible for hierarchically organizing<br />
processes. A cgroup controller is usually responsible for<br />
distributing a specific type of system resource along the hierarchy<br />
although there are utility controllers which serve purposes other than<br />
resource distribution.<br />
<br />
cgroups form a tree structure and every process in the system belongs<br />
to one and only one cgroup. All threads of a process belong to the<br />
same cgroup. On creation, all processes are put in the cgroup that<br />
the parent process belongs to at the time. A process can be migrated<br />
to another cgroup. Migration of a process doesn't affect already<br />
existing descendant processes.<br />
<br />
Following certain structural constraints, controllers may be enabled or<br />
disabled selectively on a cgroup. All controller behaviors are<br />
hierarchical - if a controller is enabled on a cgroup, it affects all<br />
processes which belong to the cgroups consisting the inclusive<br />
sub-hierarchy of the cgroup. When a controller is enabled on a nested<br />
cgroup, it always restricts the resource distribution further. The<br />
restrictions set closer to the root in the hierarchy can not be<br />
overridden from further away.<br />
</blockquote><br />
<br />
So processes are organized into a tree structure of control groups and controllers are responsible for the distribution of resources. So what kind of controllers exist?<br />
<br />
<blockquote><br />
5. Controllers<br />
<br />
5-1. CPU<br />
<br />
The "cpu" controllers regulates distribution of CPU cycles. This<br />
controller implements weight and absolute bandwidth limit models for<br />
normal scheduling policy and absolute bandwidth allocation model for<br />
realtime scheduling policy.<br />
<br />
5-2. Memory<br />
<br />
The "memory" controller regulates distribution of memory.<br />
...<br />
While not completely water-tight, all major memory usages by a given<br />
cgroup are tracked so that the total memory consumption can be<br />
accounted and controlled to a reasonable extent.<br />
<br />
5-3. IO<br />
<br />
The "io" controller regulates the distribution of IO resources. This<br />
controller implements both weight based and absolute bandwidth or IOPS<br />
limit distribution; however, weight based distribution is available<br />
only if cfq-iosched is in use and neither scheme is available for<br />
blk-mq devices.<br />
</blockquote><br />
<br />
The open-xchange service is simply put into the default ''system.slice'' without applying further limits.<br />
<br />
singlenode$ systemd-cgls<br />
├─1 /sbin/init<br />
├─system.slice<br />
│ ├─avahi-daemon.service<br />
│ │ ├─501 avahi-daemon: running [singlenode]<br />
│ │ └─514 avahi-daemon: chroot helper<br />
│ ├─console-kit-daemon.service<br />
│ │ └─16164 /usr/sbin/console-kit-daemon --no-daemon<br />
│ ├─dbus.service<br />
│ │ └─508 /usr/bin/dbus-daemon --system --address=systemd: --nofork --nopidfile --systemd-activation<br />
│ ├─munin-node.service<br />
│ │ └─4290 /usr/bin/perl -wT /usr/sbin/munin-node<br />
│ ├─open-xchange.service<br />
│ │ └─6037 /usr/bin/java -Dsun.net.inetaddr.ttl=3600 -Dnetworkaddress.cache.ttl=3600 -Dnetworkaddress.cache.negative.ttl=10 ...<br />
<br />
To check all the details use<br />
<br />
singlenode:~ # systemctl show system.slice<br />
<br />
==== Limits besides control groups ====<br />
Besides control groups systemd allows you to apply other limits to the execution environment of your service. Here we can apply the limits that would normally be applied via ''limits.conf'' or ''ulimit''. Systemd uses [http://man7.org/linux/man-pages/man2/setrlimit.2.html setrlimit] for this. The options that we set by default are:<br />
<br />
* LimitNOFILE<br />
* LimitNPROC<br />
<br />
which you have already seen when we took a look at the [[#Service_definiton|service definition]] and [[#Drop-in_config|drop-in configs]]<br />
<br />
<br />
== Verify limits ==<br />
<br />
=== System V ===<br />
<br />
Get the proper pid of the java process started for the middleware by either checking '''ps faux'''<br />
root 4103 0.0 0.0 66528 1868 pts/0 S 22:05 0:00 su -s /bin/bash open-xchange -c /opt/open-xchange/sbin/open-xchange<br />
494 4109 9.5 19.8 2224644 381312 ? Ssl 22:05 1:12 \_ /usr/bin/java -Dsun.net.inetaddr.ttl=3600 -Dnetworkaddress.cache.ttl=3600<br />
<br />
or programmatically<br />
singlenode:~ # pid=$(pgrep -P $(</var/run/open-xchange.pid) java)<br />
<br />
and check the limits applied for that process<br />
singlenode:~ # cat /proc/$pid/limits<br />
Limit Soft Limit Hard Limit Units<br />
Max cpu time unlimited unlimited seconds<br />
Max file size unlimited unlimited bytes<br />
Max data size unlimited unlimited bytes<br />
Max stack size 8388608 unlimited bytes<br />
Max core file size 0 unlimited bytes<br />
Max resident set unlimited unlimited bytes<br />
Max processes 65536 65536 processes<br />
Max open files 65536 65536 files<br />
Max locked memory 65536 65536 bytes<br />
Max address space unlimited unlimited bytes<br />
Max file locks unlimited unlimited locks<br />
Max pending signals 24254 24254 signals<br />
Max msgqueue size 819200 819200 bytes<br />
Max nice priority 0 0<br />
Max realtime priority 0 0<br />
Max realtime timeout unlimited unlimited us<br />
<br />
=== Systemd ===<br />
singlenode:~ # systemctl show open-xchange | grep Limit<br />
StartLimitInterval=10000000<br />
StartLimitBurst=5<br />
StartLimitAction=none<br />
MemoryLimit=18446744073709551615<br />
LimitCPU=18446744073709551615<br />
LimitFSIZE=18446744073709551615<br />
LimitDATA=18446744073709551615<br />
LimitSTACK=18446744073709551615<br />
LimitCORE=18446744073709551615<br />
LimitRSS=18446744073709551615<br />
LimitNOFILE=65536<br />
LimitAS=18446744073709551615<br />
LimitNPROC=65536<br />
LimitMEMLOCK=65536<br />
LimitLOCKS=18446744073709551615<br />
LimitSIGPENDING=19827<br />
LimitMSGQUEUE=819200<br />
LimitNICE=0<br />
LimitRTPRIO=0<br />
LimitRTTIME=18446744073709551615<br />
<br />
<br />
== Open-Xchange middleware on specific distros ==<br />
The support for the mentioned mechanism of resource control differ depending on the distribution and the init system in use.<br />
<br />
=== Debian 7 ===<br />
;Init<br />
: System V style<br />
;OX Configurable Limits/Defaults<br />
: nofile, nproc<br />
<br />
The mentioned limits can be configured via '''/opt/open-xchange/etc/ox-scriptconf.sh'''. The limits are applied via ulimit in the service's init script. The open-xchange service is finally started via start-stop-daemon which doesn't doesn't consider '''/etc/security/limits.*'''<br />
<br />
=== RHEL 6 / CentOS 6 ===<br />
;Init<br />
: Upstart, System V compatible<br />
;OX Configurable Limits/Defaults<br />
: nofile, nproc<br />
<br />
The mentioned limits can be configured via '''/opt/open-xchange/etc/ox-scriptconf.sh'''. The limits are applied via ulimit in the service's init script. Furthermore as the open-xchange service is finally started via '''su ... open-xchange''' on this distro a user session is opened via su/pam and the default CentOS pam config reads the '''/etc/security/limits.*''' configuration by loading the pam stack like:<br />
<br />
: '''/etc/pam.d/su'''<br />
:: '''-> /etc/pam.d/system-auth'''<br />
::: '''-> pam_limits.so'''<br />
<br />
If NPROC isn't configured for the open-xchange-server it's restricted to '''1024''' globally by default to prevent accidental fork bombs, see '''/etc/security/limits.d/90-nproc.conf''' which can result in severe problems modern multithreaded applications.<br />
<br />
=== RHEL 7 / CentOS 7 / Debian 8 / SLE 12 ===<br />
;Init<br />
: Systemd<br />
;OX Configurable Limits/Defaults<br />
: nofile, nproc<br />
<br />
For systemd the default limits are configured directly in the service's unit file that is shipped by OX and located at '''/usr/lib/systemd/system/open-xchange.service'''. The drop-in config to override or extend the default unit file is located at '''/etc/systemd/system/open-xchange.service.d/limits.conf'''. [https://www.freedesktop.org/software/systemd/man/systemd.exec.html Systemd.exec] shows a whole lot of options that can be used by admins to adapt the default service to their specific needs.<br />
<br />
== Evaluate JVM commandline options ==<br />
<br />
Before starting Java and the OSGi framework with all the Open-Xchange middleware bundles we have to evaluate several important JVM commandline options that influence the performance and stability of the Open-Xchange middleware stack. These commandline options are specified in '''/opt/open-xchange/etc/ox-scriptconf.sh'''<br />
<br />
=== Format < 7.8.4 ===<br />
For version up to 7.8.3 the JVM configuration options were all specified via a single property '''JAVA_XTRAOPTS''' which resulted in a rather long configuration string that was hard to read and maintain.<br />
<br />
JAVA_XTRAOPTS="-Dsun.net.inetaddr.ttl=3600 -Dnetworkaddress.cache.ttl=3600 -Dnetworkaddress.cache.negative.ttl=10 \<br />
-Dlogback.threadlocal.put.duplicate=false -server -Djava.awt.headless=true -Xmx512M -XX:MaxPermSize=256m -XX:+UseConcMarkSweepGC \<br />
-XX:+UseParNewGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:NewRatio=3 -XX:+UseTLAB \<br />
-Dosgi.compatibility.bootdelegation=false -XX:-OmitStackTraceInFastThrow"<br />
<br />
During the startup of the open-xchange service (step 11 of diagram Service startup using systemd and step 10 of diagram Service startup using upstart.) this JAVA_XTRAOPTS string goes through a basic validity check and is then passed to the JVM. <br />
<br />
=== Format >= 7.8.4 ===<br />
Beginning from version 7.8.4 the options are split up and sorted into different categories like:<br />
<br />
JAVA_OPTS_GC="-XX:+UseConcMarkSweepGC -XX:+UseParNewGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:NewRatio=3 -XX:+DisableExplicitGC"<br />
JAVA_OPTS_LOG="-Dlogback.threadlocal.put.duplicate=false -XX:-OmitStackTraceInFastThrow"<br />
JAVA_OPTS_MEM="-XX:MaxHeapSize=512M -XX:MaxPermSize=256M -XX:+UseTLAB"<br />
JAVA_OPTS_NET="-Dsun.net.inetaddr.ttl=3600 -Dnetworkaddress.cache.ttl=3600 -Dnetworkaddress.cache.negative.ttl=10"<br />
JAVA_OPTS_OSGI="-Dosgi.compatibility.bootdelegation=false"<br />
JAVA_OPTS_SERVER="-server -Djava.awt.headless=true"<br />
<br />
JAVA_OPTS_OTHER=""<br />
<br />
#JAVA_OPTS_DEBUG="-XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/mnt/heapdump -Xloggc:/var/log/open-xchange/gc.log -verbose:gc -XX:+PrintGCDateStamps \<br />
-XX:+PrintHeapAtGC -XX:+PrintGCApplicationStoppedTime -XX:+PrintTenuringDistribution"<br />
<br />
which is much easier to read an maintain. Furthermore we added a useful '''JAVA_OPTS_DEBUG''' configuration parameter that comes preconfigured but unused/commented so that customers can quickly enable the most commonly used debugging options in case they have to provide detailed system information to Open-Xchange.<br />
<br />
During the startup of the open-xchange service (step 11 of diagram ''Service startup using systemd'' and step 10 of diagram ''Service startup using upstart''.) all uncommented JAVA_OPTS_* categories will then be evaluated and combined again into a single string of commandline options that can then be passed to the JVM.<br />
<br />
=== Upgrade procedure ===<br />
During the upgrade '''7.8.3-revX -> 7.8.4-revY''' existing JAVA_XTRAOPTS will automatically be split up and migrated to the new categories approach. A backup of the file (ox-scriptconf.sh.timestamp) will be created and admins are urged to check for proper migration before deleting the backup file like it's already done with *.dpkg-dist or *.rpmnew files on Debian or RHEL/SLES distributions.<br />
<br />
== Encoding detection during JVM startup ==<br />
<br />
=== Background Infos ===<br />
All of en_US.UTF-8, .utf8, .utf-8, .UTF_8 will result in a valid UTF-8 charmap<br />
<br />
==== JVM startup ====<br />
<br />
The default file.encoding detection is done at JVM startup based upon LC_CTYPE which makes sense as it's documented as:<br />
<br />
CODESET (LC_CTYPE)<br />
Return a string with the name of the character encoding used in the selected locale, such as "UTF-8", "ISO-8859-1",<br />
or "ANSI_X3.4-1968" (better known as US-ASCII). This is the same string that you get with "locale charmap".<br />
For a list of character encoding names, try "locale -m", cf. locale(1).<br />
<br />
Furthermore this influences the defaultCharset selection of the JVM.<br />
<br />
/**<br />
* Returns the default charset of this Java virtual machine.<br />
*<br />
* The default charset is determined during virtual-machine startup and<br />
* typically depends upon the locale and charset of the underlying<br />
* operating system.<br />
*<br />
* @return A charset object for the default charset<br />
*<br />
* @since 1.5<br />
*/<br />
public static Charset defaultCharset() {<br />
if (defaultCharset == null) {<br />
synchronized (Charset.class) {<br />
String csn = AccessController.doPrivileged(<br />
new GetPropertyAction("file.encoding"));<br />
Charset cs = lookup(csn);<br />
if (cs != null)<br />
defaultCharset = cs;<br />
else<br />
defaultCharset = forName("UTF-8");<br />
}<br />
}<br />
return defaultCharset;<br />
}<br />
<br />
Although we try to specify valid unicode encodings for every IO operations the Open-Xchange middleware does some parts of the JVM don't allow to specify the encoding to use and falls back to the detected default encoding. That's why admins have to make sure to provide a proper unicode environment for the middleware service during every start or use documented workarounds provided by Open-Xchange (e.g. )<br />
<br />
=== Query which file encoding was detected by the JVM during startup ===<br />
<br />
/opt/open-xchange/sbin/showruntimestats -r | sed -ne 's/.*\({key=file.encoding, value=[^}]*}\).*/\1/p'<br />
{key=file.encoding, value=UTF-8}<br />
<br />
=== Locale passing from client to server via SSH ===<br />
<br />
The following setting of sshd allows clients to specify which locale to use on the server and might lead to some confusion or unexpected behaviour when checking the system locale or even to the usage of broken locales during middleware startup if your client sends misconfigured or missing locale configuration to the server.<br />
<br />
grep AcceptEnv /etc/ssh/sshd_config <br />
AcceptEnv LANG LC_*<br />
<br />
For example when i connect to the machine my local german locale is sent to the server <br />
<br />
[vagrant@showcase-community-centos-7 ~]$ locale -v <br />
LANG=de_DE.UTF-8<br />
LC_CTYPE="de_DE.UTF-8"<br />
LC_NUMERIC=de_DE.UTF-8<br />
LC_TIME=de_DE.UTF-8<br />
LC_COLLATE="de_DE.UTF-8"<br />
LC_MONETARY=de_DE.UTF-8<br />
LC_MESSAGES="de_DE.UTF-8"<br />
LC_PAPER=de_DE.UTF-8<br />
LC_NAME=de_DE.UTF-8<br />
LC_ADDRESS=de_DE.UTF-8<br />
LC_TELEPHONE=de_DE.UTF-8<br />
LC_MEASUREMENT=de_DE.UTF-8<br />
LC_IDENTIFICATION=de_DE.UTF-8<br />
LC_ALL=<br />
<br />
But after specifying that i want a new login shell the default locale configured for that user/system will be used instead!<br />
<br />
[root@showcase-community-centos-7 vagrant]# su -s /bin/bash open-xchange -l<br />
-bash-4.2$ locale -v <br />
LANG=en_US.UTF-8<br />
LC_CTYPE="en_US.UTF-8"<br />
LC_NUMERIC="en_US.UTF-8"<br />
LC_TIME="en_US.UTF-8"<br />
LC_COLLATE="en_US.UTF-8"<br />
LC_MONETARY="en_US.UTF-8"<br />
LC_MESSAGES="en_US.UTF-8"<br />
LC_PAPER="en_US.UTF-8"<br />
LC_NAME="en_US.UTF-8"<br />
LC_ADDRESS="en_US.UTF-8"<br />
LC_TELEPHONE="en_US.UTF-8"<br />
LC_MEASUREMENT="en_US.UTF-8"<br />
LC_IDENTIFICATION="en_US.UTF-8"<br />
LC_ALL=<br />
<br />
Add details from https://bugs.open-xchange.com/show_bug.cgi?id=51074<br />
<br />
== Warnings during early ==<br />
<br />
Early warnings before the start of the JVM and related logging libraries like logback and logstash forwarding are redirected to /var/log/open-xchange/open-xchange-console.log.<br />
<br />
=== Limits ===<br />
If the open-xchange startup script isn't able to set the proper limits on CentOS 6 due to e.g. hard limits being enforced via pam_limits you'll see the following warning in the console log during startup<br />
<br />
singlenode open-xchange # cat open-xchange-console.log<br />
/opt/open-xchange/sbin/open-xchange: line 115: ulimit: max user processes: cannot modify limit: Operation not permitted<br />
<br />
=== Locale ===<br />
If the open-xchange startup script isn't able to detect a unicode locale you'll see a warning like the following (the current charmap might differ)<br />
<br />
WARNING: Couldn't detect proper unicode charmap in locale category LC_CTYPE.<br />
WARNING: Current charmap: "ANSI_X3.4-1968" might cause encoding problems.</div>Marenshttps://oxpedia.org/wiki/index.php?title=Sandbox:Middleware_Startup/sandbox&diff=23218Sandbox:Middleware Startup/sandbox2017-04-24T17:58:11Z<p>Marens: /* Encoding detection during JVM startup */</p>
<hr />
<div><br />
== Systemd ==<br />
<br />
==== Service definition ====<br />
The default service definition as shipped by Open-Xchange is rather concise and easy to read. You can check by looking at the service file that is installed to either /usr/lib/systemd or /lib/systemd depending on your distribution of choice.<br />
<br />
singlenode:~ # cat /usr/lib/systemd/system/open-xchange.service <br />
[Unit]<br />
After=remote-fs.target<br />
After=time-sync.target ypbind.service sendmail.service cyrus.service <br />
<br />
[Service]<br />
User=open-xchange<br />
PermissionsStartOnly=true<br />
TimeoutStartSec=0<br />
ExecStartPre=/opt/open-xchange/sbin/triggerupdatethemes -u<br />
ExecStart=/opt/open-xchange/sbin/open-xchange<br />
ExecStop=/opt/open-xchange/sbin/shutdown -w<br />
ExecReload=/opt/open-xchange/sbin/triggerreloadconfiguration -d<br />
KillMode=process<br />
LimitNOFILE=65536<br />
LimitNPROC=65536<br />
<br />
[Install]<br />
WantedBy=multi-user.target<br />
<br />
==== Drop-in configs ====<br />
Drop in configs allow administrators to easily adapt and override default service unit files. So if you want to change the default limits or add additional limits have a look at our default example drop-in config which is located at /etc/systemd/system/open-xchange.service.d/limits.conf<br />
<br />
singlenode:~ # cat /etc/systemd/system/open-xchange.service.d/limits.conf <br />
# Override and add options in this file<br />
# See systemd.exec(5) for other limits<br />
<br />
[Service]<br />
#LimitNPROC=65536<br />
#LimitNOFILE=65536<br />
<br />
The systemd.exec documentation shows a whole lot of options that can be used by admins to adapt the default service to their specific needs.<br />
<br />
<br clear=all><br />
[[File:mw_start_sysd.png|thumb|Service startup using systemd|left|1000px]]<br />
<br clear=all><br />
<br />
== System V / Upstart ==<br />
<br />
==== Service definition ====<br />
<br />
The default service definition as shipped by Open-Xchange can be found at /etc/init.d/open-xchange and will differ slightly depending on the distro in use.<br />
<br />
<br clear=all><br />
[[File:mw_start_sysv.png|thumb|Service startup using upstart|left|1000px]]<br />
<br clear=all><br />
== Apply Resource limits ==<br />
<br />
As you can see in the two diagrams above one of the differences is that systemd reads all the infos wrt. resource limitations from the original service file and then applies possible further restrictions from the drop-in configs (steps 2 and 8 in the diagram). Limits configured in /opt/open-xchange/etc/ox-scriptconf.sh aren't considered anylonger when using systemd while the older System V style init has to source /opt/open-xchange/etc/ox-scriptconf.sh to be able to read the configs for the maximum number of processes (NPROC) and files(NRFILES) that the middleware shall be allowed to create/open (steps 3,4 and 15 in the diagram).<br />
<br />
=== Background Infos ===<br />
<br />
Several ways exist to restrict resources on a linux system from a global level down to user/groups or even shells and the processes started by them.<br />
<br />
==== Sysctl ====<br />
Sysctl is used to modify kernel parameters at runtime. E.g. to set the maximum number of files<br />
<br />
$ sysctl -w fs.file-max=100000<br />
<br />
To permanently set them append to the main configuration file and reload the settings<br />
<br />
$ echo fs.file-max=100000 >> /etc/sysctl.conf<br />
$ sysctl -p<br />
<br />
More infos can be found via '''man sysctl'''<br />
<br />
==== Limits.conf ====<br />
<br />
Allows to restrict resources an a global, group or user level. E.g:<br />
<br />
$ cat /etc/security/limits.d/90-nproc.conf <br />
# Default limit for number of user's processes to prevent<br />
# accidental fork bombs.<br />
# See rhbz #432903 for reasoning.<br />
<br />
* soft nproc 1024<br />
<br />
From '''man limits.conf''':<br />
<blockquote><br />
''Also, please note that all limit settings are set per login. They are not global, nor are they permanent; existing only for the duration of the session.''<br />
</blockquote><br />
<br />
The limits per login are applied via the ''pam'' stack. See '''man pam''' and '''man pam_limits''' for more details. As those limits are bound to sessions they don't affect most daemons started by our supported init systems or init utils. Most state that they are ignored by design, see [https://bugs.launchpad.net/ubuntu/+source/upstart/+bug/938669/comments/4 upstart], [https://bugzilla.redhat.com/show_bug.cgi?id=754285#c1 systemd] and [https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=302079 start-stop-daemon] <br />
<br />
===== Ulimit =====<br />
From '''man bash'''<br />
<blockquote><br />
ulimit [-HSTabcdefilmnpqrstuvx [limit]]<br />
Provides control over the resources available to the shell and to processes started by it, on systems that allow such control.<br />
</blockquote><br />
This is what we use in our System V compatible init scripts to increase resources for the open-xchange process across multiple distros. Currently only ''the maximum number of processes'' and ''the maximum number of open file descriptors available to a single user'' are increased via ulimit. The values are specified in '''/opt/open-xchange/ox-scriptconf.sh'''<br />
<br />
==== Systemd ====<br />
<br />
===== Control Groups=====<br />
<br />
Control groups should only affect the OX middleware if you create/manage them yourself of if you are using a ''modern'' distribution that already uses systemd as init.<br />
<br />
Citing from the kernel cgroup [https://www.kernel.org/doc/Documentation/cgroup-v2.txt documentation]:<br />
<br />
<blockquote><br />
1-2. What is cgroup?<br />
<br />
cgroup is a mechanism to organize processes hierarchically and<br />
distribute system resources along the hierarchy in a controlled and<br />
configurable manner.<br />
<br />
cgroup is largely composed of two parts - the core and controllers.<br />
cgroup core is primarily responsible for hierarchically organizing<br />
processes. A cgroup controller is usually responsible for<br />
distributing a specific type of system resource along the hierarchy<br />
although there are utility controllers which serve purposes other than<br />
resource distribution.<br />
<br />
cgroups form a tree structure and every process in the system belongs<br />
to one and only one cgroup. All threads of a process belong to the<br />
same cgroup. On creation, all processes are put in the cgroup that<br />
the parent process belongs to at the time. A process can be migrated<br />
to another cgroup. Migration of a process doesn't affect already<br />
existing descendant processes.<br />
<br />
Following certain structural constraints, controllers may be enabled or<br />
disabled selectively on a cgroup. All controller behaviors are<br />
hierarchical - if a controller is enabled on a cgroup, it affects all<br />
processes which belong to the cgroups consisting the inclusive<br />
sub-hierarchy of the cgroup. When a controller is enabled on a nested<br />
cgroup, it always restricts the resource distribution further. The<br />
restrictions set closer to the root in the hierarchy can not be<br />
overridden from further away.<br />
</blockquote><br />
<br />
So processes are organized into a tree structure of control groups and controllers are responsible for the distribution of resources. So what kind of controllers exist?<br />
<br />
<blockquote><br />
5. Controllers<br />
<br />
5-1. CPU<br />
<br />
The "cpu" controllers regulates distribution of CPU cycles. This<br />
controller implements weight and absolute bandwidth limit models for<br />
normal scheduling policy and absolute bandwidth allocation model for<br />
realtime scheduling policy.<br />
<br />
5-2. Memory<br />
<br />
The "memory" controller regulates distribution of memory.<br />
...<br />
While not completely water-tight, all major memory usages by a given<br />
cgroup are tracked so that the total memory consumption can be<br />
accounted and controlled to a reasonable extent.<br />
<br />
5-3. IO<br />
<br />
The "io" controller regulates the distribution of IO resources. This<br />
controller implements both weight based and absolute bandwidth or IOPS<br />
limit distribution; however, weight based distribution is available<br />
only if cfq-iosched is in use and neither scheme is available for<br />
blk-mq devices.<br />
</blockquote><br />
<br />
The open-xchange service is simply put into the default ''system.slice'' without applying further limits.<br />
<br />
singlenode$ systemd-cgls<br />
├─1 /sbin/init<br />
├─system.slice<br />
│ ├─avahi-daemon.service<br />
│ │ ├─501 avahi-daemon: running [singlenode]<br />
│ │ └─514 avahi-daemon: chroot helper<br />
│ ├─console-kit-daemon.service<br />
│ │ └─16164 /usr/sbin/console-kit-daemon --no-daemon<br />
│ ├─dbus.service<br />
│ │ └─508 /usr/bin/dbus-daemon --system --address=systemd: --nofork --nopidfile --systemd-activation<br />
│ ├─munin-node.service<br />
│ │ └─4290 /usr/bin/perl -wT /usr/sbin/munin-node<br />
│ ├─open-xchange.service<br />
│ │ └─6037 /usr/bin/java -Dsun.net.inetaddr.ttl=3600 -Dnetworkaddress.cache.ttl=3600 -Dnetworkaddress.cache.negative.ttl=10 ...<br />
<br />
To check all the details use<br />
<br />
singlenode:~ # systemctl show system.slice<br />
<br />
==== Limits besides control groups ====<br />
Besides control groups systemd allows you to apply other limits to the execution environment of your service. Here we can apply the limits that would normally be applied via ''limits.conf'' or ''ulimit''. Systemd uses [http://man7.org/linux/man-pages/man2/setrlimit.2.html setrlimit] for this. The options that we set by default are:<br />
<br />
* LimitNOFILE<br />
* LimitNPROC<br />
<br />
which you have already seen when we took a look at the [[#Service_definiton|service definition]] and [[#Drop-in_config|drop-in configs]]<br />
<br />
<br />
== Verify limits ==<br />
<br />
=== System V ===<br />
<br />
Get the proper pid of the java process started for the middleware by either checking '''ps faux'''<br />
root 4103 0.0 0.0 66528 1868 pts/0 S 22:05 0:00 su -s /bin/bash open-xchange -c /opt/open-xchange/sbin/open-xchange<br />
494 4109 9.5 19.8 2224644 381312 ? Ssl 22:05 1:12 \_ /usr/bin/java -Dsun.net.inetaddr.ttl=3600 -Dnetworkaddress.cache.ttl=3600<br />
<br />
or programmatically<br />
singlenode:~ # pid=$(pgrep -P $(</var/run/open-xchange.pid) java)<br />
<br />
and check the limits applied for that process<br />
singlenode:~ # cat /proc/$pid/limits<br />
Limit Soft Limit Hard Limit Units<br />
Max cpu time unlimited unlimited seconds<br />
Max file size unlimited unlimited bytes<br />
Max data size unlimited unlimited bytes<br />
Max stack size 8388608 unlimited bytes<br />
Max core file size 0 unlimited bytes<br />
Max resident set unlimited unlimited bytes<br />
Max processes 65536 65536 processes<br />
Max open files 65536 65536 files<br />
Max locked memory 65536 65536 bytes<br />
Max address space unlimited unlimited bytes<br />
Max file locks unlimited unlimited locks<br />
Max pending signals 24254 24254 signals<br />
Max msgqueue size 819200 819200 bytes<br />
Max nice priority 0 0<br />
Max realtime priority 0 0<br />
Max realtime timeout unlimited unlimited us<br />
<br />
=== Systemd ===<br />
singlenode:~ # systemctl show open-xchange | grep Limit<br />
StartLimitInterval=10000000<br />
StartLimitBurst=5<br />
StartLimitAction=none<br />
MemoryLimit=18446744073709551615<br />
LimitCPU=18446744073709551615<br />
LimitFSIZE=18446744073709551615<br />
LimitDATA=18446744073709551615<br />
LimitSTACK=18446744073709551615<br />
LimitCORE=18446744073709551615<br />
LimitRSS=18446744073709551615<br />
LimitNOFILE=65536<br />
LimitAS=18446744073709551615<br />
LimitNPROC=65536<br />
LimitMEMLOCK=65536<br />
LimitLOCKS=18446744073709551615<br />
LimitSIGPENDING=19827<br />
LimitMSGQUEUE=819200<br />
LimitNICE=0<br />
LimitRTPRIO=0<br />
LimitRTTIME=18446744073709551615<br />
<br />
<br />
== Open-Xchange middleware on specific distros ==<br />
The support for the mentioned mechanism of resource control differ depending on the distribution and the init system in use.<br />
<br />
=== Debian 7 ===<br />
;Init<br />
: System V style<br />
;OX Configurable Limits/Defaults<br />
: nofile, nproc<br />
<br />
The mentioned limits can be configured via '''/opt/open-xchange/etc/ox-scriptconf.sh'''. The limits are applied via ulimit in the service's init script. The open-xchange service is finally started via start-stop-daemon which doesn't doesn't consider '''/etc/security/limits.*'''<br />
<br />
=== RHEL 6 / CentOS 6 ===<br />
;Init<br />
: Upstart, System V compatible<br />
;OX Configurable Limits/Defaults<br />
: nofile, nproc<br />
<br />
The mentioned limits can be configured via '''/opt/open-xchange/etc/ox-scriptconf.sh'''. The limits are applied via ulimit in the service's init script. Furthermore as the open-xchange service is finally started via '''su ... open-xchange''' on this distro a user session is opened via su/pam and the default CentOS pam config reads the '''/etc/security/limits.*''' configuration by loading the pam stack like:<br />
<br />
: '''/etc/pam.d/su'''<br />
:: '''-> /etc/pam.d/system-auth'''<br />
::: '''-> pam_limits.so'''<br />
<br />
If NPROC isn't configured for the open-xchange-server it's restricted to '''1024''' globally by default to prevent accidental fork bombs, see '''/etc/security/limits.d/90-nproc.conf''' which can result in severe problems modern multithreaded applications.<br />
<br />
=== RHEL 7 / CentOS 7 / Debian 8 / SLE 12 ===<br />
;Init<br />
: Systemd<br />
;OX Configurable Limits/Defaults<br />
: nofile, nproc<br />
<br />
For systemd the default limits are configured directly in the service's unit file that is shipped by OX and located at '''/usr/lib/systemd/system/open-xchange.service'''. The drop-in config to override or extend the default unit file is located at '''/etc/systemd/system/open-xchange.service.d/limits.conf'''. [https://www.freedesktop.org/software/systemd/man/systemd.exec.html Systemd.exec] shows a whole lot of options that can be used by admins to adapt the default service to their specific needs.<br />
<br />
== Evaluate JVM commandline options ==<br />
<br />
Before starting Java and the OSGi framework with all the Open-Xchange middleware bundles we have to evaluate several important JVM commandline options that influence the performance and stability of the Open-Xchange middleware stack. These commandline options are specified in '''/opt/open-xchange/etc/ox-scriptconf.sh'''<br />
<br />
=== Format < 7.8.4 ===<br />
For version up to 7.8.3 the JVM configuration options were all specified via a single property '''JAVA_XTRAOPTS''' which resulted in a rather long configuration string that was hard to read and maintain.<br />
<br />
JAVA_XTRAOPTS="-Dsun.net.inetaddr.ttl=3600 -Dnetworkaddress.cache.ttl=3600 -Dnetworkaddress.cache.negative.ttl=10 \<br />
-Dlogback.threadlocal.put.duplicate=false -server -Djava.awt.headless=true -Xmx512M -XX:MaxPermSize=256m -XX:+UseConcMarkSweepGC \<br />
-XX:+UseParNewGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:NewRatio=3 -XX:+UseTLAB \<br />
-Dosgi.compatibility.bootdelegation=false -XX:-OmitStackTraceInFastThrow"<br />
<br />
During the startup of the open-xchange service (step 11 of diagram Service startup using systemd and step 10 of diagram Service startup using upstart.) this JAVA_XTRAOPTS string goes through a basic validity check and is then passed to the JVM. <br />
<br />
=== Format >= 7.8.4 ===<br />
Beginning from version 7.8.4 the options are split up and sorted into different categories like:<br />
<br />
JAVA_OPTS_GC="-XX:+UseConcMarkSweepGC -XX:+UseParNewGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:NewRatio=3 -XX:+DisableExplicitGC"<br />
JAVA_OPTS_LOG="-Dlogback.threadlocal.put.duplicate=false -XX:-OmitStackTraceInFastThrow"<br />
JAVA_OPTS_MEM="-XX:MaxHeapSize=512M -XX:MaxPermSize=256M -XX:+UseTLAB"<br />
JAVA_OPTS_NET="-Dsun.net.inetaddr.ttl=3600 -Dnetworkaddress.cache.ttl=3600 -Dnetworkaddress.cache.negative.ttl=10"<br />
JAVA_OPTS_OSGI="-Dosgi.compatibility.bootdelegation=false"<br />
JAVA_OPTS_SERVER="-server -Djava.awt.headless=true"<br />
<br />
JAVA_OPTS_OTHER=""<br />
<br />
#JAVA_OPTS_DEBUG="-XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/mnt/heapdump -Xloggc:/var/log/open-xchange/gc.log -verbose:gc -XX:+PrintGCDateStamps \<br />
-XX:+PrintHeapAtGC -XX:+PrintGCApplicationStoppedTime -XX:+PrintTenuringDistribution"<br />
<br />
which is much easier to read an maintain. Furthermore we added a useful '''JAVA_OPTS_DEBUG''' configuration parameter that comes preconfigured but unused/commented so that customers can quickly enable the most commonly used debugging options in case they have to provide detailed system information to Open-Xchange.<br />
<br />
During the startup of the open-xchange service (step 11 of diagram ''Service startup using systemd'' and step 10 of diagram ''Service startup using upstart''.) all uncommented JAVA_OPTS_* categories will then be evaluated and combined again into a single string of commandline options that can then be passed to the JVM.<br />
<br />
=== Upgrade procedure ===<br />
During the upgrade '''7.8.3-revX -> 7.8.4-revY''' existing JAVA_XTRAOPTS will automatically be split up and migrated to the new categories approach. A backup of the file (ox-scriptconf.sh.timestamp) will be created and admins are urged to check for proper migration before deleting the backup file like it's already done with *.dpkg-dist or *.rpmnew files on Debian or RHEL/SLES distributions.<br />
<br />
== Encoding detection during JVM startup ==<br />
<br />
=== Background Infos ===<br />
<br />
=== Query which file encoding was detected by the JVM during startup ===<br />
<br />
/opt/open-xchange/sbin/showruntimestats -r | sed -ne 's/.*\({key=file.encoding, value=[^}]*}\).*/\1/p'<br />
{key=file.encoding, value=UTF-8}<br />
<br />
=== Locale passing from client to server via SSH ===<br />
<br />
The following setting of sshd allows clients to specify which locale to use on the server and might lead to some confusion or unexpected behaviour when checking the system locale or even to the usage of broken locales during middleware startup if your client sends misconfigured or missing locale configuration to the server.<br />
<br />
grep AcceptEnv /etc/ssh/sshd_config <br />
AcceptEnv LANG LC_*<br />
<br />
For example when i connect to the machine my local german locale is sent to the server <br />
<br />
[vagrant@showcase-community-centos-7 ~]$ locale -v <br />
LANG=de_DE.UTF-8<br />
LC_CTYPE="de_DE.UTF-8"<br />
LC_NUMERIC=de_DE.UTF-8<br />
LC_TIME=de_DE.UTF-8<br />
LC_COLLATE="de_DE.UTF-8"<br />
LC_MONETARY=de_DE.UTF-8<br />
LC_MESSAGES="de_DE.UTF-8"<br />
LC_PAPER=de_DE.UTF-8<br />
LC_NAME=de_DE.UTF-8<br />
LC_ADDRESS=de_DE.UTF-8<br />
LC_TELEPHONE=de_DE.UTF-8<br />
LC_MEASUREMENT=de_DE.UTF-8<br />
LC_IDENTIFICATION=de_DE.UTF-8<br />
LC_ALL=<br />
<br />
But after specifying that i want a new login shell the default locale configured for that user/system will be used instead!<br />
<br />
[root@showcase-community-centos-7 vagrant]# su -s /bin/bash open-xchange -l<br />
-bash-4.2$ locale -v <br />
LANG=en_US.UTF-8<br />
LC_CTYPE="en_US.UTF-8"<br />
LC_NUMERIC="en_US.UTF-8"<br />
LC_TIME="en_US.UTF-8"<br />
LC_COLLATE="en_US.UTF-8"<br />
LC_MONETARY="en_US.UTF-8"<br />
LC_MESSAGES="en_US.UTF-8"<br />
LC_PAPER="en_US.UTF-8"<br />
LC_NAME="en_US.UTF-8"<br />
LC_ADDRESS="en_US.UTF-8"<br />
LC_TELEPHONE="en_US.UTF-8"<br />
LC_MEASUREMENT="en_US.UTF-8"<br />
LC_IDENTIFICATION="en_US.UTF-8"<br />
LC_ALL=<br />
<br />
Add details from https://bugs.open-xchange.com/show_bug.cgi?id=51074<br />
<br />
== Warnings during early ==<br />
<br />
Early warnings before the start of the JVM and related logging libraries like logback and logstash forwarding are redirected to /var/log/open-xchange/open-xchange-console.log.<br />
<br />
=== Limits ===<br />
If the open-xchange startup script isn't able to set the proper limits on CentOS 6 due to e.g. hard limits being enforced via pam_limits you'll see the following warning in the console log during startup<br />
<br />
singlenode open-xchange # cat open-xchange-console.log<br />
/opt/open-xchange/sbin/open-xchange: line 115: ulimit: max user processes: cannot modify limit: Operation not permitted<br />
<br />
=== Locale ===<br />
If the open-xchange startup script isn't able to detect a unicode locale you'll see a warning like the following (the current charmap might differ)<br />
<br />
WARNING: Couldn't detect proper unicode charmap in locale category LC_CTYPE.<br />
WARNING: Current charmap: "ANSI_X3.4-1968" might cause encoding problems.</div>Marenshttps://oxpedia.org/wiki/index.php?title=Sandbox:Middleware_Startup/sandbox&diff=23217Sandbox:Middleware Startup/sandbox2017-04-24T17:38:01Z<p>Marens: /* Evaluate JVM commandline options */</p>
<hr />
<div><br />
== Systemd ==<br />
<br />
==== Service definition ====<br />
The default service definition as shipped by Open-Xchange is rather concise and easy to read. You can check by looking at the service file that is installed to either /usr/lib/systemd or /lib/systemd depending on your distribution of choice.<br />
<br />
singlenode:~ # cat /usr/lib/systemd/system/open-xchange.service <br />
[Unit]<br />
After=remote-fs.target<br />
After=time-sync.target ypbind.service sendmail.service cyrus.service <br />
<br />
[Service]<br />
User=open-xchange<br />
PermissionsStartOnly=true<br />
TimeoutStartSec=0<br />
ExecStartPre=/opt/open-xchange/sbin/triggerupdatethemes -u<br />
ExecStart=/opt/open-xchange/sbin/open-xchange<br />
ExecStop=/opt/open-xchange/sbin/shutdown -w<br />
ExecReload=/opt/open-xchange/sbin/triggerreloadconfiguration -d<br />
KillMode=process<br />
LimitNOFILE=65536<br />
LimitNPROC=65536<br />
<br />
[Install]<br />
WantedBy=multi-user.target<br />
<br />
==== Drop-in configs ====<br />
Drop in configs allow administrators to easily adapt and override default service unit files. So if you want to change the default limits or add additional limits have a look at our default example drop-in config which is located at /etc/systemd/system/open-xchange.service.d/limits.conf<br />
<br />
singlenode:~ # cat /etc/systemd/system/open-xchange.service.d/limits.conf <br />
# Override and add options in this file<br />
# See systemd.exec(5) for other limits<br />
<br />
[Service]<br />
#LimitNPROC=65536<br />
#LimitNOFILE=65536<br />
<br />
The systemd.exec documentation shows a whole lot of options that can be used by admins to adapt the default service to their specific needs.<br />
<br />
<br clear=all><br />
[[File:mw_start_sysd.png|thumb|Service startup using systemd|left|1000px]]<br />
<br clear=all><br />
<br />
== System V / Upstart ==<br />
<br />
==== Service definition ====<br />
<br />
The default service definition as shipped by Open-Xchange can be found at /etc/init.d/open-xchange and will differ slightly depending on the distro in use.<br />
<br />
<br clear=all><br />
[[File:mw_start_sysv.png|thumb|Service startup using upstart|left|1000px]]<br />
<br clear=all><br />
== Apply Resource limits ==<br />
<br />
As you can see in the two diagrams above one of the differences is that systemd reads all the infos wrt. resource limitations from the original service file and then applies possible further restrictions from the drop-in configs (steps 2 and 8 in the diagram). Limits configured in /opt/open-xchange/etc/ox-scriptconf.sh aren't considered anylonger when using systemd while the older System V style init has to source /opt/open-xchange/etc/ox-scriptconf.sh to be able to read the configs for the maximum number of processes (NPROC) and files(NRFILES) that the middleware shall be allowed to create/open (steps 3,4 and 15 in the diagram).<br />
<br />
=== Background Infos ===<br />
<br />
Several ways exist to restrict resources on a linux system from a global level down to user/groups or even shells and the processes started by them.<br />
<br />
==== Sysctl ====<br />
Sysctl is used to modify kernel parameters at runtime. E.g. to set the maximum number of files<br />
<br />
$ sysctl -w fs.file-max=100000<br />
<br />
To permanently set them append to the main configuration file and reload the settings<br />
<br />
$ echo fs.file-max=100000 >> /etc/sysctl.conf<br />
$ sysctl -p<br />
<br />
More infos can be found via '''man sysctl'''<br />
<br />
==== Limits.conf ====<br />
<br />
Allows to restrict resources an a global, group or user level. E.g:<br />
<br />
$ cat /etc/security/limits.d/90-nproc.conf <br />
# Default limit for number of user's processes to prevent<br />
# accidental fork bombs.<br />
# See rhbz #432903 for reasoning.<br />
<br />
* soft nproc 1024<br />
<br />
From '''man limits.conf''':<br />
<blockquote><br />
''Also, please note that all limit settings are set per login. They are not global, nor are they permanent; existing only for the duration of the session.''<br />
</blockquote><br />
<br />
The limits per login are applied via the ''pam'' stack. See '''man pam''' and '''man pam_limits''' for more details. As those limits are bound to sessions they don't affect most daemons started by our supported init systems or init utils. Most state that they are ignored by design, see [https://bugs.launchpad.net/ubuntu/+source/upstart/+bug/938669/comments/4 upstart], [https://bugzilla.redhat.com/show_bug.cgi?id=754285#c1 systemd] and [https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=302079 start-stop-daemon] <br />
<br />
===== Ulimit =====<br />
From '''man bash'''<br />
<blockquote><br />
ulimit [-HSTabcdefilmnpqrstuvx [limit]]<br />
Provides control over the resources available to the shell and to processes started by it, on systems that allow such control.<br />
</blockquote><br />
This is what we use in our System V compatible init scripts to increase resources for the open-xchange process across multiple distros. Currently only ''the maximum number of processes'' and ''the maximum number of open file descriptors available to a single user'' are increased via ulimit. The values are specified in '''/opt/open-xchange/ox-scriptconf.sh'''<br />
<br />
==== Systemd ====<br />
<br />
===== Control Groups=====<br />
<br />
Control groups should only affect the OX middleware if you create/manage them yourself of if you are using a ''modern'' distribution that already uses systemd as init.<br />
<br />
Citing from the kernel cgroup [https://www.kernel.org/doc/Documentation/cgroup-v2.txt documentation]:<br />
<br />
<blockquote><br />
1-2. What is cgroup?<br />
<br />
cgroup is a mechanism to organize processes hierarchically and<br />
distribute system resources along the hierarchy in a controlled and<br />
configurable manner.<br />
<br />
cgroup is largely composed of two parts - the core and controllers.<br />
cgroup core is primarily responsible for hierarchically organizing<br />
processes. A cgroup controller is usually responsible for<br />
distributing a specific type of system resource along the hierarchy<br />
although there are utility controllers which serve purposes other than<br />
resource distribution.<br />
<br />
cgroups form a tree structure and every process in the system belongs<br />
to one and only one cgroup. All threads of a process belong to the<br />
same cgroup. On creation, all processes are put in the cgroup that<br />
the parent process belongs to at the time. A process can be migrated<br />
to another cgroup. Migration of a process doesn't affect already<br />
existing descendant processes.<br />
<br />
Following certain structural constraints, controllers may be enabled or<br />
disabled selectively on a cgroup. All controller behaviors are<br />
hierarchical - if a controller is enabled on a cgroup, it affects all<br />
processes which belong to the cgroups consisting the inclusive<br />
sub-hierarchy of the cgroup. When a controller is enabled on a nested<br />
cgroup, it always restricts the resource distribution further. The<br />
restrictions set closer to the root in the hierarchy can not be<br />
overridden from further away.<br />
</blockquote><br />
<br />
So processes are organized into a tree structure of control groups and controllers are responsible for the distribution of resources. So what kind of controllers exist?<br />
<br />
<blockquote><br />
5. Controllers<br />
<br />
5-1. CPU<br />
<br />
The "cpu" controllers regulates distribution of CPU cycles. This<br />
controller implements weight and absolute bandwidth limit models for<br />
normal scheduling policy and absolute bandwidth allocation model for<br />
realtime scheduling policy.<br />
<br />
5-2. Memory<br />
<br />
The "memory" controller regulates distribution of memory.<br />
...<br />
While not completely water-tight, all major memory usages by a given<br />
cgroup are tracked so that the total memory consumption can be<br />
accounted and controlled to a reasonable extent.<br />
<br />
5-3. IO<br />
<br />
The "io" controller regulates the distribution of IO resources. This<br />
controller implements both weight based and absolute bandwidth or IOPS<br />
limit distribution; however, weight based distribution is available<br />
only if cfq-iosched is in use and neither scheme is available for<br />
blk-mq devices.<br />
</blockquote><br />
<br />
The open-xchange service is simply put into the default ''system.slice'' without applying further limits.<br />
<br />
singlenode$ systemd-cgls<br />
├─1 /sbin/init<br />
├─system.slice<br />
│ ├─avahi-daemon.service<br />
│ │ ├─501 avahi-daemon: running [singlenode]<br />
│ │ └─514 avahi-daemon: chroot helper<br />
│ ├─console-kit-daemon.service<br />
│ │ └─16164 /usr/sbin/console-kit-daemon --no-daemon<br />
│ ├─dbus.service<br />
│ │ └─508 /usr/bin/dbus-daemon --system --address=systemd: --nofork --nopidfile --systemd-activation<br />
│ ├─munin-node.service<br />
│ │ └─4290 /usr/bin/perl -wT /usr/sbin/munin-node<br />
│ ├─open-xchange.service<br />
│ │ └─6037 /usr/bin/java -Dsun.net.inetaddr.ttl=3600 -Dnetworkaddress.cache.ttl=3600 -Dnetworkaddress.cache.negative.ttl=10 ...<br />
<br />
To check all the details use<br />
<br />
singlenode:~ # systemctl show system.slice<br />
<br />
==== Limits besides control groups ====<br />
Besides control groups systemd allows you to apply other limits to the execution environment of your service. Here we can apply the limits that would normally be applied via ''limits.conf'' or ''ulimit''. Systemd uses [http://man7.org/linux/man-pages/man2/setrlimit.2.html setrlimit] for this. The options that we set by default are:<br />
<br />
* LimitNOFILE<br />
* LimitNPROC<br />
<br />
which you have already seen when we took a look at the [[#Service_definiton|service definition]] and [[#Drop-in_config|drop-in configs]]<br />
<br />
<br />
== Verify limits ==<br />
<br />
=== System V ===<br />
<br />
Get the proper pid of the java process started for the middleware by either checking '''ps faux'''<br />
root 4103 0.0 0.0 66528 1868 pts/0 S 22:05 0:00 su -s /bin/bash open-xchange -c /opt/open-xchange/sbin/open-xchange<br />
494 4109 9.5 19.8 2224644 381312 ? Ssl 22:05 1:12 \_ /usr/bin/java -Dsun.net.inetaddr.ttl=3600 -Dnetworkaddress.cache.ttl=3600<br />
<br />
or programmatically<br />
singlenode:~ # pid=$(pgrep -P $(</var/run/open-xchange.pid) java)<br />
<br />
and check the limits applied for that process<br />
singlenode:~ # cat /proc/$pid/limits<br />
Limit Soft Limit Hard Limit Units<br />
Max cpu time unlimited unlimited seconds<br />
Max file size unlimited unlimited bytes<br />
Max data size unlimited unlimited bytes<br />
Max stack size 8388608 unlimited bytes<br />
Max core file size 0 unlimited bytes<br />
Max resident set unlimited unlimited bytes<br />
Max processes 65536 65536 processes<br />
Max open files 65536 65536 files<br />
Max locked memory 65536 65536 bytes<br />
Max address space unlimited unlimited bytes<br />
Max file locks unlimited unlimited locks<br />
Max pending signals 24254 24254 signals<br />
Max msgqueue size 819200 819200 bytes<br />
Max nice priority 0 0<br />
Max realtime priority 0 0<br />
Max realtime timeout unlimited unlimited us<br />
<br />
=== Systemd ===<br />
singlenode:~ # systemctl show open-xchange | grep Limit<br />
StartLimitInterval=10000000<br />
StartLimitBurst=5<br />
StartLimitAction=none<br />
MemoryLimit=18446744073709551615<br />
LimitCPU=18446744073709551615<br />
LimitFSIZE=18446744073709551615<br />
LimitDATA=18446744073709551615<br />
LimitSTACK=18446744073709551615<br />
LimitCORE=18446744073709551615<br />
LimitRSS=18446744073709551615<br />
LimitNOFILE=65536<br />
LimitAS=18446744073709551615<br />
LimitNPROC=65536<br />
LimitMEMLOCK=65536<br />
LimitLOCKS=18446744073709551615<br />
LimitSIGPENDING=19827<br />
LimitMSGQUEUE=819200<br />
LimitNICE=0<br />
LimitRTPRIO=0<br />
LimitRTTIME=18446744073709551615<br />
<br />
<br />
== Open-Xchange middleware on specific distros ==<br />
The support for the mentioned mechanism of resource control differ depending on the distribution and the init system in use.<br />
<br />
=== Debian 7 ===<br />
;Init<br />
: System V style<br />
;OX Configurable Limits/Defaults<br />
: nofile, nproc<br />
<br />
The mentioned limits can be configured via '''/opt/open-xchange/etc/ox-scriptconf.sh'''. The limits are applied via ulimit in the service's init script. The open-xchange service is finally started via start-stop-daemon which doesn't doesn't consider '''/etc/security/limits.*'''<br />
<br />
=== RHEL 6 / CentOS 6 ===<br />
;Init<br />
: Upstart, System V compatible<br />
;OX Configurable Limits/Defaults<br />
: nofile, nproc<br />
<br />
The mentioned limits can be configured via '''/opt/open-xchange/etc/ox-scriptconf.sh'''. The limits are applied via ulimit in the service's init script. Furthermore as the open-xchange service is finally started via '''su ... open-xchange''' on this distro a user session is opened via su/pam and the default CentOS pam config reads the '''/etc/security/limits.*''' configuration by loading the pam stack like:<br />
<br />
: '''/etc/pam.d/su'''<br />
:: '''-> /etc/pam.d/system-auth'''<br />
::: '''-> pam_limits.so'''<br />
<br />
If NPROC isn't configured for the open-xchange-server it's restricted to '''1024''' globally by default to prevent accidental fork bombs, see '''/etc/security/limits.d/90-nproc.conf''' which can result in severe problems modern multithreaded applications.<br />
<br />
=== RHEL 7 / CentOS 7 / Debian 8 / SLE 12 ===<br />
;Init<br />
: Systemd<br />
;OX Configurable Limits/Defaults<br />
: nofile, nproc<br />
<br />
For systemd the default limits are configured directly in the service's unit file that is shipped by OX and located at '''/usr/lib/systemd/system/open-xchange.service'''. The drop-in config to override or extend the default unit file is located at '''/etc/systemd/system/open-xchange.service.d/limits.conf'''. [https://www.freedesktop.org/software/systemd/man/systemd.exec.html Systemd.exec] shows a whole lot of options that can be used by admins to adapt the default service to their specific needs.<br />
<br />
== Evaluate JVM commandline options ==<br />
<br />
Before starting Java and the OSGi framework with all the Open-Xchange middleware bundles we have to evaluate several important JVM commandline options that influence the performance and stability of the Open-Xchange middleware stack. These commandline options are specified in '''/opt/open-xchange/etc/ox-scriptconf.sh'''<br />
<br />
=== Format < 7.8.4 ===<br />
For version up to 7.8.3 the JVM configuration options were all specified via a single property '''JAVA_XTRAOPTS''' which resulted in a rather long configuration string that was hard to read and maintain.<br />
<br />
JAVA_XTRAOPTS="-Dsun.net.inetaddr.ttl=3600 -Dnetworkaddress.cache.ttl=3600 -Dnetworkaddress.cache.negative.ttl=10 \<br />
-Dlogback.threadlocal.put.duplicate=false -server -Djava.awt.headless=true -Xmx512M -XX:MaxPermSize=256m -XX:+UseConcMarkSweepGC \<br />
-XX:+UseParNewGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:NewRatio=3 -XX:+UseTLAB \<br />
-Dosgi.compatibility.bootdelegation=false -XX:-OmitStackTraceInFastThrow"<br />
<br />
During the startup of the open-xchange service (step 11 of diagram Service startup using systemd and step 10 of diagram Service startup using upstart.) this JAVA_XTRAOPTS string goes through a basic validity check and is then passed to the JVM. <br />
<br />
=== Format >= 7.8.4 ===<br />
Beginning from version 7.8.4 the options are split up and sorted into different categories like:<br />
<br />
JAVA_OPTS_GC="-XX:+UseConcMarkSweepGC -XX:+UseParNewGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:NewRatio=3 -XX:+DisableExplicitGC"<br />
JAVA_OPTS_LOG="-Dlogback.threadlocal.put.duplicate=false -XX:-OmitStackTraceInFastThrow"<br />
JAVA_OPTS_MEM="-XX:MaxHeapSize=512M -XX:MaxPermSize=256M -XX:+UseTLAB"<br />
JAVA_OPTS_NET="-Dsun.net.inetaddr.ttl=3600 -Dnetworkaddress.cache.ttl=3600 -Dnetworkaddress.cache.negative.ttl=10"<br />
JAVA_OPTS_OSGI="-Dosgi.compatibility.bootdelegation=false"<br />
JAVA_OPTS_SERVER="-server -Djava.awt.headless=true"<br />
<br />
JAVA_OPTS_OTHER=""<br />
<br />
#JAVA_OPTS_DEBUG="-XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/mnt/heapdump -Xloggc:/var/log/open-xchange/gc.log -verbose:gc -XX:+PrintGCDateStamps \<br />
-XX:+PrintHeapAtGC -XX:+PrintGCApplicationStoppedTime -XX:+PrintTenuringDistribution"<br />
<br />
which is much easier to read an maintain. Furthermore we added a useful '''JAVA_OPTS_DEBUG''' configuration parameter that comes preconfigured but unused/commented so that customers can quickly enable the most commonly used debugging options in case they have to provide detailed system information to Open-Xchange.<br />
<br />
During the startup of the open-xchange service (step 11 of diagram ''Service startup using systemd'' and step 10 of diagram ''Service startup using upstart''.) all uncommented JAVA_OPTS_* categories will then be evaluated and combined again into a single string of commandline options that can then be passed to the JVM.<br />
<br />
=== Upgrade procedure ===<br />
During the upgrade '''7.8.3-revX -> 7.8.4-revY''' existing JAVA_XTRAOPTS will automatically be split up and migrated to the new categories approach. A backup of the file (ox-scriptconf.sh.timestamp) will be created and admins are urged to check for proper migration before deleting the backup file like it's already done with *.dpkg-dist or *.rpmnew files on Debian or RHEL/SLES distributions.<br />
<br />
== Encoding detection during JVM startup ==<br />
Add details from https://bugs.open-xchange.com/show_bug.cgi?id=51074<br />
<br />
== Warnings during early ==<br />
<br />
Early warnings before the start of the JVM and related logging libraries like logback and logstash forwarding are redirected to /var/log/open-xchange/open-xchange-console.log.<br />
<br />
=== Limits ===<br />
If the open-xchange startup script isn't able to set the proper limits on CentOS 6 due to e.g. hard limits being enforced via pam_limits you'll see the following warning in the console log during startup<br />
<br />
singlenode open-xchange # cat open-xchange-console.log<br />
/opt/open-xchange/sbin/open-xchange: line 115: ulimit: max user processes: cannot modify limit: Operation not permitted<br />
<br />
=== Locale ===<br />
If the open-xchange startup script isn't able to detect a unicode locale you'll see a warning like the following (the current charmap might differ)<br />
<br />
WARNING: Couldn't detect proper unicode charmap in locale category LC_CTYPE.<br />
WARNING: Current charmap: "ANSI_X3.4-1968" might cause encoding problems.</div>Marenshttps://oxpedia.org/wiki/index.php?title=Sandbox:Middleware_Startup/sandbox&diff=23216Sandbox:Middleware Startup/sandbox2017-04-24T14:38:00Z<p>Marens: </p>
<hr />
<div><br />
== Systemd ==<br />
<br />
==== Service definition ====<br />
The default service definition as shipped by Open-Xchange is rather concise and easy to read. You can check by looking at the service file that is installed to either /usr/lib/systemd or /lib/systemd depending on your distribution of choice.<br />
<br />
singlenode:~ # cat /usr/lib/systemd/system/open-xchange.service <br />
[Unit]<br />
After=remote-fs.target<br />
After=time-sync.target ypbind.service sendmail.service cyrus.service <br />
<br />
[Service]<br />
User=open-xchange<br />
PermissionsStartOnly=true<br />
TimeoutStartSec=0<br />
ExecStartPre=/opt/open-xchange/sbin/triggerupdatethemes -u<br />
ExecStart=/opt/open-xchange/sbin/open-xchange<br />
ExecStop=/opt/open-xchange/sbin/shutdown -w<br />
ExecReload=/opt/open-xchange/sbin/triggerreloadconfiguration -d<br />
KillMode=process<br />
LimitNOFILE=65536<br />
LimitNPROC=65536<br />
<br />
[Install]<br />
WantedBy=multi-user.target<br />
<br />
==== Drop-in configs ====<br />
Drop in configs allow administrators to easily adapt and override default service unit files. So if you want to change the default limits or add additional limits have a look at our default example drop-in config which is located at /etc/systemd/system/open-xchange.service.d/limits.conf<br />
<br />
singlenode:~ # cat /etc/systemd/system/open-xchange.service.d/limits.conf <br />
# Override and add options in this file<br />
# See systemd.exec(5) for other limits<br />
<br />
[Service]<br />
#LimitNPROC=65536<br />
#LimitNOFILE=65536<br />
<br />
The systemd.exec documentation shows a whole lot of options that can be used by admins to adapt the default service to their specific needs.<br />
<br />
<br clear=all><br />
[[File:mw_start_sysd.png|thumb|Service startup using systemd|left|1000px]]<br />
<br clear=all><br />
<br />
== System V / Upstart ==<br />
<br />
==== Service definition ====<br />
<br />
The default service definition as shipped by Open-Xchange can be found at /etc/init.d/open-xchange and will differ slightly depending on the distro in use.<br />
<br />
<br clear=all><br />
[[File:mw_start_sysv.png|thumb|Service startup using upstart|left|1000px]]<br />
<br clear=all><br />
== Apply Resource limits ==<br />
<br />
As you can see in the two diagrams above one of the differences is that systemd reads all the infos wrt. resource limitations from the original service file and then applies possible further restrictions from the drop-in configs (steps 2 and 8 in the diagram). Limits configured in /opt/open-xchange/etc/ox-scriptconf.sh aren't considered anylonger when using systemd while the older System V style init has to source /opt/open-xchange/etc/ox-scriptconf.sh to be able to read the configs for the maximum number of processes (NPROC) and files(NRFILES) that the middleware shall be allowed to create/open (steps 3,4 and 15 in the diagram).<br />
<br />
=== Background Infos ===<br />
<br />
Several ways exist to restrict resources on a linux system from a global level down to user/groups or even shells and the processes started by them.<br />
<br />
==== Sysctl ====<br />
Sysctl is used to modify kernel parameters at runtime. E.g. to set the maximum number of files<br />
<br />
$ sysctl -w fs.file-max=100000<br />
<br />
To permanently set them append to the main configuration file and reload the settings<br />
<br />
$ echo fs.file-max=100000 >> /etc/sysctl.conf<br />
$ sysctl -p<br />
<br />
More infos can be found via '''man sysctl'''<br />
<br />
==== Limits.conf ====<br />
<br />
Allows to restrict resources an a global, group or user level. E.g:<br />
<br />
$ cat /etc/security/limits.d/90-nproc.conf <br />
# Default limit for number of user's processes to prevent<br />
# accidental fork bombs.<br />
# See rhbz #432903 for reasoning.<br />
<br />
* soft nproc 1024<br />
<br />
From '''man limits.conf''':<br />
<blockquote><br />
''Also, please note that all limit settings are set per login. They are not global, nor are they permanent; existing only for the duration of the session.''<br />
</blockquote><br />
<br />
The limits per login are applied via the ''pam'' stack. See '''man pam''' and '''man pam_limits''' for more details. As those limits are bound to sessions they don't affect most daemons started by our supported init systems or init utils. Most state that they are ignored by design, see [https://bugs.launchpad.net/ubuntu/+source/upstart/+bug/938669/comments/4 upstart], [https://bugzilla.redhat.com/show_bug.cgi?id=754285#c1 systemd] and [https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=302079 start-stop-daemon] <br />
<br />
===== Ulimit =====<br />
From '''man bash'''<br />
<blockquote><br />
ulimit [-HSTabcdefilmnpqrstuvx [limit]]<br />
Provides control over the resources available to the shell and to processes started by it, on systems that allow such control.<br />
</blockquote><br />
This is what we use in our System V compatible init scripts to increase resources for the open-xchange process across multiple distros. Currently only ''the maximum number of processes'' and ''the maximum number of open file descriptors available to a single user'' are increased via ulimit. The values are specified in '''/opt/open-xchange/ox-scriptconf.sh'''<br />
<br />
==== Systemd ====<br />
<br />
===== Control Groups=====<br />
<br />
Control groups should only affect the OX middleware if you create/manage them yourself of if you are using a ''modern'' distribution that already uses systemd as init.<br />
<br />
Citing from the kernel cgroup [https://www.kernel.org/doc/Documentation/cgroup-v2.txt documentation]:<br />
<br />
<blockquote><br />
1-2. What is cgroup?<br />
<br />
cgroup is a mechanism to organize processes hierarchically and<br />
distribute system resources along the hierarchy in a controlled and<br />
configurable manner.<br />
<br />
cgroup is largely composed of two parts - the core and controllers.<br />
cgroup core is primarily responsible for hierarchically organizing<br />
processes. A cgroup controller is usually responsible for<br />
distributing a specific type of system resource along the hierarchy<br />
although there are utility controllers which serve purposes other than<br />
resource distribution.<br />
<br />
cgroups form a tree structure and every process in the system belongs<br />
to one and only one cgroup. All threads of a process belong to the<br />
same cgroup. On creation, all processes are put in the cgroup that<br />
the parent process belongs to at the time. A process can be migrated<br />
to another cgroup. Migration of a process doesn't affect already<br />
existing descendant processes.<br />
<br />
Following certain structural constraints, controllers may be enabled or<br />
disabled selectively on a cgroup. All controller behaviors are<br />
hierarchical - if a controller is enabled on a cgroup, it affects all<br />
processes which belong to the cgroups consisting the inclusive<br />
sub-hierarchy of the cgroup. When a controller is enabled on a nested<br />
cgroup, it always restricts the resource distribution further. The<br />
restrictions set closer to the root in the hierarchy can not be<br />
overridden from further away.<br />
</blockquote><br />
<br />
So processes are organized into a tree structure of control groups and controllers are responsible for the distribution of resources. So what kind of controllers exist?<br />
<br />
<blockquote><br />
5. Controllers<br />
<br />
5-1. CPU<br />
<br />
The "cpu" controllers regulates distribution of CPU cycles. This<br />
controller implements weight and absolute bandwidth limit models for<br />
normal scheduling policy and absolute bandwidth allocation model for<br />
realtime scheduling policy.<br />
<br />
5-2. Memory<br />
<br />
The "memory" controller regulates distribution of memory.<br />
...<br />
While not completely water-tight, all major memory usages by a given<br />
cgroup are tracked so that the total memory consumption can be<br />
accounted and controlled to a reasonable extent.<br />
<br />
5-3. IO<br />
<br />
The "io" controller regulates the distribution of IO resources. This<br />
controller implements both weight based and absolute bandwidth or IOPS<br />
limit distribution; however, weight based distribution is available<br />
only if cfq-iosched is in use and neither scheme is available for<br />
blk-mq devices.<br />
</blockquote><br />
<br />
The open-xchange service is simply put into the default ''system.slice'' without applying further limits.<br />
<br />
singlenode$ systemd-cgls<br />
├─1 /sbin/init<br />
├─system.slice<br />
│ ├─avahi-daemon.service<br />
│ │ ├─501 avahi-daemon: running [singlenode]<br />
│ │ └─514 avahi-daemon: chroot helper<br />
│ ├─console-kit-daemon.service<br />
│ │ └─16164 /usr/sbin/console-kit-daemon --no-daemon<br />
│ ├─dbus.service<br />
│ │ └─508 /usr/bin/dbus-daemon --system --address=systemd: --nofork --nopidfile --systemd-activation<br />
│ ├─munin-node.service<br />
│ │ └─4290 /usr/bin/perl -wT /usr/sbin/munin-node<br />
│ ├─open-xchange.service<br />
│ │ └─6037 /usr/bin/java -Dsun.net.inetaddr.ttl=3600 -Dnetworkaddress.cache.ttl=3600 -Dnetworkaddress.cache.negative.ttl=10 ...<br />
<br />
To check all the details use<br />
<br />
singlenode:~ # systemctl show system.slice<br />
<br />
==== Limits besides control groups ====<br />
Besides control groups systemd allows you to apply other limits to the execution environment of your service. Here we can apply the limits that would normally be applied via ''limits.conf'' or ''ulimit''. Systemd uses [http://man7.org/linux/man-pages/man2/setrlimit.2.html setrlimit] for this. The options that we set by default are:<br />
<br />
* LimitNOFILE<br />
* LimitNPROC<br />
<br />
which you have already seen when we took a look at the [[#Service_definiton|service definition]] and [[#Drop-in_config|drop-in configs]]<br />
<br />
<br />
== Verify limits ==<br />
<br />
=== System V ===<br />
<br />
Get the proper pid of the java process started for the middleware by either checking '''ps faux'''<br />
root 4103 0.0 0.0 66528 1868 pts/0 S 22:05 0:00 su -s /bin/bash open-xchange -c /opt/open-xchange/sbin/open-xchange<br />
494 4109 9.5 19.8 2224644 381312 ? Ssl 22:05 1:12 \_ /usr/bin/java -Dsun.net.inetaddr.ttl=3600 -Dnetworkaddress.cache.ttl=3600<br />
<br />
or programmatically<br />
singlenode:~ # pid=$(pgrep -P $(</var/run/open-xchange.pid) java)<br />
<br />
and check the limits applied for that process<br />
singlenode:~ # cat /proc/$pid/limits<br />
Limit Soft Limit Hard Limit Units<br />
Max cpu time unlimited unlimited seconds<br />
Max file size unlimited unlimited bytes<br />
Max data size unlimited unlimited bytes<br />
Max stack size 8388608 unlimited bytes<br />
Max core file size 0 unlimited bytes<br />
Max resident set unlimited unlimited bytes<br />
Max processes 65536 65536 processes<br />
Max open files 65536 65536 files<br />
Max locked memory 65536 65536 bytes<br />
Max address space unlimited unlimited bytes<br />
Max file locks unlimited unlimited locks<br />
Max pending signals 24254 24254 signals<br />
Max msgqueue size 819200 819200 bytes<br />
Max nice priority 0 0<br />
Max realtime priority 0 0<br />
Max realtime timeout unlimited unlimited us<br />
<br />
=== Systemd ===<br />
singlenode:~ # systemctl show open-xchange | grep Limit<br />
StartLimitInterval=10000000<br />
StartLimitBurst=5<br />
StartLimitAction=none<br />
MemoryLimit=18446744073709551615<br />
LimitCPU=18446744073709551615<br />
LimitFSIZE=18446744073709551615<br />
LimitDATA=18446744073709551615<br />
LimitSTACK=18446744073709551615<br />
LimitCORE=18446744073709551615<br />
LimitRSS=18446744073709551615<br />
LimitNOFILE=65536<br />
LimitAS=18446744073709551615<br />
LimitNPROC=65536<br />
LimitMEMLOCK=65536<br />
LimitLOCKS=18446744073709551615<br />
LimitSIGPENDING=19827<br />
LimitMSGQUEUE=819200<br />
LimitNICE=0<br />
LimitRTPRIO=0<br />
LimitRTTIME=18446744073709551615<br />
<br />
<br />
== Open-Xchange middleware on specific distros ==<br />
The support for the mentioned mechanism of resource control differ depending on the distribution and the init system in use.<br />
<br />
=== Debian 7 ===<br />
;Init<br />
: System V style<br />
;OX Configurable Limits/Defaults<br />
: nofile, nproc<br />
<br />
The mentioned limits can be configured via '''/opt/open-xchange/etc/ox-scriptconf.sh'''. The limits are applied via ulimit in the service's init script. The open-xchange service is finally started via start-stop-daemon which doesn't doesn't consider '''/etc/security/limits.*'''<br />
<br />
=== RHEL 6 / CentOS 6 ===<br />
;Init<br />
: Upstart, System V compatible<br />
;OX Configurable Limits/Defaults<br />
: nofile, nproc<br />
<br />
The mentioned limits can be configured via '''/opt/open-xchange/etc/ox-scriptconf.sh'''. The limits are applied via ulimit in the service's init script. Furthermore as the open-xchange service is finally started via '''su ... open-xchange''' on this distro a user session is opened via su/pam and the default CentOS pam config reads the '''/etc/security/limits.*''' configuration by loading the pam stack like:<br />
<br />
: '''/etc/pam.d/su'''<br />
:: '''-> /etc/pam.d/system-auth'''<br />
::: '''-> pam_limits.so'''<br />
<br />
If NPROC isn't configured for the open-xchange-server it's restricted to '''1024''' globally by default to prevent accidental fork bombs, see '''/etc/security/limits.d/90-nproc.conf''' which can result in severe problems modern multithreaded applications.<br />
<br />
=== RHEL 7 / CentOS 7 / Debian 8 / SLE 12 ===<br />
;Init<br />
: Systemd<br />
;OX Configurable Limits/Defaults<br />
: nofile, nproc<br />
<br />
For systemd the default limits are configured directly in the service's unit file that is shipped by OX and located at '''/usr/lib/systemd/system/open-xchange.service'''. The drop-in config to override or extend the default unit file is located at '''/etc/systemd/system/open-xchange.service.d/limits.conf'''. [https://www.freedesktop.org/software/systemd/man/systemd.exec.html Systemd.exec] shows a whole lot of options that can be used by admins to adapt the default service to their specific needs.<br />
<br />
== Evaluate JVM commandline options ==<br />
<br />
== Encoding detection during JVM startup ==<br />
Add details from https://bugs.open-xchange.com/show_bug.cgi?id=51074<br />
<br />
== Warnings during early ==<br />
<br />
Early warnings before the start of the JVM and related logging libraries like logback and logstash forwarding are redirected to /var/log/open-xchange/open-xchange-console.log.<br />
<br />
=== Limits ===<br />
If the open-xchange startup script isn't able to set the proper limits on CentOS 6 due to e.g. hard limits being enforced via pam_limits you'll see the following warning in the console log during startup<br />
<br />
singlenode open-xchange # cat open-xchange-console.log<br />
/opt/open-xchange/sbin/open-xchange: line 115: ulimit: max user processes: cannot modify limit: Operation not permitted<br />
<br />
=== Locale ===<br />
If the open-xchange startup script isn't able to detect a unicode locale you'll see a warning like the following (the current charmap might differ)<br />
<br />
WARNING: Couldn't detect proper unicode charmap in locale category LC_CTYPE.<br />
WARNING: Current charmap: "ANSI_X3.4-1968" might cause encoding problems.</div>Marenshttps://oxpedia.org/wiki/index.php?title=Sandbox:Middleware_Startup/sandbox&diff=23215Sandbox:Middleware Startup/sandbox2017-04-24T14:24:42Z<p>Marens: </p>
<hr />
<div><br />
== Systemd ==<br />
<br />
==== Service definition ====<br />
The default service definition as shipped by Open-Xchange is rather concise and easy to read. You can check by looking at the service file that is installed to either /usr/lib/systemd or /lib/systemd depending on your distribution of choice.<br />
<br />
singlenode:~ # cat /usr/lib/systemd/system/open-xchange.service <br />
[Unit]<br />
After=remote-fs.target<br />
After=time-sync.target ypbind.service sendmail.service cyrus.service <br />
<br />
[Service]<br />
User=open-xchange<br />
PermissionsStartOnly=true<br />
TimeoutStartSec=0<br />
ExecStartPre=/opt/open-xchange/sbin/triggerupdatethemes -u<br />
ExecStart=/opt/open-xchange/sbin/open-xchange<br />
ExecStop=/opt/open-xchange/sbin/shutdown -w<br />
ExecReload=/opt/open-xchange/sbin/triggerreloadconfiguration -d<br />
KillMode=process<br />
LimitNOFILE=65536<br />
LimitNPROC=65536<br />
<br />
[Install]<br />
WantedBy=multi-user.target<br />
<br />
==== Drop-in configs ====<br />
Drop in configs allow administrators to easily adapt and override default service unit files. So if you want to change the default limits or add additional limits have a look at our default example drop-in config which is located at /etc/systemd/system/open-xchange.service.d/limits.conf<br />
<br />
singlenode:~ # cat /etc/systemd/system/open-xchange.service.d/limits.conf <br />
# Override and add options in this file<br />
# See systemd.exec(5) for other limits<br />
<br />
[Service]<br />
#LimitNPROC=65536<br />
#LimitNOFILE=65536<br />
<br />
The systemd.exec documentation shows a whole lot of options that can be used by admins to adapt the default service to their specific needs.<br />
<br />
[[File:mw_start_sysd.png|1000px]]<br />
<br />
== System V / Upstart ==<br />
<br />
==== Service definition ====<br />
<br />
The default service definition as shipped by Open-Xchange can be found at /etc/init.d/open-xchange and will differ slightly depending on the distro in use.<br />
<br />
[[File:mw_start_sysv.png|1000px]]<br />
<br />
== Apply Resource limits ==<br />
<br />
As you can see in the two diagrams above one of the differences is that systemd reads all the infos wrt. resource limitations from the original service file and then applies possible further restrictions from the drop-in configs (steps 2 and 8 in the diagram). Limits configured in /opt/open-xchange/etc/ox-scriptconf.sh aren't considered anylonger when using systemd while the older System V style init has to source /opt/open-xchange/etc/ox-scriptconf.sh to be able to read the configs for the maximum number of processes (NPROC) and files(NRFILES) that the middleware shall be allowed to create/open (steps 3,4 and 15 in the diagram).<br />
<br />
=== Background Infos ===<br />
<br />
Several ways exist to restrict resources on a linux system from a global level down to user/groups or even shells and the processes started by them.<br />
<br />
==== Sysctl ====<br />
Sysctl is used to modify kernel parameters at runtime. E.g. to set the maximum number of files<br />
<br />
$ sysctl -w fs.file-max=100000<br />
<br />
To permanently set them append to the main configuration file and reload the settings<br />
<br />
$ echo fs.file-max=100000 >> /etc/sysctl.conf<br />
$ sysctl -p<br />
<br />
More infos can be found via '''man sysctl'''<br />
<br />
==== Limits.conf ====<br />
<br />
Allows to restrict resources an a global, group or user level. E.g:<br />
<br />
$ cat /etc/security/limits.d/90-nproc.conf <br />
# Default limit for number of user's processes to prevent<br />
# accidental fork bombs.<br />
# See rhbz #432903 for reasoning.<br />
<br />
* soft nproc 1024<br />
<br />
From '''man limits.conf''':<br />
<blockquote><br />
''Also, please note that all limit settings are set per login. They are not global, nor are they permanent; existing only for the duration of the session.''<br />
</blockquote><br />
<br />
The limits per login are applied via the ''pam'' stack. See '''man pam''' and '''man pam_limits''' for more details. As those limits are bound to sessions they don't affect most daemons started by our supported init systems or init utils. Most state that they are ignored by design, see [https://bugs.launchpad.net/ubuntu/+source/upstart/+bug/938669/comments/4 upstart], [https://bugzilla.redhat.com/show_bug.cgi?id=754285#c1 systemd] and [https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=302079 start-stop-daemon] <br />
<br />
===== Ulimit =====<br />
From '''man bash'''<br />
<blockquote><br />
ulimit [-HSTabcdefilmnpqrstuvx [limit]]<br />
Provides control over the resources available to the shell and to processes started by it, on systems that allow such control.<br />
</blockquote><br />
This is what we use in our System V compatible init scripts to increase resources for the open-xchange process across multiple distros. Currently only ''the maximum number of processes'' and ''the maximum number of open file descriptors available to a single user'' are increased via ulimit. The values are specified in '''/opt/open-xchange/ox-scriptconf.sh'''<br />
<br />
==== Systemd ====<br />
<br />
===== Control Groups=====<br />
<br />
Control groups should only affect the OX middleware if you create/manage them yourself of if you are using a ''modern'' distribution that already uses systemd as init.<br />
<br />
Citing from the kernel cgroup [https://www.kernel.org/doc/Documentation/cgroup-v2.txt documentation]:<br />
<br />
<blockquote><br />
1-2. What is cgroup?<br />
<br />
cgroup is a mechanism to organize processes hierarchically and<br />
distribute system resources along the hierarchy in a controlled and<br />
configurable manner.<br />
<br />
cgroup is largely composed of two parts - the core and controllers.<br />
cgroup core is primarily responsible for hierarchically organizing<br />
processes. A cgroup controller is usually responsible for<br />
distributing a specific type of system resource along the hierarchy<br />
although there are utility controllers which serve purposes other than<br />
resource distribution.<br />
<br />
cgroups form a tree structure and every process in the system belongs<br />
to one and only one cgroup. All threads of a process belong to the<br />
same cgroup. On creation, all processes are put in the cgroup that<br />
the parent process belongs to at the time. A process can be migrated<br />
to another cgroup. Migration of a process doesn't affect already<br />
existing descendant processes.<br />
<br />
Following certain structural constraints, controllers may be enabled or<br />
disabled selectively on a cgroup. All controller behaviors are<br />
hierarchical - if a controller is enabled on a cgroup, it affects all<br />
processes which belong to the cgroups consisting the inclusive<br />
sub-hierarchy of the cgroup. When a controller is enabled on a nested<br />
cgroup, it always restricts the resource distribution further. The<br />
restrictions set closer to the root in the hierarchy can not be<br />
overridden from further away.<br />
</blockquote><br />
<br />
So processes are organized into a tree structure of control groups and controllers are responsible for the distribution of resources. So what kind of controllers exist?<br />
<br />
<blockquote><br />
5. Controllers<br />
<br />
5-1. CPU<br />
<br />
The "cpu" controllers regulates distribution of CPU cycles. This<br />
controller implements weight and absolute bandwidth limit models for<br />
normal scheduling policy and absolute bandwidth allocation model for<br />
realtime scheduling policy.<br />
<br />
5-2. Memory<br />
<br />
The "memory" controller regulates distribution of memory.<br />
...<br />
While not completely water-tight, all major memory usages by a given<br />
cgroup are tracked so that the total memory consumption can be<br />
accounted and controlled to a reasonable extent.<br />
<br />
5-3. IO<br />
<br />
The "io" controller regulates the distribution of IO resources. This<br />
controller implements both weight based and absolute bandwidth or IOPS<br />
limit distribution; however, weight based distribution is available<br />
only if cfq-iosched is in use and neither scheme is available for<br />
blk-mq devices.<br />
</blockquote><br />
<br />
The open-xchange service is simply put into the default ''system.slice'' without applying further limits.<br />
<br />
singlenode$ systemd-cgls<br />
├─1 /sbin/init<br />
├─system.slice<br />
│ ├─avahi-daemon.service<br />
│ │ ├─501 avahi-daemon: running [singlenode]<br />
│ │ └─514 avahi-daemon: chroot helper<br />
│ ├─console-kit-daemon.service<br />
│ │ └─16164 /usr/sbin/console-kit-daemon --no-daemon<br />
│ ├─dbus.service<br />
│ │ └─508 /usr/bin/dbus-daemon --system --address=systemd: --nofork --nopidfile --systemd-activation<br />
│ ├─munin-node.service<br />
│ │ └─4290 /usr/bin/perl -wT /usr/sbin/munin-node<br />
│ ├─open-xchange.service<br />
│ │ └─6037 /usr/bin/java -Dsun.net.inetaddr.ttl=3600 -Dnetworkaddress.cache.ttl=3600 -Dnetworkaddress.cache.negative.ttl=10 ...<br />
<br />
To check all the details use<br />
<br />
singlenode:~ # systemctl show system.slice<br />
<br />
==== Limits besides control groups ====<br />
Besides control groups systemd allows you to apply other limits to the execution environment of your service. Here we can apply the limits that would normally be applied via ''limits.conf'' or ''ulimit''. Systemd uses [http://man7.org/linux/man-pages/man2/setrlimit.2.html setrlimit] for this. The options that we set by default are:<br />
<br />
* LimitNOFILE<br />
* LimitNPROC<br />
<br />
which you have already seen when we took a look at the [[#Service_definiton|service definition]] and [[#Drop-in_config|drop-in configs]]<br />
<br />
<br />
== Verify limits ==<br />
<br />
=== System V ===<br />
<br />
Get the proper pid of the java process started for the middleware by either checking '''ps faux'''<br />
root 4103 0.0 0.0 66528 1868 pts/0 S 22:05 0:00 su -s /bin/bash open-xchange -c /opt/open-xchange/sbin/open-xchange<br />
494 4109 9.5 19.8 2224644 381312 ? Ssl 22:05 1:12 \_ /usr/bin/java -Dsun.net.inetaddr.ttl=3600 -Dnetworkaddress.cache.ttl=3600<br />
<br />
or programmatically<br />
singlenode:~ # pid=$(pgrep -P $(</var/run/open-xchange.pid) java)<br />
<br />
and check the limits applied for that process<br />
singlenode:~ # cat /proc/$pid/limits<br />
Limit Soft Limit Hard Limit Units<br />
Max cpu time unlimited unlimited seconds<br />
Max file size unlimited unlimited bytes<br />
Max data size unlimited unlimited bytes<br />
Max stack size 8388608 unlimited bytes<br />
Max core file size 0 unlimited bytes<br />
Max resident set unlimited unlimited bytes<br />
Max processes 65536 65536 processes<br />
Max open files 65536 65536 files<br />
Max locked memory 65536 65536 bytes<br />
Max address space unlimited unlimited bytes<br />
Max file locks unlimited unlimited locks<br />
Max pending signals 24254 24254 signals<br />
Max msgqueue size 819200 819200 bytes<br />
Max nice priority 0 0<br />
Max realtime priority 0 0<br />
Max realtime timeout unlimited unlimited us<br />
<br />
=== Systemd ===<br />
singlenode:~ # systemctl show open-xchange | grep Limit<br />
StartLimitInterval=10000000<br />
StartLimitBurst=5<br />
StartLimitAction=none<br />
MemoryLimit=18446744073709551615<br />
LimitCPU=18446744073709551615<br />
LimitFSIZE=18446744073709551615<br />
LimitDATA=18446744073709551615<br />
LimitSTACK=18446744073709551615<br />
LimitCORE=18446744073709551615<br />
LimitRSS=18446744073709551615<br />
LimitNOFILE=65536<br />
LimitAS=18446744073709551615<br />
LimitNPROC=65536<br />
LimitMEMLOCK=65536<br />
LimitLOCKS=18446744073709551615<br />
LimitSIGPENDING=19827<br />
LimitMSGQUEUE=819200<br />
LimitNICE=0<br />
LimitRTPRIO=0<br />
LimitRTTIME=18446744073709551615<br />
<br />
<br />
== Open-Xchange middleware on specific distros ==<br />
The support for the mentioned mechanism of resource control differ depending on the distribution and the init system in use.<br />
<br />
=== Debian 7 ===<br />
;Init<br />
: System V style<br />
;OX Configurable Limits/Defaults<br />
: nofile, nproc<br />
<br />
The mentioned limits can be configured via '''/opt/open-xchange/etc/ox-scriptconf.sh'''. The limits are applied via ulimit in the service's init script. The open-xchange service is finally started via start-stop-daemon which doesn't doesn't consider '''/etc/security/limits.*'''<br />
<br />
=== RHEL 6 / CentOS 6 ===<br />
;Init<br />
: Upstart, System V compatible<br />
;OX Configurable Limits/Defaults<br />
: nofile, nproc<br />
<br />
The mentioned limits can be configured via '''/opt/open-xchange/etc/ox-scriptconf.sh'''. The limits are applied via ulimit in the service's init script. Furthermore as the open-xchange service is finally started via '''su ... open-xchange''' on this distro a user session is opened via su/pam and the default CentOS pam config reads the '''/etc/security/limits.*''' configuration by loading the pam stack like:<br />
<br />
: '''/etc/pam.d/su'''<br />
:: '''-> /etc/pam.d/system-auth'''<br />
::: '''-> pam_limits.so'''<br />
<br />
If NPROC isn't configured for the open-xchange-server it's restricted to '''1024''' globally by default to prevent accidental fork bombs, see '''/etc/security/limits.d/90-nproc.conf''' which can result in severe problems modern multithreaded applications.<br />
<br />
=== RHEL 7 / CentOS 7 / Debian 8 / SLE 12 ===<br />
;Init<br />
: Systemd<br />
;OX Configurable Limits/Defaults<br />
: nofile, nproc<br />
<br />
For systemd the default limits are configured directly in the service's unit file that is shipped by OX and located at '''/usr/lib/systemd/system/open-xchange.service'''. The drop-in config to override or extend the default unit file is located at '''/etc/systemd/system/open-xchange.service.d/limits.conf'''. [https://www.freedesktop.org/software/systemd/man/systemd.exec.html Systemd.exec] shows a whole lot of options that can be used by admins to adapt the default service to their specific needs.<br />
<br />
== Evaluate JVM commandline options ==<br />
<br />
== Encoding detection during JVM startup ==<br />
Add details from https://bugs.open-xchange.com/show_bug.cgi?id=51074<br />
<br />
== Warnings during early ==<br />
<br />
Early warnings before the start of the JVM and related logging libraries like logback and logstash forwarding are redirected to /var/log/open-xchange/open-xchange-console.log.<br />
<br />
=== Limits ===<br />
If the open-xchange startup script isn't able to set the proper limits on CentOS 6 due to e.g. hard limits being enforced via pam_limits you'll see the following warning in the console log during startup<br />
<br />
singlenode open-xchange # cat open-xchange-console.log<br />
/opt/open-xchange/sbin/open-xchange: line 115: ulimit: max user processes: cannot modify limit: Operation not permitted<br />
<br />
=== Locale ===<br />
If the open-xchange startup script isn't able to detect a unicode locale you'll see a warning like the following (the current charmap might differ)<br />
<br />
WARNING: Couldn't detect proper unicode charmap in locale category LC_CTYPE.<br />
WARNING: Current charmap: "ANSI_X3.4-1968" might cause encoding problems.</div>Marenshttps://oxpedia.org/wiki/index.php?title=Sandbox:Middleware_Startup/sandbox&diff=23214Sandbox:Middleware Startup/sandbox2017-04-24T13:49:54Z<p>Marens: </p>
<hr />
<div><br />
== Systemd ==<br />
<br />
[[File:mw_start_sysd.png|1000px]]<br />
<br />
==== Service definition ====<br />
The default service definition as shipped by Open-Xchange is rather concise and easy to read. You can check by looking at the service file that is installed to either /usr/lib/systemd or /lib/systemd depending on your distribution of choice.<br />
<br />
singlenode:~ # cat /usr/lib/systemd/system/open-xchange.service <br />
[Unit]<br />
After=remote-fs.target<br />
After=time-sync.target ypbind.service sendmail.service cyrus.service <br />
<br />
[Service]<br />
User=open-xchange<br />
PermissionsStartOnly=true<br />
TimeoutStartSec=0<br />
ExecStartPre=/opt/open-xchange/sbin/triggerupdatethemes -u<br />
ExecStart=/opt/open-xchange/sbin/open-xchange<br />
ExecStop=/opt/open-xchange/sbin/shutdown -w<br />
ExecReload=/opt/open-xchange/sbin/triggerreloadconfiguration -d<br />
KillMode=process<br />
LimitNOFILE=65536<br />
LimitNPROC=65536<br />
<br />
[Install]<br />
WantedBy=multi-user.target<br />
<br />
==== Drop-in configs ====<br />
Drop in configs allow administrators to easily adapt and override default service unit files. So if you want to change the default limits or add additional limits have a look at our default example drop-in config which is located at /etc/systemd/system/open-xchange.service.d/limits.conf<br />
<br />
singlenode:~ # cat /etc/systemd/system/open-xchange.service.d/limits.conf <br />
# Override and add options in this file<br />
# See systemd.exec(5) for other limits<br />
<br />
[Service]<br />
#LimitNPROC=65536<br />
#LimitNOFILE=65536<br />
<br />
The systemd.exec documentation shows a whole lot of options that can be used by admins to adapt the default service to their specific needs.<br />
<br />
== System V / Upstart ==<br />
<br />
[[File:mw_start_sysv.png|1000px]]<br />
<br />
==== Service definition ====<br />
<br />
== Apply Resource limits ==<br />
<br />
=== Background Infos ===<br />
<br />
Several ways exist to restrict resources on a linux system from a global level down to user/groups or even shells and the processes started by them.<br />
<br />
==== Sysctl ====<br />
Sysctl is used to modify kernel parameters at runtime. E.g. to set the maximum number of files<br />
<br />
$ sysctl -w fs.file-max=100000<br />
<br />
To permanently set them append to the main configuration file and reload the settings<br />
<br />
$ echo fs.file-max=100000 >> /etc/sysctl.conf<br />
$ sysctl -p<br />
<br />
More infos can be found via '''man sysctl'''<br />
<br />
==== Limits.conf ====<br />
<br />
Allows to restrict resources an a global, group or user level. E.g:<br />
<br />
$ cat /etc/security/limits.d/90-nproc.conf <br />
# Default limit for number of user's processes to prevent<br />
# accidental fork bombs.<br />
# See rhbz #432903 for reasoning.<br />
<br />
* soft nproc 1024<br />
<br />
From '''man limits.conf''':<br />
<blockquote><br />
''Also, please note that all limit settings are set per login. They are not global, nor are they permanent; existing only for the duration of the session.''<br />
</blockquote><br />
<br />
The limits per login are applied via the ''pam'' stack. See '''man pam''' and '''man pam_limits''' for more details. As those limits are bound to sessions they don't affect most daemons started by our supported init systems or init utils. Most state that they are ignored by design, see [https://bugs.launchpad.net/ubuntu/+source/upstart/+bug/938669/comments/4 upstart], [https://bugzilla.redhat.com/show_bug.cgi?id=754285#c1 systemd] and [https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=302079 start-stop-daemon] <br />
<br />
===== Ulimit =====<br />
From '''man bash'''<br />
<blockquote><br />
ulimit [-HSTabcdefilmnpqrstuvx [limit]]<br />
Provides control over the resources available to the shell and to processes started by it, on systems that allow such control.<br />
</blockquote><br />
This is what we use in our System V compatible init scripts to increase resources for the open-xchange process across multiple distros. Currently only ''the maximum number of processes'' and ''the maximum number of open file descriptors available to a single user'' are increased via ulimit. The values are specified in '''/opt/open-xchange/ox-scriptconf.sh'''<br />
<br />
==== Systemd ====<br />
<br />
===== Control Groups=====<br />
<br />
Control groups should only affect the OX middleware if you create/manage them yourself of if you are using a ''modern'' distribution that already uses systemd as init.<br />
<br />
Citing from the kernel cgroup [https://www.kernel.org/doc/Documentation/cgroup-v2.txt documentation]:<br />
<br />
<blockquote><br />
1-2. What is cgroup?<br />
<br />
cgroup is a mechanism to organize processes hierarchically and<br />
distribute system resources along the hierarchy in a controlled and<br />
configurable manner.<br />
<br />
cgroup is largely composed of two parts - the core and controllers.<br />
cgroup core is primarily responsible for hierarchically organizing<br />
processes. A cgroup controller is usually responsible for<br />
distributing a specific type of system resource along the hierarchy<br />
although there are utility controllers which serve purposes other than<br />
resource distribution.<br />
<br />
cgroups form a tree structure and every process in the system belongs<br />
to one and only one cgroup. All threads of a process belong to the<br />
same cgroup. On creation, all processes are put in the cgroup that<br />
the parent process belongs to at the time. A process can be migrated<br />
to another cgroup. Migration of a process doesn't affect already<br />
existing descendant processes.<br />
<br />
Following certain structural constraints, controllers may be enabled or<br />
disabled selectively on a cgroup. All controller behaviors are<br />
hierarchical - if a controller is enabled on a cgroup, it affects all<br />
processes which belong to the cgroups consisting the inclusive<br />
sub-hierarchy of the cgroup. When a controller is enabled on a nested<br />
cgroup, it always restricts the resource distribution further. The<br />
restrictions set closer to the root in the hierarchy can not be<br />
overridden from further away.<br />
</blockquote><br />
<br />
So processes are organized into a tree structure of control groups and controllers are responsible for the distribution of resources. So what kind of controllers exist?<br />
<br />
<blockquote><br />
5. Controllers<br />
<br />
5-1. CPU<br />
<br />
The "cpu" controllers regulates distribution of CPU cycles. This<br />
controller implements weight and absolute bandwidth limit models for<br />
normal scheduling policy and absolute bandwidth allocation model for<br />
realtime scheduling policy.<br />
<br />
5-2. Memory<br />
<br />
The "memory" controller regulates distribution of memory.<br />
...<br />
While not completely water-tight, all major memory usages by a given<br />
cgroup are tracked so that the total memory consumption can be<br />
accounted and controlled to a reasonable extent.<br />
<br />
5-3. IO<br />
<br />
The "io" controller regulates the distribution of IO resources. This<br />
controller implements both weight based and absolute bandwidth or IOPS<br />
limit distribution; however, weight based distribution is available<br />
only if cfq-iosched is in use and neither scheme is available for<br />
blk-mq devices.<br />
</blockquote><br />
<br />
The open-xchange service is simply put into the default ''system.slice'' without applying further limits.<br />
<br />
singlenode$ systemd-cgls<br />
├─1 /sbin/init<br />
├─system.slice<br />
│ ├─avahi-daemon.service<br />
│ │ ├─501 avahi-daemon: running [singlenode]<br />
│ │ └─514 avahi-daemon: chroot helper<br />
│ ├─console-kit-daemon.service<br />
│ │ └─16164 /usr/sbin/console-kit-daemon --no-daemon<br />
│ ├─dbus.service<br />
│ │ └─508 /usr/bin/dbus-daemon --system --address=systemd: --nofork --nopidfile --systemd-activation<br />
│ ├─munin-node.service<br />
│ │ └─4290 /usr/bin/perl -wT /usr/sbin/munin-node<br />
│ ├─open-xchange.service<br />
│ │ └─6037 /usr/bin/java -Dsun.net.inetaddr.ttl=3600 -Dnetworkaddress.cache.ttl=3600 -Dnetworkaddress.cache.negative.ttl=10 ...<br />
<br />
To check all the details use<br />
<br />
singlenode:~ # systemctl show system.slice<br />
<br />
==== Limits besides control groups ====<br />
Besides control groups systemd allows you to apply other limits to the execution environment of your service. Here we can apply the limits that would normally be applied via ''limits.conf'' or ''ulimit''. Systemd uses [http://man7.org/linux/man-pages/man2/setrlimit.2.html setrlimit] for this. The options that we set by default are:<br />
<br />
* LimitNOFILE<br />
* LimitNPROC<br />
<br />
You can check this by looking at the service file that is shipped by default ('''Note:''' Depending on your distro the service files may be located either in /usr/lib/systemd or /lib/systemd. We'll use /usr/lib in this oxpedia article)<br />
<br />
singlenode:~ # cat /usr/lib/systemd/system/open-xchange.service <br />
[Unit]<br />
After=remote-fs.target<br />
After=time-sync.target ypbind.service sendmail.service cyrus.service <br />
<br />
[Service]<br />
User=open-xchange<br />
PermissionsStartOnly=true<br />
TimeoutStartSec=0<br />
ExecStartPre=/opt/open-xchange/sbin/triggerupdatethemes -u<br />
ExecStart=/opt/open-xchange/sbin/open-xchange<br />
ExecStop=/opt/open-xchange/sbin/shutdown -w<br />
ExecReload=/opt/open-xchange/sbin/triggerreloadconfiguration -d<br />
KillMode=process<br />
LimitNOFILE=65536<br />
LimitNPROC=65536<br />
<br />
[Install]<br />
WantedBy=multi-user.target<br />
<br />
===== Drop-in configs =====<br />
Drop in configs allow administrators to easily override the default service unit files. So if you want to change the default limits or add additional limits have a look at<br />
<br />
singlenode:~ # cat /etc/systemd/system/open-xchange.service.d/limits.conf <br />
# Override and add options in this file<br />
# See systemd.exec(5) for other limits<br />
<br />
[Service]<br />
#LimitNPROC=65536<br />
#LimitNOFILE=65536<br />
<br />
== Verify limits ==<br />
<br />
=== System V ===<br />
<br />
Get the proper pid of the java process started for the middleware by either checking '''ps faux'''<br />
root 4103 0.0 0.0 66528 1868 pts/0 S 22:05 0:00 su -s /bin/bash open-xchange -c /opt/open-xchange/sbin/open-xchange<br />
494 4109 9.5 19.8 2224644 381312 ? Ssl 22:05 1:12 \_ /usr/bin/java -Dsun.net.inetaddr.ttl=3600 -Dnetworkaddress.cache.ttl=3600<br />
<br />
or programmatically<br />
singlenode:~ # pid=$(pgrep -P $(</var/run/open-xchange.pid) java)<br />
<br />
and check the limits applied for that process<br />
singlenode:~ # cat /proc/$pid/limits<br />
Limit Soft Limit Hard Limit Units<br />
Max cpu time unlimited unlimited seconds<br />
Max file size unlimited unlimited bytes<br />
Max data size unlimited unlimited bytes<br />
Max stack size 8388608 unlimited bytes<br />
Max core file size 0 unlimited bytes<br />
Max resident set unlimited unlimited bytes<br />
Max processes 65536 65536 processes<br />
Max open files 65536 65536 files<br />
Max locked memory 65536 65536 bytes<br />
Max address space unlimited unlimited bytes<br />
Max file locks unlimited unlimited locks<br />
Max pending signals 24254 24254 signals<br />
Max msgqueue size 819200 819200 bytes<br />
Max nice priority 0 0<br />
Max realtime priority 0 0<br />
Max realtime timeout unlimited unlimited us<br />
<br />
=== Systemd ===<br />
singlenode:~ # systemctl show open-xchange | grep Limit<br />
StartLimitInterval=10000000<br />
StartLimitBurst=5<br />
StartLimitAction=none<br />
MemoryLimit=18446744073709551615<br />
LimitCPU=18446744073709551615<br />
LimitFSIZE=18446744073709551615<br />
LimitDATA=18446744073709551615<br />
LimitSTACK=18446744073709551615<br />
LimitCORE=18446744073709551615<br />
LimitRSS=18446744073709551615<br />
LimitNOFILE=65536<br />
LimitAS=18446744073709551615<br />
LimitNPROC=65536<br />
LimitMEMLOCK=65536<br />
LimitLOCKS=18446744073709551615<br />
LimitSIGPENDING=19827<br />
LimitMSGQUEUE=819200<br />
LimitNICE=0<br />
LimitRTPRIO=0<br />
LimitRTTIME=18446744073709551615<br />
<br />
<br />
== Open-Xchange middleware on specific distros ==<br />
The support for the mentioned mechanism of resource control differ depending on the distribution and the init system in use.<br />
<br />
=== Debian 7 ===<br />
;Init<br />
: System V style<br />
;OX Configurable Limits/Defaults<br />
: nofile, nproc<br />
<br />
The mentioned limits can be configured via '''/opt/open-xchange/etc/ox-scriptconf.sh'''. The limits are applied via ulimit in the service's init script. The open-xchange service is finally started via start-stop-daemon which doesn't doesn't consider '''/etc/security/limits.*'''<br />
<br />
=== RHEL 6 / CentOS 6 ===<br />
;Init<br />
: Upstart, System V compatible<br />
;OX Configurable Limits/Defaults<br />
: nofile, nproc<br />
<br />
The mentioned limits can be configured via '''/opt/open-xchange/etc/ox-scriptconf.sh'''. The limits are applied via ulimit in the service's init script. Furthermore as the open-xchange service is finally started via '''su ... open-xchange''' on this distro a user session is opened via su/pam and the default CentOS pam config reads the '''/etc/security/limits.*''' configuration by loading the pam stack like:<br />
<br />
: '''/etc/pam.d/su'''<br />
:: '''-> /etc/pam.d/system-auth'''<br />
::: '''-> pam_limits.so'''<br />
<br />
If NPROC isn't configured for the open-xchange-server it's restricted to '''1024''' globally by default to prevent accidental fork bombs, see '''/etc/security/limits.d/90-nproc.conf''' which can result in severe problems modern multithreaded applications.<br />
<br />
=== RHEL 7 / CentOS 7 / Debian 8 / SLE 12 ===<br />
;Init<br />
: Systemd<br />
;OX Configurable Limits/Defaults<br />
: nofile, nproc<br />
<br />
For systemd the default limits are configured directly in the service's unit file that is shipped by OX and located at '''/usr/lib/systemd/system/open-xchange.service'''. The drop-in config to override or extend the default unit file is located at '''/etc/systemd/system/open-xchange.service.d/limits.conf'''. [https://www.freedesktop.org/software/systemd/man/systemd.exec.html Systemd.exec] shows a whole lot of options that can be used by admins to adapt the default service to their specific needs.<br />
<br />
<br />
== Warnings ==<br />
If the open-xchange startup script isn't able to set the proper limits on CentOS 6 due to e.g. hard limits being enforced via pam_limits you'll see the following warning in the console log during startup<br />
<br />
singlenode open-xchange # cat open-xchange-console.log<br />
/opt/open-xchange/sbin/open-xchange: line 115: ulimit: max user processes: cannot modify limit: Operation not permitted<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
== Evaluate JVM commandline options ==<br />
<br />
<br />
<br />
== Encoding detection during JVM startup ==<br />
Add details from https://bugs.open-xchange.com/show_bug.cgi?id=51074</div>Marenshttps://oxpedia.org/wiki/index.php?title=Sandbox:Middleware_Startup/sandbox&diff=23213Sandbox:Middleware Startup/sandbox2017-04-24T13:23:14Z<p>Marens: </p>
<hr />
<div><br />
== Systemd ==<br />
<br />
[[File:mw_start_sysd.png|1000px]]<br />
<br />
=== Systemd details explained ===<br />
<br />
For systemd the default limits are configured directly in the service's unit file that is shipped by OX and located at /usr/lib/systemd/system/open-xchange.service. The drop-in config to override or extend the default unit file is located at /etc/systemd/system/open-xchange.service.d/limits.conf. Systemd.exec shows a whole lot of options that can be used by admins to adapt the default service to their specific needs. <br />
<br />
==== 2 Drop-in Configs ====<br />
<br />
== System V / Upstart ==<br />
<br />
[[File:mw_start_sysv.png|1000px]]<br />
<br />
== System V / Upstart details explained ==<br />
<br />
=== Recheck resource limits ===<br />
<br />
Explain why it's needed and linkg to resource limits<br />
<br />
== Apply Resource limits ==<br />
<br />
{{AppSuite:ResourceLimits}}<br />
<br />
== Evaluate JVM commandline options ==<br />
<br />
<br />
<br />
== Encoding detection during JVM startup ==<br />
Add details from https://bugs.open-xchange.com/show_bug.cgi?id=51074</div>Marenshttps://oxpedia.org/wiki/index.php?title=File:Mw_start_sysd.png&diff=23212File:Mw start sysd.png2017-04-24T13:07:21Z<p>Marens: </p>
<hr />
<div></div>Marenshttps://oxpedia.org/wiki/index.php?title=Sandbox:Middleware_Startup/sandbox&diff=23211Sandbox:Middleware Startup/sandbox2017-04-24T13:07:08Z<p>Marens: /* Systemd */</p>
<hr />
<div><br />
= Middleware start =<br />
<br />
== Systemd ==<br />
<br />
[[File:mw_start_sysd.png|1000px]]<br />
<br />
== System V / Upstart ==<br />
<br />
[[File:mw_start_sysv.png|1000px]]<br />
<br />
== Details explained ==<br />
<br />
=== Evaluate JVM commandline options ===<br />
<br />
=== Apply Resource limits ===<br />
<br />
{{AppSuite:ResourceLimits}}<br />
<br />
=== Encoding detection during JVM startup ===<br />
Add details from https://bugs.open-xchange.com/show_bug.cgi?id=51074</div>Marenshttps://oxpedia.org/wiki/index.php?title=File:Mw_start_sysv.png&diff=23210File:Mw start sysv.png2017-04-24T13:06:39Z<p>Marens: Marens uploaded a new version of &quot;File:Mw start sysv.png&quot;</p>
<hr />
<div>middleware start with system v / upstart</div>Marenshttps://oxpedia.org/wiki/index.php?title=Sandbox:Middleware_Startup/sandbox&diff=23209Sandbox:Middleware Startup/sandbox2017-04-21T12:15:03Z<p>Marens: /* Encoding detection during JVM startup */</p>
<hr />
<div><br />
= Middleware start =<br />
<br />
== Systemd ==<br />
<br />
[[File:mw_start.png|1000px]]<br />
<br />
== System V / Upstart ==<br />
<br />
[[File:mw_start_sysv.png|1000px]]<br />
<br />
== Details explained ==<br />
<br />
=== Evaluate JVM commandline options ===<br />
<br />
=== Apply Resource limits ===<br />
<br />
{{AppSuite:ResourceLimits}}<br />
<br />
=== Encoding detection during JVM startup ===<br />
Add details from https://bugs.open-xchange.com/show_bug.cgi?id=51074</div>Marenshttps://oxpedia.org/wiki/index.php?title=Sandbox:Middleware_Startup/sandbox&diff=23208Sandbox:Middleware Startup/sandbox2017-04-20T23:07:31Z<p>Marens: </p>
<hr />
<div><br />
= Middleware start =<br />
<br />
== Systemd ==<br />
<br />
[[File:mw_start.png|1000px]]<br />
<br />
== System V / Upstart ==<br />
<br />
[[File:mw_start_sysv.png|1000px]]<br />
<br />
== Details explained ==<br />
<br />
=== Evaluate JVM commandline options ===<br />
<br />
=== Apply Resource limits ===<br />
<br />
{{AppSuite:ResourceLimits}}<br />
<br />
=== Encoding detection during JVM startup ===</div>Marenshttps://oxpedia.org/wiki/index.php?title=File:Mw_start_sysv.png&diff=23207File:Mw start sysv.png2017-04-20T23:05:56Z<p>Marens: middleware start with system v / upstart</p>
<hr />
<div>middleware start with system v / upstart</div>Marenshttps://oxpedia.org/wiki/index.php?title=Sandbox:Middleware_Startup/sandbox&diff=23206Sandbox:Middleware Startup/sandbox2017-04-20T23:05:14Z<p>Marens: /* System V / Upstart */</p>
<hr />
<div><br />
= Middleware start =<br />
<br />
== Systemd ==<br />
<br />
[[File:mw_start.png|1000px]]<br />
<br />
== System V / Upstart ==<br />
<br />
[[File:mw_start_sysv.png]]<br />
<br />
== Details explained ==<br />
<br />
=== 6 Evaluate java commandline options ===<br />
<br />
=== Apply Resource limits ===<br />
<br />
{{AppSuite:ResourceLimits}}</div>Marenshttps://oxpedia.org/wiki/index.php?title=Sandbox:Middleware_Startup/sandbox&diff=23205Sandbox:Middleware Startup/sandbox2017-04-20T08:15:49Z<p>Marens: /* Systemd */</p>
<hr />
<div><br />
= Middleware start =<br />
<br />
== Systemd ==<br />
<br />
[[File:mw_start.png|1000px]]<br />
<br />
== System V / Upstart ==<br />
<br />
[[File:mw_start_old.png]]<br />
<br />
== Details explained ==<br />
<br />
=== 6 Evaluate java commandline options ===<br />
<br />
=== Apply Resource limits ===<br />
<br />
{{AppSuite:ResourceLimits}}</div>Marens