AppSuite:DocumentsMonitoring

OX Documents / OX Documentconverter Monitoring

General

The Open-Xchange JMX offers the ability to fetch runtime information of the Java virtual machine, and about the Open-Xchange Middleware (aka groupware backend) including OX Documents, Documentconverter and Imageconverter.

For general information about the JMX interface see: OX_monitoring_interface

The usage of Jolokia to access monitoring data is described in the article Jolokia

Plugins are available for integration in the munin monitoring framework: OX_munin_scripts

OX Documentconverter

Monitoring data for OX Documentconverter can be obtained after installation of the package open-xchange-munin-scripts-jolokia. It uses Jolokia if available and configured.

The following table shows all available Documentconverter server metrics, provided via the JMX interface:

Documentconverter server metrics
Metrics Name	Description	Instance / LongTerm	Remarks
CacheEntryCount	The current count of entries, stored within the persistent cache	Instance
CacheFreeVolumeSize	The current free size in bytes of the volume, the cache root directory is located on. The size is calculated on a volume block size basis to reflect the correct size of the free volume.	Instance
CacheHitRatio	The ratio of successful job results that are retrieved from the cache against the total number of processed jobs as simple quotient	Instance
CacheOldestEntrySeconds	The age of the oldest entry within the cache in seconds	Instance	From 7.10.3 on
CachePersistentSize	The current size in bytes of all persistent entries, stored within the cache directory. The size is calculated on a volume block size basis to reflect the correct summed up size.	Instance
ConversionCount	The total number of all physical ReaderEngine and PDFTool conversions	Instance
ConversionCountPDFTool	The total number of all physical PDFTool conversions	Instance
ConversionCountReaderEngine	The total number of all physical ReaderEngine conversions	Instance
ConversionCountWithMultipleRuns	The total number of physical ReaderEngine conversions, that needed a second, new ReaderEngine instance	Instance
ConversionErrorsGeneral	The total number of physical ReaderEngine conversions, that resulted in a general error	Instance
ConversionErrorsTimeout	The total number of physical ReaderEngine conversions, that resulted in a tiemout error	Instance
ConversionErrorsDisposed	The total number of physical ReaderEngine conversions, that resulted in a disposed error (ReaderEngine instance vanished during conversion)	Instance
ConversionErrorsPassword	The total number of physical ReaderEngine conversions, that resulted in a password error	Instance
ConversionErrorsNoContent	The total number of physical ReaderEngine conversions, that resulted in a no content error (e.g. source document of 0 bytes length)	Instance
ConversionErrorsOutOfMemory	The total number of physical ReaderEngine conversions, that resulted in an out of memory error	Instance
ConversionErrorsPDFTool	The total number of physical PDFTool conversions, that resulted in an error	Instance
ErrorCacheHitRatio	The ratio of job errors that are retrieved from the errror cache against the total number of job errors as simple quotient	Instance
JobErrorsTimeout	The total number of jobs, that returned with a timeout error	Instance
JobErrorsTotal	The total number of jobs, that returned with an error	Instance
JobsProcessed	The total number of jobs that have been processed	Instance
MedianConversionTimeMillis_Background	The median time in milliseconds, a physical ReaderEngine conversion with background priority lasted, measured from the beginning of the last reset/initialization on (default period: 5 minutes)	Instance
MedianConversionTimeMillis_Low	The median time in milliseconds, a physical ReaderEngine conversion with low priority lasted, measured from the beginning of the last reset/initialization on (default period: 5 minutes)	Instance
MedianConversionTimeMillis_Medium	The median time in milliseconds, a physical ReaderEngine conversion with medium priority lasted, measured from the beginning of the last reset/initialization on (default period: 5 minutes)	Instance
MedianConversionTimeMillis_High	The median time in milliseconds, a physical ReaderEngine conversion with high priority lasted, measured from the beginning of the last reset/initialization on (default period: 5 minutes)	Instance
MedianConversionTimeMillis_Instant	The median time in milliseconds, a physical ReaderEngine conversion with instant priority lasted, measured from the beginning of the last reset/initialization on (default period: 5 minutes)	Instance
MedianConversionTimeMillis_Total	The median time in milliseconds, a physical ReaderEngine conversion lasted, measured from the beginning of the last reset/initialization on (default period: 5 minutes)	Instance
MedianJobTimeMillis_Background	The median time in milliseconds, a job with background priority needed to complete, measured from the beginning of the last reset/initialization on (default period: 5 minutes)	Instance
MedianJobTimeMillis_Low	The median time in milliseconds, a job with low priority needed to complete, measured from the beginning of the last reset/initialization on (default period: 5 minutes)	Instance
MedianJobTimeMillis_Medium	The median time in milliseconds, a job with medium priority needed to complete, measured from the beginning of the last reset/initialization on (default period: 5 minutes)	Instance
MedianJobTimeMillis_High	The median time in milliseconds, a job with high priority needed to complete, measured from the beginning of the last reset/initialization on (default period: 5 minutes)	Instance
MedianJobTimeMillis_Instant	The median time in milliseconds, a job with instant priority needed to complete, measured from the beginning of the last reset/initialization on (default period: 5 minutes)	Instance
MedianJobTimeMillis_Total	The median time in milliseconds, a job needed to complete, measured from the beginning of the last reset/initialization on (default period: 5 minutes)	Instance
MedianQueueTimeMillis_Background	The median time in milliseconds, a background job waited in the queue for processing, measured from the beginning of the last reset/initialization on (default period: 5 minutes)	Instance
MedianQueueTimeMillis_Low	The median time in milliseconds, a low priority job waited in the queue for processing, measured from the beginning of the last reset/initialization on (default period: 5 minutes)	Instance
MedianQueueTimeMillis_Medium	The median time in milliseconds, a medium priority job waited in the queue for processing, measured from the beginning of the last reset/initialization on (default period: 5 minutes)	Instance
MedianQueueTimeMillis_High	The median time in milliseconds, a high priority job waited in the queue for processing, measured from the beginning of the last reset/initialization on (default period: 5 minutes)	Instance
MedianQueueTimeMillis_Instant	The median time in milliseconds, an instant priority job waited in the queue for processing, measured from the beginning of the last reset/initialization on (default period: 5 minutes)	Instance
MedianQueueTimeMillis_Total	The median time in milliseconds, a job waited in the queue for processing, measured from the beginning of the last reset/initialization on (default period: 5 minutes)	Instance
PeakJobCountInQueue_Background	The peak number of background priority jobs in queue, measured from the beginning of the last reset/initialization on (default period: 5 minutes)	Instance
PeakJobCountInQueue_Low	The peak number of low priority jobs in queue, measured from the beginning of the last reset/initialization on (default period: 5 minutes)	Instance
PeakJobCountInQueue_Medium	The peak number of medium priority jobs in queue, measured from the beginning of the last reset/initialization on (default period: 5 minutes)	Instance
PeakJobCountInQueue_High	The peak number of high priority jobs in queue, measured from the beginning of the last reset/initialization on (default period: 5 minutes)	Instance
PeakJobCountInQueue_Instant	The peak number of instant priority jobs in queue, measured from the beginning of the last reset/initialization on (default period: 5 minutes)	Instance
PeakJobCountInQueue_Total	The peak number of jobs in queue, measured from the beginning of the last reset/initialization on (default period: 5 minutes)	Instance
PeakJobCountInAsyncQueue_Total	The peak number of jobs in the queue for asynchronous jobs, measured from the beginning of the last reset/initialization on (default period: 5 minutes)	Instance
PendingStatefulJobCount	The number of stateful job conversions for that no endConvert has been called up to now	Instance	From 7.10.3 on
QueueLimitHighReachedCountInstance	The accumulated number of times the job queue high limit has been reached for the current DocumentConverter server instance	Instance	From 7.10.3 on
QueueLimitLowReachedCountInstance	The accumulated number of times the job queue low limit has been reached for the current DocumentConverter server instance	Instance	From 7.10.3 on
QueueLimitHighReachedCountLongTerm	The accumulated number of times the job queue high limit has been reached for all previously started DocumentConverter server instances and the current DocumentConverter server instance	Long term	From 7.10.3 on
QueueLimitLowReachedCountLongTerm	The accumulated number of times the job queue low limit has been reached all previously started DocumentConverter server instances and the current DocumentConverter server instance	Long term	From 7.10.3 on
QueueTimeoutCountInstance	The accumulated number of times the job queue timeout has been reached for the current DocumentConverter server instance	Instance	From 7.10.3 on
QueueTimeoutCountLongTerm	The accumulated number of times the job queue timeout has been reached for all previously started DocumentConverter server instances and the current DocumentConverter server instance	Long term	From 7.10.3 on
RuntimeSecondsInstance	The runtime in seconds for the current DocumentConverter server instance	Instance
RuntimeSecondsLongTerm	The accumulated runtime in seconds for all previously started DocumentConverter server instances and the current DocumentConverter server instance	Long term
ScheduledJobCountInQueue	The current number of all scheduled, synchronous jobs within the queue, waiting to be processed	Instance	From 7.10.3 on
UserRequestCountInstance_Total	The number of all WebService requests, that have the user flag set to true, for the current DocumentConverter server instance	Instance
UserRequestCountInstance_Text	The number of text document related WebService requests, that have the user flag set to true, for the current DocumentConverter server instance	Instance
UserRequestCountInstance_Spreadsheet	The number of spreadsheet document related WebService requests, that have the user flag set to true, for the current DocumentConverter server instance	Instance
UserRequestCountInstance_Presentation	The number of presentation document related WebService requests, that have the user flag set to true, for the current DocumentConverter server instance	Instance
UserRequestCountInstance_Image	The number of image related WebService requests, that have the user flag set to true, for the current DocumentConverter server instance	Instance
UserRequestCountLongTerm_Total	The accumulated number of all WebService requests, that have the user flag set to true, for all previously started DocumentConverter server instances and the current DocumentConverter server instance	Long term
UserRequestCountLongTerm_Text	The accumulated number of text document related WebService requests, that have the user flag set to true, for all previously started DocumentConverter server instances and the current DocumentConverter server instance	Long term
UserRequestCountLongTerm_Spreadsheet	The accumulated number of spreadsheet document related WebService requests, that have the user flag set to true, for all previously started DocumentConverter server instances and the current DocumentConverter server instance	Long term
UserRequestCountLongTerm_Presentation	The accumulated number of presentation document related WebService requests, that have the user flag set to true, for all previously started DocumentConverter server instances and the current DocumentConverter server instance	Long term
UserRequestCountLongTerm_Image	The accumulated number of image related WebService requests, that have the user flag set to true, for all previously started DocumentConverter server instances and the current DocumentConverter server instance	Long term
UserRequestCountInstanceFailed_Total	The number of all WebService requests, that failed with an error and that have the user flag set to true, for the current DocumentConverter server instance	Instance
UserRequestCountInstanceFailed_Text	The number of text document related WebService requests, that failed with an error and that have the user flag set to true, for the current DocumentConverter server instance	Instance
UserRequestCountInstanceFailed_Spreadsheet	The number of spreadsheet document related WebService requests, that failed with an error and that have the user flag set to true, for the current DocumentConverter server instance	Instance
UserRequestCountInstanceFailed_Presentation	The number of presentation document related WebService requests, that failed with an error and that have the user flag set to true, for the current DocumentConverter server instance	Instance
UserRequestCountInstanceFailed_Image	The number of image related WebService requests, that failed with an error and that have the user flag set to true, for the current DocumentConverter server instance	Instance
UserRequestCountLongTermFailed_Total	The accmulated number of all WebService requests, that failed with an error and that have the user flag set to true, for all previously started DocumentConverter server instances and the current DocumentConverter server instance	Long term
UserRequestCountLongTermFailed_Text	The accmulated number of text document related WebService requests, that failed with an error and that have the user flag set to true, for all previously started DocumentConverter server instances and the current DocumentConverter server instance	Long term
UserRequestCountLongTermFailed_Spreadsheet	The accmulated number of spreadsheet document related WebService requests, that failed with an error and that have the user flag set to true, for all previously started DocumentConverter server instances and the current DocumentConverter server instance	Long term
UserRequestCountLongTermFailed_Presentation	The accmulated number of presentation document related WebService requests, that failed with an error and that have the user flag set to true, for all previously started DocumentConverter server instances and the current DocumentConverter server instance	Long term
UserRequestCountLongTermFailed_Image	The accmulated number of image related WebService requests, that failed with an error and that have the user flag set to true, for all previously started DocumentConverter server instances and the current DocumentConverter server instance	Long term
UserRequestSuccessRatioInstance_Total	The ratio of successful WebService requests against failed WebService requests, that have the user flag set to true, for the current DocumentConverter server instance	Instance
UserRequestSuccessRatioInstance_Text	The ratio of successful text document related WebService requests against failed text document related WebService requests, that have the user flag set to true, for the current DocumentConverter server instance	Instance
UserRequestSuccessRatioInstance_Spreadsheet	The ratio of successful spreadsheet document related WebService requests against failed spreadsheet document related WebService requests, that have the user flag set to true, for the current DocumentConverter server instance	Instance
UserRequestSuccessRatioInstance_Presentation	The ratio of successful presentation document related WebService requests against failed presentation document related WebService requests, that have the user flag set to true, for the current DocumentConverter server instance	Instance
UserRequestSuccessRatioInstance_Image	The ratio of successful image related WebService requests against failed image related WebService requests, that have the user flag set to true, for the current DocumentConverter server instance	Instance
UserRequestSuccessRatioLongTerm_Total	The accumulated ratio of successful WebService requests against failed WebService requests, that have the user flag set to true, for all previously started DocumentConverter server instances and the current DocumentConverter server instance	Long term
UserRequestSuccessRatioLongTerm_Text	The accumulated ratio of successful text document related WebService requests against failed text document related WebService requests, that have the user flag set to true, for all previously started DocumentConverter server instances and the current DocumentConverter server instance	Long term
UserRequestSuccessRatioLongTerm_Spreadsheet	The accumulated ratio of successful spreadsheet document related WebService requests against failed spreadsheet document related WebService requests, that have the user flag set to true, for all previously started DocumentConverter server instances and the current DocumentConverter server instance	Long term
UserRequestSuccessRatioLongTerm_Presentation	The accumulated ratio of successful presentation document related WebService requests against failed presentation document related WebService requests, that have the user flag set to true, for all previously started DocumentConverter server instances and the current DocumentConverter server instance	Long term
UserRequestSuccessRatioLongTerm_Image	The accumulated ratio of successful image related WebService requests against failed image related WebService requests, that have the user flag set to true, for all previously started DocumentConverter server instances and the current DocumentConverter server instance	Long term

Here is a list of sample data provided by

# /opt/open-xchange/sbin/showruntimestats -y -p 9998|sed -e 's/com.openexchange.documentconverter:name=DocumentConverterInformation,//g' | sort

CacheEntryCount = 10258
CacheFreeVolumeSize = 2709409792
CacheHitRatio = 0.9642857142857143
CacheOldestEntrySeconds = 1998029
CachePersistentSize = 564215808
ConversionCount = 1
ConversionCountPDFTool = 0
ConversionCountReaderEngine = 1
ConversionCountWithMultipleRuns = 0
ConversionErrorsDisposed = 0
ConversionErrorsGeneral = 0
ConversionErrorsNoContent = 0
ConversionErrorsOutOfMemory = 0
ConversionErrorsPassword = 1
ConversionErrorsPDFTool = 0
ConversionErrorsTimeout = 0
ErrorCacheHitRatio = 0.0
JobErrorsTimeout = 0
JobErrorsTotal = 1
JobsProcessed = 28
MedianConversionTimeMillis_Background = 0
MedianConversionTimeMillis_High = 0
MedianConversionTimeMillis_Instant = 0
MedianConversionTimeMillis_Low = 0
MedianConversionTimeMillis_Medium = 0
MedianConversionTimeMillis_Total = 0
MedianJobTimeMillis_Background = 0
MedianJobTimeMillis_High = 0
MedianJobTimeMillis_Instant = 0
MedianJobTimeMillis_Low = 0
MedianJobTimeMillis_Medium = 0
MedianJobTimeMillis_Total = 0
MedianQueueTimeMillis_Background = 0
MedianQueueTimeMillis_High = 0
MedianQueueTimeMillis_Instant = 0
MedianQueueTimeMillis_Low = 0
MedianQueueTimeMillis_Medium = 0
MedianQueueTimeMillis_Total = 0
PeakJobCountInAsyncQueue_Total = 0
PeakJobCountInQueue_Background = 0
PeakJobCountInQueue_High = 0
PeakJobCountInQueue_Instant = 0
PeakJobCountInQueue_Low = 0
PeakJobCountInQueue_Medium = 0
PeakJobCountInQueue_Total = 0
PendingStatefulJobCount = 0
QueueLimitHighReachedCountInstance = 33
QueueLimitHighReachedCountLongTerm = 66
QueueLimitLowReachedCountInstance = 33
QueueLimitLowReachedCountLongTerm = 66
QueueTimeoutCountInstance = 22
QueueTimeoutCountLongTerm = 44
RuntimeSecondsInstance = 6707
RuntimeSecondsLongTerm = 28607051
ScheduledJobCountInQueue = 0
UserRequestCountInstanceFailed_Image = 0
UserRequestCountInstanceFailed_Presentation = 0
UserRequestCountInstanceFailed_Spreadsheet = 0
UserRequestCountInstanceFailed_Text = 0
UserRequestCountInstanceFailed_Total = 0
UserRequestCountInstance_Image = 0
UserRequestCountInstance_Presentation = 0
UserRequestCountInstance_Spreadsheet = 0
UserRequestCountInstance_Text = 0
UserRequestCountInstance_Total = 0
UserRequestCountLongTermFailed_Image = 0
UserRequestCountLongTermFailed_Presentation = 10694
UserRequestCountLongTermFailed_Spreadsheet = 2598
UserRequestCountLongTermFailed_Text = 6369
UserRequestCountLongTermFailed_Total = 19661
UserRequestCountLongTerm_Image = 0
UserRequestCountLongTerm_Presentation = 121394
UserRequestCountLongTerm_Spreadsheet = 33453
UserRequestCountLongTerm_Text = 130801
UserRequestCountLongTerm_Total = 285648
UserRequestSuccessRatioInstance_Image = 1.0
UserRequestSuccessRatioInstance_Presentation = 1.0
UserRequestSuccessRatioInstance_Spreadsheet = 1.0
UserRequestSuccessRatioInstance_Text = 1.0
UserRequestSuccessRatioInstance_Total = 1.0
UserRequestSuccessRatioLongTerm_Image = 1.0
UserRequestSuccessRatioLongTerm_Presentation = 0.911906684020627
UserRequestSuccessRatioLongTerm_Spreadsheet = 0.9223388036947359
UserRequestSuccessRatioLongTerm_Text = 0.9513077117147423
UserRequestSuccessRatioLongTerm_Total = 0.9311705315633227

OX Documentconverter and munin

The Documentconverter uses a different port to provide monitoring data via Jolokia. The corresponding oxJolokiaURL is has to be configured with an entry in the settings.

[ox_documentconverter*]
env.oxJolokiaUrl http://localhost:8008/monitoring/jolokia

OX Documents

Monitoring data for OX Documents can be obtained after installation of the package open-xchange-documents-monitoring.

Here is a list of sample data provided by

# /opt/open-xchange/sbin/showruntimestats -f
DocumentsSizeMedian_Spreadsheet = 0
DocumentsCreated_Presentation = 2
DocumentsCreated_Spreadsheet = 2
RestoreDocCurrent = 40
RestoreDocRestoredSuccess = 0
RestoreDocRestoredFailure = 0
DocumentsSizeMedian_Text = 0
DocumentsSizeMedian_Presentation = 0
RestoreDocCreated_Total = 40
RestoreDocRemoved_Total = 0
DocumentsSizeMedian_Total = 0
DocumentsCreated_Total = 4
DocumentsCreated_Text = 0
DocumentsClosed_Text_Timeout = 0
DocumentsSaved_Text_Total = 2
DocumentsSaved_Text_Close = 2
DocumentsSaved_Text_100ops = 0
DocumentsSaved_Text_15mins = 0
DocumentsOperations_Text_Incoming = 8
DocumentsOperations_Text_Distributed = 0
DocumentsOpened_Text_Total = 2
DocumentsOpened_Text_OOXML = 1
DocumentsOpened_Text_Binary = 0
DocumentsOpened_Text_ODF = 1
DocumentsClosed_Text_Total = 2
DocumentsOpened_Spreadsheet_Total = 22
DocumentsOpened_Spreadsheet_OOXML = 16
DocumentsOpened_Spreadsheet_Binary = 0
DocumentsOpened_Spreadsheet_ODF = 6
DocumentsClosed_Spreadsheet_Total = 19
DocumentsClosed_Spreadsheet_Timeout = 0
DocumentsSaved_Spreadsheet_Total = 10
DocumentsSaved_Spreadsheet_Close = 10
DocumentsSaved_Spreadsheet_100ops = 0
DocumentsSaved_Spreadsheet_15mins = 0
DocumentsOperations_Spreadsheet_Incoming = 46
DocumentsOperations_Spreadsheet_Distributed = 0

DCS

JMX and access via Jolokia is disabled by default.

For the required settings in dcs.properties see https://documentation.open-xchange.com/components/documents/documents-collaboration/config/7.10.3/

jmx.enabled=true
jmx.username
jmx.password

The default port is 9994. So the JMX connection can be established via

service:jmx:rmi:///jndi/rmi://10.20.28.108:9994/jmxrmi

The DCS offers Spring Boot Actuator functionality for management and monitoring. By default endpoints are not accessible via Jolokia (except info and health). Additional endpoints have to be exposed and user credentials have to be configured.

management.endpoints.web.exposure.include=*
spring.security.user.name=ausername
spring.security.user.password=thepass

For details see https://docs.spring.io/spring-boot/docs/2.0.x/reference/html/production-ready-endpoints.html

Some examples for useful endpoints:

# curl  "http://localhost:8004/dcs/management/health"
{"status":"UP"}

Note that the health endpoint returns additional data if the call is authenticated.

# curl "http://ausername:thepass@localhost:8004/dcs/management/health"
{"status":"UP","details":{"db":{"status":"UP","details":...

The Spring Boot framework offers general metrics about system, JVM, ... As an example the metric process.cpu.usage is picked from the list at https://docs.spring.io/spring-boot/docs/2.0.10.BUILD-SNAPSHOT/actuator-api/html/#metrics :

# curl "http://ausername:thepass@localhost:8004/dcs/management/metrics/process.cpu.usage" 
{"name": "process.cpu.usage","measurements":[{"value": 0.03215434083601286, ...

These general metrics can also be retrieved in prometheus format

# curl "http://ausername:thepass@localhost:8004/dcs/management/prometheus"
# HELP jvm_threads_peak_threads ...
# TYPE jvm_threads_peak_threads gauge
jvm_threads_peak_threads 58.0
...

Data specific to ActiveMQ is explained in the ActiveMQ MBeans Reference.

Further Reading

Further details about accessing JMX data via Jolokia are described in Middleware documentation.

A call derived from this for OX Documentconverter looks like

$ curl http://yourname:yourpassword@localhost:8008/monitoring/jolokia/read/com.openexchange.documentconverter:name=DocumentConverterInformation/JobsProcessed
{"timestamp":1471868314,"status":200,"request":{"mbean":"com.openexchange.documentconverter:name=DocumentConverterInformation","attribute":"JobsProcessed","type":"read"},"value":6616}