AppSuite:Oxsysreport

Creating an OX Support Tarball

The /opt/open-xchange/sbin/oxsysreport tool will collect a lot of useful log and system information into a tar archive. This so called OX Support Tarball contains Apache and OX configurations, OX logs and system information. This and a qualified error report makes sure that our support department get all needed details for fast effective help. We always recommend to encrypt your e-mail with our Support Key when attaching this tarball.

Parameter Overview

Option	Default Values	Possible Values	Required Tools
--tempdir	/tmp	<path>
--logdir	ox_support_infos-<timestamp>	<folder name>
--log-archive	/tmp/ox_support_infos-<timestamp>.tar.gz	<path to tarball>
--keep-tmp-files	false	true false
--thread-dump	false	true false
--heap-dump	false	true false	OX heapdump or jmap and sudo (Only if not IBM Java)
--report-client	false	true false	Runs the OX Report Client
--skip-compression	false	true false
--skip-system-tools	false	true false
--exclude-old-logs	disabled	0 ... n Days
--exclude-ox-secrets	false (true up from OX 7.6.1)	true false
-h,--help	disabled

Required Tools

We recommend to check if all required tools are installed before an incident happens. In case you are running IBM Java on e.g. SuSE Linux Enterprise, heap dumps will be created differently. Please note, we do not list package names which usually get installed in a default installation. Please note, up from OX 7.6.1 we will use instead of 'jmap' the '/opt/open-xchange/sbin/heapdump' tool for heap dump creation by default, if supported by your JVM.

Tool	Debian	SuSE	Redhead	CentOS	Univention
bzip2	bzip2
jmap	sun-java<version>-jdk openjdk-<version>-jdk	N/A	java-<version>-openjdk-devel	java-<version>-openjdk-devel	openjdk-<version>-jdk
sudo	sudo

Please note, on Univention systems you have to enable the 'Unmaintained' Repositories checkbox via 'App Center' --> 'Repository Settings' to get access to package listed above.

Features

Directory for Temporary Process Data (--tempdir)

If, e.g. due to space limitations, you need to switch from /tmp to a different drive for collecting and processing the data, you can choose an alternative but existing path here.

Output Log Directory Name (--logdir)

We do not recommend to use this option, but you can choose the output log directory name which is also root folder of the OX Support Tarball. Please note, this temporary folder will be removed by default and you need to choose an not existing directory name.

OX Support Tarball Name and Storage Location (--log-archive)

This allows to choose an alternative tarball name and storage location. Please note, the script will check for and break on already existing file arguments.

Keep Temporary Process Data (--keep-tmp-files)

If you like to observe the data collection of the log directory without unpacking the tarball or to add additional files for repacking, feel free to use this option. The log directory path will be printed at the first lines of the oxsysreport output.

This information is valid from 7.6.0 on.

Create Thread Dumps (--thread-dump)

This option normally should only be used in case it is requested by our Support department. The collected thread dumps may help to identify the root cause of long running threads, critical load and very high CPU usage. In case of an incident with very high CPU usage and a lack of resources, please consider the addditional usage of --skip-compression.

Further Details of Thread Dump Creation

This option triggers the thread dump creation via 'kill -3' and logs the related top output into the '/commands/top -d 3 -bHn 5 (thread-dump)' file. The thread dump on systems other than IBM Java will be stored within the /var/log/open-xchange/open-xchange-console.log, on IBM Java they will be stored separately inside the support tarball as /tmp/javacore.<timestamp>.txt.

This information is valid from 7.6.0 on.

Create Heap Dumps (--heap-dump)

This option normally should only be used in case it is requested by our Support department. The collected heap dumps could be very helpful in memory or OX service outage incidents. Please keep in mind that you have to call oxsysreport with this option before restarting the service.

First you will get a warning dialog with some system details and with the choice to proceed or to cancel. While creating Java heap dumps, your JVM will be stopped and the OX Service is not functional. This step may take several minutes and you might have to manually restart the open-xchange service afterwards. You also have to make sure that you have enough free disk space for processing the data in directories which are defined by --tempdir and --log-archive. Depending on your max heap size '-Xmx' defined by your ox-scriptconf.sh of your groupware Java Virtual Machine and amount of logs, several gigabyte could be needed!

Further Details of Heap Dump Creation

This option triggers the creation of an heap dump which will be stored on systems other than IBM Java Systems within '/commands/jmap -dump:file (heap-dump).bz2' and '/commands/jmap -histo (heap-dump)'. The two jmap files beside without '(heap-dump)' marker contain the stdout/stderr of jmap by running this command. Please note, the '.bz2' suffix (and compression) will be suppressed by the --skip-compression option.

On IBM Java Systems the jmap command is not available and 'kill -ABRT' will do the job instead. This will kill the JVM completely by writing the heap dump. Afterwards the dump will be moved within the tarball and processed by the tool jextract. Depending on --skip-compression option jextract will compress it as zip or not. The heap dump will be located in /tmp/core.<timestamp>.dmp.zip within the tarball. The stdout/stderr of jextract will be written to '/commands/jextract core.<timestamp>.dmp (heap-dump)'. On IBM Java the OX groupware service will be restarted by this script.

This information is valid from 7.8.2 on.

Run OX Report Client (--report-client)

The OXReportClient generates an overview about features and users from your database. On larger installations this tool may take very long and so we disabled it by default.

This information is valid from 7.6.0 on.

Skip Support Tarball Compression (--skip-compression)

Please use this option only in case your system is affected by very high CPU usage and the oxsysreport compression stuck by the lack of resources. If this option is set, no compression takes place and this will affect the OX Support Tarball, as well as nested tarballs and possible heap dumps. The resulting tarball can get huge and we recommend to keep the --exclude-old-logs option in mind to also drop potentially outdated logs. Please always compress this tarball before handing it over to our support department.

This information is valid from 7.6.0 on.

Exclude Old OX Log Files from Support Tarball (--exclude-old-logs)

We recommend to use this option if you are able to locate the period of a specific issue or by handing over an follow up tarball. This option will exclude all separate log files beyond the modified period which are stored on your system in /var/log/open-xchange or /var/log/univention. This option does not affect mandatory files like e.g. open-xchange-console.log. If logs are dropped they are listed in the generated overview file /modified-and-excluded-files.txt of the tarball.

This information is valid from 7.6.0 on.

Remove Confidential Details from OX Configuration Files (--exclude-ox-secrets)

This option removes all unexpected, as well as whole confidential known files from the /opt/open-xchange/etc/ copy of the support tarball. Also confidential information like passwords and keys will be replaced by a white and blacklist heuristic with a <REMOVED BY OXSYSREPORT> marker. The replacement works only on OX properties files and the /modified-and-excluded-files.txt file will contain the details of affected files.

Examples

Please keep in mind that the following examples just give a rough overview and arguments should be adapted to your own needs.

Default Usage

   root@server# /opt/open-xchange/sbin/oxsysreport

This information is valid from 7.6.0 on.

Example Output on Debian7

 Debian system detected
 Creating temporary log directory: /tmp/ox_support_infos-20140530-074449
 Collecting output of system tools into /tmp/ox_support_infos-20140530-074449/commands
 Collecting httpd configuration into /tmp/ox_support_infos-20140530-074449/etc/apache2
 Collecting Open-Xchange configurations into /tmp/ox_support_infos-20140530-074449/opt/open-xchange/etc
 Collecting other system configurations into /tmp/ox_support_infos-20140530-074449/etc
 Collecting Open-Xchange logs into /tmp/ox_support_infos-20140530-074449/open-xchange-logs.tar.bz2
 Creating an overview about modified and excluded files: /tmp/ox_support_infos-20140530-074449/modified-and-excluded-files.txt
 Creating final archive: /tmp/ox_support_infos-20140530-074449.tar.gz

This information is valid from 7.6.0 on.

Exclude all Logs Older than Two Days and Remove OX Configuration Secrets

   root@server# /opt/open-xchange/sbin/oxsysreport --exclude-old-logs 2 --exclude-ox-secrets true

Example Output on Debian7

 Debian system detected
 Creating temporary log directory: /tmp/ox_support_infos-20140530-074504
 Collecting output of system tools into /tmp/ox_support_infos-20140530-074504/commands
 Collecting httpd configuration into /tmp/ox_support_infos-20140530-074504/etc/apache2
 Collecting Open-Xchange configurations into /tmp/ox_support_infos-20140530-074504/opt/open-xchange/etc
 Removing secrets from Open-Xchange configurations: /tmp/ox_support_infos-20140530-074504/opt/open-xchange/etc
 Collecting other system configurations into /tmp/ox_support_infos-20140530-074504/etc
 Collecting Open-Xchange logs into /tmp/ox_support_infos-20140530-074504/open-xchange-logs.tar.bz2
 Creating an overview about modified and excluded files: /tmp/ox_support_infos-20140530-074504/modified-and-excluded-files.txt
 Creating final archive: /tmp/ox_support_infos-20140530-074504.tar.gz

This information is valid from 7.6.0 on.

Create Thread Dumps, Skip Compression and Include only Today's Logs

   root@server# /opt/open-xchange/sbin/oxsysreport --thread-dump true \
                          --skip-compression true \
                          --exclude-old-logs 0

Example Output on Debian7

 Debian system detected
 Creating temporary log directory: /tmp/ox_support_infos-20140530-074517
 Collecting output of system tools into /tmp/ox_support_infos-20140530-074517/commands
 Collecting httpd configuration into /tmp/ox_support_infos-20140530-074517/etc/apache2
 Collecting Open-Xchange configurations into /tmp/ox_support_infos-20140530-074517/opt/open-xchange/etc
 Collecting other system configurations into /tmp/ox_support_infos-20140530-074517/etc
 Creating Java thread dumps: 1 2 3 4 5
 Collecting Open-Xchange logs into /tmp/ox_support_infos-20140530-074517/open-xchange-logs.tar
 Creating an overview about modified and excluded files: /tmp/ox_support_infos-20140530-074517/modified-and-excluded-files.txt
 Creating final archive: /tmp/ox_support_infos-20140530-074517.tar

This information is valid from 7.6.0 on.

Create a Heap Dump, Choose a Different Storage Location for the Tarball and Process Data

   root@server# /opt/open-xchange/sbin/oxsysreport --heap-dump true \
             --tempdir /var/tmp/ \
             --log-archive /root/ox_support_infos-$(date +%Y%m%d-%H%M%S).tar.gz

Example Output on Debian7

 ###############################################################################
                                   WARNING
 While creating Java heap dumps your JVM will be stopped and the OX Service is
 not functional. This step may take several minutes and it could happen that you
 need to restart the open-xchange service afterwards manually again.
 
 Please also make sure that you have enough free disk space for processing the
 data in directories which are defined by '--tempdir' and '--log-archive',
 depending on your ox-scriptconf.sh '-Xmx' size of your groupware Java virtual
 machine and amount of logs, several gigabyte could be needed!
 
 System Info:
 Groupware Max Heap Size: -Xmx1024m
 OX Log Storage: 2.5M    /var/log/open-xchange
 
 Filesystem     Size  Used Avail Use% Mounted on
 /dev/sdb1      500G  100G  400G  20% /var     Processing data within: /var/tmp/ox_support_infos-20140530-081700
 /dev/sdc1      500G  50G   450G  10% /root    Final tarball storage: /root/ox_support_infos-20140530-081700.tar.gz
 
 ###############################################################################
 Press ENTER to continue or [ctrl+c] to cancel...
 
 Debian system detected
 Creating temporary log directory: /var/tmp/ox_support_infos-20140530-081700
 Collecting output of system tools into /var/tmp/ox_support_infos-20140530-081700/commands
 Collecting httpd configuration into /var/tmp/ox_support_infos-20140530-081700/etc/apache2
 Collecting Open-Xchange configurations into /var/tmp/ox_support_infos-20140530-081700/opt/open-xchange/etc
 Collecting other system configurations into /var/tmp/ox_support_infos-20140530-081700/etc
 Creating Java heap dump of Open-Xchange groupware...
   Dumping heap to /var/tmp/ox_support_infos-20140530-081700/commands/jmap -dump:file (heap-dump) ...
   Heap dump file created
 Compressing Java heap dump: /var/tmp/ox_support_infos-20140530-081700/commands/jmap -dump:file (heap-dump).bz2
 Collecting Open-Xchange logs into /var/tmp/ox_support_infos-20140530-081700/open-xchange-logs.tar.bz2
 Creating an overview about modified and excluded files: /var/tmp/ox_support_infos-20140530-081700/modified-and-excluded-files.txt
 Creating final archive: /root/ox_support_infos-20140530-081700.tar.gz