AppSuite:S3 File Store

From Open-Xchange
Revision as of 08:03, 9 June 2021 by WolfgangRosenauer (talk | contribs) (→‎Overview)
(diff) ← Older revision | Latest revision (diff) | Newer revision → (diff)

Overview

Besides ordinary local- or NFS-filesystem-based filestores, OX may also be used with a cloud storage solution offering an S3 interface. The required functionality is added with the package open-xchange-filestore-s3. The filestore integration was tested against the regular Amazon Simple Storage Service (S3), however, other object storage implementation offering an S3-compatible API should work, too. The list of officially supported S3 storages can be found on the system requirements page.

Installation & Configuration

This chapter shows which components need to be installed and where configuration is done.

Install on OX App Suite

Debian GNU/Linux 11.0

Add the following entry to /etc/apt/sources.list.d/open-xchange.list if not already present: 

deb https://software.open-xchange.com/products/appsuite/stable/backend/DebianBullseye/ /
# if you have a valid maintenance subscription, please uncomment the 
# following and add the ldb account data to the url so that the most recent
# packages get installed
# deb https://[CUSTOMERID:PASSWORD]@software.open-xchange.com/products/appsuite/stable/backend/updates/DebianBullseye/ /

and run 

$ apt-get update
$ apt-get install open-xchange-filestore-s3

Debian GNU/Linux 12.0

Add the following entry to /etc/apt/sources.list.d/open-xchange.list if not already present: 

deb https://software.open-xchange.com/products/appsuite/stable/backend/DebianBookworm/ /
# if you have a valid maintenance subscription, please uncomment the 
# following and add the ldb account data to the url so that the most recent
# packages get installed
# deb https://[CUSTOMERID:PASSWORD]@software.open-xchange.com/products/appsuite/stable/backend/updates/DebianBookworm/ /

and run 

$ apt-get update
$ apt-get install open-xchange-filestore-s3



Configuration

S3 filestores can be configured in the configuration file filestore-s3.properties once they have been registered on the server with the registerfilestore command.


Register an S3 filestore

Similarly to local- or NFS-filesystem-based filestores, S3 filestore need to be registered using the commandline tool registerfilestore before they can be used. The scheme-part of the filestore URI must be set to s3 in this case. Besides the common s3:// prefix of such filestore URIs, the remaining authority component of the URI specifies an identifier of the filestore that will be used to refer to this filestore registration in the configuration file. It's recommended to use the bucket name here, for example: 

root@ox01:/opt/open-xchange/sbin# ./registerfilestore -A oxadminmaster -P secret -t s3://ox-filestore-s3 -s 1048576000 -x 5000
filestore 7433 registered


Configure the filestore

Additional configuration for each registered filestore is done in the configuration file filestore-s3.properties. To map configuration properties to specific filestore registrations, the filestore identifier corresponding to the defined authority part of the filestore URI is used in the property names. Please refer to the inline documentation or the examples below for details.

Note: In most cases, you want to use the same configuration throughout the whole ox cluster, so you should make sure to use the same filestore-s3.properties configuration file on all nodes.


Step-by-step guide to setup an AWS S3 storage

The following walktrhorugh outlines all steps necessary to complete the setup of a S3 filestore running on the regular Amazon Simple Storage Service (S3).


Prerequisites

  • Installed open-xchange-filestore-s3 package
  • A valid account for Amazon S3
  • AWS access key, e.g. AKIAIOSFODNN7EXAMPLE
  • AWS secret key, e.g. wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY


Create a bucket

  • Login to the S3 Management Console at https://console.aws.amazon.com/s3/
  • Click the Create Bucket button
  • Select a bucket name of your choice, e.g. ox-filestore-s3
  • Select the region according to your needs, e.g. EU (Ireland)
  • Click Create


Register the filestore

  • Open a terminal on a running backend server of your OX cluster
  • Navigate to the directory where the commandline utilities are installed, usually at /opt/open-xchange/sbin
  • Run the registerfilestore tool using a storepath URL starting with the s3 scheme (use s3:// as prefix literally) followed by the filestore ID (the bucket name seems most appropriate here, however, you can still use a different name), e.g.:
./registerfilestore -A oxadminmaster -P secret -t s3://ox-filestore-s3 -s 1048576000 -x 5000

(execute "./registerfilestore -h" for more information regarding possible commandline arguments)

  • The internal ID of the new filestore is printed out


Configure the filestore

  • Open the configuration file filestore-s3.properties in your favourite editor
  • Insert a new set of properties using the filestore ID assigned during registration of the filestore in the previous step, e.g. ox-filestore-s3:
com.openexchange.filestore.s3.ox-filestore-s3.endpoint=
com.openexchange.filestore.s3.ox-filestore-s3.bucketName=
com.openexchange.filestore.s3.ox-filestore-s3.region=
com.openexchange.filestore.s3.ox-filestore-s3.pathStyleAccess=
com.openexchange.filestore.s3.ox-filestore-s3.accessKey=
com.openexchange.filestore.s3.ox-filestore-s3.secretKey=
com.openexchange.filestore.s3.ox-filestore-s3.encryption=
com.openexchange.filestore.s3.ox-filestore-s3.signerOverride=
com.openexchange.filestore.s3.ox-filestore-s3.chunkSize=

com.openexchange.filestore.s3.ox-filestore-s3.endpoint=https://s3-eu-west-1.amazonaws.com
  • Set the bucket name for the filestore, e.g.:
com.openexchange.filestore.s3.ox-filestore-s3.bucketName=ox-filestore-s3
  • Define the location constraint according to the chosen region - valid values are listed in the comments of the properties file - e.g.:
com.openexchange.filestore.s3.ox-filestore-s3.region=eu-west-1

com.openexchange.filestore.s3.ox-filestore-s3.pathStyleAccess=false
  • Specify your API access and secret keys, e.g.:
com.openexchange.filestore.s3.ox-filestore-s3.accessKey=AKIAIOSFODNN7EXAMPLE
com.openexchange.filestore.s3.ox-filestore-s3.secretKey=wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY
  • Configure encryption as needed (none in this example):
com.openexchange.filestore.s3.ox-filestore-s3.encryption=none
  • Specify the signing algorithm if applicable. Please mind that this setting needs to be adjusted to "AWSS3V4SignerType" when targeting the newer "v4-only" regions like Frankfurt or Beijing.
com.openexchange.filestore.s3.ox-filestore-s3.encryption=S3SignerType
  • Optionally override the default minimum chunk size to use when doing multipart uploads.
com.openexchange.filestore.s3.ox-filestore-s3.chunkSize=5MB

Create contexts using the filestore

  • New contexts that should use the configured filestore can be created by suppliying the ID of the registered filestore in the createcontext commandline tool. Use the internal ID returned by the previously used registerfilestore command executed above, e.g., assuming the ID returned from the registerfilestore command was 6:
./createcontext -A oxadminmaster -P secret -u oxadmin -d "Context Admin" -g Admin -s User -p secret -e oxadmin@example.com -q 1000 -L example.com --access-combination-name=all -F 6


Check the configuration

  • Login to the App Suite webinterface with a user from a context using the S3 filestore
  • Upload a new file in the Drive module and download it again to verify basic functionality
  • Check the S3 management console to verify the creation of the new file; here, the file should show up under a virtual prefix based on the parent context
Retrieved from "https://wiki.open-xchange.com/wiki/index.php?title=AppSuite:S3_File_Store&oldid=26023"