AppSuite:UI build system

Build System

Abstract: This document describes the build system of OX App Suite. It is intended for app developers who use the build system to create apps as well as for OX App Suite developers who not only use the build system but also may wish to extend it.

Introduction

The build system is used to create source archives from source files. These source archives can be used to compile installable packages for various Linux distributions. The build system can generate archives for the core UI as well as for independently installed apps.

The OX App Suite build system uses Jake, a port of Rake from Ruby to Node.js. Both Rake and Jake are dependency-based build systems like Make, which allows quick incremental builds. But unlike Make, Jake doesn't have its own syntax. Instead, it provides an API in JavaScript. The API is used not only to specify dependencies between files using a full programming language, but also to implement the generation of files in the same language. This allows easy implementation of complex build systems, of which the OX App Suite build system is an example. Using the same language for the developed project and for its build system also allows any core developer to quickly extend the build system without having to switch to another language.

Using the Build System

While easily extensible, most of the time the build system will be used as-is. This chapter describes how to set up and use the build system to develop apps and the OX App Suite core.

All command examples are given for the Debian operating system. The instructions should work similarly on other POSIX systems. The first character of each command indicates whether it should be executed as root (#) or as a normal user ($).

Installing

The build system comes in two variants: as part of the OX App Suite source, and as a Software Development Kit (SDK). The SDK contains only the build system and can be installed as a package. Its only external dependency is Node.js, which should be installed automatically by your package manager. While the core of OX App Suite is supposed to be built using the included version of the build system, both the source and the SDK can be used to build external (i.e. independently installable) apps.

The actual installation depends on the chosen variant:

SDK

First, if not already done, add the Open-Xchange repository to the list of Apt sources.

  # echo deb http://software.open-xchange.com/products/appsuite/\
  stable/appsuiteui/DebianSqueeze/ / >> /etc/apt/sources.list.d/ox.list
  # aptitude update

Then, the actual installation is a single command.

  # aptitude install open-xchange-appsuite-dev

Finally, the main executable of the build system should be put in the executable path for easier calling. This is typically done by either extending the $PATH environment variable

  $ export PATH=$PATH:/opt/open-xchange-appsuite-dev/bin

or copying or symlinking the binary to a directory already in the path.

  # ln -s /opt/open-xchange-appsuite-dev/bin/build-appsuite /usr/local/bin

Source

Using the source variant can avoid the requirement for root permissions. But, when building external apps and the build system executable is not invoked using its full path, the environment variable $OX_APPSUITE_DEV needs to be set to specify the path to the build system.

  $ git clone https://git.open-xchange.com/git/wd/frontend/web
  $ export OX_APPSUITE_DEV=$(pwd)/web/ui

Just like with the SDK variant, the executable should be put in the executable path either by extending the $PATH variable

  $ export PATH=$PATH:$OX_APPSUITE_DEV/bin

or by copying or symlinking the executable.

  # ln -s $OX_APPSUITE_DEV/bin/build-appsuite /usr/local/bin

Running

The build system is executed by invoking the command build-appsuite. Similar to most build systems, the build system can perform multiple tasks, which are specified as parameters on the command line. Each task can require any number of parameters. These parameters can be specified either on the command line, using the syntax name=value, or as environment variables.

If present, the file local.conf is sourced by a shell script before the build process starts. This file can export environment variables which are specific to the local system, without checking them into a version control system. Typically, it defines values for builddir and debug.

Since the build system is based on Jake, it also accepts all other Jake options. In addition, the environment variable $nodeopts can be used to pass command line parameters to Node.js. One of the most useful parameters is --debug-brk, which can be used to debug the build system.

When developing external apps, the build system must be run from the top directory of the app's source. As a safety precaution, execution is aborted if the subdirectory apps, which usually contains JavaScript source code, is not found.

Workflow

The build system is used not only to create source archives for packaging. It can also directly install and update the built UI in a directory during development, help with the setup of the source of a new external app and more. While all of these tasks are described in the reference section, daily work involves just a few of them.

init-packaging

The first task which should be executed when creating a new external app is init-packaging. It creates packaging metadata in the current directory. It requires the parameter package to specify the package name in the generated files. All subsequent tasks will automatically extract the package name from the files generated by init-packaging.

Before execution, the apps subdirectory must be created. It indicates that the current directory is actually a valid source directory.

  $ mkdir apps
  $ build-appsuite init-packaging package=example-app
  Build path: build
  Build version: 7.0.0-1.20130111.105125
  
  Version [7.0.0]: 0.0.1
  
  Maintainer (Name <e-mail>): Maintainer <maintainer@example.com>
  
  Copyright line [2013 Open-Xchange, Inc]: 
  
  License name [CC-BY-NC-SA-3.0]: 

Short description: Hello World app to demonstrate usage of the build system The task presents a number of interactive prompts to get the necessary information about the generated packages. The entered values should follow the Debian Maintainer's Guide. Some or even all prompts can be skipped by explicitly specifying the information as a build variable. The list of variable names is available in the reference of init-packaging.

After the task is finished, the generated files can be customized manually to account for any additional packaging requirements.

app

The main workhorse task is app. It is used to process all source files to an optimized deployable app. During development, it is usually called automatically by an IDE or a directory monitoring tool. Since the builds are incremental, the task can be called after every source modification.

For quickest roundtrip times, the directory with the generated files of the app should be made available through the OX App Suite back end by overlaying an existing OX App Suite installation, for example, by symlinking it.

  # ln -s $(readlink -m build/apps/com.example) \
  /opt/open-xchange/appsuite/apps/com.example

To make the manifest accessible to the backend, it should be symlinked into the directory specified by the com.openexchange.apps.manifestPath setting (from manifests.properties) of a locally installed OX App Suite backend.

  # ln -s $(readlink -m build/manifests/example-app.json) \
  /opt/open-xchange/appsuite/manifests/example-app.json

Now, any changes can be tested by reloading OX App Suite in the browser, and any changes in the app manifest can be applied by restarting the backend.

WARNING: Do not use build variables builddir or manifestDir when developing an external app to generate the files directly in their final destination! The clean task will delete these directories and all their contents! In general, don't point builddir or any other *Dir variables at existing directories.

As an example, let's create a small app and build it. It requires only two files: apps/com.example/main.js for the source code of the app

 
define('com.example/main', function () {
  'use strict';
  var app = ox.ui.createApp({ name: 'com.example' });
  app.setLauncher(function () {
    var win = ox.ui.createWindow({
      name: 'com.example',
      title: 'Hello World App'
    });
    app.setWindow(win);
    win.nodes.main.append($('<h1>').text('Hello, World!'));
    win.show();
  });
  return { getApp: app.getInstance };
});

and apps/com.example/manifest.json for the manifest.

{ title: 'Hello World App' } Building the app is as easy as calling

$ build-appsuite app default

The default task is used instead of the app task when building the core OX App Suite. Since it is the default Jake task, it is not necessary to specify it on the command line when it's the only task.

The top directory of OX App Suite source code includes the script build.sh, which should be used instead of calling a potentially unrelated version of build-appsuite. The script changes the current directory to its own, so that it can be called from any directory. Furthermore, it sets BASEDIR, so that $OX_APPSUITE_DIR is neither used nor necessary.

$ ./build.sh clean

The build system uses dependencies and file timestamps to decide which files to rebuild. This assumes that any change to a file increases its timestamp to a value which is greater than the timestamp of any existing file. When this assumption is violated (e.g. after switching to a different source control branch with older files) it may become necessary to rebuild everything to restore the assumption about timestamps. The simplest way to achieve this is the clean task, which simply deletes all generated files.

WARNING: This can be potentially dangerous, since the clean task simply deletes the directories specified by the variables builddir, destDir, l10nDir, manifestDir, and helpDir.

dist

When the app is ready to be shipped, or rather all the time on a continuous build system, the app needs to be packaged in a format suitable for installation on a production system. Since there already exist tools to create packages from suitably arranged source code archives, the OX App Suite build system merely prepares such source archives.

The dist task creates an archive with the source (the one ending in .orig.tar.bz2) and a few additional files necessary for Debian packaging. RPM packages can be generated using the same source archive and the .spec file created by init-packaging. The version of the package is extracted from the newest entry in the file debian/changelog.

Debian packages can also be generated manually either from the temporary directory left behind by dist, or even directly from the source tree. The second option pollutes the source tree with generated files, so it is not recommended, although the .gitignore file created by init-packaging can handle these generated files.

$ build-appsuite dist $ ls tmp/packaging/ example-app-0.0.1 example-app_0.0.1-1.dsc example-app_0.0.1-1.debian.tar.bz2 example-app_0.0.1.orig.tar.bz2 $ cd tmp/packaging/example-app-0.0.1/ $ dpkg-buildpackage -b Reference

Variables

BASEDIR The top directory of the build system. Used by: all tasks. Default: installation directory of the build system Required to build external apps, since in this case, the build system is not installed in the current directory. This variable is automatically set as an environment variable by the build system executable based on $OX_APPSUITE_DEV. branch The Subversion branch of the CLDR to checkout. Used by: update-i18n. builddir The target directory for generated files. Used by: app, clean, default, dist, docs, jakedeps. Default: build copyright The copyright line to be included in packaging metadata. Used by: init-packaging. Example: 2012 Open-Xchange, Inc debug Enables a debug build. Used by: app, default. Default: false To simplify debugging of OX App Suite, compression of source code can be disabled by specifying on, yes, true or 1. description Used by: init-packaging. The description of the app to be included in packaging metadata. destDir Output directory for source archives created by the dist task. Used by: clean, dist. Default: tmp/packaging disableStrictMode Removes all "use strict" directives from processed JavaScript code. Used by: app, default. Default: false Some debugging tools which use code instrumentation have problems when the debugged code uses strict mode. This setting enables code processing even whe using debug mode, so line numbers will not match the original source code. from The root of the printed dependency tree between Jake tasks. Used by: jakedeps. helpDir The location of online help files. Used by: clean, default, dist. Default: same as builddir If the value contains the string @lang@, it will be replaced by the lowercase language code (e.g. en-us) to allow per-language directories. l10nDir The location of compiled l10n files. Used by: app, clean, default, dist. Default: same as builddir If the value contains the string @lang@, it will be replaced by the lowercase language code (e.g. en-us) to allow per-language directories. license File name of the full text of the distribution license. Used by: init-packaging. Default: based on licenseName. licenseName Name of the distribution license to be included in packaging metadata. Used by: init-packaging. Default: CC-BY-NC-SA-3.0 manifestDir The location of the combined manifest file. Used by: app, clean, default, dist. Default: same as builddir maintainer Name and email address of the package maintainer. Used by: init-packaging. Format: Name <email> package The name of the package for the built app. Used by: all tasks. Default: the package name in the first line of debian/changelog Since the name of the manifest file contains the package name and it is required to determine build dependencies, the package name must be always known. This means either debian/changelog must exist and contain at least one entry, or the parameter must be explicitly specified. reverse Reverses the direction of printed dependencies. Used by: deps. When specified, the deps task prints modules which depend on the specified modules, instead of modules on which the specified module depends. revision Revision number of the package for the app. Used by: app, default, dist. Default: 1 The revision number must increase with each rebuild of the same version to enable the creation of unique version strings. These are required in package names and to control content caching in clients. root Specifies for which module to print the dependencies. Used by: deps. Default: print all roots If specified, only the dependencies of the specified module are printed. Otherwise, the dependencies of all modules are printed. skipDeb Whether to skip the generation of Debian source packages. Used by: dist. Default: false This is useful when Debian packages are not required and/or dpkg-source is not available, e.g. on RPM based systems. Even when using this flag, at least the file debian/changelog is still required, because it is used to store the package name and version. skipLess Whether to skip the preprocessing of LessCSS files. Used by: app, default. Default: false This flag skips the generation of CSS files in the apps/themes/*/less directories of all themes. It is used by the packaging system, where the LessCSS files are precompiled after installation on the target system instead of while building the package. tag The Subversion tag of the CLDR to checkout. Used by: update-i18n. to The leaf task in the printed dependency path between Jake tasks. Used by: jakedeps. When specified, only a single path from the root to the leaf task is printed (in reverse order). version Version number of the app. Used by: app, default, dist, init-packaging. Default: 0.0.1 The version should consist of a major, minor and patch version separated by dots. Tasks

An up-to-date list of tasks can be printed using the -T command line option.

app Builds an external app. Variables: BASEDIR, builddir, debug, disableStrictMode, l10nDir, manifestDir, package, revision, version. This is the main task used to build external apps. It works without explicitly specifying any variables, but during development, builddir is usually pointed to the directory of a locally installed OX App Suite UI to avoid additional copying steps. For debugging, debug is also often used. Note that clean must be called when changing any of the variables. clean Removes all generated files. Variables: BASEDIR, builddir, destDir, l10nDir, manifestDir, package. This task should be executed before a normal build using app or default after changing any build variables and after a switch between Git branches. Normal incremental builds can miss changed files if a branch switch replaces files by older versions. default Builds OX App Suite. Variables: BASEDIR, builddir, debug, disableStrictMode, l10nDir, manifestDir, package, revision, version. This is the main task to build OX App Suite. Since it is the default Jake task, it does not need to be specified explicitly on the command line when it is the only task. It works without explicitly specifying any variables, but during development, builddir is usually pointed to a directory which is accessible to a local web server. For debugging, debug is also often used. Note that clean must be called when changing any of the variables. deps Prints module dependencies. Variables: BASEDIR, package, reverse, root. This task visualizes dependencies of RequireJS modules. It prints a tree of dependencies to stdout. If root is specified, then only the dependencies of that module are printed. Otherwise, the dependencies of all modules, on which no other module depends are printed in sequence. If reverse is specified, then this task prints dependants instead of dependencies, i.e. modules which depend on the specified module instead of modules on which the specified module depends. dist Creates source packages. Variables: BASEDIR, builddir, destDir, l10nDir, manifestDir, package, revision, skipDeb, version. This task cleans the source tree by calling clean and packs the source into an archive which can be used to create Debian and RPM packages. The necessary Debian metadata is created alongside the source archive. All files necessary for Debian packaging are placed in the directory specified by destDir. The generated .orig.tar.bz2 archive can also be used together with the .spec file to generate RPM packages. Unless the variable skipDeb is set to true, the program dpkg-source is required by this task. It is used to generate Debian-specific packaging metadata. The variables version and revision specify the version of the package which will be built from the created files, and should therefore be specified explicitly. docs Generates developer documentation. Variables: BASEDIR, builddir, package. This task generates developer documentation for OX App Suite. The generated HTML files are put into builddir/doc. init-packaging Initializes packaging information for a new app. Variables: BASEDIR, copyright, description, license, licenseName, maintainer, package, version. This task is the first task called when starting with the development of a new external app. The directory from which it is called must already contain at least the apps subdirectory. This is also the only task where the package variable must be specified explicitly. Afterwards, the package name is looked up automatically in the file debian/changelog, which is created by this task. The variables version, maintainer, copyright, licenseName, license, and description are required to fill out all necessary packaging metadata. Any of these variables which are not specified explicitly will cause an interactive prompt. This avoids the need to remember the list of variables before one can start developing an app. jakedeps Shows the dependency chain between two Jake tasks. Variables: BASEDIR, from, package, to. This task visualized dependencies between Jake tasks. The variable from specifies the root of a dependency tree, with all dependencies of from as inner and leaf nodes. If to is not specified, then that entire tree is printed. If to is also specified, then only the first found dependency path from from to to is .printed with to at the top and from at the bottom. merge Updates all .po files with the generated ox.pot. Variables: BASEDIR, builddir, debug, package, revision, version. This task updates the list of extracted i18n strings in ox.pot and calls the GNU Gettext tool msgmerge for every language in i18n/*.po. update-i18n Updates CLDR data in the source tree. Variables: BASEDIR, branch, package, tag. This task downloads data from the Unicode CLDR and updates date translations for all languages in i18n/*.po. The exact version of CLDR data is specified as Suubversion tag or branch by the variables tag and branch, respectively. If neither is specified, then the Subversion trunk is checked out. If both are specified, tag is used. verify-doc Generates a documentation skeleton for extension points. Variables: BASEDIR, package. This task is still under development. Currently, it creates a list of all extension points, which have a constant name in the source code. In the future it is supposed to update an existing list instead of overwriting it, and to handle non-constant extension point names. The list of extension points is stored as an HTML snippet in doc/extensionpoints.html.