AppSuite:UI build system
This document describes the build system of OX App Suite. It is intended for app developers who use the build system to create apps as well as for OX App Suite developers who not only use the build system but also may wish to extend it.
The build system is used to create source archives from source files. These source archives can be used to compile installable packages for various Linux distributions. The build system can generate archives for the core UI as well as for independently installed apps.
The OX App Suite build system uses Jake, a port of Rake from Ruby to Node.js. Both Rake and Jake are dependency-based build systems like Make, which allows quick incremental builds. But unlike Make, Jake doesn't have its own syntax. Instead, it provides an API in JavaScript. The API is used not only to specify dependencies between files using a full programming language, but also to implement the generation of files in the same language. This allows easy implementation of complex build systems, of which the OX App Suite build system is an example. Using the same language for the developed project and for its build system also allows any core developer to quickly extend the build system without having to switch to another language.
Using the Build System
While easily extensible, most of the time the build system will be used as-is. This chapter describes how to set up and use the build system to develop apps and the OX App Suite core.
All command examples are given for the Debian operating system. The instructions should work similarly on other POSIX systems. The first character of each command indicates whether it should be executed as root (#) or as a normal user ($).
Installing
The build system comes in two variants: as part of the OX App Suite source, and as a Software Development Kit (SDK). The SDK contains only the build system and can be installed as a package. Its only external dependency is Node.js, which should be installed automatically by your package manager. While the core of OX App Suite is supposed to be built using the included version of the build system, both the source and the SDK can be used to build external (i.e. independently installable) apps.
The actual installation depends on the chosen variant:
SDK
First, if not already done, add the Open-Xchange repository to the list of Apt sources.
# echo deb http://software.open-xchange.com/products/appsuite/\ stable/appsuiteui/DebianSqueeze/ / >> /etc/apt/sources.list.d/ox.list # aptitude update
Then, the actual installation is a single command.
# aptitude install open-xchange-appsuite-dev
Finally, the main executable of the build system should be put in the executable path for easier calling. This is typically done by either extending the $PATH environment variable
$ export PATH=$PATH:/opt/open-xchange-appsuite-dev/bin
or symlinking the binary to a directory already in the path.
# ln -s /opt/open-xchange-appsuite-dev/bin/build-appsuite /usr/local/bin
Source
Using the source variant can avoid the requirement for root permissions.
$ git clone https://git.open-xchange.com/git/wd/frontend/web
Just like with the SDK variant, the executable should be put in the executable path either by extending the $PATH variable
$ export PATH="$PATH:$(pwd)/web/ui/bin"
or by symlinking the executable.
# ln -s "$(pwd)/web/ui/bin/build-appsuite" /usr/local/bin
Running
The build system is executed by invoking the command build-appsuite. Similar to most build systems, the build system can perform multiple tasks, which are specified as parameters on the command line. Each task can require any number of parameters. These parameters can be specified either on the command line, using the syntax name=value, or as environment variables.
If present, the file local.conf is sourced by a shell script before the build process starts. This file can export environment variables which are specific to the local system, without checking them into a version control system. Typically, it defines values for builddir and debug.
Since the build system is based on Jake, it also accepts all other Jake options. In addition, the environment variable $nodeopts can be used to pass command line parameters to Node.js. One of the most useful parameters is --debug-brk, which can be used to debug the build system.
When developing external apps, the build system must be run from the top directory of the app's source. As a safety precaution, execution is aborted if the subdirectory apps, which usually contains JavaScript source code, is not found.
Workflow
The build system is used not only to create source archives for packaging. It can also directly install and update the built UI in a directory during development, help with the setup of the source of a new external app and more. While all of these tasks are described in the reference section, daily work involves just a few of them.
init-packaging
The first task which should be executed when creating a new external app is init-packaging. It creates packaging metadata in the current directory. It requires the parameter package to specify the package name in the generated files. All subsequent tasks will automatically extract the package name from the files generated by init-packaging.
Before execution, the apps subdirectory must be created. It indicates that the current directory is actually a valid source directory.
$ mkdir apps $ build-appsuite init-packaging package=example-app Build path: build Build version: 7.0.0-1.20130111.105125 Version [7.0.0]: 0.0.1 Maintainer (Name <e-mail>): Maintainer <maintainer@example.com> Copyright line [2013 Open-Xchange, Inc]: License name [CC-BY-NC-SA-3.0]: Short description: Hello World app to demonstrate usage of the build system
The task presents a number of interactive prompts to get the necessary information about the generated packages. The entered values should follow the Debian Maintainer's Guide. Some or even all prompts can be skipped by explicitly specifying the information as a build variable. The list of variable names is available in the reference of init-packaging.
After the task is finished, the generated files can be customized manually to account for any additional packaging requirements.
app
The main workhorse task is app. It is used to process all source files to an optimized deployable app. During development, it is usually called automatically by an IDE or a directory monitoring tool. Since the builds are incremental, the task can be called after every source modification.
For quickest roundtrip times, the directory with the generated files of the app should be made available through the OX App Suite back end by overlaying an existing OX App Suite installation, for example, by symlinking it.
# ln -s $(readlink -m build/apps/com.example) \ /opt/open-xchange/appsuite/apps/com.example
To make the manifest accessible to the backend, it should be symlinked into the directory specified by the com.openexchange.apps.manifestPath setting (from manifests.properties) of a locally installed OX App Suite backend.
# ln -s $(readlink -m build/manifests/example-app.json) \ /opt/open-xchange/appsuite/manifests/example-app.json
Now, any changes can be tested by reloading OX App Suite in the browser, and any changes in the app manifest can be applied by restarting the backend.
WARNING: Do not use build variables builddir or manifestDir when developing an external app to generate the files directly in their final destination! The clean task will delete these directories and all their contents! In general, don't point builddir or any other *Dir variables at existing directories.
As an example, let's create a small app and build it. It requires only two files: apps/com.example/main.js for the source code of the app
define('com.example/main', function () {
'use strict';
var app = ox.ui.createApp({ name: 'com.example' });
app.setLauncher(function () {
var win = ox.ui.createWindow({
name: 'com.example',
title: 'Hello World App'
});
app.setWindow(win);
win.nodes.main.append($('<h1>').text('Hello, World!'));
win.show();
});
return { getApp: app.getInstance };
});
and apps/com.example/manifest.json for the manifest:
{ title: 'Hello World App' }
Building the app is as easy as calling
$ build-appsuite app
default
The default task is used instead of the app task when building the core OX App Suite. Since it is the default Jake task, it is not necessary to specify it on the command line when it's the only task.
The top directory of OX App Suite source code includes the script build.sh, which should be used instead of calling a potentially unrelated version of build-appsuite. The script changes the current directory to its own, so that it can be called from any directory.
$ ./build.sh
clean
The build system uses dependencies and file timestamps to decide which files to rebuild. This assumes that any change to a file increases its timestamp to a value which is greater than the timestamp of any existing file. When this assumption is violated (e.g. after switching to a different source control branch with older files) it may become necessary to rebuild everything to restore the assumption about timestamps. The simplest way to achieve this is the clean task, which simply deletes all generated files.
WARNING: This can be potentially dangerous, since the clean task simply deletes the directories specified by the variables builddir, destDir, l10nDir, manifestDir, and helpDir.
dist
When the app is ready to be shipped, or rather all the time on a continuous build system, the app needs to be packaged in a format suitable for installation on a production system. Since there already exist tools to create packages from suitably arranged source code archives, the OX App Suite build system merely prepares such source archives.
The dist task creates an archive with the source (the one ending in .orig.tar.bz2) and a few additional files necessary for Debian packaging. RPM packages can be generated using the same source archive and the .spec file created by init-packaging. The version of the package is extracted from the newest entry in the file debian/changelog.
Debian packages can also be generated manually either from the temporary directory left behind by dist, or even directly from the source tree. The second option pollutes the source tree with generated files, so it is not recommended, although the .gitignore file created by init-packaging can handle these generated files.
$ build-appsuite dist $ ls tmp/packaging/ example-app-0.0.1 example-app_0.0.1-1.dsc example-app_0.0.1-1.debian.tar.bz2 example-app_0.0.1.orig.tar.bz2 $ cd tmp/packaging/example-app-0.0.1/ $ dpkg-buildpackage -b
Reference
Variables
BASEDIR
The top directory of the build system.
Used by: all tasks.
Default: installation directory of the build system
Required to build external apps, since in this case, the build
system is not installed in the current directory. This variable is
automatically set as an environment variable by the build system
executable based on $OX_APPSUITE_DEV.
branch
The Subversion branch of the CLDR to checkout.
Used by: update-i18n.
builddir
The target directory for generated files.
Used by: app, clean,
default, dist, docs,
jakedeps.
Default: build
copyright
The copyright line to be included in packaging metadata.
Used by: init-packaging.
Example: 2012 Open-Xchange, Inc
debug
Enables a debug build.
Used by: app, default.
Default: false
To simplify debugging of OX App Suite, compression of source code
can be disabled by specifying on, yes,
true or 1.
description
Used by: init-packaging.
The description of the app to be included in packaging metadata.
destDir
Output directory for source archives created by
the dist task.
Used by: clean, dist.
Default: tmp/packaging
disableStrictMode
Removes all "use strict" directives from processed
JavaScript code.
Used by: app, default.
Default: false
Some debugging tools which use code instrumentation have problems
when the debugged code uses strict mode. This setting enables code
processing even whe using debug mode, so line numbers
will not match the original source code.
forceDeb
Whether an error during the generation of Debian source packages is an error.
Used by: dist.
Default: false
This is useful when Debian packages are generated automatically, and a failure of dpkg-source is also a failure of the entire build. By default it merely produces a warning.
from
The root of the printed dependency tree between Jake tasks.
Used by: jakedeps.
helpDir
The location of online help files.
Used by: clean, default,
dist.
Default: same as builddir
If the value contains the string @lang@, it will be
replaced by the lowercase language code (e.g. en-us) to
allow per-language directories.
l10nDir
The location of compiled l10n files.
Used by: app, clean,
default, dist.
Default: same as builddir
If the value contains the string @lang@, it will be
replaced by the lowercase language code (e.g. en-us) to
allow per-language directories.
license
File name of the full text of the distribution license.
Used by: init-packaging.
Default: based on licenseName.
licenseName
Name of the distribution license to be included in packaging metadata.
Used by: init-packaging.
Default: CC-BY-NC-SA-3.0
manifestDir
The location of the combined manifest file.
Used by: app, clean,
default, dist.
Default: same as builddir
maintainer
Name and email address of the package maintainer.
Used by: init-packaging.
Format: Name <email>
package
The name of the package for the built app.
Used by: all tasks.
Default: the package name in the first line of
debian/changelog
Since the name of the manifest file contains the package name and it
is required to determine build dependencies, the package name must
be always known. This means either debian/changelog
must exist and contain at least one entry, or the parameter must be
explicitly specified.
reverse
Reverses the direction of printed dependencies.
Used by: deps.
When specified, the deps task prints modules which
depend on the specified modules, instead of modules on which
the specified module depends.
revision
Revision number of the package for the app.
Used by: app, default,
dist.
Default: 1
The revision number must increase with each rebuild of the same version to enable the creation of unique version strings. These are required in package names and to control content caching in clients.
root
Specifies for which module to print the dependencies.
Used by: deps.
Default: print all roots
If specified, only the dependencies of the specified module are printed. Otherwise, the dependencies of all modules are printed.
skipDeb
Whether to skip the generation of Debian source packages.
Used by: dist.
Default: false
This is useful when Debian packages are not required and/or
dpkg-source is not available, e.g. on RPM based
systems.
Even when using this flag, at least the file
debian/changelog is still required, because it is used
to store the package name and version.
skipLess
Whether to skip the preprocessing of LessCSS files.
Used by: app, default.
Default: false
This flag skips the generation of CSS files in
the apps/themes/*/less directories of all themes.
It is used by the packaging system, where the LessCSS files are
precompiled after installation on the target system instead of
while building the package.
tag
The Subversion tag of the CLDR to checkout.
Used by: update-i18n.
to
The leaf task in the printed dependency path between Jake tasks.
Used by: jakedeps.
When specified, only a single path from the root to the leaf task is printed (in reverse order).
version
Version number of the app.
Used by: app, default,
dist, init-packaging.
Default: 0.0.1
The version should consist of a major, minor and patch version separated by dots.
Tasks
An up-to-date list of tasks can be printed using the -T command line option.
app
Builds an external app.
Variables: BASEDIR, builddir,
debug, disableStrictMode,
l10nDir, manifestDir,
package, revision,
version.
This is the main task used to build external apps.
It works without explicitly specifying any variables, but during
development, builddir is usually pointed to
the directory of a locally installed OX App Suite UI to avoid
additional copying steps. For debugging, debug is also
often used. Note that clean must be called when
changing any of the variables.
clean
Removes all generated files.
Variables: BASEDIR, builddir,
destDir, l10nDir,
manifestDir, package.
This task should be executed before a normal build using
app or default after changing any build
variables and after a switch between Git branches. Normal
incremental builds can miss changed files if a branch switch
replaces files by older versions.
default
Builds OX App Suite.
Variables: BASEDIR, builddir,
debug, disableStrictMode,
l10nDir, manifestDir,
package, revision,
version.
This is the main task to build OX App Suite. Since it is the default Jake task, it does not need to be specified explicitly on the command line when it is the only task.
It works without explicitly specifying any variables, but during
development, builddir is usually pointed to
a directory which is accessible to a local web server. For
debugging, debug is also often used. Note that
clean must be called when changing any of
the variables.
deps
Prints module dependencies.
Variables: BASEDIR, package,
reverse, root.
This task visualizes dependencies of RequireJS modules. It prints
a tree of dependencies to stdout.
If root is specified, then only the dependencies of
that module are printed. Otherwise, the dependencies of all modules,
on which no other module depends are printed in sequence.
If reverse is specified, then this task prints
dependants instead of dependencies, i.e. modules which depend on
the specified module instead of modules on which the specified
module depends.
dist
Creates source packages.
Variables: BASEDIR, builddir,
destDir, forceDeb, l10nDir,
manifestDir, package,
revision, skipDeb,
version.
This task cleans the source tree by calling clean and
packs the source into an archive which can be used to create Debian
and RPM packages. The necessary Debian metadata is created alongside
the source archive. All files necessary for Debian packaging are
placed in the directory specified by destDir.
The generated .orig.tar.bz2 archive can also be used
together with the .spec file to generate RPM packages.
Unless the variable skipDeb is set to
true, the program dpkg-source is required
by this task. It is used to generate Debian-specific packaging
metadata.
The variables version and revision specify
the version of the package which will be built from the created
files, and should therefore be specified explicitly.
init-packaging
Initializes packaging information for a new app.
Variables: BASEDIR, copyright,
description, license,
licenseName, maintainer,
package, version.
This task is the first task called when starting with
the development of a new external app. The directory from which it
is called must already contain at least the apps
subdirectory. This is also the only task where
the package variable must be specified explicitly.
Afterwards, the package name is looked up automatically in the file
debian/changelog, which is created by this task.
The variables version, maintainer,
copyright, licenseName,
license, and description are required to
fill out all necessary packaging metadata. Any of these variables
which are not specified explicitly will cause an interactive prompt.
This avoids the need to remember the list of variables before one
can start developing an app.
jakedeps
Shows the dependency chain between two Jake tasks.
Variables: BASEDIR, from,
package, to.
This task visualized dependencies between Jake tasks. The variable
from specifies the root of a dependency tree, with all
dependencies of from as inner and leaf nodes. If
to is not specified, then that entire tree is printed.
If to is also specified, then only the first found
dependency path from from to to is
.printed with to at the top and from at
the bottom.
merge
Updates all .po files with the generated
ox.pot.
Variables: BASEDIR, builddir,
debug, package, revision,
version.
This task updates the list of extracted i18n strings in
ox.pot and calls the GNU Gettext tool
msgmerge for every language in i18n/*.po.
update-i18n
Updates CLDR data in the source tree.
Variables: BASEDIR, branch,
package, tag.
This task downloads data from the Unicode CLDR and updates date
translations for all languages in i18n/*.po.
The exact version of CLDR data is specified as Suubversion tag or
branch by the variables tag and branch,
respectively. If neither is specified, then the Subversion trunk is
checked out. If both are specified, tag is used.
verify-doc
Generates a documentation skeleton for extension points.
Variables: BASEDIR, package.
This task is still under development. Currently, it creates a list of all extension points, which have a constant name in the source code. In the future it is supposed to update an existing list instead of overwriting it, and to handle non-constant extension point names.
The list of extension points is stored as an HTML snippet in doc/extensionpoints.html.