any-workdir – the local autonomous dir with complete build environment for any(1) engine.

Description

Build with any(1) is localised inside directory, containing everything project-related: sources, build directories, all temporary files, final binary results and the files of engine itself. That local directory is referenced as working directory here and further on.

No impact on filesystem outside working directory is done during the build. The build is expected for non-priveleged user accounts. Unlimited quantity of working directories can be created by different users on a single machine.

In case of containerised usage (which is intended to be the normal default one) the working dir contains local host executable binaries, isolating a build process from the global host system. That results in reproducible build environment, distributable across different hosts. The engine is copied into each working directory, making them independent from each other and from global system settings.

When host container is not used, user should support on his own the reliable environment from global OS for the needed build tasks. The most common reason for not using a container is a lack of permissions for any-do(1) (for example, no possibility to retrieve sudo(1) permissions on some public server).

Installation

The way of installation depends on distribution of files for working dir.

Archives with settle.sh

If file settle.sh is present among recieved files, launch it with a desired name of your working directory:
        /path/to/settle.sh ~/my/path/my_dir
        cd ~/my/path/my_dir
Script will create my_dir with all files, provided along with it for working directory.

With no arguments it creates workdir in current directory:

        /path/to/settle.sh
        cd ./workdir

Plain archives

No settle.sh is seen among archives. Create new directory under desired name, walk into it and manually extract all available archives inside:
        mkdir my_dir
        cd my_dir
        tar xf /path/to/archive-one.txz
        tar xf /path/to/archive-two.txz
        ...
Extraction cycle may be used:
        mkdir my_dir
        cd my_dir
        for i in /path/to/*.txz /path/to/*.tgz /path/to/*.tar ; do
            tar xf $i
        done

Update with separate archives

It may be needed to update already existing working directory with some stuff. The update is done as in previous case: extract the contents of the archive into your directory.
        tar xf /path/to/arcive-updated.txz -C ~/my/path/my_dir/

Installation with any settle

The any-settle(1) command creates new working directory from template, pre-installed on the host. any-settle(1) may have more complex implementation as well. For example, it can construct working directories with distributed sources from network infrastructure. For desired project my_dir:
        any settle my_dir
        cd my_dir

Set up the executables

For the build you will need access to commands, provided by the engine. It can be done in two ways.

Modify shell environment

Executables, needed for the work, are available in directory rrr/bin/ in each working directory. For quick access to executables without installation, one may add that directory to his PATH variable and launch everything locally:
        cd my_dir
        export PATH=$(pwd)/rrr/bin:$PATH
The command must be done in every new terminal session.

This way is good to execute binaries from single pinned working directory. For regular work it is better to install commands to some permanent place.

Install commands

To install build-related executables to ~/rrr/bin/:
        cd my_dir
        ./install.sh
After that you will probably want to add path ~/rrr/bin/ to your shell environment (which is done through file ~/.profile or others alike, depending on the shell).

To install to some other place:

        cd my_dir
        DESTDIR=~/other/dir ./install.sh
The executables will be stored in ~/other/dir/bin/ for the example above.

Mind the installation of commands into home dir in the examples above. These executables are intended for regular user and not system-wide.

Set up the build data

Files for working directory may contain packages with libraries and other applications, which are needed as dependencies for your build. Their format vastly depends on the project. The libraries may be packed in different formats, may be distributed as installable images, and so on. The conventions to install that binary data for usage in the build also differ.

There is generic script any/bin/deploy.sh for all such build data. If it exists in your dir, launch it and check out its usage. The build executables must be available, as described in previous section.

        cd my_dir
        ./any/bin/deploy.sh --help
It will print out expected input parameters for deploying, if it needs any.

Essential directories

The essential set of directories inside a working dir is:
any/
rdd.conf.d/
rrr/
build/
ports/
release/

The short overview of each dir:

any/
is the directory with files of any build engine itself. The library files are copied locally to each working dir and not shared across the whole system.

rdd.conf.d/, rdd.def
are the places for input data of engine. rdd.conf.d/ is the directory with rdd(1) configuration files. rdd.def is the single file with the same purpose.
All settings for build process are done through that configs. The possible options are detailed in any-conf(7).

rrr/
is the directory with installed utils, used by engine.
rdd(1) and companions are mandatory. Other utils can be installed here also as parts of the engine. Path rrr/bin/ is added to PATH variable inside build environment.

build/
is the directory to write files during the build, as given by default any.conf config.
Here the build directories with sources and object files are stored, compilation takes place, ready package files are installed and binary archives are created. All subdirectories of build/ are created during build process.

The dir must not keep any unique or unproducible data, concerning the engine or input. It is safe to clear it and rebuild the same project from the start. This dir is absent until the first build. The locally saved sources or patches, which are not intended to be available from the network, should not be kept here, ports/ is used for that instead.

build/pack/
is the directory where packaged results of successfull build are stored.

ports/
is the directory with the content of the distro, as given by default any.conf config.
In the first turn, those are scripts, personal for each package, with commands to build them. Etalon input data for checkers is kept here as well. Opposed to that, generic build commands and methods, same for all packages, are stored inside engine any/.

In case of several various distro projects, which share little common between each other, it is possible to keep their data in separate directories, analogues to ports/, and choose the needed one between them with configs.

release/
is the directory with final release results of entire distro project. Its contents depend on the meaning of 'final result', taken by the project. It can be single installable image or some predefined set of packages.

The directory is not normally populated automatically after buildage of single packages.

Host environment

The entire host environment is copied with new working directory creation by any-settle(1), unless configured otherwise. It includes all executables to enter new shell session in containerised environment and execute generic build procedure: libc, POSIX baseutils, shell implementation, make. It must include toolchain with compiler and linker. Many utils will be needed for documentation generation and other help tasks during the build.

The host files are installed into root of the working directory with the same layout as they have in generic OS: bin/, lib/, usr/lib/, usr/bin/, etc/ and so on. After chroot(1) or other container implementation the build process will see general filesystem, localised inside working dir.

The build process with any is designed to use vast majority of files from build/, which is populated from packages, which are built inside the project or installed by some request. Installations inside build/ are granulated by profiles, can be tweaked and rebuilt by lots of ways.

The purpose of host is bare run-time support for all utilities which handle the build, while introducing minimal impact on compilation and linkage with host libraries and include files. The ideal host from that point of view contains no libraries and includes at all, besides the ones for native toolchain support. Such host is not so easy to reach, especially to get rid off dynamic shared libraries for binaries. Much easier to remove or hide everything from usr/lib/pkgconfig/ and all usr/bin/*-config scripts, and leave only those files from usr/include/ to keep compiler working.

Engine directories

The entire set of engine directories inside a working dir is:

any/bin/
any/corelib/
any/include/
any/lib/
build/context/
build/empty/
build/image/
build/log/
build/pack/
build/patch/
build/src/
build/srcstatic/
build/tmp/
build/work/
ports/packages/
ports/list/
ports/meta/
ports/patch/
rdd.conf.d/
release/
rrr/bin/
var/dump/
var/lib/

any/bin/
Directory for inner executables of any itself. Such executables are supposed non-usable outside of engine. That can be C binaries, implementing engine methods instead of ones in shell. The path is added to PATH variable inside build environment.

any/corelib/
Directory with main engine libraries, which implement core functional and declare all mandatory interfaces. All libraries from here are always included in the launch disregarding used configuration and modules. Some interfaces contain empty or trivial implementation: that means useful implementation is provided by some additional code and depends on user's choice (one or another control version system, for example). Such support is distributed through additional modules, installed into any/lib/.

any/include/
Directory for shell API constructing beyond core functionality. If some module declares global variables, it should place their assignments in the file inside this dir.

any/lib/
Directory for all possible modules with additional functional. Some of them are included in pack of any, some of them come from anymods(7), still more may be third-party.
Modules within this dir must contain only method implementations. No calls out of methods' bodies are allowed. No declarations of global variables are allowed as well.
The files of the same module may be placed in both any/include/module.sh and any/lib/module.sh. The first file will contain variable declarations, the second one methods.

build/context/
Directory to keep file context for package building. Context is composed from files of other packages, used by the current building one. Composing is done with symbolic links from real installed files of other packages into context dir. Several types of text files are copied, as they are edited by engine. Each package has its own separate context directory. The set of files, used for context, is defined by any-conf(7) settings. The list of packages, needed for current one (build dependencies), is defined by anybuild of current package, as described in any-build(7).

Personal context of each package is stored into dir build/context/PROF/PACKAGENAME/root/.
For example, context of hello-world-1.0 for amd64 will be kept in build/context/amd64/hello-world-1.0/root/ dir inside working directory.

The given scheme means the following:
-
The build environment is controlled explicitly by dependencies. The obscure non-obvious influence on the build process is reduced significantly.
-
Build context is lightweight and quickly composed, as the most files are handled with symbolic links and heavy copies are avoided.
-
Correctness of build dependencies is checked automatically - in case of missing dependency in anybuild the build process will not have access to corresponding files.
-
There is no changing resources, shared between building of several packages. So different packages can be launched in parallel without any synchronisation at all, while they do not depend on each other (in other case synchronisation of dependencies handling comes to a game).


Directory build/context/ does not limit the influence of host environment, i.e. the binaries of container, installed into the root of working directory, or the entire host system without a container.

Interfaces pkg_context(3) and pkg_root(3) write to this dir.

build/context/PROF/platform/
Directory with context, equal for all packages. The context is contained in tar archives. Every archive from this dir is extracted to personal context directory of each package (into build/context/amd64/hello-world-1.0/root/ for the example above).
The example of platform context may be system libc or some minimal filesystem, which is not granulated by packages.

build/context/PROF/static/root/
Directory with single context, shared for all packages, used in staticdeproot mode. That mode makes no personal context for each package, but maintains single shared one, as in traditional file system. See any-do(1) for description of staticdeproot.

build/empty/
Always empty dir. It is used to prevent expansion of meta-symbols, such as '*', in some shell methods.

build/image/
The dir for installed binary files of packages. Each package is installed into separated dir build/image/PROF/PACKAGENAME/.

For example, the package hello-world-1.0 for amd64 will have installed files:
build/image/amd64/hello-world-1.0/etc/
build/image/amd64/hello-world-1.0/usr/bin/
build/image/amd64/hello-world-1.0/usr/lib/
build/image/amd64/hello-world-1.0/usr/include/
build/image/amd64/hello-world-1.0/usr/share/
...


Interfaces src_install(3), pkg_install(3), pkg_rminstall(3), pkg_config(3) write to this dir.

build/log/
Directory to keep logs of build process. Each package PACKAGENAME for profile PROF keeps its log file in build/log/PROF/PACKAGENAME.log.
For example, the package hello-world-1.0 for amd64 will keep log file in build/log/amd64/hello-world-1.0.log.

Under the hood, the logs in this directory are symbolic links to files inside var/dump/, as logs are total output redirect, constructed by rdd(1), and that command uses var/dump/ unconditionally for all its operations.

Each time new build of a package starts its log file is rewritten.

The log file of entire build session is kept in build/log/PROF/PROF.log file. That file is never truncated. Each time new build session starts, summarised information of the build is appended to profile log. That summarised information is the same as outputed to terminal, besides the explicit link to the log in case of error.

build/pack/
Directory for binary archives with packages. When no additional naming convention is included, each built package is saved into installable archive build/pack/PROF/PACKAGENAME-postfix.tgz. postfix part depends on target platform for the package.
For example, the package hello-world-1.0 for amd64 will have simple binary archive build/pack/amd64/hello-world-1.0-x86_64.tgz.
Modules can change the binary format for a package and portion of data in its file name.

When some convention of archive naming takes place, each package has one or more corresponding binary archives with arbitrary names. That names are usually described in anybuild. To match package names in terms of any and names of their binary archives the method pack_name(3) is used.

Interface pkg_pack(3) writes to this dir.

build/patch/
Directory for patches, applied for packages. Each package PACKAGENAME gets his patches from build/patch/PACKAGENAME/ directory.
For example, the package hello-world-1.0 will use patches from build/patch/hello-world-1.0/.

There are different places, where patches may be stored, but all resulting files will be finally copied to build/patch/.

Interface src_fetch(3) writes to this dir.

build/src/
Directory to store sources of packages, retrieved from version control systems. Each package PACKAGENAME stores its sources in build/src/PACKAGENAME/ directory.
For example, the package hello-world-1.0 will fetch development sources into build/src/hello-world-1.0/.

That sources are used to generate some patches, applied before building. That is done by fetch_gen_patches(3) interface. Core any engine does not generate any patches, but modules may implement this interface for git(1), hg(1), svn(1) or another content tracker.
It is possible that sources from this dir are copied 'as is' on top of previously prepared ones.

The default value of that dir can be altered with any_dev_src_dir option.

Interface src_fetch(3) writes to this dir.

build/srcstatic/
Directory to store source archives, or tarballs. All prepared patches will be applied to the sources from a found archive. So these archives are supposed to be original or unchanged upstream sources, while development changes are done through patches or other explicit actions.

The alternative usage is to keep actual full-pledged development sources in build/srcstatic/ and not to use patches at all. That variant can be handy at inner debug, when still there is no need to share changes done.

In simple case, sources are recognised by expression build/srcstatic/PACKAGENAME.EXT.
For example, the package hello-world-1.0 will catch source tarball build/srcstatic/hello-world-1.0.tgz.

That is not the only scheme, though, as sources may have alternate name. This is due to possibly different names of package at authors domain and in naming convention of current distro project. The alternate name for sources is described in any-api(7) and fetch_remote_src(3) method.

If plain directory with matching name is found inside build/srcstatic/, it will be used. Otherwise the tar archive will be searched for.
Known tarball extensions are:
.tar.xz
.txz
.tar.gz
.tgz
.tar.bz2
.tbz2


The default value of the dir can be altered with any_src_dir option.

Interface src_fetch(3) writes to this dir.

build/tmp/
Directory to write and keep all intermediate files during the work of any engine or modules.

build/work/
Directory to do the actual build. The final sources are prepared here, configure, make and make install are launched. Each package has that dir in build/work/PROF/PACKAGENAME/.
For example, the package hello-world-1.0 for amd64 will be built in build/work/amd64/hello-world-1.0/ directory.

The directory is kept after the successfull build, as it contains all intermediate results of the process and irreplaceable for debug.

Interfaces src_fetch(3), src_prepare(3), src_config(3), src_compile(3) write to this dir.

ports/packages/
Directory with individual build scripts, called anybuilds. Each package has its own separate directory, named after package. Anybuild script inside is named as package-name-1.2.3.build. This file is included by default for given package inside any.
For example, anybuild for hello-world-1.0 will be in ports/packages/hello-world-1.0/hello-world-1.0.build file.
More separate shell modules can reside in the directory. That modules are included after data words, saved in rdd_prf_all variable, as described in section Modular code inclusion of any-do(1).

Package directory keeps the config file of rdd(1), personal for package, named atom.conf. That file is analogues to the ones in rdd.conf.d/, but applies only to that package.

The directory ports/packages/package-name-1.2.3/patch, if one exists, is used as source of patches for the package. That dir allows to keep all input data for the package build in a single place (anybuild and patches all together). Besides this dir, patches may be fetched from other places separately from ports/.

When binary archive format for some package manager is used, the various postinstall, preinstall and other scripts alike are needed. They are stored in the package directory inside ports/packages/ as well.
Personal package directory may contain other materials, used by modules (for example, sanity or regression checkers).

The default value of the dir can be altered with rdd_atom_path option.

ports/list/
Directory for package lists of rdd(1). Here text lists are kept with content of some buildable sets, composing final project such as OS distribution.

The convention is to name general hand-written lists with .src ending, and script generated ones with .txt ending.

List all.src has special meaning – it is used to keep all packages, available in project by default. Besides the other, that value is used as data source to resolve build dependencies. The name of the list can be changed through the option any_list_all.

The default value of the dir can be altered with rdd_list_path option.

ports/meta/
Directory for various meta-data, concerning the entire distro project inside ports/: etalon data for sanity checkers, cache for cross-compilation.

The default value of the dir can be altered with any_meta_dir option.

ports/patch/
Directory to keep locally saved patches for packages. Each package has its room in ports/patch/PACKAGE-NAME/.

The default value of the dir can be altered with any_static_patch_dir option.

rdd.conf.d/, config file rdd.def
Directory with configuration files of rdd(1). That's the place with all engine properties. Some of them belong to rdd(1) domain, some of them are read by any part of the engine and described in any-conf(7).

Directory with any source code ships example.conf.d/ subdir. If configs from any/example.conf.d/ are added to rdd.conf.d/, working directory will get reasonable default values to begin the work. any.conf is the file with all default values for engine, described in the manuals. Example architecture-specific settings, describing properties of target OS and processor architecture, are stored in build.conf. Arbitrary new files can be added to describe new entities or customise the build.

File rdd.def works as general config, but with higher priority. It is used for current settings, non-saveable for all working directories or situations. It can be used for experiments, usage of testing code, storage of dynamically generated or fetched information.

release/
Directory with release results of distro project. It can be single installable image or some predefined set of packages.

rrr/bin/
rrr/ is the destination directory to install utilities, used by engine. The content of used utilities is determined by any-settle(1) configuration. Changes to utilities should not be done here on a regular basis, better change util at its source and reinstall (though it's handy to edit something right in rrr/ for quick test purposes). Path rrr/bin/ is added to PATH variable inside build engine.

var/dump/
Directory where rdd(1) keeps its intermediate files while constructing the build scripts. The log files of build process are kept here. This dir is the place for dumps – shell files with full any environment and gathered data for some package. The dumps are used at several parts of the engine.

var/lib/
Directory to track installed modules. Each module MOD from anymods(7) leaves text file var/lib/anypkg/MOD, containing list of files of that module.

Set up any settle

It is possible to set up centralized template, which will be used as a source for creation of all working directories on the host.

Template installation may be done with ./install.sh script inside source directory of any.

        cd src/any
        ./install.sh
It saves files for new installations into PREFIX/lib/any. Everything, placed into that directory, will be deployed for new working directories. So the user may add arbitrary files into the template.

Script recognizes DESTDIR as well as PREFIX:

        cd src/any
        DESTDIR=~/test PREFIX=/usr/local ./install.sh
The command above will place files for working dir into ~/test/usr/local.

After template installation it becomes possible to create new working directories with any-settle(1) tool. One should say for his desired project my_dir:

        any settle my_dir
        cd my_dir

See also

any(1), any-do(1), any-settle(1), any-conf(7), anymods(7), any-build(7), any-api(7)