any – the engine to build software, distributable in POSIX universe.

Description

any is the engine designed to create binary program packages for any possible system, blessed by POSIX compliance.

any at its core is the set of POSIX shell functions. The functions are designed for launch with anch do or non-container any do commands (see description of both in any-do(1)).

The build of each package is described by file, called anybuild, build file, build script or just build. Commands there are written in shell and grouped into aimed methods like source fetching, configure, compile or packing.

Single build file may produce installable binary archives in formats for different package managers, for several platforms and with conformance to various development policies.

Documentation navigation

This page is top overview of work with any to gather solid picture of the matter in single place. The page mostly consists of extracts from more detailed manuals on dedicated topics, with reference links to them.

Before diving into dedicated man it is recommended to read this page first to get familiar with the whole concept.
Here is the guide for existing docs.

To work with the existing packages:

any-workdir(7) describes the project directory to work with: its installation on the host, inner structure and the purpose of each directory inside.
any-do(1) describes the main launch utility and possibilities of the build through command line.
any-howto(7) contains quick references to the most typical build tasks.

To debug and enhance the build:

any-map(7) describes all essential methods and interfaces, which do the actual building.
any-build(7) describes the structure of single anybuild and tips to write new ones.

To write advanced code for anybuilds:

any-api(7) describes all interfaces, provided by the engine for use inside build scripts.
any-dev(7) describes conventions to write new code for the engine.

Specialized areas:

any-conf(7) describes all options inside configuration files.
anymods(7) lists the available additional modules with support of version control systems, package formats and other outer infrastructure.
rdd(1) describes the principles of data gathering, taken from configuration files, and more low-level features.

Installation

The way of installation depends on distribution of files for working dir.

Archives with settle.sh

If file settle.sh is present among recieved files, launch it with a desired name of your working directory:
        /path/to/settle.sh ~/my/path/my_dir
Script will create my_dir with all files, provided along with it for working directory.

With no arguments it creates workdir in current directory:

        /path/to/settle.sh

Plain archives

No settle.sh is seen among archives. Create new directory under desired name, walk into it and manually extract all available archives inside:
        mkdir my_dir
        cd my_dir
        tar xf /path/to/archive-one.txz
        tar xf /path/to/archive-two.txz
        ...
Extraction cycle may be used:
        mkdir my_dir
        cd my_dir
        for i in /path/to/*.txz /path/to/*.tgz /path/to/*.tar ; do
            tar xf $i
        done

Update with separate archives

It may be needed to update already existing working directory with some stuff. The update is done as in previous case: extract the contents of the archive into your directory.
        tar xf /path/to/arcive-updated.txz -C ~/my/path/my_dir/

Installation with any settle

The any-settle(1) command creates new working directory from template, pre-installed on the host. any-settle(1) may have more complex implementation as well. For example, it can construct working directories with distributed sources from network infrastructure. For desired project my_dir:
        any settle my_dir
        cd my_dir

Set up the executables

For the build you will need access to commands, provided by the engine. It can be done in two ways.

Modify shell environment

Executables, needed for the work, are available in directory rrr/bin/ in each working directory. For quick access to executables without installation, one may add that directory to his PATH variable and launch everything locally:
        cd my_dir
        export PATH=$(pwd)/rrr/bin:$PATH
The command must be done in every new terminal session.

This way is good to execute binaries from single pinned working directory. For regular work it is better to install commands to some permanent place.

Install commands

To install build-related executables to ~/rrr/bin/:
        cd my_dir
        ./install.sh
After that you will probably want to add path ~/rrr/bin/ to your shell environment (which is done through file ~/.profile or others alike, depending on the shell).

To install to some other place:

        cd my_dir
        DESTDIR=~/other/dir ./install.sh
The executables will be stored in ~/other/dir/bin/ for the example above.

Mind the installation of commands into home dir in the examples above. These executables are intended for regular user and not system-wide.

Set up the build data

Files for working directory may contain packages with libraries and other applications, which are needed as dependencies for your build. Their format vastly depends on the project. The libraries may be packed in different formats, may be distributed as installable images, and so on. The conventions to install that binary data for usage in the build also differ.

There is generic script any/bin/deploy.sh for all such build data. If it exists in your dir, launch it and check out its usage. The build executables must be available, as described in previous section.

        cd my_dir
        ./any/bin/deploy.sh --help
It will print out expected input parameters for deploying, if it needs any.

Working directory

Build with any is localised inside directory, containing everything project-related: sources, build directories, all temporary files, final binary results and the files of engine itself. That local directory is referenced as working directory here and further on.

No impact on filesystem outside working directory is done during the build. The build is expected for non-priveleged user accounts. Unlimited quantity of working directories can be created by different users on a single machine.

The structure of any's working directory is detailed in any-workdir(7).
The essential set of directories inside a working dir is:

any/
rdd.conf.d/
rrr/
build/
ports/
release/

The short overview of each dir:

any/
is the directory with files of any build engine itself. The library files are copied locally to each working dir and not shared across the whole system.

rdd.conf.d/, rdd.def
are the places for input data of engine. rdd.conf.d/ is the directory with rdd(1) configuration files. rdd.def is the single file with the same purpose.
All settings for build process are done through that configs. The possible options are detailed in any-conf(7).

rrr/
is the directory with installed utils, used by engine.
rdd(1) and companions are mandatory. Other utils can be installed here also as parts of the engine. Path rrr/bin/ is added to PATH variable inside build environment.

build/
is the directory to write files during the build, as given by default any.conf config.
Here the build directories with sources and object files are stored, compilation takes place, ready package files are installed and binary archives are created. All subdirectories of build/ are created during build process.

The dir must not keep any unique or unproducible data, concerning the engine or input. It is safe to clear it and rebuild the same project from the start. This dir is absent until the first build. The locally saved sources or patches, which are not intended to be available from the network, should not be kept here, ports/ is used for that instead.

build/pack/
is the directory where packaged results of successfull build are stored.

ports/
is the directory with the content of the distro, as given by default any.conf config.
In the first turn, those are scripts, personal for each package, with commands to build them. Etalon input data for checkers is kept here as well. Opposed to that, generic build commands and methods, same for all packages, are stored inside engine any/.

In case of several various distro projects, which share little common between each other, it is possible to keep their data in separate directories, analogues to ports/, and choose the needed one between them with configs.

release/
is the directory with final release results of entire distro project. Its contents depend on the meaning of 'final result', taken by the project. It can be single installable image or some predefined set of packages.

The directory is not normally populated automatically after buildage of single packages.

See any-workdir(7) for further details.

Build existing packages

How to build something, using prewritten build scripts. Typical build tasks are described in any-do(1).

To list all packages, available by default for current project:

        lspkg amd64

Usual and most frequent build command looks like:

        any do amd64 hello-world-1.0
In case of successfull build a binary package, which accords to hello-world-1.0, will be created in directory build/pack/amd64/. Its format is .tgz when using core implementation, but that can be altered with extensions from anymods(7).
In case of error, path to the log file with build process will be displayed. The file can be examined to detect error reason.

More generally, the same string for simple launch with parametrised args is:

any do dataid package-1.2.3

dataid
is an id for data, used for the launch. For example, if one needs to build a package for x86 32 bit architecture, he would use x86 as data id. By that id specific x86 data will be selected from the configs, and that id will inherit more generic data, used for all architectures, such as prefixes and paths to write down the files.
package-1.2.3
is the name of a package to build. The process requires sources at predefined path, possibly patches, and the build script, called anybuild. That file contains methods and data, used during the build, specific for that very package.

Available packages are stored in text lists inside ports/list/ directory. All packs, available by default, are taken from all.src list. To see the contents of lists, chosen by current configuration, lspkg(1) is used:

        lspkg amd64 somelist.src

Launch out of container looks like:

        rdd amd64 hello-world-1.0

The same as previous launch, but no container is entered before build starts. Instead of localised container the whole host system affects the process. The command can be used when chroot(1) or other containers are out of reach.

Details and examples of data gathering, naming of packages, package lists can be found in any-do(1), and more low-level details in rdd(1), describing front-end.

The workflow of a single build

The building of a single package is handled by shell method map. It resides in any/corelib/map.sh file. Main method is divided into tasks, each one in turn is handled by its own dedicated method. Methods depend on each other, so each one normally works as expected after the previous one.

The essential build tasks and corresponding interface methods are as follows:

pkg_pretend(3)
performs read-only early check of the build. For example, the work with a package can be skipped if it does not work under some OS or architecture.

pkg_setup(3)
performs early preparation of the build with permission to write files. It resets temporary files, generated during previous build session. It also prepares the wrappers for toolchain, if they are used.

pkg_context(3)
resolves package dependencies and prepares the context for the build. If declared dependencies are not found, it prints the missing part. The context is separate directory with all files from dependent packages, needed for the build.

src_fetch(3)
gets the sources available at the local directory. It can fetch them from remote hosts or take files from another local storage.

src_prepare(3)
applies patches to sources and prepares them in any other way.

src_config(3)
configures the build before the compilation.

src_compile(3)
does the compilation of source code and linkage, generates documentation and performs all other essential work.

pkg_rminstall(3)
removes the files, installed by previous build. It handles only directory of the current package, if there had been one.

src_install(3)
copies the build results to separated directory with needed layout.

pkg_install(3)
installs additional files, prescribed by build politics, or removes redundant ones. Any other preparation of files for further extraction on target system goes here.

pkg_config(3)
prepares everything for installable archives: generates and installs meta-information for package manager, splits the files to sub-packages.

pkg_root(3)
mirrors the files of current package to directory with build context, making them available to another packages. It is mainly called during the build of other packages, when they use the current one. However, the method is also checked during current build to make sure it works properly.

pkg_pack(3)
creates the binary installable packages. Single package is possible. That is final result of single build process.

Let's examine the simple launch of:

    anch do amd64 hello-world-1.0
At the top level, while performing the sample build above, the following code is executed in order:
any/bin/dobefore is called by anch do before entering the sandbox (if the file exists).

any/bin/enter is called by anch do inside the sandbox to set up the build environment. It usually creates device files and mounts virtual filesystems. It is up to the user to place some another actions to that file, though.
You may read more about any/bin/ scripts in any-do(1).

Method map_group is called once for the launch. It is used to do something for the whole build task and not for separate packages. Default implementation prints here the summary information about build session. Method may also maintain the host enivironment in terms of package management, which is more complex task, then simple starter ones in any/bin/enter. Host management may be done with help of modules.

Method map_autopre is called for each package. It remains, when essential actions are changed:
            any do amd64 hello-world-1.0 changed_action
In the example above map is not executed anymore, it is replaced with changed_action. But map_autopre remains as it was. So the described method contains simple preparations, which are needed for each package before any kind of actions. For example, it prints the summary string about the package to output.

Method map is called for each package and does all the actual build.

Method map_autopost is called for each package. It is alike map_autopre, but at the end. Default implementation prints status of the finished build and spent time.

Method map_postmortem is called for each package, returned non-null status. By default it does the same, as map_autopost.

any/bin/doafter is called by anch do at the end of entire build session out of the sanbox (if the file exists).

With any do usage, the sequence remains the same, but without any/bin/ scripts.

There is possibility to call arbitrary methods directly from any-do(1) or anch do command line, thus starting the build process in some middle point or doing some limited part of the whole build.
Arbitrary entry point is useful during debug. That is used also to accomplish some custom task, using already existing methods.
For example, one may need to get some sources from remote places, save them and prepare for building locally, eliminating further remote access. To accomplish that, one can call src_fetch(3) and src_store(3) methods. The corresponding any-do(1) string would be:

        any do all somelist.src src_fetch,src_store

See any-map(7) for full description of build interfaces inside map. The any-do(1) page describes the composing of advanced commands from engine methods.

Modular code inclusion

The list of data sections, gathered from config files and options by rdd(1) (so-called datawords), is used to determine which additional modules of any(1) engine should be used in the launch.

Simple example of the launch:

        any do amd64,packdeb hello-world-1.0
The launch above will result in inclusion of module any/lib/packdeb.sh and creation of binary package in debian format inside pkg_pack(3) method.

        any do amd64,packslack hello-world-1.0
The string above includes any/lib/packslack.sh module and creates slackware binary package instead.

The list of used data words is stored in AUSE variable (in the order of their inclusion, the first one is the most right). For each used data word WORD and each used PATH the file PATH/WORD.sh is checked and included, if there is such one. The following paths are checked for modules in order:

any/include/
any/lib/
ports/packages/PACKAGE/

There is no need to assign data words as in example above explicitly in command line to get the effect of modular inclusion. Almost all used data words are generally taken from config files, but they participate in the described mechanism as well.

To add some functional module, which can be loaded and used by demand with the whole engine, one should add shell file any/lib/MODULE.sh. If the module declares global variables, proposed for use by others, the module with such variables must be placed inside any/include/. If actions of this module should take place only for some package, then it should be placed in ports/packages/PACKAGE/MODULE.sh.

The option any_profile_engine inside any.conf contains directories, where modular inclusion takes place. By adding new entries there one can gather modules from more directories or even exclude standard directories inside any/ from gathering. See any-conf(7) for further description of any_profile_engine.

Build new package

The build of each package is described in separate file, called anybuild or build script.
any-build(7) describes anybuild conventions and writing guide. It also describes another actions to make new package buildable by any, such as sources preparation.

Each anybuild is kept in separate directory, named after the full name of the package, including version.
The default any.conf defines the root of this directory as ports/packages/.

To create anybuild for package hello-world-1.0, one should create directory ports/packages/hello-world-1.0/ and create file ports/packages/hello-world-1.0/hello-world-1.0.build. There one should put methods and variables, describing the build process.

The base minimum to get package buildable is:

-
src_config(3) method
-
src_compile(3) method
-
src_install(3) method
-
DEPEND variable with build time dependencies
-
patches for package sources in directory build/patch/package-name-1.2.3/

The simple illustration of anybuild with described set:

#!/bin/sh

DEPEND="
>zlib-1.2
>=gmp-4.3.2
"

src_config()
{
    ./configure \
        --enable-shared \
        --enable-static \
        --enable-utf
}

src_compile()
{
    make
}

src_install()
{
    make install DESTDIR=$D
}

The first part of build script contains auxiliary information, such as shell interpreter name, copyright, comments from revision control system and so on. The example above contains just string #!/bin/sh
The second part contains global variables, defined by the anybuild. Global variables are assigned at the beginning of the script before all methods. In that case they will be correctly initialised for all interfaces.
The third part contains methods, defined by the build. The methods are part of any-map(7) scheme, which contains overall sequence of build methods. It can be viewed in any/corelib/map.sh.

The content of given methods is very close to the generic build scheme, used by vast quantity of programs, distributed for POSIX operating systems. The difference is grouping commands by methods with dedicated semantic purpose.

src_config
does the configure stage: all the set up before the compiling is done here.
The most well-known variant is classic ./configure call. If there are any modifications of input data before the compilation stage, they should be done here as well. For example, it can be some edit of the generated header with sed(1). If there are variable assignments, which cannot be calculated out of method call (and placed at the top of build file), this method is best place for them.
src_compile
does the actual compiling, linking, generating documentation and everything else to produce resulting content of a package.
The wide-spread code to do this is plain call of make(1) program. It is used when compiling of the package is based on makefiles – files with content-generating directives, written in their own language. These files are integral part of package sources and are not provided by build engines of higher level, such as any(1). When package uses another inner building scheme, there will be another commands to invoke the process.
src_install
does the copying of the ready files to separate installation place.
The same classic scheme does the installation with make install call. DESTDIR argument of make points to the root installation directory, where the subtree of installed package begins. The value $D contains the empty directory, dedicated for installation of current package. That variable, among many others, is provided by engine.

Put together in single file, methods above provide spectacular picture of package building.
It is technically possible to omit each method with typical repeated implementation and not put it in the anybuild (it will be taken from the engine then). But the methods, listed above, are recommended to be present explicitly.

Variable DEPEND holds build time dependencies of current package.
Dependencies are the list of entries, separated by spaces or new lines. Single entry is the range of allowed package versions and can be one of the forms:

>=used-program-1.2.3
Package used-program must be of version 1.2.3 or higher.
>used-program-1.2.3
Package used-program must be of version strictly higher then 1.2.3.
=used-program-1.2.3
Package used-program must be strictly of version 1.2.3.
used-program-1.2.3
Package used-program must be of version 1.2.3 or higher (same as >=).
<used-program-1.2.3
Package used-program must be of version strictly lower then 1.2.3.
<=used-program-1.2.3
Package used-program must be of version 1.2.3 or lower.

Versions are compared as version strings by algorithm of sort -V.
See any-api(7) for all details, which concern the work of DEPEND and its neighbours.

The anybuild example, given above, has two entries:

DEPEND="
>zlib-1.2
>=gmp-4.3.2
"
The first entry >zlib-1.2 means package zlib with any version which is higher then 1.2. This condition matches any version, beginning with 1.2.0, continuing with 1.2.1, 1.2.N, 1.3.0 and so on.
The second entry >=gmp-4.3.2 means package gmp of exact version 4.3.2 or higher, like 4.3.3 or 4.4.

Anybuild is allowed to use variables and to call methods, described by any-api(7), and is restricted to the rest. That is due to a necessity to keep every construction of anybuild stable working: names, conventions, call results. When inner engine method changes, appropriate change of all its calls is not a problem. When method is used in lots of builds, distributed asynchronously, such change is not affordable.

Any API

any provides the set of shell variables and methods, described by any-api(7). Anybuilds rely on that API to access data, gathered by the engine, and to do their build tasks.

Extensions and further development

any-dev(7) describes principles and conventions around any code to write extensions and customisations of the entire engine.

Glossary

Working directory
Autonomous directory with full build suite. Working directories are isolated from each other and, in case of chrooted / containerised environment, from host system as well. See any-workdir(7) for more details.

Configs, configuration files, configuration options
All settings for the build process are stored in the configuration files of rdd(1) at the root of working directory. Together with ports/ directory and fixed version of any aforesaid configuration files make up exhaustive input data to initialise the build.

Ports directory
Directory with the individual build scripts for packages. It also contains materials, generic to the package set, such as etalon data for checkers. See any-workdir(7) for more details.

Package
Single unit of software to build. Name of package or list with a number of names is the second in any-do(1) launch string.

Package may be also referenced as binary package, meaning installable archive with target files and metainformation, intended for use with package manager. One source package can produce one or several binary archives. Software name inside package database may differ from original source one. In the any series of manuals the term package means source package or build-time entity, unless specified otherwise.

Package list
Plain text file with a list of packages to build them all in a single launch. Each package is on a separate line, comments and empty lines are ignored. Lists must be located at their own directory, ports/list/ as given by default any.conf config.

Build id, build data, data words
One or several words, identifying the data to use for a build. It's the first argument in any-do(1) launch string. The examples from the above commands are amd64 or amd64,packslack.

The first id, usually single short word, silently loads the whole data array with build options. That happens due to data sections in configs: they are chained together with reserved option rdd_prf_id in a manner that including of top level section (for example, amd64) involves the rest sections and their more generic data (say, all or any).
Variable AUSE inside the shell code contains all loaded data sections. See rdd(1) for details of data loading and any-api(7) for AUSE variable description.

Build profile
Single word, identifying the build inside working directory. In simple case it is equal to build id, but in others not. For build id amd64,packslack build profile remains amd64. Build profile is contained in paths to installed binary files, logs and other resulting stuff.

In general case build profile is constructed explicitly inside rdd(1) config files. The user must explicitly define which data words change the build profile and which do not.

Build map
Shell method, launched for build by default. It contains sequence of essential methods, each with its own definite unique task. See any-map(7) for more details. There can be other maps also, for example, map_autopre, which contains sequence of inner auxiliary methods to prepare the building: output starting information to stdout, create needed temporary files.

Launched methods
It is possible to launch arbitrary methods instead of predefined map method. They are submitted as the third argument to any-do(1) launch string. Methods can refer to whatever subject, not only package building. The main limitation to launched methods is absence of input parameters, but they can use shell variables, provided by any-api(7).

Anybuild, build file, build script, build
The shell file with methods and variables, unique for the package. The most often methods to customise are src_config(3), src_compile(3) and src_install(3). Anybuild also contains interface variables, needed for generic build procedures, such as dependencies handling. See any-build(7) for details.

Interface method
The method with fixed name, input data and conventions of its effect. Interfaces describe the structure of some subsystem, allowing its various implementations. Different implementations of the structurally same functionality are provided by modules and extensions, and the user can choose one of them on the fly with cmd or by config file. For example, pkg_config(3) and pkg_pack(3) can be implemented with packdeb or packslack modules, resulting the binary packages in debian or slackware format, respectively.

There are build interfaces, used inside anybuilds, and engine interfaces. Build interfaces are part of any-api(7) and any-map(7). Engine interfaces are described in any-dev(7).

Any API
Set of global shell variables and methods, allowed for use inside anybuilds. Described by any-api(7).

Engine module
Enhancement of the general engine, which is turned on with explicit setting in config file or with data word, added to build id in command line. Usually it is one or several shell libraries, installed to any/lib/. But in general case module may contain new executable files and another materials as well. Module may include its own config settings, declare new interfaces and variables for anybuilds or engine, add new behaviour to standard interfaces. Each module should ship its own description.

Modules are used to keep functionality, which is not generic to include it into core engine. For example, it is the work with some outer tool, which is not needed for all users and configurations. They are also used to reduce the minimal set of included functions. anymods(7) lists modules, available in parallel with core any engine.

See also

any-workdir(7), any-do(1), any-build(7), any-api(7), any-map(7), any-conf(7), any-dev(7), anymods(7), rdd(1), any-settle(1), any-howto(7)