any do – tool to build packages by any-engine.

Synopsis

any do DATA [options]
any do DATA PACKAGES [options]
any do DATA PACKAGES METHODS [options]
any do [options]

PROGSU=/usr/bin/sudo PROGCHROOT="sudo chroot" anch do [any arguments above]
any do --help|--version

Description

any do does building tasks in working directory of any(1). It constructs shell script with the task and launches it. Data from configuration files and command line are transformed to variables and used as input data. After them libraries with shell functions are included. Then calls of needed methods are added. The result is plain standalone script with build code, which can be launched manually. For each package the separate script is created in directory var/dump/packagename_postfix. packagename is named after the name of the package, and postfix depends on used data.

The anch do command is the wrapper around any-do(1), which performs the actual work. Commands share all syntax and options. The purpose of the wrapper is to enter the containerised environment with chroot(1) or other tool before the launch of rdd. So the user stays out of container, but all his build tasks are sandboxed. User account is recreated automatically inside container, thus working files have proper permissions.

Entire concept and all details of any do are also correct according to anch do. This manual page describes exclusive properties of anch do as well. So it is possible to study the behaviour of main building commands in single place.

Essential launch

any do syntax contains three positional arguments: DATA, PACKAGES and METHODS. Each of them has fixed position in command line. Some or all of them may be ommited, but for the remaining ones the order is the same. When some argument is not given, its value is taken from configuration file. If any of three arguments has no resulting value whether from command line or from config, launch of any do terminates.

The most frequent form to launch any do is:

    any do amd64 hello-world-1.0
This accords to the format:
    any do DATA PACKAGES

Here amd64 is the first argument with DATA. It is used as start identificator to read all input data from configuration files inside rdd.conf.d/ directory and from rdd.def file. That data is transformed to shell variables, which initialise the whole build environment. hello-world-1.0 is the second argument with PACKAGES, single name in this case.

The full output of build process is redirected to a log file by default. Single string with summary result is printed for each package instead. The possible results of the command above are:

hello-world-1.0 |                                                     (12)   OK
The build of hello-world-1.0 has succeeded in 12 seconds and status OK is printed.

hello-world-1.0 |                                                     (3)  FAIL
/home/user/workdir/build/log/amd64/hello-world-1.0.log
The build of hello-world-1.0 has failed in 3 seconds and status FAIL is printed. The path to the full log file is shown below to examine an error reason.

hello-world-1.0 |                                                     (0)  SKIP
The build has been skipped by the engine with status SKIP. That is caused by special condition in the package: it may be marked as non-buildable for chosen architecture, OS or some other used dataword.

any do usage requires enough components to be installed inside working directory to host chrooted build tasks: base system, toolchain and developer tools. The build of certain large packages may require lots of additional programs.

In general case, any do can be used for arbitrary operations on packages, not only for the building. The possibilities of the usage are defined by shell methods, available in working directory. Base methods, provided by any(1), are described in any-map(7). More methods are available from anymods(7).
To launch any do with single method src_store:

    any do amd64 hello-world-1.0 src_store
This accords to the format:
    any do DATA PACKAGES METHODS

Here third argument src_store accords to METHODS, single entry in this case. When argument with METHODS is not given explicitly in command line, it contains default method map.
The standard output with launch of separate methods is the same, as with default one.

It is possible to launch any do just with data assignment:

    any do amd64
This accords to the format:
    any do DATA

In the case above PACKAGES argument will be initialised with all.src package list. That name is reserved to keep names of all packages, available for successfull build in current project.

Besides three essential positional arguments, described above, it is possible to give separate options in command line for any do. Each option is single data entry, the same as in configuration files. Options are in format of key=value. All options, provided in command line, redefine values of same options from any configuration files. Options may be put at any position of command line after any do.
All available options of engine core are described in any-conf(7). Options, used by additional modules, are described in their manual pages.

    any do amd64 hello-world-1.0 any_src_dir=newdir/srcstatic

Here new value of any_src_dir option is given, so the directory with package sources is changed from default value build/srcstatic/ to new value newdir/srcstatic/. Note the relative path: the engine automatically prepends the paths with absolute path to the working directory, so all materials could be used only from inside of it.

All positional arguments, described above, may be provided as a list of values instead of single one. List entries are connected with comma symbol without spaces between.

To build three packages onepack-1.0, twopack-2.0, threepack-3.0:

    any do amd64 onepack-1.0,twopack-2.0,threepack-3.0

To execute two methods src_fetch and src_store for each of packages onepack-1.0 and threepack-3.0:

    any do amd64 onepack-1.0,threepack-3.0 src_fetch,src_store

To add properties plus and minus to the main build id amd64:

    any do amd64,plus,minus hello-world-1.0

Additional launch modes

There are several important additional modes of any do usage, provided by default engine set.
The sections Data gathering and Modular code inclusion below explain in more details, why these additional modes have that syntax.

Output to terminal

To see the output directly on the terminal without redirection to log file, add output to the launch:
        any do amd64,output hello-world-1.0

New log files are not created with that usage. The launch in that way can be used when there is a need to preserve old build logs.

Disable dependencies check: staticdeproot

Turn off the build dependencies handling.
Instead of separate context directory for each package, populated with its dependencies, single directory with globally shared context is maintained. Each package installs its files to that directory.
        any do amd64,staticdeproot hello-world-1.0

For the example above single directory build/context/amd64/static/root/ will be used.

To build successfully in this mode, one need to populate context directory with all packages, which may be required by the desired program. It can be done in two ways:

build all packages sequentially in the same mode;
deploy them from pre-built binary packages in the same mode.
The first case is straightforward:
        any do amd64,staticdeproot needed-packs.src
For the latter case it will be:
        any deploy amd64,staticdeproot needed-packs.src
See any-deploy(1) for description of that command.
If required binary packages are available, the second way is much more fast and simple.

The mode is used in the first turn to gather the information about build dependencies for the first time. Specialized module any-builddep(1) is used for that.
Static directory with all available packages is created by one of two ways above, required package is built with the module, and build dependencies are caught automatically.

Verify the build results: sane

The mode sane exists to verify the resulting binary package with installed sanity checkers. This mode does not launch new build, but uses its results.
        any do amd64,sane hello-world-1.0

Any output after sane means some potential or real problem. No output is no worries.

More modes

Modules from anymods(7) may add further extensions and modifications of the standard build behaviour.

User and container management

anch do by default uses the same account for build process, as original user has.

The script any/bin/dobefore automatically creates account of plain user inside containerised environment. That account has the same name, id, group and group id as the original user, who has launched anch do. That is done with simple records to local etc/passwd and etc/group files.

If command sudo(1) had been found inside container, it is used to switch from root account, got after chroot(1), to original user with help of preparations above. Default location of sudo is /usr/bin/sudo. To point to another location of the command, variable PROGSU may be used:

        PROGSU=/usr/pkg/bin/sudo anch do amd64 hello-world-1.0
The new value can be as well exported in shell once, the effect will be for all further launches:
        export PROGSU=/usr/pkg/bin/sudo
        anch do amd64 hello-world-1.0
        anch do amd64 package-1.2.3
Another implementation of sudo(1) functionality is possible, such as doas(1).

To turn off automatic user switch, assign non-existent prog name to PROGSU:

        PROGSU=dumb anch do amd64 hello-world-1.0
For the command above the owner of build files will be root. Mind that empty value of PROGSU has no effect, as default value will be taken in that case.

PROGCHROOT variable keeps program used to enter the containerised environment. By default it is sudo chroot. Unlike PROGSU this value does not contain absolute paths. While PROGSU (/usr/bin/sudo in example above) is launched inside container already, PROGCHROOT is launched in generic host environment, thus taking user's PATH into account. To use fakechroot instead of default implementation:

        PROGCHROOT=fakechroot anch do amd64 hello-world-1.0

Package listing

Packages can be listed with comma in command line argument. But instead their names may be placed in plain text lists and that list names may be given in command line. That form of listing allows handling large sets of packages.

Lists must be located in dedicated directory, ports/list/ by default. That directory can be changed with rdd_list_path option. The convention about list names is to end them with .src, so they are well seen in command line. If a list is generated by some other tool, its name should be ended with .txt, so it distincts from statically saved lists.

Lists must contain package names each on new line. Empty strings and comments, beginning with #, are skipped.

The simple example of work with a list short.src:

    cat ports/list/short.src

one-package-1.0
# added later
two-package-2.0

# added last, should be checked
three-package-3.0

Launch build for the list:

    any do amd64 short.src

one-package-1.0 |                                                     (33)   OK
two-package-2.0 |                                                     (5)    OK
three-package-3.0 |                                                   (14) FAIL 
/home/user/workdir/build/log/amd64/three-package-3.0.log

So the list name, given in command line, is expanded to package names, containing inside.

Lists may be nested, so instead of a package they may contain names of other lists. In that case they are all expanded in order of inclusion.

    cat ports/list/all.src

core.src
main.src
last.src

# sort this out
fresh-package-0.99
fresh-modules-0.1.11

To list available packages, lspkg(1) util may be used.

List the content of short.src:

    lspkg amd64 short.src

one-package-1.0
two-package-2.0
three-package-3.0

List everything available for amd64 (two equal forms):

    lspkg amd64
    lspkg amd64 all.src

core-one-package-1.0
core-two-package-2.0
core-three-package-3.0
main-one-1.0
main-two-2.0
main-three-3.0
last-one-1.0
fresh-package-0.99
fresh-modules-0.1.11

Package listing pitfalls

lspkg(1) does not check the status of a package, set up by KEYWORDS or other shell conditions, as it does not load any shell code at all (if no options are given). So if a package is masked for amd64, it will be listed by lspkg amd64 nevertheless.

When changing the list location directory with rdd_list_path, it is important to leave access to default lists, such as all.src. If list directory is changed and the new one does not contain all.src, functions with its usage will not work, such as resolve_rebuild(3).

Data gathering

Configuration files of any do contain the same key=value pairs, as options for command line. But files keep them grouped by sections, each starting with section name, as in:
[name]

some_option = value
...

[other_name]
another_option = another value
When first argument of any do is some word, amd64 for example, that means the section with name amd64 will be read from configuration files. All its options become actual input data for the build.

More interesting thing happens, when a section contains option rdd_prf_id. That option keeps names of new sections, which should be read in turn. So, when amd64 contains the string:

[amd64]
rdd_prf_id = x86, toolchain, hardened
after reading of amd64 section rdd(1) will continue to read the content of hardened, toolchain and x86 sections, if there are any. When rdd_prf_id will be met somewhere again, it will load new sections again and repeat deepening read.

The same thing happens, when several datawords are given in command line, like that:

    any do x86,toolchain,hardened hello-world-1.0

Key features of the reading are:

-
datawords from the list are taken from right to left;
-
when the same options with different values are met, the result is taken from the option, which was met first;
-
already met datawords are ignored (no cyclic reading).
So, if section hardened in the example contains value use_hardened_patch=1, that value will be saved to final result despiting all other values of use_hardened_patch, which may be under other sections. That's because dataword hardened is the most right in the launch and has the highest priority.

Dedicated variable rdd_prf_all is generated by rdd(1), which contains all read datawords sections in the order of inclusion, the most right is first met section. The list is space separated. any(1)-engine saves that list to variable AUSE, which is used in all shell methods.

Sections in configuration files are chained together, so that picking appropriate one of them, such as amd64, will automatically load the whole bunch of needed options and settings. When another configuration is needed, they may be rearranged in config files or the command line launch may set them up in another order. If one need to launch some build for amd64 without hardening, and section x86 contains generic non-hardening options, he may perform:

    any do amd64,x86 hello-world-1.0
Now generic options come first and will redefine non-needed values of hardening section inside amd64.

There are the following sources of data available, in priority order:

-
rdd_prf_id in command line;
-
first argument DATA in command line;
-
file rdd.def in the root of working directory;
-
file atom.conf in the directory with build script for a package (ports/packages/hello-world-1.0/atom.conf)
-
all files *.conf inside rdd.conf.d/ in the root of working directory;
-
only if there is no rdd.conf.d/ in the working directory, and environment variable RCR_ROOT is set to a valid directory, files *.conf inside $RCR_ROOT/../etc/rdd and $RCR_ROOT/../../etc/rdd directories.

To view the result of gathered data, datardd(1) utility may be used. It outputs whole set of read data. It needs single argument with datawords.

    datardd amd64

To look only at loaded datawords:

    datardd amd64 | grep rdd_prf_all

The output of datardd(1) is not correct shell language. To view the constructed ready-to-use shell script, dumprdd(1) is used:

    dumprdd amd64 | grep rdd_prf_all

dumprdd(1) prints not only the input data, but the whole constructed script (without method calls). It makes sense to grep that output with needed value, as along with inlined listing of api.sh it is quite large.

Data gathering pitfalls

There is side difference between launches:
    any do amd64,more hello-1.0
    any do amd64 hello-1.0 rdd_prf_id="more"

In the first case additional dataword more will change the directory, where rdd(1) stores its constructed scripts and log file. Instead of var/dump/hello-1.0_amd64/ it will become var/dump/hello-1.0_amd64_more. That happens because rdd uses first DATA argument to name dump directory.
In the second case the dump directory will stay the same var/dump/hello-1.0_amd64/. That means the previous log file after launch with more will be overwritten. Note that in your script writing. To avoid that effect, add more dataword to rdd_prf_entry instead:

    any do amd64 hello-1.0 rdd_prf_entry="amd64,more"
When you actually don't know what the old datawords are (inside some generic wrapper, for example), that workaround may be used:
    eval "$(dumprdd "${@}" | grep -e '^rdd_prf_entry=' )"
    any do "${@}" rdd_prf_entry="${rdd_prf_entry},more"

Modular code inclusion

The list of data sections, gathered from config files and options by rdd(1) (so-called datawords), is used to determine which additional modules of any(1) engine should be used in the launch.

Simple example of the launch:

        any do amd64,packdeb hello-world-1.0
The launch above will result in inclusion of module any/lib/packdeb.sh and creation of binary package in debian format inside pkg_pack(3) method.

        any do amd64,packslack hello-world-1.0
The string above includes any/lib/packslack.sh module and creates slackware binary package instead.

The list of used data words is stored in AUSE variable (in the order of their inclusion, the first one is the most right). For each used data word WORD and each used PATH the file PATH/WORD.sh is checked and included, if there is such one. The following paths are checked for modules in order:

any/include/
any/lib/
ports/packages/PACKAGE/

There is no need to assign data words as in example above explicitly in command line to get the effect of modular inclusion. Almost all used data words are generally taken from config files, but they participate in the described mechanism as well.

To add some functional module, which can be loaded and used by demand with the whole engine, one should add shell file any/lib/MODULE.sh. If the module declares global variables, proposed for use by others, the module with such variables must be placed inside any/include/. If actions of this module should take place only for some package, then it should be placed in ports/packages/PACKAGE/MODULE.sh.

The option any_profile_engine inside any.conf contains directories, where modular inclusion takes place. By adding new entries there one can gather modules from more directories or even exclude standard directories inside any/ from gathering. See any-conf(7) for further description of any_profile_engine.

Automatic container scripts

While entering container, anch do executes several scripts, residing in the working directory. These scripts are to set up the container environment and keep it consistent, so the user should not do it manually. As the scripts are local and not shared system-wide, each working directory may have personal set of settings to keep.

If it is needed to turn off some script, removing its executable permissions is enough:

        chmod -x any/bin/enter

any/bin/dobefore
Script is executed before chrooting, thus it has access to entire host filesystem. It can be used to prepare access to host or network resources inside container.
By default it creates local user account inside chroot with the same id and group id, as current user has. Local account is used to emulate authentic user.
any/bin/enter
Script is executed in chroot as separate process from any do launch (environment is not shared). It creates device files, needed for the build, and mounts appropriate filesystems to directories /proc/, /dev/shm/ and /dev/pts/.
any/bin/doafter
Script is executed out of chroot after any do has finished in separate process. It has no default content. It can be used to clean up the working directory.

Scripts are given WORKDIR variable, containing the root of working directory. The variable is not exported anywhere else besides these scripts.
All scripts above normally do not have access to data, generated by any(1) engine, such as API variables. They are expected to perform tasks, which do not require in-depth build-specific knowledge.
However, it is possible there to get all variables, available inside build environment. All scripts recieve the same set of input arguments, as any do itself. So the read for separate config option can be done as such:

    eval "$(dumprdd "${@}" | grep -e '^any_some_option=' )"
For the access to whole API:
    eval "$( dumprdd "${@}" )"

The access to any(1) resources from any/bin/ scripts should be done only when it is crucially neccessary. The better alternative is usage of map_group method, which is executed once for a session alike any/bin/, but has whole engine context.

Options

Incomplete list of rdd(1) options, mentioned or touched by this manual.
rdd_prf_id=word1[,word2,...]
Add new data words to overall list and read new sections with such names from config files.
rdd_prf_all="word1 word2 ..."
List of used datawords, generated by rdd. Used as AUSE variable inside any(1).
rdd_list_path=dir
Set up the directory with lists of packages.
rdd_prf_entry=word1[,word2,...]
Explicit option with primary list of DATAWORDS, independent from position in command line.
rdd_list_entry=object1[,object2,...]
Explicit option with list of PACKAGES, independent from position in command line.
rdd_map_entry=func1[,func2,...]
Explicit option with list of METHODS, independent from position in command line.

Environment

PROGSU
variable contains the full path to sudo(1) command, installed inside container (not to enter the container). It it used to switch from local root, got after chroot(1), to proper original user.
Default value is /usr/bin/sudo.
PROGCHROOT
variable contains command to enter isolated environment.
Default value is sudo chroot.

See also

rdd(1), any(1), any-workdir(7), any-map(7), any-conf(7), anymods(7), lspkg(1), datardd(1), dumprdd(1)