any do DATA [options]
any do DATA PACKAGES [options]
any do DATA PACKAGES METHODS [options]
any do [options]

PROGSU=/usr/bin/sudo PROGCHROOT="sudo chroot" anch do [any arguments above]
any do --help|--version

any do constructs a script in shell language, which builds a package. One should reside in a working directory of any(1).

The configuration is assigned for the launch, which will be used for the build.

The data from configuration files are read by configuration name and are saved inside build script as input data in form of variables in shell. The options from command line are added there as well.

After data the script includes the libraries of any engine, they contain functions for building.

Then the methods are called sequently, which do the package building.

Package series may be built at once. Package set is assigned in command line or in text file. Generated script for each package is available to relaunch it manually afterwards. The given scripts are stored in dir

var/dump/packagename_postfix.

packagename matches the name of a package, postfix reflects the chosen configuration.

anch description

The command anch do exists, which is total analogue of any do. They share syntax and options. The aim of anch is to enter the isolated environment before the execution of its command, in the stated case before the do, which launches the build.
With anch do it is possible to act from your own user in proper environment, but build processes will go inside isolated and more reliable environment.

The user account is created automatically inside container, imitating the account of current user. So the permissions and owner of files, created inside isolated environment, will match the current user.

Arbitrary command from any set is callable with anch, and it will be executed in isolation:

any do and anch do

any deploy and anch deploy

...

The given manual besides other things describes features, which are unique for the launch with anch, as well. All information about any do concerns anch do.

any do reads three main unnamed arguments: DATA, PACKAGES and METHODS. They are located in order relatively to each other in command line. Some of them may be omitted, but the remaining ones keep the same order in line. When the argument is not given, its value is read from configuration file. If any of three arguments has no resulting value neither from command line nor from config, the launch of any do is terminated.

The options like option=value may be located at any place in command line: before main arguments DATA, PACKAGES and METHODS, after them or between them.

The most frequent form to launch any do is:

    any do amd64 hello-world-1.0

This matches the format:

    any do DATA PACKAGES

amd64

the first argument with data, DATA. It is used as the starting identificator to read all input data from configuration files in rdd.conf.d/ dir and from rdd.def file. These data are translated into shell variables, which initialise entire build environment.

hello-world-1.0

the second argument with package names, PACKAGES. There is a single package in the given case, specified directly in line.

The full output of a build process is redirected to a log file by default. The single string with brief summary is printed to stdout. The examples of possible results for the command above are:

hello-world-1.0 |                                                     (12)   OK

The build of hello-world-1.0 has succeeded in 12 seconds and status OK is printed.

hello-world-1.0 |                                                     (3)  FAIL
/home/user/workdir/build/log/amd64/hello-world-1.0.log

The build of hello-world-1.0 has failed in 3 seconds and status FAIL is printed. The path to the full log file is shown below to examine an error reason.

hello-world-1.0 |                                                     (0)  SKIP

The build has been skipped by the engine with status SKIP. That is caused by special code in the package: it had been marked as non-buildable for chosen architecture, OS or some other used dataword.

anch do usage requires complete host filesystem to be deployed in working directory, enough to keep isolated build processes: base utils, toolchain and developer tools. With regard to package requirements the host may demand lots of additional programs.

Launch of arbitrary methods

In general case any do can be used for arbitrary operations on packages, not only for the building. To perform some action one should have method, implemented inside on of shell libraries, included for chosen configuration. The list of libraries, included in current build, can be seen in the beginning of build log. The list has the form:

Loaded modules:
/any/include/atomprint.sh
/lfs/include/lfs.sh
/lfs/lib/lfs.sh

Directories */include/ have files with declarations of variables. Directories */lib/ have libraries with methods.

With idea of method with needed effect one may launch it apart. The example of single src_store method execution for the package hello-world-1.0:

    any do amd64 hello-world-1.0 src_store

The given example will store all stuff for hello-world-1.0 in regular place: archive sources, patches, data with inner version. If something named has been downloaded from the network, then after the work of src_store(3) the package is ready for the autonomous build without access to servers.

The launch above matches the format:

    any do DATA PACKAGES METHODS

The third argument here accords to method names METHODS, which are the single method src_store in the given case. When argument METHODS is not explicitly given in command line, it equals to default method map. map performs entire package building from auxiliary checks until the creation of binary archive.
The standard output after the launch of separate methods is the same as without explicit assigning of methods.

All methods of the main engine are described in any-map(7), which perform the build. They can be launched separately, as in the example above.
Additional modules and libraries may be added from anymods(7).

Default packages

It is possible to launch any do just with data assignment:

    any do amd64

This matches the format:

    any do DATA

The argument PACKAGES will be initialised with package list all.src in the example above. all.src is reserved name for the list, containing all available packages of the project.

Command line options

Besides three main positional arguments, described above, it is possible to assign particular options in any do command line. Each option is single data element like in configuration files.

Options have the format of key=value.

All options, given in command line, redefine the values of the same options from any sections inside any configuration files.

Options may be placed at arbitrary position in command line after any do.

All available options of engine core are described in any-conf(7).

    any do amd64 hello-world-1.0 any_src_dir=newdir/srcstatic

Here the new value of any_src_dir option is given. So the directory to search for package sources is changed from default value ports/srcstatic/ to the new value newdir/srcstatic/. Additionally for this example note the relative path: the engine prepends the paths automatically with the root of working directory.

Additional modules may provide their own options, described in according manual pages.

Config options

To store working option and not to mention it each time in command line, one may save it inside rdd.def file in the root of working directory:

[amd64]
any_src_dir=newdir/srcstatic

The given entry is equivalent to mention any_src_dir=newdir/srcstatic in command line for amd64 configuration:

    any do amd64 hello-world-1.0 any_src_dir=newdir/srcstatic

For arch-independent options, like sources properties, it is more prefered to use more generic section id:

[id]
any_src_dir=newdir/srcstatic

Value lists

All positional arguments, described above, may contain a list of values instead of a single entry. List elements are separated with comma symbol without spaces between.

To build three packages onepack-1.0, twopack-2.0, threepack-3.0:

    any do amd64 onepack-1.0,twopack-2.0,threepack-3.0

To execute two methods src_fetch and src_store for each of the packages onepack-1.0 and threepack-3.0:

    any do amd64 onepack-1.0,threepack-3.0 src_fetch,src_store

To add properties plus and minus to the configuration amd64:

    any do amd64,plus,minus hello-world-1.0

Additions to configuration

It is possible to add new property to the used configuration from command line.

    any do amd64,cap32 hello-world-1.0

In the example above we use amd64 as configuration, which sets up essential properties of our build. Additionally we specify the word cap32 to the right of amd64 through a comma without spaces.

The additions to configuration provide the following effect:

There is a new word inside AUSE variable in build environment, cap32 in the example above. That means the following code under the condition will begin to work:

if ause 'cap32' ; then
    ...
fi

If configuration files have the section cap32:

[cap32]
data_cap32 = something

the data from that section will be read, that is data_cap32 option in the example above.

One may check the addition to configuration from command line outside the build environment in the following way:

        datardd amd64,cap32 | grep rdd_prf_all

Sections Data gathering and Modular code inclusion below explain in more details, how additions work and why they have such syntax.

anch do does the building inside the environment from the local user account, which imitates the original user.

The script any/bin/dobefore automatically creates a local account of plain user inside containerised environment. That account has the same name, id, group and group id as the original user, who has launched anch do. Simple text records are done for that to local etc/passwd and etc/group files.

If sudo(1) command is present inside the container, it is used to switch user account to the preared one. Without switch the user inside container after the chroot(1) would be root user. Default location of sudo is /usr/bin/sudo. To set up another path, the variable PROGSU may be used:

        PROGSU=/usr/pkg/bin/sudo anch do amd64 hello-world-1.0

The variable with a new value can be once set up in the shell as well, and the effect will remain for all new launches in the same terminal:

        export PROGSU=/usr/pkg/bin/sudo
        anch do amd64 hello-world-1.0
        anch do amd64 package-1.2.3

Another program with functions of sudo(1) is possible, such as doas(1).

To turn off automatic user switch, assign non-existent prog name to PROGSU:

        PROGSU=dumb anch do amd64 hello-world-1.0

The build process will be from root for the command above. Mind that empty value of PROGSU has no effect, as the default value will be taken in that case.

The second variant to turn off the user switch is to correct the default value of PROGSU in the anch executable itself. It can be done during the install from directory with any distro:

        ./install.sh PROGSU=

After the installation above the PROGSU will be empty by default and there will be no switching of users inside a container.

The PROGCHROOT variable keeps program to enter a container. It defaults to sudo chroot. Unlike PROGSU that value does not contain absolute paths. While PROGSU (/usr/bin/sudo in the example above) is launched inside container already, PROGCHROOT is launched in generic host environment, thus taking into account user's PATH. To use fakechroot instead of default implementation:

        PROGCHROOT=fakechroot anch do amd64 hello-world-1.0

In a similar way one can change once the default value for PROGCHROOT during an install:

        ./install.sh PROGCHROOT=fakechroot

After the installation above the default value for the program to enter the container is assigned with fakechroot. That means the containerisation is turned off, but the install archives are created with root-owned files inside.

Packages can be listed in command line argument, separated with comma. But instead their names may be placed into a text list and that list name may be given in command line. Such form allows to handle large sets of packages.

Lists must be located in dedicated directory, ports/list/ by default. That directory can be changed with rdd_list_path option. The convention about list names is to end them with .src, so they are well recognizable in command line. If a list is generated by some tool, name of a list should be ended with .txt, so it would visually distinguish from statically saved lists.

Lists must contain package names each on new line. Empty strings and comments, beginning with # symbol, are skipped.

The simple example of work with a list short.src:

    cat ports/list/short.src

one-package-1.0
# added later
two-package-2.0

# added last, should be checked
three-package-3.0

Launch of the build for the list:

    any do amd64 short.src

one-package-1.0 |                                                     (33)   OK
two-package-2.0 |                                                     (5)    OK
three-package-3.0 |                                                   (14) FAIL 
/home/user/workdir/build/log/amd64/three-package-3.0.log

So the list name, given in command line, is expanded to package names from that list.

Lists may be nested, so instead of a package they may contain names of other lists. In that case they are all expanded in order of inclusion.

    cat ports/list/all.src

core.src
main.src
last.src

# sort this out
fresh-package-0.99
fresh-modules-0.1.11

To list available packages, lspkg(1) util may be used.

List the content of short.src:

    lspkg amd64 short.src

one-package-1.0
two-package-2.0
three-package-3.0

List base of all packages, available for amd64 (two equal forms):

    lspkg amd64
    lspkg amd64 all.src

core-one-package-1.0
core-two-package-2.0
core-three-package-3.0
main-one-1.0
main-two-2.0
main-three-3.0
last-one-1.0
fresh-package-0.99
fresh-modules-0.1.11

Package listing pitfalls

The package will be listed by lspkg amd64 command, even if it is masked for amd64 configuration with KEYWORDS mechanism. lspkg(1) without special options does not check the status of a package or other conditions, described in the shell build file.

When changing the list location directory with rdd_list_path, it is important to leave access to key lists, such as all.src. If list directory is changed and the new location does not contain all.src, the functions with usage of available package base will stop to work, such as resolve_rebuild(3).

Configuration files of any do contain the same key=value pairs, as options for command line. But files keep them grouped by sections, each starting with section name, as in:

[name]

some_option = value
...

[other_name]
another_option = another value

When first argument of any do is word amd64, that means the section with name amd64 will be read from configuration files. All its options become actual input data for the build.

More interesting thing happens, when read section contains the option rdd_prf_id. That option contains names of new sections, which should be read further.

So, when amd64 contains the string:

[amd64]
rdd_prf_id = x86, toolchain, hardened

after reading of amd64 section any do will continue to read the contents of hardened, toolchain and x86 sections, if there are any. When rdd_prf_id is met inside any of them, the procedure is repeated: next batch of config sections from fresh rdd_prf_id is queued for the reading. The reading continues until all met sections are checked up.

The names of config sections with data blocks are also called data words.

Similar process is going on when several data words are assigned in command line like so:

    any do x86,toolchain,hardened hello-world-1.0

In the example above the search inside configs will start from three sections hardened, toolchain, x86.

The key features of the reading are:

-

data words from the command line or from rdd_prf_id option are read from right to left;

-

if the same option is met several times during the reading, then the value for this option is chosen from the section, which has been read more early; concerning the first rule this leads to more priority of those data words, which are placed closer to the right;

-

the single section in config is read only once (no cyclic reading).

So, if section hardened in the example contains the value use_hardened_patch=1, that value will be saved as final result despiting all other values of use_hardened_patch, which may be inside other sections. That's because data word hardened is the most right in the launch string and has the highest priority.

The specialized program reads the data, that is frontend rdd(1). It forms consistent data set by the rules above and transforms them into variables in shell language, so that code from any could use them. Besides data from config files, the frontend generates some service variables of its own. They begin with prefix rdd_.

The variable rdd_prf_all contains all read data words sections in order of their inclusion, the most right is the first included. The list is space-separated. The any engine stores the list in the AUSE variable, which is used in ause* methods.

The described scheme results in the single short id like amd64 loads complete set of all settings and options, needed for the proper initialisation.

If one needs to turn off some section, included somewhere inside on of the typical configurations, he well need to create a new section, containing options with his needed values. After that put new section at the right inside launch string:

    any do amd64,debug hello-world-1.0

The example above contains additional section debug. Write inside:

[debug]
use_hardened_patch = 0

The given combination will cancel the effect of the option from hardened section.

Data sources

There are the following sources of data available, in priority order:

-

assigning the option rdd_prf_id in command line;

-

assigning the data words in the first argument DATA in command line;

-

file rdd.def in the root of working directory;

-

file atom.conf in the directory with build script for a package (ports/packages/hello-world-1.0/atom.conf)

-

all files *.conf inside rdd.conf.d/ in the root of working directory;

-

files from exec/rdd/../etc/rdd/ or from exec/rdd/../../etc/rdd/ directories, where exec/rdd is the path to rdd executable. The given configs are read only in the case, if the working directory lacks at the same time config rdd.def and dir rdd.conf.d/.

Data view

To view the resulting data the datardd(1) utility may be used. It outputs whole set of read data. It needs single argument with configuration.

    datardd amd64

To view only the loaded data words:

    datardd amd64 | grep rdd_prf_all

The output of datardd(1) is not correct script in shell language. Script in shell language, built from data and libraries, may be viewed with dumprdd(1).

    dumprdd amd64 | grep rdd_prf_all

dumprdd(1) prints not only the input data, but the entire constructed script (without method calls). It makes sense to filter that output, as it is quite large along with inlined listing of api.sh.

Data gathering pitfalls

There is the side difference between launches:

    any do amd64,more hello-1.0

    any do amd64 hello-1.0 rdd_prf_id="more"

In the first case additional data word more will change the directory, where rdd(1) stores its constructed scripts and log file. Instead of var/dump/hello-1.0_amd64/ the dir will become var/dump/hello-1.0_amd64_more. That happens because rdd uses first DATA argument to name dump directory.

In the second case the dump directory will stay the same: var/dump/hello-1.0_amd64/. That means the previous log file after the launch with more will be overwritten. Mind that in your script writing.

To avoid the effect above, add more data word to rdd_prf_entry instead:

    any do amd64 hello-1.0 rdd_prf_entry="amd64,more"

When you actually don't know what the old data words are (inside some generic wrapper, for example), the following workaround may be used:

    eval "$(dumprdd "${@}" | grep -e '^rdd_prf_entry=' )"
    any do "${@}" rdd_prf_entry="${rdd_prf_entry},more"

The list of data sections, gathered during the reading of configuration files (so-called data words), is used to determine which additional modules of any(1) engine should be used in the launch.

Consider the example with packdeb word:

        any do amd64,packdeb hello-world-1.0

The launch above will result in inclusion of module any/lib/packdeb.sh and creation of binary package in debian format by the pkg_pack(3) method.

        any do amd64,packslack hello-world-1.0

The string above includes any/lib/packslack.sh module and creates slackware binary package.

The list of used data words is stored inside the AUSE variable in order of their inclusion, the first read word with highest priority is the most right in the list. For each used data word WORD and each involved PATH the file PATH/WORD.sh is checked. If such file is found, it is included into build script.

The following paths are checked in order for modules:

any/include/

The place for additional files with global variables.

any/lib/

The place for additional library files with methods.

ports/packages/PACKAGE/

The place for any code, specific to particular package PACKAGE.

Modular inclusion provides the opportunity to have functionality, which would be loaded only at needed configuration. Besides, different modules may implement the same functions. Depending on chosen configuration one or another implementation will be applied. In the example above modules packdeb and packslack implement the same functions for binary packages creation.

If two different functions with the same name will be included at the same time, then the last loaded will be actually used.

Modular inclusion works for all data words, not just for the ones, explicitly noted in command line, like in the example above.

Directories to search for files with global variables are set up by the option any_profile_engine inside any.conf.

Directories to search for files with methods are set up by the option any_profile_lib inside any.conf.

With new value for these options it is possible to add new dirs to the search or remove the standard search paths. Look at any-conf(7) for description of any_profile_engine, any_profile_lib.

While entering a container, anch do executes several scripts, residing in the working directory. These scripts set up the container environment to fit the build process. At the same time the user does not have to do that manipulations manually. As the scripts are installed locally, each working directory may be set up individually.

any/bin/dobefore

Script is executed before chrooting, thus it has access to entire host filesystem. For example, it may set up the access to network resources from within the chroot.

By default the script creates local user account inside chroot with the same user and group id, as the current user has. Further build is done from that account. That is for the current user to access the files easily, which have been created inside the chroot.

any/bin/enter

Script is executed inside chroot before the any do launch in separate process. The script does not share the environment with any do. It creates device files, needed for the build, and mounts virtual filesystems to directories /proc/, /dev/shm/, /dev/fd/, /dev/pts/.

any/bin/doafter

Script is executed out of chroot in separate process after any do has finished. It has no default actions. One may do some cleaning up of the working directory in that script.

If it is needed to turn off some script, removing its executable permissions is enough:

        chmod -x any/bin/enter

Scripts are given WORKDIR variable with the root of the working directory. The variable is not exported anywhere else besides these scripts.

Automatic scripts do not have access to methods and variables from API of build system. They perform auxiliary tasks, which do no require that.

Nevertheless, scripts are given the same input arguments, as anch do itself. So it is possible to read the configuration inside them independently.

The single option is read in the example below:

    eval "$(dumprdd "${@}" | grep -e '^any_some_option=' )"

If it is needed to perform something once at the beginning of the build session, and to have the access to the build environment, the code for that should be added to the method map_group. Scripts any/bin/ should not be used for that.

There are several important additional modes of any do usage and of the build system in general. They considerably alter some aspect of the building, or even perform another tasks instead of building.

Results to terminal: output

To output the entire build log instead of saving it into a file, add output to the launch:

        any do amd64,output hello-world-1.0

Log files are not created with that launch. It comes handy when one needs to build something and not to erase the previous logs.

Keep previous build directory: keepsrc

To keep untouched the directory with previous build, add keepsrc to the launch:

        any do amd64,keepsrc hello-world-1.0

Directory after previous build of hello-world-1.0 will not be cleaned, and nothing is performed with sources. That can be used during the debug of heavy and long-compiling projects.

Note as well: to avoid recompiling with make, you will need to turn off the configuration and src_config inside your build file:

src_config()
{
    return
    <old src_config content>
}

Simplistic directory layout: plainpath

The mode plainpath makes the layout of sources and build materials look more familiar to slackbuild:

./package-1.2.3/

directory with package materials: build file, postinstall script and other metainformation.

./src/package-1.2.3.tgz

main sources of a package.

./patch/package-1.2.3/one.patch

./patch/package-1.2.3/two.patch

./patch/package-1.2.3/...

patches for a package, applied by the project.

Thus build materials are placed closer to the root directory and the entire picture is more clear. That is desirable for the work with separate packages, without large database of build files and lists.

To turn on the plainpath mode:

        any do amd64,plainpath hello-world-1.0

Likely you would like to use simplistic layout together with plainroot mode:

        any do amd64,plainroot,plainpath hello-world-1.0

Disable control over build context: plainroot

To build something in simplistic and straightforward way, like in manual step-by-step process, you may do:

        any do amd64,plainroot hello-world-1.0

The plainroot mode does not check the build dependencies. DEPEND variable is ignored. Also the mode sets up the variable with root directory to host filesystem like this:

        export ROOT=/

After that our build context contains entire host filesystem, beginning with /. The same situation takes place, when the package building is performed manually.

If you have some custom directory with your build files, you may point to it instead of host filesystem:

        export ROOT=/manual/dir/
        any do amd64,plainroot hello-world-1.0

The engine does not try to write anything to ROOT in plainroot mode. You should watch for read-only usage of ROOT in your builds, when plainroot is used. If all writings to ROOT had been done as recommended in specialized methods pkg_preroot and pkg_postroot, everything will work just fine: methods from pkg_root family are not called with plainroot mode.

plainroot also automatically turns on the plaincc mode, so wrappers over toolchain are not used as well. If you need plainroot but want to use wrappers over toolchain, turn them on explicitly:

        any do amd64,plainroot,cc hello-world-1.0

Disable toolchain wrapper: plaincc

The plaincc mode makes toolchain variables CC, CXX and so on exactly the same as PLAIN_CC, PLAIN_CXX and the appropriate options any_tch_cc, any_tch_cxx and so on.

        any do amd64,plaincc hello-world-1.0

Without plaincc it is possible to use wrappers around toolchain, so CC and PLAIN_CC would differ. Wrappers may implement additional functionality, depending on the projects.

Disable individual context: staticdeproot

The separate directory with build context for each package is not created, which would be filled with its build dependencies in default mode. Instead the directory with build context becomes united for all packages. Each package after the build installs its stuff there.

        any do amd64,staticdeproot hello-world-1.0

In the example above the single directory build/context/amd64/static/root/ is used.

United directory still contains links and not real files, like in the mode with individual context.

One of the main applications of the given mode - automatic gathering of build dependencies, done by any-builddep(1) module. It tracks all files, used during the build, chooses the files from united context directory, and with them detects the packages, which should be written as build dependencies.

To build something in staticdeproot mode, it is needed to fill the dir with shared context. It can be done in two ways:

build packages in staticdeproot mode;

deploy files from binary archives in staticdeproot mode.

The first case:

        any do amd64,staticdeproot needed-packs.src

The second case:

        any deploy amd64,staticdeproot needed-packs.src

The second way is easier and faster, if needed pre-built packages are already present. See any-deploy(1) for description of that command.

Verify the build results: sane

The sane mode exists to verify the binary package with sanity checkers. The mode does not build anything, but checks already built stuff.

        any do amd64,sane hello-world-1.0

Any output after sane means some potential or real problem. The details depend on each checker. No output is no problems found.

More modes

Modules from anymods(7) may add new modes and various modifications of the standard behaviour.

Incomplete list of rdd(1) options, mentioned or affected by this manual.

rdd_prf_id=word1[,word2,...]

Add new data words to overall list and read new sections with such names from config files.

rdd_prf_all="word1 word2 ..."

List of used datawords, generated by rdd. Saved to AUSE variable inside any(1).

rdd_list_path=dir

Set up the directory with lists of packages.

rdd_prf_entry=word1[,word2,...]

Explicit option with primary list of DATAWORDS, which is independent from position in command line.

rdd_list_entry=object1[,object2,...]

Explicit option with list of PACKAGES, which is independent from position in command line.

rdd_map_entry=func1[,func2,...]

Explicit option with list of METHODS, which is independent from position in command line.

PROGSU

variable contains the full path to sudo(1) command, installed inside container. The program is not used to enter the container. The program switches from root account after the chroot(1) to unprivileged user.
Default value is /usr/bin/sudo.

PROGCHROOT

variable contains the command to enter isolated environment.
Default value is sudo chroot.

any-workdir(7), lspkg(1), any-map(7), any-conf(7), anymods(7), datardd(1), dumprdd(1), rdd(1), any(1)