The building of a single package is handled by shell method map. It resides in any/corelib/map.sh file. Main method is divided into tasks, each one in turn is handled by its own dedicated method. Methods depend on each other, so each one normally works as expected after the previous one.

The essential build tasks and corresponding interface methods are as follows:

pkg_pretend(3)

performs read-only early check of the build. For example, the work with a package can be skipped if it does not work under some OS or architecture.

pkg_setup(3)

performs early preparation of the build with permission to write files. It resets temporary files, generated during previous build session. It also prepares the wrappers for toolchain, if they are used.

pkg_context(3)

resolves package dependencies and prepares the context for the build. If declared dependencies are not found, it prints the missing part. The context is separate directory with all files from dependent packages, needed for the build.

src_fetch(3)

gets the sources available at the local directory. It can fetch them from remote hosts or take files from another local storage.

src_prepare(3)

applies patches to sources and prepares them in any other way.

src_config(3)

configures the build before the compilation.

src_compile(3)

does the compilation of source code and linkage, generates documentation and performs all other essential work.

pkg_rminstall(3)

removes the files, installed by previous build. It handles only directory of the current package, if there had been one.

src_install(3)

copies the build results to separated directory with needed layout.

pkg_install(3)

installs additional files, prescribed by build politics, or removes redundant ones. Any other preparation of files for further extraction on target system goes here.

pkg_config(3)

prepares everything for installable archives: generates and installs meta-information for package manager, splits the files to sub-packages.

pkg_root(3)

mirrors the files of current package to directory with build context, making them available to another packages. It is mainly called during the build of other packages, when they use the current one. However, the method is also checked during current build to make sure it works properly.

pkg_pack(3)

creates the binary installable packages. Single package is possible. That is final result of single build process.

The following interfaces are not called inside main map method, but used inside another generic procedures:

src_store(3)

saves input build data to local place. Input data include all sources, original and development ones, and various helper data, like inner package versions. That local place has higher priority over remote sources, so after src_store(3) call the further builds will use local materials and not try to retrieve or generate them.
The method is used to prepare autonomous build set without access to outer networks.

pkg_deploy(3)

extracts files from binary packages into inner directory, making them available as dependencies for building of other packages. Searching for dependencies is done in pkg_context(3).
Method calls pkg_pretend(3) to do the skipping check.

Method also calls pkg_root(3) to check the proper installation of files. That call is also essentially needed in staticdeproot mode (see its description in any-do(1)).

Let's examine the simple launch of:

    any do amd64 hello-world-1.0

At the top level, while performing the sample build above, the following code is executed in order:

any/bin/dobefore is called by any-do(1) before entering the sandbox (if the file exists).

any/bin/enter is called by any-do(1) inside the sandbox to set up the build environment. It usually creates device files and mounts virtual filesystems. It is up to the user to place some another actions to that file, though.
You may read more about any/bin/ scripts in any-do(1).

Method map_group is called once for the launch. It is used to do something for the whole build task and not for separate packages. Default implementation prints here the summary information about build session. Method may also maintain the host enivironment in terms of package management, which is more complex task, then simple starter ones in any/bin/enter. Host management may be done with help of modules.

Method map_autopre is called for each package. It remains, when essential actions are changed:

            any do amd64 hello-world-1.0 changed_action

In the example above map is not executed anymore, it is replaced with changed_action. But map_autopre remains as it was. So the described method contains simple preparations, which are needed for each package before any kind of actions. For example, it prints the summary string about the package to output.

Method map is called for each package and does all the actual build.

Method map_autopost is called for each package. It is alike map_autopre, but at the end. Default implementation prints status of the finished build and spent time.

Method map_postmortem is called for each package, returned non-null status. By default it does the same, as map_autopost.

any/bin/doafter is called by any-do(1) at the end of entire build session out of the sanbox (if the file exists).

With rdd(1) usage, the sequence remains the same, but without any/bin/ scripts.

Now let's walk through the same launch with more details about each stage.

    any do amd64 hello-world-1.0

Chosen sample name hello-world-1.0 and profile amd64 are shown in all illustrative paths below, so they are real-launch examples. Build tasks of map remain the same conceptually, but corresponding methods may behave differently in dependence of loaded extending modules. Details for map methods are given for core engine implementation. Some tasks are left for additional modules on purpose and core engine has only stubs for them. In that case the task is described as "possible" or "may be done".

Here steps of the example build go:

any/bin/dobefore does those settings for working directory, which require access to entire host filesystem. For example, it can copy network configuration files to local etc/, so that network would be available inside container.

any/bin/enter creates device files and mounts virtual filesystems inside container as chrooted privileged user.

map_group possibly updates packages, installed in host environment. This is done as privileged user. Then it prints to terminal the data for build session: date, used profile, included datawords, architecture, toolchain and other values.

map_autopre creates timestamp and sets up needed inner engine variables. Timestamp is used to calculate the time, spent for the build. Here the log file build/log/amd64/hello-world-1.0.log is prepared: real file var/dump/hello-world-1.0_amd64/log, created by rdd(1), is taken and symbolic link, pointing from build/log/amd64/hello-world-1.0.log to it, is created.
Then the name of the package hello-world-1.0 is printed along with attributes, used for build: for example, make(1) parallel options. Also the extended summary about the build is printed to the log file.
The check by KEYWORDS is done: if that variable declares non-supported configuration, and chosen configuration fits that declaration, the further build is skipped.
Finally, attempt to enter the directory build/work/amd64/hello-world-1.0 is done (which would be rather more useful in launch with separate methods, then for described complete sequence).

pkg_pretend does the check by KEYWORDS as well. The duplication is historical and may be changed.

pkg_setup removes temporary files, left from the previous builds. It may also prepare the wrappers for toolchain, if they are used.

pkg_context handles directory with build context. That context contains symbolic links to include and library files of other packages, used at the build of the current one. Some files are hard-copied, so they can be edited to work correctly for context directory.

Firstly, method removes previous directory. Then, if build/context/amd64/platform/ exists and contains some tar files inside, these archives are extracted to context dir of the package, located at build/context/amd64/hello-world-1.0/ and available as ROOT in build scripts. The content of platform context should be needed for every packages of the project. That can be files of toolchain libc or the larger non-packaged entire filesystem, when using pre-defined non-granulated program platform. The same platform context is used for all buildable packages in the project.

After that dependencies for the current package are resolved. Method takes variable DEPEND, reads package requirements from there and searches fitting packages in build/image/amd64/. If DEPEND contains entry >=zlib-1.2.8, method expects to find directory build/image/amd64/zlib-1.2.8/, build/image/amd64/zlib-1.2.9/ or another appropriate one, hitting the range. The directory build/image/amd64/zlib-1.2.8/ must contain installed binary package zlib-1.2.8. If zlib-1.2.8 had been previously built with any, that is the case. Symbolic links are created from needed real files inside found directory to context directory.
The standart set of files, used for context, is defined by any_program_prefix_list and any_root_tracked_dirs options. Default values define set of usr/include and usr/lib directories. For more detailed description of context creation, see root_dep_populate(3) and description of any_root_* options in any-conf(7). Moreover, each package may declare its own addition to context it desires to provide for other packages, see pkg_root below.
Each entry from dependencies adds something to package context. If the package for some dependency is not found in build/image/amd64/, the build process stops with failure.
Finally, build/context/amd64/hello-world-1.0/ contains mini-root filesystem with files, needed for the build. It is restricted by declared dependencies, so there are no files from unaccounted sources there. Build engine automatically passes paths with includes and libs from that root directory for toolchain and configure scripts, so that compilation procedures would use them.

src_fetch removes the directory build/src/hello-world-1.0, which possibly remained after the previous launch. It is also possible to retain this directory with any_builddir_keep=1. The option is useful for debugging. Then the method retrieves the original sources of the package in the form of tarball. It can do it from local or remote host, by different protocols, which are defined by modules. In simple case packed original sources can be placed manually in ports/srcstatic/hello-world-1.0.tgz, where method will find them further. Main tarball extenstions .tgz, .txz, tbz2 are supported. Plain directory with sources may be placed to ports/srcstatic/hello-world-1.0/ as well. Tarball may be absent, if sources are entirely managed by revision control system, which is handled in the next step. When original sources are found somehow, they are extracted or copied to directory build/work/amd64/hello-world-1.0/, which is available as S variable in the engine.
Then the development sources are searched for. These are sources for the package, which are managed locally by git, cvs or other revision control system. Core engine does not have implementation for them. The local development may change some subsystem of the package or work with its entire sources. When something matching is found, it is downloaded to build/src/hello-world-1.0/.
Then the inner version of the package is written. When development sources are in use, the number is calculated automatically, basing on the revision history (the logic is implemented by engine module, handling that revision system). It is possible to prepare the inner version of the package manually in the text file ports/meta/all/hello-world-1.0/inner_version. In that case it will be used with highest priority, ignoring other sources. The final result, used by further methods, resides in build/tmp/amd64/hello-world-1.0/inner_version.
Then the patches for the package are prepared in build/patch/hello-world-1.0/. If there are patches in ports/patch/hello-world-1.0/, they will be taken. Otherwise patches from ports/packages/hello-world-1.0/patch/ are checked and taken. Otherwise patches are generated from development sources, if there are ones. At any case the final set of patches lies in build/patch/hello-world-1.0/.
The directory ports/packages/hello-world-1.0/files/ is checked. If it is present, and directory build/patch/hello-world-1.0/files/ is absent after all previous preparations, ports/packages/hello-world-1.0/files/ is copied to build/patch/hello-world-1.0/files/. That allows to keep non-patch files in separate directory, but have unified access to them with PATCHDIR variable from anybuilds.
At the end of this method current directory is changed to S, so tools of build system like make(1) and configure are able to work.

src_prepare applies patches from build/patch/hello-world-1.0/ to sources, residing in build/work/amd64/hello-world-1.0/. By default, it applies all found patches in the order of ls(1) output. If variable PATCHLIST is set, only patches from PATCHLIST are applied.
If PATCHLIST/files/ is present, its contents are copied without modifications into the resulting sources. The dir is used for handling binary or large files, which do not suite well for patch technique.
Additional automatic preparation of the sources may be done in the method, like installation of configure caches.

src_config does the configuring of the package sources. The content of this method is normally defined by individual build scripts.

src_compile does the compilation of source code and linkage, generates documentation and performs all other essential work. The content of this method is normally defined by individual build scripts.

pkg_rminstall removes the directory build/image/amd64/hello-world-1.0/, if it has remained after previous build. That dir is available as variable D in build scripts.

src_install does the installation of ready files to destination directory build/image/amd64/hello-world-1.0/, available as D in build scripts. Technically it is possible to install package anywhere, but installation outside D should never be done except for very specific cases. Engine relies on installation into separate D. The content of this method is normally defined by individual build scripts.

pkg_install removes all files with .la extension from D. That files contain lots of hardcode inside, together with libtool badly mess up the build strings and therefore should be eliminated.
Then text files inside D are searched for paths like ROOT or D. They are replaced with short paths, available at target filesystem. For example, if installation hardcodes path /build/image/amd64/hello-world-1.0/usr/lib in the installed file, it will be replaced by /usr/lib.

pkg_config optionally splits up the installed package files to subpackages, if it contains several binary subpackages in BPN or BP variables. The module support is needed for that as well.
Then meta-information for binary package (or several subpackages) may be generated. Its format depends on the chosen format for binary archives. During generation of meta-information inner version of the package may be used.

pkg_root installs pre-defined set of package files to context directory, or ROOT. Installation here is actually creating the symbolic links from D to ROOT. That way several filesystems can be created in fast and space-efficient way, each filesystem containing its own set of packages.
At described stage the method mirrors the files of current package to its current ROOT. That is done to make sure the method works correctly and provides needed files. But at the moment current ROOT is not used for building anymore. The main usage of pkg_root is inside pkg_context of other packages. When other package outer-1.2.3 uses the current hello-world-1.0, it will call the pkg_root of hello-world-1.0 to fill in the ROOT directory of outer-1.2.3.
With current method each package may describe, what files it provides to other packages. Typically that is standart set (includes, libraries), in which case no code for pkg_root in anybuild is needed. When package performs some personalised procedures for ROOT, its build script contains that actions, usually in pkg_postroot or pkg_preroot methods, which are additions to pkg_root default implementation. In rare cases anybuild may declare completely new pkg_root, when typical engine code does not fit.
See description of pkg_context for more details on context directory creation.

pkg_pack creates binary installable packages. In simple case it takes files from D and creates single archive. If several subpackages are declared in BP or BPN, method expects to find binary files for each of them inside IMGDIR or build/image/amd64/. For example, with two declared subpackages BPN="hello world" binary files will be taken from build/image/amd64/hello-1.0/ for first subpackage and build/image/amd64/world-1.0/ for second one. The directories are normally created by pkg_config method. As file name with archive may contain inner package version, it is read as well by the engine here.

map_autopost prints the status of finished build. If the process has reached final point, it is considered successfull. Here status OK is printed along with time in seconds, spent for the build. This method is called at the end of each package.

any/bin/doafter possibly cleans up some stuff after the whole build session, which may include many packages. It is called by any-do(1), if the file exists.

Many methods of map call mandatory submethods before and after actual implementation. These submethods are named after the main method with pre and post prefixes before the main part of the name. For example, pkg_install(3) method has according pkg_preinstall and pkg_postinstall methods.
Submethods are reserved for usage in anybuilds and must have empty implementation in the engine. When a package needs customized behaviour of some map method, needed submethod is declared inside build script and new behaviour is placed there. That scheme allows to add custom per-package code without repeat of main engine functional. Which methods actually have submethods, see below.

Like for individual build scripts, reserved submethods for engine extensions exist. These submethods are named with _extend and _extend_pre postfixes. For example, pkg_install(3) has according pkg_install_extend and pkg_install_extend_pre methods.
These submethods must be empty in the engine core, but have the actual implementation in some module. If some module wants to add customization to essential map method for all packages, while it is loaded, it defines such submethods.

Here is the scheme of all essential map methods, their submethods and related engine interfaces:

-

pkg_pretend(3)

-

pkg_pretend_extend_pre

-

keywords_check(3)

-

status_check(3)

-

pkg_pretend_extend

-

pkg_postpretend

-

pkg_setup(3)

-

pkg_presetup

-

pkg_setup_extend_pre

-

prepare_clean_tmp_timestamp(3)

-

pkg_setup_extend

-

pkg_postsetup

-

pkg_context(3)

-

pkg_precontext

-

pkg_context_extend_pre

-

root_clean_by_fire(3)

-

root_platform_populate(3)

-

root_dep_populate(3)

-

depend_resolve_current_atom(3)

-

root_setup_paths_to_install(3)

-

get_plain_tracked_dirs(3)

-

get_installed_tracked_dirs(3)

-

pkg_context_extend

-

pkg_postcontext

-

src_fetch(3)

-

src_prefetch

-

src_fetch_extend_pre

-

fetch_build_remove(3)

-

fetch_remote_src(3)

-

fetch_local_src(3)

-

fetch_dev(3)

-

fetch_dev_inner_version(3)

-

fetch_gen_patches(3)

-

src_fetch_extend

-

src_postfetch

-

src_prepare(3)

-

src_prepare_extend_pre

-

patches_all(3)

-

patches_files(3)

-

patches_auto(3)

-

src_prepare_extend

-

src_postprepare

-

src_config(3)

-

src_compile(3)

-

pkg_rminstall(3)

-

pkg_prerminstall

-

pkg_rminstall_extend_pre

-

image_clean(3)

-

pkg_rminstall_extend

-

pkg_postrminstall

-

src_install(3)

-

pkg_install(3)

-

pkg_preinstall

-

pkg_install_extend_pre

-

image_remove_la(3)

-

pkg_install_extend

-

pkg_postinstall

-

pkg_config(3)

-

pkg_preconfig

-

pkg_config_extend_pre

-

image_split_pack(3)

-

pack_gen_meta(3)

-

fetch_inner_version(3)

-

pkg_config_extend

-

pkg_postconfig

-

pkg_root(3)

-

pkg_preroot

-

pkg_root_extend_pre

-

root_clean_by_fire(3)

-

root_install(3)

-

get_installed_tracked_dirs(3)

-

get_installed_tracked_exec_dirs(3)

-

get_installed_tracked_exec_files(3)

-

root_register(3)

-

pkg_root_extend

-

pkg_postroot

-

pkg_pack(3)

-

pkg_prepack

-

pkg_pack_extend_pre

-

pack_create(3)

-

pack_name_file(3)

-

fetch_inner_version(3)

-

pack_name(3)

-

pack_create_archive(3)

-

pack_info(3)

-

pack_name_file(3)

-

pkg_pack_extend

-

pkg_postpack

-

src_store(3)

-

fetch_store_mint(3)

-

fetch_packinstall(3)

-

fetch_store_patches(3)

-

pkg_deploy(3)

-

pkg_pretend(3)

-

pack_unpack(3)

-

pkg_root(3)

-

pkg_context(3) for staticdeproot mode

-

pkg_precontext

-

pkg_context_extend_pre

-

root_clean_pack(3)

-

root_unregister(3)

-

pkg_context_extend

-

pkg_postcontext

Remaining non-default methods, related to the scheme above, are:

-

image_setup_paths(3)

-

get_installed_tracked_dirs(3)

-

pack_get_version(3)

-

pack_get_inner_version(3)

-

pack_define_by_filename(3)

-

pack_dump_content(3)

-

pack_dump_content_archive(3)

-

resolve_rebuild(3)

-

fetch_clean_patches(3)

Here is the short summary for each engine interface, used inside map methods. Names are alphabetically sorted, unlike in the structured scheme above.

depend_resolve_current_atom(3)

Print the list of packages, matching the dependencies of the current package.

fetch_build_remove(3)

Remove the directory with build files of a package.

fetch_clean_patches(3)

Remove the directory with generated patches of a package.

fetch_dev(3)

Fetch all development materials of a package from remote places.

fetch_dev_inner_version(3)

Fetch inner version of a package and write it down.

fetch_gen_patches(3)

Generate patches from development sources of a package.

fetch_inner_version(3)

Print inner version of a package.

fetch_local_src(3)

Unpack local sources of a package to its build directory.

fetch_packinstall(3)

Save sources from build directory of a package into source storage directory.

fetch_remote_src(3)

Fetch original sources of a package from remote place into local directory.

fetch_store_mint(3)

Save current data as etalon into storage directory.

fetch_store_patches(3)

Save generated patches into patch storage directory.

get_installed_tracked_dirs(3)

Print existing directories of a package, used for ROOT directory.

get_installed_tracked_exec_dirs(3)

Print existing directories of a package, containing executables, used for ROOT directory.

get_installed_tracked_exec_files(3)

Print existing executables of a package, used for ROOT directory.

get_plain_tracked_dirs(3)

Print all directories, declared for use in ROOT directory, without basedir D.

image_clean(3)

Remove installed files of a package.

image_remove_la(3)

Remove all files with .la extension from installed set of a package.

image_setup_paths(3)

Filter non-binary files in installed set of a package and remove ROOT basedir part from all paths.

image_split_pack(3)

Split installed set of a package to sub-packages.

keywords_check(3)

Parse the KEYWORDS variable and set up the status of the build.

pack_create(3)

Create binary packages from installed set of files of a package.

pack_create_archive(3)

Create single binary archive for a package manager.

pack_define_by_filename(3)

Print the name of installable package, which contains given file.

pack_gen_meta(3)

Generate and store metainformation for package manager.

pack_get_inner_version(3)

Print the inner version of a given installable archive.

pack_get_version(3)

Print the full version of a given installable archive.

pack_info(3)

Print the summarised information about installable archives of a package.

pack_name(3)

Print names of all declared installable packages, provided by a current one.

pack_name_file(3)

Print file names of all declared installable packages, provided by a current one.

pack_unpack(3)

Extract the contents of all binary packages, provided by current package, to D.

patches_all(3)

Apply all needed patches to the original sources of a package.

patches_auto(3)

Do automatic source change of a package.

patches_files(3)

Copy prepared set of files to sources of a package.

resolve_rebuild(3)

Print tree of dependencies for given set of packages.

root_clean_by_fire(3)

Remove directory with build context of a package.

root_clean_pack(3)

Remove files of a package from build context directory.

root_dep_populate(3)

Create build context directory for a package and fill it with files of dependencies.

root_install(3)

Create links and real copies of according files of a package in build context directory.

root_platform_populate(3)

Create build context directory with platform files, common for all packages of the project.

root_register(3)

Create record of a package in simple database of installed packages in build context directory.

root_setup_paths_to_install(3)

Create real copies instead of links in build context directory for package files, configured for that.

root_unregister(3)

Remove record of a package from database of installed packages in build context directory.

status_check(3)

Do the action according to the global status: OK, SKIP, FAIL.

Besides main map, there are several auxiliary method sequences. They are contained in map.sh file together with map. Additional maps are the following:

map_autopre

methods, called at the beginning of build of each package. They are called independently from main map. When methods to launch are given explicitly, map_autopre part remains the same as with default map:

        any do all somelist.src src_fetch,src_store
        any do all somelist.src

map_autopre contains methods:

prepare_all

Set up temporary files, directories, global variables, needed for the engine work.

print_atom

Print the package name to the terminal and session log file.

print_log_header

Print header with summarised information to the build log file of a package.

keywords_check

Check the status of a package by KEYWORDS mechanism.

status_check

Check, whether the build process has been skipped or terminated at the top of build script.

tobuild

Move to build directory, if it exists. The method is called here to use separate methods of map easier, as many of them expect their residing in build directory.

map_autopost

methods, called after build process ends. They do not depend on the main map sequence, like the map_autopre.

status_sum

Set up global variables, keeping the information on build result: status and total time.

print_final_atom

Print the resulting build information to the terminal and to the profile log file.

map_postmortem

methods, called after build process has ended up with non-zero status, which signals some error.

status_terminate

Like status_sum, method handles the terminated status.

print_final_atom

See map_autopost above.

print_path_log

Method to print path to the log file of a package to terminal. Path is not written to profile log file.

map_group

methods, called once at the beginning of each separate rdd(1) or any-do(1) execution, before any actions for personal packages.

admin_host

perform the administrative tasks over host environment. These may require according privileges. By default this method is empty and can be actually implemented with some module. The typical case is updating the installed host packages with package manager.

admin_owner_mini

make temporary directories writable for general user. Method is called to clean up after work of privileged user.

print_session

print header information for the build session of entire set of packages to terminal and profile log file.

Many extension modules have their own map, called as module_map. For example, module buildfresh has buildfresh_map. Such method contains whole sequence of calls, which implement function of a module. Separate map name for each module makes sure not to mess it up with default map, when module shell library is included to overall engine set. So usage of module requires explicit call of its map by the name.
On the opposite to the scheme above, module may contain redefenition of map method. That means user will be able to use only the module after its inclusion and not the main engine. That can be useful in several situations. There is also possibility to define alternative default map in configuration file like this:

[module]
rdd_map_entry = module_map

In that case module_map is executed instead of main map after inclusion of the module, but main map still may be called explicitly by the name:

        any do amd64,module all.src map

Preferred way of usage depends on module specification.

There is convention for modules, doing some regression or quality checking. Their maps are ended with _checker_map postfix. For example, trackelf module has the map trackelf_checker_map.
The checker convention is used by the engine to launch all available checkers automatically. The method sane_map detects the list of available methods with _checker_map name ending and launches them all in order of module appearence in AUSE. Method is provided by engine module sane.

The example of the launch of all available checkers for all packages:

        any do amd64,sane all.src

The same for single package hello-world-1.0.0:

        any do amd64,sane hello-world-1.0.0

The checker module must be included in config file (or explicitly in command line) to be used by automatic launch of sane.
The same technique of automatic method detection may be used for other classes of modules, not just the checkers.

any(1), any-api(7)