------------- Spec Examples ------------- templates from catalyst package /usr/share/doc/catalyst-/examples/ specs for official releases livecd-specs package /usr/share/livecd-specs-/ Or, directly from CVS > Working specs can be found in our ViewCVS in gentoo/src/releng/specs. Therein > are contained the specs used to build prior releases for most archs. > --Kumba http://www.gentoo.org/cgi-bin/viewcvs.cgi/src/releng/specs/2005.1/?root=gentoo Gentoo devs only: cvs co gentoo/src/releng/specs ----------------------------- Catalyst Features ("options") ----------------------------- snapcache: unpack portage snapshot under snapshot_cache_path and bind mount it into /usr/portage of chroot directory (skip having to unpack portage from .tar.bz2 into chroot every time) seedcache: copy stage in source_path dir, if it exists, to initialize chroot (skip having to unpack of seed stage from .tar.bz2 every time) Note: it looks like the seed stage must be manually unpacked to take advantage of this -- if source_path dir doesn't already exist, the seed stage will be unpacked straight to chroot_path rather than leaving a copy in source_dir pkgcache: save up binary packages in the pkgcache_path directory (pkgcache_path directory is bind mounted into the chroot as /usr/portage/packages, and --buildpkg is added to default emerge flags) and then, use the binary packages whenever emerging the same package again with same USE flags (pkgcache_path directory is bind mounted into the chroot as /usr/portage/packages, and --usepkg is added to default emerge flags) This is especially useful when used with the autoresume feature to save having to recompile all the packages preceding a failed one, but packages could also be shared between two different targets that share the same toolchain and the same USE flags ----- Paths ----- * main config file /etc/catalyst2/catalyst2.conf OR file specified by command-line -c option * base dir for catalyst scripts sharedir -- set in main config for catalyst2, should either be set to /usr/share/catalyst2 OR a copy of this directory with possible local additions (e.g. new target files) [/mnt/experimental/catalyst/catalyst2.0_pre13] * root working directory for catalyst builds storedir -- set in main config [/mnt/experimental/catalyst/work] * path to seed stage used for populating initial build environment source_path -- inside storedir, either an archive or a dir generic_stage: if "seedcache" enabled source_path = storedir + "/tmp/" + source_subpath [/mnt/experimental/catalyst/work/builds/tmp/default/stage3-ppc-2005.1] otherwise, source_path = storedir + "/builds/" + source_subpath + ".tar.bz2" [/mnt/experimental/catalyst/work/builds/default/stage3-ppc-2005.1.tar.bz2] exception -- stage2: if "seedcache" enabled source_path = storedir + "/tmp/" + source_subpath + "/tmp/stage1root/" [/mnt/experimental/catalyst/work/builds/tmp/default/stage3-ppc-2005.1/tmp/stage1root/] otherwise, source_path = storedir + "/builds/" + source_subpath + ".tar.bz2" [/mnt/experimental/catalyst/work/builds/default/stage3-ppc-2005.1.tar.bz2] * path for final build product (or nearly final -- e.g. for livecd it contains CD contents not iso image) target_path -- subdir of storedir, additional portion of path is constructed by code generic_stage: target_path = storedir + "/builds/" + target_subpath + ".tar.bz2" = storedir + "/builds/" + rel_type + "/" + target + "-" + subarch + "-" version_stamp + ".tar.bz2" [/mnt/experimental/catalyst/work/builds/default/stage1-g3-20051209.tar.bz2] exceptions -- livecd_stage1, libecd_stage2, grp: target_path = storedir + target_subpath = storedir + "/builds/" + rel_type + "/" + target + "-" + subarch + "-" version_stamp [/mnt/experimental/catalyst/work/builds/default/stage1-g3-20051209] exception -- snapshot: target_path = storedir + "/builds/" + target_subpath + ".tar.bz2" = storedir + "/builds/portage.tar.bz2" [/mnt/experimental/catalyst/work/builds/portage.tar.bz2] * dir containing chroot environment for building chroot_path -- subdir of storedir, additional portion of path is constructed by code chroot_path = storedir + "/tmp/" + target_subpath + "/" = storedir + "/tmp/" + rel_type + "/" + target + "-" + subarch + "-" version_stamp + "/" [/mnt/experimental/catalyst/work/tmp/default/stage1-g3-20051209/] * ROOT dir for emerges inside chroot (path is relative to chroot_path) root_path -- @@@@@@ * full path to ROOT dir for chroot (path to chroot + path to root inside chroot) destpath -- constructed by code destpath = chroot_path + root_path * path to files built as part of a stage (like destpath for a stage target, since new stage gets built under ROOT...) stage_path -- constructed by code stage_path = chroot_path + root_path * compressed portage snapshot file snapshot_path -- constructed by code snapshot_path = storedir + "/snapshots/portage-" + snapshot + ".tar.bz2" (snapshot = name of snapshot, and is normally set in spec) [/mnt/experimental/catalyst/work/snapshots/portage-20051207.tar.bz2] * toplevel dir for storing unpacked portage snapshots when snapcache is on snapshot_cache -- normally set in main config [/mnt/experimental/catalyst/snapshots] * dir where a particular portage snapshot has been unpacked for reuse snapshot_cache_path -- constructed by code snapshot_cache_path = snapshot_cache + snapshot (snapshot = name of snapshot, and is normally set in spec) [/mnt/experimental/catalyst/snapshots/20051207] * portage overlay dir that should be used for emerges portage_overlay -- supplied in spec if overlay is desired [/mnt/experimental/catalyst/portage] * dir for storing built binary packages when "pkgcache" feature is enabled pkgcache_path -- can be constructed by code, sometimes able to be overridden in spec generic_stage: pkgcache_path = storedir + "/packages/" + target_subpath + "/" = storedir + "/packages/" + rel_type + "/" + target + "-" + subarch + "-" version_stamp + "/" [/mnt/experimental/catalyst/work/packages/default/stage1-g3-20051209] exceptions -- stage4, grp, livecd_stage1, tinderbox: if "pkgcache_path" is defined in spec pkgcache_path = value from spec [/mnt/experimental/catalyst/packages] otherwise, same as generic_stage * dir used by autoresume feature for storing build-stamps that indicate a build step was already completed successfully and can be skipped when resuming aborted build autoresume_path -- hidden subdir of storedir, additional portion of path is constructed by code autoresume_path = storedir + "/tmp/" + rel_type + "/.autoresume-" + target_subpath + "/" = storedir + "/tmp/" + rel_type + "/.autoresume-" + target + "-" + subarch + "-" + version_stamp + "/" [/mnt/experimental/catalyst/work/tmp/default/.autoresume-stage1-g3-20051209/] Summary of chroot_path, root_path, destpath, stage_path, target_path * chroot build environment goes in chroot_path (name is suggestive...) * when building inside chroot, root_path is used as ROOT for installing packages (note that it is relative to chroot_path, since it is used inside chroot) * after software has been emerged, it is accessible from outside the chroot using destpath (chroot_path + root_path) * stage_path looks like the same thing as destpath when referring to files for a stage target - the path to the installed packages from outside chroot * the typical meaning of target_path is the path for the final output product of the build (snapshot file, stage file, etc) but it looks like iso targets are exceptions - the target_path is a directory where the contents of the iso are assembled, and the iso image itself goes elsewhere ---------------- Config variables ---------------- * config settings come from 4 places, starting with lowest precedence: * builtin defaults * config file * spec file (if any) * command line arguments -C "VAR1=VALUE1" "VAR2=VALUE2" ... * python code * main program parses config file settings into conf_values hash note: catalyst features included in the "options" variable are also stored as individual settings named after the feature but in all caps line in config file: options = "ccache seedcache" parses as: conf_values["CCACHE"]="1" conf_values["SEEDCACHE"]="1" * main program parses spec file settings into addlargs hash * main program parses command line settings into addlargs hash (overwrites spec values if there is a conflict) * both conf_values and addlargs are passed to targets in constructor and stored in settings hash (see constructor for generic_target class) * target uses whatever config variables it wants to from the settings hash and constructs new config variables as needed # check if feature is enabled if self.settings.has_key("SEEDCACHE"): print "About to do seed-cachey stuff" # create a custom setting based on value of a standard one self.settings["jay_home"]=get_seed_stash(self.settings["storedir"]) * the target is responsible for exporting settings as environment variables targets make sure "setup_environment" action is run prior to executing any shell scripts that make use of config settings * bash code * the convention for all variables exported from the python module is that the python name is prefixed by "clst_" (an abbrv of catalyst I guess) python: self.settings["root_path"] shell: $clst_root_path * not all of the "clst_*" variables are config settings exported from the python; variables derived from config settings also tend to have "clst_" prefix if [ -n "${clst_PKGCACHE}" ] ; then export clst_myemergeopts="--usepkg --buildpkg --newuse" fi -------------- Adding targets -------------- * Add target to valid_build_targets array in /modules/catalyst_support.py valid_build_targets=["stage1_target","stage2_target", blah blah... "netboot_target", "newstuff_target"] * New python module /modules/newstuff_target.py * import from catalyst_support and generic_stage_target from catalyst_support import * from generic_stage_target import * * inherit class from generic_stage_target (or, rarely, generic_target) class newstuff_target(generic_stage_target): * provide a register callback that associates target name with target class def register(foo): foo.update({"tinderbox":tinderbox_target}) return foo Note: main program runs "module.register(targetmap)" for each module; the argument "foo" will be the targetmap being constructed * custom stuff! Examples: * override methods from generic_stage_target that set path variables def set_root_path(self): self.settings["root_path"]=normpath("/tmp/special_root") * override set_action_sequence to change the sequence of build steps def set_action_sequence(self): self.settings["action_sequence"]=["unpack","special_stuff", blah blah ... "clean","capture","clear_autoresume"] * define a method for any of the build steps (must define any custom steps) def special_stuff(self): print "Aha! I've got you now!!" # call "special_stuff" section from our controller file (see below) cmd("/bin/bash "+self.settings["controller_file"]+" special_stuff ",env=self.env) * New target subdir (optional) Most targets will have a subdir of shell scripts: /targets/ e.g. /usr/lib/catalyst2/targets/newstuff (snapshot is an example of a target that does not need one...) There are useful support functions available from files in /targets/support, and the config settings from the python code can be read from the shell scripts as environment variables $clst_ Note: the target python module is responsible for exporting settings as environment variables; targets should run "setup_environment" action prior to running any shell scripts that use config settings (generic_stage_target supplies a setup_environment method that should not need to be overriden) * controller script Most targets will have a controller script -controller.sh e.g. targets/newstuff/newstuff-controller.sh This is a shell helper script that has sections called from one or more build steps in the python module #!/bin/bash case $1 in unpack) cd /tmp/special_root tar -xzf /tmp/special_stuff.tgz ;; special_stuff) echo "frOOt LoopS" > /tmp/special_root/.sekret_stuph ;; *) ;; esac exit $? * chroot script Often targets will have a chroot script -chroot.sh that handles the actual emerge in package-building steps (typically making heavy use of functions from the chroot-functions.sh support file) #!/bin/bash . /tmp/chroot-functions.sh update_env_settings check_portage_version setup_myfeatures setup_myemergeopts # setup our environment export FEATURES="${clst_myfeatures}" # START BUILD run_emerge ${clst_myemergeopts} ${clst_packages}