Native (automatic) Multilib for Portage

Recent

Relevant IRC discussion with kugelfang, blubb, ferringb, and geoman.

Introduction

Many moons ago there was a bug (75420) that proposed to add real multilib support into portage. The blocking issues in gcc and libtool have been fixed since the bug was closed. Additionally I am currently using this functionality daily on real life systems and it works. This is an attempt to resurrect that idea. I have filed bug 145737 to restart the discussion.

Besides the bug and IRC discussion, there are two other documents that provide some background for the discussion. Eradicator's original multilib notes and blubb's GLEP draft for multilib.


Basic funcionality

I treat the installation of non-default ABI libraries/headers as equivalent to having a USE flag. This is essentially what toolchain packages (glibc, gcc, libstc++-v3) have used the "multilib" USE flag to mean.

The current behavior of portage is that, when you emerge a package, all executables, libraries, headers, and config files are installed and merged for the default ABI of the system. With native multilib support, if the "multilib" USE flag is set and the package supports multilib (IUSE="multilib"), then portage will first process each non-default ABI up to the install phase. Then it will process the default ABI up to the install stage. Finally, portage will merge all the installed files to the file-system.

Because the default ABI is processed last it will overwrite any files installed by the alternate ABI installs, however, files from the alternate ABIs that have different paths (or names) will remain and be merged. In the case of library files, the install location will change for each ABI because the $(get_libdir) function is ABI sensitive. The net result is that you get non-ABI specific files for the default ABI, and ABI specific files for each ABI that is requested.

Rationale

Combined package install vs. slotted ABI approach:
One approach to native mulitlib support in portage is make an "ABI-slot" for each ABI of a package that is installed. However, I don't think this is the best approach. The main advantage I see to the slotted ABI approach is that alternate ABIs can be treated as seperate units for dependencies and packages that already have an installed ABI don't need that ABI to be re-installed. However, ABI slotting with correct dependency handling and correct handling of files that are owned by multiple ABI slots is complex.

The biggest downside of having separately installed "atoms" for each ABI is the risk of version skew between the ABIs. If a user has one ABI installed for a library and later installs another ABI for the same library there is a very good chance that the two will be different versions. However, there are two other more subtle and potentially more problematic version skew scenarios. First, the ebuild for the library or (more likely) the eclasses that it uses may have changed between ABI installs without the ebuild version actually being bumped. The other case is that the user may have installed other packages or changed their system in some way (CFLAGS), such that build process (especially configure checks) of the new ABI is different than for the original ABI. In both cases, the new ABI files appears to be the same configuration as the original package ABI, but they are not.

I think there are several advantages to treating alternate ABIs libraries and headers as an additional feature or option to activate for a package. First, the impact to portage is less and there is almost no change to the code path if multilib is not being used. Another advantage is that /var/db/pkg and the related code that handles it does not require restructuring to support a new form of slotting (ABI slotting). And there is the "show me the code" factor: I'm using it right now to build systems that I am using in real life.

ABI+gcc-config wrapper vs. all ABI info in CHOST:
My current method uses the existing ABI support in gcc-config which listens to certain environment variables to determine which ABI mode the compiler should run in. Another option for changing the compiler output during the build of each ABI is to have a different CHOST/compiler executable for each ABI.

My opinion is that the ABI in CHOST route is not as well trodden ground as the ABI+gcc-config approach. On MIPS (and maybe some others) it will require doing some non-standard modifications of the CHOST GNU configuration names because ABI information isn't guaranteed to be fully encoded in CHOST. For example, right now mips64el-unknown-linux-gnu is a GNU configuration name for both n32 and n64 ABIs (because from kernel down they are both 64-bit ABIs). One option is overload the vendor field. I consider this approach to be more "icky" than the ABI+gcc-config approach, but regardless this is not a standard practice and I'm concerned about some subtle problems with this approach. I think it would result in more struggle against upstream maintainers than the ABI+gcc-config option I'm currently using.

However, I'm less dogmatic about the ABI in CHOST approach than I am about the slotted ABI approach. But the ABI+gcc-config wrapper has been working well so far and very few packages have needed to be patched to work with this approach.


Basic Usage

Requirements:
  • patched portage-2.1.1-r1
  • patched gcc-config 1.3.14 (wrapper.c checks that CHOST matches target)
  • libtool >=1.5.22
  • profile with full multilib info
  • patched gcc "-print-search-dirs" to do multilib
  • Other package specific pathces listed below
Important Variables:
  • IUSE="multilib" -> indicates this ebuild supports multilib (either native multilib or multilib.eclass based multilib)
  • IUSE="multilib" and USE="multilib" -> activate multilib ABIs
  • USE="multilib_abis_amd64 multilibs_abis_x86 multilib_abis_n32" -> The actual multilib ABI flags.
  • RESTRICT="auto-multilib" and IUSE="multilib" -> means this package handles multilib internally (probably via multilib.eclass) and native multilib should not do it's magic for this package.
  • MULTILIB_ABIS -> list of ABIs to do multilib for.
  • DEFAULT_ABI -> last (default) ABI to process (i.e. executables)
Try it out:
  • Download and unpack my auto-multilib overlay (yes, this should probably be on overlays.gentoo.org).
  • Add the overlay to /etc/make.conf. 
    PORTDIR_OVERLAY="/usr/local/auto-multilib-overlay"
  • Set /etc/make.profile symlink to point to the correct profile in the overlay: 
    rm /etc/make.profile

(For amd64 systems)
ln -sf /usr/local/auto-multilib-overlay/profiles/auto-multilib/amd64 /etc/make.profile

(For mips64el systems)
ln -sf /usr/local/auto-multilib-overlay/profiles/auto-multilib/mips64el /etc/make.profile
  • Install portage-2.1.1-r801 from the overlay
  • Install gcc-config-1.3.14-r800 from the overlay
  • Install libtool 1.5.22 or greater
  • (Re-)Install gcc-4.X and make sure that print-search-dirs patch is applied.
  • Add multilib IUSE flag to zlib and expat ebuilds and digest them.
  • Do some install tests: 
    (Save original packages)
# quickpkg zlib
# quickpkg expat

(install multilib zlib and expat)
# USE="multilib" emerge zlib
# USE="multilib" emerge expat

(install for x86 only)
# USE="multilib" MULTILIB_ABIS="x86" emerge expat

(install x86 and amd64 headers/libraries but install x86 executables (i.e. /usr/bin/xmlwf).)
# USE="multilib" DEFAULT_ABI="x86" emerge expat


Internal Operation

The four portage files that have been modified are:

In ebuild.sh, the functions dyn_unpack, dyn_compile and dyn_install each have an new loop that interates through each ABI with the DEFAULT_ABI occuring last. Each of those functions still calls src_unpack, src_compile and src_install but they are called for each ABI. Each time they are called certain environment variables and the on disk structure is changed so that the resulting action applies to the ABI currently being processed.

Here is the psuedo-code for the original portage process in ebuild.sh:
...
dyn_unpack()
    src_unpack

dyn_compile()
    src_compile

dyn_install()
    src_install
...
Here is the psuedo-code for the new algorithm in ebuild.sh:
...
dyn_unpack()
    for each ABI
        call set_abi
            create ABI workdir
            set ABI envronment variables
        call src_unpack

dyn_compile()
    for each ABI
        call set_abi
            use workdir for current ABI
            set ABI environment variables
        call src_compile

dyn_install()
    for each ABI
        call set_abi
            use workdir for current ABI
            set ABI environment variables
        call src_install
        call _finalize_abi_install
            create gentoo-multilib headers (if needed)
            create ABI aware /usr/bin/*-config files
...

The code changes to ebuild.sh are fairly minor. The majority of the auto-multilib code, including the functions set_abi, _finalize_abi_install are in auto-multilib.sh. The majority of the code in auto-multilib.sh is contained in _finalize_abi_install and supporting routines, which are derived from header processing routines in the multilib.eclass.

Also, it is worth noting that the /usr/bin/*-config processing in _finalize_abi_install uses an environment variable ABI_REDIRECT_PROGS to determine whether to create an ABI aware redirect script in place of the real script.

In order to support multilib dependencies I created special USE flags of the form multilib_abis_XXX where XXX is the ABI to be installed. These USE flags are special in several ways.
  • They are a partial implementation of USE flag dependencies. This allows a user to emerge a non-multilib package such as firefox for a non-default ABI, and the required multilib packages will be automatically added to the dependency list with the appropriate multilib_abis_XXX flags automatically enable for those dependencies.
  • They are displayed like USE_EXPAND flags in portage --pretend output.
  • They are automatically added by portage when USE="multilib" but only if the package has IUSE="multilib".
  • They are only added to the /var/db/pkg/CAT/NAME/USE file if they are actually active for the package (unlike other USE flags). This allows portage to be able to test later whether the package was installed with an ABI or not.

The changes to support the multilib_abi_XXX USE flags are in emerge and portage.py.


Current Testing

I have built a multilib amd64 system from a stage1 with both ABIs (x86, amd64). There are some packages I needed to patch listed below (mostly because of incorrect ebuild assumptions). I have 294 packages installed including modular X. Remote X applications seem to work ok.

I have an automated daily script that uses auto-multilib to cross-compile, boot, and test a mips64el system image with the three MIPS ABIs: o32, n32, and n64. This is basically emerge system plus a bunch of other packages (125 total).

I also have a python regression test script that uses pexpect and several test packages in the overlay to verify the native multilib functionality.

Overall the multilib changes seem to be pretty robust and flexible.


Additional Notes

gcc-wrapper:

The gcc wrapper in gcc-config-1* needs to be patched because it doesn't do the right thing in certain cross-compiles situations. The broken case is when ABI is set to the HOST ABI, but the BUILD gcc is called it still tries to use CFLAGS_${ABI} which gives wrong CFLAGS for the native toolchain. If CHOST matches the compilers build architecture then use CFLAGS_${ABI}, otherwise don't use it.

Notable packages:
  • packages that currently have multilib IUSE:
    gcc, glibc, libstdc++-v3
  • packages that currently RESTRICT=multilib-pkg-force":
    opengl-update, eselect-opengl, ati-drivers, sandbox, glibc, nvidia-glx
Additional Package Patches in the overlay:
  • Fixes specific to working with native multilib functionality
    • db-4* to do pkg_postinst stuff for each ABI.
    • libpng to do pkg_postinst stuff for each ABI.
    • *-config redirects: freetype, pkgconfig, audiofile, libpng, python, gtk+-1*, xmkmf, imake
    • glib needs ac_cv_host set to CHOST_${ABI}
    • nspr and nss ebuilds to honor ABI when enabling 64 bit configuring options.
    • mesa to honor ABI when setting CONFIG, and use libdir in .la files.
    • xorg-cf-files to set imake config to get_libdir, freeglut configure to be multilib aware
  • Generic fixes needed for/revealed by auto-multilib
    • gmp to not set ABI variable in makefiles.
    • procps to always install /bin/ps
    • imlib, libtool library pathing
Outstanding native multilib issues:
  • Add RESTRICT="auto-multilib" to glibc, libstdc++-v3
  • problematic packages: java-config, python distutils, etc
  • mit-krb5 configure with tcl is hardcoded /usr/lib/tclConfig.sh
  • postgresql uses "perl -MConfig -e 'print $Config{privlibexp}'" to figure out dirs which forces /usr/lib64.
  • mysql has -L/usr/lib64 on link line
  • vim uses -L/usr/lib64 because of ruby
  • motif-config wants to set the config in a file in a libdir, so motif-config -s has to be run for each ABI
  • pango and gtk's /etc/gtk-2.0 and /etc/pango files are not multilib aware
  • Maybe use config-redirect solution from bug 118815. I don't really like it though. It's too heavy and requires another portage phase.
  • Make it work in pkgcore?
Outstanding non native multilib issues found:
  • gnome-libs missing dep on libXpm
  • loop in cyrus-sasl/openldap requires merge of openldap first without sasl
  • Xaw3d should DEPEND on gccmakedep
Ebuild Writing/Fixing Rules and Hints:
  • In install phase, move files, not directories. Otherwise you can end up with failures or files that are the wrong ABI.
  • Don't use ABI in Makefiles
  • Makefiles should not use install locations as makefile targets, otherwise second ABIs may not install binaries (timestamp issue).
  • Certain pkg_post and pkg_pre install functions that are adjusting the install, should either do their work in src_install if possible, or do all ABIs.
  • Global variables that change depending on ABI cannot be set in pkg_setup because it only runs once
  • Global variables that affect ABI can not be declared read-only (declare -r).


Related Discussion