[00/30] RFC: elf: glibc-hwcaps support

Message ID cover.1592836143.git.fweimer@redhat.com
Headers show
Series
  • RFC: elf: glibc-hwcaps support
Related show

Message

Chang S. Bae via Libc-alpha June 22, 2020, 3:12 p.m.
This series implements searching for alternative library implementations
under glibc-hwcaps subdirectories on the library search path.  It's
lacking tests, documentation, and a NEWS file update, but I wanted to
submit it early for discussion (and perhaps review of some of the
cleanup patches).

For background, it is best reference first the discussion about the
current hwcaps subdirectory processing:

  hwcaps subdirectory selection in the dynamic loader
  <https://sourceware.org/pipermail/libc-alpha/2020-May/113757.html>

The new approach uses a special subdirectory, called "glibc-hwcaps",
which contains nested subdirectories with all the alternative library
implementations. The glibc-hwcaps subdirectory can be present in every
subdirectory of the library search path.

ldconfig recognizes the "glibc-hwcaps" name, and it receives special
treatment if it exists: subdirectories of the directory (but the
directory itself) are scanned for libraries.  If a library is found, it
is put into the cache, and associated with the name of the subdirectory
(of glibc-hwcaps).  ldconfig does this for all subdirectories (whose
name does not begin with '.'), so it does not have to be updated if new
subdirectories are added.  There is currently no symbolic link
processing for these subdirectories.  In the future, they might store
multiple implementations with different GNU property notes, and we might
want to select those through the cache as well.  (I'm open to changing
this part to align more with traditional cache processing.)

At run time, the dynamic loader has a built-in list of HWCAP
subdirectory names (which are different from the legacy HWCAP
directories).  The new “ld.so --help” option shows these subdirectory
names.  For LD_LIBRARY_PATH processing, ld.so simply attempts to open
libraries under their soname in the active/supported glibc-hwcaps
subdirectories.  (This part is quite similar to the original HWCAP
subdirectory handling, except that the new subdirectories do not nest.)
For ld.so.cache processing, ld.so sorts the active/supported HWCAP
subdirectory names and merges this list against the list of subdirectory
names as stored in a new ld.so.cache extension section.  Each
glibc-hwcaps library listed in the cache has a HWCAP field that contains
an index into the HWCAP subdirectory array of the cache file.  (A
previously unused HWCAP bit is set so that these entries are ignored by
existing dynamic linkers.)  The dynamic linker loads the library with
has the lowest positive preference value (essentially from the
subdirectory that comes first in its consolidated HWCAP name list).

There are new ld.so options, --glibc-hwcaps-prepend and
--glibc-hwcaps-mask, to change the list of glibc-hwcaps subdirectories
searched.  Due to the way ldconfig has been modified, it is possible to
pick up completely new subdirectories (not known to the implementation)
using --glibc-hwcaps-prepend even if ld.so.cache is used.

With these changes, on a current x86-64 machine (with AVX2-level CPU
features), you can drop a shared object into the directory

  /usr/lib64/glibc-hwcaps/x86-102

and the dynamic linker will load it, either via ld.so.cache or
LD_LIBRARY_PATH processing.

Under strace, it look like this if the cache is used:

| openat(AT_FDCWD, "/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 3
| openat(AT_FDCWD, "/lib64/glibc-hwcaps/x86-102/libz.so.1", O_RDONLY|O_CLOEXEC) = 3

Diagnostics output from “ld.so --help” for this machine is this:

| Shared library search path:
|   (libraries located via /etc/ld.so.cache)
|   /lib64 (system search path)
|   /usr/lib64 (system search path)
| 
| Subdirectories of glibc-hwcaps directories, in priority order:
|   x86-103
|   x86-102 (supported, searched)
|   x86-101 (supported, searched)
|   x86-100 (supported, searched)
| 
| Legacy HWCAP subdirectories under library search path directories:
|   haswell (AT_PLATFORM)
|   tls (supported, searched)
|   avx512_1
|   x86_64 (supported, searched)

I've also included support for a "power9" subdirectory on powerpc64le,
some subdirectories on s390x, and "atomics" on aarch64.  The exact
definitions of the directories are still subject to change.  They are
there to that there is at least something new to be searched.  For the
x86-64 part, I'm going to start a separate, cross-project discussion
about the desired CPU feature flags for each micro-architecture baseline
level.

The series does not yet check if the glibc-hwcaps directory itself
exists.  If the directory does not exist, we can skip searching all its
subdirectories.  This is another motivation for having the magic
glibc-hwcaps subdirectory.  But I have not implemented this yet because
it tends to hide the search activity from strace.

Patches up to (but excluding) “elf: Add glibc-hwcaps support for
LD_LIBRARY_PATH” are more or less cleanups, and so are these patches:

  elf: Unify old and new format cache handling code in ld.so
  elf: Implement a string table for ldconfig, with tail merging
  elf: Implement tail merging of strings in ldconfig
  elf: In ldconfig, extract the new_sub_entry function from search_dir

These cleanups could be reviewed and installed separately.

Thanks,
Florian

Florian Weimer (30):
  elf: Include <stdbool.h> in <dl-tunables.h> because bool is used
  elf: Include <stddef.h> (for size_t), <sys/stat.h> in <ldconfig.h>
  elf: Do not search HWCAP subdirectories in statically linked binaries
  elf: Implement __rtld_malloc_is_full
  elf: Implement _dl_write
  elf: Extract command-line/environment variables state from rtld.c
  elf: Move ld.so error/help output to _dl_usage
  elf: Record whether paths come from LD_LIBRARY_PATH or --library-path
  elf: Implement ld.so --help
  elf: Implement ld.so --version
  scripts/update-copyrights: Update csu/version.c, elf/dl-usage.c
  elf: Use the term "program interpreter" in the ld.so help message
  elf: Print the full name of the dynamic loader in the ld.so help
    message
  elf: Make __rtld_env_path_list and __rtld_search_dirs global variables
  elf: Add library search path information to ld.so --help
  elf: Enhance ld.so --help to print HWCAP subdirectories
  elf: Do not pass GLRO(dl_platform), GLRO(dl_platformlen) to
    _dl_important_hwcaps
  elf: Add glibc-hwcaps support for LD_LIBRARY_PATH
  x86_64: Add glibc-hwcaps support
  powerpc64le: Add glibc-hwcaps support
  s390x: Add Add glibc-hwcaps support
  aarch64: Add glibc-hwcaps support
  elf: Add endianness markup to ld.so.cache
  elf: Add extension mechanism to ld.so.cache
  elf: Unify old and new format cache handling code in ld.so
  elf: Implement a string table for ldconfig, with tail merging
  elf: Implement tail merging of strings in ldconfig
  elf: In ldconfig, extract the new_sub_entry function from search_dir
  elf: Process glibc-hwcaps subdirectories in ldconfig
  elf: Add glibc-hwcaps subdirectory support to ld.so cache processing

 NEWS                                          |   4 +
 elf/Makefile                                  |  22 +-
 elf/cache.c                                   | 400 +++++++++++++--
 elf/dl-cache.c                                | 483 ++++++++++++------
 elf/dl-hwcaps-subdirs.c                       |  29 ++
 elf/dl-hwcaps.c                               | 225 +++++++-
 elf/dl-hwcaps.h                               | 102 ++++
 elf/dl-hwcaps_split.c                         |  77 +++
 elf/dl-load.c                                 |  75 +--
 elf/dl-main.h                                 | 120 +++++
 elf/dl-minimal.c                              |   8 +
 elf/dl-support.c                              |   5 +-
 elf/dl-tunables.h                             |   2 +
 elf/dl-usage.c                                | 267 ++++++++++
 elf/dl-write.c                                |  56 ++
 elf/ldconfig.c                                | 187 +++++--
 elf/rtld.c                                    | 248 ++++-----
 elf/stringtable.c                             | 201 ++++++++
 elf/stringtable.h                             |  61 +++
 elf/stringtable_free.c                        |  32 ++
 elf/tst-dl-hwcaps_split.c                     | 139 +++++
 elf/tst-stringtable.c                         | 140 +++++
 include/link.h                                |   4 +
 include/rtld-malloc.h                         |   4 +
 scripts/update-copyrights                     |   6 +
 sysdeps/aarch64/dl-hwcaps-subdirs.c           |  31 ++
 sysdeps/generic/dl-cache.h                    | 232 ++++++++-
 sysdeps/generic/ldconfig.h                    |  20 +-
 sysdeps/generic/ldsodefs.h                    |  34 +-
 .../powerpc/powerpc64/le/dl-hwcaps-subdirs.c  |  31 ++
 sysdeps/s390/s390-64/dl-hwcaps-subdirs.c      |  54 ++
 sysdeps/unix/sysv/linux/dl-write.c            |  30 ++
 sysdeps/x86_64/dl-hwcaps-subdirs.c            |  73 +++
 33 files changed, 2980 insertions(+), 422 deletions(-)
 create mode 100644 elf/dl-hwcaps-subdirs.c
 create mode 100644 elf/dl-hwcaps_split.c
 create mode 100644 elf/dl-main.h
 create mode 100644 elf/dl-usage.c
 create mode 100644 elf/dl-write.c
 create mode 100644 elf/stringtable.c
 create mode 100644 elf/stringtable.h
 create mode 100644 elf/stringtable_free.c
 create mode 100644 elf/tst-dl-hwcaps_split.c
 create mode 100644 elf/tst-stringtable.c
 create mode 100644 sysdeps/aarch64/dl-hwcaps-subdirs.c
 create mode 100644 sysdeps/powerpc/powerpc64/le/dl-hwcaps-subdirs.c
 create mode 100644 sysdeps/s390/s390-64/dl-hwcaps-subdirs.c
 create mode 100644 sysdeps/unix/sysv/linux/dl-write.c
 create mode 100644 sysdeps/x86_64/dl-hwcaps-subdirs.c

-- 
2.25.4

Comments

Jim Wilson July 10, 2020, 9:47 p.m. | #1
On Mon, Jun 22, 2020 at 8:13 AM Florian Weimer via Libc-alpha
<libc-alpha@sourceware.org> wrote:
> With these changes, on a current x86-64 machine (with AVX2-level CPU

> features), you can drop a shared object into the directory

>   /usr/lib64/glibc-hwcaps/x86-102


This appears to violate the Filesystem Hierarchy Standard (FHS)
    https://refspecs.linuxfoundation.org/FHS_3.0/fhs/index.html
which states that libraries can only be in /usr/lib or /usr/lib<qual>.
And it is implied but not clearly stated that <qual> is not allowed to
contain a slash.  There is an exception for applications that can have
a dir under /usr/lib or /usr/lib<qual>, but glibc isn't an
application.  Unless maybe you want to claim that ld.so is an
application and these are ld.so specific files?  But then all
libraries used by ld.so must be in there which is not true.

I'm not sure if anyone cares about strict compliance with FHS though.
Debian (and hence Ubuntu) deliberately violate it with their multiarch
scheme.  RISC-V violates it with our two level scheme for word size
and abi, e.g. /usr/lib64/lp64d for 64-bit hard-float libraries.
Though one person did care enough to report this as a bug.  I sent
email to the fhs-discuss mailing list to ask about this, but never got
a reply.  The email I sent in April is the last email on the mailing
list.  Anyways, I don't care about FHS personally, I just care that I
don't need to change the RISC-V ABI.  If this scheme is OK then maybe
the RISC-V scheme is OK too?

Jim
Chang S. Bae via Libc-alpha July 11, 2020, 9:02 a.m. | #2
* Jim Wilson:

> On Mon, Jun 22, 2020 at 8:13 AM Florian Weimer via Libc-alpha

> <libc-alpha@sourceware.org> wrote:

>> With these changes, on a current x86-64 machine (with AVX2-level CPU

>> features), you can drop a shared object into the directory

>>   /usr/lib64/glibc-hwcaps/x86-102

>

> This appears to violate the Filesystem Hierarchy Standard (FHS)

>     https://refspecs.linuxfoundation.org/FHS_3.0/fhs/index.html

> which states that libraries can only be in /usr/lib or /usr/lib<qual>.

> And it is implied but not clearly stated that <qual> is not allowed to

> contain a slash.  There is an exception for applications that can have

> a dir under /usr/lib or /usr/lib<qual>, but glibc isn't an

> application.  Unless maybe you want to claim that ld.so is an

> application and these are ld.so specific files?  But then all

> libraries used by ld.so must be in there which is not true.


I thought about this.  For use with the library path, it has to be a
subdirectory not a sibling directory, otherwise the behavior would be
really surprising.

In the end, it's no different from the "tls" directory we already search
(along with a bunch of others).  I posted a list of the subdirectories
searched on s390x:

  <https://sourceware.org/pipermail/libc-alpha/2020-May/113757.html>

At least the "glibc-hwcaps" name is architecture-independent, and we can
add it to FHS if we wanted.

Thanks,
Florian