• [gentoo-dev] [PATCH 0/4] distutils-r1.eclass: symlinking support

    From =?UTF-8?q?Micha=C5=82=20G=C3=B3rny?@21:1/5 to All on Sat May 3 17:20:01 2025
    Hi,

    This batch includes some minor refactoring, plus two major experimental features (to be enabled via `make.conf` by people ready to test them):

    1. DISTUTILS_ALLOW_PYC_SYMLINKS to symlink matching .pyc files across
    optimization levels, e.g.:

    __init__.opt-2.pyc -> __init__.opt-1.pyc
    __init__.opt-1.pyc -> __init__.pyc

    2. DISTUTILS_ALLOW_CROSS_IMPL_SYMLINKS to symlink matching files across
    different Python implementations and save lots of space, e.g.:

    python3.11/site-packages/frobnicate.py
    python3.12/site-packages/frobnicate.py -> ../../python3.11/site-packages/frobnicate.py
    python3.13/site-packages/frobnicate.py -> ../../python3.11/site-packages/frobnicate.py

    This can specifically help with large .py files, lots of data
    installed in package directory and stable ABI extensions.

    Both features are opt-in, and at this point considered experimental
    and dangerous. We may consider making the first one opt-out
    in the future, but I want to test both extensively first.

    Note that we may independently of this start moving package data
    out of site-packages and into /usr/share for some large packages.


    Micha艂 G贸rny (4):
    python-utils-r1.eclass: Simplify _python_impl_matches
    distutils-r1.eclass: Quote DISTUTILS_ALLOW_WHEEL_REUSE
    distutils-r1.eclass: Support .pyc symlinking
    distutils-r1.eclass: Support cross-impl symlinks

    eclass/distutils-r1.eclass | 81 ++++++++++++++++++++++++++++++++++-
    eclass/python-utils-r1.eclass | 4 +-
    2 files changed, 81 insertions(+), 4 deletions(-)

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From =?UTF-8?q?Micha=C5=82=20G=C3=B3rny?@21:1/5 to All on Sat May 3 17:20:01 2025
    Remove special case for '3.10' in _python_impl_matches -- it was only
    necessary because we needed to handle 'pypy3' target specially,
    and that is no longer the case. Technically, the code checks for
    'pypy3_10' but that's not a problem, since there is no such a thing.

    Signed-off-by: Micha艂 G贸rny <mgorny@gentoo.org>
    ---
    eclass/python-utils-r1.eclass | 4 +---
    1 file changed, 1 insertion(+), 3 deletions(-)

    diff --git a/eclass/python-utils-r1.eclass b/eclass/python-utils-r1.eclass index ef05a58e1b13..3a4a3f19a1be 100644
    --- a/eclass/python-utils-r1.eclass
    +++ b/eclass/python-utils-r1.eclass
    @@ -230,9 +230,7 @@ _python_impl_matches() {
    fi
    return 0
    ;;
    - 3.10)
    - ;;
    - 3.8|3.9|3.1[1-3])
    + 3.[89]|3.1[0-3])
    [[ ${impl%t} == python${pattern/./_} || ${impl} == pypy${pattern/./_} ]] &&
    return 0
    ;;

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From =?UTF-8?q?Micha=C5=82=20G=C3=B3rny?@21:1/5 to All on Sat May 3 17:30:01 2025
    Reported-by: Ionen Wolkens <ionen@gentoo.org>
    Signed-off-by: Micha艂 G贸rny <mgorny@gentoo.org>
    ---
    eclass/distutils-r1.eclass | 2 +-
    1 file changed, 1 insertion(+), 1 deletion(-)

    diff --git a/eclass/distutils-r1.eclass b/eclass/distutils-r1.eclass
    index 880e33cd741f..799206f7eb03 100644
    --- a/eclass/distutils-r1.eclass
    +++ b/eclass/distutils-r1.eclass
    @@ -215,7 +215,7 @@
    # This is an optimization that can avoid the overhead of calling into
    # the build system in pure Python packages and packages using the stable
    # Python ABI.
    -: ${DISTUTILS_ALLOW_WHEEL_REUSE=1}
    +: "${DISTUTILS_ALLOW_WHEEL_REUSE=1}"

    # @ECLASS_VARIABLE: BUILD_DIR
    # @OUTPUT_VARIABLE

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From =?UTF-8?q?Micha=C5=82=20G=C3=B3rny?@21:1/5 to All on Sat May 3 17:30:02 2025
    Support replacing installed files with cross-implementation symlinks
    to save space. Opt-in via `DISTUTILS_ALLOW_CROSS_IMPL_SYMLINKS`
    variable.

    Closes: https://bugs.gentoo.org/954762
    Signed-off-by: Micha艂 G贸rny <mgorny@gentoo.org>
    ---
    eclass/distutils-r1.eclass | 66 ++++++++++++++++++++++++++++++++++++++
    1 file changed, 66 insertions(+)

    diff --git a/eclass/distutils-r1.eclass b/eclass/distutils-r1.eclass
    index 32cff457e996..d2e4f9ce7ba0 100644
    --- a/eclass/distutils-r1.eclass
    +++ b/eclass/distutils-r1.eclass
    @@ -227,6 +227,18 @@
    # Note that it requires >=dev-python/gpep517-19.
    : "${DISTUTILS_ALLOW_PYC_SYMLINKS=}"

    +# @ECLASS_VARIABLE: DISTUTILS_ALLOW_CROSS_IMPL_SYMLINKS
    +# @USER_VARIABLE
    +# @DESCRIPTION:
    +# If set to a non-empty value, the eclass will replace identical files
    +# from installed wheels with symlinks, across different Python
    +# implementations. This may include .py files, variety of data files
    +# and stable ABI extensions.
    +#
    +# This is an optimization that can reduce disk space usage when packages
    +# are built for multiple Python implementations.
    +: "${DISTUTILS_ALLOW_CROSS_IMPL_SYMLINKS=}"
    +
    # @ECLASS_VARIABLE: BUILD_DIR
    # @OUTPUT_VARIABLE
    # @DEFAULT_UNSET
    @@ -1233,6 +1245,33 @@ distut
  • From =?UTF-8?q?Micha=C5=82=20G=C3=B3rny?@21:1/5 to All on Sat May 3 17:30:02 2025
    Support replacing identical .pyc files with symlinks, using gpep517-19
    option `--symlink-pyc`. Opt in via `DISTUTILS_ALLOW_PYC_SYMLINKS`
    variable.

    Signed-off-by: Micha艂 G贸rny <mgorny@gentoo.org>
    ---
    eclass/distutils-r1.eclass | 13 +++++++++++++
    1 file changed, 13 insertions(+)

    diff --git a/eclass/distutils-r1.eclass b/eclass/distutils-r1.eclass
    index 799206f7eb03..32cff457e996 100644
    --- a/eclass/distutils-r1.eclass
    +++ b/eclass/distutils-r1.eclass
    @@ -217,6 +217,16 @@
    # Python ABI.
    : "${DISTUTILS_ALLOW_WHEEL_REUSE=1}"

    +# @ECLASS_VARIABLE: DISTUTILS_ALLOW_PYC_SYMLINKS
    +# @USER_VARIABLE
    +# @DESCRIPTION:
    +# If set to a non-empty value, the eclass is allowed to create symlinks
    +# between different optimization levels of compiled .pyc files.
    +#
    +# This is an optimization that can slightly reduce disk space usage.
    +# Note that it requires >=dev-python/gpep517-19.
    +: "${DISTUTILS_ALLOW_PYC_SYMLINKS=}"
    +
    # @ECLASS_VARIABLE: BUILD_DIR
    # @OUTPUT_VARIABLE
    # @DEFAULT_UNSET
    @@ -1220,6 +1230,9 @@ distutils_wheel_install() {
    --optimize=all
    "${wheel}"
    )
    + if [[ ${DISTUTILS_ALLOW_PYC_SYMLINKS} ]]; then
    + cmd+=( --symlink-pyc )
    + fi
    printf '%s\n' "${cmd[*]}"
    "$