This is an OpenPGP/MIME signed message (RFC 4880 and 3156) --------------h9uso10o8VlfoRxIEbgSOMrB
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

On 1/27/25 11:36 AM, Matthew Brooks wrote:

Hello!

Not sure where the most appropriate place to ask this is, so if some
other list or something would be more appropriate, please let me
know.

I'm interested in trying to solve the potential system python
breakage that PEP 668 was meant to address, but solving it in a way
that *doesn't* irrevocably break users' python package install
workflows for all eternity like PEP 668 does. From what I've read,
it looks like the issue is at least potentially addressable from the
distro side, and since Gentoo is the only distro I care about, I
figured I'd check here first.

So, does anyone know if there was any particular discussions of what alternatives might have been available, or what things specifically could/would break in Gentoo as a result of user-installed python
packages interacting with system ones? I know I can't be the only
one who was rather bothered by the manner it was addressed, so I
figure there was probably at least some exploratory testing or
theorizing done before PEP 668 was addopted into Gentoo, which would
give me a starting point to work from.

The PyPA community motivating factors here were:

- if you pip install --user, it can upgrade copies of packages such that
running /usr/bin/emerged-program as non-root sees the copy in $HOME
which it is incompatible with. Fedora discussed at the time that it is
possible to simply patch the shebang to use `python -s`, then decided
to not use that approach in the end because their package manager
supports pip install --user plugins and using `-s` would prevent
detecting plugins, therefore "better to forbid pip install --user".
Yes, that conclusion confused me too.

- if you `pip install --user some-app` it will use system copies of e.g.
scipy/numpy. If you then emerge -uDU @world and upgrade to a new
version of numpy, your app might not be compatible with it. This is
not actually a problem with the distro, and the famous byword of the
Python community is "the consenting adults principle" which states
that users should be permitted to choose whether the risk of breakage
is acceptable to them.


This "consenting adults" principle does not apply with regard to PEP
668. The PyPA community insists that the adult in the room is the
distro, and that PyPA has not forbidden anything -- distros are free to
use PEP 668 if they wish to, or not use it if they don't wish to.


The distro motivating factor here varies according to distro.

For Debian, users are definitely not adults. Debian got bug reports for
users whose --user installed software was causing issues, and did not
wish to support this use case. They wrote a PEP and got it approved, to
allow the distro to indicate "go jump in a lake, we don't support you",
which eventually coalesced into "either pass the --break-system-packages
option or jump in a lake, either way you agree that we don't support you".

The existence of that option highlights the motivating purpose of the
PEP. It's not about whether you can or cannot do stuff, it's about
"whether you are allowed to ask for help after you do it". Pass the
option and you "become an adult", to use the PyPA expression.


I asked about this with regard to Gentoo. The answer I received was that previously, Gentoo patched pip to error out if you run `pip install` as
root, to prevent pip deleting or overwriting files managed by portage,
and that was undesirable because distro patches need to be "maintained". Furthermore, my offer to maintain the patch is not an acceptable solution.

It is acknowledged by Gentoo Python (apparently) that PEP 668 is a
terrible idea and absolutely not what anyone wants (especially since it
does not prevent pip from deleting or overwriting files managed by
portage), but that it ***will*** be used anyway because it allows
avoiding a pip patch.

Here is my 2023 updated patch for pip, it still applies today without additional maintenance: https://github.com/eli-schwartz/pip/commit/f5c63c0deff8e90862fb3586c381c8664a98a2ef

Here is my 2023 PR to gpep517 to allow updating shebangs to add `-s`
when installing a python program with emerge: https://github.com/projg2/gpep517/pull/13

Assuming this PR could someday get merged, though it's not clear how
much interest there is (I've tried asking for reviews several times, as
have others), it would be trivial to pass the newly added flag in the
::gentoo eclass (distutils-r1) that currently invokes gpep517 to install
python software. It would also need to be separately reimplemented in python-single-r1 for packages that ship python scripts directly.


A plan of action for how I can proceed would be amazing. I'm out of ideas!

For clarity, I'm *not* looking for a workaround for my particular
system. My end goal (and not an easy one, granted) is to hopefully
eventually make a proper fix that makes it so no user of any distro
ever needs to touch a venv again if they don't want to.

I'm only a hobbyist programmer, but since nearly every distro under
the sun went with the "externally managed environment" thing, it's
clearly more an issue of motivation/time rather than raw skill. So I
figure maybe my extreme distaste for the current solution might
allow me to bash my head against the problem long enough to make
some headway.

I feel your frustration, especially the motivation part. :(


--
Eli Schwartz

--------------h9uso10o8VlfoRxIEbgSOMrB--

-----BEGIN PGP SIGNATURE-----

wnsEABYIACMWIQTnFNnmK0TPZHnXm3qEp9ErcA0vVwUCZ5fEDwUDAAAAAAAKCRCEp9ErcA0vV5HE AQDG9uCrmJlsh+I8IblZejW8G9ZDMERR3DFDU345M5gQxgEA69tKB0D4rnnbT/w9v+0/HLiVvAqU 77sKntqnTgaFBQI=
=Bxyp
-----END PGP SIGNATURE-----

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

This is a multi-part message in MIME format.
On 28/1/25 01:16, Mark Knecht wrote:

On Mon, Jan 27, 2025 at 9:37 AM Matthew Brooks
<matthewfbrooks@posteonet <mailto:matthewfbrooks@posteo.net>> wrote:

Hello!

Hi.

First, I cannot offer any insight into the PEP668 stuff. For me, it is
what it is.

For clarity, I'm *not* looking for a workaround for my particular

system. My end goal (and not an easy one, granted) is to hopefully
eventually make a proper fix that makes it so no user of any distro
ever needs to touch a venv again if they don't want to.

I'm only a hobbyist programmer, but since nearly every distro under

the sun went with the "externally managed environment" thing, it's
clearly more an issue of motivation/time rather than raw skill. So I
figure maybe my extreme distaste for the current solution might allow
me to bash my head against the problem long enough to make some headway.

Like you I am a hobbyist programmer and, in a _very_ general way agree
with your setup of "so no user of any distro ever needs to touch a
venv again if they don't want to". However my experience programming
Python always leads me towards the use of off-brand and sometimes
poorly maintained Python libraries that don't work in all python
releases so I'm forced to use venv, which for me works great. I also
use a lot of CUDA stuff within Python and tend, possibly due to my own
lack of skills as a hobbyist, or just my general paranoia about having
to debug things I don't want to mess with, to do _everything_ personal
in a venv. Grant you, I never intend to release a program to the
public so to me it's most important to keep it working, and clearly
nothing about what you might end up suggesting stops me from doing
what you are interested in.

I wish you the best in your adventure.

Cheers,
Mark

Same here, I regard current python and python based programs a mostly
broken despite the considerable effort of the gentoo devs ... if you
want privacy, do not want to use someone elses docker containers or just
use portage installed programs.  Example Frigate, example Homeassistant. example python itself ... venv's and all these walled gardens that are
not talking to each other ...

I am developing quite an aversion to python.

BillK


<!DOCTYPE html>
<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
</head>
<body>
<p><br>
</p>
<div class="moz-cite-prefix">On 28/1/25 01:16, Mark Knecht wrote:<br>
</div>
<blockquote type="cite" cite="mid:CAK2H+efwY7x5+CQdn1yh62ihXaHQ_rsQADBVp5UGnaYhQC4g7w@mail.gmail.com">
<meta http-equiv="content-type" content="text/html; charset=UTF-8">
<div dir="ltr"><br>
<br>
On Mon, Jan 27, 2025 at 9:37 AM Matthew Brooks &lt;<a
href="mailto:matthewfbrooks@posteo.net" moz-do-not-send="true">matthewfbrooks@posteonet</a>&gt;
wrote:<br>
&gt;<br>
&gt; Hello!<br>
&gt;<br>
<br>
Hi.
<div><br>
</div>
<div>First, I cannot offer any insight into the PEP668 stuff.
For me, it is what it is.<br>
<div><br>
&gt; For clarity, I'm *not* looking for a workaround for my
particular system. My end goal (and not an easy one,
granted) is to hopefully eventually make a proper fix that
makes it so no user of any distro ever needs to touch a venv
again if they don't want to.</div>
<div>&gt;</div>
<div>&gt; I'm only a hobbyist programmer, but since nearly
every distro under the sun went with the "externally managed
environment" thing, it's clearly more an issue of
motivation/time rather than raw skill. So I figure maybe my
extreme distaste for the current solution might allow me to
bash my head against the problem long enough to make some
headway.
<div><br>
</div>
</div>
<div>Like you I am a hobbyist programmer and, in a _very_
general way agree with your setup of "so no user of any
distro ever needs to touch a venv again if they don't want
to". However my experience programming Python always leads
me towards the use of off-brand and sometimes poorly
maintained Python libraries that don't work in all python
releases so I'm forced to use venv, which for me works
great. I also use a lot of CUDA stuff within Python and
tend, possibly due to my own lack of skills as a hobbyist,
or just my general paranoia about having to debug things I
don't want to mess with, to do _everything_ personal in a
venv. Grant you, I never intend to release a program to the
public so to me it's most important to keep it working, and
clearly nothing about what you might end up suggesting stops
me from doing what you are interested in.</div>
<div><br>
</div>
<div>I wish you the best in your adventure.</div>
<div><br>
</div>
<div>Cheers,</div>
<div>Mark</div>
<div><br>
</div>
</div>
</div>
</blockquote>
<p>Same here, I regard current python and python based programs a
mostly broken despite the considerable effort of the gentoo devs
... if you want privacy, do not want to use someone elses docker
containers or just use portage installed programs.  Example
Frigate, example Homeassistant. example python itself ... venv's
and all these walled gardens that are not talking to each other
...</p>
<p>I am developing quite an aversion to python.<br>
</p>
<p>BillK</p>
<p><br>
</p>
</body>
</html>

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)