On 14/04/25 11:10, Jonathan Gossage via Python-list wrote:

I am using *Python 3.13* in a virtual environment under *Ubuntu Linux 24.04* .
The version of Python was compiled from source code and installed with make altinstall. I attempted to use *pip* to install the *Sphinx* package into
the virtual environment using the command *pip install sphinx* in the
virtual environment*.* I expected that *sphinx* would be installed in the *site-packages* directory in the virtual environment. Instead, it was installed into the site-packages directory in */home/jonathan/.locals/lib/python3.13/site-packages* even though I did not specify *--user* to the *pip install* command. Is this expected behavior? I wanted Sphinx to be installed in the virtual environment so that it would
be accessible to all users of the virtual environment.

Which command did you type?
Which type of virtual-environment, and was it activated at the time?

--
Regards,
=dn

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

On Sun, 13 Apr 2025 19:10:47 -0400, Jonathan Gossage wrote:

The version of Python was compiled from source code and installed with
make altinstall. I attempted to use *pip* to install the *Sphinx*
package into the virtual environment using the command *pip install
sphinx* in the virtual environment*.

How did you create the venv? Was it activated? I would use 'python -m pip install sphinx' but a bare pip should work.

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

Please include the group in your response; don't just send it to me.

On 4/14/2025 5:09 AM, Jonathan Gossage wrote:

The virtual environment was owned by the user running pip. It was not
owned by root. Does pip not support virtual environments that are owned
by a non-root user and have a multi-user secondary group? The actual
command was *pip install mypy flake8 sphinx*. The other packages were
also installed into the user .local tree and work properly for the user
doing the installation, but not for other members of the group that have access to the virtual environment.

Pip doesn't know about the environment it runs in. It seems to me that
you didn't active the venv before you installed using pip. So nothing
would have gotten installed into the venv. So where is the venv that you
set up? I usually put them into ~/venv. For example, a venv named "gf4"
is at ~/venv/gf4.

To activate a venv, you have to source its activate script, which is in
the venv. First you have to mark it as executable. Then you source it -

source ~/venv/gf4/bin/activate

Now when you run python (or more likely, python3), it will find the
venv's directories before it will find the system's or user's. You know
the activation has been successful because the prompt changes to show
you. The activation applies to the terminal session in which you
activated the venv.

On Sun, Apr 13, 2025 at 10:11 PM Thomas Passin <list1@tompassin.net <mailto:list1@tompassin.net>> wrote:

On 4/13/2025 7:10 PM, Jonathan Gossage via Python-list wrote:
> I am using *Python 3.13* in a virtual environment under *Ubuntu
Linux 24.04*
> .
> The version of Python was compiled from source code and installed
with make
> altinstall. I attempted to use *pip* to install the *Sphinx*
package into
> the virtual environment using the command *pip install sphinx* in the
> virtual environment*.* I expected that *sphinx* would be
installed in the
> *site-packages* directory in the virtual environment. Instead, it was
> installed into the site-packages directory in
> */home/jonathan/.locals/lib/python3.13/site-packages* even though
I did not
> specify *--user* to the *pip install* command. Is this expected
behavior? I
> wanted Sphinx to be installed in the virtual environment so that
it would
> be accessible to all users of the virtual environment.

If you ran the command as a user, then pip would not have root
privileges and could not install into the system directory. It would
fall back to installing into the user's tree. It usually prints a
message to that effect. That's standard behavior if you don't have the
venv activated.

If you want to install something into a virtual environment, you
have to
activate the environment before installing the something.

A complication could occur if the system's Python version is the
same as
the one you built. You might inadvertently run the system's binary of
python 3.13 instead of your own. I'm not familiar with the make
altinstall command but I doubt that it changes the ordinary rules for
invoking python and using a venv.

--
Jonathan Gossage

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

On Mon, 14 Apr 2025 09:55:09 -0400, Thomas Passin wrote:

Pip doesn't know about the environment it runs in. It seems to me that
you didn't active the venv before you installed using pip. So nothing
would have gotten installed into the venv. So where is the venv that you
set up? I usually put them into ~/venv. For example, a venv named "gf4"
is at ~/venv/gf4.

Are you sure about that? activate has


VIRTUAL_ENV="/home/rbowman/work/python/weather"
export VIRTUAL_ENV

_OLD_VIRTUAL_PATH="$PATH"
PATH="$VIRTUAL_ENV/bin:$PATH"
export PATH

$PATH is modified to point to the bin directory in the venv, and it
includes python and pip. You can use 'python3' if you really want to since
it is there also.

$ which pip
/home/rbowman/work/python/weather/bin/pip

'deactivate' restores $PATH.

I'll agree it sounds like the venv wasn't activated.

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

On Mon, 14 Apr 2025 15:20:13 -0700, Keith Thompson wrote:

Making the active script executable introdues the risk that you'll accidentally execute it rather than sourcing it. If you do that, it
will probably set up the environment in a new shell process which then immediately terminates.

The 'activate' script starts with

# This file must be used with "source bin/activate" *from bash*
# You cannot run it directly

The csh and fish variants have similar warnings. I don't have a Windows
box up at the moment but iirc activate.bat or Activate.ps1 is in the
Scripts subdirectory and you do run it directly. On Linux the python in
bin is usually a symlink, although you can specify it to be copied with a parameter to venv. Symlinks on Windows are problematic but the process is
more or less the same.

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

On 4/14/2025 6:20 PM, Keith Thompson via Python-list wrote:

Thomas Passin <list1@tompassin.net> writes:
[...]

To activate a venv, you have to source its activate script, which is
in the venv. First you have to mark it as executable. Then you source
it -

source ~/venv/gf4/bin/activate

[...]

No, you don't have to (and probably shouldn't) mark the script as
executable.

Making a script executable (chmod +x) is required before *executing* it,
but when you *source* a script (using "source" or "."), your current
shell reads it and evaluates its content.

Making the active script executable introdues the risk that you'll accidentally execute it rather than sourcing it. If you do that, it
will probably set up the environment in a new shell process which then immediately terminates.

You are right, my bad. When I went to check on what the venv prompt
looked like after activation, I hadn't run my Linux VM for too long and
forgot that the activate script needs to be sourced - in Windows it just
gets run as any other script. I noticed it wasn't marked executable and
blindly "fixed" that. Then of course I remembered the script has to be
sourced - and forgot to undo the execute flag.

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

On 4/14/2025 6:20 PM, Keith Thompson via Python-list wrote:

Thomas Passin <list1@tompassin.net> writes:
[...]

To activate a venv, you have to source its activate script, which is
in the venv. First you have to mark it as executable. Then you source
it -

source ~/venv/gf4/bin/activate

[...]

No, you don't have to (and probably shouldn't) mark the script as
executable.

Making a script executable (chmod +x) is required before *executing* it,
but when you *source* a script (using "source" or "."), your current
shell reads it and evaluates its content.

Making the active script executable introdues the risk that you'll accidentally execute it rather than sourcing it. If you do that, it
will probably set up the environment in a new shell process which then immediately terminates.

You are right, my bad. When I went to check on what the venv prompt
looked like after activation, I hadn't run my Linux VM for too long and
forgot that the activate script needs to be sourced - in Windows it just
gets run as any other script. I noticed it wasn't marked executable and
blindly "fixed" that. Then of course I remembered the script has to be
sourced - and forgot to undo the execute flag.

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

On Tue, 15 Apr 2025 18:43:43 +0200, Friedrich Romstedt wrote:

Many people put emphasis on that you need to *activate* a virtualenv
before using it, but no-one so far stressed the fact that you got Sphinx installed to ~/jonathan/.local/lib/python3.13/site-packages *without
using *--user.

This may depend on the OS. On Ubuntu 24.10 with python 3.12.7 I get

$ pip install black
error: externally-managed-environment

× This environment is externally managed
╰─> To install Python packages system-wide, try apt install
python3-xyz, where xyz is the package you are trying to
install.
If you wish to install a non-Debian-packaged Python package,
create a virtual environment using python3 -m venv path/to/venv.
Then use path/to/venv/bin/python and path/to/venv/bin/pip. Make
sure you have python3-full installed.

I do have '/home/rbowman/.local/lib/python3.10/site-packages' left over
from earlier versions of Ubuntu.


However on Fedora 42 with 3.13.2

Defaulting to user installation because normal site-packages is not
writable.

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

On 4/15/25 10:43, Friedrich Romstedt via Python-list wrote:

Many people put emphasis on that you need to *activate* a virtualenv before using it, but no-one so far stressed the fact that you got Sphinx installed to ~/jonathan/.local/lib/python3.13/site-packages *without using *--user.

To be clear: you do not have to activate a virtualenv to use *Python*
from it. If you just call the python by the path it's in, it figures
everything out (and sets some variables you can query vi sysconfig if
you have reason to actually need those details (look for installation
schemes).

What activating gives you is some path fiddling, and some prompt
fiddling (although the prompt fiddling keeps saying it's deprecated).
The latter is a visual clue; the former means you can also find *other* commands installed in the virtualenv - including pip.

/path/to/virtualenv//bin/python -m pip install ... will work whether
you activated or not.

pip install ... finds the first command in your PATH named pip, which
may or may not be the one in the virtualenv, *unless* you've activated
it, because that's the only way the virtualenv bin directory ends up
early in your path.

Meanwhile - the install to ~/.local : some distros just default to that behavior whether you ask for it or not, because they consider it unsafe
to install in the system location. Some have chosen to yell at you even
if you try a "user install" with the system Python. Up to them what they
do...

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

Various responses have been provided but the OP has not yet replied
on-list (as verified by Archive). Is this an error with the
list-processor or have some posts been sent to one person only (using
Reply instead of ReplyList)?

There are always others who would like to learn from list-discussions -
but will hearing only half of some of the conversation help them?


On 14/04/25 11:33, dn via Python-list wrote:

On 14/04/25 11:10, Jonathan Gossage via Python-list wrote:

I am using *Python 3.13* in a virtual environment under *Ubuntu Linux
24.04*
.
The version of Python was compiled from source code and installed with
make
altinstall. I attempted to use *pip* to install the *Sphinx* package into
the virtual environment using the command *pip install sphinx* in the
virtual environment*.* I expected that *sphinx* would be installed in the
*site-packages* directory in the virtual environment. Instead, it was
installed into the site-packages directory in
*/home/jonathan/.locals/lib/python3.13/site-packages* even though I
did not
specify *--user* to the *pip install* command. Is this expected
behavior? I
wanted Sphinx to be installed in the virtual environment so that it would
be accessible to all users of the virtual environment.

Which command did you type?
Which type of virtual-environment, and was it activated at the time?

--
Regards,
=dn

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

On 4/15/2025 3:41 PM, dn via Python-list wrote:

Various responses have been provided but the OP has not yet replied on-
list (as verified by Archive). Is this an error with the list-processor
or have some posts been sent to one person only (using Reply instead of ReplyList)?

There are always others who would like to learn from list-discussions -
but will hearing only half of some of the conversation help them?

The OP sent some replies to me and I asked them to include the list.

On 14/04/25 11:33, dn via Python-list wrote:

On 14/04/25 11:10, Jonathan Gossage via Python-list wrote:

I am using *Python 3.13* in a virtual environment under *Ubuntu Linux
24.04*
.
The version of Python was compiled from source code and installed
with make
altinstall. I attempted to use *pip* to install the *Sphinx* package
into
the virtual environment using the command *pip install sphinx* in the
virtual environment*.* I expected that *sphinx* would be installed in
the
*site-packages* directory in the virtual environment. Instead, it was
installed into the site-packages directory in
*/home/jonathan/.locals/lib/python3.13/site-packages* even though I
did not
specify *--user* to the *pip install* command. Is this expected
behavior? I
wanted Sphinx to be installed in the virtual environment so that it
would
be accessible to all users of the virtual environment.

Which command did you type?
Which type of virtual-environment, and was it activated at the time?

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

On 4/15/2025 12:43 PM, Friedrich Romstedt via Python-list wrote:

Am Mo., 14. Apr. 2025 um 01:14 Uhr schrieb Jonathan Gossage via Python-list <python-list@python.org>:

I am using *Python 3.13* in a virtual environment under *Ubuntu Linux
24.04*
.
[...]

Instead, it was
installed into the site-packages directory in
*/home/jonathan/.locals/lib/python3.13/site-packages* even though I did not >> specify *--user* to the *pip install* command. Is this expected behavior? I >> wanted Sphinx to be installed in the virtual environment so that it would
be accessible to all users of the virtual environment.

Hi Jonathan,

Many people put emphasis on that you need to *activate* a virtualenv before using it, but no-one so far stressed the fact that you got Sphinx installed to ~/jonathan/.local/lib/python3.13/site-packages *without using *--user.

On Linux, at least, it's standard for pip to install into the user's site-packages location if it's not invoked with admin privileges - even
without --user. Pip will emit a message saying so. Well, that used to be
true but nowadays Pip wants you to use the --break-system-packages flag
if you want to insist on installing into the system's Python install,
even if it's going to go into --user. I'm not sure if the restriction
will be in place given that the OP built his own Python version.

Do you have, by any chance, one of the following pip-related configuration files present?

- /etc/pip.conf
- ~/.config/pip/pip.conf
- ~/.pip/pip.conf (lagacy)

I took this list from https://pip.pypa.io/en/stable/topics/configuration/. *If* there is any such file present, you might look out for a line:

user = true

but I am not entirely certain about this.

Looking forward to your reply!

Best,
Friedrich

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

On Tue, 15 Apr 2025 14:12:19 -0400, Thomas Passin wrote:

On Linux, at least, it's standard for pip to install into the user's site-packages location if it's not invoked with admin privileges - even without --user. Pip will emit a message saying so. Well, that used to be
true but nowadays Pip wants you to use the --break-system-packages flag
if you want to insist on installing into the system's Python install,
even if it's going to go into --user. I'm not sure if the restriction
will be in place given that the OP built his own Python version.

Is that pip or a distro's version of pip? On Fedora I get the message
about defaulting to user. On Ubuntu I get a message to use a venv or if I really want a global install to use something like 'pip install python3- black'. Ubuntu's is pip 24.2, Fedor's is 24.3.1 but neither of them show '--break-system-packages' in --help.

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

On 4/15/25 16:07, Grant Edwards via Python-list wrote:

On 2025-04-15, Thomas Passin via Python-list <python-list@python.org> wrote:

On Linux, at least, it's standard for pip to install into the user's
site-packages location if it's not invoked with admin privileges - even
without --user. Pip will emit a message saying so. Well, that used to be
true but nowadays Pip wants you to use the --break-system-packages flag
if you want to insist on installing into the system's Python install,
even if it's going to go into --user.

I've always been a little baffled by that message when installing with --user. How can that possibly break system stuff?

Your user install dir is in your python path, so when you go to run an installed Python program which imports other packages, it might pick up
the version you have in your user space rather than the system one it
was tested with. It's about a whole curated Python environment that the
distro spends time validating - a different version picked up elsewhere
*might* be fine. Or it might not. I think they've had enough "it's not"
bug reports to have gotten quite prickly about the subject.

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

On 4/15/2025 5:38 PM, rbowman via Python-list wrote:

On Tue, 15 Apr 2025 14:12:19 -0400, Thomas Passin wrote:

On Linux, at least, it's standard for pip to install into the user's
site-packages location if it's not invoked with admin privileges - even
without --user. Pip will emit a message saying so. Well, that used to be
true but nowadays Pip wants you to use the --break-system-packages flag
if you want to insist on installing into the system's Python install,
even if it's going to go into --user. I'm not sure if the restriction
will be in place given that the OP built his own Python version.

Is that pip or a distro's version of pip? On Fedora I get the message
about defaulting to user. On Ubuntu I get a message to use a venv or if I really want a global install to use something like 'pip install python3- black'. Ubuntu's is pip 24.2, Fedor's is 24.3.1 but neither of them show '--break-system-packages' in --help.

The behavior is specifed in

https://packaging.python.org/en/latest/specifications/externally-managed-environments/#externally-managed-environments

Exactly how pip works and what messages it emits are specified by this document, and the details depend on how the distro's packagers configure
it. For example, here is a bit of the spec:

"Before a Python-specific package installer (that is, a tool such as pip
- not an external tool such as apt) installs a package into a certain
Python context, it should make the following checks by default:

Is it running outside of a virtual environment? It can determine
this by whether sys.prefix == sys.base_prefix.

Is there an EXTERNALLY-MANAGED file in the directory identified by sysconfig.get_path("stdlib", sysconfig.get_default_scheme())?

If both of these conditions are true, the installer should exit with an
error message indicating that package installation into this Python interpreter’s directory are disabled outside of a virtual environment.

The installer should have a way for the user to override these rules,
such as a command-line flag --break-system-packages. This option should
not be enabled by default and should carry some connotation that its use
is risky."

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

Op 15/04/2025 om 20:31 schreef Mats Wichmann via Python-list:

To be clear: you do not have to activate a virtualenv to use *Python*
from it. If you just call the python by the path it's in, it figures everything out (and sets some variables you can query vi sysconfig if
you have reason to actually need those details (look for installation schemes).

What activating gives you is some path fiddling, and some prompt
fiddling (although the prompt fiddling keeps saying it's deprecated).
The latter is a visual clue; the former means you can also find
*other* commands installed in the virtualenv - including pip.

/path/to/virtualenv//bin/python -m pip install ...   will work whether
you activated or not.

pip install ...  finds the first command in your PATH named pip, which
may or may not be the one in the virtualenv, *unless* you've activated
it, because that's the only way the virtualenv bin directory ends up
early in your path.

And the pip command that is found and run will use it's own settings
regarding where to install packages, even if you activated a virtualenv.
For example, you can't use /usr/bin/pip to install something in a
virtualenv. To install something in a virtualenv, you need to use the
pip in that virtualenv (either by first activating that venv, or by
running something like venv/bin/pip, or venv). (Of course to do that pip
needs to be installed in that venv. That might or might not be the case depending on how the venv was created.)

I kinda get the feeling that something like that is going on here.

--
"There is a theory which states that if ever anyone discovers exactly what the Universe is for and why it is here, it will instantly disappear and be
replaced by something even more bizarre and inexplicable.
There is another theory which states that this has already happened."
-- Douglas Adams, The Restaurant at the End of the Universe

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

On 4/17/2025 4:58 AM, Roel Schroeven via Python-list wrote:

Op 15/04/2025 om 20:31 schreef Mats Wichmann via Python-list:

To be clear: you do not have to activate a virtualenv to use *Python*
from it. If you just call the python by the path it's in, it figures
everything out (and sets some variables you can query vi sysconfig if
you have reason to actually need those details (look for installation
schemes).

What activating gives you is some path fiddling, and some prompt
fiddling (although the prompt fiddling keeps saying it's deprecated).
The latter is a visual clue; the former means you can also find
*other* commands installed in the virtualenv - including pip.

/path/to/virtualenv//bin/python -m pip install ...   will work whether
you activated or not.

pip install ...  finds the first command in your PATH named pip, which
may or may not be the one in the virtualenv, *unless* you've activated
it, because that's the only way the virtualenv bin directory ends up
early in your path.

Or you can cd to the venv directory and it will be first on the pythonpath.

And the pip command that is found and run will use it's own settings regarding where to install packages, even if you activated a virtualenv.
For example, you can't use /usr/bin/pip to install something in a
virtualenv.

pip detects if you are running in a venv and changes its behavior to match.

To install something in a virtualenv, you need to use the

pip in that virtualenv (either by first activating that venv, or by
running something like venv/bin/pip, or venv). (Of course to do that pip needs to be installed in that venv. That might or might not be the case depending on how the venv was created.)

I kinda get the feeling that something like that is going on here.

IMHO pip should always be run as a module, e.g., python3 -m pip. This
way you always get the same pip and environment as the python executable
that is being used.

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

On 4/17/25 15:15, Grant Edwards via Python-list wrote:

On 2025-04-17, Left Right via Python-list <python-list@python.org> wrote:

Also... when installing stuff with pip --user, it is always a package
that is not installed for the system (usually not even available for
the system). How can that "break system packages"?

pip installs dependencies. Dependencies may disagree on the version
with the system packages.

Good point. I don't generally allow pip to install dependencies, so I
forgot about that source of conflict. But as far as I can see, it
still can only break stuff for the user, not for the system. The
system is never going to look in my ~/.local/lib/python* directories.

Sure, and it's "your fault" (sic) if you break yourself. You and I can
maybe accept that. But it's usually not obvious that you're done
anything to break stuff - you just asked to install something, according
to the instructions you read on the project's website, which it's
reasonable (?) to expect reflects best practices. Usually a simple "pip install".

Say you've been using "MLwiz" installed via a system package. It has
two dozen dependencies, some of which have further deps. And it works
great. Then you install "Chatbridge" via pip, and it brings with it its
own dependencies. It's not really useful to refuse to let it bring its
own deps with it since there may be version-specific requirements on its
deps, which that project has reasons for. So now you've got a few pkgs
in the system with different versions in your .local, and you can't run
MLwiz any longer, because a .local pkg (picked up first) conflicts with
the requirements of MLwiz.

There's just not a really great answer to this.

"Use a virtualenv". Fine, but that's still daunting to many, and not necessarily as well described as it could be, and the choice of the word "virtual" in the name makes it sound more complex than it actually is.
Projects can certainly do better at describing installation scenarios -
at least if they have more than a trivial number of deps, where the
conflict chances go up rapidly.

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

On 18/04/25 9:41 am, Mats Wichmann wrote:

There's just not a really great answer to this.

Seems to me a system-installed application shouldn't be looking in the
user's .local packages in the first place. That should only be for
things the user has installed "for this user".

--
Greg

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

On 2025-04-18 13:24:28 +1200, Greg Ewing via Python-list wrote:

On 18/04/25 9:41 am, Mats Wichmann wrote:

There's just not a really great answer to this.

Seems to me a system-installed application shouldn't be looking in the
user's .local packages in the first place. That should only be for things
the user has installed "for this user".

It's not the application that looks into .local, it's Python. If you say
that a system-installed Python shouldn't look into ~/.local, then you've
just disabled that mechanism completely. If not then Python would
somehow have to distinguish between system-installed and user-installed scripts. This isn't as easy as checking whether the path starts with
/usr/bin or whether it belongs to root. Tying into the system's package
manager doesn't look appealing to me (not to mention that it might be unacceptably slow).

hjp

--
_ | Peter J. Holzer | Story must make more sense than reality.
|_|_) | |
| | | hjp@hjp.at | -- Charles Stross, "Creative writing
__/ | http://www.hjp.at/ | challenge!"

-----BEGIN PGP SIGNATURE-----

iQIzBAABCgAdFiEETtJbRjyPwVTYGJ5k8g5IURL+KF0FAmgCcgIACgkQ8g5IURL+ KF1crxAAohZV8Zn9PAMD+EmDnHFVqmp1CeDDLoMYcedcxWZhsDW5lBVJKr59c8yv jz6bUVZJdOvpeXOXz52GkUIV2mghqo2sI7TLYYNVd4y62mVHCglPytx1mCxYF1Aa zQrdbcJuPCYdS1Q5aENc2h+/KYcW+pii+dP1LaViWg+xb8Du09BYsY09Psr1ogRl r25/6IOPBXdl0GJXhjE3Er8DJCDlbaxgvMGK/5S3kceK3gvFIDWyp/ZctHmbZDdM kPsbeW5el3qMFfUs7Y+4Osw36kMi5/BcY9Ef0qNy0vfn0Mj4qmQUR0BlsLicByha 59obkuNEgHLTphLhlzPT0bPGaVISp81I2GCHUsueVM05JXPqyU0NegIi5QdLnKhS q9OpMtEQBYKvS3g+cH3ddxzrFY/FQkj5V3xsKQdnXCiWNl+3mo7hAmpWnrt8wf9T 4X+vuzYJY2hJslGiQlHiwO6UI66ObB+aNEhJHxLMJr8HNOUj1cXoH99Oc0UHltyN eyVFTL4zRvVi1mXnoZJGiCA1jL9Ssdm7JOT+cnf0d6ZMapEHqQ2W8gIs/CQIVLTV PHsgiuuPY9LS0XiOG90OVVCb5EDBt8hfhVBnozjQE5o+W1EJyNTncaePIpnXkBeo mxog3RB9/FCihbvRZJinTQNKgHvpfamDZHk/vMh

On 4/18/2025 11:38 AM, Peter J. Holzer via Python-list wrote:

On 2025-04-18 13:24:28 +1200, Greg Ewing via Python-list wrote:

On 18/04/25 9:41 am, Mats Wichmann wrote:

There's just not a really great answer to this.

Seems to me a system-installed application shouldn't be looking in the
user's .local packages in the first place. That should only be for things
the user has installed "for this user".

It's not the application that looks into .local, it's Python. If you say
that a system-installed Python shouldn't look into ~/.local, then you've
just disabled that mechanism completely. If not then Python would
somehow have to distinguish between system-installed and user-installed scripts. This isn't as easy as checking whether the path starts with
/usr/bin or whether it belongs to root. Tying into the system's package manager doesn't look appealing to me (not to mention that it might be unacceptably slow).

Let's try a specific example. Let's say that PyQt6 v6.8.3 is installed
in the system site directory. The OS uses PyQt6 for some system
purposes. Now the user comes along and forces an install of PyQt6 v6.9.0
in the user site directory. 6.9.0 has a bug that would crash one of the system's application but not the user's programs. (This is not a
far-fetched scenario).

When the system launches its application the PYTHONPATH will start with
system site directories; local user site directories will be on the
PYTHONPATH but since they come later, the python will use PyQt6 v6.8.3
because that will come first on the path. No crash here.

If the user has a program that actually does require the use of v6.9.0,
he's going to have to make sure that the user's local site directories
come first on the path. One way to do that is to set the PYTHONPATH to
point to the user's location.

In what scenario is a system application going to load a wrong version
of a dependency from a user's site location? The only one I can think of
is for the user, with the help of sudo, or by editing some
system-enabled script, were to change the global PYTHONPATH. That seems
a stretch.

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

On 2025-04-18 17:11:33 +0100, Oscar Benjamin via Python-list wrote:

On Fri, 18 Apr 2025 at 16:50, Peter J. Holzer via Python-list <python-list@python.org> wrote:

On 2025-04-18 13:24:28 +1200, Greg Ewing via Python-list wrote:

On 18/04/25 9:41 am, Mats Wichmann wrote:

There's just not a really great answer to this.

Seems to me a system-installed application shouldn't be looking in the user's .local packages in the first place. That should only be for things the user has installed "for this user".

It's not the application that looks into .local, it's Python. If you say that a system-installed Python shouldn't look into ~/.local, then you've just disabled that mechanism completely. If not then Python would
somehow have to distinguish between system-installed and user-installed scripts. This isn't as easy as checking whether the path starts with /usr/bin or whether it belongs to root. Tying into the system's package manager doesn't look appealing to me (not to mention that it might be unacceptably slow).

Couldn't the system-installed scripts have a shebang like:

#!/usr/bin/python3 -s

Yes, that should work. At least I can't think of any downsides at the
moment.

hjp

--
_ | Peter J. Holzer | Story must make more sense than reality.
|_|_) | |
| | | hjp@hjp.at | -- Charles Stross, "Creative writing
__/ | http://www.hjp.at/ | challenge!"

-----BEGIN PGP SIGNATURE-----

iQIzBAABCgAdFiEETtJbRjyPwVTYGJ5k8g5IURL+KF0FAmgDYREACgkQ8g5IURL+ KF0z/xAAgIhbMqE4Ompkwiexp2e7jrFbcKcAoWUehTQc8QQpfynNxK1AGQ66XWXc 5bIjOOCZbp4XYpLfpIfd+cPlGYmaANTnRcBdWLNwrjF//uYx6HuvtBOqKZKGGq1e 5aQbh0+3gq4Ade4JKCQXWh1YQ8vF0ux66vSbRKNa7PAe9hKcrKaBe0pMrmstNfk8 caUE1oWdxvGvgx0waH1NOwhaJiiwhItTAAvo9xxcbgzBhbIizogwuYAzCJ+xkiP+ R4IPcwLz0lReTJSSnXprUDrh17mi4wv3aNXbdpxLB82Efm7ZxQd14/lBC4kSqlYL yS5NOujbj8Po/y+nzi9XGQHJVn4P9Naom8zAcqBalbInj4qu1Xtvw0MEMrLvvQL2 eoS26KYCXbDR+0jxV/kGhPqn2TTr/HbYvQoFxPZiH/tfd8v0xXQQarZ1Cd/Jm+Ho YPMvCR403/uSvb1ilvgBujlQNTaVkadI0aUkWBNlg3F4evRdstF2IcsA2Lnbu3Ns pXjsGKBQDCq02v4DLNIr0uge7Ia9NcnyAhYdRvEMv6pjcy5RsyepR3UAcfiNnjR/ KW1ysFKe1CDkDFxnAb6Ei6202ADA+DtQZIx9Qre3RBFMR7Vuh0SjlyYtYQMvuHDl Zzooid9mr4EPpjYuGTq9kxS8I+wMTPLrjwI+xcR

On 2025-04-18 13:08:36 -0400, Thomas Passin via Python-list wrote:

On 4/18/2025 11:38 AM, Peter J. Holzer via Python-list wrote:

On 2025-04-18 13:24:28 +1200, Greg Ewing via Python-list wrote:

On 18/04/25 9:41 am, Mats Wichmann wrote:

There's just not a really great answer to this.

Seems to me a system-installed application shouldn't be looking in the user's .local packages in the first place. That should only be for things the user has installed "for this user".

It's not the application that looks into .local, it's Python. If you say that a system-installed Python shouldn't look into ~/.local, then you've just disabled that mechanism completely. If not then Python would
somehow have to distinguish between system-installed and user-installed scripts. This isn't as easy as checking whether the path starts with /usr/bin or whether it belongs to root. Tying into the system's package manager doesn't look appealing to me (not to mention that it might be unacceptably slow).

Let's try a specific example. Let's say that PyQt6 v6.8.3 is installed in
the system site directory. The OS uses PyQt6 for some system purposes. Now the user comes along and forces an install of PyQt6 v6.9.0 in the user site directory. 6.9.0 has a bug that would crash one of the system's application but not the user's programs. (This is not a far-fetched scenario).

Agreed. Also since PyQt6 is a GUI framework, I'm going to assume that
those applications are supposed to be invoked by the user on their
desktop, not by a cronjob controlled by the system.

When the system launches its application the PYTHONPATH will start with system site directories; local user site directories will be on the PYTHONPATH but since they come later, the python will use PyQt6 v6.8.3 because that will come first on the path. No crash here.

If the user has a program that actually does require the use of v6.9.0, he's going to have to make sure that the user's local site directories come first on the path. One way to do that is to set the PYTHONPATH to point to the user's location.

This is IMHO not practical. The user would have to set PYTHONPATH for
some programs, but not for others. You can't do this with .bashrc (or
similar), the user would have to write a wrapper script for each of
their programs which depend on something in ~/.local. Possible of course
but cumbersome.

I like Oscar's suggestion that Python scripts provided by the
distribution include -s in the shebang much better.

Or - what I tend to do - simply use a virtual environment for each
script that needs a package that the system doesn't provide. But of
course that's basically "disable/don't use .local" and all those venvs
take space and need to be maintained.

In what scenario is a system application going to load a wrong version of a dependency from a user's site location?

When the user sets PYTHONPATH and then accidentally invokes the
system-supplied application without resetting it first.

The only one I can think of is for the user, with the help of sudo, or
by editing some system-enabled script, were to change the global
PYTHONPATH. That seems a stretch.

No, there doesn't have to be a global (in the sense that it applies to
all users) PYTHONPATH for that to happen. You just need a PYTHONPATH
that is set for all processes of that user - which the user can
certainly set without sudo (usually by editing .bashrc or maybe using
their desktop environment's settings dialog).

hjp

--
_ | Peter J. Holzer | Story must make more sense than reality.
|_|_) | |
| | | hjp@hjp.at | -- Charles Stross, "Creative writing
__/ | http://www.hjp.at/ | challenge!"

-----BEGIN PGP SIGNATURE-----

iQIzBAABCgAdFiEETtJbRjyPwVTYGJ5k8g5IURL+KF0FAmgDZSgACgkQ8g5IURL+ KF2Fgw/+JSjASdQ79OXceKZHRToWr38cbCb9O5Q74xaoIjbOjrBK/Rgz3OFsSFxU NmwxxXvRiPkOa1KnvawDXmj/iedEqx1hIFJ+Nc66t7lyrWAtu7MSW/gmxXawGbL+ Y40/jMN6US0Ra9ZeYt1Jv/nU7qqgdT/p3mtDnQXlG3LLOZZEHhu//pS09tUHD7a/ y2B88/QzCe2WqnNEGnId+LUIzocEGTGKKWss7MVguKAw86ZxqA6tdv5W7GNTkHfb l77rJcaiJx/tAH29lbheo34WfHW3BpdACxBwgqLQJZmqqrLb4rY36mh540r2yGpU bVfdTWbRvFUkJVcRdvdDpKagShSLA5a2nxU9D8BVlSz8rbAOofHzKkDN+ZiC3OWb f6Xgz/eBVuxWDENOTobJOCiI8pCiPay7qlT9v9UVmKYotY7Rj3PysSUfyvhIPqSf +rarT1puLYAnorj+S54KkVBFKsU21wOMKGqg0YONn5rXRzbo1H6HIGcgwdz/7uNn hYZp5y0G0L+lIacueemMXQrhvfcu9OSSMqzakLMfzT1IAY5BtiDD7k07jD0ZFIAy cFJPuq3mLYVZi40xOlkuC5kPb72cVOMgIN1BXgZWhMqVDrSClqnXa2ASZu4WxWpr fTtM8+HFRzgmtN5ZK7Naa47yzff0xqlmZXDgbVD

Peter J. Holzer wrote:

On 2025-04-18 13:08:36 -0400, Thomas Passin via Python-list wrote:

...

When the system launches its application the PYTHONPATH will start with
system site directories; local user site directories will be on the
PYTHONPATH but since they come later, the python will use PyQt6 v6.8.3
because that will come first on the path. No crash here.

If the user has a program that actually does require the use of v6.9.0, he's >> going to have to make sure that the user's local site directories come first >> on the path. One way to do that is to set the PYTHONPATH to point to the
user's location.

This is IMHO not practical. The user would have to set PYTHONPATH for
some programs, but not for others. You can't do this with .bashrc (or similar), the user would have to write a wrapper script for each of
their programs which depend on something in ~/.local. Possible of course
but cumbersome.

currently in my .bashrc i have it set up to look for which
directory the terminal is in and then runs the activate script
for that environment. i like to keep my desktops/projects
open with various numbers of terminals available so this way
they are all ready to go when the system boots up.

I like Oscar's suggestion that Python scripts provided by the
distribution include -s in the shebang much better.

Or - what I tend to do - simply use a virtual environment for each
script that needs a package that the system doesn't provide. But of
course that's basically "disable/don't use .local" and all those venvs
take space and need to be maintained.

i like that they do not change until i specificly ask them to
be changed.


...

The only one I can think of is for the user, with the help of sudo, or
by editing some system-enabled script, were to change the global
PYTHONPATH. That seems a stretch.

No, there doesn't have to be a global (in the sense that it applies to
all users) PYTHONPATH for that to happen. You just need a PYTHONPATH
that is set for all processes of that user - which the user can
certainly set without sudo (usually by editing .bashrc or maybe using
their desktop environment's settings dialog).

yes, that is part of what .bashrc is for, making sure your
environment variables are set how you'd like them.


songbird

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

On 4/19/2025 4:56 AM, Peter J. Holzer via Python-list wrote:

On 2025-04-18 13:08:36 -0400, Thomas Passin via Python-list wrote:

On 4/18/2025 11:38 AM, Peter J. Holzer via Python-list wrote:

On 2025-04-18 13:24:28 +1200, Greg Ewing via Python-list wrote:

On 18/04/25 9:41 am, Mats Wichmann wrote:

There's just not a really great answer to this.

Seems to me a system-installed application shouldn't be looking in the >>>> user's .local packages in the first place. That should only be for things >>>> the user has installed "for this user".

It's not the application that looks into .local, it's Python. If you say >>> that a system-installed Python shouldn't look into ~/.local, then you've >>> just disabled that mechanism completely. If not then Python would
somehow have to distinguish between system-installed and user-installed
scripts. This isn't as easy as checking whether the path starts with
/usr/bin or whether it belongs to root. Tying into the system's package
manager doesn't look appealing to me (not to mention that it might be
unacceptably slow).

Let's try a specific example. Let's say that PyQt6 v6.8.3 is installed in
the system site directory. The OS uses PyQt6 for some system purposes. Now >> the user comes along and forces an install of PyQt6 v6.9.0 in the user site >> directory. 6.9.0 has a bug that would crash one of the system's application >> but not the user's programs. (This is not a far-fetched scenario).

Agreed. Also since PyQt6 is a GUI framework, I'm going to assume that
those applications are supposed to be invoked by the user on their
desktop, not by a cronjob controlled by the system.

When the system launches its application the PYTHONPATH will start with
system site directories; local user site directories will be on the
PYTHONPATH but since they come later, the python will use PyQt6 v6.8.3
because that will come first on the path. No crash here.

If the user has a program that actually does require the use of v6.9.0, he's >> going to have to make sure that the user's local site directories come first >> on the path. One way to do that is to set the PYTHONPATH to point to the
user's location.

This is IMHO not practical. The user would have to set PYTHONPATH for
some programs, but not for others. You can't do this with .bashrc (or similar), the user would have to write a wrapper script for each of
their programs which depend on something in ~/.local. Possible of course
but cumbersome.

I like Oscar's suggestion that Python scripts provided by the
distribution include -s in the shebang much better.

Or - what I tend to do - simply use a virtual environment for each
script that needs a package that the system doesn't provide. But of
course that's basically "disable/don't use .local" and all those venvs
take space and need to be maintained.

My problem with venvs, especially if I have more than one, is that I
eventually forget what they were for and what is different about each
one. If there's only one and it's used for all non-system work, that's
OK but beyond that and they tend to suffer knowledge rot.

In what scenario is a system application going to load a wrong version of a >> dependency from a user's site location?

When the user sets PYTHONPATH and then accidentally invokes the system-supplied application without resetting it first.

The only one I can think of is for the user, with the help of sudo, or
by editing some system-enabled script, were to change the global
PYTHONPATH. That seems a stretch.

No, there doesn't have to be a global (in the sense that it applies to
all users) PYTHONPATH for that to happen. You just need a PYTHONPATH
that is set for all processes of that user - which the user can
certainly set without sudo (usually by editing .bashrc or maybe using
their desktop environment's settings dialog).

Yes, I agree and I wasn't thinking locally enough.

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

On Sat, 19 Apr 2025 15:56:16 -0400, Thomas Passin wrote:

My problem with venvs, especially if I have more than one, is that I eventually forget what they were for and what is different about each
one. If there's only one and it's used for all non-system work, that's
OK but beyond that and they tend to suffer knowledge rot.

My Python directory has

apple/ create/ fastapi/ lunar/ numerical/ pyside6/ weather/
comics/ django/ folium/ ml/ sqlite/ coursera/ impractical/
nn/ pyqt/ torch/

Not all like sqlite are venvs since no additional modules are needed. Even
if I'm a little hazy about 'apple' after a while a quick look at the
Python file shows 'https://itunes.apple.com/search' and I remember it is
to pull down artist/track info from the itunes database.

I also try to remember to run 'pythom -m pip freeze > requirements.txt'
after I get everything set up. That way if a OS update installs a new
version of Python something like

python3 -m venv folium
cd folium
. bin/activate
python -m pip install -r requirements.txt

will refresh the venv and install the appropriate packages for the new
Python version. That's faster than looking at lib/python3.10/site-packages
or whatever the previous version was and adding the dependencies or
trying to run the py file and getting

Traceback (most recent call last):
File "/home/rbowman/work/python/impractical/benford.py", line 5, in
<module>
import matplotlib.pyplot as plt
ModuleNotFoundError: No module named 'matplotlib'

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)