On 02/05/2025 22:15, Alan Mackenzie wrote:

I think my first summary of my problem was fair: The python update scheme
is fragile. In particular if the emerge is interrupted for any reason,
it cannot be restarted, and cannot be reverted to a workable state,
without a great deal of time and expert knowledge.

Just don't blame that on gentoo. Python is notorious for being a
"dangerous" language. Look at the mess with the Python 2 -> 3 upgrade.

You said a project using Rust should assume that is a serious risk
factor. I suspect any project using Python should assume that is an even
bigger risk factor ... the fact that emerge is written in Python should
be seen as very risky!

Cheers,
Wol

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

On 5/2/25 2:15 PM, Alan Mackenzie wrote:

If that's the way they see things, it would seem that having a project dependent on rust should figure high on that project's risk list.

This is ... a non-actionable take, regardless of if I agree with it or not.

Do you expect Gentoo to replace Firefox? pycryptography? a heavily used
SVG library? kernel drivers for interesting (really, mac) hardware?
There's only so much that the downstream distribution has control over,
and I appreciate that Gentoo, generally, has accepted that the state of
the world is that rust can't easily be avoided.

There are a bunch of volunteers keeping the lights on, navigating some
of these pitfalls -- even though they themselves might actually agree
with you -- because they represent the reality of the OSS landscape.

...but complaining about it here is unlikely to actually change
anything, and only dishearten the folks who are trying to keep it all
working. It's nicer to focus our mental energy and time on changes that
are achievable as a project -- or specific, fixable technical issues,
like the wedged install this started with (and the reply with direct information on how to fix it).


My $.02,

JayF

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

Hello, Eli.

On Fri, May 02, 2025 at 16:07:17 -0400, Eli Schwartz wrote:

On 5/2/25 3:07 PM, Alan Mackenzie wrote:

On Fri, May 02, 2025 at 13:47:02 -0400, Eli Schwartz wrote:

On 5/2/25 1:07 PM, Alan Mackenzie wrote:

I've just been trying the update for python 3.13. It went well on my
new machine (well, after unmerging app-portage/unsymlink-lib, which was >>> debris from some 2019 update).

On my old machine, however, there was a seg fault while merging rust.
This is a known problem in certain first generation Ryzen processors.

However, this left the build wedged: emerge says that there's a circular >>> dependency from rust to itself. More precisely, the error message looks >>> like:

This is clearly not a python update, then.

Well, it was. ;-)

Your saying that it was, doesn't make it the case. So you can repeat a
false claim if you want -- or explain why you think it is true.

It was an update. It was for the purpose of replacing python 3.12 with
python 3.13. Thus it was a python update.

This would appear to be a bug in the rust ebuild - it has set itself as a >>> builtime dependency before it had been built - or something like that.

How do I best recover from this?

It is not a bug in the rust ebuild -- the rust ebuild has always,
always, always depended on itself. In the past, the rust ebuild silently >> downloaded a temporary copy of dev-lang/rust-bin to work around this.

I disagree. By attempting this emerging of rust, and failing, some persistent information indicating this dependency has been created. This persistent information, wherever it is, should not have been created, or
it should have been removed on the emerge failure.

Since you do not know what this "persistent information" is, I assert in response that it doesn't exist, therefore it hasn't been created.

Well, thanks for being so generous and understanding.

At the start of this python update, I was able to start emerge, which
carried on until the seg fault part way through emerging rust.

On attempting to restart the python update, portage reported a "circular dependency" in rust and failed to build any packages.

Some persistent information thus clearly differed between these two
attempts. I don't know what was changed, and I shouldn't have to find
out, being a user of emerge, not a maintainer.

If you are nonetheless certain that it exists, you are welcome to
explain what "it" is and where I can find it.

See above.

This was changed in November 2024.

Hmmm... OK.

The rust compiler is written in the rust programming language, using
bleeding-edge features of the most recent version of rust. They are
aware that this is a bootstrap problem and don't consider it a problem,
because rust is officially installed by downloading prebuilt binaries
into $HOME.

This sounds bad, but how is it different from C and C++ in GCC?

Because:
- GCC 15 can be compiled with GCC 5.4
- GCC 11 - 14 can be compiled with GCC 4.8.3
- GCC 4.8 - 10 can be compiled with a C++98 compiler
- GCC 4.7 can be compiled with a C compiler

The bootstrappable.org project uses TinyCC to build GCC 2.95.0, which
then builds GCC 4.7.4, which gives you a very nearly C++11 compiler.

It takes you 4 hops from TinyCC to GCC 15, which is a quarter of a
decade of technology.

?Quarter of a century, perhaps?

OK, thanks. That was more or less what I suspected. GCC was at the
heart of the GNU project right from the beginning, and having people able
to build it easily was always important. I'm glad that's still the case.

For rust, the best bootstrap we have is unofficial -- mrustc -- and
relies on first bootstrapping GCC with 4 hops, then compiling 13
versions of Rust released between December 2023 and today. And that's
our hack to avoid building 80 versions released over the course of 10
years (rust 1.0 was released on May 15, 2015).

Somehow that feels like it's pushing the definition of free software in a direction it was never intended to go.

Compiling rust officially requires building the pre 1.0 version of rust, >> itself written in ocaml, and then building *every* version of rust since >> then, one by one, in turn, without skipping a single one. That is over
80 different versions of rust, you need to compile all of them in order
to get to the current one.

No thanks! There's only so much time left before the Sun becomes a red giant. But this attitude must limit the portability of rust to different processors, such as are found in embedded systems.

On amd64, you can use the unofficial dev-lang/mrustc (third-party
reimplementation of rust written in C++) which allows you to skip all
the way to rust 1.74, and from there it is "only" 13 or so versions to
build.

Doesn't sound much like a solution.

You're entirely correct. The upstream rust developers imposed this constraint.

If that's the way they see things, it would seem that having a project dependent on rust should figure high on that project's risk list.

Or, you may use rust-bin. Alternatively, you may use:

```
emerge --getbinpkg dev-lang/rust
```

I think I'll be going for rust-bin. This is quite new in Gentoo (in the last few years, at any rate), but I couldn't find the news item about it
I vaguely remember reading.

Hopefully there's nothing tricky about moving from package foo to
foo-bin.

And then I can, hopefully, finish the update of the python stuff. Though
I would classify it as a design bug that all python programs need to be rebuilt just because there's a minor version update in python itself. We don't need to do this when a new version of GCC comes out.

It's not a minor version update in python itself. Python doesn't really
use semver.

Use what?

GCC, and in general, C/C++, tend to take this sort of compatibility a
lot more serious than other languages. e.g. rust and golang simply
statically compile, and that's why @rust-rebuild and @golang-rebuild
exist. Ruby has to be rebuilt on version updates just like python. Zig software only ever supports a single version of the compiler, and has to
be rewritten in tandem with the compiler upgrade.

Thank goodness for C!

Anyhow, my old machine is now in an unusable state. When I repeat the
emerge incantation from the news item, emerge fails with obscure messages
about conflicting package versions. These messages take several hours to
parse and understand, being basically dumps of the internal state of
emerge.

One way or another, I'll get that machine back in working order. But I
can't help thinking it shouldn't be this difficult, even in Gentoo.

I think my first summary of my problem was fair: The python update scheme
is fragile. In particular if the emerge is interrupted for any reason,
it cannot be restarted, and cannot be reverted to a workable state,
without a great deal of time and expert knowledge.

--
Eli Schwartz

--
Alan Mackenzie (Nuremberg, Germany).

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

This is an OpenPGP/MIME signed message (RFC 4880 and 3156) --------------mS6kssE3AXmJ7rnIA7GzGvp3
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

On 5/2/25 5:15 PM, Alan Mackenzie wrote:

This is clearly not a python update, then.

Well, it was. ;-)

Your saying that it was, doesn't make it the case. So you can repeat a
false claim if you want -- or explain why you think it is true.

It was an update. It was for the purpose of replacing python 3.12 with python 3.13. Thus it was a python update.

I am quite certain that at least once before in your life, while using
Gentoo, you have encountered a system update that had two different
updates for unrelated software, which were both ready at the same time.

https://en.wikipedia.org/wiki/Correlation_does_not_imply_causation

The bootstrappable.org project uses TinyCC to build GCC 2.95.0, which
then builds GCC 4.7.4, which gives you a very nearly C++11 compiler.

It takes you 4 hops from TinyCC to GCC 15, which is a quarter of a
decade of technology.

?Quarter of a century, perhaps?

Oops, typo. :)

OK, thanks. That was more or less what I suspected. GCC was at the
heart of the GNU project right from the beginning, and having people able
to build it easily was always important. I'm glad that's still the case.

For rust, the best bootstrap we have is unofficial -- mrustc -- and
relies on first bootstrapping GCC with 4 hops, then compiling 13
versions of Rust released between December 2023 and today. And that's
our hack to avoid building 80 versions released over the course of 10
years (rust 1.0 was released on May 15, 2015).

Somehow that feels like it's pushing the definition of free software in a direction it was never intended to go.

Well, the rust developers do say that rust itself has "no warranty of
any kind, express or implied, including but not limited to the warranty
of fitness for a particular purpose". :)

So perhaps they are leaning into that -- rust is provided without a
guarantee that it is fit for installing on your system.

It's not a minor version update in python itself. Python doesn't really
use semver.

Use what?

Semantic Versioning (semver) is the rule that says "version 1.5 and
version 2.0 of software are a major version update. Version 1.5 and
version 1.6 is just a minor update, so not a lot changed probably, and
it should still be compatible".

GCC, and in general, C/C++, tend to take this sort of compatibility a
lot more serious than other languages. e.g. rust and golang simply
statically compile, and that's why @rust-rebuild and @golang-rebuild
exist. Ruby has to be rebuilt on version updates just like python. Zig
software only ever supports a single version of the compiler, and has to
be rewritten in tandem with the compiler upgrade.

Thank goodness for C!

Anyhow, my old machine is now in an unusable state. When I repeat the
emerge incantation from the news item, emerge fails with obscure messages about conflicting package versions. These messages take several hours to parse and understand, being basically dumps of the internal state of
emerge.

One way or another, I'll get that machine back in working order. But I
can't help thinking it shouldn't be this difficult, even in Gentoo.

I think my first summary of my problem was fair: The python update scheme
is fragile. In particular if the emerge is interrupted for any reason,
it cannot be restarted, and cannot be reverted to a workable state,
without a great deal of time and expert knowledge.

Maybe it would help to know the exact command you used, and the full
output? If it's quite long, include it as an email attachment or use a pastebin.

I hope you did not use

```
emerge --resume
```

at least. Because that option is far too broken to ever be used, in my experience, besides for having no useful purpose.

Also note:
"emerge fails with obscure messages about conflicting package versions"

The news item did suggest a "safe" upgrade approach that takes longer
but is, well, safer.

It's also possible to greatly simplify large upgrades using emerge
--getbinpkg, or, it would if not for the fact that the Gentoo Binhost
has been working for two days now to finish building the binaries so it
can publish them. :(


--
Eli Schwartz

--------------mS6kssE3AXmJ7rnIA7GzGvp3--

-----BEGIN PGP SIGNATURE-----

wnsEABYIACMWIQTnFNnmK0TPZHnXm3qEp9ErcA0vVwUCaBU/ngUDAAAAAAAKCRCEp9ErcA0vVyae APwN867FPi6lU+0z5VgWjn6hoK80HoF4/2JCTzxFdW1uDwD9Em+Hm3CGLU6lSkSdBnzCpxPjwFfO M0QzAhhjJm21dQs=
=6DNb
-----END PGP SIGNATURE-----

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

Hello, Jay.

On Fri, May 02, 2025 at 14:44:06 -0700, Jay Faulkner wrote:

On 5/2/25 2:15 PM, Alan Mackenzie wrote:

If that's the way they see things, it would seem that having a project dependent on rust should figure high on that project's risk list.

This is ... a non-actionable take, regardless of if I agree with it or not.

OK.

Do you expect Gentoo to replace Firefox? pycryptography? a heavily used
SVG library? kernel drivers for interesting (really, mac) hardware?
There's only so much that the downstream distribution has control over,
and I appreciate that Gentoo, generally, has accepted that the state of
the world is that rust can't easily be avoided.

No, I don't expect any of the above.

I was thinking more as a putative project manager at the start of, say, a project to design and produce a device with embedded software. The
question would arise as to what language to program it in. Rust would
likely be a contender. But if it were up to me, I'd veto it. The
language isn't stable, doesn't have a standards body (such as ISO)
defining it, there is only one compiler, there aren't that many
competent practitioners, and so on. Add to that the difficulty of
recreating an up to date compiler if the current infrastructure
collapsed for any reason, then I think it fair to say that current users
of rust are courageous.

None of this has anything to do with Gentoo, really, it's just a
tangent.

There are a bunch of volunteers keeping the lights on, navigating some
of these pitfalls -- even though they themselves might actually agree
with you -- because they represent the reality of the OSS landscape.

...but complaining about it here is unlikely to actually change
anything, and only dishearten the folks who are trying to keep it all working. It's nicer to focus our mental energy and time on changes that
are achievable as a project -- or specific, fixable technical issues,
like the wedged install this started with (and the reply with direct information on how to fix it).

I wasn't wanting to dishearten anybody. Apologies for doing so unintentionally.

My $.02,

JayF

--
Alan Mackenzie (Nuremberg, Germany).

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

On 5/2/25 3:17 PM, Alan Mackenzie wrote:

None of this has anything to do with Gentoo, really, it's just a
tangent.

:) Yeah, fair enough for sure! I read that "a project" in the original
email as a backhanded reference to Gentoo itself. I'm friends with a
handful of folks who have been (unfairly, imo) roasted for introducing
python libraries so I am a bit trigger happy on trying to jump in front
of potential flames (so to speak).

-JayF

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

This is an OpenPGP/MIME signed message (RFC 4880 and 3156) --------------JSzxPkOtjwmolBvU0SxmIj3l
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

On 5/4/25 12:41 PM, Alan Mackenzie wrote:

Hello, Eli.

[ Added in some hours after writing the main bit below ]

I've managed to solve my problem. In the error output from emerge there
was a message about having to update some configuration files. Normally
this message just means "we advise you to get around to this sometime"
but this time it meant "your emerge will be broken until you do this".

So a quick dispatch-conf later, which inserted an atom spec into some
config file (I've forgotten which), and emerge was able to run. Just
under two hours later it was complete.

So thanks for all the help!

:) that does happen, yes.

Maybe it would help to know the exact command you used, and the full
output? If it's quite long, include it as an email attachment or use a
pastebin.

The command I used was:

# time emerge -1avUD @world

After a lot of error messages (see below), I got the following obscure message:

!!! All ebuilds that could satisfy ">=dev-python/installer-0.5.0[python_targets_pypy3_11(-)?,python_targets_python3_13t(-)?,python_targets_python3_11(-)?,python_targets_python3_12(-)?,python_targets_python3_13(-)?]" have been masked.
!!! One of the following masked packages is required to complete your request:
- dev-python/installer-0.7.0::gentoo (masked by: )

(dependency required by "dev-python/gpep517-17::gentoo" [installed]) (dependency required by "dev-python/setuptools-78.1.0::gentoo" [installed]) (dependency required by "dev-python/setuptools-scm-8.2.1::gentoo[-test]" [installed])
For more information, see the MASKED PACKAGES section in the emerge
man page or refer to the Gentoo Handbook.

On the line beginning "!!! All ebuilds ..." I do not understand what has
been written. What do the square brackets mean? What does the several
times repeated incantation "(-)?" mean?

The square brackets mean that it needs the package to be installed with specific USE flags. The (-) means that if the package does NOT support
that USE flag at all, whether set or unset, then it is equal to having
the USE flag with the "unset" value.

This is the same format used to describe a dependency inside a *.ebuild
file.

See also "man 5 ebuild" under:

- Atom USE

- Atom USE defaults

I can't parse the "(masked by: )". Is it saying that installer-0.7.0 is masked by "(dependency required by ...)". What would that even mean?
There are too few finite verbs in this message.

I believe this means, masked by the package as required not existing at
all (due to USE flags).

Where is this output documented? It is not documented in the MASKED
PACKAGES section of the emerge man page, or in the Gentoo Handbook, at
least as far as I could find.

Dunno about that, sorry. :)

Also note:
"emerge fails with obscure messages about conflicting package versions"

The news item did suggest a "safe" upgrade approach that takes longer
but is, well, safer.

Safer than what? There was no guidance in the news item as to what the dangers might be, or when the safer upgrade option is recommended.

The upgrade added a USE flag to a large number of packages, and removed
a different USE flag.

If your system is in a state where some packages depend on the removed
USE flag, you can hit issues. Simply adding a USE flag is "safer".

If some packages depend on the removed USE flag, but will be rebuilt by
portage to stop depending on that USE flag, then it's safer to add a USE
flag to all packages, get your system into a consistent state, then
remove the other USE flag from all those packages, since now you have
the new version of all those packages safely installed and the unneeded
bits aren't used by anything anymore.


From the news:

"""
Please note that the default upgrade method switches packages to the new
Python versions as they are rebuilt. This means that all interdependent packages have to support the new version for the upgrade to proceed,
and that some programs may temporarily fail to find their dependencies throughout the upgrade (although programs that are already started
are unlikely to be affected).
"""

That also includes programs which you aren't running in another window,
but rather, programs which are used to build other packages.




--
Eli Schwartz

--------------JSzxPkOtjwmolBvU0SxmIj3l--

-----BEGIN PGP SIGNATURE-----

wnsEABYIACMWIQTnFNnmK0TPZHnXm3qEp9ErcA0vVwUCaBedbQUDAAAAAAAKCRCEp9ErcA0vVy3B AP4/zt+/8xafueBOfRNpy08wCiHuezJPiiWyxgbS9/hHeAD8DDef0VATkEnn8Bv040tgBFiyH/Kh AgEQkSr3n5YJTAs=
=UFus
-----END PGP SIGNATURE-----

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

Hello, Eli.

[ Added in some hours after writing the main bit below ]

I've managed to solve my problem. In the error output from emerge there
was a message about having to update some configuration files. Normally
this message just means "we advise you to get around to this sometime"
but this time it meant "your emerge will be broken until you do this".

So a quick dispatch-conf later, which inserted an atom spec into some
config file (I've forgotten which), and emerge was able to run. Just
under two hours later it was complete.

So thanks for all the help!

On Fri, May 02, 2025 at 17:56:46 -0400, Eli Schwartz wrote:

On 5/2/25 5:15 PM, Alan Mackenzie wrote:

[ .... ]

Anyhow, my old machine is now in an unusable state. When I repeat the emerge incantation from the news item, emerge fails with obscure messages about conflicting package versions. These messages take several hours to parse and understand, being basically dumps of the internal state of emerge.

One way or another, I'll get that machine back in working order. But I can't help thinking it shouldn't be this difficult, even in Gentoo.

I think my first summary of my problem was fair: The python update scheme is fragile. In particular if the emerge is interrupted for any reason,
it cannot be restarted, and cannot be reverted to a workable state,
without a great deal of time and expert knowledge.

Maybe it would help to know the exact command you used, and the full
output? If it's quite long, include it as an email attachment or use a pastebin.

The command I used was:

# time emerge -1avUD @world

After a lot of error messages (see below), I got the following obscure
message:

!!! All ebuilds that could satisfy ">=dev-python/installer-0.5.0[python_targets_pypy3_11(-)?,python_targets_python3_13t(-)?,python_targets_python3_11(-)?,python_targets_python3_12(-)?,python_targets_python3_13(-)?]" have been masked.
!!! One of the following masked packages is required to complete your request: - dev-python/installer-0.7.0::gentoo (masked by: )

(dependency required by "dev-python/gpep517-17::gentoo" [installed]) (dependency required by "dev-python/setuptools-78.1.0::gentoo" [installed]) (dependency required by "dev-python/setuptools-scm-8.2.1::gentoo[-test]" [installed])
For more information, see the MASKED PACKAGES section in the emerge
man page or refer to the Gentoo Handbook.

On the line beginning "!!! All ebuilds ..." I do not understand what has
been written. What do the square brackets mean? What does the several
times repeated incantation "(-)?" mean?

I can't parse the "(masked by: )". Is it saying that installer-0.7.0 is
masked by "(dependency required by ...)". What would that even mean?
There are too few finite verbs in this message.

Where is this output documented? It is not documented in the MASKED
PACKAGES section of the emerge man page, or in the Gentoo Handbook, at
least as far as I could find.

It would appear that dev-python/installer-0.7.0 should satisfy the dependencies. But it doesn't. It is currently installed.

I hope you did not use

```
emerge --resume
```

at least. Because that option is far too broken to ever be used, in my experience, besides for having no useful purpose.

Yes I did indeed use emerge --resume. It usually works for me when an
emerge operation is interrupted by a sporadic seg fault.

Also note:
"emerge fails with obscure messages about conflicting package versions"

The news item did suggest a "safe" upgrade approach that takes longer
but is, well, safer.

Safer than what? There was no guidance in the news item as to what the
dangers might be, or when the safer upgrade option is recommended.

It's also possible to greatly simplify large upgrades using emerge --getbinpkg, or, it would if not for the fact that the Gentoo Binhost
has been working for two days now to finish building the binaries so it
can publish them. :(

Anyhow, here's the full output of the command

# time emerge -1avUD @world > ~acm/rust-bug.20250504b.txt 2>&1:

#########################################################################
[ Removed, due to me having solved the problem. ] #########################################################################

--
Eli Schwartz

--
Alan Mackenzie (Nuremberg, Germany).

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)