• Re: [gentoo-user] Migrating existing Gentoo to binpkg

    From Victor Ivanov@21:1/5 to gentoo-user at lists.gentoo.org on Sun Nov 17 14:44:35 2024
    This is an OpenPGP/MIME signed message (RFC 4880 and 3156) DQpPbiAxNS8xMS8yMDI0IDE1OjQxLCBFbGkgU2Nod2FydHogd3JvdGU6DQo+IGdlbnRvby1zb3Vy Y2VzIGlzIG5vdCBwYXJ0IG9mIHRoZSBndWFyYW50ZWVkIHBhY2thZ2Ugc2V0IGZvciB0aGUNCj4g YmluaG9zdC4gQW55IHBhY2thZ2UqbWF5KiBlbmQgdXAgYnVpbHQgb24gdGhlIGJpbmhvc3QsIGVp dGhlciBhcyBhDQo+IGRlcGVuZGVuY3kgZm9yIGFub3RoZXIgcGFja2FnZSB0aGF0IHN0b3BzIGJl aW5nIGEgZGVwZW5kZW5jeSBiZWNhdXNlIHRoZQ0KPiBvdGhlciBwYWNrYWdlIGhhcyBjaGFuZ2Vk LCBvciBkdWUgdG8gdGhlIGZhY3QgdGhhdCB3ZSBydW4gYSBmZXcNCj4gZGlmZmVyZW50IGJ1aWxk IGNvbmZpZ3VyYXRpb25zLCBpbmNsdWRpbmcgY2hvb3NpbmcgYSBjb3VwbGUgcGFja2FnZXMgYnkN Cj4gbG90dGVyeSBlYWNoIGRheS4gVGhvc2UgbWF5IGJlIGF2YWlsYWJsZSBvbmx5IGZvciBhbiBv bGRlciB2ZXJzaW9uIGluDQo+IHRoZSA6OmdlbnRvbyB0cmVlIGlmIHRoZXkgZG9uJ3QgZ2V0IGx1 Y2t5IGEgc2Vjb25kIHRpbWUuDQoNCkluZGVlZCwgYWxzbyB3aHkgc2hvcnRseSBhZnRlciBteSBy ZXNwb25zZSBJIGFsc28gYWRkZWQgDQoiLS11c2Vwa2ctZXhjbHVkZSAnc3lzLWtlcm5lbC9nZW50 b28tc291cmNlcyB2aXJ0dWFsLyoiIHRvIA0KRU1FUkdFX0RFRkFVTFRfT1BUUywgYXMgd2FzIHJl Y29tbWVuZGVkIGJ5IG9uZSBvZiB0aGUgZ3VpZGVzIGFueXdheSA6KQ0KDQpJIGRpZCBzdXNwZWN0 IHBhY2thZ2VzIG1heSBvciBtYXkgbm90IGJlIGJ1aWx0IGZyb20gc291cmNlIGFueXdheSANCmRl cGVuZGluZyBvbiB0aGUgY3VycmVudCBzdGF0ZSBvZiBkZXBlbmRlbmNpZXMgYW5kIHdoYXQncyBh dmFpbGFibGUgb24gDQp0aGUgdXBzdHJlYW0gYmluaG9zdC4gVGhpcyBpcyBhbHJlYWR5IHRoZSBj YXNlIGZvciBhYm91dCAzMDAgb3Igc28gDQpwYWNrYWdlcyB0aGF0IEkgaGF2ZS4NCg0KSSB3YXMg bW9zdGx5IGludGVyZXN0ZWQgaW4gdGhlIGJpbmFyeSBidWlsZHMgZHVlIHRvIHVzdWFsIHN1c3Bl Y3RzIHRoYXQgDQp0YWtlIGZvcmV2ZXIgdG8gYnVpbGQuIEJ1dCBhbHNvIG91dCBvZiBnZW5lcmFs IGN1cmlvc2l0eSwgaGF2aW5nIA0KcHJldmlvdXNseSBkYWJibGVkIGEgYml0IGludG8gaGF2aW5n IGEgbG9jYWwgYmluaG9zdCBhIGZldyB5ZWFycyBiYWNrLiBJIA0KaGF2ZSB0byBhZG1pdCwgdGhl IGZsZXhpYmlsaXR5IGlzIHByZXR0eSBzd2VldC4NCg0KPiBJdCBpcyBvbmx5IHRoZSBkZWZhdWx0 IHdoZW4gdXNpbmcgLS1nZXRiaW5wa2csIG5vdCB3aGVuIHlvdSBhcmUgaW5zdGVhZA0KPiB1c2lu ZyAtLWdldGJpbnBrZ29ubHkuDQoNCkFoISBPZiBjb3Vyc2UsIHlvdSdyZSByaWdodC4gU2hvdWxk IGhhdmUgY2FsaWJyYXRlZCBteSBleWVzIGJldHRlciB3aGlsZSANCnJlYWRpbmcgdGhlIG1hbiBw YWdlLg0KDQo+IFlvdXIgQ0ZMQUdTIGFyZSBpcnJlbGV2YW50IGhlcmUuIENGTEFHUyBvbiB0aGUg YmluaG9zdCBzZXJ2ZXIgbXVzdCBiZQ0KPiBjb21wYXRpYmxlIHdpdGggeW91ciBsb2NhbCBtYWNo aW5lLCBvciB5b3Ugd2lsbCBzdWNjZXNzZnVsbHkgaW5zdGFsbA0KPiBwYWNrYWdlcyB0aGF0IHRo ZW4gYWJvcnQgd2l0aCBTSUdJTEwgd2hlbiB5b3UgdHJ5IHRvIHJ1biB0aGUgcHJvZ3JhbXMuDQo+ IA0KPiBUaGUgcmV2ZXJzZSBpcyBub3QgdHJ1ZSAtLSBDRkxBR1Mgb24geW91ciBsb2NhbCBtYWNo aW5lIGp1c3QgbmVlZCB0byBiZQ0KPiBjb21wYXRpYmxlIHdpdGgsIHdlbGwsIHlvdXIgbG9jYWwg bWFjaGluZSAod2hlcmUgdGhleSBnZXQgaW5zdGFsbGVkKSwNCj4gbm90IHRoZSBiaW5ob3N0LCB3 aGVyZSB0aGV5IGRvbid0IGdldCBpbnN0YWxsZWQuIPCfmYINCj4gLi4uDQo+IEtlZXAgeW91ciBl eGlzdGluZyBDRkxBR1MgdGhhdCB5b3Ugd2VyZSB1c2luZyBiZWZvcmUgZW5hYmxpbmcgdGhlDQo+ IGJpbmhvc3QuIEl0J3MgYSBmcmVlIG9wdGltaXphdGlvbiBmb3IgYW55IHBhY2thZ2VzIHRoYXQg eW91IGVuZGVkIHVwDQo+IGJ1aWxkaW5nIGZyb20gc291cmNlIGFueXdheS4NCg0KSnVzdCB3aGF0 IEkgc3VzcGVjdGVkIPCfmYINCg0KQWxsIHdvcmtpbmcgcGVyZmVjdGx5IGxpa2UgYSBjaGFybS4N Cg0KVGhhbmtzIGFnYWluIHRvIGV2ZXJ5b25lIGZvciB0aGUgcmVzcG9uc2VzISBUaGF0LCBhbmQg aGF2aW5nIGtlcHQgYW4gZXllIA0Kb24gc29tZSBvdGhlciBkaXNjdXNzaW9ucywgdGhlIEdlbnRv byBjb21tdW5pdHkgbmV2ZXIgc2VpemVzIHRvIA0KcG9zaXRpdmVseSBhbWF6ZS4gVW5saWtlIHRo ZSByZXNwZWN0aXZlIGNvbW11bml0aWVzIG9mIHNvbWUgb3RoZXIuLi4gDQpwb3B1bGFyIGRpc3Ry b3MsIHRoYXQgSSBzaGFsbCByZWZyYWluIGZyb20gbWVudGlvbmluZy4NCg0KQ2hlZXJzLA0KVmlj dG9yDQo=
    -----BEGIN PGP SIGNATURE-----
    Version: ProtonMail

    wnUEARYIACcFAmc6AVgJEMuXJJI+vh0uFiEE59g+WZi8eZ7/aQNqy5ckkj6+ HS4AAIhSAQDytclsuIhB8ZV4MwJ0RQ7dVD0HpQEZRuIg6SXTCsYJqAEAjkvN 8I1QgnapgE4amgMWRGq1XIMzH4dTDON1A2V0ng8=
    =QykT
    -----END PGP SIGNATURE-----

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Victor Ivanov@21:1/5 to gentoo-user at lists.gentoo.org on Tue Nov 19 12:06:33 2024
    This is an OpenPGP/MIME signed message (RFC 4880 and 3156)
    On 18/11/2024 09:21, Dr Rainer Woitok wrote:
    I also added
    "--usepkg-exclude 'sys-kernel/gentoo-sources virtual/*" to
    EMERGE_DEFAULT_OPTS, as was recommended by one of the guides anyway :)

    Hm, I can't see any logic in Gentoo developers wasting electrons and key strokes to provide at least one "sys-kernel/gentoo-sources" and a ple- thora of "virtual/*" packages on the binhost and then advising to ex- clude them from being used. Which guide recommended that?


    The in-depth "Binary package guide" [1], section 3.7 "Additional client settings".

    I guess I misspoke by referring to it as "recommendation" but the
    example given, which excludes "sys-kernel/gentoo-sources", can be seen
    as de facto recommendation as it is quite sound.

    It makes sense as the "sys-kernel/gentoo-sources" package does not have
    a "binary" package in the sense of a running kernel, which is provided
    by "sys-kernel/gentoo-kerne
    l-bin".

    Emerging gentoo-source simply shows it as an "[ebuild]" which is what I
    would expect for a package that is supposed to only contain source code. Meanwhile, gentoo-kernel-bin is correctly being pulled from the binhost
    as "[binary]".

    As for "virtual/", well they don't really contain anything so there's no
    binary to provide. I'm not sure if there's anything "[binary]" on the
    binhost for them as I would struggle to see what that would be other
    than a label for a given... version/revision?

    Regards,
    Victor

    [1] https://wiki.gentoo.org/wiki/Binary_package_guide

    -----BEGIN PGP SIGNATURE-----
    Version: ProtonMail

    wnUEARYIACcFAmc8f1UJEMuXJJI+vh0uFiEE59g+WZi8eZ7/aQNqy5ckkj6+ HS4AAEYAAPwMKjMgMue1n+URYXU1Mfr/x73MtUooNMunZ7MC3tuS0wD+Od5T LIQnfV5z1Qk9p1L0klt/3bnwpUNYgIdQye7bAwU=
    =tZYt
    -----END PGP SIGNATURE-----

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Victor Ivanov@21:1/5 to gentoo-user at lists.gentoo.org on Fri Nov 15 14:52:55 2024
    This is an OpenPGP/MIME signed message (RFC 4880 and 3156)
    Thank you, both!


    On 15/11/2024 14:05, Jacques Montier wrote:
    What if you try this :
    emerge -auvDN --getbinpkgonly --with-bdeps=y --binpkg-respect-use=y -- keep-going world

    This does indeed suggest to replace existing builds with the upstream
    binary ones. I traced this to --getbinpkgonly which seems to force the
    use of the binary packages over source builds where possible.

    There are, however, minor differences between this and my earlier
    example with --rebuilt-binaries. Using "--getbinpkgonly" suggests a few downgrades, including to sys-kernel/gentoo-sources.

    "--binpkg-respect-use=y" makes no difference as emerge(1) suggests this
    is the default.

    On 15/11/2024 14:15, Matt Jolly wrote:
    You want to match the binhost flags:

    https://wiki.gentoo.org/wiki/Gentoo_Binary_Host_Quickstart#x86-64-v3_variant

    That was my assumption as well. But doing so and running a rebuild of
    @world with --emptytree did not yield any differences in which binar
    y
    packages are used between "-march=x86-64-v3" and "-march=znver4".

    What caught my attention was a passage from the detailed guide:

    """
    * The builder and client architecture and CHOST must match.
    * The CFLAGS and CXXFLAGS variables used to build the binary packages
    must be compatible with all clients.

    ...

    Portage can not validate if these requirements match. In case of doubt,
    it is the responsibility of the system administrator to guard these
    settings.
    """

    This made me wonder if CFLAGS need to match exactly in a situation, such
    as mine, where -march=znver4, being newer, is compatible and anything
    built with has "-march=x86-64-v3" /should/ run fine.

    If my understanding is correct, then the local value for CFLAGS should
    only affect the source builds.

    If you're currently using x86-64-v4 there's no benefit to replacing the existing packages. Just let them age out and be replaced over time as
    as new versions are released.

    Indeed, that's my
    understanding too. Combined with being the same
    version, I suspect this is why emerge is not suggesting to replace the
    existing source builds with the binary builds, but is happy to do so
    with the nuclear "--emptytree" approach.


    Best Regards,
    Victor

    -----BEGIN PGP SIGNATURE-----
    Version: ProtonMail

    wnUEARYIACcFAmc3YEgJEMuXJJI+vh0uFiEE59g+WZi8eZ7/aQNqy5ckkj6+ HS4AAOjUAP0bFGAwgcsLI5prMsCT1VM29lQ+kY64NUCIUHAXu6ToQAD9Ftxc 9tAMWLeImn65I5SuiNqipQ74jZGsDvprS+k5RQ8=
    =kaUJ
    -----END PGP SIGNATURE-----

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Matt Jolly@21:1/5 to All on Fri Nov 15 15:20:01 2024
    Hi,

    Since I'm planning to use binary packages from x86-64-v3, I presume this should be changed to:

        COMMON_FLAGS="-march=x86-64-v3 -O2 -pipe"

    or, perhaps:

        COMMON_FLAGS="-march=x86-64-v3 -mtune=znver4 -O2 -pipe" ?

    You want to match the binhost flags:

    https://wiki.gentoo.org/wiki/Gentoo_Binary_Host_Quickstart#x86-64-v3_variant


    I also have a hefty $CPU_FLA
    GS_X86 (also added to $USE) from "cpuid2cpuflags" but am not worried
    about this as packages that don't fit will simply be built from source
    as usual.

    You could also match with the flags the binhost has if you want to avoid compilation altogether. See above.

    1) What would be the preferred CFLAGS configuration

    See above.

    2) To reinstall the current source based packages with their binary equivalent, "--rebuilt-binaries" sufficient or should I just go for "-- emptytree @world"?


    If you're currently using x86-64-v4 there's no benefit to replacing the existing packages. Just let them age out and be replaced over time as
    as new versions are released.

    Cheers,

    Matt

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Eli Schwartz@21:1/5 to byte.size226@simplelogin.com on Fri Nov 15 16:40:02 2024
    This is an OpenPGP/MIME signed message (RFC 4880 and 3156) --------------FUkaFOHq9QUFk0cdOAmqPjEr
    Content-Type: text/plain; charset=UTF-8
    Content-Transfer-Encoding: quoted-printable

    On 11/15/24 7:42 AM, byte.size226@simplelogin.com wrote:
    My only question that remains is whether I should change the existing
    value for CFLAGS (I presume so). Currently, I have:

        COMMON_FLAGS="-march=znver4 -O2 -pipe"

    (yes CFLAGS etc are set to use $COMMON_FLAGS).

    Since I'm planning to use binary packages from x86-64-v3, I presume this should be changed to:

        COMMON_FLAGS="-march=x86-64-v3 -O2 -pipe"

    or, perhaps:

        COMMON_FLAGS="-march=x86-64-v3 -mtune=znver4 -O2 -pipe" ?


    Your CFLAGS are irrelevant here. CFLAGS on the binhost server must be compatible with your local machine, or you will successfully install
    packages that then abort with SIGILL when you try to run the programs.

    The reverse is not true -- CFLAGS on your local machine just need to be compatible with, well, your local machine (where they get installed),
    not the binhost, where they don't get installed. :)

    It's fully possible to change your CFLAGS in make.conf whenever you
    want, run emerge to do world updates etc, and portage doesn't care and effectively doesn't know you changed it since all installed packages are "compatible" with that make.conf -- this is extremely unlike USE flags,
    which are actually treated as match specifiers and if you change USE,
    then emerge insists on rebuilding / reinstalling. Which brings us to...


    I also have a hefty $CPU_FLA
    GS_X86 (also added to $USE) from "cpuid2cpuflags" but am not worried
    about this as packages that don't fit will simply be built from source
    as usual.


    Yup. CPU_FLAGS_X86 don't need to be added to $USE because they already
    *are* USE. And USE flags will force rebuilds where necessary, or prevent
    seeing binary packages as compatible. The binhost already builds with
    those USE flags as appropriate for v3 systems, but you may have a couple
    more enabled so packages which specifically can take advantage of that,
    will be built locally.


    Anyway, in both cases running my usual:

        emerge -aqvND --update --keep-going --with-bdeps=y @world

    doesn't yield any changes to the system. I presume because, as of now, everything is up to date.

    In contrast, adding "--rebuilt-binaries" shows a lot of binary packages
    being pulled in and "--emptytree", as expected, shows a full rebuild
    with a lot of binary packages being pulled in.

    So, to the actual questions:

    1) What would be the preferred CFLAGS configuration or, since "- march=znver4" is a newer subset of x86-64-v3, can I simply keep the
    existing one?


    Keep your existing CFLAGS that you were using before enabling the
    binhost. It's a free optimization for any packages that you ended up
    building from source anyway.


    2) To reinstall the current source based packages with their binary equivalent, "--rebuilt-binaries" sufficient or should I just go for "-- emptytree @world"?


    --rebuilt-binaries will update all packages with their binary
    equivalents, if the binary on the server has a BUILD_TIME newer than the
    time you rebuilt it locally. This isn't really the same as reinstalling
    all source based packages with their binary equivalent.

    --emptytree will update all packages with their binary equivalents, and
    also update all other packages by building them from source, which isn't actually what you asked for, though it does what you asked for as a side effect. Some people encourage running weekly --emptytree anyways,
    though. Usually the idea is doing it as an experiment to see whether all packages still build and run successfully, and report bugs / commit
    fixes to gentoo.git whenever issues are encountered. It may also go well
    with running ACCEPT_KEYWORDS="**" for sys-devel/gcc and reporting
    regressions to the GCC bugtracker. ;)


    It's not actually strictly necessary to reinstall all source based
    packages with their binary equivalent. If you want to do that though,
    you can do that by looping over directories in /var/db/pkg/*/* and
    checking which ones have a file called BINPKGMD5 in them. Those were
    installed from a binpackage. Try installing all the rest with --ask,
    check if it says [binary] or [ebuild], and repeat after manually
    narrowing the list to just the ones that offer binaries.



    --
    Eli Schwartz

    --------------FUkaFOHq9QUFk0cdOAmqPjEr--

    -----BEGIN PGP SIGNATURE-----

    wnsEABYIACMWIQTnFNnmK0TPZHnXm3qEp9ErcA0vVwUCZzdqdwUDAAAAAAAKCRCEp9ErcA0vV9eH AQCNK4Mc+ZwlG7Z2bxqIA04FPZxKRruT/gB5QAFIxpzx3gD/af3ngGNALrB+ES8Kv5wHIe/A0f0Y xeNRER5VvT2kEgI=
    =IouC
    -----END PGP SIGNATURE-----

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Eli Schwartz@21:1/5 to byte.size226@simplelogin.com on Fri Nov 15 16:50:01 2024
    This is an OpenPGP/MIME signed message (RFC 4880 and 3156) --------------s8xwXb3omDEDQPvaMm8Z6Glw
    Content-Type: text/plain; charset=UTF-8
    Content-Transfer-Encoding: quoted-printable

    On 11/15/24 9:52 AM, byte.size226@simplelogin.com wrote:
    Thank you, both!


    On 15/11/2024 14:05, Jacques Montier wrote:
    What if you try this :
    emerge -auvDN --getbinpkgonly --with-bdeps=y --binpkg-respect-use=y --
    keep-going world

    This does indeed suggest to replace existing builds with the upstream
    binary ones. I traced this to --getbinpkgonly which seems to force the
    use of the binary packages over source builds where possible.

    There are, however, minor differences between this and my earlier
    example with --rebuilt-binaries. Using "--getbinpkgonly" suggests a few downgrades, including to sys-kernel/gentoo-sources.


    gentoo-sources is not part of the guaranteed package set for the
    binhost. Any package *may* end up built on the binhost, either as a
    dependency for another package that stops being a dependency because the
    other package has changed, or due to the fact that we run a few
    different build configurations, including choosing a couple packages by
    lottery each day. Those may be available only for an older version in
    the ::gentoo tree if they don't get lucky a second time.


    "--binpkg-respect-use=y" makes no difference as emerge(1) suggests this
    is the default.


    It is only the default when using --getbinpkg, not when you are instead
    using --getbinpkgonly.


    --
    Eli Schwartz

    --------------s8xwXb3omDEDQPvaMm8Z6Glw--

    -----BEGIN PGP SIGNATURE-----

    wnsEABYIACMWIQTnFNnmK0TPZHnXm3qEp9ErcA0vVwUCZzdrpQUDAAAAAAAKCRCEp9ErcA0vVxNx APkBDivGlqWjbkVfDyOjaUyYuwWXejifCw7G1X4LfZrhiAD/Y/d3vTS32gx6B5Nwtga/LbRGUKuh dqmrohAiUkSvAA4=
    =cdFq
    -----END PGP SIGNATURE-----

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)