| Sysop: | Amessyroom |
|---|---|
| Location: | Fayetteville, NC |
| Users: | 28 |
| Nodes: | 6 (1 / 5) |
| Uptime: | 46:05:32 |
| Calls: | 422 |
| Calls today: | 1 |
| Files: | 1,024 |
| Messages: | 90,336 |
Dear all,
This concept has turned out to be more controversial then I anticipated,
so I think it warrants broader discussion here on the mailing list. Let
me first explain some background information so everyone knows what we
are talking about, and then afterwards concretely discuss my idea for implementing it in Gentoo.
The Dynamic Kernel Module System (DKMS, sys-kernel/dkms) is a
distro-agnostic framework for managing out-of-tree kernel modules.
You'll find it in most Linux distributions, though users usually do not
have to interact with it directly. In Gentoo this is packaged as sys-kernel/dkms, though unlike most other package managers Portage
currently has no support for it. As such the usefulness of this package
is limited to some edge cases.
The operating principle of DKMS is simple. The package manager installs
the sources for one or more kernel modules to /usr/src/$PACKAGE_NAME-$PACKAGE_VERSION/, and installs a configuration
file (dkms.conf) to the root of this source directory. The configuration
file describes which kernel modules are contained in this source
directory, how to build them, how to install them, as well as some metadata.
Then the dkms command line utility can be used to register (dkms add ...
) these kernel sources in the local dkms tree (/var/lib/dkms). After
this the contained kernel modules can be built (dkms build ...) and
installed (dkms install ...). Compared to the alternative approach where
the package manager installs (pre-)built kernel modules directly to /lib/modules/KV_FULL, the DKMS approach comes with several advantages:
- Via init system service, kernel modules can be automatically built and installed when the system boots into some new kernel.
- Via kernel-install hook, kernel modules can be automatically built and installed for multiple kernels in one go (kernel-install add-all).
- The above makes it possible to propagate updates of a kernel module to
an arbitrary number of kernels (and accompanying initramfs) without user intervention.
End result is a "just works" experience when it comes to out-of-tree
kernel modules. This is something that in my opinion is currently sorely lacking in Gentoo. Instead we currently have to fiddle with symlinks
(eselect kernel ...) or environment variables (KERNEL_DIR), and then iteratively invoke the package manager multiple times. This is slow and difficult to automate. And yes we could in theory create some solution
using multibuild.eclass and a KERNEL_TARGETS (similar to the
PYTHON_TARGETS). But this I think will be messy since the kernel has significantly more slots to target then e.g. python, which means these targets would have to be updated on every minor kernel version bump. Not
to mention that we would also have to support a wide variety of kernel (source) packages. And then still we would not have the same level of
"just works" experience compared to dynamically compiling missing
modules while the system is booting, this we cannot reasonably achieve
using the package manager.
For this reason I propose to not invent our own new solution to improve
the situation, but instead adopt the proven industry standard solution:
DKMS.
I've opened a Pull Request (https://github.com/gentoo/gentoo/pull/40704)
with a working implementation of a dkms.eclass. The idea of the eclass
is relatively simple. We inherit linux-mod-r1.eclass so we don't have to duplicate most of the setup and preparation logic. Then in the compile
phase we create a DKMS configuration file using the linux-mod-r1 modlist
and modargs defined by the ebuild, alternatively an existing upstream
DKMS configuration file may be used. The configuration file will contain
the users compiler etc. preference, *FLAGS and MAKEOPTS (i.e.
linux-mod-r1's MODULES_MAKEARGS), this ensures that we do not lose any
user customization in the process. Then in the install phase we find all dkms.conf files and install those and the associated sources to
/usr/src/. Finally in the pkg_postinst and pkg_prerm phase we call dkms
to add/build/install/uninstall/unbuild/remove the contained kernel
modules. For convenience we also add a pkg_config function to call dkms
again to rebuild and reinstall the kernel modules.
Note that all of this is toggled by a new "dkms" USE flag, if this flag
is disabled then modules are built and installed as usual via linux-mod-r1.eclass. This is intended to add a new option to Gentoo, not
to replace linux-mod-r1.eclass. A downside of this is that the existing USE=modules-compress and USE=modules-sign are ineffective when USE=dkms
is enabled. Though DKMS does support compressing and signing kernel
modules, which is configured via /etc/dkms/framework.conf. The default
is to compress kernel modules when the kernel modules currently in /lib/modules/KV_FULL are also compressed, and to sign kernel modules
whenever it is able to do so (note a fix for DKMS to automatically
respect our MODULES_SIGN_KEY and MODULES_SIGN_CERT has already been
merged upstream).
Now I already know Sam and Ionen are not enthusiastic about this idea,
they are concerned about having to support two paths to achieve similar
ends and are not (yet) convinced of the added value of the DKMS pathway.
But I'd love to hear more opinions. If really everyone hates this then
I'll drop it, but currently I am still convinced this is useful and
worth the effort. Because DKMS support adds a "just works" level of experience that we cannot reasonably achieve with the package manager
and because it is the industry standard solution and thus is often
tested and supported by kernel modules upstream.
Please find the proposed eclass, as well as implementation in a bunch of ebuilds, in my PR: https://github.com/gentoo/gentoo/pull/40704
I could send it to the mailinglist, but it's 41 commits large and I
don't want to spam everyone.
Best regards,
Nowa
Dear all,
This concept has turned out to be more controversial then I anticipated,
so I think it warrants broader discussion here on the mailing list. Let
me first explain some background information so everyone knows what we
are talking about, and then afterwards concretely discuss my idea for implementing it in Gentoo.
The Dynamic Kernel Module System (DKMS, sys-kernel/dkms) is a distro- agnostic framework for managing out-of-tree kernel modules. You'll find
it in most Linux distributions, though users usually do not have to
interact with it directly. In Gentoo this is packaged as sys-kernel/
dkms, though unlike most other package managers Portage currently has no support for it. As such the usefulness of this package is limited to
some edge cases.
The operating principle of DKMS is simple. The package manager installs
the sources for one or more kernel modules to /usr/src/$PACKAGE_NAME- $PACKAGE_VERSION/, and installs a configuration file (dkms.conf) to the
root of this source directory. The configuration file describes which
kernel modules are contained in this source directory, how to build
them, how to install them, as well as some metadata.
Then the dkms command line utility can be used to register (dkms
add ... ) these kernel sources in the local dkms tree (/var/lib/dkms).
After this the contained kernel modules can be built (dkms build ...)
and installed (dkms install ...). Compared to the alternative approach
where the package manager installs (pre-)built kernel modules directly
to /lib/modules/KV_FULL, the DKMS approach comes with several advantages:
- Via init system service, kernel modules can be automatically built and installed when the system boots into some new kernel.
- Via kernel-install hook, kernel modules can be automatically built and installed for multiple kernels in one go (kernel-install add-all).
- The above makes it possible to propagate updates of a kernel module to
an arbitrary number of kernels (and accompanying initramfs) without user intervention.
End result is a "just works" experience when it comes to out-of-tree
kernel modules. This is something that in my opinion is currently sorely lacking in Gentoo. Instead we currently have to fiddle with symlinks
(eselect kernel ...) or environment variables (KERNEL_DIR), and then iteratively invoke the package manager multiple times. This is slow and difficult to automate. And yes we could in theory create some solution
using multibuild.eclass and a KERNEL_TARGETS (similar to the
PYTHON_TARGETS). But this I think will be messy since the kernel has significantly more slots to target then e.g. python, which means these targets would have to be updated on every minor kernel version bump. Not
to mention that we would also have to support a wide variety of kernel (source) packages. And then still we would not have the same level of
"just works" experience compared to dynamically compiling missing
modules while the system is booting, this we cannot reasonably achieve
using the package manager.
For this reason I propose to not invent our own new solution to improve
the situation, but instead adopt the proven industry standard solution:
DKMS.
I've opened a Pull Request (https://github.com/gentoo/gentoo/pull/40704)
Β with a working implementation of a dkms.eclass. The idea of the eclass
is relatively simple. We inherit linux-mod-r1.eclass so we don't have to duplicate most of the setup and preparation logic. Then in the compile
phase we create a DKMS configuration file using the linux-mod-r1 modlist
and modargs defined by the ebuild, alternatively an existing upstream
DKMS configuration file may be used. The configuration file will contain
the users compiler etc. preference, *FLAGS and MAKEOPTS (i.e. linux-mod-
r1's MODULES_MAKEARGS), this ensures that we do not lose any user customization in the process. Then in the install phase we find all
dkms.conf files and install those and the associated sources to /usr/
src/. Finally in the pkg_postinst and pkg_prerm phase we call dkms to add/build/install/uninstall/unbuild/remove the contained kernel modules.
For convenience we also add a pkg_config function to call dkms again to rebuild and reinstall the kernel modules.
Note that all of this is toggled by a new "dkms" USE flag, if this flag
is disabled then modules are built and installed as usual via linux-mod- r1.eclass. This is intended to add a new option to Gentoo, not to
replace linux-mod-r1.eclass. A downside of this is that the existing USE=modules-compress and USE=modules-sign are ineffective when USE=dkms
is enabled. Though DKMS does support compressing and signing kernel
modules, which is configured via /etc/dkms/framework.conf. The default
is to compress kernel modules when the kernel modules currently in /lib/ modules/KV_FULL are also compressed, and to sign kernel modules whenever
it is able to do so (note a fix for DKMS to automatically respect our MODULES_SIGN_KEY and MODULES_SIGN_CERT has already been merged upstream).
Now I already know Sam and Ionen are not enthusiastic about this idea,
they are concerned about having to support two paths to achieve similar
ends and are not (yet) convinced of the added value of the DKMS pathway.
But I'd love to hear more opinions. If really everyone hates this then
I'll drop it, but currently I am still convinced this is useful and
worth the effort. Because DKMS support adds a "just works" level of experience that we cannot reasonably achieve with the package manager
and because it is the industry standard solution and thus is often
tested and supported by kernel modules upstream.
Please find the proposed eclass, as well as implementation in a bunch of ebuilds, in my PR: https://github.com/gentoo/gentoo/pull/40704
I could send it to the mailinglist, but it's 41 commits large and I
don't want to spam everyone.
Best regards,
Nowa