Package: debconf
Version: 1.5.88
Severity: normal
X-Debbugs-Cc: deity@lists.debian.org, debian-cross@lists.debian.org

Hi,

since the upload of debconf 1.5.88 a few days ago, building chrootless Debian chroot tarballs is broken if not using fakeroot. Specifically, since that upload, /var/cache/debconf/ and /var/cache/debconf/tmp.ci/ outside the chroot will get modified or created, respectively. I tried to come up with a patch but am left with more questions than answers and am thus reaching out to the debconf maintainers as well as the readers of the deity list. There is a bit to unpack, lets start with a patch that works around the issue in our CI [1]:

--- a/dpkg-preconfigure
+++ b/dpkg-preconfigure
@@ -140,7 +140,10 @@ elsif (! @debs) {
}

my $apt_extracttemplates;
-if (-x '/usr/lib/apt/apt-extracttemplates') {
+if (($ENV{USER} // '') ne 'user') {
+ warn gettext("delaying package configuration, since apt-extracttemplates does not support DPKG_ROOT");
+ exit;
+} elsif (-x '/usr/lib/apt/apt-extracttemplates') {
$apt_extracttemplates = '/usr/lib/apt/apt-extracttemplates';
} elsif (Debconf::Path::find('apt-extracttemplates')) {
$apt_extracttemplates = 'apt-extracttemplates';
EOF

Before debconf 1.5.88, dpkg-preconfigure did not find apt-extracttemplates (because apt-utils was not installed) and thus bailed out early. But since apt 2.9.11, apt-extracttemplates is part of the "apt" package and thus always available. Thus, dpkg-preconfigure does not bail out and ends up touching files that it should not. Notably, this only happens when running mmdebstrap in chrootless mode *without* fakeroot around it. With fakeroot we are lacking
pr

On Thu, Dec 26, 2024 at 12:41:42PM +0100, Johannes Schauer Marin Rodrigues wrote:

since the upload of debconf 1.5.88 a few days ago, building chrootless Debian chroot tarballs is broken if not using fakeroot. Specifically, since that upload, /var/cache/debconf/ and /var/cache/debconf/tmp.ci/ outside the chroot will get modified or created, respectively. I tried to come up with a patch but
am left with more questions than answers and am thus reaching out to the debconf maintainers as well as the readers of the deity list. There is a bit to
unpack, lets start with a patch that works around the issue in our CI [1]:

Eww. What a mess.

But dpkg-preconfigure is called by /etc/apt/apt.conf.d/70debconf via DPkg::Pre-Install-Pkgs and thus does *not* have the DPKG_ROOT variable set as it is not executed by dpkg but by apt itself. We are thus not able to figure out that this is supposed to be a chroot installation.

I'm am looking for ideas of how to fix this.

Since chroot installation worked well before apt-extracttemplates moved to "apt" I'm tending to look for a solution which just exits dpkg-preconfigure early for chrootless installations. But on what condition should this happen?

This is made worse by the fact, that apt is not being told that this is a chrootless installation -- dpkg is being told via apt options.

Should apt gain support for being told that it's doing a chrootless installation and then pass the right options or environment variables to the DPkg::Pre-Install-Pkgs scripts it calls?

Should the caller of apt set an environment variable or touch a special file to
indicate to dpkg-preconfigure that it should please exit early?

dpkg-preconfigure arguably shouldn't need to bail out; since debconf
supports DPKG_ROOT, it would be sufficient if DPKG_ROOT were set when dpkg-preconfigure is called, since that would mean it would use a
debconf database in the right place. (Well, nearly. dpkg-preconfigure currently hardcodes /var/cache/debconf/tmp.ci, but that's easily fixed.)
In general it seems better to me if chrootless installations take more
similar code paths to normal ones.

How viable/evil would it be for something outside dpkg to set DPKG_ROOT? Perhaps the caller of apt could just set DPKG_ROOT, since it knows the
intended installation path and that the installation is going to be
chrootless? Then hopefully all we need is something like:

diff --git a/dpkg-preconfigure b/dpkg-preconfigure
index 6239603a..a5f9d37d 100755
--- a/dpkg-preconfigure
+++ b/dpkg-preconfigure
@@ -156,7 +156,8 @@ if (! $have_tty && $frontend->need_tty) {
exit 0;
}

-my $tempdir='/var/cache/debconf/tmp.ci';
+my $root=$ENV{DPKG_ROOT} // '';
+my $tempdir="$root/var/cache/debconf/tmp.ci";
remove_tree($tempdir, { safe => 1, keep_root => 1 });
make_path($tempdir);


--
Colin Watson (he/him) [cjwatson@debian.org]

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

Hi,

On Thu, Dec 26, 2024 at 01:51:15PM +0000, Colin Watson wrote:

On Thu, Dec 26, 2024 at 12:41:42PM +0100, Johannes Schauer Marin Rodrigues wrote:

since the upload of debconf 1.5.88 a few days ago, building chrootless Debian
chroot tarballs is broken if not using fakeroot. Specifically, since that upload, /var/cache/debconf/ and /var/cache/debconf/tmp.ci/ outside the chroot
will get modified or created, respectively. I tried to come up with a patch but
am left with more questions than answers and am thus reaching out to the debconf maintainers as well as the readers of the deity list. There is a bit to
unpack, lets start with a patch that works around the issue in our CI [1]:

Eww. What a mess.

Thanks for noticing!

But dpkg-preconfigure is called by /etc/apt/apt.conf.d/70debconf via DPkg::Pre-Install-Pkgs and thus does *not* have the DPKG_ROOT variable set as
it is not executed by dpkg but by apt itself. We are thus not able to figure
out that this is supposed to be a chroot installation.

This is a corner of Debian I'm not experienced in, so I am only catching
up now and writing down my limited understanding.

Contrary to its name, dpkg-preconfigure is not part of dpkg, but
debconf. It is run by apt via DPkg::Pre-Install-Pkgs before any package unpacking and extracts and runs the *.config scripts contained in
control.tar. The benefit here being that the installation does not
scatter debconf prompts to various maintainer scripts and asking (most
of) them ahead of installation in one batch.

(Please now point out how wrong I am.)

Now mmdebstrap goes to great lengths to avoid debconf questions by
setting DEBIAN_FRONTEND=noninteractive and
DEBCONF_NONINTERACTIVE_SEEN=true. It really does not want to answer
those questions interactively. If you really want to answer them, man mmdebstrap suggests running debconf-set-selections from an
--essential-hook. So really, what is the benefit of running any of these
when run from mmdebstrap? We want the answers to be deterministic in the interest of reproducibility. Unless I am mistaken, we also want the
.config scripts to not have any effect when being non-interactive as you
cannot rely on them being run (e.g. when installing a package with dpkg directly).

I'm am looking for ideas of how to fix this.

I question why we run dpkg-preconfigure from mmdebstrap. In what
situation can it have any beneficial effect?

(Not running it also speeds stuff up.)

dpkg-preconfigure arguably shouldn't need to bail out; since debconf
supports DPKG_ROOT, it would be sufficient if DPKG_ROOT were set when dpkg-preconfigure is called, since that would mean it would use a
debconf database in the right place. (Well, nearly. dpkg-preconfigure currently hardcodes /var/cache/debconf/tmp.ci, but that's easily fixed.)
In general it seems better to me if chrootless installations take more similar code paths to normal ones.

debconf was required to support DPKG_ROOT, because it is run from
maintainer scripts. I see way less reason to require dpkg-preconfigure
to support this. The DPKG_ROOT variable is meant to be exported by dpkg
for maintainer scripts, but dpkg-preconfigure is not run from a
maintainer script, so the attached semantics do not apply. If we want it
to support on a directory, I argue that it should gain a --root option
instead. I'm not convinced that we should change it at all given the
above though.

How viable/evil would it be for something outside dpkg to set DPKG_ROOT?

I think it would be evil. Presently, we kinda imply that if DPKG_ROOT is
set, we're inside a maintainer script and that property gets violated
here. That's certainly debatable, but it also is a significant departure
from the present design.

Perhaps the caller of apt could just set DPKG_ROOT, since it knows the intended installation path and that the installation is going to be chrootless? Then hopefully all we need is something like:

diff --git a/dpkg-preconfigure b/dpkg-preconfigure
index 6239603a..a5f9d37d 100755
--- a/dpkg-preconfigure
+++ b/dpkg-preconfigure
@@ -156,7 +156,8 @@ if (! $have_tty && $frontend->need_tty) {
exit 0;
}

-my $tempdir='/var/cache/debconf/tmp.ci';
+my $root=$ENV{DPKG_ROOT} // '';
+my $tempdir="$root/var/cache/debconf/tmp.ci";
remove_tree($tempdir, { safe => 1, keep_root => 1 });
make_path($tempdir);

Given the above I'd rank things like this:

1. Do not call dpkg-preconfigure from mmdebstrap.
2. Add a --root option to dpkg-preconfigure.
3. Your patch.

This is assuming that my limited understanding is lacking errors. Now
please tell me where I'm wrong.

Helmut

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)

Am Thu, Dec 26, 2024 at 12:41:42PM +0100, schrieb Johannes Schauer Marin Rodrigues:

Should apt gain support for being told that it's doing a chrootless installation and then pass the right options or environment variables to the DPkg::Pre-Install-Pkgs scripts it calls?

I think it should… after all, you kinda already do tell apt to go look elsewhere with an assortment of options for sources.list, config,
dpkg.status, lists/, archives/, … so adding yet another option can't
hurt that much, can it?

That said, semantics might be tricky…

I am assuming you use the "chroot" apt config, but the various hooks
tend to use absolute paths ~ in other words paths on the host.
Many of those hook interfaces are invoked via $SHELL and I am certainly
not looking forward to implementing a $SHELL parser so we could mangle
the absolute paths into the "chroot" ……… sometimes?

Random example from my host:
| $ apt-config dump --no-empty | grep -i '::\(pre\|post\)-'
| APT::Update::Post-Invoke-Success:: "if /usr/bin/test -w /var/cache/swcatalog -a -e /usr/bin/appstreamcli; then appstreamcli refresh --source=os > /dev/null || true; fi";
| DPkg::Pre-Install-Pkgs:: "/usr/bin/apt-listchanges --apt || test $? -lt 10"; | DPkg::Pre-Install-Pkgs:: "/usr/sbin/dpkg-preconfigure --apt || true";
| DPkg::post-invoke:: "[ -x /usr/lib/libdvd-pkg/b-i_libdvdcss.sh ] && /usr/lib/libdvd-pkg/b-i_libdvdcss.sh || true";

So, in a sense, I think you are just "lucky" that you ran into this
problem class with dpkg-preconfigure/apt-extracttemplates as depending
on what you install in your chrootless chroots other fun stuff might
happen.
(And that "ignoring" the preconfigure has no consequences for you as the
config intermixed with the installation is silenced anyhow)

How is this "solved" for maintainer scripts? I mean the ones which don't
use just sed and grep which you can easily provide and use from the host
but lets say texlive or the libdvdcss from above?


The most immediate option here might just be to have a
| $ cat /path/to/chroot/etc/apt/apt.conf.d/99z-disable-hooks.conf
| #clear APT::Update::Auth-Failure;
| #clear APT::Update::Pre-Invoke;
| #clear APT::Update::Post-Invoke;
| #clear APT::Update::Post-Invoke-Stats;
| #clear APT::Update::Post-Invoke-Success;
| #clear APT::Install::Pre-Invoke;
| #clear APT::Install::Post-Invoke-Success;
| #clear DPkg::Pre-Install-Pkgs;
| #clear DPkg::Pre-Invoke;
| #clear DPkg::Post-Invoke;

Those are at least the ones I found with `git grep RunScripts`,
I kinda suspect more to exist… who knows, right? 😉

It /might/ makes sense to deprecate all the various hooks and do
something slightly more controlled aka declarative at least on the
"what on earth are we supposed to run" part of the interface.
I suppose Julian is screaming JSON hook or varlink at me now,
but that might be yet another beast to deal with.


Another more specific option might be to "just" react in
dpkg-preconfigure to DEBIAN_FRONTEND=noninteractive and exit as
I suppose the point of preconfiguring is to front load interactivity,
but if there will be none no extractions have to happen.

(obligatory mention of it being sad that this functionality is in apt to
begin with as this is not something apt should be doing. It doesn't
usually poke into deb files…)

A very fine time sink you stumbled over.
Package management is clearly a hard problem, not just for Santa.


Best regards

David Kalnischkies

-----BEGIN PGP SIGNATURE-----

iQIzBAABCgAdFiEE5sn+Q4uCja/tn0GrMRvlz3HQeIMFAmdu6GQACgkQMRvlz3HQ eINZuw/7B81rJMvgZbly6yaPkEAZjF7RyNfORNZzb1u1DLSG5OogrKMs5UyL25WL 2r6gw9jpmApf8cKROdVJiIZr/kkhseLGlSTKncqAy6gPGFLp66NUEu6/QImwh6nT Qw748Lic/fwI6aM0yGS4otkdYFzM18MR6vHcM4nqeYVzKtrez572KQuEaauDFhYt J32PeSELXWyy16ld+9JHCPZ1K/3HoChGxk0jWWS8+RI2pys7s2o4uPkFQ38x2cjS hE23ZZMXkD6CrIbGoTX8h//GQTKf5o6b4oi0H6VMfR/9V/13+e5Ieqp875xUgV2g cCcCFVyOeUcs75m5lOI6tHDKXiKXFZNxW76+o/tf1YzcLF/mrWV4Csjnpl1b+QsP o7ufp4y2kN6D0wkMPCW2Illk3zbBd4bUrmHaoYGb2a44TtUfYsazGunsCDC10j8u iW02GXCugM4MaHs61VR8ox7WbcZuBM/SuugT4bg/VfziqEO7nXNxTPpHFrtf6dea gj3FVzJLgz0Eyd2NkVzwTDDend9jsd6eH+02+Z+OMH65RDe4x/Hg2Qk4TeKj4nhb ccsut+CyDMWs04O9A2qoooCjfmD5jUbfsYA5rKBUK0r74ImA6KTceJwf8sbb65yS 8yWQovW7qNx21cX0oO6lu67YrTYWJmDCt7w/CnPcK6XPIjjtWMk=
=iW41
-----END PGP SIGNATURE-----

--- SoupGate-Win32 v1.05
* Origin: fsxNet Usenet Gateway (21:1/5)