• RFC: Running Postfix chrooted in Debian

    From Michael Tokarev@21:1/5 to All on Mon Dec 16 16:00:01 2024
    Hi!

    For 25 years, Postfix the MTA in Debian has been setup to run chrooted by default (that's where most postfix internal components run chrooted in /var/spool/postfix/, to limit possible system damage after a possible compromise).

    This setup has been criticized for 25 years, because of significant pain
    it caused to users and upstream and postfix-users support. The conclusion
    by Wietse Venema, who is the author of Postfix:


    Look, I changed the Postfix default to not chroot 10+ years ago,
    because it was already impossible on LINUX.

    I expected tht package maintainers understand that non-chroot
    shall be the norm for most installs.

    On LINUX systems, chroot is for people who want to suffer pain.
    On my FreeBSD server, Postfix chroot is painles.

    So chroot is 'nice to have' but not for LINUX.


    (https://marc.info/?l=postfix-users&m=173435292323792&w=2).

    But what's interesting is that actually, running postfix chrooted
    in Linux is *exactly* as painful (or painless) as on FeeBSD. Exactly.
    The chroot setup for postfix on FreeBSD, examples/chroot-setup/FreeBSD2 -
    where it's suggested to copy resolv.conf localtime host.conf hosts
    services files to the chroot jail - is exactly which I come with on
    linux when I tried it myself.

    The problem though, arises in 2 places.

    1. Extra nsswitch modules, such as mdns, systemd-resolved (which is
    optional since resolv.conf works), and so on, which expects their
    files to be in the chroot jail (exacly like on FreeBSD with this
    same mechanism).

    2. Cyrus SASL, - for any non-trivial (PLAIN or LOGIN) methods, it
    needs the secrets database to be accessible in the chroot, and
    people on the 'net suggest really crazy things to fix this (like
    moving /etc/sasl2 userdb to /var/spool/postfix/etc/ and symlinking
    it back to /etc/sasl2).

    3. Various postfix map lookup types which require additional stuff -
    for these, there's an easy solution in postfix exists for over
    20 years, which is a proxy: map. So this is not an issue.

    My initial intention was to turn chroot in postfix immediately.
    However, I can't do anything without understanding the root issues
    first. And my discovery turned out to be quite interesting. So I
    really wonder...

    What do you think about this aspect of postfix on debian?

    A security expert in me think it's worth the effort to make it work
    as much as it is easy to do. And for *most* things, it actually is
    rather easy. For certain things (like SASL), I can write a guide to
    be served as a *good* example; or to recommend un-chrooting postfix
    when it detects sasl usage and chroot, something like that.

    Thanks,

    /mjt

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Daniel Baumann@21:1/5 to Michael Tokarev on Mon Dec 16 17:00:02 2024
    Hi,

    first - thanks a lot for working on postfix packaging, it really needs
    some love.

    On 12/16/24 15:51, Michael Tokarev wrote:
    What do you think about this aspect of postfix on debian?

    my opinion in short: I would get rid of the chrooted complexity, it's
    not worth it and introduces way more problems that it could solve/more
    extra work that it could ever save. Time and effort are better spend to
    harden postfix elsewhere where it makes actually a difference.


    smtpd daemons are a rare thing on the majority of users systems (even if
    popcon reports a whopping 12% install base for postfix), so I mainly see
    two scenarios for running postfix:

    a) single power-user system (notebook/desktop) which has a local MTA
    to send their own mail out to a proper mail server somewhere on the
    internet
    b) running a proper mail server on the internet

    I do both and would welcome non-chrooted by default for both scenarios
    in order to have a nicer, simpler and better integrated experience with
    the rest of the system with less special casing. postfix would e.g. way
    more profit from with namespace, capability and process restrictions set
    (via systemd units?).

    Regarding security concerns, imho:

    For a) not really much of a problem because one sends his/her own stuff
    out without accepting mail from the outside, so relatively small chance
    of malicious mails that could trigger some postfix bug to compromise the system.

    For b), the real abuse/danger of the system isn't a malicious mail to
    take out/over postfix, but the bazillion non-malicious-yet-unwanted
    mails aka spam.

    Regards,
    Daniel

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Russ Allbery@21:1/5 to Michael Tokarev on Mon Dec 16 18:10:02 2024
    Michael Tokarev <mjt@tls.msk.ru> writes:

    The problem though, arises in 2 places.

    1. Extra nsswitch modules, such as mdns, systemd-resolved (which is
    optional since resolv.conf works), and so on, which expects their
    files to be in the chroot jail (exacly like on FreeBSD with this
    same mechanism).

    2. Cyrus SASL, - for any non-trivial (PLAIN or LOGIN) methods, it
    needs the secrets database to be accessible in the chroot, and
    people on the 'net suggest really crazy things to fix this (like
    moving /etc/sasl2 userdb to /var/spool/postfix/etc/ and symlinking
    it back to /etc/sasl2).

    3. Various postfix map lookup types which require additional stuff -
    for these, there's an easy solution in postfix exists for over
    20 years, which is a proxy: map. So this is not an issue.

    My initial intention was to turn chroot in postfix immediately.
    However, I can't do anything without understanding the root issues
    first. And my discovery turned out to be quite interesting. So I
    really wonder...

    What do you think about this aspect of postfix on debian?

    I personally have been very happy with the default chroot of Postfix and
    have not had any trouble with it, but I also don't do any of the three
    things you list, so that makes sense. It's a nice default for simple
    systems, but I understand why it's frustrating for people who are doing
    more complex things.

    So, I wouldn't object to undoing that given upstream's stance, but maybe
    it would be good to do that in conjunction with adding more hardening to
    the default configuration with systemd? systemd-analyze security
    postfix@- shows a whole lot of things that could potentially be improved
    in hardening settings, and while a lot of those won't work becuase of the number of things Postfix needs to be able to do, a lot of them are
    probably reasonable changes to the defaults if accompanied by instructions
    for how to turn them off with an override file. There is some obvious
    stuff like ProtectSystem, PrivateDevices, or ProtectKernelTunables that
    seems quite unlikely to break anything.

    --
    Russ Allbery (rra@debian.org) <https://www.eyrie.org/~eagle/>

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From rhys@21:1/5 to All on Mon Dec 16 17:50:01 2024
    Please read all the way to the bottom before replying. It will save time.

    a) single power-user system (notebook/desktop) which has a local MTA
    to send their own mail out to a proper mail server somewhere on the
    internet
    b) running a proper mail server on the internet

    I do both and would welcome non-chrooted by default for both scenarios
    in order to have a nicer, simpler and better integrated experience with
    the rest of the system with less special casing. postfix would e.g. way
    more profit from with namespace, capability and process restrictions set
    (via systemd units?).

    I also do both of these things.

    For a) not really much of a problem because one sends his/her own stuff
    out without accepting mail from the outside, so relatively small chance
    of malicious mails that could trigger some postfix bug to compromise the system.

    Locally exploitable vulnerabilities generally rate lower than remotely exploitable vulnerabilities. This part is true.

    However, privilege escalation is still a serious issue and should not be minimized by its likelihood. The sort of attackers who exploit these vulnerabilities are not sitting around doing it by hand any more. They are highly automated nowadays and very
    capable of doing things like this:

    a) Determine that a remote system can be induced to execute code unprivileged. b) Use remote unprivileged code execution to examine the rest of the system.
    c) Use information from b) and abilities from a) to escalate privileges and take over the system.

    For b), the real abuse/danger of the system isn't a malicious mail to
    take out/over postfix, but the bazillion non-malicious-yet-unwanted
    mails aka spam.

    No, the COMMON abuse is from that. The "REAL" danger is the system takeover, as it is much more damaging to surrounding systems, harder to detect, and harder to recover from.

    No one wants to join the botnet hordes.


    ALL THAT SAID


    This is not to say that we have to have chroot and it's the only way to make things safe. Quite the opposite. Using chroot is sometimes used as an excuse to leave things UNsafe, for obvious reasons. Better to fix the underlying issues and have a less
    complex system. Less complexity means easier to support, troubleshoot, AND keep secure.

    But let us not minimize the importance of keeping our systems "un-pwned" by botnet operators. It's not about YOU. It's not about YOUR data. It's about not allowing your resources to become tools for malicious actors to use against everyone else.

    --J

    P.S. I have never run into any issues with postfix related to running in a chroot, myself, but I acknowledge that it's exactly the sort of thing that can suddenly become a real pain, so I won't be sad to see it become "not the default" any more.

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Daniel Baumann@21:1/5 to rhys on Mon Dec 16 18:50:01 2024
    On 12/16/24 17:45, rhys wrote:
    However, privilege escalation is still a serious issue and should not be minimized by its likelihood.

    I didn't, my point is that I think they are better/more effectively
    adressed with other mechanims (systemd unit hardening) than chroot.

    The "REAL" danger is the system takeover, as it is much more damaging to surrounding systems, harder to detect, and harder to recover from.

    compromised mail relays can happen (as with any other system) but they
    are relatively easy to spot, taken out of service and resetup
    automatically, ymmv.

    Using chroot is sometimes used as an excuse to leave things UNsafe, for obvious reasons. Better to fix the underlying issues and have a less complex system. Less complexity means easier to support, troubleshoot, AND keep secure.

    which is why I'm in favour of dropping chroot here.

    But let us not minimize the importance of keeping our systems "un-pwned" by botnet operators. It's not about YOU. It's not about YOUR data. It's about not allowing your resources to become tools for malicious actors to use against everyone else.

    nobody disagrees that the security of our/our users data and systems
    matter and is of importance to us. thanks nevertheless for putting an
    emphasis on it.

    Regards,
    Daniel

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Marco d'Itri@21:1/5 to Michael Tokarev on Mon Dec 16 18:50:01 2024
    On Dec 16, Michael Tokarev <mjt@tls.msk.ru> wrote:

    What do you think about this aspect of postfix on debian?
    I do not remember ever having any issues about this, and I have been
    using Postfix since before it was called Postfix. But if Wietse says
    that a chroot default is not worth it then I fully trust him.

    The security track record of Postfix is good enough that I believe that chrooting is overkill.

    --
    ciao,
    Marco

    -----BEGIN PGP SIGNATURE-----

    iHUEABYIAB0WIQQnKUXNg20437dCfobLPsM64d7XgQUCZ2BnNgAKCRDLPsM64d7X gQKfAPkBEOBgTZRcw7pnh9xYWxfAMWF8ibpFfs8/ty64e88aBgD8DsUB7pVMoOX5 ydGa2+EjNsAkt1YM6IWQl+BlrebMdgc=
    =CngV
    -----END PGP SIGNATURE-----

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Michael Tokarev@21:1/5 to Russ Allbery on Mon Dec 16 19:10:02 2024
    16.12.2024 20:08, Russ Allbery wrote:
    So, I wouldn't object to undoing that given upstream's stance, but maybe
    it would be good to do that in conjunction with adding more hardening to
    the default configuration with systemd? systemd-analyze security
    postfix@- shows a whole lot of things that could potentially be improved
    in hardening settings, and while a lot of those won't work becuase of the number of things Postfix needs to be able to do, a lot of them are
    probably reasonable changes to the defaults if accompanied by instructions for how to turn them off with an override file. There is some obvious
    stuff like ProtectSystem, PrivateDevices, or ProtectKernelTunables that
    seems quite unlikely to break anything.
    Yes, that's a very good suggestion. I'll definitely take a look at this list. It'd be nice to have some helping hand there, too.

    BTW, is there a way for a systemd unit to take/inherit (security) settings from another unit? I'm about to ship 2 service units for different configs of postfix,
    and it'd be nice to have one set of settings. Or just create 2nd one from the first at build time.

    Thanks,

    /mjt

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Michael Tokarev@21:1/5 to Marco d'Itri on Mon Dec 16 19:30:01 2024
    16.12.2024 20:45, Marco d'Itri wrote:
    On Dec 16, Michael Tokarev <mjt@tls.msk.ru> wrote:

    What do you think about this aspect of postfix on debian?
    I do not remember ever having any issues about this, and I have been
    using Postfix since before it was called Postfix. But if Wietse says
    that a chroot default is not worth it then I fully trust him.

    It turns out the reason for this is a myth, which we believed to for
    25 years - a myth that "On FreeBSD, chroot is painless, but on Linux,
    chroot never works and is only suitable for the ones who want pain".
    Actually, it looks like, chroot on linux is *exactly* the same as on
    FreeBSD, and the pain level completely depends on which features you
    use (I mentioned all 3 possible issues in my initial email). It feels
    like this is the sole source of this opinion.

    I'd love to debunk this myth somehow, just for the sake of debunking
    a good big old myth (which I've been part of all this time too).

    *And*, after this is done, happily turn chroot by default off in postfix package in debian.

    The security track record of Postfix is good enough that I believe that chrooting is overkill.
    /mjt

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Henrik Ahlgren@21:1/5 to Michael Tokarev on Mon Dec 16 22:50:02 2024
    On Mon, 2024-12-16 at 21:21 +0300, Michael Tokarev wrote:
    It turns out the reason for this is a myth, which we believed to for
    25 years - a myth that "On FreeBSD, chroot is painless, but on Linux,
    chroot never works and is only suitable for the ones who want pain". Actually, it looks like, chroot on linux is *exactly* the same as on
    FreeBSD, and the pain level completely depends on which features you
    use (I mentioned all 3 possible issues in my initial email). It feels
    like this is the sole source of this opinion.

    I have never heard about such myth. Perhaps you are referring to the
    FreeBSD jail feature, which obviously is superiour to plain chroot.
    chroot(2) is a very simple and ancient Unix mechanism from 1979 and I
    believe it is exactly the same on all Unix/Posix-style systems.

    Anyway, systemd's hardening features are so easy and effective that I
    would really like to see not only postfix, but ALL services use them as
    much as possible. Why we still have major packages like nginx shipping
    without any hardening out-of-the-box?

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Michael Tokarev@21:1/5 to Michael Tokarev on Tue Dec 17 07:10:01 2024
    16.12.2024 20:55, Michael Tokarev wrote:
    16.12.2024 20:08, Russ Allbery wrote:
    So, I wouldn't object to undoing that given upstream's stance, but maybe
    it would be good to do that in conjunction with adding more hardening to
    the default configuration with systemd?  systemd-analyze security
    postfix@- shows a whole lot of things that could potentially be improved
    in hardening settings, and while a lot of those won't work becuase of the
    number of things Postfix needs to be able to do, a lot of them are
    probably reasonable changes to the defaults if accompanied by instructions >> for how to turn them off with an override file.  There is some obvious
    stuff like ProtectSystem, PrivateDevices, or ProtectKernelTunables that
    seems quite unlikely to break anything.
    Yes, that's a very good suggestion.  I'll definitely take a look at this list.
    It'd be nice to have some helping hand there, too.

    I gave it a quick try, attacking capabilities (which is the first thing to do in such cases). The result: https://marc.info/?l=postfix-users&m=173441390930745&w=2 -
    it needs an easy small change to decouple setup from runtime.

    Dunno yet if it's a good idea to restrict it this way though: maybe someone does reboot of their system by sending special email to special address
    (which is gone now), or maybe someone run suid binary from ~/.forward,
    or something.. but I guess in each such weird case one can add the required capability locally.

    With SystemCallFilter things are much more complex though.

    Thanks,

    /mjt

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Jonathan Dowland@21:1/5 to Henrik Ahlgren on Wed Dec 18 11:40:01 2024
    On Mon Dec 16, 2024 at 9:31 PM GMT, Henrik Ahlgren wrote:
    Anyway, systemd's hardening features are so easy and effective that I
    would really like to see not only postfix, but ALL services use them as
    much as possible. Why we still have major packages like nginx shipping without any hardening out-of-the-box?

    I would also like to see this. Perhaps it's because the maintainers
    don't want to close the door to alternative init systems, by making
    their package depend on systemd features?



    --
    Please do not CC me for listmail.

    👱🏻 Jonathan Dowland
    jmtd@debian.org
    🔗 https://jmtd.net

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Henrik Ahlgren@21:1/5 to Jonathan Dowland on Wed Dec 18 12:00:02 2024
    On Wed, 2024-12-18 at 10:38 +0000, Jonathan Dowland wrote:
    I would also like to see this. Perhaps it's because the maintainers
    don't want to close the door to alternative init systems, by making
    their package depend on systemd features?

    Adding a couple of lines to the systemd unit should not add any new dependincies to the package, or affect users of other init systems in
    any way. Of course, only systemd users will benefit from systemd
    features, but I don't see why a package should only use the lowest
    common deminator features available in all init systems.

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Jonathan Dowland@21:1/5 to Henrik Ahlgren on Wed Dec 18 12:20:01 2024
    On Wed Dec 18, 2024 at 10:58 AM GMT, Henrik Ahlgren wrote:
    Adding a couple of lines to the systemd unit should not add any new dependincies to the package, or affect users of other init systems in
    any way.

    That's a very good point. In which case that's likely not why there is a
    lag in adopting these things.

    Perhaps we could have a release goal to set a certain bar in terms of hardening features for trixie+1?


    --
    Please do not CC me for listmail.

    👱🏻 Jonathan Dowland
    jmtd@debian.org
    🔗 https://jmtd.net

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Simon Richter@21:1/5 to Jonathan Dowland on Wed Dec 18 14:20:01 2024
    Hi,

    On 12/18/24 19:38, Jonathan Dowland wrote:

    I would also like to see this. Perhaps it's because the maintainers
    don't want to close the door to alternative init systems, by making
    their package depend on systemd features?

    Most likely it is because BSD does not have systemd, and quite a lot of
    Postfix installations, especially larger ones, run on BSD.

    Postfix's "master" process effectively implements an orchestrator for a microservice architecture, similar to what systemd does. This could be
    replaced by systemd by writing a generator that creates units from
    master.cf.

    Simply converting the default master.cf to unit files and shipping these
    as default would still be a massive regression, because it would
    obsolete a lot of documentation on how the Postfix system is
    orchestrated and how to integrate new services like clamav and
    spamassassin or to deliver to courier.

    All of these recipes would become "instead of this one line in
    master.cf, you need to create a service unit, socket unit and possibly
    timer unit." My expectation would be that the next sentence would be
    "Please refer to the systemd documentation on how to do that.", and
    while the systemd documentation is exhaustive, it is significantly less targeted to the operation of mail systems than the manual page for the master.cf configuration file format.

    Specifically, changes to master.cf are frequently of the form "make this service local-only, and instead expose a copy of it with a special
    instance name that makes it write to a special queue." As a text file modification, that is easy, but replicating this with systemd units is
    going to be a complexity nightmare. Do we want to use template units?
    Drop-in directories? Templated drop-ins?

    I don't see this happening without upstream support. However, upstream
    has little incentive to do so: they currently have an orchestrator that
    works, is completely controlled by them and not subject to another
    project's interface changes (rare as they may be), and is free to make
    the assumption that the services being started are vaguely related to
    mail delivery.

    If they were to ship unit files, they'd end up in the same documentation nightmare as we would if we were to start doing so, and if they create a generator and start using systemd as the controlling process, they would
    not gain anything, because neither can they drop their own orchestrator
    (so this creates additional testing surface, as their services would now
    need to implement *two* service manager protocols), nor would that
    directly provide any additional functionality.

    Simon

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Matthias Urlichs@21:1/5 to All on Wed Dec 18 16:10:01 2024
    This is an OpenPGP/MIME signed message (RFC 4880 and 3156) --------------M8Fr8SDAUJXTCyeZ9aoQ71Oi
    Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: base64

    T24gMTYuMTIuMjQgMTU6NTEsIE1pY2hhZWwgVG9rYXJldiB3cm90ZToNCj4gV2hhdCBkbyB5 b3UgdGhpbmsgYWJvdXQgdGhpcyBhc3BlY3Qgb2YgcG9zdGZpeCBvbiBkZWJpYW4/IA0KDQpX ZSBub3cgaGF2ZSBzeXN0ZW1kLCB3aGljaCBjYW4gc2FuZGJveCBmYXIgbW9yZSBhc3BlY3Rz IG9mIHBvc3RmaXggKG9yIA0KaW5kZWVkIGFueSBvdGhlciBkYWVtb24pJ3MgZXhlY3V0aW9u IHRoYW4gaXMgcG9zc2libGUgd2l0aCBhIGNocm9vdCwgDQplc3AuIGNvbnNpZGVyaW5nIHRo ZSBoYWNrcyB3ZSBuZWVkIGZvciB0aGF0IHRvIGFjdHVhbGx5IHdvcmsuDQoNClRodXMgSSBy ZWNvbW1lbmQgdG8gZHJvcCB0aGUgY2hyb290ZXJ5LiBJdCBoYXMgc2VydmVkIGl0cyBwdXJw b3NlLg0KDQotLSANCi0tIHJlZ2FyZHMNCi0tIA0KLS0gTWF0dGhpYXMgVXJsaWNocw0KDQo=


    --------------M8Fr8SDAUJXTCyeZ9aoQ71Oi--

    -----BEGIN PGP SIGNATURE-----

    wsF5BAABCAAjFiEEr9eXgvO67AILKKGfcs+OXiW0wpMFAmdi3N8FAwAAAAAACgkQcs+OXiW0wpM+ 2hAAjn7ZqVMn3LkJBuMod8wZzmY3wWd0CAh7JQLOhGW6eZxAzjmUhNp1JFogVh7qplQe6rqXcLzV /pvmFpfa95Y1HznYnwax/kLNxyAoRSYaQYGLBPKpF6BX7xi4AFNKEVfltbe4z6+ulVCXG+pDa72K Zqr7IyiNRFM4ikBHjBY11wKizvH3+sNWJP+P/TsJ7VMyrQ89ZkEK4/asDw8nxX1aGn9s2Ig3Ee9y SchvAwVs2MA9y/f6MyntcwvpwW520U/BXaAix9DXLsMST3iQKxkbrtqaGkIDtRvp4hChNNKLT4QC Ew+2cNwcJQDakqNm/mPp/ct3hWu56uB3b20buj//7OQX3grhh4R7YzoDhxHCBCKRu/+Ru22TN5eX 4I3hHJEEnHIuMZKnnzVdSOd/J1+AMgrjfbjwWgb3Mr97jerg39/x7zuod/ysQAot5MInnZwlvsne Sycnf0+Tdj9vlD8XNCDX1Ue1IZsSqUVYJH06yN6+ZmVvklBTy9qrS1+t3kFjBeNCpCGkaF5TNNEP K8Mf6SqMrzBPW1cgmmmXN15iA4I48Z45+76ftIzYZuiXh30j3iyy+Bgsp9OPcu87MzPwW2t8CEqB KW1oGNtu18Kru99KCt03Ofw1pMteCkLte9/5nY/2Be5Rq4gja9H9lVgoswbYS3KBTWs81Oe4E05L vEs=
    =8rEm
    -----END PGP SIGNATURE-----

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Gioele Barabucci@21:1/5 to Simon Richter on Wed Dec 18 18:10:01 2024
    On 18/12/24 14:16, Simon Richter wrote:
    Most likely it is because BSD does not have systemd, and quite a lot of Postfix installations, especially larger ones, run on BSD.

    [...]

    I don't see this happening without upstream support. However, upstream
    has little incentive to do so: they currently have an orchestrator that works, is completely controlled by them and not subject to another
    project's interface changes (rare as they may be), and is free to make
    the assumption that the services being started are vaguely related to
    mail delivery.

    If they were to ship unit files, they'd end up in the same documentation nightmare as we would if we were to start doing so, and if they create a generator and start using systemd as the controlling process, they would
    not gain anything, because neither can they drop their own orchestrator
    (so this creates additional testing surface, as their services would now
    need to implement *two* service manager protocols), nor would that
    directly provide any additional functionality.

    This situation is similar to that of OpenSSH, although on a reduced
    scale. Upstream OpenSSH does not support systemd (nor Linux...),
    nevertheless Debian runs sshd through systemd via service files provided
    by Debian itself. There is indeed a documentation schism, but its
    effects do not seem to be tragic.

    As a postfix user, I'd be happy to test and contribute to a (perhaps cross-distro) postfix-systemd project that replaces master.cf with a set
    of systemd unit files.

    Regards,

    --
    Gioele Barabucci

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Florian Lohoff@21:1/5 to Michael Tokarev on Wed Dec 18 19:20:02 2024
    Hola,

    On Mon, Dec 16, 2024 at 05:51:34PM +0300, Michael Tokarev wrote:
    Hi!

    For 25 years, Postfix the MTA in Debian has been setup to run chrooted by default (that's where most postfix internal components run chrooted in /var/spool/postfix/, to limit possible system damage after a possible compromise).

    This setup has been criticized for 25 years, because of significant pain
    it caused to users and upstream and postfix-users support. The conclusion
    by Wietse Venema, who is the author of Postfix:

    I am running (also professionally) postfix mail relays and i had never
    the requirement to "un-chroot" - there is always a way to map the
    required resource, and only those, into the postfix chroot.

    And i am running postfix with uucp and satellite connections, imap setups
    of various sizes, high volume mail exchanger frontends for MS Exchange
    setups etc etc.

    As an admin i like the "security first" approach here and thats the way
    debian should handle it.

    The default setting should be to be chrooted as thats what 99.5% ours
    users are using today without issues, and it its another layer in the
    cheese model for security. Even if we find a vulnerability in one of the network facing services, it makes it harder to abuse the machine and
    lets me sleep well.

    And in the end - its easy as changing a "y" to a "n" in master.cf to
    unchroot for the setups where users fail to adapt their postfix from the chroot.

    But the default should be security first.

    Flo
    --
    Florian Lohoff f@zz.de
    Any sufficiently advanced technology is indistinguishable from magic.

    -----BEGIN PGP SIGNATURE-----

    iQIzBAEBCgAdFiEEdb9o7oebX2papQ/KkN1BIMsJ8i8FAmdjDrcACgkQkN1BIMsJ 8i+YtQ/+MycH/ZzNT0P91dpcjcUn/jtoPcR4n3aFQOsRO1qVkuCND7btYL6WZlIP GS+2ITeqC7FM2+IM1vRVmqslwTp9cFRgsZFtXO9PENrhDvBL+kF0Vm3y2lx/Z0MZ TZDjmstHCwrnKta6bUWRKMgxJnkcT+oYqzfvzAEyvavueFx4Spf703oSMliWKCiz XZFVHY0tvaZvcBa9s4883WgyQ56wSUgic69SEN//tB1Zv29KO+/WgabBfw1CCgpa HmFI6R8tJo6JOL5m4R2snOJqcROA7+JBYpBB0DIPj9sp4XRhbMsh0Jd80S3ZFwdz sc2YtCKyJ5SE/btX9kevYEVKOo+NgBfKK8GrxHr1dFga/bBbXhiBA/SvJTlx5XfH NcNMuDsi1DUWbNIMZTI2ui/p467QO6+w1xKNYIpqYMNMTb25PZSXc/73zrjTZIvA J/Mb5pt9jPQ7yIHfBf4DskhnWCrzPz2ZFRpbRNvOmbo4tmrwNJpQ6isghADLL6GL jJEOVaGFKiyh9p5uC+CY9QKNoAh+Z4kcbPhCp83Ubran5qByMdA2XqvZZgIgRPuR XK5I1i+liq75ydilQ7VX0FJn0GFx8uwrj6XIH0cFnIaZZG9WjVsgrhwL4aV2BeB3 MxdOvhhgIrqjLTydFioHtQjJi1pCKGB+897f+6C4KuPG8n5Mwlg=
    =m/0Y
    -----END PGP SIGNATURE-----

    --- SoupGate-Win3
  • From Richard Lewis@21:1/5 to Jonathan Dowland on Wed Dec 18 22:50:01 2024
    "Jonathan Dowland" <jmtd@debian.org> writes:

    On Wed Dec 18, 2024 at 10:58 AM GMT, Henrik Ahlgren wrote:
    Adding a couple of lines to the systemd unit should not add any new
    dependincies to the package, or affect users of other init systems in
    any way.

    That's a very good point. In which case that's likely not why there is a
    lag in adopting these things.

    Perhaps we could have a release goal to set a certain bar in terms of hardening features for trixie+1?

    (This would be great - i dont think debian's policy says anything at all
    about this. there's also a bit of a gap in how to replace cron scripts
    since timers are much more flexible)

    It might help if there was an analysis tool -- i saw https://github.com/synacktiv/shh/tree/master/ which might be interesting
    (nb: untested!)

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Richard Lewis@21:1/5 to Michael Tokarev on Wed Dec 18 22:40:02 2024
    Michael Tokarev <mjt@tls.msk.ru> writes:

    BTW, is there a way for a systemd unit to take/inherit (security) settings from
    another unit?

    You might want to look at the examples here https://github.com/cyberitsolutions/prisonpc-systemd-lockdown/tree/main/systemd/system/postfix%40.service.d
    (from Trent W. Buck)

    i think they use symlinks

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Michael Tokarev@21:1/5 to Henrik Ahlgren on Thu Dec 19 08:20:01 2024
    17.12.2024 00:31, Henrik Ahlgren wrote:
    On Mon, 2024-12-16 at 21:21 +0300, Michael Tokarev wrote:
    It turns out the reason for this is a myth, which we believed to for
    25 years - a myth that "On FreeBSD, chroot is painless, but on Linux,
    chroot never works and is only suitable for the ones who want pain".
    Actually, it looks like, chroot on linux is *exactly* the same as on
    FreeBSD, and the pain level completely depends on which features you
    use (I mentioned all 3 possible issues in my initial email). It feels
    like this is the sole source of this opinion.

    I have never heard about such myth. Perhaps you are referring to the
    FreeBSD jail feature, which obviously is superiour to plain chroot.
    chroot(2) is a very simple and ancient Unix mechanism from 1979 and I
    believe it is exactly the same on all Unix/Posix-style systems.

    No, we are (me and Wietse whom I quited) referring to chroot(2), and
    operations within with might require extra files in the jail (like
    presence of /etc/hosts /etc/services etc for host lookups).

    Anyway, systemd's hardening features are so easy and effective that I
    would really like to see not only postfix, but ALL services use them as
    much as possible. Why we still have major packages like nginx shipping without any hardening out-of-the-box?

    It is not the case with postfix, who is a service manager internally, -
    it is like a small systemd inside, which does its own hardening. But it
    does this within POSIX entirely, not adding any extra layers. Parts of
    postfix (the service manager and local delivery agent just to name a
    few) needs elevated privs, and most other stuff is running with reduced
    privs.

    It is definitely not a simple task to reduce postfix privileges above
    postfix itself, - it will be either too much so postfix wont do its
    work anymore, or it will be just for the sake of doing it, without
    actual real gain. For a real solution, postfix need to be made aware
    of - for example, capabilities - *inside*, and apply them on a per-
    service or per-context basis. Attempts to apply this from outside are
    useless.

    Thanks,

    /mjt

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Michael Tokarev@21:1/5 to Henrik Ahlgren on Thu Dec 19 09:10:02 2024
    17.12.2024 00:31, Henrik Ahlgren wrote:

    Anyway, systemd's hardening features are so easy and effective that I
    would really like to see not only postfix, but ALL services use them as
    much as possible. Why we still have major packages like nginx shipping without any hardening out-of-the-box?
    I would advise against this simplistic view on security and "hardening" -
    it often makes false sense of security instead of real security.

    Besides, I yet to see an actual explanation how current postfix chroot
    is not as good as systemd unit hardening can bring us (omitting the
    complexity of postfix internal services architecture for now, let's
    say, just for smtpd service alone). Which treats linux abilities
    above POSIX (which can be applied using systemd or in other ways)
    are handled "better" than postfix already handles with its internal
    hardening, including its own chroot?

    /mjt

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Michael Tokarev@21:1/5 to Simon Richter on Thu Dec 19 09:00:01 2024
    18.12.2024 16:16, Simon Richter wrote:

    Postfix's "master" process effectively implements an orchestrator for a microservice architecture, similar to what systemd does. This could be
    replaced by systemd by writing a generator that creates units from master.cf.

    Simply converting the default master.cf to unit files and shipping these as default would still be a massive regression, because it would obsolete a
    lot of documentation on how the Postfix system is orchestrated and how to integrate new services like clamav and spamassassin or to deliver to courier.

    All of these recipes would become "instead of this one line in master.cf, you need to create a service unit, socket unit and possibly timer unit." My
    expectation would be that the next sentence would be "Please refer to the systemd documentation on how to do that.", and while the systemd
    documentation is exhaustive, it is significantly less targeted to the operation of mail systems than the manual page for the master.cf configuration
    file format.
    Yes Simon, this is a very good summary, exactly to the point. Adding restrictions on top of postfix master(8) is ineffective, it should be
    done at postfix service level and sometimes context level.

    Implementing each postfix service as a separate service in systemd
    is an interesting idea, and I even tried to implement it, because it
    feels.. "right". It definitely needs code changes within postfix
    itself though.

    /mjt

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Henrik Ahlgren@21:1/5 to Michael Tokarev on Thu Dec 19 10:10:02 2024
    On Thu, 2024-12-19 at 11:04 +0300, Michael Tokarev wrote:
    I would advise against this simplistic view on security and "hardening" -
    it often makes false sense of security instead of real security.

    Yes I agree, also blindly applied "hardening" settings can cause the
    program not function correctly in use cases not tested by the DD doing
    the packagking, and possibly even weaken security.

    In any case, postfix is rather special having an internal complicated sandboxing and privilege separation architecture, which most services
    don't have. So my comment was perhaps more about adding them to many
    other services (especially Internet facing) that lack any such
    features. 

    Take bind9 named(8) for example – it can chroot (with -t) but AFAIK
    Debian does not use it by default, and I think using the various
    systemd sandboxing would be much more friendly approach. (The security
    track record of BIND is sadly not the greatest.)

    Of course, we already do have plenty of packages with fairly strict
    systemd security settings, e.g. chrony or tor. I do believe those
    "hardenings" bring real security benefits, and they are much more easy
    to understand (or fine tune) than SELinux or AppArmor profiles – which
    chrony and tor also have, BTW, belt and suspenders I guess.

    Besides, I yet to see an actual explanation how current postfix chroot
    is not as good as systemd unit hardening can bring us (omitting the complexity of postfix internal services architecture for now, let's
    say, just for smtpd service alone). Which treats linux abilities
    above POSIX (which can be applied using systemd or in other ways)
    are handled "better" than postfix already handles with its internal hardening, including its own chroot?

    At least I would be very careful with patching postfix code to drop any
    chroot based security it has from the actual source. Better work with
    the upstream author. But let's not forget, chroot only sandboxes the
    file system, the are other things that can be limited as well, like CapabilityBoundingSet.

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Marco d'Itri@21:1/5 to Henrik Ahlgren on Thu Dec 19 11:20:01 2024
    On Dec 19, Henrik Ahlgren <pablo@seestieto.com> wrote:

    Take bind9 named(8) for example – it can chroot (with -t) but AFAIK
    Debian does not use it by default, and I think using the various
    Because it makes managing it much harder, since /etc/bind/ then moves to /var/.
    Systemd directives like ProtectSystem, ReadOnlyPaths, etc... are much
    easier to use than a blunt chroot.

    systemd sandboxing would be much more friendly approach. (The security
    track record of BIND is sadly not the greatest.)
    This is a urban legend: actually you are thinking of BIND 8.
    BIND 9 is a totally different code base and it reliably kills itself
    when something goes wrong.

    --
    ciao,
    Marco

    -----BEGIN PGP SIGNATURE-----

    iHQEABYIAB0WIQQnKUXNg20437dCfobLPsM64d7XgQUCZ2PwGQAKCRDLPsM64d7X gXd3APdJu2e5irY8o8Kgzxzs1QRO6Ifnnl84RF+7RKn9IydKAP9PwC4heiDTJLp6 7KBv/yN0MaFu89m6B5ax8ZyOoN5HAA==
    =mZZv
    -----END PGP SIGNATURE-----

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Helmut Grohne@21:1/5 to Simon Richter on Sun Dec 22 08:50:01 2024
    Hi Michael and Simon,

    On Wed, Dec 18, 2024 at 10:16:12PM +0900, Simon Richter wrote:
    Postfix's "master" process effectively implements an orchestrator for a microservice architecture, similar to what systemd does. This could be replaced by systemd by writing a generator that creates units from
    master.cf.

    Thank you for the excellent explanation.

    Simply converting the default master.cf to unit files and shipping these as default would still be a massive regression, because it would obsolete a lot of documentation on how the Postfix system is orchestrated and how to integrate new services like clamav and spamassassin or to deliver to
    courier.

    I would like to dive slightly deeper into the prospect of eliminating
    the master process. You already gave a very good account on the
    usability aspects, but the technical aspects seem worth a bit more
    exploration to me.

    Postfix does not chroot all services. If it were, local delivery would
    not work. The value we gain from the orchestrator here is that it
    separates functionality into different processes somewhat isolated from
    one another to limit possible blast radius. The network facing smtp
    process has no business accessing user's $HOME. The way we get there
    today is chroot.

    If viewing this from a systemd angle, we would not want to chroot the
    smtp process and rather hand it its own mount namespace where /home (and
    other pieces) are rendered inaccessible (e.g. via ProtectHome or it
    being implied). Postfix upstream cannot rely on these mechanics, because
    it wants to support non-Linux deployments where mount namespaces are unavailable.

    If they were to ship unit files, they'd end up in the same documentation nightmare as we would if we were to start doing so, and if they create a generator and start using systemd as the controlling process, they would not gain anything, because neither can they drop their own orchestrator (so this creates additional testing surface, as their services would now need to implement *two* service manager protocols), nor would that directly provide any additional functionality.

    I'm not sure the generator approach needs to be ruled out that quickly.
    The fields of master.cf map quite nicely to systemd. I'd interpret
    "chroot" diferently and map to ProtectSystem (or similar) instead and at
    that point we alleviate the initial concerns that Michael posted in his
    initial mail (e.g. nsswitch). This is where we gain something compared
    to keeping Postfix' master process. Many of the units are
    socket-activated in a very similar way to how systemd understands socket activation. Most postfix processes perform an idle-exit, so your typical
    system only sees few of them at the same time.

    This approach is not without downsides. I'm not going to repeat Simon's.
    We'd also end up with a *lot* of .service units (30 in a typical
    installation). systemd developers may disagree with using a generator
    for orhestrating systemd units inside a master.cf file.

    Going back to the initial question of chrooting, I guess the main work
    is done as chroot is what installations do by default. In the event that
    chroot poses a problem (which seems to be rare), individual processes
    may be opted out of chrooting in master.cf at ease. Users who strongly
    care about security would likely separate their installation into
    multiple Postfix instances one of which is facing to the internet and
    another to perform delivery to local users (likely running on different machines). The status quo does not look bad enough to me to warrant
    serious investment here, but maybe I am not enough of a power user of
    Postfix to understand the pain caused by chroots.

    Going

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Michael Tokarev@21:1/5 to Helmut Grohne on Sun Dec 22 12:40:02 2024
    22.12.2024 10:24, Helmut Grohne wrote:
    Hi Michael and Simon,

    Hello Helmut! :)

    On Wed, Dec 18, 2024 at 10:16:12PM +0900, Simon Richter wrote:

    If they were to ship unit files, they'd end up in the same documentation
    ..
    I'm not sure the generator approach needs to be ruled out that quickly.
    The fields of master.cf map quite nicely to systemd. I'd interpret

    Heh. Please pause and exhale here. We are not going to ship unit files
    for individual Postfix components.

    Yes, the idea is very interesting, and it's quite something which goes to
    mind when one think about it from a distant perspective. However, for this
    to be anyhow real, postfix has to be changed. Maybe not much, but it definitely needs quite some tweaks. Because while the concepts are similar
    to what's available in systemd, the actual implementation is different and
    at the very least, some shim layer needs to be implemented for this to work.

    Also, postfix processes does the sandboxing internally and voluntary, it
    is not the master process who does the sandboxing, - master process merely gives args to a common postfix main() routine, which does the rest of the
    work - because in many cases, root-time init is needed.

    We're NOT shipping unit files for Postfix daemons.

    ...
    Going back to the initial question of chrooting, I guess the main work
    is done as chroot is what installations do by default. In the event that chroot poses a problem (which seems to be rare), individual processes
    may be opted out of chrooting in master.cf at ease. Users who strongly
    care about security would likely separate their installation into
    multiple Postfix instances one of which is facing to the internet and
    another to perform delivery to local users (likely running on different machines). The status quo does not look bad enough to me to warrant
    serious investment here, but maybe I am not enough of a power user of
    Postfix to understand the pain caused by chroots.

    I dug into this hole (or swamp). Initially I was strongly against using
    chroot by default, - for 25 year already. The same is true for the
    upstream.

    However, after seeing all the bug reports in our BTS, re-reading numerous questions on postfix-users@, various guides on the 'net related to Postfix
    and Debian... it feels like most issues we caused to Postfix community and users are not because chroot is really painful, but because of insufficient support of this concept on the Debian end. Most of the pain could've been avoided entirely if the correct approach were taken at the beginning and sticked to.

    However, *some* aspects are painful indeed. It took me quite some time to figure it all out.

    For example, when the talk is about SASL authentication, the first thing
    which comes to mind is Cyrus SASL (there's also Dovecot SASL, and it is
    quite capable to be used by its own, but in people minds it is an integral
    part of Dovecot and "since I don't use Dovecot, its SASL is not for me").
    So, people choose Cyrus SASL. Which has little understood defect or a limitation: its initial saslauthd implementation and protocol is limited
    to verifying just username+password (it is a pwcheck method). So it only supports PLAIN/LOGIN, nothing more. All the other SASL mechanisms are
    done as plugins (auxprop, ie, using auxiliary properties, not just user
    and password) within application address space, - in our case, they all
    work within Postfix smtp[d] - which, by default, is running chrooted. So
    all guides on the 'net end up doing `mv /etc/sasldb2 /var/spool/postfix/etc/`, which is an insane thing to do. And there's no way to fix this in current infrastructure, besides switching to dovecot auth (which works by implementing a higher-level protocol than saslauthd does).

    Also, current glibc has interesting properties wrt chroot and nsswitch.conf (and hence libnss modules). It is neither bad or good thing, it is just something which needs to be taken into account when running anything in
    a chroot - postfix packaging hasn't changed in this place since 2001 or
    so, - current glibc works differently. Hopefully I'll fix this part in
    postfix for good before trixie.

    So overall, things seemed rather good, despite all the hate (from me and
    whole postfix community) this chroot-by-default decision in debian received, except a few cases, some of which are easy to fix while some aren't and
    it's actually better to un-chroot parts of postfix.

    Again, there was no easy way to turn the chroot off - current postfix package in debian provides a trivial command for this: `postfix chroot off' (and `postfix chroot on' to turn it on).

    Please note also that there are a LOT of guides on the 'net about setting
    up postfix on debian/ubuntu, all of them assumes postfix is running chrooted, and most of them will just break if we're to un-chroot it out of the sudden - people will start asking "why it doesn't work anymore".

    With all this, I have a small hope there might be no need to revert that
    25 years old decision which made so much pain.

    I think there should be a debconf question asking if the user wants to (un)chroot
    it. Dunno which default answer to have though. With a really easy way to toggle
    already present.

    Thanks,

    /mjt

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)
  • From Simon McVittie@21:1/5 to Michael Tokarev on Sun Dec 22 13:20:01 2024
    On Sun, 22 Dec 2024 at 14:39:35 +0300, Michael Tokarev wrote:
    And there's no way to fix [an in-process plugin architecture] in current infrastructure, besides switching to dovecot auth (which works by implementing
    a higher-level protocol than saslauthd does).

    This is a recurring pattern: if we want a component to be able to run in
    a container (and chroots can be viewed as a simple case of containers), in-process plugins and C-library-ABI-based protocols are often
    troublesome, and it's often necessary to have a design where some amount
    of "the real work" is done in a separate process, with which the
    containerized component communicates via a somewhat stable IPC protocol.

    We see this with audio in Flatpak apps working best with Pulseaudio or
    Pipewire (a stable IPC protocol to a daemon that does the real work), erratically with ALSA dmix (an undocumented IPC protocol), and not at all
    with JACK (a stable C library ABI that wraps an unstable IPC protocol).
    We also see this with Wayland and X11 working quite well across container boundaries in practice, and protocols over a generic AF_UNIX transport
    like D-Bus or Varlink often being container-friendly (if they have been designed that way - there are plenty of ways to get this wrong).

    smcv

    --- SoupGate-Win32 v1.05
    * Origin: fsxNet Usenet Gateway (21:1/5)