Hello FreeBSD-Ports,
I switched a few months ago from running my main Poudriere builder on an
old Dell PowerEdge server to running it on an AWS EC2 instance
(c5.12xlarge in us-east-2). The instance was initially deployed from the "FreeBSD 14.4-RELEASE-amd64 UEFI-PREFERRED base ZFS" AMI (ami-075b8663dbd980c35) and has since been updated to 14.5-RELEASE. This instance has a 200G EBS volume of which 180G is a ZFS pool with roughly
80G free and 20G of swap to complement 96G of RAM. The instance does
nothing else besides shipping built packages via Nginx.
With relatively high regularity---most, but not all "bulk"
runs---poudriere will terminate with some variation of the following mix
of errors.
_mktemp: mkstemp failed on /srv/poudriere/data/logs/bulk/145amd64-default-general/2026-09-26_22h00m19s/.tmp-.poudriere.snap_loadavg.Ms7EnmmR:
No space left on device
echo: write error on stdout
_mktemp: mkstemp failed on /srv/poudriere/data/logs/bulk/145amd64-default-general/2026-09-26_22h00m19s/.tmp-.poudriere.status.10gWzEcY:
No space left on device
err: Recursive error detected: write_atomic unable to create tmpfile in /srv/poudriere/data/logs/bulk/145amd64-default-general/2026-09-26_22h00m19s _mktemp: mkstemp failed on /srv/poudriere/data/logs/bulk/145amd64-default-general/2026-09-26_22h00m19s/.tmp-.poudriere.ended.ld5bXm8y:
No space left on device
err: Recursive error detected: write_atomic unable to create tmpfile in /srv/poudriere/data/logs/bulk/145amd64-default-general/2026-09-26_22h00m19s echo: write error on stdout
This usually leaves one or two builder jails running, and I have gotten
in the habit of rebooting the instance before re-starting the build.
A typical build pulls in between 1000 and 5000 ports. I have not
detected a correlation between the crash and a specific port, but I have also not been keeping especially close track. Subjectively, the failure usually occurs in the latter third or latter quarter by port count.
My poudriere.conf is only lightly adapted from stock, and none of the
memory limit parameters are modified from "(default: none)".
I'm not sure what I should be looking at to narrow down the cause of
these failures. Any suggestions?
Thank you,
John
Hello FreeBSD-Ports,
I switched a few months ago from running my main Poudriere builder on an
old Dell PowerEdge server to running it on an AWS EC2 instance
(c5.12xlarge in us-east-2). The instance was initially deployed from the "FreeBSD 14.4-RELEASE-amd64 UEFI-PREFERRED base ZFS" AMI (ami-075b8663dbd980c35) and has since been updated to 14.5-RELEASE. This instance has a 200G EBS volume of which 180G is a ZFS pool with roughly
80G free and 20G of swap to complement 96G of RAM. The instance does
nothing else besides shipping built packages via Nginx.
With relatively high regularity---most, but not all "bulk" runs---
poudriere will terminate with some variation of the following mix of
errors.
_mktemp: mkstemp failed on /srv/poudriere/data/logs/bulk/145amd64-
default-
general/2026-09-26_22h00m19s/.tmp-.poudriere.snap_loadavg.Ms7EnmmR: No
space left on device
echo: write error on stdout
_mktemp: mkstemp failed on /srv/poudriere/data/logs/bulk/145amd64- default-general/2026-09-26_22h00m19s/.tmp-.poudriere.status.10gWzEcY: No space left on device
err: Recursive error detected: write_atomic unable to create tmpfile
in /srv/poudriere/data/logs/bulk/145amd64-default- general/2026-09-26_22h00m19s
_mktemp: mkstemp failed on /srv/poudriere/data/logs/bulk/145amd64- default-general/2026-09-26_22h00m19s/.tmp-.poudriere.ended.ld5bXm8y: No space left on device
err: Recursive error detected: write_atomic unable to create tmpfile
in /srv/poudriere/data/logs/bulk/145amd64-default- general/2026-09-26_22h00m19s
echo: write error on stdout
This usually leaves one or two builder jails running, and I have gotten
in the habit of rebooting the instance before re-starting the build.
A typical build pulls in between 1000 and 5000 ports. I have not
detected a correlation between the crash and a specific port, but I have also not been keeping especially close track. Subjectively, the failure usually occurs in the latter third or latter quarter by port count.
My poudriere.conf is only lightly adapted from stock, and none of the
memory limit parameters are modified from "(default: none)".
I'm not sure what I should be looking at to narrow down the cause of
these failures. Any suggestions?
Thank you,
John
| Sysop: | Amessyroom |
|---|---|
| Location: | Fayetteville, NC |
| Users: | 76 |
| Nodes: | 6 (0 / 6) |
| Uptime: | 59:29:56 |
| Calls: | 1,265 |
| Calls today: | 1 |
| Files: | 1,362 |
| D/L today: |
4 files (9,632K bytes) |
| Messages: | 286,017 |