Enable the systemd-resolved stub resolver and make it available on the
hassio host network interface (172.30.32.1). This allows to use
systemd-resolved directly from all containers.
Note that this makes /etc/resolv.conf point to the stub resolver running
at 127.0.0.53 by default. This stub resolver isn't reachable from within
containers. However, Docker does regnize this situation [1] and falls back
to the alternate path at /run/systemd/resolve/resolv.conf, which is what
/etc/resolv.conf is today. So this should not affect the initial
/etc/resolv.conf in containers in practise.
This will however bind to port 53 and affect add-on potentially attempt
to use that port. Add-ons should not bind to 127.0.0.53 or the hassio
host network (172.30.32.1).
[1] https://github.com/moby/moby/blob/v28.0.4/libnetwork/internal/resolvconf/resolvconf_path.go#L51C32-L51C45
* Relocate HAOS Systemd drop-ins to /usr/lib/systemd
With some exceptions, Systemd drop-ins overriding default unit configuration
have been placed to `/etc/systemd/system`. This is meant for user overrides of
those, or per `man 5 systemd.unit` for "system unites created by the
administrator". Relocate all of these to `/usr/lib/systemd` which should be
used as path for units "installed by the distribution package manager" which is
closer to what we're trying to achieve.
This will make it easier to detect changes to unit files once we enable the
possibility to edit the content of /etc.
* Patch systemd-timesyncd.service instead of replacing it fully
Reduce verbosity from deactivated Docker mounts, triggered by the Docker
healthcheck. These messages do not carry any value for us and logs supplied by
users are often spammed mostly with these. Moreover, they sometimes cause
confusion that something is wrong, see for example #3021.
Unfortunately, it's not possible to use LogFilterPatterns= here, because it's
not applied to these messages, as explicitly said in the docs:
Filtering is based on the unit for which LogFilterPatterns= is defined
meaning log messages coming from systemd(1) about the unit are not taken
into account.
runc 1.2.0 supposedly should fix this, but it's unclear when it would be
available, so let's stick to this solution (reducing verbosity from debug to
notice for all units `run-docker-*.mount`) for the time being.
In case a system takes a bit longer to boot (e.g. due to SWAP
initialization on first boot, especially on a system with lots of memory
and not very fast strage, e.g. an ODROID-M1 using an SD card) we might
time-out waiting for time synchronization before the time
synchronization service even got started. By ordering the
systemd-time-wait-sync.service after the network is online, the timeout
of this service should be started much later. With that the
systemd-time-wait-sync.service shouldn't timeout any longer.
By default systemd kills the service which causes an OOM. That make
sense for a typical service, however, for SSH we don't want this
behavior: The connection should continue, just the command which caused
OOM should be killed.
This is more readable than passing arguments to the daemon directly. It
also shortens the ExecStart command significantly, which is stored in
every log entry in systemd-journald.
It seems that Docker can fail to start if there is no space left on the
device. Try to free up some space in that case by asking journald to
limit its size to 256MiB.
This should work for any storage larger than ~2.5GiB (as the journals
maximum size is 10% of the disk size). It still should leave enough
logs to diagnose problems if necessary.
Note: We could also limit the size of the journal in first place, but
that isn't sustainable: Once that space is used up, we run into the
same problem again.
By only asking journalctl to free up if necessary, we kinda (miss)use
the journal as way to "reserve" some space which we can free up at boot
if necessary.
This can be helpful when debugging HAOS issues. Dropbear is only started
for users which actually enabled it by configuring a SSH key, so this
change won't have an effect for most people.
* Fix delaying systemd-timesyncd
Setting WantedBy=time-sync.target in a service.d config file does not
clear previous assignments of WantedBy. This caused the services to still
be pulled in by the sysinit.target, causing a ordering cycle and the
system to not start essential services.
* Remove sysinit.target from Before ordering
With commit 2d3119ef22 ("Delay Supervisor start until time has been
sychronized (#1360)") systemd-time-wait-sync.service got enabled, which
waits until systemd-timesyncd synchronizes time with a NTP server.
By default systemd-timesyncd.service and systemd-time-wait-sync.service
are pulled in by sysinit.target. This starts the services before full
network connectivity is established. The first sychronization fails and
systemd-timesyncd only retries after a ratelimit mechanism times out.
This causes a dealy of 30s during startup. While systemd-timesyncd has
a mechanism to (re)try time synchronization when network becomes
online, it seems that those only work properly when systemd-networkd
is used, see also https://github.com/systemd/systemd/issues/24298.
Simply reordering systemd-timesyncd.service after network-online.target
does not work as it causes circular dependencies (NetworkManager itself
depends ultimately on the sysinit.target).
With this change, the services are only pulled in by time-sync.target.
That allows to order the service after network-online.target. With that
the first synchronization succeeds.
This mechanism also works when a NTP server is provided through DHCP.
In that case, a the systemd-timesyncd service is started by the dispatch
script /usr/lib/NetworkManager/dispatcher.d/10-ntp before the systemd
even considers starting the service. Tests show that the default
fallback NTP is not contacted, only the DHCP provided service.
Currently systemd-timesyncd tries to connect to the NTP server quite
early at boot-up. At this time the network connection has not been
established yet. This causes resolving the NTP server to fail and
a rate limit kicks in which makes systemd-timesyncd wait for 30s until
the next attempt.
Lowering the retry attempt to 10s makes systemd-timesyncd connecting
shortly after.
Note: The rate limit is 10 attempts per 10s. Because the attempts are
immediately exhausted lowering connection retry attempt below 10s
adds no benefit.
See also: https://github.com/systemd/systemd/issues/24298
* Bump buildroot
* buildroot 99b62b8bd3...97287bbebf (3):
> package/dbus-broker: bump to release 32
> package/dbus-broker: new package
> Merge pull request #3 from home-assistant/2022.02.x-haos-cgroup-v2
* Use dbus-broker as default D-Bus broker
The dbus-broker (Linux D-Bus Message Broker) aims to be a high
performance and reliable D-Bus broker which can be used as a drop in
replacement to the reference implementation D-Bus broker. In tests it
showed significantly better performance especially when routing BLE
messages.
* Allow dbus-broker to start early
For HAOS device wipe feature we need haos-agent.service and
udisk2.service early. Both require a working D-Bus broker.
The options PrivateTmp and PrivateDevices add additional After=
orderings which doesn't allow dbus-broker to be started early.
* Fix D-Bus dependency
D-Bus services should just depend on dbus.socket.
* Disable real-time scheduling
It seems that Linux' cgroup v2 currenlty does not support RT scheduling.
* Remove Supervisor RT support flag
With CGroups v2 we can no longer support CPU resource allocation for
realtime scheduling.
* Bump OS Agent to 1.3.0 for CGroups v2 support
* Avoid duplicate log entries
So far the hassos-supervisor.service starts the hassos-supervisor script
which in turn attaches to the Supervisor container. This causes stdout
and stderr to be forwarded to the service unit, which in turn logs it in
the journal.
However, Docker too logs all stdout/stderr to the journal through the
systemd-journald log driver.
Do not attach to the Supervisor container to avoid logging the
Supervisor twice.
Note that this no longer forwards signals to the container. However, the
hassos-supervisor.service uses the ExecStop= setting to make sure the
container gets gracefully stopped.
* Use image and container name as syslog identifier
By default Docker users the container id as syslog identifier. This
leads to log messages which cannot easily be attributed to a particular
container (since the container id is a random hex string).
Use the image and container name as syslog identifier.
Note that the Docker journald log driver still stores the container id
as a separate field (CONTAINER_ID), in case the particular instance need
to be tracked.
* Start ha-cli on tty1 instead of a getty
Instead of starting a getty start the ha-cli directly. This will show
the banner right on startup with the important information such as IP
address of the instance or the URL to reach it.
* Use default shell as root shell instead of HA CLI
Instead of using the ha-cli.sh script as login shell use the regular
shell. Amongst other things, this allows to run VS Code devcontainers
remotely via SSH or using scp. The HA CLI is still available using the
`ha` command.
* Enable systemd-time-wait-sync.service by default
Enable the systemd-time-wait-sync.service by default. This allows to use
the time-sync.target which allows to make sure services only get started
once the time is synchronized.
* Make sure time is synchronized when starting hassos-supervisor.service
Use the time-sync.target to make sure that the Supervisor gets stsarted
after the time has been synchronized.
* Set timeout for systemd-time-wait-sync.service
Don't delay startup forever in case time synchronization doesn't work.
This allows to boot the system even without Internet connection.
* Disable systemd-logind support for udisks2
Currently udisks2 uses systemd-logind to prevent the system from
rebooting or similar operations while udisks operations are ongoing.
Unfortunately this stops us from using udisks2 during early boot since
systemd-logind is not ready at this point. Make the dependency
configureable so we can opt-out of using systemd-logind.
* Make dbus.service/socket and udisks2.service/socket available early
Disable default dependencies. This avoids those services to be ordered
after sysinit.target, and makes them available before local-fs.target
is reached. All mounts like mnt-data.mount are ordered before
local-fs.target, so breaking this dependency allows to use D-Bus before
mounting local file systems.
This seems fine when using the system bus directly from /run (instead of
/var/run, which is anyway a symlink to /run normally). It seems that
udisks misses /var/lib/udisks2 but it seems not to be required for the
features used so far.
So far the exit code has been evaluated, which seems to be non-zero even
with a regular term signal. With that systemd assumed the service is in
a failed state, when in fact this seems the regular behavior of dropbear
when shutting it down.
* Add --cpu-rt-runtime to allow Docker allocate real-time CPU time (#1235)
* Enable Supervisor's CPU bandwith allocation feature (#1235)
Since we have CONFIG_RT_GROUP_SCHED enabled in the Home Assistant OS
kernel the Supervisor needs to enable CPU bandwith allocation for
Add-Ons which need real-time scheduling. Set the appropriate environment
variable.
It seems that on certain setups the default DNS over TLS mode
"opportunistic" causes delays of ~10s when trying to resolve names. This
is probably caused by providers and/or firewall setups not properly rejecting
connections on port 853.
It seems that also other distributions (such as Arch Linux) still
disable DNS over TLS currently. Side step issues with DNS over TLS by
disabling it for now.
Old Laptops are a popular choice to run Home Assistant: They have low
power consumption, are relatively fast and cheap to come by. However,
closing their lid by default puts a Linux system into suspend. This is
not what the typical user of Home Assistant OS wants. Ignore lid
activity in any state by default.
* Add resolved.conf to disable stub resolver and DNSSEC
There are Add-Ons which try to bind port 53 on all interfaces including
127.0.0.53. Disable the stub resolver to make them continue working. We
don't need the resolver currently anyway.
Also disable DNSSEC to make sure the baords can access a NTP time server
even when their time is incorrect (since DNSSEC validation may fail).
This is a known chicken-egg problem with systemd-resolved/systemd-timesyncd
and might be addressed in a future version, with what we can reenable
DNSSEC:
https://github.com/systemd/systemd/issues/5873
* Make sure resolve gets added only once to nsswitch.conf
Only add resolve to nsswitch.conf if not already present.
On systems where ACPI support is present as inidcated by the presence of
/proc/acpi (e.g. on OVA compatible hypervisors), we want to properly
shut down the system when the power button is pressed (or the hypervisor
simulates this kind of event to the guest machine that executes hassos).
This changeset provides the following basic infrastructure for this
feature to work as expected:
* a systemd service to start acpid, if ACPI support can be assumed
* an acpid configuration directory
* a trivial shutdown script to invoke when a PWR event is registered
