With https://github.com/home-assistant/cli/pull/604, there is no
--content-trust option anymore. Remove the call and only check if Supervisor is
healthy and there are no issues. This replaces #4370 which is too broad.
Because the OS downgrade performed in tests now triggers change in container
snapshotters, all containers need to be redownloaded. Make sure that CLI
container exists and increase the timeout for the time being.
The tests that are involving reboots are flaky and fail when waiting for the
command to return or when waiting for a new login prompt. To mitigate this, do
not use run_check, as it needs the shell prompt to reappear, and instead use
sendline and wait up to a minute for the GRUB message.
Add timeout to expect call when waiting for the OS reboot after
switching slots. While it never fails for me locally, it regularly
breaks tests in GHA.
Add test that OS update works - use the whole stack using CLI to update to the
latest stable version (unless executed manually on the latest stable release,
this version should never be the same as the currently tested one).
With this test in place, we can also test command for switching the slots, so
add an extra test for that too.
Fixes#4103
When there's a problem with connectivity, it may result in obscure errors later
in the testing. Add checks testing three scenarions:
* connectivity in host - both using curl and nmcli
* connectivity in Supervisor container (uses docker0 as default via)
* connectivity in CLI container (uses hassio as default via)
Also make sure that Supervisor upgrade isn't attempted when the version
information is missing.
We check that landing page is working when the network is down but we don't
check it in the happy path. Add its test to make it more obvious when the
just landing page is broken.
* Add test checking journal logs for dependency cycles
* Run some test cases to get their output also when full init fails
* Remove high timeouts from the times when GHA couldn't use KVM
* Enable logging durations for future optimizations
Allow configuration of the swap size via /etc/default/haos-swapfile file. By
setting the SWAPSIZE variable in this file, swapfile get recreated on the next
reboot to the defined size. Size can be either in bytes or with optional units
(B/K/M/G, accepting some variations but always interpreted as power of 10). The
size is then rounded to 4k block size. If no override is defined or the value
can't be parsed, it falls back to previously used 33% of system RAM.
Fixes#968
I have only tested that it fails for unreleased 6.6.72 kernel but haven't
tested the happy path and missed that it also failed because the types were
different. Stupid me.
Add test that the kernel isn't tainted at the end of the basic and supervisor
test suites, allowing us to catch e.g. kernel warnings that may left unnoticed
if dmesg isn't checked. There is no other source of tainting, so the value
should be always zero.
Bump labgrid to latest release. None of the changes require adjustments in the
tests. Remove pytest from requirements.txt, it's not needed anymore, so let pip
to resolve the correct (latest) version from labgrid's dependencies. With these
new dependencies, previous DeprecationWarnings on Python 3.12 are gone now.
* Test landing page is reachable without internet connection
Add test that checks user is able to access the landing page even when HAOS has
no internet connection. We still need some sort of outgoing connectivity, so
outgoing connection attempts don't end up with "network is unreachable". To
simulate this, restricted network is created for the QEMU instance used in the
test, and when everything is started, unresponsive default gateway is added.
This intents to test regression that was fixed by
home-assistant/supervisor#5204, Supervisor 2024.7.0+ is thus needed for this
test to pass.
* Bump requirements for tests
RAUC currently doesn't know the version of the booted slot when booted for the
first time or after wiping the data partition. As a result `ha os info` is
missing this information too.
As there's no built-in mechanism for generating these data by RAUC itself, add
a oneshot service that checks if the boot slot information is contained in the
rauc.db and if not, then generate it.
RAUC seems to cope quite well even with bogus data contained in rauc.db but in
any case, a test has been added to check that everything works as expected.
On the very first boot, grubenv doesn't exist and loading and saving it
silently fails. However, it is later created by the hassos-persists script and
the missing information are added by grub.cfg on the next boot. As the
consequence, when RAUC tries to get information from grubenv on the first boot,
ORDER variable is missing and the slot is reported as bad.
Fixes#3376
* Improve handling of timeouts in tests
Make timeout handling in tests more transparent. Added a custom shell
driver that allows to define global timeout for commands in the config
file, and replaced for/sleep constructs with infinite loops that will be
eventually terminated by pytest-timeout plugin. Current timeouts taken
from last runs on Github CI with some extra headroom.
* test_supervisor_is_updated shouldn't be skipped if no update was needed
* Allow more time for system startup
* Allow even more time for system startup
* Add test suite for Supervisor tests
* test_supervisor_is_updated should depend on test_update_supervisor
Co-authored-by: Stefan Agner <stefan@agner.ch>
---------
Co-authored-by: Stefan Agner <stefan@agner.ch>
* Create foundation for Labgrid-based OS tests
Add foundation for Labgrid-based tests of OS builds. Currently uses just
the QEMU driver, which starts a virtual machine with pristine OS, and
generates few log reports which are saved as build artifacts.
Workflow is currently triggered either manually by specifying an OS
version, or by OS build job, which now saves an artifact of the OVA
image. This allows for some modularity. If we eventually add the
possibility to run builds on PRs, we could also add the workflow_call
trigger and turn the workflow into a reusable one.
TBD (in future PRs): some meaningful tests and possibility to test on
real hardware (either local or distributed).
* Apply suggestions from @agners
Co-authored-by: Stefan Agner <stefan@agner.ch>
* Wrap test command in a script, create venv for local tests
* Make shellcheck happy
---------
Co-authored-by: Stefan Agner <stefan@agner.ch>
