* Gracefully end log stream when journal gateway connection is lost
When systemd-journal-gatewayd is stopped while a client follows logs
(e.g. on host reboot with the log viewer open), aiohttp raises
ClientPayloadError and advanced_logs_handler converted it to an
APIError. For the /supervisor/logs endpoints this got logged as an
unexpected error with a full traceback and captured to Sentry on every
occurrence (#7103, SUPERVISOR-1FHT).
Once the streaming response has started, an error response can no
longer be delivered anyway, so treat a lost connection to
systemd-journal-gatewayd like a client-side disconnect and end the
stream gracefully. The APIError is still raised when the connection is
lost before any data was sent to the client.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Treat unreachable systemd-journal-gatewayd as a known API error
Manual testing of the previous commit (killing systemd-journal-gatewayd
while Supervisor is running) showed that every log API request hitting
the dead gateway logs an "Unexpected error during API call" traceback
and captures HostServiceError to Sentry (SUPERVISOR-K8C), in addition
to the ERROR already logged at the raise site in journald_logs().
Make HostServiceError inherit from APIError as well, following the
HostContainerLogEpochError precedent, so the api_process decorators
return a plain 400 response without the redundant traceback and Sentry
capture. Also treat it like HostNotSupportedError in the supervisor
logs fallback wrapper: fall back to Docker container logs with a
warning instead of an exception log plus Sentry capture.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
* Use dedicated exception for journal gateway connection errors
HostServiceError is also raised by ServiceManager for systemd units
called through the API (e.g. /os/config/sync), where inheriting from
APIError would hide genuine service breakage from Sentry. Introduce
HostJournalGatewaydConnectionError subclassing HostServiceError and
APIError, and raise it for the gatewayd connection failure only, as
suggested in the PR review.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
---------
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* Fix typos in comments, docstrings and log messages
Correct 39 spelling mistakes across comments, docstrings and log/error
message strings throughout the package (e.g. "conection" -> "connection",
"Incomming" -> "Incoming", "Rasie" -> "Raise"). All changes are confined
to human-readable text; no identifiers, attributes or D-Bus contracts are
touched, so there is no behavior change. Found with codespell.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Fix typos in tests and CI workflow
Correct spelling mistakes in test comments, docstrings and data, plus one
in the builder workflow, so the whole tree is clean for the codespell hook
added next. The assertion in test_network_manager.py is updated to match
the corrected "Unknown error while processing" log message in the source.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Add codespell pre-commit hook
Wire up codespell so spelling mistakes in comments, docstrings and strings
are caught automatically. The vendored frontend panel is excluded, and
"hass" and "astroid" are added to the ignore list as known false positives
(the Home Assistant abbreviation and the pylint dependency package).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Address review feedback
Improve grammar in several of the touched comments and docstrings: use the
plural "ignore conditions" for the list-returning property, add the missing
auxiliary verb and fix agreement in the timezone-filter comment, fix
"backups ... use" agreement, and reword "underlay" to "underlying" in the
arch module docstring.
Also drop the "*.json" skip from the codespell hook. It was carried over
from another project but is unnecessary here (all tracked JSON is clean),
and skipping it would needlessly leave translation and data JSON unchecked.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* Reword onboarding comment
"overflight" was a literal calque of the German "überflogen"; use the
idiomatic "skimmed through" instead.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
* tests: enable flake8-pytest-style (PT) ruff rules
Enable the `PT` ruff rule set and fix the resulting violations across the
test suite:
- PT006: pass parametrize argument names as tuples instead of a single
comma-separated string.
- PT022: switch fixtures that have no teardown from `yield` to `return`
so the lack of cleanup is obvious at a glance.
- PT011: add `match=` to broad `pytest.raises(ValueError)` blocks so the
expected error is anchored to a specific message.
- PT012: hoist setup (patches, branching) out of `pytest.raises()`
blocks so only the call that is expected to raise remains inside.
- PT013: replace `from pytest import X` with `import pytest` and access
attributes via the module.
- PT015: replace `try/except` + `assert False` patterns with
`pytest.raises(...)`.
- PT017: replace `assert` on exceptions inside `except` blocks with
`pytest.raises(...) as exc_info` and assert on `exc_info.value`.
No behavioral changes to the tests; the full suite still passes.
* tests: address review feedback on PT ruff rule enablement
- Fix fixture return-type annotations after switching `yield` to `return`
in tests/conftest.py: drop the `Generator[...]`/`AsyncGenerator[...]`
wrapper for `dns_manager_service`, `supervisor_internet`, `websession`,
and `mock_update_data` so the annotation matches what the fixture
actually returns.
- Correct the return-type annotation of `fixture_ip6config_service` from
`IP4ConfigService` to `IP6ConfigService`.
- Fix recurring "excepiton" typo in tests/utils/test_exception_helper.py.
* tests: verify backup cleanup on permission error
After `test_new_backup_permission_error` raises `BackupPermissionError`,
assert that no tarfile was left behind and `tmp_path` is empty. The
previous version only checked that the exception was raised, which
missed any regression where a partial tarfile would survive the failed
create.
* tests: rename DNS_GOOD_V6 to DNS_V6_UNSUPPORTED
The constant was named "good" but its tests assert that the URLs are
rejected by the DNS validator. The IPv6 URLs are well-formed but
currently rejected because IPv6 doesn't work with the Docker network
(see `dns_url` in supervisor/validate.py). Rename the constant and the
related test to make the intent obvious.
The pytest config sets ``asyncio_mode = "auto"``, which already
auto-marks every ``async def test_*`` as a coroutine test. The 38
``@pytest.mark.asyncio`` decorators sprinkled across the suite were
no-ops kept around from before that flag was set. Remove them along
with the now-unused ``import pytest`` lines they were the only
consumer of.
Pure mechanical cleanup; no test behavior changes.
Follow-up on #6739: with HassioError now logged and captured by Sentry
in api_process, hostname rejections from systemd-hostnamed surfaced as
"unexpected" 400s with noisy log entries and a Sentry event, even
though the user had simply submitted an invalid hostname.
Map this through properly so the API returns a clean, structured 400:
- Split ErrorType.INVALID_ARGS out of DBusInterfaceMethodError into its
own DBusInvalidArgsError. The two cases collapsed there before are
semantically different: UNKNOWN_METHOD / INVALID_SIGNATURE mean the
call is broken (method missing or types wrong); INVALID_ARGS means
the call is valid but the service rejected an argument's value.
- Add HostInvalidHostnameError(HostError, APIError) with error_key and
extra_fields so clients get a normalized message and a stable key
rather than systemd's raw "Invalid static hostname '...'" text.
- Translate DBusInvalidArgsError to HostInvalidHostnameError in
SystemControl.set_hostname. @api_process turns the result into a 400
without logging or Sentry capture, since this is now a modeled
client-input error rather than an unexpected one.
Validation continues to live in hostnamed (hostname_is_valid() in
systemd's src/basic/hostname-util.c); Supervisor only translates the
rejection.
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* Refactor API registration to support v1/v2 via shared methods
- Add AppVersion StrEnum (V1, V2) to supervisor/api/const.py
- Replace self.v2_app with self._v2_app and expose a versions property
(dict[AppVersion, web.Application]) computed dynamically so that test
fixtures reassigning self.webapp are automatically reflected in V1
- All _register_* methods now accept a required app: web.Application
parameter; version-specific routes are gated with
"if app is self.versions[AppVersion.V1/V2]:"
- load() loops over enabled_versions (V1 always, V2 when feature-flagged)
and calls each registration method once per version, no duplication
- Static resources are registered before webapp.add_subapp() to avoid
registering into a frozen router
- add_subapp uses self.webapp directly for readability
- Fold _register_v2_apps/_register_v2_backups/_register_v2_store into
their respective unified methods; remove the now-defunct _register_v2_*
helpers and the _api_apps/_api_backups/_api_store instance vars
- _register_proxy and _register_ingress updated to accept app; legacy
/homeassistant/* proxy routes gated behind V1 conditional
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Add dual v1/v2 parametrization to API tests
All 163 tests across 17 API modules that register identically on both
v1 and v2 now run against both versions via api_client_with_prefix.
- tests/api/conftest.py: advanced_logs_tester switched to
api_client_with_prefix so log-endpoint tests are auto-parametrized;
accepts optional v2_path_prefix kwarg for paths that differ by version
- tests/api/test_{auth,discovery,dns,docker,hardware,host,ingress,
jobs,mounts,network,os,resolution,security,services,supervisor}.py:
api_client -> api_client_with_prefix with path prefix unpacking
- supervisor/api/__init__.py: _register_panel() moved outside the
version loop -- frontend static assets are V1-only
- tests/api/test_panel.py: kept on plain api_client (V1-only)
Tests intentionally kept V1-only:
- auth/discovery: use indirect api_client parametrize for addon context
- homeassistant: all tests call legacy /homeassistant/* paths (V1-only)
- jobs (4 tests): inner @Job-decorated classes register names into a
module-level set; re-running the same test raises RuntimeError
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Extend dual v1/v2 parametrization to homeassistant and jobs tests
tests/api/conftest.py:
- Add core_api_client_with_root fixture parametrized over three paths:
v1-core: /core/... (canonical v1 path)
v1-legacy: /homeassistant/... (legacy v1 alias, same handlers)
v2-core: /v2/core/... (canonical v2 path)
tests/api/test_homeassistant.py:
- Switch all 17 api_client tests to core_api_client_with_root so each
test runs against all three access paths (v1 canonical, v1 legacy
alias, v2 canonical), exercising every registered route
tests/api/test_jobs.py:
- Promote four inner TestClass definitions to module-level helpers
(_JobsTreeTestHelper, _JobManualCleanupTestHelper,
_JobsSortedTestHelper, _JobWithErrorTestHelper) so that @Job name
registration into the global _JOB_NAMES set only happens once at
import time rather than on each parametrized test run
- Replace closure references to outer-scope coresys with self.coresys
- Use api_client_with_prefix for dual-version coverage
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Fix typo
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
---------
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
* Rename addon→app in docstrings and comments
Updates all docstrings and inline comments across supervisor/ and
tests/ to use the new app/apps terminology. No runtime behaviour
is changed by this commit.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Rename addon→app in code (variables, args, class names, functions)
Renames all internal Python identifiers from addon/addons to app/apps:
- Variable and argument names
- Function and method names
- Class names (Addon→App, AddonManager→AppManager, DockerAddon→DockerApp,
all exception, check, and fixup classes, etc.)
- String literals used as Python identifiers (pytest fixtures,
parametrize param names, patch.object attribute strings,
URL route match_info keys)
External API contracts are preserved: JSON keys, error codes,
discovery protocol fields, TypedDict/attr.s field names.
Import module paths (supervisor/addons/) are also unchanged.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Fix partial backup/restore API to remap addons key to apps
The external API accepts `addons` as the request body key (since
ATTR_APPS = "addons"), but do_backup_partial and do_restore_partial
now take an `apps` parameter after the rename. The **body expansion
in both endpoints would pass `addons=...` causing a TypeError.
Remap the key before expansion in both backup_partial and
restore_partial:
if ATTR_APPS in body:
body["apps"] = body.pop(ATTR_APPS)
Also adds test_restore_partial_with_addons_key to verify the restore
path correctly receives apps= when addons is passed in the request
body. This path had no existing test coverage.
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
* Fix merge error
* Adjust AppLoggerAdapter to use app_name
Co-authored-by: Stefan Agner <stefan@agner.ch>
---------
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Stefan Agner <stefan@agner.ch>
The /os/info API endpoint has been using D-Bus property TimeUSec which got
cached between requests, so the time returned was not always the same as
current time on the host system at the time of the request. Since there's no
reason to use D-Bus API for the time, as Supervisor runs on the same machine
and time is global, simply format current datetime object with Python and
return it in the response.
Fixes#6581
* Handle missing Accept header in host logs
Avoid indexing request headers directly in the host advanced logs handler when Accept is absent, preventing KeyError crashes on valid requests without that header. Fixes SUPERVISOR-1939.
* Add pytest
Add support for `no_colors` query parameter on all advanced logs API endpoints,
allowing users to optionally strip ANSI color sequences from log output. This
complements the existing color stripping on /latest endpoints added in #6319.
* Strip ANSI escape color sequences from /latest log responses
Strip ANSI sequences of CSI commands [1] used for log coloring from
/latest log endpoints. These endpoint were primarily designed for log
downloads and colors are mostly not wanted in those. Add optional
argument for stripping the colors from the logs and enable it for the
/latest endpoints.
[1] https://en.wikipedia.org/wiki/ANSI_escape_code#CSIsection
* Refactor advanced logs' tests to use fixture factory
Introduce `advanced_logs_tester` fixture to simplify testing of advanced logs
in the API tests, declaring all the needed fixture in a single place. # Please
enter the commit message for your changes. Lines starting
* Add endpoint for complete logs of the latest container startup
Add endpoint that returns complete logs of the latest startup of
container, which can be used for downloading Core logs in the frontend.
Realtime filtering header is used for the Journal API and StartedAt
parameter from the Docker API is used as the reference point. This means
that any other Range header is ignored for this parameter, yet the
"lines" query argument can be used to limit the number of lines. By
default "infinite" number of lines is returned.
Closes#6147
* Implement fallback for latest logs for OS older than 16.0
Implement fallback which uses the internal CONTAINER_LOG_EPOCH metadata
added to logs created by the Docker logger. Still prefer the time-based
method, as it has lower overhead and using public APIs.
* Address review comments
* Only use CONTAINER_LOG_EPOCH for latest logs
As pointed out in the review comments, we might not be able to get the
StartedAt for add-ons that are not running. Thus we need to use the only
reliable mechanism available now, which is the container log epoch.
* Remove dead code for 'Range: realtime' header handling
* Storage space usage API
* Move to host API
* add tests
* fix test url
* more tests
* fix tests
* fix test
* PR comments
* update test
* tweak format and url
* add .DS_Store to .gitignore
* update tests
* test coverage
* update to new struct
* update test
Since Systemd v256 the Range header must not end with a trailing colon.
We relied on this undocumented feature when following logs, and the
frontend or CLI may still use it in requests. To fix the requests
failing with new Systemd version, intercept the header and fill in the
num_entries to maximum possible value, which avoids the journal-gatewayd
returning the response prematurely and also works on older Systemd
versions.
The journal-gatewayd would still return response if follow flag is used
along with num_entries, but this behavior is unchanged and would be
better fixed in the backend.
Link: https://github.com/systemd/systemd/issues/37172
* Add blockbuster library and find I/O from unit tests
* Fix lint and test issue
* Fixes from feedback
* Avoid modifying webapp object in executor
* Split su options validation and only validate timezone on change
When developing/testing in a Supervised environment, the
systemd-journal-gatewayd socket is actually available. Mock the
socket Path file to make the test independent of the pytest
environment.
If no cursor is specified and negative num_skip is used, we're pointing
one record back from the last one, so host logs always returned 101
lines as the default. This was also the case of the lines query argument
that used the number directly as num_skip. Instead of doing that, point
N-1 records to the back and then get N records. Handle 1 record and
invalid numbers silently to avoid the need for error handling in
unpractical edge cases.
Since headers are clumsy considering the Core proxy between the frontend
and Supervisor, add a way to adjust number of lines and verbose log
format using query parameters as well. If both query parameters and
headers are supplied, prefer the former, as it's more prominent when
reading through the request logs.
* Use Systemd Journal API for all logs endpoints in API
Replace all logs endpoints using container logs with wrapped
advanced_logs function, adding possibility to get logs from previous
boots and following the logs. Supervisor logs are an excetion where
Docker logs are still used - in case an exception is raised while
accessing the Systemd logs, they're used as fallback - otherwise we
wouldn't have an easy way to see what went wrong.
* Refactor testing of advanced logs endpoints to a common method
* Send error while fetching Supervisor logs to Sentry; minor cleanup
* Properly handle errors and use consistent content type in logs endpoints
* Replace api_process_custom with reworked api_process_raw per @mdegat01 suggestion
* Unsupported if wrong image used on virtualization
* Add generic-aarch64 as supported image
* Add virtualization field to API
* Change startup to setup in check
* Use Journal Export Format for host (advanced) logs
Add methods for handling Journal Export Format and use it for fetching
of host logs. This is foundation for colored streaming logs for other
endpoints as well.
* Make pylint happier - remove extra pass statement
* Rewrite journal gateway tests to mock ClientResponse's StreamReader
* Handle connection refused error when connecting to journal-gatewayd
* Use SYSTEMD_JOURNAL_GATEWAYD_SOCKET global path also for connection
* Use parsing algorithm suggested by @agners in review
* Fix timestamps in formatting, always use UTC for now
* Add tests for Accept header in host logs
* Apply suggestions from @agners
Co-authored-by: Stefan Agner <stefan@agner.ch>
* Bail out of parsing earlier if field is not in required fields
* Fix parsing issue discovered in the wild and add test case
* Make verbose formatter more tolerant
* Use some bytes' native functions for some minor optimizations
* Move MalformedBinaryEntryError to exceptions module, add test for it
---------
Co-authored-by: Stefan Agner <stefan@agner.ch>
* Add udisks2 dbus support
* assert mountpoints
* Comment
* Add reference links
* docstring
* fix type
* fix type
* add typing extensions as import
* isort
* additional changes
* Simplify classes and conversions, fix bugs
* More simplification
* Fix imports
* fix pip
* Add additional properties and fix requirements
* fix tests maybe
* Handle optionality of certain configuration details
* black
* connect to devices before returning them
* Refactor for latest dbus work
* Not .items
* fix mountpoints logic
* use variants
* Use variants for options too
* isort
* Switch to dbus fast
* Move import to parent
* Add some fixture data
* Add another fixture and reduce the block devices list
* Implement changes discussed with mike
* Add property fixtures
* update object path
* Fix get_block_devices call
* Tests and refactor to minimize dbus reconnects
* Call super init in DBusInterfaceProxy
* Fix permissions on introspection files
---------
Co-authored-by: Mike Degatano <michael.degatano@gmail.com>
* Add enhanced logging REST endpoints using systemd-journal-gatewayd
Add /host/logs/entries and /host/logs/{identifier}/entries to expose log
entries from systemd-journald running on the host. Use
systemd-journal-gatewayd which exposes the logs to the Supervisor via
Unix socket.
Current two query string parameters are allowed: "boot" and "follow".
The first will only return logs since last boot. The second will keep
the HTTP request open and send new log entries as they get added to the
systemd-journal.
* Allow Range header
Forward the Range header to systemd-journal-gatewayd. This allows to
select only a certain amount of log data. The Range header is a standard
header to select only partial amount of data. However, the "entries="
prefix is custom for systemd-journal-gatewayd, denoting that the numbers
following represent log entries (as opposed to bytes or other metrics).
* Avoid connecting if systemd-journal-gatewayd is not available
* Use path for all options
* Add pytests
* Address pylint issues
* Boot ID offsets and slug to identifier
* Fix tests
* API refactor from feedback
* fix tests and add identifiers
* stop isort and pylint fighting
* fix tests
* Update default log identifiers
* Only modify /host/logs endpoints
* Fix bad import
* Load log caches asynchronously at startup
* Allow task to complete in fixture
* Boot IDs and identifiers loaded on demand
* Add suggested identifiers
* Fix tests around boot ids
Co-authored-by: Mike Degatano <michael.degatano@gmail.com>
Mike DegatanoandGitHub
Jan ČermákandGitHub
Petar PetrovandGitHub
Pascal VizeliandGitHub