* Return cursor of the first host logs entry via headers
Return first entry's cursor via custom `X-First-Cursor` header that can
be consumed by the client and used for continual requesting of the
historic logs. Once the first fetch returns data, the cursor can be
supplied as the first argument to the Range header in another call,
fetching accurate slice of the journal with the previous log entries
using the `Range: entries=cursor[[:num_skip]:num_entries]` syntax.
Let's say we fetch logs with the Range header `entries=:-19:20` (to
fetch very last 20 lines of the logs, see below why not
`entries:-20:20`) and we get `cursor50` as the reply (the actual value
will be much more complex and with no guaranteed format). To fetch
previous slice of the logs, we use `entries=cursor50:-20:20`, which
would return 20 lines previous to `cursor50` and `cursor30` in the
cursor header. This way we can go all the way back to the history.
One problem with the cursor is that it's not possible to determine when
the negative num_skip points beyond the first log entry. In that case
the client either needs to know what the first entry is (via
`entries=:0:1`) or can iterate naively and stop once two subsequent
requests return the same first cursor.
Another caveat, even though it's unlikely it will be hit in real usage,
is that it's not possible to fetch the last line only - if no cursor is
provided, negative num_skip argument is needed, and in that case we're
pointing one record back from the current cursor, which is the previous
record. The least we can return without knowing any cursor is thus
`entries=:-1:2` (where the `2` can be omitted, however with
`entries=:-1:1` we would lose the last line). This also explains why
different `num_skip` and `num_entries` must be used for the first fetch.
* Fix typo (fallback->callback)
* Refactor journal_logs_reader to always return the cursor
* Update tests for new cursor handling
* Use Journal Export Format for host (advanced) logs
Add methods for handling Journal Export Format and use it for fetching
of host logs. This is foundation for colored streaming logs for other
endpoints as well.
* Make pylint happier - remove extra pass statement
* Rewrite journal gateway tests to mock ClientResponse's StreamReader
* Handle connection refused error when connecting to journal-gatewayd
* Use SYSTEMD_JOURNAL_GATEWAYD_SOCKET global path also for connection
* Use parsing algorithm suggested by @agners in review
* Fix timestamps in formatting, always use UTC for now
* Add tests for Accept header in host logs
* Apply suggestions from @agners
Co-authored-by: Stefan Agner <stefan@agner.ch>
* Bail out of parsing earlier if field is not in required fields
* Fix parsing issue discovered in the wild and add test case
* Make verbose formatter more tolerant
* Use some bytes' native functions for some minor optimizations
* Move MalformedBinaryEntryError to exceptions module, add test for it
---------
Co-authored-by: Stefan Agner <stefan@agner.ch>
* Migrate to Ruff for lint and format
* Fix pylint issues
* DBus property sets into normal awaitable methods
* Fix tests relying on separate tasks in connect
* Fixes from feedback
* Add enhanced logging REST endpoints using systemd-journal-gatewayd
Add /host/logs/entries and /host/logs/{identifier}/entries to expose log
entries from systemd-journald running on the host. Use
systemd-journal-gatewayd which exposes the logs to the Supervisor via
Unix socket.
Current two query string parameters are allowed: "boot" and "follow".
The first will only return logs since last boot. The second will keep
the HTTP request open and send new log entries as they get added to the
systemd-journal.
* Allow Range header
Forward the Range header to systemd-journal-gatewayd. This allows to
select only a certain amount of log data. The Range header is a standard
header to select only partial amount of data. However, the "entries="
prefix is custom for systemd-journal-gatewayd, denoting that the numbers
following represent log entries (as opposed to bytes or other metrics).
* Avoid connecting if systemd-journal-gatewayd is not available
* Use path for all options
* Add pytests
* Address pylint issues
* Boot ID offsets and slug to identifier
* Fix tests
* API refactor from feedback
* fix tests and add identifiers
* stop isort and pylint fighting
* fix tests
* Update default log identifiers
* Only modify /host/logs endpoints
* Fix bad import
* Load log caches asynchronously at startup
* Allow task to complete in fixture
* Boot IDs and identifiers loaded on demand
* Add suggested identifiers
* Fix tests around boot ids
Co-authored-by: Mike Degatano <michael.degatano@gmail.com>
