Commit Graph

43 Commits

Author SHA1 Message Date
Dominik 1f9cb007cd Port API tests from BATS to pytest and fix ResponseVerifyer
Move all API-related tests (HTTP endpoints, config validation, auth,
search, history, lists, Lua pages, OpenAPI spec validation) from BATS
shell tests to pytest. BATS retains DNS, regex, CLI, and system-level
tests. The auth test suite now removes the password at the end, leaving
no net state change.

Fix ResponseVerifyer to respect the OpenAPI "required" field: optional
properties absent from FTL's response are silently skipped instead of
flagged as errors. The required list is propagated through all recursive
calls (top-level objects, allOf, nested objects, array items).

Add a set_config() helper that changes FTL configuration via the API
instead of shelling out to the CLI. The API PATCH is synchronous, so no
log polling or sleeps are needed.

Add build.sh test-api target to run only the pytest API tests (skips
BATS and perf tests).

Test migration mapping (25 BATS tests removed, all covered in pytest):

| # | Removed BATS test | Pytest counterpart |
|---|---|---|
| 1 | HTTP server responds with JSON error 404 to unknown API path | TestHTTPErrors::test_api_404_returns_json |
| 2 | HTTP server responds with error 404 to path outside /admin | TestHTTPErrors::test_non_admin_path_returns_404 |
| 3 | Config validation working on the API (type-based checking) | TestConfigValidationAPIType (2 tests) |
| 4 | Config validation working on the API (validator-based checking) | TestConfigValidationAPIValidator (4 tests) |
| 5 | Changing a config option set forced by ENVVAR is not possible via the API | TestEnvvarProtectedConfig::test_api_rejects_envvar_override |
| 6 | API domain search: Non-existing domain | TestDomainSearch::test_nonexistent_domain |
| 7 | API domain search: antigravity.ftl | TestDomainSearch::test_antigravity_domain |
| 8 | API domain search: Internationalized/partially capital domain | TestDomainSearch::test_punycode_normalization |
| 9 | API history: Returns full 24 hours | TestHistory::test_history_returns_24h |
| 10 | API history/clients: Returns full 24 hours | TestHistory::test_history_clients_returns_24h |
| 11 | Check /api/lists?type=block | TestLists::test_block_lists_only |
| 12 | Check /api/lists?type=allow | TestLists::test_allow_lists_only |
| 13 | Check /api/lists without type parameter | TestLists::test_all_lists_includes_both_types |
| 14 | API: No UNKNOWN reply in API | TestQueries::test_no_unknown_reply |
| 15 | API: No UNKNOWN status in API | TestQueries::test_no_unknown_status |
| 16 | Lua server page outside /admin is not served by default | TestLuaServerPages::test_lua_page_outside_admin_not_served_by_default |
| 17 | Lua server page is generating proper backtrace | TestLuaServerPages::test_lua_page_generates_proper_backtrace |
| 18 | Lua server page outside of webhome is served without login | TestLuaServerPages::test_lua_page_outside_webhome_served_without_login |
| 19 | API validation (checkAPI.py) | TestEndpointCoverage (3 tests) + TestEndpointResponses + TestTeleporter |
| 20 | API authorization (without password): No login required | TestAuthWorkflow::test_01_no_password_means_session_valid |
| 21 | Create, set, and use application password | TestAuthWorkflow::test_02 + test_03 + test_04 |
| 22 | CLI password file is as expected | TestAuthWorkflow::test_04b_cli_password_file |
| 23 | API authorization: Setting password | TestAuthWorkflow::test_05_set_password |
| 24 | API authorization (with password): Incorrect password is rejected | TestAuthWorkflow::test_06_incorrect_password_rejected |
| 25 | API authorization (with password): Correct password is accepted | TestAuthWorkflow::test_07_correct_password_accepted |

New tests without a BATS predecessor:
- TestAuthWorkflow::test_08_rate_limiting_enforced
- TestAuthWorkflow::test_09_remove_password
- TestAuthWorkflow::test_10_no_password_after_removal

Final count: 192 BATS + 38 pytest = 230 total (was 216 BATS). No tests
lost, 14 net new.

Signed-off-by: Dominik <dl6er@dl6er.de>
2026-03-25 08:51:04 +01:00
Dominik 868733abb4 Check for allOf definitions in an array defining arrays of objects where all components must be checked. Also remove a bit pointless comparison between YAML examples and FTL output. We already validate the YAML examples against the YAML schema which is enough. This avoids false-positives when FTL happens to return a different number of items than we have examples (which is perfectly fine)
Signed-off-by: Dominik <dl6er@dl6er.de>
2025-12-14 18:47:45 +01:00
DL6ER 5d3972a54d Add ability to trace API validator script (off by default)
Signed-off-by: DL6ER <dl6er@dl6er.de>
2025-06-01 07:53:41 +02:00
DL6ER dd39c5eb0c Expose EDE information via API if available
Signed-off-by: DL6ER <dl6er@dl6er.de>
2024-12-18 10:58:44 +01:00
DL6ER 35166fd00c Fix a small bug in the API response verifier and ensure we always favor using 64 bit interface statistics if available. The reason is that the legacy statistics use 32 bit conters which overflow every 4 GB of interface traffic
Signed-off-by: DL6ER <dl6er@dl6er.de>
2024-07-14 15:45:29 +02:00
DL6ER f2a7662e95 Be more verbose in which tables are imported during teleporter importing
Signed-off-by: DL6ER <dl6er@dl6er.de>
2024-04-02 21:59:45 +02:00
DL6ER e3c4dcf215 Restructure API response from /history/clients and /history/database/clients to allow for sparse data.
Add new config option webserver.api.client_history_global_max controling if the activities chart should sort and show the *global* (integrated over 24 hours) or the `local` (measured individually in each time slot) most active clients
Allow setting webserver.api.maxClients to 0 to always return all clients in /api/history/clients

Signed-off-by: DL6ER <dl6er@dl6er.de>
2024-02-16 17:15:20 +01:00
Dominik 933e6f605f Merge pull request #1728 from pi-hole/tweak/query_auth
Add authentication via query string
2024-01-07 07:50:58 +01:00
DL6ER 2dbb7f3b63 Merge pull request #1731 from pi-hole/fix/dhcp-range
Improve DHCP handling
2023-11-16 22:25:09 +01:00
DL6ER 503c0538ed IPv4 address 0.0.0.0 and IPv6 address :: correspond to empty strings in FTL settings
Signed-off-by: DL6ER <dl6er@dl6er.de>
2023-11-16 12:43:47 +01:00
DL6ER a69130585e Use package ipaddress for IP address validation in API tests
Signed-off-by: DL6ER <dl6er@dl6er.de>
2023-11-15 11:49:03 +01:00
DL6ER 90de6c1b83 Merge pull request #1727 from pi-hole/fix/toml_utf8
Declare pihole.toml as UTF-8 document
2023-11-14 22:25:04 +01:00
DL6ER 11127f0f13 Fix OpenAPI checker not being able to discover properties that are returned by FTL but not documented in the OpenAPI specs
Signed-off-by: DL6ER <dl6er@dl6er.de>
2023-11-04 17:11:37 +01:00
DL6ER 75cd372d0e Add string format verification in API checker
Signed-off-by: DL6ER <dl6er@dl6er.de>
2023-11-04 12:38:24 +01:00
DL6ER a96c283c0c Add authentication via query string
Signed-off-by: DL6ER <dl6er@dl6er.de>
2023-11-03 19:41:08 +01:00
DL6ER f1c59db0f6 Declare pihole.toml as UTF-8 document. we add a compile-time switch to use ASCII with UTF-8 escaping instead in case this is a necessity for anyone (I don't really expect this in the third millenial but we also know people are still using Windows XP on the web...)
Signed-off-by: DL6ER <dl6er@dl6er.de>
2023-11-03 10:12:07 +01:00
DL6ER 15ae21e39c Clarify error wording
Signed-off-by: DL6ER <dl6er@dl6er.de>
2023-11-02 05:55:27 +01:00
DL6ER 3a67a77a08 Report number of checked endpoints in the result and warn if the number of specified endpoints in FTL and the OpenAPI specs do not match
Signed-off-by: DL6ER <dl6er@dl6er.de>
2023-11-02 05:50:32 +01:00
DL6ER 5993b66bca Add checking of all endpoints defined in FTL but not in the OpenAPI specs and vice versa
Signed-off-by: DL6ER <dl6er@dl6er.de>
2023-11-02 05:28:07 +01:00
DL6ER f4e6dfc061 Implement deep-recursion of API arrays
Signed-off-by: DL6ER <dl6er@dl6er.de>
2023-10-30 07:42:51 +01:00
DL6ER 2141db3d64 Add rate-limiting on password login attempts
Signed-off-by: DL6ER <dl6er@dl6er.de>
2023-10-07 19:59:32 +02:00
DL6ER 813509841b Accept cookie authentication only when CSRF header is provided (and correct)
Signed-off-by: DL6ER <dl6er@dl6er.de>
2023-06-04 19:29:54 +02:00
Christian König 62cfc25b95 Fix spelling in v6
Signed-off-by: Christian König <ckoenig@posteo.de>
2023-05-30 22:42:13 +02:00
DL6ER 19c72d354e !!! BREAKING CHANGE !!! Switch to the proven memory-hard password-hashing alogorithm BALLOON. The stored password hash will be upgraded on the first successdful login. To wave the necessity to implement BALLOON with every client trying to access the API, we remove the existing challenge-response authentication in favor of allowing login straight with the password. This has been avoided in the past, however, seems now acceptable that FTL (even by default) offers secure end-to-end encryption over HTTPS.
Signed-off-by: DL6ER <dl6er@dl6er.de>
2023-05-30 21:22:45 +02:00
DL6ER 149ec4e0dd Add test for re-importing the just exported Teleporter file during the tests
Signed-off-by: DL6ER <dl6er@dl6er.de>
2023-01-26 20:11:55 +01:00
DL6ER 48fc06d46b Add POST /api/teleporter to upload and install backed up configuration
Signed-off-by: DL6ER <dl6er@dl6er.de>
2023-01-25 21:51:12 +01:00
DL6ER e4383775d1 Add /api/action/gravity which can be used to trigger a run of pihole -g. The output is live streamed using HTTP/1.1 chunked encoding.
Signed-off-by: DL6ER <dl6er@dl6er.de>
2023-01-24 20:15:45 +01:00
DL6ER 13168c377b Add GET /api/teleporter
Signed-off-by: DL6ER <dl6er@dl6er.de>
2023-01-23 21:56:11 +01:00
DL6ER 88e8ab9fd5 !!! BREAKING CHANGE !!! Redesign TOML config structure
Signed-off-by: DL6ER <dl6er@dl6er.de>
2023-01-22 12:32:10 +01:00
DL6ER 47ac129a53 !!! BREAKING CHANGE !!! Rename pihole-FTL.toml to pihole.toml and it is a Pi-hole wide config file also covering all the dnsmasq settings, etc.
Signed-off-by: DL6ER <dl6er@dl6er.de>
2023-01-22 10:01:54 +01:00
DL6ER 2d6c25d573 Add GET /config/{element} for more specific requests as well as PATCH and DELETE /config/{element}/{value} for direct array manipulation
Signed-off-by: DL6ER <dl6er@dl6er.de>
2023-01-18 18:19:19 +01:00
DL6ER c34975180e Rename /api/ftl/endpoints -> /api/endpoints
Signed-off-by: DL6ER <dl6er@dl6er.de>
2023-01-16 21:37:11 +01:00
DL6ER 5e96022e63 Group endpoints in /api/ftl/endpoints by supported methods
Signed-off-by: DL6ER <dl6er@dl6er.de>
2023-01-16 21:11:57 +01:00
DL6ER 140a365806 Tests: Set api.pwhash and dns.blocking.mode using PATCH /api/config
Signed-off-by: DL6ER <dl6er@dl6er.de>
2023-01-08 21:29:55 +01:00
DL6ER f199ac2f08 Automatically migrate the API password hash and the lists of clients and domains to be excluded from setupVars.conf
Signed-off-by: DL6ER <dl6er@dl6er.de>
2023-01-08 21:21:38 +01:00
DL6ER 1414e0d397 Ensure checkAPI.py also accepts situations with localAPIauth = false
Signed-off-by: DL6ER <dl6er@dl6er.de>
2023-01-08 20:59:18 +01:00
DL6ER 4ac52263e9 Implement login for python API checking script
Signed-off-by: DL6ER <dl6er@dl6er.de>
2023-01-08 16:04:26 +01:00
DL6ER 3c5c1d52b7 Transform log_debug() into a function-like macro to save some time when we are not in debugging mode (the function is not called in this case)
Signed-off-by: DL6ER <dl6er@dl6er.de>
2023-01-08 09:55:51 +01:00
DL6ER b8354c515c Fix debug flag parsing after changing from bit-wise flags to individual bools
Signed-off-by: DL6ER <dl6er@dl6er.de>
2023-01-07 20:08:38 +01:00
DL6ER 8efd253529 Merge remote-tracking branch 'origin/development' into new/http
Signed-off-by: DL6ER <dl6er@dl6er.de>
2023-01-07 18:37:13 +01:00
DL6ER e145d20d28 Rewrite the entire config-related code to allow for changing data without having to restart. Hereby, we greatly reduce code duplication in the TOML routines so we won't have to touch tme in the future when adding additional options.
Signed-off-by: DL6ER <dl6er@dl6er.de>
2023-01-07 18:06:41 +01:00
DL6ER 6d7239dbc8 Add example verification when (possibly multiple) global examples are provided below the schema level
Signed-off-by: DL6ER <dl6er@dl6er.de>
2023-01-03 22:08:55 +01:00
DL6ER f2d68f20d7 Also verify endpoint structure: Query endpoints from FTL and check if all properties mentioned in the docs are present (and of correct type) and that there are no extra properties we forgot to document. Furthermore, also verify that the provided examples are of correct type, too.
Signed-off-by: DL6ER <dl6er@dl6er.de>
2023-01-03 21:51:13 +01:00