2ac91560e7
database: Guard query-count counters against uint64_t underflow
...
memdb_queries_count and diskdb_queries_count are declared as uint64_t.
In delete_old_queries_from_db(), both counters are decremented by
`deleted` (int64_t from sqlite3_changes64). If the counter is already
at 0 — which can happen when import_queries_from_disk() fails and
resets the counter to 0 mid-import — subtracting any positive `deleted`
value wraps the counter to UINT64_MAX (≈1.84×10¹⁹).
db_counts() returns these counters as sqlite3_int64 (signed), so
UINT64_MAX becomes -1. The /api/queries endpoint then casts -1 to
unsigned long for recordsTotal, producing the observed
1.8446744073709552E+19 value and making the query log appear to have
millions of pages.
Fix: clamp the subtraction to 0 instead of allowing underflow, using
a guarded conditional assignment.
Signed-off-by: Dominik <dl6er@dl6er.de >
2026-03-20 15:56:30 +01:00
5dbeeb8e60
Always mark database import as done even when database.DBimport = false or on database errors. Otherwise, garbage collection will never run, causing the memory allocation to always grow and overtime information to be incorrect (handled in GC code as well)
...
Signed-off-by: Dominik <dl6er@dl6er.de >
2026-02-26 07:22:53 +01:00
0789ad27bd
Merge branch 'development' into tweak/async_import
...
Signed-off-by: Dominik <dl6er@dl6er.de >
2026-02-09 19:10:42 +01:00
a8872e636a
Cache earliest timestamp in both databases (mem and disk) as well
...
Signed-off-by: Dominik <dl6er@dl6er.de >
2026-02-08 10:04:33 +01:00
ba2db9c10f
Keep track of number of queries in the database ourselves. No need to fully count the number of queries in the disk after startup. Even though this is happening using a covering index, it turned out to be very slow on heavily I/O limited devices when the index tree does not fit into available cache memory (scanning can take upwards of one minute for a simple SELECT COUNT(*) FROM query_storage)
...
Signed-off-by: Dominik <dl6er@dl6er.de >
2026-01-10 17:33:20 +01:00
6ef8b074c4
Simplify delete_old_queries_in_DB() algorithm and reduce nesting in disk filesize getting
...
Signed-off-by: Dominik <dl6er@dl6er.de >
2026-01-07 19:36:02 +01:00
9aa6daf84d
Warn if too few queries were imported
...
Signed-off-by: Dominik <dl6er@dl6er.de >
2025-12-13 11:37:04 +01:00
d63247b067
Remove the compile-time option to batch locking during import. We are not using it. It adds unecessary maintainance workload
...
Signed-off-by: Dominik <dl6er@dl6er.de >
2025-12-13 09:59:32 +01:00
a3d0abaf55
Do not use a long open transaction as it may have side-effects on really slow machines when other threads are trying to work with the database
...
Signed-off-by: Dominik <dl6er@dl6er.de >
2025-12-13 09:35:15 +01:00
f8643a8f3d
Rearrange initial database interactions to avoid blocking. Also, move some uncritical parts into the corresponding threads for more parallelism.
...
Signed-off-by: Dominik <dl6er@dl6er.de >
2025-12-11 17:47:00 +01:00
4d3bae5c4f
Ensure stmt2 cannot be uninitialized upon failure
...
Signed-off-by: Dominik <dl6er@dl6er.de >
2025-12-08 19:52:33 +01:00
a690bee3d2
Fix a few possible SQLite3-related memory leaks on user errors and simplify code by removing unnecessary reset calls
...
Signed-off-by: Dominik <dl6er@dl6er.de >
2025-12-08 17:22:12 +01:00
07e1e4b07c
Also make the actual importing from disk into memory (happening before the parsing) taking place asynchronously.
...
Signed-off-by: Dominik <dl6er@dl6er.de >
2025-12-08 17:14:52 +01:00
6f436651c1
Queries already imported from disk should not be re-exported as these queries cannot receive any further updates. This is a performance optimization mostly relevant for very active (= many queries per second) systems.
...
Signed-off-by: Dominik <dl6er@dl6er.de >
2025-12-08 17:02:57 +01:00
cfcef205ac
Do not filter on the history windows twice. This can lead to missing queries in the second run, causing "empty" query slots. They are - themselves - entirely harmless, however, they will cuase "memory error" warnings until they are garbage collected away after a few minutes.
...
Signed-off-by: Dominik <dl6er@dl6er.de >
2025-12-08 17:01:44 +01:00
80e14ec11d
Queries are not recycled so we also don't need to check against a query having been recycled
...
Signed-off-by: Dominik <dl6er@dl6er.de >
2025-12-07 22:08:58 +01:00
184f80a2bf
Queries should not be counted twice during async import
...
Signed-off-by: Dominik <dl6er@dl6er.de >
2025-12-05 19:49:22 +01:00
b8bdf7f2df
Log when queries are imported based on the really imported queries
...
Signed-off-by: Dominik <dl6er@dl6er.de >
2025-12-02 20:32:02 +01:00
fd6eaabfe8
Remove pre-allocation of domains, clients, and upstreams as we will not import all of them from disk during import. This will make the SHM objects unnecessarily large - they can grow naturally during importing
...
Signed-off-by: Dominik <dl6er@dl6er.de >
2025-12-01 20:18:16 +01:00
6cd4622b60
Optionally batch querys during import to reduce locking overhead. This is experimental and disabled for now as it will reduce the time being unlocked, i.e., it reduces the number of ocassions where normal query handling could take place (which is the entire point about implementing async importing)
...
Signed-off-by: Dominik <dl6er@dl6er.de >
2025-12-01 20:17:47 +01:00
a27b105d3b
Postpose GC runs until async database import is done
...
Signed-off-by: Dominik <dl6er@dl6er.de >
2025-12-01 20:11:14 +01:00
dd4e4af691
Ensure commiting the transaction to the in-memory database is happening outside of the SHM lock. This is not really needed for the in-memory database, however, it may be an issue when users use the new forceDisk feature
...
Signed-off-by: Dominik <dl6er@dl6er.de >
2025-11-28 19:14:36 +01:00
f5b401a46e
Add database.forceDisk to force FTL into a low-memory mode
...
Signed-off-by: Dominik <dl6er@dl6er.de >
2025-11-28 18:59:11 +01:00
5e49b8ba5a
Implement look-ahead shared memory resizing and move database initialization into a dedicated asynchronous thread
...
Signed-off-by: Dominik <dl6er@dl6er.de >
2025-11-25 19:41:10 +01:00
144e0d9adf
Merge branch 'development' into tweak/db_performance
...
Signed-off-by: Dominik <dl6er@dl6er.de >
2025-11-25 19:40:11 +01:00
5d20b9f97d
Merge pull request #2700 from pi-hole/tweak/dont-lock-on-export
...
Reduce DNS resolver locking during database interaction
2025-11-21 16:49:03 +01:00
0b2b6dff0b
Increase buffer length for query string
...
Signed-off-by: Michael Woolweaver <michael@woolweaver.bid >
2025-11-11 06:53:33 -06:00
5df02b5578
Expands the get_number_of_queries_in_DB() function to optionally return
...
the earliest timestamp found in the database.
Also gets number of queries/earliest timestamp in on-disk database
Useful for determining the "All Time" range on the web interface
Signed-off-by: Adam Warner <me@adamwarner.co.uk >
2025-11-09 14:37:43 +00:00
719eb57f44
Limit the scan table size when exporting queries to disk. This will mean speedup espectially when only few entries are exported (typical case)
...
Signed-off-by: Dominik <dl6er@dl6er.de >
2025-11-08 14:03:18 +01:00
6a141f620d
We do not need to lock the SHM objects during export to disk database inside a TRANSACTION
...
Signed-off-by: Dominik <dl6er@dl6er.de >
2025-11-08 14:02:53 +01:00
9423573404
We do not need to loch the SHM objects during export to disk database inside a TRANSACTION
...
Signed-off-by: Dominik <dl6er@dl6er.de >
2025-11-08 12:16:01 +01:00
d6c56f5f64
Configure a database busy handler giving us control (and debugging capabilities) of what happens while the database is busy.
...
Signed-off-by: DL6ER <dl6er@dl6er.de >
2025-10-04 09:15:46 +02:00
92d4383e36
Merge branch 'development' into fix/insertion_counting
...
Signed-off-by: Dominik <dl6er@dl6er.de >
2025-09-26 19:37:47 +02:00
95fdc34e76
Automatic code review
...
Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com>
Signed-off-by: Dominik <DL6ER@users.noreply.github.com >
2025-09-24 19:58:12 +02:00
f8d9b5ea42
Update outdated CodeQL dependencies to allow running them
...
Signed-off-by: Dominik <dl6er@dl6er.de >
2025-09-24 19:58:12 +02:00
f34058fd4c
Ensure queries with ID 0 are stored to the long-term queries database
...
Signed-off-by: Dominik <dl6er@dl6er.de >
2025-09-24 19:58:12 +02:00
5e6fe84f30
Apply suggestions from code review
...
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com >
Signed-off-by: Dominik <DL6ER@users.noreply.github.com >
2025-09-17 09:15:03 +02:00
8413b161a2
Sunset last_disk_db_idx and instead realize the same with a subquery
...
Signed-off-by: DL6ER <dl6er@dl6er.de >
2025-07-27 11:41:44 +02:00
00c520190a
Reuse prepared SQLite3 statements instead of repeating the ever-same compilation steps over and over. So far, this was not an issue as this doesn't happen often but - anyway - it is needlessly duplicated work.
...
Signed-off-by: DL6ER <dl6er@dl6er.de >
2025-07-25 18:39:54 +02:00
dc16d6c4b7
Avoid double-incrementing the last_disk_db_idx counter. This leads to a situation where some queries are not stored in the on-disk database. How much this affects you depends on the activity of the aforegoing database storing interval and, hence, is rather subtle.
...
Signed-off-by: DL6ER <dl6er@dl6er.de >
2025-07-25 18:35:11 +02:00
b96e5294d6
Do not store queries when database.maxDBdays is == 0, however, we still want to store the remaining meta-tables for the network table to work
...
Signed-off-by: DL6ER <dl6er@dl6er.de >
2025-03-11 04:59:11 +01:00
4c98b5d33d
Non-queries are not worth warnings during database importing / history restoration
...
Signed-off-by: DL6ER <dl6er@dl6er.de >
2025-02-06 06:04:55 +01:00
1eaa9ea0f0
Improve handling of non-queries. This has recently been fixed in dnsmasq (see commit 8132969, Dec, 19, 2024)
...
Signed-off-by: DL6ER <dl6er@dl6er.de >
2025-01-24 05:13:12 +01:00
a47c35e971
Prevent a few code paths from running when we know the disk database is not available (e.g. broken or more recent than this particular FTL binary can handle (like checking out a very old branch))
...
Signed-off-by: DL6ER <dl6er@dl6er.de >
2025-01-03 22:14:27 +01:00
dd39c5eb0c
Expose EDE information via API if available
...
Signed-off-by: DL6ER <dl6er@dl6er.de >
2024-12-18 10:58:44 +01:00
787021b1cf
Try to use DNS cache ID if known to avoid searching when possible
...
Signed-off-by: DL6ER <dl6er@dl6er.de >
2024-10-21 20:05:11 +02:00
9a9615ad78
Set lastQuery property of CNAME child domains
...
Signed-off-by: DL6ER <dl6er@dl6er.de >
2024-10-19 19:05:49 +02:00
93a06c0124
Remove unused code segments
...
Signed-off-by: DL6ER <dl6er@dl6er.de >
2024-10-15 21:36:43 +02:00
2f8355ed25
Further code maintainance (adding "const", etc.)
...
Signed-off-by: DL6ER <dl6er@dl6er.de >
2024-10-14 19:27:52 +02:00
33645777e9
Add basic lookup table implementation and create new SHM objects for them. Also simplify code by changing index counters to unsigned types.
...
Signed-off-by: DL6ER <dl6er@dl6er.de >
2024-10-12 22:01:57 +02:00
Dominik
Dominik
Michael Woolweaver
Adam Warner