fbee810f52
fix: correct uint64 underflow in performance timing macros
...
When ts1.tv_nsec < ts0.tv_nsec (nanosecond counter wraps across a second
boundary), the unsigned subtraction produced a huge positive value instead
of a small negative one. Compute the full difference in signed int64_t
nanoseconds first, then convert to unsigned microseconds.
Signed-off-by: Dominik <dl6er@dl6er.de >
2026-03-21 21:41:30 +01:00
a50ebb47d4
Add debug.performance
...
Signed-off-by: Dominik <dl6er@dl6er.de >
2026-03-21 21:36:54 +01:00
2ac91560e7
database: Guard query-count counters against uint64_t underflow
...
memdb_queries_count and diskdb_queries_count are declared as uint64_t.
In delete_old_queries_from_db(), both counters are decremented by
`deleted` (int64_t from sqlite3_changes64). If the counter is already
at 0 — which can happen when import_queries_from_disk() fails and
resets the counter to 0 mid-import — subtracting any positive `deleted`
value wraps the counter to UINT64_MAX (≈1.84×10¹⁹).
db_counts() returns these counters as sqlite3_int64 (signed), so
UINT64_MAX becomes -1. The /api/queries endpoint then casts -1 to
unsigned long for recordsTotal, producing the observed
1.8446744073709552E+19 value and making the query log appear to have
millions of pages.
Fix: clamp the subtraction to 0 instead of allowing underflow, using
a guarded conditional assignment.
Signed-off-by: Dominik <dl6er@dl6er.de >
2026-03-20 15:56:30 +01:00
601880fea4
Update embedded SQLite3 to 3.51.3
...
Signed-off-by: Dominik <dl6er@dl6er.de >
2026-03-14 17:57:29 +01:00
80ddb8c2cd
Revert "Update embedded SQLite3 to 3.52.0"
2026-03-14 17:52:34 +01:00
b0419a9651
Merge pull request #2791 from pi-hole/fix/groups_by_name
...
Fix obtaining client groups by name
2026-03-10 12:02:16 +01:00
caa02e8233
Merge pull request #2790 from pi-hole/fix/names_behind_router
...
Add new option for controling name resolution via MAC address
2026-03-09 22:06:38 +01:00
dcff36ede3
Update embedded SQLite3 to 3.52.0
...
Signed-off-by: Dominik <dl6er@dl6er.de >
2026-03-06 20:33:08 +01:00
c5622e86b6
Return early in find_device_by_hwaddr() if resolver.macNames is false
...
Signed-off-by: Dominik <dl6er@dl6er.de >
2026-03-05 19:30:42 +01:00
1df6676536
Fix obtaining client groups by name
...
Signed-off-by: Dominik <dl6er@dl6er.de >
2026-02-27 13:13:56 +01:00
e53bfc6fc1
Add config option resolver.macNames which can be used to control whether FTL tries to resolve host names via MAC addresses. This may not work in special network configurations where clients aren't all on the same link (Lay2 but not Layer 3 connection)
...
Signed-off-by: Dominik <dl6er@dl6er.de >
2026-02-27 12:37:55 +01:00
5dbeeb8e60
Always mark database import as done even when database.DBimport = false or on database errors. Otherwise, garbage collection will never run, causing the memory allocation to always grow and overtime information to be incorrect (handled in GC code as well)
...
Signed-off-by: Dominik <dl6er@dl6er.de >
2026-02-26 07:22:53 +01:00
0789ad27bd
Merge branch 'development' into tweak/async_import
...
Signed-off-by: Dominik <dl6er@dl6er.de >
2026-02-09 19:10:42 +01:00
05ca076ff3
Merge pull request #2757 from pi-hole/tweak/track_sqlite_memory
...
Low-memory hardware optimizations
2026-02-08 21:13:53 +01:00
a8872e636a
Cache earliest timestamp in both databases (mem and disk) as well
...
Signed-off-by: Dominik <dl6er@dl6er.de >
2026-02-08 10:04:33 +01:00
ba2db9c10f
Keep track of number of queries in the database ourselves. No need to fully count the number of queries in the disk after startup. Even though this is happening using a covering index, it turned out to be very slow on heavily I/O limited devices when the index tree does not fit into available cache memory (scanning can take upwards of one minute for a simple SELECT COUNT(*) FROM query_storage)
...
Signed-off-by: Dominik <dl6er@dl6er.de >
2026-01-10 17:33:20 +01:00
90313fdaf4
Update SQLite3 to 3.51.2
...
Signed-off-by: Dominik <dl6er@dl6er.de >
2026-01-10 11:04:45 +01:00
b65de38a62
Use localtime_r() instead of localtime() for thread-safety reasons
...
Signed-off-by: Dominik <dl6er@dl6er.de >
2026-01-08 20:23:41 +01:00
dbdb67005d
Delete old queries once in the night instead of after each GC run
...
Signed-off-by: Dominik <dl6er@dl6er.de >
2026-01-08 19:42:29 +01:00
6ef8b074c4
Simplify delete_old_queries_in_DB() algorithm and reduce nesting in disk filesize getting
...
Signed-off-by: Dominik <dl6er@dl6er.de >
2026-01-07 19:36:02 +01:00
589649d6ce
Merge branch 'development' into tweak/track_sqlite_memory
...
Signed-off-by: Dominik <dl6er@dl6er.de >
2026-01-04 19:04:37 +01:00
be247cfda1
Add new misc.hide_connection_error setting defaulting to false
...
Signed-off-by: Dominik <dl6er@dl6er.de >
2025-12-21 12:14:30 +01:00
43c17efb7a
Fix gcc-15 caveat: {0} initializer in C or C++ for unions no longer guarantees clearing of the whole union (except for static storage duration initialization), it just initializes the first union member to zero.
...
Signed-off-by: Dominik <dl6er@dl6er.de >
2025-12-14 19:08:55 +01:00
8e7bcd48f9
Merge pull request #2601 from pi-hole/tweak/hostname_warn
...
Escape unprintable characters in invalid host names
2025-12-14 06:24:04 +01:00
88fef50bc0
Merge pull request #2727 from rrobgill/networktab
...
Network Overview - obtain MAC and hostname from dhcp.leases
2025-12-13 20:37:59 +01:00
9aa6daf84d
Warn if too few queries were imported
...
Signed-off-by: Dominik <dl6er@dl6er.de >
2025-12-13 11:37:04 +01:00
d63247b067
Remove the compile-time option to batch locking during import. We are not using it. It adds unecessary maintainance workload
...
Signed-off-by: Dominik <dl6er@dl6er.de >
2025-12-13 09:59:32 +01:00
a3d0abaf55
Do not use a long open transaction as it may have side-effects on really slow machines when other threads are trying to work with the database
...
Signed-off-by: Dominik <dl6er@dl6er.de >
2025-12-13 09:35:15 +01:00
f8643a8f3d
Rearrange initial database interactions to avoid blocking. Also, move some uncritical parts into the corresponding threads for more parallelism.
...
Signed-off-by: Dominik <dl6er@dl6er.de >
2025-12-11 17:47:00 +01:00
6d7da8d89f
Enable SQLite3 EXPLAIN comments
...
Signed-off-by: Dominik <dl6er@dl6er.de >
2025-12-08 19:52:11 +01:00
4d3bae5c4f
Ensure stmt2 cannot be uninitialized upon failure
...
Signed-off-by: Dominik <dl6er@dl6er.de >
2025-12-08 19:52:33 +01:00
a690bee3d2
Fix a few possible SQLite3-related memory leaks on user errors and simplify code by removing unnecessary reset calls
...
Signed-off-by: Dominik <dl6er@dl6er.de >
2025-12-08 17:22:12 +01:00
07e1e4b07c
Also make the actual importing from disk into memory (happening before the parsing) taking place asynchronously.
...
Signed-off-by: Dominik <dl6er@dl6er.de >
2025-12-08 17:14:52 +01:00
6f436651c1
Queries already imported from disk should not be re-exported as these queries cannot receive any further updates. This is a performance optimization mostly relevant for very active (= many queries per second) systems.
...
Signed-off-by: Dominik <dl6er@dl6er.de >
2025-12-08 17:02:57 +01:00
cfcef205ac
Do not filter on the history windows twice. This can lead to missing queries in the second run, causing "empty" query slots. They are - themselves - entirely harmless, however, they will cuase "memory error" warnings until they are garbage collected away after a few minutes.
...
Signed-off-by: Dominik <dl6er@dl6er.de >
2025-12-08 17:01:44 +01:00
80e14ec11d
Queries are not recycled so we also don't need to check against a query having been recycled
...
Signed-off-by: Dominik <dl6er@dl6er.de >
2025-12-07 22:08:58 +01:00
184f80a2bf
Queries should not be counted twice during async import
...
Signed-off-by: Dominik <dl6er@dl6er.de >
2025-12-05 19:49:22 +01:00
277a86402d
Merge branch 'development' into tweak/async_import
2025-12-05 19:06:26 +01:00
b8bdf7f2df
Log when queries are imported based on the really imported queries
...
Signed-off-by: Dominik <dl6er@dl6er.de >
2025-12-02 20:32:02 +01:00
fd6eaabfe8
Remove pre-allocation of domains, clients, and upstreams as we will not import all of them from disk during import. This will make the SHM objects unnecessarily large - they can grow naturally during importing
...
Signed-off-by: Dominik <dl6er@dl6er.de >
2025-12-01 20:18:16 +01:00
6cd4622b60
Optionally batch querys during import to reduce locking overhead. This is experimental and disabled for now as it will reduce the time being unlocked, i.e., it reduces the number of ocassions where normal query handling could take place (which is the entire point about implementing async importing)
...
Signed-off-by: Dominik <dl6er@dl6er.de >
2025-12-01 20:17:47 +01:00
a27b105d3b
Postpose GC runs until async database import is done
...
Signed-off-by: Dominik <dl6er@dl6er.de >
2025-12-01 20:11:14 +01:00
4714a378b6
Update embedded SQLIte3 to 3.51.1
...
Signed-off-by: Dominik <dl6er@dl6er.de >
2025-12-01 19:42:50 +01:00
dd4e4af691
Ensure commiting the transaction to the in-memory database is happening outside of the SHM lock. This is not really needed for the in-memory database, however, it may be an issue when users use the new forceDisk feature
...
Signed-off-by: Dominik <dl6er@dl6er.de >
2025-11-28 19:14:36 +01:00
f5b401a46e
Add database.forceDisk to force FTL into a low-memory mode
...
Signed-off-by: Dominik <dl6er@dl6er.de >
2025-11-28 18:59:11 +01:00
f1a4124817
Also log table sizes
...
Signed-off-by: Dominik <dl6er@dl6er.de >
2025-11-28 06:54:36 +01:00
bb048cc4c5
Print even more memory debugging information
...
Signed-off-by: Dominik <dl6er@dl6er.de >
2025-11-28 06:46:32 +01:00
c2c74c8b10
Implement lightweight SQLite3 memory tracking
...
Signed-off-by: Dominik <dl6er@dl6er.de >
2025-11-27 20:40:40 +01:00
c56a086813
database/network-table - obtain MAC and hostname from dhcp.leases
...
Insert option to search dhcp.leases for client details.
This only occurs if Pi-hole is the dhcp server, and if there is no mac
address available from arp cache or recent EDNS records.
If no record is present it hands over to the existing mock-hwaddr code.
The search only runs the first time the IP is seen. Once the entry is
populated it will not search the file again.
Signed-off-by: Rob Gill <rrobgill@protonmail.com >
2025-11-27 07:22:49 +10:00
e1231acd43
Merge branch 'development' into tweak/hostname_warn
...
Signed-off-by: Dominik <dl6er@dl6er.de >
2025-11-25 22:14:22 +01:00
Dominik
Dominik
Rob Gill