Update documentation for v5.0

Signed-off-by: DL6ER <dl6er@dl6er.de>

.markdownlint.json

@@ -9,6 +9,9 @@
   "MD007": {
     "indent": 4
   },
+  "MD012": {
+    "maximum": 2
+  },
   "MD013": false,
   "MD024": false,
   "MD025": false,
@@ -22,7 +25,8 @@
       "strong",
       "p",
       "sub",
-      "table", "tbody", "th", "tr", "td"
+      "table", "tbody", "th", "tr", "td",
+      "details", "summary"
     ]
   },
   "MD035": {

README.md

@@ -30,13 +30,14 @@ When working on this repo, it is advised that you review your changes locally be
 
 Please make sure you fork the repo and change the clone URL in the example below for your fork:
 
-- Linux Mint / Ubuntu instructions (tested on Linux Mint 18):
+- Linux Mint / Ubuntu 18.04 LTS / 19.10:
     - Preparations (only required once):
 
     ```bash
     git clone https://github.com/YOUR-USERNAME/docs
     cd docs
-    sudo pip install -r requirements.txt
+    sudo apt install python3-pip
+    sudo pip3 install -r requirements.txt
     ```
 
     - Running the docs server:

docs/database/ftl.md

@@ -0,0 +1,124 @@
+Pi-hole *FTL*DNS uses the well-known relational database management system SQLite3 as its long-term storage of query data. In contrast to many other database management solutions, *FTL*DNS does not need a server database engine as the database engine is directly embedded in *FTL*DNS. It seems an obvious choice as it is probably the most widely deployed database engine - it is used today by several widespread web browsers, operating systems, and embedded systems (such as mobile phones), among others. Hence, it is rich in supported platforms and offered features. SQLite implements most of the SQL-92 standard for SQL and can be used for high-level queries.
+
+The long-term query database was the first database that was added to the Pi-hole project.
+We update this database periodically and on the exit of *FTL*DNS (triggered e.g. by a `service pihole-FTL restart`). The updating frequency can be controlled by the parameter [`DBINTERVAL`](../ftldns/configfile.md#dbinterval) and defaults to once per minute. We think this interval is sufficient to protect against data losses due to power failure events. *FTL*DNS needs the database to populate its internal history of the most recent 24 hours. If the database is disabled, *FTL*DNS will show an empty query history after a restart.
+
+The location of the database can be configured by the config parameter [`DBFILE`](../ftldns/configfile.md#dbfile). It defaults to `/etc/pihole/pihole-FTL.db`. If the given file does not exist, *FTL*DNS will create a new (empty) database file.
+
+Another way of controlling the size of the long-term database is by setting a maximum age for log queries to keep using the config parameter [`MAXDBDAYS`](../ftldns/configfile.md#maxdbdays). It defaults to 365 days, i.e. queries that are older than one year get periodically removed to limit the growth of the long-term database file.
+
+The config parameter [`DBIMPORT`](../ftldns/configfile.md#dbimport) controls whether `FTL` loads information from the database on startup. It needs to do this to populate the internal data structure with the most recent history. However, as importing from the database on disk can delay FTL on very large deploys, it can be disabled using this option.
+
+---
+
+### Split database
+
+You can split your long-term database by periodically rotating the database file (do this only when `pihole-FTL` is *not* running). The individual database contents can easily be merged when required.
+This could be implemented by running a monthly `cron` job such as:
+
+```bash
+sudo service pihole-FTL stop
+sudo mv /etc/pihole/pihole-FTL.db /media/backup/pihole-FTL_$(date +"%m-%y").db
+sudo service pihole-FTL start
+```
+
+Note that DNS resolution will not be available as long as `pihole-FTL` is stopped.
+
+### Backup database
+
+The database can be backed up while FTL is running when using the SQLite3 Online backup method, e.g.,
+
+```bash
+sqlite3 /etc/pihole/pihole-FTL.db ".backup /home/pi/pihole-FTL.db.backup"
+```
+
+will create `/home/pi/pihole-FTL.db.backup` which is a copy of your long-term database.
+
+---
+
+The long-term database contains three tables:
+
+### Query Table
+
+Label | Type | Allowed to by empty | Content
+--- | --- | ---- | -----
+`id` | integer | No | autoincrement ID for the table, only used by SQLite3, not by *FTL*DNS
+`timestamp` | integer | No | Unix timestamp when this query arrived at *FTL*DNS (used as index)
+`type` | integer | No | Type of this query (see [Supported query types](ftl.md#supported-query-types))
+`status` | integer | No | How was this query handled by *FTL*DNS? (see [Supported status types](ftl.md#supported-status-types))
+`domain` | text | No | Requested domain
+`client` | text | No | Requesting client (IP address)
+`forward` | text | Yes | Forward destination used for this query (only set if `status == 2`)
+
+### Counters table
+
+This table contains counter values integrated over the entire lifetime of the table
+
+Label | Type | Allowed to by empty | Content
+--- | --- | ---- | -----
+`id` | integer | No | ID for the table used to select a counter (see below)
+`value` | integer | No | Value of a given counter
+
+ID | Interpretation
+--- | ---
+0 | Total number of queries
+1 | Total number of blocked queries
+
+### FTL table
+
+The FTL table contains some data used by *FTL*DNS for determining which queries to save to the database. This table does not contain any entries of general interest.
+
+SQLite3 syntax used to create this table:
+
+### Supported query types
+
+ID | Query Type
+--- | ---
+1 | A
+2 | AAAA
+3 | ANY
+4 | SRV
+5 | SOA
+6 | PTR
+7 | TXT
+
+<!-- ID | 1 | 2 | 3 | 4 | 5 | 6 | 7 -->
+<!-- -- | -- | -- | -- | -- | -- | -- | -- -->
+<!-- Query | A | AAAA | ANY | SRV | SOA | PTR | TXT -->
+
+### Supported status types
+
+ID | Status | | Details
+--- | --- | --- | ---
+0 | Unknown | ❔ | was not answered by forward destination
+1 | Blocked | ❌ | Domain contained in [gravity database](../database/gravity/index.md#gravity-table-gravity)
+2 | Allowed | ✅ | Forwarded
+3 | Allowed | ✅ | Known, replied to from cache
+4 | Blocked | ❌ | Domain matched by a [regex blacklist](../database/gravity/index.md#regex-table-regex) filter
+5 | Blocked | ❌ | Domain contained in [exact blacklist](../database/gravity/index.md#blacklist-table-blacklist)
+6 | Blocked | ❌ | By upstream server (known blocking page IP address)
+7 | Blocked | ❌ | By upstream server (`0.0.0.0` or `::`)
+8 | Blocked | ❌ | By upstream server (`NXDOMAIN` with `RA` bit unset)
+9 | Blocked | ❌ | Domain contained in [gravity database](../database/gravity/index.md#gravity-table-gravity)<br>*Blocked during deep CNAME inspection*
+10 | Blocked | ❌ | Domain matched by a [regex blacklist](../database/gravity/index.md#regex-table-regex) filter<br>*Blocked during deep CNAME inspection*
+11 | Blocked | ❌ | Domain contained in [exact blacklist](../database/gravity/index.md#blacklist-table-blacklist)<br>*Blocked during deep CNAME inspection*
+
+### Example for interaction with the long-term query database
+
+In addition to the interactions the Pi-hole database API offers, you can also run your own SQL commands against the database. If you want to obtain the three most queries domains for all time, you could use
+
+```bash
+sqlite3 "/etc/pihole/pihole-FTL.db" "SELECT domain,count(domain) FROM queries WHERE (STATUS == 2 OR STATUS == 3) GROUP BY domain ORDER BY count(domain) DESC LIMIT 3"
+```
+
+which would return something like
+
+```text
+discourse.pi-hole.net|421095
+www.pi-hole.net|132483
+posteo.de|130243
+```
+
+showing the domain and the number of times it was found in the long-term database. Note that such a request might take a very long time for computation as the entire history of queries has to be processed for this.
+
+{!abbreviations.md!}

docs/database/gravity/example.md

@@ -0,0 +1,264 @@
+# Per-client blocking example
+
+In this example, we describe how to set up a blocking rule for three specific clients. All remaining (and newly added) clients in the network are "unmanaged", i.e., they use Pi-hole as usual. The examples shown here are built upon each other, i.e., example 5 might make no sense without the context of example 3.
+
+Don't forget to run
+
+```bash
+pihole restartdns reload-lists
+```
+
+after your database modifications to have FTL flush its internal domain-blocking cache (separate from the DNS cache).
+
+## Prerequisites
+
+1. **Add three groups.**
+
+    The `Unassociated` group has a special meaning and cannot be deleted. All domains, clients, and adlists without a specific group assignment are automatically managed through this group. Disabling this group will disable Pi-hole blocking for all unmanaged devices.
+
+    ![Adding three groups](example-groups.png)
+
+    <details><summary>Raw database instructions</summary>
+    ```sql
+    INSERT INTO "group" (id, name) VALUES (1, 'Group 1');
+    INSERT INTO "group" (id, name) VALUES (2, 'Group 2');
+    INSERT INTO "group" (id, name) VALUES (3, 'Group 3');
+    ```
+    </details>
+
+2. **Add three clients.**
+
+    Add three clients at your will, their IP addresses might differ from the ones in this example.
+
+    ![Adding three clients](example-clients-1.png)
+
+    <details><summary>Raw database instructions</summary>
+    ```sql
+    INSERT INTO client (id, ip) VALUES (1, '192.168.0.101');
+    INSERT INTO client (id, ip) VALUES (2, '192.168.0.102');
+    INSERT INTO client (id, ip) VALUES (3, '192.168.0.103');
+    ```
+    </details>
+
+3. **Link the clients to the created groups.** Don't forget to save each assignment by clicking on the corresponding green pen icon in the same row.
+
+    ![Link groups and clients](example-clients-2.png)
+
+    <details><summary>Raw database instructions</summary>
+    ```sql
+    INSERT INTO client_by_group (client_id, group_id) VALUES (1, 1);
+    INSERT INTO client_by_group (client_id, group_id) VALUES (2, 2);
+    INSERT INTO client_by_group (client_id, group_id) VALUES (3, 3);
+    ```
+    </details>
+
+## Example 1: Exclude from blocking
+
+**Task:** Exclude client 1 from Pi-hole's blocking by removing client 1 from the `Unassociated` group (see comment above).
+
+![Change client groups assignment](example-clients-3.png)
+
+<details><summary>Raw database instructions</summary>
+```sql
+DELETE FROM client_by_group WHERE client_id = 1 AND group_id = 0;
+```
+</details>
+
+**Result**
+
+Client        | Group | Domain | Blocked
+------------- | ----- | ------ | -------
+*all other*   |   0   | doubleclick.net | Yes
+192.168.0.101 |   1   | doubleclick.net | **No**
+192.168.0.102 |   2   | doubleclick.net | Yes
+192.168.0.103 |   3   | doubleclick.net | Yes
+192.168.0.104 |   4   | doubleclick.net | Yes
+
+
+All three clients got automatically assigned to the default (`Unassociated`) group when they were added. The default group includes all adlists and list domains (if not already changed by the user). When we remove the default group for client `192.168.0.101`, we effectively remove all associations to any adlists and domains. This leaves this client completely unblocked.
+
+## Example 2: Blocklist management
+
+**Task:** Assign adlist with ID 1 to group 1 (in addition to the default assignment to group 0). This results in client `192.168.0.101` using *only this* adlist (we removed the default association in the last step).
+
+![Change blocklist group assignment](example-adlists-1.png)
+
+<details><summary>Raw database instructions</summary>
+```sql
+INSERT INTO adlist_by_group (adlist_id, group_id) VALUES (1,1);
+```
+</details>
+
+**Result**
+
+Client        | Group | Domain | Blocked
+------------- | ----- | ------ | -------
+*all other*   |   0   | doubleclick.net | Yes
+192.168.0.101 |   1   | doubleclick.net | **Yes**
+192.168.0.102 |   2   | doubleclick.net | Yes
+192.168.0.103 |   3   | doubleclick.net | Yes
+192.168.0.104 |   4   | doubleclick.net | Yes
+
+
+`192.168.0.101` gets `doubleclick.net` blocked as it uses an adlist including this domain. All other clients stay unchanged.
+
+## Example 3: Blacklisting
+
+**Task:** Add a single domain that should be **blacklisted only for group 1** (client `192.168.0.101`).
+
+### Step 1
+
+Add the domain to be blocked
+
+![Add new exact blacklist domain](example-new-black.png)
+
+![Resulting row in list of domains](example-domain-1.png)
+
+<details><summary>Raw database instructions</summary>
+```sql
+INSERT INTO domainlist (type, domain, comment) VALUES (1, 'blacklisted.com', 'Blacklisted for members of group 1');
+```
+</details>
+
+**Result**
+
+Client        | Group | Domain | Blocked
+------------- | ----- | ------ | -------
+*all other*   |   0   | blacklisted.com | Yes
+192.168.0.101 |   1   | blacklisted.com | **No**
+192.168.0.102 |   2   | blacklisted.com | Yes
+192.168.0.103 |   3   | blacklisted.com | Yes
+192.168.0.104 |   4   | blacklisted.com | Yes
+
+
+Note that Pi-hole is *not* blocking this domain for client `192.168.0.101` as we removed the default assignment through group 0 above. All remaining clients are linked through the unassociated group to this domain and see it as being blocked.
+
+### Step 2
+
+Assign this domain to group 1
+
+![Assign group to a new domain](example-domain-2.png)
+
+<details><summary>Raw database instructions</summary>
+```sql
+INSERT INTO domainlist_by_group (domainlist_id, group_id) VALUES (1, 1);
+```
+(the `domainlist_id` might be different for you, check with `SELECT last_insert_rowid();` after step 1)
+</details>
+
+**Result**
+
+Client        | Group | Domain | Blocked
+------------- | ----- | ------ | -------
+*all other*   |   0   | blacklisted.com | Yes
+192.168.0.101 |   1   | blacklisted.com | **Yes**
+192.168.0.102 |   2   | blacklisted.com | Yes
+192.168.0.103 |   3   | blacklisted.com | Yes
+192.168.0.104 |   4   | blacklisted.com | Yes
+
+All clients see this domain as being blocked. Client 1 due to a direct assignment through group 1, all remaining clients through the default group 0 (unchanged).
+
+### Step 3
+
+Remove default assignment to all clients not belonging to a group
+
+![Remove unassociated group from new domain](example-domain-3.png)
+
+<details><summary>Raw database instructions</summary>
+```sql
+DELETE FROM domainlist_by_group WHERE domainlist_id = 1 AND group_id = 0;
+```
+(the `domainlist_id` might be different for you, see above)
+</details>
+
+**Result**
+
+Client        | Group | Domain | Blocked
+------------- | ----- | ------ | -------
+*all other*   |   0   | blacklisted.com | **No**
+192.168.0.101 |   1   | blacklisted.com | Yes
+192.168.0.102 |   2   | blacklisted.com | **No**
+192.168.0.103 |   3   | blacklisted.com | **No**
+192.168.0.104 |   4   | blacklisted.com | **No**
+
+While client 1 keeps its explicit assignment through group 1, the remaining clients lost their unassignments when we removed group 0 from the assignment.
+
+## Example 4: Whitelisting
+
+**Task:** Add a single domain that should be **whitelisted only for group 2** (client `192.168.0.102`).
+
+### Step 1
+
+Add the domain to be whitelisted
+
+![Add a new exact whitelist domain](example-new-white.png)
+
+![Resulting row in list of domains](example-domain-4.png)
+
+<details><summary>Raw database instructions</summary>
+```sql
+INSERT INTO domainlist (type, domain, comment) VALUES (0, 'doubleclick.net', 'Whitelisted for members of group 2');
+```
+</details>
+
+**Result**
+
+Client        | Group | Domain | Blocked
+------------- | ----- | ------ | -------
+*all other*   |   0   | doubleclick.net | **No**
+192.168.0.101 |   1   | doubleclick.net | Yes
+192.168.0.102 |   2   | doubleclick.net | **No**
+192.168.0.103 |   3   | doubleclick.net | **No**
+192.168.0.104 |   4   | doubleclick.net | **No**
+
+Client `192.168.0.101` is not whitelisting this domain as we removed the default assignment through group 0 above. All remaining clients are linked through the default group to this domain and see it as being whitelisted. Note that this is completely analog to [step 1](#step-1_1) of [example 3](#example-3-blacklisting).
+
+### Step 2
+
+Remove default group assignment
+
+![Remove default group assignment](example-domain-5.png)
+
+<details><summary>Raw database instructions</summary>
+```sql
+DELETE FROM domainlist_by_group WHERE domainlist_id = 2 AND group_id = 0;
+```
+</details>
+
+**Result**
+
+Client        | Group | Domain | Blocked
+------------- | ----- | ------ | -------
+*all other*   |   0   | doubleclick.net | **Yes**
+192.168.0.101 |   1   | doubleclick.net | Yes
+192.168.0.102 |   2   | doubleclick.net | **Yes**
+192.168.0.103 |   3   | doubleclick.net | **Yes**
+192.168.0.104 |   4   | doubleclick.net | **Yes**
+
+Requests from all clients are blocked as the new whitelist entry is not associated with any group and, hence, is not used by any client.
+
+### Step 3
+
+Assign this domain to group 2
+
+![Assign group to a new domain](example-domain-6.png)
+
+<details><summary>Raw database instructions</summary>
+```sql
+INSERT INTO domainlist_by_group (domainlist_id, group_id) VALUES (2, 2);
+```
+(the `domainlist_id` might be different for you, check with `SELECT last_insert_rowid();` after step 1)
+</details>
+
+
+**Result**
+
+Client        | Group | Domain | Blocked
+------------- | ----- | ------ | -------
+*all other*   |   0   | doubleclick.net | Yes
+192.168.0.101 |   1   | doubleclick.net | **No**
+192.168.0.102 |   2   | doubleclick.net | Yes
+192.168.0.103 |   3   | doubleclick.net | Yes
+192.168.0.104 |   4   | doubleclick.net | Yes
+
+Client 2 got the whitelist entry explicitly assigned to. Accordingly, client 2 does not get the domain blocked whereas all remaining clients still see this domain as blocked.

docs/database/gravity/groups.md

@@ -0,0 +1,38 @@
+## Group management
+
+Any blocklist or domain on the white-/black-/regex-lists can be managed through groups. This allows not only grouping them together to highlight their relationship, but also enabling/disabling them together if one, for instance, wants to visit a specific service only temporarily.
+
+Groups are defined in the `group` table and can have an optional description in addition to the mandatory name of the group.
+
+Label | Type | Uniqueness enforced | Content
+----- | ---- | ------------------- | --------
+`id` | integer | Yes | Unique ID for database operations
+`enabled` | boolean | No | Flag whether domains in this group should be used<br>(`0` = disabled, `1` = enabled)
+`name` | text | No | Mandatory group name
+`comment` | text | No | Optional field for arbitrary user comments
+
+Group management is implemented using so-called linking tables. Hence, it is possible to
+
+- associate domains (and clients!) with any number of groups,
+- manage adlists together with groups,
+- use the same groups for black- and whitelisted domains at the same time.
+
+The linking tables are particularly simple, as they only link group `id`s with list `id`s. As an example, we describe the `domainlist_by_group` table. The `adlist` and `client` linking tables are constructed similarly.
+
+Label | Type | Content
+----- | ---- | -------
+`domainlist_id` | integer | `id` of domain in the `domainlist` table
+`group_id` | integer | `id` of associated group in the `group` table
+
+Group `0` is special as it is automatically assigned to domains and clients not being a member of other groups. Each newly added client or domain gets assigned to group zero when being added.
+
+## Effect of group management
+
+The great flexibility to manage domains in no, one, or multiple groups may result in unexpected behavior when, e.g., the domains are enabled in some but disabled in other groups. For the sake of convenience, we describe the possible configurations and whether *FTL*DNS uses these domains (✔) or not (✘) in such cases.
+
+- Domain disabled: ✘<br>Note that the domain is never imported by *FTL*DNS, even if it is contained in an enabled group.
+
+- Domain enabled: It depends...
+    - Not managed by a group: ✔
+    - Contained in one or more groups (at least one enabled): ✔
+    - Contained in one or more groups (all disabled): ✘

docs/database/gravity/index.md

@@ -0,0 +1,69 @@
+Pi-hole uses the well-known relational database management system SQLite3 for managing the various domains that are used to control the DNS filtering system. The database-based domain management has been added with Pi-hole v5.0.
+
+## Domain tables (`domainlist`)
+
+The database stores white-, and blacklists which are directly relevant for Pi-hole's domain blocking behavior. The `domainlist` table contains all domains on the white- and blacklists. It has a few extra fields to store data related to a given domain such as the `enabled` state, the dates when the domain was added and when it was last modified, and an optional comment.
+
+The date fields are defined as `INTEGER` fields as they expect numerical timestamps also known as *UNIX time*. The `date_added` and `date_modified` fields are initialized with the current timestamp converted to UNIX time. The `comment` field is optional and can be empty.
+
+Pi-hole's *FTL*DNS reads the tables through the various view, omitting any disabled domains.
+
+Label | Type | Uniqueness enforced | Content
+----- | ---- | ------------------- | --------
+`id` | integer | Yes | Unique ID for database operations
+`type` | integer | No | `0` = exact whitelist,<br> `1` = exact blacklist,<br> `2` = regex whitelist,<br> `3` = regex blacklist
+`domain` | text | Yes | Domain
+`enabled` | boolean | No | Flag whether domain should be used by `pihole-FTL`<br>(`0` = disabled, `1` = enabled)
+`date_added` | integer | No | Timestamp when domain was added
+`date_modified` | integer | No | Timestamp when domain was last modified, automatically updated when a record is changed
+`comment` | text | No | Optional field for arbitrary user comments, only field that is allowed to be `NULL`
+
+## Adlist Table (`adlist`)
+
+The `adlist` table contains all sources for domains to be collected by `pihole -g`. Just like the other tables, it has a few extra fields to store metadata related to a given source.
+
+Label | Type | Uniqueness enforced | Content
+----- | ---- | ------------------- | --------
+`id` | integer | Yes | Unique ID for database operations
+`address` | text | Yes | The URL of the list
+`enabled` | boolean | No | Flag whether domain should be used by `pihole-FTL`<br>(`0` = disabled, `1` = enabled)
+`date_added` | integer | No | Timestamp when domain was added
+`date_modified` | integer | No | Timestamp when domain was last modified, automatically updated when a record is changed
+`comment` | text | No | Optional field for arbitrary user comments
+
+## Gravity Table (`gravity`)
+
+The `gravity` table consists of the domains that have been processed by Pi-hole's `gravity` (`pihole -g`) command. The domains in this list are the collection of domains sourced from the configured sources (see the [`adlist` table](index.md#adlist-table-adlist).
+
+During each run of `pihole -g`, this table is flushed and completely rebuilt from the newly obtained set of domains to be blocked.
+
+Label | Type | Content
+----- | ---- | -------
+`domain` | text | Blocked domain compiled from adlist referenced by `adlist_id`
+`adlist_id` | integer | ID associated to adlists in table `adlist`
+
+Uniqueness is enforced on pairs of (`domain`, `adlist_id`). In other words: domains can be added multiple times, however, only when they are referencing different adlists as their origins.
+
+## Client table (`client`)
+
+Clients are identified by their IP addresses. Each client automatically gets a unique identifier (`id`).
+
+Label | Type | Content
+----- | ---- | -------
+`id` | integer | Client ID (autoincrementing)
+`ip` | text | IP address of the client (IPv4 or IPv6), Uniqueness is enforced
+`date_added` | integer | Timestamp when a client was added
+`date_modified` | integer | Timestamp when a client was last modified, automatically updated when a record is changed
+`comment` | text | Optional field for arbitrary user comments, the only field that is allowed to be `NULL`
+
+## Audit Table (`domain_audit`)
+
+The `domain_audit` table contains domains that have been audited by the user on the web interface.
+
+Label | Type | Uniqueness enforced | Content
+----- | ---- | ------------------- | --------
+`id` | integer | Yes | Unique ID for database operations
+`domain` | text | Yes | Domain
+`date_added` | integer | No | Unix timestamp when domain was added
+
+{!abbreviations.md!}

docs/database/index.md

@@ -0,0 +1,8 @@
+Pi-hole uses the well-known relational database management system SQLite3 both for its long-term storage of query data and for its domain management. In contrast to many other database management solutions, Pi-hole does not need a server database engine as the database engine is directly embedded in *FTL*DNS. It seems an obvious choice as it is probably the most widely deployed database engine - it is used today by several widespread web browsers, operating systems, and embedded systems (such as mobile phones), among others. Hence, it is rich in supported platforms and offered features. SQLite implements most of the SQL-92 standard for SQL and can be used for high-level queries.
+
+Details concerning the databases, their contained tables and exemplary SQL commands allowing even complex requests to Pi-hole's databases are described on the subpages of this category.
+
+- [Query database `/etc/pihole/pihole-FTL.db`](ftl.md)
+- [Domain database `/etc/pihole/gravity.db`](gravity/index.md)
+
+{!abbreviations.md!}

docs/ftldns/compile.md

@@ -19,12 +19,12 @@ sudo dnf install gcc gmp-devel gmp-static m4
 
 ---
 
-You'll also need to compile `nettle` as *FTL*DNS uses `libnettle` for handling DNSSEC. Compile and install a recent version of `nettle` (we tested 3.4):
+You'll also need to compile `nettle` as *FTL*DNS uses `libnettle` for handling DNSSEC. Compile and install a recent version of `nettle` (we recommend 3.5):
 
 ```bash
-wget https://ftp.gnu.org/gnu/nettle/nettle-3.4.tar.gz
-tar -xvzf nettle-3.4.tar.gz
-cd nettle-3.4
+wget https://ftp.gnu.org/gnu/nettle/nettle-3.5.tar.gz
+tar -xvzf nettle-3.5.tar.gz
+cd nettle-3.5
 ./configure
 make
 sudo make install

docs/ftldns/configfile.md

@@ -2,6 +2,8 @@ You can create a file `/etc/pihole/pihole-FTL.conf` that will be read by *FTL*DN
 
 Possible settings (**the option shown first is the default**):
 
+---
+
 ### DNS settings
 
 #### `BLOCKINGMODE=NULL|IP-NODATA-AAAA|IP|NXDOMAIN` {#blocking_mode data-toc-label='Blocking Mode'}
@@ -9,87 +11,83 @@ Possible settings (**the option shown first is the default**):
 How should `FTL` reply to blocked queries?<br>
 **[More details](blockingmode.md)**
 
+#### `CNAME_DEEP_INSPECT=true|false` {#cname_deep_inspect data-toc-label='Deep CNAME inspection'}
+
+Use this option to disable deep CNAME inspection. This might be beneficial for very low-end devices
+
+---
+
 ### Statistics settings
 
 #### `MAXLOGAGE=24.0` {#maxlogage data-toc-label='Max Log Age'}
 
 Up to how many hours of queries should be imported from the database and logs? Maximum is 24.0
 
----
-
 #### `PRIVACYLEVEL=0|1|2|3|4` {#privacylevel data-toc-label='Privacy Level'}
 
 Which privacy level is used?<br>
 **[More details](privacylevels.md)**
 
----
-
 #### `IGNORE_LOCALHOST=no|yes` {#ignore_localhost data-toc-label='Ignore localhost'}
 
 Should `FTL` ignore queries coming from the local machine?
 
----
-
 #### `AAAA_QUERY_ANALYSIS=yes|no` {#aaaa_query_analysis data-toc-label='AAAA Query Analysis'}
 
 Allow `FTL` to analyze AAAA queries from pihole.log?
 
----
-
 #### `ANALYZE_ONLY_A_AND_AAAA=false|true` {#analyze_only_a_and_aaaa data-toc-label='Analyze A and AAAA Only'}
 
 Should `FTL` only analyze A and AAAA queries?
 
-### Socket settings
+---
+
+### Other settings
 
 #### `SOCKET_LISTENING=localonly|all` {#socket_listening data-toc-label='Socket Listening'}
 
 Listen only for local socket connections or permit all connections
 
----
-
 #### `FTLPORT=4711` {#ftlport data-toc-label='FTLDNS Port'}
 
 On which port should FTL be listening?
 
-### Hostname resolution
-
 #### `RESOLVE_IPV6=yes|no` {#resolve_ipv6 data-toc-label='Resolve IPV6'}
 
 Should `FTL` try to resolve IPv6 addresses to hostnames?
 
----
-
 #### `RESOLVE_IPV4=yes|no` {#resolve_ipv4 data-toc-label='Resolve IPV4'}
 
 Should `FTL` try to resolve IPv4 addresses to hostnames?
 
-### Database settings
+#### `DELAY_STARTUP=0` {#delay_startup data-toc-label='Delay resolver startup'}
 
-**[Further details concerning the database](database.md)**
+In certain configurations you may want FTL to wait a given amount of time before trying to start the DNS revolver. This is typically found when network interfaces appear only late during system startup and the interface startup prority are configured incorrectly. This setting takes any integer value between 0 and 300 seconds
 
-#### `DBIMPORT=yes|no` {#dbimport data-toc-label='DB Import'}
+---
+
+### Long-term database settings
+
+**[Further details concerning the database](../database/index.md)**
+
+#### `DBIMPORT=yes|no` {#dbimport data-toc-label='Use database'}
 
 Should `FTL` load information from the database on startup to be aware of the most recent history?
 
----
-
-#### `MAXDBDAYS=365` {#maxdbdays data-toc-label='Max DB Days'}
+#### `MAXDBDAYS=365` {#maxdbdays data-toc-label='Max age'}
 
 How long should queries be stored in the database? Setting this to `0` disables the database
 
----
-
-#### `DBINTERVAL=1.0` {#dbinterval data-toc-label='DB Interval'}
+#### `DBINTERVAL=1.0` {#dbinterval data-toc-label='Storing Interval'}
 
 How often do we store queries in FTL's database [minutes]?
 
----
-
-#### `DBFILE=/etc/pihole/pihole-FTL.db` {#dbfile data-toc-label='DB File'}
+#### `DBFILE=/etc/pihole/pihole-FTL.db` {#dbfile data-toc-label='Database Filename'}
 
 Specify path and filename of FTL's SQLite3 long-term database. Setting this to `DBFILE=` disables the database altogether
 
+---
+
 ### File options
 
 #### `LOGFILE=/var/log/pihole-FTL.log` {#file_LOGFILE data-toc-label='Log file'}
@@ -112,70 +110,54 @@ File containing the socket FTL's API is listening on.
 
 Config file of Pi-hole containing, e.g., the current blocking status (do not change).
 
-#### `AUDITLISTFILE=/etc/pihole/auditlog.list` {#file_AUDITLISTFILE data-toc-label='Audit list file'}
-
-List containing the audited domains.
-
 #### `MACVENDORDB=/etc/pihole/macvendor.db` {#file_MACVENDORDB data-toc-label='MacVendor database file'}
 
 Database containing MAC -> Vendor information for the network table.
 
+#### `GRAVITYDB=/etc/pihole/gravity.db` {#file_GRAVITYDB data-toc-label='Gravity database'}
+
+Specify path and filename of FTL's SQLite3 gravity database. This database contains all domains relevant for Pi-hole's DNS blocking
+
+---
+
 ### Debugging options
 
 #### `DEBUG_ALL=false|true` {#debug_all data-toc-label='Debug All'}
 
 Enable all debug flags. If this is set to true, all other debug config options are ignored.
 
----
-
 #### `DEBUG_DATABASE=false|true` {#debug_database data-toc-label='Debug Database'}
 
 Print debugging information about database actions. This prints performed SQL statements as well as some general information such as the time it took to store the queries and how many have been saved to the database.
 
----
-
 #### `DEBUG_NETWORKING=false|true` {#debug_networking data-toc-label='Debug networking'}
 
 Prints a list of the detected interfaces on startup of `pihole-FTL`. Also, prints whether these interfaces are IPv4 or IPv6 interfaces.
 
----
-
 #### `DEBUG_LOCKS=false|true` {#debug_locks data-toc-label='Debug Locks'}
 
 Print information about shared memory locks. Messages will be generated when waiting, obtaining, and releasing a lock.
 
----
-
 #### `DEBUG_QUERIES=false|true` {#debug_queries data-toc-label='Debug Queries'}
 
 Print extensive query information (domains, types, replies, etc.). This has always been part of the legacy `debug` mode of `pihole-FTL`.
 
----
-
 #### `DEBUG_FLAGS=false|true` {#debug_flags data-toc-label='Debug Flags'}
 
 Print flags of queries received by the DNS hooks. Only effective when `DEBUG_QUERIES` is enabled as well.
 
----
-
 #### `DEBUG_SHMEM=false|true` {#debug_shmem data-toc-label='Debug Shared Memory'}
 
 Print information about shared memory buffers. Messages are either about creating or enlarging shmem objects or string injections.
 
----
-
 #### `DEBUG_GC=false|true` {#debug_gc data-toc-label='Debug GC'}
 
 Print information about garbage collection (GC): What is to be removed, how many have been removed and how long did GC take.
 
----
-
 #### `DEBUG_ARP=false|true` {#debug_arp data-toc-label='Debug ARP'}
 
 Print information about ARP table processing: How long did parsing take, whether read MAC addresses are valid, and if the `macvendor.db` file exists.
 
----
-
 #### `DEBUG_REGEX=false|true` {#debug_regex data-toc-label='Debug REGEX'}
 
 Controls if *FTL*DNS should print extended details about regex matching into `pihole-FTL.log`.<br>
@@ -184,28 +166,32 @@ Due to legacy reasons, we also support the following setting to be used for enab
 Note that if one of them is set to `true`, the other one cannot be used to disable this setting again.<br>
 **[More details](regex/overview.md)**
 
----
-
 #### `DEBUG_API=false|true` {#debug_api data-toc-label='Debug Telnet'}
 
 Print extra debugging information during telnet API calls. Currently only used to send extra information when getting all queries.
 
----
-
 #### `DEBUG_OVERTIME=false|true` {#debug_overtime data-toc-label='Debug overTime'}
 
 Print information about overTime memory operations, such as initializing or moving overTime slots.
 
----
-
 #### `DEBUG_EXTBLOCKED=false|true` {#debug_extblocked data-toc-label='Debug externally blocked'}
 
 Print information about why FTL decided that certain queries were recognized as being externally blocked.
 
----
-
 #### `DEBUG_CAPS=false|true` {#debug_caps data-toc-label='Debug Linux capabilities'}
 
 Print information about capabilities granted to the pihole-FTL process. The current capabilities are printed on receipt of `SIGHUP`, i.e., the current set of capabilities can be queried without restarting `pihole-FTL` (by setting `DEBUG_CAPS=true` and thereafter sending `killall -HUP pihole-FTL`).
 
+#### `DEBUG_DNSMASQ_LINES=false|true` {#debug_dnsmasq_lines data-toc-label='Debug dnsmasq lines'}
+
+Print file and line causing a `dnsmasq` event into FTL's log files. This is handy to implement additional hooks missing from FTL.
+
+#### `DEBUG_VECTORS=false|true` {#debug_vectors data-toc-label='Debug FTL vectors'}
+
+FTL uses dynamically allocated vectors for various tasks. This config option enables extensive debugging information such as information about allocation, referencing, deletion, and appending.
+
+#### `DEBUG_RESOLVER=false|true` {#debug_resolver data-toc-label='Debug FTL resolver'}
+
+Extensive information about host name resolution like which DNS servers are used in the first and second host name resolving tries (only affecting internally generated PTR queries).
+
 {!abbreviations.md!}

docs/ftldns/database.md

@@ -1,137 +1,3 @@
-Pi-hole *FTL*DNS uses the well-known relational database management system SQLite3 as its long-term storage of query data. In contrast to many other database management solutions, *FTL*DNS does not need a server database engine as the database engine is directly embedded in *FTL*DNS. It seems an obvious choice as it is probably the most widely deployed database engine - it is used today by several widespread web browsers, operating systems, and embedded systems (such as mobile phones), among others. Hence, it is rich in supported platforms and offered features. SQLite implements most of the SQL-92 standard for SQL and can be used for high-level queries.
-
-We update the database file periodically and on exit of *FTL*DNS (triggered e.g. by a `service pihole-FTL restart`). The updating frequency can be controlled by the parameter [`DBINTERVAL`](configfile.md#dbinterval) and defaults to once per minute. We think this interval is sufficient to protect against data losses due to power failure events. *FTL*DNS needs the database to populate its internal history of the most recent 24 hours. If the database is disabled, *FTL*DNS will show an empty query history after a restart.
-
-The location of the database can be configured by the config parameter [`DBFILE`](configfile.md#dbfile). It defaults to `/etc/pihole/pihole-FTL.db`. If the given file does not exist, *FTL*DNS will create a new (empty) database file.
-
-Another way of controlling the size of the long-term database is setting a maximum age for log queries to keep using the config parameter [`MAXDBDAYS`](configfile.md#maxdbdays). It defaults to 365 days, i.e. queries that are older than one year get periodically removed to limit the growth of the long-term database file.
-
-The config parameter [`DBIMPORT`](configfile.md#dbimport) controls whether `FTL` loads information from the database on startup. It needs to do this to populate the internal data structure with the most recent history. However, as importing from the database on disk can delay FTL on very large deploys, it can be disabled using this option.
-
 ---
-
-### Split database
-
-You can split your long-term database by periodically rotating the database file (do this only when `pihole-FTL` is *not* running). The individual database contents can easily be merged when required.
-This could be implemented by running a monthly `cron` job such as:
-
-```bash
-sudo service pihole-FTL stop
-sudo mv /etc/pihole/pihole-FTL.db /media/backup/pihole-FTL_$(date +"%m-%y").db
-sudo service pihole-FTL start
-```
-
-Note that DNS resolution will not be available as long as `pihole-FTL` is stopped.
-
-### Backup database
-
-The database can be backed up while FTL is running when using the SQLite3 Online backup method, e.g.,
-
-```bash
-sqlite3 /etc/pihole/pihole-FTL.db ".backup /home/pi/pihole-FTL.db.backup"
-```
-
-will create `/home/pi/pihole-FTL.db.backup` which is a copy of your long-term database.
-
+redirect: /database/
 ---
-
-The long-term database contains three tables:
-
-### Query Table
-
-Label | Type | Allowed to by empty | Content
---- | --- | ---- | -----
-`id` | integer | No | autoincrement ID for the table, only used by SQLite3, not by *FTL*DNS
-`timestamp` | integer | No | Unix timestamp when this query arrived at *FTL*DNS (used as index)
-`type` | integer | No | Type of this query (see [Supported query types](database.md#supported-query-types))
-`status` | integer | No | How was this query handled by *FTL*DNS? (see [Supported status types](database.md#supported-status-types))
-`domain` | text | No | Requested domain
-`client` | text | No | Requesting client (IP address)
-`forward` | text | Yes | Forward destination used for this query (only set if `status == 2`)
-
-SQLite3 syntax used to create this table:
-
-```sql
-CREATE TABLE queries ( id INTEGER PRIMARY KEY AUTOINCREMENT, timestamp INTEGER NOT NULL, type INTEGER NOT NULL, status INTEGER NOT NULL, domain TEXT NOT NULL, client TEXT NOT NULL, forward TEXT );
-CREATE INDEX idx_queries_timestamps ON queries (timestamp);
-```
-
-### Counters table
-
-This table contains counter values integrated over the entire lifetime of the table
-
-Label | Type | Allowed to by empty | Content
---- | --- | ---- | -----
-`id` | integer | No | ID for the table used to select a counter (see below)
-`value` | integer | No | Value of a given counter
-
-Counter ID | Interpretation
---- | ---
-0 | Total number of queries
-1 | Total number of blocked queries (Query `status` 1, 4 or 5)
-
-SQLite3 syntax used to create this table:
-
-```sql
-CREATE TABLE counters ( id INTEGER PRIMARY KEY NOT NULL, value INTEGER NOT NULL );
-```
-
-### FTL table
-
-The FTL table contains some data used by *FTL*DNS for determining which queries to save to the database. This table does not contain any entries of general interest.
-
-SQLite3 syntax used to create this table:
-
-```sql
-CREATE TABLE ftl ( id INTEGER PRIMARY KEY NOT NULL, value BLOB NOT NULL );
-```
-
-### Supported query types
-
-ID | Query Type
---- | ---
-1 | A
-2 | AAAA
-3 | ANY
-4 | SRV
-5 | SOA
-6 | PTR
-7 | TXT
-
-<!-- ID | 1 | 2 | 3 | 4 | 5 | 6 | 7 -->
-<!-- -- | -- | -- | -- | -- | -- | -- | -- -->
-<!-- Query | A | AAAA | ANY | SRV | SOA | PTR | TXT -->
-
-### Supported status types
-
-ID | Query Type
---- | ---
-0 | Unknown status (was not answered by forward destination)
-1 | Blocked by `gravity.list`
-2 | Permitted + forwarded
-3 | Permitted + replied to from cache
-4 | Blocked by wildcard
-5 | Blocked by `black.list`
-6 | Blocked by upstream server (known blocking page IP address)
-7 | Blocked by upstream server (`0.0.0.0` or `::`)
-8 | Blocked by upstream server (`NXDOMAIN` with `RA` bit unset)
-
-### Example for interaction with the FTL long-term database
-
-In addition to the interactions the Pi-hole database API offers, you can also run your own SQL commands against the database. If you want to obtain the three most queries domains for all time, you could use
-
-```bash
-sqlite3 "/etc/pihole/pihole-FTL.db" "SELECT domain,count(domain) FROM queries WHERE (STATUS == 2 OR STATUS == 3) GROUP by domain order by count(domain) desc limit 3"
-```
-
-which would return something like
-
-```text
-discourse.pi-hole.net|421095
-www.pi-hole.net|132483
-posteo.de|130243
-```
-
-showing the domain and the number of times it was found in the long-term database. Note that such a request might take very long for computation as the entire history of queries have to be processed for this.
-
-{!abbreviations.md!}

docs/ftldns/debugging.md

@@ -1,13 +1,22 @@
 Once you are used to it, you can skip most of the steps. Debugging *FTL*DNS is quite easy. `pihole-FTL` has been designed so that a debugger can be attached to an already running process. This will give you insights into how software (not limited to `pihole-FTL`) works.
 
+### Prerequeirements (only required once)
+
 1. Install `screen` and `gdb` using `sudo apt-get install screen gdb`
 2. Start a screen session (it will allow you to come back even if the SSH connection died)
     * If you don't know about `screen`, then read about it (you *will* love it!)
 3. Start a screen session using `screen`
-4. Use `sudo gdb -p $(pidof pihole-FTL)` to attach the debugger to the already running `pihole-FTL` process
-5. Once loading of the symbols has finished (the `(gdb)` input prompt is shown), run `handle SIGHUP nostop SIGPIPE nostop`
-6. Enter `continue` to continue the operation of `pihole-FTL` inside the debugger. All debugger features are now available.
-7. When `pihole-FTL` has crashed, copy & paste the terminal output into a (new) issue. Also, type `backtrace` and include its output. We might ask for additional information in order to isolate your particular issue.
+4. Configure `gdb` by installing a globally valid intialization file:
+   ```bash
+   echo "handle SIGHUP nostop SIGPIPE nostop SIGTERM nostop SIG32 nostop SIG34 nostop SIG35 nostop" | sudo tee /root/.gdbinit
+   ```
+   You can omit this step, however, you will have to remember to run the quoted line on *every start* of `gdb` in order to properly debug FTL.
+
+### Start of debugging session
+
+1. Use `sudo gdb -p $(pidof pihole-FTL)` to attach the debugger to the already running `pihole-FTL` process
+2. Once loading of the symbols has finished (the `(gdb)` input prompt is shown), enter `continue` to continue the operation of `pihole-FTL` inside the debugger. All debugger features are now available.
+3. When `pihole-FTL` has crashed, copy & paste the terminal output into a (new) issue. Also, type `backtrace` and include its output. We might ask for additional information in order to isolate your particular issue.
 
 <!-- When you want to detach the debugger from `FTL` without terminating the process, you can hit `Ctrl+C` and enter `detach` followed by `quit`. -->
 

docs/ftldns/regex/overview.md

@@ -1,17 +1,27 @@
-A regular expression, or RegEx for short, is a pattern that can be used for building arbitrarily complex blocking rules in *FTL*DNS.
+A regular expression, or RegEx for short, is a pattern that can be used for building arbitrarily complex filter rules in *FTL*DNS.
 We implement the POSIX Extended Regular Expressions similar to the one used by the UNIX `egrep` (or `grep -E`) command.
 
 Our implementation is light and fast as each domain is only checked once for a match (if you query `google.com`, it will be checked against your RegEx. Any subsequent query to the same domain will not be checked again until you restart `pihole-FTL`).
 
-## How to use regular expressions for blocking
+## Hierarchy of regex filters in *FTL*DNS
 
-*FTL*DNS reads in regular expression filters from `/etc/pihole/regex.list` (one expression per line, lines starting with `#` will be skipped).
-To tell *FTL*DNS to reload the list, either:
+*FTL*DNS uses a specific hierarchy to ensure regex filters work as you expect them to. Whitelisting always has priority over blacklisting.
+There are two locations where regex filters are important:
 
-- Execute the `>recompile-regex` API command (`echo ">recompile-regex" | nc localhost 4711`) or
+1. On loading the blocking domains form the `gravity` database table, *FTL*DNS skips not only exactly whitelisted domains but also those that match enabled whitelist regex filters.
+2. When a queried domain matches a blacklist regex filter, the query will *not* be blocked if the domain *also* matches an exact or a regex whitelist entry.
+
+## How to use regular expressions for filtering domains
+
+*FTL*DNS reads in regular expression filters from the two [`regex` database views](../../database/gravity/index.md).
+To tell *FTL*DNS to reload the list of regex filters, either:
+
+- Execute `pihole restartdns reload-lists` or
 - Send `SIGHUP` to `pihole-FTL` (`sudo killall -SIGHUP pihole-FTL`) or
 - Restart the service (`sudo service pihole-FTL restart`)
 
+The first command is to be preferred as it ensures that the DNS cache itself remains intact. Hence, it is also the fastest of the available options.
+
 ## Pi-hole Regex debugging mode
 
 To ease the usage of regular expression filters in *FTL*DNS, we offer a regex debugging mode. Set
@@ -25,11 +35,11 @@ in your `/etc/pihole/pihole-FTL.conf` and restart `pihole-FTL` to enable or disa
 Once the debugging mode is enabled, each match will be logged to `/var/log/pihole-FTL.log` in the following format:
 
 ```text
-[2018-07-17 17:40:51.304] Regex in line 2 "((^)|(\.))twitter\." matches "whatever.twitter.com"
+[2018-07-17 17:40:51.304] Regex blacklist (DB ID 15) >> MATCH: "whatever.twitter.com" vs. "((^)|(\.))twitter\."
 ```
 
-The given line number corresponds to the line in the file `/etc/pihole/regex.list`.
+The given DB ID corresponds to the ID of the corresponding row in the `domainlist` database table.
 
-Note that validation is only done on the first occurrence of a domain to increase the computational efficiency of *FTL*DNS.
+Note that validation is only done on the first occurrence of a domain to increase the computational efficiency of *FTL*DNS. The result of this evaluation is stored in an internal DNS cache that is separate from `dnsmasq`'s own DNS cache. This allows us to only flush this special cache when modifying the black- and whitelists *without* having to flush the entire DNS cache collected so far.
 
 {!abbreviations.md!}

docs/ftldns/regex/tutorial.md

@@ -1,6 +1,6 @@
 # Pi-hole regular expressions tutorial
 
-We provide a short but thorough introduction to our regular expressions implementation. This may come in handy if you are designing blocking rules (see also our cheat sheet below!). In our implementation, all characters match themselves except for the following special characters: `.[{}()\*+?|^$`. If you want to match those, you need to escape them like `\.` for a literal period, but no rule without exception (see character groups below for further details).
+We provide a short but thorough introduction to our regular expressions implementation. This may come in handy if you are designing blocking or whitelisting rules (see also our cheat sheet below!). In our implementation, all characters match themselves except for the following special characters: `.[{}()\*+?|^$`. If you want to match those, you need to escape them like `\.` for a literal period, but no rule without exception (see character groups below for further details).
 
 ## Anchors (`^` and `$`)
 

mkdocs.yml

@@ -60,6 +60,13 @@ nav:
     - 'Updating': main/update.md
   - 'Pi-hole Core':
     - 'The <samp>pihole</samp> command': core/pihole-command.md
+  - 'Databases':
+    - 'Overview': database/index.md
+    - 'Query database': database/ftl.md
+    - 'Domain database':
+      - 'Overview': database/gravity/index.md
+      - 'Group management': database/gravity/groups.md
+      - 'Examples': database/gravity/example.md
   - 'FTLDNS':
     - 'Overview': ftldns/index.md
     - 'Configuration': ftldns/configfile.md
@@ -70,7 +77,6 @@ nav:
       - "Overview": ftldns/regex/overview.md
       - "Tutorial": ftldns/regex/tutorial.md
     - 'Privacy levels': ftldns/privacylevels.md
-    - 'Long-term database': ftldns/database.md
     - 'Telnet API': ftldns/telnet-api.md
     - 'Compatibility': ftldns/compatibility.md
     - 'Install from source': ftldns/compile.md