37 commits

Author	SHA1	Message	Date
DL6ER	10c2dad48a	Improve gravity performance (#3100) ... * Gravity performance improvements. Signed-off-by: DL6ER <dl6er@dl6er.de> * Do not move downloaded lists into migration_backup directory. Signed-off-by: DL6ER <dl6er@dl6er.de> * Do not (strictly) sort domains. Random-leaf access is faster than always-last-leaf access (on average). Signed-off-by: DL6ER <dl6er@dl6er.de> * Append instead of overwrite gravity_new collection list. Signed-off-by: DL6ER <dl6er@dl6er.de> * Rename table gravity_new to gravity_temp to clarify that this is only an intermediate table. Signed-off-by: DL6ER <dl6er@dl6er.de> * Add timers for each of the calls to compute intense parts. They are to be removed before this finally hits the release/v5.0 branch. Signed-off-by: DL6ER <dl6er@dl6er.de> * Fix legacy list files import. It currently doesn't work when the gravity database has already been updated to using the single domainlist table. Signed-off-by: DL6ER <dl6er@dl6er.de> * Simplify database_table_from_file(), remove all to this function for gravity lost downloads. Signed-off-by: DL6ER <dl6er@dl6er.de> * Update gravity.db.sql to version 10 to have newle created databases already reflect the most recent state. Signed-off-by: DL6ER <dl6er@dl6er.de> * Create second gravity database and swap them on success. This has a number of advantages such as instantaneous gravity updates (as seen from FTL) and always available gravity blocking. Furthermore, this saves disk space as the old database is removed on completion. * Add timing output for the database swapping SQLite3 call. Signed-off-by: DL6ER <dl6er@dl6er.de> * Explicitly generate index as a separate process. Signed-off-by: DL6ER <dl6er@dl6er.de> * Remove time measurements. Signed-off-by: DL6ER <dl6er@dl6er.de>	2020-01-24 09:39:13 -08:00
DL6ER	3cb4f6d9d4	We cannot create vw_gravity before having created vw_whitelist as the former depends onthe later. This commit changes the order in which the tables are created. ... Signed-off-by: DL6ER <dl6er@dl6er.de>	2019-09-26 13:50:54 +02:00
DL6ER	acc50b709e	Only migrate files once (domain and adlist lists druing initial creation of gravity.db auditlog.list on database upgrade from version 1 to 2. ... Signed-off-by: DL6ER <dl6er@dl6er.de>	2019-07-07 10:33:08 +02:00
DL6ER	0405aaa3da	Review comments and fixing stickler complaints. ... Signed-off-by: DL6ER <dl6er@dl6er.de>	2019-07-06 09:32:41 +02:00
DL6ER	2fb4256f84	Rename table to "auditlist" ... Signed-off-by: DL6ER <dl6er@dl6er.de>	2019-07-05 16:28:36 +02:00
DL6ER	1dbe6c83c3	Add database upgrading mechanism for adding the audit table. ... Signed-off-by: DL6ER <dl6er@dl6er.de>	2019-07-05 13:54:18 +02:00
DL6ER	23b3a9a650	Add DISTINCT to the view\'s SELECT queries to avoid domain duplication in case they are enabled in multiple groups at the same time. ... Signed-off-by: DL6ER <dl6er@dl6er.de>	2019-07-01 09:44:20 +02:00
DL6ER	f91421418a	Rename adlists -> adlist table. ... Signed-off-by: DL6ER <dl6er@dl6er.de>	2019-06-30 23:21:10 +02:00
DL6ER	3a14e8b013	Check whitelist_by_group.group_id instead of group.enabled against NULL. ... Signed-off-by: DL6ER <dl6er@dl6er.de>	2019-06-30 23:20:13 +02:00
DL6ER	ad97e95f2f	Add required name column in group table. ... Signed-off-by: DL6ER <dl6er@dl6er.de>	2019-06-30 23:06:15 +02:00
DL6ER	5b01facd73	Fix logix on the views: AND ( OR ) ... Signed-off-by: DL6ER <dl6er@dl6er.de>	2019-06-30 20:03:22 +02:00
DL6ER	6fe637b9ee	Rename groups -> group ... Signed-off-by: DL6ER <dl6er@dl6er.de>	2019-06-30 19:43:18 +02:00
DL6ER	94d83dbb2d	Unify the two group tables into a single "groups" table. This allows managing adlists and domains together. ... Signed-off-by: DL6ER <dl6er@dl6er.de>	2019-06-30 19:33:02 +02:00
DL6ER	87c115fc86	Change implementation to use linking tables (domain_groups, adlists_groups). This allows adding domains to multiple lists conveniently. ... Signed-off-by: DL6ER <dl6er@dl6er.de>	2019-06-30 19:04:38 +02:00
DL6ER	0774c4e5ca	vw_gravity should actually filter using vw_whitelist. ... Signed-off-by: DL6ER <dl6er@dl6er.de>	2019-06-16 18:52:23 +02:00
DL6ER	01850c2128	Automatically recreate standard group after deletion to avoid violating foreign key constraints even if foreign key enforcing is disabled (SQLite's default behavior) ... Signed-off-by: DL6ER <dl6er@dl6er.de>	2019-06-16 17:15:36 +02:00
DL6ER	cbc210b014	Add adlist_groups and mark group_id columns as NOT NULL. ... Signed-off-by: DL6ER <dl6er@dl6er.de>	2019-06-16 17:05:33 +02:00
DL6ER	c53ee4202b	Add filtering by domain groups to gravity database. ... Signed-off-by: DL6ER <dl6er@dl6er.de>	2019-06-16 16:50:51 +02:00
DL6ER	d46ff1d2d7	Remove UNIQUE on gravity table as the only column is already PRIMARY KEY. Primary keys must contain UNIQUE values, and cannot contain NULL values ... Signed-off-by: DL6ER <dl6er@dl6er.de>	2019-05-01 22:08:30 +02:00
DL6ER	2f698904e4	Use PRIMARY KEY instead of NOT NULL also in gravity and info tables ... Signed-off-by: DL6ER <dl6er@dl6er.de>	2019-05-01 18:03:35 +02:00
DL6ER	106f079afc	Improve indentation (single tab) in SQLite template ... Signed-off-by: DL6ER <dl6er@dl6er.de>	2019-05-01 17:57:23 +02:00
DL6ER	6c1e3a17f7	Remove NOT NULL constraint on list ids as this is implied by PRIMARY KEY (even though most, if not all, examples on the web still use both attributes at the same time) ... Signed-off-by: DL6ER <dl6er@dl6er.de>	2019-05-01 17:54:29 +02:00
DL6ER	5e26e8245b	Order results of queries to list views by id key ... Signed-off-by: DL6ER <dl6er@dl6er.de>	2019-05-01 17:51:10 +02:00
DL6ER	983117d788	Add id (primary key, autoincrement) field to lists tables ... Signed-off-by: DL6ER <dl6er@dl6er.de>	2019-05-01 17:47:15 +02:00
DL6ER	d5781fb110	Fix extra space in front on END statement in CREATE TRIGGER commands ... Signed-off-by: DL6ER <dl6er@dl6er.de>	2019-05-01 16:38:41 +02:00
DL6ER	487718512d	Remove DISTINCT attribute from views. They are not needed as the corresponding data is obtained from fields with UNIQUE attribute so duplications are already prevented at domain insertion. The elimination of the duplication check on imort through the views slightly reduces the load during import (I observed about 8 percent time saved on my NanoPi NEO). ... Signed-off-by: DL6ER <dl6er@dl6er.de>	2019-05-01 16:37:23 +02:00
DL6ER	2180531a84	Add trigger for automatic updates of the date_modified column and improve formating in the gravity.db template ... Signed-off-by: DL6ER <dl6er@dl6er.de>	2019-04-29 19:43:35 +02:00
DL6ER	a932209143	Set default for date_modified to be the current time ... Signed-off-by: DL6ER <dl6er@dl6er.de>	2019-04-29 17:21:59 +02:00
DL6ER	126f7101ff	Add (optional) date_modified field and use vw_whitelist view in vw_blacklist ... Signed-off-by: DL6ER <dl6er@dl6er.de>	2019-04-28 14:44:03 +02:00
DL6ER	bd0215524b	Add vw_adlists view which returns only enabled adlists ... Signed-off-by: DL6ER <dl6er@dl6er.de>	2019-04-26 17:36:00 +02:00
DL6ER	5bb8163674	Change type of whitelist, blacklist, regex, and adlists time column to INTEGER and insert a DEFAULT clause that returns the current Unix time as such an integer ... Signed-off-by: DL6ER <dl6er@dl6er.de>	2019-04-26 17:26:15 +02:00
DL6ER	61cfd2f9f9	Add adlists table ... Signed-off-by: DL6ER <dl6er@dl6er.de>	2019-04-24 19:48:06 +02:00
DL6ER	644ec36e64	Review comments ... Signed-off-by: DL6ER <dl6er@dl6er.de>	2019-02-06 18:57:48 +01:00
DL6ER	9984647ebb	After renaming column in regex, it also needs to be renamed in vw_regex ... Signed-off-by: DL6ER <dl6er@dl6er.de>	2019-02-05 19:41:11 +01:00
DL6ER	c3c60e10f1	Rename regex field from domain to filter because this described better what it is ... Signed-off-by: DL6ER <dl6er@dl6er.de>	2019-02-05 19:07:12 +01:00
DL6ER	8a2363621d	Rename fields from DateAdded to dateadded ... Signed-off-by: DL6ER <dl6er@dl6er.de>	2019-02-05 19:06:46 +01:00
DL6ER	dbbf21071b	Invert logic to use ENABLED instead of DISABLED. We make sure to set "enabled = 1 WHERE enabled IS NULL" in gravity_store_in_database() ... Signed-off-by: DL6ER <dl6er@dl6er.de>	2019-02-05 19:05:11 +01:00