Commit graph

4440 commits

Author SHA1 Message Date
DL6ER
10c2dad48a Improve gravity performance (#3100)
* Gravity performance improvements.

Signed-off-by: DL6ER <dl6er@dl6er.de>

* Do not move downloaded lists into migration_backup directory.

Signed-off-by: DL6ER <dl6er@dl6er.de>

* Do not (strictly) sort domains. Random-leaf access is faster than always-last-leaf access (on average).

Signed-off-by: DL6ER <dl6er@dl6er.de>

* Append instead of overwrite gravity_new collection list.

Signed-off-by: DL6ER <dl6er@dl6er.de>

* Rename table gravity_new to gravity_temp to clarify that this is only an intermediate table.

Signed-off-by: DL6ER <dl6er@dl6er.de>

* Add timers for each of the calls to compute intense parts. They are to be removed before this finally hits the release/v5.0 branch.

Signed-off-by: DL6ER <dl6er@dl6er.de>

* Fix legacy list files import. It currently doesn't work when the gravity database has already been updated to using the single domainlist table.

Signed-off-by: DL6ER <dl6er@dl6er.de>

* Simplify database_table_from_file(), remove all to this function for gravity lost downloads.

Signed-off-by: DL6ER <dl6er@dl6er.de>

* Update gravity.db.sql to version 10 to have newle created databases already reflect the most recent state.

Signed-off-by: DL6ER <dl6er@dl6er.de>

* Create second gravity database and swap them on success. This has a number of advantages such as instantaneous gravity updates (as seen from FTL) and always available gravity blocking. Furthermore, this saves disk space as the old database is removed on completion.

* Add timing output for the database swapping SQLite3 call.

Signed-off-by: DL6ER <dl6er@dl6er.de>

* Explicitly generate index as a separate process.

Signed-off-by: DL6ER <dl6er@dl6er.de>

* Remove time measurements.

Signed-off-by: DL6ER <dl6er@dl6er.de>
2020-01-24 09:39:13 -08:00
Dan Schaper
52e2a2610e
Merge pull request #3089 from pi-hole/tweak/gravity_db_10
Add gravity database 9->10 update script
2020-01-24 09:23:34 -08:00
DL6ER
a809624356
Update blocked strings for pihole -t.
Signed-off-by: DL6ER <dl6er@dl6er.de>
2020-01-23 19:19:44 +01:00
DL6ER
29f06a4444
Merge pull request #3090 from pi-hole/tweak/debug_group_humanreadable_timestamps
Print human-readable timestamps in the debugger's gravity output
2020-01-20 20:20:51 +01:00
DL6ER
3f9e79f152
Print human-readable timestamps in the debugger's gravity output
Signed-off-by: DL6ER <dl6er@dl6er.de>
2020-01-20 20:13:44 +01:00
DL6ER
633e56e8a9
Add gravity database 9->10 update script.
Signed-off-by: DL6ER <dl6er@dl6er.de>
2020-01-20 17:59:24 +01:00
DL6ER
bf01f725f7
Merge pull request #3087 from pi-hole/fix/blocking_page
Remove dead code causing failure from the blocking page
2020-01-19 21:50:41 +01:00
DL6ER
276b191845
Remove dead code causing failure from the blocking page.
Signed-off-by: DL6ER <dl6er@dl6er.de>
2020-01-19 21:39:49 +01:00
DL6ER
c7bc58e94b
Merge pull request #3082 from pi-hole/tweak/gravity_database_locked
Add timeout to gravity database writing
2020-01-14 20:55:12 +01:00
DL6ER
8f22203d24
Wait 30 seconds for obtaining a database lock instead of immediately failing if the database is busy.
Signed-off-by: DL6ER <dl6er@dl6er.de>
2020-01-14 20:02:00 +01:00
DL6ER
782fec841e
Merge pull request #3076 from pi-hole/new/intl_domains
Add package php-intl for AdminLTE#1130
2020-01-13 17:19:49 +01:00
DL6ER
cfa909a93d
Add package php-intl for AdminLTE#1130.
Signed-off-by: DL6ER <dl6er@dl6er.de>
2020-01-12 14:09:14 +01:00
Adam Warner
e0fde41d87
Merge pull request #3066 from pi-hole/centos8_support
Update installer to support CentOS 8
2020-01-05 14:39:24 +00:00
Adam Warner
574f7c1a1f
Merge pull request #2962 from bcambl/remove_debconf-apt-progress
Remove debconf apt progress
2020-01-04 16:04:50 +00:00
bcambl
ec8f4050d0 Update installer to support CentOS 8
PHP dependency php-json is now required for both the latest Fedora and CentOS.
Package php-json will now be a default web dependency and removed from PIHOLE_WEB_DEPS when installing on CentOS7.

Signed-off-by: bcambl <blayne@blaynecampbell.com>
2020-01-02 06:52:23 -06:00
bcambl
60c51886e0 remove unused debian deps (apt-utils debconf)
Signed-off-by: bcambl <blayne@blaynecampbell.com>
2020-01-01 13:24:02 -06:00
bcambl
cbb1461010 add stdout horizontal rule to install_dependent_packages()
Signed-off-by: bcambl <blayne@blaynecampbell.com>
2020-01-01 13:23:31 -06:00
bcambl
07cc5b501c replace debconf-apt-progress with apt-get in install_dependent_packages()
Removes the need for conditional debconf-apt-progress dependency checking

Signed-off-by: bcambl <blayne@blaynecampbell.com>
2020-01-01 13:11:41 -06:00
bcambl
ebb1a730c1 remove unused fedora/centos dependency: dialog
Signed-off-by: bcambl <blayne@blaynecampbell.com>
2020-01-01 13:11:41 -06:00
MichaIng
9dff55b212 Installer | Remove "dialog" from Debian/Ubuntu installer deps
+ The installer uses `whiptail`, thus `dialog` is not required.

Signed-off-by: MichaIng <micha@dietpi.com>
2020-01-01 13:11:41 -06:00
DL6ER
8ae03b64d7
Merge pull request #3060 from pi-hole/propsed_8_to_9
Add a new migration script to fix the previous one
2019-12-30 11:58:57 +01:00
DL6ER
bb30c818ab
Update database version during migration.
Signed-off-by: DL6ER <dl6er@dl6er.de>
2019-12-30 09:21:30 +00:00
Adam Warner
c944f6a320
Add a new migration script to fix the previous one
Signed-off-by: Adam Warner <me@adamwarner.co.uk>
2019-12-29 23:32:31 +00:00
DL6ER
62ec7de963
Merge pull request #3058 from pi-hole/tweak/7_to_8
Don't create trigger with duplicate name until after old table is del…
2019-12-29 23:05:46 +01:00
Adam Warner
aa4c0ff329
Don't create trigger with duplicate name until after old table is deleted
Signed-off-by: Adam Warner <me@adamwarner.co.uk>
2019-12-29 20:35:11 +00:00
DL6ER
37217ece73
Merge pull request #3049 from pi-hole/tweak/unique_group_name
Group table enhancements
2019-12-28 14:19:04 +01:00
DL6ER
28d4f4b142
Merge pull request #3045 from pi-hole/tweak/gravity.db_permissions
Set permissions and ownership of gravity.db on pihole -g
2019-12-28 14:17:50 +01:00
DL6ER
8d5d423adb
Merge pull request #3052 from pi-hole/revert/76460f0
Revert "Change the regex used for domain validation"
2019-12-21 13:16:12 +01:00
DL6ER
cda0133dd1
Revert "Change the regex used for domain validation"
This reverts commit 76460f01e9.

Signed-off-by: DL6ER <dl6er@dl6er.de>
2019-12-21 11:15:18 +00:00
DL6ER
eda7f40fef
Reinstall trigger that prevents group zero from being deleted.
Signed-off-by: DL6ER <dl6er@dl6er.de>
2019-12-20 00:42:59 +00:00
DL6ER
e589e665a7
Also add date_added and date_modified fields to group table.
Signed-off-by: DL6ER <dl6er@dl6er.de>
2019-12-20 00:21:25 +00:00
DL6ER
b32b5ad6e9
Update gravity database to version 8. This enforces uniqueness on the group name.
Signed-off-by: DL6ER <dl6er@dl6er.de>
2019-12-20 00:09:10 +00:00
DL6ER
e2de199f47
Merge pull request #3037 from pi-hole/new/group_zero
Add special group zero to gravity database
2019-12-18 22:36:43 +01:00
DL6ER
948f4a8827
Ensure permissions and ownership of gravity.db are correctly set on each run of pihole -g. This would have prevented https://github.com/pi-hole/AdminLTE/issues/1077
Signed-off-by: DL6ER <dl6er@dl6er.de>
2019-12-16 09:55:46 +00:00
DL6ER
a1633123aa
Merge pull request #3035 from pi-hole/fix/query_gravity
pihole -q should also scan gravity table
2019-12-16 01:45:10 +01:00
DL6ER
2444296348
Again, Mr. Stickler
Signed-off-by: DL6ER <dl6er@dl6er.de>
2019-12-15 11:55:19 +00:00
DL6ER
4be7ebe61f
Scan domainlist instead of view to also catch disabled domains.
Signed-off-by: DL6ER <dl6er@dl6er.de>
2019-12-15 11:47:53 +00:00
DL6ER
a720fe1789
Add client trigger.
Signed-off-by: DL6ER <dl6er@dl6er.de>
2019-12-12 22:49:21 +00:00
DL6ER
2cec9eaf65
Merge pull request #3033 from pi-hole/fix/duplicates_in_adlists
Remove duplicates from adlists before importing
2019-12-12 21:37:26 +01:00
DL6ER
313f999af4
Merge pull request #3034 from pi-hole/tweak/gravity_url_displaying
Show full URL during gravity download
2019-12-12 21:37:20 +01:00
DL6ER
0b0ec43bf5
Merge pull request #3036 from pi-hole/fix/reload-lists
Improve list reloading
2019-12-12 21:37:09 +01:00
DL6ER
f0439c8d12
Add special group zero to gravity database.
Signed-off-by: DL6ER <dl6er@dl6er.de>
2019-12-12 16:39:02 +00:00
DL6ER
40e8657137
Please Mr. Stickler
Signed-off-by: DL6ER <dl6er@dl6er.de>
2019-12-12 11:18:46 +00:00
DL6ER
52dd72dfa5
Ensure output is always correct and also display if domain has been found but is disabled
Signed-off-by: DL6ER <dl6er@dl6er.de>
2019-12-12 11:08:19 +00:00
DL6ER
922ce7359c
pihole -q should also scan gravity table
Signed-off-by: DL6ER <dl6er@dl6er.de>
2019-12-12 10:58:41 +00:00
DL6ER
779fe670f7
Show full URL during gravity download instead of only domain and file
Signed-off-by: DL6ER <dl6er@dl6er.de>
2019-12-12 10:29:44 +00:00
DL6ER
570a7a5c11
Use sort -u instead of uniq as it is guaranteed to be safe when doing inline file operations.
Signed-off-by: DL6ER <dl6er@dl6er.de>
2019-12-12 10:17:59 +00:00
DL6ER
bd1b004d94
Remove possible duplicates found in lower-quality adlists
Signed-off-by: DL6ER <dl6er@dl6er.de>
2019-12-12 10:13:51 +00:00
DL6ER
5457b2c6ea
Merge pull request #2935 from pi-hole/new/internal-blocking
Per-client blocking changes
2019-12-12 09:49:02 +01:00
Adam Warner
02f3316710
Merge pull request #3031 from pi-hole/fix/do_not_force_local_resolver
Do not force nameserver 127.0.0.1 through resolvconf
2019-12-11 22:11:44 +00:00