Pi-hole/pi-hole
Pi-hole/pi-hole
1
0
Fork 0
mirror of https://github.com/pi-hole/pi-hole.git synced 2024-11-27 08:33:18 +00:00
Code Issues Projects Releases Packages Wiki Activity
6470 commits 8 branches 109 tags 27 MiB
Commit graph

573 commits

Author SHA1 Message Date
DL6ER
6b33b8b4c0
Store status of downloaded list (downloaded, using cache, some error, ...) and number of (in-)valid domains on this list in the gravity database. This updates the gravity databaes to version 14. 
Signed-off-by: DL6ER <dl6er@dl6er.de>
 2020-12-28 08:08:49 +01:00
Matej Dujava
ef0bdf6470 Fix validation of adlist url 
Already existing regex validation will be used on url after removing @ (in case
its in separating userinfo and host).

Signed-off-by: Matej Dujava <mdujava@kocurkovo.cz>
Fixes: https://github.com/pi-hole/pi-hole/issues/3911
Fixes: 7d19ee1b: validate blocklist URL before adding to the database (#3237)
 2020-12-21 12:50:02 +01:00
user
922f795fe6 Adding comma to the adlist table format 
To account for the Date Updated column in the adlist table adding a comma to the adlist to be imported into the database. This was causing intermittent failures on updates and therefore no domains would be blocked.

Signed-off-by: user <user@fedbrew.home>

 Date:      Sun Dec 6 06:03:30 2020 -0500
 Committer: user <user@fedbrew.home>

 On branch fix/gravity_date_updated
 Your branch and 'origin/fix/gravity_date_updated' have diverged,
 and have 1 and 1 different commits each, respectively.
   (use "git pull" to merge the remote branch into yours)

 Changes to be committed:
	modified:   gravity.sh

Signed-off-by: user <user@fedbrew.home>
 2020-12-06 06:08:30 -05:00
Adam Warner
948337dd8e
Merge pull request #3740 from pi-hole/new/adlist_date_updated 
Add date_updated field to adlist table
 2020-10-09 14:04:13 +01:00
DL6ER
4fd412d7c3
Add date_updated field in adlist table set when a list changes. 
Signed-off-by: DL6ER <dl6er@dl6er.de>
 2020-09-07 20:59:23 +02:00
DL6ER
aa7c3b6852
Remove gravity optimization. Further investigations have shown that it is useless because the standard gravity run already produces an optimal database. 
Signed-off-by: DL6ER <dl6er@dl6er.de>
 2020-09-07 20:50:11 +02:00
Adam Warner
bb1a6243d6
Merge pull request #3669 from pi-hole/release/v5.1.2 
Release/v5.1.2 to development
 2020-08-14 16:28:53 +01:00
Christian König
27399a762a Check for compression onyl once and print result 
Signed-off-by: Christian König <ckoenig@posteo.de>
 2020-08-03 22:46:14 +02:00
DL6ER
87da9084e6
Use compression (if available) when downloading the ad lists. 
Signed-off-by: DL6ER <dl6er@dl6er.de>
 2020-08-02 16:21:58 +01:00
DL6ER
527fc3c5cf
Also pidof -> pgrep in gravity.sh 
Signed-off-by: DL6ER <dl6er@dl6er.de>
 2020-07-21 22:03:54 +02:00
Christian König
73963fecda Use gravity's adlist_id in filename when saving downloaded adlist locally 
Signed-off-by: Christian König <ckoenig@posteo.de>
 2020-07-05 14:32:33 +02:00
Peter Lowe
dd205c55c1 Updated gravity.sh to allow semicolons 
Signed-off-by: Peter Lowe <pgl@yoyo.org>
 2020-05-26 15:53:10 +02:00
Adam Warner
9e96ff37de
Fix merge conflicts 
Signed-off-by: Adam Warner <me@adamwarner.co.uk>
 2020-05-12 22:16:43 +01:00
DL6ER
288d487fc0
Allow ( and ) in adlist URLs. 
Signed-off-by: DL6ER <dl6er@dl6er.de>
 2020-04-26 09:33:09 +02:00
DL6ER
fa57c457f3
Boldify number of unique domains as this is the actually interesting number 
Signed-off-by: DL6ER <dl6er@dl6er.de>
 2020-04-21 09:10:21 +02:00
Dan Schaper
0f20470a38
Merge pull request #3269 from pi-hole/tweak/hosts-comments 
Add support for comments in HOSTS-like files
 2020-04-15 11:11:10 -07:00
DL6ER
982c1b0059
Merge branch 'release/v5.0' into development 
Signed-off-by: DL6ER <dl6er@dl6er.de>
 2020-04-08 12:02:53 +02:00
Adam Warner
7d19ee1b25
validate blocklist URL before adding to the database (#3237) 
Signed-off-by: Adam Warner <me@adamwarner.co.uk>

Co-authored-by: DL6ER <dl6er@dl6er.de>
 2020-03-31 21:48:10 +01:00
DL6ER
7b15a88dc4
Strip comments from downloaded lists instead of discarding lines with comments altogether 
Signed-off-by: DL6ER <dl6er@dl6er.de>
 2020-03-31 18:36:40 +00:00
XhmikosR
7b8611ced0
Assorted typo fixes. (#3126) 
* Assorted typo fixes.

Signed-off-by: XhmikosR <xhmikosr@gmail.com>

Co-authored-by: Dan Schaper <dan.schaper@pi-hole.net>
 2020-03-08 16:53:14 -07:00
DL6ER
3dd05606ca
Call it the received number of domains instead of the imported number as importing does only happen a bit later. Only show the number of invalid domains if there are invalid domains. 
Signed-off-by: DL6ER <dl6er@dl6er.de>
 2020-02-24 07:06:15 +01:00
DL6ER
1e8bfd33f5
Improve output 
Signed-off-by: DL6ER <dl6er@dl6er.de>
 2020-02-23 22:50:06 +01:00
DL6ER
8131b5961c
Add comments to the code describing the changes. 
Signed-off-by: DL6ER <dl6er@dl6er.de>
 2020-02-22 15:22:29 +01:00
DL6ER
81d4531e10
Implement performant list checking routine. 
Signed-off-by: DL6ER <dl6er@dl6er.de>
 2020-02-22 13:01:55 +01:00
DL6ER
050e2963c7
Remove redundant code. 
Signed-off-by: DL6ER <dl6er@dl6er.de>
 2020-02-21 22:28:53 +01:00
DL6ER
3c09cd4a3a
Experimental output of matching line from shown warnings. 
Signed-off-by: DL6ER <dl6er@dl6er.de>
 2020-02-21 21:47:56 +01:00
DL6ER
839fe32042
Fix issue with missing newline at the end of adlists (#3144) 
* Also display non-fatal warnings during the database importing. Previously, we have only show warnings when there were also errors (errors are always fatal).

Signed-off-by: DL6ER <dl6er@dl6er.de>

* Ensure there is always a newline on the last line.

Signed-off-by: DL6ER <dl6er@dl6er.de>

* Stickler linting

Signed-off-by: Dan Schaper <dan.schaper@pi-hole.net>

* Move sed command into subroutine to avoid code duplication.

Signed-off-by: DL6ER <dl6er@dl6er.de>

* Also unify comments.

Signed-off-by: DL6ER <dl6er@dl6er.de>

* Also unify comments.

Signed-off-by: DL6ER <dl6er@dl6er.de>

Co-authored-by: Dan Schaper <dan@glacialmagma.com>
 2020-02-21 18:56:48 +00:00
DL6ER
d14ee26d6a
Merge pull request #3139 from pi-hole/fix/count_before_calling_FTL 
Fix wrong number of blocking domains shown on the dashboard
 2020-02-17 21:32:45 +01:00
DL6ER
52398052e9
Compute number of domains (and store it in the database) BEFORE calling FTL to re-read said value. 
Signed-off-by: DL6ER <dl6er@dl6er.de>
 2020-02-17 21:07:48 +01:00
DL6ER
601f9048cd
Merge pull request #3130 from pi-hole/fix/gravity_updated_timestamp 
Store gravity update timestamp only after database swapping
 2020-02-17 06:07:05 +01:00
DL6ER
dc2fce8e1d
Store gravity update timestamp only after database swapping. 
Signed-off-by: DL6ER <dl6er@dl6er.de>
 2020-02-12 19:26:25 +01:00
DL6ER
2a5cf221fa
Store number of distinct gravity domains in database after counting. 
Signed-off-by: DL6ER <dl6er@dl6er.de>
 2020-02-02 23:46:33 +01:00
DL6ER
10c2dad48a Improve gravity performance (#3100) 
* Gravity performance improvements.

Signed-off-by: DL6ER <dl6er@dl6er.de>

* Do not move downloaded lists into migration_backup directory.

Signed-off-by: DL6ER <dl6er@dl6er.de>

* Do not (strictly) sort domains. Random-leaf access is faster than always-last-leaf access (on average).

Signed-off-by: DL6ER <dl6er@dl6er.de>

* Append instead of overwrite gravity_new collection list.

Signed-off-by: DL6ER <dl6er@dl6er.de>

* Rename table gravity_new to gravity_temp to clarify that this is only an intermediate table.

Signed-off-by: DL6ER <dl6er@dl6er.de>

* Add timers for each of the calls to compute intense parts. They are to be removed before this finally hits the release/v5.0 branch.

Signed-off-by: DL6ER <dl6er@dl6er.de>

* Fix legacy list files import. It currently doesn't work when the gravity database has already been updated to using the single domainlist table.

Signed-off-by: DL6ER <dl6er@dl6er.de>

* Simplify database_table_from_file(), remove all to this function for gravity lost downloads.

Signed-off-by: DL6ER <dl6er@dl6er.de>

* Update gravity.db.sql to version 10 to have newle created databases already reflect the most recent state.

Signed-off-by: DL6ER <dl6er@dl6er.de>

* Create second gravity database and swap them on success. This has a number of advantages such as instantaneous gravity updates (as seen from FTL) and always available gravity blocking. Furthermore, this saves disk space as the old database is removed on completion.

* Add timing output for the database swapping SQLite3 call.

Signed-off-by: DL6ER <dl6er@dl6er.de>

* Explicitly generate index as a separate process.

Signed-off-by: DL6ER <dl6er@dl6er.de>

* Remove time measurements.

Signed-off-by: DL6ER <dl6er@dl6er.de>
 2020-01-24 09:39:13 -08:00
DL6ER
8f22203d24
Wait 30 seconds for obtaining a database lock instead of immediately failing if the database is busy. 
Signed-off-by: DL6ER <dl6er@dl6er.de>
 2020-01-14 20:02:00 +01:00
DL6ER
948f4a8827
Ensure permissions and ownership of gravity.db are correctly set on each run of pihole -g. This would have prevented https://github.com/pi-hole/AdminLTE/issues/1077 
Signed-off-by: DL6ER <dl6er@dl6er.de>
 2019-12-16 09:55:46 +00:00
DL6ER
2cec9eaf65
Merge pull request #3033 from pi-hole/fix/duplicates_in_adlists 
Remove duplicates from adlists before importing
 2019-12-12 21:37:26 +01:00
DL6ER
779fe670f7
Show full URL during gravity download instead of only domain and file 
Signed-off-by: DL6ER <dl6er@dl6er.de>
 2019-12-12 10:29:44 +00:00
DL6ER
570a7a5c11
Use sort -u instead of uniq as it is guaranteed to be safe when doing inline file operations. 
Signed-off-by: DL6ER <dl6er@dl6er.de>
 2019-12-12 10:17:59 +00:00
DL6ER
bd1b004d94
Remove possible duplicates found in lower-quality adlists 
Signed-off-by: DL6ER <dl6er@dl6er.de>
 2019-12-12 10:13:51 +00:00
Adam Warner
d29947ba32
optimise gravity list inserts 
Signed-off-by: Adam Warner <me@adamwarner.co.uk>
 2019-12-09 22:30:41 +00:00
Adam Warner
1f03faddef
shell check recomends 
Signed-off-by: Adam Warner <me@adamwarner.co.uk>
 2019-12-09 21:35:54 +00:00
DL6ER
3231e5c3ba
Address stickler requests. 
Signed-off-by: DL6ER <dl6er@dl6er.de>
 2019-12-09 16:52:03 +00:00
DL6ER
0c5185f8ba
Also display how many unique domains we have caught in the event horizon. 
Signed-off-by: DL6ER <dl6er@dl6er.de>
 2019-12-04 21:02:46 +00:00
DL6ER
037d52104a
New command "pihole -g -r" recreates gravity.db based on files backed up in /etc/pihole/migration_update. This is useful to restore a working version of the database when the user destroyed the original database. Also, update gravity.db to version 5 because of a fix we needed to implement. 
Signed-off-by: DL6ER <dl6er@dl6er.de>
 2019-11-26 10:58:39 +01:00
DL6ER
756c99653e
Merge branch 'development' into new/internal-blocking 2019-10-03 12:01:27 +02:00
Adam Warner
61a40c1b43
merge devel into 4.3.2 And Resolve merge conflicts 
Signed-off-by: Adam Warner <me@adamwarner.co.uk>
 2019-09-22 01:16:44 +01:00
DL6ER
a8af2e1837
Store domains without sorting and unifying them first. This allows us to preserve the relationship of the individual domains to the lists they came from. 
Signed-off-by: DL6ER <dl6er@dl6er.de>
 2019-09-04 23:14:29 +02:00
Adam Warner
8bef5dc805
remove n from -ne 
Signed-off-by: Adam Warner <me@adamwarner.co.uk>
 2019-09-03 23:56:23 +01:00
Adam Warner
ad41bcca5a
Remove support for adblock style lists to prevent false positives 
Signed-off-by: Adam Warner <me@adamwarner.co.uk>
 2019-09-03 23:43:11 +01:00
DL6ER
ca8982494b
Store timestamp when the gravity table was last updated successfully. This fixes https://github.com/pi-hole/AdminLTE/issues/989 
Signed-off-by: DL6ER <dl6er@dl6er.de>
 2019-09-01 14:42:07 +02:00
DL6ER
1820c2c598
Merge branch 'development' into new/whitelist-regex-support 
Signed-off-by: DL6ER <dl6er@dl6er.de>
 2019-08-22 14:19:51 +02:00
Andreas
c53be459c6
quick fix for when dig also returns a CNAME 
Signed-off-by: Andreas <ryrun@online.de>
 2019-08-21 04:37:06 -07:00
DL6ER
4371c9ba03
Ensure proper permissions are set for gravity.db after creation. 
Signed-off-by: DL6ER <dl6er@dl6er.de>
 2019-08-05 21:20:07 +02:00
Andreas
63230cb72d
quick fix for when dig also returns a CNAME 
Signed-off-by: ryrun <ryrun@online.de>
 2019-08-04 21:21:08 +02:00
DL6ER
5ff9052200
Review comments 
Signed-off-by: DL6ER <dl6er@dl6er.de>
 2019-07-09 11:41:44 +02:00
DL6ER
f5121c64be
We should still add the regex lines (initially) to the regex table as the renaming will happen only after the importing. 
Signed-off-by: DL6ER <dl6er@dl6er.de>
 2019-07-08 21:39:30 +02:00
DL6ER
054c7a2c05
Create new table + view regex_whitelist + rename old regex table to regex_blacklist. This updates the gravity.db version to 3. 
Signed-off-by: DL6ER <dl6er@dl6er.de>
 2019-07-08 21:35:31 +02:00
DL6ER
be3e198f9a
Address linting errors. 
Signed-off-by: DL6ER <dl6er@dl6er.de>
 2019-07-07 10:46:20 +02:00
DL6ER
acc50b709e
Only migrate files once (domain and adlist lists druing initial creation of gravity.db auditlog.list on database upgrade from version 1 to 2. 
Signed-off-by: DL6ER <dl6er@dl6er.de>
 2019-07-07 10:33:08 +02:00
DL6ER
0405aaa3da
Review comments and fixing stickler complaints. 
Signed-off-by: DL6ER <dl6er@dl6er.de>
 2019-07-06 09:32:41 +02:00
DL6ER
2fb4256f84
Rename table to "auditlist" 
Signed-off-by: DL6ER <dl6er@dl6er.de>
 2019-07-05 16:28:36 +02:00
DL6ER
4f4a12bb40
Upgrade database if necessary and store audit domains therein. 
Signed-off-by: DL6ER <dl6er@dl6er.de>
 2019-07-05 14:03:57 +02:00
DL6ER
7a1b6fd205
Merge branch 'development' into new/gravity.db_domain_groups 2019-07-03 19:21:55 +02:00
DL6ER
f91421418a
Rename adlists -> adlist table. 
Signed-off-by: DL6ER <dl6er@dl6er.de>
 2019-06-30 23:21:10 +02:00
DL6ER
87c115fc86
Change implementation to use linking tables (domain_groups, adlists_groups). This allows adding domains to multiple lists conveniently. 
Signed-off-by: DL6ER <dl6er@dl6er.de>
 2019-06-30 19:04:38 +02:00
DL6ER
d5e8f1a781
Add 10 seconds timeout to the sqlite3 command that writes tables to the gravity database. This prevents gravity from failing when the database file is locked for a short time. 
Signed-off-by: DL6ER <dl6er@dl6er.de>
 2019-06-26 18:20:36 +02:00
DL6ER
cbc210b014
Add adlist_groups and mark group_id columns as NOT NULL. 
Signed-off-by: DL6ER <dl6er@dl6er.de>
 2019-06-16 17:05:33 +02:00
Chris Crocker-White
69dba022c4 Handle an empty local.list 
Handle the case of an empty local.list file which would otherwise prevent the system from starting

Change-type: patch
Signed-off-by: Chris Crocker-White <chriscw@balena.io>
 2019-05-27 10:27:28 -07:00
Dan Schaper
7de6bbca61
Merge branch 'development' into release/v4.3 2019-05-18 13:18:14 -07:00
Mcat12
8e1bf6e3d4
Merge branch 'development' into new/gravitydb 
# Conflicts:
#	advanced/Scripts/list.sh
#	gravity.sh
#	pihole
 2019-05-12 15:20:52 -07:00
pvogt09
3f90261520 Moves chmod after every creation of ${destination} instead of end of function 
Signed-off-by: pvogt09 <50047961+pvogt09@users.noreply.github.com>
 2019-05-12 11:25:48 +02:00
pvogt09
225285cb97 Removes chmod for ${source} and superfluous chmod for ${destination} 
Signed-off-by: pvogt09 <50047961+pvogt09@users.noreply.github.com>
 2019-05-11 11:14:07 +02:00
pvogt09
a3d2a1062f Makes blocklists world readable 
Signed-off-by: pvogt09 <50047961+pvogt09@users.noreply.github.com>
 2019-05-10 11:13:19 +02:00
Mcat12
ec06fd0ad4
Fix incorrect adlist query when an adlist is blocked during gravity 
hosts-file.net was hard-coded as the domain to check instead of the
actual domain.

Signed-off-by: Mcat12 <newtoncat12@yahoo.com>
 2019-05-04 11:35:21 -07:00
DL6ER
0cfecd31d3
Invert logic to reduce nesting 
Signed-off-by: DL6ER <dl6er@dl6er.de>
 2019-05-01 22:07:14 +02:00
DL6ER
008e88b84b
Only migrate legacy list files when we create gravity.db in the same gravity run 
Signed-off-by: DL6ER <dl6er@dl6er.de>
 2019-05-01 18:04:31 +02:00
DL6ER
540c57f755
Add comment for automigrated list entries 
Signed-off-by: DL6ER <dl6er@dl6er.de>
 2019-05-01 17:51:25 +02:00
DL6ER
983117d788
Add id (primary key, autoincrement) field to lists tables 
Signed-off-by: DL6ER <dl6er@dl6er.de>
 2019-05-01 17:47:15 +02:00
DL6ER
b372f808dd
Fix lint errors 
Signed-off-by: DL6ER <dl6er@dl6er.de>
 2019-05-01 17:06:14 +02:00
pvogt09
8a92fb24c4 Incorporates changes requested by @Mcat12 
Signed-off-by: pvogt09 <50047961+pvogt09@users.noreply.github.com>
 2019-05-01 11:20:26 +02:00
pvogt09
5c575e73c7 Adds world readable attribute to files created by Pi-hole to circumvent #2724 
Signed-off-by: pvogt09 <50047961+pvogt09@users.noreply.github.com>
 2019-04-30 22:41:12 +02:00
DL6ER
a932209143
Set default for date_modified to be the current time 
Signed-off-by: DL6ER <dl6er@dl6er.de>
 2019-04-29 17:21:59 +02:00
DL6ER
4946b5907b
Move migrated files to backup directory instead of deleting them. This has the benefit that gravity.db can be recreated at any time by: 1. deleting gravity.db, 2. copying all files from the migration_backup directory back to /etc/pihole, 3. running pihole -g again. 
Signed-off-by: DL6ER <dl6er@dl6er.de>
 2019-04-28 22:15:47 +02:00
Mark Drobnak
4596d9a577
Merge branch 'development' into new/gravitydb 2019-04-26 23:40:28 -04:00
DL6ER
8524aecfed
Don't set gravity database permissions 
Signed-off-by: DL6ER <dl6er@dl6er.de>
 2019-04-26 17:56:11 +02:00
DL6ER
3dcb9722d3
Remove global haveSourceUrls variable and just skip downloading and processing adlists if there aren't any to be downloaded 
Signed-off-by: DL6ER <dl6er@dl6er.de>
 2019-04-26 17:39:35 +02:00
DL6ER
bd0215524b
Add vw_adlists view which returns only enabled adlists 
Signed-off-by: DL6ER <dl6er@dl6er.de>
 2019-04-26 17:36:00 +02:00
DL6ER
53e3ff2b24
Only add non-empty lines when moving file content into database tables 
Signed-off-by: DL6ER <dl6er@dl6er.de>
 2019-04-26 17:32:05 +02:00
DL6ER
0efc46260f
Use both old and new strings for detecting whether this is a Pi-hole dnsmasq config file 
Signed-off-by: DL6ER <dl6er@dl6er.de>
 2019-04-26 16:16:30 +02:00
DL6ER
d94bbfca7f
Finish migrating gravity.db to using our new gravity database. This removes the --skip-download, --blacklist-only, --whitelist-only, and --wildcard-only options as it is not necessary to run gravity when modifying them from now on (gravity really only builds the gravity database and the local.list file). 
Signed-off-by: DL6ER <dl6er@dl6er.de>
 2019-04-25 11:23:06 +02:00
DL6ER
465a39427b
Source adlists from gravity database 
Signed-off-by: DL6ER <dl6er@dl6er.de>
 2019-04-25 10:46:18 +02:00
DL6ER
2664ac0efc
Improve error reporting 
Signed-off-by: DL6ER <dl6er@dl6er.de>
 2019-04-25 10:18:37 +02:00
DL6ER
c1277705d8
Copy existing whitelist.txt, blacklist.txt, regex.list, and adlists.list to the database. We remove the files afterwards as the content lives in the database now 
Signed-off-by: DL6ER <dl6er@dl6er.de>
 2019-04-25 10:14:45 +02:00
DL6ER
e57d0fb93c
Remove the empty templates gravity.list, white.list, black.list. The old source files whitelist.txt, blacklist.txt, and regex.list still exist and need to be removed in a follow-up commit 
Signed-off-by: DL6ER <dl6er@dl6er.de>
 2019-04-24 19:05:08 +02:00
DL6ER
ee20164207
Remove detect_pihole_blocking_status() as we can modify the database at any time 
Signed-off-by: DL6ER <dl6er@dl6er.de>
 2019-04-24 19:01:31 +02:00
Dan Schaper
54ae9d83b0 Stickler-lints 
Signed-off-by: Dan Schaper <dan.schaper@pi-hole.net>
 2019-02-23 13:42:15 -08:00
DL6ER
bbc392caeb
More review comments 
Signed-off-by: DL6ER <dl6er@dl6er.de>
 2019-02-22 22:49:02 +01:00
DL6ER
61840edb30
Fill comment field with NULL (default) instead of empty string and use a variable for the timestamp instead of periodically calling date 
Signed-off-by: DL6ER <dl6er@dl6er.de>
 2019-02-22 22:46:19 +01:00
DL6ER
f9250d91a5
Explictly escape the escape. 
Signed-off-by: DL6ER <dl6er@dl6er.de>
 2019-02-06 19:17:13 +01:00
DL6ER
491c828661
Improved code, ensure that gravity.sh passes shellcheck as a whole 
Signed-off-by: DL6ER <dl6er@dl6er.de>
 2019-02-06 19:09:09 +01:00
DL6ER
644ec36e64
Review comments 
Signed-off-by: DL6ER <dl6er@dl6er.de>
 2019-02-06 18:57:48 +01:00
DL6ER
ec5b16ef85
Fix incomplete comment 
Signed-off-by: DL6ER <dl6er@dl6er.de>
 2019-02-05 19:08:08 +01:00
DL6ER
dbbf21071b
Invert logic to use ENABLED instead of DISABLED. We make sure to set "enabled = 1 WHERE enabled IS NULL" in gravity_store_in_database() 
Signed-off-by: DL6ER <dl6er@dl6er.de>
 2019-02-05 19:05:11 +01:00
DL6ER
b2f4385232
Avoid infinity loop if gravity optimization fails 
Signed-off-by: DL6ER <dl6er@dl6er.de>
 2019-02-03 17:05:00 +01:00
DL6ER
1f9a7d0b0c
Only run VACUUM on "pihole -g -o" 
Signed-off-by: DL6ER <dl6er@dl6er.de>
 2019-02-03 17:01:38 +01:00
DL6ER
d5feffa117
Do not backup the database 
Signed-off-by: DL6ER <dl6er@dl6er.de>
 2019-02-03 16:45:19 +01:00
DL6ER
fc62cf7e2f
Only run VACUUM on "full" gravity runs 
Signed-off-by: DL6ER <dl6er@dl6er.de>
 2019-02-03 16:45:06 +01:00
DL6ER
93f1859bab
Store regex in gravity.db as well 
Signed-off-by: DL6ER <dl6er@dl6er.de>
 2019-02-03 15:13:18 +01:00
DL6ER
dcf0a605cf
Use views for all tables and set disabled column to false for those tables that support it 
Signed-off-by: DL6ER <dl6er@dl6er.de>
 2019-02-03 14:14:14 +01:00
DL6ER
8a91fe6992
Store domains in gravit tables: gravity, whitelist, blacklist 
Signed-off-by: DL6ER <dl6er@dl6er.de>
 2019-02-03 13:21:26 +01:00
DL6ER
09c4c88a6d
Create and fill gravity.db during pihole -g 
Signed-off-by: DL6ER <dl6er@dl6er.de>
 2019-02-03 13:04:31 +01:00
xch12i5
cd8a423b32 Solves RegEx issue and updates documentation. 
Signed-off-by: xch12i5 <40517505+xCh12i5@users.noreply.github.com>
 2019-01-20 16:09:25 +01:00
xch12i5
e02da90faa Replaces double quotes by single quotes and updates documentation. 
Signed-off-by: xch12i5 <40517505+xCh12i5@users.noreply.github.com>
 2019-01-19 17:52:28 +01:00
xch12i5
0de5132e2b Solves RegEx issue. 
Signed-off-by: xch12i5 <40517505+xCh12i5@users.noreply.github.com>
 2019-01-19 17:07:27 +01:00
DL6ER
05bd88b4d8
Update agent string: the selected string is the most abundant one on 06 Dec 2018 with a percentage of almost 10% 
Signed-off-by: DL6ER <dl6er@dl6er.de>
 2018-12-06 18:04:17 +01:00
Adam Warner
83f7bd66e6
Don't mv preEventHorizon, copy it. 
Signed-off-by: Adam Warner <adamw@rner.email>
 2018-12-05 22:29:01 +00:00
Dan Schaper
5c621fa253
Lints 
Signed-off-by: Dan Schaper <dan.schaper@pi-hole.net>
 2018-11-12 17:21:34 -08:00
Dan Schaper
f9e71bee0a
Account for port, or lack of, in PIHOLE_DNS_1 
Signed-off-by: Dan Schaper <dan.schaper@pi-hole.net>
 2018-11-12 17:15:24 -08:00
Mark Drobnak
33939a30ea
Merge pull request #2347 from Gusher123/patch-1 
Don't let one blocklist block an other blocklist
 2018-09-30 21:24:45 -04:00
Adam Warner
0131055ec9
Submission stale, addressing @mcat12's comments 
Signed-off-by: Adam Warner <adamw@rner.email>
 2018-09-30 20:00:06 +01:00
DL6ER
337cc5ca18
BLOCKING -> BLOCKING_ENABLED 
Signed-off-by: DL6ER <dl6er@dl6er.de>
 2018-08-13 17:22:10 +02:00
DL6ER
b011adc453
Use new setupVars BLOCKING variable to determine which file to update via gravity 
Signed-off-by: DL6ER <dl6er@dl6er.de>
 2018-08-13 14:19:59 +02:00
DL6ER
6cde066edd
Have gravity detect the presence of files like gravity.list.bck and update this file if present (assuming blocking is disabled) 
Signed-off-by: DL6ER <dl6er@dl6er.de>
 2018-08-13 13:43:14 +02:00
Gusher123
2feb5f22cc
Signed-off-by 
This is my commit message

Signed-off-by: Gusher123 <t.siebeling@gmail.com>
 2018-08-12 10:05:58 +02:00
Gusher123
03a9d3bc5b Not hardcoding dns and covering all blockingmodes 
Using double brackets

Mode double brackets
 2018-08-12 03:05:53 +02:00
Gusher123
e58142b4ac
Added comments 2018-08-11 15:14:45 +02:00
Gusher123
3e5c6afaee
Don't let one blocklist, block an other 
If a blocklist is blocked by an other and 'pihole -g' is run, update the blocked blocklist by useing an alternate dns server (in this case 1.1.1.1) to acquire the IP adres and download the blocklist using the 'curl --resolve' (https://curl.haxx.se/docs/manpage.html#--resolve)
 2018-08-11 14:33:33 +02:00
DL6ER
760dd01b50
Use grep invert-match 
Signed-off-by: DL6ER <dl6er@dl6er.de>
 2018-08-07 18:36:06 +02:00
DL6ER
5bf90cb77d
Use alternative logic for counting regex file entries, skipping commented out lines 
Signed-off-by: DL6ER <dl6er@dl6er.de>
 2018-08-07 18:02:01 +02:00
Mcat12
1a8d5fb4a9
Allow PHP to edit the regex file if installed 
Fixes permissions error on the web interface when modifying the regex.

Signed-off-by: Mcat12 <newtoncat12@yahoo.com>
 2018-07-28 21:57:39 -04:00
Dan Schaper
dbc82cfb6a
Merge pull request #2236 from pi-hole/new/regex-lists 
Use regex lists instead of wildcards for blocking
 2018-07-14 06:10:44 -07:00
Mcat12
72be7ad5f0
Don't count commented out regex lines 
Signed-off-by: Mcat12 <newtoncat12@yahoo.com>
 2018-07-12 21:21:06 -04:00
WaLLy3K
7ddae8f2eb
Treat URLs without a protocol as HTTP 
Signed off by WaLLy3K <wally3k@pi-hole.net>
 2018-07-02 19:59:22 +10:00
Mcat12
0f4968d75c
Merge remote-tracking branch 'origin/development' into new/regex-lists 2018-06-29 23:12:34 -04:00
Mcat12
55175087c4
Set regex file permissions for web access 
Signed-off-by: Mcat12 <newtoncat12@yahoo.com>
 2018-06-28 22:52:44 -04:00
Dan Schaper
d5547f5c7c
Revert "Fix error: /opt/pihole/gravity.sh: 385: Warning: command substitution: ignored null byte in input" 2018-06-24 16:06:55 -07:00
Mark Drobnak
4fc3546887
Merge pull request #2069 from pi-hole/fix/nullbyte 
Fix error: /opt/pihole/gravity.sh: 385: Warning: command substitution: ignored null byte in input
 2018-06-20 20:42:24 -04:00
DL6ER
bc705aac03
Add automated wildcard list -> regex filter conversion 
Signed-off-by: DL6ER <dl6er@dl6er.de>
 2018-06-17 14:39:19 +02:00
Mark Drobnak
3149a95d6a
Fix gravity_ParseFileIntoDomains Awk comment 
Signed-off-by: Mark Drobnak <newtoncat12@yahoo.com>
 2018-06-03 13:33:33 -04:00
Adam Warner
5f2217a079
Fix for gravity.list not being emptied with an empty/missing adlist file. 
Variable output depending on content of /existence of adlist

Signed-off-by: Adam Warner <adamw@rner.email>
 2018-04-30 21:56:33 +01:00
Adam Warner
3216d65353
Merge branch 'development' into new/RemoveDefaultBlocklists 2018-04-29 13:39:17 +01:00
Adam Warner
9c29bd627e
fix user blacklisting 2018-04-21 23:55:33 +01:00
DL6ER
d24b141d8b
Get rid of numBlocked file, FTLDNS can now determine number of domains thanks to the new lists format 
Signed-off-by: DL6ER <dl6er@dl6er.de>
 2018-04-21 23:58:42 +02:00
DL6ER
bac9e005ae
Generate gravity.list and black.list in simple domain lists format for FTLDNS. Leave local.list in HOSTS format 
Signed-off-by: DL6ER <dl6er@dl6er.de>
 2018-04-21 23:27:36 +02:00
Dan Schaper
d4d1ec2568
Merge pull request #2124 from pi-hole/development 
Internal
 2018-04-17 11:54:50 -07:00
Adam Warner
88d4108f92
- Add new whipstail to choose from example blocklists on new install 
- remove adlists.default
- do not copy non-existent adlists.default on run of gravity

Signed-off-by: Adam Warner <adamw@rner.email>
 2018-04-17 10:33:06 +01:00
Jacob Salmela
187848660c
try another fix for #2028 
Signed-off-by: Jacob Salmela <jacob.salmela@pi-hole.net>
 2018-03-31 16:12:29 -05:00
Jacob Salmela
c3f391dc5a
maybe fixes #2028 by deleting null bytes 
Signed-off-by: Jacob Salmela <jacob.salmela@pi-hole.net>
 2018-03-31 14:08:48 -05:00
DL6ER
ff47ab12fb
Save number of effectively blocked domains in /etc/pihole/numBlocked 
Signed-off-by: DL6ER <dl6er@dl6er.de>
 2018-03-30 21:01:51 +02:00
Riley Avron
f390671018 Improve scheme matching to avoid inadvertent blocking 
Signed-off-by: Riley Avron <riley.avron@gmail.com>
 2018-03-06 22:03:56 -08:00
Riley Avron
512ec7fb87 Remove unneeded ifs in awk 
Signed-off-by: Riley Avron <riley.avron@gmail.com>
 2018-03-06 22:03:43 -08:00
Adam Warner
24219422df
Merge branch 'development' into FTLDNSremoveDnsmasq 2018-03-06 20:43:27 +00:00
Adam Warner
ad1f210528
As we no longer add source lists to the whitelist by default, gravity falls over on new installs. 
Signed-off-by: Adam Warner <adamw@rner.email>
 2018-03-06 18:29:28 +00:00
Adam Warner
cadab61a21
Merge branch 'development' into FTLDNS 2018-03-05 21:33:53 +00:00
DL6ER
7ed45d26c2
Remove " " at some places where they shouldn't be 
Signed-off-by: DL6ER <dl6er@dl6er.de>
 2018-02-26 17:26:51 +01:00
DL6ER
e8ec0dc701
Use pihole-FTL resolver in pihole and gravity 
Signed-off-by: DL6ER <dl6er@dl6er.de>
 2018-02-25 09:34:04 +01:00
Adam Warner
6b4685b333
much more elegant to export LC_ALL at the beginning of the script! 
Signed-off-by: Adam Warner <adamw@rner.email>
 2018-02-21 11:33:29 +00:00
Adam Warner
d4ac818a0b
Specifty LC_ALL=C when dealing with sorted lists. 
Signed-off-by: Adam Warner <adamw@rner.email>
 2018-02-21 11:23:07 +00:00
DL6ER
fc5034696d
Don't auto-whitelist blocking list domains 
Signed-off-by: DL6ER <dl6er@dl6er.de>
 2018-02-16 19:42:33 +01:00
Dan Schaper
9f2ce0e296
Merge pull request #1910 from pi-hole/fix/BinaryFileLineInGravity 
Potential fix for #1909
 2018-01-28 03:23:19 -08:00
Adam Warner
55e10d8287
replace another grep -F as pointed out by @StarPicard 
Signed-off-by: Adam Warner <adamw@rner.email>
 2018-01-28 00:50:15 +00:00
Adam Warner
f1956119fb
proposed gravity function renames 
Signed-off-by: Adam Warner <adamw@rner.email>
 2018-01-14 20:38:39 +00:00
Adam Warner
876170767c
Print actual domain instead of grep output of "Binary file /etc/pihole/list.preEventHorizon matches" when a domain with a unicode char is in a source list. 
Signed-off-by: Adam Warner <adamw@rner.email>
 2018-01-13 16:43:20 +00:00
Adam Warner
2d3906450a
add back in str local var, and change ticks to info on whitelist lines 
Signed-off-by: Adam Warner <adamw@rner.email>
 2017-12-18 20:14:53 +00:00
Adam Warner
5ecc46d85b
Accidentally too many spaces 
Signed-off-by: Adam Warner <adamw@rner.email>
 2017-12-16 14:03:12 +00:00
Adam Warner
ddd6f3dbe5
Grammar changes. Lets not work out programmatically whether we need to pluralise a word. It's not needed. 
Signed-off-by: Adam Warner <adamw@rner.email>
 2017-12-16 13:55:52 +00:00
Adam Warner
761fd13a70
Remove quotes from the uniqDomains array to actually whitelist the sources 
Signed-off-by: Adam Warner <adamw@rner.email>
 2017-12-16 13:15:26 +00:00
Adam Warner
c24e5cd3f6
:codacy: :codacy2: :codacy3: 
Signed-off-by: Adam Warner <adamw@rner.email>
 2017-12-12 12:45:25 +00:00
Adam Warner
cc3035267f
Revert parsing method to that of 3.1.4. New Method is allowing illegal lists through, and that is causing issues such as blocking www.google.com. This method will just throw away invalid list entries. 
Signed-off-by: Adam Warner <adamw@rner.email>
 2017-12-12 12:36:09 +00:00
Jacob Salmela
80e17ab721
potentially fixes #1806 by falling back to dig if getent fails 
Signed-off-by: Jacob Salmela <github@decoy.email>
 2017-12-07 22:38:47 -06:00
Mcat12
8d721d086c
Modify indentation 2017-11-21 18:37:38 -05:00
DL6ER
6f0bb30def
Improve indentation 
Signed-off-by: DL6ER <dl6er@dl6er.de>
 2017-11-21 20:58:27 +01:00
DL6ER
c54f04ef4c
Repair username:password filtering 
Signed-off-by: DL6ER <dl6er@dl6er.de>
 2017-11-21 20:55:47 +01:00
DL6ER
eb83081a5c
Clarify error message: File could also have been of zero size 
Signed-off-by: DL6ER <dl6er@dl6er.de>
 2017-11-21 18:36:34 +01:00
DL6ER
8976930e20
Properly detect if local file was successfully downloaded 
Signed-off-by: DL6ER <dl6er@dl6er.de>
 2017-11-21 18:35:58 +01:00
DL6ER
164a81776e
Allow to add local lists to gravity using e.g. file:///path/to/my.list in adlists.list 
Signed-off-by: DL6ER <dl6er@dl6er.de>
 2017-11-21 18:30:40 +01:00
DL6ER
39ab1e1ea7
When using the Pi-hole DHCP server, local host names are called "something.local". Thir PR ensures that clients that are connected via VPN are augmented by a similar suffix like "someother.vpn". 
Signed-off-by: DL6ER <dl6er@dl6er.de>
 2017-09-30 13:03:49 +02:00
spacedingo
60365ad36a Update gravity.sh 2017-09-26 09:25:47 +04:00
spacedingo
466e6d9b30 Redo pull #1687 
Fix for case sensitive duplicate domains
The change (https://github.com/pi-hole/pi-hole/pull/1687) was recently undone.
 2017-09-26 05:53:07 +04:00
WaLLy3K
b9bcfe36ff Place errors on newline 2017-09-24 11:57:15 +10:00
WaLLy3K
cd7c00ef8e Fix Ctrl-C inadvertently deleting domains files 2017-09-23 10:32:56 +10:00
WaLLy3K
3aa525b0c0 Ensure domains files are not deleted upon w/blist 2017-09-22 14:17:56 +10:00
WaLLy3K
47099e2855 Ensure resolver test occurs each second 
* Ensure that gravity will run the second the resolver is available
* Increase timeout to 120s
 2017-09-22 03:56:53 +10:00
WaLLy3K
2deb2bf03f Fix broken whitelist functionality 2017-09-22 02:23:43 +10:00
WaLLy3K
a6f9272d4b Fix gravity from only parsing one adlist URL 
* Redirect `grep` correctly to $sources (instead of using `awk`)
* Redirect $sourceDomains correctly
* Replace use of ${COL_LIGHT_BLUE}
* Add numeric count informing user of unique source domains being whitelisted
 2017-09-20 22:25:33 +10:00
Adam Warner
8d8482d60b
<<<$() back to < <() 
Signed-off-by: Adam Warner <adamw@rner.email>
 2017-09-19 19:02:50 +01:00
WaLLy3K
d02bf258af Improve non-standard list parsing 
* Add 504 status (Gateway connection timed out)
* Add text for non-standard list parsing
* Improve adblock parsing
* Ensure adblock exception rules are removed from file
* Ensure "www." is not treated as a URL-format list
* Corrected typo
* Ensure script does not fail if "-f" is used when there are no blocklists generated

Signed off by WaLLy3K <wally3k@pi-hole.net>
 2017-09-18 17:36:03 +10:00
WaLLy3K
c957124fad Optimised parsing of domains on IPv6 servers 
* Remove WHITELIST_COMMAND
* Place IPv4/IPv6 availability test underneath setupVars.conf source
* Improved clarity on comments
* Define default lookupDomain on local line
* Use `getent hosts` instead of nslookup (faster)
* Make gravity_DNSLookup() function more readable
* Use bold on "Neutrino emissions detected"
* Swap conditionals around on adlists file handling
* Add comments to both gravity_Collapse() `awk`s
* Removed unnecessary "${str}" from gravity_Pull()
* Merge function variables into local line
* Place .phgbp suffice on mktemp, so patternbuffers can be cleaned up all at once in gravity_Cleanup()
* Removed success="false" from $httpCode case, placed empty success var in local
* Reordered $httpCode case numerically because I can
* Provide error if Dnsmasq format list is being parsed
* Remove IPv4 check when determining URL list (too slow on large lists)
* Check ${#sources[@]} to ensure we're checking the number of entries and not the character count
* Define empty plural in local line, removing unnecessary plural=;
* Optimised readability of gravity_Whitelist()
* Removed uninformative "Nothing to blacklist"/"No wildcards used" text
* Optimised parsing of domains into hosts format on IPv6 enabled servers
* Ensure /etc/hostname is non-zero
* Use `: >` instead of `rm` as consistent with the rest of the script
* Ensured that gravity_Cleanup() removes ${localList}.tmp
* Optimised readability of gravity_ParseUserDomains()
* Moved dnsRestart to ${var} case statement, renaming it to dnsRestartType for readability
* Set default $listType to ensure script passes "bash strict mode"
 2017-09-15 22:39:17 +10:00
WaLLy3K
ff5411a93a Add 'Connection Refused' for 000 status 
* Shift default dnsRestart value into unset default parameter expansion value
* Change nslookup timeout to 5 seconds 
* Use &> instead of >
* Standardise plural code
* Update some comments
* Add "000" when connection is refused
* Condense adblock detection logic
* Add Dnsmasq format detection and parsing logic
* Removed unnecessary echo
* Add dnsWasOffline variable to ensure that if DNS service has been stopped, that it doesn't start and also get reloaded
 2017-09-14 20:23:49 +10:00
WaLLy3K
d3073e5e23 Fix array of unique $sourceDomains 2017-09-14 17:09:52 +10:00
WaLLy3K
8191ec01e5 Gravity Bugfixes 
* Merge development OpenVPN code
* Determine which domain to resolve depending on existence of $localList
* Re-add code to remove $localList, preventing duplicate local entries
* Minor shellcheck validation fix
 2017-09-14 16:39:30 +10:00
WaLLy3K
0a00936e99 Update resolver test & added more comments 
* Add/update code comments
* Change resolver check to test for pi.hole
* Make resolver check timeout after 10 seconds
* Use > instead of &> where appropriate
* Make resolver check sleep for 30 seconds (effectively waiting up to 50s for dnsmasq to be resolvable)
* Provide confirmation upon success of resolver check availability
* Quotes and Braced remaining variables as appropriate
* Removed duplicate local
 2017-08-28 11:36:02 +10:00
WaLLy3K
f24ab8508e WIP cleanup 
* Changed supernova/eventHorizon variables to match their purpose
* Add gravity_DNSLookup() function
* Ensure all comments are clear and relevant
* Use && instead of || in gravity_Collapse()
* Renamed existing functions, and placed them in order of script execution
* Use \t instead of literal tab in gravity_ParseFileIntoDomains()
* Replace instances of "truncate" with : > (e.g: gravity_Schwarzschild())
* Ensure correct variables are local'd
* Use phrase "Cleaning up stray matter" when gravity_Cleanup() is called
* Add black/white/wildcard switches for list.sh
* Ensure necessary functions are called when modifying black/white/wildcards

Signed-off-by: WaLLy3K <wally3k@pi-hole.net>
 2017-07-27 12:34:35 +10:00
WaLLy3K
6830b08723 Clean up and optimise Gravity 
* Shellcheck validation
* Made variable names, function names, comments and output clearer to understand
* Quoted and braced variables and conditionals
* Fix adlists.list handling logic, and remove any CR line endings
* Make CP/MV/RM provide user-friendly output upon failure
* Change adlists.list retrieval logic
* Moved and fixed adlists.list domain parsing logic
* Create gravity_ParseFileAsDomains() function to handle parsing of source files
* If no changes to a list is detected, print no output
* Ensure each source blocklist has a final newline
* Format number output as currency
* Make array of adlists domain sources unique to prevent redundant whitelisting
* Merged bash IPv4/IPv6 hosts formatting IF statement into an awk one-liner
* Trap Ctrl-C cancellations and run gravity_Cleanup()
* Use new gravity_Cleanup() function on errors and script completion
* Ensure that dnsmasq uses force-reload when gravity is invoked
* Add --wildcard option to ensure dnsmasq is restarted upon b/wlisting of a wildcard

Signed-off-by: WaLLy3K <wally3k@pi-hole.net>
 2017-07-24 21:24:34 +10:00
Mcat12
1e31fa89aa Remove /* from IPv6 as well as IPv4 
Fixes Discourse issue: https://discourse.pi-hole.net/t/ipv6-aaaa-dns-issue/3830
 2017-06-26 09:51:21 -04:00
Adam Warner
536585b846 Colourise Core Output Text (#1471) 
* Define colours within COL_TABLE
* Do not output colours for non-terminal instances
* Removed ":::"
* Fixed indenting & spacing
* Made output consistent throughout project
* Reworded text to fit on standard 80 char wide Terminal screen
* Made 'sudo raspi-config' warning (insufficient disk space) only show on RPi
* Make "Installation/Update Complete" the final msg
* Remove redundant messages
* Simplify update available message
* Confirm user would like to begin uninstall
* If "git pull" string says "Already up-to-date.", place [i] before it
* Colour Temp/Interface output
* Made `pihole disable 5z` invalid
* Added error fallback if invalid argument (not s/m) is detected
* Quoted "$2" for consistency
* Updated help text
* L185/286: Replaced echo with redirect
* User agents for adblock.mahakala.is/adaway.org unnecessary 
* Print newline on confirmation of repository reset
* Add output to admin-related dnsmasq restarts
* Return error message for "pihole -q"
* Imply default checkout behaviour with y/N
* Fix uninstall failing to remove pihole user
* Print checkout 'git remote show origin' STDERR on new line
* Replaced checkout "AdminLTE" wording with "Web Admin"
 2017-06-21 21:49:05 +10:00
DL6ER
05798fe07a
cp + rm === mv (well, almost) 2017-06-19 20:22:03 +02:00
DL6ER
8bad56e897
Improve displayed messages and overall logic 2017-06-17 14:24:30 +02:00
DL6ER
92e691408f
Remove useless cat 2017-06-17 14:14:07 +02:00
DL6ER
0a9c234127
Add "pihole -g -b" to *only* update black.list (saves a bunch of time when adding/changing only blacklisted files - won'tdownload lal lists, but only processes the blacklist and restars dnsmasq) 2017-06-17 13:57:27 +02:00