Storing the output from 'curl' commands directly as shell variables is
very inefficent, and requires much more RAM gravity.sh any time there is
an update to the block lists (and especially on the first run). Store
the raw blocklists in a temporary file on disk, and process those.
This will increase the swap file to 500MB before downloading the lists.
Most of the issue comes from the mahakala list, which is so large. If
no swap file is found, one is created.
Prepend "^" to start of latentWhitelist.txt lines.
The -x switch requires a full line match of the regexp, where as -w
will try to find the match somewhere in the line, looking for work
breaks. Combined with turning the whitelist lines into full regexps,
this results in significantly faster parsing.
Having "^" prepended to the lines also keeps false whitelisting from
occuring, such as the following example:
If whitelist.txt contains "google.com" it would whitelist many other
sites that end in "google.com" as long as there is a non-word
character preceeding the google (such as "-", or ".").
manually running gravity.sh
This will print "Getting $domain list... " for each domain, followed
by either "Done" if data was received and validated, or "Skipping
list because it does not have any new entries" if no updates were
needed.
I also wanted to replace the for loop iterating over indices with
something like:
`for url in $sources[@]}`
It made the use of `$i` in the save location more annoying though.
It's pretty slow because /etc/hosts is so large, but it works. You can
add any number of domains as arguments and it will search for and
remove each one.
Maybe someone smarter knows how to make it faster...
Some people use a hostname other than raspberrypi, so their hostname
did not resolve to 127.0.0.1. I replaced that hardcoded value with a
variable so that does not happen.
I also added a few comments and minor formatting adjustments,.
Detects if the Pi had an original Pi-hole by looking for adList.conf.
If it exists, it backs up all the files to /etc/pihole/original. Then,
the script continues as normal and creates the new files. This helps
ensure that /etc/hosts and /etc/dnsmasq.d/adList.conf are not being
used simultaneously.
You can add your own config file to permanently set variables used in
the gravity script. If the file exists, gravity.sh will detect it and
apply your custom variables. This is useful so when there is an update
to the gravity script, you do not need to adjust the variables every
time.
This script is for use with an Adafruit LCD so that you can peek at the
domains as they are blocked in real time. It's not necessary for the
Pi-hole to function, but something fun I am working on. There will be
a video in the next article I post.
