If a blocklist is blocked by an other and 'pihole -g' is run, update the blocked blocklist by useing an alternate dns server (in this case 1.1.1.1) to acquire the IP adres and download the blocklist using the 'curl --resolve' (https://curl.haxx.se/docs/manpage.html#--resolve)
* Redirect `grep` correctly to $sources (instead of using `awk`)
* Redirect $sourceDomains correctly
* Replace use of ${COL_LIGHT_BLUE}
* Add numeric count informing user of unique source domains being whitelisted
* Add 504 status (Gateway connection timed out)
* Add text for non-standard list parsing
* Improve adblock parsing
* Ensure adblock exception rules are removed from file
* Ensure "www." is not treated as a URL-format list
* Corrected typo
* Ensure script does not fail if "-f" is used when there are no blocklists generated
Signed off by WaLLy3K <wally3k@pi-hole.net>
* Remove WHITELIST_COMMAND
* Place IPv4/IPv6 availability test underneath setupVars.conf source
* Improved clarity on comments
* Define default lookupDomain on local line
* Use `getent hosts` instead of nslookup (faster)
* Make gravity_DNSLookup() function more readable
* Use bold on "Neutrino emissions detected"
* Swap conditionals around on adlists file handling
* Add comments to both gravity_Collapse() `awk`s
* Removed unnecessary "${str}" from gravity_Pull()
* Merge function variables into local line
* Place .phgbp suffice on mktemp, so patternbuffers can be cleaned up all at once in gravity_Cleanup()
* Removed success="false" from $httpCode case, placed empty success var in local
* Reordered $httpCode case numerically because I can
* Provide error if Dnsmasq format list is being parsed
* Remove IPv4 check when determining URL list (too slow on large lists)
* Check ${#sources[@]} to ensure we're checking the number of entries and not the character count
* Define empty plural in local line, removing unnecessary plural=;
* Optimised readability of gravity_Whitelist()
* Removed uninformative "Nothing to blacklist"/"No wildcards used" text
* Optimised parsing of domains into hosts format on IPv6 enabled servers
* Ensure /etc/hostname is non-zero
* Use `: >` instead of `rm` as consistent with the rest of the script
* Ensured that gravity_Cleanup() removes ${localList}.tmp
* Optimised readability of gravity_ParseUserDomains()
* Moved dnsRestart to ${var} case statement, renaming it to dnsRestartType for readability
* Set default $listType to ensure script passes "bash strict mode"
* Shift default dnsRestart value into unset default parameter expansion value
* Change nslookup timeout to 5 seconds
* Use &> instead of >
* Standardise plural code
* Update some comments
* Add "000" when connection is refused
* Condense adblock detection logic
* Add Dnsmasq format detection and parsing logic
* Removed unnecessary echo
* Add dnsWasOffline variable to ensure that if DNS service has been stopped, that it doesn't start and also get reloaded
* Merge development OpenVPN code
* Determine which domain to resolve depending on existence of $localList
* Re-add code to remove $localList, preventing duplicate local entries
* Minor shellcheck validation fix
* Add/update code comments
* Change resolver check to test for pi.hole
* Make resolver check timeout after 10 seconds
* Use > instead of &> where appropriate
* Make resolver check sleep for 30 seconds (effectively waiting up to 50s for dnsmasq to be resolvable)
* Provide confirmation upon success of resolver check availability
* Quotes and Braced remaining variables as appropriate
* Removed duplicate local
* Changed supernova/eventHorizon variables to match their purpose
* Add gravity_DNSLookup() function
* Ensure all comments are clear and relevant
* Use && instead of || in gravity_Collapse()
* Renamed existing functions, and placed them in order of script execution
* Use \t instead of literal tab in gravity_ParseFileIntoDomains()
* Replace instances of "truncate" with : > (e.g: gravity_Schwarzschild())
* Ensure correct variables are local'd
* Use phrase "Cleaning up stray matter" when gravity_Cleanup() is called
* Add black/white/wildcard switches for list.sh
* Ensure necessary functions are called when modifying black/white/wildcards
Signed-off-by: WaLLy3K <wally3k@pi-hole.net>
* Shellcheck validation
* Made variable names, function names, comments and output clearer to understand
* Quoted and braced variables and conditionals
* Fix adlists.list handling logic, and remove any CR line endings
* Make CP/MV/RM provide user-friendly output upon failure
* Change adlists.list retrieval logic
* Moved and fixed adlists.list domain parsing logic
* Create gravity_ParseFileAsDomains() function to handle parsing of source files
* If no changes to a list is detected, print no output
* Ensure each source blocklist has a final newline
* Format number output as currency
* Make array of adlists domain sources unique to prevent redundant whitelisting
* Merged bash IPv4/IPv6 hosts formatting IF statement into an awk one-liner
* Trap Ctrl-C cancellations and run gravity_Cleanup()
* Use new gravity_Cleanup() function on errors and script completion
* Ensure that dnsmasq uses force-reload when gravity is invoked
* Add --wildcard option to ensure dnsmasq is restarted upon b/wlisting of a wildcard
Signed-off-by: WaLLy3K <wally3k@pi-hole.net>
* Define colours within COL_TABLE
* Do not output colours for non-terminal instances
* Removed ":::"
* Fixed indenting & spacing
* Made output consistent throughout project
* Reworded text to fit on standard 80 char wide Terminal screen
* Made 'sudo raspi-config' warning (insufficient disk space) only show on RPi
* Make "Installation/Update Complete" the final msg
* Remove redundant messages
* Simplify update available message
* Confirm user would like to begin uninstall
* If "git pull" string says "Already up-to-date.", place [i] before it
* Colour Temp/Interface output
* Made `pihole disable 5z` invalid
* Added error fallback if invalid argument (not s/m) is detected
* Quoted "$2" for consistency
* Updated help text
* L185/286: Replaced echo with redirect
* User agents for adblock.mahakala.is/adaway.org unnecessary
* Print newline on confirmation of repository reset
* Add output to admin-related dnsmasq restarts
* Return error message for "pihole -q"
* Imply default checkout behaviour with y/N
* Fix uninstall failing to remove pihole user
* Print checkout 'git remote show origin' STDERR on new line
* Replaced checkout "AdminLTE" wording with "Web Admin"
Mish-mash of tabs and spaces used for indentation. Mostly tabs though,
so going with those. Also indenting blocks and enforcing the
loop statement; do
done
style over the newline for do since it seemed to be more common.
There were no fewer than three styles of function declaration going
on in these files, so I attempted to standardize them to the typical
myfucn() {
}
style.
- Test if /etc/hostname exists before attempting to cat the file.
- Add alternative hostname discovery via hostname command
- Add an error message to stdout if unable to determine host fqdn
* Change dummy domain
Allows users to access Pi-hole console with my.pi-hole.net instead of the IP address
* Cleaned up formatting
* Put device hostname at the top of the list
Changed my.pi-hole.net to pi.hole and placed it under the device hostname. This way, ping will return the device hostname but you still can access the web interface with pi.hole/admin
Removed symlink of scripts to u/l/b
Added copy for setupLCD to /opt/pihole/
New spinner added to gravity.sh to reflect new spinner in basic-install.sh and uninstall.sh
Failed to be populated if /tmp/piholeIP does not exist. This may happen under
the following conditions:
- was not generated during install
- gravity.sh being run again after the original install (the file is deleted
during the installation process)
Updated piholeIP because if there are multiple network adapters, the gravity list will populate incorrectly with both IP addresses. Only one address should be used.
Temporary hack for /etc/pihole directory and file permissions to pull out sudo requirement. (Irony is that is requires sudo.) This will be fixed later, but for now it's needed for existing installs so they don't break.
This seems to be the more appropriate URL:
http://hosts-file.net/ad_servers.txt.
I compared it to the original URL using opendiff and they are exactly
the same.
In my case there had been no whitespace in piholeIP (I configured it statically because 'hostname -I' returns multiple IP addresses, which ends in errornous hosts file), so it's more safe to add one between IP and hostname.
Further, I removed the obsolete latentBlacklist variable.
Due to so many users having issues with this list blocking legitimate
domains such as microsoft.com, apple.com, xkcd.com and more, I am
turning it off by default. While this drastically reduces the amount
of domains blocked, ad-blocking performance still seems to function
quite well.
Long-time users can simply uncomment the list, but this will make it so
new users have a more pleasant experience.
If you already have the mahakala list, you will need to remove it first
with this command:
sudo rm /etc/pihole/list.2.adblock.mahakala.is.domains
There are a few other lists I am looking at including, but this will
provide an immediate fix.
I'm not 100% sure if you can add multiple IP addresses to one domain on
a single line, so I just broke it up into two. It makes the file
bigger and longer, but I'll have to see how it works
Due to users having issues with the IP not being entered properly, I am restoring the original way to store `piholeIP` until a better solution can be formed.
This will increase the swap file to 500MB before downloading the lists.
Most of the issue comes from the mahakala list, which is so large. If
no swap file is found, one is created.
Prepend "^" to start of latentWhitelist.txt lines.
The -x switch requires a full line match of the regexp, where as -w
will try to find the match somewhere in the line, looking for work
breaks. Combined with turning the whitelist lines into full regexps,
this results in significantly faster parsing.
Having "^" prepended to the lines also keeps false whitelisting from
occuring, such as the following example:
If whitelist.txt contains "google.com" it would whitelist many other
sites that end in "google.com" as long as there is a non-word
character preceeding the google (such as "-", or ".").
manually running gravity.sh
This will print "Getting $domain list... " for each domain, followed
by either "Done" if data was received and validated, or "Skipping
list because it does not have any new entries" if no updates were
needed.
I also wanted to replace the for loop iterating over indices with
something like:
`for url in $sources[@]}`
It made the use of `$i` in the save location more annoying though.
Some people use a hostname other than raspberrypi, so their hostname
did not resolve to 127.0.0.1. I replaced that hardcoded value with a
variable so that does not happen.
I also added a few comments and minor formatting adjustments,.
Pushing files so they are available when the new article gets posted.
If the Pi's loopback is set in the hosts file, clients using it as a
DNS server will try to connect to their own loopback, which does not
have a Web server. So the real IP of the Pi is used. It is
recommended to use a static IP since this will be acting as a server.
Made one small change from some hard coded values to a variable.
Originally, I had this set to /run/shm (in RAM) but ran into errors
when the list reached 900,000 entries.
Then I moved it to /tmp.
Finally, I decided to just put the files in the pihole dir so they are
available after reboots. This will help with only downloading the
lists when absolutely needed--respecting the bandwidth of the people
serving the lists.
It is also possible to add addn-hosts=/path/to/hosts.conf within the
dnsmasq.conf file if you don't want to use hosts. For simplicity and
speed, I just use the regular hosts file.
Still need to get lighted to use IPv6. I am doing this because some
ads can get through using IPv6 if the IPv4 version is blocked. Also,
it seems to work fine as far as performance even though it doubles the
file size...
Also added a few comments for better documentation.
