diff --git a/.editorconfig b/.editorconfig new file mode 100644 index 00000000..e5626a07 --- /dev/null +++ b/.editorconfig @@ -0,0 +1,38 @@ +# EditorConfig is awesome: http://EditorConfig.org + +# top-most EditorConfig file +root = true + +# Unix-style newlines with a newline ending every file +[*] +end_of_line = lf +insert_final_newline = true +indent_style = space +indent_size = tab +tab_width = 2 +charset = utf-8 +trim_trailing_whitespace = true + +# Matches multiple files with brace expansion notation +# Set default charset +[*.{js,py}] +charset = utf-8 + +# 4 space indentation +[*.py] +indent_style = space +indent_size = 4 + +# Tab indentation (no size specified) +[Makefile] +indent_style = tab + +# Indentation override for all JS under lib directory +[scripts/**.js] +indent_style = space +indent_size = 2 + +# Matches the exact files either package.json or .travis.yml +[{package.json,.travis.yml}] +indent_style = space +indent_size = 2 diff --git a/.github/ISSUE_TEMPLATE.md b/.github/ISSUE_TEMPLATE.md index c985b972..3014625b 100644 --- a/.github/ISSUE_TEMPLATE.md +++ b/.github/ISSUE_TEMPLATE.md @@ -2,7 +2,7 @@ - [] I have read and understood the [contributors guide](https://github.com/pi-hole/pi-hole/blob/master/CONTRIBUTING.md). - [] The issue I am reporting can be *replicated* -- [] The issue I'm reporting isn't a duplicate (see [FAQs](https://github.com/pi-hole/pi-hole/wiki/FAQs), [closed issues](https://github.com/pi-hole/pi-hole/issues?utf8=%E2%9C%93&q=is%3Aissue%20is%3Aclosed%20), and [open issues](https://github.com/pi-hole/pi-hole/issues)). +- [] The issue I am reporting isn't a duplicate (see [FAQs](https://github.com/pi-hole/pi-hole/wiki/FAQs), [closed issues](https://github.com/pi-hole/pi-hole/issues?utf8=%E2%9C%93&q=is%3Aissue%20is%3Aclosed%20), and [open issues](https://github.com/pi-hole/pi-hole/issues)). **How familiar are you with the codebase?:** diff --git a/.gitignore b/.gitignore index c4b497a0..91014dcd 100644 --- a/.gitignore +++ b/.gitignore @@ -4,4 +4,3 @@ __pycache__ .cache .pullapprove.yml - diff --git a/.pullapprove.yml b/.pullapprove.yml index ff970b29..39566b34 100644 --- a/.pullapprove.yml +++ b/.pullapprove.yml @@ -1,7 +1,7 @@ version: 2 always_pending: - title_regex: '(WIP|wip)' + title_regex: '(WIP|wip)' labels: - wip explanation: 'This PR is a work in progress...' @@ -22,11 +22,11 @@ groups: enabled: true conditions: branches: - - development + - development required: 2 teams: - approvers - + master: approve_by_comment: enabled: true diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index d342a8b5..8cb7ccb9 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -35,4 +35,5 @@ When requesting or submitting new features, first consider whether it might be u - Before Submitting your Pull Request, merge `development` with your new branch and fix any conflicts. (Make sure you don't break anything in development!) - Please use the [Google Style Guide for Shell](https://google.github.io/styleguide/shell.xml) for your code submission styles. - Commit Unix line endings. +- Please use the Pi-hole brand: **Pi-hole** (Take a special look at the capitalized 'P' and a low 'h' with a hyphen) - (Optional fun) keep to the theme of Star Trek/black holes/gravity. diff --git a/README.md b/README.md index 0a5f7f4c..75c548e5 100644 --- a/README.md +++ b/README.md @@ -5,12 +5,12 @@

- +

## The multi-platform, network-wide ad blocker -Block ads for **all** your devices _without_ the need to install client-side software. The Pi-hole blocks ads at the DNS-level, so all your devices are protected. +Block ads for **all** your devices _without_ the need to install client-side software. The Pi-hole™ blocks ads at the DNS-level, so all your devices are protected. - Web Browsers - Cell Phones @@ -53,9 +53,9 @@ wget -O basic-install.sh https://install.pi-hole.net bash basic-install.sh ``` -Once installed, [configure your router to have **DHCP clients use the Pi as their DNS server**](https://discourse.pi-hole.net/t/how-do-i-configure-my-devices-to-use-pi-hole-as-their-dns-server/245) and then any device that connects to your network will have ads blocked without any further configuration. Alternatively, you can manually set each device to use Pi-hole as their DNS server. +Once installed, [configure your router to have **DHCP clients use the Pi as their DNS server**](https://discourse.pi-hole.net/t/how-do-i-configure-my-devices-to-use-pi-hole-as-their-dns-server/245) and then any device that connects to your network will have ads blocked without any further configuration. Alternatively, you can manually set each device to use Pi-hole™ as their DNS server. -## What is Pi-hole and how do I install it? +## What is Pi-hole™ and how do I install it?

@@ -73,7 +73,7 @@ Once installed, [configure your router to have **DHCP clients use the Pi as thei ## Technical Details -The Pi-hole is an **advertising-aware DNS/Web server**. If an ad domain is queried, a small Web page or GIF is delivered in place of the advertisement. +The Pi-hole™ is an **advertising-aware DNS/Web server**. If an ad domain is queried, a small Web page or GIF is delivered in place of the advertisement. ### Gravity @@ -83,7 +83,7 @@ The [gravity.sh](https://github.com/pi-hole/pi-hole/blob/master/gravity.sh) does #### Other Operating Systems -The automated install is only for a clean install of a Debian family or Fedora based system, such as the Raspberry Pi. However, this script will work for most UNIX-like systems, some with some slight **modifications** that we can help you work through. If you can install `dnsmasq` and a Webserver, it should work OK. If there are other platforms you'd like supported, let us know. +The automated install is only for a clean install of a Debian family or Fedora based system, such as the Raspberry Pi. However, this script will work for most UNIX-like systems, some with some slight **modifications** that we can help you work through. If you can install `dnsmasq` and a web server, it should work OK. If there are other platforms you'd like supported, let us know. ### Web Interface @@ -102,7 +102,7 @@ Domains can be whitelisted and blacklisted using either the web interface or the ### Settings -The settings page lets you control and configure your Pi-hole. You can do things like: +The settings page lets you control and configure your Pi-hole™. You can do things like: - enable Pi-hole's built-in DHCP server - exclude domains from the graphs @@ -113,7 +113,7 @@ The settings page lets you control and configure your Pi-hole. You can do thing #### Built-in DHCP Server -Pi-hole ships with a built-in DHCP server. This allows you to let your network devices use Pi-hole as their DNS server if your router does not let you adjust the DHCP options. +Pi-hole™ ships with a built-in DHCP server. This allows you to let your network devices use Pi-hole™ as their DNS server if your router does not let you adjust the DHCP options.

@@ -137,14 +137,14 @@ The same output can be achieved on the CLI by running `chronometer.sh -j` You can view [real-time stats](https://discourse.pi-hole.net/t/how-do-i-view-my-pi-holes-stats-over-ssh-or-on-an-lcd-using-chronometer/240) via `ssh` or on an [2.8" LCD screen](http://amzn.to/1P0q1Fj). This is accomplished via [`chronometer.sh`](https://github.com/pi-hole/pi-hole/blob/master/advanced/Scripts/chronometer.sh). ![Pi-hole LCD](http://i.imgur.com/nBEqycp.jpg) -## Pi-hole Projects +## Pi-hole™ Projects - [An ad blocking Magic Mirror](https://zonksec.com/blog/magic-mirror-dns-filtering/#dnssoftware) - [Pi-hole stats in your Mac's menu bar](https://getbitbar.com/plugins/Network/pi-hole.1m.py) - [Get LED alerts for each blocked ad](http://thetimmy.silvernight.org/pages/endisbutton/) - [Pi-hole on Ubuntu 14.04 on VirtualBox](http://hbalagtas.blogspot.com/2016/02/adblocking-with-pi-hole-and-ubuntu-1404.html) - [Docker Pi-hole container (x86 and ARM)](https://hub.docker.com/r/diginc/pi-hole/) -- [Splunk: Pi-hole Visualizser](https://splunkbase.splunk.com/app/3023/) +- [Splunk: Pi-hole Visualiser](https://splunkbase.splunk.com/app/3023/) - [Pi-hole Chrome extension](https://chrome.google.com/webstore/detail/pi-hole-list-editor/hlnoeoejkllgkjbnnnhfolapllcnaglh) ([open source](https://github.com/packtloss/pihole-extension)) - [Go Bananas for CHiP-hole ad blocking](https://www.hackster.io/jacobsalmela/chip-hole-network-wide-ad-blocker-98e037) - [Sky-Hole](http://dlaa.me/blog/post/skyhole) @@ -154,7 +154,7 @@ You can view [real-time stats](https://discourse.pi-hole.net/t/how-do-i-view-my- - [Minibian Pi-hole](http://munkjensen.net/wiki/index.php/See_my_Pi-Hole#Minibian_Pi-hole) - [Windows Tray Stat Application](https://github.com/goldbattle/copernicus) - [Let your blink1 device blink when Pi-hole filters ads](https://gist.github.com/elpatron68/ec0b4c582e5abf604885ac1e068d233f) -- [Pi-Hole Prometheus exporter](https://github.com/nlamirault/pihole_exporter) : a [Prometheus](https://prometheus.io/) exporter for Pi-Hole +- [Pi-hole Prometheus exporter](https://github.com/nlamirault/pihole_exporter): a [Prometheus](https://prometheus.io/) exporter for Pi-hole - [Pi-hole Droid - open source Android client](https://github.com/friimaind/pi-hole-droid) ## Coverage diff --git a/adlists.default b/adlists.default index 4a5bca3c..cbc1bfb3 100644 --- a/adlists.default +++ b/adlists.default @@ -1,53 +1,23 @@ -## Pi-hole ad-list default sources. Updated 29/10/2016 ######################### -# # -# To make changes to this file: # -# 1. run `cp /etc/pihole/adlists.default /etc/pihole/adlists.list` # -# 2. run `nano /etc/pihole/adlists.list` # -# 3. Uncomment or comment any of the below lists # -# # -# Know of any other lists? Feel free to let us know about them, or add them # -# to this file! # -################################################################################ - # The below list amalgamates several lists we used previously. # See `https://github.com/StevenBlack/hosts` for details +##StevenBlack's list https://raw.githubusercontent.com/StevenBlack/hosts/master/hosts -# Other lists we consider safe: +##MalwareDomains https://mirror1.malwaredomains.com/files/justdomains + +##Cameleon http://sysctl.org/cameleon/hosts + +##Zeustracker https://zeustracker.abuse.ch/blocklist.php?download=domainblocklist + +##Disconnect.me Tracking https://s3.amazonaws.com/lists.disconnect.me/simple_tracking.txt + +##Disconnect.me Ads https://s3.amazonaws.com/lists.disconnect.me/simple_ad.txt -# hosts-file.net list. Updated frequently, but has been known to block legitimate sites. +##Hosts-file.net https://hosts-file.net/ad_servers.txt -# Mahakala list. Has been known to block legitimate domains including the entire .com range. -# Warning: Due to the sheer size of this list, the web admin console will be unresponsive. -#https://adblock.mahakala.is/ - -# ADZHOSTS list. Has been known to block legitimate domains -#http://pilotfiber.dl.sourceforge.net/project/adzhosts/HOSTS.txt - -# Windows 10 telemetry list -#https://raw.githubusercontent.com/crazy-max/WindowsSpyBlocker/master/data/hosts/win10/spy.txt - -# Securemecca.com list - Also blocks "adult" sites (pornography/gambling etc) -#http://securemecca.com/Downloads/hosts.txt - -# Quidsup's tracker list -#https://raw.githubusercontent.com/quidsup/notrack/master/trackers.txt - -# Block the BBC News website Breaking News banner -#https://raw.githubusercontent.com/BreakingTheNews/BreakingTheNews.github.io/master/hosts - -# Untested Lists: -#https://raw.githubusercontent.com/reek/anti-adblock-killer/master/anti-adblock-killer-filters.txt -#https://raw.githubusercontent.com/Dawsey21/Lists/master/main-blacklist.txt -#http://malwaredomains.lehigh.edu/files/domains.txt -# Following two lists should be used simultaneously: (readme https://github.com/notracking/hosts-blocklists/) -#https://raw.github.com/notracking/hosts-blocklists/master/hostnames.txt -#https://raw.github.com/notracking/hosts-blocklists/master/domains.txt -# Combination of several host files on the internet (warning some facebook domains are also blocked but you can go to facebook.com). See https://github.com/mat1th/Dns-add-block for more information. -#https://raw.githubusercontent.com/mat1th/Dns-add-block/master/hosts diff --git a/advanced/Scripts/chronometer.sh b/advanced/Scripts/chronometer.sh index 7bb50745..67ff495b 100755 --- a/advanced/Scripts/chronometer.sh +++ b/advanced/Scripts/chronometer.sh @@ -8,34 +8,55 @@ # This file is copyright under the latest version of the EUPL. # Please see LICENSE file for your rights under this license. - - - #Functions############################################################################################################## piLog="/var/log/pihole.log" gravity="/etc/pihole/gravity.list" . /etc/pihole/setupVars.conf -# Borrowed/modified from https://gist.github.com/cjus/1047794 -function GetJSONValue { - retVal=$(echo $1 | sed 's/\\\\\//\//g' | \ - sed 's/[{}]//g' | \ - awk -v k="text" '{n=split($0,a,","); for (i=1; i<=n; i++) print a[i]}' | \ - sed 's/\"\:/\|/g' | \ - sed 's/[\,]/ /g' | \ - sed 's/\"//g' | \ - grep -w $2) - echo ${retVal##*|} +function GetFTLData { + # Open connection to FTL + exec 3<>/dev/tcp/localhost/"$(cat /var/run/pihole-FTL.port)" + + # Test if connection is open + if { >&3; } 2> /dev/null; then + # Send command to FTL + echo -e ">$1" >&3 + + # Read input + read -r -t 1 LINE <&3 + until [ ! $? ] || [[ "$LINE" == *"EOM"* ]]; do + echo "$LINE" >&1 + read -r -t 1 LINE <&3 + done + + # Close connection + exec 3>&- + exec 3<&- + fi } outputJSON() { - json=$(curl -s -X GET http://127.0.0.1/admin/api.php?summaryRaw) - echo ${json} + get_summary_data + echo "{\"domains_being_blocked\":${domains_being_blocked_raw},\"dns_queries_today\":${dns_queries_today_raw},\"ads_blocked_today\":${ads_blocked_today_raw},\"ads_percentage_today\":${ads_percentage_today_raw}}" +} + +get_summary_data() { + local summary=$(GetFTLData "stats") + domains_being_blocked_raw=$(grep "domains_being_blocked" <<< "${summary}" | grep -Eo "[0-9]+$") + domains_being_blocked=$(printf "%'.f" ${domains_being_blocked_raw}) + dns_queries_today_raw=$(grep "dns_queries_today" <<< "$summary" | grep -Eo "[0-9]+$") + dns_queries_today=$(printf "%'.f" ${dns_queries_today_raw}) + ads_blocked_today_raw=$(grep "ads_blocked_today" <<< "$summary" | grep -Eo "[0-9]+$") + ads_blocked_today=$(printf "%'.f" ${ads_blocked_today_raw}) + ads_percentage_today_raw=$(grep "ads_percentage_today" <<< "$summary" | grep -Eo "[0-9.]+$") + LC_NUMERIC=C ads_percentage_today=$(printf "%'.f" ${ads_percentage_today_raw}) } normalChrono() { for (( ; ; )); do + get_summary_data + domain=$(GetFTLData recentBlocked) clear # Displays a colorful Pi-hole logo echo " ___ _ _ _" @@ -49,20 +70,12 @@ normalChrono() { #uptime -p #Doesn't work on all versions of uptime uptime | awk -F'( |,|:)+' '{if ($7=="min") m=$6; else {if ($7~/^day/) {d=$6;h=$8;m=$9} else {h=$6;m=$7}}} {print d+0,"days,",h+0,"hours,",m+0,"minutes."}' echo "-------------------------------" - # Uncomment to continually read the log file and display the current domain being blocked - #tail -f /var/log/pihole.log | awk '/\/etc\/pihole\/gravity.list/ {if ($7 != "address" && $7 != "name" && $7 != "/etc/pihole/gravity.list") print $7; else;}' + echo "Recently blocked:" + echo " $domain" - json=$(curl -s -X GET http://127.0.0.1/admin/api.php?summaryRaw) - - domains=$(printf "%'.f" $(GetJSONValue ${json} "domains_being_blocked")) #add commas in - queries=$(printf "%'.f" $(GetJSONValue ${json} "dns_queries_today")) - blocked=$(printf "%'.f" $(GetJSONValue ${json} "ads_blocked_today")) - LC_NUMERIC=C percentage=$(printf "%0.2f\n" $(GetJSONValue ${json} "ads_percentage_today")) #2 decimal places - - echo "Blocking: ${domains}" - echo "Queries: ${queries}" - - echo "Pi-holed: ${blocked} (${percentage}%)" + echo "Blocking: ${domains_being_blocked}" + echo "Queries: ${dns_queries_today}" + echo "Pi-holed: ${ads_blocked_today} (${ads_percentage_today}%)" sleep 5 done diff --git a/advanced/Scripts/piholeCheckout.sh b/advanced/Scripts/piholeCheckout.sh index 3b7abbef..09f20d6b 100644 --- a/advanced/Scripts/piholeCheckout.sh +++ b/advanced/Scripts/piholeCheckout.sh @@ -104,7 +104,7 @@ checkout() #This is unlikely if ! is_repo "${PI_HOLE_FILES_DIR}" ; then - echo "::: Critical Error: Core Pi-Hole repo is missing from system!" + echo "::: Critical Error: Core Pi-hole repo is missing from system!" echo "::: Please re-run install script from https://github.com/pi-hole/pi-hole" exit 1; fi diff --git a/advanced/Scripts/piholeDebug.sh b/advanced/Scripts/piholeDebug.sh index 0b8a2571..945cd81c 100755 --- a/advanced/Scripts/piholeDebug.sh +++ b/advanced/Scripts/piholeDebug.sh @@ -43,16 +43,11 @@ cat << EOM ::: Please read and note any issues, and follow any directions advised during this process. EOM -# Ensure the file exists, create if not, clear if exists. -truncate --size=0 "${DEBUG_LOG}" -chmod 644 ${DEBUG_LOG} -chown "$USER":pihole ${DEBUG_LOG} - source ${VARSFILE} ### Private functions exist here ### log_write() { - echo "${1}" >> "${DEBUG_LOG}" + echo "${@}" >&3 } log_echo() { @@ -77,14 +72,14 @@ log_echo() { header_write() { log_echo "" - log_echo "${1}" + log_echo "---= ${1}" log_write "" } file_parse() { while read -r line; do if [ ! -z "${line}" ]; then - [[ "${line}" =~ ^#.*$ || ! "${line}" ]] && continue + [[ "${line}" =~ ^#.*$ || ! "${line}" || "${line}" == "WEBPASSWORD="* ]] && continue log_write "${line}" fi done < "${1}" @@ -231,6 +226,7 @@ ipv6_check() { ip_check() { local protocol=${1} local gravity=${2} + header_write "Checking IPv${protocol} Stack" local ip_addr_list="$(ip -${protocol} addr show dev ${PIHOLE_INTERFACE} | awk -F ' ' '{ for(i=1;i<=NF;i++) if ($i ~ '/^inet/') print $(i+1) }')" if [[ -n ${ip_addr_list} ]]; then @@ -305,62 +301,78 @@ daemon_check() { } testResolver() { - header_write "Resolver Functions Check" + local protocol="${1}" + header_write "Resolver Functions Check (IPv${protocol})" + local IP="${2}" + local g_addr + local l_addr + local url + local testurl + local localdig + local piholedig + local remotedig + + if [[ ${protocol} == "6" ]]; then + g_addr="2001:4860:4860::8888" + l_addr="::1" + r_type="AAAA" + else + g_addr="8.8.8.8" + l_addr="127.0.0.1" + r_type="A" + fi # Find a blocked url that has not been whitelisted. - TESTURL="doubleclick.com" - if [ -s "${WHITELISTMATCHES}" ]; then - while read -r line; do - CUTURL=${line#*" "} - if [ "${CUTURL}" != "Pi-Hole.IsWorking.OK" ]; then - while read -r line2; do - CUTURL2=${line2#*" "} - if [ "${CUTURL}" != "${CUTURL2}" ]; then - TESTURL="${CUTURL}" - break 2 - fi - done < "${WHITELISTMATCHES}" - fi - done < "${GRAVITYFILE}" - fi + url=$(shuf -n 1 "${GRAVITYFILE}" | awk -F ' ' '{ print $2 }') - log_write "Resolution of ${TESTURL} from Pi-hole:" - LOCALDIG=$(dig "${TESTURL}" @127.0.0.1) - if [[ $? = 0 ]]; then - log_write "${LOCALDIG}" + testurl="${url:-doubleclick.com}" + + + log_write "Resolution of ${testurl} from Pi-hole (${l_addr}):" + if localdig=$(dig -"${protocol}" "${testurl}" @${l_addr} +short "${r_type}"); then + log_write "${localdig}" else - log_write "Failed to resolve ${TESTURL} on Pi-hole" + log_write "Failed to resolve ${testurl} on Pi-hole (${l_addr})" + fi + log_write "" + + log_write "Resolution of ${testurl} from Pi-hole (${IP}):" + if piholedig=$(dig -"${protocol}" "${testurl}" @"${IP}" +short "${r_type}"); then + log_write "${piholedig}" + else + log_write "Failed to resolve ${testurl} on Pi-hole (${IP})" fi log_write "" - log_write "Resolution of ${TESTURL} from 8.8.8.8:" - REMOTEDIG=$(dig "${TESTURL}" @8.8.8.8) - if [[ $? = 0 ]]; then - log_write "${REMOTEDIG}" + log_write "Resolution of ${testurl} from ${g_addr}:" + if remotedig=$(dig -"${protocol}" "${testurl}" @${g_addr} +short "${r_type}"); then + log_write "${remotedig:-NXDOMAIN}" else - log_write "Failed to resolve ${TESTURL} on 8.8.8.8" + log_write "Failed to resolve ${testurl} on upstream server ${g_addr}" fi log_write "" - - log_write "Pi-hole dnsmasq specific records lookups" - log_write "Cache Size:" - dig +short chaos txt cachesize.bind >> ${DEBUG_LOG} - log_write "Upstream Servers:" - dig +short chaos txt servers.bind >> ${DEBUG_LOG} - log_write "" } +testChaos(){ + # Check Pi-hole specific records + + log_write "Pi-hole dnsmasq specific records lookups" + log_write "Cache Size:" + log_write $(dig +short chaos txt cachesize.bind) + log_write "Upstream Servers:" + log_write $(dig +short chaos txt servers.bind) + log_write "" + +} checkProcesses() { header_write "Processes Check" - echo "::: Logging status of lighttpd and dnsmasq..." - PROCESSES=( lighttpd dnsmasq ) + echo "::: Logging status of lighttpd, dnsmasq and pihole-FTL..." + PROCESSES=( lighttpd dnsmasq pihole-FTL ) for i in "${PROCESSES[@]}"; do - log_write "" - log_write "${i}" - log_write " processes status:" - systemctl -l status "${i}" >> "${DEBUG_LOG}" + log_write "Status for ${i} daemon:" + log_write $(systemctl is-active "${i}") done log_write "" } @@ -388,49 +400,6 @@ countdown() { tuvix=$(( tuvix - 5 )) done } -### END FUNCTIONS ### - -# Gather version of required packages / repositories -version_check || echo "REQUIRED FILES MISSING" -# Check for newer setupVars storage file -source_file "/etc/pihole/setupVars.conf" -# Gather information about the running distribution -distro_check || echo "Distro Check soft fail" -# Gather processor type -processor_check || echo "Processor Check soft fail" - -ip_check 6 ${IPV6_ADDRESS} -ip_check 4 ${IPV4_ADDRESS} - -daemon_check lighttpd http -daemon_check dnsmasq domain -checkProcesses -testResolver -debugLighttpd - -files_check "${DNSMASQFILE}" -dir_check "${DNSMASQCONFDIR}" -files_check "${WHITELISTFILE}" -files_check "${BLACKLISTFILE}" -files_check "${ADLISTFILE}" - - -header_write "Analyzing gravity.list" - - gravity_length=$(grep -c ^ "${GRAVITYFILE}") \ - && log_write "${GRAVITYFILE} is ${gravity_length} lines long." \ - || log_echo "Warning: No gravity.list file found!" - -header_write "Analyzing pihole.log" - - pihole_length=$(grep -c ^ "${PIHOLELOG}") \ - && log_write "${PIHOLELOG} is ${pihole_length} lines long." \ - || log_echo "Warning: No pihole.log file found!" - - pihole_size=$(du -h "${PIHOLELOG}" | awk '{ print $1 }') \ - && log_write "${PIHOLELOG} is ${pihole_size}." \ - || log_echo "Warning: No pihole.log file found!" - # Continuously append the pihole.log file to the pihole_debug.log file dumpPiHoleLog() { @@ -442,7 +411,7 @@ dumpPiHoleLog() { if [ -e "${PIHOLELOG}" ]; then # Dummy process to use for flagging down tail to terminate countdown & - tail -n0 -f --pid=$! "${PIHOLELOG}" >> ${DEBUG_LOG} + tail -n0 -f --pid=$! "${PIHOLELOG}" >&4 else log_write "No pihole.log file found!" printf ":::\tNo pihole.log file found!\n" @@ -453,6 +422,16 @@ dumpPiHoleLog() { finalWork() { local tricorder echo "::: Finshed debugging!" + + # Ensure the file exists, create if not, clear if exists. + truncate --size=0 "${DEBUG_LOG}" + chmod 644 ${DEBUG_LOG} + chown "$USER":pihole ${DEBUG_LOG} + # copy working temp file to final log location + cat /proc/$$/fd/3 >> "${DEBUG_LOG}" + # Straight dump of tailing the logs, can sanitize later if needed. + cat /proc/$$/fd/4 >> "${DEBUG_LOG}" + echo "::: The debug log can be uploaded to tricorder.pi-hole.net for sharing with developers only." if [[ "${AUTOMATED}" ]]; then echo "::: Debug script running in automated mode, uploading log to tricorder..." @@ -480,6 +459,70 @@ finalWork() { echo "::: A local copy of the Debug log can be found at : /var/log/pihole_debug.log" } +### END FUNCTIONS ### +# Create temporary file for log +TEMPLOG=$(mktemp /tmp/pihole_temp.XXXXXX) +# Open handle 3 for templog +exec 3>"$TEMPLOG" +# Delete templog, but allow for addressing via file handle. +rm "$TEMPLOG" + +# Create temporary file for logdump using file handle 4 +DUMPLOG=$(mktemp /tmp/pihole_temp.XXXXXX) +exec 4>"$DUMPLOG" +rm "$DUMPLOG" + +# Gather version of required packages / repositories +version_check || echo "REQUIRED FILES MISSING" +# Check for newer setupVars storage file +source_file "/etc/pihole/setupVars.conf" +# Gather information about the running distribution +distro_check || echo "Distro Check soft fail" +# Gather processor type +processor_check || echo "Processor Check soft fail" + +ip_check 6 ${IPV6_ADDRESS} +ip_check 4 ${IPV4_ADDRESS} + +daemon_check lighttpd http +daemon_check dnsmasq domain +daemon_check pihole-FTL 4711 +checkProcesses + +# Check local/IP/Google for IPv4 Resolution +testResolver 4 "${IPV4_ADDRESS%/*}" +# If IPv6 enabled, check resolution +if [[ "${IPV6_ADDRESS}" ]]; then + testResolver 6 "${IPV6_ADDRESS%/*}" +fi +# Poll dnsmasq Pi-hole specific queries +testChaos + +debugLighttpd + +files_check "${DNSMASQFILE}" +dir_check "${DNSMASQCONFDIR}" +files_check "${WHITELISTFILE}" +files_check "${BLACKLISTFILE}" +files_check "${ADLISTFILE}" + + +header_write "Analyzing gravity.list" + + gravity_length=$(grep -c ^ "${GRAVITYFILE}") \ + && log_write "${GRAVITYFILE} is ${gravity_length} lines long." \ + || log_echo "Warning: No gravity.list file found!" + +header_write "Analyzing pihole.log" + + pihole_length=$(grep -c ^ "${PIHOLELOG}") \ + && log_write "${PIHOLELOG} is ${pihole_length} lines long." \ + || log_echo "Warning: No pihole.log file found!" + + pihole_size=$(du -h "${PIHOLELOG}" | awk '{ print $1 }') \ + && log_write "${PIHOLELOG} is ${pihole_size}." \ + || log_echo "Warning: No pihole.log file found!" + trap finalWork EXIT ### Method calls for additional logging ### diff --git a/advanced/Scripts/piholeLogFlush.sh b/advanced/Scripts/piholeLogFlush.sh index 63754db2..fd66b255 100755 --- a/advanced/Scripts/piholeLogFlush.sh +++ b/advanced/Scripts/piholeLogFlush.sh @@ -3,18 +3,22 @@ # (c) 2017 Pi-hole, LLC (https://pi-hole.net) # Network-wide ad blocking via your own hardware. # -# Flushes /var/log/pihole.log +# Flushes Pi-hole's log file # # This file is copyright under the latest version of the EUPL. # Please see LICENSE file for your rights under this license. - - echo -n "::: Flushing /var/log/pihole.log ..." # Test if logrotate is available on this system if command -v /usr/sbin/logrotate &> /dev/null; then + # Flush twice to move all data out of sight of FTL + /usr/sbin/logrotate --force /etc/pihole/logrotate /usr/sbin/logrotate --force /etc/pihole/logrotate else + # Flush both pihole.log and pihole.log.1 (if existing) echo " " > /var/log/pihole.log + if [ -f /var/log/pihole.log.1 ]; then + echo " " > /var/log/pihole.log.1 + fi fi echo "... done!" diff --git a/advanced/Scripts/update.sh b/advanced/Scripts/update.sh index 9ac12940..4fceb931 100755 --- a/advanced/Scripts/update.sh +++ b/advanced/Scripts/update.sh @@ -75,13 +75,26 @@ GitCheckUpdateAvail() { fi } +FTLcheckUpdate() { + + local FTLversion=$(/usr/bin/pihole-FTL tag) + local FTLlatesttag=$(curl -sI https://github.com/pi-hole/FTL/releases/latest | grep 'Location' | awk -F '/' '{print $NF}' | tr -d '\r\n') + + if [[ "${FTLversion}" != "${FTLlatesttag}" ]]; then + return 0 + else + return 1 + fi +} + main() { local pihole_version_current local web_version_current + source "${setupVars}" #This is unlikely if ! is_repo "${PI_HOLE_FILES_DIR}" ; then - echo "::: Critical Error: Core Pi-Hole repo is missing from system!" + echo "::: Critical Error: Core Pi-hole repo is missing from system!" echo "::: Please re-run install script from https://github.com/pi-hole/pi-hole" exit 1; fi @@ -96,6 +109,21 @@ main() { echo "::: Pi-hole Core: up to date" fi + if FTLcheckUpdate ; then + FTL_update=true + echo "::: FTL: update available" + else + FTL_update=false + echo "::: FTL: up to date" + fi + + if ${FTL_update}; then + echo ":::" + echo "::: FTL out of date" + FTLdetect + echo ":::" + fi + if [[ ${INSTALL_WEB} == true ]]; then if ! is_repo "${ADMIN_INTERFACE_DIR}" ; then echo "::: Critical Error: Web Admin repo is missing from system!" @@ -122,9 +150,11 @@ main() { # pull pihole repo run install --unattended if ! ${core_update} && ! ${web_update} ; then - echo ":::" - echo "::: Everything is up to date!" - exit 0 + if ! ${FTL_update} ; then + echo ":::" + echo "::: Everything is up to date!" + exit 0 + fi elif ! ${core_update} && ${web_update} ; then echo ":::" @@ -139,7 +169,7 @@ main() { elif ${core_update} && ${web_update} ; then echo ":::" - echo "::: Updating Everything" + echo "::: Updating Pi-hole core and web admin files" getGitFiles "${PI_HOLE_FILES_DIR}" "${PI_HOLE_GIT_URL}" ${PI_HOLE_FILES_DIR}/automated\ install/basic-install.sh --unattended || echo "Unable to complete update, contact Pi-hole" && exit 1 else @@ -148,9 +178,11 @@ main() { fi else # Web Admin not installed, so only verify if core is up to date if ! ${core_update}; then - echo ":::" - echo "::: Everything is up to date!" - exit 0 + if ! ${FTL_update} ; then + echo ":::" + echo "::: Everything is up to date!" + exit 0 + fi else echo ":::" echo "::: Pi-hole core files out of date" @@ -173,6 +205,15 @@ main() { echo "::: If you had made any changes in '/etc/.pihole/', they have been stashed using 'git stash'" fi + if [[ ${FTL_update} == true ]]; then + FTL_version_current="$(/usr/bin/pihole-FTL tag)" + echo ":::" + echo "::: FTL version is now at ${FTL_version_current}" + start_service pihole-FTL + enable_service pihole-FTL + fi + + echo "" exit 0 diff --git a/advanced/Scripts/webpage.sh b/advanced/Scripts/webpage.sh index 5f032ed1..7804fc8f 100755 --- a/advanced/Scripts/webpage.sh +++ b/advanced/Scripts/webpage.sh @@ -81,19 +81,28 @@ SetWebPassword(){ exit 1 fi - # Set password only if there is one to be set - if (( ${#args[2]} > 0 )) ; then + read -s -p "Enter New Password (Blank for no password): " PASSWORD + echo "" + + if [ "${PASSWORD}" == "" ]; then + change_setting "WEBPASSWORD" "" + echo "Password Removed" + exit 0 + fi + + read -s -p "Confirm Password: " CONFIRM + echo "" + if [ "${PASSWORD}" == "${CONFIRM}" ] ; then # Compute password hash twice to avoid rainbow table vulnerability - hash=$(echo -n ${args[2]} | sha256sum | sed 's/\s.*$//') + hash=$(echo -n ${PASSWORD} | sha256sum | sed 's/\s.*$//') hash=$(echo -n ${hash} | sha256sum | sed 's/\s.*$//') # Save hash to file change_setting "WEBPASSWORD" "${hash}" echo "New password set" else - change_setting "WEBPASSWORD" "" - echo "Password removed" + echo "Passwords don't match. Your password has not been changed" + exit 1 fi - } ProcessDNSSettings() { @@ -319,6 +328,25 @@ SetWebUILayout(){ } +CustomizeAdLists() { + + list="/etc/pihole/adlists.list" + + if [[ "${args[2]}" == "enable" ]] ; then + sed -i "\\@${args[3]}@s/^#http/http/g" "${list}" + elif [[ "${args[2]}" == "disable" ]] ; then + sed -i "\\@${args[3]}@s/^http/#http/g" "${list}" + elif [[ "${args[2]}" == "add" ]] ; then + echo "${args[3]}" >> ${list} + elif [[ "${args[2]}" == "del" ]] ; then + var=$(echo "${args[3]}" | sed 's/\//\\\//g') + sed -i "/${var}/Id" "${list}" + else + echo "Not permitted" + return 1 + fi +} + SetPrivacyMode(){ if [[ "${args[2]}" == "true" ]] ; then @@ -410,6 +438,12 @@ SetListeningMode(){ } +Teleporter() +{ + local datetimestamp=$(date "+%Y-%m-%d_%H-%M-%S") + php /var/www/html/admin/scripts/pi-hole/php/teleporter.php > "pi-hole-teleporter_${datetimestamp}.zip" +} + main() { args=("$@") @@ -435,6 +469,8 @@ main() { "removestaticdhcp" ) RemoveDHCPStaticAddress;; "hostrecord" ) SetHostRecord;; "-i" | "interface" ) SetListeningMode;; + "-t" | "teleporter" ) Teleporter;; + "adlist" ) CustomizeAdLists;; * ) helpFunc;; esac diff --git a/advanced/index.php b/advanced/index.php index c076f92d..30bd131c 100644 --- a/advanced/index.php +++ b/advanced/index.php @@ -1,15 +1,47 @@ + Website Blocked diff --git a/advanced/pihole-FTL.service b/advanced/pihole-FTL.service new file mode 100644 index 00000000..30cd140f --- /dev/null +++ b/advanced/pihole-FTL.service @@ -0,0 +1,80 @@ +#!/bin/bash +### BEGIN INIT INFO +# Provides: pihole-FTL +# Required-Start: $remote_fs $syslog +# Required-Stop: $remote_fs $syslog +# Default-Start: 2 3 4 5 +# Default-Stop: 0 1 6 +# Short-Description: pihole-FTL daemon +# Description: Enable service provided by pihole-FTL daemon +### END INIT INFO + +FTLUSER=pihole +PIDFILE=/var/run/pihole-FTL.pid + +get_pid() { + pidof "pihole-FTL" +} + +is_running() { + ps "$(get_pid)" > /dev/null 2>&1 +} + +# Start the service +start() { + if is_running; then + echo "pihole-FTL is already running" + else + touch /var/log/pihole-FTL.log /run/pihole-FTL.pid /run/pihole-FTL.port + chown pihole:pihole /var/log/pihole-FTL.log /run/pihole-FTL.pid /run/pihole-FTL.port + chmod 0644 /var/log/pihole-FTL.log /run/pihole-FTL.pid /run/pihole-FTL.port + su -s /bin/sh -c "/usr/bin/pihole-FTL" "$FTLUSER" + echo + fi +} + +# Stop the service +stop() { + if is_running; then + kill "$(get_pid)" + for i in {1..5}; do + if ! is_running; then + break + fi + + echo -n "." + sleep 1 + done + echo + + if is_running; then + echo "Not stopped; may still be shutting down or shutdown may have failed, killing now" + kill -9 "$(get_pid)" + exit 1 + else + echo "Stopped" + fi + else + echo "Not running" + fi + echo +} + +### main logic ### +case "$1" in + stop) + stop + ;; + status) + status pihole-FTL + ;; + start|restart|reload|condrestart) + stop + start + ;; + *) + echo $"Usage: $0 {start|stop|restart|reload|status}" + exit 1 +esac + +exit 0 diff --git a/automated install/basic-install.sh b/automated install/basic-install.sh index f168675e..cb5dd6fc 100755 --- a/automated install/basic-install.sh +++ b/automated install/basic-install.sh @@ -54,13 +54,39 @@ skipSpaceCheck=false reconfigure=false runUnattended=false +show_ascii_berry() { + echo " + .;;,. + .ccccc:,. + :cccclll:. ..,, + :ccccclll. ;ooodc + 'ccll:;ll .oooodc + .;cll.;;looo:. + .. ','. + .',,,,,,'. + .',,,,,,,,,,. + .',,,,,,,,,,,,.... + ....''',,,,,,,'....... + ......... .... ......... + .......... .......... + .......... .......... + ......... .... ......... + ........,,,,,,,'...... + ....',,,,,,,,,,,,. + .',,,,,,,,,'. + .',,,,,,'. + ..'''. +" +} + + # Compatibility distro_check() { if command -v apt-get &> /dev/null; then #Debian Family ############################################# PKG_MANAGER="apt-get" - UPDATE_PKG_CACHE="${PKG_MANAGER} update" + UPDATE_PKG_CACHE="test_dpkg_lock; ${PKG_MANAGER} update" PKG_INSTALL=(${PKG_MANAGER} --yes --no-install-recommends install) # grep -c will return 1 retVal on 0 matches, block this throwing the set -e with an OR TRUE PKG_COUNT="${PKG_MANAGER} -s -o Debug::NoLocking=true upgrade | grep -c ^Inst || true" @@ -87,6 +113,17 @@ if command -v apt-get &> /dev/null; then LIGHTTPD_CFG="lighttpd.conf.debian" DNSMASQ_USER="dnsmasq" + test_dpkg_lock() { + i=0 + while fuser /var/lib/dpkg/lock >/dev/null 2>&1 ; do + sleep 0.5 + ((i=i+1)) + done + # Always return success, since we only return if there is no + # lock (anymore) + return 0 + } + elif command -v rpm &> /dev/null; then # Fedora Family if command -v dnf &> /dev/null; then @@ -196,7 +233,7 @@ find_IPv4_information() { get_available_interfaces() { # Get available UP interfaces. - availableInterfaces=$(ip -o link | grep -v "state DOWN\|lo" | awk '{print $2}' | cut -d':' -f1 | cut -d'@' -f1) + availableInterfaces=$(ip --oneline link show up | grep -v "lo" | awk '{print $2}' | cut -d':' -f1 | cut -d'@' -f1) } welcomeDialogs() { @@ -231,7 +268,7 @@ verifyFreeDiskSpace() { # - Insufficient free disk space elif [[ ${existing_free_kilobytes} -lt ${required_free_kilobytes} ]]; then echo "::: Insufficient Disk Space!" - echo "::: Your system appears to be low on disk space. pi-hole recommends a minimum of $required_free_kilobytes KiloBytes." + echo "::: Your system appears to be low on disk space. Pi-hole recommends a minimum of $required_free_kilobytes KiloBytes." echo "::: You only have ${existing_free_kilobytes} KiloBytes free." echo "::: If this is a new install you may need to expand your disk." echo "::: Try running 'sudo raspi-config', and choose the 'expand file system option'" @@ -401,7 +438,7 @@ setStaticIPv4() { cp "${IFCFG_FILE}" "${IFCFG_FILE}".pihole.orig # Build Interface configuration file: { - echo "# Configured via Pi-Hole installer" + echo "# Configured via Pi-hole installer" echo "DEVICE=$PIHOLE_INTERFACE" echo "BOOTPROTO=none" echo "ONBOOT=yes" @@ -582,14 +619,14 @@ version_check_dnsmasq() { local dnsmasq_conf="/etc/dnsmasq.conf" local dnsmasq_conf_orig="/etc/dnsmasq.conf.orig" local dnsmasq_pihole_id_string="addn-hosts=/etc/pihole/gravity.list" - local dnsmasq_original_config="/etc/.pihole/advanced/dnsmasq.conf.original" - local dnsmasq_pihole_01_snippet="/etc/.pihole/advanced/01-pihole.conf" + local dnsmasq_original_config="${PI_HOLE_LOCAL_REPO}/advanced/dnsmasq.conf.original" + local dnsmasq_pihole_01_snippet="${PI_HOLE_LOCAL_REPO}/advanced/01-pihole.conf" local dnsmasq_pihole_01_location="/etc/dnsmasq.d/01-pihole.conf" if [ -f ${dnsmasq_conf} ]; then echo -n "::: Existing dnsmasq.conf found..." if grep -q ${dnsmasq_pihole_id_string} ${dnsmasq_conf}; then - echo " it is from a previous pi-hole install." + echo " it is from a previous Pi-hole install." echo -n "::: Backing up dnsmasq.conf to dnsmasq.conf.orig..." mv -f ${dnsmasq_conf} ${dnsmasq_conf_orig} echo " done." @@ -597,7 +634,7 @@ version_check_dnsmasq() { cp ${dnsmasq_original_config} ${dnsmasq_conf} echo " done." else - echo " it is not a pi-hole file, leaving alone!" + echo " it is not a Pi-hole file, leaving alone!" fi else echo -n "::: No dnsmasq.conf found.. restoring default dnsmasq.conf..." @@ -669,9 +706,9 @@ installScripts() { } installConfigs() { - # Install the configs from /etc/.pihole to their various locations + # Install the configs from PI_HOLE_LOCAL_REPO to their various locations echo ":::" - echo "::: Installing configs..." + echo "::: Installing configs from ${PI_HOLE_LOCAL_REPO}..." version_check_dnsmasq #Only mess with lighttpd configs if user has chosen to install web interface @@ -682,7 +719,7 @@ installConfigs() { elif [ -f "/etc/lighttpd/lighttpd.conf" ]; then mv /etc/lighttpd/lighttpd.conf /etc/lighttpd/lighttpd.conf.orig fi - cp /etc/.pihole/advanced/${LIGHTTPD_CFG} /etc/lighttpd/lighttpd.conf + cp ${PI_HOLE_LOCAL_REPO}/advanced/${LIGHTTPD_CFG} /etc/lighttpd/lighttpd.conf mkdir -p /var/run/lighttpd chown ${LIGHTTPD_USER}:${LIGHTTPD_GROUP} /var/run/lighttpd mkdir -p /var/cache/lighttpd/compress @@ -739,10 +776,11 @@ update_package_cache() { echo ":::" echo -n "::: Updating local cache of available packages..." - if eval ${UPDATE_PKG_CACHE} &> /dev/null; then + if eval "${UPDATE_PKG_CACHE}" &> /dev/null; then echo " done!" else - echo -n "\n!!! ERROR - Unable to update package cache. Please try \"${UPDATE_PKG_CACHE}\"" + echo -en "\n!!! ERROR - Unable to update package cache. Please try \"${UPDATE_PKG_CACHE}\"" + return 1 fi } @@ -759,7 +797,7 @@ notify_package_updates_available() { echo "::: Your system is up to date! Continuing with Pi-hole installation..." else echo "::: There are ${updatesToInstall} updates available for your system!" - echo "::: We recommend you update your OS after installing Pi-Hole! " + echo "::: We recommend you update your OS after installing Pi-hole! " echo ":::" fi else @@ -789,6 +827,7 @@ install_dependent_packages() { fi done if [[ ${#installArray[@]} -gt 0 ]]; then + test_dpkg_lock debconf-apt-progress -- "${PKG_INSTALL[@]}" "${installArray[@]}" return fi @@ -835,7 +874,7 @@ installPiholeWeb() { echo "::: Existing index.php detected, not overwriting" else echo -n "::: index.php missing, replacing... " - cp /etc/.pihole/advanced/index.php /var/www/html/pihole/ + cp ${PI_HOLE_LOCAL_REPO}/advanced/index.php /var/www/html/pihole/ echo " done!" fi @@ -843,7 +882,7 @@ installPiholeWeb() { echo "::: Existing index.js detected, not overwriting" else echo -n "::: index.js missing, replacing... " - cp /etc/.pihole/advanced/index.js /var/www/html/pihole/ + cp ${PI_HOLE_LOCAL_REPO}/advanced/index.js /var/www/html/pihole/ echo " done!" fi @@ -851,14 +890,14 @@ installPiholeWeb() { echo "::: Existing blockingpage.css detected, not overwriting" else echo -n "::: blockingpage.css missing, replacing... " - cp /etc/.pihole/advanced/blockingpage.css /var/www/html/pihole + cp ${PI_HOLE_LOCAL_REPO}/advanced/blockingpage.css /var/www/html/pihole echo " done!" fi else echo "::: Creating directory for blocking page" install -d /var/www/html/pihole - install -D /etc/.pihole/advanced/{index,blockingpage}.* /var/www/html/pihole/ + install -D ${PI_HOLE_LOCAL_REPO}/advanced/{index,blockingpage}.* /var/www/html/pihole/ if [ -f /var/www/html/index.lighttpd.html ]; then mv /var/www/html/index.lighttpd.html /var/www/html/index.lighttpd.orig else @@ -871,7 +910,7 @@ installPiholeWeb() { echo ":::" echo -n "::: Installing sudoer file..." mkdir -p /etc/sudoers.d/ - cp /etc/.pihole/advanced/pihole.sudo /etc/sudoers.d/pihole + cp ${PI_HOLE_LOCAL_REPO}/advanced/pihole.sudo /etc/sudoers.d/pihole # Add lighttpd user (OS dependent) to sudoers file echo "${LIGHTTPD_USER} ALL=NOPASSWD: /usr/local/bin/pihole" >> /etc/sudoers.d/pihole @@ -889,7 +928,7 @@ installCron() { # Install the cron job echo ":::" echo -n "::: Installing latest Cron script..." - cp /etc/.pihole/advanced/pihole.cron /etc/cron.d/pihole + cp ${PI_HOLE_LOCAL_REPO}/advanced/pihole.cron /etc/cron.d/pihole echo " done!" } @@ -903,7 +942,7 @@ runGravity() { fi # Test if /etc/pihole/adlists.default exists if [[ ! -e /etc/pihole/adlists.default ]]; then - cp /etc/.pihole/adlists.default /etc/pihole/adlists.default + cp ${PI_HOLE_LOCAL_REPO}/adlists.default /etc/pihole/adlists.default fi echo "::: Running gravity.sh" { /opt/pihole/gravity.sh; } @@ -926,7 +965,7 @@ configureFirewall() { whiptail --title "Firewall in use" --yesno "We have detected a running firewall\n\nPi-hole currently requires HTTP and DNS port access.\n\n\n\nInstall Pi-hole default firewall rules?" ${r} ${c} || \ { echo -e ":::\n::: Not installing firewall rulesets."; return 0; } echo -e ":::\n:::\n Configuring FirewallD for httpd and dnsmasq." - firewall-cmd --permanent --add-port=80/tcp --add-port=53/tcp --add-port=53/udp + firewall-cmd --permanent --add-service=http --add-service=dns firewall-cmd --reload return 0 # Check for proper kernel modules to prevent failure @@ -978,7 +1017,7 @@ finalExports() { # Look for DNS server settings which would have to be reapplied source "${setupVars}" - source "/etc/.pihole/advanced/Scripts/webpage.sh" + source "${PI_HOLE_LOCAL_REPO}/advanced/Scripts/webpage.sh" if [[ "${DNS_FQDN_REQUIRED}" != "" ]] ; then ProcessDNSSettings @@ -993,7 +1032,7 @@ installLogrotate() { # Install the logrotate script echo ":::" echo -n "::: Installing latest logrotate script..." - cp /etc/.pihole/advanced/logrotate /etc/pihole/logrotate + cp ${PI_HOLE_LOCAL_REPO}/advanced/logrotate /etc/pihole/logrotate # Different operating systems have different user / group # settings for logrotate that makes it impossible to create # a static logrotate file that will work with e.g. @@ -1032,6 +1071,7 @@ installPihole() { fi installCron installLogrotate + FTLdetect || echo "::: FTL Engine not installed." configureFirewall finalExports #runGravity @@ -1063,6 +1103,7 @@ updatePihole() { fi installCron installLogrotate + FTLdetect || echo "::: FTL Engine not installed." finalExports #re-export setupVars.conf to account for any new vars added in new versions #runGravity } @@ -1156,10 +1197,98 @@ if [[ "${reconfigure}" == true ]]; then fi } +FTLinstall() { + # Download and install FTL binary + local binary="${1}" + local latesttag + local orig_dir + echo -n "::: Installing FTL... " + + orig_dir="${PWD}" + latesttag=$(curl -sI https://github.com/pi-hole/FTL/releases/latest | grep "Location" | awk -F '/' '{print $NF}') + # Tags should always start with v, check for that. + if [[ ! "${latesttag}" == v* ]]; then + echo "failed (error in getting latest release location from GitHub)" + return 1 + fi + if curl -sSL --fail "https://github.com/pi-hole/FTL/releases/download/${latesttag%$'\r'}/${binary}" -o "/tmp/${binary}"; then + # Get sha1 of the binary we just downloaded for verification. + curl -sSL --fail "https://github.com/pi-hole/FTL/releases/download/${latesttag%$'\r'}/${binary}.sha1" -o "/tmp/${binary}.sha1" + # Check if we just downloaded text, or a binary file. + cd /tmp + if sha1sum --status --quiet -c "${binary}".sha1; then + echo -n "transferred... " + stop_service pihole-FTL &> /dev/null + install -T -m 0755 /tmp/${binary} /usr/bin/pihole-FTL + cd "${orig_dir}" + install -T -m 0755 "${PI_HOLE_LOCAL_REPO}/advanced/pihole-FTL.service" "/etc/init.d/pihole-FTL" + echo "done." + return 0 + else + echo "failed (download of binary from Github failed)" + cd "${orig_dir}" + return 1 + fi + else + cd "${orig_dir}" + echo "failed (URL not found.)" + fi +} + +FTLdetect() { + # Detect suitable FTL binary platform + echo ":::" + echo "::: Downloading latest version of FTL..." + + local machine + local binary + + machine=$(uname -m) + + if [[ $machine == arm* || $machine == *aarch* ]]; then + # ARM + local rev=$(uname -m | sed "s/[^0-9]//g;") + local lib=$(ldd /bin/ls | grep -E '^\s*/lib' | awk '{ print $1 }') + if [[ "$lib" == "/lib/ld-linux-aarch64.so.1" ]]; then + echo "::: Detected ARM-aarch64 architecture" + binary="pihole-FTL-aarch64-linux-gnu" + elif [[ "$lib" == "/lib/ld-linux-armhf.so.3" ]]; then + if [ "$rev" -gt "6" ]; then + echo "::: Detected ARM-hf architecture (armv7+)" + binary="pihole-FTL-arm-linux-gnueabihf" + else + echo "::: Detected ARM-hf architecture (armv6 or lower)" + echo "::: Using ARM binary" + binary="pihole-FTL-arm-linux-gnueabi" + fi + else + echo "::: Detected ARM architecture" + binary="pihole-FTL-arm-linux-gnueabi" + fi + elif [[ $machine == x86_64 ]]; then + # 64bit + echo "::: Detected x86_64 architecture" + binary="pihole-FTL-linux-x86_64" + else + # Something else - we try to use 32bit executable and warn the user + if [[ ! $machine == i686 ]]; then + echo "::: Not able to detect architecture (unknown: ${machine}), trying 32bit executable" + echo "::: Contact Pi-hole support if you experience problems (like FTL not running)" + else + echo "::: Detected 32bit (i686) architecture" + fi + binary="pihole-FTL-linux-x86_32" + fi + + FTLinstall "${binary}" || return 1 + +} + main() { ######## FIRST CHECK ######## # Must be root to install + show_ascii_berry echo ":::" if [[ ${EUID} -eq 0 ]]; then echo "::: You are root." @@ -1210,7 +1339,7 @@ main() { fi # Update package cache - update_package_cache + update_package_cache || exit 1 # Notify user of package availability notify_package_updates_available @@ -1300,6 +1429,9 @@ main() { runGravity + start_service pihole-FTL + enable_service pihole-FTL + echo "::: done." if [[ "${useUpdateVars}" == false ]]; then @@ -1328,7 +1460,7 @@ main() { echo "::: ${pw}" echo ":::" echo "::: You can always change it using" - echo "::: pihole -a -p new_password" + echo "::: pihole -a -p" fi fi diff --git a/gravity.sh b/gravity.sh index 08590251..0dccd77b 100755 --- a/gravity.sh +++ b/gravity.sh @@ -26,10 +26,12 @@ EOM exit 0 } +PIHOLE_COMMAND="/usr/local/bin/pihole" adListFile=/etc/pihole/adlists.list -adListDefault=/etc/pihole/adlists.default -whitelistScript="pihole -w" +adListDefault=/etc/pihole/adlists.default #being deprecated +adListRepoDefault=/etc/.pihole/adlists.default +whitelistScript="${PIHOLE_COMMAND} -w" whitelistFile=/etc/pihole/whitelist.txt blacklistFile=/etc/pihole/blacklist.txt readonly wildcardlist="/etc/dnsmasq.d/03-pihole-wildcard.conf" @@ -70,36 +72,34 @@ fi ########################### # collapse - begin formation of pihole gravity_collapse() { + + #New Logic: + # Does /etc/pihole/adlists.list exist? If so leave it alone + # If not, cp /etc/.pihole/adlists.default /etc/pihole/adlists.list + # Read from adlists.list + + #The following two blocks will sort out any missing adlists in the /etc/pihole directory, and remove legacy adlists.default + if [ -f ${adListDefault} ] && [ -f ${adListFile} ]; then + rm ${adListDefault} + fi + + if [ ! -f ${adListFile} ]; then + cp ${adListRepoDefault} ${adListFile} + fi + echo "::: Neutrino emissions detected..." echo ":::" - #Decide if we're using a custom ad block list, or defaults. - if [ -f ${adListFile} ]; then - #custom file found, use this instead of default - echo -n "::: Custom adList file detected. Reading..." - sources=() - while IFS= read -r line || [[ -n "$line" ]]; do - #Do not read commented out or blank lines - if [[ ${line} = \#* ]] || [[ ! ${line} ]]; then - echo "" > /dev/null - else - sources+=(${line}) - fi - done < ${adListFile} - echo " done!" - else - #no custom file found, use defaults! - echo -n "::: No custom adlist file detected, reading from default file..." - sources=() - while IFS= read -r line || [[ -n "$line" ]]; do - #Do not read commented out or blank lines - if [[ ${line} = \#* ]] || [[ ! ${line} ]]; then - echo "" > /dev/null - else - sources+=(${line}) - fi - done < ${adListDefault} - echo " done!" - fi + echo -n "::: Pulling source lists into range..." + sources=() + while IFS= read -r line || [[ -n "$line" ]]; do + #Do not read commented out or blank lines + if [[ ${line} = \#* ]] || [[ ! ${line} ]]; then + echo "" > /dev/null + else + sources+=(${line}) + fi + done < ${adListFile} + echo " done!" } # patternCheck - check to see if curl downloaded any new files. @@ -167,7 +167,11 @@ gravity_transport() { # Process result gravity_patternCheck "${patternBuffer}" ${success} "${err}" - + + # Delete temp file if it hasn't been moved + if [[ -f "${patternBuffer}" ]]; then + rm "${patternBuffer}" + fi } # spinup - main gravity function @@ -188,17 +192,21 @@ gravity_spinup() { # Use a case statement to download lists that need special cURL commands # to complete properly and reset the user agent when required case "${domain}" in - "adblock.mahakala.is") + "adblock.mahakala.is") agent='Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2228.0 Safari/537.36' cmd_ext="-e http://forum.xda-developers.com/" ;; + + "adaway.org") + agent='Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2228.0 Safari/537.36' + ;; "pgl.yoyo.org") cmd_ext="-d mimetype=plaintext -d hostformat=hosts" ;; - # Default is a simple request - *) cmd_ext="" + # Default is a simple request + *) cmd_ext="" esac if [[ "${skipDownload}" == false ]]; then echo -n "::: Getting $domain list..." @@ -385,7 +393,7 @@ gravity_reload() { #Now replace the line in dnsmasq file # sed -i "s/^addn-hosts.*/addn-hosts=$adList/" /etc/dnsmasq.d/01-pihole.conf - pihole restartdns + "${PIHOLE_COMMAND}" restartdns echo " done!" } @@ -403,8 +411,6 @@ if [[ "${forceGrav}" == true ]]; then echo " done!" fi -#Overwrite adlists.default from /etc/.pihole in case any changes have been made. Changes should be saved in /etc/adlists.list -cp /etc/.pihole/adlists.default /etc/pihole/adlists.default gravity_collapse gravity_spinup if [[ "${skipDownload}" == false ]]; then @@ -423,4 +429,4 @@ gravity_hostFormat gravity_blackbody gravity_reload -pihole status +"${PIHOLE_COMMAND}" status diff --git a/pihole b/pihole index dda6d750..83e13000 100755 --- a/pihole +++ b/pihole @@ -287,17 +287,17 @@ piholeCheckoutFunc() { helpFunc() { cat << EOM -::: Control all PiHole specific functions! +::: Control all Pi-hole specific functions ::: ::: Usage: pihole [options] -::: Add -h after -w (whitelist), -b (blacklist), -c (chronometer), or -a (admin) for more information on usage +::: Add -h after -w (whitelist), -b (blacklist), -c (chronometer), or -a (admin) for more information on usage ::: ::: Options: ::: -w, whitelist Whitelist domain(s) ::: -b, blacklist Blacklist domain(s) (exact match) ::: -wild, wildcard Blacklist whole domain(s) (wildcard) ::: -d, debug Start a debugging session -::: Automated debugging can be enabled with `-a`. +::: Automated debugging can be enabled with '-a'. ::: 'pihole -d -a' ::: -f, flush Flush the 'pihole.log' file ::: -t, tail Output the last lines of the 'pihole.log' file. Lines are appended as the file grows @@ -306,15 +306,15 @@ helpFunc() { ::: -g, updateGravity Update the list of ad-serving domains ::: -c, chronometer Calculates stats and displays to an LCD ::: -h, help Show this help dialog -::: -v, version Show installed versions of Pi-Hole and Web-Admin +::: -v, version Show installed versions of Pi-hole and Web-Admin ::: -q, query Query the adlists for a specific domain ::: 'pihole -q domain -exact' shows exact matches only ::: -l, logging Enable or Disable logging (pass 'on' or 'off') ::: -a, admin Admin webpage options -::: uninstall Uninstall Pi-Hole from your system :(! -::: status Is Pi-Hole Enabled or Disabled -::: enable Enable Pi-Hole DNS Blocking -::: disable Disable Pi-Hole DNS Blocking +::: uninstall Uninstall Pi-hole from your system! :( +::: status Display if Pi-hole is Enabled or Disabled +::: enable Enable Pi-hole DNS Blocking +::: disable Disable Pi-hole DNS Blocking ::: Blocking can also be disabled only temporarily, e.g., ::: 'pihole disable 5m' - will disable blocking for 5 minutes ::: restartdns Restart dnsmasq diff --git a/test/debian.Dockerfile b/test/debian.Dockerfile index 931c0ba7..66436f1a 100644 --- a/test/debian.Dockerfile +++ b/test/debian.Dockerfile @@ -1,4 +1,4 @@ -FROM debian:jessie +FROM buildpack-deps:jessie-scm ENV GITDIR /etc/.pihole ENV SCRIPTDIR /opt/pihole diff --git a/test/test_automated_install.py b/test/test_automated_install.py index 89d9a0e0..8e36fc96 100644 --- a/test/test_automated_install.py +++ b/test/test_automated_install.py @@ -78,7 +78,7 @@ def test_configureFirewall_firewalld_running_no_errors(Pihole): assert expected_stdout in configureFirewall.stdout firewall_calls = Pihole.run('cat /var/log/firewall-cmd').stdout assert 'firewall-cmd --state' in firewall_calls - assert 'firewall-cmd --permanent --add-port=80/tcp --add-port=53/tcp --add-port=53/udp' in firewall_calls + assert 'firewall-cmd --permanent --add-service=http --add-service=dns' in firewall_calls assert 'firewall-cmd --reload' in firewall_calls def test_configureFirewall_firewalld_disabled_no_errors(Pihole): @@ -297,6 +297,111 @@ def test_update_package_cache_failure_no_errors(Pihole): assert 'ERROR' in updateCache.stdout assert 'done!' not in updateCache.stdout +def test_FTL_detect_aarch64_no_errors(Pihole): + ''' confirms only aarch64 package is downloaded for FTL engine ''' + # mock uname to return aarch64 platform + mock_command('uname', {'-m':('aarch64', '0')}, Pihole) + # mock ldd to respond with aarch64 shared library + mock_command('ldd', {'/bin/ls':('/lib/ld-linux-aarch64.so.1', '0')}, Pihole) + detectPlatform = Pihole.run(''' + source /opt/pihole/basic-install.sh + FTLdetect + ''') + expected_stdout = 'Detected ARM-aarch64 architecture' + assert expected_stdout in detectPlatform.stdout + +def test_FTL_detect_armv6l_no_errors(Pihole): + ''' confirms only armv6l package is downloaded for FTL engine ''' + # mock uname to return armv6l platform + mock_command('uname', {'-m':('armv6l', '0')}, Pihole) + # mock ldd to respond with aarch64 shared library + mock_command('ldd', {'/bin/ls':('/lib/ld-linux-armhf.so.3', '0')}, Pihole) + detectPlatform = Pihole.run(''' + source /opt/pihole/basic-install.sh + FTLdetect + ''') + expected_stdout = 'Detected ARM-hf architecture (armv6 or lower)' + assert expected_stdout in detectPlatform.stdout + +def test_FTL_detect_armv7l_no_errors(Pihole): + ''' confirms only armv7l package is downloaded for FTL engine ''' + # mock uname to return armv7l platform + mock_command('uname', {'-m':('armv7l', '0')}, Pihole) + # mock ldd to respond with aarch64 shared library + mock_command('ldd', {'/bin/ls':('/lib/ld-linux-armhf.so.3', '0')}, Pihole) + detectPlatform = Pihole.run(''' + source /opt/pihole/basic-install.sh + FTLdetect + ''') + expected_stdout = 'Detected ARM-hf architecture (armv7+)' + assert expected_stdout in detectPlatform.stdout + +def test_FTL_detect_x86_64_no_errors(Pihole): + ''' confirms only x86_64 package is downloaded for FTL engine ''' + detectPlatform = Pihole.run(''' + source /opt/pihole/basic-install.sh + FTLdetect + ''') + expected_stdout = 'Detected x86_64 architecture' + assert expected_stdout in detectPlatform.stdout + +def test_FTL_detect_unknown_no_errors(Pihole): + ''' confirms only generic package is downloaded for FTL engine ''' + # mock uname to return generic platform + mock_command('uname', {'-m':('mips', '0')}, Pihole) + detectPlatform = Pihole.run(''' + source /opt/pihole/basic-install.sh + FTLdetect + ''') + expected_stdout = 'Not able to detect architecture (unknown: mips)' + assert expected_stdout in detectPlatform.stdout + +def test_FTL_download_aarch64_no_errors(Pihole): + ''' confirms only aarch64 package is downloaded for FTL engine ''' + # mock uname to return generic platform + download_binary = Pihole.run(''' + source /opt/pihole/basic-install.sh + FTLinstall pihole-FTL-aarch64-linux-gnu + ''') + expected_stdout = 'done' + assert expected_stdout in download_binary.stdout + assert 'failed' not in download_binary.stdout + +def test_FTL_download_unknown_fails_no_errors(Pihole): + ''' confirms unknown binary is not downloaded for FTL engine ''' + # mock uname to return generic platform + download_binary = Pihole.run(''' + source /opt/pihole/basic-install.sh + FTLinstall pihole-FTL-mips + ''') + expected_stdout = 'failed' + assert expected_stdout in download_binary.stdout + assert 'done' not in download_binary.stdout + +def test_FTL_binary_installed_and_responsive_no_errors(Pihole): + ''' confirms FTL binary is copied and functional in installed location ''' + installed_binary = Pihole.run(''' + source /opt/pihole/basic-install.sh + FTLdetect + pihole-FTL version + ''') + expected_stdout = 'v' + assert expected_stdout in installed_binary.stdout + +# def test_FTL_support_files_installed(Pihole): +# ''' confirms FTL support files are installed ''' +# support_files = Pihole.run(''' +# source /opt/pihole/basic-install.sh +# FTLdetect +# stat -c '%a %n' /var/log/pihole-FTL.log +# stat -c '%a %n' /run/pihole-FTL.port +# stat -c '%a %n' /run/pihole-FTL.pid +# ls -lac /run +# ''') +# assert '644 /run/pihole-FTL.port' in support_files.stdout +# assert '644 /run/pihole-FTL.pid' in support_files.stdout +# assert '644 /var/log/pihole-FTL.log' in support_files.stdout + # Helper functions def mock_command(script, args, container): ''' Allows for setup of commands we don't really want to have to run for real in unit tests '''