Merge branch 'PiholeDebug'

Conflicts:
	README.md

README.md

@@ -1,14 +1,19 @@
+<<<<<<< HEAD
 # Automated Install
 ##### Designed For Raspberry Pi A+, B, B+, 2, Zero (with an Ethernet adapter), and also works on most Debian based distros!
+=======
+# Automated Install 
+##### Designed For Raspberry Pi A+, B, B+, 2, Zero, and 3B (with an Ethernet/Wi-Fi adapter)
+>>>>>>> PiholeDebug
 
 1. Install Raspbian on Raspberry Pi variants or a Debian based distribution on other hardware or virtual machines
 2. Run the command below
 
-### ```curl -L install.pi-hole.net | bash```
+### ```curl -L https://install.pi-hole.net | bash```
 
 #### Alternative Semi-Automated install ####
 ```
-wget -O basic-install.sh install.pi-hole.net
+wget -O basic-install.sh https://install.pi-hole.net
 chmod +x basic-install.sh
 ./basic-install.sh
 ```
@@ -18,13 +23,20 @@ Once installed, [configure your router to have **DHCP clients use the Pi as thei
 ## Pi-hole Is Free, But Powered By Your Donations
 [![Donate](https://www.paypalobjects.com/en_US/i/btn/btn_donateCC_LG.gif "Free, but powered by donations")](https://www.paypal.com/cgi-bin/webscr?cmd=_s-xclick&hosted_button_id=3J2L3Z4DHW9UY "Donate")
 
+## Catch us out on the net:
+Twitter: [@The_Pi_Hole](https://twitter.com/The_Pi_Hole)
+
+reddit: [/r/pihole](https://www.reddit.com/r/pihole/)
+
 ## How Does It Work?
 **Watch the 60-second video below to get a quick overview**
 
 [![Pi-hole exlplained](http://i.imgur.com/qNybJDX.png)](https://vimeo.com/135965232)
 
 ## Pi-hole Projects
+- [Pi-hole on Ubuntu 14.04 on VirtualBox](http://hbalagtas.blogspot.com/2016/02/adblocking-with-pi-hole-and-ubuntu-1404.html)
 - [x86 Docker container that runs Pi-hole](https://hub.docker.com/r/diginc/pi-hole/)
+- [Splunk: Pi-hole Visualizser](https://splunkbase.splunk.com/app/3023/)
 - [Pi-hole Chrome extension](https://chrome.google.com/webstore/detail/pi-hole-list-editor/hlnoeoejkllgkjbnnnhfolapllcnaglh) ([open source](https://github.com/packtloss/pihole-extension))
 - [Go Bananas for CHiP-hole ad blocking](https://www.hackster.io/jacobsalmela/chip-hole-network-wide-ad-blocker-98e037)
 - [Sky-Hole](http://dlaa.me/blog/post/skyhole)
@@ -34,7 +46,7 @@ Once installed, [configure your router to have **DHCP clients use the Pi as thei
 - [Minibian Pi-hole](http://munkjensen.net/wiki/index.php/See_my_Pi-Hole#Minibian_Pi-hole)
 
 ## Coverage
-- [Splunk: Pi-hole Visualizser](https://splunkbase.splunk.com/app/3023/)
+- [Pi-hole on Adafruit's blog](https://blog.adafruit.com/2016/03/04/pi-hole-is-a-black-hole-for-internet-ads-piday-raspberrypi-raspberry_pi/)
 - [The Defrag Show - MSDN/Channel 9](https://channel9.msdn.com/Shows/The-Defrag-Show/Defrag-Endoscope-USB-Camera-The-Final-HoloLens-Vote-Adblock-Pi-and-more?WT.mc_id=dlvr_twitter_ch9#time=20m39s)
 - [MacObserver Podcast 585](http://www.macobserver.com/tmo/podcast/macgeekgab-585)
 - [Medium: Block All Ads For $53](https://medium.com/@robleathern/block-ads-on-all-home-devices-for-53-18-a5f1ec139693#.gj1xpgr5d)
@@ -67,6 +79,8 @@ The [Web interface](https://github.com/jacobsalmela/AdminLTE#pi-hole-admin-dashb
 
 `http://192.168.1.x/admin/index.php`
 
+![Pi-hole Advanced Stats Dashboard](http://i.imgur.com/rTlLYPh.png)
+
 ### API
 
 A basic read-only API can be accessed at `/admin/api.php`. It returns the following JSON:
@@ -92,4 +106,4 @@ You can view [real-time stats](http://pi-hole.net/faq/install-the-real-time-lcd-
 - There is also an [FAQ section on pi-hole.net](http://pi-hole.net)
 
 ## Other Operating Systems
-This script will work for other UNIX-like systems with some slight **modifications**.  As long as you can install `dnsmasq` and a Webserver, it should work OK.  The automated install only works for a clean install of Raspiban right now since that is how the project originated.
+This script will work for other UNIX-like systems with some slight **modifications**.  As long as you can install `dnsmasq` and a Webserver, it should work OK.  The automated install is only for a clean install of a Debian based system, such as the Raspberry Pi.

adlists.default

@@ -25,6 +25,7 @@ https://s3.amazonaws.com/lists.disconnect.me/simple_ad.txt
 http://hosts-file.net/ad_servers.txt
 
 # Mahakala list. Has been known to block legitimage domains including the entire .com range.
+# Warning: Due to the sheer size of this list, the web admin console will be unresponsive.
 #http://adblock.mahakala.is/
 
 # ADZHOSTS list. Has been known to block legitimate domains

advanced/Scripts/piholeDebug.sh

@@ -0,0 +1,292 @@
+#!/usr/bin/env bash
+# Pi-hole: A black hole for Internet advertisements
+# (c) 2015, 2016 by Jacob Salmela
+# Network-wide ad blocking via your Raspberry Pi
+# http://pi-hole.net
+# Generates pihole_debug.log in /var/log/ to be used for troubleshooting.
+#
+# Pi-hole is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 2 of the License, or
+# (at your option) any later version.
+
+# Nate Brandeburg
+# nate@ubiquisoft.com
+# 3/24/2016
+
+######## GLOBAL VARS ########
+DEBUG_LOG="/var/log/pihole_debug.log"
+DNSMASQFILE="/etc/dnsmasq.conf"
+PIHOLECONFFILE="/etc/dnsmasq.d/01-pihole.conf"
+LIGHTTPDFILE="/etc/lighttpd/lighttpd.conf"
+GRAVITYFILE="/etc/pihole/gravity.list"
+HOSTSFILE="/etc/hosts"
+WHITELISTFILE="/etc/pihole/whitelist.txt"
+BLACKLISTFILE="/etc/pihole/blacklist.txt"
+ADLISTSFILE="/etc/pihole/adlists.list"
+PIHOLELOG="/var/log/pihole.log"
+WHITELISTMATCHES="/tmp/whitelistmatches.list"
+
+
+######## FIRST CHECK ########
+# Must be root to debug
+if [[ $EUID -eq 0 ]]; then
+	echo "::: You are root... Beginning debug!"
+else
+	echo "::: Sudo will be used for debugging."
+	# Check if sudo is actually installed
+	if [[ $(dpkg-query -s sudo) ]]; then
+		export SUDO="sudo"
+	else
+		echo "::: Please install sudo or run this as root."
+		exit 1
+	fi
+fi
+
+# Ensure the file exists, create if not, clear if exists.
+if [ ! -f "$DEBUG_LOG" ]; then
+	$SUDO touch $DEBUG_LOG
+	$SUDO chmod 644 $DEBUG_LOG
+	$SUDO chown "$USER":root $DEBUG_LOG
+else 
+	truncate -s 0 $DEBUG_LOG
+fi
+
+### Private functions exist here ###
+function compareWhitelist {
+	if [ ! -f "$WHITELISTMATCHES" ]; then
+		$SUDO touch $WHITELISTMATCHES
+		$SUDO chmod 644 $WHITELISTMATCHES
+		$SUDO chown "$USER":root $WHITELISTMATCHES
+	else
+		truncate -s 0 $WHITELISTMATCHES
+	fi
+
+	echo "#######################################" >> $DEBUG_LOG
+	echo "######## Whitelist Comparison #########" >> $DEBUG_LOG
+	echo "#######################################" >> $DEBUG_LOG
+	while read -r line; do
+		TMP=$(grep -w ".* $line$" "$GRAVITYFILE")
+		if [ ! -z "$TMP" ]; then
+			echo "$TMP" >> $DEBUG_LOG
+			echo "$TMP"	>> $WHITELISTMATCHES
+		fi
+	done < "$WHITELISTFILE"
+	echo >> $DEBUG_LOG
+}
+
+function compareBlacklist {
+	echo "#######################################" >> $DEBUG_LOG
+	echo "######## Blacklist Comparison #########" >> $DEBUG_LOG
+	echo "#######################################" >> $DEBUG_LOG
+	while read -r line; do
+		if [ ! -z "$line" ]; then
+			grep -w ".* $line$" "$GRAVITYFILE" >> $DEBUG_LOG
+		fi
+	done < "$BLACKLISTFILE"
+	echo >> $DEBUG_LOG
+}
+
+function testNslookup {
+	TESTURL="doubleclick.com"
+	echo "#######################################" >> $DEBUG_LOG
+	echo "############ NSLookup Test ############" >> $DEBUG_LOG
+	echo "#######################################" >> $DEBUG_LOG
+	# Find a blocked url that has not been whitelisted.
+	if [ -s "$WHITELISTMATCHES" ]; then
+		while read -r line; do
+			CUTURL=${line#*" "}
+			if [ "$CUTURL" != "Pi-Hole.IsWorking.OK" ]; then
+				while read -r line2; do
+					CUTURL2=${line2#*" "}
+					if [ "$CUTURL" != "$CUTURL2" ]; then
+						TESTURL="$CUTURL"
+						break 2
+					fi
+				done < "$WHITELISTMATCHES"
+			fi
+		done < "$GRAVITYFILE"
+	fi
+
+	echo "NSLOOKUP of $TESTURL from PiHole:" >> $DEBUG_LOG
+	nslookup "$TESTURL" >> $DEBUG_LOG
+	echo >> $DEBUG_LOG
+	echo "NSLOOKUP of $TESTURL from 8.8.8.8:" >> $DEBUG_LOG
+	nslookup "$TESTURL" 8.8.8.8 >> $DEBUG_LOG
+	echo >> $DEBUG_LOG
+}
+
+### Check Pi internet connections ###
+# Log the IP addresses of this Pi
+IPADDR=$(ifconfig | perl -nle 's/dr:(\S+)/print $1/e')
+echo "::: Writing local IPs to debug log"
+echo "IP Addresses of this Pi:" >> $DEBUG_LOG
+echo "$IPADDR" >> $DEBUG_LOG
+echo >> $DEBUG_LOG
+
+# Check if we can connect to the local gateway
+GATEWAY_CHECK=$(ping -q -w 1 -c 1 "$(ip r | grep default | cut -d ' ' -f 3)" > /dev/null && echo ok || echo error)
+echo "Gateway check:" >> $DEBUG_LOG
+echo "$GATEWAY_CHECK" >> $DEBUG_LOG
+echo >> $DEBUG_LOG
+
+# Test the nslookup here
+compareWhitelist
+compareBlacklist
+testNslookup
+
+echo "::: Writing dnsmasq.conf to debug log..."
+echo "#######################################" >> $DEBUG_LOG
+echo "############### Dnsmasq ###############" >> $DEBUG_LOG
+echo "#######################################" >> $DEBUG_LOG
+if [ -e "$DNSMASQFILE" ]
+then
+	#cat $DNSMASQFILE >> $DEBUG_LOG
+	while read -r line; do
+		if [ ! -z "$line" ]; then
+			[[ "$line" =~ ^#.*$ ]] && continue
+			echo "$line" >> $DEBUG_LOG
+        fi
+	done < "$DNSMASQFILE"
+	echo >> $DEBUG_LOG
+else
+	echo "No dnsmasq.conf file found!" >> $DEBUG_LOG
+	printf ":::\tNo dnsmasq.conf file found!\n"
+fi
+
+echo "::: Writing 01-pihole.conf to debug log..."
+echo "#######################################" >> $DEBUG_LOG
+echo "########### 01-pihole.conf ############" >> $DEBUG_LOG
+echo "#######################################" >> $DEBUG_LOG
+if [ -e "$PIHOLECONFFILE" ]
+then
+	#cat "$PIHOLECONFFILE" >> $DEBUG_LOG
+	while read -r line; do
+		if [ ! -z "$line" ]; then
+			[[ "$line" =~ ^#.*$ ]] && continue
+			echo "$line" >> $DEBUG_LOG
+        fi
+	done < "$PIHOLECONFFILE"
+	echo >> $DEBUG_LOG
+else
+	echo "No 01-pihole.conf file found!" >> $DEBUG_LOG
+	printf ":::\tNo 01-pihole.conf file found\n"
+fi
+
+echo "::: Writing lighttpd.conf to debug log..."
+echo "#######################################" >> $DEBUG_LOG
+echo "############ lighttpd.conf ############" >> $DEBUG_LOG
+echo "#######################################" >> $DEBUG_LOG
+if [ -e "$LIGHTTPDFILE" ]
+then
+	#cat "$PIHOLECONFFILE" >> $DEBUG_LOG
+	while read -r line; do
+		if [ ! -z "$line" ]; then
+			[[ "$line" =~ ^#.*$ ]] && continue
+			echo "$line" >> $DEBUG_LOG
+        fi
+	done < "$LIGHTTPDFILE"
+	echo >> $DEBUG_LOG
+else
+	echo "No lighttpd.conf file found!" >> $DEBUG_LOG
+	printf ":::\tNo lighttpd.conf file found\n"
+fi
+
+echo "::: Writing size of gravity.list to debug log..."
+echo "#######################################" >> $DEBUG_LOG
+echo "############ gravity.list #############" >> $DEBUG_LOG
+echo "#######################################" >> $DEBUG_LOG
+if [ -e "$GRAVITYFILE" ]
+then
+	wc -l "$GRAVITYFILE" >> $DEBUG_LOG
+	echo >> $DEBUG_LOG
+else
+	echo "No gravity.list file found!" >> $DEBUG_LOG
+	printf ":::\tNo gravity.list file found\n"
+fi
+
+# Write the hostname output to compare against entries in /etc/hosts, which is logged next
+echo "Hostname of this pihole is: " >> $DEBUG_LOG
+hostname >> $DEBUG_LOG
+
+echo "::: Writing hosts file to debug log..."
+echo "#######################################" >> $DEBUG_LOG
+echo "################ Hosts ################" >> $DEBUG_LOG
+echo "#######################################" >> $DEBUG_LOG
+if [ -e "$HOSTSFILE" ]
+then
+	cat "$HOSTSFILE" >> $DEBUG_LOG
+	echo >> $DEBUG_LOG
+else
+	echo "No hosts file found!" >> $DEBUG_LOG
+	printf ":::\tNo hosts file found!\n"
+fi
+
+### PiHole application specific logging ###
+echo "::: Writing whitelist to debug log..."
+echo "#######################################" >> $DEBUG_LOG
+echo "############## Whitelist ##############" >> $DEBUG_LOG
+echo "#######################################" >> $DEBUG_LOG
+if [ -e "$WHITELISTFILE" ]
+then
+	cat "$WHITELISTFILE" >> $DEBUG_LOG
+	echo >> $DEBUG_LOG
+else
+	echo "No whitelist.txt file found!" >> $DEBUG_LOG
+	printf ":::\tNo whitelist.txt file found!\n"
+fi
+
+echo "::: Writing blacklist to debug log..."
+echo "#######################################" >> $DEBUG_LOG
+echo "############## Blacklist ##############" >> $DEBUG_LOG
+echo "#######################################" >> $DEBUG_LOG
+if [ -e "$BLACKLISTFILE" ]
+then
+	cat "$BLACKLISTFILE" >> $DEBUG_LOG
+	echo >> $DEBUG_LOG
+else
+	echo "No blacklist.txt file found!" >> $DEBUG_LOG
+	printf ":::\tNo blacklist.txt file found!\n"
+fi
+
+echo "::: Writing adlists.list to debug log..."
+echo "#######################################" >> $DEBUG_LOG
+echo "############ adlists.list #############" >> $DEBUG_LOG
+echo "#######################################" >> $DEBUG_LOG
+if [ -e "$ADLISTSFILE" ]
+then
+	cat "$ADLISTSFILE" >> $DEBUG_LOG
+	echo >> $DEBUG_LOG
+else
+	echo "No adlists.list file found!" >> $DEBUG_LOG
+	printf ":::\tNo adlists.list file found!\n"
+fi
+
+
+# Continuously append the pihole.log file to the pihole_debug.log file
+function dumpPiHoleLog {
+	trap '{ echo -e "\nFinishing debug write from interrupt... Quitting!" ; exit 1; }' INT
+	echo -e "::: Writing current pihole traffic to debug log...\n:::\tTry loading any/all sites that you are having trouble with now... \n:::\t(Press ctrl+C to finish)"
+	echo "#######################################" >> $DEBUG_LOG
+	echo "############# pihole.log ##############" >> $DEBUG_LOG
+	echo "#######################################" >> $DEBUG_LOG
+	if [ -e "$PIHOLELOG" ]
+	then
+		while true; do
+			tail -f "$PIHOLELOG" >> $DEBUG_LOG
+			echo >> $DEBUG_LOG
+		done
+	else
+		echo "No pihole.log file found!" >> $DEBUG_LOG
+		printf ":::\tNo pihole.log file found!\n"
+	fi
+}
+
+# Anything to be done after capturing of pihole.log terminates
+function finalWork {
+	echo "::: Finshed debugging!"
+}
+trap finalWork EXIT
+
+### Method calls for additional logging ###
+dumpPiHoleLog

automated install/basic-install.sh

@@ -477,7 +477,8 @@ installScripts() {
 	$SUDO cp /etc/.pihole/advanced/Scripts/blacklist.sh /usr/local/bin/blacklist.sh
 	$SUDO cp /etc/.pihole/advanced/Scripts/piholeLogFlush.sh /usr/local/bin/piholeLogFlush.sh
 	$SUDO cp /etc/.pihole/advanced/Scripts/updateDashboard.sh /usr/local/bin/updateDashboard.sh
-	$SUDO chmod 755 /usr/local/bin/{gravity,chronometer,whitelist,blacklist,piholeLogFlush,updateDashboard}.sh
+	$SUDO cp /etc/.pihole/advanced/Scripts/piholeDebug.sh /usr/local/bin/piholeDebug.sh
+	$SUDO chmod 755 /usr/local/bin/{gravity,chronometer,whitelist,blacklist,piholeLogFlush,updateDashboard,piholeDebug}.sh
 	$SUDO echo " done."
 }