Merge branch 'development' into new/RemoveDefaultBlocklists

README.md

@@ -3,7 +3,7 @@
 <b>Network-wide ad blocking via your own Linux hardware</b><br/>
 </p>
 
-The Pi-hole is a [DNS sinkhole](https://en.wikipedia.org/wiki/DNS_Sinkhole) that protects your devices from unwanted content, without installing any client-side software.
+The Pi-hole[®](https://pi-hole.net/trademark-rules-and-brand-guidelines/) is a [DNS sinkhole](https://en.wikipedia.org/wiki/DNS_Sinkhole) that protects your devices from unwanted content, without installing any client-side software.
 
 - **Easy-to-install**: our versatile installer walks you through the process, and [takes less than ten minutes](https://www.youtube.com/watch?v=vKWjx1AQYgs)
 - **Resolute**: content is blocked in _non-browser locations_, such as ad-laden mobile apps and smart TVs
@@ -27,7 +27,7 @@ Those who want to get started quickly and conveniently, may install Pi-hole usin
 #### `curl -sSL https://install.pi-hole.net | bash`
 
 ## Alternative Install Methods
-[Piping to `bash` is controversial](https://pi-hole.net/2016/07/25/curling-and-piping-to-bash), as it prevents you from [reading code that is about to run](https://github.com/pi-hole/pi-hole/blob/master/automated%20install/basic-install.sh) on your system. Therefore, we provide these alternative installation methods which allow code review before installation: 
+[Piping to `bash` is controversial](https://pi-hole.net/2016/07/25/curling-and-piping-to-bash), as it prevents you from [reading code that is about to run](https://github.com/pi-hole/pi-hole/blob/master/automated%20install/basic-install.sh) on your system. Therefore, we provide these alternative installation methods which allow code review before installation:
 
 ### Method 1: Clone our repository and run
 ```
@@ -60,16 +60,22 @@ Make no mistake: **your support is absolutely vital to help keep us innovating!*
 ### Donations
 Sending a donation using our links below is **extremely helpful** in offsetting a portion of our monthly expenses:
 
-&nbsp;<img src="https://pi-hole.github.io/graphics/Badges/paypal-badge-black.svg" width="24" height="24" alt="PP"/> <a href="https://www.paypal.com/cgi-bin/webscr?cmd=_s-xclick&hosted_button_id=3J2L3Z4DHW9UY">Donate via PayPal</a><br/>
+- <img src="https://pi-hole.github.io/graphics/Badges/paypal-badge-black.svg" width="24" height="24" alt="PP"/> <a href="https://www.paypal.com/cgi-bin/webscr?cmd=_s-xclick&hosted_button_id=3J2L3Z4DHW9UY">Donate via PayPal</a><br/>
-&nbsp;<img src="https://pi-hole.github.io/graphics/Badges/bitcoin-badge-black.svg" width="24" height="24" alt="BTC"/> Bitcoin Address: <code>1GKnevUnVaQM2pQieMyeHkpr8DXfkpfAtL</code>
+- <img src="https://pi-hole.github.io/graphics/Badges/bitcoin-badge-black.svg" width="24" height="24" alt="BTC"/> [Bitcoin](https://commerce.coinbase.com/checkout/fb7facaf-bebd-46be-bb77-b358f4546763): <code>
+3MDPzjXu2hjw5sGLJvKUi1uXbvQPzVrbpF</code></br>
+- <img src="https://pi-hole.github.io/graphics/Badges/bitcoin-badge-black.svg" width="24" height="24" alt="BTC"/> [Bitcoin Cash](https://commerce.coinbase.com/checkout/fb7facaf-bebd-46be-bb77-b358f4546763): <code>qzqsz4aju2eecc6uhs7tus4vlwhhela24sdruf4qp5</code></br>
+- <img src="https://pi-hole.github.io/graphics/Badges/ethereum-badge-black.svg" width="24" height="24" alt="BTC"/> [Ethereum](https://commerce.coinbase.com/checkout/fb7facaf-bebd-46be-bb77-b358f4546763): <code>0x79d4e90A4a0C732819526c93e21A3F1356A2FAe1</code>
 
 ### Alternative support
-If you'd rather not donate (_which is okay!_), there are other ways you can help support us:
+If you'd rather not [donate](https://pi-hole.net/donate/) (_which is okay!_), there are other ways you can help support us:
 
-- [Digital Ocean](http://www.digitalocean.com/?refcode=344d234950e1) affiliate link
+- [Digital Ocean](http://www.digitalocean.com/?refcode=344d234950e1) _affiliate link_
-- [Vultr](http://www.vultr.com/?ref=7190426) affiliate link
+- [UNIXstickers.com](http://unixstickers.refr.cc/jacobs) _save $5 when you spend $9 using our affiliate link_
-- [UNIXstickers.com](http://unixstickers.refr.cc/jacobs) affiliate link
+- [Pi-hole Swag Store](https://pi-hole.net/shop/) _affiliate link_
-- [Pi-hole Swag Store](https://pi-hole.net/shop/)
+- [Amazon](http://www.amazon.com/exec/obidos/redirect-home/pihole09-20) _affiliate link_
+- [Ho-ost](https://clients.ho-ost.com/aff.php?aff=19) _save 50% with our affiliate link_
+- [DNS Made Easy](https://cp.dnsmadeeasy.com/u/133706) _affiliate link_
+- [Vultr](http://www.vultr.com/?ref=7190426) _affiliate link_
 - Spreading the word about our software, and how you have benefited from it
 
 ### Contributing via GitHub
@@ -127,7 +133,7 @@ You can read our [Core Feature Breakdown](https://github.com/pi-hole/pi-hole/wik
 ### The Web Interface Dashboard
 This [optional dashboard](https://github.com/pi-hole/AdminLTE) allows you to view stats, change settings, and configure your Pi-hole. It's the power of the Command Line Interface, with none of the learning curve!
 
-<a href="https://pi-hole.github.io/graphics/Screenshots/dashboard.png"><img src="https://pi-hole.github.io/graphics/Screenshots/dashboard.png" width="888" height="522" alt="Pi-hole Dashboard"/></a>
+<img src="https://pi-hole.github.io/graphics/Screenshots/pihole-dashboard.png"  alt="Pi-hole Dashboard"/></a>
 
 Some notable features include:
 * Mobile friendly interface
@@ -142,11 +148,11 @@ Some notable features include:
 There are several ways to [access the dashboard](https://discourse.pi-hole.net/t/how-do-i-access-pi-holes-dashboard-admin-interface/3168):
 
 1. `http://<IP_ADDPRESS_OF_YOUR_PI_HOLE>/admin/`
-2. `http:/pi.hole/admin/` (when using Pi-hole as your DNS server)
+2. `http://pi.hole/admin/` (when using Pi-hole as your DNS server)
 3. `http://pi.hole/` (when using Pi-hole as your DNS server)
 
-## The Faster-Than-Light Engine
+## Faster-than-light Engine
-The [FTL Engine](https://github.com/pi-hole/FTL) is a lightweight, purpose-built daemon used to provide statistics needed for the Web Interface, and its API can be easily integrated into your own projects. As the name implies, FTL does this all *very quickly*!
+FTLDNS[™](https://pi-hole.net/trademark-rules-and-brand-guidelines/) is a lightweight, purpose-built daemon used to provide statistics needed for the Web Interface, and its API can be easily integrated into your own projects. As the name implies, FTLDNS does this all *very quickly*!
 
 Some of the statistics you can integrate include:
 * Total number of domains being blocked
@@ -194,6 +200,7 @@ While quite outdated at this point, [this original blog post about Pi-hole](http
 - [Magic Mirror with DNS Filtering](https://zonksec.com/blog/magic-mirror-dns-filtering/#dnssoftware)
 - [Pi-hole Droid: Android client](https://github.com/friimaind/pi-hole-droid)
 - [Windows DNS Swapper](https://github.com/roots84/DNS-Swapper), see [#1400](https://github.com/pi-hole/pi-hole/issues/1400)
+- [Pi-hole Visualizer](https://www.reddit.com/r/pihole/comments/82ikgb/pihole_visualizer_update/)
 -----
 
 ## Coverage

advanced/Scripts/piholeCheckout.sh

@@ -133,7 +133,7 @@ checkout() {
   Please re-run install script from https://github.com/pi-hole/pi-hole${COL_NC}"
     exit 1;
   fi
-  if [[ "${INSTALL_WEB}" == "true" ]]; then
+  if [[ "${INSTALL_WEB_INTERFACE}" == "true" ]]; then
     if ! is_repo "${webInterfaceDir}" ; then
      echo -e "  ${COL_LIGHT_RED}Error: Web Admin repo is missing from system!
   Please re-run install script from https://github.com/pi-hole/pi-hole${COL_NC}"
@@ -157,7 +157,7 @@ checkout() {
     echo ""
     echo -e "  ${INFO} Pi-hole Core"
     fetch_checkout_pull_branch "${PI_HOLE_FILES_DIR}" "development" || { echo "  ${CROSS} Unable to pull Core developement branch"; exit 1; }
-    if [[ "${INSTALL_WEB}" == "true" ]]; then
+    if [[ "${INSTALL_WEB_INTERFACE}" == "true" ]]; then
       echo ""
       echo -e "  ${INFO} Web interface"
       fetch_checkout_pull_branch "${webInterfaceDir}" "devel" || { echo "  ${CROSS} Unable to pull Web development branch"; exit 1; }
@@ -173,7 +173,7 @@ checkout() {
     echo -e "  ${INFO} Shortcut \"master\" detected - checking out master branches..."
     echo -e "  ${INFO} Pi-hole core"
     fetch_checkout_pull_branch "${PI_HOLE_FILES_DIR}" "master" || { echo "  ${CROSS} Unable to pull Core master branch"; exit 1; }
-    if [[ ${INSTALL_WEB} == "true" ]]; then
+    if [[ ${INSTALL_WEB_INTERFACE} == "true" ]]; then
       echo -e "  ${INFO} Web interface"
       fetch_checkout_pull_branch "${webInterfaceDir}" "master" || { echo "  ${CROSS} Unable to pull Web master branch"; exit 1; }
     fi
@@ -209,7 +209,7 @@ checkout() {
       exit 1
     fi
     checkout_pull_branch "${PI_HOLE_FILES_DIR}" "${2}"
-  elif [[ "${1}" == "web" ]] && [[ "${INSTALL_WEB}" == "true" ]] ; then
+  elif [[ "${1}" == "web" ]] && [[ "${INSTALL_WEB_INTERFACE}" == "true" ]] ; then
     str="Fetching branches from ${webInterfaceGitUrl}"
     echo -ne "  ${INFO} $str"
     if ! fully_fetch_repo "${webInterfaceDir}" ; then

advanced/Scripts/update.sh

@@ -113,7 +113,7 @@ main() {
     echo -e "  ${INFO} Pi-hole Core:\\t${COL_LIGHT_GREEN}up to date${COL_NC}"
   fi
 
-  if [[ "${INSTALL_WEB}" == true ]]; then
+  if [[ "${INSTALL_WEB_INTERFACE}" == true ]]; then
     if ! is_repo "${ADMIN_INTERFACE_DIR}" ; then
       echo -e "\\n  ${COL_LIGHT_RED}Error: Web Admin repo is missing from system!
   Please re-run install script from https://pi-hole.net${COL_NC}"

advanced/Scripts/webpage.sh

@@ -149,6 +149,10 @@ ProcessDNSSettings() {
 		let COUNTER=COUNTER+1
 	done
 
+  if [ ! -z "${LOCAL_DNS_PORT}" ]; then
+    add_dnsmasq_setting "server" "127.0.0.1#${LOCAL_DNS_PORT}"
+  fi
+
 	delete_dnsmasq_setting "domain-needed"
 
 	if [[ "${DNS_FQDN_REQUIRED}" == true ]]; then
@@ -525,6 +529,16 @@ SetPrivacyLevel() {
 		changeFTLsetting "PRIVACYLEVEL" "${args[2]}"
 	fi
 }
+SetLocalDNSport() {
+  # Ensure port is a natural number { 0, 1, 2, 3, ... }
+  if [[ "${1}" == "0" ]]; then
+    delete_setting "LOCAL_DNS_PORT"
+    ProcessDNSSettings
+  elif [[ "${1}" =~ ^[0-9]+$ ]]; then
+    change_setting "LOCAL_DNS_PORT" "${1}"
+    ProcessDNSSettings
+  fi
+}
 
 main() {
 	args=("$@")
@@ -556,6 +570,7 @@ main() {
 		"adlist"              ) CustomizeAdLists;;
 		"audit"               ) audit;;
 		"-l" | "privacylevel" ) SetPrivacyLevel;;
+		"localdnsport"        ) SetLocalDNSport "$3";;
 		*                     ) helpFunc;;
 	esac
 

automated install/basic-install.sh

@@ -56,7 +56,7 @@ IPV4_ADDRESS=""
 IPV6_ADDRESS=""
 # By default, query logging is enabled and the dashboard is set to be installed
 QUERY_LOGGING=true
-INSTALL_WEB=true
+INSTALL_WEB_INTERFACE=true
 
 
 # Find the rows and columns will default to 80x24 if it can not be detected
@@ -77,6 +77,16 @@ c=$(( c < 70 ? 70 : c ))
 skipSpaceCheck=false
 reconfigure=false
 runUnattended=false
+INSTALL_WEB_SERVER=true
+# Check arguments for the undocumented flags
+for var in "$@"; do
+  case "$var" in
+  "--reconfigure" ) reconfigure=true;;
+  "--i_do_not_follow_recommendations" ) skipSpaceCheck=true;;
+  "--unattended" ) runUnattended=true;;
+  "--disable-install-webserver" ) INSTALL_WEB_SERVER=false;;
+  esac
+done
 
 # If the color table file exists,
 if [[ -f "${coltable}" ]]; then
@@ -164,7 +174,7 @@ if command -v apt-get &> /dev/null; then
   # These programs are stored in an array so they can be looped through later
   INSTALLER_DEPS=(apt-utils dialog debconf dhcpcd5 git ${iproute_pkg} whiptail)
   # Pi-hole itself has several dependencies that also need to be installed
-  PIHOLE_DEPS=(bc cron curl dnsutils iputils-ping lsof netcat sudo unzip wget idn2 sqlite3 libcap2-bin dns-root-data resolvconf)
+  PIHOLE_DEPS=(bc cron curl dnsutils iputils-ping lsof netcat psmisc sudo unzip wget idn2 sqlite3 libcap2-bin dns-root-data resolvconf)
   # The Web dashboard has some that also need to be installed
   # It's useful to separate the two since our repos are also setup as "Core" code and "Web" code
   PIHOLE_WEB_DEPS=(lighttpd ${phpVer}-common ${phpVer}-cgi ${phpVer}-${phpSqlite})
@@ -946,14 +956,36 @@ setAdminFlag() {
       "On (Recommended)")
         echo -e "  ${INFO} Web Interface On"
         # Set it to true
-        INSTALL_WEB=true
+        INSTALL_WEB_INTERFACE=true
         ;;
       Off)
         echo -e "  ${INFO} Web Interface Off"
         # or false
-        INSTALL_WEB=false
+        INSTALL_WEB_INTERFACE=false
         ;;
     esac
+
+  # Request user to install web server, if --disable-install-webserver has not been used (INSTALL_WEB_SERVER=true is default).
+  if [[ "${INSTALL_WEB_SERVER}" == true ]]; then
+    WebToggleCommand=(whiptail --separate-output --radiolist "Do you wish to install the web server (lighttpd)?\\n\\nNB: If you disable this, and, do not have an existing webserver installed, the web interface will not function." "${r}" "${c}" 6)
+    # with the default being enabled
+    WebChooseOptions=("On (Recommended)" "" on
+        Off "" off)
+    WebChoices=$("${WebToggleCommand[@]}" "${WebChooseOptions[@]}" 2>&1 >/dev/tty) || (echo -e "  ${COL_LIGHT_RED}Cancel was selected, exiting installer${COL_NC}" && exit 1)
+    # Depending on their choice
+    case ${WebChoices} in
+      "On (Recommended)")
+        echo -e "  ${INFO} Web Server On"
+        # set it to true, as clearly seen below.
+        INSTALL_WEB_SERVER=true
+        ;;
+      Off)
+        echo -e "  ${INFO} Web Server Off"
+        # or false
+        INSTALL_WEB_SERVER=false
+        ;;
+    esac
+  fi
 }
 
 # A function to display a list of example blocklists for users to select
@@ -1129,7 +1161,7 @@ installConfigs() {
   version_check_dnsmasq
 
   # If the user chose to install the dashboard,
-  if [[ "${INSTALL_WEB}" == true ]]; then
+  if [[ "${INSTALL_WEB_SERVER}" == true ]]; then
     # and if the Web server conf directory does not exist,
     if [[ ! -d "/etc/lighttpd" ]]; then
       # make it
@@ -1497,7 +1529,7 @@ configureFirewall() {
 #
 finalExports() {
   # If the Web interface is not set to be installed,
-  if [[ "${INSTALL_WEB}" == false ]]; then
+  if [[ "${INSTALL_WEB_INTERFACE}" == false ]]; then
     # and if there is not an IPv4 address,
     if [[ "${IPV4_ADDRESS}" ]]; then
       # there is no block page, so set IPv4 to 0.0.0.0 (all IP addresses)
@@ -1512,7 +1544,7 @@ finalExports() {
   # If the setup variable file exists,
   if [[ -e "${setupVars}" ]]; then
     # update the variables in the file
-    sed -i.update.bak '/PIHOLE_INTERFACE/d;/IPV4_ADDRESS/d;/IPV6_ADDRESS/d;/PIHOLE_DNS_1/d;/PIHOLE_DNS_2/d;/QUERY_LOGGING/d;/INSTALL_WEB/d;/LIGHTTPD_ENABLED/d;' "${setupVars}"
+    sed -i.update.bak '/PIHOLE_INTERFACE/d;/IPV4_ADDRESS/d;/IPV6_ADDRESS/d;/PIHOLE_DNS_1/d;/PIHOLE_DNS_2/d;/QUERY_LOGGING/d;/INSTALL_WEB_SERVER/d;/INSTALL_WEB_INTERFACE/d;/LIGHTTPD_ENABLED/d;' "${setupVars}"
   fi
   # echo the information to the user
     {
@@ -1522,7 +1554,8 @@ finalExports() {
   echo "PIHOLE_DNS_1=${PIHOLE_DNS_1}"
   echo "PIHOLE_DNS_2=${PIHOLE_DNS_2}"
   echo "QUERY_LOGGING=${QUERY_LOGGING}"
-  echo "INSTALL_WEB=${INSTALL_WEB}"
+  echo "INSTALL_WEB_SERVER=${INSTALL_WEB_SERVER}"
+  echo "INSTALL_WEB_INTERFACE=${INSTALL_WEB_INTERFACE}"
   echo "LIGHTTPD_ENABLED=${LIGHTTPD_ENABLED}"
     }>> "${setupVars}"
 
@@ -1566,32 +1599,35 @@ installPihole() {
   create_pihole_user
 
   # If the user wants to install the Web interface,
-  if [[ "${INSTALL_WEB}" == true ]]; then
+  if [[ "${INSTALL_WEB_INTERFACE}" == true ]]; then
     if [[ ! -d "/var/www/html" ]]; then
       # make the Web directory if necessary
       mkdir -p /var/www/html
     fi
-    # Set the owner and permissions
+
-    chown ${LIGHTTPD_USER}:${LIGHTTPD_GROUP} /var/www/html
+    if [[ "${INSTALL_WEB_SERVER}" == true ]]; then
-    chmod 775 /var/www/html
+      # Set the owner and permissions
-    # Give pihole access to the Web server group
+      chown ${LIGHTTPD_USER}:${LIGHTTPD_GROUP} /var/www/html
-    usermod -a -G ${LIGHTTPD_GROUP} pihole
+      chmod 775 /var/www/html
-    # If the lighttpd command is executable,
+      # Give pihole access to the Web server group
-    if [[ -x "$(command -v lighty-enable-mod)" ]]; then
+      usermod -a -G ${LIGHTTPD_GROUP} pihole
-      # enable fastcgi and fastcgi-php
+      # If the lighttpd command is executable,
-      lighty-enable-mod fastcgi fastcgi-php > /dev/null || true
+      if [[ -x "$(command -v lighty-enable-mod)" ]]; then
-    else
+        # enable fastcgi and fastcgi-php
-      # Othweise, show info about installing them
+        lighty-enable-mod fastcgi fastcgi-php > /dev/null || true
-      echo -e  "  ${INFO} Warning: 'lighty-enable-mod' utility not found
+      else
-      Please ensure fastcgi is enabled if you experience issues\\n"
+        # Othweise, show info about installing them
-    fi
+        echo -e  "  ${INFO} Warning: 'lighty-enable-mod' utility not found
+        Please ensure fastcgi is enabled if you experience issues\\n"
+      fi
+	fi
   fi
   # Install scripts,
   installScripts
   # configs,
   installConfigs
   # If the user wants to install the dashboard,
-  if [[ "${INSTALL_WEB}" == true ]]; then
+  if [[ "${INSTALL_WEB_INTERFACE}" == true ]]; then
     # do so
     installPiholeWeb
   fi
@@ -1618,6 +1654,15 @@ accountForRefactor() {
   sed -i 's/piholeIPv6/IPV6_ADDRESS/g' ${setupVars}
   sed -i 's/piholeDNS1/PIHOLE_DNS_1/g' ${setupVars}
   sed -i 's/piholeDNS2/PIHOLE_DNS_2/g' ${setupVars}
+  sed -i 's/^INSTALL_WEB=/INSTALL_WEB_INTERFACE=/' ${setupVars}
+  # Add 'INSTALL_WEB_SERVER', if its not been applied already: https://github.com/pi-hole/pi-hole/pull/2115
+  if ! grep -q '^INSTALL_WEB_SERVER=' ${setupVars}; then
+    local webserver_installed=false
+    if grep -q '^INSTALL_WEB_INTERFACE=true' ${setupVars}; then
+      webserver_installed=true
+    fi
+    echo -e "INSTALL_WEB_SERVER=$webserver_installed" >> ${setupVars}
+  fi
 }
 
 updatePihole() {
@@ -1625,9 +1670,9 @@ updatePihole() {
   # Install base files and web interface
   installScripts
   # Install config files
-  installConfigs  
+  installConfigs
   # If the user wants to install the dasboard,
-  if [[ "${INSTALL_WEB}" == true ]]; then
+  if [[ "${INSTALL_WEB_INTERFACE}" == true ]]; then
     # do so
     installPiholeWeb
   fi
@@ -1677,7 +1722,7 @@ displayFinalMessage() {
     pwstring="NOT SET"
   fi
    # If the user wants to install the dashboard,
-   if [[ "${INSTALL_WEB}" == true ]]; then
+   if [[ "${INSTALL_WEB_INTERFACE}" == true ]]; then
        # Store a message in a variable and display it
        additional="View the web interface at http://pi.hole/admin or http://${IPV4_ADDRESS%/*}/admin
 
@@ -1745,7 +1790,7 @@ clone_or_update_repos() {
         exit 1; \
       }
     # If the Web interface was installed,
-    if [[ "${INSTALL_WEB}" == true ]]; then
+    if [[ "${INSTALL_WEB_INTERFACE}" == true ]]; then
       # reset it's repo
       resetRepo ${webInterfaceDir} || \
         { echo -e "  ${COL_LIGHT_RED}Unable to reset ${webInterfaceDir}, exiting installer${COL_NC}"; \
@@ -1760,7 +1805,7 @@ clone_or_update_repos() {
         exit 1; \
       }
       # If the Web interface was installed,
-      if [[ "${INSTALL_WEB}" == true ]]; then
+      if [[ "${INSTALL_WEB_INTERFACE}" == true ]]; then
         # get the Web git files
         getGitFiles ${webInterfaceDir} ${webInterfaceGitUrl} || \
         { echo -e "  ${COL_LIGHT_RED}Unable to clone ${webInterfaceGitUrl} into ${webInterfaceDir}, exiting installer${COL_NC}"; \
@@ -2065,15 +2110,6 @@ main() {
   # Check for supported distribution
   distro_check
 
-  # Check arguments for the undocumented flags
-  for var in "$@"; do
-    case "$var" in
-      "--reconfigure" ) reconfigure=true;;
-      "--i_do_not_follow_recommendations" ) skipSpaceCheck=true;;
-      "--unattended" ) runUnattended=true;;
-    esac
-  done
-
   # If the setup variable file exists,
   if [[ -f "${setupVars}" ]]; then
     # if it's running unattended,
@@ -2127,7 +2163,7 @@ main() {
       fi
     fi
 
-    if [[ "${INSTALL_WEB}" == true ]]; then
+    if [[ "${INSTALL_WEB_SERVER}" == true ]]; then
       stop_service lighttpd
     fi
     # Determine available interfaces
@@ -2147,21 +2183,19 @@ main() {
     # Clone/Update the repos
     clone_or_update_repos
 
-    # Install packages used by the Pi-hole
+    # Install the Core dependencies
-    if [[ "${INSTALL_WEB}" == true ]]; then
+    local dep_install_list=("${PIHOLE_DEPS[@]}")
+    if [[ "${INSTALL_WEB_SERVER}" == true ]]; then
       # Install the Web dependencies
-      DEPS=("${PIHOLE_DEPS[@]}" "${PIHOLE_WEB_DEPS[@]}")
+      dep_install_list+=("${PIHOLE_WEB_DEPS[@]}")
-    # Otherwise,
-    else
-      # just install the Core dependencies
-      DEPS=("${PIHOLE_DEPS[@]}")
     fi
 
-    install_dependent_packages DEPS[@]
+    install_dependent_packages dep_install_list[@]
+    unset dep_install_list
 
     # On some systems, lighttpd is not enabled on first install. We need to enable it here if the user
     # has chosen to install the web interface, else the `LIGHTTPD_ENABLED` check will fail
-    if [[ "${INSTALL_WEB}" == true ]]; then
+    if [[ "${INSTALL_WEB_SERVER}" == true ]]; then
       enable_service lighttpd
     fi
 
@@ -2182,16 +2216,15 @@ main() {
     # Clone/Update the repos
     clone_or_update_repos
 
-    # Install packages used by the Pi-hole
+    # Install the Core dependencies
-    if [[ "${INSTALL_WEB}" == true ]]; then
+    local dep_install_list=("${PIHOLE_DEPS[@]}")
+    if [[ "${INSTALL_WEB_SERVER}" == true ]]; then
       # Install the Web dependencies
-      DEPS=("${PIHOLE_DEPS[@]}" "${PIHOLE_WEB_DEPS[@]}")
+      dep_install_list+=("${PIHOLE_WEB_DEPS[@]}")
-    # Otherwise,
-    else
-      # just install the Core dependencies
-      DEPS=("${PIHOLE_DEPS[@]}")
     fi
-    install_dependent_packages DEPS[@]
+
+    install_dependent_packages dep_install_list[@]
+    unset dep_install_list
 
     if [[ -x "$(command -v systemctl)" ]]; then
       # Value will either be 1, if true, or 0
@@ -2206,7 +2239,7 @@ main() {
   # Copy the temp log file into final log location for storage
   copy_to_install_log
 
-  if [[ "${INSTALL_WEB}" == true ]]; then
+  if [[ "${INSTALL_WEB_INTERFACE}" == true ]]; then
     # Add password to web UI if there is none
     pw=""
     # If no password is set,
@@ -2228,7 +2261,7 @@ main() {
   fi
 
   # If the Web server was installed,
-  if [[ "${INSTALL_WEB}" == true ]]; then
+  if [[ "${INSTALL_WEB_SERVER}" == true ]]; then
 
     if [[ "${LIGHTTPD_ENABLED}" == "1" ]]; then
       start_service lighttpd
@@ -2255,7 +2288,7 @@ main() {
   fi
 
   # If the Web interface was installed,
-  if [[ "${INSTALL_WEB}" == true ]]; then
+  if [[ "${INSTALL_WEB_INTERFACE}" == true ]]; then
     # If there is a password,
     if (( ${#pw} > 0 )) ; then
       # display the password
@@ -2267,7 +2300,7 @@ main() {
   #
   if [[ "${useUpdateVars}" == false ]]; then
     # If the Web interface was installed,
-    if [[ "${INSTALL_WEB}" == true ]]; then
+    if [[ "${INSTALL_WEB_INTERFACE}" == true ]]; then
       echo -e "  View the web interface at http://pi.hole/admin or http://${IPV4_ADDRESS%/*}/admin"
       echo ""
     fi

automated install/uninstall.sh

@@ -46,13 +46,10 @@ source "${setupVars}"
 distro_check
 
 # Install packages used by the Pi-hole
-if [[ "${INSTALL_WEB}" == true ]]; then
+DEPS=("${INSTALLER_DEPS[@]}" "${PIHOLE_DEPS[@]}")
+if [[ "${INSTALL_WEB_SERVER}" == true ]]; then
   # Install the Web dependencies
-  DEPS=("${INSTALLER_DEPS[@]}" "${PIHOLE_DEPS[@]}" "${PIHOLE_WEB_DEPS[@]}")
+  DEPS+=("${PIHOLE_WEB_DEPS[@]}")
-# Otherwise,
-else
-  # just install the Core dependencies
-  DEPS=("${INSTALLER_DEPS[@]}" "${PIHOLE_DEPS[@]}")
 fi
 
 # Compatability

gravity.sh

@@ -343,13 +343,18 @@ gravity_ParseFileIntoDomains() {
     # Scanning for "^IPv4$" is too slow with large (1M) lists on low-end hardware
     echo -ne "  ${INFO} Format: URL"
 
-    awk '{
+    awk '
-      # Remove URL protocol, optional "username:password@", and ":?/;"
+      # Remove URL scheme, optional "username:password@", and ":?/;"
-      if ($0 ~ /[:?\/;]/) { gsub(/(^.*:\/\/(.*:.*@)?|[:?\/;].*)/, "", $0) }
+      # The scheme must be matched carefully to avoid blocking the wrong URL
-      # Remove lines which are only IPv4 addresses
+      # in cases like:
-      if ($0 ~ /^[0-9]+\.[0-9]+\.[0-9]+\.[0-9]+$/) { $0="" }
+      #   http://www.evil.com?http://www.good.com
-      if ($0) { print $0 }
+      # See RFC 3986 section 3.1 for details.
-    }' "${source}" 2> /dev/null > "${destination}"
+      /[:?\/;]/ { gsub(/(^[a-zA-Z][a-zA-Z0-9+.-]*:\/\/(.*:.*@)?|[:?\/;].*)/, "", $0) }
+      # Skip lines which are only IPv4 addresses
+      /^[0-9]+\.[0-9]+\.[0-9]+\.[0-9]+$/ { next }
+      # Print if nonempty
+      length { print }
+    ' "${source}" 2> /dev/null > "${destination}"
 
     echo -e "${OVER}  ${TICK} Format: URL"
   else
@@ -503,14 +508,12 @@ gravity_ParseBlacklistDomains() {
 
   # Empty $accretionDisc if it already exists, otherwise, create it
   : > "${piholeDir}/${accretionDisc}"
-  
+
   if [[ -f "${piholeDir}/${whitelistMatter}" ]]; then
-    gravity_ParseDomainsIntoHosts "${piholeDir}/${whitelistMatter}" "${piholeDir}/${accretionDisc}"
+    mv "${piholeDir}/${whitelistMatter}" "${piholeDir}/${accretionDisc}"
-    grep -c "^" "${piholeDir}/${whitelistMatter}" > "${piholeDir}/numBlocked" 2> /dev/null
   else
     # There was no whitelist file, so use preEventHorizon instead of whitelistMatter.
-    gravity_ParseDomainsIntoHosts "${piholeDir}/${preEventHorizon}" "${piholeDir}/${accretionDisc}"
+    mv "${piholeDir}/${preEventHorizon}" "${piholeDir}/${accretionDisc}"
-    grep -c "^" "${piholeDir}/${preEventHorizon}" > "${piholeDir}/numBlocked" 2> /dev/null
   fi
 
   # Move the file over as /etc/pihole/gravity.list so dnsmasq can use it
@@ -528,11 +531,10 @@ gravity_ParseUserDomains() {
   if [[ ! -f "${blacklistFile}" ]]; then
     return 0
   fi
-
+  
-  gravity_ParseDomainsIntoHosts "${blacklistFile}" "${blackList}.tmp"
   # Copy the file over as /etc/pihole/black.list so dnsmasq can use it
-  mv "${blackList}.tmp" "${blackList}" 2> /dev/null || \
+  cp "${blacklistFile}" "${blackList}" 2> /dev/null || \
-    echo -e "\\n  ${CROSS} Unable to move ${blackList##*/}.tmp to ${piholeDir}"
+    echo -e "\\n  ${CROSS} Unable to move ${blacklistFile##*/} to ${piholeDir}"
 }
 
 # Trap Ctrl-C