Recursive gobuster script

I was working on some automated tasks to include in my workflow and realized I wanted to use gobuster for launching dictionary-based enumeration on targets. I was currently using dirb for this but gobuster seems to be the faster tool to use. I fell behind on my scanning efforts during my last engagement and am trying to squeeze more juice from the time I’m being given.

Run it with something like.

./gobuster_recurse.sh http://www.target.com /path/to/wordlist.txt 5

You can also find this in my Gists at github at https://gist.github.com/ryan-wendel/b2c0545b3b76e86ff1afac5e1849dafe

The entire script…

#!/bin/bash

TARGET="$1"
WORDLIST="$2"
LEVELS="$3"

TMP_FILE_PREFIX="/tmp/gobuster_$$"
USER_AGENT='Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)'
BACKUP_WORDLIST="/usr/local/wordlists/custom/rw-common-dirs.txt"
RESPONSE_CODES="200,301,307,401,403"
THREADS="10"

print_help() {
        echo "Usage: $(basename $0) <url> <wordlist> <levels>"
}

if [ -z "$TARGET" ]; then
    echo "Error: Provide me with a URL"
    echo
    print_help
    exit 1
fi

if [ -z "$WORDLIST" ]; then
    echo "Error: You did not provide me with a wordlist."
    echo
    WORDLIST="${BACKUP_WORDLIST}"
    echo "Using ${WORDLIST}, instead."
    #print_help
    exit 2
fi

if [ ! -e "$WORDLIST" ]; then
    echo "Error: Wordlist file doesn't exist."
    echo
    print_help
    #exit 3
fi

if [ -z "$LEVELS" ]; then
    echo "Error: Provide me with a number of levels to recurse"
    echo
    print_help
    exit 4
elif [[ ! "$LEVELS" =~ ^[0-9]+$ ]]; then
    echo "Error: Provide me with an integer"
    echo
    print_help
    exit 5
fi

run_gobuster() {
    local TARGET=$1
    local LEVEL=$2
    local NEXT_LEVEL=$((LEVEL + 1))

    #echo "[-] Level = $LEVEL"
    #echo "[+] Busting $TARGET"

    if [ "${LEVEL}" -lt "${LEVELS}" ]; then
        #echo gobuster -f -q -e -k -r -t ${THREADS} -m dir -w "${WORDLIST}" -s "${RESPONSE_CODES}" -u ${TARGET} -a "${USER_AGENT}" 
        gobuster -f -q -e -k -r -t ${THREADS} -m dir -w "${WORDLIST}" -s "${RESPONSE_CODES}" -u ${TARGET} -a "${USER_AGENT}" | grep 'http.*Status: [234]' | sed 's/ (Status.*//' | while read HIT; do
            echo "[+] Found $HIT"
            run_gobuster ${HIT} ${NEXT_LEVEL}
        done
    fi
}

STATUS=$(curl -k -o /dev/null --silent --head --write-out '%{http_code}\n' "$TARGET")

if [ "$STATUS" -ge "100" -a "$STATUS" -lt "500" ]; then
    echo "[+] Found $TARGET"
    run_gobuster $TARGET 0
fi

The plan is to use the output of this script and feed it into Chris Truncer’s EyeWitness. This will help me quickly get a feel for the web application surface-area I am working with while on engagements.

Something like:

./gobuster_recurse.sh http://192.168.0.115 3 | grep Found | sed 's/.*http/http/' > /tmp/web_enum.tmp
python /usr/share/eyewitness/EyeWitness.py --no-prompt -f /tmp/web_enum.tmp --timeout 30 --threads 4 --web -d /tmp/project_foo/web/eyewitness/192.168.0.115

You can loop over your URL list and spit out that same command for each URL. Nice and easy way to perform a quick visual scan of a webapp.