How To Find Hidden Web Directories Using Dirsearch

When a security analyst performing website penetration testing the initial step should be finding hidden directories of a vulnerable website.

These hidden web directories are essential because they can give useful information i.e. potential attack vectors that would not be visible on the public facing website.

One of the ways to achieve this is by attempting brute-forcing site structure that includes directories and files in websites and for that, you have to choose a powerful tool.

Although there are many tools available used to perform site brute-forcing includes Dirbuster or Dirb but these have their own limitation such as Dirbuster only offers GUI interface that is not feasible all the time and Dirb does not include multithreading feature.

The most popular choice among penetration testers for website brute-forcing is Dirsearch.

Dirsearch, written in python is a command-line website directory scanner. It has a lot of features making it the complete winner in terms of performance:

  1. It includes Multithreading, making it faster than any other site scanner tool
  2. It performs Recursive brute-forcing
  3. It has HTTP proxy support
  4. Dirsearch effectively detects invalid web pages
  5. It has User agent randomization and Batch processing
  6. Supports Request delaying

This tool can be run on any operating system (Windows, Linux, mac) making it more compatible and simpler, yet a powerful tool.

In this setup we will be using Kali Linux as an attacking machine and DVWA on Metasploitable 2 as the target.

Install Dirsearch

You need to install Dirsearch in your Kali Linux attacking machine. First update all the repositories from the command # apt-get update

After that install Dirsearch from Github by using the link: https://github.com/maurosoria/dirsearch

Type command:

# git clone https://github.com/maurosoria/dirsearch

to install Dirsearch in your Kali Linux attacking machine

After successfully installing the tool, change the root directory into Dirsearch.

# cd dirsearch/

to install Dirsearch in your Kali Linux attacking machine

Type ls to see the content under Dirsearch directoy.

to install Dirsearch in your Kali Linux attacking machine

You can see there is a configuration file and a python file named ‘dirsearch.py’ which we are going to execute.

Configuring the Tool

Next step will be configuring the Dirsearch. You can do it in three ways.

As this ‘dirsearch.py’ file needs python 3 to execute correctly, just simply run it with python.

# python3 dirsearch.py

configuring the Dirsearch

It gives a usage example, stating we need to specify a valid URL.

The second way to run Dirsearch is to run it with Bash. Type ls -la to give us the permissions of everything in this directory.

configuring the Dirsearch

As we can see from above that this tool is executable, now run it using the dot-slash.

# ./dirsearch.py

configuring the Dirsearch

The third and the easiest way to run Dirsearch is by creating a symbolic link in the /bin directory as this will permit us to run this tool not only with the directory cloned from GitHub but anywhere in the prompt.

Change into the bin directory and create a symbolic link naming it ‘dirsearch’ by using the ln -s command.

# ln -s ln -s ~/dirsearch/dirsearch.py dirsearch

configuring the Dirsearch

Site Scanning Using Dirsearch

As we have created the symbolic link, just type dirsearch in any directory to execute.

Site Scanning Using Dirsearch

Use the flag ‘-h’ with the dirsearch to get full help menu that will show a ton of options and potential configuration settings.

Site Scanning Using Dirsearch

Site Scanning Using Dirsearch

So now for searching the site it needs a valid target’s URL and a file extension to run. You can specify a valid URL with the -u flag, and a file extension to search for with the -e flag.

In our case we will give it the URL of our DVWA on Metasploitable2 machine.

# dirsearch -u http://192.168.0.102/dvwa -e php

Site Scanning Using Dirsearch

Site Scanning Using Dirsearch

This showing us a lot of information regarding  the extensions, HTTP methods in use, number of threads, and size of the current wordlist. After that it starts to dig into the directories and returns with its findings that includes the status code, size, and directory name.

If you want to exclude certain HTTP status codes use -x flag.

# dirsearch -u http://192.168.0.102/dvwa -e php -x 403

Site Scanning Using Dirsearch

Site Scanning Using Dirsearch

Here we are excluding all the 403 codes by specifying after -x flag. It leads us to get what we required, making it cleaner and simpler.

Also, you can use -w flag to use a wordlist of your choice.

# dirsearch -u http://192.168.0.102/dvwa -e php -x 403,301,302 -w /usr/share/wordlists/wfuzz/general/common.txt

Site Scanning Using Dirsearch

You can see it does not find a large pool of results with this mentioned wordlist because the size is smaller.

Now coming to its best feature i.e. performing recursive directory scanning. Use -r flag to run the recursive search.

# dirsearch -u http://192.168.0.102/dvwa -e php -x 403,301,302 -r

Site Scanning Using Dirsearch

Site Scanning Using Dirsearch

Now see there is an additional entry of ‘Recursive Level’. On completing the initial scan this will go back and starts scanning each directory it found recursively, in this case it starts config/ then docs/ and so on.

While scanning if you want to exit the scan parse ‘e’ from keyboard. To continue it from the point you have stopped, parse ‘c’. Parse ‘n’ to move to the next directory. These steps will let you have a control over the results as recursive scanning is a time-consuming process.

If you want to set the recursion level to a deeper value, you can use the -R flag and a value of how many levels deep you want to dig.

# dirsearch -u http://192.168.0.102/dvwa -e php -x 403,301,302 -r -R 3

Site Scanning Using Dirsearch

Site Scanning Using DirsearchSo, you can see this starts scanning the /administrator directory which is obviously could not be found on the top level.

Conclusion

This article demonstrates how to use an effective web directory brute-forcing tool Dirsearch to scan and search hidden web directories which may not be visible to a user. This is very a very first task in penetration tasting and Dirsearch do this job much faster than the traditional Dirbuster and Dirb.

Irfan Shakeel
Irfan Shakeel, the founder of ehacking project, he also hosts cyber security training classes at EH Academy. He has discovered many vulnerabilities in the famous platforms (like Google, Dailymotion, Harvard University & etc.). He specializes in Network hacking, VoIP pentesting & digital forensics. He is the author of the book title “Hacking from Scratch”.

Most Popular

Top 10 things to Do After Installing Kali Linux

Kali Linux is considered to be one of the best hacking distribution of this era, it is developed by Offensive Security to give an...

Become a spy in your own right with Xnspy Android spying app

Having become widely popular among parents and employers, spying apps have become quite the norm nowadays. Android spying apps have made it a lot...

e-Services Portals Potentially Expose Government Infrastructure to File-based Attacks

More and more users are embracing technology to perform their day-to-day activities. It’s not only private businesses that are forced to establish digital channels...

What is Nmap? How to use Nmap for Information Gathering

Nmap stands for Network Mapper, a powerful network scanning and host detection tool that is being used to perform reconnaissance in a very first...