DirBuster: Website Link Discovery

TL;DR

DirBuster can find many links and files on a static website, but it’s not perfect. It relies on wordlists and may miss content if the site uses unusual naming or doesn’t have an index file in discovered directories.

How DirBuster Works

DirBuster is a tool that sends requests to a web server, trying common filenames and directory names. If it finds something (like a webpage or image), it lists it. It’s like systematically guessing where things are on the website.

Steps to List Website Links & Information

1. Download and Install DirBuster: Get it from DirBuster’s official site. It runs on Java, so make sure you have that installed too.
2. Start DirBuster: Open the application after installation.
3. Enter the Target URL: In the ‘URL’ field, type the website address you want to scan. For example:

   http://www.example.com

4. Choose a Wordlist: This is crucial! DirBuster uses lists of common names to guess at links.
   • Click ‘Wordlists’.
   • Select a wordlist that suits the website you’re scanning. Common choices include:
     • directory-list-2.3-medium.txt: A good starting point for many sites.
     • common.txt: For very basic scans.
     • You can also download more specialised lists from the internet.
5. Configure Scan Options (Optional): Click ‘Options’ to adjust settings.
   • Recursion Depth: How many levels deep DirBuster will search. Higher numbers take longer but find more. Start with 2 or 3.
   • Threads: The number of requests sent at once. More threads are faster, but can overload the server (be polite!). 10-20 is usually safe.
6. Start the Scan: Click ‘Start’. DirBuster will begin sending requests and displaying results.
7. Review the Results: The main window shows a list of found files and directories.
   • Status Codes: Pay attention to these:
     • 200 OK: Found a webpage or file.
     • 301 Moved Permanently/302 Found: Redirects – follow the redirect to see where it goes.
     • 403 Forbidden/404 Not Found: Access denied or page doesn’t exist (but might still be interesting!).

Limitations & How to Improve Results

1. Wordlist Coverage: DirBuster only finds what’s in its wordlists. If the website uses unusual names, you’ll miss things.
   • Custom Wordlists: Create your own list of likely filenames and directories based on what you know about the site.
2. Index Files: DirBuster often looks for ‘index.html’, ‘index.php’ etc. If a directory doesn’t have one, it might not be listed.
   • Try adding ‘?dir=’ to the URL in your browser to see if there are hidden directories.
3. JavaScript-Generated Content: Static websites rarely use this but if they do, DirBuster won’t find links created by JavaScript.
   • Use a browser developer tool (F12) to inspect the website’s code and identify dynamically generated links.
4. Robots.txt: Check the robots.txt file on the website. It tells search engines which parts of the site *not* to scan – it might give you clues about interesting areas.

Alternative Tools

If DirBuster isn’t finding everything, consider these:

• Burp Suite: A more advanced web security testing tool.
• OWASP ZAP: Another powerful and free web scanner.