Unveiling Hidden URLs - Web Enumeration with Hakrawler

In the ever-evolving realm of cybersecurity, having a tool that can efficiently traverse the web for potential vulnerabilities is not just an asset – it's a necessity. Meet Hakrawler, a robust and versatile web crawler meticulously crafted for hunting down subdomains, paths, and other critical information. Whether you're a seasoned penetration tester, a dedicated bug bounty hunter, or simply curious about your digital footprint, Hakrawler stands as your unwavering companion in the quest for web exploration and security.

Installation

Embark on your Hakrawler journey with a seamless installation process:

Go install:

Install Go: Ensure you have Go installed on your system.

Download and Compile Hakrawler: Run the following command to download and compile the latest version of Hakrawler

go install github.com/hakluke/hakrawler@latest

This sets the stage for harnessing the power of Hakrawler.

Run Hakrawler: Execute Hakrawler using the command ~/go/bin/hakrawler. For a more convenient experience, export the PATH:

export PATH="~/go/bin/:$PATH"
Alternatively, for persistent configuration, add this line to your ~/.bashrc file.

Docker Install (from DockerHub):

Experience a hassle-free installation via DockerHub:

echo https://www.google.com | docker run --rm -i hakluke/hakrawler:v2 -subs

Local Docker Install:

For those preferring a local Docker install, follow these steps:

Clone the Repository:

git clone https://github.com/hakluke/hakrawler
cd hakrawler

Navigate to the heart of Hakrawler's codebase.

Build and Run Docker:

sudo docker build -t hakluke/hakrawler .
sudo docker run --rm -i hakluke/hakrawler --help

Unleash Hakrawler's capabilities with a local Docker setup.

Kali Linux: Using apt

Please note that this method installs an older version of Hakrawler without all features and may exhibit some bugs. It is recommended to explore other installation methods for the latest and most stable version.

sudo apt install hakrawler

Usage

Now that Hakrawler is at your disposal, let's delve into its capabilities through practical usage scenarios:

Single URL Crawling:

echo https://google.com | hakrawler

Initiate a precise crawl on a single URL, uncovering valuable insights.

Multiple URLs Exploration:

cat urls.txt | hakrawler

Efficiently traverse a list of URLs, automating your reconnaissance process.

Custom Timeout Setting:

cat urls.txt | hakrawler -timeout 5

Tailor the timeout for each line of input, ensuring efficient and controlled crawling.

Seamless Proxy Integration:

cat urls.txt | hakrawler -proxy http://localhost:8080

Channel your requests through a proxy, enhancing your anonymity and security.

Inclusive Subdomain Discovery:

echo https://google.com | hakrawler -subs

Uncover not just the surface but the hidden subdomains, expanding your web exploration.

Advanced Features

Comprehensive Output Analysis:

Hakrawler provides extensive output, allowing you to analyze and interpret the results effectively.

HTML, JavaScript, and More:

Explore Hakrawler's ability to detect and analyze HTML and JavaScript, adding a layer of depth to your web reconnaissance.

Conclusion

In the vast landscape of cybersecurity, having a tool like Hakrawler is akin to possessing a keen-eyed guide in the uncharted territories of the web. As you explore its features, customize commands, and witness the precision it brings to your web crawling experience, Hakrawler becomes an indispensable asset in your cybersecurity toolkit.

If you have any questions about this tutorial please leave a comment below or reach out to me on Twitter @amrelsagaei.