Knull and Knulleye Project

Overview

The Knull and Knulleye projects are designed for network scanning and web content retrieval. Knull focuses on scanning IP ranges using Masscan and fetching HTTP/HTTPS content, while Knulleye takes screenshots of the web pages identified by Knull. These tools are useful for security researchers, penetration testers, and network administrators who need to analyze web services and their content.

Features

Knull

• Masscan Integration: Scans specified CIDR ranges for open ports 80 (HTTP) and 443 (HTTPS).
• Multithreaded Fetching: Concurrently fetches web page content from discovered services.
• Session Management: Allows resuming scans from where they left off in case of interruptions.
• Custom SSL Handling: Bypasses SSL certificate verification for HTTPS requests.

Knulleye

• Selenium Integration: Takes full-page screenshots of web pages.
• Session Management: Tracks processed URLs to avoid reprocessing.
• Headless Browser: Operates in headless mode for efficiency.
• Filename Sanitization: Ensures valid filenames based on URLs.

Helper Scripts

• count_ips.sh: Counts the total number of IP addresses in a file containing CIDR ranges.
• extract_cidr.sh: Extracts CIDR blocks from a given input file.

Prerequisites

Before using these scripts, ensure you have the following installed:

• Python 3
• Masscan: For network scanning.
• Selenium: For web page screenshotting.
• ChromeDriver: Required for Selenium to control Chrome.
• Nmap: Used by count_ips.sh to list IPs.

Installation

1. Clone the Repository:

   git clone https://github.com/laudarch/knull.git
   cd knull
   python3 -m venv venv
   source ./venv/bin/activate

2. Install Python Dependencies:

   pip install -r requirements.txt

3. Install Masscan:

   sudo apt-get install masscan

4. Install ChromeDriver:

   sudo apt-get install chromium-chromedriver

5. Install Nmap:

   sudo apt-get install nmap

Usage

Knull

To scan a list of CIDR ranges and fetch web content:

./knull.py cidr_ranges.txt session_name

• cidr_ranges.txt: File containing CIDR ranges (one per line).
• session_name: Name of the session/project.

Knulleye

To take screenshots of URLs discovered by Knull:

./knulleye.py [file] session_name

• file: optional file containing urls to scan,(useful when you already have urls).
• session_name: Name of the session/project.

Helper Scripts

Count IPs

To count the total number of IP addresses in a file containing CIDR ranges:

./tools/count_ips.sh cidr_ranges.txt

• cidr_ranges.txt: File containing CIDR ranges (one per line).

Extract CIDR

To extract CIDR blocks from a file:

./tools/extract_cidr.sh input_file output_file

• input_file: File containing CIDR ranges.
• output_file: File to store the extracted CIDR ranges (one per line).

Session Management

The Knull and Knulleye projects use a session-based approach to organize and store output files. Each session (or project) is stored in its own directory under the Sessions folder. This allows you to manage multiple scans or projects independently, keeping their data isolated and organized.

Directory Structure

Here’s the directory structure for a session named session_name:

Sessions/                  # Root directory for all sessions
  session_name/            # Directory for a specific session/project
    ips.txt                # List of discovered IPs with ports
    scanner_session.json   # JSON file tracking the session state (completed/in-progress CIDR ranges)
    masscan/               # Directory containing Masscan output files
    screenshots/           # Directory containing screenshots taken by Knulleye
    results/               # Directory containing HTTP/HTTPS results
      http_results.txt     # Content of HTTP pages
      https_results.txt    # Content of HTTPS pages

Explanation of Files and Directories

1. ips.txt:

   • Contains a list of discovered IP addresses and their corresponding ports (e.g., http://192.168.1.1:80, https://192.168.1.1:443).

2. scanner_session.json:

   • Tracks the state of the session, including which CIDR ranges have been completed and which are in progress. This allows the scan to be resumed if interrupted.

3. masscan/:

   • Stores the raw output files generated by Masscan for each CIDR range scanned. Files are named in the format masscan_CIDR.json.

4. screenshots/:

   • Contains screenshots of web pages taken by Knulleye. Screenshots are saved with filenames derived from the URLs (e.g., scr_192_168_1_1_http.png).

5. results/:

   • Contains the fetched content of HTTP and HTTPS pages:
     • http_results.txt: Content of HTTP pages (port 80).
     • https_results.txt: Content of HTTPS pages (port 443).

Using Session Names

The session_name parameter is used to create a unique directory for each session or project. This allows you to run multiple scans or projects without overwriting data. For example:

• If you run Knull with session_name=project1, all output will be stored in Sessions/project1/.
• If you run Knulleye with session_name=project1, it will look for ips.txt and save screenshots in Sessions/project1/screenshots/.

Example Workflow

1. Run Knull:

   ./knull.py cidr_ranges.txt project1

   • This creates the directory Sessions/project1/ and stores all scan results there.

2. Run Knulleye:

   ./knulleye.py project1

   • This reads ips.txt from Sessions/project1/ and saves screenshots in Sessions/project1/screenshots/.

3. Resume a Session:

   • If the scan is interrupted, you can rerun Knull with the same session_name to resume from where it left off.

Benefits of Session Management

• Isolation: Each session is stored in its own directory, preventing data from different projects from mixing.
• Resumability: The scanner_session.json file allows scans to be resumed after interruptions.
• Organization: All related files (IPs, results, screenshots) are stored in a structured way, making it easy to analyze and share results.

This session-based approach ensures that your scans and projects are well-organized, easy to manage, and resumable.

Docker Integration

To simplify setup and ensure a consistent environment, this project includes a Dockerfile that allows you to run Knull and Knulleye inside a Docker container. This is particularly useful if you want to avoid installing dependencies directly on your system or if you need to run the tools in an isolated environment.

Prerequisites

Before using Docker, ensure you have the following installed:

• Docker: Install Docker by following the official Docker installation guide.

Building the Docker Image

1. Clone the Repository (if you haven't already):

   git clone https://github.com/laudarch/knull.git
   cd knull
   python3 -m venv venv
   source ./venv/bin/activate

2. Build the Docker Image: Run the following command to build the Docker image:

   docker build -t knull .

   This will create a Docker image named knull with all the necessary dependencies installed.

Running the Docker Container

Once the Docker image is built, you can run the container interactively or execute specific commands.

Interactive Mode

To start an interactive shell inside the container:

docker run -it --rm knull

From here, you can run any of the scripts (e.g., knull.py, knulleye.py, tools/count_ips.sh, tools/extract_cidr.sh) as needed.

Running Specific Commands

To run a specific command directly, use the following syntax:

docker run -it --rm knull ./knull.py cidr_ranges.txt session_name
docker run -it --rm knull ./knulleye.py session_name
docker run -it --rm knull ./tools/count_ips.sh cidr_ranges.txt
docker run -it --rm knull ./tools/extract_cidr.sh input_file output_file

Persisting Session Data

To persist session data (e.g., scan results, screenshots) between container runs, you can use Docker volumes. For example:

docker run -it --rm -v $(pwd)/sessions:/app/sessions knull

This will mount the sessions directory from your host machine to the /app/sessions directory inside the container, ensuring that session data is saved even after the container is stopped.

Notes

• Headless Mode: The Docker container is configured to run Chrome in headless mode, which is ideal for server environments.
• Resource Limits: If you're scanning large networks, consider increasing the container's resource limits (e.g., CPU and memory) using Docker's --cpus and --memory flags.

Example Workflow

1. Build the Docker image:

   docker build -t knull .

2. Run a Masscan and fetch web content:

   docker run -it --rm -v $(pwd)/sessions:/app/sessions knull./knull.py cidr_ranges.txt session_name

3. Take screenshots of the discovered URLs:

   docker run -it --rm -v $(pwd)/sessions:/app/sessions knull./knulleye.py session_name

4. Count IPs in a CIDR file:

   docker run -it --rm knull./count_ips.sh cidr_ranges.txt

5. Extract CIDR blocks from a file:

   docker run -it --rm knull./extract_cidr.sh input_file output_file

Using Docker ensures a clean and reproducible environment for running Knull and Knulleye, making it easier to manage dependencies and share the project with others.

Notes

• Ensure you have the necessary permissions to run network scans and access web content.
• Use these tools responsibly and in compliance with applicable laws and regulations.

License

This project is licensed under the MIT License. See the LICENSE file for details.

Contributing

Contributions are welcome! Please open an issue or submit a pull request for any improvements or bug fixes.

Acknowledgments

• Masscan: For fast and efficient network scanning.
• Selenium: For web page automation and screenshotting.
• Nmap: For IP address listing.

---

For any questions or issues, please open an issue on the GitHub repository.