Merge pull request #784 from shelld3v/several-updates

Several updates

Author: maurosoria

CHANGELOG.md

@@ -8,6 +8,8 @@ Currently working...
 - Fixed the CSV Injection vulnerability (https://www.exploit-db.com/exploits/49370)
 - Raw request supported
 - Can setup the default URL scheme (will be used when there is no scheme in the URL)
+- Option to skip target by specified status codes
+- Added max-runtime option
 
 v0.4.1 (2020.12.8)
 ---------

README.md

@@ -1,5 +1,3 @@
-![dirsearch](https://user-images.githubusercontent.com/59408894/103289759-87a0ce80-4a1a-11eb-89c9-2feb7e6db25f.png) - *Hacking is not a crime*
-
 dirsearch - Web path scanner
 =========
 
@@ -16,13 +14,13 @@ dirsearch - Web path scanner
 
 Overview
 --------
-- Dirsearch is a mature command-line tool designed to brute force directories and files in webservers. 
+- "dirsearch" is a mature command-line tool designed to brute force directories and files in webservers.
 
 - With 6 years of growth, dirsearch now has become the top web content scanner.
 
 - As a feature-rich tool, dirsearch gives users the opportunity to perform a complex web content discovering, with many vectors for the wordlist, high accuracy, impressive performance, advanced connection/request settings, modern brute-force techniques and nice output.
 
-- Dirsearch is being actively developed by [@maurosoria](https://twitter.com/_maurosoria) and [@shelld3v](https://github.com/shelld3v)
+- "dirsearch" is being actively developed by [@maurosoria](https://twitter.com/_maurosoria) and [@shelld3v](https://github.com/shelld3v)
 
 
 Installation & Usage
@@ -36,9 +34,9 @@ python3 dirsearch.py -u <URL> -e <EXTENSIONS>
 
 - To can use SOCKS proxy or work with `../` in the wordlist, you need to install pips with `requirements.txt`: `pip3 install -r requirements.txt`
 
-- If you are using Windows and don't have git, you can install the ZIP file [here](https://github.com/maurosoria/dirsearch/archive/master.zip). Dirsearch also supports [Docker](https://github.com/maurosoria/dirsearch#support-docker)
+- If you are using Windows and don't have git, you can install the ZIP file [here](https://github.com/maurosoria/dirsearch/archive/master.zip). dirsearch also supports [Docker](https://github.com/maurosoria/dirsearch#support-docker)
 
-*Dirsearch requires python 3 or greater*
+*dirsearch requires python 3 or greater*
 
 
 Features
@@ -84,22 +82,20 @@ Features
 
 About wordlists
 ---------------
-**Summary**: Wordlist must be a text file, each line will be an endpoint. About extensions, unlike other tools, dirsearch won't append extensions to every word, if you don't use the `-f` flag. By default, only the `%EXT%` keyword in the wordlist will be replaced with extensions (`-e <extensions>`).
+**Summary**: Wordlist must be a text file, each line will be an endpoint. About extensions, unlike other tools, dirsearch doesn't append extensions to every word, if you don't use the `-f` flag. By default, only the `%EXT%` keyword in the wordlist will be replaced with extensions (`-e <extensions>`).
 
 **Details**:
 - Each line in the wordlist will be processed as such, except when the special keyword *%EXT%* is used, it will generate one entry for each extension (-e | --extensions) passed as an argument.
 
 Example:
 
 ```
-root/
 index.%EXT%
 ```
 
 Passing the extensions "asp" and "aspx" (`-e asp,aspx`) will generate the following dictionary:
 
 ```
-root/
 index
 index.asp
 index.aspx
@@ -111,7 +107,6 @@ Example:
 
 ```
 admin
-home.%EXT%
 api%NOFORCE%
 ```
 
@@ -122,9 +117,6 @@ admin
 admin.php
 admin.html
 admin/
-home
-home.php
-home.html
 api
 ```
 
@@ -169,8 +161,8 @@ Options:
     --suffixes=SUFFIXES
                         Add custom suffixes to all entries, ignore directories
                         (separated by commas)
-    --only-selected     Only entries with selected extensions or no extension
-                        + directories
+    --only-selected     Only directories + files with selected extensions (or
+                        no extension)
     --remove-extensions
                         Remove extensions in all wordlist entries (Example:
                         admin.php -> admin)
@@ -207,10 +199,11 @@ Options:
     --exclude-redirects=REGEXPS
                         Exclude responses by redirect regexps or texts,
                         separated by commas (Example: 'https://okta.com/*')
-    --calibration=PATH  Path to test for calibration
-    --random-agent      Choose a random User-Agent for each request
+    --exclude-content=PATH
+                        Exclude responses by response content of this path
     --minimal=LENGTH    Minimal response length
     --maximal=LENGTH    Maximal response length
+    --skip-on-429       Skip target whenever 429 status code is returned
     -q, --quiet-mode    Quiet mode
     --full-url          Print full URLs in the output
     --no-color          No colored output
@@ -226,6 +219,7 @@ Options:
     --header-list=FILE  File contains HTTP request headers
     -F, --follow-redirects
                         Follow HTTP redirects
+    --random-agent      Choose a random User-Agent for each request
     --user-agent=USERAGENT
     --cookie=COOKIE
 
@@ -237,7 +231,7 @@ Options:
     --proxy=PROXY       Proxy URL, support HTTP and SOCKS proxies (Example:
                         localhost:8080, socks5://localhost:8088)
     --proxy-list=FILE   File contains proxy servers
-    --matches-proxy=PROXY
+    --replay-proxy=PROXY
                         Proxy to replay with found paths
     --scheme=SCHEME     Default scheme (for raw request or if there is no
                         scheme in the URL)
@@ -257,7 +251,7 @@ Options:
     --csv-report=OUTPUTFILE
 ```
 
- **NOTE**: 
+ **NOTE**:
  You can change the dirsearch default configurations (default extensions, timeout, wordlist location, ...) by editing the **default.conf** file.
 
 
@@ -306,7 +300,7 @@ python3 dirsearch.py -e php,htm,js,bak,zip,tgz,txt -u https://target -t 30
 - **--prefixes**: Adding custom prefixes to all entries
 
 ```
-python3 dirsearch.py -e php -u https://target --prefixes .,admin,_,~
+python3 dirsearch.py -e php -u https://target --prefixes .,admin,_
 ```
 Base wordlist:
 
@@ -319,7 +313,6 @@ Generated with prefixes:
 .tools
 admintools
 _tools
-~tools
 ```
 
 - **--suffixes**: Adding custom suffixes to all entries
@@ -342,34 +335,48 @@ internal~
 internal/
 ```
 
-### Exclude extensions
-Use **-X | --exclude-extensions** with your exclude-extension list to remove all entries in the wordlist that have the given extensions
+### Blacklist
+Inside the `db` folder, there are several "blacklist files". Paths in those files will be filtered from the scan result if they have the same status as mentioned in the filename.
+
+Example: If you add `admin.php` into `db/403_blacklist.txt`, whenever you do a scan that `admin.php` returns 403, it (`admin.php`) will be excluded.
+
+### Filters
+Use **-i | --include-status** and **-x | --exclude-status** to select allowed and not allowed response status codes
 
 ```
-python3 dirsearch.py -e asp,aspx,htm,js -u https://target -X php,jsp,jspx
+python3 dirsearch.py -e php,html,js -u https://target -i 200,204,400,403 -x 500,502,429
 ```
 
-Base wordlist:
+**--exclude-sizes**, **--exclude-texts**, **--exclude-regexps**, **--exclude-redirects** and **--exclude-content** are also supported for a more advanced filter
 
 ```
-admin
-admin.%EXT%
-index.html
-home.php
-test.jsp
+python3 dirsearch.py -e php,html,js -u https://target --exclude-sizes 1B,243KB
 ```
 
-After:
+```
+python3 dirsearch.py -e php,html,js -u https://target --exclude-texts "403 Forbidden"
+```
 
 ```
-admin
-admin.asp
-admin.aspx
-admin.htm
-admin.js
-index.html
+python3 dirsearch.py -e php,html,js -u https://target --exclude-regexps "^Error$"
+```
+
+```
+python3 dirsearch.py -e php,html,js -u https://target --exclude-content "admin.php"
+```
+
+### Raw requests
+dirsearch allows you to import the raw request from a file. The raw file content will be looked something like this:
+
+```
+GET /admin HTTP/1.1
+Host: admin.example.com
+Cache-Control: max-age=0
+Accept: */*
 ```
 
+Since there is no way for dirsearch to know what the URI scheme is (`http` or `https`), you need to set it using the `--scheme` flag. By default, the scheme is `http`, which is not popular in modern web servers now. That means, without setting up the scheme, you may brute-force with the wrong protocol, and will end up with false negatives.
+
 ### Wordlist formats
 Supported wordlist formats: uppercase, lowercase, capitalization
 
@@ -378,58 +385,45 @@ Supported wordlist formats: uppercase, lowercase, capitalization
 ```
 admin
 index.html
-test
 ```
 
 #### Uppercase:
 
 ```
 ADMIN
 INDEX.HTML
-TEST
 ```
 
 #### Capital:
 
 ```
 Admin
 Index.html
-Test
-```
-
-### Filters
-Use **-i | --include-status** and **-x | --exclude-status** to select allowed and not allowed response status codes
-
-```
-python3 dirsearch.py -e php,html,js -u https://target -i 200,204,400,403 -x 500,502,429
 ```
 
-**--exclude-sizes**, **--exclude-texts**, **--exclude-regexps** and **--exclude-redirects** are also supported for a more advanced filter
+### Exclude extensions
+Use **-X | --exclude-extensions** with your exclude-extension list to remove all entries in the wordlist that have the given extensions
 
 ```
-python3 dirsearch.py -e php,html,js -u https://target --exclude-sizes 1B,243KB
+python3 dirsearch.py -e asp,aspx -u https://target -X jsp
 ```
 
-```
-python3 dirsearch.py -e php,html,js -u https://target --exclude-texts "403 Forbidden"
-```
+Base wordlist:
 
 ```
-python3 dirsearch.py -e php,html,js -u https://target --exclude-regexps "^Error$"
+admin
+admin.%EXT%
+test.jsp
 ```
 
-### Raw requests
-dirsearch allows you to import the raw request from a file. The raw file content will be looked something like this:
+After:
 
 ```
-GET /admin HTTP/1.1
-Host: admin.example.com
-Cache-Control: max-age=0
-Accept: */*
+admin
+admin.asp
+admin.aspx
 ```
 
-Since there is no way for dirsearch to know what the URI scheme is (`http` or `https`), you need to set it using the `--scheme` flag. By default, the scheme is `http`, which is not popular in modern web servers now. That means, without setting up the scheme, you may brute-force with the wrong protocol, and will end up with false negatives.
-
 ### Scan sub-directories
 From an URL, you can scan sub-directories with **--subdirs**.
 
@@ -516,10 +510,11 @@ python3 dirsearch.py -u https://target -w db/dicc.txt --no-extension
 Tips
 ---------------
 - To run dirsearch with a rate of requests per second, try `-t <rate> -s 1`
+- The server has a request limit? That's bad, but feel free to bypass it, by randomizing proxy with `--proxy-list`
 - Want to findout config files or backups? Try out `--suffixes ~` and `--prefixes .`
 - For some endpoints that you do not want to force extensions, add `%NOFORCE%` at the end of them
 - Want to find only folders/directories? Combine `--no-extension` and `--suffixes /`!
-- The combination of `--cidr`, `-F` and `-q` will reduce most of the noise + false negatives when brute-forcing with a CIDR
+- The combination of `--cidr`, `-F`, `-q` and a low `--timeout` will reduce most of the noise + false negatives when brute-forcing with a CIDR
 
 
 Support Docker

db/400_blacklist.txt

@@ -3,3 +3,5 @@
 %2e%2e;/test
 %3f/
 %C0%AE%C0%AE%C0%AF
+..;/
+

db/500_blacklist.txt

@@ -1,3 +1,6 @@
 %ff
 %3f/
 %C0%AE%C0%AE%C0%AF
+%2e%2e;/test
+..;/
+

default.conf

@@ -13,8 +13,9 @@ recursive = False
 recursion-depth = 0
 exclude-subdirs = %%ff/
 random-user-agents = False
+max-time = 0
+## 0 means no maximal running time
 save-logs-home = False
-skip-on-429 = False
 full-url = False
 quiet-mode = False
 color = True
@@ -23,7 +24,8 @@ color = True
 # exclude-sizes = 0b,123gb
 # exclude-texts = "Not found"
 # exclude-regexps = 403 [a-z]{1,25}
-# calibration-path = InvalidPath123
+# exclude-content = index.html
+# skip-on-status = 429,999
 
 [reports]
 autosave-report = True

lib/connection/requester.py

@@ -30,8 +30,6 @@ class Requester(object):
     def __init__(
         self,
         url,
-        cookie=None,
-        useragent=None,
         maxPool=1,
         maxRetries=5,
         timeout=20,
@@ -102,13 +100,6 @@ def __init__(
         ):
             self.headers["Host"] += ":{0}".format(self.port)
 
-        # Set cookie and user-agent headers
-        if cookie:
-            self.setHeader("Cookie", cookie)
-
-        if useragent:
-            self.setHeader("User-agent", useragent)
-
         self.maxRetries = maxRetries
         self.maxPool = maxPool
         self.timeout = timeout
@@ -164,7 +155,7 @@ def request(self, path, proxy=None):
                 url = self.url + self.basePath + path
 
                 if self.randomAgents:
-                    self.headers["User-agent"] = random.choice(self.randomAgents)
+                    self.headers["User-Agent"] = random.choice(self.randomAgents)
 
                 response = self.session.request(
                     self.httpmethod,