Reconnaissance | Information Gathering

Reconnaissance  |  Information Gathering

Reconnaissance is the act of gathering preliminary data or intelligence on your target. The data is gathered in order to better plan for your attack. Reconnaissance can be performed actively (meaning that you are directly touching the target) or passively (meaning that your recon is being performed through an intermediary).

 Reconnaissance divide into two phases :-

1. Passive Reconnaissance :-


During Information Gathering phase, Passive information gathering will take place before we move on to Active information gathering. Passive information gathering is relatively less aggressive than active information gathering. Unlike active information gathering which requires much more direct engagement with the target, passive does not. Passive information gathering uses publicly published information about the target organization by using Google Hacking (aka. Google Dorks), The Wayback Machine, Job postings, NetCraft, Whois search.

The process of collecting information about an intended target of a malicious hack without the target knowing what is occurring. Typical passive reconnaissance can include physical observation of an enterprise's building, sorting through discarded computer equipment in an attempt to find equipment that contains data or discarded paper with usernames and passwords, eavesdropping on employee conversations, researching the target through common Internet tools such as Whois, impersonating an employee in an attempt to collect information, and packet sniffing.


2. Active Reconnaissance :-

Active information gathering involves direct engagement with the target organization through such techniques like social engineering, nmap scan. Since it makes a direct contact to the target Active Information Gathering would trigger the target's IDS, IPS if there are any and this is where we draw the line between Passive and Active Information Gatherings.

The process of collecting information about an intended target of a malicious hack by probing the target system. Active reconnaissance typically involves port scanning in order to find weaknesses in the target system (i.e., which ports are left vulnerable and/or if there are ways around the firewall and routers). The process of exploiting the system can then be carried out once the hacker has found a way to access the system.


Tools which we are going to use for information gathering ( both active and passive ) about our target :-


1. Altdns:-

Altdns - Subdomain discovery through alterations and permutations

Altdns is a DNS recon tool that allows for the discovery of subdomains that conform to patterns. Altdns takes in words that could be present in subdomains under a domain (such as test, dev, staging) as well as takes in a list of subdomains that you know of.
From these two lists that are provided as input to altdns, the tool then generates a massive output of "altered" or "mutated" potential subdomains that could be present. It saves this output so that it can then be used by your favourite DNS bruteforcing tool.
Alternatively, the -r flag can be passed to altdns so that once this output is generated, the tool can then resolve these subdomains (multi-threaded) and save the results to a file.
Altdns works best with large datasets. Having an initial dataset of 200 or more subdomains should churn out some valid subdomains via the alterations generated.

Installation

pip install -r requirements.txt

Usage

# ./altdns.py -i subdomains.txt -o data_output -w words.txt -r -s results_output.txt

• subdomains.txt contains the known subdomains for an organization
• data_output is a file that will contain the massive list of altered and permuted subdomains
• words.txt is your list of words that you'd like to permute your current subdomains with (i.e. admin, staging, dev, qa) - one word per line
• the -r command resolves each generated, permuted subdomain
• the -s command tells altdns where to save the results of the resolved permuted subdomains. results_output.txt will contain the final list of permuted subdomains found that are valid and have a DNS record.
• the -t command limits how many threads the resolver will use simultaneously
• -d 1.2.3.4 overrides the system default DNS resolver and will use the specified IP address as the resolving server. Setting this to the authoritative DNS server of the target domain may increase resolution performance

Screenshots

 

Download

2. Amass:-

OWASP Amass

 Amass is now an OWASP project and the OWASP GitHub organization repository is where all further development and releases will take place.

Community

• Discord Server - Discussing OSINT, network recon and developing security tools using Go

Example Amass Terminal Capture

Presented at Facebook (and shared publically) for the Sept. 2018 OWASP London Chapter meeting:

Download

3. Aquatone:-

AQUATONE

AQUATONE is a set of tools for performing reconnaissance on domain names. It can discover subdomains on a given domain by using open sources as well as the more common subdomain dictionary brute force approach. After subdomain discovery, AQUATONE can then scan the hosts for common web ports and HTTP headers, HTML bodies and screenshots can be gathered and consolidated into a report for easy analysis of the attack surface.

Installation

Dependencies

AQUATONE depends on Node.js and NPM package manager for its web page screenshotting capabilities. Follow this guide for Installation instructions.
You will also need a newer version of Ruby installed. If you plan to use AQUATONE in Kali Linux, you are already set up with this. If not, it is recommended to install Ruby with RVM.
Finally, the tool itself can be installed with the following command in a terminal:

$ gem install aquatone

IMPORTANT: AQUATONE's screenshotting capabilities depend on being run on a system with a graphical desktop environment. It is strongly recommended to install and run AQUATONE in a Kali linux virtual machine. I will not provide support or bug fixing for other systems than Kali Linux.

Usage

Discovery

The first stage of an AQUATONE assessment is the discovery stage where subdomains are discovered on the target domain using open sources, services and the more common dictionary brute force approach:

$ aquatone-discover --domain example.com

aquatone-discover will find the target's nameservers and shuffle DNS lookups between them. Should a lookup fail on the target domain's nameservers, aquatone-discover will fall back to using Google's public DNS servers to maximize discovery. The fallback DNS servers can be changed with the --fallback-nameservers option:

$ aquatone-discover --domain example.com --fallback-nameservers 87.98.175.85,5.9.49.12

Tuning

aquatone-discover will use 5 threads as default for concurrently performing DNS lookups. This provides reasonable performance but can be tuned to be more or less aggressive with the --threads option:

$ aquatone-discover --domain example.com --threads 25

Hammering a DNS server with failing lookups can potentially be picked up by intrusion detection systems, so if that is a concern for you, you can make aquatone-discover a bit more stealthy with the --sleep and --jitter options. --sleep accepts a number of seconds to sleep between each DNS lookup while --jitter accepts a percentage of the --sleep value to randomly add or subtract to or from the sleep interval in order to break the sleep pattern and make it less predictable.

$ aquatone-discover --domain example.com --sleep 5 --jitter 30

Please note that setting the --sleep option will force the thread count to one. The --jitter option will only be considered if the --sleep option has also been set.

API keys

Some of the passive collectors will require API keys or similar credentials in order to work. Setting these values can be done with the --set-key option:

$ aquatone-discover --set-key shodan o1hyw8pv59vSVjrZU3Qaz6ZQqgM91ihQ

All keys will be saved in ~/aquatone/.keys.yml.

Results

When aquatone-discover is finished, it will create a hosts.txt file in the ~/aquatone/<domain> folder, so for a scan of example.com it would be located at ~/aquatone/example.com/hosts.txt. The format will be a comma-separated list of hostnames and their IP, for example:

example.com,93.184.216.34
www.example.com,93.184.216.34
secret.example.com,93.184.216.36
cdn.example.com,192.0.2.42
...

In addition to the hosts.txt file, it will also generate a hosts.json which includes the same information but in JSON format. This format might be preferable if you want to use the information in custom scripts and tools. hosts.json will also be used by the aquatone-scan and aquatone-gather tools.
See aquatone-discover --help for more options.

Scanning

The scanning stage is where AQUATONE will enumerate the discovered hosts for open TCP ports that are commonly used for web services:

$ aquatone-scan --domain example.com

The --domain option will look for hosts.json in the domain's AQUATONE assessment directory, so in the example above it would look for ~/aquatone/example.com/hosts.json. This file should be present if aquatone-discover --domain example.com has been run previously.

Ports

By default, aquatone-scan will scan the following TCP ports: 80, 443, 8000, 8080 and 8443. These are very common ports for web services and will provide a reasonable coverage. Should you want to specifiy your own list of ports, you can use the --ports option:

$ aquatone-scan --domain example.com --ports 80,443,3000,8080

Instead of a comma-separated list of ports, you can also specify one of the built-in list aliases:

• small: 80, 443
• medium: 80, 443, 8000, 8080, 8443 (same as default)
• large: 80, 81, 443, 591, 2082, 2095, 2096, 3000, 8000, 8001, 8008, 8080, 8083, 8443, 8834, 8888, 55672
• huge: 80, 81, 300, 443, 591, 593, 832, 981, 1010, 1311, 2082, 2095, 2096, 2480, 3000, 3128, 3333, 4243, 4567, 4711, 4712, 4993, 5000, 5104, 5108, 5280, 5281, 5800, 6543, 7000, 7396, 7474, 8000, 8001, 8008, 8014, 8042, 8069, 8080, 8081, 8083, 8088, 8090, 8091, 8118, 8123, 8172, 8222, 8243, 8280, 8281, 8333, 8337, 8443, 8500, 8834, 8880, 8888, 8983, 9000, 9043, 9060, 9080, 9090, 9091, 9200, 9443, 9800, 9981, 11371, 12443, 16080, 18091, 18092, 20720, 55672

Example:

$ aquatone-scan --domain example.com --ports large

Tuning

Like aquatone-discover, you can make the scanning more or less aggressive with the --threads option which accepts a number of threads for concurrent port scans. The default number of threads is 5.

$ aquatone-scan --domain example.com --threads 25

As aquatone-scan is performing port scanning, it can obviously be picked up by intrusion detection systems. While it will attempt to lessen the risk of detection by randomising hosts and ports, you can tune the stealthiness more with the --sleep and --jitter options which work just like the similarly named options for aquatone-discover. Keep in mind that setting the --sleep option will force the number of threads to one.

Results

When aquatone-scan is finished, it will create a urls.txt file in the ~/aquatone/<domain> directory, so for a scan of example.com it would be located at ~/aquatone/example.com/urls.txt. The format will be a list of URLs, for example:

http://example.com/
https://example.com/
http://www.example.com/
https://www.example.com/
http://secret.example.com:8001/
https://secret.example.com:8443/
http://cdn.example.com/
https://cdn.example.com/
...

This file can be loaded into other tools such as EyeWitness.
aquatone-scan will also generate a open_ports.txt file, which is a comma-separated list of hosts and their open ports, for example:

93.184.216.34,80,443
93.184.216.34,80
93.184.216.36,80,443,8443
192.0.2.42,80,8080
...

See aquatone-scan --help for more options.

Gathering

The final stage is the gathering part where the results of the discovery and scanning stages are used to query the discovered web services in order to retrieve and save HTTP response headers and HTML bodies, as well as taking screenshots of how the web pages look like in a web browser to make analysis easier. The screenshotting is done with the Nightmare.js Node.js library. This library will be installed automatically if it's not present in the system.

$ aquatone-gather --domain example.com

aquatone-gather will look for hosts.json and open_ports.txt in the given domain's AQUATONE assessment directory and request and screenshot every IP address for each domain name for maximum coverage.

Tuning

Like aquatone-discover and aquatone-scan, you can make the gathering more or less aggressive with the --threads option which accepts a number of threads for concurrent requests. The default number of threads is 5.

$ aquatone-gather --domain example.com --threads 25

As aquatone-gather is interacting with web services, it can be picked up by intrusion detection systems. While it will attempt to lessen the risk of detection by randomising hosts and ports, you can tune the stealthiness more with the --sleep and --jitter options which work just like the similarly named options for aquatone-discover. Keep in mind that setting the --sleep option will force the number of threads to one.

Results

When aquatone-gather is finished, it will have created several directories in the domain's AQUATONE assessment directory:

• headers/: Contains text files with HTTP response headers from each web page
• html/: Contains text files with HTML response bodies from each web page
• screenshots/: Contains PNG images of how each web page looks like in a browser
• report/ Contains report files in HTML displaying the gathered information for easy analysis

Subdomain Takeover

Subdomain takeover is a very prevalent and potentially critical security issue which commonly occurs when an organization assigns a subdomain to a third-party service provider and then later discontinues use, but forgets to remove the DNS configuration. This leaves the subdomain vulnerable to complete takover by attackers by signing up to the same service provider and claiming the dangling subdomain.
aquatone-takeover can be used to check hosts uncovered by aquatone-discover for potential domain takeover vulnerabilities:

$ aquatone-takeover --domain example.com

aquatone-takeover can detect potential subdomain takeover situations from 25 different service providers, including GitHub Pages, Heroku, Amazon S3, Desk and WPEngine.

Results

aquatone-takeover will create a takeovers.json file in the domain's assessment directory which will contain information in JSON format about any potential subdomain takeover vulnerabilities:

{
  "shop.example.com": {
    "service": "Shopify",
    "service_website": "https://www.shopify.com/",
    "description": "Ecommerce platform",
    "resource": {
      "type": "CNAME",
      "value": "shops.myshopify.com"
    }
  },
  "help.example.com": {
    "service": "Desk",
    "service_website": "https://www.desk.com/",
    "description": "Customer service and helpdesk ticket software",
    "resource": {
      "type": "CNAME",
      "value": "example.desk.com"
    }
  },
  ...
} 

 Download

4.  Bfac:-

BFAC

Advanced Backup-File Artifacts Testing for Web-Applications

BFAC (Backup File Artifacts Checker) is an automated tool that checks for backup artifacts that may disclose the web-application's source code. The artifacts can also lead to leakage of sensitive information, such as passwords, directory structure, etc.
The goal of BFAC is to be an all-in-one tool for backup-file artifacts black-box testing.

Features

• Multithreaded scanning.
• Includes request rate throttling.
• HTTP proxy support.
• Uses multiple algorithms for automatically detecting valid and invalid pages.
• HTTP proxy support
• User agent randomization.
• Batch processing.
• Works both as a command-line tool and Python module.
• Support for Windows, MacOS, and Linux operating systems.
• Reporting: simple, verbose, CSV, JSON.

Usage

Description	Command
Help	bfac --help
Check a single URL.	bfac --url http://example.com/test.php
Check a list of URLs.	bfac --list testing_list.txt
Single URL with a different level (level 2 for example).	bfac --url http://example.com/test.php --level 2
Single URL and show the results only.	bfac --no-text --url http://example.com/test.php
Limit the test to exposed DVCS tests.	bfac --dvcs-test --url http://example.com/
Verify existence of files using Content-Length checks only.	bfac --detection-technique content_length http://example.com/test.php
Verify existence of files using Status-Code checks only.	bfac --detection-technique status_code http://example.com/test.php
Exclude results with specific status-codes.	bfac --exclude-status-codes 301,999 http://example.com/test.php

Using BFAC as a module

import bfac


testing_level = 5

# Returns a list of BFA patterns for http://example.com/test.php
bfa_urls = bfac.generate_bfa_urls(
    'http://example.com/test.php', testing_level=testing_level)

# Performs BFA testing using BFAC, and returns a list of findings, if any.
# If nothing is identified, it returns an empty list.
bfa_testing_result = bfac.test_url(
    'http://example.com/test.php', testing_level=testing_level)

Requirements

• Python2 or Python3
• requests
• colorama

Installation [Optional]

sudo python setup.py install

Compatibility

The project currently supports all platforms that run Python. The project is compatible with both Python 2 and Python 3.

Download

5. Dirsearch:-

Overview

dirsearch is a simple command line tool designed to brute force directories and files in websites.

Operating Systems supported

• Windows XP/7/8/10
• GNU/Linux
• MacOSX

Features

• Multithreaded
• Keep alive connections
• Support for multiple extensions (-e|--extensions asp,php)
• Reporting (plain text, JSON)
• Heuristically detects invalid web pages
• Recursive brute forcing
• HTTP proxy support
• User agent randomization
• Batch processing
• Request delaying

About wordlists

Dictionaries must be text files. Each line will be processed as such, except that the special word %EXT% is used, which will generate one entry for each extension (-e | --extension) passed as an argument.
Example:

• example/
• example.%EXT%

Passing the extensions "asp" and "aspx" will generate the following dictionary:

• example/
• example.asp
• example.aspx

You can also use -f | --force-extensions switch to append extensions to every word in the wordlists (like DirBuster).

Download

6. Discover:-


7. DnsEnum
8. DnsRecon
9. Enum4Linux
10. Eyewitness
11. Fierce
12. GithubCloner
13. Gitrob
14. Gobuster
15. Goofile
16. Gowitness
17. Hacks
18.Hardcidr
19. Httpscreenshot
20. Ipcrawl
21. Knockpy
22. Masscan
23. Massdns
24. Msfenum
25. Nullinux
26. Onesixtyone
27. Osrframework
28. Prowl
29. Rawr
30. Recon-ng
31. Ridenum
32. Sap-dissector-wireshark
33. Scancannon
34. Server-status_pwn
35. Shell-storm-api
36. Simplyemail
37. Skiptracer
38. Smtp-user-enum
39. Sniper
40. Spiderfoot
41. SSH-audit
42. Subfinder
43. Subjack
44. Sublist3r
45. Subover
46. theHarvester
47. Turbolist3r
48. Tweets_analyzer
49. UDP-proto-scanner
50. URLcrazy
51. Wafw00f
52. Windows-exploit-suggester
53. Xdotool
54. Yapscan
55. InSpy



Thank You !