Open Source Intelligence (OSINT)

Introduction to OSINT

Open Source Intelligence (OSINT) involves collecting publicly available information from a wide range of sources to support cybersecurity investigations, ethical hacking, and research. This process is crucial in the reconnaissance phase of penetration testing, where understanding a target's digital footprint is key to identifying potential vulnerabilities. OSINT allows security professionals to gather valuable data, such as personal details, domain registration information, exposed assets, network configurations, and even leaked credentials, which can be used to build a profile of the target. By leveraging search engines, social media, domain lookup services, and publicly accessible databases, OSINT provides insight into potential weaknesses that may not be visible through direct scanning or active probing. This passive reconnaissance is highly effective in reducing unknowns about a target's infrastructure, network topology, or organizational structure, enabling better planning and precision in penetration tests or cybersecurity defenses. Effective OSINT helps identify security gaps, such as unpatched systems, exposed APIs, and improperly configured services, which can be mitigated before adversaries exploit them.

OSINT Framework

Video Credit: Gary Ruddell - OSINT Framework

Google Dorks

Google Dorks utilize advanced search operators to uncover information not easily accessible through standard search queries. By combining specific search commands, you can reveal hidden directories, unsecured files, and exposed admin panels.

Reverse Image Search

Reverse Image Search allows you to find the origin of an image, discover similar images, or gather details about where the image appears online. It is an invaluable tool in OSINT for tracking down image sources, verifying authenticity, identifying individuals or objects, and analyzing the online spread of images. Below are several tools that can assist you in performing detailed reverse image searches:

Reverse image search tools are particularly useful for verifying the authenticity of photos, identifying misinformation, and conducting digital forensics. They are a crucial part of an investigator's toolkit in tracing the digital trail of images and uncovering hidden insights.

Extract Metadata from Files

Metadata often contains hidden information about files, such as location, author, or timestamp, that can be useful for investigations. Tools like ExifTool can extract metadata from various file types.

Install ExifTool: To install ExifTool on your system, use the following command:

sudo apt-get install exiftool

Extract Metadata from an Image: To extract metadata from an image, use the command:

exiftool <image_name>.jpg

Note: Extracted metadata can reveal sensitive information such as GPS coordinates or device details used to create an image.

theHarvester

theHarvester is a powerful tool used for gathering email addresses, employee names, subdomains, and other publicly available information from various search engines and platforms. It is often used in the early stages of penetration testing to build a list of targets.

Install theHarvester:

sudo apt-get install theharvester

Run theHarvester:

theharvester -d example.com -l 500 -b google,bing,linkedin

Note: theHarvester can also utilize APIs for more advanced queries if configured with relevant keys.

Shodan

Video Credit: Gary Ruddell - Shodan

Shodan is known as the "search engine for the Internet of Things." It allows you to search for internet-connected devices, such as routers, webcams, and industrial control systems. Unlike traditional search engines that index websites, Shodan indexes devices and the services they run. This makes it a powerful tool for security researchers, penetration testers, and IT administrators who need to assess the exposure of devices within their network.

Getting Started with Shodan

To use Shodan, you need to create an account on the Shodan website (Shodan.io). A basic free account will provide you with limited query access, while a paid subscription unlocks more advanced features and higher query limits.

Understanding Shodan's Search Queries

Shodan allows you to perform highly specific searches to find exposed devices. Here are some common query parameters:

Searching for Exposed Devices

Shodan can be used to locate exposed devices on the internet. Here are a few examples of practical searches:

Using Shodan Filters

Shodan provides a wide range of filters to help you narrow down your search results:

Exporting Search Results

Shodan allows you to export search results for further analysis. Paid users can download results as CSV or JSON files, which can be useful for large-scale investigations or integrations into other tools.

Shodan API

Shodan provides an API that allows developers to integrate its functionality into custom applications. You can use the API to automate tasks like device discovery and vulnerability assessment. To get started with the API, you will need an API key, which you can find on your Shodan account page.

Practical Use Cases for Shodan

Shodan Command-Line Interface (CLI)

Shodan also offers a command-line interface that can be installed using Python's package manager:

pip install shodan

Once installed, you need to initialize it with your API key:

shodan init YOUR_API_KEY

Here are some common CLI commands:

Shodan Alerts

Shodan Alerts is a feature that allows you to monitor specific IP ranges for changes or vulnerabilities. This is useful for companies that want to stay aware of any new devices or potential security risks in their network.

To create an alert, go to your Shodan account and navigate to the "Create Alert" section. You can specify IP ranges or CIDR blocks that you want Shodan to monitor.

Shodan can identify devices with open ports, vulnerable services, and even exposed databases. This information is crucial for understanding potential attack surfaces.

Maltego

Maltego is a data mining tool that helps build relationships between various types of data such as domains, IP addresses, email addresses, and social media profiles. It is extremely valuable for visualizing complex relationships.

Download and Install Maltego:

Visit: Maltego Download

Creating a New Graph:

Maltego is particularly useful for mapping out a company’s digital footprint and identifying relationships that could lead to vulnerabilities.

Recon-ng

Recon-ng is a reconnaissance framework that automates the process of gathering information. It is modular and can be extended to perform different types of OSINT tasks, such as domain, subdomain, and employee discovery.

Install Recon-ng:

sudo apt-get install recon-ng

Start Recon-ng:

recon-ng

Setting Up a Workspace:

workspaces create example

Loading Modules and Running Commands:

modules load recon/domains-hosts/google_site_web
set SOURCE example.com
run

FOCA

FOCA (Fingerprinting Organizations with Collected Archives) is a tool that scans for metadata in documents obtained from public sources, helping to identify potential vulnerabilities in document properties.

Download FOCA: FOCA is a Windows-based tool, available for download from its official website.

Use Cases for FOCA:

Sublist3r

Sublist3r is a tool designed to enumerate subdomains of websites using OSINT. It is a valuable tool for understanding the infrastructure of a target.

Install Sublist3r:

git clone https://github.com/aboul3la/Sublist3r.git
cd Sublist3r
sudo python setup.py install

Run Sublist3r:

sublist3r -d example.com

Social Media and Public Registries

Social media platforms and public registries are important resources for OSINT. Platforms like LinkedIn, Facebook, and Twitter can reveal employee details, job descriptions, and other sensitive information.

whois example.com

DNS Enumeration

DNS enumeration involves gathering information about a domain’s DNS infrastructure, which can reveal subdomains, IP addresses, and potential points of vulnerability.

Install dnsenum:

sudo apt-get install dnsenum

Run dnsenum:

dnsenum example.com

Note: DNS enumeration is useful for uncovering the digital landscape of a target organization and potential misconfigurations.

SSL Certificates

SSL certificates are crucial for secure communication between clients and servers. Understanding how to inspect SSL certificates can provide insight into an organization’s security posture.

View SSL Certificates in Web Browser: Click on the lock icon next to the URL to inspect certificate details, such as the issuer and expiration date.

Retrieve SSL Certificate Information via CLI:

openssl s_client -connect example.com:443 | openssl x509 -noout -text

Inspecting SSL certificates helps determine the encryption used and ensures that the target has up-to-date and secure certificates.

Social-Engineer Toolkit (SET)

The Social-Engineer Toolkit (SET) is an open-source tool that automates the process of social engineering attacks. It can be used for phishing, credential harvesting, and other types of social engineering campaigns.

Install SET:

git clone https://github.com/trustedsec/social-engineer-toolkit.git
cd social-engineer-toolkit
sudo python setup.py install

Use Cases: SET can be utilized to create custom phishing emails, clone websites for credential harvesting, and generate payloads that can exploit vulnerabilities on the target's machine.