urlfinder

A high-speed tool for passively gathering URLs, optimized for efficient web asset discovery without active scanning.

Features • Installation • Usage • Examples • Join Discord

Overview

URLFinder is a high-speed, passive URL discovery tool designed to simplify and accelerate web asset discovery, ideal for penetration testers, security researchers, and developers looking to gather URLs without active scanning.

Features

Curated Passive Sources to maximize comprehensive URL discovery
Supports multiple output formats (JSON, file, stdout)
Optimized for Speed and resource efficiency
STDIN/OUT support for easy integration into existing workflows

Installation

URLFinder requires Go 1.21. Install it using the following command or download a pre-compiled binary from the releases page.

go install -v github.com/projectdiscovery/urlfinder/cmd/urlfinder@latest

This command displays help for URLFinder. Below are some common switches and options.

A streamlined tool for discovering associated URLs.

Usage:
  ./urlfinder [flags]

Flags:
INPUT:
   -d, -list string[]  target domain or list of domains

SOURCE:
   -s, -sources string[]           specific sources for discovery (e.g., -s censys,dnsrepo)
   -es, -exclude-sources string[]   sources to exclude (e.g., -es censys,dnsrepo)
   -all                             use all sources (may be slower)

FILTER:
   -m, -match string[]       URLs or list to match (file or comma-separated)
   -f, -filter string[]      URLs or list to filter (file or comma-separated)

RATE-LIMIT:
   -rl, -rate-limit int      max HTTP requests per second (global)
   -rls, -rate-limits value  per-provider HTTP request limits (e.g., -rls waybackarchive=15/m)

UPDATE:
   -up, -update              update URLFinder to the latest version
   -duc, -disable-update-check  disable automatic update checks

OUTPUT:
   -o, -output string       specify output file
   -j, -jsonl               JSONL output format
   -od, -output-dir string  specify output directory
   -cs, -collect-sources    include all sources in JSON output

CONFIGURATION:
   -config string           config file (default "$CONFIG/urlfinder/config.yaml")
   -pc, -provider-config string  provider config file (default "$CONFIG/urlfinder/provider-config.yaml")
   -proxy string            HTTP proxy

DEBUG:
   -silent                  show only URLs in output
   -version                 display URLFinder version
   -v                       verbose output
   -nc, -no-color           disable colored output
   -ls, -list-sources       list all available sources
   -stats                   display source statistics

OPTIMIZATION:
   -timeout int   timeout in seconds (default 30)
   -max-time int  max time in minutes for enumeration (default 10)

Examples

Basic Usage

This command enumerates URLs for the target domain projectdiscovery.io.

Example run:

$ urlfinder -d tesla.com

  __  _____  __   _____         __       
 / / / / _ \/ /  / __(_)__  ___/ /__ ____
/ /_/ / , _/ /__/ _// / _ \/ _  / -_) __/
\____/_/|_/____/_/ /_/_//_/\_,_/\__/_/    										

		projectdiscovery.io

[INF] Current urlfinder version v0.0.1 (latest)
[INF] Enumerating urls for tesla.com
https://www.tesla.com/akam/13/7e68a6e8
https://www.tesla.com/akam/13/pixel_4e07b670
https://www.tesla.com/da_dk/en/node/30788?redirect=no
https://www.tesla.com/de_at/findus/location/charger/dc6290
https://www.tesla.com/akam/13/7ade0a44
https://www.tesla.com/cs_cz/referral/teslaapp23713?redirect=no
https://www.tesla.com/da_dk/findus/location/charger/dc253
https://www.tesla.com/akam/13/pixel_76102729
https://www.tesla.com/da_dk/blog/modules//system/system.messages.js
...
[INF] Found 202435 urls for tesla.com in 2 minutes 37 seconds

Filtering Options

Use the -m (match) and -f (filter) options to refine results based on URL patterns.

Examples

Include URLs Matching Specific Patterns

To include only URLs containing "shop" or "model":
```
urlfinder -d tesla.com -m shop,model
```
Exclude URLs Matching Specific Patterns

To exclude URLs containing "privacy" or "terms":
```
urlfinder -d tesla.com -f privacy,terms
```
Combined Match and Filter

To find URLs containing "support" but exclude those with "faq":
```
urlfinder -d tesla.com -m support -f faq
```

Using Files for Matching and Filtering

Provide patterns in files:

urlfinder -d tesla.com -m include-patterns.txt -f exclude-patterns.txt

JSONL Output Example

Use the -j or --jsonl flag to output results in JSONL (JSON Lines) format, where each line is a separate JSON object. This format is useful for processing large outputs in a structured way.

Command Example

urlfinder -d tesla.com -j

Example JSONL Output

{"url":"https://shop.tesla.com/product/model-s-plaid","input":"tesla.com","source":"waybackarchive"}
{"url":"https://www.tesla.com/inventory/used/ms","input":"tesla.com","source":"waybackarchive"}
{"url":"https://forums.tesla.com/discussion/101112/model-3-updates","input":"tesla.com","source":"waybackarchive"}

Each JSON object contains:

url: The discovered URL.
input: The target domain (e.g., tesla.com).
source: The data source for the URL discovery (e.g., waybackarchive).