Getting started
Introduction
GNU Wget downloads files from the web non-interactively over HTTP, HTTPS, and FTP. This reference covers common options and recipes, current through Wget 1.25.
- GNU Wget manual (gnu.org)
- Wget man page (man7.org)
Basic download
| Syntax | Effect |
|---|---|
wget URL | Save to the current directory |
-i urls.txt | Read URLs from a file (--input-file) |
Wget runs without a UI and keeps downloading after you log out.
See: Invoking
Output filename
| Flag | Effect |
|---|---|
-O file | Write the body to file |
-O - | Write the body to stdout |
--content-disposition | Use the server's suggested filename |
--restrict-file-names=windows | Escape unsafe path characters |
-O - streams; --content-disposition needs server support.
See: Download options
Quiet and verbose
| Flag | Effect |
|---|---|
-q | Suppress all output (--quiet) |
-v | Show headers and debug (--verbose) |
-nv | Between quiet and verbose |
-o log | Write messages to log |
-a log | Append to log |
-o overwrites; -a appends. Use -q in scripts.
Resume and timestamping
| Flag | Effect |
|---|---|
-c | Resume a partial file |
-N | Fetch only newer files |
--no-clobber | Skip files that exist |
-c, -N and -nc answer different "file exists" cases;
pick one rather than stacking them.
See: Download options
HTTP
Headers
| Flag | Effect |
|---|---|
--header='X-Api-Key: abc' | Add a request header |
--user-agent='Agent/1.0' | Set the User-Agent header |
Repeat --header for more headers; quote values with spaces.
See: HTTP options
POST data
| Flag | Effect |
|---|---|
--post-data='a=1&b=2' | Send a form-urlencoded body |
--post-file=body.txt | Send a file's contents as body |
--post-data sets the content type automatically.
See: HTTP options
Authentication
| Flag | Effect |
|---|---|
--user=alice | Username |
--password=secret | Password |
.netrc | Keep credentials out of the process list |
Command-line credentials appear in process listings; prefer
a .netrc entry with machine, login and password.
See: HTTP options
Cookies
| Flag | Effect |
|---|---|
--load-cookies=cookies.txt | Send cookies from a jar |
--save-cookies=jar.txt | Save cookies at the end |
--keep-session-cookies | Persist session cookies on save |
Netscape-format cookie jars work with both flags.
See: HTTP options
TLS
| Flag | Effect |
|---|---|
--no-check-certificate | Skip certificate verification |
--certificate=cert.pem | Send a client certificate (PEM) |
Verification is on by default; disable it only for controlled endpoints.
Recursive download
Recursive
| Flag | Effect |
|---|---|
-r -l 2 | Follow links two levels deep |
-l inf | Remove the depth cap |
-l counts hops from the start URL.
Mirror
| Flag | Effect |
|---|---|
-m | Recurse, timestamp, no depth cap (--mirror) |
-m also keeps FTP directory listings.
Page requisites
| Flag | Effect |
|---|---|
-p | Fetch images and CSS (--page-requisites) |
-k | Point links at local files |
-E | Add .html to responses |
Use -p -k to view one page offline with its assets.
Directory layout
| Flag | Effect |
|---|---|
-np | Do not ascend from the start URL |
--no-directories | Save all files into one directory |
-x | Keep the host path |
-np is the usual guard against crawling into a parent site.
See: Directory options
Filtering
Accept and reject
| Flag | Effect |
|---|---|
-A pdf | Keep only PDF files |
-R png,jpg | Skip images |
Patterns match filenames; separate several with commas.
Domains and hosts
| Flag | Effect |
|---|---|
--domains=example.com | Follow only this domain |
--exclude-domains=ads.example.com | Skip a specific host |
--domains includes subdomains and accepts comma-separated names.
Directory filters
| Flag | Effect |
|---|---|
-I css,img | Allow these paths |
-X private | Skip these paths |
Patterns match directory names in the URL path.
Link following
| Flag | Effect |
|---|---|
--follow-ftp | Follow FTP links found in HTML |
--span-hosts | Allow downloads from other hosts |
Pair --span-hosts with --domains to bound a cross-host crawl.
See: Following links
Rate limiting and retries
Retries
| Flag | Effect |
|---|---|
-t 3 | Retry each URL up to three times |
--retry-connrefused | Retry refused connections too |
Connection refusals are not retried unless enabled.
See: Download options
Wait and rate
| Flag | Effect |
|---|---|
--wait=2 | Pause two seconds between requests |
--random-wait | Vary the pause around --wait |
--limit-rate=200k | Cap download speed at 200 KB/s |
--random-wait scales the pause; k and m set rate units.
See: Download options
Timeouts
| Flag | Effect |
|---|---|
--timeout=30 | Set every timeout at once |
--dns-timeout=5 | DNS lookup limit |
--connect-timeout=10 | TCP connection limit |
--read-timeout=30 | Inactivity limit on reads |
--read-timeout measures inactivity, not total transfer time.
See: Download options
Quota and background
| Flag | Effect |
|---|---|
--quota=100m | Stop after 100 MB downloaded |
-b | Detach and keep downloading |
Background output goes to wget-log unless -o is set.
Examples
Common recipes
# Mirror a site for offline browsing
wget -m -k -p -np https://example.com/
# Resume an interrupted download
wget -c https://example.com/large.iso
# Download all PDFs from a page
wget -r -l 1 -A pdf -np https://example.com/docs/
# Fetch with a custom header and save the body
wget --header='Authorization: Bearer TOKEN' \
-O out.json https://api.example.com/data
# Check a URL without downloading
wget --spider -q https://example.com && echo OK
See: Examples
More recipes
# Post JSON with an explicit content type
wget --header='Content-Type: application/json' \
--post-data='{"name":"Ada"}' https://api.example.com/users
# Quiet download with a progress bar only
wget -q --show-progress https://example.com/file
# Limit download speed
wget --limit-rate=100k https://example.com/file
See: Examples
0 Comments for this cheatsheet. Write yours!