Getting started

Introduction

GNU Wget downloads files from the web non-interactively over HTTP, HTTPS, and FTP. This reference covers common options and recipes, current through Wget 1.25.

Basic download

SyntaxEffect
wget URLSave to the current directory
-i urls.txtRead URLs from a file (--input-file)

Wget runs without a UI and keeps downloading after you log out.

See: Invoking

Output filename

FlagEffect
-O fileWrite the body to file
-O -Write the body to stdout
--content-dispositionUse the server's suggested filename
--restrict-file-names=windowsEscape unsafe path characters

-O - streams; --content-disposition needs server support.

See: Download options

Quiet and verbose

FlagEffect
-qSuppress all output (--quiet)
-vShow headers and debug (--verbose)
-nvBetween quiet and verbose
-o logWrite messages to log
-a logAppend to log

-o overwrites; -a appends. Use -q in scripts.

See: Logging and input file options

Resume and timestamping

FlagEffect
-cResume a partial file
-NFetch only newer files
--no-clobberSkip files that exist

-c, -N and -nc answer different "file exists" cases; pick one rather than stacking them.

See: Download options

HTTP

Headers

FlagEffect
--header='X-Api-Key: abc'Add a request header
--user-agent='Agent/1.0'Set the User-Agent header

Repeat --header for more headers; quote values with spaces.

See: HTTP options

POST data

FlagEffect
--post-data='a=1&b=2'Send a form-urlencoded body
--post-file=body.txtSend a file's contents as body

--post-data sets the content type automatically.

See: HTTP options

Authentication

FlagEffect
--user=aliceUsername
--password=secretPassword
.netrcKeep credentials out of the process list

Command-line credentials appear in process listings; prefer a .netrc entry with machine, login and password.

See: HTTP options

Cookies

FlagEffect
--load-cookies=cookies.txtSend cookies from a jar
--save-cookies=jar.txtSave cookies at the end
--keep-session-cookiesPersist session cookies on save

Netscape-format cookie jars work with both flags.

See: HTTP options

TLS

FlagEffect
--no-check-certificateSkip certificate verification
--certificate=cert.pemSend a client certificate (PEM)

Verification is on by default; disable it only for controlled endpoints.

See: HTTPS (SSL/TLS) options

Recursive download

Recursive

FlagEffect
-r -l 2Follow links two levels deep
-l infRemove the depth cap

-l counts hops from the start URL.

See: Recursive retrieval options

Mirror

FlagEffect
-mRecurse, timestamp, no depth cap (--mirror)

-m also keeps FTP directory listings.

See: Recursive retrieval options

Page requisites

FlagEffect
-pFetch images and CSS (--page-requisites)
-kPoint links at local files
-EAdd .html to responses

Use -p -k to view one page offline with its assets.

See: Recursive retrieval options

Directory layout

FlagEffect
-npDo not ascend from the start URL
--no-directoriesSave all files into one directory
-xKeep the host path

-np is the usual guard against crawling into a parent site.

See: Directory options

Filtering

Accept and reject

FlagEffect
-A pdfKeep only PDF files
-R png,jpgSkip images

Patterns match filenames; separate several with commas.

See: Recursive accept/reject options

Domains and hosts

FlagEffect
--domains=example.comFollow only this domain
--exclude-domains=ads.example.comSkip a specific host

--domains includes subdomains and accepts comma-separated names.

See: Recursive accept/reject options

Directory filters

FlagEffect
-I css,imgAllow these paths
-X privateSkip these paths

Patterns match directory names in the URL path.

See: Directory-based limits

FlagEffect
--follow-ftpFollow FTP links found in HTML
--span-hostsAllow downloads from other hosts

Pair --span-hosts with --domains to bound a cross-host crawl.

See: Following links

Rate limiting and retries

Retries

FlagEffect
-t 3Retry each URL up to three times
--retry-connrefusedRetry refused connections too

Connection refusals are not retried unless enabled.

See: Download options

Wait and rate

FlagEffect
--wait=2Pause two seconds between requests
--random-waitVary the pause around --wait
--limit-rate=200kCap download speed at 200 KB/s

--random-wait scales the pause; k and m set rate units.

See: Download options

Timeouts

FlagEffect
--timeout=30Set every timeout at once
--dns-timeout=5DNS lookup limit
--connect-timeout=10TCP connection limit
--read-timeout=30Inactivity limit on reads

--read-timeout measures inactivity, not total transfer time.

See: Download options

Quota and background

FlagEffect
--quota=100mStop after 100 MB downloaded
-bDetach and keep downloading

Background output goes to wget-log unless -o is set.

See: Basic startup options

Examples

Common recipes

# Mirror a site for offline browsing
wget -m -k -p -np https://example.com/
# Resume an interrupted download
wget -c https://example.com/large.iso
# Download all PDFs from a page
wget -r -l 1 -A pdf -np https://example.com/docs/
# Fetch with a custom header and save the body
wget --header='Authorization: Bearer TOKEN' \
  -O out.json https://api.example.com/data
# Check a URL without downloading
wget --spider -q https://example.com && echo OK

See: Examples

More recipes

# Post JSON with an explicit content type
wget --header='Content-Type: application/json' \
  --post-data='{"name":"Ada"}' https://api.example.com/users
# Quiet download with a progress bar only
wget -q --show-progress https://example.com/file
# Limit download speed
wget --limit-rate=100k https://example.com/file

See: Examples

Also see

0 Comments for this cheatsheet. Write yours!