Getting started

Introduction

awk is a line-by-line text processor built into every Unix. Each record is matched against patterns, and each pattern runs an action.

This sheet uses portable awk, noting where GNU awk adds more.

Invocation

awk '{ print $1 }' file.log       # First column
awk -F: '{ print $1 }' /etc/passwd
awk -v limit=10 '$3 > limit' data.tsv
awk -f report.awk data.tsv

-F sets the field separator and -v defines a variable before the program runs.

See: Invoking awk

Program shape

BEGIN      { ... }        # Once, before any input
/pattern/  { ... }        # Lines matching the regex
$3 > 100   { ... }        # Lines matching a condition
NR == 1                   # Pattern only: print the line
           { print $2 }   # Action only: every line
END        { ... }        # Once, after all input

A program is a list of pattern { action } rules; either part is optional.

See: Patterns and actions

Ranges

awk '/BEGIN/,/END/' file           # Between two markers
awk '/start/ { on = 1 } on { print } /end/ { on = 0 }'

A comma pattern turns on at the first match and off after the second.

See: Range patterns

Options

-F fs          # Field separator
-v var=value   # Set a variable
-f prog.awk    # Read the program from a file
--posix        # Strict POSIX mode (gawk)
--version      # Version (gawk)

gawk -i inplace edits files in place, but it is a GNU extension.

See: Command-line options

Fields

$0 and $n

awk '{ print $0 }' file            # The whole line
awk '{ print $1, $NF }' file       # First and last field
awk '{ print $(NF-1) }' file       # Second to last

Fields are split on FS before the action runs; $0 is the whole record.

See: Fields

FS and OFS

awk -F, '{ print $2 }' file.csv
awk 'BEGIN { FS = ","; OFS = "\t" } { $1 = $1; print }'

Assigning a field rebuilds $0 with OFS, which is the usual CSV-to-TSV trick. The default FS = " " splits on runs of spaces and tabs.

See: Field separators

RS and ORS

awk 'BEGIN { RS = "" } { print NF }'     # Paragraph mode
gawk 'BEGIN { RS = "\n\n"; ORS = "\n--\n" } { print }'

RS splits input into records and ORS terminates each print.

See: Record separators

Variables

Built-in variables

VariableHolds
$0The current record
$1 … $NFFields
NFNumber of fields
NRRecords read so far
FNRRecords in the current file
FILENAMECurrent file name
FS / OFSInput / output field separator
RS / ORSInput / output record separator
RSTART / RLENGTHmatch() result
SUBSEPKey separator for a[i,j]
ENVIRON["HOME"]Environment variables

ARGV and ARGC are standard; ARGIND is a gawk addition.

See: Built-in variables

Setting variables

awk -v n=3 'NR <= n' file
awk '{ total += $2 } END { print total }' file
awk 'BEGIN { OFS = ":" } { $1 = $1; print }' file

-v sets a variable before BEGIN; an assignment after the program sets one while reading.

See: Variables

Strings

sub and gsub

awk '{ sub(/foo/, "bar"); print }' file     # First match
awk '{ gsub(/[0-9]+/, "N"); print }' file   # Every match
awk '{ gsub(/a/, "&!"); print }' file       # & is the match

sub and gsub edit $0 unless you pass a target variable.

See: String functions

split

awk '{ n = split($0, parts, ","); print n, parts[1] }' file

split fills an array from 1 and returns its length.

See: split

substr and index

awk '{ print substr($0, 1, 10) }' file     # First 10 chars
awk '{ print substr($0, 5) }' file         # From char 5
awk '{ print index($0, "error") }' file    # 0 if absent

String positions start at 1, and length($0) counts characters.

See: substr

match and case

awk '{ if (match($0, /[0-9]+/)) print RSTART, RLENGTH }' file
awk '{ print toupper($1), tolower($2) }' file

match() sets RSTART and RLENGTH; ~ tests a regex without setting them.

See: match

Numbers and printf

printf formats

awk '{ printf "%-10s %5.2f\n", $1, $2 }' file
awk 'BEGIN { printf "%03d %x %c\n", 7, 255, 65 }'
awk '{ s = sprintf("%s: %d", $1, $2); print s }' file

printf takes a format and a comma-separated list; print separates its arguments with OFS and ends with ORS.

See: Printf

Math functions

awk '{ sum += $1 } END { print sum, sum/NR }' file
awk 'BEGIN { print int(3.9), sqrt(16), 2^10 }'
awk 'BEGIN { srand(); print int(rand() * 100) }'

int() truncates toward zero; % is modulo; ^ means power.

See: Numeric functions

Strings and numbers

awk '{ print $1 + 0 }' file          # Coerce to a number
awk 'BEGIN { print "10" < "9" }'     # String compare: 1
awk 'BEGIN { print 10 < 9 }'         # Numeric compare: 0

A field is a numeric string when it looks like a number; +0 forces a number.

See: Conversion

Arrays

Counting

awk '{ count[$1]++ } END { for (k in count) print count[k], k }' file
awk '$0 in seen { next } { seen[$0] = 1; print }' file

Array keys are strings, and a missing entry reads as 0.

See: Arrays

Iteration

gawk 'BEGIN { PROCINFO["sorted_in"] = "@ind_str_asc" }
      { count[$1]++ }
      END { for (k in count) print k }' file

for (k in a) visits keys in unspecified order unless gawk's sorted_in is set.

See: Array traversal

Delete and multiple keys

awk '{ delete count[$1] }' file
awk '{ total[$1, $2] += $3 } END { for (k in total) print k }'
awk 'BEGIN { a["x", "y"] = 1; if (("x", "y") in a) print "yes" }'

a[i,j] joins the keys with SUBSEP. delete a clears the whole array in gawk.

See: Delete

Control flow

if and else

awk '{ if ($3 > 100) print "high"; else print "low" }' file
awk '$3 > 100 && $4 ~ /ok/ { count++ } END { print count }' file

Patterns are conditions, so simple filters need no if.

See: Control statements

Loops

awk '{ for (i = 1; i <= NF; i++) print i, $i }' file
awk 'BEGIN { while ((getline line < "file") > 0) print line }'

break and continue work as they do in C.

See: While statement

next and exit

awk '/^#/ { next } { print }' file
awk 'NR == 10 { exit } { print }' file

next skips the rest of the record; exit runs END unless it is already in END.

See: Next statement

One-liners

Lines and columns

awk 'NR >= 2 && NR <= 5' file      # Lines 2 to 5
awk 'NF' file                      # Skip blank lines
awk '{ print NR": "$0 }' file      # Number lines
awk '!seen[$0]++' file             # Drop duplicates

NF is false on a blank line, so awk 'NF' filters them out.

See: Simple examples

Summaries

awk '{ s += $1 } END { print s }' file
awk '{ s += $1 } END { print s/NR }' file
awk '{ c[$1]++ } END { for (k in c) print c[k], k }' file

An unset variable is empty, which counts as 0 in arithmetic.

See: END

Two files

awk 'NR == FNR { seen[$0] = 1; next }
     $0 in seen' a.txt b.txt       # Lines in both files

NR == FNR holds while the first file is read; an empty first file breaks the trick.

See: FNR

Output

Redirection

awk '{ print > "out.txt" }' file
awk '{ print >> "out.txt" }' file
awk '{ print > "/dev/stderr" }' file

> truncates on first use and appends afterwards; >> always appends.

See: Redirection

Pipes

awk '{ print $1 }' file | sort | uniq -c | sort -rn
awk '{ print $1 | "sort -u" }' file

A pipe inside the program stays open; the shell pipe is usually simpler.

See: Pipes

0 Comments for this cheatsheet. Write yours!