Getting started
Introduction
awk is a line-by-line text processor built into every Unix. Each record is matched against patterns, and each pattern runs an action.
This sheet uses portable awk, noting where GNU awk adds more.
- GNU awk manual (gnu.org)
- POSIX awk (pubs.opengroup.org)
- awk(1) (man7.org)
Invocation
awk '{ print $1 }' file.log # First column
awk -F: '{ print $1 }' /etc/passwd
awk -v limit=10 '$3 > limit' data.tsv
awk -f report.awk data.tsv
-F sets the field separator and -v defines a variable before the
program runs.
See: Invoking awk
Program shape
BEGIN { ... } # Once, before any input
/pattern/ { ... } # Lines matching the regex
$3 > 100 { ... } # Lines matching a condition
NR == 1 # Pattern only: print the line
{ print $2 } # Action only: every line
END { ... } # Once, after all input
A program is a list of pattern { action } rules; either part is optional.
See: Patterns and actions
Ranges
awk '/BEGIN/,/END/' file # Between two markers
awk '/start/ { on = 1 } on { print } /end/ { on = 0 }'
A comma pattern turns on at the first match and off after the second.
See: Range patterns
Options
-F fs # Field separator
-v var=value # Set a variable
-f prog.awk # Read the program from a file
--posix # Strict POSIX mode (gawk)
--version # Version (gawk)
gawk -i inplace edits files in place, but it is a GNU extension.
See: Command-line options
Fields
$0 and $n
awk '{ print $0 }' file # The whole line
awk '{ print $1, $NF }' file # First and last field
awk '{ print $(NF-1) }' file # Second to last
Fields are split on FS before the action runs; $0 is the whole record.
See: Fields
FS and OFS
awk -F, '{ print $2 }' file.csv
awk 'BEGIN { FS = ","; OFS = "\t" } { $1 = $1; print }'
Assigning a field rebuilds $0 with OFS, which is the usual
CSV-to-TSV trick. The default FS = " " splits on runs of spaces and
tabs.
See: Field separators
RS and ORS
awk 'BEGIN { RS = "" } { print NF }' # Paragraph mode
gawk 'BEGIN { RS = "\n\n"; ORS = "\n--\n" } { print }'
RS splits input into records and ORS terminates each print.
See: Record separators
Variables
Built-in variables
| Variable | Holds |
|---|---|
$0 | The current record |
$1 … $NF | Fields |
NF | Number of fields |
NR | Records read so far |
FNR | Records in the current file |
FILENAME | Current file name |
FS / OFS | Input / output field separator |
RS / ORS | Input / output record separator |
RSTART / RLENGTH | match() result |
SUBSEP | Key separator for a[i,j] |
ENVIRON["HOME"] | Environment variables |
ARGV and ARGC are standard; ARGIND is a gawk addition.
See: Built-in variables
Setting variables
awk -v n=3 'NR <= n' file
awk '{ total += $2 } END { print total }' file
awk 'BEGIN { OFS = ":" } { $1 = $1; print }' file
-v sets a variable before BEGIN; an assignment after the program sets
one while reading.
See: Variables
Strings
sub and gsub
awk '{ sub(/foo/, "bar"); print }' file # First match
awk '{ gsub(/[0-9]+/, "N"); print }' file # Every match
awk '{ gsub(/a/, "&!"); print }' file # & is the match
sub and gsub edit $0 unless you pass a target variable.
See: String functions
split
awk '{ n = split($0, parts, ","); print n, parts[1] }' file
split fills an array from 1 and returns its length.
See: split
substr and index
awk '{ print substr($0, 1, 10) }' file # First 10 chars
awk '{ print substr($0, 5) }' file # From char 5
awk '{ print index($0, "error") }' file # 0 if absent
String positions start at 1, and length($0) counts characters.
See: substr
match and case
awk '{ if (match($0, /[0-9]+/)) print RSTART, RLENGTH }' file
awk '{ print toupper($1), tolower($2) }' file
match() sets RSTART and RLENGTH; ~ tests a regex without setting
them.
See: match
Numbers and printf
printf formats
awk '{ printf "%-10s %5.2f\n", $1, $2 }' file
awk 'BEGIN { printf "%03d %x %c\n", 7, 255, 65 }'
awk '{ s = sprintf("%s: %d", $1, $2); print s }' file
printf takes a format and a comma-separated list; print separates its
arguments with OFS and ends with ORS.
See: Printf
Math functions
awk '{ sum += $1 } END { print sum, sum/NR }' file
awk 'BEGIN { print int(3.9), sqrt(16), 2^10 }'
awk 'BEGIN { srand(); print int(rand() * 100) }'
int() truncates toward zero; % is modulo; ^ means power.
See: Numeric functions
Strings and numbers
awk '{ print $1 + 0 }' file # Coerce to a number
awk 'BEGIN { print "10" < "9" }' # String compare: 1
awk 'BEGIN { print 10 < 9 }' # Numeric compare: 0
A field is a numeric string when it looks like a number; +0 forces a
number.
See: Conversion
Arrays
Counting
awk '{ count[$1]++ } END { for (k in count) print count[k], k }' file
awk '$0 in seen { next } { seen[$0] = 1; print }' file
Array keys are strings, and a missing entry reads as 0.
See: Arrays
Iteration
gawk 'BEGIN { PROCINFO["sorted_in"] = "@ind_str_asc" }
{ count[$1]++ }
END { for (k in count) print k }' file
for (k in a) visits keys in unspecified order unless gawk's
sorted_in is set.
See: Array traversal
Delete and multiple keys
awk '{ delete count[$1] }' file
awk '{ total[$1, $2] += $3 } END { for (k in total) print k }'
awk 'BEGIN { a["x", "y"] = 1; if (("x", "y") in a) print "yes" }'
a[i,j] joins the keys with SUBSEP. delete a clears the whole array
in gawk.
See: Delete
Control flow
if and else
awk '{ if ($3 > 100) print "high"; else print "low" }' file
awk '$3 > 100 && $4 ~ /ok/ { count++ } END { print count }' file
Patterns are conditions, so simple filters need no if.
See: Control statements
Loops
awk '{ for (i = 1; i <= NF; i++) print i, $i }' file
awk 'BEGIN { while ((getline line < "file") > 0) print line }'
break and continue work as they do in C.
See: While statement
next and exit
awk '/^#/ { next } { print }' file
awk 'NR == 10 { exit } { print }' file
next skips the rest of the record; exit runs END unless it is
already in END.
See: Next statement
One-liners
Lines and columns
awk 'NR >= 2 && NR <= 5' file # Lines 2 to 5
awk 'NF' file # Skip blank lines
awk '{ print NR": "$0 }' file # Number lines
awk '!seen[$0]++' file # Drop duplicates
NF is false on a blank line, so awk 'NF' filters them out.
See: Simple examples
Summaries
awk '{ s += $1 } END { print s }' file
awk '{ s += $1 } END { print s/NR }' file
awk '{ c[$1]++ } END { for (k in c) print c[k], k }' file
An unset variable is empty, which counts as 0 in arithmetic.
See: END
Two files
awk 'NR == FNR { seen[$0] = 1; next }
$0 in seen' a.txt b.txt # Lines in both files
NR == FNR holds while the first file is read; an empty first file breaks
the trick.
See: FNR
Output
Redirection
awk '{ print > "out.txt" }' file
awk '{ print >> "out.txt" }' file
awk '{ print > "/dev/stderr" }' file
> truncates on first use and appends afterwards; >> always appends.
See: Redirection
Pipes
awk '{ print $1 }' file | sort | uniq -c | sort -rn
awk '{ print $1 | "sort -u" }' file
A pipe inside the program stays open; the shell pipe is usually simpler.
See: Pipes
0 Comments for this cheatsheet. Write yours!