Shell Scripting

Welcome to the Shell Scripting Tutorial! On the left is a code editor; on the right is a real Linux terminal. Files you save in the editor are synced to /tutorial/ on the VM.

Why shell scripting?

Every time you repeat a task in the terminal — renaming files, checking server logs, running builds — you are a candidate for automation. A shell script captures those commands in a file so you can re-run, share, and schedule them without retyping anything.

Shell scripts are the foundation of CI/CD pipelines, Docker entrypoints, deployment scripts, and system administration. The skills you learn here transfer directly to real production workflows.

Two lines every script needs

Open morning.sh in the editor. It already has:

#!/bin/bash
set -e

Line 1 — the shebang (#!): When you run a file, Linux reads the first two bytes to decide how to execute it. #! followed by a path tells the OS which interpreter to use. Without it, the OS guesses — and often guesses wrong. Always write #!/bin/bash on line 1.

Line 2 — the safety net (set -e): By default, Bash happily continues running after a failed command. set -e stops the script immediately when something goes wrong, preventing a cascade of confusing failures. Always include it.

Your task

Add the following three lines below the existing ones:

echo "Good morning!"
echo "Today is $(date +%A)."
echo "You are logged in as: $(whoami)"

Then save (Ctrl+S / Cmd+S) and run in the terminal:

chmod +x morning.sh
./morning.sh

Breaking it down:

• chmod +x grants execute permission. Linux requires this explicit step before running a file as a program — a deliberate security feature so files don’t accidentally become executable.
• ./morning.sh — the ./ prefix means “look in the current directory.” The shell only searches directories listed in $PATH for commands; your local folder is not in $PATH by default.
• $(date +%A) is command substitution: the shell runs date +%A first, captures its output, and injects the result into your string. Any command can go inside $(...) — this is one of Bash’s most useful features.

The pipe operator | is one of the most powerful ideas in Unix. It connects programs so that the output of one becomes the input of the next, letting you build data-processing pipelines from small, single-purpose tools.

The Unix philosophy

Each Unix tool does one thing well:

• grep finds matching lines
• sort sorts lines alphabetically or numerically
• uniq removes consecutive duplicate lines
• wc counts lines, words, or characters
• cut extracts specific columns

Pipes let you compose them into powerful chains without writing a program. Data flows through memory from one process to the next — no intermediate files needed.

Explore the data

A file called server_log.txt is provided. Browse it first:

cat server_log.txt

Pipeline exercises

Type these yourself — building pipelines step by step trains you to think in terms of data flowing through stages.

# 1. Count total lines
wc -l < server_log.txt

# 2. Show only ERROR lines
grep "ERROR" server_log.txt

# 3. Count how many errors
grep "ERROR" server_log.txt | wc -l

# 4. Extract just the timestamps (first field)
cut -d' ' -f1 server_log.txt | head -5

# 5. Which message types appear most often?
cut -d' ' -f2 server_log.txt | sort | uniq -c | sort -rn | head -3

Read pipeline 5 left to right: extract column 2 → sort alphabetically (so duplicates become adjacent) → count consecutive duplicates → sort by count descending → show top 3. Each step is trivial; the combination is powerful.

Redirection: connecting commands to files

Pipes connect commands to commands. Redirection connects commands to files:

grep "ERROR" server_log.txt > errors_only.txt   # create/overwrite
echo "extra line" >> errors_only.txt             # append (safe)
wc -l < errors_only.txt                          # read from file

Note: > overwrites without warning. Use >> to append and preserve existing content.

Now create errors_only.txt by running the grep "ERROR" redirection above — the tests below will check it.

Variables store values for reuse. In Bash, you assign with = and read with $.

The spaces rule — easy to break, hard to debug

color="blue"      # correct
color = "blue"    # WRONG — shell sees three words: "color", "=", "blue"

There must be no spaces around =. The shell is a command language: color = "blue" looks like running a command named color with arguments = and blue. You will get a “command not found” error that can be confusing at first.

The quoting problem

When you write $variable, the shell replaces it with the value — then word-splits the result on spaces and runs glob expansion. This causes chaos when values contain spaces:

file="my report.txt"
wc -l $file      # shell splits into: wc -l my report.txt  (TWO args!)
wc -l "$file"    # correct: one argument, treated as a unit

Rule: always double-quote your variables unless you have a specific reason not to.

Single vs. double quotes

Use double quotes almost always. Use single quotes when you want a literal $ — for example, in awk programs or regex patterns.

See the bug

buggy.sh has a deliberate quoting mistake. Run it first:

bash buggy.sh

Read the error carefully — it shows exactly what happens when a space-containing variable is unquoted.

Fix it

1. Open buggy.sh in the editor.
2. Find the unquoted $filename on the wc -l line.
3. Wrap it in double quotes: "$filename".
4. Save and run again — the error should disappear.

Now build your own

Open inventory.sh and write a script that:

1. Stores a project name and a version number in variables.
2. Uses $(...) to capture the number of .sh files in the current directory (hint: ls *.sh | wc -l).
3. Prints a summary like: Project: mytools v1.0 — 3 scripts found

chmod +x inventory.sh
./inventory.sh

Scripts need to react to different situations. Bash’s if statement runs commands conditionally based on whether a test succeeds.

Syntax

if [ condition ]; then
    # runs when condition is true
elif [ other_condition ]; then
    # runs when first is false but this is true
else
    # runs when all conditions are false
fi

Why the spaces inside [ ] are mandatory

[ is not special syntax — it is an actual command (try which [). Like any command, its arguments must be separated by spaces:

[ -f "$file" ]    # correct: "[" receives "-f" and "$file" as args
[-f "$file"]      # WRONG: shell tries to run a command named "[-f"

Common tests

Important: use -eq, -lt, -gt for numbers; use = and != for strings. Mixing them gives wrong results silently.

Printing to stderr

Error messages should go to standard error (>&2), not standard output. This lets callers separate normal output from errors:

echo "Error: file not found" >&2
exit 1

Your task

Edit health_check.sh and fill in the three _____ placeholders:

1. Test if the file does not exist — hint: ! -f "$file".
2. Test if error_count is greater than 3.
3. Test if error_count is greater than 0.

chmod +x health_check.sh
./health_check.sh server_log.txt    # should report CRITICAL (4 errors)
./health_check.sh nonexistent.txt   # should print an error and exit 1

When you run ./script.sh one two three, the shell sets special variables automatically:

Always validate your arguments

A script that silently does the wrong thing when called incorrectly is worse than one that fails clearly. Good scripts check their inputs:

if [ $# -eq 0 ]; then
    echo "Usage: $0 <file1> [file2...]" >&2
    exit 1
fi

Using $0 in the usage message means it always shows the correct script name, even if the script is renamed.

Looping over arguments

"$@" expands to all arguments as separate, properly-quoted words:

for f in "$@"; do
    echo "Processing: $f"
done

Always quote "$@" — unquoted $@ has the same space-splitting problem as unquoted variables.

Your task

Write file_info.sh from scratch. It should:

1. Print a usage message and exit with code 1 if no arguments given.
2. Loop over all arguments with for f in "$@".
3. For each argument:
   • Print <name>: not found if it doesn’t exist.
   • Print <name>: directory if it’s a directory.
   • Print <name>: <N> lines otherwise (use wc -l < "$f").

Test with:

chmod +x file_info.sh
./file_info.sh server_log.txt morning.sh /tmp nope.txt

You should see a line count, a directory label, and a “not found” message — all from one call.

Loops eliminate repetition. Instead of writing the same logic ten times for ten files, you write it once and let the loop do the rest.

for loops — iterate over a list

for fruit in apple banana cherry; do
    echo "I like $fruit"
done

The list can be literal words, a glob, or command output:

for f in *.sh; do          # expands to all matching filenames
    echo "Found: $f"
done

while loops — repeat until a condition changes

counter=1
while [ $counter -le 5 ]; do
    echo "Step $counter"
    counter=$((counter + 1))
done

$(( )) is arithmetic expansion — it evaluates integer math expressions. Common operations: +, -, *, /, and $((total * 100 / count)) for integer percentages.

Accumulating totals

A common pattern is keeping running counts across loop iterations:

passed=0
failed=0
# ... inside loop:
passed=$((passed + 1))

Your task

Write batch_check.sh that:

1. Uses for f in *.sh to loop over every .sh file.
2. Reads the first line of each file with head -1 "$f" and checks whether it equals #!/bin/bash.
3. Prints ✓ filename if correct, ✗ filename (missing shebang) if not.
4. After the loop, prints: Checked N files: P passed, F failed

Tip: store the first line in a variable before comparing: first=$(head -1 "$f") Then: if [ "$first" = "#!/bin/bash" ]; then

chmod +x batch_check.sh
./batch_check.sh

As scripts grow, repeating the same logic in multiple places becomes a maintenance problem. Functions let you name a block of code and call it anywhere, changing it in one place when needed.

Defining and calling

greet() {
    local name="$1"
    echo "Hello, ${name}!"
}

greet "engineer"   # → Hello, engineer!
greet "world"      # → Hello, world!

Functions are defined first, then called by name — just like external commands, but without a separate file.

Key concepts

local variables: Without local, variables set inside a function leak into the rest of the script, overwriting outer variables with the same name. Always use local for variables that belong only to the function.

Arguments: Functions receive $1, $2, etc. independently of the script’s own arguments — each call gets its own set.

Return values: return 0 means success, return 1 means failure. This lets callers use functions directly in if:

if is_number "$input"; then
    echo "Valid number"
fi

Capturing output: Use $(func_name arg) to capture what a function prints, just like with any command:

upper=$(to_upper "hello")   # upper="HELLO"

Your task

Write toolkit.sh with three utility functions:

1. to_upper — converts its argument to uppercase. Hint: echo "$1" | tr '[:lower:]' '[:upper:]'
2. file_ext — prints the file extension of its argument. Hint: echo "${1##*.}" — ##*. strips everything up to the last dot.
3. is_number — exits 0 if the argument is a valid integer, 1 otherwise. Hint: [[ "$1" =~ ^-?[0-9]+$ ]] (a Bash regex test)

After defining the functions, add a demo section that calls each one and prints the results.

chmod +x toolkit.sh
./toolkit.sh

Three tools handle most text-processing work in shell scripts. Knowing when to reach for each one is the key skill.

Choosing the right tool

A useful heuristic: grep to filter rows, sed to edit them, awk to compute across them.

grep — search for patterns

grep "pattern" file          # Lines matching pattern
grep -i "pattern" file       # Case-insensitive
grep -c "pattern" file       # Count of matching lines
grep -v "pattern" file       # Lines NOT matching (invert)
grep -E "pat1|pat2" file     # Extended regex: OR

sed — stream editor

sed 's/old/new/' file        # Replace first match per line
sed 's/old/new/g' file       # Replace ALL matches (global flag)
sed '/pattern/d' file        # Delete matching lines
sed -n '5,10p' file          # Print only lines 5–10

The s/old/new/ syntax is a substitution expression. The / characters are delimiters — you can use any character if your pattern contains /: e.g. s|/old/path|/new/path|.

awk — column-oriented processing

awk splits each line into fields ($1, $2, …) and lets you run a program on each one:

awk '{print $1}' file              # First field of every line
awk -F, '{print $2}' file.csv      # Comma-delimited: field 2
awk '$3 == 404 {print $2}' file    # Print field 2 where field 3 is 404
awk '{sum += $4} END {print sum / NR}' file  # Average of field 4

The END block runs once after all lines — useful for totals and averages. NR is the total number of records (lines) processed.

Your task

access.log is provided. Write log_analysis.sh that:

1. Prints the total number of requests.
2. Shows the count of each HTTP method (GET, POST, etc.) — hint: awk '{print $1}' | sort | uniq -c
3. Lists all unique URLs that returned a 404 — hint: awk '$3 == 404 {print $2}'
4. Calculates the average response time from the last column — hint: strip the ms suffix with sed 's/ms//' before summing.

chmod +x log_analysis.sh
./log_analysis.sh

case — readable multi-way branching

When you need to check one variable against many possible values, a chain of if/elif becomes hard to scan. case is cleaner:

case "$input" in
    start)   echo "Starting..."  ;;
    stop)    echo "Stopping..."  ;;
    restart) echo "Restarting..." ;;
    *)       echo "Unknown: $input" ;;
esac

Each branch ends with ;;. The * pattern is the catch-all default. esac is case spelled backwards — a quirky Bash tradition.

case also supports glob patterns in branches:

case "$filename" in
    *.sh)       echo "Shell script" ;;
    *.py)       echo "Python file"  ;;
    *.txt|*.md) echo "Text file"    ;;
esac

Exit codes: the language of success and failure

Every command exits with a number. 0 always means success; any other value means failure. This is the opposite of most programming languages (where 0 is falsy) — it trips people up at first.

exit 0    # success
exit 1    # general error
exit 2    # misuse / wrong arguments (a common convention)

Exit codes are what make scripts composable. Other scripts and CI systems check your exit code to decide what to do next:

if ./health_check.sh logfile.txt; then
    echo "All clear"
else
    echo "Problems found"
fi

Your task

Write service.sh — a simulated service controller. Use functions for the start, stop, and status logic, then dispatch with a case statement on $1.

Requirements:

• start — prints Starting service..., exits 0.
• stop — prints Stopping service..., exits 0.
• status — checks if /tmp/my_service.pid exists. If yes: Service is running, exit 0. If no: Service is stopped, exit 1.
• restart — calls your stop and start functions.
• No input or unknown input — print usage to stderr, exit 2.

chmod +x service.sh
./service.sh start
./service.sh status
./service.sh        # should print usage

The shell expands patterns before commands ever run. Understanding this explains many surprising behaviors.

Globs vs. regular expressions — a common source of confusion

These look similar but work in completely different contexts:

Key insight: globs match filenames in the shell before the command runs. Regexes match text content inside a running program.

Glob patterns (file matching)

*.sh          # any filename ending in .sh
file?.txt     # file1.txt, fileA.txt (exactly one character)
[0-9]*        # anything starting with a digit

If no files match a glob, the shell passes the literal pattern string to the command — usually causing a “file not found” error.

Brace expansion

Brace expansion generates lists and happens before glob expansion. It does not require matching files to exist:

echo {A,B,C}              # → A B C
echo file{1..5}.txt       # → file1.txt file2.txt ... file5.txt
mkdir -p project/{src,tests,docs}   # three directories at once

Regular expressions in grep

Your task

Write organizer.sh that:

1. Creates test files using brace expansion: touch report_{jan,feb,mar}.csv log_{jan,feb,mar}.txt notes.md
2. Uses a for loop with a glob to list all .csv files.
3. Uses grep -E on access.log to print lines where the URL starts with /api/.
4. Uses grep -E on access.log to print lines with 4xx status codes (hint: 4[0-9]{2}).
5. Cleans up the test files with rm at the end.

chmod +x organizer.sh
./organizer.sh

You now have all the core building blocks. These final exercises give you less scaffolding — focus on designing the solution before writing any code.

Process substitution <(...)

Some commands expect filenames, not pipes. Process substitution makes the output of a command look like a file:

diff file1.txt file2.txt                       # compare two files
diff <(sort file1.txt) <(sort file2.txt)       # compare sorted versions

Without process substitution, you would create two temporary files, sort into them, diff them, then delete them. Process substitution handles all of that automatically.

tee — split a stream in two

tee writes its input to a file and passes it through to stdout:

grep "ERROR" server_log.txt | tee errors.txt | wc -l
#                             ↑ saves to file  ↑ also counts

Useful when you want to save an intermediate result for debugging without breaking the pipeline.

Reading diff output

When diff compares two inputs, lines prefixed with < come from the first source, and lines prefixed with > come from the second. This tells you which entries are unique to each side.

Your task

Write compare_logs.sh that accepts two log filenames as arguments:

1. Validate both files exist (exit with an error if not).
2. Print the line count of each file.
3. Use process substitution with diff to compare the sorted, unique error messages from each file. Extract just the message text (everything after the log level keyword).
4. Summarize which errors appear only in file 1 and only in file 2.

server_log2.txt is provided for testing:

chmod +x compare_logs.sh
./compare_logs.sh server_log.txt server_log2.txt

Time to combine everything into a real tool. No starter code — design the script yourself.

Challenge

Write monitor.sh — a log-monitoring tool that analyzes server_log.txt and produces a complete status report.

Requirements:

1. Accept an optional filename argument (default: server_log.txt). Validate it exists.
2. Print a header: === Log Monitor Report ===
3. Summary section:
   • Total entries
   • Count per log level (INFO, WARN, ERROR)
   • Time range (first and last timestamp)
4. Error details: for each ERROR line, print the timestamp and message in a readable format.
5. Recommendations: based on the data, print at least one actionable recommendation (e.g., “High error rate — investigate upstream services” if errors > 20% of total).
6. Exit with code 0 if no errors, code 1 if errors were found.

Design approach — plan before you code

Sketch your structure in comments first, then fill it in:

1. What data do I need? (counts, timestamps, error lines)
2. Which tool extracts each piece?
   • grep -c for counts per level
   • head -1 / tail -1 for first and last timestamps
   • grep "ERROR" piped to awk or cut for error details
   • awk for integer percentage: $((errors * 100 / total))
3. What functions make sense? e.g. print_summary, print_errors, print_recommendations

chmod +x monitor.sh
./monitor.sh

You have learned variables, quoting, pipes, conditionals, loops, functions, text processing, and exit codes. Now combine them all into a production-quality tool.

The challenge

Build dashboard.sh — a system health dashboard that gathers information and produces a formatted report. No starter code. No hints. Design it yourself.

Parsing flags — the while/case pattern

Real command-line tools accept flags like -f file and -o output. The idiomatic Bash pattern uses a while loop with case:

while [ $# -gt 0 ]; do
    case "$1" in
        -f) logfile="$2"; shift 2 ;;
        -o) outfile="$2"; shift 2 ;;
        -h) usage; exit 0 ;;
        *)  echo "Unknown option: $1" >&2; exit 2 ;;
    esac
done

shift N discards the first N positional parameters, sliding subsequent arguments into $1, $2, etc. for the next iteration.

Minimum requirements

1. A usage() function printed with -h.
2. Flag parsing: -f <logfile>, -o <outfile>, -h.
3. System info section: hostname, current date/time, disk usage.
4. Log analysis section:
   • Entry counts per level (INFO, WARN, ERROR)
   • Top 3 most frequent messages (strip timestamps)
   • Error rate as a percentage
5. Health verdict:
   • HEALTHY — 0 errors (exit 0)
   • DEGRADED — error rate < 30% (exit 1)
   • CRITICAL — error rate ≥ 30% (exit 2)
6. If -o is given, write the report to that file; otherwise print to stdout.

chmod +x dashboard.sh
./dashboard.sh
./dashboard.sh -f server_log.txt -o report.txt
cat report.txt