CS 35L

---

Current CS35L Deck

Includes all flash cards taught until today

You need to see a list of all the files and folders in your current directory. What command do you use?

ls

The ls command lists directory contents. You can also use flags like ls -l for a detailed list or ls -a to see hidden files.

You are currently in your home directory and need to navigate into a folder named ‘Documents’. Which command achieves this?

cd Documents

The cd (change directory) command is used to change the current working directory in the command line operating system.

You want to quickly view the entire contents of a small text file named ‘config.txt’ printed directly to your terminal screen.

cat config.txt

The cat (concatenate) command reads data from the file and gives its content as output. It is widely used to display file contents.

You need to find every line containing the word ‘ERROR’ inside a massive log file called ‘server.log’.

grep 'ERROR' server.log

grep searches for patterns in each file. It prints each line that matches the given pattern, making it invaluable for parsing logs.

You wrote a new bash script named ‘script.sh’, but when you try to run it, you get a ‘Permission denied’ error. How do you make the file executable?

chmod +x script.sh

The chmod command changes file modes or Access Control Lists. The +x flag specifically adds execute permissions to the file.

You want to rename a file from ‘draft_v1.txt’ to ‘final_version.txt’ without creating a copy.

mv draft_v1.txt final_version.txt

The mv command is used to move files or directories, but it is also the standard command used to rename a file by moving it to a new name in the same directory.

You are starting a new project and need to create a brand new, empty folder named ‘src’ in your current location.

mkdir src

mkdir stands for ‘make directory’. It creates a new directory with the specified name, provided you have the correct permissions.

You want to view the contents of a very long text file called ‘manual.txt’ one page at a time so you can scroll through it.

less manual.txt

The less command is a terminal pager program that allows viewing (but not editing) the contents of a text file one screen at a time, allowing both forward and backward navigation.

You need to create an exact duplicate of a file named ‘report.pdf’ and save it as ‘report_backup.pdf’.

cp report.pdf report_backup.pdf

The cp (copy) command is used to copy files or directories from one location to another.

You have a temporary file called ‘temp_data.csv’ that you no longer need and want to permanently delete from your system.

rm temp_data.csv

The rm (remove) command deletes files or directories. Use with caution, as deleted files are generally not recoverable.

You want to quickly print the phrase ‘Hello World’ to the terminal or pass that string into a pipeline.

echo 'Hello World'

The echo command outputs the strings it is being passed as arguments. It is commonly used in scripts to display messages or write data to a file using redirection.

You want to know exactly how many lines are contained within a file named ‘essay.txt’.

wc -l essay.txt

The wc (word count) command prints newline, word, and byte counts for a file. Adding the -l flag restricts the output to only the line count.

You need to perform an automated find-and-replace operation on a stream of text to change the word ‘apple’ to ‘orange’.

sed 's/apple/orange/g'

The sed (stream editor) s: Stands for ‘substitute’. /: The delimiter that separates the search and replacement terms. apple: The string you want to find. orange: The string you want to replace it with. g: Stands for ‘global’. This tells sed to replace every instance of ‘apple’ on a line, rather than just the first one it finds.

You have a space-separated log file and want a tool to extract and print only the 3rd column of data.

awk '{print $3}'

awk is a versatile command-line utility and programming language designed for text processing and data extraction, particularly excelling at handling structured, column-based data.

You want to store today’s date (formatted as YYYY-MM-DD) in a variable called TODAY so you can use it to name a backup file dynamically.

TODAY=$(date +%Y-%m-%d)

Command substitution $(...) executes the inner command in a subshell and replaces itself with the command’s standard output. This is the standard way to capture the result of a command into a variable.

A variable FILE holds the value my report.pdf. Running rm $FILE fails with a ‘No such file or directory’ error for both ‘my’ and ‘report.pdf’. How do you fix this?

rm "$FILE"

Without quotes, the shell performs word splitting on the expanded variable, breaking my report.pdf into two separate arguments: my and report.pdf. Wrapping the variable in double quotes ("$FILE") prevents word splitting and passes the entire value as a single argument.

You are writing a script that requires exactly two arguments. How do you check how many arguments were passed to the script so you can print a usage error if the count is wrong?

$#

$# expands to the total count of positional arguments passed to the script. A typical guard looks like: if [ $# -ne 2 ]; then echo 'Usage: script.sh src dest'; exit 1; fi. Other useful special variables: $1, $2 (individual args), $0 (script name), $@ (all args).

You want to create a directory called ‘build’ and then immediately run cmake .. inside it, but only if the directory creation succeeded — all in a single command.

mkdir build && cd build && cmake ..

The && (AND) operator runs the next command only if the previous one succeeded (exit code 0). If mkdir fails, neither cd nor cmake will execute. This is a concise alternative to a full if/then/fi block for simple success-guarded sequences.

At the start of a script, you need to change into /deploy/target. If that directory doesn’t exist, the script must abort immediately — write a defensive one-liner.

cd /deploy/target || exit 1

The || (OR) operator runs the right-hand command only if the left-hand command fails (non-zero exit code). This is the standard idiom for aborting a script when a critical precondition is not met, without needing a full if statement.

You want to delete all files ending in .tmp in the current directory using a single command, without listing each filename explicitly.

rm *.tmp

The * wildcard is a glob pattern — it is expanded by the shell itself (filename expansion) before rm runs. Globbing is entirely separate from regular expressions: in RegEx, * means ‘zero or more of the preceding character’, not ‘match anything’.

You have some uncommitted, incomplete changes in your working directory, but you need to switch to another branch to urgently fix a bug. How do you temporarily save your current work without making a messy commit?

git stash

git stash temporarily shelves your staged and unstaged changes for tracked files. To include brand new, untracked files in the stash, you must use git stash -u.

You know a bug was introduced recently, but you aren’t sure which commit caused it. How do you perform a binary search through your commit history to find the exact commit that broke the code?

git bisect

git bisect systematically halves your commit history, asking you to test if the bug is present at each step, making it highly efficient for tracking down regressions.

You are looking at a file and want to know exactly who last modified a specific line of code, and in which commit they did it.

git blame

git blame annotates each line of a file with the author and the commit hash of the last modification, which is very useful for figuring out who to ask about a piece of code.

You want to safely ‘undo’ a previous commit that introduced an error, but you don’t want to rewrite history or force-push. How do you create a new commit with the exact inverse changes?

git revert

git revert is the safest way to undo changes on a shared branch because it adds a new commit that nullifies the targeted commit, rather than deleting history.

You want to see exactly what has changed in your working directory compared to your last saved snapshot (the most recent commit).

git diff HEAD

Using git diff HEAD compares your current modified tracked files on disk with the most recent commit. Note that it does not show completely untracked files.

You have a feature branch with several experimental commits, but you only want to move one specific, completed commit over to your main branch.

git cherry-pick

git cherry-pick allows you to selectively grab a specific commit from one branch and apply it onto your current branch, without merging the entire branch history.

You want to integrate a feature branch into main, but instead of bringing over all 15 tiny incremental commits, you want them combined into one clean commit on the main branch.

git merge --squash

Squashing combines all the patches from the feature branch into a single set of changes, allowing you to make one clean commit on the target branch and keep the history tidy with git merge --squash.

You are building a massive project and want to include an entirely separate external Git repository as a subdirectory within your project, while keeping its history independent.

git submodule

git submodule allows you to nest an independent repository inside your main project. It essentially creates a pointer to a specific commit of that external library.

You are starting a brand new project in an empty folder on your computer and want Git to start tracking changes in this directory.

git init

This command initializes a new, empty Git repository in the current directory via git init, creating the hidden .git folder that manages all version control data.

You have just installed Git on a new computer and need to set up your username and email address so that your commits are properly attributed to you.

git config

Using git config --global user.name and user.email establishes your default identity, which can be overridden for specific projects using git config --local.

You’ve made changes to three different files, but you only want two of them to be included in your next snapshot. How do you move those specific files to the staging area?

git add <filename>

The git add command moves file modifications from the working directory into the staging area (index) to prepare them for the next commit.

You’ve lost track of what you’ve been doing. You want a quick overview of which files are modified, which are staged, and which are completely untracked by Git.

git status

This command git status provides a high-level summary of the working tree and staging area statuses, making it one of the most frequently used Git commands.

You have staged all the files for a completed feature and are ready to permanently save this snapshot to your local repository’s history with a descriptive message.

git commit -m 'message'

Committing takes everything in the staging area and records it as a permanent new version in your local Git history via git commit.

You want to review the chronological history of all past commits on your current branch, including their author, date, and commit message.

git log

This command git log prints out the commit history. You can also add flags like -p to see the exact patch/diff introduced in each commit.

You’ve made edits to a file but haven’t staged it yet. You want to see the exact lines of code you added or removed compared to what is currently in the staging area.

git diff

Running git diff without any arguments compares your current working directory against the staging area.

You want to start working on a completely new feature in isolation without affecting the main codebase.

git branch <branch-name>

This command git branch creates a new branch pointer at your current commit, allowing you to diverge from the main line of development.

You are currently on your feature branch and need to switch your working directory back to the ‘main’ branch.

git switch main

The modern git switch command navigates between branches, updating your working directory and moving HEAD. The legacy equivalent git checkout main still works and is widely seen in older documentation and scripts.

Your feature branch is complete, and you want to integrate its entire commit history into your current ‘main’ branch.

git merge <branch-name>

Merging with git merge takes the divergent histories of two branches and combines them, often resulting in a new ‘merge commit’ with two parents.

Instead of creating a merge commit, you want to take the commits from your feature branch and re-apply them directly on top of the latest ‘main’ branch to create a clean, linear history.

git rebase main

Rebasing with git rebase rewrites history by moving the base of your current branch to the tip of another, which creates a cleaner history but should be avoided on public/shared branches.

You want to start working on an open-source project hosted on GitHub. How do you download a full local copy of that repository to your machine?

git clone <url>

Cloning with git clone downloads the entire repository, including its full history and all branches, from a remote server to your local disk.

Your team members have uploaded new commits to the shared remote repository. You want to fetch those changes and immediately integrate them into your current local branch.

git pull

The git pull command is effectively a combination of git fetch and git merge (the default) or git rebase (if configured), integrating remote changes into your local branch.

You have finished making several commits locally and want to upload them to the remote GitHub repository so your team can see them.

git push

git push transmits your local commits to the remote server, updating the remote branch to match your local branch.

You have a specific commit hash and want to see detailed information about it, including the commit message, author, and the exact code diff it introduced.

git show <commit-hash>

git show displays the details of a specific Git object, most commonly used to inspect exactly what changed in a single commit.

You want to start working on a new feature in isolation. How do you create a new branch called ‘feature-auth’ and immediately switch to it in a single command?

git switch -c feature-auth

git switch -c creates a new branch and switches to it in one step. The -c flag stands for ‘create’. The legacy equivalent is git checkout -b feature-auth, which you will still encounter in older scripts and documentation.

You accidentally staged a file you didn’t intend to include in your next commit. How do you move it back to the working directory without losing your modifications?

git restore --staged <filename>

git restore --staged unstages a file by copying the version from the last commit back into the staging area, leaving your working directory modifications completely untouched. It is the modern equivalent of the older git reset HEAD <file>.

You made some experimental changes to a file but want to discard them entirely and revert to the version from your last commit.

git restore <filename>

git restore <file> discards unstaged changes in the working directory, reverting it to the staged version (or the last commit if nothing is staged). To explicitly revert to the last commit, use git restore --source=HEAD <file>.

You merge a feature branch into main, and Git performs the merge without creating a new merge commit — it simply moves the ‘main’ pointer forward. What type of merge is this, and when does it occur?

A fast-forward merge — occurs when ‘main’ has no new commits since the feature branch was created.

A fast-forward merge happens when the target branch hasn’t diverged. Git can simply advance its pointer in a straight line to the tip of the incoming branch, keeping the commit history perfectly linear. If both branches have diverged, Git performs a three-way merge and creates a merge commit instead.

You want to safely inspect the codebase at a specific older commit without modifying any branch. How do you do this?

git switch --detach <commit-hash>

git switch --detach places you in ‘detached HEAD’ state, where HEAD points directly to a specific commit rather than a branch. You can look around freely, but any commits made here are not anchored to a branch and can be lost when switching away. Use git switch -c <name> to anchor them to a new branch if needed.

Session Complete!

Your Score: 0/48

Come back later to improve your recall!

CS 35L Quiz

Includes all quizzes taught until today

A developer needs to parse a massive log file, extract IP addresses, sort them, and count unique occurrences. Instead of writing a 500-line Python script, they use cat | awk | sort | uniq -c. Why is this approach fundamentally preferred in the UNIX environment?

Correct Answer:

Explanation

The UNIX design philosophy revolves around writing programs that ‘do one thing and do it well’, and having them work together by handling text streams. Pipelines allow complex tasks to be solved elegantly by chaining these simple tools together.

A script runs a command that generates both useful output and a flood of permission error messages. The user runs script.sh > output.txt, but the errors still clutter the terminal screen while the useful data goes to the file. What underlying concept explains this behavior?

Correct Answer:

Explanation

In UNIX, processes have three default streams: stdin, stdout, and stderr. Redirection with > only captures stdout. To capture the errors, you would need to redirect stderr specifically (e.g., using 2>).

A C++ developer writes a Bash script with a for loop. Inside the loop, they declare a variable temp_val. After the loop finishes, they try to print temp_val expecting it to be undefined or empty, but it prints the last value assigned in the loop. Why did this happen?

Correct Answer:

Explanation

Bash does not use block-level scoping. Unless explicitly declared as local inside a function, variables assigned within control structures like loops or conditionals bleed into the rest of the script.

You want to use a command that requires two file inputs (like diff), but your data is currently coming from the live outputs of two different commands. Instead of creating temporary files on the disk, you use the <(command) syntax. What is this concept called and what does it achieve?

Correct Answer:

Explanation

Process substitution <() allows the standard output of a command to be treated as a file. The operating system creates a temporary file descriptor (like /dev/fd/63) that the consuming command can read from as if it were a physical file.

A script contains entirely valid Python code, but the file is named script.sh and has #!/bin/bash at the very top. When executed via ./script.sh, the terminal throws dozens of ‘command not found’ and syntax errors. What is the fundamental misunderstanding here?

Correct Answer:

Explanation

In UNIX systems, file extensions are largely superficial. The shebang (#!) on the first line is what tells the operating system’s program loader which interpreter program (e.g., Bash, Python, Node) to use to parse the rest of the file.

A developer uses the regular expression [0-9]{4} to validate that a user’s input is exactly a four-digit PIN. However, the system incorrectly accepts ‘12345’ and ‘A1234’. What crucial RegEx concept did the developer omit?

Correct Answer:

Explanation

Without anchors, a regular expression looks for a match anywhere within the string. Because ‘12345’ contains ‘1234’, the pattern matches. Adding ^ (start) and $ (end) ensures the pattern must account for the entire string.

You are designing a data pipeline in the shell. Which of the following statements correctly describe how UNIX handles data streams and command chaining? (Select all that apply)

Correct Answers:

Explanation

Pipes (|) link stdout to stdin, >> appends stdout to a file, and < feeds a file into stdin. By default, pipes do NOT pass stderr (errors will still print to the screen). >! is not the standard way to redirect both streams in Bash (typically >& or &> is used).

You’ve written a shell script deploy.sh but it throws a ‘Permission denied’ error or fails to run when you type ./deploy.sh. Which of the following are valid reasons or necessary steps to successfully execute a script as a standalone program? (Select all that apply)

Correct Answers:

Explanation

Scripts require execute permissions (chmod +x) to be run as programs, and a shebang tells the OS how to interpret the text. They are interpreted, not compiled. The ./ explicitly tells the OS to look in the current directory, bypassing the need for the directory to be in the $PATH.

In Bash, exit codes are crucial for determining if a command succeeded or failed. Which of the following statements are true regarding how Bash handles exit statuses and control flow? (Select all that apply)

Correct Answers:

Explanation

In UNIX, an exit code of 0 means success, while non-zero (like 1) means failure/error. if statements work by running a command and proceeding if its exit code is 0. set -e is a common safety mechanism to fail fast if an error occurs.

When you type a command like python or grep into the terminal, the shell knows exactly what program to run without you providing the full file path. How does the $PATH environment variable facilitate this, and how is it managed? (Select all that apply)

Correct Answers:

Explanation

$PATH is an explicit list of directories, not a global search crawler (which would be extremely slow). The OS checks these directories in order. Modifications in the terminal are bound to that specific session unless saved in a configuration file like .bashrc.

A developer writes LOGFILE="access errors.log" and then runs wc -l $LOGFILE. The command fails with ‘No such file or directory’ errors for both ‘access’ and ‘errors.log’. What is the root cause?

Correct Answer:

Explanation

When a variable is expanded without double quotes, the shell splits the result on whitespace. wc -l $LOGFILE is interpreted as wc -l access errors.log — two separate arguments. The fix is to always quote variable expansions: wc -l "$LOGFILE".

A script is invoked with ./deploy.sh production 8080 myapp. Inside the script, which variable holds the value 8080?

Correct Answer:

Explanation

$0 is the script name (./deploy.sh), $1 is the first argument (production), $2 is the second argument (8080), and $3 would be myapp. $# holds the total argument count (3).

A script contains the line: cd /deploy/target && ./run_tests.sh && echo 'All tests passed!'. If ./run_tests.sh exits with a non-zero status code, what happens next?

Correct Answer:

Explanation

The && operator short-circuits on failure. Since ./run_tests.sh returned a non-zero exit code, the shell skips all remaining &&-linked commands. Only if every command in the chain succeeds does echo 'All tests passed!' execute.

Which of the following statements correctly describe Bash quoting and command substitution behavior? (Select all that apply)

Correct Answers:

Explanation

Single quotes suppress all expansion; double quotes allow variable and command substitution while protecting spaces. Quoting "$VARIABLE" is essential to prevent word splitting on values containing spaces. Single and double quotes are NOT interchangeable when variable expansion is needed. $(command) is command substitution — the standard way to capture command output into a variable.

Which of the following best describes the core difference between centralized and distributed version control systems (like Git)?

Correct Answer:

Explanation

This defines the distributed model, giving developers offline access to the full project history and the ability to work independently.

What are the three primary local states that a file can reside in within a standard Git workflow?

Correct Answer:

Explanation

Files are edited in the working directory, moved to the staging area to prep for a snapshot, and finally committed to the local repository.

What does the command git diff HEAD compare?

Correct Answer:

Explanation

Adding HEAD tells Git to compare the uncommitted files currently on your disk with the snapshot of the most recent commit.

Which Git command should you NEVER use on a shared branch because it can permanently overwrite and destroy work pushed by other team members?

Correct Answer:

Explanation

Force-pushing overwrites the remote repository’s history to exactly match yours, which will permanently delete any new commits made by your collaborators.

You have some uncommitted, incomplete changes in your working directory, but you need to switch to another branch to urgently fix a bug. Which command is best suited to temporarily save your current work without making a messy commit?

Correct Answer:

Explanation

git stash temporarily shelves your staged and unstaged changes, leaving you with a clean working directory so you can safely switch contexts.

What happens when you enter a ‘Detached HEAD’ state in Git?

Correct Answer:

Explanation

Checking out a specific commit hash detaches HEAD. Any new commits made in this state won’t update a branch pointer and can easily be lost.

Which Git command utilizes a binary search through your commit history to help you pinpoint the exact commit that introduced a bug?

Correct Answer:

Explanation

git bisect systematically halves your commit history, asking you to test if the bug is present at each step, to efficiently find the exact commit that broke the code.

What is the primary purpose of Git Submodules?

Correct Answer:

Explanation

Submodules allow you to nest an independent repository inside your main project, pointing to a specific commit of that external library.

Which of the following are advantages of a Distributed Version Control System (like Git) compared to a Centralized one? (Select all that apply)

Correct Answers:

Explanation

In distributed systems like Git, developers have a full local copy of the repository, allowing for offline work and eliminating the single point of failure found in centralized systems. This makes it ideal for large, distributed teams. However, it does not prevent merge conflicts.

Which of the following represent the core local states (or areas) where files can reside in a standard Git architecture? (Select all that apply)

Correct Answers:

Explanation

The three primary local states in Git’s architecture are the Working Directory (files on disk), the Staging Area (prepped for the next commit), and the Local Repository (the compressed history of commits). The remote server is external, and ‘Detached HEAD’ is a state of the HEAD pointer, not a file area.

Which of the following commands are primarily used to review changes, history, or differences in a Git repository? (Select all that apply)

Correct Answers:

Explanation

git log shows the commit history, git diff shows differences between states or commits, and git show displays details about a specific Git object (like a commit). git push uploads data, and git init creates a new repository.

In which of the following scenarios would using git stash be considered an appropriate and helpful practice? (Select all that apply)

Correct Answers:

Explanation

git stash temporarily shelves your local modifications (both staged and unstaged), leaving you with a clean working directory. This is perfect for quick context switches or pulling updates without committing messy, incomplete work. It is not used for permanent deletion or applying commits (which would be git cherry-pick).

Which of the following are valid methods or strategies for integrating changes from a feature branch back into the main codebase? (Select all that apply)

Correct Answers:

Explanation

Merging, rebasing, and squashing (git merge --squash) are all techniques used to integrate branch histories together. git bisect is a debugging tool for finding bugs, and git blame is used to see who last modified specific lines of code.

A faulty commit was pushed to a shared ‘main’ branch last week and your teammates have already synced it. Why should you use git revert to fix this rather than git reset --hard followed by a force-push?

Correct Answer:

Explanation

git revert is the safe choice on shared branches because it adds a new forward-moving commit that nullifies the targeted commit, without rewriting or destroying history. Using git reset --hard and force-pushing would overwrite the shared history, causing serious disruption for any teammate who has already synced that commit.

When integrating a feature branch into ‘main’, under what condition will Git perform a fast-forward merge rather than creating a three-way merge commit?

Correct Answer:

Explanation

A fast-forward merge occurs when the target branch’s history has not diverged — Git can simply slide its pointer forward to incorporate the changes linearly. If both branches have unique commits since they split, Git must perform a three-way merge using their common ancestor, resulting in a merge commit with two parents.

What does the file .git/HEAD contain when you are checked out on a branch, compared to when you are in a detached HEAD state?

Correct Answer:

Explanation

Normally, .git/HEAD is a symbolic reference like ref: refs/heads/main — a pointer to a branch pointer. When you enter detached HEAD state (e.g., with git switch --detach <hash>), HEAD is disconnected from any branch and contains the raw SHA hash of a specific commit directly. Any commits made in this state are not anchored to a branch and can be lost when switching away.

What is the primary mechanism make uses to determine if a target needs to be rebuilt?

Correct Answer:

Explanation

make looks at the file modification timestamps. If a prerequisite (like a .c file) has a newer timestamp than the target (like a .o file), make knows the source code has changed and recompiles the target. This enables efficient incremental builds.

What specific whitespace character MUST be used to indent the command/recipe lines in a Makefile rule?

Correct Answer:

Explanation

Makefile syntax strictly requires command lines (recipes) to be indented with a single Tab character. Using spaces will result in a ‘missing separator’ error.

What does the automatic variable $@ represent in a Makefile rule?

Correct Answer:

Explanation

$@ evaluates to the target name. It is heavily used in compilation commands (e.g., gcc ... -o $@) to dynamically specify the output file name.

Why is the .PHONY directive used in Makefiles (e.g., .PHONY: clean)?

Correct Answer:

Explanation

If you have a rule named clean, and a file literally named clean is accidentally created in the directory, make clean will say ‘clean is up to date’ and do nothing. .PHONY: clean forces make to execute the recipe regardless of whether a file named clean exists.

If a user runs the make command in their terminal without specifying a target, what will make do?

Correct Answer:

Explanation

By default, make begins execution with the first target it encounters in the file. By convention, developers often make the first target all (which depends on the main executables) so that running make builds the entire project.

You have a pattern rule: %.o: %.c. What does the % symbol do?

Correct Answer:

Explanation

The % symbol is a wildcard in pattern rules. %.o: %.c acts as a generic template, avoiding the need to write individual rules for every single source file in a project.

Which of the following are primary benefits of using a Makefile instead of a standard procedural Bash script (build.sh)? (Select all that apply)

Correct Answers:

Explanation

Makefiles save time via incremental compilation, manage complex dependency graphs automatically (declarative vs procedural), and provide standardized commands. They do not run the compiler faster, nor do they write code for you.

Which of the following are valid Automatic Variables in Make? (Select all that apply)

Correct Answers:

Explanation

$@, $<, and $^ are standard automatic variables used to make Makefile rules concise and dynamic. $# and $$ are Bash variables, not Make automatic variables.

In standard C/C++ project Makefiles, which of the following variables are common conventions used to increase flexibility? (Select all that apply)

Correct Answers:

Explanation

CC, CFLAGS, and LDFLAGS are universal conventions in Makefiles. Using them allows developers to easily swap compilers or add flags via command line arguments (e.g., make CC=clang) without editing the actual Makefile. MAKE_IT_FAST is not a standard convention.

How does the evaluation logic of a Makefile differ from a standard cookbook recipe or procedural script? (Select all that apply)

Correct Answers:

Explanation

Makefiles are declarative, not procedural. They do not execute top-to-bottom sequentially, nor do they execute blindly. They build a dependency graph from the top-down (starting at the target), and then execute the necessary commands from the bottom-up, skipping anything that is already up to date.

Quiz Complete!

Your Score: 0/40

Welcome to Computer Science 35L - Software Construction at UCLA

User Stories

---

User stories are the most commonly used format to specify requirements in a light-weight, informal way (particulalry in Agile projects). Each user story is a high-level description of a software feature written from the perspective of the end-user.

User stories act as placeholders for a conversation between the technical team and the “business” side to ensure both parties understand the why and what of a feature.

Format

User stories follow this format:

---

As a [user role],

I want [to perform an action]

so that [I can achieve a goal]

---

For example:

(Smart Grocery Application): As a home cook, I want to swap out ingredients in a recipe so that I can accommodate my dietary restrictions and utilize what I already have in my kitchen.

(Travel Itinerary Planner): As a frequent traveler, I want to discover unique, locally hosted activities so that I can experience the authentic culture of my destination rather than just the standard tourist traps.

This structure makes the team to identify not just the “what”, but also the “who” and — most importantly — the “why”.

The main requirement of the user story is captured in the I want part. The so that part clarifies the goal the user wants to achieve. It does not add additional requirements or constraints to the described requirements.

Be specific about the actor. Avoid generic labels like “user” in the As a clause. Instead, name the specific role that benefits from the feature (e.g., “job seeker”, “hiring manager”, “store owner”). A precise actor clarifies who needs the feature and why, helps the team understand the context, and prevents stories from becoming vague catch-alls. If you find yourself writing “As a user,” ask: which user?

Acceptance Criteria

While the story itself is informal, we make it actionable using Acceptance Criteria. They define the boundaries of the feature and act as a checklist to determine if a story is “done”. Acceptance criteria define the scope of a user story.

They follow this format:

---

Given [pre-condition / initial state]

When [action]

Then [post-condition / outcome]

---

For example:

(Smart Grocery Application): As a home cook, I want to swap out ingredients in a recipe so that I can accommodate my dietary restrictions and utilize what I already have in my kitchen.

• Given the user is viewing a recipe’s ingredient list, when they tap on a specific ingredient, then a modal should appear suggesting a list of viable alternatives.
• Given the user selects a substitute from the alternatives list, when they confirm the swap, then the recipe’s required quantities and nutritional estimates should recalculate and update on the screen.
• Given the user has modified a recipe with substitutions, when they tap the “Save to My Cookbook” button, then the customized version of the recipe should be stored in their personal profile without altering the original public recipe.

These acceptance criteria add clarity to the user story by defining the specific conditions under which the feature should work as expected. They also help to identify potential edge cases and constraints that need to be considered during development. The acceptance criteria define the scope of conditions that check whether an implementation is “correct” and meets the user’s needs. So naturally, acceptance criteria must be specific enough to be testable but should not be overly prescriptive about the implementation details, not to constraint the developers more than really needed to describe the true user need.

Here is another example:

(Travel Itinerary Planner): As a frequent traveler, I want to discover unique, locally hosted activities so that I can experience the authentic culture of my destination rather than just the standard tourist traps.

• Given the user has set their upcoming trip destination to a city, when they navigate to the “Local Experiences” tab, then they should see a dynamically populated list of activities hosted by verified local residents.
• Given the user is browsing the experiences list, when they apply the “Under $50” budget filter, then the list should refresh to display only the activities that fall within that price range.
• Given the user selects a specific local experience, when they tap “Check Availability”, then a calendar widget should expand displaying open booking slots for their specific travel dates.

INVEST

To evaluate if a user story is well-written, we apply the INVEST criteria:

• Independent: Stories should not depend on each other so they can be implemented and released in any order.
• Negotiable: They capture the essence of a need without dictating specific design decisions (like which database to use).
• Valuable: The feature must deliver actual benefit to the user, not just the developer.
• Estimable: The scope must be clear enough for developers to predict the effort required.
• Small: A story should be a manageable chunk of work that isn’t easily split into smaller, still-valuable pieces.
• Testable: It must be verifiable through its acceptance criteria.

We will now look at these criteria in more detail below.

Independent

An independent story does not overlap with or depend on other stories—it can be scheduled and implemented in any order.

What it is and Why it Matters The “Independent” criterion states that user stories should not overlap in concept and should be schedulable and implementable in any order (Wake 2003). An independent story can be understood, tracked, implemented, and tested on its own, without requiring other stories to be completed first.

This criterion matters for several fundamental reasons:

• Flexible Prioritization: Independent stories allow the business to prioritize the backlog based strictly on value, rather than being constrained by technical dependencies (Wake 2003). Without independence, a high-priority story might be blocked by a low-priority one.
• Accurate Estimation: When stories overlap or depend on each other, their estimates become entangled. For example, if paying by Visa and paying by MasterCard are separate stories, the first one implemented bears the infrastructure cost, making the second one much cheaper (Cohn 2004). This skews estimates.
• Reduced Confusion: By avoiding overlap, independent stories reduce places where descriptions contradict each other and make it easier to verify that all needed functionality has been described (Wake 2003).

How to Evaluate It To determine if a user story is independent, ask:

1. Does this story overlap with another story? If two stories share underlying capabilities (e.g., both involve “sending a message”), they have overlap dependency—the most painful form (Wake 2003).
2. Must this story be implemented before or after another? If so, there is an order dependency. While less harmful than overlap (the business often naturally schedules these correctly), it still constrains planning (Wake 2003).
3. Was this story split along technical boundaries? If one story covers the UI layer and another covers the database layer for the same feature, they are interdependent and neither delivers value alone (Cohn 2004).

How to Improve It If stories violate the Independent criterion, you can improve them using these techniques:

• Combine Interdependent Stories: If two stories are too entangled to estimate separately, merge them into a single story. For example, instead of separate stories for Visa, MasterCard, and American Express payments, combine them: “A company can pay for a job posting with a credit card” (Cohn 2004).
• Partition Along Different Dimensions: If combining makes the story too large, re-split along a different dimension. For overlapping email stories like “Team member sends and receives messages” and “Team member sends and replies to messages”, repartition by action: “Team member sends message”, “Team member receives message”, “Team member replies to message” (Wake 2003).
• Slice Vertically: When stories have been split along technical layers (UI vs. database), re-slice them as vertical “slices of cake” that cut through all layers. Instead of “Job Seeker fills out a resume form” and “Resume data is written to the database”, write “Job Seeker can submit a resume with basic information” (Cohn 2004).

Examples of Stories Violating ONLY the Independent Criterion

Example 1: Overlap Dependency

Story A: “As a team member, I want to send and receive messages so that I can communicate with my colleagues.”

• Given I am on the messaging page, When I compose a message and click “Send”, Then the message appears in the recipient’s inbox.
• Given a colleague has sent me a message, When I open my inbox, Then I can read the message.

Story B: “As a team member, I want to reply to messages so that I can indicate which message I am responding to.”

• Given I have received a message, When I click the “Reply” button and submit my response, Then the reply is sent to the original sender.
• Given the reply has been received, When the original sender views the message, Then it is displayed has a reply to the original message.

• Negotiable: Yes. Neither story dictates a specific UI or technology.
• Valuable: Yes. Communication features are clearly valuable to users.
• Estimable: Difficult. The overlapping “send” capability makes it unclear how to estimate each story independently.
• Small: Yes. Each story is as small as it can be without losing value. Sending without receiving would be incomplete and thus not valuable, so we cannot split story A into seperate stories.
• Testable: Yes. Clear acceptance criteria can be written for sending, receiving, and replying.
• Why it violates Independent: Both stories include “sending a message.” If Story A is implemented first, parts of Story B are already done. If Story B is implemented first, parts of Story A are already done. This creates confusion about what is covered and makes estimation unreliable.
• How to fix it: Repartition into three non-overlapping stories: “As a team member, I want to send a message”, “As a team member, I want to receive messages”, and “As a team member, I want to reply to a message.”

Example 2: Technical (Horizontal) Splitting

Story A: “As a job seeker, I want to fill out a resume form so that I can enter my information.”

• Given I am on the resume page, When I fill in my name, address, and education, Then the form displays my entered information.

Story B: “As a job seeker, I want my resume data to be saved so that it is available when I return.”

• Given I have filled out the resume form, When I click “Save”, Then my resume data is available when I log back in.

• Negotiable: No. Both stories dictate internal technical steps rather than user-facing capabilities.
• Valuable: No. Neither story delivers value on its own—a form that does not save is useless, and saving data without a form to collect it is equally useless.
• Estimable: Yes. Developers can estimate each technical task.
• Small: Yes. Each is a small piece of work.
• Testable: Yes. Each can be verified in isolation.
• Why it violates Independent: Story B is meaningless without Story A, and Story A is useless without Story B. They are completely interdependent because the feature was split along technical boundaries (UI layer vs. persistence layer) instead of user-facing functionality (Cohn 2004).
• How to fix it: Combine into a single vertical slice: “As a job seeker, I want to submit a resume with basic information (name, address, education) so that employers can find me.” This cuts through all layers and delivers value independently (Cohn 2004).

Negotiable

A negotiable story captures the essence of a user’s need without locking in specific design or technology decisions—the details are worked out collaboratively.

What it is and Why it Matters The “Negotiable” criterion states that a user story is not an explicit contract for features; rather, it captures the essence of a user’s need, leaving the details to be co-created by the customer and the development team during development (Wake 2003). A good story captures the essence, not the details (see also “Requirements Vs. Design”).

This criterion matters for several fundamental reasons:

• Enabling Collaboration: Because stories are intentionally incomplete, the team is forced to have conversations to fill in the details. Ron Jeffries describes this through the three C’s: Card (the story text), Conversation (the discussion), and Confirmation (the acceptance tests) (Cohn 2004). The card is merely a token promising a future conversation (Wake 2003).
• Evolutionary Design: High-level stories define capabilities without over-constraining the implementation approach (Wake 2003). This leaves room to evolve the solution from a basic form to an advanced form as the team learns more about the system’s needs.
• Avoiding False Precision: Including too many details early creates a dangerous illusion of precision (Cohn 2004). It misleads readers into believing the requirement is finalized, which discourages necessary conversations and adaptation.

How to Evaluate It To determine if a user story is negotiable, ask:

1. Does this story dictate a specific technology or design decision? Words like “MongoDB”, “HTTPS”, “REST API”, or “dropdown menu” in a story are red flags that it has left the space of requirements and entered the space of design.
2. Could the development team solve this problem using a completely different technology or layout, and would the user still be happy? If the answer is yes, the story is negotiable. If the answer is no, the story is over-constrained.
3. Does the story include UI details? Embedding user interface specifics (e.g., “a print dialog with a printer list”) introduces premature assumptions before the team fully understands the business goals (Cohn 2004).

How to Improve It If a story violates the Negotiable criterion, you can improve it using these techniques:

• Focus on the “Why”: Use “So that” clauses to clarify the underlying goal, which allows the team to negotiate the “How”.
• Specify What, Not How: Replace technology-specific language with the user need it serves. Instead of “use HTTPS”, write “keep data I send and receive confidential.”
• Define Acceptance Criteria, Not Steps: Define the outcomes that must be true, rather than the specific UI clicks or database queries required.
• Keep the UI Out as Long as Possible: Avoid embedding interface details into stories early in the project (Cohn 2004). Focus on what the user needs to accomplish, not the specific controls they will use.

Examples of Stories Violating ONLY the Negotiable Criterion

Example 1: The Technology-Specific Story

“As a subscriber, I want my profile settings saved in a MongoDB database so that they load quickly the next time I log in.”

• Given I am logged in and I change my profile settings, When I log out and log back in, Then my profile settings are still applied.

• Independent: Yes. Saving profile settings does not depend on other stories.
• Valuable: Yes. Remembering user settings is clearly valuable.
• Estimable: Yes. A developer can estimate the effort to implement settings persistence.
• Small: Yes. This is a focused piece of work.
• Testable: Yes. You can verify that settings persist across sessions.
• Why it violates Negotiable: Specifying “MongoDB” is a design decision. The user does not care where the data lives. The engineering team might realize that a relational SQL database or local browser caching is a much better fit for the application’s architecture.
• How to fix it: “As a subscriber, I want the system to remember my profile settings so that I don’t have to re-enter them every time I log in.”

Example 2: The UI-Specific Story

“As a student, I want the website to use HTTPS so that my data is safe.”

• Given I am submitting personal data on the website, When the data is transmitted to the server, Then the connection uses HTTPS encryption.

• Independent: Yes. Security does not depend on other stories.
• Valuable: Yes. Data safety is clearly valuable to the user.
• Estimable: Yes. Enabling HTTPS is a well-understood task.
• Small: Yes. This is a single, focused change.
• Testable: Yes. You can verify that traffic is encrypted.
• Why it violates Negotiable: “HTTPS” is a specific design decision. The user’s actual need is data confidentiality, which could be achieved in multiple ways depending on the system’s architecture.
• How to fix it: “As a student, I want the website to keep data I send and receive confidential so that my privacy is ensured.”

Valuable

A valuable story delivers tangible benefit to the customer, purchaser, or user—not just to the development team.

What it is and Why it Matters The “Valuable” criterion states that every user story must deliver tangible value to the customer, purchaser, or user—not just to the development team (Wake 2003). A good story focuses on the external impact of the software in the real world: if we frame stories so their impact is clear, product owners and users can understand what the stories bring and make good prioritization choices (Wake 2003).

This criterion matters for several fundamental reasons:

• Informed Prioritization: The product owner prioritizes the backlog by weighing each story’s value against its cost. If a story’s business value is opaque—because it is written in technical jargon—the customer cannot make intelligent scheduling decisions (Cohn 2004).
• Avoiding Waste: Stories that serve only the development team (e.g., refactoring for its own sake, adopting a trendy technology) consume iteration capacity without moving the product closer to its users’ goals. The IRACIS framework provides a useful lens for value: does the story Increase Revenue, Avoid Costs, or Improve Service? (Wake 2003)
• User vs. Purchaser Value: It is tempting to say every story must be valued by end-users, but that is not always correct. In enterprise environments, the purchaser may value stories that end-users do not care about (e.g., “All configuration is read from a central location” matters to the IT department managing 5,000 machines, not to daily users) (Cohn 2004).

How to Evaluate It To determine if a user story is valuable, ask:

1. Would the customer or user care if this story were dropped? If only developers would notice, the story likely lacks user-facing value.
2. Can the customer prioritize this story against others? If the story is written in “techno-speak” (e.g., “All connections go through a connection pool”), the customer cannot weigh its importance (Cohn 2004).
3. Does this story describe an external effect or an internal implementation detail? Valuable stories describe what happens on the edge of the system—the effects of the software in the world—not how the system is built internally (Wake 2003).

How to Improve It If stories violate the Valuable criterion, you can improve them using these techniques:

• Rewrite for External Impact: Translate the technical requirement into a statement of benefit for the user. Instead of “All connections to the database are through a connection pool”, write “Up to fifty users should be able to use the application with a five-user database license” (Cohn 2004).
• Let the Customer Write: The most effective way to ensure a story is valuable is to have the customer write it in the language of the business, rather than in technical jargon (Cohn 2004).
• Focus on the “So That”: A well-written “so that” clause forces the author to articulate the real-world benefit. If you cannot complete “so that [some user benefit]” without referencing technology, the story is likely not valuable.
• Complete the Acceptance Criteria: A story may appear valuable but have incomplete acceptance criteria that leave out essential functionality, effectively making the delivered feature useless.

Examples of Stories Violating ONLY the Valuable Criterion

Example 1: The Developer-Centric Story

“As a developer, I want to rewrite the core authentication API in Rust so that I can use a more modern programming language.”

• Given the authentication API currently runs on Node.js, When a developer deploys the new Rust-based API, Then all existing authentication endpoints return identical responses.

• Independent: Yes. Rewriting the auth API does not depend on other stories.
• Negotiable: Yes. The story is phrased as a goal (rewrite auth), leaving room to discuss scope and approach.
• Estimable: Yes. A developer experienced with Rust can estimate the effort of a rewrite.
• Small: Yes. Rewriting a single API component can fit within a sprint.
• Testable: Yes. You can verify the new API passes all existing authentication tests.
• Why it violates Valuable: The story is written entirely from the developer’s perspective. The user does not care which programming language the API uses. The “so that” clause (“use a more modern programming language”) describes a developer preference, not a user benefit (Cohn 2004).
• How to fix it: If there is a legitimate user-facing reason (e.g., performance), rewrite the story around that benefit: “As a registered member, I want to log in without noticeable delay so that I can start using the application immediately.”

Example 2: The Incomplete Story

“As a smart home owner, I want to schedule my porch lights to turn on automatically at a specific time so that I don’t have to walk up to a dark house in the evening.” Given I am logged into the smart home mobile app, When I set the porch light schedule to turn on at 6:00 PM, Then the porch lights will illuminate at exactly 6:00 PM every day.

• Independent: Yes. Scheduling lights does not depend on other stories.
• Negotiable: Yes. The specific UI and scheduling mechanism are open to discussion.
• Estimable: Yes. Implementing a time-based trigger is well-understood work.
• Small: Yes. A single scheduling feature fits within a sprint.
• Testable: Yes. The acceptance criteria define a clear pass/fail condition.
• Why it violates Valuable: On first glance, this story looks valuable. But the acceptance criteria are missing the ability to turn off the lights. If lights stay on forever, they waste energy and the feature becomes a nuisance rather than a benefit. The story as written delivers incomplete value because its acceptance criteria do not capture the full scope needed to make the feature genuinely useful.
• How to fix it: Add the missing acceptance criterion: “Given I am logged into the smart home mobile app, When I set the porch light schedule to turn off at 6:00 AM and the lights are illuminated, Then the porch lights will turn off at 6:00 AM.” Now the story delivers complete value.

Estimable

An estimable story has a scope clear enough for the development team to make a reasonable judgment about the effort required.

What it is and Why it Matters The “Estimable” criterion states that the development team must be able to make a reasonable judgment about a story’s size, cost, or time to deliver (Wake 2003). While precision is not the goal, the estimate must be useful enough for the product owner to prioritize the story against other work (Cohn 2004).

This criterion matters for several fundamental reasons:

• Enabling Prioritization: The product owner ranks stories by comparing value to cost. If a story cannot be estimated, the cost side of this equation is unknown, making informed prioritization impossible (Cohn 2004).
• Supporting Planning: Stories that cannot be estimated cannot be reliably scheduled into an iteration. Without sizing information, the team risks committing to more (or less) work than they can deliver.
• Surfacing Unknowns Early: An unestimable story is a signal that something important is not understood—either the domain, the technology, or the scope. Recognizing this early prevents costly surprises later.

How to Evaluate It Developers generally cannot estimate a story for one of three reasons (Cohn 2004):

1. Lack of Domain Knowledge: The developers do not understand the business context. For example, a story saying “New users are given a diabetic screening” could mean a simple web questionnaire or an at-home physical testing kit—without clarification, no estimate is possible (Cohn 2004).
2. Lack of Technical Knowledge: The team understands the requirement but has never worked with the required technology. For example, a team asked to expose a gRPC API when no one has experience with Protocol Buffers or gRPC cannot estimate the work (Cohn 2004).
3. The Story is Too Big: An epic like “A job seeker can find a job” encompasses so many sub-tasks and unknowns that it cannot be meaningfully sized as a single unit (Cohn 2004).

How to Improve It The approach to fixing an unestimable story depends on which barrier is blocking estimation:

• Conversation (for Domain Knowledge Gaps): Have the developers discuss the story directly with the customer. A brief conversation often reveals that the requirement is simpler (or more complex) than assumed, making estimation possible (Cohn 2004).
• Spike (for Technical Knowledge Gaps): Split the story into two: an investigative spike—a brief, time-boxed experiment to learn about the unknown technology—and the actual implementation story. The spike itself is always given a defined maximum time (e.g., “Spend exactly two days investigating credit card processing”), which makes it estimable. Once the spike is complete, the team has enough knowledge to estimate the real story (Cohn 2004).
• Disaggregate (for Stories That Are Too Big): Break the epic into smaller, constituent stories. Each smaller piece isolates a specific slice of functionality, reducing the cognitive load and making estimation tractable (Cohn 2004).

Examples of Stories Violating ONLY the Estimable Criterion

Example 1: The Unknown Domain

“As a patient, I want to receive a personalized wellness screening so that I can understand my health risks.”

• Given I am a new patient registering on the platform, When I complete the wellness screening, Then I receive a personalized health risk summary based on my answers.

• Independent: Yes. The screening feature does not depend on other stories.
• Negotiable: Yes. The specific questions and screening logic are open to discussion.
• Valuable: Yes. Personalized health screening is clearly valuable to patients.
• Small: Yes. A single screening workflow can fit within a sprint—once the scope is clarified.
• Testable: Yes. Acceptance criteria can define specific screening outcomes for specific patient profiles.
• Why it violates Estimable: The developers do not know what “personalized wellness screening” means in this context. It could be a simple 5-question web form or a complex algorithm that integrates with lab data. Without domain knowledge, the team cannot estimate the effort (Cohn 2004).
• How to fix it: Have the developers sit down with the customer (e.g., a qualified nurse or medical expert) to clarify the scope. Once the team learns it is a simple web questionnaire, they can estimate it confidently.

Example 2: The Unknown Technology

“As an enterprise customer, I want to access the system’s data through a gRPC API so that I can integrate it with my existing microservices infrastructure.”

• Given an enterprise client sends a gRPC request for user data, When the system processes the request, Then the system returns the requested data in the correct Protobuf-defined format.

• Independent: Yes. Adding an integration interface does not depend on other stories.
• Negotiable: Partially. The customer has specified gRPC, but the service contract and data schema are open to discussion.
• Valuable: Yes. Enterprise integration is clearly valuable to the purchasing organization.
• Small: Yes. A single service endpoint can fit within a sprint—once the team understands the technology.
• Testable: Yes. You can verify the interface returns the correct data in the correct format.
• Why it violates Estimable: No one on the development team has ever built a gRPC service or worked with Protocol Buffers. They understand what the customer wants but have no experience with the technology required to deliver it, making any estimate unreliable (Cohn 2004).
• How to fix it: Split into two stories: (1) a time-boxed spike—”Investigate gRPC integration: spend at most two days building a proof-of-concept service”—and (2) the actual implementation story. After the spike, the team has enough knowledge to estimate the real work (Cohn 2004).

Small

A small story is a manageable chunk of work that can be completed within a single iteration—not so large it becomes an epic, not so small it loses meaningful context. A user story should be as small as it can be while still delivering value.

What it is and Why it Matters The “Small” criterion states that a user story should be appropriately sized so that it can be comfortably completed by the development team within a single iteration (Cohn 2004). Stories typically represent at most a few person-weeks of work; some teams restrict them to a few person-days (Wake 2003). If a story is too large, it is called an epic and must be broken down. If a story is too small, it should be combined with related stories.

This criterion matters for several fundamental reasons:

• Predictability: Large stories are notoriously difficult to estimate accurately. The smaller the story, the higher the confidence the team has in their estimate of the effort required (Cohn 2004).
• Risk Reduction: If a massive story spans an entire sprint (or spills over into multiple sprints), the team risks delivering zero value if they hit a roadblock. Smaller stories ensure a steady, continuous flow of delivered value.
• Faster Feedback: Smaller stories reach a “Done” state faster, meaning they can be tested, reviewed by the product owner, and put in front of users much sooner to gather valuable feedback.

How to Evaluate It To determine if a user story is appropriately sized, ask:

1. Can it be completed in one sprint? If the answer is no, or “maybe, if everything goes perfectly,” the story is too big. It is an epic and must be split (Cohn 2004).
2. Is it a compound story? Words like and, or, and but in the story description (e.g., “I want to register and manage my profile and upload photos”) often indicate that multiple stories are hiding inside one. A compound story is an epic that aggregates multiple easily identifiable shorter stories (Cohn 2004).
3. Is it a complex story? If the story is large because of inherent uncertainty (new technology, novel algorithm), it is a complex story and should be split into a spike and an implementation story (Cohn 2004).
4. Is it too small? If the administrative overhead of writing and estimating the story takes longer than implementing it, the story is too small and should be combined with related stories (Cohn 2004).

How to Improve It The approach to fixing a story that violates the Small criterion depends on whether it is too big or too small:

Stories that are too big:

• Split by Workflow Steps (CRUD): Instead of “As a job seeker, I want to manage my resume,” split along operations: create, edit, delete, and manage multiple resumes (Cohn 2004).
• Split by Data Boundaries: Instead of splitting by operation, split by the data involved: “add/edit education”, “add/edit job history”, “add/edit salary” (Cohn 2004).
• Slice the Cake (Vertical Slicing): Never split along technical boundaries (one story for UI, one for database). Instead, split into thin end-to-end “vertical slices” where each story touches every architectural layer and delivers complete, albeit narrow, functionality (Cohn 2004).
• Split by Happy/Sad Paths: Build the “happy path” (successful transaction) as one story, and handle the error states (declined cards, expired sessions) in subsequent stories.

Stories that are too small:

• Combine Related Stories: Merge tiny, related items (e.g., a batch of small UI tweaks or minor bug fixes) into a single story representing a half-day to several days of work (Cohn 2004).

Examples of Stories Violating ONLY the Small Criterion

Example 1: The Epic (Too Big)

“As a traveler, I want to plan a vacation so that I can book all the arrangements I need in one place.”

• Given I have selected travel dates and a destination, When I search for vacation packages, Then I see available flights, hotels, and rental cars with pricing.
• Given I have selected a flight, hotel, and rental car, When I click “Book”, Then all reservations are confirmed and I receive a booking confirmation email.

• Independent: Yes. Planning a vacation does not overlap with other stories.
• Negotiable: Yes. The specific features and UI are open to discussion.
• Valuable: Yes. End-to-end vacation planning is clearly valuable to travelers.
• Estimable: No. The scope is so vast that developers cannot reliably predict the effort. (Violations of Small often cause violations of Estimable, since epics contain hidden complexity.)
• Testable: Yes. Acceptance criteria can be written for individual planning features.
• Why it violates Small: “Planning a vacation” involves searching for flights, comparing hotels, booking rental cars, managing an itinerary, handling payments, and much more. This is an epic containing many stories. It cannot be completed in a single sprint (Cohn 2004).
• How to fix it: Disaggregate into smaller vertical slices: “As a traveler, I want to search for flights by date and destination so that I can find available options”, “As a traveler, I want to compare hotel prices for my destination so that I can choose one within my budget”, etc.

Example 2: The Micro-Story (Too Small)

“As a job seeker, I want to edit the date for each community service entry on my resume so that I can correct mistakes.”

• Given I am viewing a community service entry on my resume, When I change the date field and click “Save”, Then the updated date is displayed on my resume.

• Independent: Yes. Editing a single date field does not depend on other stories.
• Negotiable: Yes. The exact editing interaction is open to discussion.
• Valuable: Yes. Correcting resume data is valuable to the user.
• Estimable: Yes. Editing a single field is trivially estimable.
• Testable: Yes. Clear pass/fail criteria can be written.
• Why it violates Small: This story is too small. The administrative overhead of writing, estimating, and tracking this story card takes longer than actually implementing the change. Having dozens of stories at this granularity buries the team in disconnected details—what Wake calls a “bag of leaves” (Wake 2003).
• How to fix it: Combine with related micro-stories into a single meaningful story: “As a job seeker, I want to edit all fields of my community service entries so that I can keep my resume accurate.” (Cohn 2004)

Testable

A testable story has clear, objective, and measurable acceptance criteria that allow the team to verify definitively when the work is done.

What it is and Why it Matters The “Testable” criterion dictates that a user story must have clear, objective, and measurable conditions that allow the team to verify when the work is officially complete. If a story is not testable, it can never truly be considered “Done.”

This criterion matters for several crucial reasons:

• Shared Understanding: It forces the product owner and the development team to align on the exact expectations. It removes ambiguity and prevents the dreaded “that’s not what I meant” conversation at the end of a sprint.
• Proving Value: A user story represents a slice of business value. If you cannot test the story, you cannot prove that it successfully delivers that value to the user.
• Enabling Quality Assurance: Testable stories allow QA engineers (and developers practicing Test-Driven Development) to write their test cases—whether manual or automated—before a single line of production code is written.

How to Evaluate It To determine if a user story is testable, ask yourself the following questions:

1. Can I write a definitive pass/fail test for this? If the answer relies on someone’s opinion or mood, it is not testable.
2. Does the story contain “weasel words”? Look out for subjective adjectives and adverbs like fast, easy, intuitive, beautiful, modern, user-friendly, robust, or seamless. These words are red flags that the story lacks objective boundaries.
3. Are the Acceptance Criteria clear? Does the story have defined boundaries that outline specific scenarios and edge cases?

How to Improve It If you find a story that violates the Testable criterion, you can improve it by replacing subjective language with quantifiable metrics and concrete scenarios:

• Quantify Adjectives: Replace subjective terms with hard numbers. Change “loads fast” to “loads in under 2 seconds.” Change “supports a lot of users” to “supports 10,000 concurrent users.”
• Use the Given/When/Then Format: Borrow from Behavior-Driven Development (BDD) to write clear acceptance criteria. Establish the starting state (Given), the action taken (When), and the expected, observable outcome (Then).
• Define “Intuitive” or “Easy”: If the goal is a “user-friendly” interface, make it testable by tying it to a metric, such as: “A new user can complete the checkout process in fewer than 3 clicks without relying on a help menu.”

Examples of Stories Violating ONLY the Testable Criterion

Below are two user stories that are not testable but still satisfy (most) other INVEST criteria.

Example 1: The Subjective UI Requirement

“As a marketing manager, I want the new campaign landing page to feature a gorgeous and modern design, so that it appeals to our younger demographic.”

• Given the landing page is deployed, When a visitor from the 18-24 demographic views it, Then the design looks gorgeous and modern.

• Independent: Yes. It doesn’t inherently rely on other features being built first.
• Negotiable: Yes. The exact layout and tech used to build it are open to discussion.
• Valuable: Yes. A landing page to attract a younger demographic provides clear business value.
• Estimable: Yes. Generally, a frontend developer can estimate the effort to build a standard landing page.
• Small: Yes. Building a single landing page easily fits within a single sprint.
• Why it violates Testable: “Gorgeous,” “modern,” and “appeals to” are completely subjective. What one developer thinks is modern, the marketing manager might think is ugly.
• How to fix it: Tie it to a specific, measurable design system or user-testing metric. (e.g., “Acceptance Criteria: The design strictly adheres to the new V2 Brand Guidelines and passes a 5-second usability test with a 4/5 rating from a focus group of 18-24 year olds.”)

Example 2: The Vague Performance Requirement

“As a data analyst, I want the monthly sales report to generate instantly, so that my workflow isn’t interrupted by loading screens.”

• Given the database contains 5 years of sales data, When the analyst requests the monthly sales report, Then the report generates instantly.

• Independent: Yes. Optimizing or building this report can be done independently.
• Negotiable: Yes. The team can negotiate how to achieve the speed (e.g., caching, database indexing, background processing).
• Valuable: Yes. Saving the analyst’s time is a clear operational benefit.
• Small: Yes. It is a focused optimization on a single report.
• Why it violates Testable: “Instantly” is physically impossible in computing, and it is a highly subjective standard. Does instantly mean 0.1 seconds, or 1.5 seconds? Without a benchmark, a test script cannot verify if the feature passes or fails.
• Estimable: No. Without a clear definition of “instantly”, the team cannot estimate the effort required to build the feature. Violations of testable are often also not estimable. In the example above, the Subjective UI was still estiable, because independent of the specific definition of “modern”, the implementation effort would not change signifiantly, just the specific UI that would be chosen would change.
• How to fix it: Replace the subjective word with a quantifiable service level indicator. (e.g., “Acceptance Criteria: Given the database contains 5 years of sales data, when the analyst requests the monthly sales report, then the data renders on screen in under 2.5 seconds at the 95th percentile.”)

Example 3: The Subjective Audio Requirement

“As a podcast listener, I want the app’s default intro chime to play at a pleasant volume, so that it doesn’t startle me when I open the app.”

• Given I open the app for the first time, When the intro chime plays, Then the volume is at a pleasant level.

• Independent: Yes. Adjusting the audio volume doesn’t rely on other features.
• Negotiable: Yes. The exact decibel level or method of adjustment is open to discussion.
• Valuable: Yes. Improving user comfort directly enhances the user experience.
• Estimable: Yes. Changing a default audio volume variable or asset is a trivial, highly predictable task (e.g., a 1-point story). The developers know exactly how much effort is involved.
• Small: Yes. It will take a few minutes to implement.
• Why it violates Testable: “Pleasant volume” is entirely subjective. A volume that is pleasant in a quiet library will be inaudible on a noisy subway. Because there is no objective baseline, QA cannot definitively pass or fail the test.
• How to fix it: “Acceptance Criteria: The default intro chime must be normalized to -16 LUFS (Loudness Units relative to Full Scale).”

How INVEST supports agile processes like Scrum

The INVEST principles matter because they act as a compass for creating high-quality, actionable user stories that align with Agile goals and principles of processes like Scrum. By ensuring stories are Independent and Small, teams gain the scheduling flexibility needed to implement and release features in any order within short iterations. If user stories are not independent, it becomes hard to always select the highest value user stories. If they are not small, it becomes hard to select a Sprint Backlog that fit the team’s velocity.
Negotiable stories promote essential dialogue between developers and stakeholders, while Valuable ones ensure that every effort translates into a meaningful benefit for the user. Finally, stories that are Estimable and Testable provide the clarity required for accurate sprint planning and objective verification of the finished product. In Scrum and XP, user stories are estimated during the Planning activity.

FAQ on INVEST

How are Estimable and Testable different?

Estimable refers to the ability of developers to predict the size, cost, or time required to deliver a story. This attribute relies on the story being understood well enough and having a clear enough scope to put useful bounds on those guesses.

Testable means that a story can be verified through objective acceptance criteria. A story is considered testable if there is a definitive “Yes” or “No” answer to whether its objectives have been achieved.

In practice, these two are closely linked: if a story is not testable because it uses vague terms like “fast” or “high accuracy,” it becomes nearly impossible to estimate the actual effort needed to satisfy it. But that is not always the case.

Here are examples of user stories that isolate those specific violations of the INVEST criteria:

Violates Testable but not Estimable User Story: “As a site administrator, I want the dashboard to feel snappy when I log in so that I don’t get frustrated with the interface.”

• Why it violates Testable: Terms like “snappy” or “fast” are subjective. Without a specific metric (e.g., “loads in under 2 seconds”), there is no objective “Yes” or “No” answer to determine if the story is done.
• Why it is still Estimable: A developer might still estimate this as a “small” task if they assume it just requires basic front-end optimization, even though they can’t formally verify the “snappy” feel.

Violates Estimable but not Testable User Story: “As a safety officer, I want the system to automatically identify every pedestrian in this complex, low-light video feed.”

• Why it violates Estimable: This is a “research project”. Because the technical implementation is unknown or highly innovative, developers cannot put useful bounds on the time or cost required to solve it.
• Why it is still Testable: It is perfectly testable; you could poll 1,000 humans to verify if the software’s identifications match reality. The outcome is clear, but the effort to reach it is not.
• What about Small? This user story is not small. It is a very large feature that takes a long time to implement.

How are Estimable and Small different?

While they are related, Estimable and Small focus on different dimensions of a user story’s readiness for development.

Estimable: Predictability of Effort

Estimable refers to the developers’ ability to provide a reasonable judgment regarding the size, cost, or time required to deliver a story.

• Requirements: For a story to be estimable, it must be understood well enough and be stable enough that developers can put “useful bounds” on their guesses.
• Barriers: A story may fail this criterion if developers lack domain knowledge, technical knowledge (requiring a “technical spike” to learn), or if the story is so large (an epic) that its complexity is hidden.
• Goal: It ensures the Product Owner can prioritize stories by weighing their value against their cost.

Small: Manageability of Scope

Small refers to the physical magnitude of the work. A story should be a manageable chunk that can be completed within a single iteration or sprint.

• Ideal Size: Most teams prefer stories that represent between half a day and two weeks of work.
• Splitting: If a story is too big, it should be split into smaller, still-valuable “vertical slices” of functionality. However, a story shouldn’t be so small (like a “bag of leaves”) that it loses its meaningful context or value to the user.
• Goal: Smaller stories provide more scheduling flexibility and help maintain momentum through continuous delivery.

Key Differences

1. Nature of the Constraint: Small is a constraint on volume, while Estimable is a constraint on clarity.
2. Accuracy vs. Size: While smaller stories tend to get more accurate estimates, a story can be small but still unestimatable. For example, a “Research Project” or investigative spike might involve a very small amount of work (reading one document), but because the outcome is unknown, it remains impossible to estimate the time required to actually solve the problem.
3. Predictability vs. Flow: Estimability is necessary for planning (knowing what fits in a release), while Smallness is necessary for flow (ensuring work moves through the system without bottlenecks).

Should bug reports be user stories?

Mike Cohn explicitly advocates for this unified approach, stating that the best method is to consider each bug report its own story (Cohn 2004). If a bug is large and requires significant effort, it should be estimated, prioritized, and treated exactly like any other typical user story (Cohn 2004). However, treating every minor bug as an independent story can cause administrative bloat. For bugs that are small and quick to fix, Cohn suggests that teams combine them into one or more unified stories (Cohn 2004). On a physical task board, this is achieved by stapling several small bug cards together under a single “cover story card”, allowing the collection to be estimated and scheduled as a single unit of work (Cohn 2004).

From the Extreme Programming (XP) perspective, translating a bug report into a narrative user story addresses only the process layer; the technical reality is that a bug is a missing test. Kent Beck argues that problem reports must come with test cases demonstrating the problem in code (Beck and Andres 2004). When a developer encounters or is assigned a problem, their immediate action must be to write an automated unit or functional test that isolates the issue (Beck and Andres 2004). In this paradigm, a bug report is fundamentally an executable specification. Writing the story card is merely a placeholder; the true confirmation of the defect’s existence—and its subsequent resolution—is proven by a test that fails, and then passes (Beck and Andres 2004).

Applicability

User stories are ideal for iterative, customer-centric projects where requirements might change frequently.

Limitations

User stories can struggle to capture non-functional requirements like performance, security, or reliability, and they are generally considered insufficient for safety-critical systems like spacecraft or medical devices

User Stories in Practice

While user stories are widely adopted for building shared understanding (Patton 2014) and fostering a pleasant workplace among developers (Lucassen et al. 2016), empirical research highlights several significant challenges in their practical application.

Common Quality Issues

• The NFR Blindspot: Practitioners systematically omit non-functional requirements (NFRs)—such as usability, security, and performance—because these constraints often do not fit neatly into the standard functional template (Lauesen and Kuhail 2022). Mike Cohn notes that forcing NFRs into the “As a… I want…” format often results in untestable statements like “The software must be easy to use” (Cohn 2004).
• Rationale Hazard: While specified rationale (“so that…”) is essential for requirements quality (Lucassen et al. 2016), practitioners often fill this field in unjustifiably to satisfy templates. This forced inclusion of “filler” goals can directly lead to unverifiable requirements that obscure true business objectives (Lauesen and Kuhail 2022).
• Ambiguity: Ambiguity manifests across lexical, syntactic, semantic, and pragmatic levels (Amna and Poels 2022). When analyzed collectively, vague stories often lead to severe cross-story defects, including logical conflicts and missing dependencies (Amna and Poels 2022).

Process Anti-Patterns

• The “Template Zombie”: This occurs when a team allows its work to be driven by templates rather than the thought process necessary to deliver a product (Patton 2014). Practitioners become “Template Zombies” when they mechanically force technical tasks or backend services into the story format, often ignoring the necessary collaborative conversation (Patton 2014).
• The Client-Vendor Anti-Pattern: Jeff Patton identifies a toxic dynamic where one party (often a business stakeholder) takes a “client” role to dictate requirements, while the other (often a developer or analyst) takes a “vendor” role to merely take orders and provide estimates. This creates a “requirements contract” that kills the collaborative problem-solving at the heart of agile development (Patton 2014).
• Story Smells: Common “smells” include Goldplating (adding unplanned features), UI Detail Too Soon (constraining design before understanding goals), and Thinking Too Far Ahead (exhaustive detailing long before implementation) (Cohn 2004).

Automation and LLMs

Recent advancements in Large Language Models (LLMs) have introduced new capabilities for requirement engineering:

• Syntactic Maturity: LLMs like GPT-4o excel at generating well-formed, atomic, and grammatically complete user stories, often outperforming novice analysts in following strict templates (Sharma and Tripathi 2025).
• The Convergence Gap: While LLMs achieve high coverage of standard requirements, they exhibit a “convergence vs. creativity” trade-off. They tend to converge on predictable patterns and may miss novel or domain-specific nuances that human analysts provide (Quattrocchi et al. 2025).
• The Power of Prompting: The quality of automated generation is highly sensitive to prompt design. Using a “Meta-Few-Shot” approach—combining structural rules with explicit positive and negative examples—can push LLM success rates significantly higher, even surpassing manual human generation in semantic accuracy (Santos et al. 2025).

Story Mapping and INVEST

The narrative flow of User Story Mapping captures the sequential and hierarchical relationships between stories (Patton 2014). From a theoretical perspective, this creates a notable tension with the INVEST criteria: while Story Mapping emphasizes the journey’s context and narrative flow, it can challenge the Independence criterion by highlighting the deep relationships between individual stories in a user journey. However, this mapping generally helps achieve the other INVEST criteria—particularly Valuable and Small—by providing a clear framework for slicing features into manageable releases.

Quiz

User Stories & INVEST Principle Flashcards

Test your knowledge on Agile user stories and the criteria for creating high-quality requirements!

What is the primary purpose of Acceptance Criteria in a user story?

To define the specific conditions that must be met for a user story to be considered ‘Done’. They define the scope of the user story.

Acceptance criteria provide clear, objective boundaries for a feature. They remove ambiguity, guide the development team on exactly what needs to be built, and form the basis for testing whether the story delivers the intended value. They make user stories testable and estimable.

What is the standard template for writing a User Story?

‘As a [role], I want [feature/action], so that [benefit/value].’

This structure ensures the team always understands who they are building for, what they are building, and why it matters to the business or user.

What does the acronym INVEST stand for?

Independent, Negotiable, Valuable, Estimable, Small, and Testable.

The INVEST principle is a widely used checklist to assess the quality and readiness of a user story before it enters a sprint.

What does ‘Independent’ mean in the INVEST principle?

A user story should be self-contained and not rely on the completion of other stories.

Dependencies make prioritization and planning difficult. If stories are tightly coupled, the team should look for ways to combine them or split them along different boundaries.

Why must a user story be ‘Negotiable’?

Because a user story describes requirements, not implementation details or design decisions..

Developers and product owners collaborate and negotiate the implementation details just-in-time, allowing for better, more flexible solutions.

What makes a user story ‘Estimable’?

The development team must have enough information to roughly gauge the effort required to complete it.

If a story isn’t estimable, it usually means it is too large or poorly understood. The team needs more discussion or a technical spike to clarify the requirements.

Why is it crucial for a user story to be ‘Small’?

It must be appropriately sized to be completed within a single Agile iteration or sprint.

Smaller stories reduce delivery risk, provide faster feedback loops, and make estimation much more accurate. If a story is too big (an ‘Epic’), it must be broken down.

How do you ensure a user story is ‘Testable’?

By defining clear, objective Acceptance Criteria.

A story is only ‘Done’ when it can be verified. If you cannot test a story, you cannot prove that it successfully delivers the intended value to the user.

What is the widely used format for writing Acceptance Criteria?

The ‘Given [pre-condition] / When [action] / Then [post-condition]’ format.

This format structures criteria as clear scenarios: Given a specific context or starting state, When a specific action is performed, Then a specific, measurable outcome or result occurs.

What is the difference between the main body of the User Story and Acceptance Criteria?

A User Story describes the what and why of a feature from the user’s perspective, while Acceptance Criteria define the scope by listing all conditions that must be met for the story to be considered ‘Done’.

Think of the User Story as the goal and the Acceptance Criteria as the checklist to verify that the goal has been achieved. The story is the ‘what’ and ‘why’, while the criteria are the ‘how we know it’s done’.

Session Complete!

Your Score: 0/10

Come back later to improve your recall!

INVEST Criteria Violations Quiz

Test your ability to identify which of the INVEST principles are being violated in various Agile user stories, now including their associated Acceptance Criteria.

Read the following user story and its acceptance criteria: “As a customer, I want to pay for my items using a credit card, so that I can complete my purchase. (Note: This story cannot be implemented until the User Registration and Cart Management stories are fully completed).

Acceptance Criteria:

• Given a user has items in their cart, when they enter valid credit card details and submit, then the payment is processed and an order confirmation is shown.
• Given a user enters an expired credit card, when they submit, then the system displays an ‘invalid card’ error message.

Which INVEST criteria are violated? (Select all that apply)

Correct Answers:

Explanation

This story violates the Independent criterion because it is explicitly blocked by other incomplete stories. All other INVEST criteria are met.

Read the following user story and its acceptance criteria: “As a user, I want the application to be built using a React.js frontend, a Node.js backend, and a PostgreSQL database, so that I can view my profile.”

Acceptance Criteria:

• Given a user is logged in, when they navigate to the profile route, then the React.js components mount and display their data.
• Given a profile update occurs, when the form is submitted, then a REST API call is made to the Node.js server to update the PostgreSQL database.

Which INVEST criteria are violated? (Select all that apply)

Correct Answers:

Explanation

This violates Negotiable because it rigidly dictates the exact technical implementation rather than stating a problem for the developers to solve. It also violates Valuable, because the end-user generally derives no direct business value from the specific tech stack being used; their value comes solely from being able to view their profile.

Read the following user story and its acceptance criteria: “As a developer, I want to add a hidden ID column to the legacy database table that is never queried, displayed on the UI, or used by any background process, so that the table structure is updated.”

Acceptance Criteria:

• Given the database migration script runs, when the legacy table is inspected, then a new integer column named ‘hidden_id’ exists.
• Given the application is running, when any database operation occurs, then the ‘hidden_id’ column remains completely unused and unaffected.

Which INVEST criteria are violated? (Select all that apply)

Correct Answers:

Explanation

This heavily violates Valuable because doing work that has no impact, utility, or visible outcome provides no return on investment. It also violates Negotiable by prescribing a hyper-specific database tweak rather than a flexible user need, completely missing the spirit of a user story.

Read the following user story and its acceptance criteria: “As a hospital administrator, I want a comprehensive software system that includes patient records, payroll, pharmacy inventory management, and staff scheduling, so that I can run the entire hospital effectively.”

Acceptance Criteria:

• Given a doctor is logged in, when they search for a patient, then their full medical history is displayed.
• Given it is the end of the month, when HR runs payroll, then all staff are paid accurately.
• Given the pharmacy receives a shipment, when it is logged, then the inventory updates automatically.
• Given a nursing manager opens the calendar, when they drag and drop shifts, then the schedule is saved and notifications are sent to staff.

Which INVEST criteria are violated? (Select all that apply)

Correct Answers:

Explanation

This violates Small because it is a massive ‘Epic’ (with wide-ranging acceptance criteria) that would take months or years to build. Furthmore, it violates Testable because ‘run the entire hospital effectively’ is too broad and subjective to objectively verify in a single iteration. Because of this, it also violates Estimable, as the team cannot possibly guess the accurate effort for an entire hospital system in one go.

Read the following user story and its acceptance criteria: “As a website visitor, I want the homepage to load blazing fast and look extremely modern, so that I have a pleasant browsing experience.”

Acceptance Criteria:

• Given a user enters the website URL, when they press enter, then the page loads blazing fast.
• Given the homepage renders, when the user looks at the UI, then the design feels extremely modern and pleasant.

Which INVEST criteria are violated? (Select all that apply)

Correct Answers:

Explanation

This mainly violates Testable because ‘blazing fast’, ‘extremely modern’, and ‘pleasant’ are highly subjective; without objective metrics (like ‘loads in < 2 seconds’), you can’t test if it’s done. Consequently, this ambiguity causes it to violate Estimable, because developers cannot estimate the effort required to satisfy a vaguely defined standard of ‘modern’ or ‘fast’.

Quiz Complete!

Your Score: 0/5

Tools

---

Shell Scripting

---

Start here: If you are new to shell scripting, begin with the Interactive Shell Scripting Tutorial — hands-on exercises in a real Linux system. This article is a reference to deepen your understanding afterward.

If you have ever found yourself performing the same repetitive tasks on your computer—renaming batches of files, searching through massive text logs, or configuring system environments—then shell scripting is the magic wand you need. Shell scripting is the bedrock of system administration, software development workflows, and server management.

In this detailed educational article, we will explore the concepts, syntax, and power of shell scripting, specifically focusing on the most ubiquitous UNIX shell: Bash.

Basics

What is the Shell?

To understand shell scripting, you first need to understand the “shell”.

An operating system (like Linux, macOS, or Windows) acts as a middleman between the physical hardware of your computer and the software applications you want to run. It abstracts away the complex details of the hardware so developers can write functional software.

The kernel is the core of the operating system that interacts directly with the hardware. The shell, on the other hand, is a command-line interface (CLI) that serves as the primary gateway for users to interact with a computer’s operating system. While many modern users are accustomed to graphical user interfaces (GUIs), the shell is a program that specifically takes text-based user commands and passes them to the operating system to execute. In the context of this course, mastering the shell is like becoming a “wizard” who can construct and manipulate complex software systems simply by typing words.

Motivation: Why the Shell is Essential

As a software engineer, you need to be familiar with the ecosystem of tools that help you build software efficiently. The Linux ecosystem offers a vast array of specialized tools that allow you to write programs faster and debug log files by combining small, powerful commands. Understanding the shell increases your productivity in a professional environment and provides a foundation for learning other domain-specific scripting languages. Furthermore, the shell allows you to program directly on the operating system without the overhead of additional interpreters or heavy libraries.

The Unix Philosophy

The shell’s power is rooted in the Unix philosophy, which dictates:

1. Write programs that do one thing and do it well.
2. Write programs to work together.
3. Write programs to handle text streams, because that is a universal interface.

By treating data as a sequence of characters or bytes—similar to a conveyor belt rather than a truck—the shell allows parallel processing and the composition of complex behaviors from simple parts.

Essential UNIX Commands

Before writing scripts, you need to know the fundamental commands that you will be stringing together. These are the building blocks of any UNIX environment.

1. File Handling

These are the foundational tools for interacting with the POSIX filesystem:

• ls: List directory contents (files and other directories).
• cd: Change the current working directory (e.g., use .. to move to a parent folder).
• pwd: Print the name of the current/working directory so you don’t get lost.
• mkdir: Create a new directory.
• cp: Copy files. Use -r (recursive) to copy a directory and its contents.
• mv: Move or rename files and directories.
• rm: Remove (delete) files. Use -r to remove a directory and its contents recursively.
• rmdir: Remove empty directories (only works on empty ones).
• touch: Create an empty file or update timestamps.

2. Text Processing and Data Manipulation

Unix treats text streams as a universal interface, and these tools allow you to transform that data:

• cat: Concatenate and print files to standard output.
• grep: Search for patterns using regular expressions.
• sed: Stream editor for filtering and transforming text (commonly search-and-replace).
• tr: Translate or delete characters (e.g., changing case or removing digits).
• sort: Sort lines of text files alphabetically; add -n for numeric order, -r to reverse.
• uniq: Filter adjacent duplicate lines; the -c flag prefixes each line with its occurrence count. Because it only compares consecutive lines, you almost always pipe sort first so that duplicates are adjacent.
• wc: Word count (lines, words, characters).
• cut: Extract specific sections/fields from lines.
• comm: Compare two sorted files line by line.
• head / tail: Output the first or last part of files.
• awk: Advanced pattern scanning and processing language.

3. Permissions, Environment, and Documentation

These tools manage how your shell operates and how you access information:

• man: Access the manual pages for other commands. This is arguably the most useful command, providing built-in documentation for every other command in the system.
• chmod: Change file mode bits (permissions). Files in a Unix-like system have three primary types of permissions: read (r), write (w), and execute (x). For security reasons, the system requires an explicit execute permission because you do not want to accidentally run a file from an unknown source. Permissions are often read in “bits” for the owner (u), group (g), and others (o).
• which / type: Locate the binary or type for a command.
• export: Set environment variables. The PATH variable is especially important; it tells the shell which directories to search for executable programs. You can temporarily update it using export or make it permanent by adding the command to your ~/.bashrc or ~/.profile file.
• source / .: Execute commands from a file in the current shell environment.

4. System, Networking, and Build Tools

Tools used for remote work, debugging, and automating the construction process:

• ssh: Secure shell to connect to remote machines like SEASnet.
• scp: Securely copy files between hosts.
• wget2 / curl: Download files or data from the internet.
• make: Build automation tool that uses shell-like syntax to manage the incremental build process of complex software, ensuring that only changed files are recompiled.
• gcc / clang: C/C++ compilers.
• tar: Manipulate tape archives (compressing/decompressing).

The Power of I/O Redirection and Piping

The true power of the shell comes from connecting commands. Every shell program typically has three standard stream ports:

1. Standard Input (stdin / 0): Usually the keyboard.
2. Standard Output (stdout / 1): Usually the terminal screen.
3. Standard Error (stderr / 2): Where error messages go, also usually the terminal.

Redirection

You can redirect these streams using special operators:

• >: Redirects stdout to a file, overwriting it. (e.g., echo "Hello" > file.txt)
• >>: Redirects stdout to a file, appending to it without overwriting.
• <: Redirects stdin from a file. (e.g., cat < input.txt)
• 2>: Redirects stderr to a specific file to specifically log errors.
• 2>&1: Redirects stderr to the standard output stream. Note: order matters — command > file.txt 2>&1 sends both streams to the file, whereas command 2>&1 > file.txt only redirects stdout to the file while stderr still goes to the terminal.

Piping

The pipe operator | is the most powerful composition tool. It takes the stdout of the command on the left and sends it directly into the stdin for the command on the right.

Example: cat access.log | grep "ERROR" | wc -l This pipeline reads a log file, filters only the lines containing “ERROR”, and then counts how many lines there are.

Here Documents and Here Strings

Sometimes you need to feed a block of text directly into a command without creating a temporary file. A here document (<<) lets you embed multi-line input inline, up to a chosen delimiter:

cat <<EOF
Server: production
Version: 1.4.2
Status: running
EOF

The shell expands variables inside the block (just like double quotes). To suppress expansion, quote the delimiter: <<'EOF'.

A here string (<<<) feeds a single expanded string to a command’s standard input — a concise alternative to echo "text" | command:

grep "ERROR" <<< "08:15:45 ERROR failed to connect"

Process Substitution

Advanced shell users often utilize process substitution to treat the output of a command as a file. The syntax looks like <(command). For example, H < <(G) >> I allows you to refer to the standard output of command G as a file, redirect it into the standard input of H, and append the output to I.

Writing Your First Shell Script

When you find yourself typing the same commands repeatedly, you should create a shell script. A shell script is written in a plain text file (often ending in .sh) and contains a sequence of commands that the shell executes as a program.

Interpreted Nature

Unlike a compiled language like C++, which is compiled into machine code before execution, shell scripts are interpreted at runtime rather than ahead of time. This allows for rapid prototyping. Bash always reads at least one complete line of input, and reads all lines that make up a compound command (such as an if block or for loop) before executing any of them. This means a syntax error on a later line inside a multi-line compound block is caught before the block starts executing — but an error in a branch that is never reached at runtime may go unnoticed. Use bash -n script.sh to check for syntax errors without running the script.

The Shebang

Every script should start with a “shebang” (#!). This tells the operating system which interpreter should be used to run the script. For Bash scripts, the first line should be:

#!/bin/bash

Execution Permissions

By default, text files are not executable for security reasons. Execute permission is required only if you want to run the script directly as a command:

chmod +x myscript.sh
./myscript.sh

Alternatively, you can bypass the execute-permission requirement entirely by passing the file as an argument to the Bash interpreter directly — no chmod needed:

bash myscript.sh

You can also run a script’s commands within the current shell (inheriting and potentially modifying its environment) using source or the . builtin: source myscript.sh.

Debugging Scripts

When a script behaves unexpectedly, Bash has built-in tracing modes that let you see exactly what the shell is doing:

• bash -n script.sh: Reads the script and checks for syntax errors without executing any commands. Always run this first when a script refuses to start.
• bash -x script.sh (or set -x inside the script): Prints a trace of each command and its expanded arguments to stderr before executing it — indispensable for logic bugs. Each traced line is prefixed with +.
• bash -v script.sh (or set -v): Prints each line of input exactly as read, before expansion — useful for seeing the raw source being interpreted.

You can combine flags: bash -xv script.sh. To turn tracing on for only a section of a script, use set -x before that section and set +x after it.

Error Handling (set -e and Exit Status)

By default, a Bash script will continue executing even if a command fails. Every command returns a numerical code known as an Exit Status; 0 generally indicates success, while any non-zero value indicates an error or failure. Continuing after a failure can be dangerous and lead to unexpected behavior. To prevent this, you should typically include set -e at the top of your scripts:

#!/bin/bash
set -e

This tells the shell to exit immediately if any simple command fails, making your scripts safer and more predictable.

Syntax and Programming Constructs

Bash is a full-fledged programming language, but because it is an interpreted scripting language rather than a compiled language (like C++ or Java), its syntax and scoping rules are quite different.

5. Scripting Constructs

In our scripts, we also treat these keywords as “commands” for building logic:

• #! (Shebang): An OS-level interpreter directive on the first line of a script file — not a Bash keyword or command. When the OS executes the file, it reads #! and uses the rest of that line as the interpreter path. Within Bash itself, any line starting with # is simply a comment and is ignored.
• read: Read a line from standard input into a variable. Common flags: -p "prompt" displays a prompt on the same line, -s silently hides typed input (useful for passwords), and -n 1 returns after exactly one character instead of waiting for Enter.
• if / then / elif / else / fi: Conditional execution.
• for / do / done / while: Looping constructs.
• case / in / esac: Multi-way branching on a single value.
• local: Declare a variable scoped to the current function.
• return: Exit a function with a numeric status code.
• exit: Terminate the script with a specific status code.

Variables

You can assign values to variables without declaring a type. Note that there are no spaces around the equals sign in Bash.

NAME="Ada"
echo "Hello, $NAME"

Parameter Expansion — Default Values and String Manipulation

Beyond simple $VAR substitution, Bash supports a powerful set of parameter expansion operators that let you handle missing values and manipulate strings entirely within the shell, without spawning external tools.

Default values:

# Use "server_log.txt" if $1 is unset or empty
file="${1:-server_log.txt}"

# Use "anonymous" if $NAME is unset or empty, AND assign it
NAME="${NAME:=anonymous}"

String trimming — remove a pattern from the start (#) or end (%) of a value:

path="/home/user/project/main.sh"
filename="${path##*/}"    # removes longest prefix up to last /  → "main.sh"
noext="${filename%.*}"    # removes shortest suffix from last .  → "main"

The double form (## / %%) removes the longest match; the single form (# / %) removes the shortest.

Search and replace:

msg="Hello World World"
echo "${msg/World/Earth}"    # replaces first match  → "Hello Earth World"
echo "${msg//World/Earth}"   # replaces all matches  → "Hello Earth Earth"

Scope Differences

Unlike C++ or Java, Bash lacks strict block-level scoping (like {} blocks). Variables assigned anywhere in a script — including inside if statements and loops — remain accessible throughout the entire script’s global scope. There are, however, several important isolation boundaries:

• Function-level scoping: variables declared with the local builtin inside a Bash function are visible only to that function and its callees.
• Subshells: commands grouped with ( list ), command substitutions $(...), and background jobs run in a subshell — a copy of the shell environment. Any variable assignments made inside a subshell do not propagate back to the parent shell.
• Per-command environment: a variable assignment placed immediately before a simple command (e.g., VAR=value command) is only visible to that command for its duration, leaving the surrounding scope untouched.

Arithmetic

Math in Bash is slightly idiosyncratic. While a language like C++ operates directly on integers with + or /, arithmetic in Bash needs to be enclosed within $(( ... )) or evaluated using the let command.

x=5
y=10
sum=$((x + y))
echo "The sum is $sum"

Control Structures: If-Statements and Loops

Bash supports standard control flow constructs.

If-Statements:

if [ "$sum" -gt 10 ]; then
    echo "Sum is greater than 10"
elif [ "$sum" -eq 10 ]; then
    echo "Sum is exactly 10"
else
    echo "Sum is less than 10"
fi

[ is a shell builtin command: The single bracket [ is not special syntax — it is a builtin command, a synonym for test. Because Bash implements it internally, its arguments must be separated by spaces just like any other command: [ -f "$file" ] is correct, but [-f "$file"] tries to run a command named [-f, which fails. This is why the spaces inside brackets are mandatory, not just stylistic. (An external binary /usr/bin/[ also exists on most systems, but Bash uses its builtin by default — you can verify with type -a [.)

The following table covers the most important tests available inside [ ]:

Test	Meaning
-f path	Path exists and is a regular file
-d path	Path exists and is a directory
-z "$var"	String is empty (zero length)
"$a" = "$b"	Strings are equal
"$a" != "$b"	Strings are not equal
$x -eq $y	Integers are equal
$x -gt $y	Integer greater than
$x -lt $y	Integer less than
! condition	Logical NOT (negates the test)

Important: use -eq, -lt, -gt for numbers and = / != for strings. Mixing them produces wrong results silently.

[ vs [[: The double bracket [[ ... ]] is a Bash keyword with additional power: it does not perform word splitting on variables, allows && and || inside the condition, and supports regex matching with =~. Prefer [[ ]] in new Bash scripts.

Loops:

for i in 1 2 3 4 5; do
    echo "Iteration $i"
done

For numeric ranges, the C-style for loop (the arithmetic for command) is often cleaner:

for (( i=1; i<=5; i++ )); do
    echo "Iteration $i"
done

This is a distinct looping construct from the standalone (( )) arithmetic compound command. In this form, expr1 is evaluated once at start, expr2 is tested before each iteration (loop runs while non-zero), and expr3 is evaluated after each iteration — the same semantics as C’s for loop.

Loop control keywords:

• break: Exit the loop immediately, regardless of the remaining iterations.
• continue: Skip the rest of the current iteration and jump to the next one.

for f in *.log; do
    [ -s "$f" ] || continue    # skip empty files
    grep -q "ERROR" "$f" || continue
    echo "Errors found in: $f"
done

Quoting and Word Splitting

How you quote text profoundly changes how Bash interprets it — this is one of the most common sources of bugs in shell scripts.

• Single quotes ('...'): All characters are literal. No variable or command substitution occurs. echo 'Cost: $5' prints exactly Cost: $5.
• Double quotes ("..."): Spaces are preserved, but $VARIABLE and $(command) are still expanded. echo "Hello $USER" prints Hello Ada.

A critical pitfall is word splitting: when you reference an unquoted variable, the shell splits its value on whitespace and treats each word as a separate argument. Consider:

FILE="my report.pdf"
rm $FILE      # WRONG: shell splits into two args: "my" and "report.pdf"
rm "$FILE"    # CORRECT: the entire value is passed as one argument

Always quote variable references with double quotes to protect against word splitting.

Command Substitution

Command substitution captures the standard output of a command and uses it as a value in-place. The modern syntax is $(command):

TODAY=$(date +%Y-%m-%d)
echo "Backup started on: $TODAY"

The shell runs the inner command in a subshell, then replaces the entire $(...) expression with its output. This is the standard way to assign the results of commands to variables.

Positional Parameters and Special Variables

Scripts receive command-line arguments via positional parameters. If you run ./backup.sh /src /dest, then inside the script:

Variable	Value	Description
$0	./backup.sh	Name of the script itself
$1	/src	First argument
$2	/dest	Second argument
$#	2	Total number of arguments passed
$@	/src /dest	All arguments as separate, properly-quoted words
$?	(exit code)	Exit status of the most recent command

When iterating over all arguments, always use "$@" (quoted). Without quotes, $@ is subject to word splitting and arguments containing spaces are silently broken into multiple words:

for f in "$@"; do
    echo "Processing: $f"
done

Command Chaining with && and ||

Because every command returns an exit status, you can chain commands conditionally without writing a full if/then/fi block:

• && (AND): The right-hand command runs only if the left-hand command succeeds (exit code 0). mkdir output && echo "Directory created" — only prints if mkdir succeeded.
• || (OR): The right-hand command runs only if the left-hand command fails (non-zero exit code). cd /target || exit 1 — exits the script immediately if the directory cannot be entered.

This compact chaining idiom is widely used in professional scripts for concise, readable error handling.

Background Jobs

Appending & to a command runs it asynchronously — the shell launches it in the background and immediately returns to the prompt without waiting for it to finish:

./long_running_build.sh &
echo "Build started, continuing with other work..."

Two special variables are useful when managing background processes:

• $$: The process ID (PID) of the current shell itself. Often used to create unique temporary file names: tmp_file="/tmp/myscript.$$".
• $!: The PID of the most recently backgrounded job. Use it to wait for or kill a specific background process.

The jobs command lists all active background jobs; fg brings the most recent one back to the foreground, and bg resumes a stopped job in the background.

Functions — Reusable Building Blocks

When the same logic appears in multiple places, extract it into a function. Functions in Bash work like small scripts-within-a-script: they accept positional arguments via $1, $2, etc. — independently of the outer script’s own arguments — and can be called just like any other command.

greet() {
    local name="$1"
    echo "Hello, ${name}!"
}

greet "engineer"   # → Hello, engineer!

The local Keyword

Without local, any variable set inside a function leaks into and overwrites the global script scope. Always declare function-internal variables with local to prevent subtle bugs:

process() {
    local result="$1"   # visible only inside this function
    echo "$result"
}

Returning Values from Functions

The return statement only carries a numeric exit code (0–255), not data. To pass a string back to the caller, have the function echo the value and capture it with command substitution:

to_upper() {
    echo "$1" | tr '[:lower:]' '[:upper:]'
}

loud=$(to_upper "hello")   # loud="HELLO"

You can also use functions directly in if statements, because a function’s exit code is treated as its truth value: return 0 is success (true), return 1 is failure (false).

Case Statements — Readable Multi-Way Branching

When you need to check one variable against many possible values, a case statement is far cleaner than a chain of if/elif:

case "$command" in
    start)   echo "Starting service..."  ;;
    stop)    echo "Stopping service..."  ;;
    status)  echo "Checking status..."   ;;
    *)       echo "Unknown command: $command" >&2; exit 2 ;;
esac

Each branch ends with ;;. The * pattern is the catch-all default, matching any value not handled by earlier branches. The block closes with esac (case backwards).

Exit Codes — The Language of Success and Failure

Every command — including your own scripts — exits with a number. 0 always means success; any non-zero value means failure. This is the opposite of most programming languages where 0 is falsy. Conventional exit codes are:

Code	Meaning
0	Success
1	General error
2	Misuse — wrong arguments or invalid input

Meaningful exit codes make scripts composable: other scripts, CI pipelines, and tools like make can call your script and take action based on the result. For example, ./monitor.sh || alert_team only triggers the alert when your monitor exits non-zero.

Shell Expansions — Brace Expansion and Globbing

The shell performs several rounds of expansion on a command line before executing it. Understanding the order helps you predict and control what the shell does.

Brace Expansion

First comes brace expansion, which generates arbitrary lists of strings. It is a purely textual operation — no files need to exist:

mkdir project/{src,tests,docs}      # creates three directories at once
cp config.yml config.yml.{bak,old}  # copies to two names simultaneously
echo {1..5}                          # → 1 2 3 4 5  (sequence expression)

Brace expansion happens before all other expansions, so you can combine it freely with variables and globbing.

Supercharging Scripts with Regular Expressions

Because the UNIX philosophy is heavily centered around text streams, text processing is a massive part of shell scripting. Regular Expressions (RegEx) is a vital tool used within shell commands like grep, sed, and awk to find, validate, or transform text patterns quickly.

Globbing vs. Regular Expressions: These look similar but are entirely different systems. Globbing (filename expansion) uses *, ?, and [...] to match filenames — the shell expands these before the command runs (e.g., rm *.log deletes all .log files). The three special pattern characters are: * matches any string (including empty), ? matches any single character, and [ opens a bracket expression [...] that matches any one of the enclosed characters — e.g., [a-z] matches any lowercase letter, and [!a-z] matches any character that is not a lowercase letter. Regular Expressions use ^, $, .*, [0-9]+, and similar constructs — they are pattern languages used by tools like grep, sed, and awk, and also natively by Bash itself via the =~ operator inside [[ ]] conditionals (which evaluates POSIX extended regular expressions directly without spawning an external tool). Critically, * means “match anything” in globbing, but “zero or more of the preceding character” in RegEx.

RegEx allows you to match sub-strings in a longer sequence. Critical to this are anchors, which constrain matches based on their location:

• ^ : Start of string. (Does not allow any other characters to come before).
• $ : End of string.

Example: ^[a-zA-Z0-9]{8,}$ validates a password that is strictly alphanumeric and at least 8 characters long, from the exact beginning of the string to the exact end.

Conclusion

Shell scripting is an indispensable skill for anyone working in tech. By viewing the shell as a set of modular tools (the “Infinity Stones” of your development environment), you can combine simple operations to perform massive, complex tasks with minimal effort. Start small by automating a daily chore on your machine, and before you know it, you will be weaving complex UNIX tools together with ease!

Quiz

Shell Commands Flashcards

Which Shell command would you use for the following scenarios?

You need to see a list of all the files and folders in your current directory. What command do you use?

ls

The ls command lists directory contents. You can also use flags like ls -l for a detailed list or ls -a to see hidden files.

You are currently in your home directory and need to navigate into a folder named ‘Documents’. Which command achieves this?

cd Documents

The cd (change directory) command is used to change the current working directory in the command line operating system.

You want to quickly view the entire contents of a small text file named ‘config.txt’ printed directly to your terminal screen.

cat config.txt

The cat (concatenate) command reads data from the file and gives its content as output. It is widely used to display file contents.

You need to find every line containing the word ‘ERROR’ inside a massive log file called ‘server.log’.

grep 'ERROR' server.log

grep searches for patterns in each file. It prints each line that matches the given pattern, making it invaluable for parsing logs.

You wrote a new bash script named ‘script.sh’, but when you try to run it, you get a ‘Permission denied’ error. How do you make the file executable?

chmod +x script.sh

The chmod command changes file modes or Access Control Lists. The +x flag specifically adds execute permissions to the file.

You want to rename a file from ‘draft_v1.txt’ to ‘final_version.txt’ without creating a copy.

mv draft_v1.txt final_version.txt

The mv command is used to move files or directories, but it is also the standard command used to rename a file by moving it to a new name in the same directory.

You are starting a new project and need to create a brand new, empty folder named ‘src’ in your current location.

mkdir src

mkdir stands for ‘make directory’. It creates a new directory with the specified name, provided you have the correct permissions.

You want to view the contents of a very long text file called ‘manual.txt’ one page at a time so you can scroll through it.

less manual.txt

The less command is a terminal pager program that allows viewing (but not editing) the contents of a text file one screen at a time, allowing both forward and backward navigation.

You need to create an exact duplicate of a file named ‘report.pdf’ and save it as ‘report_backup.pdf’.

cp report.pdf report_backup.pdf

The cp (copy) command is used to copy files or directories from one location to another.

You have a temporary file called ‘temp_data.csv’ that you no longer need and want to permanently delete from your system.

rm temp_data.csv

The rm (remove) command deletes files or directories. Use with caution, as deleted files are generally not recoverable.

You want to quickly print the phrase ‘Hello World’ to the terminal or pass that string into a pipeline.

echo 'Hello World'

The echo command outputs the strings it is being passed as arguments. It is commonly used in scripts to display messages or write data to a file using redirection.

You want to know exactly how many lines are contained within a file named ‘essay.txt’.

wc -l essay.txt

The wc (word count) command prints newline, word, and byte counts for a file. Adding the -l flag restricts the output to only the line count.

You need to perform an automated find-and-replace operation on a stream of text to change the word ‘apple’ to ‘orange’.

sed 's/apple/orange/g'

The sed (stream editor) s: Stands for ‘substitute’. /: The delimiter that separates the search and replacement terms. apple: The string you want to find. orange: The string you want to replace it with. g: Stands for ‘global’. This tells sed to replace every instance of ‘apple’ on a line, rather than just the first one it finds.

You have a space-separated log file and want a tool to extract and print only the 3rd column of data.

awk '{print $3}'

awk is a versatile command-line utility and programming language designed for text processing and data extraction, particularly excelling at handling structured, column-based data.

You want to store today’s date (formatted as YYYY-MM-DD) in a variable called TODAY so you can use it to name a backup file dynamically.

TODAY=$(date +%Y-%m-%d)

Command substitution $(...) executes the inner command in a subshell and replaces itself with the command’s standard output. This is the standard way to capture the result of a command into a variable.

A variable FILE holds the value my report.pdf. Running rm $FILE fails with a ‘No such file or directory’ error for both ‘my’ and ‘report.pdf’. How do you fix this?

rm "$FILE"

Without quotes, the shell performs word splitting on the expanded variable, breaking my report.pdf into two separate arguments: my and report.pdf. Wrapping the variable in double quotes ("$FILE") prevents word splitting and passes the entire value as a single argument.

You are writing a script that requires exactly two arguments. How do you check how many arguments were passed to the script so you can print a usage error if the count is wrong?

$#

$# expands to the total count of positional arguments passed to the script. A typical guard looks like: if [ $# -ne 2 ]; then echo 'Usage: script.sh src dest'; exit 1; fi. Other useful special variables: $1, $2 (individual args), $0 (script name), $@ (all args).

You want to create a directory called ‘build’ and then immediately run cmake .. inside it, but only if the directory creation succeeded — all in a single command.

mkdir build && cd build && cmake ..

The && (AND) operator runs the next command only if the previous one succeeded (exit code 0). If mkdir fails, neither cd nor cmake will execute. This is a concise alternative to a full if/then/fi block for simple success-guarded sequences.

At the start of a script, you need to change into /deploy/target. If that directory doesn’t exist, the script must abort immediately — write a defensive one-liner.

cd /deploy/target || exit 1

The || (OR) operator runs the right-hand command only if the left-hand command fails (non-zero exit code). This is the standard idiom for aborting a script when a critical precondition is not met, without needing a full if statement.

You want to delete all files ending in .tmp in the current directory using a single command, without listing each filename explicitly.

rm *.tmp

The * wildcard is a glob pattern — it is expanded by the shell itself (filename expansion) before rm runs. Globbing is entirely separate from regular expressions: in RegEx, * means ‘zero or more of the preceding character’, not ‘match anything’.

Session Complete!

Your Score: 0/20

Come back later to improve your recall!

Self-Assessment Quiz: Shell Scripting & UNIX Philosophy

Test your conceptual understanding of shell environments, data streams, and scripting paradigms beyond basic command memorization.

A developer needs to parse a massive log file, extract IP addresses, sort them, and count unique occurrences. Instead of writing a 500-line Python script, they use cat | awk | sort | uniq -c. Why is this approach fundamentally preferred in the UNIX environment?

Correct Answer:

Explanation

The UNIX design philosophy revolves around writing programs that ‘do one thing and do it well’, and having them work together by handling text streams. Pipelines allow complex tasks to be solved elegantly by chaining these simple tools together.

A script runs a command that generates both useful output and a flood of permission error messages. The user runs script.sh > output.txt, but the errors still clutter the terminal screen while the useful data goes to the file. What underlying concept explains this behavior?

Correct Answer:

Explanation

In UNIX, processes have three default streams: stdin, stdout, and stderr. Redirection with > only captures stdout. To capture the errors, you would need to redirect stderr specifically (e.g., using 2>).

A C++ developer writes a Bash script with a for loop. Inside the loop, they declare a variable temp_val. After the loop finishes, they try to print temp_val expecting it to be undefined or empty, but it prints the last value assigned in the loop. Why did this happen?

Correct Answer:

Explanation

Bash does not use block-level scoping. Unless explicitly declared as local inside a function, variables assigned within control structures like loops or conditionals bleed into the rest of the script.

You want to use a command that requires two file inputs (like diff), but your data is currently coming from the live outputs of two different commands. Instead of creating temporary files on the disk, you use the <(command) syntax. What is this concept called and what does it achieve?

Correct Answer:

Explanation

Process substitution <() allows the standard output of a command to be treated as a file. The operating system creates a temporary file descriptor (like /dev/fd/63) that the consuming command can read from as if it were a physical file.

A script contains entirely valid Python code, but the file is named script.sh and has #!/bin/bash at the very top. When executed via ./script.sh, the terminal throws dozens of ‘command not found’ and syntax errors. What is the fundamental misunderstanding here?

Correct Answer:

Explanation

In UNIX systems, file extensions are largely superficial. The shebang (#!) on the first line is what tells the operating system’s program loader which interpreter program (e.g., Bash, Python, Node) to use to parse the rest of the file.

A developer uses the regular expression [0-9]{4} to validate that a user’s input is exactly a four-digit PIN. However, the system incorrectly accepts ‘12345’ and ‘A1234’. What crucial RegEx concept did the developer omit?

Correct Answer:

Explanation

Without anchors, a regular expression looks for a match anywhere within the string. Because ‘12345’ contains ‘1234’, the pattern matches. Adding ^ (start) and $ (end) ensures the pattern must account for the entire string.

You are designing a data pipeline in the shell. Which of the following statements correctly describe how UNIX handles data streams and command chaining? (Select all that apply)

Correct Answers:

Explanation

Pipes (|) link stdout to stdin, >> appends stdout to a file, and < feeds a file into stdin. By default, pipes do NOT pass stderr (errors will still print to the screen). >! is not the standard way to redirect both streams in Bash (typically >& or &> is used).

You’ve written a shell script deploy.sh but it throws a ‘Permission denied’ error or fails to run when you type ./deploy.sh. Which of the following are valid reasons or necessary steps to successfully execute a script as a standalone program? (Select all that apply)

Correct Answers:

Explanation

Scripts require execute permissions (chmod +x) to be run as programs, and a shebang tells the OS how to interpret the text. They are interpreted, not compiled. The ./ explicitly tells the OS to look in the current directory, bypassing the need for the directory to be in the $PATH.

In Bash, exit codes are crucial for determining if a command succeeded or failed. Which of the following statements are true regarding how Bash handles exit statuses and control flow? (Select all that apply)

Correct Answers:

Explanation

In UNIX, an exit code of 0 means success, while non-zero (like 1) means failure/error. if statements work by running a command and proceeding if its exit code is 0. set -e is a common safety mechanism to fail fast if an error occurs.

When you type a command like python or grep into the terminal, the shell knows exactly what program to run without you providing the full file path. How does the $PATH environment variable facilitate this, and how is it managed? (Select all that apply)

Correct Answers:

Explanation

$PATH is an explicit list of directories, not a global search crawler (which would be extremely slow). The OS checks these directories in order. Modifications in the terminal are bound to that specific session unless saved in a configuration file like .bashrc.

A developer writes LOGFILE="access errors.log" and then runs wc -l $LOGFILE. The command fails with ‘No such file or directory’ errors for both ‘access’ and ‘errors.log’. What is the root cause?

Correct Answer:

Explanation

When a variable is expanded without double quotes, the shell splits the result on whitespace. wc -l $LOGFILE is interpreted as wc -l access errors.log — two separate arguments. The fix is to always quote variable expansions: wc -l "$LOGFILE".

A script is invoked with ./deploy.sh production 8080 myapp. Inside the script, which variable holds the value 8080?

Correct Answer:

Explanation

$0 is the script name (./deploy.sh), $1 is the first argument (production), $2 is the second argument (8080), and $3 would be myapp. $# holds the total argument count (3).

A script contains the line: cd /deploy/target && ./run_tests.sh && echo 'All tests passed!'. If ./run_tests.sh exits with a non-zero status code, what happens next?

Correct Answer:

Explanation

The && operator short-circuits on failure. Since ./run_tests.sh returned a non-zero exit code, the shell skips all remaining &&-linked commands. Only if every command in the chain succeeds does echo 'All tests passed!' execute.

Which of the following statements correctly describe Bash quoting and command substitution behavior? (Select all that apply)

Correct Answers:

Explanation

Single quotes suppress all expansion; double quotes allow variable and command substitution while protecting spaces. Quoting "$VARIABLE" is essential to prevent word splitting on values containing spaces. Single and double quotes are NOT interchangeable when variable expansion is needed. $(command) is command substitution — the standard way to capture command output into a variable.

Quiz Complete!

Your Score: 0/14

After finishing these quizzes, you are now ready to practice in a real Linux system. Try the Interactive Shell Scripting Tutorial!

Regular Expressions

---

New to RegEx? Start here: The RegEx Tutorial: Basics teaches you Regular Expressions step by step with hands-on exercises and real-time feedback. Then continue with the Advanced Tutorial for greedy/lazy matching, groups, lookaheads, and integration challenges. Come back to this page as a reference.

This page is a reference guide for Regular Expression syntax, engine mechanics, and worked examples. It is designed to be consulted alongside or after the interactive tutorial — not as a replacement for hands-on practice.

Overview

The Core Purpose of RegEx

At its heart, RegEx solves three primary problems in software engineering:

1. Validation: Ensuring user input matches a required format (e.g., verifying an email address or checking if a password meets complexity rules).
2. Searching & Parsing: Finding specific substrings within a massive text document or extracting required data (e.g., scraping phone numbers from a website).
3. Substitution: Performing advanced search-and-replace operations (e.g., reformatting dates from YYYY-MM-DD to MM/DD/YYYY).

The Conceptual Power of Pattern Matching: What RegEx Actually Does

Before we dive into the specific symbols and syntax, we need to understand the fundamental shift in thinking required to use Regular Expressions.

When we normally search through text (like using Ctrl + F or Cmd + F in a word processor), we perform a Literal Search. If you search for the word cat, the computer looks for the exact character c, followed immediately by a, and then t.

However, real-world data is rarely that predictable. Regular Expressions allow you to perform a Structural Search. Instead of telling the computer exactly what characters to look for, you describe the shape, rules, and constraints of the text you want to find.

Let’s look at one simple and two complex examples to illustrate this conceptual leap.

The Simple Example: The “Cat” Problem

Imagine you are proofreading a document and want to find every instance of the animal “cat.”

If you do a literal search for cat, your text editor will highlight the “cat” in “The cat is sleeping,” but it will also highlight the “cat” in “catalog”, “education”, and “scatter”. Furthermore, a literal search for cat will completely miss the plural “cats” or the capitalized “Cat”.

Conceptually, a Regular Expression allows you to tell the computer:

“Find the letters C-A-T (ignoring uppercase or lowercase), but only if they form their own distinct word, and optionally allow an ‘s’ at the very end.” By defining the rules of the word rather than just the literal letters, RegEx eliminates the false positives (“catalog”) and captures the edge cases (“Cats”).

Complex Example 1: The Phone Number Problem

Suppose you are given a massive spreadsheet of user data and need to extract everyone’s phone number to move into a new database. The problem? The users typed their phone numbers however they wanted. You have:

• 123-456-7890
• (123) 456-7890
• 123.456.7890
• 1234567890

A literal search is useless here. You cannot Ctrl + F for a phone number if you don’t already know what the phone number is!

With RegEx, you don’t search for the numbers themselves. Instead, you describe the concept of a North American phone number to the engine:

“Find a sequence of exactly 3 digits (which might optionally be wrapped in parentheses). This might be followed by a space, a dash, or a dot, but it might not. Then find exactly 3 more digits, followed by another optional space, dash, or dot. Finally, find exactly 4 digits.”

With one single Regular Expression, the engine will scan millions of lines of text and perfectly extract every phone number, regardless of how the user formatted it, while ignoring random strings of numbers like zip codes or serial numbers.

Complex Example 2: The Server Log Problem

Imagine you are a backend engineer, and your company’s website just crashed. You are staring at a server log file containing 500,000 lines of system events, timestamps, IP addresses, and status codes. You need to find out which specific IP addresses triggered a “Critical Timeout” error in the last hour.

The data looks like this: [2023-10-25 14:32:01] INFO - IP: 192.168.1.5 - Status: OK [2023-10-25 14:32:05] ERROR - IP: 10.0.4.19 - Status: Critical Timeout

You can’t just search for “Critical Timeout” because that won’t extract the IP address for you. You can’t search for the IP address because you don’t know who caused the error.

Conceptually, RegEx allows you to create a highly specific, multi-part extraction rule:

“Scan the document. First, find a timestamp that falls between 14:00:00 and 14:59:59. If you find that, keep looking on the same line. If you see the word ‘ERROR’, keep going. Find the letters ‘IP: ‘, and then permanently capture and save the mathematical pattern of an IP address (up to three digits, a dot, up to three digits, etc.). Finally, ensure the line ends with the exact phrase ‘Critical Timeout’. If all these conditions are met, hand me back the saved IP address.”

This is the true power of Regular Expressions. It transforms text searching from a rigid, literal matching game into a highly programmable, logic-driven data extraction pipeline.

The Anatomy of a Regular Expression

A regular expression is composed of two types of characters:

• Literal Characters: Characters that match themselves exactly (e.g., the letter a matches the letter “a”).
• Metacharacters: Special characters that have a unique meaning in the pattern engine (e.g., *, +, ^, $).

Let’s explore the most essential metacharacters and constructs.

Anchors: Controlling Position

Anchors do not match any actual characters; instead, they constrain a match based on its position in the string.

• ^ (Caret): Asserts the start of a string. ^Hello matches “Hello world” but not “Say Hello”.
• $ (Dollar Sign): Asserts the end of a string. end$ matches “The end” but not “endless”.

Practice this: Anchors exercises in the Interactive Tutorial

Character Classes: Matching Sets of Characters

Character classes (or sets) allow you to match any single character from a specified group.

• [abc]: Matches either “a”, “b”, or “c”.
• [a-z]: Matches any lowercase letter.
• [A-Za-z0-9]: Matches any alphanumeric character.
• [^0-9]: The caret inside the brackets means negation. This matches any character that is not a digit.

Practice this: Character Classes exercises in the Interactive Tutorial

Metacharacters

Because certain character sets are used so frequently, RegEx provides handy meta characters:

• \d: Matches any digit (equivalent to [0-9]).
• \w: Matches any “word” character (alphanumeric plus underscore: [a-zA-Z0-9_]).
• \s: Matches any whitespace character (spaces, tabs, line breaks).
• . (Dot): The wildcard. Matches any single character except a newline. (To match a literal dot, you must escape it with a backslash: \.).

Practice this: Meta Characters exercises in the Interactive Tutorial

Quantifiers: Controlling Repetition

Quantifiers tell the RegEx engine how many times the preceding element is allowed to repeat.

• * (Asterisk): Matches 0 or more times. (a* matches “”, “a”, “aa”, “aaa”)
• + (Plus): Matches 1 or more times. (a+ matches “a”, “aa”, but not “”)
• ? (Question Mark): Matches 0 or 1 time (makes the preceding element optional).
• {n}: Matches exactly n times.
• {n,m}: Matches between n and m times.

Practice this: Quantifiers exercises in the Interactive Tutorial

Real-World Examples

Let’s look at how we can combine these rules to solve practical problems.

Example A: Password Validation

Suppose we need to validate a password that must be at least 8 characters long and contain only letters and digits.

The Pattern: ^[a-zA-Z0-9]{8,}$

Breakdown:

• ^ : Start of the string.
• [a-zA-Z0-9] : Allowed characters (any letter or number).
• {8,} : The previous character class must appear 8 or more times.
• $ : End of the string. (This ensures no special characters sneak in at the end).

Example B: Email Validation

Validating an email address perfectly according to the RFC standard is notoriously difficult, but a highly effective, standard RegEx looks like this:

The Pattern: ^[a-zA-Z0-9.-]+@[a-zA-Z0-9.-]+\.[a-zA-Z]{2,}$

Breakdown:

1. ^[a-zA-Z0-9.-]+ : Starts with one or more alphanumeric characters, dots, or dashes (the username).
2. @ : A literal “@” symbol.
3. [a-zA-Z0-9.-]+ : The domain name (e.g., “ucla” or “google”).
4. \. : A literal dot (escaped).
5. [a-zA-Z]{2,}$ : The top-level domain (e.g., “edu” or “com”), consisting of 2 or more letters, extending to the end of the string.

Grouping and Capturing

Often, you don’t just want to know if a string matched; you want to extract specific parts of the string. This is done using Groups, denoted by parentheses ().

Standard Capture Groups

If you want to extract the domain from an email, you can wrap that section in parentheses: ^.+@(.+\.[a-zA-Z]{2,})$ The engine will save whatever matched inside the () into a numbered variable that you can access in your programming language.

Named Capture Groups

When dealing with complex patterns, remembering group numbers gets confusing. Modern RegEx engines (like Python’s) support Named Capture Groups using the syntax (?P<name>pattern).

Example: Parsing HTML Hex Colors Imagine you want to extract the Red, Green, and Blue values from a hex color string like #FF00A1:

The Pattern: #(?P<R>[0-9a-fA-F]{2})(?P<G>[0-9a-fA-F]{2})(?P<B>[0-9a-fA-F]{2})

Here, we define three named groups (R, G, and B). When this runs against #FF00A1, our code can cleanly extract:

• Group “R”: FF
• Group “G”: 00
• Group “B”: A1

Seeing it in Action: Step-by-Step Worked Examples

Let’s put the theory of pattern pointers, bumping along, and backtracking into practice. Here is exactly how the RegEx engine steps through the three conceptual examples we discussed earlier.

Worked Example 1: The “Cat” Problem

The Goal: Find the distinct word “cat” or “cats” (case-insensitive), ignoring words where “cat” is just a substring. The Regex: \b[Cc][Aa][Tt][Ss]?\b (Note: \b is a “word boundary” anchor. It matches the invisible position between a word character and a non-word character, like a space or punctuation).

The Input String: "cats catalog cat"

Step-by-Step Execution:

1. Index 0 (c in “cats”):
   • The pattern pointer starts at \b. Since c is the start of a word (a transition from the start of the string to a word character), the \b assertion passes (zero characters consumed).
   • [Cc] matches c.
   • [Aa] matches a.
   • [Tt] matches t.
   • [Ss]? looks for an optional ‘s’. It finds s and matches it.
   • \b checks for a word boundary at the current position (between ‘s’ and the space). Because ‘s’ is a word character and the following space is a non-word character, the boundary assertion passes. Match successful!
   • Match 1 Saved: "cats"
2. Resuming at Index 4 (the space):
   • The engine resumes exactly where it left off to look for more matches.
   • \b matches the boundary. [Cc] fails against the space. The engine bumps along.
3. Index 5 (c in “catalog”):
   • \b matches. [Cc] matches c. [Aa] matches a. [Tt] matches t.
   • The string pointer is now positioned between the t and the a in “catalog”.
   • The pattern asks for [Ss]?. Is ‘a’ an ‘s’? No. Since the ‘s’ is optional (?), the engine says “That’s fine, I matched it 0 times,” and moves to the next pattern token.
   • The pattern asks for \b (a word boundary). The string pointer is currently between t (a word character) and a (another word character). Because there is no transition to a non-word character, the boundary assertion fails.
   • Match Fails! The engine drops everything, resets the pattern, and bumps along to the next letter.
4. Index 13 (c in “cat”):
   • The engine bumps along through “atalog “ until it hits the final word.
   • \b matches. [Cc] matches c. [Aa] matches a. [Tt] matches t.
   • [Ss]? looks for an ‘s’. The string is at the end. It matches 0 times.
   • \b looks for a boundary. The end of the string counts as a boundary. Match successful!
   • Match 2 Saved: "cat"

Worked Example 2: The Phone Number Problem

The Goal: Extract a uniquely formatted phone number from a string. The Regex: \(?\d{3}\)?[- .]?\d{3}[- .]?\d{4}

The Input String: "Call (123) 456-7890 now"

Step-by-Step Execution:

1. The engine starts at C. \( (an optional literal opening parenthesis) is not C, so it skips it. Next token \d{3} fails because C is not a digit. Bump along.
2. It bumps along through “Call “ until it reaches index 5: (.
3. Index 5 (():
   • \(? matches the (. (Consumed).
   • \d{3} matches 123. (Consumed).
   • \)? matches the ). (Consumed).
   • [- .]? looks for an optional space, dash, or dot. It finds the space after the parenthesis and matches it. (Consumed).
   • \d{3} matches 456. (Consumed).
   • [- .]? finds the - and matches it. (Consumed).
   • \d{4} matches 7890. (Consumed).
4. The pattern is fully satisfied.
   • Match Saved: "(123) 456-7890"

Worked Example 3: The Server Log (with Backtracking)

The Goal: Extract the IP address from a specific error line. The Regex: ^.*ERROR.*IP: (?P<IP>\d{1,3}(?:\.\d{1,3}){3}).*Critical Timeout$ (Note: We use .* to skip over irrelevant parts of the log).

The Input String: [14:32:05] ERROR - IP: 10.0.4.19 - Status: Critical Timeout

Step-by-Step Execution:

1. Start of String: ^ asserts we are at the beginning.
2. The .*: The pattern token .* tells the engine to match everything. The engine consumes the entire string all the way to the end: [14:32:05] ERROR - IP: 10.0.4.19 - Status: Critical Timeout.
3. Hitting a Wall: The next pattern token is the literal word ERROR. But the string pointer is at the absolute end of the line. The match fails.
4. Backtracking: The engine steps the string pointer backward one character at a time. It gives back t, then u, then o… all the way back until it gives back the space right before the word ERROR.
5. Moving Forward: Now that the .* has settled for matching [14:32:05] , the engine moves to the next token.
   • ERROR matches ERROR.
   • The next .* consumes the rest of the string again.
   • It has to backtrack again until it finds IP: .
6. The Capture Group: The engine enters the named capture group (?P<IP>...).
   • \d{1,3} matches 10.
   • (?:\.\d{1,3}){3} matches .0, then matches .4, then matches .19.
   • The engine saves the string "10.0.4.19" into a variable named “IP”.
7. The Final Stretch: * The final .* consumes the rest of the string again, backtracking until it can match the literal phrase Critical Timeout.
   • $ asserts the end of the string.
   • Match Saved! The group “IP” successfully holds "10.0.4.19".

Advanced

Advanced Pattern Control: Greediness vs. Laziness

Once you understand the basics of matching characters and using quantifiers, you will inevitably run into scenarios where your regular expression matches too much text. To solve this problem, we use Lazy Quantifiers.

By default, regular expression quantifiers (*, +, {n,m}) are greedy. This means they will consume as many characters as mathematically possible while still allowing the overall pattern to match.

The Greedy Problem: Imagine you are trying to extract the text from inside an HTML tag: <div>Hello World</div>. You might write the pattern: <.*>

Because .* is greedy, the engine sees the first < and then the .* swallows the entire rest of the string. It then backtracks just enough to find the final > at the very end of the string. Instead of matching just <div>, your greedy regex matched the entire string: <div>Hello World</div>.

The Lazy Solution (Non-Greedy): To make a quantifier lazy (meaning it will match as few characters as possible), you simply append a question mark ? immediately after the quantifier.

• *? : Matches 0 or more times, but as few times as possible.
• +? : Matches 1 or more times, but as few times as possible.

If we change our pattern to <div>(.*?)</div>, the engine matches the tags and captures only the text inside. Running this against <div>Hello World</div> will successfully yield a match where the first capture group is exactly “Hello World”.

Advanced Pattern Control: Lookarounds

Sometimes you need to assert that a specific pattern exists (or doesn’t exist) immediately before or after your current position, but you don’t want to include those characters in your final match result. To solve this problem, we use Lookarounds.

Lookarounds are “zero-width assertions.” Like anchors (^ and $), they check a condition at a specific position, but they do not “consume” any characters. The engine’s pointer stays exactly where it is.

Positive and Negative Lookaheads

Lookaheads look forward in the string from the current position.

• Positive Lookahead (?=...): Asserts that what immediately follows matches the pattern.
• Negative Lookahead (?!...): Asserts that what immediately follows does not match the pattern.

Example: The Password Condition Lookaheads are the secret to writing complex password validators. Suppose a password must contain at least one number. You can use a positive lookahead at the very start of the string: ^(?=.*\d)[A-Za-z\d]{8,}$

• ^ asserts the position at the beginning of the string.
• (?=.*\d) looks ahead through the string from the current position. If it finds a digit, the condition passes. Crucially, because lookaheads are zero-width, they do not consume characters. After the check passes, the engine’s string pointer resets back to the exact position where the lookahead started (which, in this specific case, is still the beginning of the string).
• [A-Za-z\d]{8,}$ then evaluates the string normally from that starting position to ensure it consists of 8+ valid characters.

Positive and Negative Lookbehinds

Lookbehinds look backward in the string from the current position.

• Positive Lookbehind (?<=...): Asserts that what immediately precedes matches the pattern.
• Negative Lookbehind (?<!...): Asserts that what immediately precedes does not match the pattern.

Example: Extracting Prices Suppose you have the text: I paid $100 for the shoes and €80 for the jacket. You want to extract the number 100, but only if it is a price in dollars (preceded by a $).

If you use \$\d+, your match will be $100. But you only want the number itself! By using a positive lookbehind, you can check for the dollar sign without consuming it: (?<=\$)\d+

• The engine reaches a position in the string.
• It peeks backward to see if there is a $.
• If true, it then attempts to match the \d+ portion. The match is exactly 100.

By mastering lazy quantifiers and lookarounds, you transition from simply searching for text to writing highly precise, surgical data-extraction algorithms!

How the RegEx Engine Finds All Matches: Under the Hood

To truly master Regular Expressions, it helps to understand exactly what the computer is doing behind the scenes. When you run a regex against a string, you are handing your pattern over to a RegEx Engine—a specialized piece of software (typically built using a theoretical concept called a Finite State Machine) that parses your text.

Here is the step-by-step breakdown of how the engine evaluates an input string to find every possible match.

The Two “Pointers”

Imagine the engine has two pointers (or fingers) tracing the text:

• The Pattern Pointer: Points to the current character/token in your RegEx pattern.
• The String Pointer: Points to the current character in your input text.

The engine always starts with both pointers at the very beginning (index 0) of their respective strings. It processes the text strictly from left to right.

Attempting a Match and “Consuming” Characters

The engine looks at the first token in your pattern and checks if it matches the character at the string pointer.

• If it matches, the engine consumes that character. Both pointers move one step to the right.
• If a quantifier like + or * is used, the engine will act greedily by default. It will consume as many matching characters as possible before moving to the next token in the pattern.

Hitting a Wall: Backtracking

What happens if the engine makes a choice (like matching a greedy .*), moves forward, and suddenly realizes the rest of the pattern doesn’t match? It doesn’t just give up.

Instead, the engine performs Backtracking. It remembers previous decision points—places where it could have made a different choice (like matching one fewer character). It physically moves the string pointer backwards step-by-step, trying alternative paths until it either finds a successful match for the entire pattern or exhausts all possibilities.

The “Bump-Along” (Failing and Retrying)

If the engine exhausts all possibilities at the current starting position and completely fails to find a match, it performs a “bump-along.”

It resets the pattern pointer to the beginning of your RegEx, advances the string pointer one character forward from where the last attempt began, and starts the entire process over again. It will continue this process, checking every single starting index of the string, until it finds a match or reaches the end of the text.

Finding All Matches (Global Search)

Usually, a RegEx engine stops the moment it finds the first valid match. However, if you instruct the engine to find all matches (usually done by appending a global modifier, like /g in JavaScript or using re.findall() in Python), the engine performs a specific sequence:

1. It finds the first successful match.
2. It saves that match to return to you.
3. It resumes the search starting at the exact character index where the previous match ended.
4. It repeats the evaluate-bump-match cycle until the string pointer reaches the absolute end of the input string.

An Example in Action: Let’s say you are searching for the pattern cat in the string "The cat and the catalog".

1. The engine starts at T. T is not c. It bumps along.
2. It eventually bumps along to the c in "cat". c matches c, a matches a, t matches t. Match #1 found!
3. The engine saves "cat" and moves its string pointer to the space immediately following it.
4. It continues bumping along until it hits the c in "catalog".
5. It matches c, a, and t. Match #2 found!
6. It resumes at the a in "catalog", bumps along to the end of the string, finds nothing else, and completes the search.

By mechanically stepping forward, backtracking when stuck, and resuming immediately after success, the engine guarantees no potential match is left behind!

Limitations of RegEx: The HTML Problem

As powerful as RegEx is, it has mathematical limitations. Under the hood, standard regular expressions are powered by Finite Automata (state machines).

Because Finite Automata have no “memory” to keep track of deeply nested structures, you cannot write a general regular expression to perfectly parse HTML or XML.

HTML allows for infinitely nested tags (e.g., <div><div><span></span></div></div>). A regular expression cannot inherently count opening and closing brackets to ensure they are perfectly balanced. Attempting to use RegEx to parse raw HTML often results in brittle code full of false positives and false negatives. For tree-like structures, you should always use a dedicated parser (like BeautifulSoup in Python or the DOM parser in JavaScript) instead of RegEx.

Conclusion

Regular Expressions might look intimidating, but they are incredibly logical once you break them down into their component parts. By mastering anchors, character classes, quantifiers, and groups, you can drastically reduce the amount of code you write for data validation and text manipulation. Start small, practice in online tools like Regex101, and slowly incorporate them into your daily software development workflow!

Quiz

Basic RegEx Syntax Flashcards (Production/Recall)

Test your ability to produce the exact Regular Expression metacharacter or syntax based on its functional description.

What metacharacter asserts the start of a string?

^ (Caret)

Placed at the very beginning of a pattern, it constrains the match to the start of the string.

What metacharacter asserts the end of a string?

$ (Dollar sign)

Placed at the very end of a pattern, it constrains the match to the end of the string.

What syntax is used to define a Character Class (matching any single character from a specified group)?

[...] (Square brackets)

Wrapping characters in square brackets tells the engine to match exactly one of the characters found inside.

What syntax is used inside a character class to act as a negation operator (matching any character NOT in the group)?

[^...] (Caret immediately after the opening bracket)

If the caret is the very first character inside the brackets, it inverts the character class.

What meta character is used to match any single digit?

\d

This is the direct equivalent of the character class [0-9].

What meta character is used to match any ‘word’ character (alphanumeric plus underscore)?

\w

This is the direct equivalent of the character class [a-zA-Z0-9_].

What meta character is used to match any whitespace character (spaces, tabs, line breaks)?

\s

This quickly targets formatting and spacing characters.

What metacharacter acts as a wildcard, matching any single character except a newline?

. (Dot)

To match a literal dot, you would need to escape it (.).

What quantifier specifies that the preceding element should match ‘0 or more’ times?

* (Asterisk)

This allows the preceding character or group to be completely absent or repeat infinitely.

What quantifier specifies that the preceding element should match ‘1 or more’ times?

+ (Plus sign)

This guarantees the character or group appears at least once, but allows it to repeat infinitely.

What quantifier specifies that the preceding element should match ‘0 or 1’ time?

? (Question mark)

This essentially makes the preceding character, class, or group optional.

What syntax is used to specify that the preceding element must repeat exactly n times?

{n} (Curly braces with a number)

You can also specify a range of repetitions using {n,m}.

What syntax is used to create a standard capture group?

(...) (Parentheses)

This groups tokens together and saves the matched substring into memory for later extraction.

What is the syntax used to create a Named Capture Group?

(?P<name>pattern)

This captures the matched substring and assigns it a specific variable name instead of just a numbered index.

Session Complete!

Your Score: 0/14

Come back later to improve your recall!

RegEx Example Flashcards

Test your knowledge on solving common text-processing problems using Regular Expressions!

Write a regex to validate a standard email address (e.g., user@domain.com).

^[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\.[a-zA-Z]{2,}$

This matches a sequence of alphanumeric characters and allowed symbols for the username, followed by an ‘@’, a domain name, and a 2+ character top-level domain.

Write a regex to match a standard US phone number, with optional parentheses and various separators (e.g., 123-456-7890 or (123) 456-7890).

^(\(\d{3}\)|\d{3})[- .]?\d{3}[- .]?\d{4}$

The first group matches either 3 digits in parentheses or just 3 digits. The [- .]? allows for an optional dash, space, or dot between the number segments.

Write a regex to match a 3 or 6 digit hex color code starting with a hashtag (e.g., #FFF or #1A2B3C).

^#([a-fA-F0-9]{6}|[a-fA-F0-9]{3})$

Matches the literal ‘#’ followed by either exactly 6 hexadecimal characters or exactly 3 hexadecimal characters, ensuring no extra characters exist using the $ anchor.

Write a regex to validate a strong password (at least 8 characters, containing at least one uppercase letter, one lowercase letter, and one number).

^(?=.*[a-z])(?=.*[A-Z])(?=.*\d)[a-zA-Z\d]{8,}$

Uses positive lookaheads (?=...) to scan ahead and guarantee the presence of a lowercase, uppercase, and digit before matching a string of 8 or more valid characters.

Write a regex to match a valid IPv4 address (e.g., 192.168.1.1).

^(?:(?:25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.){3}(?:25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)$

Ensures that each of the 4 octets falls between 0 and 255, separating the first three with literal dots.

Write a regex to extract the domain name from a URL, ignoring the protocol and ‘www’ (e.g., extracting ‘example.com’ from ‘https://www.example.com/page’).

^(?:https?:\/\/)?(?:www\.)?([^\/]+)

The non-capturing groups (?:...) make the ‘http(s)://’ and ‘www.’ optional. It then captures all characters up to the first forward slash.

Write a regex to match a date in the format YYYY-MM-DD (checking for valid month and day ranges).

^\d{4}-(0[1-9]|1[0-2])-(0[1-9]|[12]\d|3[01])$

Matches exactly 4 digits for the year, restricts the month to 01-12, and restricts the day to 01-31.

Write a regex to match a time in 24-hour format (HH:MM).

^([01]\d|2[0-3]):([0-5]\d)$

The hour group allows either 00-19 or 20-23. The minute group strictly allows 00-59.

Write a regex to match an opening or closing HTML tag.

<\/?[a-z][\s\S]*>

Matches the opening <, an optional / for closing tags, a starting letter, and then any characters (including newlines) until the closing >.

Write a regex to find all leading and trailing whitespaces in a string (commonly used for string trimming).

^\s+|\s+$

Matches one or more whitespace characters \s+ at the start ^ of the string, OR | one or more whitespace characters at the end $ of the string.

Session Complete!

Your Score: 0/10

Come back later to improve your recall!

RegEx Quiz

Test your understanding of regular expressions beyond basic syntax, focusing on underlying mechanics, performance, and theory.

You are tasked with extracting all data enclosed in HTML <div> tags. You write a regular expression, but it consistently fails on deeply nested divs (e.g., <div><div>text</div></div>). From a theoretical computer science perspective, why is standard RegEx the wrong tool for this?

Correct Answer:

Explanation

Standard regular expressions correlate to regular languages in formal language theory. They are evaluated by Finite State Machines, which do not have a ‘stack’ or memory mechanism to keep track of balanced, nested pairs (unlike context-free grammars used by HTML parsers).

A developer writes a regex to parse a log file: ^.*error.*$. They notice that while it works, it runs much slower than expected on very long log lines. What underlying behavior of the .* token is causing this inefficiency?

Correct Answer:

Explanation

Greedy quantifiers (like * and +) match as much as possible right away. .* consumes the whole string, realizes it missed ‘error’, and has to backtrack (step backwards) one character at a time to find a match, which is computationally expensive on long strings.

You need to validate user input to ensure a password contains both a number and a special character, but you don’t know what order they will appear in. What mechanism allows a RegEx engine to assert these conditions without actually ‘consuming’ the string character by character?

Correct Answer:

Explanation

Lookaheads and lookbehinds are ‘zero-width assertions’. They look ahead in the string to verify a pattern exists, but they do not consume any characters. This allows you to chain multiple lookaheads at the start of a string to act like a logical ‘AND’ statement.

A junior engineer writes a regex to validate an email address and deploys it. Later, the system crashes because the regex engine hangs infinitely when evaluating the malicious input: aaaaaaaaaaaaaaaaaaaaaaaaa!. What vulnerability did the engineer likely introduce?

Correct Answer:

Explanation

Catastrophic backtracking (often leading to a ReDoS - Regular Expression Denial of Service) occurs when a regex has multiple ways to match the same sequence (often via nested quantifiers). When a match fails at the very end (like the !), the engine attempts every possible exponential combination of the preceding characters before giving up.

When writing a complex regex to extract phone numbers, you group the area code using parentheses (...) so you can apply a ? quantifier to it. However, you don’t actually need to save the area code data in memory for later. What is the most optimized approach?

Correct Answer:

Explanation

Standard parentheses (...) create ‘capturing groups’, meaning the regex engine allocates memory to store the matched substring for later extraction. If you only need parentheses to group tokens for a quantifier, a non-capturing group (?:...) prevents this unnecessary memory overhead.

You write a regex to ensure a username is strictly alphanumeric: [a-zA-Z0-9]+. However, a user successfully submits the username admin!@#. Why did this happen?

Correct Answer:

Explanation

Without positional anchors (^ for the start, $ for the end), a regex will search for any substring that satisfies the pattern. It found ‘admin’, considered it a successful match, and ignored the trailing invalid characters. Anchors are mandatory for strict input validation.

Which of the following scenarios are highly appropriate use cases for Regular Expressions? (Select all that apply)

Correct Answers:

Explanation

RegEx excels at pattern matching flat strings, validating strict formats, and doing structural find-and-replace operations. It should NOT be used to parse complex, tree-like structures like JSON or HTML, as it cannot handle infinite nesting and is prone to security bypasses.

In the context of evaluating a regex for data extraction, what represents a ‘False Positive’ and a ‘False Negative’? (Select all that apply)

Correct Answers:

Explanation

A False Positive is an unwanted match (the regex was too loose). A False Negative is a missed match (the regex was too strict and missed valid data). These are the two primary edge cases engineers must balance when writing patterns.

Which of the following strategies are effective ways to prevent Catastrophic Backtracking (ReDoS)? (Select all that apply)

Correct Answers:

Explanation

Catastrophic backtracking is solved by removing ambiguity in the pattern. You can do this by refactoring nested quantifiers, or by using advanced engine features like Atomic Groups and Possessive Quantifiers, which explicitly forbid the engine from backtracking over a match.

Which of the following statements about Lookaheads (?=...) are true? (Select all that apply)

Correct Answers:

Explanation

Lookaheads do not consume characters, can be chained to check multiple conditions from the exact same starting position, and are the standard way to enforce logical ‘AND’ in regex. However, they are a feature of PCRE (Perl Compatible Regular Expressions) and are generally NOT supported by basic POSIX tools like standard sed.

Quiz Complete!

Your Score: 0/10

RegEx Tutorial: Basics

---

0 / 16 exercises completed

This hands-on tutorial will walk you through Regular Expressions step by step. Each section builds on the last. Complete exercises to unlock your progress. Don’t worry about memorizing everything — focus on understanding the patterns.

Regular expressions look intimidating at first — that’s completely normal. Even experienced developers regularly look up regex syntax. The key is to break patterns into small, logical pieces. By the end of this tutorial, you’ll be able to read and write patterns that would have looked like gibberish an hour ago. If you get stuck, that means you’re learning — every programmer has been exactly where you are.

Three exercise types appear throughout:

• Build it (Parsons): drag and drop regex fragments into the correct order.
• Write it (Free): type a regex from scratch.
• Fix it (Fixer Upper): a broken regex is given — debug and repair it.

Your progress is saved in your browser automatically.

Literal Matching

The simplest regex is just the text you want to find. The pattern cat matches the exact characters c, a, t — in that order, wherever they appear. This means it matches inside words too: cat appears in “education” and “scatter”.

Key points:

• RegEx is case-sensitive by default: cat does not match “Cat” or “CAT”.
• The engine scans left-to-right, reporting every non-overlapping match.

Character Classes

A character class [...] matches any single character listed inside the brackets. For example, [aeiou] matches any one lowercase vowel.

You can also use ranges: [a-z] matches any lowercase letter, [0-9] matches any digit, and [A-Za-z] matches any letter regardless of case.

To negate a class, place ^ right after the opening bracket: [^a-z] matches any character that is not a lowercase letter — digits, punctuation, spaces, etc.

Meta Characters

Writing out full character classes every time gets tedious. RegEx provides meta character escape sequences:

meta character	Meaning	Equivalent Class
\d	Any digit	[0-9]
\D	Any non-digit	[^0-9]
\w	Any “word” character	[a-zA-Z0-9_]
\W	Any non-word character	[^a-zA-Z0-9_]
\s	Any whitespace	[ \t\n\r\f]
\S	Any non-whitespace	[^ \t\n\r\f]

The dot . is a wildcard that matches any single character (except newline). Because the dot matches almost everything, it is powerful but easy to overuse. When you actually need to match a literal period, escape it: \.

Anchors

Before reading this section, try the first exercise below. Use what you already know to write a regex that matches only if the entire string is digits. You’ll discover a gap in your toolkit — that’s the point!

So far every pattern matches anywhere inside a string. Anchors constrain where a match can occur without consuming characters:

Anchor	Meaning
^	Start of string (or line in multiline mode)
$	End of string (or line in multiline mode)
\b	Word boundary — the point between a “word” character (\w) and a “non-word” character (\W), or vice versa

Anchors are critical for validation. Without them, the pattern \d+ would match the 42 inside "hello42world". Adding anchors — ^\d+$ — ensures the entire string must be digits.

Word boundaries (\b) let you match whole words. \bgo\b matches the standalone word “go” but not “goal” or “cargo”.

Quantifiers

Quantifiers control how many times the preceding element must appear:

Quantifier	Meaning
*	Zero or more times
+	One or more times
?	Zero or one time (optional)
{n}	Exactly n times
{n,}	n or more times
{n,m}	Between n and m times

Common misconception: * vs +

Students frequently confuse these two. The key difference:

• a*b matches b, ab, aab, aaab, … — the a is optional (zero or more).
• a+b matches ab, aab, aaab, … — at least one a is required.

If you want “one or more”, reach for +. If you genuinely mean “zero or more”, use *. Getting this wrong is one of the most common sources of regex bugs.

Alternation & Combining

The pipe | works like a logical OR: cat|dog matches either “cat” or “dog”. Alternation has low precedence, so gray|grey matches the full words — you don’t need parentheses for simple cases.

When you combine multiple regex features, patterns become expressive:

• gr[ae]y — character class for the spelling variant.
• \d{2}:\d{2} — two digits, a colon, two digits (time format).
• ^(0[1-9]|1[0-2])/(0[1-9]|[12]\d|3[01])$ — a full date validator.

Start simple and add complexity only when tests demand it.

---

You’ve completed the basics! You now know how to match literal text, use character classes, metacharacters, anchors, quantifiers, and alternation.

Ready for more? Continue to the Advanced RegEx Tutorial to learn greedy vs. lazy matching, groups, lookaheads, and tackle integration challenges.

RegEx Tutorial: Advanced

---

0 / 13 exercises completed

This is the second part of the Interactive RegEx Tutorial. If you haven’t completed the Basics Tutorial yet, start there first — the exercises here assume you’re comfortable with literal matching, character classes, metacharacters, anchors, quantifiers, and alternation.

Warm-Up Review

Before diving into advanced features, let’s make sure the basics are solid. These exercises combine concepts from the Basics tutorial. If any feel rusty, revisit the Basics.

Greedy vs. Lazy

By default, quantifiers are greedy — they match as much text as possible. This often surprises beginners.

Consider matching HTML tags with <.*> against the string <b>bold</b>:

• Greedy <.*> matches <b>bold</b> — the entire string! The .* gobbles everything up, then backtracks just enough to find the last >.
• Lazy <.*?> matches <b> and then </b> separately. Adding ? after the quantifier makes it match as little as possible.

The lazy versions: *?, +?, ??, {n,m}?

Use the step-through visualizer in the first exercise below to see exactly how the engine behaves differently in each mode.

Groups & Capturing

Parentheses (...) serve two purposes:

1. Grouping: Treat multiple characters as a single unit for quantifiers. (na){2,} means “the sequence na repeated 2 or more times” — matching nana, nanana, etc.

2. Capturing: The engine remembers what each group matched, which is useful in search-and-replace operations (backreferences like \1 or $1).

If you only need grouping without capturing, use a non-capturing group: (?:...)

Lookaheads & Lookbehinds

Lookaround assertions check what comes before or after the current position without including it in the match. They are “zero-width” — they don’t consume characters.

Syntax	Name	Meaning
(?=...)	Positive lookahead	What follows must match ...
(?!...)	Negative lookahead	What follows must NOT match ...
(?<=...)	Positive lookbehind	What precedes must match ...
(?<!...)	Negative lookbehind	What precedes must NOT match ...

A classic use case: password validation. To require at least one digit AND one uppercase letter, you can chain lookaheads at the start: ^(?=.*\d)(?=.*[A-Z]).+$. Each lookahead checks a condition independently, and the .+ at the end actually consumes the string.

Lookbehinds are useful for extracting values after a known prefix — like capturing dollar amounts after a $ sign without including the $ itself.

Putting It All Together

You’ve learned every major regex feature. The real skill is knowing which tools to combine for a given problem. These exercises don’t tell you which section to draw from — you’ll need to decide which combination of character classes, anchors, quantifiers, groups, and lookarounds to use.

This is where regex goes from “I can follow along” to “I can solve problems on my own.”

Python

---

Welcome to Python! Since you already know C++, you have a strong foundation in programming logic, control flow, and object-oriented design. However, moving from a compiled, statically typed systems language to an interpreted, dynamically typed scripting language requires a shift in how you think about memory and execution.

To help you make this transition, we will anchor Python’s concepts directly against the C++ concepts you already know, adjusting your mental model along the way.

The Execution Model: Scripts vs. Binaries

In C++, your workflow is Write $\rightarrow$ Compile $\rightarrow$ Link $\rightarrow$ Execute. The compiler translates your source code directly into machine-specific instructions.

Python is a scripting language. You do not explicitly compile and link a binary. Instead, your workflow is simply Write $\rightarrow$ Execute.

Under the hood, when you run python script.py, the Python interpreter reads your code, translates it into an intermediate “bytecode,” and immediately runs that bytecode on the Python Virtual Machine (PVM).

What this means for you:

• No main() boilerplate: Python executes from top to bottom. You don’t need a main() function to make a script run, though it is often used for organization.
• Rapid Prototyping: Because there is no compilation step, you can write and test code iteratively and quickly.
• Runtime Errors: In C++, the compiler catches syntax and type errors before the program ever runs. In Python, errors are caught at runtime when the interpreter actually reaches the problematic line.

C++:

#include <iostream>
int main() {
    std::cout << "Hello, World!" << std::endl;
    return 0;
}

Python:

print("Hello, World!")

The Mental Model of Memory: Dynamic Typing

This is the largest paradigm shift you will make.

In C++ (Statically Typed), a variable is a box in memory. When you declare int x = 5;, the compiler reserves 4 bytes of memory, labels that specific memory address x, and restricts it to only hold integers.

In Python (Dynamically Typed), a variable is a name tag attached to an object. The object has a type, but the variable name does not.

Let’s look at an example:

x = 5         # Python creates an integer object '5'. It attaches the name tag 'x' to it.
print(x)      

x = "Hello"   # Python creates a string object '"Hello"'. It moves the 'x' tag to the string.
print(x)      # The integer '5' is now nameless and will be garbage collected.

Because variables are just name tags (references) pointing to objects, you don’t declare types. The Python interpreter figures out the type of the object at runtime.

Syntax and Scoping: Whitespace Matters

In C++, scope is defined by curly braces {} and statements are terminated by semicolons ;.

Python uses indentation to define scope, and newlines to terminate statements. This enforces highly readable code by design.

C++:

for (int i = 0; i < 5; i++) {
    if (i % 2 == 0) {
        std::cout << i << " is even\n";
    }
}

Python:

for i in range(5):
    if i % 2 == 0:
        print(f"{i} is even") # Notice the 'f' string, Python's modern way to format strings

Note: range(5) generates a sequence of numbers from 0 up to (but not including) 5.

Passing Arguments: “Pass-by-Object-Reference”

In C++, you explicitly choose whether to pass variables by value (int x), by reference (int& x), or by pointer (int* x).

How does Python handle this? Because everything in Python is an object, and variables are just “name tags” pointing to those objects, Python uses a model often called “Pass-by-Object-Reference”.

When you pass a variable to a function, you are passing the name tag.

• If the object the tag points to is Mutable (like a List or a Dictionary), changes made inside the function will affect the original object.
• If the object the tag points to is Immutable (like an Integer, String, or Tuple), any attempt to change it inside the function simply creates a new object and moves the local name tag to it, leaving the original object unharmed.

# Modifying a Mutable object (similar to passing by reference/pointer in C++)
def modify_list(my_list):
    my_list.append(4) # Modifies the actual object in memory

nums = [1, 2, 3]
modify_list(nums)
print(nums) # Output: [1, 2, 3, 4]

# Modifying an Immutable object (behaves similarly to pass by value)
def attempt_to_modify_int(my_int):
    my_int += 10 # Creates a NEW integer object, moves the local 'my_int' tag to it

val = 5
attempt_to_modify_int(val)
print(val) # Output: 5. The original object is unchanged.

Here is the continuation of your introduction to Python, building on the mental models we established in the first part.

Here is the next part of your Python introduction, covering string formatting, core data structures, and the powerful slicing syntax.

String Formatting: The Magic of f-strings

In C++, building a complex string with variables traditionally requires chaining << operators with std::cout, using sprintf, or utilizing the modern std::format. This can get verbose quickly.

Python revolutionized string formatting in version 3.6 with the introduction of f-strings (formatted string literals). By simply prefixing a string with the letter f (or F), you can embed variables and even evaluate expressions directly inside curly braces {}.

C++:

std::string name = "Alice";
int age = 30;
std::cout << name << " is " << age << " years old and will be " 
          << (age + 1) << " next year.\n";

Python:

name = "Alice"
age = 30

# The f-string automatically converts variables to strings and evaluates the math
print(f"{name} is {age} years old and will be {age + 1} next year.")

Pedagogical Note: Under the hood, Python calls the __str__() method of the objects placed inside the curly braces to get their string representation.

Core Collections: Lists, Sets, and Dictionaries

Because Python does not enforce static typing, its built-in collections are highly flexible. You do not need to #include external libraries to use them; they are native to the language syntax.

Lists (C++ Equivalent: std::vector)

A List is an ordered, mutable sequence of elements. Unlike a C++ std::vector<T>, a Python list can contain objects of entirely different types. Lists are defined using square brackets [].

# Heterogeneous list
my_list = [1, "two", 3.14, True]

my_list.append("new item") # Adds to the end (like push_back)
my_list.pop()              # Removes and returns the last item
print(len(my_list))        # len() gets the size of any collection

Sets (C++ Equivalent: std::unordered_set)

A Set is an unordered collection of unique elements. It is implemented using a hash table, making membership testing (in) exceptionally fast—$O(1)$ on average. Sets are defined using curly braces {}.

unique_numbers = {1, 2, 2, 3, 4, 4}
print(unique_numbers) # Output: {1, 2, 3, 4} - duplicates are automatically removed

# Fast membership testing
if 3 in unique_numbers:
    print("3 is present!")

Dictionaries (C++ Equivalent: std::unordered_map)

A Dictionary (or “dict”) is a mutable collection of key-value pairs. Like Sets, they are backed by hash tables for incredibly fast $O(1)$ lookups. Dicts are defined using curly braces {} with a colon : separating keys and values.

player_scores = {"Alice": 50, "Bob": 75}

# Accessing and modifying values
player_scores["Alice"] += 10 
player_scores["Charlie"] = 90 # Adding a new key-value pair

print(f"Bob's score is {player_scores['Bob']}")

Memory Management: RAII vs. Garbage Collection

In C++, you are the absolute master of memory. You allocate it (new), you free it (delete), or you utilize RAII (Resource Acquisition Is Initialization) and smart pointers to tie memory management to variable scope. If you make a mistake, you get a memory leak or a segmentation fault.

In Python, memory management is entirely abstracted away. You do not allocate or free memory. Instead, Python primarily uses Reference Counting backed by a Garbage Collector.

Every object in Python keeps a running tally of how many “name tags” (variables or references) are pointing to it. When a variable goes out of scope, or is reassigned to a different object, the reference count of the original object decreases by one. When that count hits zero, Python immediately reclaims the memory.

C++ (Manual / RAII):

void createArray() {
    // Dynamically allocated, must be managed
    int* arr = new int[100]; 
    // ... do something ...
    delete[] arr; // Forget this and you leak memory!
}

Python (Automatic):

def create_list():
    # Creates a list object in memory and attaches the 'arr' tag
    arr = [0] * 100 
    # ... do something ...
    
    # When the function ends, 'arr' goes out of scope. 
    # The list object's reference count drops to 0, and memory is freed automatically.

Data Structures and “Pythonic” Iteration

In C++, you rely heavily on the Standard Template Library (STL) for data structures like std::vector and std::unordered_map. Because C++ is statically typed, a std::vector<int> can only hold integers.

Because Python is dynamically typed (variables are just tags), its built-in data structures are incredibly flexible. A single Python List can hold an integer, a string, and another list simultaneously.

Additionally, while C++ traditionally relies on index-based for loops (though modern C++ has range-based loops), Python strongly encourages iterating directly over the elements of a collection. This is considered writing “Pythonic” code.

C++ (Index-based iteration):

std::vector<std::string> fruits = {"apple", "banana", "cherry"};
for (size_t i = 0; i < fruits.size(); i++) {
    std::cout << fruits[i] << std::endl;
}

Python (Pythonic Iteration):

fruits = ["apple", "banana", "cherry"]

# Do not do: for i in range(len(fruits)): ...
# Instead, iterate directly over the object:
for fruit in fruits:
    print(fruit)

# Python's equivalent to std::unordered_map is a Dictionary
student_grades = {"Alice": 95, "Bob": 82}

for name, grade in student_grades.items():
    print(f"{name} scored {grade}")

Object-Oriented Programming: Explicit self and “Duck Typing”

If you are used to C++ classes, Python’s approach to OOP will feel radically open and simplified.

1. No Header Files: Everything is declared and defined in one place.
2. Explicit self: In C++, instance methods have an implicit this pointer. In Python, the instance reference is passed explicitly as the first parameter to every instance method. By convention, it is always named self.
3. No True Privacy: C++ enforces public, private, and protected access specifiers at compile time. Python operates on the philosophy of “we are all consenting adults here.” There are no true private variables. Instead, developers use a convention: prefixing a variable with a single underscore (e.g., _internal_state) signals to other developers, “This is meant for internal use, please don’t touch it,” but the language will not stop them from accessing it.
4. Duck Typing: In C++, if a function expects a Bird object, you must pass an object that inherits from Bird. Python relies on “Duck Typing”—If it walks like a duck and quacks like a duck, it must be a duck. Python doesn’t care about the object’s actual class hierarchy; it only cares if the object implements the methods being called on it.

C++:

class Rectangle {
private:
    int width, height; // Enforced privacy
public:
    Rectangle(int w, int h) : width(w), height(h) {} // Constructor
    
    int getArea() {
        return width * height; // 'this->' is implicit
    }
};

Python:

class Rectangle:
    # __init__ is Python's constructor. 
    # Notice 'self' must be explicitly declared in the parameters.
    def __init__(self, width, height):
        self._width = width   # The underscore is a convention meaning "private"
        self._height = height # but it is not strictly enforced by the interpreter.

    def get_area(self):
        # You must explicitly use 'self' to access instance variables
        return self._width * self._height

# Instantiating the object (Note: no 'new' keyword in Python)
my_rect = Rectangle(10, 5)
print(my_rect.get_area())

Dunder Methods: __str__ vs. operator<<

In the OOP section, we covered the __init__ constructor method. Python uses several of these “dunder” (double underscore) methods to implement core language behavior.

In C++, if you want to print an object using std::cout, you have to overload the << operator. In Python, you simply implement the __str__(self) method. This method returns a “user-friendly” string representation of the object, which is automatically called whenever you use print() or an f-string.

Python:

class Book:
    def __init__(self, title, author, year):
        self.title = title
        self.author = author
        self.year = year
        
    def __str__(self):
        # This is what print() will call
        return f'"{self.title}" by {self.author} ({self.year})'

my_book = Book("Pride and Prejudice", "Jane Austen", 1813)
print(my_book) # Output: "Pride and Prejudice" by Jane Austen (1813)

Substring Operations and Slicing

In C++, if you want a substring, you call my_string.substr(start_index, length). Python takes a much more elegant and generalized approach called Slicing.

Slicing works not just on strings, but on any ordered sequence (like Lists and Tuples). The syntax uses square brackets with colons: sequence[start:stop:step].

• start: The index where the slice begins (inclusive).
• stop: The index where the slice ends (exclusive).
• step: The stride between elements (optional, defaults to 1).

Negative Indexing: This is a crucial Python paradigm. While index 0 is the first element, index -1 is the last element, -2 is the second-to-last, and so on.

text = "Software Engineering"

# Basic slicing
print(text[0:8])    # Output: 'Software' (Indices 0 through 7)

# Omitting start or stop
print(text[:8])     # Output: 'Software' (Defaults to the very beginning)
print(text[9:])     # Output: 'Engineering' (Defaults to the very end)

# Negative indexing
print(text[-11:])   # Output: 'Engineering' (Starts 11 characters from the end)
print(text[-1])     # Output: 'g' (The last character)

# Using the step parameter
print(text[0:8:2])  # Output: 'Sfwr' (Every 2nd character of 'Software')

# The ultimate Pythonic trick: Reversing a sequence
print(text[::-1])   # Output: 'gnireenignE erawtfoS' (Steps backwards by 1)

Because variables in Python are references to objects, it is important to note that slicing a list or a string creates a shallow copy—a brand new object in memory containing the sliced elements.

Tuple Unpacking and Variable Swapping

The lecture introduces the concept of Syntactic Sugar—language features that don’t add new functional capabilities but make programming significantly easier and more readable.

A prime example is unpacking. In C++, swapping two variables requires a temporary third variable (or utilizing std::swap). Python handles this natively with multiple assignment.

C++:

int temp = a;
a = b;
b = temp;

Python:

a, b = b, a # Syntactic sugar that swaps the values instantly

Exception Handling: try / except

While we discussed that Python catches errors at runtime, the Week 2 materials highlight how to handle these errors gracefully using try and except blocks (Python’s equivalent to C++’s try and catch).

In C++, exceptions are often reserved for critical failures, but in Python, using exceptions for control flow (like catching a ValueError when a user inputs a string instead of an integer) is standard practice.

try:
    guess = int(input("> "))
except ValueError:
    print("Invalid input, please enter a number.")

Robust Command-Line Arguments (argparse)

In C++, you typically handle command-line inputs by parsing int argc and char* argv[] directly in main(). While Python does have a direct equivalent (sys.argv), the course materials emphasize using the built-in argparse module. It automatically generates help/usage messages, enforces types, and parses flags, saving you from writing boilerplate C++ parsing code.

Regular Expressions (re module)

Since Python is a scripting language, it is heavily utilized for text processing. The lecture transitions into using Python to read and parse text files (like User Stories) using the re module. While C++ has the <regex> library, Python’s integration with RegEx and string manipulation is much more central to its everyday use case as a scripting tool.

Git

---

Want to practice? Try the Interactive Git Tutorial — hands-on exercises in a real Linux system right in the browser!

In modern software construction, version control is not just a convenience—it is a foundational practice, solving several major challenges associated with managing code. Git is by far the most common tool for version control. Let’s dive into both!

Basics

What is Version Control?

Version control (also known as source control or revision control) is the software engineering practice of controlling, organizing, and tracking different versions in the history of computer files. While it works best with text-based source code, it can theoretically track any file type.

We call a tool that supports version control a Version Control System (VCS). The most common version control systems are:

• Git (most common for open source systems, also used by Microsoft, Apple, and most other companies)
• Mercurial (used by Meta, formerly Facebook (Goode and Rain 2014), Jane Street, and some others)
• Piper (internal tool used by Google (Potvin and Levenberg 2016))
• Subversion (used by some older projects)

Why is it Essential?

Manual version control—saving files with names like Homework_final_v2_really_final.txt—is cumbersome and error-prone. Automated systems like Git solve several critical problems:

• Collaboration: Multiple developers can work concurrently on the same project without overwriting each other’s changes.
• Change Tracking: Developers can see exactly what has changed since they last worked on a file.
• Traceability: It provides a summary of every modification: who made it, when it happened, and why.
• Reversion/Rollback: If a bug is introduced, you can easily revert to a known stable version.
• Parallel Development: Branching allows for the isolated development of new features or bug fixes without affecting the main codebase.

Centralized vs. Distributed Version Control

There are two primary models of version control systems:

Feature	Centralized (e.g., Subversion, Piper)	Distributed (e.g., Git, Mercurial)
Data Storage	Data is stored in a single central repository.	Each developer has a full copy of the entire repository history.
Offline Work	Requires a connection to the central server to make changes.	Developers can work and commit changes locally while offline.
Best For	Small teams requiring strict centralized control.	Large teams, open-source projects, and distributed workflows.

The Git Architecture: The Three States

To understand Git, you must understand where your files live at any given time. Git operates across three main “states” or areas:

1. Working Directory (or Working Tree): This is where you currently edit your files. It contains the files as they exist on your disk.
2. Staging Area (or Index): This is a middle ground where you “stage” changes you want to include in your next snapshot.
3. Local Repository: This is where Git stores the compressed snapshots (commits) of your project’s history.

Fundamental Git Workflow

A typical Git workflow follows these steps:

1. Initialize: Turn a directory into a Git repo using git init.
2. Stage: Add file contents to the staging area with git add <filename>.
3. Commit: Record the snapshot of the staged changes with git commit -m "message".
4. Check Status: Use git status to see which files are modified, staged, or untracked.
5. Review History: Use git log to see the sequence of past commits.

Inspecting Differences

git diff is used to compare different versions of your code:

• git diff: Compares the working directory to the staging area.
• git diff HEAD: Compares the working directory to the latest commit.
• git diff HEAD^ HEAD: Compares the parent commit to the latest commit (shows what the latest commit changed).

Branching and Merging

A branch in Git is like a pointer to a commit (implemented as a lightweight, 41-byte text file stored in .git/refs/heads/ that contains the SHA checksum of the commit it currently points to). Creating or destroying a branch is nearly instantaneous — Git writes or deletes a tiny reference, not a copy of your project. The HEAD pointer (stored in .git/HEAD) normally holds a symbolic reference to the current branch, such as ref: refs/heads/main.

Integrating Changes

When you want to bring changes from a feature branch back into the main codebase, Git typically uses one of two automatic merge strategies:

• Fast-Forward Merge: When the target branch (main) has received no new commits since the feature branch was created, Git simply advances the main pointer to the tip of the feature branch. No merge commit is created; the history stays perfectly linear.
• Three-Way Merge: When both branches have diverged — each has commits the other doesn’t — Git compares both tips against their common ancestor and creates a new merge commit with two parents. The commit graph forms a diamond shape where the two diverging paths converge.

Alternative Integration Workflows

For more control over your project’s history, you can use these manual techniques:

• Rebasing: Re-applies commits from one branch onto a new base, producing new commit objects with new SHA hashes. Creates a linear history but must never be used on shared branches, as it rewrites history that collaborators may already have.
• Squashing: git merge --squash collapses all commits from a feature branch into a single commit on the target branch, keeping the main history tidy.

Complications

• Merge Conflict: Happens when Git cannot automatically reconcile differences — usually when the same lines of code were changed in both branches.
• Detached HEAD: Occurs when HEAD points directly to a commit hash rather than a branch reference — for example, when using git switch --detach <commit> to inspect an older version of the codebase. New commits made in this state are not anchored to any branch and can easily be lost when switching away. To preserve work from a detached HEAD, create a new branch with git switch -c <name> before switching elsewhere.

Advanced Power Tools

Git includes several advanced commands for debugging and project management:

• git stash: Temporarily saves local changes (staged and unstaged) so you can switch branches without committing messy or incomplete work.
• git cherry-pick: Selectively applies a specific commit from one branch onto another.
• git bisect: Uses a binary search through your commit history to find the exact commit that introduced a bug.
• git blame: Annotates each line of a file with the name of the author and the commit hash of the last person to modify it.
• git revert: Safely “undoes” a previous commit by creating a new commit with the inverse changes, preserving the original history.

Managing Large Projects: Submodules

For very large projects, Git Submodules allow you to keep one Git repository as a subdirectory of another. This is ideal for including external libraries or shared modules while maintaining their independent history. Internally, a submodule is represented as a file pointing to a specific commit ID in the external repo.

Best Practices for Professional Use

• Write Meaningful Commit Messages: Messages should explain what was changed and why. Avoid vague messages like “bugfix” or “small changes”.
• Commit Small and Often: Aim for small, coherent commits rather than massive, “everything” updates.
• Never Force-Push (git push -f) on Shared Branches: Force-pushing overwrites the remote history to match your local copy, permanently deleting any commits your collaborators have already pushed.
• Use git revert to Undo Shared History: When a bad commit has already been pushed, use git revert <hash> to create a new “anti-commit” that safely inverts the change while preserving the full history. Never use git reset --hard on shared branches — it rewrites history and breaks every collaborator’s local copy.
• Use .gitignore: Always include a .gitignore file to prevent tracking unnecessary or sensitive files, such as build artifacts or private keys.
• Pull Frequently: Regularly pull the latest changes from the main branch to catch merge conflicts early.

Git Command Manual

Common Git commands can be categorized into several functional groups, ranging from basic setup to advanced debugging and collaboration.

Configuration and Initialization

Before working with Git, you must establish your identity and initialize your project.

• git config: Used to set global or repository-specific settings. Common configurations include setting your username, email, and preferred text editor.
• git init: Initializes a new, empty Git repository in your current directory, allowing Git to begin tracking files.

The Core Workflow (Local Changes)

These commands manage the lifecycle of your changes across the three Git states: the working directory, the staging area (index), and the repository history.

• git add: Adds file contents to the staging area to be included in the next commit.
• git status: Provides an overview of which files are currently modified, staged for the next commit, or untracked by Git.
• git commit: Records a snapshot of all changes currently in the staging area and saves it as a new version in the local repository’s history. Professional practice encourages writing meaningful commit messages to help team members understand the “what” and “why” of changes.
• git log: Displays the sequence of past commits. Using git log -p allows you to see the actual changes (patches) introduced in each commit.
• git diff: Compares different versions of your project:
  • git diff: Compares the working directory to the staging area.
  • git diff HEAD: Compares the working directory to the latest commit.
  • git diff HEAD^ HEAD: Compares the parent commit to the latest commit (shows what the latest commit changed).
• git restore (Git 2.23+): The modern command for undoing file changes, replacing the file-restoration uses of the older git checkout and git reset:
  • git restore --staged <file>: Unstages a file, moving it out of the staging area while leaving working directory modifications untouched.
  • git restore <file>: Discards all uncommitted changes to a file in the working directory, restoring it to its last staged or committed state. This is irreversible — uncommitted changes will be permanently lost.

Branching and Merging

Branching allows for parallel development, such as working on a new feature without affecting the main codebase.

• git branch: Lists, creates, or deletes branches. A branch is a lightweight pointer (a 41-byte file in .git/refs/heads/) to a specific commit.
• git switch (recommended, Git 2.23+): The modern, dedicated command for navigating branches.
  • git switch <branch>: Switches to an existing branch.
  • git switch -c <new-branch>: Creates a new branch and immediately switches to it.
  • git switch --detach <commit>: Checks out an arbitrary commit in detached HEAD state for safely inspecting older code without affecting any branch.
• git checkout (legacy): The older multi-purpose command that handled both branch switching and file restoration. Still widely encountered in documentation and scripts. git checkout <branch> is equivalent to git switch <branch>; git checkout -b <name> is equivalent to git switch -c <name>.
• git merge: Integrates changes from one branch into another.
  • git merge --squash: Combines all commits from a feature branch into a single commit on the target branch to maintain a cleaner history.
• git rebase: Re-applies commits from one branch onto a new base. This is often used to create a linear history, though it must never be used on shared branches.

Remote Operations

These commands facilitate collaboration by syncing your local work with a remote server (like GitHub).

• git clone: Creates a local copy of an existing remote repository.
• git pull: Fetches changes from a remote repository and immediately merges them into your current local branch.
• git push: Uploads your local commits to a remote repository. Note: Never use git push -f (force-push) on shared branches, as it can overwrite and destroy work pushed by other team members.

Advanced and Debugging Tools

Git includes powerful utilities for handling complex scenarios and tracking down bugs.

• git stash / git stash pop: Temporarily saves uncommitted changes (both staged and unstaged) so you can switch contexts without making a messy commit. Use pop to re-apply those changes later.
• git cherry-pick: Selectively applies a single specific commit from one branch onto another.
• git bisect: Uses a binary search through commit history to find the exact commit that introduced a bug.
• git blame: Annotates each line of a file with the author and commit ID of the last person to modify it.
• git revert <commit>: Creates a new “anti-commit” that applies the exact inverse changes of a previous commit, safely undoing it without rewriting history. Prefer this over git reset whenever the commit to undo has already been pushed to a shared branch.
• git show: Displays detailed information about a specific Git object, such as a commit.
• git submodule: Allows you to include an external Git repository as a subdirectory of your project while maintaining its independent history.

Quiz

Git Commands Flashcards

Which Git command would you use for the following scenarios?

You have some uncommitted, incomplete changes in your working directory, but you need to switch to another branch to urgently fix a bug. How do you temporarily save your current work without making a messy commit?

git stash

git stash temporarily shelves your staged and unstaged changes for tracked files. To include brand new, untracked files in the stash, you must use git stash -u.

You know a bug was introduced recently, but you aren’t sure which commit caused it. How do you perform a binary search through your commit history to find the exact commit that broke the code?

git bisect

git bisect systematically halves your commit history, asking you to test if the bug is present at each step, making it highly efficient for tracking down regressions.

You are looking at a file and want to know exactly who last modified a specific line of code, and in which commit they did it.

git blame

git blame annotates each line of a file with the author and the commit hash of the last modification, which is very useful for figuring out who to ask about a piece of code.

You want to safely ‘undo’ a previous commit that introduced an error, but you don’t want to rewrite history or force-push. How do you create a new commit with the exact inverse changes?

git revert

git revert is the safest way to undo changes on a shared branch because it adds a new commit that nullifies the targeted commit, rather than deleting history.

You want to see exactly what has changed in your working directory compared to your last saved snapshot (the most recent commit).

git diff HEAD

Using git diff HEAD compares your current modified tracked files on disk with the most recent commit. Note that it does not show completely untracked files.

You have a feature branch with several experimental commits, but you only want to move one specific, completed commit over to your main branch.

git cherry-pick

git cherry-pick allows you to selectively grab a specific commit from one branch and apply it onto your current branch, without merging the entire branch history.

You want to integrate a feature branch into main, but instead of bringing over all 15 tiny incremental commits, you want them combined into one clean commit on the main branch.

git merge --squash

Squashing combines all the patches from the feature branch into a single set of changes, allowing you to make one clean commit on the target branch and keep the history tidy with git merge --squash.

You are building a massive project and want to include an entirely separate external Git repository as a subdirectory within your project, while keeping its history independent.

git submodule

git submodule allows you to nest an independent repository inside your main project. It essentially creates a pointer to a specific commit of that external library.

You are starting a brand new project in an empty folder on your computer and want Git to start tracking changes in this directory.

git init

This command initializes a new, empty Git repository in the current directory via git init, creating the hidden .git folder that manages all version control data.

You have just installed Git on a new computer and need to set up your username and email address so that your commits are properly attributed to you.

git config

Using git config --global user.name and user.email establishes your default identity, which can be overridden for specific projects using git config --local.

You’ve made changes to three different files, but you only want two of them to be included in your next snapshot. How do you move those specific files to the staging area?

git add <filename>

The git add command moves file modifications from the working directory into the staging area (index) to prepare them for the next commit.

You’ve lost track of what you’ve been doing. You want a quick overview of which files are modified, which are staged, and which are completely untracked by Git.

git status

This command git status provides a high-level summary of the working tree and staging area statuses, making it one of the most frequently used Git commands.

You have staged all the files for a completed feature and are ready to permanently save this snapshot to your local repository’s history with a descriptive message.

git commit -m 'message'

Committing takes everything in the staging area and records it as a permanent new version in your local Git history via git commit.

You want to review the chronological history of all past commits on your current branch, including their author, date, and commit message.

git log

This command git log prints out the commit history. You can also add flags like -p to see the exact patch/diff introduced in each commit.

You’ve made edits to a file but haven’t staged it yet. You want to see the exact lines of code you added or removed compared to what is currently in the staging area.

git diff

Running git diff without any arguments compares your current working directory against the staging area.

You want to start working on a completely new feature in isolation without affecting the main codebase.

git branch <branch-name>

This command git branch creates a new branch pointer at your current commit, allowing you to diverge from the main line of development.

You are currently on your feature branch and need to switch your working directory back to the ‘main’ branch.

git switch main

The modern git switch command navigates between branches, updating your working directory and moving HEAD. The legacy equivalent git checkout main still works and is widely seen in older documentation and scripts.

Your feature branch is complete, and you want to integrate its entire commit history into your current ‘main’ branch.

git merge <branch-name>

Merging with git merge takes the divergent histories of two branches and combines them, often resulting in a new ‘merge commit’ with two parents.

Instead of creating a merge commit, you want to take the commits from your feature branch and re-apply them directly on top of the latest ‘main’ branch to create a clean, linear history.

git rebase main

Rebasing with git rebase rewrites history by moving the base of your current branch to the tip of another, which creates a cleaner history but should be avoided on public/shared branches.

You want to start working on an open-source project hosted on GitHub. How do you download a full local copy of that repository to your machine?

git clone <url>

Cloning with git clone downloads the entire repository, including its full history and all branches, from a remote server to your local disk.

Your team members have uploaded new commits to the shared remote repository. You want to fetch those changes and immediately integrate them into your current local branch.

git pull

The git pull command is effectively a combination of git fetch and git merge (the default) or git rebase (if configured), integrating remote changes into your local branch.

You have finished making several commits locally and want to upload them to the remote GitHub repository so your team can see them.

git push

git push transmits your local commits to the remote server, updating the remote branch to match your local branch.

You have a specific commit hash and want to see detailed information about it, including the commit message, author, and the exact code diff it introduced.

git show <commit-hash>

git show displays the details of a specific Git object, most commonly used to inspect exactly what changed in a single commit.

You want to start working on a new feature in isolation. How do you create a new branch called ‘feature-auth’ and immediately switch to it in a single command?

git switch -c feature-auth

git switch -c creates a new branch and switches to it in one step. The -c flag stands for ‘create’. The legacy equivalent is git checkout -b feature-auth, which you will still encounter in older scripts and documentation.

You accidentally staged a file you didn’t intend to include in your next commit. How do you move it back to the working directory without losing your modifications?

git restore --staged <filename>

git restore --staged unstages a file by copying the version from the last commit back into the staging area, leaving your working directory modifications completely untouched. It is the modern equivalent of the older git reset HEAD <file>.

You made some experimental changes to a file but want to discard them entirely and revert to the version from your last commit.

git restore <filename>

git restore <file> discards unstaged changes in the working directory, reverting it to the staged version (or the last commit if nothing is staged). To explicitly revert to the last commit, use git restore --source=HEAD <file>.

You merge a feature branch into main, and Git performs the merge without creating a new merge commit — it simply moves the ‘main’ pointer forward. What type of merge is this, and when does it occur?

A fast-forward merge — occurs when ‘main’ has no new commits since the feature branch was created.

A fast-forward merge happens when the target branch hasn’t diverged. Git can simply advance its pointer in a straight line to the tip of the incoming branch, keeping the commit history perfectly linear. If both branches have diverged, Git performs a three-way merge and creates a merge commit instead.

You want to safely inspect the codebase at a specific older commit without modifying any branch. How do you do this?

git switch --detach <commit-hash>

git switch --detach places you in ‘detached HEAD’ state, where HEAD points directly to a specific commit rather than a branch. You can look around freely, but any commits made here are not anchored to a branch and can be lost when switching away. Use git switch -c <name> to anchor them to a new branch if needed.

Session Complete!

Your Score: 0/28

Come back later to improve your recall!

Version Control and Git Quiz

Test your knowledge of core version control concepts, Git architecture, branching strategies, and advanced commands.

Which of the following best describes the core difference between centralized and distributed version control systems (like Git)?

Correct Answer:

Explanation

This defines the distributed model, giving developers offline access to the full project history and the ability to work independently.

What are the three primary local states that a file can reside in within a standard Git workflow?

Correct Answer:

Explanation

Files are edited in the working directory, moved to the staging area to prep for a snapshot, and finally committed to the local repository.

What does the command git diff HEAD compare?

Correct Answer:

Explanation

Adding HEAD tells Git to compare the uncommitted files currently on your disk with the snapshot of the most recent commit.

Which Git command should you NEVER use on a shared branch because it can permanently overwrite and destroy work pushed by other team members?

Correct Answer:

Explanation

Force-pushing overwrites the remote repository’s history to exactly match yours, which will permanently delete any new commits made by your collaborators.

You have some uncommitted, incomplete changes in your working directory, but you need to switch to another branch to urgently fix a bug. Which command is best suited to temporarily save your current work without making a messy commit?

Correct Answer:

Explanation

git stash temporarily shelves your staged and unstaged changes, leaving you with a clean working directory so you can safely switch contexts.

What happens when you enter a ‘Detached HEAD’ state in Git?

Correct Answer:

Explanation

Checking out a specific commit hash detaches HEAD. Any new commits made in this state won’t update a branch pointer and can easily be lost.

Which Git command utilizes a binary search through your commit history to help you pinpoint the exact commit that introduced a bug?

Correct Answer:

Explanation

git bisect systematically halves your commit history, asking you to test if the bug is present at each step, to efficiently find the exact commit that broke the code.

What is the primary purpose of Git Submodules?

Correct Answer:

Explanation

Submodules allow you to nest an independent repository inside your main project, pointing to a specific commit of that external library.

Which of the following are advantages of a Distributed Version Control System (like Git) compared to a Centralized one? (Select all that apply)

Correct Answers:

Explanation

In distributed systems like Git, developers have a full local copy of the repository, allowing for offline work and eliminating the single point of failure found in centralized systems. This makes it ideal for large, distributed teams. However, it does not prevent merge conflicts.

Which of the following represent the core local states (or areas) where files can reside in a standard Git architecture? (Select all that apply)

Correct Answers:

Explanation

The three primary local states in Git’s architecture are the Working Directory (files on disk), the Staging Area (prepped for the next commit), and the Local Repository (the compressed history of commits). The remote server is external, and ‘Detached HEAD’ is a state of the HEAD pointer, not a file area.

Which of the following commands are primarily used to review changes, history, or differences in a Git repository? (Select all that apply)

Correct Answers:

Explanation

git log shows the commit history, git diff shows differences between states or commits, and git show displays details about a specific Git object (like a commit). git push uploads data, and git init creates a new repository.

In which of the following scenarios would using git stash be considered an appropriate and helpful practice? (Select all that apply)

Correct Answers:

Explanation

git stash temporarily shelves your local modifications (both staged and unstaged), leaving you with a clean working directory. This is perfect for quick context switches or pulling updates without committing messy, incomplete work. It is not used for permanent deletion or applying commits (which would be git cherry-pick).

Which of the following are valid methods or strategies for integrating changes from a feature branch back into the main codebase? (Select all that apply)

Correct Answers:

Explanation

Merging, rebasing, and squashing (git merge --squash) are all techniques used to integrate branch histories together. git bisect is a debugging tool for finding bugs, and git blame is used to see who last modified specific lines of code.

A faulty commit was pushed to a shared ‘main’ branch last week and your teammates have already synced it. Why should you use git revert to fix this rather than git reset --hard followed by a force-push?

Correct Answer:

Explanation

git revert is the safe choice on shared branches because it adds a new forward-moving commit that nullifies the targeted commit, without rewriting or destroying history. Using git reset --hard and force-pushing would overwrite the shared history, causing serious disruption for any teammate who has already synced that commit.

When integrating a feature branch into ‘main’, under what condition will Git perform a fast-forward merge rather than creating a three-way merge commit?

Correct Answer:

Explanation

A fast-forward merge occurs when the target branch’s history has not diverged — Git can simply slide its pointer forward to incorporate the changes linearly. If both branches have unique commits since they split, Git must perform a three-way merge using their common ancestor, resulting in a merge commit with two parents.

What does the file .git/HEAD contain when you are checked out on a branch, compared to when you are in a detached HEAD state?

Correct Answer:

Explanation

Normally, .git/HEAD is a symbolic reference like ref: refs/heads/main — a pointer to a branch pointer. When you enter detached HEAD state (e.g., with git switch --detach <hash>), HEAD is disconnected from any branch and contains the raw SHA hash of a specific commit directly. Any commits made in this state are not anchored to a branch and can be lost when switching away.

Quiz Complete!

Your Score: 0/16

Make

---

Motivation

Imagine you are building a small C program. It just has one file, main.c. To compile it, you simply open your terminal and type:

gcc main.c -o myapp

Easy enough, right?

Want to practice? Try the Interactive Makefile Tutorial — 10 hands-on exercises that build from basic rules to automatic variables and pattern rules, with real-time feedback.

Now, imagine your project grows. You add utils.c, math.c, and network.c. Your command grows too:

gcc main.c utils.c math.c network.c -o myapp

Still manageable. But what happens when you join a real-world software team? An operating system kernel or a large application might have thousands of source files. Typing them all out is impossible.

First Attempt: The Shell Script

To solve this, you might write a simple shell script (build.sh) that just compiles everything in the directory: gcc *.c -o myapp

This works, but it introduces a massive new problem: Time. Compiling a massive codebase from scratch can take minutes or even hours. If you fix a single typo in math.c, your shell script will blindly recompile all 9,999 other files that didn’t change. That is incredibly inefficient and will destroy your productivity as a developer.

The “Aha!” Moment: Incremental Builds

What you actually need is a smart tool that asks two questions before doing any work:

1. What exactly depends on what? (e.g., “The executable depends on the object files, and the object files depend on the C files and Header files”).
2. Has the source file been modified more recently than the compiled file?

If math.c was saved at 10:05 AM, but math.o (its compiled object file) was created at 9:00 AM, the tool knows math.c has changed and must be recompiled. If utils.c hasn’t been touched since yesterday, the tool completely skips recompiling it and just reuses the existing utils.o.

This is exactly why make was created in 1976, and why it remains a staple of software engineering today. While the original utility was created at Bell Labs, modern development primarily relies on GNU Make, a powerful and widely-extended implementation that reads a configuration file called a Makefile.

So GNU make is the project’s engine that reads recipes from Makefiles to build complex products.

How It Works

Inside a Makefile, you define three main components:

• Targets: What you want to build or the task you want to run.
• Prerequisites: The files that must exist (or be updated) before the target can be built.
• Commands: The exact terminal steps required to execute the target.

When you type make in your terminal, the tool analyzes the dependency graph and checks file modification timestamps. It then executes the bare minimum number of commands required to bring your program up to date.

The Dual Purpose

Makefiles are incredibly powerful—but their design can be confusing at first glance because they serve two distinct purposes:

1. Building Artifacts: Their primary, traditional use is for compiling languages (like C and C++), where they manage the complex process of turning source code into executable files.
2. Running Tasks: In modern development, they are frequently used with interpreted languages (like Python) as a convenient shortcut for common project tasks (e.g., make install, make test, make lint, make deploy).

Why We Need Makefiles

Ultimately, Makefiles are heavily relied upon because they:

1. Save massive amounts of time by enabling incremental builds (only recompiling the specific files that have changed).
2. Automate complex processes so developers don’t have to memorize long or tedious terminal commands.
3. Standardize workflows across teams by providing predictable, universal commands (like make test to run all tests or make clean to delete generated files).
4. Document dependencies, making it perfectly clear how all the individual pieces of a software system fit together.

The Cake Analogy

Think of Makefiles as a receipe book for baking a complex, multi-layered cake. Let’s make a spectacular three-tier chocolate cake with raspberry filling and buttercream frosting. A Makefile is your ultimate, highly-efficient kitchen manager and master recipe combined.

Here is how the concepts map together:

Concepts

1. The Targets (What you are making)

In a Makefile, a target is the file you want to generate.

• The Final Target (The Executable): This is the fully assembled, frosted, and decorated cake ready for the display window.
• Intermediate Targets (e.g., Object Files in C): These are the individual components that must be made before the final cake can be assembled. In this case, your intermediate targets are the baked chocolate layers, the raspberry filling, and the buttercream frosting. If we know how to bake each individual component and we know how to combine each of them together, we can bake the cake. Makefiles allow you to define the targets and the dependencies in a structured, isolated way that describes each component individually.

2. The Dependencies (What you need to make it)

Every target in a Makefile has dependencies—the things required to build it.

• Raw Source Code (Source Files): These are your raw ingredients: flour, sugar, cocoa powder, eggs, butter, and fresh raspberries.
• Chain of Dependencies: The Final Cake depends on the chocolate layers, filling, and frosting. The chocolate layers depend on flour, sugar, eggs, and cocoa powder.

Worked example of the Cake Recipe

Let’s build the Makefile for our cake recipe.

Iteration 1: The Basic Rule (The Blueprint)

The Need: We need to tell our kitchen manager (make) what our final goal is, what it requires, and how to put it together.

The Syntax: The most fundamental building block of a Makefile is a Rule. A rule has three parts:

1. Target: What you want to build (followed by a colon :).
2. Dependencies: What must exist before you can build it (separated by spaces).
3. Command: The actual terminal command to build it. CRITICAL: This line must start with a literal Tab character, not spaces.

# Step 1: The Basic Rule
cake: chocolate_layers raspberry_filling buttercream
	echo "Stacking chocolate_layers, raspberry_filling, and buttercream to make the cake."
	touch cake

Note: If you run this now (i.e., ask the kitchen manager to bake the cake), make cake will complain: “No rule to make target ‘chocolate_layers’”. It knows it needs them, but it doesn’t know how to bake them.

Iteration 2: The Dependency Chain

The Need: We need to teach make how to create the missing intermediate ingredients so it can satisfy the requirements of the final cake.

The Syntax: We simply add more rules. make reads top-to-bottom, but executes bottom-to-top based on what the top target needs.

# Step 2: Adding the Chain
cake: chocolate_layers raspberry_filling buttercream
	echo "Stacking layers, filling, and frosting to make the cake."
	touch cake

chocolate_layers: flour.txt sugar.txt eggs.txt cocoa.txt
	echo "Mixing ingredients and baking at 350 degrees."
	touch chocolate_layers

raspberry_filling: raspberries.txt sugar.txt
	echo "Simmering raspberries and sugar."
	touch raspberry_filling

buttercream: butter.txt powdered_sugar.txt
	echo "Whipping butter and sugar."
	touch buttercream

Now the kitchen works! But notice we hardcoded “350 degrees”. If we get a new convection oven that bakes at 325 degrees, we have to manually find and change that number in every single baking rule.

Iteration 3: Variables (Macros)

The Need: We want to define our kitchen settings in one place at the top of the file so they are easy to change later.

The Syntax: You define a variable with NAME = value and you use it by wrapping it in a dollar sign and parentheses: $(NAME).

# Step 3: Variables
OVEN_TEMP = 350
MIXER_SPEED = high

cake: chocolate_layers raspberry_filling buttercream
	echo "Stacking layers to make the cake."
	touch cake

chocolate_layers: flour.txt sugar.txt eggs.txt cocoa.txt
	echo "Baking at $(OVEN_TEMP) degrees."
	touch chocolate_layers

buttercream: butter.txt powdered_sugar.txt
	echo "Whipping at $(MIXER_SPEED) speed."
	touch buttercream

(I’ve omitted the filling rule here just to keep the example short, but you get the idea).

---

Iteration 4: Automatic Variables (The Shortcuts)

The Need: Look at the chocolate_layers rule. We list all the ingredients in the dependencies, but in a real C++ program, you also have to list all those exact same files again in the compiler command. Typing things twice causes typos.

The Syntax: Makefiles have built-in “Automatic Variables” that act as shortcuts:

• $@ automatically means “The name of the current target.”
• $^ automatically means “The names of ALL the dependencies.”

# Step 4: Automatic Variables
OVEN_TEMP = 350

cake: chocolate_layers raspberry_filling buttercream
	echo "Making $@" 
	touch $@

chocolate_layers: flour.txt sugar.txt eggs.txt cocoa.txt
	echo "Taking $^ and baking them at $(OVEN_TEMP) to make $@"
	touch $@

Now, the command echo "Taking $^ ..." will automatically print out: “Taking flour.txt sugar.txt eggs.txt cocoa.txt…”. If you add a new ingredient to the dependency list later, the command updates automatically!

---

Iteration 5: Phony Targets (.PHONY)

The Need: Sometimes we make a terrible mistake and just want to throw everything in the trash and start completely over. We want a command to wipe the kitchen clean.

The Syntax: We create a rule called clean that deletes files. However, what if you accidentally create a real text file named “clean” in your folder? make will look at the file, see it has no dependencies, and say “The file ‘clean’ is already up to date. I don’t need to do anything.”

To fix this, we use .PHONY. This tells make: “Hey, this isn’t a real file. It’s just a command name. Always run it when I ask.”

# Step 5: The Final, Complete Scaffolding
OVEN_TEMP = 350

cake: chocolate_layers raspberry_filling buttercream
	echo "Making $@" 
	touch $@

chocolate_layers: flour.txt sugar.txt eggs.txt cocoa.txt
	echo "Taking $^ and baking them at $(OVEN_TEMP) to make $@"
	touch $@

# ... (other recipes) ...

.PHONY: clean
clean:
	echo "Throwing everything in the trash!"
	rm -f cake chocolate_layers raspberry_filling buttercream

By typing make clean in your terminal, the kitchen is reset. By typing make cake (or just make, as it defaults to the first rule), your fully automated bakery springs to life.

Now we get this completete Makefile:

# ---------------------------------------------------------
# Complete Makefile for a Three-Tier Chocolate Raspberry Cake
# ---------------------------------------------------------

# Variables (Kitchen settings)
OVEN_TEMP = 350F
MIXER_SPEED = medium-high

# 1. The Final Target: The Cake
# Depends on the baked layers, filling, and frosting
cake: chocolate_layers raspberry_filling buttercream
	@echo "🎂 Assembling the final cake!"
	@echo "-> Stacking layers, spreading filling, and covering with frosting."
	@touch cake
	@echo "✨ Cake is ready for the display window! ✨"

# 2. Intermediate Target: Chocolate Layers
# Depends on raw ingredients (our source files)
chocolate_layers: flour.txt sugar.txt eggs.txt cocoa.txt
	@echo "🥣 Mixing flour, sugar, eggs, and cocoa..."
	@echo "🔥 Baking in the oven at $(OVEN_TEMP) for 30 minutes."
	@touch chocolate_layers
	@echo "✅ Chocolate layers are baked."

# 3. Intermediate Target: Raspberry Filling
raspberry_filling: raspberries.txt sugar.txt lemon_juice.txt
	@echo "🍓 Simmering raspberries, sugar, and lemon juice."
	@touch raspberry_filling
	@echo "✅ Raspberry filling is thick and ready."

# 4. Intermediate Target: Buttercream Frosting
buttercream: butter.txt powdered_sugar.txt vanilla.txt
	@echo "🧁 Whipping butter and sugar at $(MIXER_SPEED) speed."
	@touch buttercream
	@echo "✅ Buttercream frosting is fluffy."

# 5. Pattern Rule: "Shopping" for Raw Ingredients
# In a real codebase, these would already exist as your code files.
# Here, if an ingredient (.txt file) is missing, Make creates it.
%.txt:
	@echo "🛒 Buying ingredient: $@"
	@touch $@

# 6. Phony Target: Clean the kitchen
# Removes all generated files so you can bake from scratch
.PHONY: clean
clean:
	@echo "🧽 Cleaning up the kitchen..."
	@rm -f cake chocolate_layers raspberry_filling buttercream *.txt
	@echo "🧹 Kitchen is spotless!"

3. The Rules (The Recipe/Commands)

In a Makefile, the rule or command is the specific action the compiler must take to turn the dependencies into the target.

• Compiling: The rule to turn flour, sugar, and eggs into a chocolate layer is: “Mix ingredients in bowl A, pour into a 9-inch pan, and bake at 350°F for 30 minutes.”
• Linking: The rule to turn the individual layers, filling, and frosting into the Final Cake is: “Stack layer, spread filling, stack layer, cover entirely with frosting.”

This can be visualized as a dependency graph:

The Real Magic: Incremental Baking (Why we use Makefiles)

The true power of a Makefile isn’t just knowing how to bake the cake; it’s knowing what doesn’t need to be baked again. Make looks at the “timestamps” of your files to save time.

Imagine you are halfway through assembling your cake. You have your baked chocolate layers sitting on the counter, your buttercream whipped, and your raspberry filling ready. Suddenly, you realize someone mislabeled the sugar. It’s actually salt! Oh no! You need to remake everything that included sugar and everything that included these intermediate target.

• Without a Makefile: You would throw away everything. You would re-bake the chocolate layers, re-whip the buttercream, and remake the raspberry filling from scratch. This takes hours (like recompiling a massive codebase from scratch).
• With a Makefile: The kitchen manager (make) looks at the counter. It sees that the buttercream is already finished and its raw ingredients haven’t changed. However, it sees your new packed of sugar (a source file was updated). The manager says: “Only remake the raspberry filling and the chocolate layers, and then reassemble the final cake. Leave the buttercream as is.”

If you look closely at the arrows of the dependency graph above and focus on the arrows leaving [sugar.txt], you can immediately see the brilliance of make:

1. The Split Path: The arrow from sugar.txt forks into two different directions: one goes to the Chocolate_Layers and the other goes to the Raspberry_Filling.
2. The Safe Zone: Notice there is absolutely no arrow connecting sugar.txt to the Buttercream (which uses powdered sugar instead).
3. The Chain Reaction: When make detects that sugar.txt has changed (because you fixed the salty sugar), it travels along those two specific arrows. It forces the Chocolate Layers and Raspberry filling to be remade. Those updates then trigger the double-lined arrows ══▶, forcing the Final Cake to be reassembled.

Because no arrow carried the “sugar update” to the Buttercream, the Buttercream is completely ignored during the rebuild!

A Recipe as a Makefile

If your cake recipe were written as a Makefile, it would look exactly like this:

Final_Cake: Chocolate_Layers Raspberry_Filling Buttercream Stack components and frost the outside.

Chocolate_Layers: Flour Sugar Eggs Cocoa Mix ingredients and bake at 350°F for 30 minutes.

Raspberry_Filling: Raspberries Sugar Lemon_Juice Simmer on the stove until thick.

Buttercream: Butter Powdered_Sugar Vanilla Whip in a stand mixer until fluffy.

Whenever you type make in your terminal, the system reads this recipe from the top down, checks what is already sitting in your “kitchen,” and only does the work absolutely necessary to give you a fresh cake.

Makefile Syntax

How Do Makefiles Work?

A Makefile is built around a simple logical structure consisting of Rules. A rule generally looks like this:

target: prerequisites
	command

• Target: The file you want to generate (like an executable or an object file), or the name of an action to carry out (like clean).
• Prerequisites (Dependencies): The files that are required to build the target.
• Commands (Recipe): The shell commands that make executes to build the target. (Note: Commands MUST be indented with a Tab character, not spaces!)

When you run make, it looks at the target. If any of the prerequisites have a newer modification timestamp than the target, make executes the commands to update the target. The relationships you define matter immensely; for example, if you remove the object files ($(OBJS)) dependency from your main executable rule (e.g., $(EXEC): $(OBJS)), make will no longer know how to re-link the executable when its constituent object files change.

Syntax Basics

To write flexible and scalable Makefiles, you will use a few specific syntactic features:

• Variables (Macros): Variables act as placeholders for command-line options, making the build rules cleaner and easier to modify. For example, you can define a variable for your compiler (CC = clang) and your compiler flags (CFLAGS = -Wall -g). When you want to use the variable, you wrap it in parentheses and a dollar sign: $(CC).
• String Substitution: You can easily transform lists of files. For example, to generate a list of .o object files from a list of .c source files, you can use the syntax: OBJS = $(SRCS:.c=.o).
• Automatic Variables: make provides special variables to make rules more concise.
  • $@ represents the target name.
  • $< represents the first prerequisite.
• Pattern Rules: Pattern rules serve as templates for creating many rules with the identical structure. For instance, %.o : %.c defines a generic rule for creating a .o (object) file from a corresponding .c (source) file.

A Worked Example

Let’s tie all of these concepts together into a stereotypical, robust Makefile for a C program.

# Variables
SRCS = mysrc1.c mysrc2.c
TARGET = myprog
OBJS = $(SRCS:.c=.o)
CC = clang
CFLAGS = -Wall

# Main Target Rule
$(TARGET): $(OBJS)
	$(CC) $(CFLAGS) -o $(TARGET) $(OBJS)

# Pattern Rule for Object Files
%.o: %.c
	$(CC) $(CFLAGS) -c $< -o $@

# Clean Target
clean:
	rm -f $(OBJS) $(TARGET)

Breaking it down:

• Line 2-6: We define our variables. If we later want to use the gcc compiler instead, or add an optimization flag like -O3, we only need to change the CC or CFLAGS variables at the top of the file.
• Line 9-10: This rule says: “To build myprog, I need mysrc1.o and mysrc2.o. To build it, run clang -Wall -o myprog mysrc1.o mysrc2.o.”
• Line 13-14: This pattern rule explains how to turn a .c file into a .o file. It tells Make: “To compile any object file, use the compiler to compile the first prerequisite ($<, which is the .c file) and output it to the target name ($@, which is the .o file)”.
• Line 17-18: The clean target is a convention used to remove all generated object files and the target executable, leaving only the original source files. You can execute it by running make clean.

Quiz

Makefile Flashcards (Syntax Production/Recall)

Test your ability to produce the exact Makefile syntax, rules, and variables based on their functional descriptions.

What is the standard syntax to define a basic build rule in a Makefile?

target: prerequisites command

A rule defines the target file to be built, the prerequisite files it depends on, and the command(s) to execute to build it.

What specific whitespace character MUST be used to indent the command/recipe lines in a Makefile rule?

A Tab character

If you use spaces instead of a Tab, make will throw an error (often ‘missing separator’).

How do you reference a variable (or macro) named ‘CC’ in a Makefile command?

$(CC) or ${CC}

Variables are defined using an equals sign (e.g., CC = gcc) and referenced by wrapping the name in parentheses or curly braces preceded by a dollar sign.

What Automatic Variable represents the file name of the target of the rule?

$@

This is commonly used in the compilation command to specify the output file name dynamically.

What Automatic Variable represents the name of the first prerequisite?

$<

This is frequently used in pattern rules to pass the specific source file (like a .c file) to the compiler.

What Automatic Variable represents the names of all the prerequisites, with spaces between them?

$^

This is often used in linking commands where all object files need to be passed to the compiler at once to create an executable.

What wildcard character is used to define a Pattern Rule (a generic rule applied to multiple files)?

% (Percent sign)

For example, %.o: %.c tells make how to compile any .o object file from a corresponding .c source file.

What special target is used to declare that a target name is an action (like ‘clean’) and not an actual file to be created?

.PHONY: target_name

This prevents conflicts if a file literally named ‘clean’ is ever created in the directory, ensuring the recipe will always run when invoked.

What metacharacter can be placed at the very beginning of a recipe command to suppress make from echoing the command to the terminal?

@ (At symbol)

For example, @echo "Done!" will print ‘Done!’ to the terminal without printing the actual ‘echo’ command first.

What syntax is used for string substitution on a variable, such as changing all .c extensions in $(SRCS) to .o?

$(SRCS:.c=.o)

This evaluates to the value of the SRCS variable, but replaces the suffix .c with .o for every word in the list.

Session Complete!

Your Score: 0/10

Come back later to improve your recall!

Makefile Flashcards (Example Generation)

Test your knowledge on solving common build automation problems using Makefile syntax and rules!

Write a basic Makefile rule to compile a single C source file (main.c) into an executable named app.

app: main.c gcc main.c -o app

This defines app as the target, main.c as the prerequisite, and provides the exact compilation command required to build the executable. (Remember the command must be indented with a Tab).

Write a Makefile snippet that defines variables for the C compiler (gcc) and standard compilation flags (-Wall -g), and uses them to compile main.c into main.o.

`CC = gcc CFLAGS = -Wall -g

main.o: main.c $(CC) $(CFLAGS) -c main.c -o main.o`

Variables make the Makefile flexible. If you later want to switch to clang or add optimization flags, you only need to change the variable definitions at the top.

Write a standard clean target that removes all .o files and an app executable, ensuring it runs even if a file literally named ‘clean’ is created in the directory.

.PHONY: clean clean: rm -f *.o app

The .PHONY declaration tells make that clean is an action, not a file. The -f flag in rm prevents errors if the files don’t exist.

Write a generic pattern rule to compile any .c file into a corresponding .o file, using automatic variables for the target name and the first prerequisite.

%.o: %.c $(CC) $(CFLAGS) -c $< -o $@

The % acts as a wildcard. $< substitutes the name of the .c file, and $@ substitutes the name of the .o file.

Given a variable SRCS = main.c utils.c, write a variable definition for OBJS that dynamically replaces the .c extension with .o for all files in SRCS.

OBJS = $(SRCS:.c=.o)

This is a substitution reference. It takes the value of SRCS, looks for the .c suffix at the end of each word, and replaces it with .o.

Write a rule to link an executable myprog from a list of object files stored in the $(OBJS) variable, using the automatic variable that lists all prerequisites.

myprog: $(OBJS) $(CC) $^ -o $@

$^ expands to a space-separated list of all prerequisites (all the .o files), and $@ expands to the target name (myprog).

Write the conventional default target rule that is used to build multiple executables (e.g., app1 and app2) when a user simply types make without specifying a target.

all: app1 app2

By convention, all is the first target in a Makefile. Since make defaults to the first target it sees, this guarantees both apps are built. No commands are needed; simply listing them as prerequisites triggers their respective build rules.

Write a run target that executes an output file named ./app, but suppresses make from printing the command to the terminal before running it.

run: app @./app

Adding the @ symbol at the very beginning of the recipe line tells make to execute the command silently (without echoing it to standard output first).

Write a variable definition SRCS that uses a Make function to dynamically find and list all .c files in the current directory.

SRCS = $(wildcard *.c)

The wildcard function tells make to search the filesystem and return a space-separated list of all files matching the given pattern.

Write a generic rule to create a build directory build/ using the mkdir command.

build/: mkdir -p $@

Directories can be targets just like regular files. The -p flag ensures mkdir doesn’t throw an error if the directory already exists.

Session Complete!

Your Score: 0/10

Come back later to improve your recall!

C Program Makefile Flashcards

Test your ability to read and understand actual Makefile snippets commonly found in real-world C projects.

Given the snippet app: main.o network.o utils.o followed by the command $(CC) $(CFLAGS) $^ -o $@, what exactly does the command evaluate to if CC=gcc and CFLAGS=-Wall?

gcc -Wall main.o network.o utils.o -o app

$^ expands to all prerequisites (the three .o files), and $@ expands to the target name (app). This is the standard way to link object files into a final C executable.

If a C project Makefile contains SRCS = main.c math.c io.c and OBJS = $(SRCS:.c=.o), what does OBJS evaluate to?

main.o math.o io.o

This is a substitution reference. It iterates through every file listed in SRCS, finds the .c extension, and replaces it with .o to dynamically generate the list of object files needed for compilation.

Read this common pattern rule: %.o: %.c followed by $(CC) $(CFLAGS) -c $< -o $@. If make uses this rule to build utils.o from utils.c, what does $< represent?

utils.c

In a pattern rule, $< is an automatic variable that evaluates to the first prerequisite. In this case, it passes the specific C source file being compiled to the compiler.

You see the line CC ?= gcc at the top of a Makefile. What happens if a developer compiles the project by typing make CC=clang in their terminal?

The compiler used will be clang, not gcc.

The ?= operator is a conditional assignment. It only sets CC to gcc if CC has not already been defined. The command-line argument overrides the default.

A C project has a rule clean: rm -f *.o myapp. Why is it critical to also include .PHONY: clean in this Makefile?

To ensure the cleanup runs even if a developer accidentally creates a file literally named clean in the project directory.

Without .PHONY, if a file named clean exists and has no prerequisites, make will think the target clean is already “up to date” and will refuse to execute the rm command.

In the rule main.o: main.c main.h types.h, what happens if you edit and save types.h?

make will recompile main.o the next time it is run.

Even though types.h is not directly passed to the compiler in the recipe, listing header files as prerequisites tells make to track their modification timestamps. If a header changes, the object files that depend on it must be rebuilt.

You are reading a Makefile and see @echo "Compiling $@..." followed by @$(CC) -c $< -o $@. What do the @ symbols do?

They suppress the terminal from printing the actual commands before executing them.

Instead of seeing the messy gcc command printed out, the developer will only see the clean, custom message: Compiling main.o....

What is the conventional purpose of the CFLAGS variable in a C Makefile?

To store flags and options passed to the C compiler (e.g., warnings and optimizations).

Common values include -Wall (enable all warnings), -Wextra (extra warnings), -g (include debugging symbols), or -O2 (optimize for speed).

What is the conventional purpose of the LDFLAGS or LDLIBS variables in a C Makefile?

To store flags and library links passed to the linker during the final executable creation.

For example, if your C program uses the math library <math.h>, you would define LDLIBS = -lm so the linker knows to include it when combining the .o files.

A C project has multiple executables: a server and a client. The Makefile starts with all: server client. What happens if you just type make?

It will build both the server and the client executables.

make defaults to the first target defined in the file (which is all here). By listing both programs as prerequisites for all, it forces make to evaluate and build the rules for both of them.

Session Complete!

Your Score: 0/10

Come back later to improve your recall!

Make and Makefiles Quiz

Test your understanding of Makefiles, including syntax rules, execution order, automatic variables, and underlying concepts like incremental compilation.

What is the primary mechanism make uses to determine if a target needs to be rebuilt?

Correct Answer:

Explanation

make looks at the file modification timestamps. If a prerequisite (like a .c file) has a newer timestamp than the target (like a .o file), make knows the source code has changed and recompiles the target. This enables efficient incremental builds.

What specific whitespace character MUST be used to indent the command/recipe lines in a Makefile rule?

Correct Answer:

Explanation

Makefile syntax strictly requires command lines (recipes) to be indented with a single Tab character. Using spaces will result in a ‘missing separator’ error.

What does the automatic variable $@ represent in a Makefile rule?

Correct Answer:

Explanation

$@ evaluates to the target name. It is heavily used in compilation commands (e.g., gcc ... -o $@) to dynamically specify the output file name.

Why is the .PHONY directive used in Makefiles (e.g., .PHONY: clean)?

Correct Answer:

Explanation

If you have a rule named clean, and a file literally named clean is accidentally created in the directory, make clean will say ‘clean is up to date’ and do nothing. .PHONY: clean forces make to execute the recipe regardless of whether a file named clean exists.

If a user runs the make command in their terminal without specifying a target, what will make do?

Correct Answer:

Explanation

By default, make begins execution with the first target it encounters in the file. By convention, developers often make the first target all (which depends on the main executables) so that running make builds the entire project.

You have a pattern rule: %.o: %.c. What does the % symbol do?

Correct Answer:

Explanation

The % symbol is a wildcard in pattern rules. %.o: %.c acts as a generic template, avoiding the need to write individual rules for every single source file in a project.

Which of the following are primary benefits of using a Makefile instead of a standard procedural Bash script (build.sh)? (Select all that apply)

Correct Answers:

Explanation

Makefiles save time via incremental compilation, manage complex dependency graphs automatically (declarative vs procedural), and provide standardized commands. They do not run the compiler faster, nor do they write code for you.

Which of the following are valid Automatic Variables in Make? (Select all that apply)

Correct Answers:

Explanation

$@, $<, and $^ are standard automatic variables used to make Makefile rules concise and dynamic. $# and $$ are Bash variables, not Make automatic variables.

In standard C/C++ project Makefiles, which of the following variables are common conventions used to increase flexibility? (Select all that apply)

Correct Answers:

Explanation

CC, CFLAGS, and LDFLAGS are universal conventions in Makefiles. Using them allows developers to easily swap compilers or add flags via command line arguments (e.g., make CC=clang) without editing the actual Makefile. MAKE_IT_FAST is not a standard convention.

How does the evaluation logic of a Makefile differ from a standard cookbook recipe or procedural script? (Select all that apply)

Correct Answers:

Explanation

Makefiles are declarative, not procedural. They do not execute top-to-bottom sequentially, nor do they execute blindly. They build a dependency graph from the top-down (starting at the target), and then execute the necessary commands from the bottom-up, skipping anything that is already up to date.

Quiz Complete!

Your Score: 0/10

Design Patterns

---

Overview

In software engineering, a design pattern is a common, acceptable solution to a recurring design problem that arises within a specific context. The concept did not originate in computer science, but rather in architecture. Christopher Alexander, an architect who pioneered the idea, defined a pattern beautifully: “Each pattern describes a problem which occurs over and over again in our environment, and then describes the core of the solution to that problem, in such a way that you can use this solution a million times over, without ever doing it the same way twice”.

In software development, design patterns refer to medium-level abstractions that describe structural and behavioral aspects of software. They sit between low-level language idioms (like how to efficiently concatenate strings in Java) and large-scale architectural patterns (like Model-View-Controller or client-server patterns). Structurally, they deal with classes, objects, and the assignment of responsibilities; behaviorally, they govern method calls, message sequences, and execution semantics.

Anatomy of a Pattern

A true pattern is more than simply a good idea or a random solution; it requires a structured format to capture the problem, the context, the solution, and the consequences. While various authors use slightly different templates, the fundamental anatomy of a design pattern contains the following essential elements:

• Pattern Name: A good name is vital as it becomes a handle we can use to describe a design problem, its solution, and its consequences in a word or two. Naming a pattern increases our design vocabulary, allowing us to design and communicate at a higher level of abstraction.
• Context: This defines the recurring situation or environment in which the pattern applies and where the problem exists.
• Problem: This describes the specific design issue or goal you are trying to achieve, along with the constraints symptomatic of an inflexible design.
• Forces: This outlines the trade-offs and competing concerns that must be balanced by the solution.
• Solution: This describes the elements that make up the design, their relationships, responsibilities, and collaborations. It specifies the spatial configuration and behavioral dynamics of the participating classes and objects.
• Consequences: This explicitly lists the results, costs, and benefits of applying the pattern, including its impact on system flexibility, extensibility, portability, performance, and other quality attributes.

GoF Design Patterns

Here are some examples of design patterns that we describe in more detail:

• State: Encapsulates state-based behavior into distinct classes, allowing a context object to dynamically alter its behavior at runtime by delegating operations to its current state object.

• Observer: Establishes a one-to-many dependency between objects, ensuring that a group of dependent objects is automatically notified and updated whenever the internal state of their shared subject changes.

Architectural Patterns

Here are some examples of architectural patterns that we describe in more detail:

• Model-View-Controller (MVC): The Model-View-Controller (MVC) architectural pattern decomposes an interactive application into three distinct components: a model that encapsulates the core application data and business logic, a view that renders this information to the user, and a controller that translates user inputs into corresponding state updates

The Benefits of a Shared Toolbox

Just as a mechanic must know their toolbox, a software engineer must know design patterns intimately—understanding their advantages, disadvantages, and knowing precisely when (and when not) to use them.

• A Common Language for Communication: The primary challenge in multi-person software development is communication. Patterns solve this by providing a robust, shared vocabulary. If an engineer suggests using the “Observer” or “Strategy” pattern, the team instantly understands the problem, the proposed architecture, and the resulting interactions without needing a lengthy explanation.
• Capturing Design Intent: When you encounter a design pattern in existing code, it communicates not only what the software does, but why it was designed that way.
• Reusable Experience: Patterns are abstractions of design experience gathered by seasoned practitioners. By studying them, developers can rely on tried-and-tested methods to build flexible and maintainable systems instead of reinventing the wheel.

Challenges and Pitfalls of Design Patterns

Despite their power, design patterns are not silver bullets. Misusing them introduces severe challenges:

• The “Hammer and Nail” Syndrome: Novice developers who just learned patterns often try to apply them to every problem they see. Software quality is not measured by the number of patterns used. Often, keeping the code simple and avoiding a pattern entirely is the best solution.
• Over-engineering vs. Under-engineering: Under-engineering makes software too rigid for future changes. However, over-applying patterns leads to over-engineering—creating premature abstractions that make the codebase unnecessarily complex, unreadable, and a waste of development time. Developers must constantly balance simplicity (fewer classes and patterns) against changeability (greater flexibility but more abstraction).
• Implicit Dependencies: Patterns intentionally replace static, compile-time dependencies with dynamic, runtime interactions. This flexibility comes at a cost: it becomes harder to trace the execution flow and state of the system just by reading the code.
• Misinterpretation as Recipes: A pattern is an abstract idea, not a snippet of code from Stack Overflow. Integrating a pattern into a system is a human-intensive, manual activity that requires tailoring the solution to fit a concrete context.

Context Tailoring

It is important to remember that the standard description of a pattern presents an abstract solution to an abstract problem. Integrating a pattern into a software system is a highly human-intensive, manual activity; patterns cannot simply be misinterpreted as step-by-step recipes or copied as raw code. Instead, developers must engage in context tailoring—the process of taking an abstract pattern and instantiating it into a concrete solution that perfectly fits the concrete problem and the concrete context of their application.

Because applying a pattern outside of its intended problem space can result in bad design (such as the notorious over-use of the Singleton pattern), tailoring ensures that the pattern acts as an effective tool rather than an arbitrary constraint.

The Tailoring Process: The Measuring Tape and the Scissors

Context tailoring can be understood through the metaphor of making a custom garment, which requires two primary steps: using a “measuring tape” to observe the context, and using “scissors” to make the necessary adjustments.

1. Observation of Context

Before altering a design pattern, you must thoroughly observe and measure the environment in which it will operate. This involves analyzing three main areas:

• Project-Specific Needs: What kind of evolution is expected? What features are planned for the future, and what frameworks is the system currently relying on?
• Desired System Properties: What are the overarching goals of the software? Must the architecture prioritize run-time performance, strict security, or long-term maintainability?
• The Periphery: What is the complexity of the surrounding environment? Which specific classes, objects, and methods will directly interact with the pattern’s participants?

2. Making Adjustments

Once the context is mapped, developers must “cut” the pattern to fit. This requires considering the broad design space of the pattern and exploring its various alternatives and variation points. After evaluating the context-specific consequences of these potential variations, the developer implements the most suitable version. Crucially, the design decisions and the rationale behind those adjustments must be thoroughly documented. Without documentation, future developers will struggle to understand why a pattern deviates from its textbook structure.

Dimensions of Variation

Every design pattern describes a broad design space containing many distinct variations. When tailoring a pattern, developers typically modify it along four primary dimensions:

Structural Variations

These variations alter the roles and responsibility assignments defined in the abstract pattern, directly impacting how the system can evolve. For example, the Factory Method pattern can be structurally varied by removing the abstract product class entirely. Instead, a single concrete product is implemented and configured with different parameters. This variation trades the extensibility of a massive subclass hierarchy for immediate simplicity.

Behavioral Variations

Behavioral variations modify the interactions and communication flows between objects. These changes heavily impact object responsibilities, system evolution, and run-time quality attributes like performance. A classic example is the Observer pattern, which can be tailored into a “Push model” (where the subject pushes all updated data directly to the observer) or a “Pull model” (where the subject simply notifies the observer, and the observer must pull the specific data it needs).

Internal Variations

These variations involve refining the internal workings of the pattern’s participants without necessarily changing their external structural interfaces. A developer might tailor a pattern internally by choosing a specific list data structure to hold observers, adding thread-safety mechanisms, or implementing a specialized sorting algorithm to maximize performance for expected data sets.

Language-Dependent Variations

Modern programming languages offer specific constructs that can drastically simplify pattern implementations. For instance, dynamically typed languages can often omit explicit interfaces, and aspect-oriented languages can replace standard polymorphism with aspects and point-cuts. However, there is a dangerous trap here: using language features to make a pattern entirely reusable as code (e.g., using include Singleton in Ruby) eliminates the potential for context tailoring. Design patterns are fundamentally about design reuse, not exact code reuse.

The Global vs. Local Optimum Trade-off

While context tailoring is essential, it introduces a significant challenge in large-scale software projects. Perfectly tailoring a pattern to every individual sub-problem creates a “local optimum”. However, a large amount of pattern variation scattered throughout a single project can lead to severe confusion due to overloaded meaning.

If developers use the textbook Observer pattern in one module, but highly customized, structurally varied Observers in another, incoming developers might falsely assume identical behavior simply because the classes share the “Observer” naming convention. To mitigate this, large teams must rely on project conventions to establish pattern consistency. Teams must explicitly decide whether to embrace diverse, highly tailored implementations (and name them distinctly) or to enforce strict guidelines on which specific pattern variants are permitted within the codebase.

Pattern Compounds

In software design, applying individual design patterns is akin to utilizing distinct compositional techniques in photography—such as symmetry, color contrast, leading lines, and a focal object. Simply having these patterns present does not guarantee a masterpiece; their deliberate arrangement is crucial. When leading lines intentionally point toward a focal object, a more pleasing image emerges. In software architecture, this synergistic combination is known as a pattern compound.

A pattern compound is a reoccurring set of patterns with overlapping roles from which additional properties emerge. Notably, pattern compounds are patterns in their own right, complete with an abstract problem, an abstract context, and an abstract solution. While pattern languages provide a meta-level conceptual framework or grammar for how patterns relate to one another, pattern compounds are concrete structural and behavioral unifications.

The Anatomy of Pattern Compounds

The core characteristic of a pattern compound is that the participating domain classes take on multiple superimposed roles simultaneously. By explicitly connecting patterns, developers can leverage one pattern to solve a problem created by another, leading to a new set of emergent properties and consequences.

Solving Structural Complexity: The Composite Builder

The Composite pattern is excellent for creating unified tree structures, but initializing and assembling this abstract object structure is notoriously difficult. The Builder pattern, conversely, is designed to construct complex object structures. By combining them, the Composite’s Component acts as the Builder’s AbstractProduct, while the Leaf and Composite act as ConcreteProducts.

This compound yields the emergent properties of looser coupling between the client and the composite structure and the ability to create different representations of the encapsulated composite. However, as a trade-off, dealing with a recursive data structure within a Builder introduces even more complexity than using either pattern individually.

Managing Operations: The Composite Visitor and Composite Command

Pattern compounds frequently emerge when scaling behavioral patterns to handle structural complexity:

• Composite Visitor: If a system requires many custom operations to be defined on a Composite structure without modifying the classes themselves (and no new leaves are expected), a Visitor can be superimposed. This yields the emergent property of strict separation of concerns, keeping core structural elements distinct from use-case-specific operations.
• Composite Command: When a system involves hierarchical actions that require a simple execution API, a Composite Command groups multiple command objects into a unified tree. This allows individual command pieces to be shared and reused, though developers must manage the consequence of execution order ambiguity.

Communicating Design Intent and Context Tailoring

Pattern compounds also naturally arise when tailoring patterns to specific contexts or when communicating highly specific design intents.

• Null State / Null Strategy: If an object enters a “do nothing” state, combining the State pattern with the Null Object pattern perfectly communicates the design intent of empty behavior. (Note that there is no Null Decorator, as a decorator must fully implement the interface of the decorated object).
• Singleton State: If State objects are entirely stateless—meaning they carry behavior but no data, and do not require a reference back to their Context—they can be implemented as Singletons. This tailoring decision saves memory and eases object creation, though it permanently couples the design by removing the ability to reference the Context in the future.

The Advantages of Compounding Patterns

The primary advantage of pattern compounds is that they make software design more coherent. Instead of finding highly optimized but fragmented patchwork solutions for every individual localized problem, compounds provide overarching design ideas and unifying themes. They raise the composition of patterns to a higher semantic abstraction, enabling developers to systematically foresee how the consequences of one pattern map directly to the context of another.

Challenges and Pitfalls

Despite their power, pattern compounds introduce distinct architectural and cognitive challenges:

• Mixed Concerns: Because pattern compounds superimpose overlapping roles, a single class might juggle three distinct concerns: its core domain functionality, its responsibility in the first pattern, and its responsibility in the second. This can severely overload a class and muddle its primary responsibility.
• Obscured Foundations: Tightly compounding patterns can make it much harder for incoming developers to visually identify the individual, foundational patterns at play.
• Naming Limitations: Accurately naming a class to reflect its domain purpose alongside multiple pattern roles (e.g., a “PlayerObserver”) quickly becomes unmanageable, forcing teams to rely heavily on external documentation to explain the architecture.
• The Over-Engineering Trap: As with any design abstraction, possessing the “hammer” of a pattern compound does not make every problem a nail. Developers must constantly evaluate whether the resulting architectural complexity is truly justified by the context.

Advanced Concepts

Patterns Within Patterns: Core Principles

When analyzing various design patterns, you will begin to notice recurring micro-architectures. Design patterns are often built upon fundamental software engineering principles:

• Delegation over Inheritance: Subclassing can lead to rigid designs and code duplication (e.g., trying to create an inheritance tree for cars that can be electric, gas, hybrid, and also either drive or fly). Patterns like Strategy, State, and Bridge solve this by extracting varying behaviors into separate classes and delegating responsibilities to them.
• Polymorphism over Conditions: Patterns frequently replace complex if/else or switch statements with polymorphic objects. For instance, instead of conditional logic checking the state of an algorithm, the Strategy pattern uses interchangeable objects to represent different execution paths.
• Additional Layers of Indirection: To reduce strong coupling between interacting components, patterns like the Mediator or Facade introduce an intermediate object to handle communication. While this centralizes logic and improves changeability, it can create long traces of method calls that are harder to debug.

Domain-Specific and Application-Specific Patterns

The Gang of Four patterns are generic to object-oriented programming, but patterns exist at all levels.

• Domain-Specific Patterns: Certain industries (like Game Development, Android Apps, or Security) have their own highly tailored patterns. Because these patterns make assumptions about a specific domain, they generally carry fewer negative consequences within their niche, but they require the team to actually possess domain expertise.
• Application-Specific Patterns: Every distinct software project will eventually develop its own localized patterns—agreed-upon conventions and structures unique to that team. Identifying and documenting these implicit patterns is one of the most critical steps when a new developer joins an existing codebase, as it massively improves program comprehension.

Conclusion

Design patterns are the foundational building blocks of robust software architecture. However, they are a substitute for neither domain expertise nor critical thought. The mark of an expert engineer is not knowing how to implement every pattern, but possessing the wisdom to evaluate trade-offs, carefully observe the context, and know exactly when the simplest code is actually the smartest design.

Observer

---

Problem 

In software design, you frequently encounter situations where one object’s state changes, and several other objects need to be notified of this change so they can update themselves accordingly.

If the dependent objects constantly check the core object for changes (polling), it wastes valuable CPU cycles and resources. Conversely, if the core object is hard-coded to directly update all its dependent objects, the classes become tightly coupled. Every time you need to add or remove a dependent object, you have to modify the core object’s code, violating the Open/Closed Principle.

The core problem is: How can a one-to-many dependency between objects be maintained efficiently without making the objects tightly coupled?

Context

The Observer pattern is highly applicable in scenarios requiring distributed event handling systems or highly decoupled architectures. Common contexts include:

• User Interfaces (GUI): A classic example is the Model-View-Controller (MVC) architecture. When the underlying data (Model) changes, multiple UI components (Views) like charts, tables, or text fields must update simultaneously to reflect the new data.

• Event Management Systems: Applications that rely on events—such as user button clicks, incoming network requests, or file system changes—where an unknown number of listeners might want to react to a single event.

• Social Media/News Feeds: A system where users (observers) follow a specific creator (subject) and need to be notified instantly when new content is posted.

Solution

The Observer design pattern solves this by establishing a one-to-many subscription mechanism.

It introduces two main roles: the Subject (the object sending updates after it has changed) and the Observer (the object listening to the updates of Subjects).

Instead of objects polling the Subject or the Subject being hard-wired to specific objects, the Subject maintains a dynamic list of Observers. It provides an interface for Observers to attach and detach themselves at runtime. When the Subject’s state changes, it iterates through its list of attached Observers and calls a specific notification method (e.g., update()) defined in the Observer interface.

This creates a loosely coupled system: the Subject only knows that its Observers implement a specific interface, not their concrete implementation details.

Design Decisions

Push vs. Pull Model:

Push Model: The Subject sends the detailed state information to the Observer as arguments in the update() method, even if the Observer doesn’t need all data. This keeps the Observer completely decoupled from the Subject but can be inefficient if large data is passed unnecessarily.

Pull Model: The Subject sends a minimal notification, and the Observer is responsible for querying the Subject for the specific data it needs. This requires the Observer to have a reference back to the Subject, slightly increasing coupling, but it is often more efficient.

State

---

Problem 

The core problem the State pattern addresses is when an object’s behavior needs to change dramatically based on its internal state, and this leads to code that is complex, difficult to maintain, and hard to extend.

If you try to manage state changes using traditional methods, the class containing the state often becomes polluted with large, complex if/else or switch statements that check the current state and execute the appropriate behavior. This results in cluttered code and a violation of the Separation of Concerns design principle, since the code different states is mixed together and it is hard to see what the behavior of the class is in different states. This also violates the Open/Closed principle, since adding additional states is very hard and requires changes in many different places in the code. 

Context

An object’s behavior depends on its state, and it must change that behavior at runtime. You either have many states already or you might need to add more states later. 

Solution

Create an Abstract State class that defines the interface that all states have. The Context class should not know any state methods besides the methods in the Abstract State so that it is not tempted to implement any state-dependent behavior itself. For each state-dependent method (i.e., for each method that should be implemented differently depending on which state the Context is in) we should define one abstract method in the Abstract State class. 

Create Concrete State classes that inherit from the Abstract State and implement the remaining methods. 

The only interactions that should be allows are interactions between the Context and Concrete States. There are no interaction among Concrete States objects.

Design Decisions

How to let the state make operations on the context object?

The state-dependent behavior often needs to make changes to the Context. To implement this, the state object can either store a reference to the Context (usually implemented in the Abstract State class) or the context object is passed into the state with every call to a state-dependent method.  

How to represent a state in which the object is never doing anything (either at initialization time or as a “final” state)

Use the Null Object pattern to create a “null state”

Model-View-Controller (MVC)

---

The Model-View-Controller (MVC) architectural pattern decomposes an interactive application into three distinct components: a model that encapsulates the core application data and business logic, a view that renders this information to the user, and a controller that translates user inputs into corresponding state updates.

Problem 

User interface software is typically the most frequently modified portion of an interactive application. As systems evolve, menus are reorganized, graphical presentations change, and customers often demand to look at the same underlying data from multiple perspectives—such as simultaneously viewing a spreadsheet, a bar graph, and a pie chart. All of these representations must immediately and consistently reflect the current state of the data. A core architectural challenge thus arises: How can multiple, simultanous user interface functionality be kept completely separate from application functionality while remaining highly responsive to user inputs and underlying data changes? Furthermore, porting an application to another platform with a radically different “look and feel” standard (or simply upgrading windowing systems) should absolutely not require modifications to the core computational logic of the application.

Context

The MVC pattern is applicable when developing software that features a graphical user interface, specifically interactive systems where the application data must be viewed in multiple, flexible ways at the same time. It is used when an application’s domain logic is stable, but its presentation and user interaction requirements are subject to frequent changes or platform-specific implementations.

Solution

To resolve these forces, the MVC pattern divides an interactive application into three distinct logical areas: processing, output, and input.

• The Model: The model encapsulates the application’s state, core data, and domain-specific functionality. It represents the underlying application domain and remains completely independent of any specific output representations or input behaviors. The model provides methods for other components to access its data, but it is entirely blind to the visual interfaces that depict it.
• The View: The view component defines and manages how data is presented to the user. A view obtains the necessary data directly from the model and renders it on the screen. A single model can have multiple distinct views associated with it.
• The Controller: The controller manages user interaction. It receives inputs from the user—such as mouse movements, button clicks, or keyboard strokes—and translates these events into specific service requests sent to the model or instructions for the view.

To maintain consistency without introducing tight coupling, MVC relies heavily on a change-propagation mechanism. The components interact through an orchestration of lower-level design patterns, making MVC a true “compound pattern”.

• First, the relationship between the Model and the View utilizes the Observer pattern. The model acts as the subject, and the views (and sometimes controllers) register as Observers. When the model undergoes a state change, it broadcasts a notification, prompting the views to query the model for updated data and redraw themselves.
• Second, the relationship between the View and the Controller utilizes the Strategy pattern. The controller encapsulates the strategy for handling user input, allowing the view to delegate all input response behavior. This allows software engineers to easily swap controllers at runtime if different behavior is required (e.g., swapping a standard controller for a read-only controller).
• Third, the view often employs the Composite pattern to manage complex, nested user interface elements, such as windows containing panels, which in turn contain buttons.

Consequences

Applying the MVC pattern yields profound architectural advantages, but it also introduces notable liabilities that an engineer must carefully mitigate.

Benefits

• Multiple Synchronized Views: Because of the Observer-based change propagation, you can attach multiple varying views to the same model. When the model changes, all views remain perfectly synchronized and updated.
• Pluggable User Interfaces: The conceptual separation allows developers to easily exchange view and controller objects, even at runtime.
• Reusability and Portability: Because the model knows nothing about the views or controllers, the core domain logic can be reused across entirely different systems. Furthermore, porting the system to a new platform only requires rewriting the platform-dependent view and controller code, leaving the model untouched.

Liabilities

• Increased Complexity: The strict division of responsibilities requires designing and maintaining three distinct kinds of components and their interactions. For relatively simple user interfaces, the MVC pattern can be heavy-handed and over-engineered.
• Potential for Excessive Updates: Because changes to the model are blindly published to all subscribing views, minor data manipulations can trigger an excessive cascade of notifications, potentially causing severe performance bottlenecks.
• Inefficiency of Data Access: To preserve loose coupling, views must frequently query the model through its public interface to retrieve display data. If not carefully designed with data caching, this frequent polling can be highly inefficient.
• Tight Coupling Between View and Controller: While the model is isolated, the view and its corresponding controller are often intimately connected. A view rarely exists without its specific controller, which hinders their individual reuse.

Sample Code

This sample code shows how MVC could be implemented in Python:

# ==========================================
# 0. OBSERVER PATTERN BASE CLASSES
# ==========================================
class Subject:
    """The 'Observable' - broadcasts changes."""
    def __init__(self):
        self._observers = []

    def attach(self, observer):
        if observer not in self._observers:
            self._observers.append(observer)

    def detach(self, observer):
        self._observers.remove(observer)

    def notify(self):
        """Alerts all observers that a change happened."""
        for observer in self._observers:
            observer.update(self)

class Observer:
    """The 'Watcher' - reacts to changes."""
    def update(self, subject):
        pass


# ==========================================
# 1. THE MODEL (The Subject)
# ==========================================
class TaskModel(Subject):
    def __init__(self):
        super().__init__() # Initialize the Subject part
        self.tasks = []

    def add_task(self, task):
        self.tasks.append(task)
        self.notify() 

    def get_tasks(self):
        return self.tasks


# ==========================================
# 2. THE VIEW (The Observer)
# ==========================================
class TaskView(Observer):
    def update(self, subject):
        # When notified, the view pulls the latest data directly from the model
        tasks = subject.get_tasks()
        self.show_tasks(tasks)

    def show_tasks(self, tasks):
        print("\n--- Live Auto-Updated List ---")
        for index, task in enumerate(tasks, start=1):
            print(f"{index}. {task}")
        print("------------------------------\n")


# ==========================================
# 3. THE CONTROLLER (The Middleman)
# ==========================================
class TaskController:
    def __init__(self, model):
        self.model = model

    def add_new_task(self, task):
        print(f"Controller: Adding task '{task}'...")
        # The controller only updates the model. It trusts the model to handle the rest.
        self.model.add_task(task)


# ==========================================
# HOW IT ALL WORKS TOGETHER
# ==========================================
if __name__ == "__main__":
    # 1. Initialize Model and View
    my_model = TaskModel()
    my_view = TaskView()
    
    # 2. Wire them up (The View subscribes to the Model)
    my_model.attach(my_view)

    # 3. Initialize Controller (Notice it only needs the Model now)
    app_controller = TaskController(my_model)

    # 4. Simulate user input. 
    # Watch how adding a task automatically triggers the View to print!
    app_controller.add_new_task("Learn the Observer pattern")
    app_controller.add_new_task("Combine Observer with MVC")

Design Principles

---

Information Hiding

Description

SOLID

Description

Information Hiding

---

In the realm of software engineering, few principles are as foundational or as frequently misunderstood as Information Hiding (IH). While often confused with simply making variables “private,” IH is a sophisticated strategy for managing the overwhelming complexity inherent in modern software systems.

Historical Context

To understand why we hide information, we must look back to the mid-1960s. During the Apollo missions, lead software engineer Margaret Hamilton noted that software complexity had already surpassed hardware complexity. By 1968, the industry reached a “Software Crisis” where projects were consistently over budget, behind schedule, and failing to meet specifications. In response, David Parnas published a landmark paper in 1972 proposing a new way to decompose systems. He argued that instead of breaking a program into steps (like a flowchart), engineers should identify “difficult design decisions” or “decisions likely to change” and encapsulate each one within its own module.

The Core Principle: Secrets and Interfaces

The Information Hiding principle states that design decisions likely to change independently should be the “secrets” of separate modules. A module is defined as an independent work unit—such as a function, class, directory, or library—that can be assigned to a single developer. Every module consists of two parts:

• The Interface (API): A stable contract that describes what the module does. It should only reveal assumptions that are unlikely to change.
• The Implementation: The “secret” code that describes how the module fulfills its contract. This part can be changed freely without affecting the rest of the system, provided the interface remains the same.

A classic real-world example is the power outlet. The interface is the standard two or three-prong socket. As a user, you do not need to know if the power is generated by solar, wind, or nuclear energy; you only care that it provides electricity. This allows the “implementation” (the power source) to change without requiring you to replace your appliances.

Common “Secrets” to Hide

Successful modularization requires identifying which details are volatile. Common secrets include:

• Data Structures: Whether data is stored in an array, a linked list, or a hash map.
• Data Storage: Whether information is stored on a local disk, in a SQL database, or in the cloud.
• Algorithms: The specific steps of a computation, such as using A* versus Dijkstra for pathfinding.
• External Dependencies: The specific libraries or frameworks used, such as choosing between Axios or Fetch for network requests.

SOLID

---

The SOLID principles are design principles for changeability in object-oriented systems.

Single Responsibility Principle

Open/Closed Principle

Liskov Substitution Principle

Interface Segregation Principle

Dependency Inversion Principle

Software Process

---

Agile

For decades, software development was dominated by the Waterfall model, a sequential process where each phase—requirements, design, implementation, verification, and maintenance—had to be completed entirely before the next began. This “Big Upfront Design” approach assumed that requirements were stable and that designers could predict every challenge before a single line of code was written. However, this led to significant industry frustrations: projects were frequently delayed, and because customer feedback arrived only at the very end of the multi-year cycle, teams often delivered products that no longer met the user’s changing needs.

Agile Manifesto

In 2001, a group of software experts met in Utah to address these failures, resulting in the Agile Manifesto. Rather than a rigid rulebook, the manifesto proposed a shift in values:

• Individuals and interactions over processes and tools
• Working software over comprehensive documentation
• Customer collaboration over contract negotiation
• Responding to change over following a plan While the authors acknowledged value in the items on the right, they insisted that the items on the left were more critical for success in complex environments.

Core Principles

The heart of Agility lies in iterative and incremental development. Instead of one long cycle, work is broken into short, time-boxed periods—often called Sprints—typically lasting one to four weeks. At the end of each sprint, the team delivers a “Working Increment” of the product, which is demonstrated to the customer to gather rapid feedback. This ensures the team is always building the “right” system and can pivot if requirements evolve. Key principles supporting this include:

• Customer Satisfaction: Delivering valuable software early and continuously.
• Simplicity: The art of maximizing the amount of work not done.
• Technical Excellence: Continuous attention to good design to enhance long-term agility.
• Self-Organizing Teams: Empowering developers to decide how to best organize their own work rather than acting as “coding monkeys”.

Common Agile Processes

The most common agile processes include:

• Scrum: The most popular framework using roles like Scrum Master, Product Owner, and Developers.
• Extreme Programming (XP): Focused on technical excellence through “extreme” versions of good practices, such as Test-Driven Development (TDD), Pair Programming, Continuous Integration, and Collective Code Ownership
• Lean Software Development: Derived from Toyota’s manufacturing principles, Lean focuses on eliminating waste

Scrum

---

0:00

--:--

While many organizations claim to be “Agile”, the vast majority (roughly 63%) implement the Scrum framework.

Scrum Theory

Scrum is a management framework built on the philosophy of Empiricism. This philosophy asserts that in complex environments like software development, we cannot rely on detailed upfront predictions. Instead, knowledge comes from experience, and decisions must be based on what is actually observed and measured in a “real” product.

To make empiricism actionable, Scrum rests on three core pillars:

• Transparency: Significant aspects of the process must be visible to everyone responsible for the outcome. “The work is on the wall”, meaning stakeholders and developers alike should see exactly where the project stands via artifacts like Kanban boards.
• Inspection: The team must frequently and diligently check their progress toward the Sprint Goal to detect undesirable variances.
• Adaptation: If inspection reveals that the process or product is unacceptable, the team must adjust immediately to minimize further issues. It is important to realize that Scrum is not a fixed process but one designed to be tailored to a team’s specific domain and needs.

Scrum Roles

0:00

--:--

Scrum defines three specific roles that are intentionally designed to exist in tension to ensure both speed and quality:

• The Product Owner (The Value Navigator): This role is responsible for maximizing the value of the product resulting from the team’s work. They “own” the product vision, prioritize the backlog, and typically communicate requirements through user stories.
• The Developers (The Builders): Developers in Scrum are meant to be cross-functional and self-organizing. This means they possess all the skills needed—UI, backend, testing—to create a usable increment without depending on outside teams. They are responsible for adhering to a Definition of Done to ensure internal quality.
• The Scrum Master (The Coach): Misunderstood as a “project manager”, the Scrum Master is actually a servant-leader. Their primary objective is to maximize team effectiveness by removing “impediments” (blockers like legal delays or missing licenses) and coaching the team on Scrum values.

Scrum Artifacts

Scrum manages work through three primary artifacts:

• Product Backlog: An emergent, ordered list of everything needed to improve the product.
• Sprint Backlog: A subset of items selected for the current iteration, coupled with an actionable plan for delivery.
• The Increment: A concrete, verified stepping stone toward the Product Goal. An increment is only “born” once a backlog item meets the team’s Definition of Done—a checklist of quality measures like functional testing, documentation, and performance benchmarks.

Scrum Events

The framework follows a specific rhythm of time-boxed events:

• The Sprint: A 1–4 week period of uninterrupted development.
• Sprint Planning: The entire team collaborates to define why the sprint is valuable (the goal), what can be done, and how it will be built.
• Daily Standup (Daily Scrum): A 15-minute sync where developers discuss what they did yesterday, what they will do today, and any obstacles in their way.
• Sprint Review: A working session at the end of the sprint where stakeholders provide feedback on the working increment. A good review includes live demos, not just slides.
• Sprint Retrospective: The team reflects on their process and identifies ways to increase future quality and effectiveness.

Scaling Scrum with SAFe

When a product is too massive for a single team of 7–10 people, organizations often use the Scaled Agile Framework (SAFe). SAFe introduces the Agile Release Train (ART)—a “team of teams” that synchronizes their sprints. It operates on Program Increments (PI), typically lasting 8–12 weeks, which align multiple teams toward quarterly goals. While SAFe provides predictability for Fortune 500 companies, critics sometimes call it “Scrum-but-for-managers” because it can reduce individual team autonomy through heavy planning requirements.

Scrum Quiz

Recalling what you just learned is the best way to form lasting memory. Use this quiz to test your understanding of the Scrum framework, roles, events, and principles.

A software development group realizes their newest feature is confusing users based on early behavioral data. They immediately halt their current plan to redesign the user interface. Which foundational philosophy of their framework does this best illustrate?

Correct Answer:

Explanation

Scrum is founded on empiricism, and adaptation requires adjusting the process or product as soon as possible if aspects deviate outside acceptable limits.

In an environment that prioritizes agility, the individuals actually building the product must possess a specific dynamic. Which description best captures how this group should operate?

Correct Answer:

Explanation

Scrum Teams are cross-functional (possessing all necessary skills) and self-managing (deciding internally who does what, when, and how).

The development group is completely blocked because they lack access to a third-party API required for their current iteration. Who is primarily responsible for facilitating the resolution of this organizational bottleneck?

Correct Answer:

Explanation

The Scrum Master serves the team by causing the removal of impediments to their progress.

To ensure the team is consistently tackling the most crucial problems first, someone must dictate the priority of upcoming work items. Who holds this responsibility?

Correct Answer:

Explanation

The Product Owner is the one who orders the work for a complex problem into a Product Backlog to maximize value.

What condition must be strictly satisfied before a newly developed feature is officially considered a completed, verifiable stepping stone toward the ultimate product vision?

Correct Answer:

Explanation

The Scrum Master helps the team focus on creating high-value Increments that specifically meet the Definition of Done (the quality measures checklist).

What is the primary objective of the Daily Scrum?

Correct Answer:

Explanation

The Daily Scrum focuses on progress toward the Sprint Goal and produces an actionable plan for the next day of work.

At the conclusion of a work cycle, the team gathers specifically to discuss how they can improve their internal collaboration and technical practices for the next cycle. Which event does this describe?

Correct Answer:

Explanation

The Sprint Retrospective is an event for the team to inspect their process, reflect on it, and adjust for the next Sprint to continuously improve.

When a massive enterprise needs to coordinate dozens of teams working on the same vast product, they might adopt a ‘team of teams’ approach. According to common critiques, what is a potential drawback of this heavily synchronized model?

Correct Answer:

Explanation

Common critiques of frameworks like SAFe suggest they can reduce individual team autonomy through heavy planning and synchronization requirements.

Quiz Complete!

Your Score: 0/8

Extreme Programming (XP)

---

Overview

Extreme Programming, or XP, emerged as one of the most influential Agile frameworks, originally proposed by software expert Kent Beck. Unlike traditional “Waterfall” models that rely on “Big Upfront Design” and assume stable requirements, XP is built for environments where requirements evolve rapidly as the customer interacts with the product. The core philosophy is to identify software engineering practices that work well and push them to their purest, most “extreme” form.

The primary objectives of XP are to maximize business value, embrace changing requirements even late in development, and minimize the inherent risks of software construction through short, feedback-driven cycles.

Applicability and Limitations

XP is specifically designed for small teams (ideally 4–10 people) located in a single workspace where working software is needed constantly. While it excels at responsiveness, it is often difficult to scale to massive organizations of thousands of people, and it may not be suitable for systems like spacecraft software where the cost of failure is absolute and working software cannot be “continuously” deployed in flight.

XP Practices

The success of XP relies on a set of loosely coupled practices that synergize to improve software quality and team responsiveness.

The Planning Game (and Planning Poker)

The goal of the Planning Game is to align business needs with technical capabilities. It involves two levels of planning:

• Release Planning: The customer presents user stories, and developers estimate the effort required. This allows the customer to prioritize features based on a balance of business value and technical cost.
• Iteration Planning: User stories are broken down into technical tasks for a short development cycle (usually 1–4 weeks).

To facilitate estimation, teams often use Planning Poker. Each member holds cards with Fibonacci numbers representing “story points”—imaginary units of effort. If estimates differ wildly, the team discusses the reasoning (e.g., a hidden complexity or a helpful library) until a consensus is reached.

Small Releases

XP teams maximize customer value by releasing working software early, often, and incrementally. This provides rapid feedback and reduces risk by validating real-world assumptions in short cycles rather than waiting years for a final delivery.

Test-Driven Development (TDD)

In XP, testing is not a final phase but a continuous activity. TDD follows a strict “Red-Green-Refactor” rhythm:

• Red: Write a tiny, failing test for a new requirement.
• Green: Write the simplest possible code to make that test pass, even taking shortcuts.
• Refactor: Clean the code and improve the design while ensuring the tests still pass.

TDD ensures high test coverage and results in “living documentation” that describes exactly what the code should do.

Pair Programming

Two developers work together on a single machine. One acts as the Driver (hands on the keyboard, focusing on local implementation), while the other is the Navigator (watching for bugs and thinking about the high-level architecture). Research suggests this improves product quality, reduces risk, and aids in knowledge management.

Continuous Integration (CI)

To avoid the “integration hell” that occurs when developers wait too long to merge their work, XP mandates integrating and testing the entire system multiple times a day. A key benchmark is the 10-minute build: if the build and test process takes longer than 10 minutes, the feedback loop becomes too slow.

Collective Code Ownership

In XP, there are no individual owners of modules; the entire team owns all the code. This increases the bus factor—the number of people who can disappear before the project stalls—and ensures that any team member can fix a bug or improve a module.

Coding Standards

To make collective ownership feasible, the team must adhere to strict coding standards so that the code looks unified, regardless of who wrote it. This reduces the cognitive load during code reviews and maintenance.

Critical Perspectives: Design vs. Agility

A common critique of XP is that focusing solely on implementing features can lead to a violation of the Information Hiding principle. Because TDD focuses on the immediate requirements of a single feature, developers may fail to step back and structure modules around design decisions likely to change.

To mitigate this, XP advocates for “Continuous attention to technical excellence”. While working software is the primary measure of progress, a team that ignores good design will eventually succumb to technical debt—short-term shortcuts that make future changes prohibitively expensive.

Testing

---

In our quest to construct high-quality software, testing stands as the most popular and essential quality assurance activity. While other techniques like static analysis, model checking, and code reviews are valuable, testing is often the primary pillar of industry-standard quality assurance.

Test Classifications

Regression Testing

As software evolves, we must ensure that new features don’t inadvertently break existing functionality. This is the purpose of regression testing—the repetition of previously executed test cases. In a modern agile environment, these are often automated within a Continuous Integration (CI) pipeline, running every time code is changed

Black-Box and White-Box

When we design tests, we usually adopt one of two mindsets. Black-box testing treats the system as a “black box” where the internal workings are invisible; tests are derived strictly from the requirements or specification to ensure they don’t overfit the implementation. In contrast, white-box testing requires the tester to be aware of the inner workings of the code, deriving tests directly from the implementation to ensure high code coverage.

The Testing Pyramid: Levels of Execution

A robust testing strategy requires a mix of tests at different levels of abstraction.

These levels include:

• Unit Testing: The execution of a complete class, routine, or small program in isolation.
• Component Testing: The execution of a class, package, or larger program element, often still in isolation.
• Integration Testing: The combined execution of multiple classes or packages to ensure they work correctly in collaboration.
• System Testing: The execution of the software in its final configuration, including all hardware and external software integrations.

Testability

Test Doubles

---

Test Stub

A Test Stub is an object that replaces a real component to allow a test to control the indirect inputs of the SUT. Indirect inputs are the values returned to the SUT by another component whose services the SUT uses, such as return values, updated parameters, or exceptions. By replacing the real DOC with a Test Stub, the test establishes a control point that forces the SUT down specific execution paths it might not otherwise take, thus helping engineers test unreachable code or unique edge cases. During the test setup phase, the Test Stub is configured to respond to calls from the SUT with highly specific values.

While Test Stubs perfectly address the injection of inputs, they inherently ignore the indirect outputs of the SUT. To observe outputs, we must shift to a different class of Test Doubles.

Test Spy

When the behavior of the SUT includes actions that cannot be observed through its public interface—such as sending a message on a network channel or writing a record to a database—we refer to these actions as indirect outputs. To verify these indirect outputs, we use a Test Spy. A Test Spy is a more capable version of a Test Stub that serves as an observation point by quietly recording all method calls made to it by the SUT during execution. Like a Test Stub, a Test Spy may need to provide values back to the SUT to allow execution to continue, but its defining characteristic is its ability to capture the SUT’s indirect outputs and save them for later verification by the test. The use of a Test Spy facilitates a technique called “Procedural Behavior Verification”. The testing lifecycle using a spy looks like this:

1. The test installs the Test Spy in place of the DOC.

2. The SUT is exercised.

3. The test retrieves the recorded information from the Test Spy (often via a Retrieval Interface).

4. The test uses standard assertion methods to compare the actual values passed to the spy against the expected values.

A software engineer should utilize a Test Spy when they want the assertions to remain clearly visible within the test method itself, or when they cannot predict the values of all attributes of the SUT’s interactions ahead of time. Because a Test Spy does not fail the test at the first deviation from expected behavior, it allows tests to gather more execution data and include highly detailed diagnostic information in assertion failure messages.

Mock Object

A Mock Object, like a Test Spy, acts as an observation point to verify the indirect outputs of the SUT. However, a Mock Object operates using a fundamentally different paradigm known as “Expected Behavior Specification”. Instead of waiting until after the SUT executes to verify the outputs procedurally, a Mock Object is configured before the SUT is exercised with the exact method calls and arguments it should expect to receive. The Mock Object essentially acts as an active verification engine during the execution phase. As the SUT executes and calls the Mock Object, the mock dynamically compares the actual arguments received against its programmed expectations. If an unexpected call occurs, or if the arguments do not match, the Mock Object fails the test immediately.

Quality Attributes

---

While functionality describes exactly what a software system does, quality attributes describe how well the system performs those functions. Quality attributes measure the overarching “goodness” of an architecture along specific dimensions, encompassing critical properties such as extensibility, availability, security, performance, robustness, interoperability, and testability.

Important quality attributes include:

• Interoperability: the degree to which two or more systems or components can usefully exchange meaningful information via interfaces in a particular context.

• Testability: degree to which a system or component can be tested via runtime observation, determining how hard it is to write effective tests for a piece of software.

The Architectural Foundation: “Load-Bearing Walls”

Quality attributes are often described as the load-bearing walls of a software system. Just as the structural integrity of a building depends on walls that cannot be easily moved once construction is finished, early architectural decisions strongly impact the possible qualities of a system. Because quality attributes are typically cross-cutting concerns spread throughout the codebase, they are extremely difficult to “add in later” if they were not considered early in the design process.

Categorizing Quality Attributes

Quality attributes can be broadly divided into two categories based on when they manifest and who they impact:

• Design-Time Attributes: These include qualities like extensibility, changeability, reusability, and testability. These attributes primarily impact developers and designers, and while the end-user may not see them directly, they determine how quickly and safely the system can evolve.
• Run-Time Attributes: these include qualities like performance, availability, and scalability. These attributes are experienced directly by the user while the program is executing.

Specifying Quality Requirements

To design a system effectively, quality requirements must be measurable and precise rather than broad or abstract. A high-quality specification requires two parts: a scenario and a metric.

• The Scenario: This describes the specific conditions or environment to which the system must respond, such as the arrival of a certain type of request or a specific environmental deviation.
• The Metric: This provides a concrete measure of “goodness”. These can be hard thresholds (e.g., “response time < 1s”) or soft goals (e.g., “minimize effort as much as possible”).

For example, a robust specification for a Mars rover would not just say it should be “robust,” but that it must “function normally and send back all information under extreme weather conditions”.

Trade-offs and Synergies

A fundamental reality of software design is that you cannot always maximize all quality attributes simultaneously; they frequently conflict with one another.

• Common Conflicts: Enhancing security through encryption often decreases performance due to the extra processing required. Similarly, ensuring high reliability (such as through TCP’s message acknowledgments) can reduce performance compared to faster but unreliable protocols like UDP.
• Synergies: In some cases, attributes support each other. High performance can improve usability by providing faster response times for interactive systems. Furthermore, testability and changeability often synergize, as modular designs that are easy to change also tend to be easier to isolate for testing.

Interoperability

---

Interoperability is defined as the degree to which two or more systems or components can usefully exchange meaningful information via interfaces in a particular context.

Motivation

In the modern software landscape, systems are rarely “islands”; they must interact with external services to function effectively

Interoperability is a fundamental business enabler that allows organizations to use existing services rather than reinventing the wheel. By interfacing with external providers, a system can leverage specialized functionality for email delivery, cloud storage, payment processing, analytics, and complex mapping services. Furthermore, interoperability increases the usability of services for the end-user; for instance, a patient can have their electronic medical records (EMR) seamlessly transferred between different hospitals and doctors, providing a level of care that would be impossible with fragmented data.

From a technical perspective, interoperability is the glue that supports cross-platform solutions. It simplifies communication between separately developed systems, such as mobile applications, Internet of Things (IoT) devices, and microservices architectures.

Specifying Interoperability Requirements

To design effectively for interoperability, requirements must be specified using two components: a scenario and a metric.

• The Scenario: This must describe the specific systems that should collaborate and the types of data they are expected to exchange.
• The Metric: The most common measure is the percentage of data exchanged correctly.

Syntactic vs Semantic Interoperability

To master interoperability, an engineer must distinguish between its two fundamental dimensions: syntactic and semantic. Syntactic interoperability is the ability to successfully exchange data structures. It relies on common data formats, such as XML, JSON, or YAML, and shared transport protocols, such as HTTP(S). When two systems can parse each other’s data packets and validate them against a schema, they have achieved syntactic interoperability.

However, a major lesson in software architecture is that syntactic interoperability is not enough. Semantic interoperability requires that the exchanged data be interpreted in exactly the same way by all participating systems. Without a shared interpretation, the system will fail even if the data is transmitted flawlessly. For example, if a client system sends a product price as a decimal value formatted perfectly in XML, but assumes the price excludes tax while the receiving server assumes the price includes tax, the resulting discrepancy represents a severe semantic failure. An even more catastrophic example occurred with the Mars Climate Orbiter, where a spacecraft was lost because one component sent thrust commands in US customary units (pounds of force) while the receiving interface expected Standard International units (Newtons).

To achieve true semantic interoperability, engineers must rigorously define the semantics of shared data. This is done by documenting the interface with a semantic view that details the purpose of the actions, expected coordinate systems, units of measurement, side-effects, and error-handling conditions. Furthermore, systems should rely on shared dictionaries, standardized terminologies.

Architectural Tactics and Patterns

When systems must interact but possess incompatible interfaces, the Adapter design pattern is the primary solution. An adapter component acts as a translator, sitting between two systems to convert data formats (syntactic translation) or map different meanings and units (semantic translation). This approach allows the systems to interoperate without requiring changes to their core business logic.

In modern microservices architectures, interoperability is managed through Bounded Contexts. Each service handles its own data model for an entity, and interfaces are kept minimal—often sharing only a unique identifier like a User ID—to separate concerns and reduce the complexity of interactions.

Trade-offs

Interoperability often conflicts with changeability. Standardized interfaces are inherently difficult to update because a change to the interface cannot be localized to a single system; it requires all participating systems to update their implementations simultaneously.

The GDS case study highlights this dilemma. Because the GDS interface is highly standardized, it struggled to adapt to the business model of Southwest Airlines, which does not use traditional seat assignments. Updating the GDS standard to support Southwest would have required every booking system and airline in the world to change their software, creating a massive implementation hurdle.

“Practical Interoperability”

In a real-world setting, a design for interoperability is evaluated based on its likelihood of adoption, which involves two conflicting measures:

1. Implementation Effort: The more complex an interface is, the less likely it is to be adopted due to the high cost of implementation across all systems.
2. Variability: An interface that supports a wide variety of use cases and potential extensions is more likely to be adopted.

Successful interoperable design requires finding the “sweet spot” where the interface provides enough variability to be useful while remaining simple enough to minimize adoption costs.

UML

---

More Notes (WIP):

• Sequence Diagrams
• State Machine Diagrams
• Class Diagrams

1. Classes, Interfaces, and Modifiers

This snippet demonstrates how to define an interface, a class, and use visibility modifiers (+, -, #, ~).

@startuml
interface "Drivable" <<interface>> {
  + startEngine(): void
  + stopEngine(): void
}

class "Car" {
  - make: String
  - model: String
  # year: int
  ~ packageLevelAttribute: String
  
  + startEngine(): void
  + getMake(): String
}
@enduml

---

2. Relationships

PlantUML uses different arrow styles to represent the various relationships. The direction of the arrow generally goes from the “child” or “part” to the “parent” or “whole.”

Generalization (Inheritance)

Use <|-- to draw a solid line with an empty, closed arrowhead.

@startuml
class "Vehicle" {
  + move(): void
}
class "Car"
class "Motorcycle"

Vehicle <|-- Car
Vehicle <|-- Motorcycle
@enduml

Interface Realization (Implementation)

Use <|.. to draw a dashed line with an empty, closed arrowhead.

@startuml
interface "Drivable"
class "Car"

Drivable <|.. Car
@enduml

Association and Multiplicities

Use -- for a standard solid line. You can add quotes around numbers at either end to define the multiplicities, and a colon followed by text to label the association.

@startuml
class "Teacher"
class "Course"
class "Student"

Teacher "1" -- "0..*" Course : teaches >
Course "1..*" -- "0..*" Student : enrolled in >
@enduml

Aggregation

Use o-- to draw a solid line with an empty diamond pointing to the “whole” class.

@startuml
class "Department"
class "Professor"

Department o-- Professor
@enduml

Composition

Use *-- to draw a solid line with a filled (black) diamond pointing to the “whole” class.

@startuml
class "House"
class "Room"

House *-- "1..*" Room : contains
@enduml

---

3. Putting It All Together: A Mini E-commerce Example

Here is a consolidated PlantUML diagram showing how these concepts interact in a simple system design.

@startuml
interface "PaymentMethod" <<interface>> {
  + pay(amount: double): boolean
}

class "CreditCard" {
  - cardNumber: String
  - expirationDate: String
  + pay(amount: double): boolean
}

class "Customer" {
  - name: String
  - email: String
  + placeOrder(): void
}

class "Order" {
  - orderId: int
  - totalAmount: double
}

class "OrderItem" {
  - productId: int
  - quantity: int
}

' Relationships
PaymentMethod <|.. CreditCard : realizes
Customer "1" -- "0..*" Order : places >
Order *-- "1..*" OrderItem : is composed of >
Customer "1" -- "0..*" PaymentMethod : uses >

@enduml

---

Would you like me to show you how to add more advanced PlantUML features, like notes, coloring, or packages to organize your classes?

Class Diagrams 

Class diagrams represent classes and their interactions.

Classes

Classes are displayed as rectangles with one to three different sections that are each separated by a horizontal line.

The top section is always the name of the class. If the class is abstract, the name is in italics. 

The middle section indicates attributes of the class (i.e., member variables). 

The bottom section should include all methods that are implemented in this class (i.e., for which the implementation of the class contains a method definition). 

Inheritance is visualized using an arrow with an empty triage pointing to the super class. 

Attributes and methods can be marked as public (+), private (-), or protected (#), to indicate the visibility. Hint: Avoid public attributes, as this leads to bad design. (Public means every class has access, private means only this class has access, protected means this class and its sub classes have access) 

When a class uses an association, the name and visibility of the attribute can be written either next to the association or in the attribute section, or both (but only if it is done consistently). Writing it on the Association is more common since it increases the readability of the diagram.

Please include types for arguments and a meaningful parameter name. Include return types in case the method returns something (e.g., + calculateTax(income: int): int) 

Interfaces

Interfaces are classes that do not have any method definitions and no attributes. Interfaces only contain method declarations. Interfaces are visualized using the <<interface>> stereotype

To realize an interface, use then arrow with an empty triage pointing to the interface and a dashed line.

Sequence Diagrams 

Sequence diagrams display the interaction between concrete objects (or component instances). 

They show one particular example of interactions (potentially with optional, alternative, or looped behavior when necessary). Sequence diagrams are not intended to show ALL possible behaviors since this would become very complex and then hard to understand.

Objects / component instances are displayed in rectangles with the label following this pattern: objectName: ClassName. If the name of the object is irrelevant, then you can just write : Classname. 

When showing interactions between objects then all arrows in the sequence diagram represent method calls being made between the two objects. So an arrow from the client object with the name handleInput to the state objects means that somewhere in the code of the class of which client is an instance of, there is a method call to the handleInput method on the object state. Important: These are interactions between particular objects, not just generally between classes. It’s always on concrete instance of this class. 

The names shown on the arrows have to be consistent with the method names shown in the class diagram, including the number or arguments, order of arguments, and types of arguments. Whenever an arrow with method x and arguments of type Y and Z are received by an object o, then either the class of which o is an instance of or one of its super classes needs to have an implementation of x(Y,Z).     

It is a modeling choice to decide whether you want to include concrete values (e.g., caclulateTax(1400)) or meaningful variable names (e.g., calculateTax(income)). If you reference a real variable that has been used before, please make sure to ensure it is the same one and it has the right type. 

State Machine Diagrams 

State machines model the transitions between different states. States are modeled either as oval, rectangles with rounded corners, or circles. 

Transitions follow the patter [condition] trigger / action. 

State machines always need an initial state but don’t always need a final state. 

---

References

1. (Amna and Poels 2022): Asma Rafiq Amna and Geert Poels (2022) “Ambiguity in user stories: A systematic literature review,” Information and Software Technology, 145, p. 106824.
2. (Beck and Andres 2004): Kent Beck and Cynthia Andres (2004) Extreme Programming Explained: Embrace Change. 2nd ed. Boston, MA: Addison-Wesley Professional.
3. (Cohn 2004): Mike Cohn (2004) User Stories Applied: For Agile Software Development. Addison-Wesley Professional.
4. (Goode and Rain 2014): Durham Goode and Rain (2014) “Scaling Mercurial at Facebook.” Engineering at Meta.
5. (Lauesen and Kuhail 2022): Soren Lauesen and Mohammad A. Kuhail (2022) “User Story Quality in Practice: A Case Study,” Software, 1, pp. 223–241.
6. (Lucassen et al. 2016): Gijs Lucassen, Fabiano Dalpiaz, Jan Martijn van der Werf, and Sjaak Brinkkemper (2016) “Improving agile requirements: the Quality User Story framework and tool,” Requirements Engineering, 21(3), pp. 383–403.
7. (Patton 2014): Jeff Patton (2014) User Story Mapping: Better Software Through Community, Discovery, and Learning. O’Reilly Media.
8. (Potvin and Levenberg 2016): Rachel Potvin and Josh Levenberg (2016) “Why Google Stores Billions of Lines of Code in a Single Repository,” Communications of the ACM, 59(7), pp. 78–87.
9. (Quattrocchi et al. 2025): Giovanni Quattrocchi, Liliana Pasquale, Paola Spoletini, and Luciano Baresi (2025) “Can LLMs Generate User Stories and Assess Their Quality?,” IEEE Transactions on Software Engineering.
10. (Santos et al. 2025): Reine Santos, Gabriel Freitas, Igor Steinmacher, Tayana Conte, Ana Carolina Oran, and Bruno Gadelha (2025) “User Stories: Does ChatGPT Do It Better?,” International Conference on Enterprise Information Systems (ICEIS). SciTePress.
11. (Sharma and Tripathi 2025): Amol Sharma and Anil Kumar Tripathi (2025) “Evaluating user story quality with LLMs: a comparative study,” Journal of Intelligent Information Systems, 63, pp. 1423–1451.
12. (Wake 2003): Bill Wake (2003) “INVEST in Good Stories: The Series.”