Taming the Terminal

Bart Busschots is the author of all of the written tutorials in the Taming the Terminal series so the lion’s share of the credit goes to him. Allison Sheridan is the student of the series asking the dumb questions, and she created the podcast. Steve Sheridan convinced Bart and Allison that instead of having the series buried inside the larger NosillaCast Podcast, that it should be a standalone podcast. He did all of the editing to pull out the audio for the 35 original episodes from the NosillaCast, top and tail with music, and pushed Bart and Allison to record the intros. Steve even created the Taming the Terminal logo.

Allison had a vision of Taming the Terminal becoming an eBook but had no idea how to accomplish this. Helma van der Linden figured out how to programmatically turn the original feature-rich set of HTML web pages into an ePub book as well as producing a PDF version, and even an HTML version. She managed the GitHub project and fixed the technical aspects of the book and kept Allison on task as she did the proofreading and editing of the entire book. Allison created the book cover as well.

In macOS Catalina, released after much of the tutorial content in this book was released, Apple replaced the default shell bash with the zsh shell. As a result you’ll notice the prompt change from $ to % partway through the book. There may be cases where the instructions given during the bash days might not work with today’s zsh.

To switch back to bash if you do run into problems, simply enter:

You’ll be shown this warning explaining how to switch your default interactive shell back to zsh, and can proceed with the lessons.

If you’d like to see these instalments in their web form, you can go to ttt.bartificer.net.

If you enjoy the format of this series, you might also enjoy the podcast and written tutorials for Bart and Allison’s next series, Programming By Stealth at pbs.bartificer.net.

Feedback on the book can be sent to allison@podfeet.com.

We hope you enjoy your journey with Taming the Terminal.

In this first segment, I just want to lay a very basic foundation. I plan to take things very slowly with this series, so I’m going to start the way I mean to continue. Let’s start with some history and some wider context.

Before the days of GUIs (Graphical User Interfaces), and even before the days of simple menu-driven non-graphical interfaces like the original Word Perfect on DOS, the way humans interacted with computers was through a “command shell”. Computers couldn’t (and still can’t) speak or interpret human languages properly, and humans find it very hard to speak in native computer languages like binary machine codes (though early programmers did actually have to do that). New languages were invented to help bridge the gap, and allow humans and computers to meet somewhere in the middle.

The really big problem is that computers have absolutely no intelligence, so they can’t deal with ambiguity at all. Command shells use commands that look Englishy, but they have very rigid structures (or grammars if you will) that remove all ambiguity. It’s this rigid structure that allows the automated translation from shell commands to binary machine code the computer can execute.

When using any command shell, the single most important thing to remember is that computers are absolutely stupid, so they will do EXACTLY what you tell them to, no matter how silly the command you give them. If you tell a computer to delete all the files on a hard drive, it will, because, well, that’s what you asked it to do! Another important effect of computers’ total lack of intelligence is that there is no such thing as “close enough” — if you give it a command that’s nearly valid, a computer can no more execute it than if you’d just mashed the keyboard with your face. Nearly right and absolute gibberish are just as unintelligible to a computer. You must be exact, and you must be explicit at all times.

No wonder we went on to invent the GUI, this command shell malarky sounds really complicated! There is no doubt that if the GUI hadn’t been invented the personal computer wouldn’t have taken off like it has. If it wasn’t for the GUI, there’s no way there would be more computers than people in my parents home (two of them, and including iPhones and tablets, 6 computers!) Even the nerdiest of nerds use GUI operating systems most of the time because they make a lot of things both easier and more pleasant. BUT — not everything.

We all know that a picture says a thousand words, but when you are using a GUI it’s the computer that is showing you a picture, all you get to do is make crude hand gestures at the computer, which I’d say is worth about a thousandth of a word — so, a single shell command can easily be worth a thousand clicks. This is why all desktop OSes still have command shells built in — not as their ONLY user interface like in times past, but as a window within the GUI environment that lets you communicate with the computer using the power of a command shell.

Even Microsoft understands the power of the command shell. DOS may be dead, but the new Windows Power Shell is giving Windows power users a new, more modern, and more powerful command shell than ever before. Windows 8 may have removed the Start menu, but Powershell is still there! All Linux and Unix distros have command shells, and OS X gives you access to an array of different Command Shells through Terminal.app.

Just like there is no one GUI interface, there is no one command shell. Also, just like most GUIs are at least somewhat similar to each other, they all use icons for example, most command shells are also quite similar, having a command prompt that accepts commands with arguments to supply input to the commands or alter their behaviour. OS X does not ship with one command shell, it ships with SIX (sh, bash, zsh, csh, tcsh, and ksh)!

You can see the list of available shells (and set your default shell) by opening System Preferences, going to the Users & Groups pref pane, unlocking it, then right-clicking on your username in the sidebar, and selecting Advanced Options ...:

The default shell on OS X is the Bourne Again Shell (Bash), so that’s the shell we’ll be using for the remainder of this series. If you’ve not altered the defaults, then whenever you open a Terminal window on OS X, what you’re presented with is a Bash command shell. Bash is an updated and improved version of the older Bourne Shell (sh), which was the default shell in Unix for many years. The original Bourne Shell dates all the way back to 1977 and is called after its creator, Stephen Bourne. The Bourne Again Shell is a ‘newer’ update to the Bourne Shell dating back to 1989, the name being a nerdy religious joke by its author, Brian Fox. The Bourne Again Shell was not the last in the line of shells tracing their origins to the Bourne Shell, there is also zsh which dates back to 1990, but despite being a more powerful shell, it hasn’t taken off like Bash has.

So, what does a shell do? Does it just let you enter a single command and then run it? Or is there more to it? Unsurprisingly, there’s a lot more to it! The shell does its own pre-processing before issuing a command for you, so a lot of things that we think of as being part of how a command works are actually features provided by the command shell. The best example is the almost ubiquitous * symbol. When you issue a command like chmod 755 *.php, the actual command is not given *.php as a single argument that it must then interpret. No, the * is interpreted and processed by the shell before being passed on to the chmod command. It’s the shell that goes and looks for all the files in the current folder that end in .php, and replaces the *.php bit of the command with a list of all the actual files that end in .php, and passes that list on to the chmod command.

As well as providing wildcard substitution (the * thing), almost all shells also provide ‘plumbing’ for routing command inputs and outputs between commands and files, the definition of variables to allow sequences of commands to be generalised in a more reusable way, simple programming constructs to enable conditional actions, looping, and the grouping of sequences of commands into named functions, and the execution of a sequence of commands inside a file (scripting), and much more. Different shells also provide their own custom features to help make life at the command prompt easier for users. My favourite is tab-completion which is the single best thing Bash has to offer over sh in my opinion. OS X also brings some unique features to the table, with superb integration between the GUI and the command shell through features like drag-and-drop support in Terminal.app and shell scripting support in Automator.app. Of all the OSes I’ve used, OS X is the one that makes it the easiest to integrate command-line programs into the GUI.

I’ll end today by explaining an important part of the Unix philosophy, a part that’s still very much alive and well within OS X today. Unix aims to provide a suite of many simple command-line tools that each do just one thing, but do it very well. Complex tasks can then be achieved by chaining these simple commands together using a powerful command shell. Each Unix/Linux command-line program can be seen as a lego brick — not all that exciting on their own, but using a bunch of them, you can build fantastic things! My hope for this series is to help readers and listeners like you develop the skills to build your own fantastic things to make your computing lives easier. Ultimately the goal is to help you create more and better things by using automation to free you from as many of your repetitive tasks as possible!

You’ll find Terminal.app in /Applications/Utilities. Unless you’ve changed some of the default settings (or are using a very old version of OS X), you will now see a white window that is running the bash command shell that looks something like this:

Let’s start at the very top of the window with its title bar. At the left of the title is a proxy icon representing the current directory for the current Bash command shell and beside it the name of that folder. (Note that directory is the Unix/Linux/DOS word for what OS X and Windows refers to as a folder.) Like Finder windows, Terminal sessions are always “in” a particular directory/folder. After the current directory will be a dash, followed by the name of the process currently running in the Terminal session (in our case, a bash shell). The current process is followed by another dash, and then the dimensions of the window in fixed-width characters.

Within the window itself you will likely see a line of output telling you when you last logged in, and from where (if it was on this computer it will say ttys followed by some number, if it was from another computer, it will give the computer’s hostname). This will be followed on the next line by the so-called command prompt, and then the input cursor.

Let’s have a closer look at the command prompt. As with almost everything in Linux/Unix, the prompt is entirely customisable, so although Bash is the default shell on lots of different operating systems, the default prompt on each of those systems can be different. Let’s first look at the default Bash command prompt on OS X:

On OS X, the prompt takes the following form:

hostname:current_directory username$

First, you have the hostname of the computer on which the command shell is running (defined in System Preferences  Sharing  Computer Name). This might seem superfluous, but it becomes exceptionally useful once you start using ssh to log in to other computers via the Terminal.

The hostname is followed by a : and then the command shell’s current directory (note that ~ is short-hand for “the current user’s home folder”, more on this next time).

The current directory is followed by a space, and then the Unix username of the user running the command shell (defined when you created your OS X account, defaults to your first name if available). Finally, there is a $ character (which changes to a # when you run bash as the root user). Again, this might not seem very useful at first, but there are many reasons you may want to switch your command shell to run as a different user from time to time, so it is also very useful information.

As an example of how the default shells differ on different operating systems, below is an example from a RedHat-style Linux distribution (CentOS in this case):

As you can see, it contains the same information, but arranged a little differently:

[username@hostname current_directory]$

Finally, Debian-style Linux distributions like Ubuntu use a different default prompt again, but also showing the same information:

username@hostname:current_directory$

Handy Tip: if you find the text in the Terminal window to small to read, you can make it bigger or smaller with ⌘++ or ⌘+-. This will affect just your current Terminal window. You can permanently change the default by editing the default profile in Terminal  Preferences  Profiles.

Now that we understand the different parts of our Terminal window, let’s have a look at the structure of the actual commands we will be typing at that cursor!

I want to start by stressing that the commands executed by a command shell are not determined by the command shell, but by the operating system. Regardless of whether you use Bash on OS X, or zsh on OS X, you will have to enter OS X commands. Similarly, if you use Bash on Linux, you will have to enter Linux commands. Thankfully Linux and Unix agree almost entirely on the structure of their basic commands, so with a very few (and very annoying) exceptions, you can use the same basic commands on any Linux or Unix distribution (remember that at its heart OS X is Free BSD Unix).

Commands take the form of the command itself optionally followed by a list of arguments separated by spaces, e.g.:

command argument_1 argument_2 …​ argument_n

Arguments are a mechanism for passing information to a command. Most commands need at least one argument to be able to perform their task, but some don’t. Both commands and arguments are case-sensitive, so beware your capitalisation!

For example, the cd (change directory) command takes one argument (a directory path):

In this example, the command is cd, and the one argument passed is /Users/Shared/.

Some commands don’t require any arguments at all, e.g. the pwd (present working directory) command:

It is up to each command to determine how it will process the arguments it is given. When the developer was creating the command he or she will have had to make decisions about what arguments are compulsory, what arguments are optional, and how to parse the list of arguments the command is given by the shell when being executed.

In theory, every developer could come up with their own mad complex scheme for parsing argument lists, but in reality most developers loath re-inventing the wheel (thank goodness), so a small number of standard libraries have come into use for parsing arguments. This means that many apps use very similar argument styles.

As well as accepting simple arguments like the cd command above, many apps accept specially formatted arguments referred to as flags. Flags are usually used to specify optional extra information, with information that is required taken as simple arguments. Flags are arguments (or pairs of arguments) that start with the - symbol.

The simplest kinds of flags are those that don’t take a value, they are specified using a single argument consisting of a - sign followed by a single letter. For example, the ls (list directory) command can accept the flag -l (long-form listing) as an argument. e.g.

The way the standard argument processing libraries work, flags can generally be specified in an arbitrary order. The ls command also accepts the flag -a (list all), so the following are both valid and equivalent:

and

The standard libraries also allow flags that don’t specify values to be compressed into a single argument like so:

Sometimes flags need to accept a value, in which case the flag stretches over two arguments which have to be contiguous. For example, the ssh (secure shell) command allows the port to be used for the connection to be specified with the -p flag, and the username to connect as with the -l flag, e.g.:

These single-letter flags work great for simple commands that don’t have too many options, but more complex commands often support many tens of optional flags. For that reason, another commonly used argument processing library came into use that accepts long-form flags that start with a -- instead of a single -. As well as allowing a command to support more flags, these longer form flags also allow values to be set within a single argument by using the = sign.

As an example, the mysql command (needs to be installed separately on OS X) allows the username and password to be used when making a database connection to be specified using long-form flags:

Many commands support both long and short form arguments, and they can be used together, e.g.:

So far we know that commands consist of a command optionally followed by a list of arguments separated by spaces, and that many Unix/Linux commands use similar schemes for processing arguments where arguments starting with - or -- are treated in a special way, and referred to as flags. That all seems very simple, but, there is one important complication that we have to address before finishing up for this segment, and that’s special characters.

Within Bash (and indeed every other command shell), there are some characters that have a special meaning, so they cannot be used in commands or arguments without signifying to the command shell in some way that is should interpret these symbols as literal symbols, and not as representations of some sort of special value or function.

The most obvious example from what we have learned today is the space character, it is used as the separator between commands and the argument list that follows, and within that argument list as the separator between individual arguments. What if we want to pass some text that contains a space to a command as an argument? This happens a lot because spaces are valid characters within file and folder names on Unix and Linux, and file and folder names are often passed as arguments.

As well as the space there are other symbols that have special meanings. I won’t explain what they mean today, but I will list them:

You have two choices for how you deal with these special characters when you need to include them within an argument, you can escape each individual special character within the argument, or you can quote the entire argument.

Escaping is easy, you simply prefix the special character in question with a \. If there are only one or two special characters in an argument this is the simplest and easiest solution. But, it can become tedious if there are many such special characters.

Let’s use the echo command to illustrate escaping. The echo command simply prints out the input it receives. The following example passes the phrase Hello World! to the echo command as a single argument. Note that this phrase contains two special characters that will need to be escaped, the space and the !:

If you don’t want to escape each special character in an argument, you can quote the argument by prepending and appending either a " or a ' symbol to it. There is a subtle difference between using ' or ".

When you quote with ' you are doing so-called full quoting, every special character can be used inside a full quote, but, it is impossible to use a ' character inside a fully quoted argument. For example:

When you quote with " on the other hand you are doing so-called partial quoting, which means you can use most special characters without escaping them, but not all. Partial quoting will become very important later when we start to use variables and things because the biggest difference between full and partial quoting is that you can’t use variable substitution with full quoting, but you can with partial quoting (don’t worry if that makes no sense at the moment, it will later in the series).

When using partial quoting you still have to escape the following special characters:

For example:

and:

There are a few other peculiar edge cases with partial quoting — for example, you can’t end a partial quote with a !, and you can’t quote just a * on its own (there may well be more edge cases I haven’t bumped into yet).

That’s where we’ll leave it for this segment. We’ve now familiarised ourselves with the OS X Terminal window, and we’ve described the anatomy of a Unix/Linux command. In the next segment, we’ll look at the Unix/Linux file system, and at some of the commands used to navigate around it.

Physical storage media are nothing more than a massive array of virtual pigeon holes, each of which can hold a single 1 or 0. All your information is stored by grouping together a whole bunch of these pigeon holes and giving that grouping of 1s and 0s some kind of name. Humans simply could not deal with remembering that the essay they were working on is stored in sectors 4 to 1024 on cylinder 213 on the disk connected to the first SATA channel on the motherboard. We need some kind of abstraction to bring order to the chaos and to allow us to organise our data in a human-friendly way.

A good analogy would be a pre-computer office where the unit of storage was a single sheet of paper. Without some sort of logical system for organising all this paper, no one would ever be able to find anything, hence, in the real world, we developed ‘systems’ for ‘filing’ paper. Or, to put it another way, we invented physical filesystems, based around different ways of grouping and naming the pieces of paper. If a single document contained so much information that it ran over multiple pages, those piece of paper were physically attached to each other using a tie, a paperclip, or a staple. To be able to recognise a given document at a glance, documents were given titles. Related documents were then put together into holders that, for some reason were generally green, and those holders were then placed into cabinets with rails designed to hold the green holders in an organised way. I.e. we had filing cabinets containing folders which contained files. The exact organisation of the files and folders were up to the individual clerks who managed the data and were dependant on the kind of data being stored. Doctors tend to store files alphabetically by surname, while libraries love the Dewey Decimal system.

When it comes to computers, the job of bringing order to the chaos falls to our operating systems. We call the many different schemes that have been devised to provide that order, filesystems. Some filesystems are media dependent, while others are operating system dependent. E.g. the Joliet file system is used on CDs and DVDs regardless of OS, while FAT and NTFS are Windows filesystems, EXT is a family of Linux file systems, and HFS+ is a Mac file system.

There is an infinite number of possible ways computer scientists could have chosen to bring order to the chaos of bits on our various media, but, as is often the case, a single real-world analogy was settled on by just about all operating system authors. Whether you use Linux, Windows, or OS X, you live in a world of filesystems that contain folders (AKA directories) that contain files and folders. Each folder and file in this recursive hierarchical structure has a name, so it allows us humans, to keep our digital documents organised in a way that we can get our heads around. Although all our modern filesystems have their own little quirks under the hood, they all share the same simple architecture, your data goes in files which go in folders which can go in other folders which eventually go into file systems.

You can have lots of files with the same name in this kind of file system, but, you can never have two items with the same name in the same folder. This means that each file and folder can be uniquely identified by listing all the folders you pass to get from the so-called ‘root’ of the filesystem as far as the file or folder you are describing. This is what we call the full path to a file or folder. Where operating systems diverge is in their choice of separator, and in the rules they impose on file and folder names. On all modern consumer operating systems, we write file paths as a list of folder and file names separated by some character, called the ‘path separator’. DOS and Windows use \ (the backslash) as the path separator, on classic MacOS it was : (old OS X apps that use Carbon instead of Cocoa still use : when showing file paths, iTunes did this up until the recent version 11!), and on Linux/Unix (including OS X), / (the forward-slash) is used.

A single floppy disk and a single CD or DVD contain a single file system to hold all the data on a given disk, but that’s not true for hard drives, thumb drives, or networks. When formatting our hard drives or thumb drives we can choose to sub-divide a single physical device into multiple so-called partitions, each of which will then contain a single filesystem.

You’ve probably guessed by now that on our modern computers we tend to have more than one filesystem. Even if we only have one internal hard disk in our computer that has been formatted to have only a single partition, every CD, DVD, or thumb drive we own contains a filesystem, and, each network share we connect to is seen by our OS as yet another file system. In fact, we can even choose to store an entire filesystem (even an encrypted one) in a single file, e.g. DMG files, or TrueCrypt vaults.

So, all operating systems have to merge lots of file systems into a single over-arching namespace for their users. Or, put another way, even if two files have identical paths on two filesystems mounted by the OS at the same time, there has to be a way to distinguish them from each other. There are lots of different ways you could combine multiple filesystems into a single unified namespace, and this is where the DOS/Windows designers parted ways with the Unix/Linux folks. Microsoft combines multiple file systems together in a very different way to Unix/Linux/OS X.

Let’s start by looking at the approach Microsoft chose. In DOS, and later Windows, each filesystem is presented to the user as a separate island of data named with a single letter, referred to as a drive letter. This approach has an obvious limitation, you can only have 26 file systems in use at any one time! For historical reasons, A:\ and B:\ were reserved for floppy drives, so, the first partition on the hard drive connected to the first IDE/SATA bus on the motherboard is given the drive letter C:\, the second one D:\ and so on. Whenever you plug in a USB thumb drive or a memory card from a camera it gets ‘mounted’ on the next free drive letter. Network shares also get mounted to drive letters.

Just like files and folders, filesystems themselves have names too, often referred to as Volume Names. Windows makes very little use of these volume names though, they don’t show up in file paths, but, Windows Explorer will show them in some situations to help you figure out which of your USB hard drives ended up as K:\ today.

An analogy you can use for file systems is that of a tree. The trunk of the tree is the base of the file system, each branch is a folder, and each leaf a file. Branches ‘contain’ branches and leaves, just like folders contain folders and files. If you bring that analogy to Microsoft’s way of handling filesystems, then the global namespace is not a single tree, but a small copse of between 1 and 26 trees, each a separate entity, and each named with a single letter.

If we continue this analogy, Linux/Unix doesn’t plant a little copse of separate trees like DOS/Windows, instead, they construct one massive Franken-tree by grafting smaller trees onto the branches of a single all-containing master tree. When Linux/Unix boots, one filesystem is considered to be the main filesystem and used as the master file system into which other file systems get inserted as folders. In OS X parlance, we call the partition containing this master file system the System Disk. Because the system disk becomes the root of the entire filesystem it is gets assigned the shortest possible file path, /.

If your system disk’s file system contained just two folders, folder_1 and folder_2, they would get the file paths /folder_1/ and /folder_2/ in Linux/Unix/OS X. The Unix/Linux command mount can then be used to ‘graft’ filesystems into the master filesystem using any empty folder as the so-called mount point.

On Linux systems, it’s common practice to keep home folders on a separate partition, and to then mount that separate partition’s file system as /home/. This means that the main filesystem has an empty folder in it called home and that as the computer boots, the OS mounts a specified partition’s file system into that folder. A folder at the root of the that partition’s file system called just allison would then become /home/allison/.

On regular Linux/Unix distributions the file /etc/fstab (file system table) tells the OS what filesystems to mount to what mount points. A basic version of this file will be created by the installer, but in the past, whenever you added a new disk to a Linux/Unix system you had to manually edit this file. Thankfully, we now have something called automount to automatically mount any readable filesystems to a predefined location on the filesystem when they are connected.

The exact details will change from OS to OS, but on Ubuntu, the folder /media/ is used to hold mount points for any file system you connect to the computer. Unlike Windows, most Linux/Unix systems make use of filesystems’ volume names and use them to give the mount points sensible names, rather than random letters. If I connect a USB drive containing a single partition with a filesystem with the volume name Allison_Pen_Drive, Ubuntu will automatically mount the filesystem on that thumb drive when you plug it in, using the mount point /media/Allison_Pen_Drive/. If that pen drive contained a single folder called myFolder containing a single file called myFile.txt, then myFile.txt would be added to the filesystem as /media/Allison_Pen_Drive/myFolder/myFile.txt.

Having the ability to mount any filesystem as any folder within a single master filesystem allows you to easily separate different parts of your OS across different drives. This is very useful if you are a Linux/Unix sysadmin or power user, but it can really confuse regular users. Because of this, OS X took a simpler route. There is no /etc/fstab by default (though if you create one OS X will correctly execute it as it boots). The OS X installer does not allow you to split OS X over multiple partitions. Everything belonging to the OS X system, including all the users home folders, are installed on a single partition, the system disk, and all other file systems, be they internal, external, network, or disk images, get automatically mounted in /Volumes/ as folders named for the file systems’ volume labels.

Going back to our imaginary thumb drive called Allison_Pen_Drive (which Ubuntu would mount as /media/Allison_Pen_Drive/), OS X will mount that as /Volumes/Allison_Pen_Drive/ when you plug it in. If you had a second partition, or a second internal drive, called, say, Fatso (a little in-joke for Allison), OS X would mount that as /Volumes/Fatso/. Likewise, if you double-clicked on a DMG file you downloaded from the net, say with the Adium installer, OS X would mount that as something like /Volumes/Adium/ until you eject the DMG. The ‘disks’ listed in the Finder sidebar in the section headed Devices are just links to the contents of /Volumes/. You can see this for yourself by opening a Finder Window and either hitting the key-combo ⌘+shift+g, or navigating to Go  Go To Folder …​ in the menubar to bring up the Go To Folder text box, and then typing the path /Volumes and hitting return.

OS X’s greatly simplified handling of mount points definitely makes OS X less confusing, but, the simplicity comes at a price. If you DO want to do more complicated things like having your home folders on a separate partition, you are stepping outside of what Apple considers the norm, and into a world of pain. On Linux/Unix separating out home folders is trivial, on OS X it’s a mine-field!

We’ll leave it here, for now, next time we’ll learn how to navigate around a Unix/Linux/OS X filesystem.

Two instalments ago we learned that, just like a Finder window, a command prompt is ‘in’ a single folder/directory at any time. That folder is known as the current working directory or the present working directory. Although the default Bash command prompt on OS X will show us the name of our current folder, it doesn’t show us the full path. To see the full path of the folder you are currently in, you need the pwd (present working directory) command. This is a very simple command that doesn’t need any arguments.

When you open an OS X Terminal, by default your current working directory will be your home directory, so, if you open a Terminal now and type just pwd you’ll see something like:

Knowing where you are is one thing, the next thing you might want to do is look around to see what’s around you, and for that, you’ll need the ls (list) command.

If you run the command without any arguments you’ll see a listing of all the visible files and folders in the current directory. On OS X, this default view is annoying in two ways. Firstly, you’ll see your files and folders spread out over multiple columns, so scanning for a file name alphabetically becomes annoyingly confusing, especially if the list scrolls. Secondly, on OS X (though not on most Linux distros), you won’t be able to tell what is a file and what is a folder at a glance, you’ll just see names, which is really dumb (even DOS does a better job by default!).

You can force ls to display the contents of a folder in a single column in two ways. You can either use the -l flag to request a long-form listing, showing lots of metadata along with each file name, or, you can use the -1 flag to specify that you just want the names but in a single column. For now, most of the metadata shown in the long-form listing is just confusing garbage, so you are probably better off using -1. If you do want to use the long-form listing, I suggest adding the -h flag to convert the file size column to human-readable file sizes like 100K, 5M, and 64G. I’ve trained myself to always use ls -lh and never to use just ls -l.

You have two options for making it easy to distinguish files from folders in the output from ls on OS X. You can either use the -F flag to append a / to the end of every folder’s name, or, the -G flag to use colour outputs (folders will be in blue). The -F flag will work on Linux and Unix, but the -G flag is a BSD Unix thing and doesn’t work on Linux. Linux users need to use the more logical --color instead.

I said that ls shows you all the visible files in your current directory, what if you want to see all the files, including hidden files? Simple, just use the -a flag.

Finally, before we move away from ls (for now), I should mention that you can use ls to show you the content of any folder, not just the content of your current folder. To show the content of any folder or folders, use the path(s) as regular arguments to ls. E.g. to see what is in your system-level library folder you can run:

Now that we can see where we are with pwd, and look around us with ls, the next obvious step is moving around the filesystem, but, we need to take a small detour before we’re ready to talk about that.

In the last instalment, we talked about file paths like the imaginary file on Allison’s thumb drive with the path /Volumes/Allison_Pen_Drive/myFolder/myFile.txt. That type of path is called an absolute path and is one of two types of path you can use as arguments to Linux/Unix commands.

Absolute paths (AKA full paths) are like full addresses, or phone numbers starting with the + symbol, they describe the location of a file without reference to anything but the root of the filesystem. They will work no matter what your present working directory is. When you need to be explicit, like say when you’re doing shell scripting, you probably want to use absolute paths, but, they can be tediously long and unwieldy.

This is where relative paths come in, relative paths don’t describe where a file or folder is relative to the root of the file system, but, instead, relative to your present working directory. If you are stopped for directions and someone wants to know where the nearest gas station is, you don’t give them the full address, you give them directions relative to where they are at that moment. Similarly, if you want to phone someone in the same street you don’t dial + then the country code then the area code then their number, you just dial the number because, like your command shell is in a current working directory, your telephone is in an area code.

With phone numbers, you can tell whether something is a relative or an absolute phone number by whether or not it starts with a +. With Unix/Linux paths the magic character is /. Any path that starts with a / will be interpreted as an absolute path by the OS, and conversely, any path that does not begin with a / will be interpreted as a relative path.

If you are in your home folder, you can describe the relative path to your iTunes library file as Music/iTunes/iTunes\ Library.xml (note the backslash to escape the space in the path). That means that your home folder contains a folder called Music, which contains a folder called iTunes, which contains a file called iTunes Library.xml.

Describing relative paths to items deeper in the file system hierarchy from you is easy, but what if you need to go the other way, not to folders contained in your current folder, but instead to the folders that contain your current folder? Have another look at the output of ls -aG1 in any folder. What are the top two entries? I don’t have to know what folder you are in to know the answer, the first entry will be a folder called ., and the second entry will be a folder called ... These are the key to allowing relative paths that go up the chain.

The folder . is a hard link to the current folder. If you are in your home folder, ls ./Documents and ls Documents will do the same thing, show you the contents of a folder called Documents in your current folder. This seems pointless, but trust me, it will prove to be important and useful in the future.

For now, the more interesting folder is .., which is a hard link to the folder that contains the current folder. I.e. it allows you to specify relative paths that move back towards / from where you are.

In OS X, home directories are stored in a folder called /Users. As well as one folder for each user (named for the user), /Users also contains a folder called Shared which is accessible by every user to facilitate easy local file sharing. Regardless of your username, the relative path from your home folder to /Users/Shared is always ../Shared (unless you moved your home folder to a non-standard location of course). .. means go back one level, then move forward to Shared. You can go back as many levels as you want until you hit / (where .. is hard-linked to itself), e.g. the relative path from your home folder to / is ../../.

Finally, the Bash shell (and all other common Unix/Linux shells) provides one other very special type of path, home folders. We have mentioned in passing in previous instalments that ~ means ‘your home directory’. No matter where on the filesystem you are, ~/Music/iTunes/iTunes\ Library.xml is always a relative path to your iTunes library file. But, the ~ character does a little more than that, it can take you to ANY user’s home folder simply by putting their username after the ~. Imagine Allison & Steve share a computer. Allison’s username is allison, and Steve’s is steve. Allison and Steve can each access their own iTunes libraries at ~/Music/iTunes/iTunes\ Library.xml, but, Allison can also access Steve’s at ~steve/Music/iTunes/iTunes\ Library.xml, and likewise, Steve can access Allison’s at ~allison/Music/iTunes/iTunes\ Library.xml (all assuming the file permissions we are ignoring for now, are set appropriately of course).

So — now that we understand that we can have absolute or relative paths, we are finally ready to start navigating the file system by changing our current directory. The command to do this is cd (change directory).

Firstly, if you ever get lost and you want to get straight back to your home directory, just run the cd command with no arguments and it will take you home!

Generally, though we want to use the cd command to navigate to a particular folder, to do that, simply use either the absolute or relative path to the folder you want to navigate to as the only argument to the cd command, and assuming the path you entered is free of typos, off you’ll go!

Finally, for this instalment, I just want to mention one other nice trick the cd command has up its sleeve, it has a (very short) memory. If you type cd - you will go back to where ever you were before you last used cd.

As an example, let’s say you spent ages navigating a very complex file system and are now 20 folders deep. You’ve forgotten how you got there, but you’ve finally found that obscure system file you need to edit to make some app do some non-standard thing. Then, you make a boo boo, and you accidentally type just cd on its own, all of a sudden, you are back in your home folder. Don’t panic, you won’t have to find that complicated path again, just type cd - and you’ll be right back where you were before your rubber-fingered the cd command!

That’s where we’ll leave things for this instalment. We now understand the structure of our file systems and how to navigate around them. Next time we’ll dive head-long into these file permissions we’ve been ignoring for the last two instalments.

For any Windows users out there, the DOS equivalents are as follows:

Let’s start with some context. Just like every command shell has a present working directory, every process on a Linux/Unix system is also owned by a user, including shell processes. So, when you execute a command in a command shell, that process has a file system location associated with it and a username. By default your shell will be running as the user you logged into your computer as, though you can become a different user if and when you need to (more on that in future instalments). You can see which user you are running as with the very intuitive command:

Secondly, users on Unix/Linux systems can be members of one or more groups. On OS X there are a number of system groups to which your user account may belong, including one called staff to which all admin users belong. You can see what groups you belong to with the command:

(You can even see the groups any username belongs to by adding the username as an argument.)

On older versions of OS X creating your own custom groups was hard. Thankfully Apple has addressed this shortcoming in more recent versions of the OS, and you can now create and manage your own custom groups in the Users & Groups preference pane (click the + button and choose group as the user type, then use the radio buttons to add or remove people from the group).

Unix/Linux file systems like EXT and HFS+ store metadata about each file and folder as part of that file or folder’s entry in the file system. Some of that metadata is purely informational, things like the date the file was created, and the date it was last modified, but that metadata also includes ownership information and a so-called Unix File Permission Mask.

There are two pieces of ownership information stored about every file and folder: a UID, and a GID. What this means is that every file and folder belongs to one user and one group.

In the standard Linux/Unix file permissions model there are only three permissions that can be granted on a file or folder:

All permutations of these three permissions are possible on any file, even if some of them are counter-intuitive and rarely needed.

The Unix file Permission Mask ties all these concepts together. The combination of the context of the executing process and the metadata in a file or folder determines the permissions that apply. You can use the ls -l command to see the ownership information and file permission mask associated with any file or folder.

The hard part is interpreting the meaning of the file permission mask.

On standard Unix/Linux systems this mask contains ten characters, though on OS X it can contain an optional 11th or even 12th character appended to the end of the mask (we’ll be ignoring these for this instalment).

The first character specifies the ‘type’ of the file:

The remaining nine characters represent three sets of read, write, and execute permissions (rwx), specified in that order. If a permission is present then it is represented by an r, w, or x, and if it’s not present, it’s represented by a -.

The first group of three permission characters are the permissions granted to the user who owns the file, the second three are the permissions granted to all users who are members of the group that owns the file, and the last three are the permissions granted to everyone, regardless of username or group membership.

To figure out what permissions you have on a file you need to know the following things:

When you try to read the contents of a file, your OS will figure out whether or not to grant you that access using the following algorithm:

Write and execute permissions are processed in exactly the same way.

When you see the output of ls -l, you need to mentally follow the same algorithm to figure out whether or not you have a given permission on a given file or folder. The three columns to look at are the mask, the file owner, and the file group.

We’ll stop here for now. In the next instalment, we will explain the meaning of the + and @ characters which can show up at the end of a file permission masks on OS X, and we’ll look at the commands for altering the permissions on a file or folder.

The command to change the user that owns one or more files or folders is chown (change owner). The command takes a minimum of two arguments, the username to change the ownership to, and one or more files or folders to modify. E.g.:

The command can also optionally take a -R flag to indicate that the changes should be applied ‘recursively’, that is that if the ownership of a folder is changed, the ownership of all files and folders contained within that folder should also be changed. The chown command is very picky about the placement of the flag though, it MUST come before any other arguments E.g.:

Similarly, the command to change the group that a file belongs to is chgrp (change group). It behaves in the same way as chown, and also supports the -R flag to recursively change the group. E.g.:

Finally, if you want to change both user and group ownership of files or folders at the same time, the chown command provides a handy shortcut. Instead of passing just a username as the first argument, you can pass a username and group name pair separated by a :, so the previous two examples can be rolled into the one example below:

The command to alter the permission mask, or file mode, is chmod (change mode). In many ways it’s similar to the chown and chgrp commands. It takes the same basic form, and supports the -R flag, however, the formatting of the first argument — the permission you want to set — can be very confusing.

The command actually supports two entirely different approaches to setting the permissions. I find both of them equally obtuse, and my advice to people is to pick one and stick with it. Long ago I chose the numeric approach to setting file permissions, so that’s the approach we’ll use here.

This approach is based on treating the three permissions, read, write, and execute as a three-digit binary number, if you have read permission, the first digit is a 1, if not, it’s a 0, and the same for the write and execute permissions. So, the permissions rwx would be represented by the binary number 111, the permissions r-x by 101, and r-- by 100. Since there are three sets of rwx permissions (user, group, everyone), a full Unix file permission mask is defined by three three-digit binary numbers.

Unfortunately, the chmod command doesn’t take the binary numbers in binary form, it expects you to convert them to decimal ^[1] first, and pass it the three sets of permissions as three digits. This sounds hard, but with a little practice, it’ll soon become second-nature.

The key to reading off the permissions is this table:

Rather than trying to memorise the table itself, you should try to learn the process for creating it instead. The lighter coloured cells in the centre of the table are the important ones to be able to re-create on demand. They are not random, they are a standard binary to decimal conversion table, and you should notice that the three columns have a distinct pattern. The right-most column alternates from 0 to 1 as you go down, the column second from the right has two 0s, then two 1s, then two 0s etc, and finally the third column from the right has four 0s, then four 1s. If you wanted to convert a 4 digit binary number to decimal you would add a fourth column that has 8 0s then 1s, if you wanted to convert a 5-bit binary number you’d add yet another column where it’s eight 0s then eight 1s, and so on — each column you go to the left doubles the number of 0s and 1s before repeating.

If you can reproduce this table on demand you’ll have learned two things — how to do Unix file permissions, and how to convert any arbitrary binary number to decimal (though there are better ways if the binary number has many digits).

Even if you don’t want to learn how to create the table, you’ll probably still be fine if you remember just the most common permissions:

If you run a website, for example, regular files like images or HTML pages and images should have permissions 644 (rw-r--r--: you get read and write, everyone gets read). Executable files and folders should have 755 (rwxr-xr-x: you get full permission, everyone can list the folder contents and read the files within).

Let’s end with a few examples. If you want to alter a file you own so that you have read, write and execute permission, but no one else can access the file in any way you would use the command:

If the file should not be executable even by you, then you would use:

Clearly, this is not intuitive, and it’s understandably very confusing to most people at first. Everyone needs to go over this a few times before it sinks in, so if it doesn’t make sense straight away, you’re not alone. Do please keep at it though, this is very important stuff.

We said last time that on OS X, a + at the end of a file permission mask signifies that the file has ACLs (access control lists) associated with it. These ACLs allow more granular permissions to be applied to files on top of the standard Unix File Permissions. If either the ACLs OR the standard Unix permissions deny you the access you are requesting, OS X will block you.

You can read the ACLs associated with files by adding the -le flags to the ls command. If a file in the folder you are listing the contents of has file ACLs, they will be listed underneath the file, one ACL per line, and indented relative to the files in the list. Each ACL associated with a file is numbered, and the numbering starts from 0.

The ACLs read as quite Englishy, so you should be able to figure out what they mean just by looking at them. As an example, let’s have a look at the extended permissions on OS X home directories:

By default, all OS X home folders are in the folder /Users, which is the folder the above commands lists the contents of. You can see here that my home folder (bart) has one or more file ACLs associated with it because it has a + at the end of the permissions mask. On the lines below you can see that there is only one ACL associated with my home folder and that it’s numbered 0. The contents of the ACL are:

As you might expect, this means that the group everyone is denied permission to delete my home folder. Everyone includes me, so while the Unix file permissions (rwxr-xr-x) give me full control over my home folder, the ACL stops me deleting it. The same is true of the standard folders within my account like Documents, Downloads, Library, Movies, Music, etc..

If you’re interested in learning to add ACLs to files or folders, you might find this link helpful: www.techrepublic.com/blog/mac/…​

In the last instalment, we mentioned that all files in a Linux/Unix file system have metadata associated with them such as their creation date, last modified date, and their ownership and file permission information. OS X allows arbitrary extra metadata to be added to any file. This metadata can be used by applications or the OS when interacting with the file.

For example, when you give a file a colour label, that label is stored in an extended attribute. If you give a file or folder a custom Finder icon, that gets stored in an extended attribute (this is how DropBox.app makes your DropBox folder look different even though it’s a regular folder.) Similarly, spotlight comments are stored in an extended attribute, and third-party tagging apps also use extended attributes to store the tags you associate with a given file (presumably OS X Mavericks will adopt the same approach for the new standard file tagging system it will introduce to OS X).

Extended attributes take the form of name-value-pairs. The name, or key, is usually quite long to prevent collisions between applications, and, like plist files, is usually named in reverse-DNS order. E.g., all extended attributes set by Apple have names that start with com.apple, which is the reverse of Apple’s domain name, apple.com. So, if I were to write an OS X app that used extended file attributes, the correct thing for me to do would be for me to prefix all my extended attribute names with ie.bartb, and if Allison were to do the same she should prefix hers with com.podfeet. (Note that this is a great way to avoid name-space collisions since every domain only has one owner. This approach is used in many places, including Java package naming.) The values associated with the keys are stored as strings, with complex data and binary data stored as 64bit encoded (i.e. HEX) strings. This means the contents of many extended attributes is not easily human-readable.

Any file that has extended attributes will have an @ symbol appended to its Unix file permission mask in the output of ls -l. To see the list of the names/keys for the extended attributes belonging to a file you can use ls -l@.

You can’t use ls to see the actual contents of the extended attributes though, only to get their names. To see the names and values of all extended attributes on one or more files use:

The nice thing about the -l flag is that if the value stored in an extended attribute looks like it’s a base 64 encoded HEX string it automatically does a conversion to ASCII for you and displays the ASCII value next to the HEX value.

Apple uses extended attributes to track where files have been downloaded from, by what app, and if they are executable, and whether or not you have dismissed the warning you get the first time you run a downloaded file. Because of this, every file in your Downloads folder will contain extended attributes, so ~/Downloads is a great place to experiment with xattr.

As an example, I downloaded the latest version of the XKpasswd library from my website (xkpasswd-v0.2.1.zip). I can now use xattr to see all the extended attributes OS X added to that file like so:

You can see that OS X has added three extended attributes to the file, com.apple.metadata:kMDItemDownloadedDate, com.apple.metadata:kMDItemWhereFroms and com.apple.quarantine.

All three of these attributes are base 64 encoded HEX. The HEX representation of the data looks meaningless to us humans of course, but OS X understands what it all means, and the xattr command is nice enough to display the ASCII next to the HEX for us. In the case of the download date, it’s encoded in such a way that even the ASCII representation of the data is of no use to us, but we can read the URL from the second extended attribute, and we can see that Safari didn’t just save the URL of the file (https://www.bartbusschots.ie/downloads/xkpasswd-v0.2.1.zip), but also the URL of the page we were on when we clicked to download the file (https://www.bartbusschots.ie/blog/?page_id=2137). Finally, the quarantine information is mostly meaningless to humans, except that we can clearly see that the file was downloaded by Safari.

The xattr command can also be used to add, edit, or remove extended attributes from a file, but we won’t be going into that here.

That’s where we’ll leave things for this instalment. Hopefully, you can now read all the metadata and security permissions associated with files and folders in OS X, and you can alter the Unix file permissions on files and folders.

We’ve almost covered all the basics when it comes to dealing with files in the Terminal now. We’ll finish up with files next time when we look at how to copy, move, delete, and create files from the Terminal.

This is one of those topics that I think is best taught through example, so let’s start by opening a Terminal window and navigating to our Documents folder:

We’ll then create a folder called TtT6n in our Documents folder with the command:

As you can see, directories/folders are created using the mkdir (make directory) command. When used normally the command can only create folders within existing folders. A handy flag to know is the -p (for path) flag which will instruct mkdir to create all parts of a path that do not yet exist in one go, e.g.:

Since the TtT6n folder already existed the command will have no effect on it, however, within that folder it will first create a folder called topLevelFolder, and then within that folder, it will create a folder called secondLevelFolder.

At this stage let’s move into the TtT6n folder from where we’ll execute the remainder of our examples:

We can now use the -R (for recursive) flag for ls to verify that the mkdir -p command did what we expect it to. I like to use the -F flag we met before with -R so that folder names have a trailing / appended:

When using ls -R the contents of each folder is separated by a blank line, and for folders deeper down than the current folder each listing is prefixed with the relative path to the folder about to be listed followed by a :.

In other words, we are expecting to see just a single entry in the first segment, a folder called topLevelFolder, then we expect to see a blank line followed by the name of the next folder to be listed, which will be the aforementioned topLevelFolder, followed by the listing of its contents, which is also just one folder, this time called secondLevelFolder. This will then be followed by a header and listing of the contents of secondLevelFolder, which is currently empty.

Let’s now create two empty files in the deepest folder within our test folder (secondLevelFolder). There are many ways to create a file in Unix/Linux, but one of the simplest is to use the touch command. The main purpose of this command is to change the last edited date of an existing file to the current time, but if you try to touch a file that doesn’t exist, touch creates it for you:

You can use ls -RF to verify that these files have indeed been created (you can use the -lh flags along with the -RF flags to see that the files are indeed empty — i.e. 0 bytes in size).

Let’s now create a second top-level folder, and copy the files to it:

As you can see, the command to copy a file is cp. Here we have used cp in its simplest form, with just two arguments, the first being what to copy, the second being where to copy it to. The first argument must be a file and the last a folder.

The cp command is a cleverer than this though — it can take any number of arguments greater than two. All arguments but the last one will be treated as sources, and the last one will be treated as the destination to copy all these sources to. So, we can re-write our two cp commands above as simply:

(You can verify that the copy has worked with the ls -RF command.)

We can be even more efficient though — we can use the shell’s wild card completion functionality to simplify things even further:

Note that the cp command will happily work with either full or relative paths. Also, you may remember that a few instalments back I mentioned that every folder contains a file called . that’s a reference to the folder containing it, and that while this sounds useless, it’s actually very useful. Well, the cp command provides a great example to illustrate this point. To copy a file to the folder you are currently in, you can use . as the destination path.

By default, cp will only copy files, but, it can copy folders (and their contents) if you use the -R (recursive) flag. The flag should precede the argument list.

Let’s now create yet another empty folder and copy the folder secondLevelFolder, and all its contents, into the new folder:

(Again, you can use ls -RF to verify that the copy has worked as expected.)

The mv (move) command works in a similar way to cp, but, it removes the source files and folders after it has copied them to the destination folder. The mv command can move folders without needing to specify any flags. As an example, let’s create yet another folder, and move our original secondLevelFolder and its contents to it:

(Again, we can use ls -RF to verify that the folder and the files within it have indeed been moved.)

Users of DOS may remember that in DOS the commands to copy and move were simply copy and move, and, that there was a separate command to rename a file or folder which was simply rename. Unix/Linux does not have a separate rename command, instead, you rename a file or folder by moving it from its old name to its new name. As an example, let’s rename our first folder (topLevelFolder) to topLevelFolder0:

We can use a simple ls command to verify that the folder has indeed been renamed.

Finally, I want to briefly mention the rm (remove) command, which can be used to delete files and/or folders. The rm command simply takes the list of files to be nuked as arguments. Unlike deleting files in the Finder, the rm command doesn’t have any kind of safety net — it does not use the recycle bin. If you mess it up, you’d better hope you have a recent backup because your files are GONE! This is why I advise people to avoid using deleting files/folders from the command line unless you absolutely have to, or, are very comfortable on the Terminal. It’s much safer to delete things in the Finder. Whenever you do use the rm command, ALWAYS check over your command before hitting enter! To adapt the older carpentry adage that you should always measure twice and cut once, my advice when using rm is to think twice and execute once.

By default, the rm command will only allow you to delete regular files, but if you add the -r flag it will delete folders and their contents too. A great additional safety net when using rm either in recursive mode or with shell wild card completion is the use the -i flag to enter interactive mode — in this mode, you will be asked to confirm the deletion of each file and folder.

Let’s end by going back up one level, and then deleting our digital playpen recursively:

Today we learned how to create files and folders with touch and mkdir, to copy files with cp, move and rename them with mv, and delete them with rm. Next time we’ll move on to looking at how Unix/Linux/OS X handled processes.

We now live in a world where multitasking is a normal and expected part of our computing experiences, be that on our servers, desktops, laptops, tablets, or phones. Multitasking is not something that comes naturally to our computers though. Until relatively recently, our home computers had a single CPU that could execute only a single task at a time. Or, in computer-science-speak, our computers could only execute a single simultaneous thread of execution. In the days of DOS that was true of the hardware as well as the software. You booted DOS, it then handed control over to the program you launched with it, which then had full control of your computer until it exited and handed control back to DOS. You could not run two programs at the same time.

Many of us got our first introduction to the concept of multitasking with Windows 3.1. Windows ran on the same single-CPU hardware as DOS, so how could it do many things at once on hardware that could only do a single thing at a time? Well, it didn’t, it just looked like it did. Even back in the early 90s, our computers were doing millions of calculations per second, so the way Windows 3.1 did multitasking was through a software abstraction. Every task that wanted to use the CPU was represented in software as a “process”. This representation could store the entire CPU-state of the thread of execution, allowing Windows to play and pause it at will. A few thousand times a second, Windows would use hardware interrupts to wrest control of the CPU from the running process, take a snap-shot of its state, save it, then load the last saved state of the next process in the queue, and let it run for a bit. If you had 10 processes and a 1 MHz CPU, then each process got about 100,000 CPU cycles to work with per second, enough to give you the impression that all your programs were all running at the same time.

Our modern hardware can do more than one thing at once, even on many of our phones. Firstly, modern CPUs are hyper-threaded. That means that they support more than one thread of execution at the same time on a single CPU (more than 1 does not mean 100s, it usually means two). Secondly, many of our CPUs now have multiple cores on the same piece of silicon. This means that they are effectively two, or even four, CPUs in one, and each one of those cores can be hyper-threaded too! Finally, many of our computers now support multiple CPUs, so if you have four quad-core multi-threaded CPUs (like the much-loved octo-macs), you have the ability to execute 4x4x2, i.e., 32 threads, at the same time. Mind you, your average Mac has many fewer than that; a dual-core hyper-threaded CPU is common, giving you ‘just’ four actually simultaneous threads of execution.

Clearly, being able to run just 4 processes at the same time is just not enough, hence, even our modern computers use the same software trick as Windows 3.1 to appear to run many tens or even hundreds of processes at the same time.

There is literally an entire university semester of material in designing strategies for efficiently dividing up available CPU-time between processes. All we’ll say on the topic for now, is that the OS gets to arbitrate which process gets how much time, and that that arbitration is a lot more involved than a simple queuing system. The OS can associate priorities with processes, and it can use those priorities to give some processes preferential access over others.

We should also clarify that there is not a one-to-one mapping between processes and applications. Each app does have at least one process associated with it, but once an app is running it can fork or spawn as many child processes as it wants/needs. You could imagine a word processing app having one process to deal with the UI and another separate process for doing spell checking simultaneously in the background.

We should also note that on modern operating systems there are two broad classes of processes, those used by the OS to provide system services (often referred to as system processes), and those instigated by users to do tasks for them (often called user processes). There is no fundamental difference between these two groups of processes though, it’s just a taxonomy thing really. If you boot up your Mac and leave it at the login screen, there will already be tens of system processes running. Exactly how many will vary from user to user depending on how many or few services are enabled.

Finally, we should note that not all running processes are represented in the UI we see in front of us. When we launch an app there is a clear mapping between the process we started and one or more windows on our screen or icons in our menubar, but there are many processes that don’t have any windows, don’t show up in the dock, and don’t have an icon in the menubar. These hidden processes are often referred to as background processes.

Each running process on a Unix/Linux/OS X computer has an associated process ID or PID. This is simply an integer number that the OS uses to identify the process. The very first process (the OS kernel), gets the PID 0, and every process that starts after that gets the next free PID. On Linux systems, the process with the PID 1 is usually init, which is the process Linux uses to manage system services, so the Kernel will start init which then starts all the other system processes. OS X uses the same idea, but instead of using init, it uses something called launchd (the launch daemon) to manage system processes. If your system has been running for a long time it’s normal to see PIDs with 5 digits or more.

As well as having a PID, each Linux/Unix/OS X process (except for the kernel), also has a reference to the process that started it, called a Parent Process ID, or a PPID. This gives us the concept of a hierarchy of processes, with the kernel at the top of the pyramid.

As well as a PID and PPID, each process also runs as a particular user. Whether a given file can or can’t be accessed by a given process is determined by the user the process is running as, and the permissions set on the file.

Now it’s time to open up the Terminal and get stuck in with some real-world commands.

Let’s start with the most basic process-related command, ps, which lists running processes. Note that ps is one of the few basic terminal commands that behave differently on Linux and Unix.

On a Mac, if you run the ps command without arguments, all that will be listed are the terminal-based processes owned by the user running the shell. In all likelihood, all you’ll see is just a single bash process, which is your current shell (if you have multiple Terminal windows/tabs open you’ll probably see a bash processes for each one).

The columns you’ll see listed are PID (process ID), TTY (ignore this for now, it’s not really relevant on modern computers), TIME (how much CPU time the process is currently using), and CMD, the running command (including arguments if any).

Most of the time, the output of ps without arguments is of little to no interest. you need to use one or more arguments to get anything useful from ps.

Let’s start by listing all the processes owned by a given user, regardless of whether or not they are terminal-based processes:

e.g.

If you’re a big multitasker like me, you may be surprised by just how many processes you have spawned. If you use Chrome as your browser you may also notice that it uses a separate process for each open tab.

Something else you’re likely to want to do is to see all current processes, regardless of who they belong to. On Linux, we would do that with the -e (for everyone) flag, while on Unix we would do that with the -A (for ALL) flag. OS X conveniently supports both, so just use whichever one you find easiest to remember.

or

At this stage, you’ll be seeing a very long list, but for each entry, all you’re seeing is the standard four headers. You’re viewing the list of all processes for all users, but there is no column to tell you which process belongs to which user! For this list to be useful you need to ask ps to give you more information about each process.

Your first instinct might well be to try the -l flag in the hope that ps behaves like ls. Give it a go:

As you can see, you now get much more information about each process, but it’s not actually particularly useful information! While giving you too much irrelevant information, -l doesn’t actually give you all the information you probably do want. For example, -l gives the UID number of the user who owns the process, rather than the username.

A better, though still imperfect, option is the -j flag (no idea what it stands for). Try it:

This still gives you more information than you need, but it does at least give you usernames rather than UIDs.

Thankfully there is a better option, you can use the -o flag to specify the list of headings you want in the output from ps. To see a list of all the possible headings, use:

To specify the headings you want, use the -o flag followed by a comma-separated list of headings without spaces after the commas. In my opinion, the following gives the most useful output format:

Finally, you can also use flags to sort the output in different ways. Of particular use are -m to sort by memory usage, and -r to sort by CPU usage.

The ps command is a good way to get an instantaneous snapshot of the processes running on your system, but usually, what you really want is a real-time sorted list of processes, and for that, we have the top command:

You’ll now see real-time statistics on memory and CPU usage as well as a list of your top processes. On most Linux distributions the default sorting for top is by CPU usage, which is actually very useful, but Apple didn’t think like that. Instead, Apple chose a default sort order of descending PID, i.e. the most recently started processes.

You can either re-sort after starting top by hitting o and then typing something like -cpu (for descending CPU sorting), or -vsize (for descending memory usage), and hitting enter.

Or, you can pass the same arguments when starting top from the command line:

Finally, to exit out of top just type q.

When looking at top, a very important thing to look at is the so-called load averages, which are shown in the metadata above the process list at the top of the top screen. There will be three of them, the first is the average over the last minute, the second is the average over the last 5 minutes, and the third is the average over the last 15 minutes. The actual definition of the load average is a bit esoteric, so we’re not going to go into it here. What you should know is that the load average is a pretty good metric for the amount of stress a computer is under. If any bottleneck starts to slow processes down, the result will be increased load averages. If your CPU is stressed, load averages will go up, if you’ve run out of RAM and your system is having to do a lot of swapping, load averages will go up, if you’re doing a lot of IO and your disk is too slow to keep up, your load averages will go up.

The next obvious question is, how high a load average is too high? A good metric is that ideally none of your load averages should cross the number of logical CPUs you have during regular user.

It’s OK for the 1-minute average to cross the number of CPUs you have occasionally, but if the 15-minute average crosses the number of CPUs you have when you’re not doing something unusually stressful like transcoding video, then your computer is probably in need of an upgrade.

Clearly, ps and top can give you a lot of information about the processes that are running on your system, but they are both quite clunky because to get the most out of them you have to use a lot of flags. On OS X, a much better choice is to use the built-in Activity Monitor app (Applications  Utilities  Activity Monitor). This will show you all the same information but in a nice easy-to-use GUI. You can choose which processes you see with a drop-down at the top right of the window, and you can sort on any column by clicking on its header.

To visually see the hierarchy of processes, you can choose All Processes, Hierarchically from the drop-down. Bear in mind though that this view is not good for sorting or filtering. If you’re trying to figure out which apps are using the most CPU or RAM, it’s best to stick with the All Processes option.

So far we’ve looked at commands for listing processes. Next time we’ll move on to commands for interacting with processes, particularly, for stopping processes that are causing problems.

Whether or not you think of it in this way, you’re always starting processes. Each time you launch an app, you’ve started a process. Also, each time you issue a command on the Terminal, you’re starting a process.

So far, each time we’ve started a process from the command shell, it’s been a so-called foreground process. We enter the command, this starts a new process, and that new process gets control of our terminal until it finishes, at which time we get returned to the command shell. When we run very simple commands like ps or ls, the processes take just milliseconds to run, so we don’t really notice that we lose our prompt for a moment while the new process takes charge, does its thing, and then hands control back.

However, sometimes you want to start a process from the command shell, and not have it take control of your Terminal. A good example might be using a command shell to start a GUI app. You want the app to start, and to stay running, but you also want your command prompt back! We can do this using something called a background process. Should that process produce any text output, it will still show in our Terminal window, but as soon as we hit enter we’ll have our command prompt back.

Backgrounding a process is actually very easy to do, just add an & to the end of the command, and it will start in the background!

Let’s illustrate this with a quick example. For this to work you’ll need to be on OS X, have FireFox installed, and have it not be running when we start.

First, let’s start FireFox from the Terminal as a regular foreground process:

You’ll see that the FireFox GUI launches as normal and that it outputs some information on what it’s doing to the Terminal. You’ll also notice that you have lost your command prompt. You can hammer the enter key all you want, but you’re not getting that command prompt back until the FireFox process exits and hands control back to your command shell. Go ahead and do that through the GUI as normal. Notice how the moment FireFox quits you get your command prompt back.

Now, let’s start FireFox as a background process:

The FireFox GUI will start just like before, and the same debugging information will be displayed in the Terminal window, but this time you just have to hit enter to get your command prompt back. Firefox is still running, but you have your command prompt back.

If you scroll up you’ll see that the very first piece of output when you ran the command was a line consisting of a number in square brackets, followed by another number, something like: [1] 1714

The number in square brackets is a counter for the number of background processes the current command shell is running, and the second number is the PID of the new background processes, in this case, FireFox’s PID.

On Unix and Linux it’s normal to launch GUI apps from the command line by backgrounding them, and as you can see, it works on the Mac too. However, the Mac has another, better, way of opening things from the command line that other versions of Linux and Unix don’t have: the open command.

The best way to think of open is as the command-line equivalent of double-clicking something in the Finder. When you pass open one or more files as arguments it will open those files using the default app for their file type. As an added bonus, it will automatically background any apps it starts, so you don’t even have to remember the &! Note that if you pass open a folder rather than a file it will open the folder in a new Finder window.

A very handy trick is to use open to get a Finder window showing the contents of your command shell’s present working directory as follows:

Another useful tip is to use open to quickly view hidden folders in the Finder. On recent versions of OS X, where the Library folder is hidden, I often use:

As well as opening things with the default app for their type, open can also open files or folders with any other app that supports the given type. The -a flag allows you to specify the app the item should be opened with.

I do a lot of Perl programming, and I have two apps installed that I regularly use to edit Perl code. I have OS X configured to open .pl and .pm files with Komodo Edit by default. Komodo Edit is a nice free Perl IDE, and I do all my big programming jobs in it. However, being a big IDE, it’s a bit of a hog — it takes so long to open that it has a splash screen! So, when I just want to make a quick edit, I prefer to use the light-weight Smultron editor instead. I can use open with the -a flag to specify that I want my Perl file opened with Smultron using a command something like:

Similarly, if I was using a computer that didn’t have Smultron installed I could open the Perl file with TextEdit using a command like:

That’s all we really need to know about starting processes, so let’s move on to ending them.

The command to quit a process is kill, which sounds very draconian indeed. Despite how scary it sounds, you don’t have to be overly afraid of it, because if used without flags, all it does is politely ask the process if it would please quit. In other words, it’s the command-line equivalent of ⌘+Q or selecting Quit from an app’s menu.

The kill command needs at least one argument: the PID of the process you’d like to end. This is where the ps and top commands we learned about last time come in very handy (or indeed the Activity Monitor app if you are on a Mac). Note that you can use kill to end as many processes at once as you like; just keep adding more PIDs as arguments.

Let’s work through another example, using FireFox again. We’ll start by opening FireFox and backgrounding it with:

Before we can quit FireFox with the kill command we need to know its PID. We can either scroll up and read it from the first line of output or, we can use the ps command to get it. Since we started FireFox from the terminal, it will show up when we use ps without any arguments at all.

Once you have the PID you can quit FireFox with (replacing [the_PID] with the actual PID of course):

You should see FireFox exit, and the next time you hit enter on your Terminal you should see a message telling you that a process you started and backgrounded has ended.

Finding PIDs can be a bit of a pain, so you’ll be happy to know that you don’t have to! There is another command for ending processes that uses process names rather than PIDs, it’s the even more scary-sounding killall command.

For what seems like the millionth time today, let’s start FireFox and background it:

Rather than looking up its PID, let’s now exit it with the killall command:

Note that you need to be careful with killall because, as its name suggests, it will kill ALL processes with a given name, not just one!

Up until now, we’ve been polite, and we’ve used kill and killall to ask processes to please quit themselves. When an app crashes or hangs, that won’t get you very far. If the app is so messed up it can’t deal with mouse input anymore, it’s also not going to respond when kill or killall politely ask it to stop.

When this happens, it’s time to bring out the big guns! Both kill and killall take an optional argument -KILL, which tells kill/killall to instruct the OS to terminate the process, rather than asking the process to terminate itself. ONLY DO THIS AS A LAST RESORT, YOU CAN LOSE UNSAVED DATA THIS WAY!

Note that on older Unix systems killall didn’t exist at all, and kill only took numeric arguments. The old numeric equivalent of -KILL is -9, and both kill and killall on OS X (and Linux) will accept this old-fashioned flag as well as the more modern -KILL.

Before we finish I want to reiterate how important it is to always try kill and killall without the -KILL option first. Think of it this way, it’s basic good manners to ask the process to please leave before you call the bouncers over to eject it!

We’ll be revisiting processes again later in the series, but we’re done with them for now.

In the next instalment, we’ll be taking a look at the built-in manual that comes with every Unix/Linux OS, including Mac OS X. You don’t need a book to tell you what flags or arguments a command expects, or what exactly they mean, you can find it all out right from the command shell, even if it takes a little practice to learn to interpret the information.

Let’s start with the practicalities of opening, closing, and navigating a man page before we look at the structure and formatting.

To open a man page simply use the man command with a single argument, the name of the command or config file you would like to read the entry for.

As an example, let’s call up the documentation for the ls command:

You’ll immediately notice that you’ve lost your command prompt, and are viewing a text document in your terminal window. The most important thing to learn is how to exit out of the man page and get back to your command prompt. To get out, simply hit the q key (for quit)!

OK, now that we know to get back out, let’s re-open the man page for ls and have a look around.

You can navigate up and down in a man page with the up and down arrow keys. You can also scroll down a single line by hitting enter, or a whole page at once with the spacebar. To scroll up a whole page at once hit b (for back). You can also go back half a page with the u key (for up).

You can search in a document by typing / followed by your search pattern, and then hitting enter. To get to the next result hit the n key (and to get to the previous result, Shift+n).

Now that we can navigate around, let’s have a closer look at the structure of a man page. The first thing to note is that each man page is divided into sections, which are labelled in all capitals, and their content is indented by one tab. Sections can contain sub-sections who’s content is indented by two tabs, and so on.

Just about every man page you’ll ever see will have the following three sections:

NAME — this will be the first section, and will simply contain the name of the thing the man page is documenting, perhaps with a very short description. E.g. the name section in the ls man page contains ls -- list directory contents.

SYNOPSIS — this is a very important section and one we’ll look at in a lot more detail below. This section uses a somewhat cryptic notation to describe the structure of the arguments a command expects.

DESCRIPTION — this is where the main body of the documentation will be contained. The description is usually the longest section by far, and often contains sub-sections. This is where you expect to find a list of all the options a command accepts and a description of what they do.

Just to reiterate, there is no formal structure every man page has to follow, but there are conventions, so most man pages will contain at least some of the sections listed below, in addition to the three above. A man page may contain custom sections though, so the list below is not exhaustive.

OVERVIEW — very long man pages sometimes contain a one-paragraph summary of what the command does between the SYNOPSIS and DESCRIPTION sections.

OPTIONS — more complicated commands may separate their list of command-line flags into a separate OPTIONS section immediately after the DESCRIPTION section.

EXAMPLES — many man pages contain annotated examples of how the command being documented can be used. If present, this is usually a very helpful section, and often worth jumping straight to.

TIPS — some man pages use this section to offer some useful advice to users.

SEE ALSO — this section is used to list related man pages, often describing related commands or associated configuration files.

FILES — if a command’s function is affected by one or more configuration files, the default file system locations for these files are often listed in this section. E.g. the FILES section from the ntpdate man page:

STANDARDS — if the command conforms to some kind of standard set out by some sort of standards authority (perhaps the IEEE or the ISO), then the relevant standards may be listed in this section. E.g. the STANDARDS section from the ls man page:

DIAGNOSTICS — for now, you can probably ignore this section. If it’s present it contains information that’s usually only useful when writing or debugging scripts.

ENVIRONMENT — we haven’t discussed the command-line environment yet in this series, although it is next on the list. For now, you can ignore this section.

COMPATIBILITY — this section will only be present if the command has potential compatibility problems; perhaps it doesn’t quite comply with a standard or something like that.

LEGACY DESCRIPTION — some commands have changed their behaviour over time. This section is where the old behaviours will be documented. This is really only useful when working with old scripts which might still be assuming the command’s old behaviour.

BUGS — if there are known problems with the command or known conditions which cause unusual or undesirable behaviour, they may be listed in this section.

HISTORY — this can be a fun section, and is usually very short, and details the origins of the command. E.g. the HISTORY section of the ls man page tells us that “An ls command appeared in Version 1 AT&T UNIX”.

AUTHOR — details the authors of the command being documented.

COPYRIGHT — the copyright information for the command being documented.

When you’re first learning about a command, the DESCRIPTION section is probably the most useful to you, but when it comes to relearning something you were once familiar with, the SYNOPSIS section is often the most useful. Although it’s short, it’s very dense with information. However, to be able to extract the meaning from this short section you need to understand the meaning of the formatting:

BOLD TEXT — any text in bold should be entered verbatim

UNDERLINED /ITALIC — any text that is either in italics or underlined (depending on your version of man, usually underline in modern OSes) needs to be replaced with a real value as described by the text. E.g. file should be replaced with the path to an actual file.

…​ — anything followed by an ellipsis (three dots) can be optionally repeated

[] — anything contained within square brackets is optional

| — the pipe symbol should be read as ‘or’

The most important thing is not to be afraid of man pages. At first, they will seem archaic and confusing, but you’ll soon get used to their style, and you might even come to like it! There is no substitute for practice though — the only way to learn to read man pages is to read man pages!

Finally, let’s end on a really bad nerd joke!

The simplest command for dealing with text files is cat. You can use cat to print the contents of a text file to the screen by calling it with one argument, the path to the text file you’d like printed out.

E.g., using cat to read the content of your computer’s network time configuration (definitely works on OS X, should work on most other Linux and Unix distros too):

cat works great for short files, but it’s not well suited for reading longer files. For example, using cat to show the config for man:

While it’s useful to be able to print out the contents of a file, what would be much more useful is a command to allow us to read a file at our own pace. In Unix-speak, what we want is a pager.

Historically the pager of choice on Unix and Linux systems was more. Like cat, you invoke more by passing it a file as an argument, but unlike cat, after more has printed a screen-full of text, it will stop. You can then use the enter key or the down arrow key to move down one line at a time, or the spacebar to move down a whole screen at a time. E.g.

While you’ll find more included in just about every modern Unix or Linux OS, it’s only there for legacy reasons. more has been superseded by a newer and more feature-rich pager, which is humorously called less (because we all know less is more).

We’re actually already familiar with less because it’s the pager used by the man command. All the commands we learned in the previous instalment for navigating around man pages are actually commands for navigation around text files with less!

The less command is invoked in the same way as cat and more, e.g.:

The less command is very powerful, and it can deal with very large files without getting bogged down. As an example, most Linux and Unix distributions contain a dictionary file, usually located at /usr/share/dict/words. This file is 235,886 lines long on OS X, and less has no problems searching or navigating it:

While less is, without doubt, the best pager on modern Unix/Linux systems, and while it should be your command of choice for reading most text files, there is another pair of text-file-related commands every command-line user should know — head and tail.

The head and tail commands really come into their own when it comes to dealing with log files. The head command will show you the first 10 lines of a file, and the tail command the last 10. Simply using head and tail on a log file will quickly answer a very simple but very important question — what date range does my log file span? (Note — Linux and most Unix users will find the system log at /var/log/messages, OS X is unusual in storing its system log in system.log)

Both head and tail can actually show any number of lines at the beginning or end of a file by using a rather strange flag, - followed by an integer number. E.g. to see the last 50 lines of the system log use:

Or, to see just the first line use:

Finally, the tail command has one more very useful trick up its sleeve, it can continue to print out new lines at the end of a file in real-time as they are added. This is perfect for monitoring log files while you’re troubleshooting. To enter this real-time mode invoke tail with the -f flag. Remember that the only way out of a tail -f is with Ctrl+c.

You could run the command below to get a live view of the system log, but it’s hard to force log entries to appear there. On OS X, a nice example to use is the WiFi log file. If you run the command below in a Terminal window and then switch networks, you should see entries appear in the log file in real-time:

You can roughly divide the command-line text editors into two categories, the quick and simple editors, and the power editors. The simpler editors are much easier to learn but much less powerful. If you spend a lot of time on the command line, learning at least one of the power editors is worth the effort in my opinion.

Anyway, let’s start simple. There are two common, quick and simple command-line text editors, pico and nano, and the chances are very high that no matter what modern Linux or Unix OS you are using, one of these two will be available. (OS X comes with nano, but like with more, it pretends to have pico too, until you run pico --version when it fesses up to really being nano.)

Once opened pico and nano are virtually indistinguishable anyway, so which you have really doesn’t matter. If you want to edit an existing file, you invoke nano (or pico) with the path to the file you want to edit. If you want to create a new file, you invoke nano with the path you’d like the new file to be created at.

Let’s play it safe and start a new file for our experimentations:

(On OSes other than OS X use nano ~/nanoTest.txt instead.)

Once in nano, you’ll see a cursor where you can start to type, and along the bottom a list of possible commands with their key combinations next to them (remember, ^ is short-hand for the Ctrl key).

Let’s keep this simple and just type the utterly clicheéd sentence:

You’re probably guessing that to save a file you’d use some kind of save option, perhaps ^s, but that would be much too simple. Instead, in nano-speak, you want to write the file out, so you use the command ^o. After hitting Ctrl+o, nano will then show you the path it’s about to write the file to (which is editable should you change your mind about the destination of your edits.) When you’re happy with the path, you hit enter to actually save the file. Note that if you try to exit nano without writing out first, nano will offer to save the file for you, so you can also save with the sequence ^+x, y, Enter.

At this stage, you actually have all the skills you’re likely to truly need, so feel free to tune out at this point. However, if you’re interested, I’m also going to give a very brief and very superficial overview of one of the two most popular modal editors.

There are two leviathans in the command-line text editing world, and both have been around since the 1970s. In nerd circles, your choice of text editor is about as polarising as the Republican and Democratic political parties in the US. You almost never meet someone who excels at both of them, and every Unix nerd has their favourite of the two. The two editors I’m talking about are Emacs and vi.

As it happens I’m a vi guy, so it’s vi that I’m going to give a quick overview of.

The single most important thing to know about vi is that it is modal, that means that at any given time vi is in one mode OR another. Specifically, vi is always in either insert mode OR command mode. In insert mode, everything you type is entered into the file where the cursor is, and in command mode, nothing you type is entered into the file, and everything you type is interpreted as a command by vi. This confuses the heck out of people, and it takes some getting used to!

You invoke vi in the same way you would nano, so for our example let’s do the following:

(on OSes other than OS X use vi ~/viTest.txt instead)

When the file opens we are in command mode. If we were editing a pre-existing file instead of creating a new one, we would be able to move the cursor around, but anything we type would be treated as a command by vi, not as input for the file.

Let’s start by switching from command mode into insert mode. To do this, hit the i key (i for insert). Notice that at the bottom of the screen it now says INSERT in all caps — you’ll always see this when you are in insert mode.

Let’s be boring and insert the same text as before:

To get back out of insert mode you use the esc key. You’ll see that when you hit escape the INSERT at the bottom of the screen goes away and there is actually a prompt down there for you to enter commands into.

The most important commands to know are the following:

You can combine those commands into one, so to save and exit you would use the command :wq.

If you start hammering away on the keyboard in command mode, erroneously assuming you are in insert mode, it’s inevitable that you’ll accidentally invoke a command you REALLY didn’t want to invoke. This is why the most important vi command to know after :wq is :q!, which is exit without saving (if you try :q without the ! when there are unsaved changes vi won’t let you exit).

So far this all sounds needlessly complex, so let’s step things up a gear, and start to make real use of the command mode in vi. Let’s start by copying a line of text, or, in vi-speak, let’s yank a line of text.

While in command mode (hit esc to make double-sure), move the cursor (with the arrow keys) so it’s somewhere on the line that says ‘Hello World!’, then type yy. You have now yanked the current line.

Now that we have a line yanked, we can paste a copy of it by hitting the p key (for put). You can keep hammering on the p key as often as you like to keep adding more copies of the line.

One of the things I like most about vi is that you can enter a number before many of the commands to repeat them that many times. To put our yanked line 500 times the command is 500p.

Let’s say our aspirations have expanded, we’d like to greet the entire universes, not just the world! We could make over 500 edits, or, we could ask vi to do a global find and replace for us with the command:

You can also use vi commands to navigate around a file. E.g. :n (where n is a number) will take you to the nth line. So to get to the 25th line you would enter the command :25.

Similarly, $ jumps the cursor to the end of the current line, and 0 jumps the cursor to the start of the current line.

vi will of course also let you easily delete content. To delete the current line just enter dd. You can probably guess how to delete 400 lines in one go, it is of course 400dd. To delete everything from the cursor to the end of the line enter D, and to delete one character use x.

One final thing to mention in this VERY brief overview is that there are multiple ways to enter into insert mode from command mode. We already know that i will start you inserting at the cursor, but it’s often useful to start inserting one character after the cursor, which you do with a (for append). You can also enter insert mode on a new blank line after the line containing the cursor with o (for open line). Similarly, O opens a new line before the line with the cursor on it.

This is just the tip of the vi-iceberg, it can do much much more. There are literally books written about it. However, in my opinion once you understand the modal nature of vi, all you really need is a good cheat sheet to help you find the commands you need until they become second nature. (I have a printout of the first diagram on this page hanging on my wall at work).

It’s very important to be able to read the content of text files from the command line, and also to be able to do at least basic edits from there. Every command-line user needs to at least remember less and tail -f. Every command-line user also needs to familiarise themselves with pico/nano at the very least. If you spend a lot of time on the command line, I think it’s definitely worth investing the time to learn vi or Emacs.

Although we’ve not used the word ‘environment’ before, we have already discussed some elements that make up a process’s environment. Specifically, we know that every process has a user ID associated with it (we say that every process runs as a user), and we have come across the concept of the present working directory. Both of these elements make up part of the basic environment that every process on your computer executes within, not just command shells. The third major pillar in the basic environment is environment variables. These are name-value pairs that can be accessed by running processes.

When one process starts another process, the child process inherits a copy of the parent process’s environment. The child process runs as the same user the parent process was running as, it starts with the same present working directory, and it gets a copy of all the environment variables that existed in the parent’s environment at the moment the child was spawned. The important thing to note is that child processes do not share a single environment with their parents, they get a duplicate that they are then free to alter without affecting the parent process’s environment. When a child process changes its present working directory, that has no effect on the parent process’s present working directory, and similarly, when a child process changes the value stored in a given environment variable, that has no effect on the value stored in the same environment variable within the parent process’s environment.

While all processes have access to a basic environment, command shells extend this basic foundation to provide a much richer environment for their users. Until now very little that we have looked at has been shell-specific, but that changes with this instalment. Each command shell gets to create its own environment and to define its own mechanisms for interacting with it. What works in Bash will not necessarily work in Ksh, Zsh, etc. In this series we’ll only be dealing with the default command shell on most modern Unix and Linux OSes (including OS X): Bash. Note that Bash is an extended version of SH, so what works in SH works in Bash, and much, though not all, of what works in Bash also works in SH.

In this instalment, we’ll be focusing on Environment Variables, and specifically, how Bash interacts with them.

The command to list the names and values of all currently set environment variables is simply env (or printenv on some systems). E.g.:

env lists the environment variables one per line. On each line, the name of the variable is the text before the first =, and the value is everything after it.

Some of these variables are purely informational, while others are used to affect how a process behaves.

Bash, like every other process, has access to all the variables set within its environment. However, Bash extends the concept and of variables into shell variables, of which the environment variables are just a subset. Bash shell variables can be local to the shell, or can exist within the shell and the environment. We already know that env lets us see all the environment variables which exist in our shell but there is another command to let us see all the variables in our shell, both those in the environment and the local ones, and that command is set. To see all the shell variables that exist, call set with no arguments. E.g.

If you compare the output of env and set you’ll see that every environment variable is a shell variable, but, there are many more shell variables than there are environment variables. Remember, when a child process is created only the environment variables get copied into the child process’s environment, even if the child process is another Bash process. Shell variables are local to a single command shell, hence they are often called local variables.

Shell variables can be used when invoking shell commands. To access the content of a variable you use the $ operator. When you enter $VARIABLE_NAME in the shell it will be replaced with the value of the variable named VARIABLE_NAME. E.g. to change to the Desktop directory in your home folder you could use:

or (if you have a Mac configured in the default way)

Way back in the second instalment we discussed quoting strings in the shell, and we mentioned that there was a very important difference between using double and single quotes and that it would become important later. Well, this is where that difference becomes important. If you use the $ operator within a string enclosed by double quotes the variable name will get replaced by the variable’s value, if you use it within a string contained within single quotes it will not!

This is why the following do work (this is an OS X-specific example):

But the following does not:

Note that you can also inhibit the $ operator by escaping it with a \ character. Hence, the following has exactly the same effect as the previous command:

Sometimes when we type the $ symbol we mean the $ operator, and sometimes we just mean the character $. If we mean the character, we have to inhibit the operator either by escaping it or by using single quotes around the string containing it. Whenever you find yourself typing the $ character, pause and think which you mean before hitting Enter, and be sure you have it escaped or not as appropriate.

While we can list the values stored in all variables with set, it’s also helpful to know how to show the value stored in a single variable. The easiest way to do this is to make use of the initially useless-seeming command echo. All echo does is print out the argument you pass to it, so, a simple example would be:

This seems pretty dull, but, when you combine echo with the $ operator it becomes much more useful:

We can even get a little more creative:

Now that we can use variables, let’s look at how we create them and alter their values. You create variables simply by assigning them a value, and you alter their value by assigning them a new value. The = operator assigns a value to a variable. In our examples, we won’t use a variable set by the system, but we’ll create our own one called MY_FIRST_VAR.

Before we start, we can verify that our variable does not exist yet:

Now let’s create our variable by giving it a value:

Now let’s verify that we did indeed initialise our new variable with the value we specified:

Now let’s get a little more creative and change the value stored in our variable using values stored in two variables inherited from the environment:

Because we used double quotes, it is the value stored in the variables LOGNAME and HOME that have been stored in MY_FIRST_VAR, not the strings $LOGNAME and $HOME.

At this stage our new variable exists only as a local shell variable, it is not stored in our process’s environment:

The export command can be used to 'promote' a variable into the environment. Simply call the command with the name of the variable to be promoted as an argument, e.g. to push our variable to the environment use:

We can now verify that we really have pushed our new variable to the environment:

So far in this series I have been a little loose with the term command, I’ve avoided putting too fine a point on exactly what a terminal command is, but we’ll remedy that today. If you remember right back to the second instalment, we said that when entering commands on the command line, the first word is the command, and the other words (separated by spaces) formed the arguments to that command. We spent a lot of time discussing the vagaries of quoting the arguments, but we didn’t discuss the command itself in any detail.

In Bash, when you enter a command, that command can actually be one of two things, a builtin Bash command, or, an executable file which Bash will execute for you. You can see the list of builtin commands on BSD-style Unixes (including OS X) with man builtin. On Linux, you need to navigate to the SHELL BUILTIN COMMANDS section of the VERY long Bash man page for the same information.

When you enter a command in Bash, the first thing it does is figure out whether or not the command is a builtin. If it is a builtin then Bash just does whatever it is you asked. Where things get interesting is when you enter a command that is not a builtin. What Bash does then is interpret the command as a request to run an executable file with that name. If Bash finds such a file it runs it, and if not, it gives an error like:

The obvious question is, how does Bash find the executable files to run? This is where PATH comes in.

Before we continue, let’s print out the current value of PATH with the echo command and $ operator we learned about in the previous instalment:

You should see a value that looks something like the following (though yours may well be shorter, mine is extra long because I use MacPorts to install Linux command-line tools onto my Mac):

The value of PATH is a : delimited ordered list of folders.

Each time you enter a command that is not a builtin, what Bash does is search each of the folders listed in PATH in order until it finds an executable file with the name you entered. The order is important, if two folders in your path contain files with the same names, it’s the files in the folders nearest the front of the list that will get executed.

The which command can be used to show you which file will be executed when you use a given command, e.g.

Something people often find confusing is that many of the builtin commands are actually executable files, as can be demonstrated with which (which is itself a builtin):

What makes these commands special is that Bash does not use PATH to figure out where they are, it maps to them directly, so, even if you delete your PATH, the builtins will continue to work.

In fact, let’s do just that (in a safe way that won’t do any harm to your computer)!

We have now blanked the PATH environment variable in our command shell — note that we have ONLY altered the copy of PATH stored in this one command shell — all other command shells, including any new ones opened in the future, are totally unaffected by this change.

But we can’t do things like:

It’s not that the executable files have gone, or no longer work, it’s that our instance of Bash has lost the ability to find them because its PATH is blank. We can still run the executables by using their full paths, e.g.:

Before we continue, let’s restore our PATH to its normal value by closing this command shell and opening a new one.

When you get to the stage of writing your own scripts (or downloading other people’s scripts), you’ll probably want your scripts to run without needing to give the full paths to the scripts each time. As an example let’s create a new folder in our home directory and create a simple script within it:

Enter the following content into the file whereAmI and save:

Then make the script executable, and test it:

At the moment we have to enter the full path to whereAmI each time we want to use it. Let’s remedy that by adding our new myScripts folder to the end of our PATH:

Note that we have to include the current value of PATH in the new value we set for PATH or we would be replacing the PATH rather than adding to it. This is a very common pitfall, and the effect would be that all non-builtin commands apart from those in the one new folder would break. Note also that we used $HOME instead of ~ because you can’t use ~ in PATH.

Verify that PATH has been updated:

We can now use our script as a command without having to specify the full path:

Now, close your command shell, and open a new one, and try to use your script as a command again:

Why was the command not found? The answer is simply that the change we made to PATH in our previous shell’s environment vanished the moment we closed that shell. What we need to do is make a permanent change, and to do that we need to understand how Bash initialises its environment.

When SH or Bash are initialising they start the environment building process by sourcing the file /etc/profile. The out-of-the-box content of this file will be determined by your choice of OS. On my Ubuntu server /etc/profile contains the following:

While OS X comes with a much shorter and easier to understand /etc/profile:

In this series we are focusing on OS X, so we’ll only look at how OS X initialises its Environment in detail.

What the above OS X /etc/profile does is two things:

As you might guess from the name, path_helper is a utility for constructing the default path. You can run it yourself to see what it produces:

If you’re curious, you can learn how it builds the path by reading the relevant man page with man path_helper. The skinny version is that it reads the system-wide default path from /etc/paths, and then adds any extra paths defined in files contained in the folder /etc/paths.d. To have a look at the default paths you can use:

(On a default OS X install the last command will fail because there are no files present in /etc/paths.d by default)

If we wanted to add our new scripts folder to the default path for all users on the system we could edit /etc/paths, or add a new file with the path or our scripts folder in /etc/paths.d, but don’t do that! These system-level paths should only be used for system-level things, as we’ll see shortly, there is a better way to make user-specific customisations.

For completeness, let’s have a look at /etc/bashrc.

What’s going on here is mostly OS X-specific customisations to Bash. The Ubuntu equivalent to this file is /etc/bash.bashrc, and just like with /etc/profile, the contents of the file is completely different to what you get on OS X.

There is really only one line in this file that I want to draw your attention to, and then, only as a preview of the next instalment. The line in questions is:

It looks like gobbledegook, but, it’s actually the line that sets the format of the command prompt. \h is the hostname, \W is the current folder, and \u the current user. You should recognise that as the format of the command prompt in your OS X Terminal windows. We’ll look at this in more detail next time.

So far there are two files doing the customisation of Bash for us, /etc/profile and /etc/bashrc. These are both system files, and if you try to edit them as a regular user you’ll find you don’t have permission:

It’s with good reason that you don’t have editing rights to these files — you could do serious damage to your system if you make a mistake in these files. Unless you really know what you are doing, never edit either of them!

The system-level configuration files are only the first half of Bash’s startup procedure. When a new Bash process has finished running those files, it moves on to a new phase where it checks the user’s home directory for certain specially-named files.

For reasons we won’t go into now, if you’re a Linux user the user-level file to create/edit is ~/.bashrc, while Mac users should create/edit ~/.bash_profile. (If you really care about why there is a difference, you can have a read of this short article).

So, any customisations we wish to make to Bash on our Macs should be made in ~/.bash_profile. Let’s go ahead and set a custom PATH that includes the folder we created earlier:

Enter the following and save the file (BE CAREFUL TO GET IT RIGHT):

To test your newly customised environment simply open a new Terminal. If you’ve done everything right you should see the warning message telling you ~/.bash_profile has been executed, and, your path should have been updated to include ~/myScripts. You can verify this by running:

You should use ~/.bash_profile to make all your Bash customisations, not just customisations to your PATH. In the next instalment, we’ll have a look at some of the other customisations you might like to configure in your ~/.bash_profile file.

Aliases are basically command shortcuts. When used as a command, the alias gets expanded to a longer string by Bash before execution. Aliases can contain arguments as well as commands. Aliases can be used to create short mnemonics for long complex commands or to add default arguments to existing commands.

Let’s start with an example of the first kind. The command ls -l often results in very long output, so it would be nice to be able to automatically run this output through a pager so we can see it one page at a time instead of having to scroll back up to find the start of the output. As we’ll learn in the next instalment, you can do this using the | (pronounced ‘pipe’) operator. To run the output of ls -l through less the command is:

Let’s create a short alias for this command, ll:

As you can see, you create an alias using the alias command. The bit before the = is the shortcut, and the bit after the = is what it will expand into. If the expansion contains spaces or other special characters then either the entire replacement needs to be quoted, or each special character needs to be escaped. The vast majority of aliases you create will contain spaces and/or at least one special character, so it’s best to get into the habit of always quoting your aliases. In the above example, both the spaces and the | would need to be escaped, so the un-quoted version of the above alias would be the following difficult to read mess:

Another common use of aliases is to add default arguments to existing commands. Let’s look at two common examples of this.

By default, the ls command does not clearly distinguish the types of the items it lists. ls -F makes things quite a bit clearer by adding a trailing / to all folders in the listing, and -G makes things even clearer still by using colours to distinguish different types of files (remember that on Linux it’s --color rather than -G). To have ls always show trailing slashes and use colours we can alias ls to ls -FG (or ls -F --color on Linux):

Secondly, we can use aliases to make commands that might unintentionally remove or override files behave in a safer manner by automatically adding the -i flag to rm, cp & mv:

Note that you can use the alias command to add multiple aliases at once. If you set these aliases then rm, cp, and mv will ask for your permission before deleting or overriding a file. If you’re manipulating many files at once this will get tedious, so remember that even if you do this, you can over-ride the -i by adding a -f (for force) to the commands. This means you have to be explicit about deleting or overriding files, instead of the commands implicitly assuming you are happy to have the files destroyed.

To see a list of all currently defined aliases, simply run the alias command without any arguments:

To remove an alias, use the unalias command with the shortcut to be removed as an argument. E.g. to remove our ll alias, run:

Aliases are part of the Bash environment and are not persistent. If you create an alias and then close your command shell it will be gone. To make aliases permanent we need to add them into our Bash startup scripts. As we learned in the previous instalment, for Mac users that means adding our aliases to ~/.bash_profile, while Linux users should add their aliases to ~/.bashrc.

If we include the examples from the previous instalment an updated ~/.bash_profile file to include the ll alias would look something like:

As we discussed right back in the first instalment, the format of your command prompt varies from system to system. This variation is not just between command shells, but also within command shells, because many allow customisations to the format of the prompt. Bash is one of the command shells that support command prompt customisation.

In BASH, the format of the prompt is defined in the shell variable PS1. You can see the current format used with the command:

On OS X the default value of PS1 is \h:\W \u\$, giving prompts that look like:

On the RedHat and CentOS variants of Linus the default is [\u@\h \W]\$, which give prompts that look like:

From the above, you should be able to deduce that any letter in the code prefixed with a \ is replaced with a value. Below is a list of some of the variables at your disposal:

As an example, let’s create a very descriptive Bash prompt:

Like with any shell variable, any changes we made to PS1 are confined to our current command shell. Opening a new Terminal window will restore PS1 to its default value. If we want to make the change permanent, we need to add it to our bash startup file (~/.bash_profile on OS X or ~/.bashrc on Linux).

A ~/.bash_profile file setting all the customisations we’ve discussed in this instalment and the previous instalment would look like:

Personally, I like to keep my prompts set to their default values — it helps me recognise the OS I’m on at a glance.

Over the past few instalments, we have looked at what the shell environment is, how we can manipulate shell and environment variables and aliases. We looked at two variables in particular, PATH and PS1. We also discussed Bash aliases, and how to make permanent changes to your Bash environment, allowing you to customise your PATH and prompt and define aliases.

This is where we’ll leave the concept of the environment for now. In the next instalment, we’ll move on to look at what I jokingly call ‘plumbing’ — how commands can be chained together, and how files can be used for input to and output from commands.

Before we can get into the nitty-gritty of chaining commands together, we need to introduce a new concept, that of a stream of data. Quite simply, a stream is a sequential flow of data — everything that goes in one end comes out the other, and it always comes out in the same order it went in.

In a Unix/Linux environment there are three standard streams:

Many Unix/Linux commands can take their input from STDIN, and just about every command will write its output to STDOUT and/or STDERR. This allows commands to be strung together by the simple act of redirecting these streams — if you redirect the output of one command to the input of another, you have chained them together!

Remember that every process has its own environment, and therefore, its own version of the three standard streams. The redirection operators alter the copies of these three variables within individual processes, so, from the command’s point of view, it always reads from STDIN, and always writes to STDOUT and/or STDERR, but where those streams flow to is determined by their environment.

Bash provides a number of stream redirection operators:

The | operator is probably the most used of the three, as it allows straight-forward command chaining. It’s also the simplest. For that reason we’re going to focus solely on the | operator in this instalment, leaving the file-related operators until the next instalment.

To facilitate an example, let’s introduce a simple command for counting words, characters, or lines — wc (word count).

A quick look at the man page for wc shows that it counts lines when used with the -l flag. Something else you might notice is that the command OPTIONALLY takes an argument of one or more file paths as input. This means we can count the number of lines in the standard Unix/Linus hosts file with the command:

But why is the list of file paths optional? What could the command possibly count if you don’t point it at a file? The answer can be found further down in the man page:

In other words, wc will read its input from STDIN if no file is specified, and, no matter what the input source, it will write its results to STDOUT.

Now that you know about the three standard streams, you’ll start to see them in man pages all over the place. E.g., you’ll find the following in the man page for ls:

Let’s combine our knowledge of ls, wc, streams, and stream redirection to build a command to determine how many files or folders there are on our Desktop:

NOTE — the | operator ONLY redirects STDOUT to STDIN, it has no effect on STDERR, so if there is an error generated by the ls command that error message will go to the screen, and not to the wc command. To illustrate this point, let’s try to count the number of files in a folder that does not exist:

We see the error message generated by ls on the screen, and it is one line long, but, wc never saw that line because it was printed to STDERR, so instead of saying there is 1 file in this fictitious folder, it tells us, correctly, that there are no files in that folder.

You’ll often hear Unix nerds tell you that in Unix, everything is a stream. This is because deep down, Unix (and Linux) treat files as streams. This is especially true of some special files which really are streams of data rather than pointers to data on a hard disk. Special files of this type have a c (for character special file, i.e. a character stream) as the first letter in their permission mask in the output of ls -l. E.g.:

There are many such files in /dev/ on a Unix/Linux machine, but it is a VERY bad idea to write to or read from any you don’t understand. These files are generally connected directly to some piece of hardware in your system, including low-level access to your hard drives, so you could destroy important data very easily. (Thankfully you need to be root to write to the hard drive connected ‘files’).

There are a few safe special files that I want to mention though:

As an example, let’s use /dev/urandom to generate 10 random characters.

Before we can begin there are two complications that we need to understand. Firstly, these special streams have no beginning or end, so we have to be sure to always read from then in a controlled way — if you ask a command like cat to print out the contents of such a file it will never stop, because cat continues until it reaches the end of file marker, and these special ‘files’ have no end! Also, /dev/urandom does not output text characters, it outputs binary data, and while some combinations of binary data map to characters on our keyboards, most don’t, so we will need to convert this stream of binary data into a stream of text characters.

We can overcome the first of these limitations by using the head command we met in part 11 of this series. Previously we’ve used head to show us the first n lines of a file, but we can use the -c flag to request a specific number of characters rather than lines.

The second problem can be overcome with the base64 command, which converts binary data to text characters using the Base64 encoding algorithm. A quick look at the man page for base64 shows that it can use streams as well as files:

putting it all this together we can assemble the following command:

This is nearly perfect, but, you’ll notice that the output always ends with ==, this is the Base64 code for ‘end of input’. We can chop that off by piping our output through head one more time to return only the first 10 characters:

This will print only the 10 random characters, and nothing more. Since this command does not print a newline character, it leaves the text stuck to the front of your prompt which is messy. To get around this you can run echo with no arguments straight after the above command:

Note we are NOT piping the output to echo, the symbol used is ;, which is the command separator, it denotes the end of the previous command and the start of the next one, allowing multiple separate commands to be written on one line. The commands will be executed in order, one after the other.

Finally, because we need to use the same number of characters in both head commands, we could use command-line variables to make this command more generic and to make it easier to customise the number of characters:

In this instalment, we have introduced the concept of streams, particularly the three standard streams provided by the environment, STDOUT, STDERR, and STDIN. We’ve seen that these streams can be redirected using a set of operators and that this redirection provides a mechanism for chaining commands together to form more complex and powerful commands. We’ve been introduced to the concept of using files as input and output, but have not looked at that in detail yet. We’ve also not yet looked at merging streams together, or independently redirecting STDOUT and STDERR to separate destinations — this is what’s on the agenda for the next instalment.

So far we’ve been redirecting the output of one command to the input of another, but we can also use files as the source of our streams using the < operator. The operator works by connecting the content of the file to the right of the operator to STDIN in the command to the left of the operator.

As an example, let’s use the wc command we learned about in the previous instalment to count the number of lines in the standard Unix/Linux hosts file again, but this time, we’ll use the < operator:

Because the wc command can take its input either from STDIN or from one or more files passed as arguments, the above command achieves the same things as the command we saw in the previous instalment:

The wc command is not in any way unusual in this, the vast majority of Unix/Linux commands which operate on text or binary data can accept that data either from STDIN or from a file path passed as an argument. For example, all the following commands we have met before can take their input from STDIN rather than by specifying a file path as an argument.

It’s easier to just pass paths as arguments through, hence the < operator is probably the least-used of the stream redirection operators. However, just because it’s the least-used, doesn’t mean it’s never needed! There are some commands that will only accept input via STDIN, and for such commands, it’s vital to have an understanding of the < operator in your command-line toolkit. In my professional life, the one example I meet regularly is the mysql command, which does not take a file path as an argument (Note that MySQL is not installed by default on OS X). To load an SQL file into a MySQL database from the command line you have to do something like:

While you’re not likely to find yourself using files as input streams very often, you are quite likely to find yourself using files as output streams. There are two operators which perform this task, and the difference between them is subtle but very important.

The first of these operators is >. This operator directs the output from the command to its left to a NEW file at the path specified to its right. If a file already exists at the specified path, it will be REPLACED. This means that after the command finishes the file will only contain the output from that one command execution. Because of this overwriting behaviour, always use the > operator with great care!

The second of the file output operators is >>. This operates in a very similar way to >, directing the output of the command to its left to the file specified to its right, but with one very important difference — if a file already exists at the specified path it will not be replaced, instead, the new output will be appended to the end of the file. This makes the >> operator much safer, but, it means you cannot easily see which content in the file came from the latest execution of the command.

As a practical example, let’s revisit our command for generating random characters from the previous instalment, but this time, rather than outputting the random characters to the terminal, we’ll send them to a file:

We can verify that we have generated 256 random characters by using the wc command with the -c flag to get it to count characters:

If we re-run the command we can verify that the file still only contains 256 characters because the original version of the file was simply replaced by a new version because we used the > operator:

Now let’s change things up and generate 8 random characters at a time, but append them to a file with the >> operator:

As before we can verify the amount of characters in the file using:

Now, each time we repeat the command we will add 8 more characters to the file rather than replacing its contents each time:

So far we have always operated on STDIN and STDOUT. This is true for our use of all four of the operators we’ve met so far (|, <, > & >>). However, there is often a need to control other streams, particularly STDERR.

Unfortunately, we now have no choice but to take a look at some rather deep Unix/Linux internals. We’ve already learned that each process has a reference to three streams within its environment which we’ve been calling by their Englishy names STDIN, STDOUT & STDERR. We now need to remove this abstraction. What the process’s environment actually contains is something called a “File Descriptor Table”, which contains a numbered table of streams. Three of these streams are created by default, and always present, but processes can add as many more streams as they wish. Within the file descriptor table, all streams are referenced by number, rather than with nice Englishy names, and the numbers start counting from zero. To make use of the file descriptor table, we need to know the following mappings:

If we were to define our own streams, the first stream we defined would get the file descriptor 3, the next one 4 and so on. We are not going to be defining our own streams in this series, so all we have to remember is the contents of the small table above.

We can use the numbers in the file descriptor table in conjunction with the <, > & >> operators to specify which streams the files should be connected to. For example, we could re-write the examples from today as follows:

Since these operators use 0 and 1 by default, you’d never write the above commands with the 0s and 1s included, but, you have to use the file descriptor table to redirect STDERR.

Let’s revisit the command we used to intentionally trigger output to STDERR in the previous instalment:

This command tries to count the files in a non-existent folder. Because the folder does not exist, the ls command writes nothing to STDOUT. Because the | only operates on STDOUT the wc command counts zero lines, and the error message which was written to STDERR is printed to the screen. We could now redirect the error message to a file as follows:

Note that we have to redirect STDERR before the | operator, otherwise we would be redirecting STDERR from the wc command rather than the ls command.

You can use multiple redirects in the one command. For example, you could use one redirect to send data from a file to a command, and another redirect the send the output to a different file. This is not something you’ll see very often, but again, it’s something MySQL command-line users will know well, where this is a common construct:

You might also want to send STDOUT to one file, and STDERR to a different file:

Unlike in the Ghostbusters universe, in the Unix/Linux universe, it’s often desirable to cross the streams — i.e. to merge two streams together. The most common reason to do this is to gather all output, regular and error, into a single stream for writing to a file. The way this is usually done is to divert STDERR to STDOUT and then redirect STDOUT to a file.

In order to construct a meaningful example, let’s preview a command we’re going to be returning to in great detail in a future instalment, the find command. This command often writes to both STDOUT and STDERR during normal operation.

As its name suggests, the find command can be used to search for files that meet certain criteria. If you run the command as a regular user and ask it to search your entire hard drive or a system folder, it will run into a lot of permission errors interspersed with the regular output as the OS prevents it from searching some protected system folders. As a simple example, let’s use find to search for .pkg files in the system library folder:

Almost straight away you’ll see a mix of permission errors and files with the .pkg extension. The key point is that there is a mix of errors and results. If we try to capture all the output with the command below we’ll see that the error messages are not sent to the file, instead, they are sent to our screen (as expected):

As we’ve just learned, we could send the errors to one file and the files to another with:

But how could we capture all the output together?

To do this we need to introduce one more operator, the & operator. This operator allows a file descriptor table entry to be used in place of a file path by the <, > & >> operator. Hence, we can redirect STDERR (2) to STDOUT (1) as follows:

This has no noticeable effect until you send STDOUT to a file, then you can see that we have indeed diverted STDERR to STDOUT, and the combined stream to a file:

IMPORTANT: notice the counter-intuitive ordering of the above command, although the redirect happens first, it MUST be specified at the end of the command or it will not work.

There is much much more than can be done with streams, but, this is all most people are likely to need in their day-to-day life on the command line, so we’ll stop here before we confuse everyone too much 🙂

We have now seen how streams, and a process’s file descriptor table, can be manipulated using the stream redirection commands to chain commands together and funnel input and output to and from files in a very flexible way. This ability to manipulate streams opens up a whole new world to us, allowing us to build up complex commands from simple commands. This ability to chain commands is a pre-requisite for our next topic — searching at the command line.

If you want to get scientific about it, regular expressions are languages for describing regular grammars, which are the simplest type of grammar in the Chomsky Hierarchy. You could easily dedicate an entire university-level course to explaining the meaning and importance of that last sentence, and in fact, if you take a degree in Computer Science, you will! However, that’s not much use to us for the purpose of this series. In effect, what it means is that regular expressions provide a means for representing patterns that can be described as a series of elements following one after the other. That means regular expressions can do a lot. They can find all currency amounts in a document (a currency symbol followed by either an integer or a decimal number), they can find percentages (an integer or decimal number followed by a percentage symbol), they can find temperatures (an integer or decimal number followed by a C, F, or K), and so on. That includes quite complex things like recognising URLs, which could be described something like:

That description is actually incomplete because you would need to describe what some of those parts mean in more detail before you could write a regular expression for them, but that’s no problem because those parts too can be described as a series of elements following each other. For example, you’d have to further break down the domain name part into something like:

The key point is that if you can describe a pattern as a series of elements that follow one after the other, then you should be able to write a regular expression to represent that pattern.

So, regular expressions are without a doubt powerful, but, they are not all-powerful — there are entire classes of problems regular expressions are powerless to help with. In fact, to get scientific again for a moment, there are three entire grammar classes in the Chomsky Hierarchy that REs are powerless to help with. In practical terms that means that REs can’t help when some kind of memory is needed to know what has gone before, or when the elements in the pattern can be arbitrarily ordered and/or nested. For example, it would be impossible to write a regular expression to test if an arbitrary piece of text contained a matched set of arbitrarily nested brackets, because, to know if a given closing bracket is or is not valid, you need to know how many opening brackets have proceeded it. Also, REs can’t be used to validate something like XML (or HTML for that matter), because tags can come in any order, and be validly nested in all sorts of different ways.

Not understanding the limits of REs leads to a lot of frustration, and a lot of very unreliable code. If you can’t describe it as a series of elements that follow each other in a given order, a regular expression is not the answer!

The fact that many programmers don’t understand the limitations of regular expressions has led to the incorrect maxim that if you have a problem and try to solve it with regular expressions you then have two problems, your original problem and a regular expression.

Regular expressions can look very intimidating, but, once you know the language they are written in, they are actually very simplistic things. Think of it as a mathematical equation, until you know what all the symbols mean, it’s a big pile of intimidating gobbledegook, but, once you understand the meanings of the symbols, you can work your way through an equation logically.

The following apparent gibberish is a regular expression describing the domain name pattern described above:

For now, that looks horrific, but, when we’ve finished this instalment and the one after, I promise it’ll make sense!

Also, I promise the following is a really funny joke — when you get it, you’ll know you get REs! (I have this on a T-shirt, and it works as a great nerd ––test.)

Just like there is no one programming language, there is no one language for regular expressions. So, that leads to an obvious question, which type of RE should we learn? Because this series is all about the Terminal, the answer is actually very easy, there’s really only one choice that makes sense, but, it happens to be a choice that conveniently gives us a very solid base to build from for other uses of REs.

Let’s start with some context. Firstly, when it comes to regular expressions you can’t ignore my favourite scripting language, Perl. Perl was developed for the purpose of processing text, which means pattern matching is at the very core of its DNA. The official backronym for Perl is the Practical Extracting and Reporting Language, and the joke backronym is the Pathologically Eclectic Rubbish Lister. Either way, Perl is all about extracting information from textual data, so it’s all about pattern matching.

Because Perl has pattern matching so deeply embedded within its being, it should come as no surprise that the gold standard for regular expressions are so-called Perl-Style Regular Expressions, which you’ll often find referenced in man pages and other documentation as PCRE, which stands for Perl Compatible Regular Expression. What we’re going to learn is not exactly PCRE, but, a sub-set of PCRE called POSIX ERE. The fact that POSIX ERE is a subset of PCRE means that everything we learn will be useful to us in any application that uses PCRE, which means we can transfer what we learn here to a LOT of other contexts, including the two common web programming languages PHP and JavaScript (which both use PCRE for their regular expression implementations).

Why POSIX ERE? In fact, more fundamentally, what is POSIX?

POSIX stands for Portable Operating System Interface, and it’s the reason that the things we learn in this series are so surprisingly portable. POSIX is the standard that unites most of the flavours of Unix and Linux and gives us a common foundation to work off of. Not all our *nix operating systems are POSIX certified, but they are all, to a very very high degree, POSIX compliant. OS X is actually POSIX certified, but Linux is not, it just implements pretty much the entire POSIX standard. POSIX covers many things, from how file systems should be presented, to a core set of terminal commands that are the same across all POSIX OSes, to a large set of programming APIs that can be used to create apps that run on all POSIX systems, to a portable regular expression syntax.

Actually, POSIX specifies two regular expression languages, POSIX Basic Regular Expressions (BRE), and POSIX Extended Regular Expressions (ERE). The reason there are two is that POSIX is literally decades old, and regular expressions have come a long way since the BRE syntax was defined. When it comes to the simple stuff, BRE and ERE are the same, but, when it comes to more complex stuff, specifically cardinalities and grouping, they are not compatible. For these advanced features, BRE is not PCRE compatible, but ERE is, making it the best kind of RE for those exploring the terminal.

For all the examples in this series, we are going to use ERE, and we are only going to use command line tools that understand ERE. However, it’s important to know that BRE exists, because you’ll see both BRE and ERE mentioned in many man pages, and, some terminal commands default to BRE for legacy reasons, but can accept ERE if a certain flag is passed.

The only way to really learn regular expressions is through practical examples, so, for this instalment and the next, we’ll be using the egrep command to search the standard Unix words file for words that match a given pattern. We’ll be looking at the egrep command in more detail later in the series, but for now, all we need to know is that egrep can be used with two arguments, the first, a regular expression in POSIX ERE format, and the second the path to a file to search. egrep will print each line that contains text that matches the given pattern. It will not print just the text that matches the pattern, it will print the entire line that contains the match.

The standard Unix words file is a text file containing a list of valid English words, one word per line. On OS X and Ubuntu Linux, the file is located at /usr/share/dict/words, though on some Unix/Linux variants you’ll find it at /usr/dict/words instead.

In this instalment we’re going to start with the simpler parts of the ERE language, and, in fact, everything we learn today will be valid ERE, BRE, and PCRE, so it will apply very very widely indeed.

Ordinary characters represent themselves in a pattern, so the POSIX ERE to represent the letter a is simply:

Similarly, the RE to represent the character t followed by the character h is simply:

Let’s start with a simple example — finding all words that contain a double e in the words file. Remember, the egrep command prints any line from the input file that matches the specified pattern, so, to find all words with a double e you could use the following command:

Let’s take things up a notch, and include line boundaries in our pattern. The special character ^ represents start of line when used at the start of a regular expression (it can have other meanings when used elsewhere as we’ll see later). Its opposite number is the special character $, which represents end of line.

So, the following command will find all words starting with the character b:

Similarly, the following command will find all words ending in the three letters ing:

*Note:* you may have noticed that I’ve been single-quoting the pattern in all the examples. This is often not necessary, because many patterns don’t contain Bash special characters, but, some do, including the one above, which contains the dollar symbol. If the string had not been single-quoted, we would have had to escape the dollar symbol which would be very messy. My advice would be to get into the habit of always single-quoting regular expressions, it’ll save you a lot of frustration over time!

Something else that’s very important is the ability to specify a wild-card character. We can do that using the period character, which you should read in an RE as any one character.

As an example, let’s say you’re stuck on a thorny crossword puzzle, and you need a word that fits into something e something something f something. You could use the following terminal command to find a list of possible answers:

Something to notice in the above command is that the specific pattern we are looking for is bounded by a ^ and a $, this is to ensure we don’t get longer words that contain the pattern returned. If you run the command again but leave those symbols out you’ll see that you get a lot of unwanted results (over 900 on OS X).

The last thing we’re going to look at in this instalment is character classes, these are used to match a single character against multiple options. You can think of everything inside a character class as being a big list of "ors". Character classes are enclosed inside square brackets, so, you should read the character class below as a or b or c or d or e or f:

As an example, let’s search for all four-letter words starting with a vowel:

You can also use the minus sign within character classes to specify ranges of characters. Some commonly used ranges include:

You don’t have to stick to those common ranges though, you can use subsets of them, and you can use multiple ranges within a single character class.

As an example, the regular expression below matches valid MAC addresses in OS X (and Linux) format. On POSIX OSes like OS X and Linux, MAC addresses are represented as a series of six two-character lower-case hexadecimal numbers separated by colon symbols, so, they could be matched with the following regular expression:

The above RE will work, but it’s quite unwieldy and full of repetition, you might imagine there’d be a simpler, more efficient way of representing this pattern, and you’d be right! I’ll stop here and leave the following as a teaser for the next instalment, the above ungainly 102-character RE can be reduced to just 29 characters using two important new concepts, cardinality and grouping.

As we have already seen, character classes can be used to specify a list of allowed characters. We’ve seen that you can simply list the characters that are allowed one after the other, and, that you can use the - operator to specify a range of characters. Something else you can do with a character class is invert it, in other words, have it match every character except the ones you list. To do this, you must start the character class with the ^ symbol.

For example, the following command will find all five-letter words that don’t start with a vowel:

Notice that the meaning of the ^ symbol changes depending on where it is used, outside character classes it means start of line, and inside character classes, it means not any of the following.

When we’re describing patterns in English, we’ll often find ourselves using the word or, so it’s not surprising that there is a POSIX ERE operator to allow us to search for one pattern, or another (or another, or another …​). The pipe symbol (|) means or in POSIX ERE. This symbol has a different meaning in Bash (it’s one of the stream redirection operators), so it’s vital you quote any RE containing the | symbol.

As an example, the following command will search the standard Unix words file for all five-letter words starting in th or ending in ing:

It’s often very helpful to be able to group together a part of a pattern, effectively defining a sub-pattern. To do this, surround the sub-pattern in round brackets (aka parentheses). We can do this to limit the scope of an or operator, or, as we’ll see shortly, to define which parts of a pattern can and cannot be repeated.

As a simple example, the following command will find all seven letter words starting with th or ab:

Many patterns contain some form of repetition, hence, regular expression languages generally contain a number of operators for expressing different ways in which a pattern, or a part of a pattern, can be repeated. There are four POSIX ERE operators that allow you to specify different amounts of repetition.

The first and most common is the * operator which you should read as zero or more occurrences of. The operator applies to just the single character or group directly to its left.

For example, the command below will find all words of any length starting with th and ending with ing, including the word thing which has no letters between the th and the ing:

The next operator we’ll look at is the + operator, which you should read as one or more occurrences of. Like the *, this operator also only operates on the single character or group directly to its left. If we repeat the above example but with a + rather than a *, then we are searching for all words starting in th and ending in ing with at least one letter between the th and the ing. In other words, the same results as before, but without the word thing, which has zero letters between the th and the ing:

The last of the simple cardinality operators is the ? operator which you should read as either zero or one occurrence of or, more succinctly optionally. Again, like the * and + operators, this operator also only operates on the single character or group directly to its left.

As an example, the following command finds all words that end in ing or ings:

The above returns both winning, and winnings.

The first three cardinality operators will usually give you what you need, but, sometimes you need to specify an arbitrary range of times a pattern may be repeated, in which case, you’ll need the final cardinality operator, {}. This operator can be used in a number of ways:

Like the other three cardinality operators, this operator also only acts on the one character or group directly to its left.

As a first example, the following command lists all 10 letter words:

As another example, the following command lists all words between 10 and 12 characters long (inclusive):

Finally, the following command list all words at least 15 letters long:

We’ve now seen all the symbols that have a meaning within a POSIX ERE (except for one which we’ll see in a moment), so, we know that all the following characters have a special meaning:

If you want to include any of these characters in your patterns, you have to escape them if they occur somewhere in the pattern where they have a meaning. The way you do this is by preceding them with the escape character, \.

If you wanted to match an actual full-stop (aka period) within your RE, you would need to escape it, so, an RE to match an optionally decimal temperature (in Celsius, Fahrenheit, or Kelvin) could be written like so:

Similarly, an RE to find all optionally decimal dollar amounts could be written as:

However, we could write this more clearly by using the fact that very few characters have a special meaning within character classes, and hence don’t need to be escaped if they are used in that context:

As a general rule, this kind of notation is easier to read than using the escape character, so, it’s generally accepted best practice to use character classes where possible to avoid having to escape symbols. This is of course not always possible, but when it is, it’s worth doing in my opinion.

As well as being used to escape special characters, the \ operator can also be used to match some special characters or sets of characters, e.g.:

Note that the above is not an exhaustive list, these are just the escape sequences you’re most likely to come across or need.

Given the above, we could re-write our regular expressions for temperatures and dollar amounts as follows:

We have also improved our regular expressions by surrounding them in word boundary markers, this means the RE will only match such amounts if they are not stuck into the middle of another word.

For our examples we have been using the standard Unix words file, which has one word per line, so, we have been able to use the start and end of line operators to specify the start and end of words. However, this would not work if we were searching a file with multiple words on the same line. To make our examples more generic, replace the ^ and $ dollar operators at the start and end of the patterns with \b (or the start with \< and the end with \>).

Given everything we now know, let’s revisit the example we ended with in the previous instalment, our big ungainly RE for matching MAC addresses:

We can now re-write it as simply:

The above will do everything our original RE did, but, actually, it’s not as good as it could be, because it really should specify that the entire MAC address should appear as a word, so we should surround it with \b escape sequences:

To really get practical, it’s time to stop using the standard Unix words file, and start using more complex input. Specifically, we’re going to use the ifconfig command which prints the details for all the network devices on a computer. We’ll be looking at this command in much more detail later in the series, but for now, we’ll just be using the command with no arguments. To see what it is we’ll be pattern-matching against, run the command on its own first:

So far we have been using the egrep command in its two-argument form, but, it can also be used with only one argument, the pattern to be tested, if the input is passed via STDIN. We’ll be using stream redirection to pipe the output of ifconfig to egrep.

Let’s use our new MAC address RE to find all the MAC addresses our computer has:

Having created an RE for MAC addresses, we can also create one for IP addresses (IPV4 to be specific):

We can use ifconfig and egrep again to find all the IP addresses our computer has:

So, let’s go right back to the examples we used at the very very start of all this. Firstly, to the RE for domain names:

Hopefully, you can now read this RE as follows:

And finally, to the RE that I promised was a funny joke:

You could read it as:

Or, you could read it as:

Given that Shakespeare’s 450th birthday was last month, it seemed appropriate to include this bit of nerd humour!

We’ve now covered most of the POSIX ERE spec, and probably more than most people will ever need to know, but if you’d like to learn more I can recommend this tutorial.

Regular expressions make sense when you want to search for things, so, it’s not surprising that you mostly find them in apps where searching is important.

You’ll very often find REs in advanced text editors (not in basic editors like TextEdit.app). Two examples are included below, the Advanced Find and Replace window in Smultron 6, and the Find dialogue in the Komodo Edit 8 cross-platform IDE (the two editors I do all my programming in):

Another place you’ll often find regular expressions is in apps for renaming files, for example, Name Mangler 3 or the bulk-renaming tool within Path Finder:

Thanks to NosillaCast listener Caleb Fong for recommending a great online RE testing tool which you can find at regex101.com

We’ve now learned enough about REs to move on to looking at command-line tools for searching for text in files, and files in the filesystem. This is what we’ll be moving on to next in this series.

The grep command goes back a very long way and has been the staple text-searching command on POSIX operating systems like Unix and Linux for decades. To this day it’s used millions of times each day for simple text searches. But, it has a shortcoming — it’s stuck in the past when it comes to regular expressions — grep pre-dates the invention of POSIX ERE! egrep is identical to grep except that it interprets patterns passed to it as POSIX EREs.

If you can, it’s probably best to get into the habit of always using egrep, and never using grep, but for those of us who’ve been around the block a few times, this could be asking for too much (old habits die hard!). What I usually do is use grep when I don’t need regular expressions, and egrep when I do. However, in this series, I’m going to follow my own advice and only use egrep.

For egrep, lines of text are atomic. In other words, egrep searches or filters text one line at a time, checking the entire line against the given pattern, and considering the whole line to match the pattern or not.

There are two basic ways in which egrep can be used — it can filter whatever is sent to via standard in (STDIN — see part 15) against a given pattern, or, it can search for lines matching a given pattern in one or more files.

Let’s start with the first use-case, using egrep to filter STDIN. When egrep is used in this mode it passes every line of text sent to STDIN that matches the given pattern to standard out (STDOUT) and ignores all others. If you send 5,000 lines of text to egrep via STDIN, and only 5 of those lines match the specified pattern, then only 5 lines will be passed to STDOUT (which is the screen unless the output is redirected elsewhere).

When content is redirected to egrep's STDIN, egrep only needs one argument — the pattern to filter on.

On Unix/Linux/OS X computers configured to act as servers, there will be a lot of log files being written to continuously, and sysadmins will very often need to filter those logs while troubleshooting an issue, or tweaking the server’s configuration. In my day-job as a Linux sysadmin, I do this all the time. Regardless of the log file to be filtered, the approach is the same, use tail -f to stream the log file in question to tail's STDOUT in real-time, then redirect that stream to egrep's STDIN with the pipe operator.

For example, on a Linux server running a BIND DNS server process, DNS log entries are mixed with other entries in the central system messages log (/var/log/messages). When debugging a problem with the DNS server, you don’t want to be distracted by all the other messages flowing into that log file. The following command will filter that log so that you only see messages from the DNS server process, which all start with the prefix named::

The log files on our personal computers are much quieter places, so PC users will rarely find themselves needing to filter log files. However, that doesn’t mean PC terminal users won’t find themselves wanting to use egrep to filter STDIN.

You can use egrep to filter the output from any command using the pipe operator. To generate a meaningful example we need a command that will generate a lot of formatted output at will. We’re going to use a command we’ll come back to in much more detail in future instalments, tcpdump. As its name suggests, tcpdump prints the details of every TCP packet that enters or leaves your computer to STDOUT. Every time your computer interacts with the network, tcpdump will generate output — in our modern connected world, that means tcpdump generates a LOT of output!

Firstly, let’s run tcpdump without filtering its output to see just how much network traffic there is on our computers:

tcpdump will keep capturing traffic until it is interrupted, so when you’ve seen enough, you can exit it with Ctrl+c.

It probably won’t be long until you start seeing packets fly by, but if it’s a bit sluggish, try checking your email or visiting a web page and the packets will soon start to fly!

Now, let’s say we want to watch what DNS queries our computer is making. Given that DNS queries are over port 53, and that your router is almost certainly your DNS server, we know that all DNS queries will be sent to your router on port 53. Before we construct the pattern to pass to egrep, we need to find the IP address of our router. We can do this by filtering the output from another command that we’ll be looking at in much more detail later, netstat. With the appropriate flags, netstat prints out our computer’s routing table, and the default route in that table is to your router, so filtering the output of netstat for a line starting with the word default will show the IP of your router:

When I run this command I get the following output:

This tells me that my router has the IP address 192.168.10.1 (yours will probably be different, very likely 10.0.0.1 or 192.168.0.1, my network is set up a little unusually).

Given this information I can now use egrep to filter the output of tcpdump to show me only my DNS queries with the following command:

You can construct a similar command for your computer by inserting your IP address into the above command. E.g. if your router’s IP address is 10.0.0.1 the command will be:

Notice that, rather confusingly, tcpdump adds the port number to the end of the IP as a fifth number.

Note that if we wanted to be really accurate with our regular expression, we would use something like the example below, which is more explicit, and hence much less prone to picking up the odd false positive:

When you execute your command, visit a few web pages, and watch as DNS queries are sent from your computer to your router. When I visit www.whitehouse.gov I get the following output:

This gives you some idea of just how many resources from disparate sources get pulled together to create a modern web page!

Let’s move on now to using egrep to search the contents of one or more files for a given pattern.

When using egrep to search file(s), it requires a minimum of two arguments, first the pattern to be searched for, and secondly at least one file to search. If you want to search multiple files, you can keep adding more file paths as arguments.

In this mode, egrep will filter the lines in the file in the same way it did when filtering a stream, but if you ask it to filter more than one file it will prepend any output with the name of the file the matching line came from. This is a very useful feature.

The vast majority of the examples we used in the previous two instalments used egrep to search the Unix words file. As a quick reminder, the following command will find all lines in the words file that start with the letters th:

A very common use-case for using egrep on a single file is to quickly check a setting in a configuration file. For example, on a Linux web server with PHP installed, you could use the command below to check the maximum file upload size the server is configured to accept:

On a server with a default PHP install that will return the following output:

Most of us are probably not running web server processes on our personal computers, so let’s look at a more practical example. On any POSIX OS (Linux, Unix or OS X), you can see what DNS server(s) are configured by searching the file /etc/resolv.conf for lines beginning with the word nameserver. The following command does just that:

So far we have only searched one file at a time, but you can point egrep at as many files as you like, either explicitly, or by using shell wild card expansion. For example, the command below looks for lines containing apple.com in all the log files in the folder /var/log:

egrep is a very powerful command that supports a staggering array of flags. We couldn’t possibly go through them all here. Remember, you can use the man pages to see everything egrep can do:

However, there are a few flags that are so useful they bear special mention. Firstly, to make egrep case-insensitive, you can use the -i flag. If you’re not sure of the capitalisation of the text you’re looking for, use egrep -i.

If you want to see the line numbers within the files for all the matches found by egrep you can use the -n flag.

And finally, the biggie, you can use the -r flag to recursively search every single file in a given directory. Be careful with this one — if you ask egrep to search too much, it will take a very long time indeed to finish!

In this instalment, we’ve seen how egrep can be used to filter a stream or to search one or more files for lines of text matching a specified pattern. This is very useful, and something sysadmins do a lot in the real world. In the next instalment we’ll be moving on to a different, but equally important, type of search — file searches. We’ll use the aptly-named find command to find files that meet one or more criteria. find supports a huge variety of different search criteria, including simple things like the name of the file, and more advanced things like the amount of time elapsed since the file was last edited. All these criteria can be combined to create powerful searches that will show all MS Office files in your Documents folder that were edited within the last week and are bigger than 1MB in size.

Regardless of the criteria you wish to use, the basic form of the find command is always the same, you first need to tell it where to look, then you tell it what criteria to use when searching:

The path is searched recursively (by default), so if you give a path of ~/Documents, it will search your documents folder and all folders within your documents folder. To search your entire computer, and all mounted drives, use a path of just /. To use the current folder as the base of your search use a path of . (which always means ‘the current folder’ as we learned in instalment 4).

To see a full list of all possible search criteria, you can, of course, read the manual entry for find with the command man find, but we’ll look at some of the more common criteria you’ll be most likely to need here.

We’ve already seen that we can use the -or and -not flags, but there is also a -and flag. In fact, if you don’t separate your criteria with a -or flag, a -and flag is implied.

The following example from above:

Is actually interpreted as:

We can even take things a step further and add subexpressions using ( and ) to start and end each subexpression (they can even be nested). Note that ( and ) have meaning in BASH, so they need to be either escaped or quoted. Since I find escaping makes everything hard to read and understand, I recommend always quoting these operators.

As a final example, the following command will find large PowerPoint presentations in your Documents folder, i.e. all files bigger than 100MiB in size that end in .ppt or .pptx.

In this instalment, we’ve seen that we can use the find command to search for files based on all sorts of criteria and that we can combine those criteria using boolean algebra to generate very powerful search queries. In the next instalment, we’ll discover that you can use the find command not only to search for files but to apply an action to every file it finds.

The find command is common to all POSIX operating systems, so it works on Linux, Unix, and OS X. OS X maintains an index of your files allowing quick searching in the Finder and via Spotlight. Because this index is kept up to date by the OS, it makes searching with Spotlight much quicker than searching with find. In the next instalment, we’ll also discover that OS X ships with a terminal command that allows you to use the power of Spotlight from the command line!

The final trick find has up its sleeves is that it can be used to execute a given command on all the files it finds. Depending on the command you choose to execute, this could be VERY dangerous indeed, so tread carefully! Definitely think twice before using the execute feature in `find`å to delete files!

To execute commands on the results of find you need to use the -exec flag in conjunction with the {} flag and either the ';' or '+' flags. The reason there are two end flags is that there are two modes in which find can execute commands. It can either execute the same command once on each file that was found or, it can execute the given command once only with all the found files as arguments.

On OSX the operating system indexes the files on your computer so you can quickly search them with Spotlight. Rather than having to search through all the files, Spotlight just searches the index, which makes it much faster than find when searching through big chunks of the file system.

Apple has very kindly exposed the power of Spotlight to the command line with the very useful mdfind command.

If you can type it into the Spotlight text box in the top-left of a Finder window and get back a list of files, you can pass the same query to mdfind, and it will find the same results. Note that I said Spotlight in the Finder, and not in the menubar — that was not by accident. mdfind is only for finding files, not for all the other fancy stuff you can do with Spotlight in the menu bar like the calculator function or the ability to get dictionary definitions. Another important caveat is that mdfind can only find files in folders indexed by Spotlight. If you add a drive or a folder to Spotlight’s ignore list, mdfind can never find files on those drives or folders.

The mdfind command is very easy to use:

For example, to find all PDFs on your computer you could use:

To find all PDFs that contain the word ‘internet’ you could use:

To find every PDF you have edited today you could use:

I haven’t been able to find a definitive list of all possible Spotlight search commands, but googling for ‘spotlight syntax’ will lead to useful articles like this one.

mdfind supports a number of arguments, and it can also search based on deep metadata, but I’m going to leave most of that as an exercise for the user — you can get the full documentation through the manual:

However, there are two flags I do want to draw attention to.

By default mdfind will search the entire Spotlight index, but you can ask it to only return results contained within a given folder using the -onlyin flag, for example, to find all PDFs in your Documents folder you could use:

Note that the search is always recursive, even when using the -onlyin flag.

The other flag I want to mention is -count. If this flag is set the number of matching files will be returned rather than the files themselves. So, to see how many music files you have in your Music folder you could use:

Or, to answer the eternal question of just how many apps you have installed:

We have now seen how to filter streams and search files with egrep, and we’ve learned how to search for files with find and mdfind. That brings us to the end of the searching topic, at least for now. The next big topic will be networking, but before we start into such a big topic we’ll take a break for a fun little tips and tricks instalment.

Hopefully, you’ll find at least some of these tips and tricks useful, and find yourself being a little more efficient on the command line.

In the next instalment, we’ll make a start on what will be a quite long series on networking. We’ll start by laying a theoretical foundation, and then get stuck in with a selection of network-related terminal commands.

The internet, and our home and work networks, are all TCP/IP networks. What we now call TCP/IP started life as ARPANET, a project run by the US Advanced Research Projects Agency, or ARPA (today this same institution is called DARPA, the D standing for Defense). For home users, the internet is a 1990s phenomenon, but DARPANET became operational way back in 1969.

The age of TCP/IP is a mixed blessing. On the one hand, it’s been around more than long enough to have most of the bugs shaken out of it. But, on the other hand, it was designed before anyone understood what it would become, so problems like security simply weren’t considered in the design. Security has had to be retrofitted afterwards, and that’s not exactly been a smooth process!

From our point of view as a user trying to communicate across our home network or the internet, the process always starts at the application layer. We are sending an email, or we are viewing a web page, or we are downloading a file, or we are playing a game, or we are chatting with friends, or whatever.

As a worked example, let’s look at what happens when you use your web browser to try to visit http://www.so-4pt.net/~bart/ttt23/.

Before you can visit that URL, I have to have put some digital ducks in a row on my end. Firstly, I have to have arranged for a DNS server to advertise to the world that my webserver has the IP address 46.22.130.125, and, I have to have the server with that IP address configured to act as a webserver.

On my server, a piece of software has to be running which speaks the HTTP protocol. We’ll call this piece of software the webserver process. This process has to have asked the OS to ‘listen’ on TCP port 80 (the standard HTTP port). That means that when any computer initiates a TCP connection to port 80 on my server, my server’s OS will hand that connection to my webserver process.

Assuming I have everything properly configured on my end, you open your favourite browser, and type http://www.so-4pt.net/~bart/ttt23/ into the address bar, and hit enter.

Your browser speaks HTTP, so it starts by formulating an HTTP request which will look something like:

It then has to figure out, based on the URL you typed, what IP address it should strike up a TCP conversation with!

To do that it asks your OS to do a DNS lookup on its behalf. For simplicity, let’s assume your OS had the answer in its cache, so it just gives the browser the answer that www.so-4pt.net maps to the IP address to 46.22.130.125. We’ll be looking at the DNS protocol in much more detail in a future instalment!

Your browser then asks your OS to open a TCP connection to port 80 on 46.22.130.125 on its behalf. Once your OS has done that, a TCP connection will exist between your browser and the webserver process on my server. As far as either end of that connection is concerned, any data written into the connection on their end will pop out on the other end exactly as it was written. This is a two-way connection, so the browser can send data to the webserver process, and the webserver process can send data back to the browser.

Your browser then sends the HTTP request it formulated to the webserver process on my server through the TCP connection your OS negotiated with my OS on your browser’s behalf. The webserver process receives the request, understands it because it speaks HTTP, and formulates a response in the form of an HTTP response that will look something like:

You might notice that this response includes the HTML that makes up the content of the page as well as some metadata in the form of an HTTP response code and some HTTP headers.

Once my webserver process has formulated this response somehow (in this case by reading a file from the hard disk on my server and copying its contents into the data part of the HTTP response) it sends the response to your browser through the TCP connection.

Your browser then interprets the response and acts accordingly. The 200 OK response code tells your browser the request was successful, and that the HTML it got back is not an error message or a redirect or anything like that, and is the HTML for the web page you requested. It then interprets that HTML and draws the resulting web page on your screen.

The key point is that your browser and my webserver app communicated using HTTP, and they were oblivious to how the data got from one to the other. All either side knew about the network was that TCP was at their service. How TCP made it possible for the data to go from your browser to my webserver is irrelevant to them both. Notice how neither the HTTP request nor the HTTP response contained either an IP address or a MAC address. Those things happen below HTTP in the stack, so they are irrelevant to HTTP.

The takeaway from this example is that at a logical level, both sides talked HTTP to each other. Logically, communication is always directly across the stack.

Of course, TCP isn’t magic, and the TCP implementations inside the operating systems on your computer and my server sent a whole load of IP packets over and back between each other to make that TCP connection happen.

The two TCP implementations were totally oblivious to how the data got between the two computers though. All our TCP implementations knew is that they could pass a packet down to the IP implementations within our operating systems and that our IP implementations would send the packet on their behalf.

So far, nothing has actually left our computers yet! All of this chatter between the network layers has been in software. It’s not until the IP implementations in our OS finally hand those packets down to the Ethernet implementation within our OSes that anything physical actually happens!

To understand how the packets actually move from one physical computer to another, let’s focus on what happens to just one single IP packet and let’s chose a packet that’s being sent from your computer to my server. Let’s assume your computer is using WiFi to connect to the internet and that you have an ADSL router.

We join our packet in the IP implementation of your computer’s OS. The first thing your computer’s IP implementation does is look at the IP address on the packet and compare it to the IP address range of your network. My server is not in your network, so the IP implementation concludes that the packet is not local and so must be routed. Because your network is properly configured, your OS knows the IP and MAC addresses of your router’s internal network interface. Your IP implementation takes the IP packet and hands it to the Ethernet implementation with your OS along with the MAC address it should be sent to, that of your router. Your OS’s Ethernet implementation then wraps some metadata around the IP packet to turn it into an Ethernet packet. Finally, using the driver software for your WiFi card, your OS’s Ethernet implementation transmits the Ethernet packet which contains the IP packet we are following as modulated radio waves.

Your router’s wireless card receives the radio waves, interprets them and hands the Ethernet packet up to the Ethernet implementation in your router’s firmware. Your router’s Ethernet firmware unwraps the packet and hands it up to your router’s IP firmware. Your router’s IP firmware looks at the destination address on the IP packet and sees that it’s destined for the Internet, so it hands the packet down to your router’s ADSL firmware which wraps the packet again and converts it into electrical impulses which run through the phone network to your ISP’s router.

Your ISP’s router strips off the ADSL wrapper and passes the packet up to its IP firmware. The router’s IP firmware looks at the destination IP and then decides which of your ISP’s many interconnections to other routers is best suited to this packet. It then wraps the packet in the appropriate Layer 1 wrapper for the connection it chooses and send it on its way. The next router along again strips off the Layer 1 wrapper, looks at the IP address, decides which way to route it, wraps it as appropriate and sends it on to the next router. This continues for as long as it takes for the packet to arrive at the router in the data centre where my webserver lives. I’m in Ireland and so is my server, and for me, it still takes 8 ‘hops’ for the packet to get from my router to the router in the data centre hosting my server. For people farther away it will be even more.

The router in my data centre will get the packet, unwrap whatever Layer 1 wrapper is around it, pass it up to its IP firmware, and look at the destination IP. Because my data centre and server are correctly configured, the router’s IP firmware will recognise that this packet is destined for a device on its local network, and know the MAC address of my server. It will pass the packet down to its Ethernet firmware one last time, asking it to send the packet to my server’s MAC address. The router’s Ethernet firmware will convert the packet to electrical signals, and send them down the UTP patch cable connected to my server.

My server’s network card will receive the Ethernet packet, unwrap it, and pass it up to my server’s OS’s IP implementation, which will remove the IP wrapper and pass the packet to TCP which will finally know what to do with the data inside the last wrapper, and eventually, those 1s and 0s will get to the webserver process as a part of the HTTP request.

The key point is how the layers interact. As we saw initially, logically the browser and the server talked HTTP to each other, but physically, the packet started in the application layer and moved down the layers to the link layer, then up and down between the link and internet layers many times as it moved across before finally being handed all the way up the layers back to the application layer when it arrived at my server. The diagram below illustrates this idea:

So, we followed the path of a single IP packet, how many IP packets were actually needed for this very simple HTTP transaction? The answer is 12 (see below). Only a subset of those 12 packets contained the actual data transmitted. The rest were IP packets sent by TCP in order to negotiate and un-negotiate the TCP connection, and to signal back and forth what packets had and had not been received safely.

This is a confusing and dense topic. I’d be very surprised if it all sank in on the first reading. For now, if just the following key points have sunk in, you’re golden:

In the following instalments, we’ll start to make our way up the stack, looking at each layer in more detail, and learning about terminal commands that interact with some of the protocols that make up each layer.

Later in the instalment, we’re going to refer back to results of the following command and explain it, but it takes time for the packets to be collected, so before we start, please open a Terminal window and leave the following command running for at least 5 or 10 minutes:

As we discussed in the previous instalment, the bottom of the four layers in the TCP/IP model is the Link Layer. Its function is to move a single packet of data from one device connected to a network to another device connected to the same network. Within our homes, we use Ethernet to provide our Layer 1 connectivity. We use two different implementations of Ethernet — we use Ethernet over copper wire (usually called Ethernet cables), and we use Ethernet over radio waves, commonly known as WiFi.

The Ethernet protocol addresses hosts on the network by their Media Access Control address, or MAC address. Every network card on your computer has a MAC address, regardless of whether it’s a wired or wireless Ethernet card. An Ethernet packet travelling through your network has a source and a destination MAC address.

Ethernet was designed to work on a shared medium — i.e., all network cards see all Ethernet packets travelling across the network. In normal use, a network card ignores all Ethernet packets that are not addressed to it, but a card can be instructed to pass all packets that reach it up to the OS, even those addressed to different MAC addresses. This is known as promiscuous mode.

The special MAC address ff:ff:ff:ff:ff:ff is used to broadcast an Ethernet packet to every device on the network. All network cards consider packets addressed to this special MAC address to be addressed to them, and pass that packet up to the OS, even when not in promiscuous mode.

You can see the MAC addresses associated with your Mac/Linux/Unix computer with the command:

(the -a stands for ‘all’ and is needed in many flavours of Linux to see network devices that are not currently active — OS X defaults to showing all devices, so the -a is optional on OS X.)

This command will list every network interface defined on your computer, both physical and virtual. The output is broken into sections with the content of the section tabbed in. Each section belongs to a different network interface, and the associated MAC address is labelled ether. The naming conventions for the network interfaces vary massively between different OSes, but one thing is always the same, they are confusing as all heck, and figuring out which name matches which physical network interface is non-trivial. Things are always confusing, but if you have a VPN installed they get even more confusing because VPNs are implemented using virtual network interfaces. On the whole, the simplest way to figure out which MAC address matches which device is to use your OS’s control panel GUI. On OS X that means the Network System Preference pane. To see which MAC address matches which interface, select a network interface in the left sidebar, then click Advanced…​ and navigate to the Hardware tab:

While the naming of network devices on Linux/Unix/OS X is confusing, there are some general rules that may help you figure out which device is which:

Realistically, if you’re running Linux or Unix the network interfaces you care about are probably the ones starting with eth, and for Mac users, it’s probably the ones starting with en.

To see all MAC addresses associated with your computer, regardless of which network card they belong to, you can use:

Because Ethernet uses a shared medium, it’s susceptible to congestion — if two network cards try to transmit a packet at the same time, they interfere with each other and both messages become garbled. This is known as a collision. When an Ethernet card detects a collision, it stops transmitting and waits a random amount of milliseconds before trying again. This simple approach has been proven to be very effective, but it’s Achilles heel is that it’s very prone to congestion. When an Ethernet network gets busy the ratio of successful transitions to collisions can collapse to the point where almost no packets actually get through.

With WiFi this shortcoming is unavoidable — a radio frequency is a broadcast medium, so collisions are always going to be a problem, and this is why it’s very important to choose a WiFi channel that’s not also being used by too many of your neighbours!

A copper cable is not the same as a radio frequency though! In order to create a copper-based Ethernet network, we need some kind of box to connect all the cables coming from all our devices together.

Originally these boxes had no intelligence at all — they simply created an electrical connection between all the cables plugged into them — creating a broadcast medium very much like a radio frequency. This kind of simplistic device is known as an Ethernet hub. An Ethernet network held together by one or more hubs is prone to congestion.

A way to alleviate this problem is to add some intelligence into the box that connects the Ethernet cables together. Rather than blindly retransmitting every packet, the device can interpret the Ethernet packet, read the destination MAC address, and then only repeat it down the cable connected to the destination MAC address. Intelligent devices like this are called Ethernet switches. In order to function, an Ethernet switch maintains a lookup table of all MAC addresses reachable via each cable plugged into it. Connections to hubs/switches are often referred to as legs or ports. These lookup tables take into account the fact that you can connect switches together, so they allow the mapping of multiple MAC addresses to each leg/port. If you have an eight-port switch with seven devices connected to it, and you then connect that switch to another switch, that second switch sees seven MAC addresses at the end of one of its legs.

Because switches intelligently repeat Ethernet packets, they are much more efficient than hubs, but congestion can still become a problem because broadcast packets have to be repeated out of every port/leg.

10 years ago you had to be careful when buying an Ethernet ‘switch’ to be sure you weren’t buying a hub by mistake. Thankfully, switches are ubiquitous today, and it’s almost impossible to find a hub.

There is a third kind of network device that we should also mention in this conversation — the router. A router is a device that has a Layer 1 connection to two or more different networks. It uses the Layer 2 IP protocol to intelligently move packets between those networks.

Our home routers cause a lot of confusion because they are actually hybrid devices happen to contain a router. The best way to think of a home router is as a box containing two or three component devices — a router to pass packets between your home network and the internet, an Ethernet switch that forms the heart of your home network, and, optionally, a wireless access point, which is the WiFi-equivalent of an Ethernet hub. Importantly, if it’s present, the wireless access point is connected to the Ethernet switch, ensuring that a single Ethernet network exists on both the copper and the airwaves. This means that an Ethernet packet can be sent from a wired network card to a wireless network card in a single Layer 1 hop — i.e. Layer 2 is not needed to get a single packet from a phone on your WiFi to a desktop computer on your wired Ethernet. Confusingly, while this single packet will pass through a device you call a router, it will not be routed — it will go nowhere near the router inside your home router, it will stay on the switch and the wireless access point inside your home router. The diagram below illustrates the typical setup:

The protocol that sits on top of Ethernet is the IP Protocol. The IP protocol moves a packet from one IP address to another, and it does so by repeatedly dropping the packet down to the link layer below to move the packet one hop at a time from a directly-connected device to another directly-connected device until it arrives at its destination. As a quick reminder, see the diagram below from the previous instalment:

Within our LAN, the Layer 1 protocol IP used to move a packet from one device on our LAN to another device on our LAN is Ethernet. Ethernet can only move a packet from one MAC address to another, and IP moves packets from one IP address to another, so how does the IP protocol figure out what MAC address matches to what IP address so it knows where to ask Ethernet to send the packet?

The Address Resolution Protocol, or ARP, is an Ethernet protocol that maps IP addresses to MAC addresses. It’s a supremely simplistic protocol. Whenever a computer needs to figure out what MAC address matches a given IP address, it sends an ARP request to the broadcast MAC address (ff:ff:ff:ff:ff:ff), and whatever computer has the requested IP answers back to the MAC address asking the question with an ARP reply saying that their MAC address matches the requested IP.

The command you’ve had running in the background since the start of this instalment has been listening for ARP packets and printing every one your computer sees. You should see output something like:

What you can see is a whole bunch of ARP requests asking the network who has various IP addresses, and, a few replies. If your entire home network uses WiFi you’ll probably see an approximately even number of requests and responses, but, if your network includes devices connected via wired Ethernet you should notice a distinct asymmetry between requests and responses, especially if your computer is connected to the network via Ethernet. This is not because requests are going un-answered, but rather because there is a switch in the mix, and that switch is only passing on Ethernet packets that are relevant to you. Requests are broadcast, so Ethernet switches send those packets to everyone, but responses are directed at a single MAC address, so those are only passed out the relevant port on the switch. In effect, what you are seeing is the efficiency of an Ethernet switch in action!

While we’re on the subject of efficiency, computers don’t send an ARP request each and every time they want to transmit an IP packet, ARP responses are cached by the OS, so new ARP requests are only sent when a mapping is not found in the cache. You can see the MAC to IP mappings currently cached by your OS with the command arp -an. You’ll get output something like:

The more devices on your LAN you are interacting with, the more mappings you’ll see.

Something you may have noticed about ARP is that it assumes all computers are truthful, that is to say, that no computer will falsely assert their MAC address maps to any given IP. This assumption is why ALL untrusted Ethernet networks are dangerous — be they wired or wireless. This is why the Ethernet port in a hotel room is just as dangerous as public WiFi. To intercept other people’s network traffic, an attacker simply has to send out false ARP replies and erroneously advertise their MAC address as matching their victim’s IP address. The attacker can then read the packets before passing them on to the correct MAC address. Users will not lose connectivity because the packets all get where they are supposed to eventually go, but the attacker can read and alter every packet. This technique is known as ARP Spoofing or ARP Poison Routing (APR) and is staggeringly easy to execute.

ARP is just the first example we have met of the Internet’s total lack of built-in security. It illustrates the point that the designers of the IP stack simply never imagined there would be malicious actors on their networks. If it didn’t have such detrimental effects on all our security, the naive innocence of those early pioneers would be very endearing!

This is the last we’ll see of Layer 1 in this series. In the next instalment, we’ll be moving up the stack to Layer 2 and the IP protocol — the real work-horse of the internet. In particular, we’ll be tackling one of the single most confusing, and most critical, networking concepts — that of the IP subnet. It’s impossible to effectively design or troubleshoot home networks without understanding subnets, and yet they are a mystery to so many.

At the IP level, packets are sent from one IP address to another. Every computer connected to an IP network has one or more IP addresses. For the purpose of this series we will be ignoring IPv6, so IP addresses are of the form n1.n2.n3.n4, where n1 to n4 are numbers between 0 and 255, E.g. 192.168.0.1.

Unlike MAC addresses, IP addresses are not hard-coded into our network interfaces. Instead, they have to be configured within the operating system. Historically this was always a manual process, but today it’s usually automated using the Dynamic Host Configuration Protocol, or DHCP. We’ll be looking at how DHCP works in detail later in the series. Also, a single network interface can have many IP addresses assigned to it, and a single computer can have many network interfaces, so it’s not uncommon for a computer to have multiple IP addresses.

The vast address-space of possible IP addresses is managed by the Internet Corporation for Assigned Names and Numbers (ICANN). ICANN assign blocks of IP addresses to organisations. There are also three special blocks of IP addresses that ICANN have reserved for use on private networks, and it’s these IPs that we use within our homes. These private IP addresses can never appear on the public internet, they must always be either completely disconnected from the internet, or isolated from the internet by a NAT router. We’ll look at NAT in more detail later, but for now, all we need to know is that just about every home router is a NAT router.

The private IP address ranges:

Additionally, there are two other special ranges of IPs you may encounter:

Unlike Ethernet, the IP protocol can send packets across different networks. The act of moving a packet through different networks from source to destination is known as routing. The heavy-lifting in routing, figuring out a route through the maze of interconnected networks that make up the internet, is done by routers, and is beyond the scope of this series. However, each and every device that speaks IP needs to make some simple routing decisions. Fundamentally our computers have to answer one simple routing question "is the destination IP address for this packet on my local network, or is it on a different network?".

If your computer determines that the destination IP address is on the local network, then it looks up the MAC address for the destination IP using ARP (as described in the previous instalment) and sends the packet directly to its destination using Ethernet.

On the other hand, if your computer determines that the destination IP is not on your local network, then it must send the packet to your router for processing. To do this your computer must know the IP address of your router, and, it must find your router’s MAC address using ARP. Once your computer has that information it sends the packet to your router using Ethernet. Your router then sends the packet on to your ISP’s router which will send it on its way across the internet.

At this stage, we know that for your computer to work on an IP network it must have an IP address, and it must know the IP address of the router it should use to send remote packets on their way. But how does it know whether or not a packet is destined for the local network? The key to answering that question is IP subnets.

When we say that a device must figure out whether or not a packet’s destination IP is local or not, what we really mean is that it must determine whether or not the destination IP is in the same subnet.

A subnet is a collection of similar IP addresses that share a single Ethernet network. In other words, every device on a subnet must be able to send Ethernet packets to every other device on the same subnet. There is not a one-to-one mapping between Ethernet networks and IP subnets though, so while a subnet can’t be spread over multiple Ethernet networks, one Ethernet network can host multiple IP subnets. Having said that, in reality, the mapping usually is one-to-one, especially within our homes.

Note that when people talk about a Local Area Network or LAN, they are often referring to an IP subnet. Do bear in mind though that many people use those terms very loosely, and often inaccurately.

In order to understand how a subnet is defined, we need a deeper understanding of IP addresses. We are accustomed to seeing them as so-called dotted quads (like 192.168.0.1), but those dotted quads are just a human-friendly way of representing what an IP address really is — a 32-bit binary number. The IP address 192.168.0.3 is really 11000000101010000000000000000011.

We create subnets of different sizes by choosing a dividing line somewhere inside this 32-bit block, and saying that everything before our dividing line will be kept constant for this subnet. In other words, all IP addresses on a given subnet share the same prefix when expressed in binary form. This means that to define a subnet we need two pieces of information — the first address in the subnet, and the location of the dividing line, or, to use the correct terms, we need a network address (or net address), and a netmask. The size of the subnet (the number of IP addresses it contains) is determined by where you choose to place the divider. The closer to the front of the 32 bits, the bigger the subnet, the closer to the back, the smaller.

Netmasks are represented using 32-bit binary numbers. All the bits before the chosen divide are set to 1, and all the bits after the divide are set to 0. When written in binary, a netmask MUST be 32 bits long and MUST consist of a series of 1s followed by a series of 0s. The chosen dividing line represented by the netmask is the point where the 1s change to 0s.

Because netmasks are 32-bit numbers, just like IP addresses, we can represent them in the same way, as dotted quads. (There are other ways to represent them too though — as we’ll see later)

Remember, the problem to be solved is that our computers need to be able to determine if a given IP address is local or remote. Local IP packets should be sent directly to the destination, while remote packets should be sent to the router.

The key fact here is that you can use a binary AND operator to calculate your net address based on just your IP address and netmask. In other words, when you take your own IP address and AND it with your netmask you get your net address. You then take the destination IP address and AND it with your netmask, and compare the result to your net address. If the result is the same as your net address, the IP is local, if not, it’s remote and needs to be routed.

Firstly, let’s define what we mean by an AND operation in a binary context:

This probably all sounds very abstract, so let’s work through an example. We are computer A, and the two computers we want to communicate with are Computers B and C. Subnet masks are not published, they are an internal OS setting, so we do not know the recipients’ netmasks, only our own. So, this is what we know:

First, let’s do some conversions:

Using the simple truth table for AND shown above, we can AND each IP address with our netmask to get the results below. Note that you need to apply the AND rule 32 times for each conversation, once for each bit. This is known in computer science as a bitwise AND operation.

What we see here is that A and B are in the same subnet, but C is not.

You can play around with this, and see all the binary calculations using my free IP Subnet calculator at www.subnetcalc.it. You can use this link to load the IP and Netmask into the interface automatically and save yourself some copying and pasting. You can then paste IP B and IP C into the IP Test text box near the bottom of the page to see why one is local and the other is not.

If we write out all the IP addresses between 192.168.0.0 and 192.168.0.9, as well as our netmask (255.255.255.248) the pattern will hopefully become clear. For extra clarity, I’ve also included the imaginary separator represented by the netmask by inserting a | character into the binary strings.

A subnet is, by definition, defined by its netmask and the first IP address within the subnet, known as the network address. A netmask of 255.255.255.248 only allows 3 bits of freedom within a subnet, so that means it defines 8 IP addresses (see above), but only SIX can actually be used. The first and last IP addresses of all subnets are reserved, and cannot be used by devices.

The first IP address is the network address, or net address, for the subnet, while the last is the so-called broadcast address.

The table below shows the subnet 192.168.0.0/255.255.255.248: