An Introduction to Unix

2.2. Basic Commands

Contents:

To run a Unix command (or any program, for that matter), you normally must type the name of the command/program file followed by any arguments. There is actually a program running that accepts your keystrokes and launches the appropriate program. The program that reads and interprets your keystrokes is called the shell. There are many shells available, all of which offer different features. The default shell for ODU CS is called tcsh, and we'll concentrate on that.

The command/program name is usually not given as a full file name. Instead, certain directories, such as /bin, are automatically searched for a program of the appropriate name. This set of directories is referred to as your “execution path” New accounts are set up so that the directories holding the most commonly used Unix commands and programs are already in the execution path.9

You can see your path by giving the command

echo $PATH

And you can modify your $PATH, if desired, to add additional directories.

Thus, one can invoke the ls command as

/bin/ls

but it's usually simpler to say

2.2.1. Command Arguments

Of course, most Unix commands consist not only of the program name but also require one or more command arguments. The command arguments indicate the details of the desired operation, including files to work with, text to use, etc.

Command arguments tend to come in four varieties:

File names: If a command needs to operate on one or more files, then we specify those files by giving a path to the file - a sequence of directories that we can follow to reach the file, ending with the file name itself.
A path may be absolute, meaning that it begins with the root directory / or with someone's home directory (starting with ~). Alternatively, a path can be relative, meaning that it is interpreted in terms of your current working directory.

Try This:
Log in and give the command
cd ~/playing

Now your current working directory should be ~/playing and one of the files in that directory should be math.h. If you wanted to refer to that file in a command argument, you could do so using of relative paths or absolute paths.
Give the commands
ls -l /home/yourLoginName/playing/math.h ls -l ~/playing/math.h ls -l ~yourLoginName/playing/math.h ls -l math.h ls -l ./math.h ls -l ../playing/math.h
The first three are absolute paths. The last two are relative paths. Note that they all refer to the same file.10
In fact, there are an infinite number of relative paths for most files, though most are pretty silly, e.g., ../playing/../playing/./././math.h.
If you aren't convinced that all of the above referred you to the same file, repeat the above commands using more instead ot ls -l.

If you change your working directory, e.g.,
cd ~
then you can refer to that same file using the same absolute paths /home/yourLoginName/playing/math.h, ~/playing/math.h, or ~yourLoginName/playing/math.h because absolute paths do not depend on your current directory. You can also use relative paths, but these would be different than before because your current directory is different: e.g., playing/math.h.

Try This:

cd ~ ls -l ~/playing/math.h ls -l math.h ls -l ../playing/math.h ls -l playing/math.h
The first ls still works because it uses an absolute path. Absolute paths are unaffected by your current working directory. The next two ls commands fail, because they were relative paths that made sense when your current working directory was ~/playing, but not when your current working directory is ~. The last ls works as it uses a relative path that makes sense from your current working directory. (Why? Use ls to explore your current directory and make sure you understand why some of these relative paths work and others fail.)
Directory names: these are also specified using absolute and relative paths. In fact, a directory in Unix is also a file - it's just a file whose special contents include a list of other files. This means that most Unix commands that expect a file can be given a directory as well. But in many cases that won't do what we want.

Try This:

The more command lists the contents of a file, a page at a time. Use the space bar to move forward a page, 'b' to move back a page, and use 'q' to quit.

cd ~ more playing/math.h more playing
As you can see, trying to list the contents of a directory file is not useful.

There are a few commands that work only on directories. mkdir (create an empty directory) and rmdir (remove a directory, if empty) are the most common.

In other cases, commands may work with either files or directories, but have slightly different behaviors in the two cases. The cp command is a good example of this.
Try This:
cd ~/playing cp yesno.h g1 ls cp math.h g1 ls rm g1 mkdir g1 cp math.h g1 ls ls g1
The three cp commands are almost identical. In the first case, we copied a file to a path that did not indicate an existing file (g1). So cp created a copy with that name. The second cp copies to a path (g1) that now indicates an existing file. cp removes that existing file and then creates a new copy with that name. The third cp, however, names an existing file that happens to be a directory. In this case, cp does not replace g1 but writes a copy (named math.h) inside the directory g1.
Text: some commands receive text as an argument. The simplest example is echo, which simply prints its arguments:
Try This:
echo Hello world echo I am $user

A more interesting example is the grep command, which searches files for lines containing desired text. For example,
Try This:

more /usr/include/math.h grep cos /usr/include/math.h
The grep command lists all lines inside /usr/include/math.h containing the string “cos” . We'll shortly see ways to write more complex search patterns.
Flags: these are special arguments used to alter or control the behavior of a command. Flags for Unix commands are usually written beginning with a “-” or occasionally “--” .
Try This:

ls ~ ls -a ~ ls /usr/include ls -l /usr/include
to see the difference in behavior caused by the -l flag.

How do you know what kind of arguments can be accepted by each command? You pretty much have to deal with this on a individual basis, though for any specific command you can consult the on-line manual for that command via man, e.g.
man ls man grep

10. In fact, there are an infinite number of relative paths for most files, though most are pretty silly, e.g., ../playing/../playing/./././math.h. (back)

2.2.2. Special Characters

As you type, some characters have a special meaning. For example, if you have entered the first few letters of a file name and hit the “Tab” key, the shell will examine what you have typed so far and attempt to complete the filename by filling in the remaining characters. If the shell is unable to complete the filename, a bell or beep sound will be given. Even in this case, the shell will fill in as many characters as it can.

Most special characters are entered by holding down the “Control” key while typing a letter. By convention, we designate this by placing the symbol “^” in front of the name of the second key. For example, if you have typed zero or more letters of a filename and want to see a list of what filenames begin with what you have typed, you could type ^D, i.e., hold down the “Control” key and type “d” .

Some other useful special keys are:

^C is used to abort a program/command that is running too long or working incorrectly. Beware: aborting a program that is updating a file may leave garbage in that file.
^D is used when a command/program is reading many lines of input from the keyboard and you want to signal the end of the input.
^H, the “Backspace” key, and the “Delete” key all delete the most recently typed character.
^B moves the cursor Backwards over what you have just typed, without deleting those characters. This is useful in correcting typing mistakes. The “left” arrow on your keyboard may also do the same thing.
^F moves the cursor Forwards over what you have just typed, without deleting those characters. The “right” arrow on your keyboard may also do the same thing.
^P retrieves the Previous command that you had typed in. Repeated ^P's may be used to look back through a number of commands that you have issued recently.
The “up” arrow on your keyboard may also do the same thing.
^N is the opposite of ^P. After one or more ^ P's, a ^N allows you to move back to the Next more recent command.
The “down” arrow on your keyboard may also do the same thing.
In many programs, ^Z pauses the program and returns you temporarily to the shell. To return to the paused program, give the command: fg

2.2.3. One Pattern, Many Instances

Contents:

2.2.3.1 Wild Cards

2.2.3.2 Regular Expressions

2.2.3.2.1 Regular Expressions and grep

2.2.3.2.2 Regular Expressions and sed

In the command examples we have used so far, we have always written a single file path or a single text string. In many cases, however, we want to supply commands with a whole list of files or text strings. Typing out the whole list, one at a time, would be tedious,11

Don't forget, though, that you can use the tab character to speed up typing of individual file names.

so we usually write some kind of pattern that describes multiple items instead.

2.2.3.1. Wild Cards

Whenever we have a command that can take multiple filenames, we can often write a single pattern for several files. Patterns for file names use wildcard characters, the most common of which is “*” , which tells the shell12

The program that reads your keyboard input, determines what command or program you want to run, then launches that program.

to substitute any combination of zero or more characters that results in an existing filename.

Try This:

ls ~/playing rm ~/playing/* ls ~/playing

What files were matched by the wildcard pattern in the rm command?

ls /usr/include

Notice that there are a number of files ending with .h

cp /usr/include/s*.h ~/playing ls ~/playing

Again, note the use of the wildcard to form a pattern for multiple file names. In cases, like this, where there are multiple possibile matched, the shell forms a list of all the matches. So the rm command actually saw a list of all the files in the ~/playing directory, and the cp command saw all the files in the in the /usr/include directory whose names began with “s” and ended with “.h” .

One good way to figure out what files will match a wildcard pattern is to use the echo command. echo simply prints out its arguments. But since the arguments in the command line are processed by the shell before invoking the echo program, any wildcard patterns will have already been expanded.

Try This:

ls /usr/include echo /usr/include echo /usr/include/*.* echo /usr/include/*

The difference between the last two may be subtle. The *.* pattern will match only files that contain a “.” . Unlike Windows, Unix does not require file names to end with a period and a three-letter extension. Some sort of period and extension is common, but directory names and executable programs often have no extension and no period. (In Windows, you can create a file with an empty extension, but Windows insists on adding a period at the end.)

echo /usr/include/f*.* echo /usr/include/*f*.*

12.

The program that reads your keyboard input, determines what command or program you want to run, then launches that program.

(back)

2.2.3.2. Regular Expressions

Contents:

2.2.3.2.1 Regular Expressions and grep

2.2.3.2.2 Regular Expressions and sed

If wildcards provide a way to write patterns for file and directory paths, can we also write patterns for text strings? Yes, but this is not built into the shell for use by every command, the way that wildcards are. Instead, most Unix programs and commands that do some kind of searching or matching for text will share a common notation for patterns of text to be matched. This notation is called regular expressions. For example, almost every text editor in any operating system will allow you to search a file for a given string. But most Unix text editors (including the emacs editor we'll study later) will allow you to search for any string matching a regular expression “pattern” . sed, a useful utility for doing simple changes to text files, is most often invoked to use its “substitute” command, which replaces any text matching a regular expression by some desired replacement text. The csplit command splits a single file into multiple pieces, where the point of division is most often indicated via a regular expression. Perl and awk, available on most Unix systems but not covered in this course, are scripting (programming) languages with a heavy emphasis on text manipulation, which is accomplished largely through matching on regular expressions.

Regular Expressions and grep

In an earlier example, we saw that the program grep can be used to list all lines of a file that match a given string. For example,

grep 'cos' /usr/include/math.h

would list all lines in the indicated file that contain the string “cos” .

expression. The above example works because of the way that regular expressions are composed. The first rules of regular expressions are:

A regular expression consisting of a single “non-special” character will match any string containing that character.

As it happens, none of the alphabetic and numeric characters are “special” , so the regular expression c would match any string containing a “c” .
Try This:

grep c /usr/include/math.h
Since grep works line by line, this would select every line containing a “c” .
If a set of regular expressions r₁, r₂, …, r_k are concatenated together to form a single larger regular expression r₁r₂…r_k, it matches any string that contains a substring formed from a concatenation of strings s₁s₂…s_k, each of which matches the corresponding regular expression.
So when we write
grep cos /usr/include/math.h
the cos is actually the concatenation of three regular expressions c, o, and s, and matches any string that contains a substring matching c followed immediately by a substring o followed immediately by a substring matching s.
That may seem an unnecessarily complex way to get to the original idea of “matches the string `cos',” but this idea of concatenation is a general one that becomes more important as we consider other combinations of smaller regular expressions.

The real power of regular expressions comes into play when we consider the various “special” characters that serve as regular expression operators.

Regular expressions can be grouped in parentheses, written as (…).
So these two commands do exactly the same thing:13
egrep is an “extended” grep that supports a richer set of regular expression operations than does regular grep.

egrep 'cos' /usr/include/math.h egrep '(cos)' /usr/include/math.h

Try This:
grep '(' /usr/include/math.h egrep '(' /usr/include/math.h egrep '\(' /usr/include/math.h
“(” is special to egrep, but not to grep. To take away the “special-ness” , we can quote the parenthesis character when presenting it to egrep.
[…] brackets containing any set of characters not beginning with ^ will match a string containing any one of those characters.

Try This:
grep sqrt /usr/include/math.h grep '[sS][qQ][rR][tT]' /usr/include/math.h
The regular expression [sS] matches any single character that is either “s” or “S” , and so on.14
Actually, doing case-insensitive matches is so common that grep provides a shortcut. The -i flag makes all character matches case insensitive, so
grep -i 'sqrt' /usr/include/math.h grep '[sS][qQ][rR][tT]' /usr/include/math.h
are identical.

Try This:
Compare the output from
grep '(' /usr/include/math.h grep ' (' /usr/include/math.h grep '[ (]' /usr/include/math.h grep '((' /usr/include/math.h grep '[ (](' /usr/include/math.h

The character set inside a regular expression square brackets can be abbreviated by giving a range of characters separated by a hyphen. For example, [a-z] would match all the lower-case alphabetic characters.
Try This:
What lines are not listed by the following?
grep '_[a-zA-Z]' /usr/include/math.h
If you can't figure it out, you can check with the command
grep -v '_[a-zA-Z]' /usr/include/math.h
The -v option causes grep to list only those lines that do not match the pattern.
[…] brackets containing any set of characters beginning with ^ will match a string containing any character not in that set.

Try This:
grep 'cos' /usr/include/math.h grep '[^a]cos' /usr/include/math.h
. matches any single printable character, including blanks, but not including the end-of-line character.

Try This:
grep 'd.f' /usr/include/math.h
If r is a regular expression, then r* matches zero or more successive strings, each of which matches r.

Try This:
grep 'ER*N' /usr/include/math.h grep '#.*if' /usr/include/math.h
Note that the behavior of * is very different in regular expressions than it is in file path wildcards.
If r is a regular expression, then r+ matches one or more successive strings, each of which matches r.

Try This:

grep 'ER*_' /usr/include/math.h egrep 'ER+_' /usr/include/math.h
If r is a regular expression, then r? matches zero or one strings matching r.
Try This:
egrep 'def' /usr/include/math.h egrep 'define' /usr/include/math.h egrep 'def(ine)?' /usr/include/math.h egrep 'def(ine)?d' /usr/include/math.h
In essence, the ? makes the preceding item optional. (So the first and third commands in this example are actually equivalent.)
If r is a regular expression, then ^r matches any string that begins with a substring matching r.

Try This:
grep 'ex' /usr/include/math.h grep '^ex' /usr/include/math.h
If r is a regular expression, then r$ matches any string that ends with a substring matching r.

Try This:
grep '_' /usr/include/math.h grep '_$' /usr/include/math.h

With so many special characters, you might wonder just what you're supposed to do if you really want to search for lines containing a “*” , or a “?” , or a … The answer is given in our final rule:

If c is a special character, then \c matches that character in a string.

Try This:
grep '3.' /usr/include/math.h grep '3\.' /usr/include/math.h

This is by no means an exhaustive list of all the regular expression operations, but it's probably enough for most purposes. The textbook has some additional information, and you can get an exhaustive description via the command

man regexp

13.

egrep is an “extended” grep that supports a richer set of regular expression operations than does regular grep.

(back)

14.

Actually, doing case-insensitive matches is so common that grep provides a shortcut. The -i flag makes all character matches case insensitive, so

grep -i 'sqrt' /usr/include/math.h grep '[sS][qQ][rR][tT]' /usr/include/math.h

are identical.

(back)

Regular Expressions and sed

As noted earlier, many other commands besides grep will use regular expressions. sed, for example, allows you to enter a variety of editing commands that will be applied to every line of a file. A common use of sed is to scan each line of the file for a pattern and to replace that pattern, wherever it occurs, by some string. The sed command to do this is

sed s/pattern/replacement/g filename

where filename is the file whose contents we want to scan and replace, pattern is a regular expression describing the text to search for in each line, and replacement is the text by which we wish to replace any thing that matches the pattern.15

The '/' characters are simply necessary to indicate the beginning and end of the pattern and replacement strings. They can be replaced by any character that does not appear in either the pattern or replacement strings.

Try This:

cd ~/playing cp ~cs252/Assignments/ftpAsst/alas.txt . more alas.txt

Now try operating on that file with sed:

sed s/e/Q/g alas.txt

for a simple text-to-text substitution, and

sed 's/[aeiou]/X/g' alas.txt

to see that the pattern really is interpreted as a regular expression.16

The quotes are required because, as explained in the next section, the '[' and ']' characters are normally considered special characters by the shell, and the quotes tell the shell to leave those characters alone, so that sed can actually see them.

The 'g' at the end of each of the prior examples indicates that the change should be applied every time a match is found. If the 'g' is dropped, only the first match in each line will be replaced.

Try This:

sed 's/[aeiou]/X/' alas.txt

to see the effect of dropping the 'g'.

15. The '/' characters are simply necessary to indicate the beginning and end of the pattern and replacement strings. They can be replaced by any character that does not appear in either the pattern or replacement strings. (back)

16.

(back)

11.

Don't forget, though, that you can use the tab character to speed up typing of individual file names.

(back)

2.2.4. Quoting

In the grep examples of the previous sections, I took care to enclose each regular expression inside 'quotes'. That's not some special way to indicate regular expressions, nor is it specific to text strings as arguments.17

Unix is amazingly tolerant of what characters get put into file names, but only masochists would use these special characters because the command shell will interpret them as something else, making it difficult to type the file name.

. Instead, it's a way to be sure that the command shell did not treat the enclosed characters as special shell characters but would pass them, unchanged, to the grep command. Among the characters that the shell would tend to treat as “special” are blanks and Fix this xref: / * < > | ? \ ; , ! $ ' ` " We've already seen special uses for the first 3 of these, and will encounter some of the others in later lessons.

What do we do if we need to type one of these special characters into a command but not have it treated specially? For example, suppose that we had a file foo.txt and we wished to list all the lines in that file that contained a “<” . We can't do

grep < foo.txt

because the < is a special (input redirection) character to the shell.

What we need to do is to quote that special character somehow to prevent the command shell from treating it specially. There are three ways we can do this:

We can place a backslash (\) in front of the special character.:
grep \< foo.txt
Note that backslashes can quote themselves. So if we wanted to print on our terminal screen a backslash followed by an asterisk, we could write
echo \\\*
The first backslash quotes the second one. The third one quotes the asterisk.
We can enclose all or part of the argument in single quotation marks. This suppresses all special characters. Also, if the enclosed portion includes blanks, it combines what would otherwise have been seen as multiple parameters into a single parameter.
grep Hello there foo.txt
This would look in the files named “there” and “foo.txt” for any lines containing the word “Hello” .
grep 'Hello there' foo.txt
This would look in the file named “foo.txt” for any lines containing the phrase “Hello there” .

grep 'Hello there!' foo.txt
This would look in the file named “foo.txt” for any lines containing the phrase “Hello there!” . Note that, without the quotes, the “!” would have been treated as a special character.
Finally, we can enclose all or part of the argument in double quotation marks. This suppresses all special characters except $, and also gathers its contents into a single parameter.

Try This:
echo $user * echo '$user *' echo "$user *"
Can you see the differences in how the two special characters are treated in each case?

17. Unix is amazingly tolerant of what characters get put into file names, but only masochists would use these special characters because the command shell will interpret them as something else, making it difficult to type the file name. (back)

9. You can see your path by giving the command

echo $PATH

And you can modify your $PATH, if desired, to add additional directories. (back)