You are reading the beginning of the
lecture notes for
CS 252,
Introduction to Unix for
Programmers, a course offered by the ODU Computer Science Department.
The course is designed to introduce students to the basic Unix
skills that they will need to work productively on the ODU CS Dept.'s
network of Sun workstations and Linux PC's.
Whether you are enrolled in that course or not, you are welcome to
peruse these notes.
You can tackle the topics in this course in different orders, depending upon your own preferences. shows how the topics that make up this course are related to one another.
An arrow from one topic to another means that you should complete the first topic before attempting the second. For example, before looking at "Program Development (via X)" you must have completed both "Program Development (via telnet)" and "Editing Files (via X)". On the other hand, you could do "Shell Games" either before or after doing "X Windows", whichever you prefer.
In addition to those seeking access to the CS Dept.'s Unix network,
some people may be interested in running Linux on a home PC. Most Linux distributions will offer a terminal/command window that runs a Unix command shell called
“bash”
. Others who don't want to abandon Windows entirely may be interested in
the
Cygwin project's free
port of the GNU C/C++ compilers for use on Windows 95/98/NT/2000
machines. Along with the compilers, this package provides a Unix
emulation layer including the bash shell.
Most of what is described in this website applies to
working in bash as well.
We can actually
interpret the title question in a number of different ways.
Do we mean,
“why does the CS department use
Unix?”
, or
“why is this course about Unix?”
, or
“why was
unix invented?”
, or even
“why does Unix behave the way that it
does?”
It's actually the last of these interpretations that I want to
address, although understanding the answer to that question will go
long ways towards explaining why the CS department uses Unix in most
of its courses and, therefore, the very reason for the existence of
this course.
I think that, to really understand a number of the fundamental
behaviors of Unix, it helps to consider how Unix is different from
what is probably a much more familiar operating system to most of you,
Microsoft Windows. Furthermore, to understand the differences between
these operating systems, you need to look at the history of computer
hardware and system software evolution in effect at the time when each
of these operating systems was designed. In particular, I want to
focus on three ideas: evolution of CPU's and process support,
evolution in display technology, and evolution in network and
technology.
1.1. Mainframe & Minicomputers
1.1.1. CPU and process support
Computer historians are fond of pointing out that mainframe computers
were huge behemoths, occupying massive rooms, drawing large amounts of
electrical power for their operation, and often requiring cooling
systems fully as large as the processor itself. For some time,
processors continued to be physically large, although the processing
power squeezed into that space grew tremendously.
On the early machines, only a single program could be run at any given
time. As processors became more powerful, both hardware and system
software evolved to permit more than one program to run simultaneously
on the same processor. This is called multiprocessing. The initial
reason for doing multiprocessing was to allow programs from many
different users (programmers) to run at once. This, in turn, is called
multiprogramming. At first, it was assumed that a single user had no
need for more than one process at a time.
Interactive programs are characterized by long periods of idleness, in
which they are awaiting the next input from the user. In an
interactive environment, it becomes natural for users to switch
attention from one process that is awaiting input or, in some cases,
conducting a lengthy calculation, to another process that has become
more interesting. For example, someone using a word processor might
want to switch over to their calendar to look up an important date
before returning to the word processor and typing that date into their
document. Fortunately, once you have support for multiprogramming, you
have most of which you need for combined multiprogramming and
multiprocessing.
In fact, there is a definite advantage to having started with
multiprogramming. In a multiprogramming environment, there is a great
danger that one programmer's buggy software could crash and, by
rewriting portions of memory or resetting machine parameters, take
down not only that programmer's program but other programs that
happened to be running on the machine at the same time. Consequently,
multiprogramming systems place a heavy emphasis on security, erecting
hardware and software barriers between processes that make it very
difficult for one process to affect others in any way.
The trend toward multiprogramming in multiprocessing persisted, not
only across families of mainframe computers, but also across the
increasing number of desk-size
“minicomputers”
. It is in this
context that Unix was developed. From the very beginning, Unix was
therefore envisioned as an operating system that would provide support
for both multiprocessing and multiprogramming.
1.1.2. Display Technology
During the heyday of the mainframe, most data was entered on punch
cards and most output went directly to a printer. Most of these
systems had a
“console”
where commands could be entered directly
from a keyboard, and output received directly on an electric
typewriter-like printer, but such input was slow and inexact. Prior to
multiprocessing, it would have been economic folly to tie up an
expensive CPU waiting for someone to type commands and read output at
merely human speeds. So the system console generally saw use only for
booting up the system, running diagnostics when something was going
wrong, or issuing commands to the computer center staff (e.g.,
“Please
mount magnetic tape #4107 on drive 2.”
).
The advent of multiprocessing and the subsequent rise of interactive
computing applications meant that the single system console, hidden
away in the computer room where only the computer center staff ever
touched it, was replaced with a number of computer terminals
accessible to the programmers and data entry staff. An early computer
terminal was, basically, a keyboard for input and an electric
typewriter for output.
Terminals were not cheap, but their lifetime cost was actually
dominated by the amount of paper they consumed. In fairly short order,
the typewriter output was replaced by a CRT screen. This opened up new
possibilities in output. A CRT screen can be cleared, output to it can
be written at different positions on the screen, and portions of the
screen can be rewritten without rewriting the entire thing. These
things can't be done when you are printing directly onto a roll of
paper. Terminal manufacturers began to add control sequences,
combinations of character codes that, instead of printing directly,
would instruct the terminal to shift the location where the next
characters would appear, to change to bold-face or underlined
characters, to clear the screen, etc. All of this wizardry was
hard-wired -- there were no integrated-circuit CPUs that could be
embedded into the box and programmed to produce the desired results.
Consequently, terminals were quite expensive (the fancier ones costing
as much as a typical new car). Different manufacturers selected their
control sequences as much based upon what they could wire in easily as
upon any desire for uniformity or compatability. Consequently, there
were eventually hundreds of models of CRT-based computer terminals,
all of which used incompatible sets of control sequences.
Embedded microprocessors eventually simplified the design of computer
terminals considerably (sinking a number of companies along the way
that had made their money leasing the older expensive models), and the
capabilities of computer terminal began to grow, including adding
graphics and color capabilities. Eventually, PCs became cheap enough
that the whole idea of a dedicated box serving merely as a terminal
came into question, and the computer terminal now exists as a separate
entity only in very special circumstances, although there are periodic
attempts to revive the idea (e.g., so-called internet appliances).
1.1.3. Network Technology
Before there was a World-Wide Web, there was an Internet. The
internet grew out of a deliberate attempt to allow researchers all
around the country access to the limited number of highly expensive
mainframe CPU's. Internet traffic originally was dominated by telnet,
a protocol for issuing text commands to a computer via the internet,
and ftp, a protocol for transferring files from machine to machine via
the Internet. Email came along later.
In imitation of (and perhaps in jealousy of) the internet,
UseNet evolved as an anarchic collection of mainframe and minicomputers that
each knew a handful of telephone numbers of other UseNet computers and could
pass email and news (a.k.a. bulletin board) entries along those connections.
As the idea of long range networking took hold, more and more sites
began installing local area networks to enable communication among
their own machines.
1.2. A Tale of Two Operating Systems
Unix evolved for minicomputers in an historical context where
- Multiprocessing
was expected, and the hardware provided safeguards for protecting
one running process from affecting or being affected by other
processes on the same CPU.
- The most common displays were computer terminals, which came in
many different models, all of which used mutually incompatable
control sequences. Most of these could display text only, or
text with simple vertical & horizontal line graphics.
“True”
graphics terminals were not unknown, and were clearly on their
way, but were so expensive as to be comparatively rare.
- Networking
was common. In fact, it was normal, perhaps even the rule, for
users to be controlling, via the network, machines that were
remote from the users' actual location.
When personal-computer (PC's) came on the scene, they represented a
revolution in terms of both decreased size and decreased cost, but
they represented a step backwards in terms of total computing power
and in terms of the sophistication of the hardware support for many
systems programming activities.
Oddly enough, PC systems seemed to recap the entire history of
computing up till that time, thoug at a somewhat faster pace:
-
“One user -- One CPU”
was a rallying
cry of the early PC proponents. They argued that, although an
individual PC presented limited CPU power compared to mainframe
or mini machines, the individual PC could still provide a single
user with more CPU power than that person would receive as their
share of a mainframe when split over a large number of
simultaneous users. So early PC operating systems returned to the
single-user, single-process model that had
gone out of fashion decades before in the world of larger
computers.
- Display technology reverted
initially to the electric-typewriter style system console. This
was quickly supplanted by a
“dumb terminal”
CRT-and-keyboard,
though early printers were still electric typewriter
based.
The very existence of integrated circuit CPUs, however, lowered
the cost of CRT displays to the point where more elaborate,
graphics-capable displays were soon available.
- Network technology was initially
spurned.
“One user -- One CPU”
, remember? Why would anyone
need access to other computers. Email and net news could be
handled by modem connection without full-fledged networking. It
took a surprisingly long time before PC operating systems and
applications began to acknowledge that not every bit of
information and not every hardware/software resource could
economically be replicated on every PC
system.
MSDOS was
developed in a context where
-
“One user -- One CPU”
was the rule.
Multiple processes for a single user were not deemed necessary.
- Most PCs had a CRT display with
limited character and graphics capabilities.
- Networking was deemed unnecessary.
As MSDOS evolved
into Windows, it did so in response to changes in the HW/SW context:
-
“One user -- One CPU”
remained the
rule, but a single user might have multiple processes.
- PC displays could show characters in
a variety of fonts, and graphics capabilities were more common.
- Some people might want local
networking, but it was supplied by third-party add-ons with minimal
support from the operating system itself. As for the internet, why would
anyone with a PC want to communicate with all those mainframe
dinosaurs?
1.3. Reflections on MS Windows & Unix
Of course, both MS Windows and Unix continued to evolve past their
earliest forms, but the contexts in which they have evolved helped
establish their fundamental philosophy and continues to influence how
they work today.
- Unix
users tend to do a lot more typing than Windows users. Graphics
capabilities were rare when Unix was developed, so commands had
to be typed out rather than always working through a window GUI,
and that practice of typing commands continues to influence the
“look and feel”
of Unix. Unix does have a windowing GUI, but the
Unix approach is to launch an application via a typed command,
then let that application open up windows if it needs
them. Compare to MS Windows, where most users never type a
command, and may not even realize that many common Windows
applications offer a variety of command line options. (Try, for
example, creating shortcuts on your Windows desktop with the
Target:
C:\WINDOWS\EXPLORER.EXE
and another with the Target:
C:\WINDOWS\EXPLORER.EXE /n,/e,c:\
Try them each and see what they do. The difference is potentially
useful, but this possibility is unknown to most Windows users. - Because
graphics capabilities were rare when Unix was developed, many
Unix applications are text based. The canonical Unix application
reads a stream of text in (from
“standard input”
) and produces a
stream of text as output (on
“standard output”
). Because the output is
often considered a slightly modified version of the input, such
programs are called filters. Unix users are more likely to
string together a bunch of filters that each do something simple
than to hunt for a massive dedicated application that does the
whole job at once. For example, if you wanted to know how many
statements occurred in some file of C++ code, an MS Windows
programmer would load that file into a visual C++ programming
environment, then search through the menu bars for a
“properties”
or
“number of statements”
item that might convey the required
information. A Unix programmer would, in a single line of text
commands, feed that file into a program that extracted each line
containing a
“;”
(because most C++ statements end with semicolons) and
then feed that set of
“;”
-containing lines through a
line-counting filter.
-
“One user -- One CPU”
thinking dominates
Windows applications even though networking is widely
available. MS Windows has never offered the same level of
protection against one process affecting others on the same
machine. In Windows, it's still all too common for a single
crashing application to lock up the machine to the point where a
reboot is required. In Unix, such reboots are quite rare.
In part, this is because Unix allows processes to interact in
only two ways:
- By having one process write out files that the other reads
- By having one process write characters into a
“pipeline”
that is read by another process.
By contrast, MS Windows has a variety of communication
mechanisms, some of which allow one process to directly
manipulate the data of another. This allows different applications to
work together in interesting and valuable ways (e.g., embedding charts
from a spreadsheet directly inside a word-processor document), but at
an inevitable cost to overall system security and stability.
- Unix
applications are often designed to be run remotely, via a network. Even
the Unix windowing GUI, called X, is based on the idea that an
application can be run on whatever machine it is installed upon, but will
actually display its windows on and accept its mouse clicks and other
inputs from any machine on the network. MS Windows, on the other hand,
assumes that the application is running on the machine where its outputs
should be displayed. If you want to use, say, a paint program that isn't
installed on the PC you are sitting at but is installed on another PC in
the same room, you can't run that program without actually getting up and
moving to the other PC.
- Another instance of this bias, one that is, I suspect, much
closer to the hearts and pocketbooks of Windows applications
developers, can be seen
in the deliberate ignorance of many applications to the realities
of the networking world. Although Windows will provide support for different users
logging in (one at a time!) to given PC, the majority of
application software that a user might install on that PC will assume
that a single data area and a single set of option and preference
settings are enough for that PC. Therefore every time you change
a setting, you affect the behavior of that application not only
for yourself, but for all other people who use that same
machine. Unix applications, on the other hand, are far more
likely to be distributed under site licenses permitting all users
at a site equal access.
If you have PCs arranged on a
network, so that they can access a common pool of disk drives,
when you purchase a new windows application, it is likely to try
to install itself onto a single PC, not allowing itself to be run
from the other PCs that can access the drive where you install the
software. And even if you could run it by from a different PC,
you will often find that all of your preferences or options
settings and all of your accumulated data is squirreled away in
the registry or systems area of the PC on which the software was
installed, making it unusable from other
PCs.
All this may help to explain why the CS Dept. makes such heavy use of
Unix. It's not that we dislike MS WIndows. But if we require a
particular software package (say, a compiler) in our classes, under
Unix we install it on a few Unix machines and let students run it from
remote locations via the internet. Under Windows we have to install it
on every CS Dept machine, ask that it be installed on every
machine in the ODU laboratories (and at the distance learning
sites). And when an updated version of that package comes out, we have
to go through the entire process all over again. It's just far, far
easier to maintain a consistent working envoronment for all students
under Unix.
There are two ways to interact with Unix and Unix programs: through a
text-only interface, or via a windows interface --- the Unix
windowing system is called X.
Which mode of interaction you use depends upon how you are accessing to
the network, the software on the machine you are sitting at, and how
fast your connection to the network is. But even if you are running
X, one of the first things you are likely to open is an
“xterm”
, a
text-only command window.
Unix users tend to launch
programs from the text-only interface. If the program itself supports
windows, mice, etc., then they can point and click to their heart's content.
In this section, we'll concentrate on access via
ssh, an internet protocol for issuing interactive commands to
remote machines. Window-based X access will be introduced
later.
There is an older text-based protocol for giving commands to Unix
called "telnet". telnet is still used for a variety of purposes, but
has fallen out of favor because it sends everything (including your
login name and password) in plain-text format, leaving you vulnerable to someone else on your network eavesdropping via "packet sniffers".
By contrast, ssh encrypts all your communications, so you are safe even if someone is eavedropping. As of Summer 2008,
the ODU CS dept will no longer support telnet connections.
1.4. Making a Connection: ssh
If you are a student registered for a CS course, and have never had a Unix
account on the CS Dept system, you can get your account by going to
the CS Dept home page and clicking on
the
“Account Creation”
link (under
“Online Services”
).
If you have had an account in the recent past, it should be
regenerated for you in any semester when you are registered for a CS course.
Otherwise, you will need to contact your instructor or from the CS systems
staff to get your account.
“ssh”
(secure shell) is an internet protocol that
allows a person connected to the internet to log into other machines
on the internet and to issue commands to those machines. To use
ssh, you must have an
“ssh client”
program on the
machine where you are seated, and you must know the name of a machine
elsewhere that is running a
“ssh server”
, the program
that accepts logins and subsequent commands from the client.
To choose an ssh server in our Dept., select a
machine from this list
at http://system.cs.odu.edu/?page=faq&id=fastmachines.
Next, you need to run an ssh client program. If you are on your
own PC, you can choose the client program to install and run. If
you are on a lab PC, at work, or for some other reason using
someone else's machine, you either need to use whatever they
have installed or bring your own client program, probably on a
USB flash drive.
-
From your own Windows PC: Install PuTTY. Then proceed to Connecting via PuTTY.
-
From Windows PCs in CS Dept Labs: Most, if not all,
will have PuTTY. Run it and proceed to Connecting via PuTTY. In some
cases, you may find OpenSSH instead.
-
From Windows PCs in the ODU OCCS Labs: As of May 2008,
OCCS machines do not have an ssh client installed. This
will probably change soon. In the meantime, install Portable
PuTTY on a USB flashdrive and proceed to Connecting via PuTTY.
-
From Linux PCs or Windows PCs with CygWin (and probably
for Apple machines running OS/X): These machines will have
(or, in the case of CygWin, can easily have installed) a
command-line ssh program. Proceed to Connecting via SSH Command Line.
1.4.1. Connecting via PuTTY
Run the PuTTY program. You will be presented
with a session dialog box like the one shown here. For the "Host Name", fill in the name of the ssh server machine that you chose earlier. For the "Connection Type", make sure that "SSH" is selected. Then click "Open".
The first time you connect to any machine, you will get a rather intimidating warning that "The server's host key is not cached in the registry" followed by a lot of details that uniquely identify the machine you are connecting to. This is a way ssh tries to protect you against people trying to capture your login info by "spoofing" or impersonating a legitimate machine. In practice, you're not likely to know whether this info is correct or not, so click "Yes" to proceed with the connection.
If you get a similar message later when reconnecting to a
machine you have previously used, that might be suspicious (or
it may just mean that the original machine crashed and has
been replaced by a different one that was given the same name
on the network.
You should now be prompted for your login name. You are ready to log in.
1.4.2. Connecting via Command-Line SSH
Open a terminal window (or a CygWin command window).
Give the command
ssh -l yourLoginName sshServerName
filling in your CS Unix login name and the name of the ssh server machine you chose earlier (including the
.cs.odu.edu ending).
If you are running a Linux or OS/X machine, you might want to try a variation on that command:
ssh -Y -f -l yourLoginName sshServerName xterm
filling in your CS Unix login name and the name of the ssh server machine you chose earlier (including the
.cs.odu.edu ending). This will usually open up a separate window for your login session.
The first time you connect to any machine, you will get a rather intimidating warning such as "The authenticity of host ... can't be established" followed by a lot of details that uniquely identify the machine you are connecting to. This is a way ssh tries to protect you against people trying to capture your login info by "spoofing" or impersonating a legitimate machine. In practice, you're not likely to know whether this info is correct or not, so click "Yes" to proceed with the connection.
If you get a similar message later when reconnecting to a
machine you have previously used, that might be suspicious (or
it may just mean that the original machine crashed and has
been replaced by a different one that was given the same name
on the network.
You should now be prompted for your login name. You are ready to log in.
Now that you have a login prompt, enter your login name. At the
“password:”
prompt, enter your password.
After a few moments, you should receive a command prompt.
1.6. Setting Your Terminal Type
Remember that Unix evolved in a time when many manufacturers made many
different models of computer terminals, each with its own set of
command codes for clearing the screen, moving the cursor to different
screen positions, setting bold face, underline and other text
characteristics, and so on.
A typical Unix installation will be equipped to communicate with any
of a few hundred different types of terminals.
Now, you're not using a terminal, but you are using a program that simulates one. telnet was originally intended to allow terminals to
issue commands to remote CPUs, and ssh is intended as a replacement for the older telnet protocol, so your ssh client program actually
works by simulating an
“old-fashioned”
computer terminal. The
authors of your client program chose one or more kinds of terminal
that they would simulate. For Unix to manipulate your screen
appropriately, it most know what kind of terminal command codes your
ssh client program is prepared to accept.
Find out the kind of terminal
being emulated by your telnet program. You may need to consult the
program documentation or help files for this. You may also be able to
deduce this information from the program's
“Options”
or
“Preferences”
menus.
Here are some common choices:
| SSH client program |
TERM type |
Comments |
| PuTTY | xterm | |
| OpenSSH | xterm? | (Need to check this) |
| Command-line SSH, basic form, Linux or OS/X | xterm or linux | |
| Command-line SSH, basic form, CygWin | xterm or rxvt | Depends on what type of window you type the command in. |
| Command-line SSH, xterm variant | xterm | |
Now, there is a protocol by which Unix systems can ask a terminal,
“What kind are you?”
, and the terminal responds with an
identification code. Many simpler clients don't implement this
feature, however, so you can't take it for granted.
Look at the messages that you received after
after logging in. If it includes a line saying something like
“Terminal type is…”
and the terminal type that it names
makes sense for your ssh client program, you're all set and you can skip
the next step. But it there terminal type seems wrong, or you see a
message indicating that you are on a
“dumb terminal”
, you need to tell the Unix system what
kind of terminal you are
really emulating.
The command to do so is
setenv TERM xxxx
where
xxxx is the kind of terminal (e.g,
setenv TERM vt100).
Some people also recommend that you follow this command with
tset -Q
which resets the terminal. In my own experience, this is usually
unnecessary,and I have
found that many communications programs don't deal well with this, but
try it if your terminal seems to be misbehaving.
Most
“dumb”
terminals provide for 24 lines of text. Many
telnet programs, however, allow more. If yours is one of
these, you should tell Unix how many lines you are using by giving the
command
stty rows nn
where
nn is the number of rows/lines.
1.7. Changing Your Password
Whether we like it or not, we need to worry about the security
of our computing environment. There are people who would take
advantage of this computer system if they had any, or more complete,
access to it. This could range from the use of computer resources
they have no right to, to the willful destruction and/or appropriation
of the information we all have online. In order to maintain the level
of security in our computing environment that we need, there are some
things we all have to take responsibility for. Even though you may
not feel like you personally have much to lose if someone had access
to your account or files, you have to realize that as soon as someone
gains ANY access to our system, it's 100 times easier for them to gain
access to ALL of it. So when you are lax with your own account, you
are endangering the work and research of everyone else working here.
Your password is the fundamental element of security not only
for your personal account, but for the whole UNIX system that we
share. Without an account and password a person has NO access to our
system. If someone discovers (or you tell someone) your password, not
only will they have access to your personal files, but they will have
a much better chance to launch attacks against the security of the
entire system.
Your account password is the key to accessing and modifying all of
your files. If another user discovers your password, he or she can
delete all your files, modify important data, read your private
correspondence, and send mail out in your name. You can lose much
time and effort recovering from such an attack. If you practice the
following suggestions, you can minimize the risk.
- NEVER give another user your password. There is no reason to do
this. You can change permissions and have groups set up if you need
to share access with other individuals. Your account should be yours
alone.
- Never write down your password. Another person can read it from your
blotter, calendar, etc. as easily as you can.
- Never use passwords that can be easily guessed. Personal
information about you (birth date, etc.) may be known to the attacker
or may be recorded in on-line databases that the attacker has
already obtained.
Passwords should not be single words (in any
language) because on-line dictionaries are widely available for
use in spelling checkers. A common approach to cracking passwords
is to compile a set of such words
and to run a program that
tries each one on each
account on the machine. Consider inserting punctuation and other
“odd”
characters into your password to foil such attacks.
A person with local knowledge can also try
your spouse's name, pets' names, etc. Your account is vulnerable to
this type of cracking unless you choose your password carefully.
- Change your password the very first time you log in, and every
few months thereafter. Security
problems are often traceable to stale passwords and accounts. These
are accounts that have become inactive for one reason or another or
the password has not changed for a long time. In our particular
environment we have had break-ins via such stale accounts. A password
that remains the same for a long time provides an intruder the
opportunity to run much more advanced and longer running programs to
break such passwords.
- Vary the system by which you choose a password. For example, don't
repeatedly use combinations like BLUEgreen and REDyellow. If an
intruder discovers your pattern, he or she can guess future passwords.
The command to change your password is
passwd
This command will first prompt you for your old password (just to
check that you really are you!) and then will ask you to type your new
password (twice, so that an inadvertent typing mistake won't leave you
with a password that even you don't know!).
To leave our Unix systems, type
exit
(not
logout, as indicated in the textbook.)
2.1. Files and Directories
Files in Unix are organized by listing them in directories.
Directories are themselves files, and so may appear within other
directories. The result is a tree-like hierarchy. At the root of this
tree is a directory known simply as
“/”
.
This directory lists
various others:
The bin directory contains many of the programs for performing
common Unix commands. The usr directory contains many of the
data files that are required by those and other commands. Of
particular interest, however, is the home directory, which
contains all of the files associated with individual users like you
and me.
Each individual user gets a directory within home bearing their
own login name. My login name is zeil.
We can expand our view of the Unix files then as:
cd and ls are two common Unix commands, as will be
explained later.
Within my own home directory, I have a directory also named
“bin”
,
containing my own personal programs. Two of these are called
“clpr”
and
“psnup”
. So these files are arranged as:
The full name of any file is given by listing the entire path from the
root of the directory tree down to the file itself, with
“/”
characters separating each directory from what follows. For example,
the full names of the four programs in the above diagram are
/bin/cd
/bin/ls
/home/zeil/bin/clpr
/home/zeil/bin/psnup
There are some common abbreviations that can be used to shorten file
names.
- You can refer to the home directory of someone with
login name name as
~name.
- You can refer to your own home directory simply as
~.
So you could refer to the file containing my clpr program
as either /home/zeil/bin/clpr or
~zeil/bin/clpr.
When I
myself am logged in, I can refer to this program by either of those two
names, or simply as
~/bin/clpr.
- At all times when
entering Unix commands, you have a
“working”
directory. If the file
you want is within that directory (or within other directories
contained in the working directory), the name of the working directory
may be omitted from the start of the filename. When you first log in,
your home directory is your working directory. For example, when I
have just logged
in, I could refer to my program simply as bin/clpr, dropping the
leading /home/zeil/ because that would be my working directory at
that time.
- The working directory itself can be referred to as simply
“.”
.
- The
“parent”
of the working directory (i.e., the directory
containing the working directory) can be referred to as
“..”
.
Unix filenames can be almost any length and may contain almost any
characters. As a practical matter, however, you should avoid using
punctuation characters other than the hyphen, the underscore, and the
period. Also, avoid blanks, and non-printable characters within
file names. All of these have special meanings when you are typing
commands and so would be very hard to enter within a filename.
Some things to keep in mind about Unix file names that may be
different from other file systems you have used:
2.1.1. Getting Started: Navigating Your Directories
If you have not yet done so, log in now so that you can work though
the following commands.
The cp command copies one or more files. You can give this
command as cp file1 file2 to make a copy of file file1,
the copy being named file2. Alternatively, you can copy one or
more files into a directory by giving the command as
cp file1 file2…filen directory
2.1.2. Some Common Unix Commands
Here are some common Unix commands. Some of these will be discussed
in more detail later, but if this list at least makes you aware of the
existence of a command that does something you want, then you can
always get the details on the command by reading its on-line manual
page via the man command:
man command-name
You might try experimenting with these in
your ~/playing directory.
- exit
- Shut down the current shell. If this shell
is the one you got at log-in, this command logs you
out.
- rlogin machine
- Logs you in to another machine on the
network. Use this if the machine you are on seems to be running slowly
and the who command indicates that there are lots of others on
the same machine.
- who
- Lists everyone logged into the same machine that you are
using.
2.1.2.2. File and Directory Manipulation
- cd directory
- Changes your current working directory to the indeicated directory, which may be absolute or relative.
- find directory instructions
- Searches the indicated directory and any subdorectories
within it for files. The instructions may serve to limit the files
found (e.g., search for files with a given name or that have been
modified after a given date) or may indicate commands to run on those files.
- ls path
- Lists all the files matching the indicated path. If that path is a directory, lists all the files within that directory. You can provide multiple paths in the same command, or omit the path entirely (in which case the contents of your current working directory will be listed). ls can be modified with a number of
“flags”
. Some of the more common are:
- ls -a
- By default, filenames beginning with
“.”
are considered
“hidden”
and not shown by the ls command. The -a (for
“all”
) option causes these files to be shown as well.
- ls -l
- This is the
“long”
form of ls. It displays
additional information about each file, such as the size and date on
which the file was last modified.
Note that options can be combined. For example, you can say
ls -la to get extra information including normally hidden files.
- ls -F
- Adds a little bit of extra information to each
file name.
If the file is an executable program, its name is marked with an
“*”
. If the file is a directory, its name is marked with
“/”
.
- mkdir directory
- Creates a directory.
- mv file1 file2
- Renames file1 as
file2. file2 may be in a different directory.
- mv file1 directory
- Moves file1 to
the given directory.
- pwd
- Prints your current working directory.
- rm file1…filen
- Deletes the listed
files. Be very careful using wildcards with this command.
rm * will delete everything in the current working
directory!
- rm -i file1…filen
- Deletes the listed
files, but firsts asks permission to delete each one.
- rm -r file1…filen
- Deletes the listed
files. If any of these files is a directory, it deletes that directory
and everything in it as well.
- rmdir directory
- Deletes a directory
(if empty).
2.1.2.3. Text File Manipuation
- cat file1…filen
- Lists the
contents of each of the listed files on your screen.
- grep text-pattern
file1…filen
-
Searches the files for lines matching the indicated text pattern and
prints the matching lines.
- more file1…filen
- Lists files one
screen at a time, pausing after each screen-full. Hit the space bar to
advance to the next screen.
A related program is less, which also allows you to move
backwards through the files by hitting
“b”
.
- sed editing-instructions file
-
Applies the editing instructions to each line of the file, writing out
the resulting changed version of the file.
- cancel request
- Remove a file from the printer queue, so
that it won't get printed. The
request identifier is found using the lpstat command.
- lp file
- Send file to printer for printing. Most sites
have multiple printers, each having its own name. One of these will be
the
“default”
printer used by the lp command. For the
others, you must give the printer name as part of the printer command:
lp -d printer file
For example, at the Norfolk ODU campus, lp by itself prints to
a fast printer in the public workstation lab for text only. lp -dcookie
prints to
“cookie”
, a printer in the same room that offers the
extra capability of printing Postscript graphics.
You will need to consult your local staff to see what printers are
available at other sites.
- lpstat
- Shows the list of files you have
“queued up”
awaiting
their turn on printers. Entered by itself,
lpstat
it lists only your own print jobs, giving a unique identifier for each
one.
To see the entire list of print jobs on some printer, enter
lpstat -o printer
As you explore Unix, you are bound to have questions. Some ways to get
answers include:
- The entire Unix manual is on-line.
man command
displays the manual page for the given command.
man -k keyword
looks up the given keyword in an index and lists the commands that may
be relevant. - The CS Department systems staff has collected a variety of additional
help documents. You can find them by going to the Dept home page
(http://www.cs.odu.edu/) and selecting
“Frequently Asked Questions”
under the
“Systems Group”
heading.
- A staff member is generally on duty or on call in the public CS lab in Hughes Hall (on the Norfolk campus) whenever that room is open.
- If none of the above help, then send e-mail to
“root@cs.odu.edu”
. This is
also how you report bugs, machine failures, etc.
To run a Unix command (or any program, for that matter), you normally
must type the name of the command/program file followed by any
arguments. There is actually a program running that accepts your
keystrokes and launches the appropriate program. The program that
reads and interprets your keystrokes is called the shell.
There are many shells available, all of which offer different
features. The default shell for ODU CS is called tcsh, and we'll
concentrate on that.
The command/program name is usually not given as a full
file name. Instead, certain directories, such as /bin, are
automatically searched for a program of the appropriate name. This set of directories is referred to as your
“execution path”
New accounts are set up so that the directories holding the most commonly used Unix commands and programs are already in the execution path.
Thus,
one can invoke the
ls command as
/bin/ls
but it's usually simpler to say
ls
Of course, most Unix commands consist not only of the program name but also require one or more command arguments. The command arguments indicate the details of the desired operation, including files to work with, text to use, etc.
Command arguments tend to come in four varieties:
- File names: If a command needs to operate on one or more files,
then we specify those files by giving a path to the file -
a sequence of directories that we can follow to reach the file, ending
with the file name itself.
A path may be absolute, meaning that it begins with the
root directory / or with someone's home directory
(starting with ~). Alternatively, a path can be
relative, meaning that it is interpreted in terms of your
current working directory.
If you change your working directory, e.g.,
cd ~
then you can refer to that same file using the same absolute paths
/home/yourLoginName/playing/math.h,
~/playing/math.h, or
~yourLoginName/playing/math.h because
absolute paths do not depend on your current directory. You can also
use relative paths, but these would be different than before because
your current directory is different: e.g.,
playing/math.h.
-
Directory names: these are also specified using absolute and
relative paths. In fact, a directory in Unix is also a file - it's
just a file whose special contents include a list of other files. This
means that most Unix commands that expect a file can be given a
directory as well. But in many cases that won't do what we want.
There are a few commands that work only on
directories. mkdir (create an empty directory) and
rmdir (remove a directory, if empty) are the most common.
In other cases, commands may work with either files or directories,
but have slightly different behaviors in the two cases. The
cp command is a good example of this.
- Text: some commands receive text as an argument. The simplest example is echo, which simply prints its arguments:
A more interesting example is the grep command, which
searches files for lines containing desired text. For example,
-
Flags: these are special arguments used to alter or control the behavior of
a command. Flags for Unix commands are usually written beginning with
a
“-”
or occasionally
“--”
.
How do you know what kind of arguments can be accepted by each
command? You pretty much have to deal with this on a individual
basis, though for any specific command you can consult the on-line
manual for that command via man, e.g.
man ls
man grep
2.2.2. Special Characters
As you type, some characters have a special meaning. For example, if
you have entered the first few letters of a file name and hit the
“Tab”
key, the shell will examine what you have typed so far and
attempt to complete the filename by filling in the remaining
characters. If the shell is unable to complete the filename, a bell or
beep sound will be given. Even in this case, the shell will fill in as
many characters as it can.
Most special characters are entered by holding down the
“Control”
key while typing a letter. By convention, we designate this by placing
the symbol
“^”
in front of the name of the second key. For
example, if you have typed zero or more letters of a filename and want
to see a list of what filenames begin with what you have typed, you
could type ^D, i.e., hold down the
“Control”
key and type
“d”
.
Some other useful special keys are:
2.2.3. One Pattern, Many Instances
In the command examples we have used so far, we have always written a
single file path or a single text string. In many cases, however, we
want to supply commands with a whole list of files or text
strings. Typing out the whole list, one at a time, would be tedious,
so we usually write some kind of pattern that describes multiple items
instead.
Whenever we have a command that can take multiple filenames, we can often write a single pattern for several files. Patterns for file names use wildcard characters, the most common of which is
“*”
, which tells the shell
to substitute any combination of zero or
more characters that results in an existing filename.
One good way to figure out what files will match a wildcard pattern is to use the echo command. echo simply prints out its arguments. But since the arguments in the command line are processed by the shell before invoking the echo program, any wildcard patterns will have already been expanded.
2.2.3.2. Regular Expressions
If wildcards provide a way to write patterns for file and directory
paths, can we also write patterns for text strings? Yes, but this is
not built into the shell for use by every command, the way that
wildcards are. Instead, most Unix programs and commands that do some
kind of searching or matching for text will
share a common notation for patterns of text to be matched. This
notation is called regular expressions. For example, almost
every text editor in any operating system will allow you to search a
file for a given string. But most Unix text editors (including the
emacs editor we'll study later) will allow you to
search for any string matching a regular expression
“pattern”
. sed, a useful utility for doing
simple changes to text files, is most often invoked to use its
“substitute”
command, which replaces any text matching a
regular expression by some desired replacement text. The
csplit command splits a single file into multiple pieces, where
the point of division is most often indicated via a regular
expression. Perl and awk, available on most
Unix systems but not covered in this course, are scripting
(programming) languages with a heavy emphasis on text manipulation,
which is accomplished largely through matching on regular expressions.
Regular Expressions and grep
In an earlier example, we saw that the program grep can
be used to list all lines of a file that match a given string. For
example,
grep 'cos' /usr/include/math.h
would list all lines in the indicated file that contain the string
“cos”
.
expression. The above example works because of the way that regular
expressions are composed. The first rules of regular expressions are:
-
A regular expression consisting of a single
“non-special”
character will match any string
containing that character.
As it happens, none of the alphabetic and numeric characters
are
“special”
, so the regular
expression c
would match any string containing a
“c”
.
-
If a set of regular expressions
r1,
r2, …,
rk are concatenated together to
form a single larger regular expression
r1r2…rk,
it matches any string that contains a substring formed from a
concatenation of strings
s1s2…sk,
each of which matches the corresponding regular expression.
So when we write
grep cos /usr/include/math.h
the cos is actually the concatenation of three
regular expressions c, o, and
s, and matches any string that contains a
substring matching c followed immediately by a
substring o followed immediately by a substring
matching s.
That may seem an unnecessarily complex way to get to the
original idea of
“matches the string `cos',”
but
this idea of concatenation is a general one that becomes more
important as we consider other combinations of
smaller regular expressions.
The real power of regular expressions comes into play when we consider
the various
“special”
characters that serve as regular
expression operators.
-
Regular expressions can be grouped in
parentheses, written as
(…).
So these two
commands do exactly the same thing:
egrep 'cos' /usr/include/math.h
egrep '(cos)' /usr/include/math.h
-
[…]
brackets containing any set of characters not beginning with ^
will match a string containing any one of those
characters.
The character set inside a regular expression square brackets can
be abbreviated by giving a range of characters separated by a
hyphen. For example, [a-z] would match all the
lower-case alphabetic characters.
-
[…]
brackets containing any set of characters beginning with ^ will
match a string containing any character not
in that set.
-
. matches any single
printable character, including blanks, but not including the
end-of-line character.
-
If r
is a regular expression, then
r*
matches zero or more successive strings, each of which matches
r.
-
If
r is a
regular expression, then
r+
matches one or more successive strings, each of which matches
r.
-
If
r is a
regular expression, then
r?
matches zero or one strings matching
r.
-
If
r
is a regular expression, then
^r
matches any string that begins with a substring matching
r.
-
If r
is a regular expression, then
r$
matches any string that ends with a substring matching
r.
With so many special characters, you might wonder just what you're
supposed to do if you really want to search for lines containing a
“*”
, or a
“?”
, or a … The answer is
given in our final rule:
This is by no means an exhaustive list of all the regular expression
operations, but it's probably enough for most purposes. The textbook
has some additional information, and you can get an exhaustive
description via the command
man regexp
Regular Expressions and sed
As noted earlier, many other commands besides grep
will use regular expressions. sed, for example, allows
you to enter a variety of editing commands that will be applied to
every line of a file. A common use of sed is to scan each line of the file for a pattern and to replace that pattern, wherever it occurs, by some string. The sed command to do this is
sed s/pattern/replacement/g filename
where
filename is the file whose
contents we want to scan and replace,
pattern is a regular expression describing
the text to search for in each line, and
replacement is the text by which we wish to
replace any thing that matches the pattern.
The 'g' at the end of each of the prior examples indicates that
the change should be applied every time a
match is found. If the 'g' is dropped, only the first match in
each line will be replaced.
