Checkpoint 3.2.1. Exercise: Try shell commands for navigation.
Try each of the following commands on your own machine.
Section 3.2 Basic Shell Commands
In this section, we explore basic shell commands related to files, directories, and navigation.
Disclaimer: It is beyond the scope of this book to cover anything but a small number of shell commands needed for contributions to open source, so the subsections below cover only the essentials.
Subsection 3.2.1 Files, Directories, and Privileges
This and subsequent subsections are written to be followed as an extended exercise with explanations given as we go.
First, follow the directions given in Subsection 3.1.3 to open a terminal window so you can use your shell. All of the shell commands will be typed in this terminal window.
Introduction to Shell Commands.
Shell commands are simply commands that are typed into the shell. As our very first shell command to try, let’s try the
echo command which simply echos or displays the argument(s) to the terminal. Try the following with or without the quotes:echo "I love open source!"
You should have seen the text "I love open source!" printed in the terminal. Now if that were all the
echo command could do, it would be a fairly pointless command. The echo command’s real power is in displaying the contents of variables and files as we will soon see.The shell is designed to work in an OS which is a multi-user environment. Type in the following command to see your username:
whoami
The shell should respond with the username that you created when you set up your system. For me that is
pearcej.Note that your environment might be case sensitive, which means that it might discriminate between uppercase and lowercase letters, where two words that in letter cases only, are not considered equal. This means that
Echo, ECHO, WhoAmI, and WHOAMI might not be recognized as commands, so you are in a case-sensitive environment and accidentally type one of these, you will elicit a response like:Command 'WhoAmI' not found
or if you are in a case-sensitive environment and you are lucky, you might get something more helpful like:
Command 'Echo' not found, did you mean:
command 'echo'
Sometimes it is helpful to be able to remember some information or look up previously stored information. This can be accomplished using environment variables. An environment variable is a value that contains information about the system environment or the currently logged-in user. Having environment variables is a way of keeping track of information about the system environment and serves as a way to pass information about the environment to programs. For example, the operating system sets a bunch of environment variables which store information about the environment. As just one example, you might wonder how the operating system keeps track of its users. For each user, there is a user environment variable called USER that is specific only to the currently logged-in user. The shell will expand environment variables and that `echo` is useful to show us the results of that, so the USER environment variable can be accessed using the
echo command that we already saw. Try the following:echo $USER
You should see the same result as you saw with the
whoami command! Note that you must precede the variable name with a dollar sign $ whenever you reference the value it contains. Isn’t it interesting to peek inside the environment variable in that way? In fact, one of the key uses of the
echo command is to echo the values of variables, which the shell can expand. Let’s see another example:echo $SHELL
You will likely see something like the following printed:
/bin/bash
The reason you see something like this is because the SHELL environment variable stores the location of the shell program. This location, called the file path, is given in the form of the path needed to find the location. We explore how this path is described in the next section.
Files and File Systems.
Before computers became popular, a file and a folder referred to a container for a collection of documents. Operating systems use these terms very similarly. A file refers to a document and a folder or, synonymously, a directory refers to a collection of files and sub-directories. The operating system handles how files and directories are represented on a physical storage medium, how they are read from, and written to. You may be most familiar with accessing files through a graphical user interface. Creating a new document corresponds to creating a new file. Often, when opening an application that deals with documents, it will suggest documents you have recently worked with. These each correspond to a file, each of which is stored in a directory somewhere, and knowing where files are in your file directory will be important to working on open source.
You might (or might not) be familiar with using folders to organize our files and to navigate the file system. (On an iphone or Android phone, you make a new folder by dropping one app on top of another app, on a Mac you long press and then choose new folder, and on a Windows machine, you right-click and choose New > folder.) This type of organization is important to understand, and the folder metaphor is useful. In fact, there are two useful metaphors to help with our understanding of file systems.
When using the shell, "folders" are more typically called "directories", but the two terms are equivalent.
Inside of a directory, we can find files and possibly additional directories. Since directories may contain other directories in addition to files, the directories on your computer form a hierarchy. Each directory is contained within another directory, which we refer to as its parent. The only exception to this rule is the root directory, which can be a parent to other directories, but has no parent itself. It is therefore the top-level directory from which we can reach any other directory. Each file and directory in the file system can be referred to by its unique path. The root directory always has the path /, and a file can thus be identified by its path through the file system beginning from the root node. If there is a folder named
myfolder located in the root directory, its path would be /myfolder. If myfolder then contains a file named hello.txt, its path would be /myfolder/hello.txt. If /myfolder contains a directory named /mydocuments, its path would be /myfolder/mydocuments. And so on." Note that the term root directory shouldn’t be confused with the term root user in a Linux-like system. The root user is a user named root, which is a special superuser account in Linux operating systems that has unrestricted read and write privileges to everything. The root directory should also not be confused with the
/root folder which is found on some Linux-based systems and which is the home directory of the root superuser.Let’s get our bearings and see which directory we are currently in. The
pwd command stands for print working directory and as the name says it prints the path of the directory you are working in. Try typing:pwd
When I type this, I see the following (but you will likely see a different path):
/home/pearcej
Reading the path from right to left, using the folder metaphor, this means I am in a directory named
pearcej which itself is a inside of a directory called home which is itself inside of the root directory, /. Note that the position with respect to the forward slash / tells us the direction of the relationship – further to the right is further inside. To use the tree terminology, a directory that has a file or second directory directly inside of it is called a parent of the file or second directory that is inside, while the file or second directory is called the child of the parent directory. Hence, pearcej is a child of home and home is a child of the root directory / in our example.Let’s try "moving" (i.e. changing our working directory) to the root directory. Use the
cd command which stands for change directory followed by the /:cd /
This should transport you to the root directory (if you weren’t there already.) Now when you use
pwd, you should see / echoed back.Next type the following
ls to get a listing of the files and subdirectories in the current working directory.ls
You should see a listing of files and directories.
Shell Command Options.
Many shell commands have options which can be invoked by using a command option, which is also called a flag. Let’s try using a command option with
ls, the listing command. Try typing the following where the -r option stands for reverse, and it reverses the sorting order:ls -r
To try another, try typing the following where the
-l option stands for long listing format, meaning that instead of output containing only the names of files and directories, the ls command will produce additional information, assuming you have at least one file or directory.ls -l
You are likely to see some lines that look something like this, each of which corresponds to one file or subdirectory in your current working directory. The exact output you get will vary based on the directory you are in and what files and folders exist on your machine:
drwxr-xr-x 3 root root 4096 Jun 19 10:55 home lrwxrwxrwx 1 root root 7 Mar 24 2022 bin -> usr/bin -rw-rw-r-- 1 pearcej friends 22 Sep 15 2022 hello.txt
Thus, the
-l flag changed how the listing is displayed.File Permissions.
Let’s return to the example from the previous section by exploring what we get in a long listing. In particular, let’s explore the permissions portions of the lines produced by the
ls -l command.cd ~ ls -l
You should get something that looks similar to the following, of course your files and directories are likely to differ.
drwxr-xr-x 3 pearcej pearcej 4096 Jun 19 10:55 home lrwxrwxrwx 1 root pearcej 7 Mar 24 2022 bin -> usr/bin -rw-rw-r-- 1 pearcej friends 22 Apr 15 2023 hello.txt
The last item on each of these lines is the filename itself. The first item in each of the lines is a 10 character-long mix of letters and minus signs. The first character of that set indicates the file type where
d tells us it is a directory, l indicates it is a symbolic link, and - indicates it is a regular file. (Note that a symbolic link, aka symlink or soft link is a special type of file that acts as a shortcut by pointing to some other file or directory.) The next nine characters display various permissions for the file. The first set of three characters (characters 2-4) are the user’s (i.e. the file owner’s) permissions, the next three are the group’s permissions, and the last three are the permissions for all others. (The user’s name is in the third column and the group name is in the fourth column.) In each of these sets of three characters, the three characters refer to read (r), write (w), and execute (x) respectively. If you see a letter then that permission is allowed, and if you see a ’-’, the permission is disallowed. For example in the last of the three lines shown above, we see that the file hello.txt has permissions of -rw-rw-r--. This means this is a file that can be read and written (changed) but not executed by the owner pearcej and also by any user who is a member of the friends group, however other users can only read the file.Occasionally, you need to change permissions of a file. For example, you might need to change permissions to make a file executable. Changing file permissions is done with the
chmod command. We will explore this in a bit.Learning More About Command Options.
If you want to see all of the command options for a given command, there are two different methods for many commands. You can often, but not always, use the
--help option on the command or you can use the manual which is accessed using the man command. For example, with the ls command, you can use either one of the following, noting that they work a bit differently because the manual may use paging. If it does, to go to the next page use the space bar.ls --help man ls
Note that since both
--help and man are not consistently both available, it is worth knowing both. That way, if one doesn’t work, you can try the other.You may be wondering about the use of one minus sign ’-’ vs the use of two minus signs ’--’ in shell options. One minus sign is typically used for single letter commands, while two is typically used for word-length commands, although even this is not a hard and fast rule. You’ll occasionally get programs that also use a single `-` for word-length commands. If you look back at the help and/or manual results for
ls, you may see that the options -a and --all both list all files including that begin with a ’.’, which are the hidden files and directories. Besides brevity, one advantage the single minus sign often offers, is that to run multiple options at the same time, all you often need to do is concatenate them. Note that this is also program-specific; not all programs will allow this, but it is convenient when it is offered! For example, ls -lra returns the result with the -l, -r, and -a option flags all activated. Give it a try!ls -lra
Creating and Removing Directories.
Next, let’s go to your home directory which is referenced by the special tilde ’~’ character as follows:
cd ~
Then print your working directory and list your files just to see that you have been transported again.
You can make a new directory using the
mkdir command. For example, let’s say you want to make a new directory with the name newdir, you can type the following:mkdir newdir
To check that the directory was created as expected, you can get a fresh listing of your files and directories with
ls -l. The output should include a line that ends in `newdir`. Try it!drwxr-xr-x 2 pearcej pearcej 4096 Nov 15 17:00 newdir
Then you can descend into the new directory and see where you are with:
cd newdir pwd
You should note that your working directory has changed! One might expect not to see anything when running the
ls command in an empty directory, but try it with the following options:ls -la
You are likely to see something like the following:
drwxr-xr-x 2 pearcej pearcej 4096 Nov 16 17:00 . drwxr-xr-x 13 pearcej pearcej 4096 Nov 16 17:00 ..
Note the period at the end of the first line where the filename is expected. A period or dot (.) represents the current directory. The double period or double dot (..) represents the parent directory of the current one.
You can use this dot notation as a reference as well, so
ls .., will list all the files and directories in the parent directory relative to where you are so you don’t have to change directories to get the listing from another directory. Try it!You can even use these dots in combination with folder names. For example, if you are inside of the
newdir directory and you want to make a sibling directory called newdir2, you can type the following:mkdir ../newdir2 ls -l ..
Here, the first command creates a new directory named
newdir2 as a child of our current parent directory. This makes newdir2 a sibling of the original newdir directory because both have the same parent directory. The second line produces a long listing of all of the files and directories in the directory that is one level up relative to the current working directory. You should see both of the directories that you made listed as subdirectories.If you want to delete either of these new empty directories, you can use the remove directory command,
rmdir followed by the name of the directory you want to remove. Note that there are other ways to remove directories, but remove directory is useful because it refuses to delete a directory that is not empty. Give it a try!A handy use of the
cd command is to use cd - which will take you to your previous working directory. This is handy if you need to move between two distant locations. Try it followed by pwda couple of times.cd - pwd
Input and Output Redirection.
There are a number of ways one can create a new file. Let’s try some. First, use
cd to descend into one of your new directories if you are not already in one so that you are in an empty directory. Then let’s try the following commands:touch newfile1.txt echo 'I love open source!' > newfile2.txt echo 'I really love open source!' >> newfile3.txt ls
You should now see three new files named newfile1.txt, newfile2.txt, and newfile3.txt respectively. This particular usage of the
touch command simply makes an empty file if no file by that name already exists in the directory. It is a niche use case for the command, but people use it this way with regularity. The intended purpose of the touch changes the last accessed/modified date. Understanding the result of the first line we typed is pretty straightforward. The second and third lines that we typed require a bit more explanation. As we learned above, the echo command simply outputs (or echoes) the argument(s) to the terminal, but here the echo command has been used in combination with output redirection into a file.
Output redirection is a feature in the shell that allows the user to redirect the output of a command using
> or >>. Recall that the echo command normally prints to the screen. What we did above in creating new files with the echo command was to use output redirection to redirect the output of the command into a file of the specified name. So, the command echo 'I love open source!' > newfile2.txt redirected the output of the echo command into the newfile2.txt file. Both > and >> will create a new file with the provided name if one does not already exist. Note that the > is the output redirection operator used for overwriting a file that might or might not already exist, while the >> is an output operator that appends the output to an existing file or creates a new one if one does not already exist. So, > should be used judiciously! Let’s look at the contents of the files that we created. Let’s use
cat to see the file contents. Type the following one at a time:cat newfile1.txt cat newfile2.txt cat newfile3.txt
You might be wondering why we are using a command called
cat. The cat command, which stands for concatenate, is used to concatenate standard input (typically the keyboard) or file(s) to the standard output. Like the name suggests, you can also use the cat command to concatenate files. For example, if you try:cat newfile2.txt newfile3.txt > newfile1.txt cat newfile1.txt
You should see that newfile1.txt instead of being empty, now contains the concatenation of newfile2.txt and newfile3.txt
You can also use the
cat command in combination with output redirection to create a multi-line file. Try typing the following lines to create a new multi-line file named newfile_multi.txt:cat > newfile_multi.txt these are multiple lines
You can complete this command sequence by pressing
Control+D on your keyboard which will cause the new file to be closed. You should now have a new file called newfile_multi.txt.Analogous to output redirection, input redirection in the shell using
< or << allows you to redirect the input of a command. So, if we want the response from a command to be written to a file instead of to the terminal, we can use output redirection, but if we want the input to a command to come from a file instead of from the keyboard, we use input redirection. Let’s look at an example of input redirection using the shell command
wc, which stands for word count. This a command that as the name suggests can be used for counting. However, it does more than count words! It actually provides the line count, the word count, and the character count in the file(s) specified in the file arguments. By default, it displays all this in four-columnar output with the file name in the final column.Let’s try the following:
wc < newfile2.txt
The shell should reply with
1 4 20 newfile2.txt because newfile2.txt contains ’I love open source!’ which is 1 line, 4 words, and 20 characters. If you only want to count the number of words, you could use the -w flag to display only the word count as follows:wc -w < newfile2.txt
Let’s practice with input and output redirection using a temporary file. Let’s redirect the output of the
ls command into a temporary file, and then use input redirection with wc -w to get the word count of this file. (This is one way to count the number of files in your current directory.) Let’s try the following:ls > temp.txt wc -w < temp.txt
If you are wondering if there is a better way than using a temporary file, there is. A pipe (|) in the shell allows you to redirect (aka to pipe) the output of one command into the input of another command. Let’s see an example:
ls | wc -w
Try it! This is an improvement because it is faster and avoids the use of a temporary file.
Scripts.
It is possible to use what we have learned thus far to make an executable script. A script is a text file that contains a sequence of commands for the operating system. Shell scripts can be useful to bring together common sets of tasks and to make repetitive tasks faster and easier.
Let’s try it! Typing
cat >> helloworld.sh will tell the shell to open a new file named helloworld.sh and to expect multiple lines of input from the keyboard. Type the following lines:cat >> helloworld.sh # This is a comment in a script typically used to explain the purpose echo 'Hello World!'
End the
cat >> helloworld.sh command sequence by pressing Ctrl+D on your keyboard. This will cause the file named helloworld.sh to be completed and closed.You should now have new file named
helloworld.sh. However, we would like to make it executable so we can run it as a script. To make it executable, we use the chmod command. The chmod stands for change file mode bits. To add execution, we will need the +x option, which stands for ’add execution’ and will change it to being executable. So, run the following in your shell to make it an executable file for all users:chmod +x helloworld.sh
To execute this script, you need to indicate the directory where to find it, which is the current directory. So, to run it, you can type the following:
./helloworld.sh
We can create a more interactive script using the
read command which reads text from the keyboard. It is frequently used to make scripts interactive. However, we need a variable to store the result. In the shell, some variables, like environment variables, always exist and you can always access them. However, you can also create your own variables. As we have seen, you can use echo to ask the shell to provide the value of a variable, but you must precede the variable name with the dollar sign ($). Let’s see how all this works by trying the following, finishing by pressing Control+D:cat >> hellouser.sh # Says hello to the user by name. echo 'What is your first name?' # In the next line, USERNAME is dynamically created as a new variable read USERNAME echo "Hello " $USERNAME
To run this new script, you will again need to make it executable. Then you can run it.
chmod +x hellouser.sh ./hellouser.sh
File Management.
We can move files around, make copies of files, or remove (delete) files from the filesystem using
mv, cp, and rm respectively. Let’s see how these work. First, list your files. You should have newfile1.txt, newfile2.txt, and newfile3.txt in your directory from the previous exercise.Let’s make a new directory named subdir below our current directory and move one of our files there.
mkdir subdir mv newfile3.txt subdir/newfile3.txt ls ls subdir
Note that the command
ls subdir provides a listing of the subdirectory named subdir. You should see that newfile3.txt is now not in the current directory, but it is instead in the subdirectory subdir.In addition to moving files around in the directory structure, we can also use the
mv command to rename files as follows:mv newfile1.txt newfile4.txt ls
One important thing to watch out for with the
mv command is that if the destination filename already exists, it might get overwritten. For this reason, you might want to use the -i flag which stands for interactive. Try it.mv -i newfile1.txt newfile2.txt
You should see a question like, "overwrite ’newfile2.txt’?"If you respond with "n", then the file will not be overwritten.
The
cp command works as you might expect:cp newfile2.txt newfile2_cp.txt cat newfile2_cp.txt
Just like with the
mv command, the cp command will overwrite the destination file if it already exists. For this reason, you might want to use the -i flag which stands for interactive here too. Try it.cp -i newfile4.txt newfile2_cp.txt
You should see a question like, "overwrite ’newfile2_cp.txt’?" If you respond with "n", the command will not be overwritten, so using the
-i flag seems a wise safeguard.The
rm command is a useful, but another dangerous command. Let’s try it:rm newfile2_cp.txt
Just as with
cp, and mv, you are probably wise to use the -i option.rm -i newfile4.txt
And, if you respond with "n", then the removal will not happen.
A few time-saving shell commands.
The up arrow key retrieves the previous shell command. If you press it multiple times, it will take you back through multiple commands in your shell history. This is a useful way to repeat a command. For example, if you had a typo, you can use the up arrow, edit the command, and push enter to fix the command. Analogously, the down arrow will move you in the reverse direction through the shell command history. For more useful shell commands, type
man bash for hints on how to search your shell history, re-execute commands, and much more.An additional efficiency-enhancing feature is the history command. This command conveniently presents a record of previously executed shell commands, enabling users to effortlessly revisit their command history.
Here’s an example of what might be displayed when the
history command is executed:1 git init 2 git add main.c 3 git commit -m "Initial commit" 4 git remote add origin https://github.com/username/repo.git 5 git push -u origin master 6 history
This paragraph is intended to alert you to some useful search features. A couple examples of very common search patterns are using wildcards for zero or more characters or for a single character. The asterisk (*) specifies zero or more characters to match. In bash the question mark (?) is used for matching exactly one single character.
For example, if we type the following:
rm -i newfile?.txt
Then the question mark will match with any single character, and we will see the following prompts:
rm: remove regular file 'newfile2.txt'? rm: remove regular file 'newfile4.txt'?
If we instead type:
rm -i newfile*.txt
Then the asterisk will match with any number of characters (including zero), and we will see the following prompts:
rm: remove regular file 'newfile2.txt'? rm: remove regular file 'newfile4.txt'? rm: remove regular file 'newfile2_cp.txt'?
As you can see, these search patterns give you a lot of power and control.
Conclusion.
Hopefully, you now feel a bit more comfortable using the shell. The shell commands discussed above are summarized in Appendix Section B.1.
Checkpoint 3.2.2.
You have a file named data.txt that contains the following lines:
apple orange banana grape
Your task is to create a new file named fruits.txt and copy the contents of data.txt to fruits.txt. Next, append the word
kiwi to fruits.txt. Then, display the contents of fruits.txt in the terminal. Next, count the number of fruits in fruits.txt and display the total count. Finally, overwrite the content of data.txt with the content of fruits.txt and rename data.txt to a new file named fruits2.Here are some commands you may need to perform the task. Rearrange the correct commands in the correct order by dragging and dropping.
You have attempted of activities on this page.
