Introduction to Shell

In order to effectively work on the SPS website, you will need to have access to a shell. This guide will help provide a minimal introduction to using shells. Since our access to the OCF computers (where we setup the development environment) is over SSH (Secure SHell), this will be necessary in order to effectively debug the website.

The most common shell (at the time of writing) is bash, the Bourne Again Shell. Bash is the default shell provided on the OCF and the only shell available through git bash (for Windows). Most Linux distributions use bash as their default shell, but a few distributions have switched to zsh, the Z shell. The Terminal program in Mac OSX uses zsh by default. These instructions will be written with bash in mind, but they should work very similarly for zsh.

A common convention for shell commands is to specify a replaceable string by square braces, e.g. [username] would represent your username (carterturn for me).

Another common rule of shells is that pressing Ctrl+c will cancel the current operation. Sometimes you will have to press it multiple times.

SSH

It will be useful to have access to a shell during this introduction. You should try to run these commands on the OCF computers while you read through this document. To open a shell on the OCF comptuers, connect via SSH. To do this, you will first need to open a shell on your computer. On Mac OSX or Linux, this can be done by running the Terminal program. On Windows, this can be done by opening git bash. Once you have done so, run the following command:

ssh [your OCF username]@ssh.ocf.berkeley.edu

The Shell as a File Browser

The most common use of the shell is to browse files, copy and move files, or delete files: the same functions that a typical GUI file browser performs. This section will go over some of the commands used to do this.

Change Directories: cd

The first command is cd, which stands for "change directory." Most shell commands take one or more arguments that tell the command what to do. cd takes a single command, the path to the directory to change to. A path can be specified in a few different ways. The most common way is relative: if a path does not start with /, the path will be relative to the current directory. If a path starts with /, the path will be relative to the root directory of the system. Finally, if a path starts with ~, the path will be relative to the home directory (on most systems, this is /home/[username], on the OCF, home is /home/[first letter of username]/[first two letters of username]/[username]. Try running cd public_html. This will change your current directory to the public_html folder. You will notice your prompt (the part before your cursor) change to include public_html. By default, the prompt shows your current directory (~ for home). If you run cd with no arguments, you will change back to your home directory. Run cd with no arguments now.

List Files: ls

You have probably noticed that cd can only be used to change directories if you already know what directories are there. To view files and folders in your current directory, use the command ls (short for "list"). If you run this command in your home directory, you should see the public_html folder. ls can also list the contents of other directories. If you specify a path, similar to cd, ls will show files in that folder (or just the file at that path if a path to a file is specified). In addition to the path argument, ls has a variety of flags: arguments that can be used to modify its behavior. To get a list of the flags, run the command ls --help. You should see a lengthy explanation of the available flags. Many flags have a short and long form. In general, the short form is a single dash followed by a single letter, while the long form is two dashes followed by a word (most commands use this convention, but there are some exceptions). An important flag for ls is the -l flag. This changes ls to a list mode, outputting additional information about the files and folders at the path. -a is also useful. This shows hidden files and folders (files and folders with names beginning with a .). Flags can be combined. Try running the command ls -la in your home directory (note that the short flags can be combined with only a single - preceding them. You should see a folder called .. This is the current directory. The folder called .. is the next directory up (the folder with the first two letters of your username). Note that you can use . and .. as part of paths (e.g. for cd).

Move, Copy, Remove: mv, cp, rm

With cd and ls you can move around and see files. Next, you will likely want to move, copy, and remove files. The commands for this are mv, cp, and rm respectively. The mv and cp commands are quite similar (except that cp copies instead of moving). To see how these work, we will first create an empty file. Run the command touch test.file to create an empty file named test.file. Run ls to check this. Let us then copy that file. cp copies all of its non-flag arguments to its last argument. If the last argument is a folder, cp copies the file or folder into the folder. Run cp test.file test_2.file. This creates another empty file (or to be precise, a copy of the empty file test.file) named test_2.file. Next, let us copy a file into a directory. Run the command cp test.file public_html/ to create a copy of test.file in the public_html folder. Sometimes you want to copy a folder. cp requires the flag -r (for recurse into folders) to do this. You can see this by running cp -r public_html/ new_folder. This will create a new folder in your home directory named new_folder, which should contain a file named test.file already. mv is quite similar to cp, so we will not go into detail on its use here. Let us now use rm to cleanup all the files we just created. First, run rm test.file test_2.file. This will delete the files we made in the current directory (verify this with ls). Note that rm will allow you to remove multiple files at once. Next, run rm public_html/test.file. Finally, we remove new_folder. If you try to do this with rm new_folder, you will get an error message: rm does not allow you to just remove folders. To delete folders, run rm -r new_folder (again, -r means recurse into folders). On the OCF computers, you will be prompted for each file that rm attempts to remove. This is a safeguard against accidentally deleting large sets of files. Type Y and press enter to confirm deletion (or N to not delete if something seems wrong).

Creating: touch, mkdir

The command touch is used to create an empty file if none exists, and updates the timestamp on a file if it does exist (many system utilities use timestamps to determine if a file has changed, so this can be useful for telling a system utility to re-run. See update_website_debug.sh for an example of this behavior). The command mkdir [folder name] is used to "make" a "directory". If you are trying to delete an empty directory, the command rmdir can be used. rm -r can also be used to delete empty directories.

Git Commands

In this section, we will review the commands used for git. All git commands start with git. Nearly all git commands have a --help flag that can be used to get more information about a command.

Cloning

The first git command you will usually run is clone. The full syntax is git clone [remote url]. This will clone a remote url into a local folder, with the folder name based on the remote url. If you want to clone to a local folder with a different name, you can use the -o [folder name] flag to specify a different local folder name.

Status

When you are in a git repository, the command git status provides useful information on the current state of your local copy of the repository. The first line of the output shows the current branch. After that, there are three lists (not all of them will be visible). The first list is a list of changes that are staged. Staged changes will be included in a commit when you run git commit. The second list is a list of changes that are staged to be included in the next commit. The third and final list is a list of files that are not part of the git repository. These files will not be included in the next commit unless they are staged.

Branching

As you have probably heard, git uses a system of branches. A branch is effectively a set of commits in a chronological order with each commit being dependent on the commits before it. Two different branches have a different set of commits. To create a new branch, you can use the command git checkout -b [new branch name]. The -b flag is needed the first time a branch is checked out. After a branch is created, you can use the command git checkout [existing branch name] to switch to it. When you checkout an existing branch, you should pull the latest version from the remote repository (Github). A pull is actually a combination of several steps, but they rarely need to be done separately, so we will defer a discussion of them. You can pull a branch by running git pull.

Adding

The command git add is used to stage changes to be committed. The simplest form is with the --all flag. This will stage all the changes you have made (including new files). If you only want to prepare some changes, you can specify a path or set of paths, for example git add file1.html folder1/file2.html. If you only want to add part of a file, you can use the -p flag. This will open an interactive editor that allows you to select parts of a file.

Committing

Once all the changes you want staged are staged, run git commit to make the commit. By default, the commit message is entered in interactive mode, so git will open the default text editor. For more information on common editors, see the Editors section of this page. git commit has an option to provide the commit message on the command line using the -m (for message) flag. For example, git commit -m "Initial commit" will make a commit with the message Initial commit without opening an editor.

Pushing

Once you have added a commit to a branch, you will want to update the remote repository. This is done by pushing the branch using the command git push. The first time you push a branch, you also need to specify the remote repository and remote branch. This is done with the --set-upstream [remote repository] [remote branch] flag. The remote repository is identified by a short name, nearly always origin. The remote branch should usually be the same as the local branch. The full command is git push --set-upstream [remote repository] [remote branch]. After the first push, you can just use git push; git will remember the remote repository and remote branch.

Merging

You will eventually want to combine your branch with dev-stable. We will make changes to dev-stable using Pull Requests in the Github web interface, but in order to make adding code to dev-stable clean, you want to minimize the changes you have relative to dev-stable. You do this by first merging dev-stable into your branch. To do this, we want to latest copy of dev-stable. To get this, run git checkout dev-stable, then git pull. Then, checkout your branch again. Finally, run git merge dev-stable. This will bring the code from the latest version of dev-stable into your branch. If all goes well, a text editor will open will a commit message for the merge already loaded. You can just save and close this to complete the merge. If you have edited a section of a file that has also changed in dev-stable, you may get an error indicating that it was not possible to complete the merge automatically. In this case, contact Carter. After completing the merge, run git push to sync the merged code to the remote repository.

Editors

Sometimes you want to edit files from the shell. To do this, you can use a text editor. All editors take a file as an argument, for example [editor name] [file] will open a file in an editor. The three most common editors, all available on the OCF computers, are nano, emacs, and vim. nano is the easiest to use; all of nano's shortcuts are written on the button of the screen when nano is running. emacs and vim are much more powerful (it has been joked that emacs is practically an entire operating system disguised as a text editor, which may be fair, considering that it has the game snake built in), but rely on a variety of keyboard shortcuts that are only apparent after reading their manuals. Reading the shortcuts can be accomplished in a few hours, but memorizing them tends to take a few weeks. A few important shortcuts for these editors are listed below.

Emacs

To start entering text in emacs, just start typing. You can move around a file using the arrow keys (though this is not the most efficient method, it is the easiest to remember). To save a file, use the keyboard shortcut Ctrl+x Ctrl+s (that is, hold down Ctrl, then press x, then release x, then press s, then release s, then release Ctrl). To close emacs, use the keyboard shortcut Ctrl+x Ctrl+c. emacs will prompt you if the file is unsaved. emacs is the best text editor, but it does take some time to learn.

Vi/Vim

vi and vim (Vi iMproved) are very similar text editors. In many cases, the command vi simply opens vim. I will use vim to refer to both here. vim is a modal text editor, which means that some commands only work at some times. Pressing Esc several times will let you exit to the "root" mode. To start entering text, press i (for "insert" mode). Depending on your keyboard configuration, the arrow keys may not allow you to move around a file while in insert mode. To leave insert mode, press Esc. To save a file, leave insert mode, then enter :w (to write a file). To quit, enter :q. To quit without saving, enter :q!.

Keyboard Shortcuts

Using a shell may seem tedious due to all of the typing required to make your instructions explicit. Fortunately, there are a variety of keyboard shortcuts that make shell usage much faster (faster than GUI usage in most cases). The keyboard shortcuts here apply to the bash shell; similar shortcuts exist for zsh, but I am less familiar with them.

The most important keyboard shortcut is the tab key. If you press tab, bash will attempt to automatically complete your current command or path (sometimes, bash is even able to complete flags. git also has excellent tab compatibility, try pressing tab in the middle of typing a git command to see what happens). This does not work too well with two letter commands, but for anything longer than three letters, it is worth trying to press tab to see if bash can automatically figure it out. If you have two files or folders that begin with the same letters, bash will fill in as many letters as are in common. Once you have cloned our repos, trying typing cd s, then pressing tab. It should automatically fill in cd sps_web_. If you then type 2 and press tab again, the entire command cd sps_web_2020 will be filled in. You only had to press seven keys!

If you use mice frequently while typing, you probably find yourself clicking on the middle of phrases to fix a typo. Your mouse does not work in the shell, but there are keyboard shortcuts for rapidly moving around a line. Ctrl+a takes you to the beginning of a line, Ctrl+e takes you to the end of a line. Alt+f jumps forward a "word" (defined as a set of letters and numbers), Alt+b jumps backwards a word. On some systems, Alt+Backspace backspaces an entire word (this depends fairly strongly on your keyboard layout).

The shell also has its own built-in cut and paste functionality (usually referred to as "kill" for cut and "yank" for paste). Ctrl+u cuts everything to the left of the cursor. Ctrl+k cuts everything to the right of the cursor. Ctrl+y then pastes that content.

bash maintains a history of commands you have run. If you press the up arrow (or Ctrl+p), the previous command will be loaded, ready to run again. You can keep moving up through your history. Use the down arrow (or Ctrl+n) to move forwards in history. You can even search backwards in the history by pressing Ctrl+r, then (with Ctrl and r released) entering a phrase. History is retained between sessions. If you log out and log in again, your history will still be there. If you visit the same directories often, using history may be faster than typing a cd command.

Conclusion

Hopefully this guide has provided you with enough information on using shells to provide "a landing you can walk away from". This is nowhere near a complete description of what shells are capable of. All shells have complete programming languages built in, with if statements, while and for loops, and variables. Nearly anything (except viewing pictures and watching videos) can be done entirely from the shell, then automated. If you would like to know more, the Advanced Bash-Scripting Guide is a good resource. Some other very powerful tools are sed and awk; they are installed with most shells.