Command-line Ninjary GSPS 2013

This is meant to be a quick reference guide to command-line usage for those of us who have the basics down, but aren't yet true ninjas. Please add your own tips or quick examples! We spend a surprising amount of our lives inside a terminal, so learn to love it and use it well. Note: this page was written for bash, but much of this stuff should translate to csh or your shell of choice.

General notes about navigation commands: First, “Ctrl” commands are the same as in the text editor emacs, so you may find that other emacs line navigation commands work as well. Second, if you are on a Unix machine, note that the line navigation commands work almost anywhere you can edit text - you will learn that accidentally if you start using these commands compulsively. Third, if you use “Ctrl” commands often, you may consider setting the “Caps Lock” key as an additional “Control” key - this is just a computer setting for Unix (e.g. System Preferences > Keyboard > Modifier Keys) but requires slightly more work on Windows (e.g. the AutoHotkey program).

• cntrl-a: go to beginning of line
• cntrl-e: go to end of line
• cntrl-k: delete everything to right
• cntrl-u: delete everything to left
• cntrl-w: delete word to left
• (option or cntrl)-(left or right): move left/right. cntrl for linux, option for mac.
• Ctrl-f: move forward one character
• Ctrl-b: move backward one character
• use backslash \ to literally interpret any command characters

• cd : go home
• cd - : go back to previous working directory
• ~ : stand-in for home directory
  • e.g.: ls ~/Documents

• !!: repeat previous command
• !xxxx: repeat most recent command that starts with xxxx
• cntrl-r xxxx: search recursively through history for match to xxxx
  • keep typing to refine search==
  • cntrl-r again to cycle through matches
  • and hit enter to execute
• ^this^that: rerun most recent command, but replace this with that
• history: shows history of recent commands
  • history | grep xxxx: search for command xxxx in history
  • !xxxx: repeat item number xxxx in history

• cntrl-x-e: invoke your editor to compose a hairy command
  • executes command upon exit
  • can be invoked in the middle of composing command

• curly braces perform parameter expansion, inserting a copied string with contents of braces swapped
  • cp file.{txt,txt.backup}: same as cp file.txt file.txt.backup
  • cp file_{,copied}one.txt: same as cp file_one.txt file_copied_one.txt
  • rm file_{a,b,d,g}.txt: same as rm file_a.txt file_b.txt file_d.txt file_g.txt

• clear: clear the terminal screen
• cntrl-z: shunt something to the background
  • fg: to get it back
  • really helpful when in some other program like a text editor or python/IRAF terminal

• man xxxx: display manual for xxxx
• (which or whereis) xxxx: display path for xxxx

• wc xxxx: character, word, and line count on file xxxx
  • wc -l: return only line count

• cat xxxx: display contents of file xxxx
• head -n 10 xxxx: display first 10 lines of file xxxx
• tail -n 10 xxxx: display last 10 lines of file xxxx

• grep xxxx yyyy: search for xxxx in file yyyy
  • xxxx can be simple string or regular expression
• find path xxxx: search for files with names matching xxxx
  • path is optional - defaults to working directory
  • xxxx can be simple string or regular expression

• kill xxxx: kill (end) process with process id xxxx
• top: display running processes and system info
  • q: quit
  • k xxxx: in linux, kills process with id xxxx
  • sadly, no equivalent to above on most macs
• ps -e: show all running processes
  • ps -e | grep xxxx: search for xxxx amongst running processes

• date: print date & time
  • can choose output format - i.e.: date +%H:%M:%S
• crontab: fully-functional command scheduler
  • good examples here
• at: a simple one-off scheduler
  • cmd | at time : schedule cmd to run at time
  • i.e.: ls ~/* > ~/home_dir.list | at 3:05pm
  • understands pretty broad range of time definitions

• screen: like VNC but only for the terminal, lets you start a session and then disconnect and reconnect at will
  • screen to connect
  • screen -d to disconnect
  • screen -r to reconnect
  • screen -list to show current sessions
  • screen -r xxx to reconnect to session with PID xxx
  • great for ssh sessions
  • full guide here

Best way to install and manage command-line tools. Native for all linux builds; have a couple good choices for macs.

• Homebrew
• MacPorts

• cmd & : run cmd in subshell
• cmd1 ; cmd2 : run cmd1 then run cmd2
• cmd1 && cmd2 : run cmd2 after cmd1 only if cmd1 is successful
• cmd1 || cmd2 : run cmd2 after cmd1 only if cmd1 is not successful

• cmd > file : sends output of cmd to file
  • overwrites file
  • cmd >> file : appends output of cmd to file

• cmd1 | cmd2 : connects stdout of cmd1 to stdin of cmd2

• cmd 2> file: sends error of cmd to file
  • cmd 2>> file : appends error of cmd to file
  • e.g.: ls not.a.file* 2> output; cat output
• cmd > output 2> errors : save output and errors of cmd separately, in files output and errors
  • e.g.: ls * not.a.file* > success.out 2> failure.err

• cmd < file : sends file to cmd

• sort: sorts stuff alphanumerically
  • ls | sort

• xargs cmd : a piping glue - runs cmd with standard input as arguments
  • can be used with a pipe or with a file
  • use it to translate between lines in a file or pipe and arguments to a function
  • best to understand by example
  • echo one two three > test.txt; xargs echo < test.txt
  • echo -e 'one\ntwo\nthree' > test.txt; xargs echo < test.txt
  • ls | sort | xargs echo

• awk: programming language to edit pipes on the fly
  • basic usages are powerful, learn more here
  • awk '{print $1}: print the first item of each line, items split by spaces/tabs
  • awk -F, '{print $2}: print the second item of each line, items split by ,
• Example: search for all running python instances, find their PID, and kill them
  • ps -e | grep python | awk '{print $1}' | xargs kill
• Example: count and display all the items in current working directory
  • ls | awk '{sum+=1; print sum “: ” $1}'
• Example: replace every occurrence of NA with the number 0 in a file
  • cat file.txt | awk '{gsub(“NA”, “0”); print $0}' > clean.file.txt

Variables that have already been declared are prefaced by a $, but you don't need one when first declaring a variable. Any variables declared normally (x=4) are considered strings, to declare a number use the let keyword: let x=4. You can also declare a variable to be a command by wrapping it in back-ticks or parentheses. I.E. LIST=(ls); $LIST. There are some system-wide variables that should always be available, like $SHELL, $PATH, and $HOME. Note that though convention says system-wide variables should be all caps, they don't need to be.

There are several ways to write loops, the most useful of which are below. Note that in bash an $ always indicates a variable, and that all variables are, by default, treated as strings. The actions to be repeated must be wrapped in do/done statements, and the semicolon marks an end-of-statement (equivalent to a line return). A few examples:

for i in {1..20}
  do
    echo $i
  done

for i in {a,b,d,g}; do echo $i; done

for i in a 2 b 6;
  do
    echo $i;
  done

for file in *filePattern*;
  do
    echo file;
  done

The shell can do arithmetic and logic tests, but I think if you're doing anything very complex you should move into Python or Perl. Logic comparisons should be put in square brackets (spacing is important), if statements must be closed with a fi, and the operators you have at your access are:

• ==: string equality
• !=: string inequality
• -eq: number equality
• -ne: number inequality
• -gt: greater than
• -lt: less than
• -ge: greater than or equal
• -le: less than or equal
• &&: and
• ||: or

Use aliases to make hard-to-remember commands less terrible.

Example: http://xkcd.com/1168/:

• alias maketar='tar -czvf '
• alias untar='tar -xzvf '

You can declare functions and use them at any time. The most common way to interact with functions is through required arguments, as shown below. Adding in flags (like -v) for optional arguments takes some more work (look up getopts).

• $#: number of parameters passed
• $@: array of all parameters
• $N: the Nth parameter passed to the function, with $0 being the function name

Some examples:

function speak () {
  echo here are all $# arguments: $@
  echo here is argument 1: $1
  echo here is argument 3: $3
}

function killme () {
  ps -e | grep $1 | awk '{print $1;}' | xargs kill
  echo $1 is dead
}

function cls () {
  cd $1
  ls
}

There are a set of system files in your home directory that are executed every time a terminal is opened - use those to define aliases, functions, or variables you want always available.