HPC Linux Tutorial: Difference between revisions

From HPC Docs
Jump to navigation Jump to search
No edit summary
No edit summary
 
(25 intermediate revisions by the same user not shown)
Line 1: Line 1:
==The Linux Command Line==
==Emergency Linux Procedures==
The Linux command line provides a way to type commands in a text terminal to perform actions on the system you are connected to, either locally or remotely. Linux supports multiple "shells" which are command interpreters. While each shell has its own special features, they tend to have a lot of overlap so it is fairly easy to become acclimated with a new shell.  The ELSA HPC cluster uses the '''bash''' shell [[Wikipedia:Bash_(Unix_shell)|(<u>B</u>ourne <u>A</u>gain <u>Sh</u>ell)]] with is currently the very command and popular shell. Other shells include the original [[Wikipedia:Bourne_shell|Bourne Shell]], [[Wikipedia:Tcsh|Tcsh]], [[Wikipedia:KornShell|KornShell]] and [[Wikipedia:Z_shell|Z Shell]].
''HELP!'' I'm stuck and can't get out!


===Command Structure===
===How do I logout?===
You can logout of Linux using one, or a combination, of these methods.


The basic format of the is <code>command options arguments</code> where <code>options</code> and <code>arguments</code> may either be optional or required depending on the command being run. Command <code>options</code> begin with a <code>-</code> or <code>--</code> to indicate they are options, not arguments.
* Type <code>exit</code> at the terminal prompt.
* Type <code>logout</code> at the terminal prompt.  This sometimes doesn't work, and you must use <code>exit</code> instead because of ... reasons.
* Hold down the <code>CTRL</code> key and press <code>d</code>.  CTRL-d in the universal Unix/Linux end-of-input key sequence.


Let's look at some real-life examples:
===Oh no, VI===
Somehow you managed to stumble into the '''vi''' text editor. VI comes with with every Linux distribution and is a good editor learn, but right now you just want to get out of it.  Follow these steps.
* Press the <code>ESC</code> key.  It doesn't hurt to press it 2 or 3 times.
* Type <code>:q!</code>
* Press the <code>RETURN</code> or <code>ENTER</code> key (depending on what it is called on your keyboard)
You should now be back at a terminal prompt.  You're welcome.


* <p><code>ls -l</code>
===But the Internet told me to do it===
: This command <code>ls</code> lists files while the optional <code>-l</code> (that's a small "L") argument tells the command to do a long listing. A long listing includes the file permissions, owners, group and other info.</p>
So you've been using the Internet to figure out how to get some piece of software working that you need for your research.  It probably told you to run <code>sudo apt-get ''blahblahblah''</code> so you did that.  Now the system is telling you that "This incident will be reported." Don't panic. Everything is okay.
<br>


* <p><code>ls myfile</code>
'''Sudo''' stands for "super user do" and is a command to elevate your privileges to the mighty '''root''' user who has full access to the entire systemThe '''root''' user is needed to install software and make system-wide changes.  Unfortunately (or fortunately for the HPC administrator) you don't have access to do this, hence the incident message.
: This <code>ls</code> doesn't include any options but does include an argument <code>myfile</code>This will list the file <code>myfile</code> to the terminal if it exists in the current directory. If not, you'll get a message such as <code>ls: cannot access myfile: No such file or directory</code> instead.</p>
<br>


* <p><code>ls -l myfile</code>
On a side note, even without using '''sudo''', you won't be able to use the command '''apt-get''' since it is a tool for installing packages on Debian Linux variants like Ubuntu.  The ELSA cluster uses Rocky Linux which does not use '''apt-get''' but instead uses '''dnf''' which is another package manager.  But don't worry, you don't have permissions to run either of these with or without '''sudo'''.  If you need software installed system-wide, contact the HPC administrator for some helpful, friendly assistance.
: In this next example, we combine command, options and arguments. This will list the file to the terminal with the file permissions, owners, etc. if it exists in the current directory.</p>
<br>


* <p><code>ls -lh myfile yourfile</code>
==Software Carpentry's Unix Shell Lesson==
: In this final example, multiple options and arguments are specified. Not all commands have multiple options or allow multiple arguments, but the <code>ls</code> command does. The <code>-lh</code> combined options tell the <code>ls</code> command to not only show all the additional information about the files but convert any the output (e.g. the file size) to "human readable" format. If multiple arguments are permitted, the command applies to all that are specified.  This command could also have been specified with the options listed separately as <code>ls -l -h myfile yourfile</code>.</p>
[https://software-carpentry.org/ Software Carpentry] focuses on teaching researchers the computing skills they need to get more done in less time and with less pain. Their lesson on the Unix Shell is listed below. Note that while they refer to it as Unix, the lesson also applies to its younger cousin Linux.
<br>


===On-line Manual Pages===
* [http://swcarpentry.github.io/shell-novice/ The Unix Shell] Lesson
Linux consists of thousands of commands. While the syntax is common among most of them as <code>command options arguments</code>, it may be daunting to remember the various options for each command. Options don't necessary represent the same thing between commands. For example, the <code>-i</code> option means different things in the <code>ls</code> command and the <code>grep</code> command. To eliminate the need to keep a thick Linux command "bible" next to you at all times, the system provides "manual pages" on-line that can be called up as needed. These are called '''man pages''' in Linux lingo. You use the <code>man</code> command to bring the manual pages for a command you provide as an argument, e.g. <code>man ls</code>.
* [http://swcarpentry.github.io/shell-novice/reference The Unix Shell Summary of Basic Commands] Reference


===Orthogonality===
==Ryan's Tutorials==
[https://ryanstutorials.net/ Ryan's Tutorials] are also excellent. Two relevant tutorials are linked below.


==Manipulating Files==
* [https://ryanstutorials.net/linuxtutorial/ Linux Tutorial]
* [https://ryanstutorials.net/bash-scripting-tutorial/ Bash Scripting Tutorial]


==Additional Resources==
==On-line Manual Pages==
Below are additional resources for learning the Linux command-line. The Lynda.com resource below may require you to provide your TCNJ login information to access.
Linux consists of thousands of commands. While the <code>command options arguments</code> syntax is common among most of them, it can be daunting to remember the various options for each command. Options don't necessary have the same meaning between different commands. For example, the <code>-i</code> option has a different meaning when used with <code>ls</code> than with <code>grep</code>. To eliminate the need to keep a thick Linux command "bible" next to you at all times (or more likely a browser window open to your favorite search engine), the system provides "manual pages" built into Linux that can be called up as needed. These are called '''man pages''' in Linux lingo. You use the <code>man</code> command to display the manual pages for a command, e.g. <code>man cat</code> will display the manual page for the '''cat''' command.


* Lynda.com course titled Learning [https://www.lynda.com/Linux-tutorials/Learn-Linux-Command-Line-Basics/435539-2.html?org=tcnj.edu Linux Command Line]. In the "What do you already know?" section, only watch the "A very quick intro to Linux" lesson before skipping down to section 2 "What is the command line?"  Watch all the videos in sections 2, 3 and 4.
Let's review an example '''man page'''. The command <code>man</code> followed by most Linux commands will display a helpful screen with its syntax, description of what the it does, all of the available options, and a few examples.  For more help using '''man''' try running <code>man man</code>.
 
<code>$ man cat</code>
<pre>
CAT(1)                                                            User Commands                                                          CAT(1)
 
 
 
NAME
      cat - concatenate files and print on the standard output
 
SYNOPSIS
      cat [OPTION]... [FILE]...
 
DESCRIPTION
      Concatenate FILE(s), or standard input, to standard output.
 
      -A, --show-all
              equivalent to -vET
 
      -b, --number-nonblank
              number nonempty output lines, overrides -n
 
      -e    equivalent to -vE
 
      -E, --show-ends
              display $ at end of each line
 
      -n, --number
              number all output lines
 
      -s, --squeeze-blank
              suppress repeated empty output lines
 
      -t    equivalent to -vT
 
      -T, --show-tabs
              display TAB characters as ^I
 
      -u    (ignored)
 
      -v, --show-nonprinting
              use ^ and M- notation, except for LFD and TAB
 
      --help display this help and exit
 
      --version
              output version information and exit
 
      With no FILE, or when FILE is -, read standard input.
 
EXAMPLES
      cat f - g
              Output f's contents, then standard input, then g's contents.
 
      cat    Copy standard input to standard output.
 
      GNU coreutils online help: <http://www.gnu.org/software/coreutils/> Report cat translation bugs to <http://translationproject.org/team/>
 
AUTHOR
      Written by Torbjorn Granlund and Richard M. Stallman.
 
COPYRIGHT
      Copyright © 2013 Free Software Foundation, Inc.  License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>.
      This is free software: you are free to change and redistribute it.  There is NO WARRANTY, to the extent permitted by law.
 
SEE ALSO
      tac(1)
 
      The  full documentation for cat is maintained as a Texinfo manual.  If the info and cat programs are properly installed at your site, the
      command
 
              info coreutils 'cat invocation'
 
      should give you access to the complete manual.
 
 
 
GNU coreutils 8.22                                                November 2015                                                          CAT(1)
</pre>
 
To search all man pages for a certain keyword, try using the '''-k''' option like the example below.
 
<code>$ man -k concatenate</code>
<pre>
g3cat (1)            - concatenate multiple g3 documents
strcat (3)          - concatenate two strings
strncat (3)          - concatenate two strings
wcscat (3)          - concatenate two wide-character strings
wcsncat (3)          - concatenate two wide-character strings
cat (1)              - concatenate files and print on the standard output
cat (1p)            - concatenate and print files
nc (1)              - Concatenate and redirect sockets
ncat (1)            - Concatenate and redirect sockets
pmlogextract (1)    - reduce, extract, concatenate and merge Performance Co-Pilot archives
strcat (3p)          - concatenate two strings
strncat (3p)        - concatenate a string with part of another
tac (1)              - concatenate and print files in reverse
Tcl_Concat (3)      - concatenate a collection of strings
wcscat (3p)          - concatenate two wide-character strings
wcsncat (3p)        - concatenate a wide-character string with part of another
zcat (1p)            - expand and concatenate data
</pre>

Latest revision as of 12:54, 29 May 2024

Emergency Linux Procedures

HELP! I'm stuck and can't get out!

How do I logout?

You can logout of Linux using one, or a combination, of these methods.

  • Type exit at the terminal prompt.
  • Type logout at the terminal prompt. This sometimes doesn't work, and you must use exit instead because of ... reasons.
  • Hold down the CTRL key and press d. CTRL-d in the universal Unix/Linux end-of-input key sequence.

Oh no, VI

Somehow you managed to stumble into the vi text editor. VI comes with with every Linux distribution and is a good editor learn, but right now you just want to get out of it. Follow these steps.

  • Press the ESC key. It doesn't hurt to press it 2 or 3 times.
  • Type :q!
  • Press the RETURN or ENTER key (depending on what it is called on your keyboard)

You should now be back at a terminal prompt. You're welcome.

But the Internet told me to do it

So you've been using the Internet to figure out how to get some piece of software working that you need for your research. It probably told you to run sudo apt-get blahblahblah so you did that. Now the system is telling you that "This incident will be reported." Don't panic. Everything is okay.

Sudo stands for "super user do" and is a command to elevate your privileges to the mighty root user who has full access to the entire system. The root user is needed to install software and make system-wide changes. Unfortunately (or fortunately for the HPC administrator) you don't have access to do this, hence the incident message.

On a side note, even without using sudo, you won't be able to use the command apt-get since it is a tool for installing packages on Debian Linux variants like Ubuntu. The ELSA cluster uses Rocky Linux which does not use apt-get but instead uses dnf which is another package manager. But don't worry, you don't have permissions to run either of these with or without sudo. If you need software installed system-wide, contact the HPC administrator for some helpful, friendly assistance.

Software Carpentry's Unix Shell Lesson

Software Carpentry focuses on teaching researchers the computing skills they need to get more done in less time and with less pain. Their lesson on the Unix Shell is listed below. Note that while they refer to it as Unix, the lesson also applies to its younger cousin Linux.

Ryan's Tutorials

Ryan's Tutorials are also excellent. Two relevant tutorials are linked below.

On-line Manual Pages

Linux consists of thousands of commands. While the command options arguments syntax is common among most of them, it can be daunting to remember the various options for each command. Options don't necessary have the same meaning between different commands. For example, the -i option has a different meaning when used with ls than with grep. To eliminate the need to keep a thick Linux command "bible" next to you at all times (or more likely a browser window open to your favorite search engine), the system provides "manual pages" built into Linux that can be called up as needed. These are called man pages in Linux lingo. You use the man command to display the manual pages for a command, e.g. man cat will display the manual page for the cat command.

Let's review an example man page. The command man followed by most Linux commands will display a helpful screen with its syntax, description of what the it does, all of the available options, and a few examples. For more help using man try running man man.

$ man cat

CAT(1)                                                            User Commands                                                           CAT(1)



NAME
       cat - concatenate files and print on the standard output

SYNOPSIS
       cat [OPTION]... [FILE]...

DESCRIPTION
       Concatenate FILE(s), or standard input, to standard output.

       -A, --show-all
              equivalent to -vET

       -b, --number-nonblank
              number nonempty output lines, overrides -n

       -e     equivalent to -vE

       -E, --show-ends
              display $ at end of each line

       -n, --number
              number all output lines

       -s, --squeeze-blank
              suppress repeated empty output lines

       -t     equivalent to -vT

       -T, --show-tabs
              display TAB characters as ^I

       -u     (ignored)

       -v, --show-nonprinting
              use ^ and M- notation, except for LFD and TAB

       --help display this help and exit

       --version
              output version information and exit

       With no FILE, or when FILE is -, read standard input.

EXAMPLES
       cat f - g
              Output f's contents, then standard input, then g's contents.

       cat    Copy standard input to standard output.

       GNU coreutils online help: <http://www.gnu.org/software/coreutils/> Report cat translation bugs to <http://translationproject.org/team/>

AUTHOR
       Written by Torbjorn Granlund and Richard M. Stallman.

COPYRIGHT
       Copyright © 2013 Free Software Foundation, Inc.  License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>.
       This is free software: you are free to change and redistribute it.  There is NO WARRANTY, to the extent permitted by law.

SEE ALSO
       tac(1)

       The  full documentation for cat is maintained as a Texinfo manual.  If the info and cat programs are properly installed at your site, the
       command

              info coreutils 'cat invocation'

       should give you access to the complete manual.



GNU coreutils 8.22                                                November 2015                                                           CAT(1)

To search all man pages for a certain keyword, try using the -k option like the example below.

$ man -k concatenate

g3cat (1)            - concatenate multiple g3 documents
strcat (3)           - concatenate two strings
strncat (3)          - concatenate two strings
wcscat (3)           - concatenate two wide-character strings
wcsncat (3)          - concatenate two wide-character strings
cat (1)              - concatenate files and print on the standard output
cat (1p)             - concatenate and print files
nc (1)               - Concatenate and redirect sockets
ncat (1)             - Concatenate and redirect sockets
pmlogextract (1)     - reduce, extract, concatenate and merge Performance Co-Pilot archives
strcat (3p)          - concatenate two strings
strncat (3p)         - concatenate a string with part of another
tac (1)              - concatenate and print files in reverse
Tcl_Concat (3)       - concatenate a collection of strings
wcscat (3p)          - concatenate two wide-character strings
wcsncat (3p)         - concatenate a wide-character string with part of another
zcat (1p)            - expand and concatenate data