OS | Haobin Tan

Introduction to Linux

Thu, 15 Aug 2024 00:00:00 +0000

Linux and Unix

Operating System (OS) is software that

manages computer hardware and resources
allows interaction with hardware to perform useful tasks

Unix

A family of operating systems
Popular Unix-based OS:
- HP-UX
- IBM AIX
- Apple macOS

Linux

Family of Unix-like OSs (usually specific distribution)
Originally developed as an effort to create a free, open-source Unix OS
Key features
- Free and open source
- Secure
- Multi-user
- Multitasking
- Portability
Use cases
- Android
- Supercomputers
- Data centers and cloud services
- PCs

Linux Distributions

Linux distribution

A specific flavor of Linux OS
Also referred to as Distro
Core component: Linux kernel

Linux distro differences

System utilities
GUI
Shell commands
Support types
- Community vs. enterprise
- Long-tem support (LTS) vs. rolling release

Common/popular distro

Debian
- One of the earliest-rooted distros: first release in 1993 (named version 0.01) and the first official, stable release in 1996 (version 1.1)
- Features
  - Stable, reliable, and fully open source
  - Supports many computer architectures (or types of hardware)
  -> Highly regarded in the server space
- Largest community-run distro
Ubuntu
- Debian-based, used a lot of the same tools as the Debian OS
- Three official editions
  - Desktop: for personal computers, laptops, and workstations
  - Server:for simple file servers or multinode clouds
  - Core: for the Internet of Things (IoT)
Red Hat Linux
- A “core” Linux distro (meaning that it is not derived from another Linux distro)
- Stable, reliable, and fully open source
- Ships as Red Hat Enterprise Linux (or RHEL), an edition focused entirely on enterprise customers
Fedora
- Supports many architectures
- Very reliable and secure
- Actively developed, with a large and growing community
SUSE Enterprise (a.k.a SLE)
- Available in two editions
  - Serve (SLES)
  - Desktop (SLED)
- Supports many architecture (e.g., ARM for Raspberry Pi)
- Uses the SUSE Package Hub -> Enable users to install packages that aren’t officially part of SLE
Arch Linux
- Do-it-yourself (DIY) approach
- Highly configurable -> Requires strong understanding of Linux and system tools
- Not focus on stability -> easy access to the newest software

Linux Architecture

Layer	Function	Tasks / Jobs
UI	Allows users to interact with the system using a keyboard or mouse	Use a web browser to send an Email
Application	Includes system daemons, programming languages, shells, user apps, tools
Operating system	Responsible for jobs that are vital for system stability such as job scheduling and keeping track of time	Assign software to users Help detect errors and prevent failures Perform file management tasks
Kernel	Core component of the OS, responsible for managing memory, processing, and securtiy Lowest-level software, starts on boot Bridge between apps and hardware	Memory management Process management Device drivers Security
Hardward	Includes all the physical or electronic devices in the computer such as processors, memory modules, input devices, and storage

Linux filesystem

Collections of files in your machine
Begins at root directory (\)
Tree-like structure
Assigns appropriate access rights

/bin
- Contains user binary files, which contain the code your machine reads to run programs and execute commands.
- It’s called “slash bin” to signify that it exists directly below the root directory.
/usr: contains user programs
/home: Your personal working directory where you should store all your personal files
/boot: Contains your system boot files, the instructions vital for system startup
/media: Contains files related to temporary media such as CD or USB drives that are connected to the system

Linux Terminal

Shell

An OS-level application that interpreets commands.
You can use shell command to (e.g.)
- Move and copy files
- Write to and read from files
- Extract and filter data
- Search for data
Popular shells
- Bash
- Zsh

Terminal

An application to interact with the shell
Enter commands and receive output from them

Linux system communication

User enters the command in a terminal, which is then relayed to the shell.
The core components of the operating system and kernel translate the command for the hardware to perform.
When the hardware completes the command, the kernel reads any changes or results and sends them back via the shell to the terminal for the user’s information.

Linux filesystem paths

The filesystem is the human-readable directory or file location.

a/b indicates that the file or directory named “b” is located inside the directory named “a”.
Special paths
- /: Root directory
- ~: Home directory, an important subdirectory of the root directory
- ..: Parent of current directory
- .: Current directory
Filesystem-related commands
- cd : change directory
- pwd: print path name for present working directory
- ls: list the contents of the directory

Creating and Editing Text Files

Popular text editors

Command-line text editors
- GNU nano: use nano <filename> to open a text file in GNU nano from the command prompt
- vi
- vim
  - Insert mode
    - Press i to enter Insert mode
    - Type some text
    - Press Esc to exist Insert mode and switch to Command mode. Now the text is written to the buffer at the cursor location.
  - Command mode
    - Enter :sav <filename> to create a file and write the buffer to the file
    - Enter :w to write the buffer to the file without exiting
    - Enter :q to quit vim session
    - Enter :q! to quit without saving
GUI-based text editors
- gedit
Command-line or GUI: emacs

Install Software and Updates

Packages: Archive files for installing new software or updating existing software

Deb packages (.deb files): For Debian-based distributions (e.g., Debian, Ubuntu, and Mint)
RPM packages (.rpm files): For Red Hat-based distributions (e.g., CentOS/RHEL, Fedora, and openSUSE)

RPM: Red Hat Package Manager

.deb and .rpm formats are equivalent. You can use alien to convert it

RPM to deb
```
alien <package-name>.rpm
```
deb to RPM
```
alien -r <package-name>.deb
```

Package managers

Manage the download and installation of packages
Available for different Linux distros
Can be GUI-based or command-line tools
Benefits
- Automatically resolve dependencies
- Notify you when updates are available
- Automatic or manual installation

For deb-based

GUI-based: Update Manager
Command line-based: apt
- sudo apt update: Find available packages for your distro
  - The output of this command lists each available package, builds a dependency tree, and lets you know how many packages can be upgraded
- sudo apt upgrade: Upgrad all installed packages on a system
- sudo apt install <package_name>: Install a specific package

For RPM-based

GUI-based: PackageKit
Command line-based: yum
- sudo yum update: Update all packages in system
- sudo yum install<package_name>: Install a specific package

Information, Navigation, and Management Commands

Thu, 15 Aug 2024 00:00:00 +0000

Overview

What is a shell?

User interface for running commands
Interactive language
Scripting language, can be used to automate tasks
Default shell: Bash. Other shells: sh, ksh, tcsh, zsh, and fish. Check shell with
```
printenv SHELL
```
This returns the path to the default shell program

Shell command applications

Getting information

whoam: User name
id: User ID and group ID
uname: OS name
ps: Running processes
top: Resource usage
df: Mounted file systems
man: Reference manual
date: Today’s date

Navigating and working with files and directories

Files

Files
- cp: Copy fiile
- mv: Change file name or path
- rm: Remove file
- touch: Create empty file, update file timestamp
- chmod: Change/modify file permissions
- ws: Get count of lines, words, characters in file
- grep: Return lines in file matching pattern
Directory
- ls: List files and directories
- find: Fined files in directory tree
- pwd: Get present working directory
- mkdir: Make directory
- cd: Change directory
- rmdir: Remove directory

Printing file and string contents

cat: Print file contents
more: Print file contents page-by-page
head: Print first N lines of file
tail: print last N lines of file
echo: Print string or variable value

File compression and archiving

tar: Archive a set of files
zip: Compress a set of files
unzip: Extract files from a compressed zip archive

Performing network operations

hostname: Print hostname
ping: Send packts to URL and print response
ifconfig: Display or configure system network interfaces
curl: Display contents of file at a URL
wget: Download file from URL

Monitoring performance and status of the system, its components and applications

Running batch jobs, such as ETL operations.

Informational Command

Display user information
Verify identity or identify user account

Command	Usage
`whoami`	Return user name
`id`	User or Group ID `id -u` returns the numerical ID of the user `id -u -n` returns the name corresponding to the numerical user ID
`uname` (Unix name)	Returns OS information Identify system or diagnose issues
`df` (disk free)	Show disk usage to monitor disk usage or check space
`ps` (process status)	Monitor or manage processes
`top` (table of processes)	Task manger: Monitor system performance and reosurce usage
`echo`	Print string or variable value
`date`	Display system date and time
`man <command>` (manual)	Show manual for any command

is a free and open-source collaborative documentation effort, which provides documentation that is more accessible than the traditional man pages, along with practical examples.

Example

ls

pwd, cd

ls

find: Find files in directory tress.

Example:
- iname: case insensitive

File and Directory Management Commands

mkdir

rm

If you want to remove multiple files, for example by using a wildcard to find all filenames matching a pattern, it’s best practice to confirm or deny each deletion by including the -i option, which creates a prompt to ask for confirmation before every deletion.

Be careful when deleting files or directories! There is normally no way to restore a deleted file once it is deleted, as there is no trash folder. This is why you should always back up, or archive, your important files. You will learn more about archiving files soon.

rmdir: Remove empty directory

touch: Create empty file, update file date

cp: Copy file or directory (-r) to destination

mv: Move file or directory

when the source and target directories are the same, you can use mv to rename a file. Example: use mv to rename users.txt to user-info.txt by entering the following command:
```
mv users.txt user-info.txt
```

You should always use caution when moving a file. If the target file already exists, it will be overwritten, or replaced, by the source file.

chmod (change mode): Change file permissions

Specify which permissions to change with a combination of the following characters:

Option Description

r, w, x Permissions: read, write, and execute

u,g, o User categories: user, group, and all others

+, - Operations: grant and revoke
Example

Option	Description
`r`, `w`, `x`	Permissions: read, write, and execute
`u`,`g`, `o`	User categories: user, group, and all others
`+`, `-`	Operations: grant and revoke

Managing File Permissions and Ownership

Three possible levels of file ownership in Linux: user, group, and other.

Whoever creates a file, namely the user at the time of creation, becomes the owner of that file by default.
A group of users can also share ownership of a file.
The other category essentially refers anyone in the universe with access to your Linux machine - careful when assigning ownership permission to this level!

Only an official owner of a file is allowed to change its permissions. This means that only owners can decide who can read the file, write to it, or execute it.

Viewing file permissions

Example:

$ echo "Who can read this file?" > my_new_file
$ more my_new_file
Who can read this file?
$ ls -l my_new_file
-rw-r--r-- 1 theia users 25 Dec 22 17:47 x

Here we’ve echoed the string "Who can read this file?" into a new file called my_new_file. The next line uses the more command to print the contents of the new file.

Finally, the ls command with the -l option displays the file’s (default) permissions: rw-r--r--.

The first three characters (rw-) define the user permissions. You, being the user, have the permission rw-, which means you have read and write permissions by default, but do not have execution permissions. Otherwise there would be an x in place of the last -.
The next three (r--) the group pemissions. The final three (r--) the other permissions.

Looking at the entire line, rw-r--r--, you can see that anyone can read the file, nobody can execute it, and you are the only user that can write to it.

The - at the very beginning of the line in the terminal means that the permissions are referring to a file. If you were getting the permissions to a directory, you would see a d in the front for “directory”.

Directory permissions

The permissions for directories are similar but distinct for files. Though directories use the same rwx format, the symbols have slightly different meanings.

The following table illustrates the meanings of each permission for directories:

Directory Permission	Permissible action(s)
`r`	List directory contents using `ls` command
`w`	Add or remove files or directories
`x`	Enter directory using `cd` command

Making a file private

You can revoke read permissions from your group and all other users by using the chmod command.

Example:

$ chmod go-r my_new_file
$ ls -l my_new_file
-rw------- 1 theia users 24 Dec 22 18:49 my_new_file

In the chmod command, go-r is the permission change to be applied, which in this case means removing for the group (g) and others (o) the read (r) permission.

Executable file

A Linux file is executable if it contains instructions that can be directly interpreted by the operating system. Basically, an exectuable file is a ready-to-run program. They’re also referred to as binaries or executables.

Script is a particular kind of executable. shell scripting, or more specifically Bash scripting, which is writing scripts in Bash (born-again shell), a very popular shell scripting language. A shell script is a plain text file that can be interpreted by a shell.

Formally speaking, for a text file to be considered an executable shell script for a given user, it needs to have two things:

Execute permissions set for that user
A directive, called a “shebang”, in its first line to declare itself to the operating system as a binary

Text Files, Networking, and Archiving Commands

Sat, 17 Aug 2024 00:00:00 +0000

Text File Commands

Viewing file content

cat (catenate): Print entire file contents

Limitation: If the file content is long, the output will take up the entire terminal window

more: Print file content page-by-page

head: Print first 10 lines of file

head -n <N>: Print first N lines of file

tail: Print last 10 lines of file

tail -n <N>: Print last N lines of file

wc (word count): Count lines, words, characters

-l/-w/-c: View only line/word/character count

Wrangling text files

sort: Sort the lines of a file alpha-numerically and prints the sorted result to standard output.

-r: Reverse order

uniq (unique): Filter out the repeated lines

Note: uniq only removes duplicated lines if they are consecutive!

grep (global regular expression print): Return lines of a file matching a specified pattern

Example

Some frequently used options for grep include:

Option	Description
`-n`	Along with the matching lines, also print the line numbers
`-c`	Get the count of matching lines
`-i`	Ignore the case of the text while matching
`-v`	Print all lines which do not contain the pattern
`-w`	Match only if the pattern matches whole words

cut: Extract a section from each file

Example
- Extract the second to ninth characters from every line
- Extract the last names
  - -d ' ': Specify space as the field delimiter
  - -f2: Return the second filed from each line

paste: Merge lines from different files

tr: Trim repeated characters to a single character.

$ echo "There are too many spaces in this sentence." | tr -s " "
There are too many spaces in this sentence.

xargs: Trim leading and trailing spaces from a string.

$ echo " Never start or end a sentence with a space. " | xargs
Never start or end a sentence with a space.

rev : Reverse the order of characters on a line of text.

$ echo ".sdrawkcab saw ecnetnes sihT" | rev
This sentence was backwards.

Networking Commands

Brief introduction to networking

Computer networks

A computer network is a set of computers that are able to communicate with each other and share resources provided by network nodes. Examples of computer networks include Local Area Networks (LANs), Wide Area Networks (WANs), and the entire Internet. The Internet, or World Wide Web, is essentially a giant network of computer networks.

A network resource is any object, such as a file or document, which can be identified by the network.

An object is identifiable if it can be assigned a unique name and address that the network can use to identify and access it.

A network node is any device that participates in a network.

A network can include any device which is not necessarily a computer but is part of the network’s infrastructure. Examples of network nodes include modems, network switches, hubs, and wifi hotspots.

Host, clients, and servers

A host is a special type of node in a computer network - it is a computer that can function as a server or a client on a network.

A server is a host computer that is able to accept a connection from a client host and fulfill certain resource requests made by the client.

Many hosts can perform either role, acting as both client and server.

Packets and pings

A network packet is a formatted chunk of data that can be transmitted over a network.

Today’s computer networks typically use communication protocols that are based on such packets of information. Every packet consists of two types of data:
the control information
the payload.

The control information is data about how and where to deliver the payload, such as the source and destination network addresses, while the payload is the message being sent.

The ping command works by sending special ’echo request’ packets to a host and waiting for a response from the host.

ping is a utility available on most operating systems that have networking capability. Linux has its own implementation of the ping command that’s used to test and troubleshoot connectivity to other network hosts.

URLs and IP adresses

IP stands for “Internet Protocol” which defines the format of data transmitted over the internet or a local network.

An IP address is a code used to uniquely identify any host on a network.

An IP address can be used to establish a connection to a host and exchange packets with it, for example using the ping command. In addition to their payload, IP packets - a type of network packet that conforms to the Internet Protocol - contain the IP addresses of the source and destination hosts.

A URL, more commonly known as a web address, stands for Uniform Resource Locator.

A URL uniquely identifies a web resource and enables access to that resource. Typically the resource that a URL points to is a web page, but it can also be used for tasks such as transferring files, sending emails, and accessing databases.

For example, the URL to the Wikipedia page for URL is https://en.wikipedia.org/wiki/URL. Just like for a typical URL, its format indicates a protocol (https), a hostname (en.wikipedia.org), and a file name (/wiki/URL).

Networking commands

hostname: Print host name

-s: Drop the domain suffix
-i: Provide the IP address of the hostname
Example

ifconfig (Interface configuration): Display or configure the system network interfaces

eth0: SHow ethernet config information

ping: Send ICMP packets to URL and print response

c: Return a set number of ping results

curl (Client URL): Transer data to and from URL(s)

wget (Web get): Download file(s) from a URL

More focused than curl, supports recursive file downloads

Archiving and Compression Commands

Archiving and compression are distinct processes, which are usually combined

Archiving: Process of storing information that you don’t use regularly but want to preserve
- An “archive file” is a collection of data files and directories that are stored as a single file.
- Archiving makes the collection more portable and serves as a backup in case of loss or corruption.
Compressing: Reducing the size of a file by taking advantage of redundancy in its information content.
- Advantages
  - Preserve storage space
  - Speed up data transfer
  - Reduce bandwidth load

tar (tape archiver): Archive and extract files, allows you to pack multiple files and directories into a single archive file.

-c: Create a new archive
-f: Tell tar to interpret its input from the file rather than from the default, which is standard input
-z: Compress the archive file with gzip
-v: Verbosely list files processed
tar -tf: List archive contents
tar -xf: Extract (“untag”) file and folders
tar -xzf: Decompress and extract

zip: Compress files and directories to an archive

zip compresses files prior to bundling them, Whereas tar, with the -z option, achieves compression by applying gzip on the entire tarball, but only after bundling it.

unzip: Extract and decompress zipped archive

Shell Scripting

Thu, 15 Aug 2024 00:00:00 +0000

Shell Scripting Basics

A script is a list of commands that can be interpreted and run by a program called scripting  language, which is usually not compiled but interpreted at runtime instead.

Generally slower to run than compiled languages, but they are also much easier and faster to develop
Widely used to automate processes, such as
- ETL jobs
- File backups and archiving
- System administration tasks

Shell script is an executable text file with an interpreter directive, a.k.a. “shebang” (#!) directive, which has the form “pound, bang, interpreter” plus an optional argument

#!interpreter [optional-arg]

interpreter: An absolute path to an executable program
optional-arg: A string representing a single argument

Example:

#!/bin/sh invokes the Bourne shell or other compatible shell program, from the bin directory.
#!/bin/bash invokes the BAsh shell
#!/usr/bin/env python3

Shell Variables

Shell variables offer a powerful way to store and later access or modify information such as numbers, character strings, and other data structures by name.

Example

$ firstname=Jeff
$ echo $firstname
Jeff

The first line assigns the value Jeff to a new variable called firstname. The next line accesses and displays the value of the variable, using the echo command along with the special character $ in front of the variable name to extract its value, which is the string Jeff. Thus, we have created a new shell variable called firstname for which the value is Jeff.

Another way to create a shell variable is to use the read command, which reads user input into a shell variable.

Example:

$ read lastname
Grossman
$ echo $lastname
Grossman

The read command is particularly useful in shell scripting. You can use it within a shell script to prompt users to input information, which is then stored in a shell variable and available for use by the shell script while it is running.

Filters, Pipes, and Variables

Filters

Filters are shell commands, which

Take input from standard input
Send output to standard output
Transform input data into output data
Examples: wc. cat. more, head, sort, grep
Filters can be chain together!

Pipe (Pipeline)

Pipe command (|) : for chaining filter commands

Use pattern

[command 1] | [command 2] | [command 3] ... | [command n]

E.g.,

command 1 | command 2

Output of command 1 is input of command 2

Example

Variables

Scope limited to shell
set: List all variables and their definitions that are visible to the current shell

Define shell variables

var_name=value # Note: NO space around "="!

Clear/Delete a variable\

unset var_name

Example

Environment variables: Just like shell variables, except they have extended scope. They persist in any child processes spawned by the shell in which they originate.
- Extend shell variable to environment variable:
```
export var_name
```
  Example
```
export GREETINGS
```
  makes the GREETINGS variable defined in the example above an environment variable
- env: List all environment variables

Useful Features of the Bash Shell

Metacharacters

Metacharacters are special characters that have meaning to the shell.

#: Precedes a comment that the shell ignores
;: Seprate commands typed on the same line, e.g.,
```
$ echo "Hello" ; whoami
```
*: Filename expansion wildcard, represents any number of consecutive characters within a filename pattern
?: Single character wildcard in filename expansion (a single-character version of * ), e.g.,
\: Escape unique character interpretation
" ": Interpret literally, interpret metacharacters within string
' ': Interpret literally, escape all metacharacters within string

Example:

I/O redirection

Input/Output, or I/O redirection, refers to a set of features used for redirecting either the standard input, the keyboard, or the standard output, the terminal.

>: Redirect output to the file. It also creates the file if it doesn’t exist and overwrites its contents if it already exists.
>>: Append output to the file
2>: Ridirect standard error to file
2>>: Append standard error to file
<: Redirect file contents to standard input

Example

Command substitution

Replace the command with its output: $(command)$ or command. Example:

Command line arguments

Program arguments specified on the command line
A way to pass arguments to a shell script

Example

$ ./MyBashScript.sh arg1 arg2

Batch vs. concurrent modes

Batch mode: Commands run sequentially

command 1; command 2

Command 2 only runs after command 1 is completed.

Concurrent mode: Commands run in parallel

command 1 & command 2

The ampersand operator, after command 1, directs command 1 to operate in the background and passes control to command 2 in the foreground.

Advanced Bash Scripting

Conditionals

Bash script conditionals use the following if-then-else syntax:

if [ condition ]
then
 statement_block_1
else
 statement_block_2
fi

You must always put spaces around your condition within the square brackets [ ].
Every if condition block must be paired with a fi to tell Bash where the condition block ends.
The else block is optional but recommended. If the condition evaluates to false without an else block, then nothing happens within the if condition block. Consider options such as echoing a comment in statement_block_2 to indicate that the condition was evaluated as false.

Example: the condition is checking whether the number of command-line arguments read by some Bash script, $#, is equal to 2.

if [[ $# == 2 ]]
then
 echo "number of arguments is equal to 2"
else
 echo "number of arguments is not equal to 2"
fi

Notice: the use of the double square brackets, which is the syntax required for making integer comparisons in the condition [[ $# == 2 ]].

$ can be used to access command-line arguments passed to the script.

Examples:

$0: The name of the script.
$1, $2, $3, …: The first, second, third, etc., arguments passed to the script.
$#: The number of arguments passed.
$@ or $*: All arguments as a list.
$?: The exit status of the last command.
$$: The process ID (PID) of the current script.

You can also make string comparisons. For example, assume you have a variable called string_var that has the value "Yes" assigned to it. Then the following statement evaluates to true:

`[ $string_var == "Yes" ]` # Only need single square brackets for string comparison

You can also include multiple conditions to be satified by using the “and” operator && or the “or” operator ||. Example

if [ condition1 ] && [ condition2 ]
then
 echo "conditions 1 and 2 are both true"
else
 echo "one or both conditions are false"
fi


if [ condition1 ] || [ condition2 ]
then
 echo "conditions 1 or 2 are true"
else
 echo "both conditions are false"
fi

if-then-elif-else Syntax:

if [ condition ]
then
 statement_block_1
elif
 statement_block_2
else
 statement_block_3
fi

Logical Operators

==: is equal to

# If a variable a has a value of 2, the following condition evaluates to true; otherwise it evalutes to false.
$a = 2

!=: is not equal to
<= or -le: is less than or equal to

More see: Advanced Bash-Scripting Guide

Arithmetic calculations

You can perform integer addition, subtraction, multiplication, and division using the notation $(()).

Example

echo $((3+2))

a=3
b=2
c=$(($a+$b))
echo $c

Bash natively handles integer arithmetic but does not handle floating-point arithmetic. As a result, it will always truncate the decimal portion of a calculation result.

Basic arithmetic operators:

Symbol	Operation
`+`	addition
`-`	subtraction
`*`	multiplication
`/`	division

Arrays

An array is a space-delimited list contained in parentheses. To create an array, declare its name and contents:

my_array=(1 2 "three" "four" 5)

If you want to add items to your array after creating it, you can add to your array by appending one element at a time:

my_array+=("six")
my_array+=(7)

This adds elements "six" and 7 to the array my_array.

By using indexing, you can access individual or multiple elements of an array:

# print the first item of the array:
echo ${my_array[0]}

# print the third item of the array:
echo ${my_array[2]}

# print all array elements:
echo ${my_array[@]}

Get the array length:

arr_len=${#arr[@]}

arr_len=${#arr[*]}

Here # gives the lenght of the array (or string).

`for` loops

You can use a for loop along with indexing to iterate over all elements of an array.

Syntax

The for loop requires a ; do component in order to cycle through the loop.
You need to terminate the for loop block with a done statement.

for item in ${arr[@]}; do
 # Handle $item
done

for i in ${!arr[@]}; do
 # Handle ${arr[$i]}
done

Another way to implement a for loop when you know how many iterations you want is as follows. For example, the following code prints the number 0 through 6.

N=6
for (( i=0; i<=$N; i++ )) ; do
 echo $i
done

Scheduling Jobs Using Cron

Job scheduling: Schedule jobs to run automatically at certain times. E.g., “Backup script to run every Sunday at 2AM”.

Cron is the general name of the tool that runs scheduled jobs consisting of shell commands or shell scripts.

Crond is the daemon or service that interprets “crontab files” every minute and submits the corresponding jobs to cron at scheduled

times.

A crontab, short for “cron table,” is a simple text file containing jobs and schedule data. Crontab is also a command that invokes a text editor to allow you to edit a crontab file.

Adding jobs in Crontab

Enter crontab -e opens the default text editor and allows you to specify a new schedule and a new command.

Syntax:

m h dom mon dow command

Five time-and-date fields m h dom mon dow stand for: minute, hour, day of month, month, and day of week. All five positions must have either a numeric entry or an asterisk (*), which is a wildcard symbol that means “any.”

The five time-and-date fields cannot contain spaces and their allowed values are as follows:

Field	Allowed values
minute	0-59
hour	0-23, 0 = midnight
day	1-31
month	1-12
weekday	0-6, 0 = Sunday

Example: Append the current date to the file ‘sundays.txt’ at 15:30 every Sunday.

30 15 * * 0 date >> sundays.txt

Then save and exit the editor.

Viewing and removing Cron jobs

Viewing: crontab -l

Removing

Use crontab -e to open the editor and remove specific line(s)
crontabl r: Remove all cron jobs

Cheatsheet

Tue, 20 Aug 2024 00:00:00 +0000

Introduction to Linux

Command	Syntax	Description	Example
List	`ls [OPTIONS] [FILE/DIRECTORY]`	List files and directories at path	`ls /home/user/documents`
Print Working Directory	`pwd`	Print present working directory	`pwd`
Change Directory	`cd [DIRECTORY]`	Change current directory	`cd /home/user/documents`
Super user do	`sudo [COMMAND]`	Run command with superuser privileges	`sudo apt update`
Text Editor	`nano [FILE]`	Open file with Nano text editor	`nano myfile.txt`

Introduction to Linux Commands

Informational, Navigational, & Management Commands

Command	Syntax	Description	Example
Who Am I	`whoami`	Return username	`whoami`
User ID	`id`	Return current user or group ID	`id`
System Information	`uname [OPTIONS]`	Display system information	`uname -a`
Manual Pages	`man [COMMAND]`	Display manual page for a command	`man ls`
Curl	`curl [OPTIONS] [URL]`	Transfer data from or to server	`curl https://some_website.com`
Date	`date [OPTIONS]`	Display current date and time	`date`
Find	`find [DIRECTORY] [OPTIONS]`	Find files and directories at specified path	`find /home/user -name '*.txt'`
Make Directory	`mkdir [DIRECTORY]`	Create new directory	`mkdir myfolder`
Remove Directory	`rmdir [DIRECTORY]`	Remove empty directory	`rmdir myfolder`
Process Status	`ps [OPTIONS]`	Display process status information	`ps -ef`
Table of Processes	`top`	Display live system resource usage	`top`
Disk Usage	`df [OPTIONS] [FILESYSTEM]`	Display disk space usage	`df -h`
Create Empty File	`touch [FILE]`	Create new file or update timestamp	`touch myfile.txt`
Copy	`cp [OPTIONS] [SOURCE] [DESTINATION]`	Copy files or directories from source to destination	`cp myfile.txt /home/user/documents`
Move	`mv [OPTIONS] [SOURCE] [DESTINATION]`	Move or rename files and directories	`mv myfile.txt /home/user/documents`
Remove	`rm [OPTIONS] [FILE/DIRECTORY]`	Remove files	`rm my_scratch_file.txt`
		Remove nonempty directory	`rm -r path_to_temp_directory`
	`rmdir [OPTIONS] [DIRECTORY]`	Remove empty directory	`rmdir path_to_my_directory`
Change Mode	`chmod [OPTIONS] [MODE] [FILE]`	Change file or directory permissions	`chmod u+x myfile.txt`

Working with Text Files, Networking & Archiving Commands

Command	Syntax	Description	Example
Concatenate	`cat [FILE]`	Display the contents of a file	`cat myfile.txt`
		Concatentate and display contents of multiple files	`cat file1 file2`
More	`more [FILE]`	Display file one screen at a time	`more myfile.txt`
Head	`head [OPTIONS] [FILE]`	Display first N lines of file	`head -5 myfile.txt`
Tail	`tail [OPTIONS] [FILE]`	Display last N lines of file	`tail -5 myfile.txt`
Echo	`echo [ARGUMENTS]`	Display arguments in console	`echo Hello, World!`
Sort	`sort [OPTIONS] [FILE]`	Alphanumerically sort file contents	`sort file.txt`
Unique	`uniq [OPTIONS] [FILE]`	Report or remove consecutively repeated lines in file	`uniq file.txt`
Word Count	`wc [OPTIONS] [FILE]`	Print the number of lines, words, and characters in a file	`wc file.txt`
Grep	`grep [OPTIONS] PATTERN [FILE]`	Search for a specified pattern in a file	`grep "hello" file.txt`
Paste	`paste [OPTIONS] [FILE1] [FILE2]`	Merge lines of files side by side	`paste file1.txt file2.txt`
Cut	`cut [OPTIONS] [FILE]`	Remove sections from each line of a file	`cut -d":" -f1 /etc/passwd`
Tar	`tar [OPTIONS] [FILE]`	Archive files together into a single file	`tar -czvf archive.tar.gz /directory`
Zip	`zip [OPTIONS] [FILE]`	Compress files into a zip archive	`zip archive.zip file1.txt file2.txt`
Unzip	`unzip [OPTIONS] [FILE]`	Uncompress files from a zip archive	`unzip archive.zip`
Hostname	`hostname`	Print the name of the current host system	`hostname`
Ping	`ping [OPTIONS] HOSTNAME/IP`	Send ICMP ECHO_REQUEST packets to a network host	`ping google.com`
Ifconfig	`ifconfig [INTERFACE]`	Display or configure network interface parameters	`ifconfig`
IP	`ip [OPTIONS]`	Show or manipulate routing, devices, policy routing, and tunnels	`ip addr`
Curl	`curl [OPTIONS] URL`	Transfer data from or to a server	`curl https://some_website.com`
Wget	`wget [OPTIONS] URL`	Download files from the web	`wget https://some_website.com/some_file.txt`

Introduction to Shell Scripting

Command	Syntax	Description	Example
Shebang	`#!/bin/[shell]`	First line of shell script	`#!/bin/bash`
Pipe	filter1 \| filter2	Chain any number of filters	`ls
Locate executable	`which [EXECUTABLE]`	Display location of `bash` executable	`which bash`
Bash	`bash [SCRIPT]`	Interpret and run script using Bash shell	`bash script.txt`
Set	`set [OPTION]`	List all shell variables	`set`
Define variable	`[VARIABLE_NAME]=[VALUE]`	Define shell variable by name and assign value	`name="John"`
Read	`read [VARIABLE]`	Read from standard input and store result in variable	`read name`
Env	`env`	Print all environment variables and their values	`env`
Export	`export [VARIABLE]`	Extend scope of local variable to all child processes	`export name`
Crontab	`crontab [OPTIONS]`	Open crontab default editor	`crontab -e`
		List all cron jobs	`crontab -l`
Schedule tasks to run at specified times using cron daemon	`m h dom mon dow command`	Append date/time to file every Sunday at 6:15 pm	`15 18 * * 0 date >> sundays.txt`
		Back up home directory every Monday at 3:00 am	`0 3 * * 1 tar -cvf my_backup_path\my_archive.tar.gz $HOME\`
		Run shell script first minute of first day of each month	`1 0 1 * * ./My_Script.sh`

export

Thu, 17 Dec 2020 00:00:00 +0000

`export` command

A built-in command of the Bash shell
Ensure environment variables and functions to be passed to child processes
Syntax
```
export [-f] [-n] [name[=value] ...] or export -p
```
- -f : Names are exported as functions
- -n : Remove names from export list
- -p : List of all names that are exported in the current shell

Usage

Display all the exported environment variables of your system

export

Display all exported variable on current shell

export -p

If you want to check specified variable:

export -p | grep <var>

Using export with functions

export -f function_name

Example

Assign a value before exporting a function or variable

export name[=value]

Set an environment variable

To create a new variable, use the export command followed by a variable name and its value.

Syntax:

export NAME=VALUE

Example:

export PROJECT_DIR=main/app
export -p | grep PROJECT_DIR

declare -x PROJECT_DIR="main/app"

Variables set directly with export are temporary variables, which means that the value defined for that variable will not take effect when you exit the current shell.

Environment varaible `PATH`

The PATH variable is an environment variable that contains an ordered list of paths that Unix/Linux will search for executables when running a command. Using these paths means that we do not have to specify an absolute path when running a command.

For example

echo $PATH

/usr/lib/lightdm/lightdm:/usr/local/sbin:/usr/local/bin

Here : is the separator. The PATH variable is itself a list of folders that are “walked” through when you run a command. In this case, the folders on PATH are:

/usr/lib/lightdm/lightdm
/usr/local/sbin
/usr/local/bin

Linux/Unix traverses these folders in order until finding an executable.

Adding a New Path

We can add a new path to the PATH variable using the export command.

To append a new path, we reassign PATH with our new path <new-path> at the end:

export PATH=$PATH:<new-path>

For example, let’s say we’re gonna add a new path usr/local/bin to thePATH variable:

export PATH=$PATH:usr/local/bin

Reference

tee

Fri, 18 Dec 2020 00:00:00 +0000

The tee command reads from the standard input and writes to both standard output and one or more files at the same time.

tee is mostly used in combination with other commands through piping.

Syntax

tee [OPTIONS] [FILE]

Copy

OPTIONS
- -a (--append) : Do not overwrite the files instead append to the given files.
- -i (--ignore-interrupts) : Ignore interrupt signals.
- Use tee --help to view all available options.
FILE One or more files. Each of which the output data is written to.

Usage

Display output and write it in a file

The most basic usage of the tee command is to display the standard output (stdout) of a program and write it in a file.

Example:

$ echo "Hello world!" | tee hello-world.txt

Hello World!

$ cat hello-world.txt

Hello World!

The output is piped to the tee command, which displays the output to the terminal and writes the same information to the file hello-world.txt.

Write to Multiple Files

The tee command can also write to multiple files. To do so, specify a list of files separated by space as arguments:

$ command | tee file1.out file2.out file3.out

Append to File

By default, the tee command will overwrite the specified file. Use the -a (--append) option to append the output to the file :

$ command | tee -a file.out

E.g.

$ echo "Hello world!" | tee hello-world.txt

Hello world!

$ cat hello-world.txt

Hello World!

$ echo "Hi world!" | tee -a hello-world.txt

Hi world!

$ cat hello-world.txt

Hello World!
Hi world!

Ignore Interrupt

To ignore interrupts use the -i (--ignore-interrupts) option. This is useful when stopping the command during execution with CTRL+C and want tee to exit gracefully.

$ command | tee -i file.out

Hide the Output

If you don’t want tee to write to the standard output, you can redirect it to /dev/null:

$ command | tee file.out >/dev/null

Reference

chmod

Tue, 05 Jan 2021 00:00:00 +0000

Check permissions

In terminal: ls –l [file_name]

For example:

$ ls -l

-rw-r--r-- 1 EckoTan staff 12 Jan 5 18:02 hello-world.txt

-rw-r--r-- : file permission
EckoTan : owner/creator of the file
staff : the group to which that owner belongs to
Jan 5 18:02 : date of creation
hello-world.txt : the file

Permissions

The permission settings, grouped in a string of characters (-, r, w, x), are classified into four sections:

File type. There are three possibilities for the type. It can either be a regular file (–), a directory (d) or a link (i).
File permission of the user (owner)
File permission of the owner’s group
File permission of other users

In permission of user, group, and others, the characters -, r, w, and x stand for None, read, write, and execute.

-: permission is NOT granted
r: Read permissions. The file can be opened, and its content viewed. But it can not be modified (nor added/removed)
w: Write permissions. The file can be edited, modified, and deleted.
x: Execute permissions. If the file is a script or a program, it can be run (executed).This option is mainly used for running scripts.

Example

-rw-r--r-- 1 EckoTan staff 12 Jan 5 18:02 hello-world.txt

hello-world.txt is a regular file with read and write permission assigned to the owner, but gives read-only access to the group and others

Change permissions

The command that executes such tasks is the chmod command.

Syntax:

$ chmod [permission] [file_name]

There are two ways to define permission:

using symbols (alphanumerical characters)
using the octal notation method

Define File Permission with Symbolic Mode

To use chmod to set permissions, we need to tell it:

Who: Who we are setting permissions for?
- u : User, meaning the owner of the file.
- g : Group, meaning members of the group the file belongs to.
- o : Others, meaning people not governed by the u and g permissions.
- a : All, meaning all of the above.
(If none of these are used, chmod behaves as if “a” had been used.)
What: What change are we making? Are we adding or removing the permission?
- – : Removes the permission.
- + : Grants the permission. The permission is added to the existing permissions.
- = : Set a permission and remove others.
Which: Which of the permissions are we setting?
- r
- w
- x

Example

$ ls -l

-rw-r--r-- 1 EckoTan staff 12 Jan 5 18:02 hello-world.txt

We want to give group members to have write permission on hello-world.txt:

$ chmod g+w hello-world.txt

$ ls -l

-rw-rw-r-- 1 EckoTan staff 12B Jan 5 18:02 hello-world.txt

We want the permission settings look like this

u: read, write, execute
g: read, write
o: read, write

$ chmod u=rwx,g=rw,o=rw hello-world.txt

$ ls -l

-rwxrw-rw- 1 EckoTan staff 12B Jan 5 18:02 hello-world.txt

Define permission using numerical shorthand

Another way to specify permission is by using the octal/numeric format. This option is faster, as it requires less typing, although it is not as straightforward as the previous method.

The following example shows how it works:

Using this method, each of the three permissions (r, w, x) is represented by one of the bits in the binary equivalent of the decimal number.

In this example,

The permission of user/owner is rwx. This could be represented as 111, which equals 7 in decimal.
The permission of group is r-x. This could be represented as 101, which equals 5 in decimal.
The permission of others is r--. This could be represented as `1010, which equals 4 in decimal.

Therefore, to achieve the same permission settings as

$ chmod u=rwx,g=rx,o=r hello-world.txt

We can also type:

$ chmod 754 hello-world.txt

-rwxr-xr-- 1 EckoTan staff 12B Jan 5 18:02 hello-world.txt

OS | Haobin Tan

Introduction to Linux

Linux and Unix

Linux Distributions

Linux Architecture

Linux filesystem

Linux Terminal

Linux system communication

Linux filesystem paths

Creating and Editing Text Files

Install Software and Updates

Information, Navigation, and Management Commands

Overview

What is a shell?

Shell command applications

Informational Command

File and Directory Navigation Commands

File and Directory Management Commands

Managing File Permissions and Ownership

Viewing file permissions

Directory permissions

Making a file private

Executable file

Text Files, Networking, and Archiving Commands

Text File Commands

Viewing file content

Wrangling text files

Networking Commands

Brief introduction to networking

Computer networks

Host, clients, and servers

Packets and pings

URLs and IP adresses

Networking commands

Archiving and Compression Commands

Shell Scripting

Shell Scripting Basics

Shell Variables

Filters, Pipes, and Variables

Filters

Pipe (Pipeline)

Variables

Useful Features of the Bash Shell

Metacharacters

I/O redirection

Command substitution

Command line arguments

Batch vs. concurrent modes

Advanced Bash Scripting

Conditionals

Logical Operators

Arithmetic calculations

Arrays

for loops

Scheduling Jobs Using Cron

Adding jobs in Crontab

Viewing and removing Cron jobs

Cheatsheet

Introduction to Linux

Introduction to Linux Commands

Informational, Navigational, & Management Commands

Working with Text Files, Networking & Archiving Commands

Introduction to Shell Scripting

export

export command

Usage

Display all the exported environment variables of your system

Display all exported variable on current shell

Using export with functions

Assign a value before exporting a function or variable

Set an environment variable

Environment varaible PATH

Adding a New Path

Reference

tee

Syntax

Usage

Display output and write it in a file

Write to Multiple Files

Append to File

`for` loops

`export` command

Environment varaible `PATH`