What is Linux?
Operating System (OS): Linux is an open-source operating system, like Windows or macOS, but with unique features and flexibility.
Kernel: The core of Linux, which manages hardware (CPU, memory, devices) and software interactions.
Free and Open Source: Linux is freely available for anyone to use, modify, and distribute.
Unix-like: Linux is based on Unix, a powerful OS used in servers and research.
Popular for Servers and Developers: Many servers, developers, and companies (e.g., Google, Facebook) prefer Linux for its security, stability, and customization options.
Why Use Linux?
Security: Linux is known for being secure, making it ideal for servers.
Customizable: You can control almost every part of Linux, tailoring it to your needs.
Community Support: A large community offers resources, support, and tools.
Variety of Distros: Linux has "distributions" (distros) like Ubuntu, Fedora, and CentOS, each with unique features.
What is an Operating System (OS)?
Definition: An OS is software that manages computer hardware and software resources. It provides an interface between the user and the computer hardware.
Functions of an OS:
Manages Hardware: Controls CPU, memory, storage, and peripheral devices.
Runs Applications: Allows you to install and run programs like browsers, games, and office software.
User Interface: Provides a way for users to interact with the computer, like through a graphical interface or command line.
Examples of OS:
Windows: Popular for personal computers.
macOS: Used in Apple computers.
Linux: Known for flexibility and security, widely used in servers.
Android and iOS: Operating systems for mobile devices.
Why is the OS Important?
- Without an OS, a computer would be hard to use, as it would have no way to understand commands or manage programs. The OS makes it easy to run multiple tasks and keep everything organized.
What is a Shell?
Definition: A shell is a program that allows users to interact with the operating system by typing commands. It interprets these commands and passes them to the OS to execute.
Purpose: It acts as a bridge between the user and the OS, enabling you to run scripts, manage files, and control processes.
Shell Types:
Command-Line Shell: Allows you to interact by typing commands (e.g., Bash, Zsh).
Graphical Shell: Provides a GUI (Graphical User Interface) for interaction (e.g., Windows Explorer in Windows OS).
Types of Shells (Popular in Linux/Unix)
Bash (Bourne Again Shell)
Most Popular: Default shell on many Linux systems.
Features: Supports scripting, command history, and job control.
Example: Commonly used in Linux and macOS for scripting.
Zsh (Z Shell)
Highly Customizable: Known for themes, plugins, and user-friendly features.
Features: Auto-completion, spelling correction, and more advanced scripting options.
Example: Often chosen by advanced users who want a modern shell.
Ksh (Korn Shell)
Balanced Performance: Combines scripting power of Bourne shell with C shell-like features.
Features: Good for scripting with enhanced functionality over older shells.
Example: Used in some Unix systems, especially for older scripts.
Tcsh (TENEX C Shell)
C-Like Syntax: Built on the C Shell (csh) and often used by developers familiar with C programming.
Features: Command-line editing and history, ideal for C programming-style scripting.
Example: Used mostly in certain Unix environments.
Fish (Friendly Interactive Shell)
User-Friendly: Offers intuitive syntax and helpful features like syntax highlighting.
Features: Colorful display, auto-suggestions, easier scripting.
Example: Popular among new users and those who want interactive features.
Choosing a Shell
For Beginners: Bash or Fish, due to simplicity and wide availability.
For Advanced Users: Zsh or Ksh for more customization and scripting power.
Linux File System Hierarchy :
/
(Root): The top directory; everything starts here. All other directories are subdirectories of root.
Key Directories in the File System
/bin
: Essential command binaries for all users (e.g.,ls
,cp
,mv
).- Example: Commands you can run even when other directories aren't accessible.
/boot
: Contains files needed to start the system, like the Linux kernel and bootloader.- Example: The
vmlinuz
file, which is the Linux kernel.
- Example: The
/dev
: Contains device files, which represent hardware devices (e.g., disks, USB drives).- Example:
sda
represents the first hard drive.
- Example:
/etc
: Configuration files for the system and applications.- Example:
passwd
file for user account settings.
- Example:
/home
: Contains personal directories for each user.- Example:
/home/safia
is Safia's personal directory for files and settings.
- Example:
/lib
: Libraries essential for system binaries in/bin
and/sbin
.- Example: Shared libraries, similar to
.dll
files in Windows.
- Example: Shared libraries, similar to
/media
and/mnt
: Temporary mount points for external drives or media.- Example:
/media/usb
where USB devices might be mounted.
- Example:
/opt
: Optional software and packages installed manually.- Example: Used to store software not included by default, like custom programs.
/proc
: Virtual filesystem providing system information.- Example:
/proc/cpuinfo
for CPU details,/proc/meminfo
for memory details.
- Example:
/root
: The home directory for the root (administrator) user.- Example: Only accessible to the root user.
/sbin
: System binaries used by the root user for system maintenance (e.g.,fsck
,reboot
).- Example: Commands restricted to system administrators.
/srv
: Data served by the system, such as web or FTP data.- Example: Used by web servers to store website data.
/tmp
: Temporary files, cleared on reboot.- Example: Used for temporary storage by applications.
/usr
: Contains user programs and utilities.Subdirectories:
/usr/bin
: General user commands./usr/local
: Locally installed software not managed by the OS package manager.
/var
: Variable files, like logs, cache, and spool files.- Example:
/var/log
contains system logs,/var/tmp
stores temporary files.
- Example:
Basic Linux Commands
ls
- Lists files and directories.- Example:
ls /home
lists all files in the/home
directory.
- Example:
cd
- Changes the directory.- Example:
cd /home/safia
navigates to thesafia
directory.
- Example:
pwd
- Displays the current directory path.- Example:
pwd
shows the full path of your current location.
- Example:
mkdir
- Creates a new directory.- Example:
mkdir projects
creates a new folder namedprojects
.
- Example:
rmdir
- Removes an empty directory.- Example:
rmdir old_folder
deletesold_folder
if it's empty.
- Example:
rm
- Deletes files and directories.- Example:
rm file.txt
removesfile.txt
;rm -r folder
removesfolder
and its contents.
- Example:
File Handling Commands
cp
- Copies files or directories.- Example:
cp file1.txt /backup/
copiesfile1.txt
to the/backup
directory.
- Example:
mv
- Moves or renames files and directories.- Example:
mv old_name.txt new_name.txt
renames a file.
- Example:
cat
- Displays file contents.- Example:
cat file.txt
shows the content offile.txt
.
- Example:
nano
/vim
- Opens file in text editors.- Example:
nano file.txt
opensfile.txt
for editing in thenano
editor.
- Example:
touch
- Creates a new empty file.- Example:
touch newfile.txt
creates an empty file callednewfile.txt
.
- Example:
System Monitoring Commands
top
- Shows real-time system processes.- Example:
top
displays active processes and resource usage.
- Example:
df
- Checks disk space usage.- Example:
df -h
shows available and used disk space in a human-readable format.
- Example:
free
- Displays available and used memory.- Example:
free -h
shows memory usage in a human-readable format.
- Example:
uptime
- Shows how long the system has been running.- Example:
uptime
displays the system's uptime and load average.
- Example:
ps
- Lists running processes.- Example:
ps aux
shows all running processes with detailed info.
- Example:
File Permission Commands
chmod
- Changes file permissions.- Example:
chmod 755
script.sh
sets read, write, and execute permissions for the owner.
- Example:
chown
- Changes file owner and group.- Example:
chown user:group file.txt
setsuser
as the owner offile.txt
.
- Example:
Networking Commands
ping
- Tests connectivity to another host.- Example:
ping
google.com
checks if Google is reachable.
- Example:
ifconfig
- Displays or configures network interfaces.- Example:
ifconfig
shows IP and network configuration.
- Example:
wget
- Downloads files from the internet.- Example:
wget
http://example.com/file.zip
downloadsfile.zip
.
- Example:
System Management Commands
sudo
- Runs commands as a superuser.- Example:
sudo apt update
updates package lists on Debian-based systems.
- Example:
shutdown
- Shuts down or restarts the system.- Example:
shutdown now
turns off the system immediately;shutdown -r now
restarts it.
- Example:
reboot
- Restarts the system.- Example:
reboot
immediately reboots the computer.
- Example:
head and tail commands :
head
Command
Purpose: Displays the first few lines of a file.
Default: Shows the first 10 lines if no number is specified.
Usage:
- Basic Syntax:
head [options] [file]
Common Options:
-n NUM
: Show the firstNUM
lines.
Practical Examples:
Show the first 10 lines of a file:
head myfile.txt
Show the first 5 lines:
head -n 5 myfile.txt
tail
Command
Purpose: Displays the last few lines of a file.
Default: Shows the last 10 lines if no number is specified.
Usage:
- Basic Syntax:
tail [options] [file]
Common Options:
-n NUM
: Show the lastNUM
lines.-f
: Follow the file in real-time (useful for logs).
Practical Examples:
Show the last 10 lines of a file:
tail myfile.txt
Show the last 5 lines:
tail -n 5 myfile.txt
Follow a log file in real-time:
tail -f /var/log/syslog
Installing Applications with apt
Install an Application:
Command:
sudo apt-get install <app_name>
Example: To install Nginx:
sudo apt-get install nginx
Update Package Index:
Command:
sudo apt update
Purpose: Checks for the latest version of packages.
Example:
sudo apt update
Upgrade Installed Packages:
Command:
sudo apt upgrade
Purpose: Installs the latest versions of all installed packages.
Example:
sudo apt upgrade
Checking Nginx Installation Directory
Check Nginx Files:
Command:
cd /var/www/html
Purpose: Navigate to the default directory for web files served by Nginx.
Checking Linux Version
Command:
lsb_release -a
Purpose: Displays detailed information about the Linux distribution.
Example:
lsb_release -a
Booting Process
Definition: The booting process is the sequence of events that occurs when you turn on your computer until the operating system is fully loaded.
Steps:
Power On: Press the power button.
POST (Power-On Self Test): The system checks hardware components (RAM, CPU, etc.).
Bootloader: The bootloader (like GRUB) loads the operating system.
Kernel Loading: The OS kernel initializes the system and hardware.
System Initialization: User-space processes and services start, leading to the login prompt.
Difference Between nmtui
and nmcli
nmtui
:Type: Text User Interface (TUI).
Usage: Provides a graphical interface for managing network connections in the terminal.
Example: Run
nmtui
to open a menu-driven interface.
nmcli
:Type: Command-Line Interface (CLI).
Usage: Allows for managing network connections using command-line commands.
Example: Use
nmcli d
to show devices.
Difference Between |
(Pipe) and &&
|
(Pipe):Function: Takes the output of one command and uses it as input for another command.
Example:
ls | grep "txt"
- Lists files and filters to show only.txt
files.
&&
:Function: Executes the next command only if the previous command succeeded (returned a zero exit status).
Example:
mkdir new_folder && cd new_folder
- Creates a folder and changes into it only if the creation was successful.
What is a Server?
Definition: A server is a computer or system that provides resources, data, services, or programs to other computers, known as clients, over a network.
Types of Servers:
Web Server: Hosts websites and delivers web pages to users. (e.g., Apache, Nginx)
File Server: Stores and manages files for multiple users. (e.g., FTP servers)
Database Server: Provides database services and manages data storage. (e.g., MySQL, PostgreSQL)
Application Server: Hosts applications and allows users to access software remotely.
Functionality:
Receives requests from clients.
Processes those requests.
Sends back the appropriate responses or data.
Example:
- When you access a website, your computer (client) sends a request to a web server, which processes the request and returns the website’s content to your browser.
What is SDLC?
- Definition: The Software Development Life Cycle (SDLC) is a structured process used for developing software applications. It outlines the stages involved in software development from conception to deployment and maintenance.
Phases of SDLC:
Requirement Gathering and Analysis:
Collect and analyze user requirements.
Document functional and non-functional requirements.
Planning:
Define the project scope, resources, timeline, and budget.
Create a project plan.
Design:
Create architectural and detailed design specifications.
Decide on technologies, system interfaces, and data structures.
Implementation (Coding):
Write the actual code based on the design specifications.
Perform unit tests to ensure individual components work correctly.
Testing:
Validate the complete system to ensure it meets requirements.
Identify and fix defects through various testing methods (e.g., integration, system, user acceptance testing).
Deployment:
Release the software to the production environment.
Make it available for users.
Maintenance:
Provide ongoing support and updates.
Fix any issues that arise after deployment.
Example:
- If a company wants to develop a mobile app, they would follow the SDLC to ensure the app is built efficiently, meets user needs, and functions correctly.
What is the Waterfall Model?
- Definition: The Waterfall Model is a linear and sequential approach to software development. Each phase must be completed before the next phase begins, resembling a waterfall flowing downwards through distinct stages.
Phases of the Waterfall Model:
Requirement Analysis:
Gather and document all requirements from stakeholders.
Define what the software needs to accomplish.
System Design:
Create a system architecture and design specifications based on requirements.
Plan system interfaces and data structures.
Implementation (Coding):
Develop the software by writing code according to the design specifications.
Focus on individual components or modules.
Integration and Testing:
Combine all modules and test the complete system for defects.
Ensure the software meets requirements through various testing methods.
Deployment:
Release the software to users in a production environment.
Make the system available for real-world use.
Maintenance:
Provide support and address any issues after deployment.
Implement updates or enhancements as necessary.
Characteristics:
Linear Process: Each phase must be completed before moving to the next, making it easy to understand and manage.
Documentation: Emphasizes thorough documentation at each stage, which can aid in future maintenance.
Advantages:
Simple and easy to manage due to its structured approach.
Clearly defined stages make progress easy to track.
Disadvantages:
Inflexible to changes; difficult to go back to a previous phase once it's completed.
Late testing may lead to discovering issues late in the process, which can be costly to fix.
Example:
- In a project to develop a banking application, requirements are gathered, the system is designed, coded, tested, deployed, and then maintained in a sequential manner.
What is the Agile Model?
- Definition: Agile is a way to develop software in small, flexible steps, allowing teams to adapt to changes quickly.
Key Features:
Customer Feedback: Involve customers throughout the project to ensure their needs are met.
Short Cycles: Work is done in short periods called iterations or sprints (usually 2-4 weeks).
Continuous Improvement: Regularly check what works well and what can be improved.
Agile Steps:
Planning: Decide what to build and list features.
Development: Create the software in short bursts (sprints).
Testing: Test features in the same sprint to catch issues early.
Review: Show what has been built to customers and get feedback.
Retrospective: Discuss what went well and what could be better for the next sprint.
Advantages:
Flexible: Can easily change based on customer needs.
Frequent Releases: Delivers working software often, keeping customers happy.
Team Collaboration: Encourages teamwork and communication.
Example:
- If building a mobile app, the team works on one feature at a time, gets feedback, and adjusts plans based on that feedback.
User Management :
- Definition: User management involves creating, modifying, and deleting user accounts and managing permissions and groups.
Key Commands for User Management:
Add a New User:
Command:
sudo adduser <username>
Description: Creates a new user account and prompts for a password and user information.
Example:
sudo adduser safia
Delete a User:
Command:
sudo deluser <username>
Description: Removes a user account. Optionally, you can remove their home directory with
--remove-home
.Example:
sudo deluser safia
Modify User Information:
Command:
sudo usermod [options] <username>
Description: Modifies an existing user's properties, such as changing the username, home directory, or shell.
Example: Change a user's shell:
sudo usermod -s /bin/bash safia
Change User Password:
Command:
passwd <username>
Description: Updates the password for a specified user.
Example:
sudo passwd safia
List All Users:
Command:
cat /etc/passwd
Description: Displays a list of all user accounts on the system.
Example:
cat /etc/passwd
How to make a group :
sudo groupadd <group_name>
Check User Information:
Command:
id <username>
Description: Shows the user ID (UID) and group ID (GID) of a user along with their groups.
Example:
id safia
Add User to a Group:
Command:
sudo gpasswd -a <username> <groupname>
Description: Adds a user to a specified group.
Example:
sudo gpasswd -a <username> <groupname>
Delete a Group:
Command:
sudo delgroup <groupname>
Description: Removes a specified group.
Example:
sudo delgroup developers
List Groups:
Command:
cat /etc/group
Description: Displays a list of all groups on the system.
Example:
cat /etc/group
Practical Example:
To add a new user named "safia," change her password, and then add her to the "sudo" group:
sudo adduser safia sudo passwd safia sudo groupadd DevOps sudo gpasswd-a safia DevOps
File Permissions in Linux
- Definition: File permissions control who can read, write, or execute a file or directory in Linux.
Types of Permissions:
Read (r):
- Allows the user to read the contents of a file or list the contents of a directory.
Write (w):
- Allows the user to modify the contents of a file or add/remove files in a directory.
Execute (x):
- Allows the user to execute a file (if it's a program or script) or access a directory.
User Types:
Owner: The user who created the file.
Group: A set of users that have shared permissions.
Others: All other users on the system.
Viewing Permissions:
Command:
ls -l
Description: Lists files and directories with detailed information, including permissions.
Example:
ls -l
Output Example:
-rwxr-xr-- 1 safia users 2048 Jan 01 10:00 example.txt
Breakdown of permissions:
- rwx r-x r--:
-
(file type)rwx
(owner has read, write, execute)r-x
(group has read, execute)r--
(others have read only)
Changing Permissions:
Command:
chmod <permissions> <file>
Description: Changes the permissions of a file or directory.
Examples:
Numeric Mode:
Command:
chmod 755 example.txt
Meaning: Owner can read, write, execute; group and others can read and execute.
Symbolic Mode:
Command:
chmod u+x example.txt
Meaning: Adds execute permission for the owner (
u
for user).
Changing Ownership:
Command:
chown <owner>:<group> <file>
Description: Changes the owner and/or group of a file or directory.
Example:
sudo chown safia example.txt
Changing Group:
Command:
chgrp <group> <file>
Description: Changes the group associated with a file.
Example:
sudo chgrp users example.txt
Practical Example:
To give everyone read and execute permissions on a script named "run.sh":
chmod a+rx run.sh
Numeric Mode in Linux File Permissions
- Definition: Numeric mode uses numbers to set file permissions for the owner, group, and others.
Permission Values:
Read (r): 4
Write (w): 2
Execute (x): 1
Calculating Permissions:
Combine the values to set permissions for each user type:
Owner (u): First digit
Group (g): Second digit
Others (o): Third digit
Examples:
No Permissions:
000
- Owner, group, and others have no permissions.
Read Only for Owner:
400
- Owner can read; group and others have no permissions.
Read and Write for Owner:
600
- Owner can read and write; group and others have no permissions.
Read, Write, and Execute for Owner:
700
- Owner can read, write, and execute; group and others have no permissions.
Read and Execute for Group:
750
- Owner has read, write, execute; group has read and execute; others have no permissions.
Full Permissions for Owner, Read for Group and Others:
744
- Owner has read, write, execute; group has read; others have read.
Setting Permissions Using Numeric Mode:
Command:
chmod <numeric_mode> <file>
Example: To set full permissions for the owner and read/execute for group and others:
chmod 755 example.txt
Breakdown of 755
:
Owner (7): Read (4) + Write (2) + Execute (1) = 7
Group (5): Read (4) + Execute (1) = 5
Others (5): Read (4) + Execute (1) = 5
Practical Example:
To set read and write permissions for the owner, and read permissions for group and others:
chmod 644 document.txt
What is SSH?
- SSH: A secure way to connect to another computer over the internet.
Connecting to an AWS Ubuntu Server via SSH
Step 1: Create a Server on AWS
Launch an EC2 Instance:
Go to the AWS Management Console.
Navigate to EC2 and click on "Launch Instance."
Select "Ubuntu Server" from the list of AMIs (Amazon Machine Images).
Choose your instance type (e.g., t2.micro) and configure settings as needed.
Make sure to create or select a key pair (this is important for SSH access).
Step 2: Connect to the Server Using SSH
Copy the SSH DNS:
After the instance is launched, go to the EC2 Dashboard.
Select your instance, and look for the "Public DNS (IPv4)" in the details.
It should look something like:
ec2-xx-xx-xx-xx.compute-1.amazonaws.com
.
Prepare Your Local Environment:
Open your terminal.
Change to the directory where your key file (the .pem file) is located:
cd ~/Downloads
Set Permissions for Your Key File:
Run the following command to restrict permissions on your key file (replace
<key_name>
with the name of your .pem file):chmod 400 <key_name>.pem
Connect to Your Instance:
Use the SSH command to connect to your instance. Replace
<username>
withubuntu
(for Ubuntu servers) and<ssh_dns>
with the DNS you copied earlier:ssh -i <key_name>.pem ubuntu@<ssh_dns>
Example Command Sequence:
# Step 2: Navigate to the Downloads directory
cd ~/Downloads
# Step 3: Set permissions for your key file
chmod 400 my-key.pem # Replace 'my-key.pem' with your actual key file name
# Step 4: Connect to the AWS instance
ssh -i my-key.pem ubuntu@ec2-xx-xx-xx-xx.compute-1.amazonaws.com # Replace with your actual DNS
Notes:
Make sure that your security group allows SSH access (port 22) from your IP address.
If prompted about the authenticity of the host, type "yes" to continue.
Connecting Server A to Server B Using SSH Key Pair
Step 1: Generate SSH Key Pair on Server A
Open Terminal on Server A:
ssh-keygen
This command generates a public and private key pair.
Press
Enter
to accept the default file location (usually~/.ssh/id_rsa
).
Navigate to the SSH Directory:
cd .ssh
Copy the Public Key to Clipboard:
cat id_rsa.pub
- Copy the output of this command (the public key).
Step 2: Add Public Key to Server B
Log in to Server B:
- Use your normal login method (username/password or other).
Navigate to the SSH Directory on Server B:
cd .ssh
Open the
authorized_keys
File:If the file doesn’t exist, you can create it:
touch authorized_keys
Open the file using
vim
or any text editor:vim authorized_keys
Paste the Public Key:
Press
i
to enter insert mode invim
.Paste the copied public key into the file.
Save and exit
vim
by pressingEsc
and typing:wq
.
Step 3: Connect from Server A to Server B
Return to Server A:
- Open your terminal on Server A.
Connect to Server B Using SSH:
- Use the private key to connect:
ssh -i id_rsa ubuntu@<server_B_dns>
- Replace
<server_B_dns>
with the DNS or IP address of Server B.
Example Command Sequence:
On Server A:
# Generate SSH Key Pair
ssh-keygen 8
# Navigate to SSH Directory
cd .ssh
# Copy Public Key to Clipboard
cat id_rsa.pub
On Server B:
# Navigate to SSH Directory
cd .ssh
# Create authorized_keys if it doesn't exist
touch authorized_keys
# Open authorized_keys with vim
vim authorized_keys
# (Paste the public key here, then save and exit)
Back to Server A:
# Connect to Server B using the private key
ssh -i id_rsa ubuntu@<server_B_dns>
Notes:
Ensure the permissions are set correctly:
- The
.ssh
directory should be700
and theauthorized_keys
file should be600
.
- The
If you encounter any issues, check that the SSH service is running on both servers and that the firewall allows SSH connections.
Using SCP to Transfer a File from Local to Server
Step 1: Prepare the Server
Connect to Your Server:
- Use SSH to connect to your server:
ssh -i <private_key_name> ubuntu@<server_dns>
- Replace
<private_key_name>
with your private key file and<server_dns>
with your server’s DNS or IP address.
Create a Directory on the Server:
- Run the following command to create a new directory:
mkdir safia
Navigate to the Directory:
- Change into the new directory:
cd safia
Get the Current Path:
- Use the following command to print the working directory:
pwd
- This should display something like
/home/ubuntu/safia
. Note this path as you'll need it for the SCP command.
Exit the Server:
- Use the following command to exit back to your local system:
exit
Step 2: Transfer the File from Local to Server
Open Your Local Terminal:
- Ensure you are in the directory where your file (e.g.,
sample.csv
) is located.
- Ensure you are in the directory where your file (e.g.,
Use SCP to Copy the File:
- Run the following command to transfer the file to the server:
scp -i <private_key_name> sample.csv ubuntu@<server_dns>:/home/ubuntu/safia
- Replace
<private_key_name>
with your private key,sample.csv
with your actual file name, and<server_dns>
with your server's DNS or IP address.
Example Command Sequence:
On the Server:
# Step 1: Connect to the server
ssh -i my-key.pem ubuntu@ec2-xx-xx-xx-xx.compute-1.amazonaws.com
# Create a directory named 'safia'
mkdir safia
# Change into the directory
cd safia
# Get the current path
pwd # Should output something like /home/ubuntu/safia
# Exit back to the local system
exit
On the Local System:
# Step 2: Transfer the file using SCP
scp -i my-key.pem sample.csv ubuntu@ec2-xx-xx-xx-xx.compute-1.amazonaws.com:/home/ubuntu/safia
Notes:
Make sure the
scp
command is available on your local machine (it's usually included with SSH).Ensure that your server's firewall allows incoming connections on port 22 (the default SSH/SCP port).
After executing the SCP command, you should see the progress of the file transfer. Once completed, the file
sample.csv
will be available in the/home/ubuntu/safia
directory on your server.
Using SCP to Transfer a File from Server to Local
Step 1: Prepare the File on the Server
Connect to Your Server:
- Use SSH to connect to your server:
ssh -i <private_key_name> ubuntu@<server_dns>
- Replace
<private_key_name>
with your private key file and<server_dns>
with your server’s DNS or IP address.
Check for the File:
If the file you want to copy is already on the server, you can skip the next step.
If not, create a new file for practice:
vim filename.txt
In
vim
, pressi
to enter insert mode and write some content in the file.After writing, press
Esc
, type:wq
, and hitEnter
to save and exit.
Confirm the File Exists:
- Use the following command to list files and ensure
filename.txt
is created:
- Use the following command to list files and ensure
ls -l
Exit the Server:
- Use the following command to exit back to your local system:
exit
Step 2: Transfer the File from Server to Local
Open Your Local Terminal:
- Ensure you are in the directory where you want to save the file.
Use SCP to Copy the File:
- Run the following command to transfer the file from the server to your local machine:
scp -i <private_key_name> ubuntu@<server_dns>:/home/ubuntu/filename.txt .
- Replace
<private_key_name>
with your private key, and<server_dns>
with your server's DNS or IP address.
Verify the File Transfer:
- Use the following command to list files and check if
filename.txt
is in your local directory:
- Use the following command to list files and check if
ls -l
Example Command Sequence:
On the Server:
# Step 1: Connect to the server
ssh -i my-key.pem ubuntu@ec2-xx-xx-xx-xx.compute-1.amazonaws.com
# Create a new file named 'filename.txt'
vim filename.txt
# (Write some content, then save and exit vim)
# Confirm the file exists
ls -l
# Exit back to the local system
exit
On the Local System:
# Step 2: Transfer the file using SCP
scp -i my-key.pem ubuntu@ec2-xx-xx-xx-xx.compute-1.amazonaws.com:/home/ubuntu/filename.txt .
# Verify the file transfer
ls -l
Notes:
Ensure that the
scp
command is available on your local machine (it's usually included with SSH).Make sure your server's firewall allows incoming connections on port 22 (the default SSH/SCP port).
After executing the SCP command, you should see the progress of the file transfer. Once completed, the file
filename.txt
will be available in your current local directory.
Updating and Upgrading with APT
1. Update Package List
This command refreshes the local package index with the latest available versions from the repositories:
sudo apt update
2. Upgrade Installed Packages
This command upgrades all installed packages to their latest versions without removing any packages:
sudo apt upgrade
3. Full Upgrade (optional)
This command intelligently handles changing dependencies and will remove obsolete packages if necessary:
sudo apt full-upgrade
Example Command Sequence:
# Update package list
sudo apt update
# Upgrade installed packages
sudo apt upgrade
# Full upgrade (optional)
sudo apt full-upgrade
Managing Services with systemctl
1. Check Status of a Service
To check the status of a specific service (e.g.,
nginx
):sudo systemctl status nginx
2. Start a Service
To start a service:
sudo systemctl start nginx
3. Stop a Service
To stop a service:
sudo systemctl stop nginx
4. Restart a Service
To restart a service:
sudo systemctl restart nginx
5. Enable a Service at Boot
To enable a service to start on boot:
sudo systemctl enable nginx
6. Disable a Service at Boot
To disable a service from starting at boot:
sudo systemctl disable nginx
Example Command Sequence for Systemctl:
# Check the status of nginx
sudo systemctl status nginx
# Start the nginx service
sudo systemctl start nginx
# Stop the nginx service
sudo systemctl stop nginx
# Restart the nginx service
sudo systemctl restart nginx
# Enable nginx to start on boot
sudo systemctl enable nginx
# Disable nginx from starting on boot
sudo systemctl disable nginx
Notes:
Always ensure to run update and upgrade commands periodically to keep your system secure and up to date.
Use
systemctl
to manage services efficiently, which is important for maintaining server applications and services.
Differences Between sudo apt
and sudo apt-get
Feature | sudo apt | sudo apt-get |
User Interface | User-friendly, more readable output | More command-line oriented, less user-friendly |
Features | Combines functionalities of apt-get and apt-cache | Provides basic package management commands |
Commands Available | Includes commands like apt update , apt upgrade , apt install , etc. | Includes commands like apt-get update , apt-get upgrade , apt-get install , etc. |
Progress Bar | Displays a progress bar for operations | No progress bar; only displays text output |
Package Management | Recommended for interactive use, simpler for users | Recommended for scripting and advanced usage |
Compatibility | Compatible with most commands from apt-get | A long-standing tool; widely used in scripts |
Key Points:
Usage: While both commands can be used for package management,
sudo apt
is designed to be more user-friendly for interactive use, whereassudo apt-get
is better suited for scripts and automated tasks.Recommendation: For everyday use,
sudo apt
is recommended because of its simplicity and additional features. For scripting or advanced usage,sudo apt-get
remains a powerful tool.
Example Commands:
Using
apt
:sudo apt update # Update package list sudo apt upgrade # Upgrade installed packages sudo apt install <package_name> # Install a package
Using
apt-get
:sudo apt-get update # Update package list sudo apt-get upgrade # Upgrade installed packages sudo apt-get install <package_name> # Install a package
Both commands ultimately achieve similar goals but do so with different levels of user interaction and output.
The purge
command in the context of APT (Advanced Package Tool) is used to completely remove a package from your system, including its configuration files. Here’s a concise overview:
Using apt purge
What Does It Do?
The
purge
command removes the specified package along with any configuration files that may have been created during its installation.This is different from
remove
, which only removes the package but keeps its configuration files.
Basic Syntax
sudo apt purge <package_name>
Example Command
- To purge a package, for example,
nginx
, you would run:
sudo apt purge nginx
Steps to Use apt purge
Open Terminal: Access your terminal.
Run the Purge Command:
sudo apt purge <package_name>
Confirm Removal: Follow any prompts to confirm the removal.
Clean Up Unused Dependencies (optional):
- After purging, you can remove any unused dependencies with:
sudo apt autoremove
Notes
Use
purge
when you want to ensure that no traces of the package, including configuration files, remain on your system.This is particularly useful when you are troubleshooting or reinstalling software and want a clean state.
grep :
The
grep
command in Linux is used to search for specific patterns or text within files. It’s very powerful for finding information in large files or across multiple files.What Does It Do?
grep
searches for a specified string or pattern within files and outputs the matching lines.It’s case-sensitive by default but can be made case-insensitive with options.
Basic Syntax
grep [options] "pattern" filename
Examples
Search for a Specific Word in a File
grep "word" filename.txt
- This finds and displays all lines containing
"word"
infilename.txt
.
- This finds and displays all lines containing
Case-Insensitive Search
grep -i "word" filename.txt
- The
-i
option makes the search case-insensitive, so it matches"word"
,"Word"
,"WORD"
, etc.
- The
Search Across Multiple Files
grep "word" *.txt
- Searches for
"word"
in all.txt
files in the current directory.
- Searches for
Display Line Numbers
grep -n "word" filename.txt
- The
-n
option shows line numbers for each matching line.
- The
Recursive Search in Directories
grep -r "word" /path/to/directory
- The
-r
option searches for"word"
recursively through all files in the specified directory.
- The
Inverted Search (Exclude Matching Lines)
grep -v "word" filename.txt
- The
-v
option inverts the search, displaying lines that do not contain"word"
.
- The
Example Commands Summary
grep "pattern" filename.txt # Basic search for "pattern" in a file
grep -i "pattern" filename.txt # Case-insensitive search
grep -n "pattern" filename.txt # Show line numbers with results
grep -r "pattern" /path/to/dir # Recursive search in a directory
grep -v "pattern" filename.txt # Exclude lines containing "pattern"
Practical Example
Suppose you have a file named log.txt
and want to find all entries that mention "error" (case-insensitive) and see the line numbers:
grep -i -n "error" log.txt
Notes
grep
is an essential tool for log analysis, finding specific data in files, and filtering outputs.You can combine
grep
with other commands or use it in scripts for more complex searches.
awk :
The
awk
command in Linux is a powerful text-processing tool used for pattern scanning, data extraction, and manipulation of file contents. It processes each line of input text and applies specified actions to the parts that match given patterns.What Does It Do?
awk
reads text line-by-line, splits each line into fields based on a delimiter (default is whitespace), and allows you to perform actions on each line or field.Commonly used for filtering text, extracting columns, and performing calculations.
Basic Syntax
awk 'pattern { action }' filename
Examples
Print Specific Columns
awk '{ print $1 }' filename.txt
Prints the first column (
$1
) of each line infilename.txt
.$1
,$2
, etc., represent columns.$0
represents the entire line.
Filter and Print Columns Based on Pattern
awk '/pattern/ { print $1, $3 }' filename.txt
- Searches for lines containing
"pattern"
and prints the first and third columns.
- Searches for lines containing
Use a Different Field Separator
awk -F ":" '{ print $1 }' /etc/passwd
- The
-F ":"
option sets:
as the field separator. This example prints the first field in each line of/etc/passwd
.
- The
Perform Calculations
awk '{ print $1, $2 * $3 }' filename.txt
- Multiplies the second and third columns for each line and prints the first column followed by the result.
Print Line Numbers with Lines
awk '{ print NR, $0 }' filename.txt
NR
is the line number. This prints each line with its line number.
Example Commands Summary
awk '{ print $1 }' filename.txt # Print first column
awk '/pattern/ { print $1, $2 }' filename.txt # Filter lines with "pattern" and print columns 1 and 2
awk -F "," '{ print $1, $2 }' filename.csv # Use comma as delimiter in CSV file
awk '{ print NR, $0 }' filename.txt # Print line numbers with lines
awk '{ print $1, $2 + $3 }' filename.txt # Add columns 2 and 3, print with column 1
Practical Example
Suppose you have a file data.txt
with this content:
safia 90 85
puja 78 95
naziya 88 82
To calculate and print the total score for each person:
awk '{ print $1, $2 + $3 }' data.txt
Output:
safia 175
puja 173
naziya 170
Notes
awk
is ideal for data extraction and manipulation, especially in structured text.You can combine
awk
with other commands for advanced data processing.
sed :
The
sed
(Stream Editor) command in Linux is a powerful tool for performing text transformations, such as find and replace, deleting lines, or inserting text in files. It processes text line-by-line, applying specified operations.What Does It Do?
sed
reads each line of input and performs specified actions like substituting text, deleting lines, or inserting text.It’s often used for quick, inline text manipulations.
Basic Syntax
sed 'command' filename
Examples of sed
Commands
Substitute Text (Find and Replace)
sed 's/oldtext/newtext/' filename.txt
Replaces the first occurrence of
"oldtext"
with"newtext"
in each line.To replace all occurrences on each line, add the
g
flag:sed 's/oldtext/newtext/g' filename.txt
Replace Text in Place (Directly Modify the File)
sed -i 's/oldtext/newtext/g' filename.txt
- The
-i
option modifies the file directly without needing to save it to a new file.
- The
Delete Specific Lines
sed '3d' filename.txt
Deletes the third line of
filename.txt
.To delete a range of lines, specify the range:
sed '2,4d' filename.txt
- This deletes lines 2 to 4.
Print Only Matching Lines
sed -n '/pattern/p' filename.txt
- The
-n
option suppresses default output, so only lines containing"pattern"
are printed.
- The
Insert Text Before a Line
sed '3i\New line of text' filename.txt
- Inserts
"New line of text"
before line 3.
- Inserts
Replace Only Lines That Match a Pattern
sed '/pattern/s/oldtext/newtext/' filename.txt
- Replaces
"oldtext"
with"newtext"
only on lines that contain"pattern"
.
- Replaces
Example Commands Summary
sed 's/old/new/' filename.txt # Replace first occurrence of "old" with "new" per line
sed 's/old/new/g' filename.txt # Replace all occurrences of "old" with "new" per line
sed -i 's/old/new/g' filename.txt # Replace "old" with "new" in file directly
sed '3d' filename.txt # Delete the third line
sed '2,4d' filename.txt # Delete lines 2 to 4
sed -n '/pattern/p' filename.txt # Print only lines with "pattern"
sed '3i\Inserted line' filename.txt # Insert text before line 3
sed '/pattern/s/old/new/' filename.txt # Replace "old" with "new" only on lines containing "pattern"
Practical Example
Suppose you have a file greetings.txt
:
Hello World
Hello Universe
Goodbye World
To replace "Hello"
with "Hi"
on lines containing "World"
:
sed '/World/s/Hello/Hi/' greetings.txt
Output:
Hi World
Hello Universe
Goodbye World
Final Notes
sed
is particularly useful for batch editing text files, such as log files or configuration files.Remember to use
-i
cautiously, as it edits files directly.
Difference between Sed and Awk
sed
andgrep
are both text-processing commands in Linux, but they serve different purposes.1. Purpose
grep
: Primarily used for searching text. It looks for lines that match a specific pattern or string and outputs those matching lines.sed
: Primarily used for editing text. It’s a stream editor that can substitute, delete, insert, or transform text in files.
2. Basic Usage
grep
Example: To find lines containing the word "error" in a file:grep "error" filename.txt
- This will print lines containing "error" without modifying the original file.
sed
Example: To replace the word "error" with "warning" in a file:sed 's/error/warning/g' filename.txt
- This will replace "error" with "warning" in each line where it appears (without changing the file unless
-i
is added).
- This will replace "error" with "warning" in each line where it appears (without changing the file unless
3. Main Functions
grep
:Searches for specific text patterns.
Displays only lines that match the pattern.
Useful for filtering data based on specific keywords.
sed
:Edits text (like replace, delete, or insert).
Can modify files directly if used with
-i
(in-place).Useful for making quick text transformations, like find and replace across multiple lines.
4. Common Flags
grep
:-i
: Case-insensitive search.-v
: Invert match (show lines that do NOT match the pattern).-n
: Show line numbers for matching lines.
sed
:-i
: Edit file in-place (modifies the original file).s
: Substitution (replace text)./pattern/d
: Delete lines matching a pattern.
5. Practical Comparison Example
Suppose you have a file log.txt
with the following lines:
Error: File not found
Warning: Disk space low
Info: Operation successful
Error: Network timeout
Using
grep
to Find Lines with "Error":grep "Error" log.txt
Output:
Error: File not found Error: Network timeout
Using
sed
to Replace "Error" with "Alert":sed 's/Error/Alert/g' log.txt
Output:
Alert: File not found Warning: Disk space low Info: Operation successful Alert: Network timeout
Summary Table
Feature | grep | sed |
Purpose | Search/filter lines | Edit/transform text |
Primary Function | Pattern matching and display | Text substitution, deletion, insertion |
Modifies File | No | Yes (with -i flag) |
Typical Use Case | Find lines containing specific text | Find and replace, delete, or insert text |
Example | grep "word" filename | sed 's/old/new/g' filename |
FIND :
The
find
command in Linux is a powerful tool used to search for files and directories in a specified directory hierarchy. It allows you to locate files based on various criteria such as name, type, size, modified date, and permissions.Basic Syntax
find [path] [options] [expression]
path
: Starting directory for the search (e.g.,/home
,.
for the current directory).options
: Specifies search criteria (e.g.,-name
,-type
).expression
: What you want to find or do with the results.
Examples of find
Commands
Find Files by Name
find /path/to/search -name "filename"
- Searches for files named
"filename"
in the specified directory.
- Searches for files named
Find Files by Partial Name (Case-Insensitive)
find /path/to/search -iname "filename"
-iname
makes the search case-insensitive, so it finds"filename"
,"FileName"
, etc.
Find Files by Extension
find /path/to/search -name "*.txt"
- Finds all
.txt
files in the directory and subdirectories.
- Finds all
Find Files by Type
Regular files:
find /path/to/search -type f
Directories:
find /path/to/search -type d
Finds files (
-type f
) or directories (-type d
).
Find Files by Size
find /path/to/search -size +50M
Finds files larger than 50 MB.
Common size suffixes:
k
(KB),M
(MB),G
(GB).
Find Files Modified in the Last X Days
find /path/to/search -mtime -7
Finds files modified in the last 7 days.
-mtime +7
finds files modified more than 7 days ago.
Find Files by Permissions
find /path/to/search -perm 755
- Finds files with the exact permission
755
.
- Finds files with the exact permission
Find and Execute a Command on Each File
find /path/to/search -name "*.log" -exec rm {} \;
Finds all
.log
files and deletes them usingrm
.{}
is a placeholder for each found file, and\;
ends the command.
Practical Examples
Find All
.conf
Files in/etc
Directoryfind /etc -name "*.conf"
- Lists all configuration files in
/etc
.
- Lists all configuration files in
Find and List Large Files
find / -type f -size +1G
- Finds files larger than 1 GB on the entire system.
Find Recently Created or Modified Files
find . -type f -ctime -1
- Finds files created or changed within the last 24 hours in the current directory.
Find Empty Directories
find /path/to/search -type d -empty
- Finds all empty directories.
Summary Table
Task | Command Example |
Find files by name | find /path -name "filename" |
Find files by extension | find /path -name "*.ext" |
Find directories only | find /path -type d |
Find large files | find /path -size +100M |
Find modified files (last X days) | find /path -mtime -X |
Find files by permission | find /path -perm 755 |
Find and delete files | find /path -name "*.tmp" -exec rm {} \; |
Tips
.
indicates the current directory.-exec
is very useful for actions like moving, copying, or deleting files automatically.Combine
find
with other commands (e.g.,grep
,xargs
) for advanced searches.
Differences between find and grep Command
find
and grep
are both Linux commands used for searching, but they serve different purposes and are suited to different types of tasks. Here’s a breakdown of their differences:
1. Purpose
find
: Used to search for files and directories within a directory structure based on criteria like name, size, type, permissions, or modification date.grep
: Used to search within files for specific text patterns. It reads the content of files and prints lines that match the specified pattern.
2. What They Search
find
: Works at the file system level. It searches for files and directories but does not look inside the contents of files.grep
: Works at the text/content level. It searches within files to find lines that match the given pattern.
3. Typical Use Cases
find
:Locating files or directories based on attributes like name, size, or permissions.
Searching for files modified within a specific timeframe.
Running commands on files found (e.g., deleting old files, changing permissions).
grep
:Finding occurrences of a particular word or phrase within a file.
Filtering output to show only lines containing specific patterns.
Searching for patterns in logs, configurations, or text files.
4. Basic Usage
find
: Searches for files and directories.find /path/to/search -name "filename"
- Finds files named
"filename"
in the specified directory and subdirectories.
- Finds files named
grep
: Searches within files for text patterns.grep "pattern" filename.txt
- Finds lines containing
"pattern"
withinfilename.txt
.
- Finds lines containing
5. Practical Comparison Example
Suppose you have a directory with multiple files, and you want to locate a specific file and search for a keyword within it.
Using
find
to Locate the File by Name:find /directory -name "target_file.txt"
- Finds
target_file.txt
within/directory
.
- Finds
Using
grep
to Search for a Pattern in a File:grep "keyword" target_file.txt
- Searches within
target_file.txt
for lines containing"keyword"
.
- Searches within
6. Combination of find
and grep
You can use both commands together to find files and search within them:
find /path -name "*.log" -exec grep "error" {} \;
- Finds all
.log
files in/path
and searches each for the term"error"
.
Summary Table
Feature | find | grep |
Purpose | Search for files and directories | Search within file content |
Scope | File and directory names, paths, properties | Text or patterns within file content |
Primary Use Case | Locating files by attributes | Finding lines with specific text patterns |
Can Modify Files | Yes (with -exec option) | No (read-only) |
Example | find /path -name "*.txt" | grep "text" filename.txt |
Summary
Use
find
when you need to locate files or directories.Use
grep
when you need to search for specific text within files.
Mounting a Volume (Basic Storage Device) :
Step 1:
Attach EBS Volume to EC2 Instance
Open EC2 Dashboard: Go to EC2 Dashboard in AWS and select Instances.
Navigate to Elastic Block Store:
In the left panel, click on Elastic Block Store > Volumes.
Choose an Available Volume. Make sure it’s in the same Availability Zone as your EC2 instance.
Attach Volume:
Select the volume, click Actions > Attach Volume.
Select your EC2 instance from the dropdown list and specify the device name (usually
/dev/xvdf
or similar).Click Attach to link the volume to your EC2 instance.
Step 2:
Format and Mount the Volume on EC2
Log in to Your EC2 Instance:
- Use SSH to connect to your EC2 instance.
Format the Volume:
Format the attached volume as an ext4 file system.
mkfs - make a file system
-t - type
ext4 - external file system
sudo mkfs -t ext4 /dev/xvdf
List block storage devices to verify the device.
lsblk
Create a Mount Directory:
Switch to root and create a directory to mount the volume.
sudo su mkdir /mnt/new_volume
Mount the Volume:
Mount the volume to the newly created directory.
mount /dev/xvdf /mnt/new_volume
Verify the Mount:
Use
df -h
to check if the volume is mounted successfully.df -h
You should see
/dev/xvdf
mounted at/mnt/new_volume
.
Optional: Make Mount Persistent
To ensure the volume mounts automatically on reboot, add it to the /etc/fstab
file.
Get the UUID of the Volume:
sudo blkid /dev/xvdf
Edit the fstab File:
sudo vim /etc/fstab
Add an Entry:
Add the following line (replace
UUID=xxxx-xxxx
with your actual UUID):UUID=xxxx-xxxx /mnt/new_volume ext4 defaults,nofail 0 2
Now your EBS volume is set up and will persist through reboots.
Physical VS volumes group VS logical volumes in linux :
In Linux, Logical Volume Manager (LVM) provides a flexible way to manage disk storage. With LVM, you can create Physical Volumes (PVs), group them into Volume Groups (VGs), and divide them into Logical Volumes (LVs) that you can resize or manage dynamically. Here’s a breakdown of each component and the steps for using LVM.
Components of LVM
Physical Volume (PV):
Represents actual physical storage devices (e.g., disks or partitions) in LVM.
Example:
/dev/xvdf
,/dev/xvdg
,/dev/xvdh
in AWS.
Volume Group (VG):
Combines multiple physical volumes into a single storage pool.
Provides more flexibility, as the space can be allocated dynamically to logical volumes.
Logical Volume (LV):
Created within a volume group and acts as a usable partition.
LVs can be resized easily, unlike traditional partitions.
You can mount LVs to directories like a typical filesystem.
LVM (Logical Volume Manager):
- A tool to manage PVs, VGs, and LVs, making it easier to handle dynamic storage requirements.
Steps for Setting Up LVM with AWS Volumes
Step 1: Create and Attach Volumes in AWS
Go to your EC2 Dashboard in AWS.
In Elastic Block Store, create and attach three volumes (e.g.,
/dev/xvdf
,/dev/xvdg
,/dev/xvdh
) to your EC2 instance.
Step 2: Set Up LVM in Linux
Switch to Root User:
sudo su
Create Physical Volumes:
- Initialize the attached volumes for LVM.
pvcreate /dev/xvdf /dev/xvdg /dev/xvdh
- Check the list of physical volumes:
pvs
Create a Volume Group:
- Combine the physical volumes into a single volume group.
vgcreate tws_vg /dev/xvdf /dev/xvdg /dev/xvdh
- List all volume groups:
vgs
Create a Logical Volume:
Allocate space for a logical volume (e.g., 10 GB) within the volume group.
if you Adding Volume With an existing volume (optional)
lvcreate -L 10G -n tws_lv tws_vg
lvextend -L +5G /dev/tws_vg/tws_lv
Display Volume Information:
To view detailed information for each component:
Physical Volumes:
pvdisplay
Volume Groups:
vgdisplay
Logical Volumes:
lvdisplay
Step 3: Format and Mount the Logical Volume
Format the Logical Volume:
mkfs.ext4 /dev/tws_vg/tws_lv
Create a Mount Point and Mount the LV:
mkdir /mnt/tws_volume mount /dev/tws_vg/tws_lv /mnt/tws_volume
Verify the Mount:
df -h
- This will display the mounted logical volume and its storage details.
Summary Table
Task | Command Example |
Create Physical Volumes | pvcreate /dev/xvdf /dev/xvdg /dev/xvdh |
List Physical Volumes | pvs |
Create Volume Group | vgcreate tws_vg /dev/xvdf /dev/xvdg /dev/xvdh |
List Volume Groups | vgs |
Create Logical Volume | lvcreate -L 10G -n tws_lv tws_vg |
Display Volume Info (PV) | pvdisplay |
Display Volume Info (VG) | vgdisplay |
Display Volume Info (LV) | lvdisplay |
Format Logical Volume | mkfs.ext4 /dev/tws_vg/tws_lv |
Mount Logical Volume | mount /dev/tws_vg/tws_lv /mnt/tws_volume |