Useful Linux Commands

A collection of useful Linux commands, scripts, and tips I’ve accumulated over time. This includes everything from package management and Docker operations to image processing with ImageMagick and video editing with ffmpeg.

Package Management Link to heading

Find versions of a package in apt and install specific version Link to heading

apt list -a terraform
apt install terraform=1.3.7

Shell and Terminal Link to heading

Recording a shell session to a GIF Link to heading

Using asciinema to record and agg to convert to GIF:

# Start recording
asciinema rec output_name.cast

# Run your commands in the new shell that opens
# Press Ctrl+D when done

# Convert to GIF
agg output_name.cast output_name.gif

Run command in background and detach from terminal Link to heading

nohup ./long-running-script.sh &> /dev/null &

Create an alias permanently Link to heading

echo "alias ll='ls -lah'" >> ~/.bashrc
source ~/.bashrc

Find and replace text in all files recursively Link to heading

find . -type f -name "*.txt" -exec sed -i 's/old_text/new_text/g' {} +

Networking Link to heading

Check which process is using a specific port Link to heading

sudo lsof -i :8080
# or
sudo netstat -tulpn | grep :8080
# or
sudo ss -tulpn | grep :8080

Test network speed between two servers using iperf Link to heading

On the server:

iperf -s

On the client:

iperf -c SERVER_IP

Download file with resume capability Link to heading

wget -c https://example.com/large-file.iso

Show all listening ports and services Link to heading

sudo ss -tulpn

Flush DNS cache Link to heading

# Ubuntu/Debian
sudo systemd-resolve --flush-caches

# macOS
sudo dscacheutil -flushcache; sudo killall -HUP mDNSResponder

Find your public IP address Link to heading

curl ifconfig.me
# or
curl icanhazip.com

Find your IPv4 IP Address Link to heading

curl -4 icanhazip.com

Security Link to heading

Allow tcpdump through AppArmor Link to heading

sudo apparmor_parser -R /etc/apparmor.d/usr.sbin.tcpdump

Generate a strong random password Link to heading

openssl rand -base64 32
# or
pwgen 32 1

Check for open ports on remote host Link to heading

nmap -p 1-65535 example.com

Create an SSH key pair Link to heading

ssh-keygen -t ed25519 -C "[email protected]"

View SSL certificate details Link to heading

openssl x509 -in certificate.crt -text -noout

Test SSL/TLS connection Link to heading

openssl s_client -connect example.com:443

Disk and Filesystem Link to heading

Show disk usage by directory, sorted by size Link to heading

du -h --max-depth=1 | sort -hr

Find largest files in directory Link to heading

find . -type f -exec du -h {} + | sort -rh | head -n 20

Monitor disk usage in real-time Link to heading

watch -n 1 df -h

Find and delete files older than N days Link to heading

find /path/to/dir -type f -mtime +30 -delete

Create a swap file Link to heading

sudo fallocate -l 4G /swapfile
sudo chmod 600 /swapfile
sudo mkswap /swapfile
sudo swapon /swapfile
echo '/swapfile none swap sw 0 0' | sudo tee -a /etc/fstab

Process Management Link to heading

List processes by memory usage Link to heading

ps aux --sort=-%mem | head -n 20

List processes by CPU usage Link to heading

ps aux --sort=-%cpu | head -n 20

Run process with lower priority (nice) Link to heading

nice -n 19 ./cpu-intensive-task.sh

Change priority of running process Link to heading

renice -n 10 -p PID

Show process tree Link to heading

pstree -p
# or for specific user
pstree -u username

Text Processing Link to heading

Extract specific columns from CSV Link to heading

awk -F',' '{print $1,$3}' file.csv
# or
cut -d',' -f1,3 file.csv

Remove duplicate lines from file Link to heading

sort file.txt | uniq
# or preserve order
awk '!seen[$0]++' file.txt

Find and replace in file (in-place) Link to heading

sed -i 's/old/new/g' file.txt
# macOS requires empty string: sed -i '' 's/old/new/g' file.txt

Print lines between two patterns Link to heading

sed -n '/START_PATTERN/,/END_PATTERN/p' file.txt

Convert file to lowercase Link to heading

tr '[:upper:]' '[:lower:]' < input.txt > output.txt

Remove empty lines Link to heading

sed '/^$/d' file.txt
# or
grep -v '^$' file.txt

Print specific line number Link to heading

sed -n '42p' file.txt
# or
awk 'NR==42' file.txt

Join lines with comma Link to heading

paste -sd ',' file.txt

Compare two files side by side Link to heading

diff -y file1.txt file2.txt
# or colored diff
colordiff file1.txt file2.txt

Finding Files Link to heading

Find files modified in last N minutes Link to heading

find /path -type f -mmin -60

Find files by size Link to heading

find /path -type f -size +100M  # larger than 100MB
find /path -type f -size -1M    # smaller than 1MB

Find and execute command on files Link to heading

find /path -name "*.log" -exec gzip {} \;

Find files with specific permissions Link to heading

find /path -type f -perm 0777

Find broken symlinks Link to heading

find /path -xtype l

Find files owned by specific user Link to heading

find /path -user username

Archive and Compression Link to heading

Compress with maximum compression Link to heading

tar -czf - directory/ | pigz -9 > archive.tar.gz
# or with xz for better compression
tar -cJf archive.tar.xz directory/

Split large file into chunks Link to heading

split -b 100M largefile.iso chunk_

Combine split files Link to heading

cat chunk_* > largefile.iso

Docker Link to heading

Install Docker Compose Link to heading

sudo curl -L "https://github.com/docker/compose/releases/download/v2.19.0/docker-compose-$(uname -s)-$(uname -m)" -o /usr/bin/docker-compose

List all Docker containers and their IP addresses Link to heading

docker ps -q | xargs -n 1 sudo docker inspect \
  -f '{{.Name}}%tab%{{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}}%tab%{{range .NetworkSettings.Networks}}{{.GlobalIPv6Address}}%tab%{{end}}' \
  | sed 's#%tab%#\t#g' \
  | sed 's#/##g' \
  | sort \
  | column -t -N NAME,IP\(s\) -o $'\t'

Clean up unused Docker resources Link to heading

# Remove all stopped containers
docker container prune -f

# Remove all dangling images
docker image prune -f

# Remove all unused volumes
docker volume prune -f

# Remove everything unused
docker system prune -a --volumes -f

Copy files to/from container Link to heading

docker cp file.txt container_name:/path/in/container
docker cp container_name:/path/in/container/file.txt ./

Show Docker resource usage Link to heading

docker stats

Export and import Docker images Link to heading

# Export
docker save myimage:latest | gzip > myimage.tar.gz

# Import
gunzip -c myimage.tar.gz | docker load

Git Link to heading

Clone repository with specific branch Link to heading

git clone -b branch_name https://github.com/user/repo.git

Undo last commit (keep changes) Link to heading

git reset --soft HEAD~1

Undo last commit (discard changes) Link to heading

git reset --hard HEAD~1

View commit history with graph Link to heading

git log --oneline --graph --all --decorate

Stash changes temporarily Link to heading

git stash
git stash list
git stash pop

Delete local branch Link to heading

git branch -d branch_name
# Force delete
git branch -D branch_name

Delete remote branch Link to heading

git push origin --delete branch_name

Show changed files in last commit Link to heading

git diff --name-only HEAD~1 HEAD

Revert a specific file to previous commit Link to heading

git checkout HEAD~1 -- path/to/file

Find commits that changed a specific file Link to heading

git log --follow -- path/to/file

Clean untracked files Link to heading

git clean -fd

File Operations Link to heading

Rename all files in a directory to match the directory name Link to heading

This script is specific to JPG files and handles both regular and _b suffixed files:

find * -type d -exec bash -c '
cd "{}"
c=1
for f in *; do
    if [[ "$f" == *_b.jpg ]]; then
        mv "$f" "{}"-$((c-1))_b.jpg
    else
        mv "$f" "{}"-$c.jpg
        ((c++))
    fi
done
' \;

Image Processing Link to heading

Python script to crop photos off a full album page scan Link to heading

This OpenCV-based script extracts individual photos from a scanned album page by detecting contours:

import cv2
import os
import sys

def extract_photos(input_path, output_folder):
    # Read the image
    image = cv2.imread(input_path)

    # Convert the image to grayscale
    gray = cv2.cvtColor(image, cv2.COLOR_BGR2GRAY)

    # Threshold the image to isolate bright regions
    _, thresh = cv2.threshold(gray, 100, 255, cv2.THRESH_BINARY)

    # Find contours in the thresholded image
    contours, _ = cv2.findContours(thresh, cv2.RETR_EXTERNAL, cv2.CHAIN_APPROX_SIMPLE)

    # Create output folder if it doesn't exist
    if not os.path.exists(output_folder):
        os.makedirs(output_folder)

    # Get existing photos in the output folder to avoid overwriting
    existing_photos = [f for f in os.listdir(output_folder) if f.startswith("photo_") and f.endswith(".jpg")]

    # If there are existing photos, find the highest count and start from the next one
    if existing_photos:
        highest_count = max([int(f.split("_")[1].split(".")[0]) for f in existing_photos])
        count = highest_count + 1
    else:
        count = 0

    # Calculate the 5% of the total image area
    min_area = 0.05 * image.shape[0] * image.shape[1]

    # Iterate through the contours
    for contour in contours:
        # Get the bounding rectangle
        x, y, w, h = cv2.boundingRect(contour)

        # Filter by area to only include photos larger than 5% of the total image size
        if w * h > min_area:
            cropped = image[y:y+h, x:x+w]
            output_path = os.path.join(output_folder, f'photo_{count}.jpg')
            cv2.imwrite(output_path, cropped)
            count += 1

    print(f"Extracted photos to {output_folder}")

if __name__ == "__main__":
    if len(sys.argv) != 2:
        print("Usage: python script_name.py path_to_your_image.jpg")
        sys.exit(1)

    input_path = sys.argv[1]
    output_folder = 'extracted_photos'

    extract_photos(input_path, output_folder)

ImageMagick commands Link to heading

Remove all image metadata Link to heading

find . -type f -name "*.jpg" -exec mogrify -strip {} \;

Increase colour vibrance on images Link to heading

for x in $(ls -1 *.jpg); do 
  convert "$x" -modulate 100,110 "$x"
done

Increase brightness and contrast on images Link to heading

convert original.jpg -brightness-contrast 10x5 bv-10.jpg

Remove outside white border (for Polaroids) Link to heading

export count=1
for x in $(ls -1 *.jpg); do 
  convert "$x" -flatten -fuzz 50% -trim +repage "${x}_${count}.jpg"
  count=$(expr $count + 1)
done

Upscale images Link to heading

find . -type f -name "*.jpg" -exec convert {} -morphology Close Diamond:1 -blur 0x1 -normalize {} \;

Flip all images horizontally Link to heading

for x in $(ls -1 *.jpg); do 
  convert "${x}" -flop "${x}"
done

Video Processing Link to heading

ffmpeg commands Link to heading

Join two videos together Link to heading

ffmpeg -i video1.mp4 -i video2.mp4 \
  -filter_complex "[0:v] [0:a] [1:v] [1:a] concat=n=2:v=1:a=1 [v] [a]" \
  -map "[v]" -map "[a]" output.mp4

Detect silent timestamps in video Link to heading

ffmpeg -hide_banner -vn -i video.mp4 \
  -af "silencedetect=n=-20dB:d=2" \
  -f null - 2>&1 | grep "silence_end" | awk '{print $5 " " $8}' > silence.txt

Crop video by time Link to heading

ffmpeg -i video.mp4 -ss 02:43:04 -to 02:55:30 -c:v copy -c:a copy output.mp4

Join videos from text file list Link to heading

Create a text file with the list of videos:

file 'video1.mp4'
file 'video2.mp4'

Then concatenate:

ffmpeg -f concat -safe 0 -i files.txt -c copy 'joined.MP4'

Vertically flip a video Link to heading

ffmpeg -i input.mp4 -vf vflip output.mp4

Reverse a video Link to heading

Video must be split into chunks first, as the entire video is loaded into memory (even a 1.4GB file will consume 64GB of memory):

# Split video into 5-minute segments
sudo ffmpeg -i source.mp4 -map 0 -c copy -f segment -segment_time 300 -reset_timestamps 1 video_%03d.mp4

# Reverse each segment
for f in *.mp4; do
    sudo ffmpeg -i "$f" -vf reverse "${f/.mp4/_reversed.mp4}"
done

# Build file list in reverse order
for f in *_reversed.mp4; do 
    echo "file '$f'" > tmp.txt
    cat fileList.txt >> tmp.txt 2>/dev/null
    rm -f fileList.txt
    mv tmp.txt fileList.txt
done

# Concatenate all reversed segments
ffmpeg -f concat -safe 0 -i fileList.txt -c copy final_reversed_video.mp4

Crop to a section of video Link to heading

Format: crop=width:height:x:y

ffmpeg -i in.mp4 -filter:v "crop=80:60:200:100" -c:a copy out.mp4

Crop all video files in a directory based on their filename Link to heading

Expects filenames like crop-HHMMSS-outputname.mp4:

for f in crop-*; do 
  time=$(echo "$f" | awk -F'-' '{print substr($2,1,2)":"substr($2,3,2)":"substr($2,5,2)}')
  outname=$(echo "$f" | sed -e 's/^crop-[0-9]*-//')
  ffmpeg -hide_banner -loglevel warning -i "$f" -ss 00:00:00 -t "$time" -c:v copy "$outname"
done

Remove sections of a video with no motion Link to heading

ffmpeg -i input.mp4 -vf "select=gt(scene\,0.03),setpts=N/(15*TB)" output.mp4

Stabilize video Link to heading

Requires ffmpeg binaries with vidstab built in. See John Van Sickle’s builds.

# Generate the stabilization file
ffmpeg -i input.mp4 -vf vidstabdetect=shakiness=7 -f null -

# Stabilize the actual video
ffmpeg -i input.mp4 -vf vidstabtransform=smoothing=30:zoom=5:input="transforms.trf" stabilized.mp4

Convert audio to mono Link to heading

ffmpeg -i input.mp4 -c:v copy -ac 1 mono-output.mp4

Upscale video to 4K Link to heading

ffmpeg -i INPUT_FILE \
  -vf scale=3840x2160:flags=lanczos \
  -c:v libx264 \
  -crf 13 \
  -c:a aac \
  -b:a 512k \
  -preset slow \
  OUTPUT_FILE

Homebrew (macOS) Link to heading

Export all installed packages Link to heading

brew bundle dump

Dotfiles Management Link to heading

Chezmoi Link to heading

Add a file to Chezmoi:

chezmoi add <file>

Navigate to the Chezmoi working directory:

chezmoi cd

Data Processing Link to heading

jq - Convert CSV to JSON Link to heading

First, create a jq script file (csv2json.jq):

def objectify(headers):
  # For jq 1.4, replace the following line by: def tonumberq: .;
  def tonumberq: tonumber? // .;
  . as $in
  | reduce range(0; headers|length) as $i ({}; .[headers[$i]] = ($in[$i] | tonumberq) );

def csv2table:
  # For jq 1.4, replace the following line by:  def trim: .;
  def trim: sub("^ +";"") |  sub(" +$";"");
  split("\n") | map( split(",") | map(trim) );

def csv2json:
  csv2table
  | .[0] as $headers
  | reduce (.[1:][] | select(length > 0) ) as $row
      ( []; . + [ $row|objectify($headers) ]);

csv2json

Then use it to convert a CSV file:

jq -R -s -f csv2json.jq credential_report.csv > credential_report.json

Google Takeout - Extract archives Link to heading

Extract all files from a Google Takeout archive:

pv takeout-* | tar xzif -

rsync -az --progress . /mnt2/photos/Takeout/Google\ Photos/

System Troubleshooting Link to heading

Fix broken sudo / etc ownership Link to heading

If /etc ownership gets broken and sudo stops working, use PolicyKit to fix it. This requires three terminal windows:

Shell 1 - Get the PID:

echo $$

Shell 2 - Start the PolicyKit agent:

pkttyagent -p <PID OF SHELL 1>
# Output:
# ==== AUTHENTICATING FOR org.freedesktop.policykit.exec ===
# Authentication is needed to run `/usr/bin/chown' as the super user
# Authenticating as: root
# Password:
# ==== AUTHENTICATION COMPLETE ===

Shell 3 - Fix the ownership:

pkexec chown -R root:root /etc

SSL Certificates Link to heading

Let’s Encrypt with Cloudflare DNS validation Link to heading

Example using Docker with Certbot for a Vault instance:

docker run --rm --name certbot --net=host \
  -v "/docker/vault/etc/letsencrypt:/etc/letsencrypt" \
  -v "/docker/vault/var:/var/lib/letsencrypt" \
  -v $PWD/cf.ini:/cf.ini \
  certbot/dns-cloudflare certonly \
  --email "[email protected]" \
  --agree-tos \
  -n \
  --dns-cloudflare \
  --dns-cloudflare-credentials /cf.ini \
  -d "vault.example.com"