Programster's Blog

Tutorials focusing on Linux, programming, and open-source

Linux CLI Cheatsheet

cheatsheet cli linux-administration

Below are some short snippets of code to help you perform day-to-day tasks in Linux. If you can't find what you are looking for, make sure to look at the BASH cheatsheet.

JSON
1. Installation
2. Filtering
Filesystem
File Manipulation
Disk Management
User Management
Group Management
Networking
Misc
References

JSON

Allo the following examples use jq which can be installed with

sudo apt install jq -y

If you have some command or script that outputs in JSON, then it may not be in a readable format. To have it output to the terminal in a readable format, just pipe it to jq like so:

cat myJsonData.json | jq

If you then want that output to go to a file, do the following:

echo '[1,2,3,4,5,6,7,8,9,10]' | jq . > my-output-file.json

Filtering

If you wish to filter down to a specific sub block, then you can do this like so:

cat someComplicatedJson.json | jq '.primaryBlockKeyName.subBlockKey.tertiaryBlockKey

You can chain sub-blocks with . as many times as you desire.

For more examples of using jq, refer to the following:

Filesystem

Copy File(s) But Don't Overwrite

Sometimes you want to copy a file to another location, but are worries about the possibility of overwriting something that already exists. If that's the case, use the -n or --no-clobber parameter.

cp -n $SOURCE $DEST

Move File(s) But Don't Overwrite

In the same situation as above, but for moving file(s).

mv -i

mv --interactive

Unlike with cp -n, this will propt you if you wish to overwrite files, whereas cp -n will just immediately exit with an error.

Get Folder Sizes

If you've ever completely filled a system to the point where you can't even install ncdu to track where all your storage is gone, you can get the size of all the folders at your current level by executing:

du -sh *

However, if you can use ncdu instead, that is a much better tool.

Move all Files of a Type To Own Folder

If you have a jumble of files with no apparent structure, perhaps from a backup, and want to move a ceratain file type (in this example PDFs) into their own directory then you can execute the following:

mkdir pdf
find . -name "*.pdf" -type f -exec /bin/mv {} ./pdf \;

Obviously if you just wanted to copy the files instead, use:

mkdir pdf
find . -name "*.pdf" -type f -exec /bin/cp {} ./pdf \;

If you forget to create the directory and it doesn't already exist then you may lose your data.

Using xargs may be better, please refer here.

List All Files Within All Subdirectories

find . -type f

If you wish to also list the directories:

find .

List All Files With Specific File Extensions

find ./ -type f \( -iname \*.jpg -o -iname \*.png \)

List all Files Over A Certain Size

The following command will recursively find individual files larger than [size] in MB. This could be useful for finding really large files to consider compressing/deleting.

find . -type f -size +[size]M

File Manipulation

Replace Text In Files (Sed)

I use variations of the sed command to manipulate config files for automated scripts

SEARCH="search text here"
REPLACE="replace text here"
FILEPATH="/path/to/file.conf"
sed -i "s;$SEARCH;$REPLACE;" $FILEPATH

If you want to replace an entire line, then just use .* as shown below.

SEARCH="bind-address.*"
REPLACE="bind-address = $PRIVATE_IP
FILEPATH="/etc/mysql/my.cnf"
sudo sed -i "s;$SEARCH;$REPLACE;" $FILEPATH

You can use [[:space:]]\+ when you don't know how many whitespaces or tabs there are. E.g.

SEARCH="datadir[[:space:]]\+= /var/lib/mysql"
REPLACE="datadir = /media/data/mysql"
FILEPATH="/etc/mysql/my.cnf"
sudo sed -i "s;$SEARCH;$REPLACE;" $FILEPATH

To replace an entire line that contains text that you are looking for in the middle of the line, you can use the following:

SEARCH="^.*\bYour pattern or sentence goes here\b.*$"
REPLACE="This will replace the entire line"
FILEPATH="$DIR/irapcms.sql"
sed -i "s;$SEARCH;$REPLACE;" $FILEPATH

source

Write To File

echo "my content here" | tee $FILEPATH

echo "my content here" > $FILEPATH

cat << EOF > [filepath]
[script content]
EOF

Write To File With Sudo Privileges

echo "my content here" | sudo tee $FILEPATH

cat << EOF  | sudo tee $FILEPATH
[script content]
EOF

cat << EOF > [filepath]
[script content]
EOF

Append To File

echo "my content here" >>  $FILEPATH

Append To File (With Sudo Privs)

echo "my content here" | sudo tee -a $FILEPATH

cat << EOF  | sudo tee -a $FILEPATH
[script content]
EOF

Split File Into Chunks

Let's say you have a massive text file, perhaps a giant log file. If that text file became unmanageable, and you wanted to split it into lot of smaller files that had up to x number of lines each, then you could do the following with the split command:

$INPUT_FILE="my-file.txt"
$OUTPUT_FILE_PREFIX="output-"
$OUTPUT_FILE_EXTENSION=".txt"
$MAX_LINES_PER_FILE=10000

split \
  --verbose \
  --numeric-suffixes \
  --suffix-length=2 \
  --lines=$MAX_LINES_PER_FILE \
  --additional-suffix=$OUTPUT_FILE_EXTENSION \
  $INPUT_FILE \
  $OUTPUT_FILE_PREFIX

If you are going to need more than 99 files, then increase the --suffix-length accordingly. E.g. 3 for up to 999 files.

Gzip A File

The following command will take the given file, and replace it with a gzipped one with the same name, but with a .gz extension. This is particularly useful for MySQL dumps and log files.

gzip myLogFile.txt

To then undo the operation do:

gzip --decompress myLogFile.txt.gz

Alternatively, the --uncompress flag is a synonym that will do the same thing:

gzip --uncompress myLogFile.txt.gz

This will replace the .gz file with the decompressed file. If you want to keep the file, simply add the --keep flag.

Disk Management

Get Utilization

df -h

Or you could install pydf for a simpler interface.

Secure Erase Disk

The best way to erase a disk is to use secure erase. For an SSD, this will reach areas that the other commands won't, and return it to peak performance.

Alternatively, you can use shred

shred -n 1 -vz /dev/sdb

... or use scrub

scrub -p dod /dev/sdb

... or you could use dd.

Check Sata Connection of Drives

dmesg | grep -i  ahci | grep -i --color Gbps

Check the SATA Connection Of Specific Drive

smartctl -a /dev/sda | grep "^SATA"

You will need to install the smartctl package if you haven't already.

Get Block Size

sudo blockdev --getbsz /dev/sd[x]

The output will be in bytes, so typically this will be 4096 to represent a 4k block size.

User Management

Add User With Adduser

The simplest way to add users is to use the command below which will ask you a series of questions and set up their home directory.

sudo adduser [username]

Add User With Useradd

If you are wanting to add a user as part of a script, or just don't want to set up a home directory, you can use

sudo useradd [username]

In that format, it will create a user without a home directory or even a password, use the options -m to automatically create a home directory, and -p to specify the encrypted password. For example, the commands below will create a user called test with a random password:

USER="test"
PASSWORD=`openssl rand -base64 16`
PASSWORD=${PASSWORD::-2}
ENCRYPTED_PASSWORD=$(openssl passwd -1 ${PASSWORD})
sudo useradd -m -p $ENCRYPTED_PASSWORD $USER
echo "Please log in with password: $PASSWORD"

Delete User

sudo deluser [user]

A full-blown command in which we specify the user ID, group ID, whether to create a home, and the shell to use would be:

sudo useradd \
  --no-create-home  \
  --uid 1019  \
  --gid 100  \
  --shell /usr/sbin/nologin  \
   myUserName

Get User's ID

id --user $USERNAME

If you just want your own user ID, you can run:

id --user

Change User's ID

The following command would change the programster user to user ID 1026

sudo usermod -u 1026 programster

You would need to run this command as another user, when the specified user is not logged in.

Get Users Primary Group ID

id --group $USER

Change Users Primary Group ID

You can also do the following to set a user's primary group:

usermod --gid primary_group_name programster

The user's primary group determines the default group ID on all files/folders the user creates, thus all files the user now creates will have group ownership set to that by default.

Add User To Group

sudo adduser $USERNAME $GROUPNAME

The adduser command may not always be available. In such a scenario, you may wish to use:

sudo usermod \
  --append $USER_NAME \
  --groups $GROUP_NAME

All other tutorials I see mention doing usermod -a -G $GROUP_NAME $USER_NAME, but I have tested that this way round works, and I feel it is more memorable and intutitive.

Append User To Multiple Groups

If you want to append a user to multiple groups in one go, then you can do the following:

sudo usermod \
  --append $USER_NAME \
  --groups $GROUP1,$GROUP2,$GROUP3

Set User's Groups

If you want to specify the list of groups a user should be part of, which would append/remove the user from groups as appropriate to ensure they match the list, then you would do the following:

sudo usermod \
  $USER_NAME \
  --groups $GROUP1,$GROUP2,$GROUP3

Remove User From Group

sudo deluser $USERNAME $GORUPNAME

List Groups User Belongs To

groups $USER_TO_CHECK

Group Management

Create Group

sudo groupadd myGroupName

If you wish to specify the ID of the group, you would use:

sudo groupadd --gid 1003 myGroupName

Get Group's ID

The following will output the ID of the specified group:

cut -d: -f3 < <(getent group $GROUP_NAME)

Get Name Of Group From ID

Use the following to convert a group ID, to the name of the group

getent group $GROUP_ID | cut -d: -f1

List All Groups In System

The following command will list all the groups on the system.

cut -d: -f1 /etc/group | sort

Change Group's ID

groupmod --gid $DESIRED_GROUP_ID $GROUP_TO_CHANGE

For example, the following command would change the "admins" group to 1026.

sudo groupmod --gid 1026 admins

If the name of the group being changed, is the primary group of the logged in users ID, then you would need to run this command as another user.

Rename Group

groupmod --new-name $NEW_GROUP_NAME $CURRENT_GROUP_NAME

e.g.

groupmod --new-name apacheUser www-data

Networking

Output IPs

hostname -I

If you want each IP on a newline, use the command below:

hostname -I | xargs | tr [:space:] '\n'

Stop Processes That Are Using Port

The following command will kill all processes that are using the port 1234.

lsof -ti:1234 | xargs kill -9

Blink NIC

If you have multiple NICs in a computer and want to know which one is which, you can make it blink. The command below will make eth0 blink for 30 seconds.

ethtool -p eth0 30

Output All Interfaces

ip addr

Output Specific Interface

If you just want details about a single interface, you can do the following:

INTERFACE="enp2s0"
ip addr show $INTERFACE

Misc

Hashes

md5sum $FILEPATH | awk '{print $1}'
sha1sum $FILEPATH | awk '{print $1}'
sha256sum $FILEPATH | awk '{print $1}'

Copy CD To Disk

If you have a CD-Rom drive (remember those things?), and want to create an ISO version on your computer, you can do it like so:

dd if=/dev/cdrom of=filename.iso

Set File Timestamp

We all know that you can use touch $FILENAME to update a file to say it was edited right now. However, if you want to manually set a time that is in the past, you can use:

touch -d '1 June 2018 11:02' $FILENAME

Here is a little php snippet for programmatically setting time of files:

<?php

$filename = "test-php-file.txt"; // specify name for file to create/edit timestmap of
$timestamp = time() - (60*60); // set a unix timestamp here.
$dateString = date("d F Y H:i:s", $timestamp);
$cmd = "touch -d '{$dateString}' {$filename}";
shell_exec($cmd);

Download File And Set Custom Name

The following command will use wget to download a file, but set your own custom name for the file when it is downloaded.

wget --output-document="my-custom-filename.zip" http://my.domain.com/path-to-file

Get Time Taken For Command To Run

Sometimes it's useful to know how long a command takes. All you have to do is wrap it like so:

CMD="YOUR COMMAND GOES HERE"
utime="$( TIMEFORMAT='%lU';time ( $CMD ) 2>&1 1>/dev/null )"
echo "$utime"

CentOS - Check If Reboot Required

#!/bin/bash
LAST_KERNEL=$(rpm -q --last kernel | perl -pe 's/^kernel-(\S+).*/$1/' | head -1)
CURRENT_KERNEL=$(uname -r)

test $LAST_KERNEL = $CURRENT_KERNEL || echo REBOOT

Get Ubuntu Release Name

lsb_release -cs

This should output something like:

xenial

Generate Random Password

This is my favourite option from How-To-Geek: 10 Ways to Generate a Random Password from the Command Line

openssl rand -base64 32

If you need it to not contain special characters, you could use:

NUM_CHARACTERS=24
head /dev/urandom | tr -dc A-Za-z0-9 | head -c$NUM_CHARACTERS

... or if hexadecimal is ok (all lowercase):

NUM_CHARACTERS=24
openssl rand -hex $NUM_CHARACTERS

Find Server's Fingerprint

If you swap change a server around on an IP or domain name, you may see the following message:

ECDSA host key for dns.programster.org has changed and you have requested strict checking.
Host key verification failed.

If you wish to manually check what your server's fingerprint is, you can run the following command:

ssh-keygen -l -f /etc/ssh/ssh_host_ecdsa_key.pub

The -l tells ssh-keygen to "show fingerprint of specified public key file." and the -f is for specifying the filepath to the file we wish to work on.

References

Last updated: 21st April 2022
First published: 16th August 2018