Difference between revisions of "Linux commands"

From Organic Design wiki
(See also: Add and remove startup items)
m (Resizing JPG's and changing quality setting)
(90 intermediate revisions by 2 users not shown)
Line 1: Line 1:
 
{{info|This is the start of a list of useful Linux shell commands (generally "one liners") we use a lot}}
 
{{info|This is the start of a list of useful Linux shell commands (generally "one liners") we use a lot}}
 +
[[File:Bash Tricks.jpg|800px|right]]
  
  
=== Ubuntu package management ===
+
=== Debian package management ===
 +
The package management system used on [[Debian]] and the "downstream" distros based on it such as [[Ubuntu]] and [[Linux Mint|Mint]].
 
   
 
   
 
==== Searching for an installed package ====
 
==== Searching for an installed package ====
 
Use dpkg and grep;
 
Use dpkg and grep;
{{code|<bash>
+
<source lang="bash">
 
dpkg -l | grep java
 
dpkg -l | grep java
</bash>}}
+
</source>
will list all packages with java in the name
+
will list all installed packages and filter to only those with "java" in the name
  
 
==== Searching for installable packages ====
 
==== Searching for installable packages ====
 
apt-cache  performs a variety of operations on APT´s package cache
 
apt-cache  performs a variety of operations on APT´s package cache
{{code|<bash>
+
<source lang="bash">
 
apt-cache search java
 
apt-cache search java
</bash>}}
+
</source>
 
lists all packages in the cache with java in the name that are installable
 
lists all packages in the cache with java in the name that are installable
  
 
== SSH & SCP ==
 
== SSH & SCP ==
 +
See also [[SSH]] for more specific uses.
  
===tar pipe===
+
=== Screen ===
{{code|<bash>
+
Screen is an essential utility to use during an SSH session over dodgy connections that can drop or when doing something mission-critical. Just use the '''screen''' command without any parameters and another shell session will start (if you ''exit'' you'll be back in the original shell session again). This new shell session will remain active even if the connection drops. To get back into the session again later, use '''screen -R'''. There are many other options for using multiple ''screen'' sessions and multiple windows onto a single session etc, see this [https://tournasdimitrios1.wordpress.com/2010/11/04/linux-the-screen-command-a-must-for-ssh/ screen cheatsheet] for more details.
## From a local to remote machine
+
 
tar -zcvf - directory | ssh remote 'cd somewhere; tar -zxvf -'
+
'''Note:''' One annoying issue with ''screen'' is that the normal scroll-back doesn't work, but you can user ''CTRL+A'' then ''ESC'' to activate "copy mode" which then allows you to scroll back using the cursor keys or page up/down. To exit "copy mode" use ESC again.
## From remote machine to local machine
 
ssh remote 'cd somewhere; tar -zcvf - directory' | tar -zxf -
 
## remote tar and pipe to tgz
 
ssh remote 'cd somewhere; tar -zcf - directory' | cat - > directory.tar.gz
 
</bash>}}
 
  
 +
=== Basic SCP syntax ===
 
Example of sending a file to a remote server:
 
Example of sending a file to a remote server:
{{code|<bash>
+
<source lang="bash">
 
scp -P 2222 /my/local/file.txt USER@example.com:/home/USER
 
scp -P 2222 /my/local/file.txt USER@example.com:/home/USER
</bash>}}
+
</source>
  
 
Example of retrieving a file from a remote server:
 
Example of retrieving a file from a remote server:
{{code|<bash>
+
<source lang="bash">
 
scp -P 2222 USER@example.com:/home/USER/file.txt ./
 
scp -P 2222 USER@example.com:/home/USER/file.txt ./
</bash>}}
+
</source>
 
''Note using port forwarding for commonly accessed sites you can define non standard ports in your  <code>.ssh/config</code> so you never
 
''Note using port forwarding for commonly accessed sites you can define non standard ports in your  <code>.ssh/config</code> so you never
 
need to explicitly state them above.''
 
need to explicitly state them above.''
Line 46: Line 45:
  
 
When this happens, go onto the target machine, check how many bytes were transferred of the target file and rename it to "xaa". Then on the local machine, do the following command to split the source file into two parts, the first being of the size of the number of bytes that have already been transferred (in this example 1234567 bytes).
 
When this happens, go onto the target machine, check how many bytes were transferred of the target file and rename it to "xaa". Then on the local machine, do the following command to split the source file into two parts, the first being of the size of the number of bytes that have already been transferred (in this example 1234567 bytes).
{{code|<bash>split -b 1234567 foo.tar.gz</bash>}}
+
<source lang="bash">
 +
split -b 1234567 foo.tar.gz
 +
</source>
  
 
The resulting files are called "xaa" and "xab", and the latter is the containing the remaining content that needs to be transferred to the target server. Once you've transferred it across, you can then join the two files (remember you renamed the first part to "xaa" so there's no need to transfer that) together using ''cat'' as follows, and then remove all the ''xa*'' files from noth source and target servers.
 
The resulting files are called "xaa" and "xab", and the latter is the containing the remaining content that needs to be transferred to the target server. Once you've transferred it across, you can then join the two files (remember you renamed the first part to "xaa" so there's no need to transfer that) together using ''cat'' as follows, and then remove all the ''xa*'' files from noth source and target servers.
{{code|<bash>cat xa* > foo.tar.gz</bash>}}
+
<source lang="bash">
 +
cat xa* > foo.tar.gz
 +
</source>
 +
 
 +
=== Tar pipes ===
 +
<source lang="bash">
 +
## From a local to remote machine
 +
tar -zcvf - directory | ssh remote 'cd somewhere; tar -zxvf -'
 +
## From remote machine to local machine
 +
ssh remote 'cd somewhere; tar -zcvf - directory' | tar -zxf -
 +
## remote tar and pipe to tgz
 +
ssh remote 'cd somewhere; tar -zcf - directory' | cat - > directory.tar.gz
 +
</source>
  
== Files ==
+
=== Port-forwarding with SSH ===
 +
{{:SSH}}
 +
 
 +
== Files & Devices ==
 +
*[https://www.digitalocean.com/community/tutorials/how-to-partition-and-format-storage-devices-in-linux Partitioning and formatting]
 +
 
 +
=== Clear the contents of a file ===
 +
<source lang="bash">
 +
> /path/to/file
 +
</source>
 +
 
 +
=== List all the storage devices and partitions attached to the system (even if they're unformatted or unmounted) ===
 +
<source lang="bash">
 +
cat /proc/partitions
 +
</source>
 +
 
 +
To check what filesystem a device has on it use ''file'', e.g.
 +
<source lang="bash">
 +
file -s /dev/sda1
 +
</source>
  
 
=== Get the size of a directory and its contents ===
 
=== Get the size of a directory and its contents ===
{{code|<bash>
+
<source lang="bash">
 
du -sh /home/foo
 
du -sh /home/foo
</bash>}}
+
</source>
  
  
 
Use this more specific version to find the size of a users Maildir folder:
 
Use this more specific version to find the size of a users Maildir folder:
{{code|<bash>
+
<source lang="bash">
 
du -sh /home/*/Maildir|sed 's|/home/||'|sed 's|/Maildir||'
 
du -sh /home/*/Maildir|sed 's|/home/||'|sed 's|/Maildir||'
</bash>}}
+
</source>
  
===Search for file content recursively===
 
Here is an example looking for library(.+) or require(.+) within R files recursively
 
  
{{code|<bash>
+
Here's another version of ''du'' that accounts for dotfiles and sorts the results:
## Printing all matches
+
<source lang="bash">
find . -name "*.R" -print | xargs perl -ne 'm/(library|require)\(\"?(.+?)\"?\)/ && print "$2\n"'
+
du -sch .[!.]* * | sort -h
## Building a hash table of unique matches
+
</source>
find . -name "*.R" -print | xargs perl -ne 'm/(library|require)\(\"?(\w+?)\"?\)/ && $x{$2}++; END{print "@{[keys(%x)]}\n"}'
 
</bash>}}
 
  
If you have spaces or newlines in file names, then you have to use -print0 option instead of -print and xargs -null so that the list of file names are exchanged with null-terminated strings.
+
=== Search for file content recursively ===
 +
Here's an example looking for a phrase within a specific file type recursively through a directory structure, and printing the file names and line numbers of the resulting matches.
  
{{code|<bash>
+
<source lang="bash">
## Printing all matches
+
grep -rn "Foo" *.php
find . -name "*.R" -print0 | xargs -0 perl -ne 'm/(library|require)\(\"?(.+?)\"?\)/ && print "$2\n"'
+
</source>
## Building a hash table of unique matches
 
find . -name "*.R" -print0 | xargs -0 perl -ne 'm/(library|require)\(\"?(\w+?)\"?\)/ && $x{$2}++; END{print "@{[keys(%x)]}\n"}'
 
</bash>}}
 
  
 
There are other tips at [http://stackoverflow.com/questions/316590/how-do-you-use-cat-recursively stackoverflow.com].
 
There are other tips at [http://stackoverflow.com/questions/316590/how-do-you-use-cat-recursively stackoverflow.com].
  
===Search for file content recursively and tar===
+
=== Search for file content recursively and tar ===
{{code|<bash>
+
<source lang="bash">
 
## Find and tar
 
## Find and tar
 
find . -name "*.R" -print0 | xargs -0 tar -cvf Rfiles.tar
 
find . -name "*.R" -print0 | xargs -0 tar -cvf Rfiles.tar
Line 96: Line 122:
 
## check contents
 
## check contents
 
tar -ztvf Rfiles.tar.gz
 
tar -ztvf Rfiles.tar.gz
</bash>}}
+
</source>
  
===Search for file content recursively and long list===  
+
=== Search for file by name recursively ===  
{{code|<bash>
+
<source lang="bash">
## Printing all matches
+
find . -name "*.R"
find . -name "*.R" -ls
+
</source>
</bash>}}
+
 
 +
=== Count occurrences of a word in a file or files ===
 +
<source lang="bash">
 +
grep -roh "WORD" file*.txt | wc -
 +
</source>
 +
 
 +
=== Count the number of lines in a file or output ===
 +
<source lang="bash">
 +
wc -l <FILE>
 +
cat <FILE> grep foo | wc -l
 +
</source>
  
 
=== Search and replace content in files ===
 
=== Search and replace content in files ===
You could also use find and sed, but I find that this little line of perl works nicely.
+
You could also use find and sed, but I find that this little line of perl works nicely.
{{code|<bash>
+
<source lang="bash">
 
perl -pi -w -e 's/SEARCH/REPLACE/g;' *.php
 
perl -pi -w -e 's/SEARCH/REPLACE/g;' *.php
</bash>}}
+
</source>
 
*-e means execute the following line of code.
 
*-e means execute the following line of code.
 
*-i means edit in-place
 
*-i means edit in-place
Line 114: Line 150:
 
*-p loop
 
*-p loop
 
EXTS="7z t7z"
 
EXTS="7z t7z"
 +
 +
=== Search for files that contain the windows BOM character (&amp;#65279;) ===
 +
Windows editors often add a BOM character to the beginning of UTF-8 encoded text files, these cause a lot of trouble for many types of scripts such as PHP and the problem can be very hard to track down. This short shell one-liner I found [http://stackoverflow.com/questions/204765/elegant-way-to-search-for-utf-8-files-with-bom here] searches the current directory recursively showing what files have a BOM, you can then use a decent editor like [[Geany]] to remove them.
 +
<source lang="bash">
 +
grep -rlI $'\xEF\xBB\xBF' .
 +
</source>
  
 
=== Mass renaming ===
 
=== Mass renaming ===
Line 119: Line 161:
  
 
;Example:
 
;Example:
{{code|
+
{{code|mmv \*.JPG \#1.jpg }}
mmv \*.JPG \#1.jpg  
 
}}
 
  
 
The first pattern matches anything with a “.JPG” and renames each file (the “#1” matches the first wildcard) to “.jpg”. Each time you use a \(wildcard) you can use a #x to get that wildcard. Where x is a positive number starting at 1. Copied off: http://tldp.org/LDP/GNU-Linux-Tools-Summary/html/mass-rename.html
 
The first pattern matches anything with a “.JPG” and renames each file (the “#1” matches the first wildcard) to “.jpg”. Each time you use a \(wildcard) you can use a #x to get that wildcard. Where x is a positive number starting at 1. Copied off: http://tldp.org/LDP/GNU-Linux-Tools-Summary/html/mass-rename.html
 +
 +
=== Compare two directory structures for differences ===
 +
This is very handy if you need to know if two directory structures are the same or not including file content. It uses the ''diff'' command with the ''r'' switch to work recursively and the ''x'' switch to allow skipping of the ''.svn'' directories.
 +
<source lang="bash">
 +
diff -qrx .svn DIR1 DIR2
 +
</source>
 +
 +
=== Mount a .iso and .img files ===
 +
See [http://www.cyberciti.biz/tips/how-to-mount-iso-image-under-linux.html this] for ISO files and [https://askubuntu.com/questions/236263/browse-img-without-mounting this] for IMG files.
 +
 +
=== Mount a USB stick ===
 +
If you don't know the device name of the stick, plug it in and find it with the following:
 +
<source lang="bash">
 +
dmesg |grep SCSI
 +
</source>
 +
 +
Then mount the first partition,
 +
<source lang="bash">
 +
mkdir -p ~/memstick
 +
mount -t vfat -o rw,users /dev/sdX1 ~/memstick
 +
</source>
 +
 +
=== Clone a device ===
 +
Remember that this is a very low-level operation and so if you're cloning a live partition you may end up with partially written files or other corruption issues on the destination. Stop as many services as you can first, do an ''fsck'' afterwards and manually copy any directories that had issues.
 +
<source>dd if=/dev/sda of=/dev/sdb bs=32M</source>
 +
 +
A good way to find more information about the devices first, for example if you have a number of unformatted devices of the same size and need more specific information about the hardware, is to use the '''lsscsi'' command which gives the following sample output:
 +
<source>
 +
[0:0:0:0]    disk    AMCC    9650SE-2LP DISK  4.10  /dev/sdb
 +
[1:0:0:0]    disk    ATA      Hitachi HUA72201 JP4O  /dev/sda
 +
[2:0:0:0]    disk    ATA      TOSHIBA MG03ACA1 FL1A  /dev/sdc
 +
</source>
 +
The ''sdb'' and ''sdc'' devices look identical with commands such as ''lsblk'' or ''fdisk -l'', but one of the is a RAID pair and the other just a normal drive.
 +
 +
'''See also:''' [[Clone a live partition over SSH]]
 +
 +
=== Changing device UUIDs ===
 +
It may sometimes be useful to change the UUIDs of partitions such as when they've been cloned with a low-level method like ''dd''.
 +
 +
'''Note:''' the disk identifier should also be changed which you can do with ''fdisk'' from the e'''x'''pert menu.
 +
 +
Check what devices and UUIDs you have in the system:
 +
<source>
 +
# blkid
 +
/dev/sda1: UUID="bc459b13-3aad-4017-a68b-ce8ab36275e1" SEC_TYPE="ext2" TYPE="ext3"
 +
/dev/sda2: UUID="86ce60b7-ad4a-44aa-88e2-2aefd5c6c396" TYPE="swap"
 +
/dev/sda3: UUID="b56d82f7-5817-4a52-9763-0b38aa360e2b" TYPE="ext4"
 +
/dev/sdc1: UUID="81d40a02-8019-4c1f-afb8-fb41d117c6d1" TYPE="ext3"
 +
/dev/sdc2: UUID="c57f1400-129a-402a-90f1-820a22c6a2fe" TYPE="swap"
 +
/dev/sdc3: UUID="c23ab65a-c32f-41e3-bd31-51ed563e0099" TYPE="ext4"
 +
</source>
 +
 +
or:
 +
<source>
 +
# ls -l /dev/disk/by-uuid/
 +
total 0
 +
lrwxrwxrwx 1 root root 10 Sep  2 12:10 81d40a02-8019-4c1f-afb8-fb41d117c6d1 -> ../../sdc1
 +
lrwxrwxrwx 1 root root 10 Sep  2 12:10 86ce60b7-ad4a-44aa-88e2-2aefd5c6c396 -> ../../sda2
 +
lrwxrwxrwx 1 root root 10 Sep  2 12:10 b56d82f7-5817-4a52-9763-0b38aa360e2b -> ../../sda3
 +
lrwxrwxrwx 1 root root 10 Sep  2 12:10 bc459b13-3aad-4017-a68b-ce8ab36275e1 -> ../../sda1
 +
lrwxrwxrwx 1 root root 10 Sep  2 12:10 c23ab65a-c32f-41e3-bd31-51ed563e0099 -> ../../sdc3
 +
lrwxrwxrwx 1 root root 10 Sep  2 12:27 c57f1400-129a-402a-90f1-820a22c6a2fe -> ../../sdc2
 +
</source>
 +
 +
Check what swap partitions you have:
 +
<source>
 +
# cat /proc/swaps
 +
Filename Type Size Used Priority
 +
/dev/sda2                              partition 3999740 0 -1
 +
</source>
 +
 +
Create a new UUID and assign it to and ext parition:
 +
<source>
 +
# uuidgen
 +
81d40a02-8019-4c1f-afb8-fb41d117c6d1
 +
#tune2fs /dev/sdb1 -U 81d40a02-8019-4c1f-afb8-fb41d117c6d1
 +
</source>
 +
 +
Do the same for a swap partition:
 +
<source>
 +
# uuidgen
 +
86ce60b7-ad4a-44aa-88e2-2aefd5c6c396
 +
# swapoff /dev/sda2
 +
# mkswap -U 86ce60b7-ad4a-44aa-88e2-2aefd5c6c396 /dev/sda2
 +
Setting up swapspace version 1, size = 3999740 KiB
 +
no label, UUID=86ce60b7-ad4a-44aa-88e2-2aefd5c6c396
 +
</source>
 +
 +
=== Updating device partition tables without rebooting ===
 +
*partprobe /dev/sdx
 +
*partx -u /dev/sdx
 +
*echo 1 > /sys/block/sdX/device/rescan
 +
*check partition information before and after with ''cat /proc/partitions''
 +
 +
=== Downloading from google drive in shell ===
 +
Thanks to [https://unix.stackexchange.com/questions/136371/how-to-download-a-folder-from-google-drive-using-terminal#148674 FrankerZ] for this! The first command outputs a code which you put into the second command.
 +
<source lang="bash">
 +
wget --save-cookies cookies.txt --keep-session-cookies --no-check-certificate 'https://docs.google.com/uc?export=download&id={!FILEID!}' -O- | sed -rn 's/.*confirm=([0-9A-Za-z_]+).*/Code: \1\n/p'
 +
</source>
 +
 +
<source lang="bash">
 +
wget --load-cookies cookies.txt 'https://docs.google.com/uc?export=download&confirm={!CODE_FROM_ABOVE!}&id={!FILEID!}' -O {!FILENAME!}
 +
</source>
  
 
== Image manipulation ==
 
== Image manipulation ==
  
 
=== Resizing JPG's and changing quality setting ===
 
=== Resizing JPG's and changing quality setting ===
The first line shows how to reduce and image to 25% and quality to 50% adding "_resized" to the results filename. The second command uses Perl to apply this same command to all JPG's in the current directory.
+
The first line shows how to reduce and image to 25% and quality to 50% adding "_resized" to the results filename.
{{code|<bash>
+
<source lang="bash">
convert foo.jpg -resize 25% -quality 50% foo_resized.jpg
+
convert foo.jpg -resize 25% -quality 75% foo_resized.jpg
 +
</source>
  
perl -e 'for (glob "*.jpg") { $img = $_; s/(....)$/_resized$1/; qx "convert \'$img\' -resize 25% -quality 50% \'$_\'"; }'
+
Here's a Perl script version used to apply this same command to all JPG's (jpg, JPG, jpeg etc) in the current directory.
</bash>}}
+
<source lang="perl">
 +
#!/usr/bin/perl
 +
use File::Glob qw(:globally :nocase);
 +
for (glob "*.jp*g") {
 +
my $img = $_;
 +
s/(....)$/_resized$1/;
 +
qx "convert \'$img\' -resize 25% -quality 75% \'$_\'";
 +
}
 +
</source>
 +
 
 +
This command rotates, normalises, greyscales and sets the output quality of set of an image:
 +
<source lang="bash">
 +
convert FILE  -normalize -set colorspace Gray -separate -average -resize 50% -rotate "-90" -quality 75% OUT.JPG
 +
</source>
  
 
=== Apply an opaque background of a specified colour to a directory of transparent PNG's ===
 
=== Apply an opaque background of a specified colour to a directory of transparent PNG's ===
 
*This command requires ImageMagick to be installed
 
*This command requires ImageMagick to be installed
 
*It loops through all PNG's in the CWD and puts them in a directory called ''processed'' which must exist
 
*It loops through all PNG's in the CWD and puts them in a directory called ''processed'' which must exist
{{code|<bash>
+
<source lang="bash">
 
perl -e 'qx "convert $_ -background #ff00ff -flatten foo/$_" for glob "*.png"'
 
perl -e 'qx "convert $_ -background #ff00ff -flatten foo/$_" for glob "*.png"'
</bash>}}
+
</source>
 +
 
 +
== Audio & Video ==
 +
 
 +
=== avconv (ffmpeg) ===
 +
'''Note:''' Many operating systems including Ubuntu are now using the [http://libav.org/ libav] fork of ''ffmpeg'' now which means you should use the '''avconv''' command instead.
  
== Audio conversion ==
 
 
The following converts a ''.wav'' file to an ''mp3'':
 
The following converts a ''.wav'' file to an ''mp3'':
{{code|<bash>
+
<source lang="bash">
ffmpeg -i test.wav test.mp3
+
ffmpeg -i foo.wav foo.mp3
</bash>}}
+
</source>
 +
 
 +
Here's a more complicated example which converts ''mp4'' video into ''mp3'' audio at a lower quality with only one channel at 11KHz and 32kbps.
 +
<source lang="bash">
 +
ffmpeg -i foo.mp4 -acodec libmp3lame -ab 16k -ar 11025 -ac 1 foo.mp3
 +
</source>
  
 
To do a whole directory you could do this:
 
To do a whole directory you could do this:
{{code|<bash>
+
<source lang="bash">
 
perl -e 'qx "ffmpeg -i $_ $_.mp3" for glob "*.wav"'
 
perl -e 'qx "ffmpeg -i $_ $_.mp3" for glob "*.wav"'
</bash>}}
+
</source>
 +
 
 +
Or a little more complex; a whole directory with a proper name change accounting for names with spaces in them. This
 +
<source lang="bash">
 +
perl -e 'for (glob "*.mp4") { $i=$_; s/.mp3$/.converted.wav/; qx "ffmpeg -i \"$i\" \"$_\"" }'
 +
</source>
  
== Video manipulation ==
+
Use the following commands to extract a small (5 seconds in this example) snippet out of a video (the -ar switch is only needed for outputting to flv I think).
Use the following commands to extract a small snippet out of a video (the -ar switch is only needed for outputting to flv I think).
+
<source lang="bash">
{{code|<bash>
+
ffmpeg -i "foo.avi" -ss 00:10:10 -t 5 -ar 22050 "foo.flv"
ffmpeg -i "foo.avi" -ss 00:10:10 -t 00:00:05 -ar 22050 "foo.flv"
+
</source>
</bash>}}
 
  
=== See also ===
+
'''See also:'''
*[[Converting_microarray_images]] for bash/imagemagick image file manipulation
 
  
== System monitoring ==
+
*[[Simple video streaming with ffserver]]
 +
*[[Converting microarray images]] for bash/imagemagick image file manipulation
 +
*[http://tipsonubuntu.com/2016/11/02/install-ffmpeg-3-2-via-ppa-ubuntu-16-04/ How to install ffmpeg 3.2 in Ubuntu 16]
 +
 
 +
=== Downloading Youtube from shell ===
 +
First install the Python [http://rg3.github.io/youtube-dl/download.html youtube-dl] tool, with '''pip''':
 +
<source lang="bash">
 +
sudo pip install --upgrade youtube_dl
 +
</source>
 +
 
 +
Then use the tool to download the video, e.g.
 +
<source lang="bash">
 +
youtube-dl https://www.youtube.com/watch?v=f9m2yReECak
 +
</source>
 +
 
 +
== System monitoring & management ==
 
List the top 10 memory consuming processes
 
List the top 10 memory consuming processes
{{code|<bash>ps -auxf | sort -nr -k 4 | head -10</bash>}}
+
<source lang="bash">
 +
ps -auxf | sort -nr -k 4 | head -10
 +
</source>
  
  
 
List the top 10 CPU consuming processes
 
List the top 10 CPU consuming processes
{{code|<bash>ps -auxf | sort -nr -k 3 | head -10</bash>}}
+
<source lang="bash">
 +
ps -auxf | sort -nr -k 3 | head -10
 +
</source>
 +
 
 +
 
 +
Limit CPU resources to a process (you may need to first install the '''cpulimit''' package):
 +
<source lang="bash">
 +
cpulimit -b -l <PERCENTAGE> -p <PID>
 +
</source>
  
 
== Network commands ==
 
== Network commands ==
  
 
=== Restart the network after changing configuration ===
 
=== Restart the network after changing configuration ===
{{code|<bash>/etc/init.d/networking restart</bash>}}
+
<source lang="bash">
 +
/etc/init.d/networking restart
 +
</source>
  
 
=== List all the listening sockets and their ports and programs ===
 
=== List all the listening sockets and their ports and programs ===
{{code|<bash>netstat -lp</bash>}}
+
<source lang="bash">
 +
netstat -lp
 +
</source>
  
 
=== Get current default gateway ===
 
=== Get current default gateway ===
{{code|<bash>netstat -nr</bash>}}
+
<source lang="bash">
 +
netstat -nr
 +
</source>
 
The default gateway is on the last line, it should have the U and G flags set
 
The default gateway is on the last line, it should have the U and G flags set
  
 
=== Release DHCP lease ===
 
=== Release DHCP lease ===
{{code|<bash>dhclient -r</bash>}}
+
<source lang="bash">
 +
dhclient -r
 +
</source>
  
 
=== Obtain a new DHCP lease ===
 
=== Obtain a new DHCP lease ===
{{code|<bash>dhclient</bash>}}
+
<source lang="bash">
 +
dhclient
 +
</source>
  
 
=== Scan a local subnet for active IP addresses ===
 
=== Scan a local subnet for active IP addresses ===
{{code|<bash>nmap -sP 192.168.1.0/24</bash>}}
+
<source lang="bash">
 +
sudo nmap -sP 192.168.1.0/24
 +
</source>
  
 
=== Get the MAC address and hostname of an IP on the current subnet ===
 
=== Get the MAC address and hostname of an IP on the current subnet ===
{{code|<bash>arp -a 192.168.1.1</bash>}}
+
<source lang="bash">
 +
arp -a 192.168.1.1
 +
</source>
 +
 
 +
=== Display traffic passing through a port ===
 +
This is very useful for debugging interactions with services, for example the following command shows traffic going between a local ''elasticsearch'' service and a local client application.
 +
<source lang="bash">
 +
sudo tcpdump port 9200 -qAi lo
 +
</source>
 +
 
 +
=== Make the system prefer IPv4 connections ===
 +
Open '''/etc/gai.conf''' and uncomment the following line:
 +
<source>
 +
#
 +
# For sites which prefer IPv4 connections change the last line to
 +
#
 +
precedence ::ffff:0:0/96 100
 +
</source>
 +
 
 +
=== Test which IPv6 address is being used for public connections ===
 +
Servers can be assigned many IPv6 addresses, but which one should you assign to ''AAAA'' records pointing at the server? This command shows which address is being used to make outgoing connections which would be the best one to use for incoming connections as well. The address used to test the outgoing connection here is google's public DNS server.
 +
<source lang="bash">
 +
ip r get to 2001:4860:4860::8888
 +
</source>
 +
*'''See also:''' [https://bgp.he.net/ Assigned IPv6 address ranges] and [http://tldp.org/HOWTO/html_single/Linux+IPv6-HOWTO/ IPv6 HOWTO]
  
 
== Internet ==
 
== Internet ==
 +
 +
=== Download an url to a local file with continuation ===
 +
<source lang="bash">
 +
curl -L -O -C - http://www.foo.bar/file
 +
</source>
 +
 +
<source lang="bash">
 +
wget -c http://www.foo.bar/file
 +
</source>
 +
  
 
=== Download a copy of an entire site ===
 
=== Download a copy of an entire site ===
 
This will download an entire site, if the site is already downloaded, then only newer files are transferred.
 
This will download an entire site, if the site is already downloaded, then only newer files are transferred.
{{code|<pre>
+
<source>
 
wget -m http://foo.bar
 
wget -m http://foo.bar
</pre>}}
+
</source>
 
 
==Devices==
 
===mount===
 
All files accessible in a Unix system are arranged in one big tree, the file hierarchy, rooted at /.  These files  can  be  spread  out over  several  devices. The mount command serves to attach the file system found on some device to the big file tree.
 
{{code|<bash>
 
mount [/dev/device] [/media/directory]
 
</bash>}}
 
 
 
===fstab===
 
Static information about the filesystems ('''fstab''') is a configuration file that contains information of all the partitions and storage devices in your computer. It is a map of devices to the point in the filesytem where the device can be accessed, it contains information of where your partitions and storage devices should be mounted and how. It acts as a set of defaults for devices that are specified using the mount command where the file system directory is not specified. See [[W:Fstab#Example]].
 
 
 
{{code|<bash>
 
cat /etc/fstab
 
</bash>}}
 
 
 
=== Mount a .iso ===
 
See [http://www.cyberciti.biz/tips/how-to-mount-iso-image-under-linux.html this HOWTO]
 
  
 
== Port forwarding ==
 
== Port forwarding ==
 
Port forwarding allows a remote client to gain access to a network so an intranet can be accessed.
 
Port forwarding allows a remote client to gain access to a network so an intranet can be accessed.
{{code|<bash>
+
<source lang="bash">
ssh -fN -L[PORT]:appserver:[PORT] username@sshdserver
+
ssh -fnNTL[PORT]:appserver:[PORT] username@sshdserver
</bash>}}
+
</source>
  
 
Then point your webbrowsers proxy server to appserver:[PORT] and access the intranet etc.
 
Then point your webbrowsers proxy server to appserver:[PORT] and access the intranet etc.
  
 
To subvert a firewalled environment where outgoing ssh is allowed.
 
To subvert a firewalled environment where outgoing ssh is allowed.
{{code|<bash>
+
<source lang="bash">
 
$ ssh -D 9000 username@remotehost
 
$ ssh -D 9000 username@remotehost
</bash>}}
+
</source>
  
 
Then point your web browser to a SOCKS proxy @ localhost:9000
 
Then point your web browser to a SOCKS proxy @ localhost:9000
  
 
Further it's possible to get ssh through a web proxy using [http://www.agroman.net/corkscrew/ corkscrew].
 
Further it's possible to get ssh through a web proxy using [http://www.agroman.net/corkscrew/ corkscrew].
{{code|<bash>
+
<source lang="bash">
 
ssh -oProxyCommand='corkscrew local_web_proxy proxy_port %h %p' username@remotehost
 
ssh -oProxyCommand='corkscrew local_web_proxy proxy_port %h %p' username@remotehost
</bash>}}
+
</source>
  
 
*[http://en.wikipedia.org/wiki/Port_forwarding Port_forwarding (Wikipedia)]
 
*[http://en.wikipedia.org/wiki/Port_forwarding Port_forwarding (Wikipedia)]
 
*[http://www.securityfocus.com/infocus/1816 SSH Port Forwarding article]
 
*[http://www.securityfocus.com/infocus/1816 SSH Port Forwarding article]
 
*[http://www.ssh.com/support/documentation/online/ssh/adminguide/32/Port_Forwarding.html Port Forwarding (ssh.com)]
 
*[http://www.ssh.com/support/documentation/online/ssh/adminguide/32/Port_Forwarding.html Port Forwarding (ssh.com)]
 +
 +
=== Monitoring network traffic ===
 +
You can get daily and monthly traffic statistics that are updated every five minutes by installing the ''vnstat'' package. This is very useful if your ISP enforces a traffic limit. You can get reports on individual network interfaces.
 +
 +
Enable monitoring on a specific interface as follows:
 +
<source lang="bash">
 +
vnstat --enable -i eth0
 +
</source>
 +
 +
And then you can get a report for an interface as follows:
 +
<source lang="bash">
 +
vnstat -i eth0
 +
</source>
 +
 +
Example output:
 +
<source>
 +
  monthly
 +
                    rx      |    tx      |    total    |  avg. rate
 +
    ------------------------+-------------+-------------+---------------
 +
      Nov '17    82.43 MiB |  15.97 MiB |  98.40 MiB |    0.35 kbit/s
 +
    ------------------------+-------------+-------------+---------------
 +
    estimated        91 MiB |      16 MiB |    107 MiB |
 +
 +
  daily
 +
                    rx      |    tx      |    total    |  avg. rate
 +
    ------------------------+-------------+-------------+---------------
 +
        today    82.43 MiB |  15.97 MiB |  98.40 MiB |  10.67 kbit/s
 +
    ------------------------+-------------+-------------+---------------
 +
    estimated        93 MiB |      17 MiB |    110 MiB |
 +
</source>
  
 
== General ==
 
== General ==
 
=== Add and remove startup items ===
 
=== Add and remove startup items ===
{{code|<bash> update-rc.d ITEM defaults </bash>}}
+
<source lang="bash">
 +
update-rc.d ITEM defaults
 +
</source>
 +
 
 +
<source lang="bash">
 +
update-rc.d -f ITEM remove
 +
</source>
 +
 
 +
=== Date and time ===
 +
Get the current (or another time specified in ''--date'') time in UNIX timestamp format:
 +
<source lang="bash">
 +
date +%s
 +
date --date='01/30/2012 01:23:45' +%s
 +
</source>
 +
 
 +
 
 +
Convert a UNIX timestamp into a date/time in the current locale format:
 +
<source lang="bash">
 +
date -d @1282368345
 +
</source>
  
{{code|<bash>update-rc.d -f ITEM remove</bash>}}
+
== Specific guides ==
 +
*[http://www.linuxstall.com/fstab/ All about fstab]
 +
*[https://www.networkworld.com/article/2693414/setting-limits-with-ulimit.html All about ulimit]
 +
*[https://www.cyberciti.biz/faq/linux-add-a-swap-file-howto/ Swap file HOWTO]
 +
*[http://www.folkstalk.com/2012/07/xargs-command-examples-in-unix-linux.html Using xargs]
 +
*[http://pubs.opengroup.org/onlinepubs/9699919799/utilities/date.html Date format codes]
 +
*[https://www.tecmint.com/customize-bash-colors-terminal-prompt-linux/ Setting shell colours]
 +
*[https://haydenjames.io/linux-networking-commands-scripts/ 59 networking commands and scripts]
 +
*[http://www.cyberciti.biz/tips/top-linux-monitoring-tools.html Top Linux monitoring commands]
 +
*[https://www.howtogeek.com/howto/30184/10-ways-to-generate-a-random-password-from-the-command-line/ 10 ways to make random passwords from the command line]
 +
*[https://www.2daygeek.com/efficient-ways-to-read-the-log-files-in-linux/ Efficient Ways To Read The Log Files In Linux]
 +
*[https://help.ubuntu.com/community/UbuntuBonding Bonding multiple network connections into one] ''- see also [http://www.linuxfoundation.org/collaborate/workgroups/networking/bonding#Frequently_Asked_Questions this] for more detail''
 +
*[http://www.terminalinflection.com/manual-grub-boot/ Manually booting from the GRUB prompt]
 +
*[https://www.digitalocean.com/community/tutorials/how-to-partition-and-format-storage-devices-in-linux Digital Ocean tutorial on formatting drives]
 +
*[https://www.howtoforge.com/converting_rpm_to_deb_with_alien Convert an RPM to a DEB with alien]
 +
*[https://docs.01.org/clearlinux/latest/guides/maintenance/cpu-performance.html CPU power & performance adjustment]
  
 
== See also ==
 
== See also ==
Line 253: Line 540:
 
*[http://files.fosswire.com/2008/04/ubunturef.pdf Ubuntu commands cheat sheet]
 
*[http://files.fosswire.com/2008/04/ubunturef.pdf Ubuntu commands cheat sheet]
 
*[https://help.ubuntu.com/community/UsingTheTerminal Using The Terminal] ''- from Ubuntu help''
 
*[https://help.ubuntu.com/community/UsingTheTerminal Using The Terminal] ''- from Ubuntu help''
*[http://www.sit.auckland.ac.nz/Category:Unix UoA Faculty of Science unix docs]
 
*[http://www.cyberciti.biz/tips/top-linux-monitoring-tools.html Top Linux monitoring commands]
 
 
*[http://www.ee.surrey.ac.uk/Teaching/Unix/ Basic Linux Tutorial]
 
*[http://www.ee.surrey.ac.uk/Teaching/Unix/ Basic Linux Tutorial]
 +
*[https://debian-handbook.info/ The Debian Administrator's Handbook]
 
*[http://www.cheatography.com/davechild/cheat-sheets/ Another excellent cheatsheet site]
 
*[http://www.cheatography.com/davechild/cheat-sheets/ Another excellent cheatsheet site]
*[http://unixhelp.ed.ac.uk/CGI/man-cgi?date Date format codes]
+
*[http://www.coker.com.au/bonnie++/ bonnie++] ''- easy to use disk benchmarking tool (good output explanation [http://www.jamescoyle.net/how-to/599-benchmark-disk-io-with-dd-and-bonnie here])
 
[[Category:Linux]][[Category:Help]]
 
[[Category:Linux]][[Category:Help]]

Revision as of 23:42, 15 December 2020

Info.svg This is the start of a list of useful Linux shell commands (generally "one liners") we use a lot
Bash Tricks.jpg


Contents

Debian package management

The package management system used on Debian and the "downstream" distros based on it such as Ubuntu and Mint.

Searching for an installed package

Use dpkg and grep;

dpkg -l | grep java

will list all installed packages and filter to only those with "java" in the name

Searching for installable packages

apt-cache performs a variety of operations on APT´s package cache

apt-cache search java

lists all packages in the cache with java in the name that are installable

SSH & SCP

See also SSH for more specific uses.

Screen

Screen is an essential utility to use during an SSH session over dodgy connections that can drop or when doing something mission-critical. Just use the screen command without any parameters and another shell session will start (if you exit you'll be back in the original shell session again). This new shell session will remain active even if the connection drops. To get back into the session again later, use screen -R. There are many other options for using multiple screen sessions and multiple windows onto a single session etc, see this screen cheatsheet for more details.

Note: One annoying issue with screen is that the normal scroll-back doesn't work, but you can user CTRL+A then ESC to activate "copy mode" which then allows you to scroll back using the cursor keys or page up/down. To exit "copy mode" use ESC again.

Basic SCP syntax

Example of sending a file to a remote server:

scp -P 2222 /my/local/file.txt USER@example.com:/home/USER

Example of retrieving a file from a remote server:

scp -P 2222 USER@example.com:/home/USER/file.txt ./

Note using port forwarding for commonly accessed sites you can define non standard ports in your .ssh/config so you never need to explicitly state them above.

Recovering from an interrupted transfer

This ability is included natively in Linux and used to be very useful for splitting large backups up so they could fit onto small media such as floppy disks. But there's one time when it's very useful nowadays too which is when a large file transfer gets interupted and there's no option for continuation such as when using SCP.

When this happens, go onto the target machine, check how many bytes were transferred of the target file and rename it to "xaa". Then on the local machine, do the following command to split the source file into two parts, the first being of the size of the number of bytes that have already been transferred (in this example 1234567 bytes).

split -b 1234567 foo.tar.gz

The resulting files are called "xaa" and "xab", and the latter is the containing the remaining content that needs to be transferred to the target server. Once you've transferred it across, you can then join the two files (remember you renamed the first part to "xaa" so there's no need to transfer that) together using cat as follows, and then remove all the xa* files from noth source and target servers.

cat xa* > foo.tar.gz

Tar pipes

## From a local to remote machine
tar -zcvf - directory | ssh remote 'cd somewhere; tar -zxvf -'
## From remote machine to local machine
ssh remote 'cd somewhere; tar -zcvf - directory' | tar -zxf -
## remote tar and pipe to tgz
ssh remote 'cd somewhere; tar -zcf - directory' | cat - > directory.tar.gz

Port-forwarding with SSH

SSH allows you to forward ports on one machine to another using the SSH pipes.

For example, using a local port-forward, you could create a local connection into a remote SQL server that's not available to external connections if you have SSH access to machine with running the SQL server:

ssh -nNTL 3333:localhost:3306 USER@SERVER

Now you can access the database locally on port 3333.

Or in the opposite direction you can use a remote port-forward you could give SSH access to a machine that's behind a firewall or router by putting its SSH port onto a remote machine:

ssh -nNTR 2222:localhost:22 USER@SERVER

You can now ssh into the local machine from within the remote server on port 2222 instead.

Note that by default the forwarded port is only accessible locally, but you can change this behaviour with the GatewayPorts SSH option.

Lets say you're doing some maintenance on your site and you'd like to redirect all the requests on port 80 to another web-server (SERVER) on port 8080. You could set up your temporary web-server on 8080 on a remote server and then after you've stopped the local web-server, forward all requests for port 80 to 8080 on the remote machine with a local port-forward. By using the GatewayPorts option, the forwarded port 80 is available externally.

ssh -nNTL 80:SERVER:8080 -o GatewayPorts=yes USER@SERVER

This option can be useful for remote forwards too, for example what if we wanted our machine behind the firewall that we want accessible via SSH to be accessible externally? In this case we need to set the GatewayPorts option in sshd_config because it applies to the remote machine, not to the one running the command. If it's set to yes then the forwarded ports will be available externally as well, or you can set it to clientspecified to restrict the access to an external IP defined in the command:

ssh -nNTR 1.2.3.4:2222:localhost:22 USER@SERVER

Now the host at 1.2.3.4 is able to SSH into our firewalled machine on port 2222 of USER@SERVER. See also this post regarding security for port-forwarding accounts.

Browsing the net through a remote server with SSH

Sometimes you need to browse using an IP address that's in another location, for example if the content you want to access is only available to local users or if you're buying something and the prices are based on the buyers location. If you have access to a server in the required location, or someone you know in that location is willing to temporarily set up an SSH server that you can access, the you can use the following syntax to set up a local port that your browser can use as a proxy server.

ssh -fnNTCD 1080 USER@SERVER
  • The f option means to Fork off into the background after successful connection
  • The nNT options mean to use this SSH session only for tunnelling, prevent reading STDIN, no pseudo terminal and not to use open up a shell session.
  • The C option means to compress the data in the tunnel which is a good idea if you're on a slow connection
  • The D option is the main one that tells SSH to set up a tunnel connected to a local port with the port number specified

Next you need to configure your browser to connect via this local port. Chromium allows you to specify the proxy details as a command-line option, so there's no need to change the network configuration and then change it back afterwards. Simply open a shell window and use the following syntax (make sure there are no other Chromium windows open when you do this).

chromium-browser --proxy-server="socks5://localhost:1080"

In Firefox you need to change the Network Proxy setting in Preferences/General. The changes take effect instantly without needing a restart or anything, but you'll need to remember to change the setting back after the SSH connection is closed.

FF-proxy.jpg


Files & Devices

Clear the contents of a file

> /path/to/file

List all the storage devices and partitions attached to the system (even if they're unformatted or unmounted)

cat /proc/partitions

To check what filesystem a device has on it use file, e.g.

file -s /dev/sda1

Get the size of a directory and its contents

du -sh /home/foo


Use this more specific version to find the size of a users Maildir folder:

du -sh /home/*/Maildir|sed 's|/home/||'|sed 's|/Maildir||'


Here's another version of du that accounts for dotfiles and sorts the results:

du -sch .[!.]* * | sort -h

Search for file content recursively

Here's an example looking for a phrase within a specific file type recursively through a directory structure, and printing the file names and line numbers of the resulting matches.

grep -rn "Foo" *.php

There are other tips at stackoverflow.com.

Search for file content recursively and tar

## Find and tar
find . -name "*.R" -print0 | xargs -0 tar -cvf Rfiles.tar
## check contents
tar -tvf Rfiles.tar

## Find and tar.gz
find . -name "*.R" -print0 | xargs -0 tar -zcvf Rfiles.tar.gz
## check contents
tar -ztvf Rfiles.tar.gz

Search for file by name recursively

find . -name "*.R"

Count occurrences of a word in a file or files

grep -roh "WORD" file*.txt | wc -

Count the number of lines in a file or output

wc -l <FILE>
cat <FILE> grep foo | wc -l

Search and replace content in files

You could also use find and sed, but I find that this little line of perl works nicely.

perl -pi -w -e 's/SEARCH/REPLACE/g;' *.php
  • -e means execute the following line of code.
  • -i means edit in-place
  • -w write warnings
  • -p loop

EXTS="7z t7z"

Search for files that contain the windows BOM character (&#65279;)

Windows editors often add a BOM character to the beginning of UTF-8 encoded text files, these cause a lot of trouble for many types of scripts such as PHP and the problem can be very hard to track down. This short shell one-liner I found here searches the current directory recursively showing what files have a BOM, you can then use a decent editor like Geany to remove them.

grep -rlI $'\xEF\xBB\xBF' .

Mass renaming

mmv is a mass move/copy/renaming tool that uses standard wildcards to perform its functions. According to the manual the “;” wildcard is useful for matching files at any depth in the directory tree (ie it will go below the current directory, recursively).

Example
mmv \*.JPG \#1.jpg

The first pattern matches anything with a “.JPG” and renames each file (the “#1” matches the first wildcard) to “.jpg”. Each time you use a \(wildcard) you can use a #x to get that wildcard. Where x is a positive number starting at 1. Copied off: http://tldp.org/LDP/GNU-Linux-Tools-Summary/html/mass-rename.html

Compare two directory structures for differences

This is very handy if you need to know if two directory structures are the same or not including file content. It uses the diff command with the r switch to work recursively and the x switch to allow skipping of the .svn directories.

diff -qrx .svn DIR1 DIR2

Mount a .iso and .img files

See this for ISO files and this for IMG files.

Mount a USB stick

If you don't know the device name of the stick, plug it in and find it with the following:

dmesg |grep SCSI

Then mount the first partition,

mkdir -p ~/memstick
mount -t vfat -o rw,users /dev/sdX1 ~/memstick

Clone a device

Remember that this is a very low-level operation and so if you're cloning a live partition you may end up with partially written files or other corruption issues on the destination. Stop as many services as you can first, do an fsck afterwards and manually copy any directories that had issues.

dd if=/dev/sda of=/dev/sdb bs=32M

A good way to find more information about the devices first, for example if you have a number of unformatted devices of the same size and need more specific information about the hardware, is to use the 'lsscsi command which gives the following sample output:

[0:0:0:0]    disk    AMCC     9650SE-2LP DISK  4.10  /dev/sdb 
[1:0:0:0]    disk    ATA      Hitachi HUA72201 JP4O  /dev/sda 
[2:0:0:0]    disk    ATA      TOSHIBA MG03ACA1 FL1A  /dev/sdc

The sdb and sdc devices look identical with commands such as lsblk or fdisk -l, but one of the is a RAID pair and the other just a normal drive.

See also: Clone a live partition over SSH

Changing device UUIDs

It may sometimes be useful to change the UUIDs of partitions such as when they've been cloned with a low-level method like dd.

Note: the disk identifier should also be changed which you can do with fdisk from the expert menu.

Check what devices and UUIDs you have in the system:

# blkid
/dev/sda1: UUID="bc459b13-3aad-4017-a68b-ce8ab36275e1" SEC_TYPE="ext2" TYPE="ext3" 
/dev/sda2: UUID="86ce60b7-ad4a-44aa-88e2-2aefd5c6c396" TYPE="swap" 
/dev/sda3: UUID="b56d82f7-5817-4a52-9763-0b38aa360e2b" TYPE="ext4" 
/dev/sdc1: UUID="81d40a02-8019-4c1f-afb8-fb41d117c6d1" TYPE="ext3" 
/dev/sdc2: UUID="c57f1400-129a-402a-90f1-820a22c6a2fe" TYPE="swap" 
/dev/sdc3: UUID="c23ab65a-c32f-41e3-bd31-51ed563e0099" TYPE="ext4"

or:

# ls -l /dev/disk/by-uuid/
total 0
lrwxrwxrwx 1 root root 10 Sep  2 12:10 81d40a02-8019-4c1f-afb8-fb41d117c6d1 -> ../../sdc1
lrwxrwxrwx 1 root root 10 Sep  2 12:10 86ce60b7-ad4a-44aa-88e2-2aefd5c6c396 -> ../../sda2
lrwxrwxrwx 1 root root 10 Sep  2 12:10 b56d82f7-5817-4a52-9763-0b38aa360e2b -> ../../sda3
lrwxrwxrwx 1 root root 10 Sep  2 12:10 bc459b13-3aad-4017-a68b-ce8ab36275e1 -> ../../sda1
lrwxrwxrwx 1 root root 10 Sep  2 12:10 c23ab65a-c32f-41e3-bd31-51ed563e0099 -> ../../sdc3
lrwxrwxrwx 1 root root 10 Sep  2 12:27 c57f1400-129a-402a-90f1-820a22c6a2fe -> ../../sdc2

Check what swap partitions you have:

# cat /proc/swaps
Filename				Type		Size	Used	Priority
/dev/sda2                               partition	3999740	0	-1

Create a new UUID and assign it to and ext parition:

# uuidgen
81d40a02-8019-4c1f-afb8-fb41d117c6d1
#tune2fs /dev/sdb1 -U 81d40a02-8019-4c1f-afb8-fb41d117c6d1

Do the same for a swap partition:

# uuidgen
86ce60b7-ad4a-44aa-88e2-2aefd5c6c396
# swapoff /dev/sda2
# mkswap -U 86ce60b7-ad4a-44aa-88e2-2aefd5c6c396 /dev/sda2
Setting up swapspace version 1, size = 3999740 KiB
no label, UUID=86ce60b7-ad4a-44aa-88e2-2aefd5c6c396

Updating device partition tables without rebooting

  • partprobe /dev/sdx
  • partx -u /dev/sdx
  • echo 1 > /sys/block/sdX/device/rescan
  • check partition information before and after with cat /proc/partitions

Downloading from google drive in shell

Thanks to FrankerZ for this! The first command outputs a code which you put into the second command.

wget --save-cookies cookies.txt --keep-session-cookies --no-check-certificate 'https://docs.google.com/uc?export=download&id=FILEID' -O- | sed -rn 's/.*confirm=([0-9A-Za-z_]+).*/Code: \1\n/p'
wget --load-cookies cookies.txt 'https://docs.google.com/uc?export=download&confirm=CODE_FROM_ABOVE&id=FILEID' -O FILENAME

Image manipulation

Resizing JPG's and changing quality setting

The first line shows how to reduce and image to 25% and quality to 50% adding "_resized" to the results filename.

convert foo.jpg -resize 25% -quality 75% foo_resized.jpg

Here's a Perl script version used to apply this same command to all JPG's (jpg, JPG, jpeg etc) in the current directory.

#!/usr/bin/perl
use File::Glob qw(:globally :nocase);
for (glob "*.jp*g") {
	my $img = $_;
	s/(....)$/_resized$1/;
	qx "convert \'$img\' -resize 25% -quality 75% \'$_\'";
}

This command rotates, normalises, greyscales and sets the output quality of set of an image:

convert FILE  -normalize -set colorspace Gray -separate -average -resize 50% -rotate "-90" -quality 75% OUT.JPG

Apply an opaque background of a specified colour to a directory of transparent PNG's

  • This command requires ImageMagick to be installed
  • It loops through all PNG's in the CWD and puts them in a directory called processed which must exist
perl -e 'qx "convert $_ -background #ff00ff -flatten foo/$_" for glob "*.png"'

Audio & Video

avconv (ffmpeg)

Note: Many operating systems including Ubuntu are now using the libav fork of ffmpeg now which means you should use the avconv command instead.

The following converts a .wav file to an mp3:

ffmpeg -i foo.wav foo.mp3

Here's a more complicated example which converts mp4 video into mp3 audio at a lower quality with only one channel at 11KHz and 32kbps.

ffmpeg -i foo.mp4 -acodec libmp3lame -ab 16k -ar 11025 -ac 1 foo.mp3

To do a whole directory you could do this:

perl -e 'qx "ffmpeg -i $_ $_.mp3" for glob "*.wav"'

Or a little more complex; a whole directory with a proper name change accounting for names with spaces in them. This

perl -e 'for (glob "*.mp4") { $i=$_; s/.mp3$/.converted.wav/; qx "ffmpeg -i \"$i\" \"$_\"" }'

Use the following commands to extract a small (5 seconds in this example) snippet out of a video (the -ar switch is only needed for outputting to flv I think).

ffmpeg -i "foo.avi" -ss 00:10:10 -t 5 -ar 22050 "foo.flv"

See also:

Downloading Youtube from shell

First install the Python youtube-dl tool, with pip:

sudo pip install --upgrade youtube_dl

Then use the tool to download the video, e.g.

youtube-dl https://www.youtube.com/watch?v=f9m2yReECak

System monitoring & management

List the top 10 memory consuming processes

ps -auxf | sort -nr -k 4 | head -10


List the top 10 CPU consuming processes

ps -auxf | sort -nr -k 3 | head -10


Limit CPU resources to a process (you may need to first install the cpulimit package):

cpulimit -b -l <PERCENTAGE> -p <PID>

Network commands

Restart the network after changing configuration

/etc/init.d/networking restart

List all the listening sockets and their ports and programs

netstat -lp

Get current default gateway

netstat -nr

The default gateway is on the last line, it should have the U and G flags set

Release DHCP lease

dhclient -r

Obtain a new DHCP lease

dhclient

Scan a local subnet for active IP addresses

sudo nmap -sP 192.168.1.0/24

Get the MAC address and hostname of an IP on the current subnet

arp -a 192.168.1.1

Display traffic passing through a port

This is very useful for debugging interactions with services, for example the following command shows traffic going between a local elasticsearch service and a local client application.

sudo tcpdump port 9200 -qAi lo

Make the system prefer IPv4 connections

Open /etc/gai.conf and uncomment the following line:

#
# For sites which prefer IPv4 connections change the last line to
#
precedence ::ffff:0:0/96 100

Test which IPv6 address is being used for public connections

Servers can be assigned many IPv6 addresses, but which one should you assign to AAAA records pointing at the server? This command shows which address is being used to make outgoing connections which would be the best one to use for incoming connections as well. The address used to test the outgoing connection here is google's public DNS server.

ip r get to 2001:4860:4860::8888

Internet

Download an url to a local file with continuation

curl -L -O -C - http://www.foo.bar/file
wget -c http://www.foo.bar/file


Download a copy of an entire site

This will download an entire site, if the site is already downloaded, then only newer files are transferred.

wget -m http://foo.bar

Port forwarding

Port forwarding allows a remote client to gain access to a network so an intranet can be accessed.

ssh -fnNTL[PORT]:appserver:[PORT] username@sshdserver

Then point your webbrowsers proxy server to appserver:[PORT] and access the intranet etc.

To subvert a firewalled environment where outgoing ssh is allowed.

$ ssh -D 9000 username@remotehost

Then point your web browser to a SOCKS proxy @ localhost:9000

Further it's possible to get ssh through a web proxy using corkscrew.

ssh -oProxyCommand='corkscrew local_web_proxy proxy_port %h %p' username@remotehost

Monitoring network traffic

You can get daily and monthly traffic statistics that are updated every five minutes by installing the vnstat package. This is very useful if your ISP enforces a traffic limit. You can get reports on individual network interfaces.

Enable monitoring on a specific interface as follows:

vnstat --enable -i eth0

And then you can get a report for an interface as follows:

vnstat -i eth0

Example output:

monthly
                     rx      |     tx      |    total    |   avg. rate
     ------------------------+-------------+-------------+---------------
       Nov '17     82.43 MiB |   15.97 MiB |   98.40 MiB |    0.35 kbit/s
     ------------------------+-------------+-------------+---------------
     estimated        91 MiB |      16 MiB |     107 MiB |

   daily
                     rx      |     tx      |    total    |   avg. rate
     ------------------------+-------------+-------------+---------------
         today     82.43 MiB |   15.97 MiB |   98.40 MiB |   10.67 kbit/s
     ------------------------+-------------+-------------+---------------
     estimated        93 MiB |      17 MiB |     110 MiB |

General

Add and remove startup items

update-rc.d ITEM defaults
update-rc.d -f ITEM remove

Date and time

Get the current (or another time specified in --date) time in UNIX timestamp format:

date +%s
date --date='01/30/2012 01:23:45' +%s


Convert a UNIX timestamp into a date/time in the current locale format:

date -d @1282368345

Specific guides

See also