Linux commands
Contents
- 1 Ubuntu package management
- 2 SSH & SCP
- 3 Files & Devices
- 3.1 List all the storage devices and partitions attached to the system (even if they're unformatted or unmounted)
- 3.2 Get the size of a directory and its contents
- 3.3 Search for file content recursively
- 3.4 Search for file content recursively and tar
- 3.5 Search for file by name recursively
- 3.6 Count occurrences of a word in a file or files
- 3.7 Search and replace content in files
- 3.8 Mass renaming
- 3.9 Compare two directory structures for differences
- 3.10 Mount a .iso
- 3.11 Mount a USB stick
- 3.12 Clone a device
- 3.13 Changing device UUIDs
- 3.14 Updating device partition tables without rebooting
- 4 Image manipulation
- 5 Audio conversion
- 6 Video manipulation
- 7 System monitoring
- 8 Network commands
- 8.1 Restart the network after changing configuration
- 8.2 List all the listening sockets and their ports and programs
- 8.3 Get current default gateway
- 8.4 Release DHCP lease
- 8.5 Obtain a new DHCP lease
- 8.6 Scan a local subnet for active IP addresses
- 8.7 Get the MAC address and hostname of an IP on the current subnet
- 9 Internet
- 10 Port forwarding
- 11 General
- 12 See also
Ubuntu package management
Searching for an installed package
Use dpkg and grep;
dpkg -l | grep java
will list all packages with java in the name
Searching for installable packages
apt-cache performs a variety of operations on APT´s package cache
apt-cache search java
lists all packages in the cache with java in the name that are installable
SSH & SCP
tar pipe
## From a local to remote machine
tar -zcvf - directory | ssh remote 'cd somewhere; tar -zxvf -'
## From remote machine to local machine
ssh remote 'cd somewhere; tar -zcvf - directory' | tar -zxf -
## remote tar and pipe to tgz
ssh remote 'cd somewhere; tar -zcf - directory' | cat - > directory.tar.gz
Example of sending a file to a remote server:
scp -P 2222 /my/local/file.txt USER@example.com:/home/USER
Example of retrieving a file from a remote server:
scp -P 2222 USER@example.com:/home/USER/file.txt ./
Note using port forwarding for commonly accessed sites you can define non standard ports in your .ssh/config
so you never
need to explicitly state them above.
Recovering from an interrupted transfer
This ability is included natively in Linux and used to be very useful for splitting large backups up so they could fit onto small media such as floppy disks. But there's one time when it's very useful nowadays too which is when a large file transfer gets interupted and there's no option for continuation such as when using SCP.
When this happens, go onto the target machine, check how many bytes were transferred of the target file and rename it to "xaa". Then on the local machine, do the following command to split the source file into two parts, the first being of the size of the number of bytes that have already been transferred (in this example 1234567 bytes).
split -b 1234567 foo.tar.gz
The resulting files are called "xaa" and "xab", and the latter is the containing the remaining content that needs to be transferred to the target server. Once you've transferred it across, you can then join the two files (remember you renamed the first part to "xaa" so there's no need to transfer that) together using cat as follows, and then remove all the xa* files from noth source and target servers.
cat xa* > foo.tar.gz
Files & Devices
List all the storage devices and partitions attached to the system (even if they're unformatted or unmounted)
cat /proc/partitions
To check what filesystem a device has on it use file, e.g.
file -s /dev/sda1
Get the size of a directory and its contents
du -sh /home/foo
Use this more specific version to find the size of a users Maildir folder:
du -sh /home/*/Maildir|sed 's|/home/||'|sed 's|/Maildir||'
Search for file content recursively
Here's an example looking for a phrase within a specific file type recursively through a directory structure, and printing the file names and line numbers of the resulting matches.
grep -rn "Foo" *.php
There are other tips at stackoverflow.com.
Search for file content recursively and tar
## Find and tar
find . -name "*.R" -print0 | xargs -0 tar -cvf Rfiles.tar
## check contents
tar -tvf Rfiles.tar
## Find and tar.gz
find . -name "*.R" -print0 | xargs -0 tar -zcvf Rfiles.tar.gz
## check contents
tar -ztvf Rfiles.tar.gz
Search for file by name recursively
find . -name "*.R"
Count occurrences of a word in a file or files
grep -roh "WORD" file*.txt | wc -
Search and replace content in files
You could also use find and sed, but I find that this little line of perl works nicely.
perl -pi -w -e 's/SEARCH/REPLACE/g;' *.php
- -e means execute the following line of code.
- -i means edit in-place
- -w write warnings
- -p loop
EXTS="7z t7z"
Mass renaming
mmv is a mass move/copy/renaming tool that uses standard wildcards to perform its functions. According to the manual the “;” wildcard is useful for matching files at any depth in the directory tree (ie it will go below the current directory, recursively).
- Example
The first pattern matches anything with a “.JPG” and renames each file (the “#1” matches the first wildcard) to “.jpg”. Each time you use a \(wildcard) you can use a #x to get that wildcard. Where x is a positive number starting at 1. Copied off: http://tldp.org/LDP/GNU-Linux-Tools-Summary/html/mass-rename.html
Compare two directory structures for differences
This is very handy if you need to know if two directory structures are the same or not including file content. It uses the diff command with the r switch to work recursively and the x switch to allow skipping of the .svn directories.
diff -qrx .svn DIR1 DIR2
Mount a .iso
See this HOWTO
Mount a USB stick
If you don't know the device name of the stick, plug it in and find it with the following:
dmesg |grep SCSI
Then mount the first partition,
mkdir -p ~/memstick
mount -t vfat -o rw,users /dev/sdX1 ~/memstick
Clone a device
Remember that this is a very low-level operation and so if you're cloning a live partition you may end up with partially written files or other corruption issues on the destination. Stop as many services as you can first, do an fsck afterwards and manually copy any directories that had issues.
dd if=/dev/sda of=/dev/sdb bs=32M
A good way to find more information about the devices first, for example if you have a number of unformatted devices of the same size and need more specific information about the hardware, is to use the 'lsscsi command which gives the following sample output:
[0:0:0:0] disk AMCC 9650SE-2LP DISK 4.10 /dev/sdb
[1:0:0:0] disk ATA Hitachi HUA72201 JP4O /dev/sda
[2:0:0:0] disk ATA TOSHIBA MG03ACA1 FL1A /dev/sdc
The sdb and sdc devices look identical with commands such as lsblk or fdisk -l, but one of the is a RAID pair and the other just a normal drive.
Changing device UUIDs
It may sometimes be useful to change the UUIDs of partitions such as when they've been cloned with a low-level method like dd.
Note: the disk identifier should also be changed which you can do with fdisk from the expert menu.
Check what devices and UUIDs you have in the system:
# blkid
/dev/sda1: UUID="bc459b13-3aad-4017-a68b-ce8ab36275e1" SEC_TYPE="ext2" TYPE="ext3"
/dev/sda2: UUID="86ce60b7-ad4a-44aa-88e2-2aefd5c6c396" TYPE="swap"
/dev/sda3: UUID="b56d82f7-5817-4a52-9763-0b38aa360e2b" TYPE="ext4"
/dev/sdc1: UUID="81d40a02-8019-4c1f-afb8-fb41d117c6d1" TYPE="ext3"
/dev/sdc2: UUID="c57f1400-129a-402a-90f1-820a22c6a2fe" TYPE="swap"
/dev/sdc3: UUID="c23ab65a-c32f-41e3-bd31-51ed563e0099" TYPE="ext4"
or:
# ls -l /dev/disk/by-uuid/
total 0
lrwxrwxrwx 1 root root 10 Sep 2 12:10 81d40a02-8019-4c1f-afb8-fb41d117c6d1 -> ../../sdc1
lrwxrwxrwx 1 root root 10 Sep 2 12:10 86ce60b7-ad4a-44aa-88e2-2aefd5c6c396 -> ../../sda2
lrwxrwxrwx 1 root root 10 Sep 2 12:10 b56d82f7-5817-4a52-9763-0b38aa360e2b -> ../../sda3
lrwxrwxrwx 1 root root 10 Sep 2 12:10 bc459b13-3aad-4017-a68b-ce8ab36275e1 -> ../../sda1
lrwxrwxrwx 1 root root 10 Sep 2 12:10 c23ab65a-c32f-41e3-bd31-51ed563e0099 -> ../../sdc3
lrwxrwxrwx 1 root root 10 Sep 2 12:27 c57f1400-129a-402a-90f1-820a22c6a2fe -> ../../sdc2
Check what swap partitions you have:
# cat /proc/swaps
Filename Type Size Used Priority
/dev/sda2 partition 3999740 0 -1
Create a new UUID and assign it to and ext parition:
# uuidgen
81d40a02-8019-4c1f-afb8-fb41d117c6d1
#tune2fs /dev/sdb1 -U 81d40a02-8019-4c1f-afb8-fb41d117c6d1
Do the same for a swap partition:
# uuidgen
86ce60b7-ad4a-44aa-88e2-2aefd5c6c396
# swapoff /dev/sda2
# mkswap -U 86ce60b7-ad4a-44aa-88e2-2aefd5c6c396 /dev/sda2
Setting up swapspace version 1, size = 3999740 KiB
no label, UUID=86ce60b7-ad4a-44aa-88e2-2aefd5c6c396
Updating device partition tables without rebooting
- partprobe /dev/sdx
- partx -u /dev/sdx
- echo 1 > /sys/block/sdX/device/rescan
- check partition information before and after with cat /proc/partitions
Image manipulation
Resizing JPG's and changing quality setting
The first line shows how to reduce and image to 25% and quality to 50% adding "_resized" to the results filename. The second command uses Perl to apply this same command to all JPG's in the current directory.
convert foo.jpg -resize 25% -quality 50% foo_resized.jpg
perl -e 'for (glob "*.jpg") { $img = $_; s/(....)$/_resized$1/; qx "convert \'$img\' -resize 25% -quality 50% \'$_\'"; }'
Apply an opaque background of a specified colour to a directory of transparent PNG's
- This command requires ImageMagick to be installed
- It loops through all PNG's in the CWD and puts them in a directory called processed which must exist
perl -e 'qx "convert $_ -background #ff00ff -flatten foo/$_" for glob "*.png"'
Audio conversion
The following converts a .wav file to an mp3:
ffmpeg -i foo.wav foo.mp3
Here's a more complicated example which converts mp4 video into mp3 audio at a lower quality with only one channel at 11KHz and 32kbps.
ffmpeg -i foo.mp4 -acodec libmp3lame -ab 16k -ar 11025 -ac 1 foo.mp3
To do a whole directory you could do this:
perl -e 'qx "ffmpeg -i $_ $_.mp3" for glob "*.wav"'
Or a little more complex; a whole directory with a proper name change accounting for names with spaces in them. This
perl -e 'for (glob "*.mp4") { $i=$_; s/.mp3$/.converted.wav/; qx "ffmpeg -i \"$i\" \"$_\"" }'
Video manipulation
Use the following commands to extract a small snippet out of a video (the -ar switch is only needed for outputting to flv I think).
ffmpeg -i "foo.avi" -ss 00:10:10 -t 00:00:05 -ar 22050 "foo.flv"
See also
- Converting_microarray_images for bash/imagemagick image file manipulation
System monitoring
List the top 10 memory consuming processes
ps -auxf | sort -nr -k 4 | head -10
List the top 10 CPU consuming processes
ps -auxf | sort -nr -k 3 | head -10
Network commands
Restart the network after changing configuration
/etc/init.d/networking restart
List all the listening sockets and their ports and programs
netstat -lp
Get current default gateway
netstat -nr
The default gateway is on the last line, it should have the U and G flags set
Release DHCP lease
dhclient -r
Obtain a new DHCP lease
dhclient
Scan a local subnet for active IP addresses
nmap -sP 192.168.1.0/24
Get the MAC address and hostname of an IP on the current subnet
arp -a 192.168.1.1
Internet
Download a copy of an entire site
This will download an entire site, if the site is already downloaded, then only newer files are transferred.
wget -m http://foo.bar
Port forwarding
Port forwarding allows a remote client to gain access to a network so an intranet can be accessed.
ssh -fN -L[PORT]:appserver:[PORT] username@sshdserver
Then point your webbrowsers proxy server to appserver:[PORT] and access the intranet etc.
To subvert a firewalled environment where outgoing ssh is allowed.
$ ssh -D 9000 username@remotehost
Then point your web browser to a SOCKS proxy @ localhost:9000
Further it's possible to get ssh through a web proxy using corkscrew.
ssh -oProxyCommand='corkscrew local_web_proxy proxy_port %h %p' username@remotehost
General
Add and remove startup items
update-rc.d ITEM defaults
update-rc.d -f ITEM remove