Difference between revisions of "Linux commands"

From Organic Design wiki
(→‎Search for file content recursively: simpler and more useful example)
(Change source-code blocks to standard format)
Line 6: Line 6:
 
==== Searching for an installed package ====
 
==== Searching for an installed package ====
 
Use dpkg and grep;
 
Use dpkg and grep;
−
{{code|<bash>
+
<source lang="bash">
 
dpkg -l | grep java
 
dpkg -l | grep java
−
</bash>}}
+
</source>
 
will list all packages with java in the name
 
will list all packages with java in the name
  
 
==== Searching for installable packages ====
 
==== Searching for installable packages ====
 
apt-cache  performs a variety of operations on APT´s package cache
 
apt-cache  performs a variety of operations on APT´s package cache
−
{{code|<bash>
+
<source lang="bash">
 
apt-cache search java
 
apt-cache search java
−
</bash>}}
+
</source>
 
lists all packages in the cache with java in the name that are installable
 
lists all packages in the cache with java in the name that are installable
  
Line 21: Line 21:
  
 
=== tar pipe ===
 
=== tar pipe ===
−
{{code|<bash>
+
<source lang="bash">
 
## From a local to remote machine
 
## From a local to remote machine
 
tar -zcvf - directory | ssh remote 'cd somewhere; tar -zxvf -'
 
tar -zcvf - directory | ssh remote 'cd somewhere; tar -zxvf -'
Line 28: Line 28:
 
## remote tar and pipe to tgz
 
## remote tar and pipe to tgz
 
ssh remote 'cd somewhere; tar -zcf - directory' | cat - > directory.tar.gz
 
ssh remote 'cd somewhere; tar -zcf - directory' | cat - > directory.tar.gz
−
</bash>}}
+
</source>
  
 
Example of sending a file to a remote server:
 
Example of sending a file to a remote server:
−
{{code|<bash>
+
<source lang="bash">
 
scp -P 2222 /my/local/file.txt USER@example.com:/home/USER
 
scp -P 2222 /my/local/file.txt USER@example.com:/home/USER
−
</bash>}}
+
</source>
  
 
Example of retrieving a file from a remote server:
 
Example of retrieving a file from a remote server:
−
{{code|<bash>
+
<source lang="bash">
 
scp -P 2222 USER@example.com:/home/USER/file.txt ./
 
scp -P 2222 USER@example.com:/home/USER/file.txt ./
−
</bash>}}
+
</source>
 
''Note using port forwarding for commonly accessed sites you can define non standard ports in your  <code>.ssh/config</code> so you never
 
''Note using port forwarding for commonly accessed sites you can define non standard ports in your  <code>.ssh/config</code> so you never
 
need to explicitly state them above.''
 
need to explicitly state them above.''
Line 46: Line 46:
  
 
When this happens, go onto the target machine, check how many bytes were transferred of the target file and rename it to "xaa". Then on the local machine, do the following command to split the source file into two parts, the first being of the size of the number of bytes that have already been transferred (in this example 1234567 bytes).
 
When this happens, go onto the target machine, check how many bytes were transferred of the target file and rename it to "xaa". Then on the local machine, do the following command to split the source file into two parts, the first being of the size of the number of bytes that have already been transferred (in this example 1234567 bytes).
−
{{code|<bash>split -b 1234567 foo.tar.gz</bash>}}
+
<source lang="bash">
 +
split -b 1234567 foo.tar.gz
 +
</source>
  
 
The resulting files are called "xaa" and "xab", and the latter is the containing the remaining content that needs to be transferred to the target server. Once you've transferred it across, you can then join the two files (remember you renamed the first part to "xaa" so there's no need to transfer that) together using ''cat'' as follows, and then remove all the ''xa*'' files from noth source and target servers.
 
The resulting files are called "xaa" and "xab", and the latter is the containing the remaining content that needs to be transferred to the target server. Once you've transferred it across, you can then join the two files (remember you renamed the first part to "xaa" so there's no need to transfer that) together using ''cat'' as follows, and then remove all the ''xa*'' files from noth source and target servers.
−
{{code|<bash>cat xa* > foo.tar.gz</bash>}}
+
<source lang="bash">
 +
cat xa* > foo.tar.gz
 +
</source>
  
 
== Files & Devices ==
 
== Files & Devices ==
  
 
=== Get the size of a directory and its contents ===
 
=== Get the size of a directory and its contents ===
−
{{code|<bash>du -sh /home/foo</bash>}}
+
<source lang="bash">
 +
du -sh /home/foo
 +
</source>
  
  
 
Use this more specific version to find the size of a users Maildir folder:
 
Use this more specific version to find the size of a users Maildir folder:
−
{{code|<bash>du -sh /home/*/Maildir|sed 's|/home/||'|sed 's|/Maildir||'</bash>}}
+
<source lang="bash">
 +
du -sh /home/*/Maildir|sed 's|/home/||'|sed 's|/Maildir||'
 +
</source>
  
 
=== Search for file content recursively ===
 
=== Search for file content recursively ===
 
Here's an example looking for a phrase within a specific file type recursively through a directory structure, and printing the file names and line numbers of the resulting matches.
 
Here's an example looking for a phrase within a specific file type recursively through a directory structure, and printing the file names and line numbers of the resulting matches.
  
−
{{code|<bash>grep -rn "Foo" *.php</bash>}}
+
<source lang="bash">
 +
grep -rn "Foo" *.php
 +
</source>
  
 
There are other tips at [http://stackoverflow.com/questions/316590/how-do-you-use-cat-recursively stackoverflow.com].
 
There are other tips at [http://stackoverflow.com/questions/316590/how-do-you-use-cat-recursively stackoverflow.com].
  
 
=== Search for file content recursively and tar ===
 
=== Search for file content recursively and tar ===
−
{{code|<bash>
+
<source lang="bash">
 
## Find and tar
 
## Find and tar
 
find . -name "*.R" -print0 | xargs -0 tar -cvf Rfiles.tar
 
find . -name "*.R" -print0 | xargs -0 tar -cvf Rfiles.tar
Line 78: Line 88:
 
## check contents
 
## check contents
 
tar -ztvf Rfiles.tar.gz
 
tar -ztvf Rfiles.tar.gz
−
</bash>}}
+
</source>
  
 
=== Search for file content recursively and long list ===  
 
=== Search for file content recursively and long list ===  
−
{{code|<bash>find . -name "*.R" -ls</bash>}}
+
<source lang="bash">
 +
find . -name "*.R" -ls
 +
</source>
  
 
=== Count occurrences of a word in a file or files ===
 
=== Count occurrences of a word in a file or files ===
−
{{code|<bash>grep -roh "WORD" file*.txt | wc -</bash>}}
+
<source lang="bash">
 +
grep -roh "WORD" file*.txt | wc -
 +
</source>
  
 
=== Search and replace content in files ===
 
=== Search and replace content in files ===
 
You could also use find and sed, but I find that this little line of perl works nicely.
 
You could also use find and sed, but I find that this little line of perl works nicely.
−
{{code|<bash>perl -pi -w -e 's/SEARCH/REPLACE/g;' *.php</bash>}}
+
<source lang="bash">
 +
perl -pi -w -e 's/SEARCH/REPLACE/g;' *.php
 +
</source>
 
*-e means execute the following line of code.
 
*-e means execute the following line of code.
 
*-i means edit in-place
 
*-i means edit in-place
Line 105: Line 121:
 
=== Compare two directory structures for differences ===
 
=== Compare two directory structures for differences ===
 
This is very handy if you need to know if two directory structures are the same or not including file content. It uses the ''diff'' command with the ''r'' switch to work recursively and the ''x'' switch to allow skipping of the ''.svn'' directories.
 
This is very handy if you need to know if two directory structures are the same or not including file content. It uses the ''diff'' command with the ''r'' switch to work recursively and the ''x'' switch to allow skipping of the ''.svn'' directories.
−
{{code|<bash>diff -qrx .svn DIR1 DIR2</bash>}}
+
<source lang="bash">
 +
diff -qrx .svn DIR1 DIR2
 +
</source>
  
 
=== Mount a .iso ===
 
=== Mount a .iso ===
Line 112: Line 130:
 
=== Mount a USB stick ===
 
=== Mount a USB stick ===
 
If you don't know the device name of the stick, plug it in and find it with the following:
 
If you don't know the device name of the stick, plug it in and find it with the following:
−
{{code|<bash>dmesg |grep SCSI</bash>}}
+
<source lang="bash">
 +
dmesg |grep SCSI
 +
</source>
  
 
Then mount the first partition,
 
Then mount the first partition,
−
{{code|<bash>mkdir -p ~/memstick
+
<source lang="bash">
−
mount -t vfat -o rw,users /dev/sdX1 ~/memstick</bash>}}
+
mkdir -p ~/memstick
 +
mount -t vfat -o rw,users /dev/sdX1 ~/memstick
 +
</source>
  
  
Line 123: Line 145:
 
=== Resizing JPG's and changing quality setting ===
 
=== Resizing JPG's and changing quality setting ===
 
The first line shows how to reduce and image to 25% and quality to 50% adding "_resized" to the results filename. The second command uses Perl to apply this same command to all JPG's in the current directory.
 
The first line shows how to reduce and image to 25% and quality to 50% adding "_resized" to the results filename. The second command uses Perl to apply this same command to all JPG's in the current directory.
−
{{code|<bash>
+
<source lang="bash">
 
convert foo.jpg -resize 25% -quality 50% foo_resized.jpg
 
convert foo.jpg -resize 25% -quality 50% foo_resized.jpg
  
 
perl -e 'for (glob "*.jpg") { $img = $_; s/(....)$/_resized$1/; qx "convert \'$img\' -resize 25% -quality 50% \'$_\'"; }'
 
perl -e 'for (glob "*.jpg") { $img = $_; s/(....)$/_resized$1/; qx "convert \'$img\' -resize 25% -quality 50% \'$_\'"; }'
−
</bash>}}
+
</source>
  
 
=== Apply an opaque background of a specified colour to a directory of transparent PNG's ===
 
=== Apply an opaque background of a specified colour to a directory of transparent PNG's ===
 
*This command requires ImageMagick to be installed
 
*This command requires ImageMagick to be installed
 
*It loops through all PNG's in the CWD and puts them in a directory called ''processed'' which must exist
 
*It loops through all PNG's in the CWD and puts them in a directory called ''processed'' which must exist
−
{{code|<bash>
+
<source lang="bash">
 
perl -e 'qx "convert $_ -background #ff00ff -flatten foo/$_" for glob "*.png"'
 
perl -e 'qx "convert $_ -background #ff00ff -flatten foo/$_" for glob "*.png"'
−
</bash>}}
+
</source>
  
 
== Audio conversion ==
 
== Audio conversion ==
 
The following converts a ''.wav'' file to an ''mp3'':
 
The following converts a ''.wav'' file to an ''mp3'':
−
{{code|<bash>
+
<source lang="bash">
 
ffmpeg -i test.wav test.mp3
 
ffmpeg -i test.wav test.mp3
−
</bash>}}
+
</source>
  
 
To do a whole directory you could do this:
 
To do a whole directory you could do this:
−
{{code|<bash>
+
<source lang="bash">
 
perl -e 'qx "ffmpeg -i $_ $_.mp3" for glob "*.wav"'
 
perl -e 'qx "ffmpeg -i $_ $_.mp3" for glob "*.wav"'
−
</bash>}}
+
</source>
  
 
== Video manipulation ==
 
== Video manipulation ==
 
Use the following commands to extract a small snippet out of a video (the -ar switch is only needed for outputting to flv I think).
 
Use the following commands to extract a small snippet out of a video (the -ar switch is only needed for outputting to flv I think).
−
{{code|<bash>
+
<source lang="bash">
 
ffmpeg -i "foo.avi" -ss 00:10:10 -t 00:00:05 -ar 22050 "foo.flv"
 
ffmpeg -i "foo.avi" -ss 00:10:10 -t 00:00:05 -ar 22050 "foo.flv"
−
</bash>}}
+
</source>
  
 
=== See also ===
 
=== See also ===
Line 158: Line 180:
 
== System monitoring ==
 
== System monitoring ==
 
List the top 10 memory consuming processes
 
List the top 10 memory consuming processes
−
{{code|<bash>ps -auxf | sort -nr -k 4 | head -10</bash>}}
+
<source lang="bash">
 +
ps -auxf | sort -nr -k 4 | head -10
 +
</source>
  
  
 
List the top 10 CPU consuming processes
 
List the top 10 CPU consuming processes
−
{{code|<bash>ps -auxf | sort -nr -k 3 | head -10</bash>}}
+
<source lang="bash">
 +
ps -auxf | sort -nr -k 3 | head -10
 +
</source>
  
 
== Network commands ==
 
== Network commands ==
  
 
=== Restart the network after changing configuration ===
 
=== Restart the network after changing configuration ===
−
{{code|<bash>/etc/init.d/networking restart</bash>}}
+
<source lang="bash">
 +
/etc/init.d/networking restart
 +
</source>
  
 
=== List all the listening sockets and their ports and programs ===
 
=== List all the listening sockets and their ports and programs ===
−
{{code|<bash>netstat -lp</bash>}}
+
<source lang="bash">
 +
netstat -lp
 +
</source>
  
 
=== Get current default gateway ===
 
=== Get current default gateway ===
−
{{code|<bash>netstat -nr</bash>}}
+
<source lang="bash">
 +
netstat -nr
 +
</source>
 
The default gateway is on the last line, it should have the U and G flags set
 
The default gateway is on the last line, it should have the U and G flags set
  
 
=== Release DHCP lease ===
 
=== Release DHCP lease ===
−
{{code|<bash>dhclient -r</bash>}}
+
<source lang="bash">
 +
dhclient -r
 +
</source>
  
 
=== Obtain a new DHCP lease ===
 
=== Obtain a new DHCP lease ===
−
{{code|<bash>dhclient</bash>}}
+
<source lang="bash">
 +
dhclient
 +
</source>
  
 
=== Scan a local subnet for active IP addresses ===
 
=== Scan a local subnet for active IP addresses ===
−
{{code|<bash>nmap -sP 192.168.1.0/24</bash>}}
+
<source lang="bash">
 +
nmap -sP 192.168.1.0/24
 +
</source>
  
 
=== Get the MAC address and hostname of an IP on the current subnet ===
 
=== Get the MAC address and hostname of an IP on the current subnet ===
−
{{code|<bash>arp -a 192.168.1.1</bash>}}
+
<source lang="bash">
 +
arp -a 192.168.1.1
 +
</source>
  
 
== Internet ==
 
== Internet ==
Line 192: Line 232:
 
=== Download a copy of an entire site ===
 
=== Download a copy of an entire site ===
 
This will download an entire site, if the site is already downloaded, then only newer files are transferred.
 
This will download an entire site, if the site is already downloaded, then only newer files are transferred.
−
{{code|<pre>
+
<source>
 
wget -m http://foo.bar
 
wget -m http://foo.bar
−
</pre>}}
+
</source>
  
 
== Port forwarding ==
 
== Port forwarding ==
 
Port forwarding allows a remote client to gain access to a network so an intranet can be accessed.
 
Port forwarding allows a remote client to gain access to a network so an intranet can be accessed.
−
{{code|<bash>
+
<source lang="bash">
 
ssh -fN -L[PORT]:appserver:[PORT] username@sshdserver
 
ssh -fN -L[PORT]:appserver:[PORT] username@sshdserver
−
</bash>}}
+
</source>
  
 
Then point your webbrowsers proxy server to appserver:[PORT] and access the intranet etc.
 
Then point your webbrowsers proxy server to appserver:[PORT] and access the intranet etc.
  
 
To subvert a firewalled environment where outgoing ssh is allowed.
 
To subvert a firewalled environment where outgoing ssh is allowed.
−
{{code|<bash>
+
<source lang="bash">
 
$ ssh -D 9000 username@remotehost
 
$ ssh -D 9000 username@remotehost
−
</bash>}}
+
</source>
  
 
Then point your web browser to a SOCKS proxy @ localhost:9000
 
Then point your web browser to a SOCKS proxy @ localhost:9000
  
 
Further it's possible to get ssh through a web proxy using [http://www.agroman.net/corkscrew/ corkscrew].
 
Further it's possible to get ssh through a web proxy using [http://www.agroman.net/corkscrew/ corkscrew].
−
{{code|<bash>
+
<source lang="bash">
 
ssh -oProxyCommand='corkscrew local_web_proxy proxy_port %h %p' username@remotehost
 
ssh -oProxyCommand='corkscrew local_web_proxy proxy_port %h %p' username@remotehost
−
</bash>}}
+
</source>
  
 
*[http://en.wikipedia.org/wiki/Port_forwarding Port_forwarding (Wikipedia)]
 
*[http://en.wikipedia.org/wiki/Port_forwarding Port_forwarding (Wikipedia)]
Line 222: Line 262:
 
== General ==
 
== General ==
 
=== Add and remove startup items ===
 
=== Add and remove startup items ===
−
{{code|<bash> update-rc.d ITEM defaults </bash>}}
+
<source lang="bash">
 +
update-rc.d ITEM defaults
 +
</source>
  
−
{{code|<bash>update-rc.d -f ITEM remove</bash>}}
+
<source lang="bash">
 +
update-rc.d -f ITEM remove
 +
</source>
  
 
== See also ==
 
== See also ==

Revision as of 18:10, 22 May 2015

Info.svg This is the start of a list of useful Linux shell commands (generally "one liners") we use a lot


Ubuntu package management

Searching for an installed package

Use dpkg and grep; 

dpkg -l | grep java

will list all packages with java in the name

Searching for installable packages

apt-cache performs a variety of operations on APT´s package cache 

apt-cache search java

lists all packages in the cache with java in the name that are installable

SSH & SCP

tar pipe

## From a local to remote machine
tar -zcvf - directory | ssh remote 'cd somewhere; tar -zxvf -'
## From remote machine to local machine
ssh remote 'cd somewhere; tar -zcvf - directory' | tar -zxf -
## remote tar and pipe to tgz
ssh remote 'cd somewhere; tar -zcf - directory' | cat - > directory.tar.gz

Example of sending a file to a remote server: 

scp -P 2222 /my/local/file.txt USER@example.com:/home/USER

Example of retrieving a file from a remote server: 

scp -P 2222 USER@example.com:/home/USER/file.txt ./

Note using port forwarding for commonly accessed sites you can define non standard ports in your .ssh/config so you never need to explicitly state them above.

Recovering from an interrupted transfer

This ability is included natively in Linux and used to be very useful for splitting large backups up so they could fit onto small media such as floppy disks. But there's one time when it's very useful nowadays too which is when a large file transfer gets interupted and there's no option for continuation such as when using SCP.

When this happens, go onto the target machine, check how many bytes were transferred of the target file and rename it to "xaa". Then on the local machine, do the following command to split the source file into two parts, the first being of the size of the number of bytes that have already been transferred (in this example 1234567 bytes). 

split -b 1234567 foo.tar.gz

The resulting files are called "xaa" and "xab", and the latter is the containing the remaining content that needs to be transferred to the target server. Once you've transferred it across, you can then join the two files (remember you renamed the first part to "xaa" so there's no need to transfer that) together using cat as follows, and then remove all the xa* files from noth source and target servers. 

cat xa* > foo.tar.gz

Files & Devices

Get the size of a directory and its contents

du -sh /home/foo


Use this more specific version to find the size of a users Maildir folder: 

du -sh /home/*/Maildir|sed 's|/home/||'|sed 's|/Maildir||'

Search for file content recursively

Here's an example looking for a phrase within a specific file type recursively through a directory structure, and printing the file names and line numbers of the resulting matches. 

grep -rn "Foo" *.php

There are other tips at stackoverflow.com.

Search for file content recursively and tar

## Find and tar
find . -name "*.R" -print0 | xargs -0 tar -cvf Rfiles.tar
## check contents
tar -tvf Rfiles.tar

## Find and tar.gz
find . -name "*.R" -print0 | xargs -0 tar -zcvf Rfiles.tar.gz
## check contents
tar -ztvf Rfiles.tar.gz

Search for file content recursively and long list

find . -name "*.R" -ls

Count occurrences of a word in a file or files

grep -roh "WORD" file*.txt | wc -

Search and replace content in files

You could also use find and sed, but I find that this little line of perl works nicely. 

perl -pi -w -e 's/SEARCH/REPLACE/g;' *.php
  • -e means execute the following line of code.
  • -i means edit in-place
  • -w write warnings
  • -p loop

EXTS="7z t7z"

Mass renaming

mmv is a mass move/copy/renaming tool that uses standard wildcards to perform its functions. According to the manual the “;” wildcard is useful for matching files at any depth in the directory tree (ie it will go below the current directory, recursively).

Example
mmv \*.JPG \#1.jpg

The first pattern matches anything with a “.JPG” and renames each file (the “#1” matches the first wildcard) to “.jpg”. Each time you use a \(wildcard) you can use a #x to get that wildcard. Where x is a positive number starting at 1. Copied off: http://tldp.org/LDP/GNU-Linux-Tools-Summary/html/mass-rename.html

Compare two directory structures for differences

This is very handy if you need to know if two directory structures are the same or not including file content. It uses the diff command with the r switch to work recursively and the x switch to allow skipping of the .svn directories. 

diff -qrx .svn DIR1 DIR2

Mount a .iso

See this HOWTO

Mount a USB stick

If you don't know the device name of the stick, plug it in and find it with the following: 

dmesg |grep SCSI

Then mount the first partition, 

mkdir -p ~/memstick
mount -t vfat -o rw,users /dev/sdX1 ~/memstick


Image manipulation

Resizing JPG's and changing quality setting

The first line shows how to reduce and image to 25% and quality to 50% adding "_resized" to the results filename. The second command uses Perl to apply this same command to all JPG's in the current directory. 

convert foo.jpg -resize 25% -quality 50% foo_resized.jpg

perl -e 'for (glob "*.jpg") { $img = $_; s/(....)$/_resized$1/; qx "convert \'$img\' -resize 25% -quality 50% \'$_\'"; }'

Apply an opaque background of a specified colour to a directory of transparent PNG's

  • This command requires ImageMagick to be installed
  • It loops through all PNG's in the CWD and puts them in a directory called processed which must exist
perl -e 'qx "convert $_ -background #ff00ff -flatten foo/$_" for glob "*.png"'

Audio conversion

The following converts a .wav file to an mp3: 

ffmpeg -i test.wav test.mp3

To do a whole directory you could do this: 

perl -e 'qx "ffmpeg -i $_ $_.mp3" for glob "*.wav"'

Video manipulation

Use the following commands to extract a small snippet out of a video (the -ar switch is only needed for outputting to flv I think). 

ffmpeg -i "foo.avi" -ss 00:10:10 -t 00:00:05 -ar 22050 "foo.flv"

See also

System monitoring

List the top 10 memory consuming processes 

ps -auxf | sort -nr -k 4 | head -10


List the top 10 CPU consuming processes 

ps -auxf | sort -nr -k 3 | head -10

Network commands

Restart the network after changing configuration

/etc/init.d/networking restart

List all the listening sockets and their ports and programs

netstat -lp

Get current default gateway

netstat -nr

The default gateway is on the last line, it should have the U and G flags set

Release DHCP lease

dhclient -r

Obtain a new DHCP lease

dhclient

Scan a local subnet for active IP addresses

nmap -sP 192.168.1.0/24

Get the MAC address and hostname of an IP on the current subnet

arp -a 192.168.1.1

Internet

Download a copy of an entire site

This will download an entire site, if the site is already downloaded, then only newer files are transferred. 

wget -m http://foo.bar

Port forwarding

Port forwarding allows a remote client to gain access to a network so an intranet can be accessed. 

ssh -fN -L[PORT]:appserver:[PORT] username@sshdserver

Then point your webbrowsers proxy server to appserver:[PORT] and access the intranet etc.

To subvert a firewalled environment where outgoing ssh is allowed. 

$ ssh -D 9000 username@remotehost

Then point your web browser to a SOCKS proxy @ localhost:9000

Further it's possible to get ssh through a web proxy using corkscrew. 

ssh -oProxyCommand='corkscrew local_web_proxy proxy_port %h %p' username@remotehost

General

Add and remove startup items

update-rc.d ITEM defaults
update-rc.d -f ITEM remove

See also

Retrieved from "https://organicdesign.nz/wiki/index.php?title=Linux_commands&oldid=119812"