Python Resources

https://python.zeef.com/luis.solis

Some Useful One Liner in Shell scripting

 Find out Top 10 Most Used Commands.

cat ~/.bash_history | tr "\|\;" "\n" | sed -e "s/^ //g" | cut -d " " -f 1 | sort | uniq -c | sort -n | tail -n 15
11 ssh
12 shutdown
15 cp
15 vncserver
22 cat
23 find
23 pwd
24 mv
25 ovc
47 grep
58 ps
67 vi
74 ll
117 ls
118 cd

Find out Top 10 Largest File or Directory Using du, sort and head

# du -sk /var/log/* | sort -r -n | head -10
1796    /var/log/audit
1200    /var/log/sa
612     /var/log/anaconda.log
512     /var/log/wtmp
456     /var/log/messages.4
92      /var/log/messages.2
76      /var/log/scrollkeeper.log
72      /var/log/secure
56      /var/log/cups
48      /var/log/messages.1

Find out which process is using up your memory using ps, awk, sort



$ ps aux | awk '{if ($5 != 0 ) print $2,$5,$6,$11}' | sort -k2n
PID VSZ RSS COMMAND
3823 3788 484 /sbin/mingetty
3827 3788 484 /sbin/mingetty
3830 3788 484 /sbin/mingetty
3833 3788 488 /sbin/mingetty
3834 3788 484 /sbin/mingetty
3873 3788 484 /sbin/mingetty
2173 3796 568 /usr/sbin/acpid
1835 3800 428 klogd
1832 5904 596 syslogd
2054 5932 540 /usr/sbin/sdpd
2281 6448 360 gpm

Find List of Unique Words in a file Using tr, sed, uniq



cat Readme1.txt
Installtion Steps:

Run the below steps as root user:

1.Copy the output file at any temporary location.
2.Unzip the file using unzip command.
3.Copy the tar to data directory
tr -c a-zA-Z '\n' < Readme1.txt  | sed '/^$/d' | sort | uniq -i -c
1 any
1 as
1 at
1 below
1 command
2 Copy
1 data
1 directory
2 file
1 Installtion
1 location
1 output
1 root
1 Run
2 steps
1 tar
1 temporary
4 the
1 to
2 unzip
1 user
1 using
$


Display Username and UID sorted by UID Using cut, sort and tr

cut -d ':' -f 1,3 /etc/passwd | sort -t ':' -k2n - | tr ':' '\t'
root    0
daemon    1
bin    2
sys    3
sync    4
games    5
man    6
lp    7
mail    8
news    9
uucp    10
proxy    13

IP Packet Filtering: iptables Explained For Beginners

iptables is a IP Filter which is shipped with kernel. Technically speaking, an IP filter will work on 

Network layer in TCP/IP stack but actually iptables work on data link and transport layer as well. 

In a broad sense, iptables consists on tables, which consists of chain which is further comprised 

of rules.

Default tables are: 

1. Raw
2. Mangle
3. NAT
4. Filter

Default chains are (yes, they are written in upper case):

1. PREROUTING: used by raw, mangle and nat tables
2. INPUT: used by mangle and filter tables
3. FORWARD: used by mangle and filter tables
4. OUTPUT: used by raw, mangle, nat and filter tables
5. POSTROUTING: used by mangle and nat tables

I'll discuss about Filter table here. It is the one which is most generally used but if you are 

interested in others as well you can find a detailed tutorial at frozentux. Filter table uses three

 chains, INPUT, FORWARD and OUTPUT. 

• INPUT chain is for the packets meant for your own local machine. Reply of a http request 
• made by your browser will go through INPUT chain.
• OUTPUT chain is for the packets going out of your machine. The http request made by your browser will go through this chain.
• FORWARD chain is for the packets which you receive but they are not meant for you. Your machine is just supposed to forward them to another device. This generally happens 
• when the machine is configured as a gateway or something similar.

Now every iptables rules have some "target" which is executed when it is matched against a 

"criteria". Following are the most common targets:

• ACCEPT: Packet is accepted and goes to the application for processing.
• DROP: Packet is dropped. No information regarding the drop is sent to the sender.
• REJECT: Packet is dropped and information (error) message is sent to the sender.
• LOG: Packet details are sent to syslogd for logging. 
• DNAT: Rewrites the destination IP of the packet
• SNAT: Rewrites the source IP of the packet

First four are used in Filter tables a lot. Now let us discuss some of the common criteria:

• -p <protocol>: It matches protocols like tcp, udp, icmp and all
• -s <ip_addr>: It matches source IP address
• -d <ip_addr>: It matches destination IP address
• --sport <port>: It matches the source port
• --dport <port>: It matches the destination port
• -i <interface>: It matches the interface from which the packet entered
• -o <interface>: It matches the interface from which the packet exits

Now we know the basic things to start building our rules. Let us try to write some rules 

for a few hypothetical (or real) situations. First we'll set default policy for iptables filter

table using -P flag.

iptables -t filter -P INPUT DROP

iptables -t filter -P OUTPUT DROP 

Now we'll allow this machine to send only http requests and ssh requests:

iptables -A INPUT -p tcp -i eth0 --dport 80

iptables -A INPUT -p tcp -i eth0 --dport 22 

Note that -A flag is used because we want to append these rules to current iptables config. 

If we do not use -A then the rules will be overwritten.

Also assume that my machine acts as an ftp server so I should allow connections to port 20 and 21.

iptables -A OUTPUT -p tcp -i eth0 --sport 20

iptables -A OUTPUT -p tcp -i eth0 --sport 21 

Also I would say that I trust my internal network:

iptables -A OUTPUT -j ACCEPT -p all -d 192.168.1.0/24 -o eth0

iptables -A INPUT -j ACCEPT -p all -s 192.168.1.0/24 -i eth0

So this is how my final iptables script would look like:

iptables -t filter -P INPUT DROP

iptables -t filter -P OUTPUT DROP

iptables -A INPUT -p tcp -i eth0 --dport 80

iptables -A INPUT -p tcp -i eth0 --dport 22

iptables -A OUTPUT -p tcp -i eth0 --sport 20

iptables -A OUTPUT -p tcp -i eth0 --sport 21

iptables -A OUTPUT -j ACCEPT -p all -d 192.168.1.0/24 -o eth0

iptables -A INPUT -j ACCEPT -p all -s 192.168.1.0/24 -i eth0 

Note that rules are read sequentially. So other than policy rules, if any rule matches

 then reading will stop and action will be taken according to that rule.

Everything Is a File

Everything Is a File
---------------------------
Let me begin this by a famous UNIX statement which emphasizes the main underlying working model of Linux operating system. The statement is that "Everything Is a File".

If you are one of that person who spends a lot time with Linux or any NIX system, then you might have already understood that statement. Each and everything in Linux is accessed by a file. To make that much more clear, even block devices like Hard disks,and CD/DVD's are nothing other than a file(yeah but a special file..but a file).

Linux Boot Process

Linux Boot Process 

Soft Link/Hard Link

We are using the 'pwd' and 'ls' command in our everyday life. Here I will discuss some interesting options we rarely use with them. If ur in doubt at any point please refer the man pages.

pwd command:
-------------------------------------
Basically it prints the present working dir.

Now lets use pwd with some useful options.

[root@localhost tmp]# ln -s /tmp/test/ temp

[root@localhost tmp]# ls -l temp
lrwxrwxrwx. 1 root root 10 Jul 19 04:40 temp -> /tmp/test/

Here I created a symnolic lik named 'temp' for the folder '/tmp/test'.

For more info on symlinks and hardlinks, please refer
--
http://www.geekride.com/hard-link-vs-soft-link/
--

[root@localhost tmp]# cd temp/

-L (logical) Use PWD from environment, even if it contains symbolic links

[root@localhost temp]# /bin/pwd -L
/tmp/temp

-P (physical) Avoid all symbolic links

[root@localhost temp]# /bin/pwd -P
/tmp/test

that with -L option it print working directory from environment even if it contains symlinks. So it prints the current dir i.e /tmp/temp. However, -P print actual physical current working directory by resolving all symbolic links i.e /tmp/test.

ls
===
Basically it list the files in the dir.

When using -F an asterisk (*) means the file is executable, at sign (@) indicates a symbolic link, an equal sign (=) is a socket, and a pipe or vertical bar (|) is a FIFO. You can refer the following link to know more about 'FIFO'

==
http://man7.org/linux/man-pages/man7/fifo.7.html
==

[root@localhost ~]# ls -F /var/run/cups/cups.sock
/var/run/cups/cups.sock=
[root@localhost ~]# touch test

[root@localhost ~]# chmod +x test
[root@localhost ~]# ls -F test
test*

[root@localhost ~]# ln -s testing test
[root@localhost ~]# ls -F test
test@

[root@localhost ~]# mkfifo a=rw MYFIFO
[root@localhost ~]# ls -F MYFIFO
MYFIFO|

-S option is used for sorting file by size.

[root@localhost ~]# ls -lS
total 68
-rw-r--r--. 1 root root 41433 Jun 17 17:49 install.log
-rw-r--r--. 1 root root 9154 Jun 17 17:47 install.log.syslog
drwxr-xr-x. 2 root root 4096 Jul 19 04:32 dir
-rw-------. 1 root root 3332 Jun 17 17:49 anaconda-ks.cfg
lrwxrwxrwx. 1 root root 7 Jul 19 04:21 test -> testing
prw-r--r--. 1 root root 0 Jul 19 04:24 a=rw
prw-r--r--. 1 root root 0 Jul 19 04:24 MYFIFO

ls -a shows hidden files. However, 'ls -a .*' doesn't work as indended.
Note
-----
* matches any string, including the null string, ? matches any single character. Characters enclosed in [ ] specify a list or range of characters, any of which will match.

[root@localhost ~]# mkdir example
[root@localhost ~]# cd example/
[root@localhost example]# touch .foo ..foo ...foo

[root@localhost ~]# cd example/
[root@localhost example]# ls .*
.foo ..foo ...foo

.:

..:
anaconda-ks.cfg dir install.log MYFIFO
a=rw example install.log.syslog test

Now here is the problem.

The problem is that .* is expanded to include . and .., which are then both displayed. Instead of getting just the dot files in the current directory, you get those files, plus all the files and directories in the current directory (.), all the files and directories in the parent directory (..), and the names and contents of any subdirectories in the current directory that start with a dot. This is very confusing. Here it prints the contents in 'example' dir as well as its parent dir.

let try with the option 'd' with 'ls'

[root@localhost example]# ls -d .*
. .. .foo ..foo ...foo

Now we will match file only starting with '.'

[root@localhost example]# ls -d .[!.]*
.foo

You can verify the same the echo command

[root@localhost example]# echo .[!.]*
.foo

[root@localhost example]#
.[!.]*
is a filename expansion pattern where [] denotes a list of characters to match, but the leading ! negates the list. So we are looking for a dot, followed by any character that is not a dot, followed by any number of any characters

Similarly

[root@localhost example]# ls -d .??f*
...foo
[root@localhost example]# ls -d .?f*
..foo
[root@localhost example]# ls -d ..?f*
...foo
[root@localhost example]#

You can use echo * as an emergency substitute for ls if the com-
mand is corrupt or not available for some reason. This works because is expanded by the shell to everything in the current directory, which results in a list similar to what you’d get with ls.

I have only included a few options. If ur curious enough, u can always refer the man page for these commands.

For any suggestions,questions,problems regarding this post, please feel free to share.

Linux df Command Usage Examples

Linux df command can be used to display disk usage statistics for the file systems present on the Linux system. It’s handy tool to know which filesystem is consuming how much memory. Also, if a particular filename is picked up and supplied as argument to df command then it displays the disk usage statistics for the file system on which the file resides. This command can be used by the system administrators to know the disk usage status of various file systems on Linux so that proper clean-up and maintenance of the Linux system can be performed. The df command provides various options through which the output can be customized in a way that is most suited to the user.

Syntax
=======

Before jumping on to the examples, lets first take a look on how to use the df command. Here is the syntax information of df command from the man page:

df [OPTION]... [FILE]...

So we see that the df command does not require any mandatory argument. The OPTION and FILE arguments are non-mandatory. While the OPTION argument tells the df command to act in a way as specified by the definition of that OPTION, the FILE argument tells the df command to print disk usage of only that file system on which the FILE resides.

Examples
1. Basic example

Here is how the df command can be used in its most basic form.

# df
Filesystem 1K-blocks Used Available Use% Mounted on
/dev/sda6 29640780 4320704 23814388 16% /
udev 1536756 4 1536752 1% /dev
tmpfs 617620 888 616732 1% /run
none 5120 0 5120 0% /run/lock
none 1544044 156 1543888 1% /run/shm

In the output above, the disk usage statistics of all the file systems were displayed when the df command was run without any argument.

The first column specifies the file system name, the second column specifies the total memory for a particular file system in units of 1k-blocks where 1k is 1024 bytes. Used and available columns specify the amount of memory that is in use and is free respectively. The use column specifies the used memory in percentage while the final column ‘Mounted on’ specifies the mount point of a file system.

2. Get the disk usage of file system through a file

As already discussed in the introduction, df can display the disk usage information of a file system if any file residing on that file system is supplied as an argument to it.

Here is an example:

# df test
Filesystem 1K-blocks Used Available Use% Mounted on
/dev/sda6 29640780 4320600 23814492 16% /

Here is another example:

# df groff.txt
Filesystem 1K-blocks Used Available Use% Mounted on
/dev/sda6 29640780 4320600 23814492 16% /

We used two different files (residing on same file system) as argument to df command. The output confirms that the df command displays the disk usage of file system on which a file resides.
3. Display inode information

There exists an option -i through which the output of the df command displays the inode information instead of block usage.

For example:

# df -i
Filesystem Inodes IUsed IFree IUse% Mounted on
/dev/sda6 1884160 261964 1622196 14% /
udev 212748 560 212188 1% /dev
tmpfs 216392 477 215915 1% /run
none 216392 3 216389 1% /run/lock
none 216392 8 216384 1% /run/shm

As we can see in the output above, the inode related information was displayed for each filesystem.

4. Produce a grand total

There exists an option –total through which the output displays an additional row at the end of the output which produces a total for every column.

Here is an example:

# df --total
Filesystem 1K-blocks Used Available Use% Mounted on
/dev/sda6 29640780 4320720 23814372 16% /
udev 1536756 4 1536752 1% /dev
tmpfs 617620 892 616728 1% /run
none 5120 0 5120 0% /run/lock
none 1544044 156 1543888 1% /run/shm
total 33344320 4321772 27516860 14%

So we see that the output contains an extra row towards the end of the output and displays total for each column.
5. Produce output in human readable format

There exists an option -h through which the output of df command can be produced in a human readable format.

Here is an example:

# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/sda6 29G 4.2G 23G 16% /
udev 1.5G 4.0K 1.5G 1% /dev
tmpfs 604M 892K 603M 1% /run
none 5.0M 0 5.0M 0% /run/lock
none 1.5G 156K 1.5G 1% /run/shm

So we can see that the output displays the figures in form of ‘G’ (gigabytes), ‘M’ (megabytes) and ‘K’ (kilobytes). This makes the output easy to read and comprehend and thus makes is human readable. Note that the name of the second column is also changed to ‘size’ in order to make it human readable.