Unix Job Interview Questions

Adding 30 days to current date using PERL inside SHELL

#! /usr/bin/ksh

dt1=`perl -l -e '$date=(localtime(time + (30 * 24 * 60 *60))); print $date;'`

dt=`echo $dt1|awk '{if($3<10) print $1" "$2" 0"$3" "$4" "$5;else print $0}'`

echo $dt|awk '{print $5"-"$2"-"$3"23:59:59"}'|sed -e 's/Jan/01/g'\;'s/Feb/02/g'\;'s/Mar/03/g'\;'s/Apr/04/g'\;'s/May/05/g'\;'s/Jun/06/g'\;'s/Jul/07/g'\;'s/Aug/08/g'\;'s/Sep/09/g'\;'s/Oct/10/g'\;'s/Nov/11/g'\;'s/Dec/12/g'





Print 3 lines before pattern match using sed and awk ..

Q: How do you display your running kernel version? (Solaris, AIX, Linux)
A: Linux # uname –r , Solaris # showrev

Q: Which command do you use to display a table of running processes? (Solaris, AIX, Linux)
A: Linux # ps –ef and top , Solaris # prstat

Q: Which file do you modify to configure a domain name resolver? (Solaris, AIX, Linux)
A: Linux # /etc/resolv.conf , Solaris # /etc/resolv.conf

Q: Which file contains a list of locally defined hostnames and corresponding IP addresses? (Solaris, AIX, Linux)
A: Linux # /etc/hosts , Solaris # /etc/hosts and linked file /etc/inet/hosts

Q: How do you display a routing table? (Solaris, AIX, Linux)
A: Linux # ip route show or #netstat –nr or #route –n and Solaris # netstat –nr and #route -n

Q: Which command would you use to view partitions and their sizes on Solaris?
A: # df -kh

Q: Which OpenBoot command would you use to print/view OpenBoot environment variables on a SUN server?
A: #printenv

Q: What does "ypwhich" command do? (Solaris, AIX, Linux)
A: # Will display NIS server to which client is connected to and which NIS Server is master for particular map specified with this command

Q: which command would you use to create an OS user on Solaris and Linux?
A: Linux # useradd and Solaris #useradd

Q: Which file contains passwords for local users on Solaris, Linux and on AIX?
A: Linux #/etc/shadow and Solaris # /etc/shadow

Q: Which command would you use to list partitions on Linux?
A: Linux # mount –l or # df -kh

Q: Which command/commands would you use to manage installed packages on RedHat Linux?
A: Linux # rpm

Q: What is the default port for SSH server?
A: 22

Q: Which command/commands would you use to manage installed packages on Solaris?
A: #pkginfo #pkgrm # pkgadd #pkgchk

Q: What command would you use to install an OS patch on Solaris?
A: #showrev –p and #patchadd -p

Q: Which Veritas command would you use to display a list of Veritas volumes?
A: # vxprint

Q: Which Veritas command would you use to display a list of disks on a system?
A: # vxdx list

Q: What is the main system configuration utility in AIX?
A:

Q: Which file has a list of filesystems to be mounted at boot time on Solaris, Linux and AIX?
A: Linux # /etc/fstab and Solaris #/etc/vfstab

Q. How to change all .my files to .txt in current directory??
Solution: for i in `ls *.my` ; do NEW=`echo $ised -e 's/.my/.txt/g'` ; mv $i $NEW ; done

Q. How to change all .extn files to .my in entire directory recursively??
Solution:
find ./ -type f grep ".extn$" >files.txt;for i in `cat files.txt`;do NEW=`echo $ised 's/.extn/.my/g'`;mv "${i}" "${NEW}";done;rm -f files.txt

Q: What is umask??
Answer: umask is used to set default permission level in entire unix system. It can be set in .kshrc/.bashrc/.cshrc or init (So that to load at the time of login into shell)

Q. What is inode??
Answer : inode represents any file/directory residing in your unix system with a numner known as inode number. It is unique for every file/dir. It is basically a reference that kernel use for any work related to that entity. "ls -i filename" will give you the inode number of that particu;ar file.

Q. Explain general commands like ps,df, export, env, ufsdump, tar, cron, system.
Answer: EASY ONE

Q. Explain the make utility in unix?? what are the parameters of a makefile and how to use them. What are macros in makefile.
Answer: Unix make utility is used to build a software package. It requires a makefile where all the bunch of the scripts/codes/programs are declared/defined. The main parameters of the make file are variable declaration, Macro declaration, Phony Target declaration and Clean-up block. Macros in a makefile represents the order in which the scripts would get executed.

Q. What are the benefits of makefile over shell script??
Answer: Using makefile one can run the different scripts or programs or codes in one go. This can also be done by making a wrapper shell script which can execute the different scripts in one go. But then it would not be termed as a package because when we say a package it means that we have one installer (.exe) which can execute the bunch more efficiently and also this .exe will be in encrypted format. So in order to maintain efficiency and security makefile is given precedence over wrapper shell scripts.

Q. What are hard links in UNIX??
Answer: A hard link is a reference to a file or directory that appears just like a file or directory, not a link. Hard links only work within a filesystem. In other words, don't use hard links between mounted filesystems. A hard link is only a reference to the original file, not a copy of the file. If the original file is deleted, the information will be lost.
To create a hard link of the file /export/home/fred/stuff to /var/tmp/thing, use:

ln /export/home/fred/stuff /var/tmp/thing

The syntax for creating a hard link of a directory is the same. To create a hard link of /var/www/html to /var/www/webroot, use:

ln /var/www/html /var/www/webroot

Q. What are the soft/symbolic links in unix ??
Answer: A symbolic link is a pointer to another file or directory. It can be used just like the original file or directory. A symbolic link appears in a long listing (ls -l) with a reference to the original file/directory. A symbolic link, as opposed to a hard link, is required when linking from one filesystem to another and can be used within a filesystem as well.
To create a symbolic link, the syntax of the command is similar to a copy or move command: existing file first, destination file second. For example, to link the directory /export/space/common/archive to /archive for easy access, use:

ln -s /export/space/common/archive /archive

To link the runtime control script /etc/init.d/httpd to /etc/rc2.d/S77httpd, use:

cd /etc/rc2.d
ln -s ../init.d/httpd S77httpd

Q. What is a command to kill the last background job??
Answer: kill $!

Q. Bring a job to foreground by specifying its job number after the percent sign.
Answer: fg %jobnumber
(Note: jobnumber can be obtained by using jobs -l command)

Q. Explain top command.
Answer: top provides an ongoing look at processor activity in real time. It displays a listing of the most CPU-intensive tasks on the system, and can provide an interactive interface for manipulating processes. It can sort the tasks by CPU usage, memory usage and runtime. can be better configured than the standard top from the procps suite. Most features can either be selected by an interactive command or by specifying the feature in the personal or system-wide configuration file.

Q. Write a script which give you those records which are having CPU utilization more than 80%.
Answer: df -k awk 'NR==1{print $1"\t"$5"\t"$6}sub("%","",$5){if($5 >= 80) print $1"\t"$5"%\t"$NF}'

Q. How to pass arguments in an awk script??Answer: Using, cat a.txtawk -v awk_var="$shell_var" '{print $1 "\t" awk_var}'
e.g. #! /usr/bin/sh
var=$*
cat ussd.txt >offpeak_input.txt
cat offpeaknawk -v var1="$var" '{print "\"1-"$1"\"""""\""$2"\"""""1006"""""$3"""0""""-1""""-1""""\""var1" .Account balance is Rs..Bonus minutes.\""}'>>offpeak_input.txt

Useful awk oneliners

FILE SPACING:
# double space a file
awk '1;{print ""}' awk 'BEGIN{ORS="\n\n"};1'

# double space a file which already has blank lines in it. Output file should contain no more than one blank line between lines of text. NOTE: On Unix systems, DOS lines which have only CRLF (\r\n) are # often treated as non-blank, and thus 'NF' alone will return TRUE.
awk 'NF{print $0 "\n"}'
# triple space a file
awk '1;{print "\n"}'
NUMBERING AND CALCULATIONS:
# precede each line by its line number FOR THAT FILE (left alignment). Using a tab (\t) instead of space will preserve margins.
awk '{print FNR "\t" $0}' files*

# precede each line by its line number FOR ALL FILES TOGETHER, with tab.
awk '{print NR "\t" $0}' files*
# number each line of a file (number on left, right-aligned) # Double the percent signs if typing from the DOS command prompt.
awk '{printf("%5d : %s\n", NR,$0)}'
# number each line of file, but only print numbers if line is not blank Remember caveats about Unix treatment of \r (mentioned above)
awk 'NF{$0=++a " :" $0};{print}' awk '{print (NF? ++a " :" :"") $0}'
# count lines (emulates "wc -l")
awk 'END{print NR}'

# print the sums of the fields of every line
awk '{s=0; for (i=1; i<=NF; i++) s=s+$i; print s}'
# add all fields in all lines and print the sum
awk '{for (i=1; i<=NF; i++) s=s+$i}; END{print s}' # print every line after replacing each field with its absolute value awk '{for (i=1; i<=NF; i++) if ($i < i =" -$i;" i="1;" i =" ($i">
# print the total number of fields ("words") in all lines
awk '{ total = total + NF }; END {print total}' file

# print the total number of lines that contain "Beth"
awk '/Beth/{n++}; END {print n+0}' file
# print the largest first field and the line that contains it Intended for finding the longest string in field 1
awk '$1 > max {max=$1; maxline=$0}; END{ print max, maxline}'

# print the number of fields in each line, followed by the line
awk '{ print NF ":" $0 } '

# print the last field of each line
awk '{ print $NF }'

# print the last field of the last line
awk '{ field = $NF }; END{ print field }'

# print every line with more than 4 fields
awk 'NF > 4'

# print every line where the value of the last field is > 4
awk '$NF > 4'

TEXT CONVERSION AND SUBSTITUTION:
# IN UNIX ENVIRONMENT: convert DOS newlines (CR/LF) to Unix format
awk '{sub(/\r$/,"");print}' # assumes EACH line ends with Ctrl-M

# IN UNIX ENVIRONMENT: convert Unix newlines (LF) to DOS format
awk '{sub(/$/,"\r");print}

# IN DOS ENVIRONMENT: convert Unix newlines (LF) to DOS format
awk 1

# IN DOS ENVIRONMENT: convert DOS newlines (CR/LF) to Unix format # Cannot be done with DOS versions of awk, other than gawk:
gawk -v BINMODE="w" '1' infile >outfile

# Use "tr" instead.
tr -d \r outfile # GNU tr version 1.22 or higher

# delete leading whitespace (spaces, tabs) from front of each line # aligns all text flush left
awk '{sub(/^[ \t]+/, ""); print}'

# delete trailing whitespace (spaces, tabs) from end of each line
awk '{sub(/[ \t]+$/, "");print}'

# delete BOTH leading and trailing whitespace from each line
awk '{gsub(/^[ \t]+[ \t]+$/,"");print}' awk '{$1=$1;print}' # also removes extra space

# insert 5 blank spaces at beginning of each line (make page offset)
awk '{sub(/^/, " ");print}'

# align all text flush right on a 79-column width
awk '{printf "%79s\n", $0}' file*

# center all text on a 79-character width
awk '{l=length();s=int((79-l)/2); printf "%"(s+l)"s\n",$0}' file*

# substitute (find and replace) "foo" with "bar" on each line
awk '{sub(/foo/,"bar");print}' # replaces only 1st instance
gawk '{$0=gensub(/foo/,"bar",4);print}' # replaces only 4th instance
awk '{gsub(/foo/,"bar");print}' # replaces ALL instances in a line
# substitute "foo" with "bar" ONLY for lines which contain "baz"
awk '/baz/{gsub(/foo/, "bar")};{print}'
# substitute "foo" with "bar" EXCEPT for lines which contain "baz" awk '!/baz/{gsub(/foo/, "bar")};{print}'
# change "scarlet" or "ruby" or "puce" to "red"
awk '{gsub(/scarletrubypuce/, "red"); print}'
# reverse order of lines (emulates "tac")
awk '{a[i++]=$0} END {for (j=i-1; j>=0;) print a[j--] }' file*
# if a line ends with a backslash, append the next line to it (fails if there are multiple lines ending with backslash...)
awk '/\\$/ {sub(/\\$/,""); getline t; print $0 t; next}; 1' file*
# print and sort the login names of all users
awk -F ":" '{ print $1 "sort" }' /etc/passwd
# print the first 2 fields, in opposite order, of every line
awk '{print $2, $1}' file
# switch the first 2 fields of every line
awk '{temp = $1; $1 = $2; $2 = temp}' file
# print every line, deleting the second field of that line
awk '{ $2 = ""; print }'
# print in reverse order the fields of every line
awk '{for (i=NF; i>0; i--) printf("%s ",i);printf ("\n")}' file
# remove duplicate, consecutive lines (emulates "uniq")
awk 'a !~ $0; {a=$0}'
# remove duplicate, nonconsecutive lines
awk '! a[$0]++' # most concise script
awk '!($0 in a) {a[$0];print}' # most efficient script
# concatenate every 5 lines of input, using a comma separator between fields
awk 'ORS=%NR%5?",":"\n"' file

SELECTIVE PRINTING OF CERTAIN LINES:
# print first 10 lines of file (emulates behavior of "head")
awk 'NR <>1{exit};1'
# print the last 2 lines of a file (emulates "tail -2")
awk '{y=x "\n" $0; x=$0};END{print y}'
# print the last line of a file (emulates "tail -1")
awk 'END{print}'
# print only lines which match regular expression (emulates "grep")
awk '/regex/'
# print only lines which do NOT match regex (emulates "grep -v")
awk '!/regex/'
# print the line immediately before a regex, but not the line # containing the regex
awk '/regex/{print x};{x=$0}' awk '/regex/{print (x=="" ? "match on line 1" : x)};{x=$0}'
# print the line immediately after a regex, but not the line containing the regex
awk '/regex/{getline;print}'
# grep for AAA and BBB and CCC (in any order)
awk '/AAA/; /BBB/; /CCC/'
# grep for AAA and BBB and CCC (in that order)
awk '/AAA.*BBB.*CCC/'
# print only lines of 65 characters or longer
awk 'length > 64'
# print only lines of less than 65 characters
awk 'length < nr="="8,NR="="12'" nr="="52'" nr="="52">
CREDITS AND THANKS:http://student.northpark.edu/pemente/awk/awk1line.txt

At below link the famous awk one liners by Eric Pement is explained.
http://www.catonmat.net/blog/awk-one-liners-explained-part-one/

Useful sed one liners

FILE SPACING:
# double space a file
sed G
# double space a file which already has blank lines in it. Output file should contain no more than one blank line between lines of text.
sed '/^$/d;G'
# triple space a file
sed 'G;G'
# undo double-spacing (assumes even-numbered lines are always blank)
sed 'n;d'
# insert a blank line above every line which matches "regex"
sed '/regex/{x;p;x;}'
# insert a blank line below every line which matches "regex"
sed '/regex/G'
# insert a blank line above and below every line which matches "regex"
sed '/regex/{x;p;x;G;}'
NUMBERING:
# number each line of a file (simple left alignment). Using a tab (see # note on '\t' at end of file) instead of space will preserve margins.
sed = filename sed 'N;s/\n/\t/'
# number each line of a file (number on left, right-aligned)
sed = filename sed 'N; s/^/ /; s/ *\(.\{6,\}\)\n/\1 /'
# number each line of file, but only print numbers if line is not blank
sed '/./=' filename sed '/./N; s/\n/ /'
# count lines (emulates "wc -l")
sed -n '$='

TEXT CONVERSION AND SUBSTITUTION:
# IN UNIX ENVIRONMENT: convert DOS newlines (CR/LF) to Unix format
sed 's/.$//' # assumes that all lines end with CR/LF
sed 's/^M$//' # in bash/tcsh, press Ctrl-V then Ctrl-M
sed 's/\x0D$//' # gsed 3.02.80, but top script is easier

# IN UNIX ENVIRONMENT: convert Unix newlines (LF) to DOS format
sed "s/$/`echo -e \\\r`/" # command line under ksh
sed 's/$'"/`echo \\\r`/" # command line under bash
sed "s/$/`echo \\\r`/" # command line under zsh
sed 's/$/\r/' # gsed 3.02.80

# IN DOS ENVIRONMENT: convert Unix newlines (LF) to DOS format
sed "s/$//" # method 1
sed -n p # method 2
# IN DOS ENVIRONMENT: convert DOS newlines (CR/LF) to Unix format Can only be done with UnxUtils sed, version 4.0.7 or higher. Cannot be done with other DOS versions of sed. Use "tr" instead.
sed "s/\r//" infile >outfile # UnxUtils
sed v4.0.7 or higher tr -d \r outfile # GNU tr version 1.22 or higher
# delete leading whitespace (spaces, tabs) from front of each line # aligns all text flush left
sed 's/^[ \t]*//' # see note on '\t' at end of file
# delete trailing whitespace (spaces, tabs) from end of each line
sed 's/[ \t]*$//' # see note on '\t' at end of file
# delete BOTH leading and trailing whitespace from each line
sed 's/^[ \t]*//;s/[ \t]*$//'
# insert 5 blank spaces at beginning of each line (make page offset)
sed 's/^/ /'
# align all text flush right on a 79-column width
sed -e :a -e 's/^.\{1,78\}$/ &/;ta' # set at 78 plus 1 space
# center all text in the middle of 79-column width. In method 1, spaces at the beginning of the line are significant, and trailing # spaces are appended at the end of the line. In method 2, spaces at # the beginning of the line are discarded in centering the line, and no trailing spaces appear at the end of lines.
sed -e :a -e 's/^.\{1,77\}$/ & /;ta' # method 1
sed -e :a -e 's/^.\{1,77\}$/ &/;ta' -e 's/\( *\)\1/\1/' # method 2

# substitute (find and replace) "foo" with "bar" on each line
sed 's/foo/bar/' # replaces only 1st instance in a line
sed 's/foo/bar/4' # replaces only 4th instance in a line
sed 's/foo/bar/g' # replaces ALL instances in a line
sed 's/\(.*\)foo\(.*foo\)/\1bar\2/' # replace the next-to-last case
sed 's/\(.*\)foo/\1bar/' # replace only the last case
# substitute "foo" with "bar" ONLY for lines which contain "baz"
sed '/baz/s/foo/bar/g'
# substitute "foo" with "bar" EXCEPT for lines which contain "baz"
sed '/baz/!s/foo/bar/g'
# change "scarlet" or "ruby" or "puce" to "red"
sed 's/scarlet/red/g;s/ruby/red/g;s/puce/red/g' # most seds gsed 's/scarlet\ruby\puce/red/g' # GNU sed only
# reverse order of lines (emulates "tac") bug/feature in HHsed v1.5 causes blank lines to be deleted
sed '1!G;h;$!d' # method 1
sed -n '1!G;h;$p' # method 2

# reverse each character on the line (emulates "rev")
sed '/\n/!G;s/\(.\)\(.*\n\)/&\2\1/;//D;s/.//'
# join pairs of lines side-by-side (like "paste")
sed '$!N;s/\n/ /'
# if a line ends with a backslash, append the next line to it
sed -e :a -e '/\\$/N; s/\\\n//; ta'
# if a line begins with an equal sign, append it to the previous line # and replace the "=" with a single space
sed -e :a -e '$!N;s/\n=/ /;ta' -e 'P;D'
# add commas to numeric strings, changing "1234567" to "1,234,567"
gsed ':a;s/\B[0-9]\{3\}\>/,&/;ta' # GNU sed
sed -e :a -e 's/\(.*[0-9]\)\([0-9]\{3\}\)/\1,\2/;ta' # other seds
# add commas to numbers with decimal points and minus signs (GNU sed)
gsed ':a;s/\(^\[^0-9.]\)\([0-9]\+\)\([0-9]\{3\}\)/\1\2,\3/g;ta'
# add a blank line every 5 lines (after lines 5, 10, 15, 20, etc.) gsed '0~5G' # GNU sed only
sed 'n;n;n;n;G;' # other seds

SELECTIVE PRINTING OF CERTAIN LINES:
# print first 10 lines of file (emulates behavior of "head")
sed 10q
# print first line of file (emulates "head -1")
sed q
# print the last 10 lines of a file (emulates "tail")
sed -e :a -e '$q;N;11,$D;ba'
# print the last 2 lines of a file (emulates "tail -2")
sed '$!N;$!D'
# print the last line of a file (emulates "tail -1")
sed '$!d' # method 1
sed -n '$p' # method 2
# print only lines which match regular expression (emulates "grep")
sed -n '/regexp/p' # method 1
sed '/regexp/!d' # method 2
# print only lines which do NOT match regexp (emulates "grep -v")
sed -n '/regexp/!p' # method 1, corresponds to above
sed '/regexp/d' # method 2, simpler syntax
# print the line immediately before a regexp, but not the line # containing the regexp
sed -n '/regexp/{g;1!p;};h'
# print the line immediately after a regexp, but not the line # containing the regexp
sed -n '/regexp/{n;p;}'
# print 1 line of context before and after regexp, with line number indicating where the regexp occurred (similar to "grep -A1 -B1")
sed -n -e '/regexp/{=;x;1!p;g;$!N;p;D;}' -e h
# grep for AAA and BBB and CCC (in any order)
sed '/AAA/!d; /BBB/!d; /CCC/!d'
# grep for AAA and BBB and CCC (in that order)
sed '/AAA.*BBB.*CCC/!d'
# grep for AAA or BBB or CCC (emulates "egrep")
sed -e '/AAA/b' -e '/BBB/b' -e '/CCC/b' -e d # most seds
gsed '/AAA\BBB\CCC/!d' # GNU sed only
# print paragraph if it contains AAA (blank lines separate paragraphs) HHsed v1.5 must insert a 'G;' after 'x;' in the next 3 scripts below
sed -e '/./{H;$!d;}' -e 'x;/AAA/!d;'
# print paragraph if it contains AAA and BBB and CCC (in any order)
sed -e '/./{H;$!d;}' -e 'x;/AAA/!d;/BBB/!d;/CCC/!d'
# print paragraph if it contains AAA or BBB or CCC
sed -e '/./{H;$!d;}' -e 'x;/AAA/b' -e '/BBB/b' -e '/CCC/b' -e d
gsed '/./{H;$!d;};x;/AAA\BBB\CCC/b;d' # GNU sed only
# print only lines of 65 characters or longer
sed -n '/^.\{65\}/p'
# print only lines of less than 65 characters
sed -n '/^.\{65\}/!p' # method 1, corresponds to above
sed '/^.\{65\}/d' # method 2, simpler syntax
# print section of file from regular expression to end of file
sed -n '/regexp/,$p'
# print section of file based on line numbers (lines 8-12, inclusive)
sed -n '8,12p' # method 1
sed '8,12!d' # method 2
# print line number 52 sed -n '52p' # method 1
sed '52!d' # method 2
sed '52q;d' # method 3, efficient on large files
# beginning at line 3, print every 7th line
gsed -n '3~7p' # GNU sed only
sed -n '3,${p;n;n;n;n;n;n;}' # other seds
# print section of file between two regular expressions (inclusive)
sed -n '/Iowa/,/Montana/p' # case sensitive

SELECTIVE DELETION OF CERTAIN LINES:
# print all of file EXCEPT section between 2 regular expressions
sed '/Iowa/,/Montana/d'
# delete duplicate, consecutive lines from a file (emulates "uniq"). First line in a set of duplicate lines is kept, rest are deleted.
sed '$!N; /^\(.*\)\n\1$/!P; D'
# delete duplicate, nonconsecutive lines from a file. Beware not to overflow the buffer size of the hold space, or else use GNU sed.
sed -n 'G; s/\n/&&/; /^\([ -~]*\n\).*\n\1/d; s/\n//; h; P'
# delete all lines except duplicate lines (emulates "uniq -d").
sed '$!N; s/^\(.*\)\n\1$/\1/; t; D'
# delete the first 10 lines of a file
sed '1,10d'
# delete the last line of a file
sed '$d'
# delete the last 2 lines of a file
sed 'N;$!P;$!D;$d'
# delete the last 10 lines of a file
sed -e :a -e '$d;N;2,10ba' -e 'P;D' # method 1
sed -n -e :a -e '1,10!{P;N;D;};N;ba' # method 2
# delete every 8th line
gsed '0~8d' # GNU sed only
sed 'n;n;n;n;n;n;n;d;' # other seds
# delete ALL blank lines from a file (same as "grep '.' ")
sed '/^$/d' # method 1
sed '/./!d' # method 2
# delete all CONSECUTIVE blank lines from file except the first; also deletes all blank lines from top and end of file (emulates "cat -s")
sed '/./,/^$/!d' # method 1, allows 0 blanks at top, 1 at EOF
sed '/^$/N;/\n$/D' # method 2, allows 1 blank at top, 0 at EOF
# delete all CONSECUTIVE blank lines from file except the first 2:
sed '/^$/N;/\n$/N;//D'
# delete all leading blank lines at top of file
sed '/./,$!d'
# delete all trailing blank lines at end of file
sed -e :a -e '/^\n*$/{$d;N;ba' -e '}' # works on all seds
sed -e :a -e '/^\n*$/N;/\n$/ba' # ditto, except for gsed 3.02*
# delete the last line of each paragraph
sed -n '/^$/{p;h;};/./{x;/./p;}'

SPECIAL APPLICATIONS:
# remove nroff overstrikes (char, backspace) from man pages. The 'echo' command may need an -e switch if you use Unix System V or bash shell.
sed "s/.`echo \\\b`//g" # double quotes required for Unix environment
sed 's/.^H//g' # in bash/tcsh, press Ctrl-V and then Ctrl-H
sed 's/.\x08//g' # hex expression for sed v1.5
# get Usenet/e-mail message header
sed '/^$/q' # deletes everything after first blank line
# get Usenet/e-mail message body
sed '1,/^$/d' # deletes everything up to first blank line
# get Subject header, but remove initial "Subject: " portion
sed '/^Subject: */!d; s///;q'
# get return address header
sed '/^Reply-To:/q; /^From:/h; /./d;g;q'
# parse out the address proper. Pulls out the e-mail address by itself from the 1-line return address header (see preceding script)
sed 's/ *(.*)//; s/>.*//; s/.*[:<] *//' # add a leading angle bracket and space to each line (quote a message) sed 's/^/> /'
# delete leading angle bracket & space from each line (unquote a message)
sed 's/^> //'
# remove most HTML tags (accommodates multiple-line tags)
sed -e :a -e 's/<[^>]*>//g;/zipup.bat dir /b *.txt sed "s/^\(.*\)\.TXT/pkzip -mo \1 \1.TXT/">>zipup.bat

TYPICAL USE: Sed takes one or more editing commands and applies all ofthem, in sequence, to each line of input. After all the commands havebeen applied to the first input line, that line is output and a secondinput line is taken for processing, and the cycle repeats. Thepreceding examples assume that input comes from the standard inputdevice (i.e, the console, normally this will be piped input). One ormore filenames can be appended to the command line if the input doesnot come from stdin. Output is sent to stdout (the screen). Thus:
cat filename sed '10q' # uses piped input
sed '10q' filename # same effect, avoids a useless "cat"
sed '10q' filename > newfile # redirects output to disk

On line selection or deletion in which you only need to output linesfrom the first part of the file, a "quit" command (q) in the scriptwill drastically reduce processing time for large files. Thus:
sed -n '45,50p' filename # print line nos. 45-50 of a file
sed -n '51q;45,50p' filename # same, but executes much faster

CREDITS AND THANKS:http://student.northpark.edu/pemente/sed/sed1line.txt

Daily Weapons In Unix for Work -2

To reverse a file using awk and print next three lines after pattern match (Including pattern):

awk '{ a[i++] = $0 } END { for (j=i-1; j>=0;) print a[j--] }' abcd.txt |sed -n '/MANU/{p;n;p;n;p;n;p;}' >abcd_new.txt

(This can be used when you want to print 3 lines before pattern match, including pattern. Logic: reverse a file and print 3 lines after pattern match.)

How to reverse a file using awk:

awk '{ a[i++] = $0 } END { for (j=i-1; j>=0;) print a[j--] }' abcd.txt

How to reverse a file using sed:

sed -n '1!G;h;$p' abcd.txt


how to get yesterday date in unix
YEST=`TZ="GMT+24" date +'%m/%d/%Y'`; echo $YEST

Replace Text : sed -e "s/Old/New/g"

Remove Text : sed -e "s/Text//g"

Remove Text From Front : sed -e "s/^Text//g"

Remove Text From End : 

sed -e "s/Text\$//g"

sed -e 's/Text$//g'

Insert Text in Front : sed -e "s/^/Text/g"

Append Text to End :

sed -e "s/\$/Text/g"

sed -e 's/$/Text/g'

Truncate :

sed -e "s/Text.*//"

cut -c5-8

Downshift and Upshift :

tr '[A-Z]' '[a-z]'

tr '[a-z]' '[A-Z]'

Change Tabs to Spaces : sed -e "s/<tab>/ /g"

Change Multiple Spaces to a Single Space :

sed -e "s/  */ /g"

Change Whitespaces to a Single Space :

sed -e "s/[<tab><space>][<tab><space>]*/<space>/g"

Delete Leading Whitespaces:

sed -e "s/^[<tab><space>]*//g"

Delete Trailing Whitespaces:

sed -e 's/[<tab><space>]*$//g'

Delete Lines:

sed -e "/Text/d" 

grep -v "Text"

Delete Empty Lines:

sed -e '/^*$/d'

sed -e '/^[<tab><space>]*$/d'

Delete First Line:

sed -e '1d'

sed -e '1,7d'

Delete Last Line : sed -e '$d'

Print First Line :

sed -n '1p'

sed -e '2,$d'

sed -e '1q'

Print Last Line :

sed -n '$p'

Delete Comments :

sed -e '/^#/d'

sed -e 's/#.*//'

Delete Text Between Keywords :

sed -e '/Keyword1/,/Keyword2/d'

sed -e '/^Keyword1$/,/^Keyword2$/d'

Extract Text Between Keywords : 

sed -e '/^Keyword1$/,/^Keyword2$/!d'

Print Odd Lines : cat -n file |  sed -e '/.....[02468]/d' -e 's/^.......//'

Print Even Lines : cat -n file |  sed -e '/.....[13579]/d' -e 's/^.......//'

Read a File Backwards : cat -n file | sort -nr | sed 's/^.......//'

Convert < > into LTGT : cat $* | sed -e 's/</\&lt;/g' -e 's/>/\&gt;/g' 

Daily Weapons of Unix in Office

Count the number of occurances of a pattern in a file using awk
awk '{ for (i=1;i<=NF;i++) if ( $i == "word" ) count++ } END{print count}' filename
This will print the count of the occurance of  a pattern "word" in a file.


How to compare the values of a column in awk in a same file and consecutive lines..
If one would like to compare the values of 2nd column of consecutive lines of same file in such a way so that if the difference between first value and second value is more than 100 it should print complete line else ignore line.

Input File:
=======

ABC 2500
ABCD 123
XYZ 122
WXYZ 2565


Desired Output:
==========

ABC 2500 (i.e. difference between 2500 and 123 is greater than 100 here)
XYZ 122 (i.e. difference between 122 and 2565 is greater than 100 here)

Command:
=======

awk 'NR % 2 != 0 {a=$1; b=$2} NR % 2 == 0 {if (b - $2 > 100 || $2 - b > 100){print a,b}}' inputfile




Count the number of files in each directory recursively
find ./ -type d -exec sh -c "echo -n {} ' ' ; ls -l {} | wc -l" \;


Use awk in following cases

Joining two files parallely using awk:
========================
awk 'NR==FNR{a[NR]=$0; next} {print a[FNR], $0}' file1.txt file2.txt

Changing extension of all .txt to .sh:
========================
for i in *.txt;do mv "$i" "${i%.txt}".sh;done



Print 3 lines before and 5 lines after pattern match abcd in a file a.txt
In Linux:

awk 'c-->0;$0~s{if(b)for(c=b+1;c>1;c--)print r[(NR-c+1)%b];print;c=a}b{r[NR%b]=$0}' b=3 a=5 s="abcd" a.txt



ftp in UNIX
ftp -n 10.200.120.12 << EOF
quote user username
quote pass password
cd /path/to/directory/
prompt off
put dump.txt
exit
EOF




Print one line before pattern match using sed in UNIX.
sed -n -e '/manu/{x;p;x;p;}' -e h test.txt




sort a delimited file on the basis of a column
sort -t":" -k3 filename.txt

(The above will sort a file filename.txt which is ":" delimited on the basis of third column.)




Print every 6th line of a file ..
The below awk command will print every 6th line of a file test.txt starting from line number one.

awk '{if(NR%6==1) print $0}' test.txt

(Note: Print every nth line using awk.)



# insert a blank line above every line which matches "regex"
 sed '/regex/{x;p;x;}' OR  sed '/regex/{x;G;}'

# insert a blank line below every line which matches "regex"
 sed '/regex/G'

# insert a blank line above and below every line which matches "regex"
 sed '/regex/{x;p;x;G;}'

# remove the header of a file
sed '/1,1d' filename

Difference between AWK and SED command

SED (which stands for Stream EDitor) is a simple but powerful computer program used to apply various pre-specified textual transformations to a sequential stream of text data. It reads input files line by line, edits each line according to rules specified in its simple language (the sed script), and then outputs the line.

AWK is a complete pattern scanning and processing language, it is most commonly used as a Unix command-line filter to reformat the output of other commands.  For example, to print only the second and sixth fields of the date command (the month and year) with a space separating them, at the Unix prompt, you would enter: date | awk ‘{print $2 ” ” $6}’

====================================================


sed reads from standard input and/or file(s), applies specified changes which are typically mostly editing changes, and writes to standard output. It is relatively similar to the text editor
ed(1), except in the case of sed it doesn't alter its input file(s)*, but writes the data to standard output, and sed also has some commands which ed does not have (and in most cases wouldn't be suitable for or make sense for ed). sed's commands are mostly limited to editing type commands.

awk is sort of like an interpreted somewhat C-like language with added built-in functionality for dealing with strings and patterns. Though not a fully general purpose programming language, awk is much closer to such than sed. 

Unix Interview Questions and Commands

1. How do you find which processes are using a particular file?

By using lsof command in UNIX. It will list down PID of all the processes which are using a particular file.

2. How do you find which remote hosts are connecting to your host on a particular port say 10123?

By using netstat command

For example: execute netstat -a | grep "port" and it will list the entire hosts which are connected to this host on port 10123.

3. How to tell if my process is running in Unix?

You can list down all the running processes using [ps] command. Then you can “grep” your user name or process name to see if the process is running.

4. What is ephemeral port in UNIX?

Ephemeral ports are port used by Operating system for client sockets. There is a specific range on which OS can open any port specified by ephemeral port range.

5. How to list down file/folder lists alphabetically?

Normally [ls –lt] command lists down file/folder list sorted by modified time. If you want to list then alphabetically, then you should simply specify: [ls –l]

6. If one process is inserting data into your MySQL database? How will you check how many rows inserted into every second?

By using "watch" command in UNIX

7. There is a file Unix_Test.txt which contains words "Unix". How will you replace all Unix to UNIX?

By using SED command in UNIX

For example: you can execute sed s/Unix/UNIX/g fileName.

8. You have a tab separated file which contains Name, Address and Phone Number. List down all Phone Number without their name and addresses?

By using either AWK or CUT command.

9. How to check if the last command was successful in Unix?

To check the status of last executed command in UNIX, you can check the value of an inbuilt bash variable [$?]. See the below example:

$> echo $?

10. How to check all the running processes in Unix?

The standard command to see this is [ps]. But [ps] only shows you the snapshot of the processes at that instance. If you need to monitor the processes for a certain period of time and need to refresh the results in each interval, consider using the [top] command.

$> ps –ef

If you wish to see the % of memory usage and CPU usage, then consider the below switches:

$> ps aux

If you wish to use this command inside some shell script, or if you want to customize the output of [ps] command, you may use “-o” switch like below. By using “-o” switch, you can specify the columns that you want [ps] to print out.

$>ps -e -o stime,user,pid,args,%mem,%cpu

11 Your application home directory is full? How will you find which directory is taking how much space?

By using disk usage (DU) command in Unix

For example du –sh . | grep G  will list down all the directories which have GIGS in Size.

12. How do you find for how many days your Server is up?

By using uptime command in UNIX

13. How to check if a file is present in a particular directory in Unix?

Using command, we can do it in many ways. Based on what we have learnt so far, we can make use of [ls] and [$?] command to do this. See below:

$> ls –l file.txt; echo $?

If the file exists, the [ls] command will be successful. Hence [echo $?] will print 0. If the file does not exist, then [ls] command will fail and hence [echo $?] will print 1.

14. You have an IP address in your network. How will you find hostname and vice versa?

By using nslookup command in UNIX

15. How to execute a database stored procedure from Shell script?

$> SqlReturnMsg=`sqlplus -s username/password@database<<EOF
BEGIN
Proc_Your_Procedure(… your-input-parameters …);
END;
/
EXIT;
EOF`
$> echo $SqlReturnMsg

16. How to check the command line arguments in a UNIX command in Shell Script?

In a bash shell, you can access the command line arguments using $0, $1, $2, … variables, where $0 prints the command name, $1 prints the first input parameter of the command, $2 the second input parameter of the command and so on.

17. How to fail a shell script programmatically?

Just put an [exit] command in the shell script with return value other than 0. This is because the exit code of successful Unix program is zero. So, suppose if you write exit -1 inside your program, then your program will throw an error and exit immediately.

18. How to print/display the first line of a file?

There are many ways to do this. However the easiest way to display the first line of a file is using the [head] command.

$> head -1 file.txt

If you specify [head -2] then it would print first 2 records of the file.

Another way can be by using [sed] command. [Sed] is a very powerful text editor which can be used for various text manipulation purposes like this.

$> sed '2,$ d' file.txt

How does the above command work? The 'd' parameter basically tells [sed] to delete all the records from display from line 2 to last line of the file (last line is represented by $ symbol). Of course it does not actually delete those lines from the file, it just does not display those lines in standard output screen. So you only see the remaining line which is the 1st line.]

19. How to print/display the last line of a file?

The easiest way is to use the [tail] command.

$> tail -1 file.txt

If you want to do it using [sed] command, here is what you should write:

$> sed -n '$ p' test

From our previous answer, we already know that '$' stands for the last line of the file. So '$ p' basically prints (p for print) the last line in standard output screen. '-n' switch takes [sed] to silent mode so that [sed] does not print anything else in the output.

20. How to display n-th line of a file?

The easiest way to do it will be by using [sed]. Based on what we already know about [sed] from our previous examples, we can quickly deduce this command:

$> sed –n '<n> p' file.txt

You need to replace <n> with the actual line number. So if you want to print the 4th line, the command will be

$> sed –n '4 p' test

Of course you can do it by using [head] and [tail] command as well like below:

$> head -<n> file.txt | tail -1

You need to replace <n> with the actual line number. So if you want to print the 4th line, the command will be

$> head -4 file.txt | tail -1

21. How to remove the first line / header from a file?

We already know how [sed] can be used to delete a certain line from the output – by using the'd' switch. So if we want to delete the first line the command should be:

$> sed '1 d' file.txt

But the issue with the above command is, it just prints out all the lines except the first line of the file on the standard output. It does not really change the file in-place. So if you want to delete the first line from the file itself, you have two options.

Either you can redirect the output of the file to some other file and then rename it back to original file like below:

$> sed '1 d' file.txt > new_file.txt
$> mv new_file.txt file.txt

Or, you can use an inbuilt [sed] switch '–i' which changes the file in-place. See below:

$> sed –i '1 d' file.txt

22. How to remove the last line/ trailer from a file in Unix script?

Always remember that [sed] switch '$' refers to the last line. So using this knowledge we can deduce the below command:

$> sed –i '$ d' file.txt

23. How to remove certain lines from a file in Unix?

If you want to remove line <m> to line <n> from a given file, you can accomplish the task in the similar method shown above. Here is an example:

$> sed –i '5,7 d' file.txt

The above command will delete line 5 to line 7 from the file file.txt

24. How to remove the last n-th line from a file?

This is bit tricky. Suppose your file contains 100 lines and you want to remove the last 5 lines. Now if you know how many lines are there in the file, then you can simply use the above shown method and can remove all the lines from 96 to 100 like below:

$> sed –i '96,100 d' file.txt   # alternative to command [head -95 file.txt]

But not always you will know the number of lines present in the file (the file may be generated dynamically, etc.) In that case there are many different ways to solve the problem. There are some ways which are quite complex and fancy. But let's first do it in a way that we can understand easily and remember easily. Here is how it goes:

$> tt=`wc -l file.txt | cut -f1 -d' '`;sed –i "`expr $tt - 4`,$tt d" test

As you can see there are two commands. The first one (before the semi-colon) calculates the total number of lines present in the file and stores it in a variable called “tt”. The second command (after the semi-colon), uses the variable and works in the exact way as shown in the previous example.

25. How to check the length of any line in a file?

We already know how to print one line from a file which is this:

$> sed –n '<n> p' file.txt

Where <n> is to be replaced by the actual line number that you want to print. Now once you know it, it is easy to print out the length of this line by using [wc] command with '-c' switch.

$> sed –n '35 p' file.txt | wc –c

The above command will print the length of 35th line in the file.txt.

26. How to get the nth word of a line in Unix?

Assuming the words in the line are separated by space, we can use the [cut] command. [cut] is a very powerful and useful command and it's real easy. All you have to do to get the n-th word from the line is issue the following command:

cut –f<n> -d' '

'-d' switch tells [cut] about what is the delimiter (or separator) in the file, which is space ' ' in this case. If the separator was comma, we could have written -d',' then. So, suppose I want find the 4th word from the below string: “A quick brown fox jumped over the lazy cat”, we will do something like this:

$> echo “A quick brown fox jumped over the lazy cat” | cut –f4 –d' '

And it will print “fox”

27. How to reverse a string in unix?

Pretty easy. Use the [rev] command.

$> echo "unix" | rev

xinu

28. How to get the last word from a line in Unix file?

We will make use of two commands that we learnt above to solve this. The commands are [rev] and [cut]. Here we go.

Let's imagine the line is: “C for Cat”. We need “Cat”. First we reverse the line. We get “taC rof C”. Then we cut the first word, we get 'taC'. And then we reverse it again.

$>echo "C for Cat" | rev | cut -f1 -d' ' | rev

Cat

29. How to get the n-th field from a Unix command output?

We know we can do it by [cut]. Like below command extracts the first field from the output of [wc –c] command

$>wc -c file.txt | cut -d' ' -f1

But I want to introduce one more command to do this here. That is by using [awk] command. [awk] is a very powerful command for text pattern scanning and processing. Here we will see how may we use of [awk] to extract the first field (or first column) from the output of another command. Like above suppose I want to print the first column of the [wc –c] output. Here is how it goes like this:

$>wc -c file.txt | awk ' ''{print $1}'

The basic syntax of [awk] is like this:

awk 'pattern space''{action space}'

The pattern space can be left blank or omitted, like below:

$>wc -c file.txt | awk '{print $1}'

In the action space, we have asked [awk] to take the action of printing the first column ($1).

30. How to replace the n-th line in a file with a new line in Unix?

This can be done in two steps. The first step is to remove the n-th line. And the second step is to insert a new line in n-th line position. Here we go.

Step 1: remove the n-th line

$>sed -i'' '10 d' file.txt       # d stands for delete

Step 2: insert a new line at n-th line position

$>sed -i'' '10 i This is the new line' file.txt     # i stands for insert

31. How to show the non-printable characters in a file?

Open the file in VI editor. Go to VI command mode by pressing [Escape] and then [:]. Then type [set list]. This will show you all the non-printable characters, e.g. Ctrl-M characters (^M) etc., in the file.

32. How to zip a file in Linux?

Use inbuilt [zip] command in Linux

33. How to unzip a file in Linux?

Use inbuilt [unzip] command in Linux.

$> unzip –j file.zip

34. How to test if a zip file is corrupted in Linux?

Use “-t” switch with the inbuilt [unzip] command

$> unzip –t file.zip

35. How to check if a file is zipped in Unix?

In order to know the file type of a particular file use the [file] command like below:
$> file file.txt
file.txt: ASCII text
If you want to know the technical MIME type of the file, use “-i” switch.
$>file -i file.txt
file.txt: text/plain; charset=us-ascii
If the file is zipped, following will be the result
$> file –i file.zip
file.zip: application/x-zip

Awesome Awk One Liners for Unix Interviews

1.To Negate some lines with AWK:

 These are called awk one liners. The First trick for today is how to negate some rows while using awk.

The Trick is : awk -F ” ” ‘NR>5 && NR<7{print $1,$2,$3}’ 1.txt

Note : This question will mostly asked in interviews where you will be asked to print some parameters from a file mostly a log file where in you will be asked to negate some rows herein refered to as records in awk. The above command here performs the following action. The field separator mostly called as delimiter here will be a space and then it will negate all the records and will only iterate through line numbers five through seven and then print the first second and third parameter.

2. To Take only a few Lines:

How would you print only lines match between Rahul and Abhishek using awk? Considering your file names.txt contains :
Rahul
Sahil
Geetika
Mamta
Abhishek
and the expected output should be like :
Sahil
Geetika
Mamta

The Trick is : $ awk ‘/Rahul/{ P=1; next } /Abhishek/ {exit} P’ names.txt

3. Context Addressing in AWK:

The above command will give you the desired output. Just try. If the Interviewer asks you what is this called, Just say this is called the CONTEXT addressing in awk. The interviewer might also be interested in asking you to print a certain range of lines from a file where in you do not have a pattern defined. We all know awk works on the principle of condition { actions }. Where condition is typically an expression and action is a series of commands. The input is split into records, where by default records are separated by newline characters so that the input is split into lines. So we know the context addressing can be used just like grep, so here you go.

The Trick is : awk ‘$0 ~ /pattern/ {print $0}’

4. Taking a range of records:

Suppose one wants to print all the lines in a file that match some pattern, the above code should be used.Lets suppose he wants you to print logs from line number 15000 to 16000.Just give him a simple blunt reply.

The Trick is  : head -16000 filename | tail -1000.

5. Deleting only files :

Now the interviewer might be tempted to ask you to delete all the files in the current directory with awk.

The Trick is  : ls -l|awk ‘$1!~/^drwx/{print $9}’|xargs rm

6. Removing Duplicates without sorting :

Be careful when trying this out in your home directory. This will remove all the files. awk syntax is not the same in every Unix system, but there is a way to learning how is it in our particular system: man awk. Now lets see our last trick, this talks about deleting the duplicate entries in file without sorting. There are thousand ways of doing one thing in unix lets see a couple of them. Suppose your file text.txt has contents the below contents.

/users/env> cat test.txt
abcd
efgh
ijkl
mnop
abcd
/users/env > cat test.txt |awk !’x[$0]++’
abcd
efgh
ijkl
mnop

Now this one lines removes the duplicate without sorting them.

Taken From http://www.computerandyou.net/2012/07/six-simple-and-awesome-awk-one-liners-for-unix-interviews/

AWK Interview Questions 1

Awk is powerful tool in Unix. Awk is an excellent tool for processing the files which have data arranged in rows and columns format. It is a good filter and report writer.
1. How to run awk command specified in a file?
awk -f filename

2. Write a command to print the squares of numbers from 1 to 10 using awk command
awk 'BEGIN { for(i=1;i<=10;i++) {print "square of",i,"is",i*i;}}'

3. Write a command to find the sum of bytes (size of file) of all files in a directory.
ls -l | awk 'BEGIN {sum=0} {sum = sum + $5} END {print sum}'

4. In the text file, some lines are delimited by colon and some are delimited by space. Write a command to print the third field of each line.

awk '{ if( $0 ~ /:/ ) { FS=":"; } else { FS =" "; } print $3 }' filename

5. Write a command to print the line number before each line?
awk '{print NR, $0}' filename

6. Write a command to print the second and third line of a file without using NR.
awk 'BEGIN {RS="";FS="\n"} {print $2,$3}' filename

7. Write a command to print zero byte size files?
ls -l | awk '/^-/ {if ($5 !=0 ) print $9 }'

8. Write a command to rename the files in a directory with "_new" as postfix?
ls -F | awk '{print "mv "$1" "$1".new"}' | sh

9. Write a command to print the fields in a text file in reverse order?
awk 'BEGIN {ORS=""} { for(i=NF;i>0;i--) print $i," "; print "\n"}' filename

10. Write a command to find the total number of lines in a file without using NR
awk 'BEGIN {sum=0} {sum=sum+1} END {print sum}' filename

Another way to print the number of lines is by using the NR. The command is
awk 'END{print NR}' filename 

perl coding

        my $Log_Path;
        my $OPLOG_PATH;
        my $Start_Time_Dur;
        my $End_Time_Dur;
        my $Category ;
        my $op_file ;
        if (scalar @ARGV < 2) {
        die "
        ***********************************************************
        USAGE:
        Option1 : $0 Log_File_Name Flag
        Option2 : $0 Log_File_Name Flag ArrayCounter
        Help :
        perl $0 Log_File_Name --flags

        flags are:

          --Help                Suggests the usage for the script and the flags available
          --Default             1. Shows The Default Behaviour {Top 15 Max Response Time URL and Top 15 Download URL}
                                2. Top ten URL's Response Time wise, Top Ten URL Download Wise, Hourly Frequency of load
          --Hourly              Outputs Hourly hit/response time/download amount from URL's
          --CategoryWise        Outputs category wise hit/response time/download amount from URL's
          --Categ_Hourly        Outputs Hourly as well as Category wise hit/response time/download amount from URL's
          --Download            Outputs download amount from URL's
          --Cat_Download        Outputs Category wise download amount from URL's
          --Hourly_Download     Outputs Hourly download amount from URL's
          --Create_Hourly_Logs  This Creates the Hourly Log as per the requirement
          --All3Ip              Output's 3 ip's which are present at IP field
          --URL                 Output's the unique URL
          --HperIP              Output's Hit's Per Ip
        "
        }
        else {
        $Log_Path = $ARGV[0] || '';
        }
        use strict;
        my $array_counter = $ARGV[2] || '15';
        print "Number of Rows Requested : ".$array_counter."\n";
        my %hash1 = ();
        my @array1;
        my $output_file;
        my @arrays;
        my @hourly_log;
        my $scalar1;
        my %seen = ();
        my $tim;
        my $waqt;
        my $series;
        my $text;
        my @hitsip;
        my @respons;
        my @topten;
        my @qwe;
        my @dada;
        my @highrespo;
        my @all3ip;
        my @sorthosize;
        my @bothvalue;
        my $oneval;
        my @topten_size;
        my @topten_cat_size;
        my @topten_onlycat_size;
        my @sorthousize;
        my @toptensize;
        my @sortsize;
        my @highsize;
        my @sortcat;
        my @sortdowcatonly;
        my @diff_value;
        my @hourly_op;
        my @topt_3;
        my @next3array;
        my $pages;
        my @special;
        my @next3soarray;
        my @sortcatonly;
        my $how;
        my @topten_hourly;
        my @highestsize;
        my @topten_onlycat;
        my %count;
        my @sortcatsize;
        my @topten_cat;
        my @nextarray;
        my @nextsoarray;
        my @sortqwe;
        my @grephours ;
        my @timarr = (1 .. 24);
        my ($begin_time, $end_time, $total_hits);
                my ($Start_Hour,$Start_Min,$End_Hour,$End_Min);
        my ($host, $ident_user, $auth_user, $date, $time, $time_zones, $farji, $farji2i, $farji3, $farji4, $farji5, $farji6, $farji7, $farji8, $farji9, $farji10, $farji11, $farji12) ;
        my ($hour, $min, $sec) ;
        print "Log File Used for Parsing is -->$Log_Path"."\n";
        open (MYFILE,$Log_Path) || die "Error : $!";

        ############### parse command line argument and choose the right request to server ############
                        if ($ARGV[1] =~ "--Hourly") {
                                                        print "Start Time of analysis Period ";
                                                        print "Start Time -->in format of HH:MM --> ";
                                                        chomp($Start_Time_Dur = <STDIN>);
                                                        ($Start_Hour, $Start_Min) = split /:/, $Start_Time_Dur, 2;
                                                        if ($Start_Hour >= 0 && $Start_Hour <= 24 && $Start_Min >= 0 && $Start_Min <= 60) {
                                                        print $Start_Hour."\n" ;
                                                        print $Start_Min."\n" ;
                                                        }
                                                        else {
                                                        print "Start Time Should be in range of 00 to 24 hours ";
                                                        exit ;
                                                        }
                                                        print "End Time of analysis Period ";
                                                        print "End Time --> in format of HH:MM --> ";
                                                        chomp($End_Time_Dur = <STDIN>);
                                                        ($End_Hour, $End_Min) = split /:/, $End_Time_Dur, 2;
                                                        if ($End_Hour >= 0 && $End_Hour <= 24 && $End_Min >= 0 && $End_Min <= 60) {
                                                        print $End_Hour."\n" ;
                                                        print $End_Min."\n" ;
                                                        }
                                                        else {
                                                        print "End Time Should be in range of 00 to 24 hours ";
                                                        exit ;
                                                        }
                                                        if ($Start_Hour <= $End_Hour) {
                                                        print "This is Right Order Of Time Duration"."\n" ;
                                                        }
                                                        else {
                                                        print "End_Hour should be greater than or equal to start hour. Give the right ordering" ;
                                                        exit ;
                                                        }
                                                        if ($End_Hour == 24){
                                                        print "Default Value of End Minute when End Hour is 24"."\n" ;
                                                        $End_Min = 00 ;
                                                        print $End_Min."\n";
                                                        }
                                                        &break_component ;
                                                        &hourly_repso ;
                                                        &hourly_size;
                        } elsif ($ARGV[1] =~ "--Default") {
                                                        print "Shows The Default Behaviour {Top 15 Max Response Time URL and Top 15 Download URL}";
                                                        &break_component ;
                                                        &genral_behav ;
                                                        &hou_only_size;
                                                        &download_wise;
                                                        &cat_only_size;
                        } elsif ($ARGV[1] =~ "--CategoryWise") {
                                                        print " Give Category you want to analyze ";
                                                        chomp($Category = <STDIN>);
                                                        print " Category Choosen --> ".$Category."\n";
                                                        &break_component ;
                                                        &category_wise ;
                                                        &categ_down_wise;
                        } elsif ($ARGV[1] =~ "--Categ_Hourly") {
                                                        print "Start Time of analysis Period ";
                                                        print "Start Time -->in format of HH:MM --> ";
                                                        chomp($Start_Time_Dur = <STDIN>);
                                                        ($Start_Hour, $Start_Min) = split /:/, $Start_Time_Dur, 2;
                                                        if ($Start_Hour >= 0 && $Start_Hour <= 24 && $Start_Min >= 0 && $Start_Min <= 60) {
                                                        print $Start_Hour."\n" ;
                                                        print $Start_Min."\n" ;
                                                        }
                                                        else {
                                                        print "Start Time Should be in range of 00 to 24 hours ";
                                                        exit ;
                                                        }
                                                        print "End Time of analysis Period ";
                                                        print "End Time --> in format of HH:MM --> ";
                                                        chomp($End_Time_Dur = <STDIN>);
                                                        ($End_Hour, $End_Min) = split /:/, $End_Time_Dur, 2;
                                                        if ($End_Hour >= 0 && $End_Hour <= 24 && $End_Min >= 0 && $End_Min <= 60) {
                                                        print $End_Hour."\n" ;
                                                        print $End_Min."\n" ;
                                                        }
                                                        else {
                                                        print "End Time Should be in range of 00 to 24 hours ";
                                                        exit ;
                                                        }
                                                        if ($Start_Hour <= $End_Hour) {
                                                        print "This is Right Order Of Time Duration"."\n" ;
                                                        }
                                                        else {
                                                        print "End_Hour should be greater than or equal to start hour. Give the right ordering" ;
                                                        exit ;
                                                        }
                                                        if ($End_Hour == 24){
                                                        print "Default Value of End Minute when End Hour is 24"."\n" ;
                                                        $End_Min = 00 ;
                                                        print $End_Min."\n";
                                                        }
                                                        print " Give Category you want to analyze ";
                                                        chomp($Category = <STDIN>);
                                                        print " Category Choosen --> ".$Category."\n";
                                                        &break_component ;
                                                        &hourcateg_repso;
                                                        &download_wise;
                                                        &hourly_cat_size;
                        } elsif ($ARGV[1] =~ "--Download") {
                                                        &break_component ;
                                                        &download_wise ;
                                                        &cat_only_size;
                        } elsif ($ARGV[1] =~ "--Cat_Download") {
                                                        print " Give Category you want to analyze ";
                                                        chomp($Category = <STDIN>);
                                                        print " Category Choosen --> ".$Category."\n";
                                                        &break_component ;
                                                        &category_wise ;
                        } elsif ($ARGV[1] =~ "--Hourly_Download") {
                                                        print " Analyze Hourly Download ";
                                                        print "Start Time of analysis Period ";
                                                        print "Start Time -->in format of HH:MM --> ";
                                                        chomp($Start_Time_Dur = <STDIN>);
                                                        ($Start_Hour, $Start_Min) = split /:/, $Start_Time_Dur, 2;
                                                        if ($Start_Hour >= 0 && $Start_Hour <= 24 && $Start_Min >= 0 && $Start_Min <= 60) {
                                                        print $Start_Hour."\n" ;
                                                        print $Start_Min."\n" ;
                                                        }
                                                        else {
                                                        print "Start Time Should be in range of 00 to 24 hours ";
                                                        exit ;
                                                        }
                                                        print "End Time of analysis Period ";
                                                        print "End Time --> in format of HH:MM --> ";
                                                        chomp($End_Time_Dur = <STDIN>);
                                                        ($End_Hour, $End_Min) = split /:/, $End_Time_Dur, 2;
                                                        if ($End_Hour >= 0 && $End_Hour <= 24 && $End_Min >= 0 && $End_Min <= 60) {
                                                        print $End_Hour."\n" ;
                                                        print $End_Min."\n" ;
                                                        }
                                                        else {
                                                        print "End Time Should be in range of 00 to 24 hours ";
                                                        exit ;
                                                        }
                                                        if ($Start_Hour <= $End_Hour) {
                                                        print "This is Right Order Of Time Duration"."\n" ;
                                                        }
                                                        else {
                                                        print "End_Hour should be greater than or equal to start hour. Give the right ordering" ;
                                                        exit ;
                                                        }
                                                        if ($End_Hour == 24){
                                                        print "Default Value of End Minute when End Hour is 24"."\n" ;
                                                        $End_Min = 00 ;
                                                        print $End_Min."\n";
                                                        }
                                                        &break_component ;
                                                        &categ_down_wise;
                                                        &hourly_size ;
                        } elsif ($ARGV[1] =~ "--Create_Hourly_Logs") {
                                                        print " Analyze Category Wise Hourly Download ";
                                                        print "Start Time of analysis Period ";
                                                        print "Start Time -->in format of HH:MM --> ";
                                                        chomp($Start_Time_Dur = <STDIN>);
                                                        ($Start_Hour, $Start_Min) = split /:/, $Start_Time_Dur, 2;
                                                        if ($Start_Hour >= 0 && $Start_Hour <= 24 && $Start_Min >= 0 && $Start_Min <= 60) {
                                                        print $Start_Hour."\n" ;
                                                        print $Start_Min."\n" ;
                                                        }
                                                        else {
                                                        print "Start Time Should be in range of 00 to 24 hours ";
                                                        exit ;
                                                        }
                                                        print "End Time of analysis Period ";
                                                        print "End Time --> in format of HH:MM --> ";
                                                        chomp($End_Time_Dur = <STDIN>);
                                                        ($End_Hour, $End_Min) = split /:/, $End_Time_Dur, 2;
                                                        if ($End_Hour >= 0 && $End_Hour <= 24 && $End_Min >= 0 && $End_Min <= 60) {
                                                        print $End_Hour."\n" ;
                                                        print $End_Min."\n" ;
                                                        }
                                                        else {
                                                        print "End Time Should be in range of 00 to 24 hours ";
                                                        exit ;
                                                        }
                                                        if ($Start_Hour <= $End_Hour) {
                                                        print "This is Right Order Of Time Duration"."\n" ;
                                                        }
                                                        else {
                                                        print "End_Hour should be greater than or equal to start hour. Give the right ordering" ;
                                                        exit ;
                                                        }
                                                        if ($End_Hour == 24){
                                                        print "Default Value of End Minute when End Hour is 24"."\n" ;
                                                        $End_Min = 00 ;
                                                        print $End_Min."\n";
                                                        }
                                                        print "Give THe File Name To wirte the Logs-->"."\n";
                                                        chomp($output_file = <STDIN>);
                                                        print "Output File Name is --> ".$output_file."\n";
                                                        #print "Give Category you want to analyze ";
                                                        #chomp($Category = <STDIN>);
                                                        #print " Category Choosen --> ".$Category."\n";
                                                        &break_component ;
                                                        &hourly_op ;
                        } elsif ($ARGV[1] =~ "--All3Ip") {
                                                        &break_component;
                                                        &new_all3_ip ;
                        }
                          elsif ($ARGV[1] =~ "--URL") {
                                                        &break_component;
                                                        &uniqe_url;
                                                        &greedy_parse;
                                                        &parse_unique;
                        }  elsif ($ARGV[1] =~ "--HperIP") {
                                                        &break_component;
                                                        &perIPhit;
                        }

        ############### parsing of command line argument over ############