Network Management (netstat)

netstat command (Network Statistics)
netstat displays the contents of various network-related data structures in depending on the options selected.

Syntax:
netstat < option/s >
multiple options can be given at one time.

-a  : displays the state of all sockets.

-r   : shows the system routing tables
-i   : gives statistics on a per-interface basis.
-m : displays information from the network memory buffers. On Solaris, this shows statistics for STREAMS
-p [proto] : retrieves statistics for the specified protocol
-s : shows per-protocol statistics. 

(some implementations allow -ss to remove fileds with a value of 0 (zero) from the display.)
-D : display the status of DHCP configured interfaces. 

-n : do not lookup hostnames, display only IP addresses.
-d (with -i) : displays dropped packets per interface.
-I [interface] retrieve information about only the specified interface.

-v : be verbose
interval :  number for continuous display of statictics.

$netstat -rn                       
Routing Table: IPv4                   
Destination    Gateway       Flags    Ref    Use    Interface
-------------------- -------------------- ----- ----- ------ ---------
192.168.1.0    192.168.1.11  U         1     1444    le0
224.0.0.0       192.168.1.11  U         1          0    le0
default          192.168.1.1    UG       1    68276   
127.0.0.1       127.0.0.1       UH       1    10497   lo0

This shows the output on a Solaris machine who's IP address is 192.168.1.11 with a default router at 192.168.1.1

Other important options with netstat: 

netstat -an | grep LISTEN | grep xxxx to grep for the port number
If you want to see what ports processes are listening on at a given moment and get a sorted list so you can see where there might be an available port, 

you can use the command below:
netstat -an | grep LISTEN | awk '{print substr($1,3)}' | sort –n

 
Netstat –anP tcp | grep ipaddress | wc –l  to grep the number of connections to the web server.

Input Output statistics ( Iostat )

Iostat command (Input Output statistics)
iostat reports terminal and disk I/O activity and CPU utilization. The first line of output is for the time period since boot & each subsequent line is for the prior interval . Kernel maintains a number of counters to keep track of the values. iostat's activity class options default to tdc (terminal, disk, and CPU). If any other option/s are specified, this default is completely overridden i.e. iostat -d will report only statistics about the disks.

syntax:
Basic synctax is iostat  < options > interval count

option - let you specify the device for which information is needed like disk , cpu or terminal. (-d , -c , -t or -tdc ) E gives error statistics and n expands disk names (-E, -n or –En). x options gives the extended statistics .
interval - is time period in seconds between two samples . 

iostat 4 will give data at each 4 seconds interval.
count - is the number of times the data is needed . 

iostat 4 5  will give data at 4 seconds interval  5 times.

Identifying the bottlenecks:
The values to look from the iostat output are:
•Reads/writes per second (r/s , w/s)
•Percentage busy (%b)
•Service time (svc_t)
If a disk shows consistently high reads/writes along with , the percentage busy (%b) of the disks is greater than 5 percent, and the average service time (svc_t) is greater than 30 milliseconds, then one of the following action needs to be taken
1.)Tune the application to use disk i/o more efficiently by modifying the disk queries and using available cache facilities of application servers .
2.) Spread the file system of the disk on to two or more disk using disk striping feature of volume manager /disksuite etc.
3.) Increase the system parameter values for inode cache , ufs_ninode , which is Number of inodes to be held in memory. Inodes are cached globally (for UFS), not on a per-file system basis.
4.) Move the file system to another faster disk /controller or replace existing disk/controller to a faster one.

The ouput The  fields  have  the  following  meanings:

disk - name of the disk
r/s  - reads per second
w/s  - writes per second
Kr/s - kilobytes read per second
Kw/s - kilobytes written per second
wait - average number of transactions waiting for service (Q length)
actv - average number of transactions actively being serviced (removed from the queue but not yet completed)
%w - percent of time there are transactions waiting for service (queue non-empty)
%b - percent of time the disk is busy (transactions in progress)

CPU Management ( psrinfo ) & ( prstat )

psrinfo  Command
To determine the number of processors in the system and their speed use the psrinfo -v command. In Solaris 10, -vp prints additional information.

prstar command

prstat  Command  (Determining User Consumption)

The CPU column of prstat always reports the percentage of system CPU resources a process is consuming and not the percentage of CPU resources of a processor or a processor set, even if the -C option is specified on the command line. For example, if there is a two-processor set on a four-processor system and a prstat -C is executed on the processor set, since the processor set has 50% of the system's CPUs, the total percentages of the CPU column will not exceed 50%.

To disable a processor the following command can be used:

psradm –f < processor number >
example: psradm –f 1

psradm –n < processor number >
example: psradm –n

Typical headers of the prstat command.

PID USERNAME SIZE RSS STATE PRI NICE TIME CPU-PROCESS/NLWP

PID – Process Id
USERNAME – Name of the user using the processor SIZE – Process Size
RSS - Resident Set Size ( Size of the process on the memory)
STATE – State of the process / Cpu information
PRI - Priority NICE – Nice value
TIME – Amount of time the process is running on the CPU
CPU PROCESS/NLWP – Command/Number of light weight processes / Threats
NPROC – Number of processes
USERNAME – Name of the User
MEMORY – Percentage of memory consumed.

Memory Management (Vmstat)

 

Vmstat

Vmstat reports virtual memory statistics of  process, virtual memory, disk, trap and CPU activity.

On multicpu systems, vmstat averages the number of CPUs into the output. For per-process statistics .Without options, vmstat displays a one-line summary of the virtual memory activity since the system was booted.

Syntax:                    
Basic syntax is vmstat interval count               
option - let you specify the type of information needed such as paging -p , cache    -c ,.interrupt -i etc.             
If no option is specified information about process, memory, paging, disk, interrupts & CPU is displayed.                            

Interval - is time period in seconds between two samples.

vmstat 4 will give data at each 4 seconds interval.                       
Count - is the number of times the data is needed.
vmstat 4 5 will give data at 4 seconds interval    5 times.   

Other essential options which can be used with vmstat:

vmstat  -s  gives  the  summary  of  the  statistics

vmstat  –p  gives  the  information  about  paging  activity.

           

The following command displays a summary of what the system is doing every five seconds.

Example:  vmstat  5

procs	memory			page						disk			faults		cpu
r  b  w	swap	free  re  mf		pi	po	fr	de	sr	s0  s1		s2	s3	in	sy	cs	us  sy  id
0  0  0	11456  4120  1		41	19	1	3	0	2	0	4	0	0	48  112		130	4	14  82
0  0  1	10132  4280  0		4	44	0	0	0	0	0  23		0	0  211  230			144	3	35  62
0  0  1	10132  4616  0		0	20	0	0	0	0	0  19		0	0  150  172			146	3	33  64
0  0  1	10132  5292  0		0	9	0	0	0	0	0  21		0	0  165  105			130	1	21  78

The fields of vmstat's display are

procs
r    in run queue
b    blocked for resources I/O, paging etc.
w    swapped

memory (in Kbytes)   
swap - amount of swap  space currently available
free  - size of the free list  
page ( in units per second).   
re    page reclaims - see -S option for how this field is modified.
mf    minor faults - see -S option for how this field is modified.
pi    kilobytes paged in   
po    kilobytes paged out   
fr    kilobytes freed   
de   anticipated short-term memory shortfall (Kbytes)
sr    pages scanned by clock algorithm   

disk ( operations per second )
There are slots for up to four disks, labeled with a single letter and number.
The letter indicates the type of disk (s = SCSI, i = IPI, etc). The number is the logical unit number.

faults
in   (non clock) device interrupts
sy   system calls
cs   CPU context switches

cpu -  breakdown of percentage usage of CPU time. On multiprocessors this is an average across all processors.
us   user time
sy   system time
id   idle time

Identifying the bottlenecks:

1.) If the number of processes in run queue (procs r) are consistently greater than the number of CPUs on the system it will slow down system as there are more processes then available CPUs .

2.) If this number is more than four times the number of available CPUs in the system then system is facing shortage of cpu power and will greatly slow down the processess on the system.

3.) If the idle time (cpu id) is consistently 0 and if the system time (cpu sy) is double the user time (cpu us) system is facing shortage of CPU resources. The system time should not be more than the user time at any give instance.

4.) Memory bottlenecks are determined by the scan rate (sr) . The scan rate is the pages scanned by the clock algorithm per second. If the scan rate (sr) is continuously over 200 pages per second then there is a memory shortage. Scan rate and page out should be zero or 1.