Monday, February 17

Monitoring

Threads, Requests & Heap from console

To Monitor Current Heap -

Log in to Admin console
Select your server for which you want to monitor heap
On right hand side, click on Monitoring - > Performance tab
Heap Current Size   : The current size (in bytes) of the JVM heap.
Heap Free Current  : The current amount of memory (in bytes) that is available in the JVM heap.
Heap Free Percent: Percentage of the maximum memory that is free.

Heap Size Max : The maximum free memory configured for this JVM.

To Monitor Threads -

Log in to Admin console
Select your server for which you want to monitor heap
On right hand side, click on Monitoring - > Threads tab


Active Execute Threads : 
An array of the threads currently processing work in the active thread pool.
Execute Thread Total Count : 
The total number of threads in the pool.
Executive Thread Idle Count :
The number of idle threads in the pool. This count does not include standby threads and stuck threads. The count indicates threads that are ready to pick up new work when it arrives
Queue Length :
The number of pending requests in the priority queue. This is the total of internal system requests and user requests.
Pending User Request Count :
The number of pending user requests in the priority queue. The priority queue contains requests from internal subsystems and users. This is just the count of all user requests.
Completed Request Count :
The number of completed requests in the priority queue.
Hogging Thread Count :
returns the threads that are being hogged by a request right now. These threads will either be declared as stuck after the configured timeout or will return to the pool before that. The self-tuning mechanism will backfill if necessary.
Stand By Thread Counts :
Returns the number of threads in the standby pool. Threads that are not needed to handle the present work load are designated as standby and added to the standby pool. These threads are activated when more threads are needed.
Throughput :
The mean number of requests completed per second


IN  Self-Tuning Thread Pool Threads
Name : Queue Name
Total Requests : The number of requests that have been processed by the queue.
Current Request : Returns a String representation of the request this thread is currently processing.
User : Returns the name of the ExecuteThread.
Idle :  Returns true if the execute thread has no work assigned to it.
Stuck : Returns true if the execute thread is being hogged by a request for much more than the normal execution time as observed by the scheduler automatically. If this thread is still busy after the stuck thread max time, then it is declared as stuck.
Hogger :  Returns true if the execute thread is not a part of the active thread pool.
Stand By :  Returns the transaction that the execute thread is currently working on behalf of.

To Monitor Requests -

Log in to Admin console
Select your server for which you want to monitor heap
On right hand side, click on Monitoring - > Workload tab

Pending Requests : The number of waiting requests in the queue or Pending requests that are waiting for an available thread.
Completed Requests : The number of requests that have been processed
Executing Requests : Number of requests that are currently executing.
Max Wait Time : The max time a request had to wait for a thread. Only requests whose execution is needed to satisfy the constraint are considered.
Deferred Requests :  Number of requests that are denied a thread for execution because the constraint is exceeded.


Friday, February 14

FTP

This one is used to copy the file to server or copy from server

Here we are discussing copy the file to server.

Open an command prompt,
Goto the file location, run the following commands.
  
D:\home\files'>'ftp SERVER_NAME
Connected to SERVER_NAME.
220 SERVER_NAME FTP server (Version wu-2.6.2+Sun) ready.
User (SERVER_NAME:(none)): KRISHNA_USERNAME
331 Password required for savvion.
Password: *******
230 User KRISHNA-SERVER logged in.
ftp'>' bin
200 Type set to I.
ftp'>' hash
Hash mark printing On  ftp: (2048 bytes/hash mark) .
ftp'>' cd /WEBLOGIC-HOME/APPS/KRISHNA/FILES/ 
250 CWD command successful.
ftp'>' get FILE_NAME
200 PORT command successful.
150 Opening BINARY mode data connection for FILE (498 bytes).

226 Transfer complete.
ftp: 498 bytes received in 0.00Seconds 498000.00Kbytes/sec.
ftp'>' bye
221-You have transferred 498 bytes in 1 files.
221-Total traffic for this session was 1001 bytes in 1 transfers.
221-Thank you for using the FTP service on KRISHNA-SERVER.

221 Goodbye.


Note: if you want to copy from server just use 'put' command.

Thread Dump


Thread dump provides a snapshot of the current active live threads. It provides the stack trace of all the java threads in the JVM. It is used when the server is hung and we want to see the threads executing and take their dump.

There are different ways to take thread dump.

1.Command line:  use the below command when you know the port number. just run the same command 3 times in 5 sec intervel of gaps.
kill -3 PID

2.WebLogic.Admin utility:  javaWebLogic.Admin adminurl -username WebLogic -password WebLogic1 THRED_DUMP

3.Admin console: 
Step1: login to the admin console
Step2: Click on server
Step3: Navigate to servers
Step4: Click monitor tab
Step5: Click on tread
Step6: Click on the dumpthread stack.

Location of the Thread Dump:  
/DOMAIN_NAME/servers/logs/KK_stdout.log

======================================================

Thread Dump:

  1. What is Thread dump?
  2. When we will take Thread dump? (Scenarios)
  3. How  Many  ways take Thread Dumps
  4. Thread Dump Generating Procedure
  5. What can I Analysis with Thread Dump?
  6. How can I analysis thread dump?
  7. Actions taken for Issue resolving
  8. References


Coming to step by step learning:
--------------------------------

What is Thread dump?

Thread Dump is a textual dump of all active threads and monitors of Java apps running in a Virtual Machine.

When we will take Thread dump? (Scenarios)

1.  Scenario 1: when server is hang Position, i.e. that time server will not respond to coming requests.

2.  Scenario 2: While sever is taking more time to restart

3.  Scenario 3: When we are Getting exception like “java.lang.OutOfMemoryException”

4.  Scenario 4: Process running out of File descriptors. Server cannot accept further requests because sockets cannot be created

5.  Scenario 5: Infinite Looping in the code


How many ways take Thread Dumps?

Many types we have to take a Thread dumps. As per your flexibility you can choose one Procedure. For analyzing take dumps some Intervals (like every 10mins, 10mins etc.).



Generating Dump Talking Procedures

1. Take Thread dump from Console by Using of below command
    $kill -3 PID
   (For Getting PID, Use this Command ps –ef | grep “java”)
Here The Output of the Thread Dump will be generated in the Server STDOUT.
(Note: If a process is not responding to kill -3 then it’s a JVM bug.)


2. Generation Thread Dump via Admin Console

a.  login to Admin Console(with Admin Username/Password)
b.  Click on Server, after choose your server
c.  Goto Monitoring TAB
d.  Goto Threads TAB, after click on “Dump Thread Stack” Button
e.  Now you can view the all the Threads in Same page
f.  Copy and paste in a txt file.

3.  We can Collect Thread Dump Using “WebLogic.Admin” which is deprecated, but still available or may be available in near future as well As i think because it is one of the best debugging utility for Admins.

  java WebLogic.Admin -url t3://hostname: port -username Weblogic -password Weblogic THREAD_DUMP


This Thread Dumps will be generated in Servers STDOUT file


4. Getting Thread Dumps by using Jstack Utility

    a.jstack –m  (to connect to a live java process)

    b. jstack –m [server_id@]
                (to connect to a remote debug server)
    (-m Means print both java and native frames (mixed mode)) 

5. By Using WLST Script, can contain extension of (.py)

connect(‘weblogic’,'weblogic’,'t3://hostname:port′) cd (”Servers’) ls() cd (‘AdminServer’) ls() threadDump()

 Execute this Script in console. 

What can I Analysis with Thread Dumps?
We need to analyze the thread dumps for analyzing running threads and their states to identifying.

How can I analysis thread dumps?

For analyze thread dumps we have lots of tools to understand easily thread states

1.  samurai tool :

    In this tool you can identify all the Thread states by     identifying colors. We need to take care about Deadlocks and waiting state threads.

   More Details:

    $ java -jar samurai.jar

     After running we will get a Screen like below



     Goto Thread dump tab
    When Samurai detects a thread dump in your log, a tab named "Thread Dump" will appear.



You can just click "Thread dumps" tab to see the analysis result. Samurai colors idle threads in gray, blocked threds in red and running threds in green. There are three result views and Samurai shows "Table view" by default. In many case you are just interested in the table view and the sequence view. Use the table view to decide which thread needs be inspected, the sequence view to understand the thread's behavior. You should takecare especially threds always in red.
  


2.  TDA Tool :

 Actions taken for Issue resolving

1.  Classic Dead Locks : Look for the threads waiting for monitor entry

For Example :

"ExecuteThread: '95' for queue: 'default'" daemon prio=5 tid=0x411cf8 nid=0x6c waiting for monitor entry [0xd0f80000..0xd0f819d8]
    at weblogic.common.internal.ResourceAllocator.release(ResourceAllocator.java:766)
    at weblogic.jdbc.common.internal.ConnectionEnv.destroy(ConnectionEnv.java:590)
Reason: The above thread is waiting to acquire lock on Resource Allocator object. The next step is to identify the thread that is holding the Resource Allocator object
"ExecuteThread: '0' for queue: '__weblogic_admin_rmi_queue'" daemon prio=5 tid=0x41b978 nid=0x77 waiting for monitor entry [0xd0480000..0xd04819d8]
    at weblogic.jdbc.common.internal.ConnectionEnv.getPrepStmtCacheHits(ConnectionEnv.java:174)
    at weblogic.common.internal.ResourceAllocator.getPrepStmtCacheHitCount   (ResourceAllocator.java:1525)
Reason: This thread is holding lock on source Allocator object, but is waiting for Connection Env object. This is a classic deadlock.

     

2.  Threads in wait() state:
   A sample dump:

"ExecuteThread: '10' for queue: 'SERV_EJB_QUEUE'" daemon prio=5 tid=0x005607f0 nid=0x30 in Object.wait() [83300000..83301998]
  at java.lang.Object.wait(Native Method)
  - waiting on <0xc357bf18> (a weblogic.ejb20.pool.StatelessSessionPool)
at weblogic.ejb20.pool.StatelessSessionPool.waitForBean(StatelessSessionPool.java:222)

Reason: The above thread would come out of wait() under two conditions
 (Depending on application logic)
1) One of the thread available in the execute queue pool would call notify() on this object when an instance is available. (If the wait() is indefinite).
  This can cause the thread to hang for ever if server never does a notify() to this object.

2) If the timeout exceeds, the thread would throw an exception and back to execute queue thread pool.

Node Manager in Oracle WebLogic Server

Node Manageris Weblogic Server utility to start, stop and restartAdministration and Managed Server Instances from remote location (There are other ways as well to start/stop Weblogic – Node Manager is optional component).

1. Node Manager Process is associated with a Machine and NOT with specific Weblogic Domain (i.e. Use one node manager for multiple domains on same machine)

2. There are two versions of Node Manager - Java-based and Script-based
Java-based node manager – runs with in JVM (Java Virtual Machine) Process and more secure than script-based node manager. Configuration for java-based node manager are stored in nodemanager.properties
Script-based node manager – is available for Linux and Unix systems only and is based on shell script.

3. Access Node Manager
- From Administration Console : Environments -> Machines -> Configuration -> Node Manager

 4. Default port on which node manager listen for requests is localhost:5556, When you configure Node Manager to accept commands from remote systems.

5. Any domains created after the Node Manager Service has been installed should not have to be enrolled against the Node Manager. The Node Manager should automatically be ‘reachable‘ by the domain.
.
How to Configure Node Manager?




Log in to weblogic Admin Console
Click on Environments -> Machines tab on left hand side













Click on new tab on right hand side








Enter name of machine and select Machine OS as unix if you are on unix machine otherwise default, click ok






Now click on created machine






Select type plain, enter listen address and port 











To check status :-
Start node manager, click on  machine name, click on Monitoring -> Node Manager Status
click on node manager tab, Status should be Reachable.









How to start Node Manager ?
$WL_HOME\server\bin\startNodeManager.sh
To Stop Node Manager 
In Unix/Linux, kill the process -
kill -9 `ps -ef | grep -i nodemanager.javahome | grep -v grep | awk {'print $2'} | head -1`

Node Manager Log Files
$WL_HOME/common/nodemanager/nodemanager.log

Important configuration file for node manager
$WL_HOME/common/nodemanager/nodemanager.properties- used by java based node manager. This file is created on first time start of nodemanager (startNodeManager.sh)

$WL_HOME/common/nodemanager/nodemanager.domains - contains mappings between the names of domains managed by Node Manager and their corresponding directories.
entry like
base_domain=D:\Oracle\Middleware\user_projects\domains\base_domain
soa_domain=D:\Oracle\Middleware\user_projects\domains\soa_domain

$WL_HOME/common/nodemanager/nm_data.properties- This file stores the encryption data the Node Manager uses as asymmetric encryption key. This file is created on first time start of nodemanager (startNodeManager.sh)

$DOMAIN_HOME/config/nodemanager/nm_password.properties- This file stores the Node Manager username and password used by Admin Server to connect to Node Manager.




Why node manager is required?
 A Node Manager process is not associated with a specific WebLogic domain but with a machine. 
You can use the same Node Manager process to control server instances in any WebLogic Server domain, as long as the server instances reside on the same machine as the Node Manager process. Node Manager must run on each computer that hosts WebLogic Server instances. whether Administration Server or Managed Server that you want to control with Node Manager.

How to configure Node Manager in Weblogic.? Can please someone provide the steps for it.?
Raja Reddy 1. Start Admin server of domain which you want to enroll with node manager

2. Go to wlst prompt ( go to your_domain, run setDomainEnv and then "java weblogic.WLST")


3. Connect with admin server
connect("user","password","host:admin_port")

4. Run nmEnroll command
nmEnroll('C:/Oracle/Middleware/wlserver_10.3/common/nodemanager')

Where "C:/Oracle/Middleware/wlserver_10.3/common/nodemanager" is the node manager 
home of node manager with which you want to enroll your domain

5. Update nodemanager.domains under node manager home and add your domain path in 
same format as earlier domains path there, like below i have added domain in red

JJTU_Domain1=C\:\\Oracle\\Middleware2\\user_projects\\domains\\JJTU_Domain1
base_domain=C\:\\Oracle\\Middleware2\\user_projects\\domains\\base_domain 

6. Update node manager password ( username also if not correct there ) from admin console
( login to admin console > click on your domain name on left hand side > click on security tab 
on right hand side > click on Advance link > update node manager password ( username 
also if not correct there )

Now you can able to connect with node manager

wls:/base_domain/serverConfig> nmConnect("weblogic","weblogic123","localhost","40046","base_domain","C:/Oracle/Middleware2/user_projects/domains/base_domain","plain")
Connecting to Node Manager ...
Successfully Connected to Node Manager.



nodemanager.properties
This is the configuration file used by the Java-based version of Node Manager.
This file is located in WL_HOME/common/nodemanager.

nodemanager.hosts
This file contains a list of all the trusted hosts that can issue commands to Node Manager.
This file is located in WL_HOME/common/nodemanager.
nodemanager.domains
This file contains mappings between the names of domains managed by Node Manager and their corresponding directories. 
This file is located in WL_HOME/common/nodemanager.
nm_data.properties
This file stores the encryption data the Node Manager uses a symmetric encryption key. The data is stored in encrypted form.
This file is located in WL_HOME/common/nodemanager.
nm_password.properties
This file stores a username/password pair specific to the Node Manager server that is managing this domain. This is known as the Node Manager secret. The username and password are appended to a salt value (obtained from the SerializedSystemIni.dat of the domain) and SHA-hashed.
This file is located in DOMAIN_HOME/config/nodemanager.