Creating JDBC Connection Pool/JDBC Data Source

Connection Pool Creation:

1.    Start admin server
 
2.    Go to http:///console and login
 
3.    Go to Services/JDBC/Connection Pools jdbc

4.    Click  Configure a new JDBC Connection Pool...

     a.    Select Database Type : Oracle
     b.    Select Database Driver  : Oracle’s Driver (Thin) version 9.0.1, 9.0.2,10
     c.    Click Continue
     d.    Enter following values
           i.    Name: XXXXXXXXXXX
           ii.    Database Name: XXXXXXX
           iii.    Host Name: XXXXXXXXXXXXX
           iv.    Port: XXXXXX
           v.    Database User Name: XXXXXXXXXXX
           vi.    Password: XXXXXX
           vii.    Confirm Password: XXXXXXXXXX
     e.    Click Continue
     f.    Click Test Driver Configuration, this must result in successful test. If not, review above steps
     g.    Select ejb and portal server as targets
     h.    Click Create and deploy
     i.    This will bring you back to Connection Pools list with new pool listed

5.    Now go to Services/JDBC/Data Sources and click

Configure a new JDBC Data Source
 
1.    Enter pcnDataSource for Name and JNDI Name
 
2.    Enable Emulate Two-Phase Commit for non-XA Driver
 
3.    Click Continue
 
4.    Select pcnPool from Pool Name list
 
5.    Select ejb and portal servers for target
 
6.    Click Create

Thread Dump will be printed to the servers standard out (by default, the shell in which the server is running)

On Windows machine:

cltr + break

On Unix Machine:

get pid using ps -ef grep java
kill -3 (process id)

 

When weblogic is running as windows service

For Out file add in JAVA_OPTIONS:
-Dweblogic.Stdout=

Using beasvc :

beasvc –dump – svcname:”bea Admin_server”

Using Weblogic Admin utility

C:\bea\user_projects\domains\rDomain>java weblogic.Admin -url t3://localhost:7001
-username weblogic -password weblogic THREAD_DUMP

Collecting thread dumps from the console:

Server > Monitoring > Threads > Dump Thread Stacks

Using WLST:

java weblogic.WLST
connect("weblogic","weblogic","t3:\\localhost:7001")
threadDump()

---

Script to take thread dump prstat and pstack.

#
# Takes an argument (PID of the WLS process) and loops three times.
# This will append the prstat information to a file called dump_high_cpu.txt.
# The thread dump information will either be in file where stdout was redirected or printed on the screen.
#

 

if test $# -eq 0
then
    echo "usage: $0 ";
else

 

                for loopnum in 1 2 3
                do
                        echo `date`
                        CURDATIME=`date '+%y%m%d_%H%M%S'`
                        echo $CURDATIME
                   prstat -L -p $1 1 1 >> ~/export/home/username/`hostname`_${CURDATIME}_$1.out
                   pstack $1  >> ~/export/home/username/`hostname`_${CURDATIME}_$1.out
                   kill -3  $1
                   echo "prstat, pstack, and thread dump done. #" $loopnum
                   sleep 1
                   echo "Done sleeping."
                done

 

---

Tools to analyze thread dumps (downloads)

1. Samurai

2. Thread dump analyzer.

Weblogic Issues Troubleshooting

1. Port Conflict Issue:

While configuring a new WebLogic instance and starting it, that might be get an issue like : "Port already in use". There could be many reasons for this one.

1. on the same machine multiple standalone instances might be running one of the instance already used that port which you have given for new configuration. 

2. apache might be running with the same port.

3. middleware might be running on the same machine with same port 

On Solaris Operating environment we have 2 options:

1. using pfiles command

netstat –na|grep --> identify port in use  

pfiles |grep -i sockname |grep port --> look for every java process is initialized by startWebLogic.sh or startManagedWebLogic.sh

2. Another way costly one (Third party package) to find the process that is using particular port is :

lsof -i tcp:

3. Best way is perl script using a method it will check only standard ports which are used by the system.

getservbyport(int port_number, const char *protocol_name) 

#!/usr/bin/perl

($name, $aliases, $port_number, $protocol_name) = getservbyport(7001, "tcp");

print "Name = $name\n";

print "Aliases = $aliases\n";

print "Port Number = $port_number\n";

print "Protocol Name = $protocol_name\n";

2.  Config.xml Repository Issue

When I copy pasted few of the JMS Queue related configurations from existing WLS config.xml to newly created WLS Domain's config.xml. I got stuck the control at following line in the WebLogic Server logs.

<="" at="" configuration="" domain="" from="" path="" repository="" span="">

Sol: Verify all the lines in the config.xml file there might be missing any xml tags missing. I have missed out one line need to correct it now it is working fine.

 3. Multicast Failure

while running the managed server on a remote server getting the following exception:

java.net.SocketException: Socket is closed
        at java.net.DatagramSocket.send(DatagramSocket.java:577)

Sol:

a).  check for the ping happening between the cluster involved machines.

b).  Verify the Multicast IP with the MulticastTest with multiple ips and with ports then use which is communicating the message between two participants.

AppMachine 1: java utils.MulticastTest -n  app1  -a 237.0.0.5 -p 9999

WebMachine 2:java utils.MulticastTest -n  app2  -a 237.0.0.5 -p 9999

Multicast Issues

The messages must exchange between them... otherwise check with Network team.

c). Multcast heart beat can be visible by the following command line:

java weblogic.cluster.MulticastMonitor mydomain mycluster

4. Application not accessible

While accessing the application url after Web App deployment.

<[ServletContext(id=29754737,name=webapp1,context-path=/contextpath)] Root cause of ServletException.
java.lang.StackOverflowError

 Sol: Check the classpath it is the communication gap between Servlet and an EJB

5. Editing the WLS password

New password is given in boot.properties when server is in offline.

 The Admin server started but while starting the managed instances got the following error:

********************************************************
The WebLogic Server did not start up properly.
Reason: weblogic.security.SecurityInitializationException: Authentication denied: Boot identity not valid; The user name and/or password from the boot identity file (boot.properties) is not valid. The boot identity may have been changed since the boot identity file was created. Please edit and update the boot identity file with the proper values of username and password. The first time the updated boot identity file is used to start the server, these new values are encrypted.
************************************************************

Sol: Resolved by editing in boot.properties, startManagedWebLogic.sh files for the new password in 'CLEARTEXT' pattern and rename ldap folder in the Managedserver instance folder present in the domain folder.

While starting up the server instance boot.properties will be encrypted and re-written back. after this ldap folder will be generated in the domain directory for the Managed instance where new file will be created with instance name with extention as .ldif

Apache Plug-in configuration

While configuring the Apache using the instructions given in the following link :

6. Deployment Error

 Module Name: apollo, Error: [HTTP:101179][HTTP] Error occurred while parsing descriptor in Web application "/domainpath/application.war" [Path="/source_code_path", URI="applicaiton.war"
org.xml.sax.SAXParseException: Element type "local-path" must be followed by either attribute specifications, ">" or "/>".
Sol:

Check the Deployment Descriptors (here weblogic.xml) edit as per the given Line.

7. JNDI Issue

While deploying application which is having multiple JMS Queues getting as:

data couldn't be saved. The exception is : Error while doSenderQueueLookUp for :QUEUEJNDIUnable to resolve 'QUEUEJNDI' Resolved

Sol:

Please verify the Persistance Store is connected to Connection Pool.

8.  New managed server Configuration issue

Here I am with set of errors when newly configuring WebLogic 9.2 MP3 managed server and adding to a webcluster. The managed server when started first time shown the following :

1. BEA-000362

2. BEA-000386

On Solaris box:

<="" span="">

There are 1 nested errors:

java.net.UnknownHostException: mywebhostname.com: myhostname.com

        at java.net.InetAddress.getAllByName0(InetAddress.java:1128)

        at java.net.InetAddress.getAllByName0(InetAddress.java:1098)

        at java.net.InetAddress.getAllByName(InetAddress.java:1061)On 

On Linux box:

  <="">

weblogic.ldap.EmbeddedLDAPException: Unable to open initial replica url: http://myadminhost.com:7001/bea_wls_management_internal2/wl_management

        at weblogic.ldap.EmbeddedLDAP.getInitialReplicaFromAdminServer(EmbeddedLDAP.java:1319)

        at weblogic.ldap.EmbeddedLDAP.start(EmbeddedLDAP.java:221)

        at weblogic.t3.srvr.SubsystemRequest.run(SubsystemRequest.java:64)

        at weblogic.work.ExecuteThread.execute(ExecuteThread.java:209)

        at weblogic.work.ExecuteThread.run(ExecuteThread.java:181)

java.io.FileNotFoundException: Response: '500: Internal Server Error' for url: 'http://myadminhost.com:7001/bea_wls_management_internal2/wl_management'

        at weblogic.net.http.HttpURLConnection.getInputStream(HttpURLConnection.java:472)

        at weblogic.ldap.EmbeddedLDAP.getInitialReplicaFromAdminServer(EmbeddedLDAP.java:1296)

        at weblogic.ldap.EmbeddedLDAP.start(EmbeddedLDAP.java:221)

        at weblogic.t3.srvr.SubsystemRequest.run(SubsystemRequest.java:64)

        at weblogic.work.ExecuteThread.execute(ExecuteThread.java:209)

        Truncated. see log file for complete stacktrace

Solution/Work around:

BEA-000362

Here we need to investigate for the network communication happening or not with the hostnames given in the config.xml. ping from managed server box to admin server hosting dns. And also check the /etc/hosts updated with the new host dns. Some times DNS table entries not updated in WAN domains may cause this issue.

BEA-000386

   Check for the hostname of the managed server listen-address given one and hosting dns/ip is matching or not using the WebLogic console(this could be a time taking process). 

   Unix aware guy always prefer grep on config.xml :)

weblogic Introduction

WebLogic Server

“WebLogic is a server software application that runs on middle tier, between backend databases, related applications and browser based thin clients. “
Paul Ambrose, Bob Pasker, Laurie Pitman, and Carl Resnikoff founded WebLogic, Inc. in September 1995. The founders worked together to pursue what eventually became the "Application Server".
BEA Systems acquired WebLogic, Inc. in 1998.
Oracle Corporation acquired BEA Systems in 2008.
Following the success of the Java platform, the term application server sometimes refers to a J2EE or Java EE 5 application server. Some of the better-known Java Enterprise Edition application servers include:
 WebSphere Application Server and WebSphere Application Server Community Edition (IBM)
 Sybase Enterprise Application Server (Sybase Inc)
 Oracle WebLogic Server (Oracle)
 ATG Dynamo (ATG)
 JBoss (Red Hat)
 JRun (Adobe Systems)
 Apache Geronimo (Apache Software Foundation)
 Oracle OC4J (Oracle)
 Sun GlassFish Enterprise Server (based on GlassFish Application Server) (Sun Microsystems)
 SAP Netweaver AS (ABAP/Java) (SAP)
 Glassfish Application Server (open source)
 WebObjects (Apple Inc.)
 Pramati Application Server (Pramati Technologies)

Application Architecture:

WebLogic Features:

WebLogic Server provides essential features for developing and deploying mission-critical e-commerce applications across distributed, heterogeneous computing environments. These features include the following:

 Standards leadership—Comprehensive enterprise Java supports to ease the implementation and deployment of application components. WebLogic Server is the first independently developed Java application server to achieve J2EE certification. In addition, BEA actively participates in the development of J2EE and Web Services standards that drive innovation and advancement in Java and XML technology.

 Rich client options—WebLogic Server supports Web browsers and other clients that use HTTP; Java clients that use RMI (Remote Method Invocation) or IIOP (Internet Inter-ORB Protocol); SOAP clients on any SOAP-enabled plaftorm; and mobile devices that use (WAP) Wireless Access Protocol. Connectors from BEA and other companies enable virtually any client or legacy application to work with a WebLogic Server application.

 Flexible Web services—WebLogic Server provides a solid platform for deploying Web services as components of a heterogeneous distributed application. Web services use a cross-platform, cross-language data model (XML) to provide interoperability among application components on diverse hardware and software platforms. Web services support user-defined data types and one-way asynchronous operations. A Web service can intercept SOAP messages for further processing. New Ant tasks automatically generate important components and package the service into a deployable EAR file.

WebLogic Server uses Web Services Description Language (WSDL) 1.1, an XML-based specification, to describe Web services. WebLogic Web services support Simple Object Access Protocol (SOAP) 1.1 and 1.2 as the message format and HTTP as a connection protocol.
Note: WebLogic Web services accept both SOAP 1.1 and 1.2 incoming requests, but produce only SOAP 1.1 outgoing responses.

 Enterprise e-business scalability—Efficient use and high availability of critical resources are achieved through Enterprise JavaBean business components and mechanisms such as WebLogic Server clustering for dynamic Web pages, backend resource pooling, and connection sharing.

 Robust administration—WebLogic Server offers a Web-based Administration Console for configuring and monitoring WebLogic Server services. A command-line interface for configuration makes it convenient to administer WebLogic Servers with scripts.

 E-commerce-ready security—WebLogic Server provides Secure Sockets Layer (SSL) support for encrypting data transmitted across WebLogic Server, clients, and other servers. User authentication and authorization for all WebLogic Server services are provided through roles and security providers. External security stores, such as Lightweight Directory Access Protocol (LDAP) servers, can still be adapted to WebLogic realms, enabling single sign-on for the enterprise. The Security Service Provider Interface makes it possible to extend WebLogic Security services and to implement WebLogic Security features in applications.

 Maximum development and deployment flexibility—WebLogic Server provides tight integration with and support for leading databases, development tools, and other environments.

 Bi-directional functional interoperability between Java/J2EE objects and Microsoft ActiveX components—BEA WebLogic jCOM provides a run-time component that implements both Component Object Model (COM)/Distributed Component Object Model (DCOM) and Remote Method Invocation (RMI) distributed components infrastructures. This makes the objects look like native objects for each environment.

 Java Message Service (JMS)—An enterprise messaging system, also referred to as message-oriented middleware (MOM), enables applications to communicate with one another through the exchange of messages. A message is a request, report, and/or event that contains information needed to coordinate communication between different applications. A message provides a level of abstraction, allowing you to separate the details about the destination system from the application code.
The Java Message Service (JMS) is a standard API for accessing enterprise messaging systems. Specifically, JMS enables Java applications sharing a messaging system to exchange messages, and it simplifies application development by providing a standard interface for creating, sending, and receiving messages.

WebLogic Versions
 WebLogic Server 11g (10.3.1) JDK 160_11
 WebLogic Server 10.3 JDK 160_05
 WebLogic Server 10.0 JDK 150_11
 WebLogic Server 9.2 JDK 150_12
 WebLogic Server 9.1 JDK 150_13
 WebLogic Server 9.0 JDK
 WebLogic Server 8.1 JDK !42_11
 WebLogic Server 7.0 JDK 131_06
 WebLogic Server 6.1 JDK 131_01
 WebLogic Server 6.0 JDK
 WebLogic Server 5.1 JDK 1.3.1_09

Hardware Requirements:
A Windows computer must have the following minimum configuration.
• A Pentium-class processor running at 400 MHz or faster
• Random Access Memory (RAM)
• 1GB recommended
• 512MB minimum
• 5G available hard drive space
• SVGA monitor
• CD-ROM drive (for installation)
• TCP/IP network interface
• Local area network (LAN) card and persistent Internet connection is required.

Protocol Support:
Client applications connect with WebLogic Server using standard networking protocols over TCP/IP. WebLogic Server listens for connection requests at a network address that can be specified as part of a Uniform Resource Identifier (URI).

Scheme Protocol
HTTP HyperText Transfer Protocol. Used by Web browsers and HTTP-capable programs.
HTTPS Hypertext Transfer Protocol over Secure Sockets Layer (SSL). Used by Web browsers and HTTPS-capable client programs.
T3 WebLogic T3 protocol for Java-to-Java connections, which multiplexes JNDI, RMI, EJB, JDBC, and other WebLogic services over a network connection.
T3S WebLogic T3 protocol over Secure Sockets Layer (SSL).
RMI Remote Method Invocation (RMI), the standard Java facility for distributed applications.
IIOP Internet Inter-ORB protocol, used by CORBA-enabled Java clients to execute WebLogic RMI objects over IIOP. Other CORBA clients connect to WebLogic Server with a CORBA naming context instead of a URI for WebLogic Server.
IIOPS Internet Inter-ORB protocol over Secure Sockets Layer (SSL).
SOAP WebLogic Web services use Simple Object Access Protocol (SOAP) 1.1 as the message format and HTTP as a connection protocol.

Installation Methods
The BEA installation program supports the following methods for installing the BEA WebLogic Platform software:

 Graphical mode
Graphical-mode installation is an interactive, GUI-based method for installing WebLogic Platform. It can be run on both Windows and UNIX systems. For installation procedures, see Installing WebLogic Platform Using Graphical-Mode Installation.
Note: If you want to run graphical-mode installation, the console attached to the machine on which you are installing the software must support a Java-based GUI. All consoles for Windows systems support Java-based GUIs, but not all consoles for UNIX systems do. If you attempt to start the installation program in graphical mode on a system that cannot support a graphical display, the installation program automatically starts console-mode installation.

 Console mode
Console-mode installation is an interactive, text-based method for installing WebLogic Platform, from the command line, on either a UNIX system or a Windows system. For instructions for using this method, see Installing WebLogic Platform Using Console-Mode Installation.

 Silent mode
Silent-mode installation is a noninteractive method of installing WebLogic Platform that requires the use of an XML properties file for selecting installation options. You can run silent-mode installation in either of two ways: as part of a script or from the command line. Silent-mode installation is a way of setting installation configurations only once and then using those configurations to duplicate the installation on many machines. For instructions, see Installing WebLogic Platform Using Silent-Mode Installation.
C:\ server923_win32.cmd –mode=silent –silent_xml=path_to_silent.xml –log=C:\temp\install.log

Difference between Application Server and Web Server

• Web Server serves pages for viewing in web browser, static HTML pages or gifs, jpegs, etc., and can also run code written in CGI, JSP etc. A Web server handles the HTTP and HTTPS protocol.
Ex: IBM web server, Apache web server, IIS web server, Sun one web server, etc.
• An Application Server is used to run business logic or dynamically generated presentation code. It can either .NET based or J2EE based.
Ex: BEA WebLogic Server, IBM WebSphere, JBoss, etc.
• The basic difference between a web server and an application server is
Webserver can execute only web applications i.e. servlets and JSPs and has only a single container known as Web container which is used to interpret/execute web applications.
• Application server can execute Enterprise application, i.e. (servlets, jsps, and EJBs) it is having two containers 1. Web Container(for interpreting/executing servlets and jsps) 2. EJB container (for executing Ejbs). It can perform operations like load balancing, transaction demarcation etc
• Webserver delegation model is fairly simple, when the request comes into the webserver; it simply passes the request to the program best able to handle it (Server side program). It may not support transactions and database connection pooling.
• Application server is more capable of dynamic behaviour than webserver. We can also configure application server to work as a webserver.Simply application server is a superset of webserver.

Administration Tools
• Configuration Wizard
GUI/scriptable tool to create and extend WebLogic domains Template based.
• Administration Console
Browser-based tool for configuring and monitoring domains, deploying applications, and controlling servers.
• WebLogic Scripting Tool (WLST)
Script or command line tool to do the same thing as the Administration Console and Configuration Wizard.
Note: that we will cover details on WLST in a separate document.
• weblogic.Admin
Deprecated command line tool for configuring a domain.
Recommend using WLST instead.
• weblogic.Deployer
Command line tool for deploying applications.

WebLogic Domains:
A domain is the basic administration unit for WebLogic Server instances. It consists of one or more WebLogic Server instances and their associated resources.
• Multiple domains can be created based on different system administrator’s responsibilities, application boundaries, or geographical locations of servers.
• A single domain can be used to centralize all WebLogic Server administration activities.
Each domain’s configuration is stored in a separate configuration file called config.xml, which is stored on the Administration Server along with other files such as logs. When the Administration Server is used to perform a configuration task, the changes apply only to the domain managed by that Administration Server.
The management of a domain is specific to the Administration Server for that domain. Accordingly, this means that server instances, applications, and resources in one domain should be treated as being independent of servers, applications, and resources in a different domain.

Contents of a Domain:
A domain can include multiple WebLogic Server clusters and non-clustered WebLogic Server instances. The scope and purpose of a domain can vary significantly. A minimal domain can contain only one WebLogic Server instance, which functions both as an Administration Server, and as a Managed server. Production Environment: 4 standalone Managed Servers, and a cluster of 4 Managed Servers. WLS Domain may contain Resources like Machine, Node Manager, etc. And services like JMS, JDBC,etc.

Administration Server
The Administration Server operates as the central control entity for the configuration of the entire domain. It maintains the domain's configuration documents and distributes changes in the configuration documents to Managed Servers. You can also use the Administration Server as a central location from which to monitor all resources in a domain.

Managed Server
Managed Servers host business applications, application components, Web services, and their associated resources. To optimize performance, Managed Servers maintain a read-only copy of the domain's configuration document. When a Managed Server starts up, it connects to the domain's Administration Server to synchronize its configuration document with the document that the Administration Server maintains.

RESET ADMIN USERNAME AND PASSW0RD

The following are the steps to be followed to reset the Admin UserName/Password.

1. Ensure that your have shutdown all servers -- Admin and Managed.

2. From the command line, run the script setDomainEnv.sh, located in $DOMAIN_HOME/bin/setDomainEnv.sh

3. cd to $DOMAIN_HOME/security
Rename the file 'DefaultAuthenticatorInit.ldift'

4. Create a new initialization file for the default authenticator in the same directory as above ($DOMAIN_HOME/security) by running command :
java weblogic.security.utils.AdminAccount <USERNAME_ADM> <PWD_ADM> $DOMAIN_HOME/security
This should produce a new 'DefaultAuthenticatorInit.ldift'
(can be verified with the time stamp of that file)

5. Rename the files 'DefaultAuthenticatormyrealmInit.initialized' and 'boot.properties'.
 - $DOMAIN_HOME/servers/AdminServer/data/ldap/DefaultAuthenticatormyrealmInit.initialized
- $DOMAIN_HOME/servers/AdminServer/security/boot.properties

6) Start the Admin server
At start time, use the newUser and newPassword values when prompted for user and password.

java -cp /WebLogic/utils/bea922/weblogic92/server/lib/weblogic.jar:$CLASSPATH weblogic.security.utils.AdminAccount

OVERVIEW OF WEBLOGIC ADMINISTRATION

OBJECTIVES

 WLS INSTALLATION
 WLS DOMAINS
 NODEMANAGER
 SERVER LIFE CYCLE
 ADMIN CONSOLE

WLS INSTALLATION

THREE MODES OF INSTALLATION
 GRAPHICAL
 CONSOLE
 SILENT

GRAPHICAL
 Installation wizard takes you through the series of steps obtaining the configuration information required for installation.
Steps for installation:
 Download the software
//Windows:
 Go to the directory where you downloaded the installation program and double-click the installation file
//Unix:
 Launch the installation by entering the following command:
 chmod a+x filename.bin
./filename.bin


CONSOLE MODE

 Download the software as described
 Go to the directory in which you downloaded the installation program.
//Windows
 net_platformXXX_win32.exe -mode=console (net installer)
 platformXXX_win32.exe -mode=console (WebLogic Platform package installer)
 serverXXX_win32.exe -mode=console (WebLogic Server package installer)
(In these filenames, XXX represents the version number of the software you are installing)
//Unix
 chmod a+x filename.bin
./filename.bin -mode=console
Configuration Information
 BEA License Agreement
 Choose BEA Home Directory
 Choose Install Type
 Choose Components
 Specify Download Options
 Choose Product Directory
 Install as Services

SILENT MODE
 The installation program reads the settings for configuration from an XML file
Steps:
 Create a template file that defines the configuration settings
 Start Installation using the following command
Windows:
filename.exe -mode=silent –silent_ xml=/path_to_silent.xml
Unix:
chmod a+x filename
./filename.bin -mode=silent -silent_xml=/path_to_silent.xml

WLS DOMAINS
What?
How?

WLS DOMAINS – What?
 basic administrative unit for a WebLogic Server installation
 logically related group of WebLogic Server resources that you manage as a unit.
 Administration Server serves as a central point of contact
 include additional WebLogic Server instances called Managed Servers
 domain's configuration is stored in a separate configuration file called config.xml
 can include multiple WebLogic Server clusters and non-clustered WebLogic Server instances

Components of Weblogic Server Domain
 Servers
 Clusters
 Machine
 Deployments
 Applications
 EJB Modules
 Web Application Modules
 Connector Modules
 Services
 JDBC
 JMS
 Virtual Hosts
 Mail

Creating a new Domain

 Using the Configuration Wizard
 weblogic.Server Command
 Using Ant Tasks

Note: A domain can be created either in development or production mode.
Development mode uses a relaxed security configuration and enables you to auto-deploy applications.
A production domain uses full security and may use clusters or other advanced features

NODE MANAGER
 Java utility that runs as separate process from WebLogic Server
 can start and stop the Managed Servers remotely using the Administration Console or from the command line
 automatically restart a Managed Server after an unexpected failure.
 Monitor the self-reported health of Managed Servers and automatically kill server instances whose health state is "failed“
 Default Configuration for Nodemanager exists and it can be directly started by executing the startNodeManager script.

Configure Nodemanager in production Environment
Production Environment:
 nodemanager.hosts - IP address or DNS name entries are made
 Configure a Machine - for Node Manager to accept commands from remote Administration Servers
 Configure SSL – for secure communication b/w nodemanager -admin server and nodemanager - managed servers
 nodemanager.properties –
 SSL properties
 Non-SSL properties

SERVER LIFECYCLE

Server States
 SHUTDOWN
- a server instance is configured but inactive

 STARTING
- a server instance prepares itself to accept requests and perform application processing
- it retrieves its configuration data
- starts its kernel-level services
- initializes subsystem-level services
- deploys applications
- loads and runs startup classes

 STANDBY
- a server has initialized all of its services and internal applications (such as the Administration Console)
- can accept administration commands
- can participate in cluster communication
- does not accept requests from external clients.

 RESUMING
- to bring back the server to running state from stand by state

 RUNNING
- it offers its services to clients
- can operate as a full member of a cluster

 SUSPENDING
- server handles a predefined portion of the work that is currently in process—referred to as "in-flight" work
- Upon completion of in-flight work, the server progresses from the SUSPENDING state to the SHUTTING_DOWN state.

 SHUTDOWN
- a result of a graceful shutdown or forced shutdown process

 FAILED
- one or more critical subsystems have failed, the server instance sets its state to FAILED

 UNKNOWN
- If a server instance cannot be contacted, it is in the UNKNOWN state.

ADMIN CONSOLE
 graphical user interface you use to manage a WebLogic Server domain
Uses of Administration Console:

 Configure, start, and stop WebLogic Server Instances

 Configure WebLogic Server Clusters

 Configure WebLogic Server Services, such as database connectivity (JDBC), and messaging (JMS).

 Configure security parameters, including managing users, groups, and roles.

 Configure and Deploy your applications.

 Monitor server and application performance.

 View server and domain log files.

 View application deployment descriptors.

 Edit selected runtime application deployment descriptor elements.

ADMIN CONSOLE VIEW
ADMIN CONSOLE VIEW depends on the security role granted to the user
The four global roles are as follows:

1.Admin
 View the server configuration, including the encrypted value of encrypted attributes.
 Modify the entire server configuration.
 Deploy enterprise applications, startup and shutdown classes, and Web application, EJB, J2EE Connector, Web Service, and WebLogic Tuxedo Connector components. If applicable, edit deployment descriptors.
 Start, resume, and stop servers by default.

2. Deployer
 View the server configuration, except for encrypted attributes.
 Change startup and shutdown classes, Web applications, JDBC data pool connections, EJB, J2EE Connector, Web Service, and WebLogic Tuxedo Connector components. If applicable, edit deployment descriptors.

3. Monitor
 View the server configuration, except for encrypted attributes.
 This security role effectively provides read-only access to the WebLogic Server Administration Console, weblogic.Admin utility and MBean APIs.

4. Operator

 View the server configuration, except for encrypted attributes.
 Start, resume, and stop servers by default.

WebLogic Server 8.1 Administration

Agenda
• Basic design concepts of WebLogic Server
• WebLogic Threading Model
• WebLogic Classloading
• WebLogic Clustering(communication between Apache and weblogic cluster)
• WebLogic Application Deployment
• WebLogic JDBC,JMS,JMX,JNDI
• WebLogic Performance tuning
• WebLogic Troubleshooting
• Enhancements in WLS9.0


Basic design concepts of WebLogic server - Domain
• Domain
Domain is a logically related group of WebLogic Server resources that you manage as a unit. A domain always includes only one instance of WebLogic Server called the Administration Server. Administration Server serves as a central point of contact for server instances and system administration tools.

A domain may also include additional WebLogic Server instance
called Managed Servers.

• All domain configuration information is stored in config.xml.
Attributes in config.xml are stored a ConfigMbeans within the memory of Admin and managed server.

• Server startup:
Admin server: The server as part of its startup process, would parse the config.xml and create Mbeans.
For eg:
From the above entries, myserver instance would create a ConfigMbean of type ServerMbean object with attributes ListenPort=7001 and name=myserver

Basic design concepts of WebLogic server - Administration Server
Admin server is just another server instance but with added functionality.

Admin server is equipped with the functionality to drive the deployment (master deployer), monitor the managed servers etc.
All the weblogic.Admin utility commands are initiated from Admin server.
Admin server would act as central point for managing the domain.

Basic design concepts of WebLogic server Managed Server
Managed Server:
Managed server would connect to admin server via http connection and gets a copy of config.xml file and creates the related Mbeans. These Mbeans exists in memory of server for the life time.

Once managed server has started up successfully, we do not need Admin server. The application hosted on managed servers can be accessed irrespective whether or not Admin server is up and running. Admin server would be needed for deployments, any administrative activity like accessing the console etc.

Basic design concepts of WebLogic server Managed Server independence Mode
• Managed server Independence mode:
Managed server can also start without Admin server if MSI mode is enabled. In this mode, managed server would look for file “msi-config.xml” under its root directory. It creates Mbeans out
of msi-config.xml to startup successfully.
msi-config.xml will be created/copied to managed server by admin server (If MSI replication is enabled- from Admin console)

For details look under:
http://e-docs.bea.com/wls/docs81/adminguide/failures.html#1118666
Basic design concepts of WebLogic server Server startup
Server startup:
WebLogic server can be started in the following ways:
• Using scripts
• Node manager
• Weblogic.Admin start command
• On windows weblogic server can be started as windows service (beasvc.exe)

Using Scripts
WebLogic installation creates the startup scripts. We can modify the created scripts to suit our environment.
All scripts would have :
java weblogic.Server

The main class is weblogic.Server that initiates the server startup sequence.
As part of server startup sequence (irrespective of how it is started) there are various services that get initialized.
RMI service, cluster service, JNDI service etc
Basic design concepts of WebLogic server
Node Manager
• Node Manager is a Java utility that runs as separate process from WebLogic Server and allows you to perform common operations tasks for a Managed Server, regardless of its location with respect to its Administration Server
• There should be one NodeManger process on each physical machine.
• NodeManager is domain independent

Basic design concepts of WebLogic server- beasvc
• WebLogic Server can be started as windows service.

The executable that initiates the server startup is beasvc.exe
The scripts used to create the service are created during installation ()

Run the script (edit the classpath and JVM arguments if needed) to create the service which will update the registry too.
Run the service through control panel.

Basic design concepts of WebLogic server Server lifecycle

The following phases are involved in server startup cycle:
SHUTDOWN—>STARTING—>STANDBY (optional) —>RUNNING

The following are high level activities that occur during server startup:
• Obtains the configuration data. While Admin server gets this from config.xml, managed server get this from Admin server.
• Starts logging and timer services & loads license.bea file.
• Initializes internal component services like RMI service, Cluster service, IIOP service, Deployment Manager service etc
• JDBC connection Pools are created.
• Startup classes (Eg Wily introscope) are executed.
• Applications are deployed.

• The following phases are involved in server graceful shutdown cycle (When weblogic.Admin utlity is used or stopped from console)

RUNNING—>SUSPENDING—>STANDBY—>SHUTTING DOWN—>SHUTDOWN

Graceful shutdown of server allows all the inflight work to be finished before the server shuts down.

The first services to shutdown are RMI service and web container, so that no new requests can come through.

Server would wait for “Graceful Shutdown Timeout” before shutting down.

All the existing httpsessions are destroyed or server would wait till they expire depending on “Ignore Sessions During Shutdown”.


Basic design concepts of WebLogic server Production Vs Development modes
Weblogic Server can be started in development mode or production mode.
In dev mode, there is an application poller.
This is a simple utility that polls the “/applications” directory every 3secs (default). If it finds a different version of application in the directory, then weblogic would redeploy the app automatically.
Obviously there is a little bit of overhead involved, but this is justified for dev environment.

In production mode application poller is turned off.
User would either have to deploy from console or run weblogic.Deployer utility

Basic design concepts of WebLogic server Domain-Wide Administration Port
Domain-Wide Administration Port
This is an optional feature available that would help in segregating the admin traffic with application related traffic.
A different port has to be configured and all admin tasks (including accessing Admin console) are picked up from the configured port.
The cons for this setup are:
• The Administration Server and all Managed Servers in your domain must be configured with support for the SSL protocol.
• After enabling the administration port, all Administration Console traffic must connect via the administration port.

Basic design concepts of WebLogic server Admin Server recovery
If Admin server machine is down and if we need to migrate to another machine:
1) Backup Config.xml file
2) Backup security data (ldap directory & SerializedSystemIni.dat)
Replace these files in the new domain and start the Admin server.
The new Admin server will automatically reconnect with running managed servers

Basic design concepts of WebLogic server
• WebLogic8.1 is J2EE1.3 compatible
• Managed Servers within the domain can be at different Service Pack levels as long as the Admin Server is at the same Service Pack Level or higher than its Managed Servers.
• All servers that are part of cluster should be of same service pack level.
Basic design concepts of WebLogic server WebLogic internal applications
WebLogic has some internal applications deployed by default to perform some of internal activities.
console.war – weblogic Admin console is a webApplication deployed only on Admin server.
FiledistributionServlet: This servlet takes care of communication activities between Admin and managed server. (Deployment, managed server startup etc)
Eg: During deployment, this servlet takes care of copying the application on to all managed servers staging directory.
uddi.war/uddiexplorer.war: These are meant to service webservice requests.

WebLogic Classloading
A classloader is part of JVM that loads classes into memory. Classloader is responsible for finding the classes at runtime.
The following classloaders are involved with weblogic server:

• Bootstrap classloader (JVM)
• Extensions classloader (JVM)
• Application classloader (WebLogic)

Points to remember:
• Java classloader would always search for a class from parent classloader. If two classes with same name exist within an EAR/WAR file and System ClassPath, the class from system classpath is picked up always because system classpath is parent classloader of application classloader (from the flow diagram above)
• Classes in WAR file can see/load EJB classes but not viceversa
because WAR classloader is child of EJB classloader

WebLogic Threading Model
• WebLogic Server is a multi-threaded java application that uses an internal component called an “Execute Thread” (which extends java.lang.Thread) to perform it’s work.

• Threads are created out of thread pools. Each thread pool is meant for specific purpose.

The important thread pools on weblogic8.1 are:
• Thread Group: weblogic.socket.Muxer
Defaults to 3 on Unix systems and 2 on Windows.Threads from this group are used for socket reading purpose.
• Thread Group : weblogic.health.CoreHealthMonitor
CoreHealthService creates an instance of this thread to do periodic monitoring of servers' memory and state of its threads. It reports stuck threads.
• Thread Group : weblogic.admin.RMI
Threads from this group are used for communication between Admin and managed servers.
Eg: Deployment of application.
• Thread Group: weblogic.kernel.System
Threads from this group are used for weblogic internal activities like RJVM heartbeats, getting http state dump for JNDI updates in cluster.
• Thread Group : weblogic.admin.HTTP
Threads from this group are used for serving all console related requests.
• Thread Group: weblogic.kernel.Non-Blocking
Threads from this group are used for session replication activities.
• There are other threads created by JVM.
For Eg, in case of SUN JVM we can see:

"Signal Dispatcher" daemon prio=10 tid=0x009226e8 nid=0xc24 waiting on condition[0..0]
The Signal Dispatcher is responsible for forwarding native events.
like when user hits kill -3, this thread takes care of initiating the process of getting thread dumps.

• Thread Group: weblogic.kernel.default
The threads from this group are the main worker threads that does the real work.
These threads service requests to the application hosted on weblogic.
There are 15 threads by default. The number of execute threads can be modified through Admin console.

WebLogic Threading
WebLogic Threading- Custom Queues

Custom thread queues are user defined.
The following are the two steps to achieve this.
• Create a thread queue. This can be done from Admin Console.
• Assign the created thread queue for an application. This is done by including the queue information in the application descriptor files.
For eg:
Weblogic Admin console (console.war) is a webapplication that has custom thread queue. If we see weblogic.xml of console.war we can see the custom queue definition as:


weblogic.admin.HTTP


If we are creating a sample thread queue say “my-sample-queue”.
To assign it our webApplication, just as these entries to weblogic.xml of our webApplication.

my-sample-queue



WebLogic Application hosting
• Applications deployed on weblogic server would be:
1) EAR ( WAR + EJB jar + Custom jars)
Descriptor files are:
META-INF/application.xml
META-INF/weblogic-application.xml

2) WAR ( Servlet/JSPs + java libraries,classes)
Descriptor files are:
WEB-INF/web.xml
WEB-INF/weblogic.xml

3) EJB (EJB interfaces, java libraries/classes)
Descriptor files:
META-INF/ejb-jar.xml
META-INF/weblogic-ejb-jar.xml

4) RAR ( J2EE connector)
Descriptor files:
META-INF/ra.xml
META-INF/weblogic-ra.xml

The applications can be deployed in archived format (EAR,WAR) or as exploded format. (Although we rarely see EJBs deployed in directory format)



Deployment modes
Deployment can be done through Admin console OR command line using webogic.Deployer utility.
More about the deployer utility can be found at:
http://e-docs.bea.com/wls/docs81/deployment/tools.html#wldeployer

The following are the modes of deployment.
• No stage
• Stage
• External_stage
Deployment
• WebLogic does deployment in two phases
(aka Two-Phase deployment)

Prepare : This phase makes sure that the application is in a state in which it can be deployed reliably. During this phase, Admin server copies files to target (managed servers)

Activate: The activate phase involves the actual deployment i.e the classes are loaded into memory. This takes place only if prepare phase is successful.
Deployment Modes

• No Stage
In this mode, weblogic will deploy the application from the
Path= tag that is specified in the tag in config.xml.
weblogic will not make a copy of the app to any staging directory, nor will it make any copy of the app to distribute to managed servers. All servers should have to share a common file system so that all the servers would reference the application from one physical location.

• Stage
weblogic will make a copy of all of the applications into the staging directory, and will deploy the app from there. weblogic will do all the file transfer of the application files to the managed servers. As part of deployment process weblogic would copy the application (WAR/EAR) on to managed servers and deploy from there.
This is default deployment mode unless changed.

• External_Stage
weblogic will deploy from the staging directory, but we (user)
are responsible for getting the application (EAR/EAR) to staging directory.Weblogic will do no file transfer to the managed servers.

The difference between nostage and external stage is that nostage is intended for people who have one shared disk, so the Path= attribute is meaningful for every server in the domain.
External stage is for people who don't use a shared disk, so Path= only has meaning only the admin server. Each managed server will have a stagedPath that makes sense for that server.

WebLogic Deployment
At Narayanasetti's company we used External_Stage as our deployment mode.

Our scripts would do the file transfer and weblogic would do the deployment.

More over we use command line utility weblogic.Deployer
for deployment purpose.

WebLogic JDBC
Connection pool:
Connection pool is a named group of identical JDBC connections to a database that are created when the connection pool is deployed. Connection pooling is for efficiency purposes to reduce the overhead of creating connection objects at runtime.

JDBC Flow logic:
When server startup, depending on the initial JDBC pool capacity, connections are created.
Connections are created based on the information provided in pool attributes.
At runtime, Client asks for a connection, weblogic would take an existing connection from pool and do a ping (query is user defined).
If ping is successful, weblogic would give connection to client, else it will recreate and give the connection.
WebLogic JDBC (Flow chart)
WebLogic JDBC configuration
1) Create a connection Pool from Admin console.
Provide all the JDBC driver attributes like Driver class, Driver URL and test table name.
2) Target the connection pool to the list of servers.
3) Create a Datasource and assign it to the pool created.
4) Target the Datasource to the servers where connection pool is targeted.

WEbLogic JDBC (Drivers)
• WebLogic has integrated JDBC drivers.
All the drivers that weblogic ship are JDBC type4 thin drivers.
(All the weblogic JDBC drivers are from DataDirect technologies)

WebLogic would also include some of the frequently used drivers. These drivers come with installation. These drivers are shipped for convenience purpose and it is advisable to use the latest drivers available from vendors website.

WebLogic JDBC
The following are the that drivers come with weblogic installation:
(Apart from weblogic type 4 integrated drivers)
They are located under /beahome/weblogic81/server/ext/jdbc :
• Oracle 10g,9.2 thin driver
• Sybase Jconnect5.5 , 4.x
Any driver that implements JDBC spec can be used with weblogic. The driver should be thread safe too. (Suitable for multithreaded apps)
WebLogic would support type2 OCI drivers too. However thin driver is recommended because it is 100% java and not prone to jvm crash or native memory leak. In case of Oracle, the performance between thin driver and OCI driver has been pretty much narrowed down with JDK1.4 (As per Oracle)

WebLogic JDBC MultiPool
1.MultiPool
This is a pool of connection Pools. The pools can be connected to different DBMS or specific setup like Oracle RAC.
You would need to choose the algorithm based on application requirements (Only one algorithm can be applied)
• High Availability: All connection requests are served from first pool in the list. *Only* if the pool is not available, (connection fails and cannot be refreshed) the connection is routed to the next pool in the list.

2. Loadbalacing:
Connection requests to the pool are roundrobined across the list of pools. However if a connection request to a connection pool
was not successful and cannot be recreated (such as when DB is down or pool is suspended) then the request is routed to next connection pool in the list.
Oracle RAC:
The multipool concept can be applied to Oracle RAC where each connection pool is connected to one of the oracle instance and we can choose the algorithm.
There are quite a few limitations to multipool when global transactions are involved (XA).

WebLogic JMX
• Weblogic implements SUN JMX1.0 (Java Management Extensions)
All WebLogic Server resources are managed through these JMX-based services.
For easy understanding, consider the following : JDBC connection Pool (snippet from config.xml)


The JDBC connection pool resource is exposed as Mbean called “JDBCConnectionPoolMBean”
• For JDBCConnectionPoolMBean, there are lot of other attributes with default values. The values that we add through console override the default values.(like MinCapacity,MaxCapacity etc)
An Mbean is just a concrete java class that is coded as per JMX specification and it provides a set of setters and getters of the attributes.
In our example, we can invoke JDBCConnectionPoolMBean and
then change the attribute like initial capacity via JMX program.
• There are another set of Mbeans called runtime Mbeans.
As the name implies, this is read only and used to monitor/get the runtime values.

Extending our same JDBC pool example, the related runtime Mbean is : JDBCConnectionPoolRuntimeMBean.
If we see the API, we can see methods like:
getActiveConnectionsCurrentCount()
These are runtime values at the particular instant.

General rule of thumb is, Mbeans for which the change involves persisting data to config.xml are termed as configuration Mbeans.
(JDBC pool capacity, driver name, timeout value etc).
Mbeans that involve read-only data is termed a runtime Mbean.
The API documentation will give a better picture of all the methods

As shown in the example, we have configMbeans as well as RuntimeMbeans for all weblogic resources. (JDBC,JMS,EJB etc)
This provides an easy way for other vendors like vignette to monitor and administer weblogic resources.
When are these configMbeans created?
yes, at startup. Admin Server creates these Mbeans (for each resource) from config.xml and managed servers when they startup, connect to admin server and get a copy of these Mbeans into their local memory. RuntimeMbeans are created at rutime and they reflect the in instantaneous values.

WebLogic JNDI
• JNDI provides naming and directory functionality to applications.
Applications use naming services to locate objects in data sources, EJBs, JMS, MailSessions, and so on in the network.
JNDI provides abstraction where the client need not worry of any changes on server side.
For eg: datasource associated to JDBC connection pool.
connection pool properties can be changed or even connect to different DB, but there would not be any code change to client.
All the client has to do is lookup the JNDI name and get the connection.
• WebLogic Server provides its own implementation, weblogic.jndi.WLInitialContextFactory, that uses the standard JNDI interfaces.
• To check the JNDI binding on the server, open weblogic Admin console, right-click on the server name and view the JNDI tree.
• In weblogic cluster, all the JNDI information is replicated across all the servers in the cluster through multicast.
WebLogic JMS
• JMS enables applications to communicate with one another through the exchange of messages. All J2EE servers implement JMS specification.
In JMS we have Message Producer which creates/sends message and Message Consumer who actually consumes the message and probably do some action further.

The common messaging models in JMS are:

1) Publish-Subscribe Messaging
When multiple applications need to receive the same messages, Publish-Subscribe Messaging is used.
The main concept in this model is “topic”.
Publisher may send messages to a Topic, and all applications that are subscribed to the topic will receive the message.
Eg: Consider a stock quote application. The publisher would broadcast the updated stock quote (topic) and all the subscribers that are interested (subscribed) in the particular stock quote will receive it.

2) Point-To-Point Messaging (PTP)
The concept in this model is “queue”.
This is 1-1 where there is one sender and one receiver.
A queue sender (producer) sends a message to a specific queue. A queue receiver (consumer) receives messages from a specific queue.

WebLogic Clustering
Basics
• WebLogic Server provides clustering services for
WebApplications
EJB and RMI applications
JNDI
• Essential services of clustering:
LoadBalancing
Failover
• Underlying mechanism/protocol
Heartbeats and JNDI updates – Multicast (UDP)
Session replication –IP sockets(TCP)
JNDI state dump – http url connection

• J2EE spec does not mention about clustering.

So this is a value add on feature that most application server vendors provide.
Since the spec does not talk about it, each vendor would implement this in their own way. However the ultimate objectives are “loadbalancing and failover”.

So all the cluster related information would be defined in
weblogic.xml (This is vendor specific descriptor file)

• Replication
WebLogic Server provides clustering support for servlets and JSPs by replicating the HTTP session state of clients that access clustered servlets and JSPs. WebLogic Server can maintain HTTP session states in memory, a filesystem, or a database.
Replication involves, copying all the session information on to another server (called secondary server). In case of weblogic this is a RMI (t3) call.

WebLogic has five different implementations of session persistence: (set in weblogic.xml)

• Memory (default – session info stored in memory)
• File system persistence (session info stored in text file)
• JDBC persistence (session info stored in database)
• Cookie-based session persistence (session info stored in client browser)
• In-memory replication (Session info stored both in primary and secondary servers memory. Most commonly used in cluster setup)
Communication between Apache and WebLogic Cluster
A typical journey of request (Narayanasetti shared platform)
Request comes through hardware loadbalancer and gets forwarded to one of the Apache server.
WebLogic Apache plugin gets invoked, delegates the http request to one of the servers in cluster. (It gets the information of weblogic instances from httpd.conf file)
WebLogic server would create a session cookie and send back the session cookie as http response header.
As long as browser is alive, the session cookie is passed (as http header) and Apache plugin would parse the session cookie and route the request to same server.

Performance tuning
• LAN Capacity
It is recommended to have 100 Megabit minimum
It is recommended to host DB on separate Hardware machine.

• JVM Heap Size
Large Heap leads to slower, but less frequent GC
Small Heap leads to faster, but more frequent GC
If heap is larger than RAM, this will cause page swapping.

• How to choose the Max Heap
Run the application with –xaprof and do the load test. After the load test kill the server process.
Upon the process exit, JVM would create a summary which includes the max amount of memory that has been utilized.
This serves as base point and max heap can be set more than this value.
There are other commercial tools available to record the heap usage like integration of LoadRunner into wily introscope so that
memory usage can be monitored and recorded.

• Garbage Collection timings

Its good to have the JVM not spend more than 5 to 10 secs
for full GC because FullGC pauses the application threads per cpu. (Some of the latest JVMs have settings that would reduce the pause time but not 100%).

In those cases, try other GC settings (parallel GC or concurrent GC or incremental GC) and see if it reduces the GC times.

• Set Execute Thread Count accordingly to utilize CPU’s
Too many threads will cause too much context switching.
OS has to schedule the threads so that all the threads get fair share.

Too little threads will cause CPU under utilization and
request waiting.

Set an optimal values based on thread/CPU usage Set.
This can be done by taking periodic thread dumps during load test
to see if threads are idle or busy for most part of the time.

JDBC tuning

• Use large enough pool to service largest number of concurrent database users
• It is recommended to have the Max pool capacity equal to Execute thread count. (This can be application dependant)
• Process most of the data inside the database to minimize network traffic
• Process a commit after a batch of statements


WebLogic tuning

• On Solaris, set the TCP timewait interval to 60secs
Suggested TCP values from BEA is at:
http://e-docs.bea.com/wls/docs81/perform/HWTuning.html#1121083

• Application would be most of the times be CPU intensive or IO intensive. Typically IO intensive applications need more threads because IO operations tend to be slow.

• Use NativeIO wherever possible.
NativeIO is fast compared to java IO because in java, the socket needs to be polled.
Troubleshooting WebLogic
Logging

Logging:

1) Server log
WebLogic server creates server log file by default under:
///.log
The location is configurable.

2) JDBC log
All SQL statements and DB related exceptions/errors.
This file is created under //jdbc.log

3) STDout log (If the process is redirected to STDout)

4) Domain log
All domain level information is logged into this file.
This is subset of server log file.
/.log

5) Access log
All http requests are recorded in this log file
//access.log

6) Transaction log
All servers record transaction in the tlog file
//.tlog

WebLogic Troubleshooting(Server crash)
Server Crash
This implies the weblogic java process no longer exists.
Server crash can occur only because of native code. (Java cannot cause a process to crash)

Determine all potential sources of native code used by the WebLogic Server.
• nativeIO.

• Type2 jdbc driver.(At Narayanasetti we use thin driver for most applications)

• Native libraries accessed with JNI calls.

• SSL native libraries.

• JVM itself. Most of the times its from JVM.

Sometimes the JVM will produce a small log file that may contain useful information as to which library the crash has originated from. (hs_err_pid*.log)


Server Crash Analysis
When a JVM is crashed, a core file(binary image of the process) is created. Run pmap and pstack against the core file to get the library that caused the crash.

Demo to figure out offending library using existing pmap & pstack out files.

Check list:

1) hs _err_pid*.log (Look for library that caused the crash)

2) pmap core (core file created in JVM root dir)
pstack core

3) Using debugger (gdb,dbx,adb) (if above two steps does not provide any information)

Server Hang

A server is said to be hung when:
1) Process is still alive
2) Server does not accept any requests because all the execute threads busy or stuck for some reason.
3) No reponse sent to clients.
4) java weblogic.Admin PING command doesn’t return a normal response

WebLogic Troubleshooting(Server hang)

Server Hang Analysis:
The first step is to take multiple thread dumps.
• A thread dump is a snapshot of the JVM at the particular instant.
• Multiple thread dumps are necessary to conclude that the threads are stuck and not progressing.

Procedure to take thread dumps:
Unix:
Open shell window and issue the command kill -3
where PID is java processID of weblogic. Thread dumps are
logged on to STDout file.

Windows:
Do ctrl-break on command window where weblogic is running.
Thread dumps are created on the same command window.

Windows Service:
Open a command prompt and issue the command(Make sure beasvc.exe is in the PATH)
c:\> beasvc -dump -svcname:service-name
Thread dumps are created in the defined log file.
While creating service, we can provide log option in installservice script as:
-log:"d:\bea\domains\mydomain\myserver-stdout.txt

• Before we analyze thread dumps, it is important to know the common thread states:
1)Runnable [marked as R in some VMs]:
This state indicates that the thread is either running currently or is ready to run the next time the OS thread scheduler schedules it.

2)Object.wait() [marked as CW in some VMs]:
Indicates that the thread is waiting for some condition to be
fulfilled.

3)Waiting for monitor entry [marked as MW in some VMs]:
Indicates that the thread is waiting to enter a synchronized block.

These threads are something to watch out because there is lock contention here. Thread is waiting for a lock on object and some other thread is holding the lock.

In case of weblogic, the main worker threads are from group weblogic.kernel.defalt:
"ExecuteThread: '1' for queue: 'weblogic.kernel.Default'“….
This is the set of threads we need to look for hang/slow performance issues.
This is a snapshot of idle thread waiting for some work to be assigned.
On an idle system you would see lot of threads in the below state:

"ExecuteThread: '1' for queue: 'weblogic.kernel.Default'" daemon prio=5 tid=0x031a6308 nid=0x980 in Object.wait() [2dff000..2dffd8c]
at java.lang.Object.wait(Native Method)
- waiting on <0x112cf2c0> (a weblogic.kernel.ExecuteThread)
at java.lang.Object.wait(Object.java:429)
at weblogic.kernel.ExecuteThread.waitForRequest(ExecuteThread.java:153)
- locked <0x112cf2c0> (a weblogic.kernel.ExecuteThread)
at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:172)

• As for thread dump analysis & conclusion, lets see a sample thread dump and drill into it further
Demo of RSD thread dump (Thread stuck issue on UAT)
WebLogic Troubleshooting(Server slow)

• Server performing Slow
There are lot of reasons for server performing slow.
First step is to take thread dumps and see what the threads are doing. If there is nothing wrong with the threads there are other reasons why server performs slow:

Process runs OutOfMemory:
If java heap is full, server process appears to be hung and not accepting any requests because each request needs heap for allocating objects.
So if heap is full, none of the requests get served, all the requests fail with java.lang.OutOfMemory

WebLogic troubleshooting

• OutOfMemory Analysis:
OutOfMemory can occur because of real memory crunch or a memory leak causing the heap to fill with orphaned objects.
First step is to enable GC and run the server again.
(-XX:printGCDetails).
The STDout file would show the garbage collection details.
If the error is because of memory leak, then we would need to use profilers like Introscope or optmizeIT to figure out the source of leak.

Process size = java heap + native memory + memory occupied by the executables and libraries.
On 32 bit operating systems, the virtual address space of a process can go up to 4 GB. This is data bit limitation (2 pow 32)

Out of this 4 GB, the OS kernel reserves some part for itself (typically 1 – 2 GB).
This is not a limitation on 64 bit machines like solaris(sparc) or windows running on Itanium (64 bit)

WebLogic Troubleshooting Fragmentation
• OutOfMemory Analysis
OOM can occur due to fragmentation. In this situation, we can see free memory available but still get OutOfMemory errors.
Before we know about fragmentation, we need to know the following fact:
Heap allocation can only be contiguous (As per JVM spec). If a request needs 2MB of memory then JVM has to provide 2MB of contiguous memory chunk.
Over a period of time, memory allocation is becomes scattered and there might not be enough contiguous memory available.
FullGC might no be able to reclaim the contiguous space.
This is called fragmentation

WebLogic Troubleshooting
For eg: The verbose:gc output might look like the following if there was a fragmentation of heap. There is free memory available, but still JVM throws OOM error.
(Most of the fragmentation bugs are resolved in Sun JDK1.4.2_xx)

[GC 4673K->3017K(32576K), 0.0050632 secs]
[GC 5047K->3186K(32576K), 0.0028928 secs]
[GC 5232K->3296K(32576K), 0.0019779 secs]
[GC 5309K->3210K(32576K), 0.0004447 secs]
java.lang.OutOfMemoryError

• OutOfMemory Analysis
Fragmentation relates issues are because of bug in JVM.
Best approach is to try the latest minor version of JVM and if does not work out, we need to work with vendor to get it fixed.
• The following commands on solaris will provide good information:
vmstat :
The vmstat command reports statistics about kernel threads, virtual memory, disks, traps and CPU activity
sar:
An OS utility that is termed as system activity reporter

• If the application uses SSL, then the server performs slow compared to non SSL.
SSL reduces the capacity of the server by about 33 to 50 percent depending upon the strength of encryption used in the SSL connections.

• Process running out of File descriptors. Server cannot accept further requests because sockets cannot be created. (Each socket created consumes a FileDescriptor)
The following exception is thrown in such cases:
java.net.SocketException: Too many open files
OR
java.io.IOException: Too many open files
In the above case, the lsof utility would help. lsof utility shows the list of all open filedescriptors. From the list of open files, we ( application owner) can easily figure out if it is a bug or expected behavior. If it is expected behavior, then the number of FDs needs to be increased. (default number is 1024)

• GC taking long times (more than 20secs).
This appears like a hang for end users.
In the above case, we need to tune the GC parameters.
In these scenarios, we should be trying other GC options available. In some cases (GC taking very long times), incremental GC has been useful (-Xincgc).

Before knowing about high CPU analysis, it would be helpful if we know about solaris threading Model.

• Process consuming High CPU

Process using high CPU is not always bad. This is a common misconception. Infact we want our application to use the CPU efficiently.
What we do not want is, a single thread or couple of threads consuming all the CPU forever(In an infinite loop)
and not allowing other threads to get the CPU share(timeslice).

High CPU analysis:

1) Run prstat to know the process that is consuming the highest CPU.

2) Run prstat –L –p where PID is processID that is consuming the highest CPU.
From the above command you would see the lwps that are consuming the high CPU.

3) Run pstack
From the pstack, we can see the lwp mapped to native thread
For eg:
----------------- lwp#