Drop Down MenusCSS Drop Down MenuPure CSS Dropdown Menu

Sunday, April 29, 2018

Basics knowledge on Weblogic Server & Administration

What is Weblogic server?
A weblogic server is a instance of java process executing in a JVM.Weblogic server is a software application that runs as a middle tire between back-end database and related applications and browser based thin clients.Two Type of servers:
1. Administration server.
2. Managed server.

What is a Admin Server?

Every domain have one and only one admin server.
It is the central point of control for the webLogic domain.
It runs Weblogic admin console.
Admin server stores the master copy of the domain config.

Usually we can deploy applications on AdminServer/ManagedServers/Clusters But for Production environments, it is Not recommend to hosting application logic or resources on the Administration Server.We can deploy applications on admin server but it is not recommended.

What is a Managed server?

A domain can have zero or more managed servers.
All the applications are hosted in managed servers only.
Admin server stores the master copy of the domain config. Each Managed server stores local copy of domain config.Managed server contacts the admin server while starting. So, to do any changes, we need to contact Admin server.Admin server will push those changes to Managed servers. We can start the managed server if admin server failed. But we cant do any changes in managed server if admin is failed(Stopped).

admin vs managed:

Managed Servers are similar to the Administration Servers, except few differences. Like Managed Servers will not have Administration Console Application (consoleapp) deployed in it.

Each Managed Server is independent of all other Managed Servers in the domain. Managed Servers can be part of Clusters to work together as a Unit. We can have one or more than one Managed Servers in a domain according to our requirement. The Managed Servers can reside inside the Physical box of AdminServer or they can run on a Separate Physical Box as well.

What Happens if the Administration Server Fails?

The failure of an Administration Server for a domain does not affect the operation of Managed Servers in the domain.If an Administration Server stops running while the Managed Servers in the domain continue to run, each Managed Server periodically attempts to reconnect to the Administration Server.Also, you can start a Managed Server even if the Administration Server is not running.

Is it possible to start Managed server without Admin server or Why availability of Admin server is important?

Oh yes...managed servers are individual entities and can serve the request on their own but it will depend on actually  how they were configured?

While building the WebLogic environment the option "Managed Server Independence Enabled" was not checked. I am not sure, what could be the reason for it.

It is found under: Environment > Servers > Configuration > Tuning > Advanced
If the "Managed Server Independence Enabled" option is enabled, Admin server will push the config.xml to all the managed servers as msi-config.xml.
When a Managed Server starts, it tries to contact the Administration Server to retrieve its configuration information.If a Managed Server cannot connect to the Administration Server during startup, it can retrieve its configuration by reading configuration (msi-config.xml)
and security (boot.properties) files directly. A Managed Server that starts in this way is running in Managed Server Independence (MSI) mode.

config.xml:

It is the most important part of weblogic server. In a distributed environment, the config.xml file only resides on admin server and gets created automatically in the domain folder as soon as the server starts for the first time. As the name suggests, it contains the configuration of the domain like entries of the servers, clusters, data-sources etc.
The passwords in config.xml file are encrypted using DES algorithm and vary from domain to domain. Said that, every domain will have SerializedSystemIni.dat, which works as a key to encrypt and decrypt any element.
Do not worry, if SerializedSystemIni.dat is not present in the domain folder as it gets created automatically as soon as you start the server.

config.xml is the central configuration repository for a domain. every resource you have  configured from admin console or by command line or by any other tool registered under  this file.
Location: $DOMAIN_HOME/config/config.xml

What is a Domain?

Domain is logical grouping of resources and services and consist of Administration Server, Managed Server and cluster.  There can only be one administration Server in domain and zero to N Managed Server.

What is a Machine in Weblogic server?

A computer that hosts the WL server instances.Used by node manager to restart a failed managed server.

What is a Node Manager?

-Node manager is a utility or process running on a physical server that enables starting, stopping,suspending or restarting admin and managed server remotely.
Node Manager is utility in Oracle WebLogic Server to start , stop, restart Managed Server or Administration Server remotely.
-It is not associated with a domain, though can start any server that reside on same physical server.
-it is required if we use Admin console to start servers.
Node Manager is optional component in WebLogic Server but it is mandatory if you want to start Managed Server from Console (other option is to start Managed Server from command line)
Node Manager is optional component in WebLogic Server but it is mandatory if you want to start Managed Server from Console (other option is to start Managed Server from command line)
There is two  types of NM,
1. java-based:(runs under JVM) (more secure)
2. Script based:(not available on windows)

1). Java Based Nodemanager: This can be used in Windiows as well as UNIX based Operating systems. Java Based Node Manager is not supported on Open VMS, OS/390, AS400, UnixWare, or Tru64 UNIX.
2). Script Based Nodemanager: This nodemanager can be used in UNIX based operating systems. The advantage of the script-based Node Manager is that it can remotely manage servers over a network that has been configured to use SSH.

Why We Need Nodemanagers?

1). It can Start, Shut Down, and Restart an Administration Server which has assigned a Machine.
2). It can Start, Shut Down, and Restart an Managed Server which has assigned a Machine.
3). It can Restart Administration and Managed Servers. (Crash Recovery of Servers is an Interesting feature)
4). It can be used to Monitor Servers and View Log Data

How to start Node Manager ?

$WL_HOME\server\bin\startNodeManager.sh
To Stop Node Manager
In Unix/Linux, kill the process -
kill -9 `ps -ef | grep -i nodemanager.javahome | grep -v grep | awk {'print $2'} | head -1`

Node Manager Log Files
$WL_HOME/common/nodemanager/nodemanager.log

Why node manager is required?
 A Node Manager process is not associated with a specific WebLogic domain but with a machine.
You can use the same Node Manager process to control server instances in any WebLogic Server domain, as long as the server instances reside on the same machine as the Node Manager process. Node Manager must run on each computer that hosts WebLogic Server instances. whether Administration Server or Managed Server that you want to control with Node Manager.

Node Manager Log Files
$WL_HOME/common/nodemanager/nodemanager.log

Important configuration file for node manager:
$WL_HOME/common/nodemanager/nodemanager.properties- used by java based node manager. This file is created on first time start of nodemanager (startNodeManager.sh)

$WL_HOME/common/nodemanager/nodemanager.domains - contains mappings between the names of domains managed by Node Manager and their corresponding directories.
entry like
base_domain=D:\Oracle\Middleware\user_projects\domains\base_domain
soa_domain=D:\Oracle\Middleware\user_projects\domains\soa_domain

$WL_HOME/common/nodemanager/nm_data.properties- This file stores the encryption data the Node Manager uses as asymmetric encryption key. This file is created on first time start of nodemanager (startNodeManager.sh)

$DOMAIN_HOME/config/nodemanager/nm_password.properties- This file stores the Node Manager username and password used by Admin Server to connect to Node Manager.


What are the Default port number for admin, node manager and ssl ? 
Default multicast address? Multicast range?
Default port no for admin :7001
ssl: 7002
node manager: 5556
Default multicast address :239.192.0.0
range 224.0.0.0 to 239.255.255.255

Development Mode
The default JDK for development domain is Sun Hotspot
You can use the demo certificates for SSL
Auto deployment is enabled
Server instances rotate their log files on startup
Admin Server uses an automatically created boot.properties during startup
The default maximum capacity for JDBC Datasource is 15
The debugFlag which is used to start the WebLogic Workshop Debugger is enabled

Production Mode
The default JDK for production domain is JRockit
If you use the demo certificates for SSL a warning is displayed
Auto deployment is disabled
Server instances rotate their log files when it reaches 5MB
Admin Server prompts for username and password during startup
The default maximum capacity for JDBC Datasource is 25
The debugFlag which is used to start the WebLogic Workshop Debugger is disabled

Weblogic Boot Properties
Using boot.properties, you can control the server startup without passing the username and password.
1. Create a boot.properties file.
Create a plain text file called boot.properties with the following content:
username=weblogic
password=weblogic01

2. Copy the boot.properties to the following location:
/usr/local/weblogic10/user_projects/domains/devdomain/servers/managedserver1/security
/u01/app/middleware/user_projects/domains/ba_bi_uat/servers/bi_server1/security

/u01/app/middleware/user_projects/domains/ba_bi_uat/config
[user@abc config]$ ls -lrt config.xml
-rw-r----- 1 user dba 38119 Oct 25  2016 config.xml
[user@abc config]$

What is a Cluster?

Two or more managed server becomes or forms the cluster in a domain and cluster handle the  load balancing across the cluster.
Group of WebLogic Managed Server Instances that work together to provide high availability and scalability for applications is called cluster. WebLogic Servers with in cluster can run on same machine or different machines. These are also called as managed Server cluster.All the servers in a cluster must be in same domain. All servers in a cluster must be same version level, though they can run on different machines.

There are two method of clustering Horizontal and Vertical.
Horizontal clustering :involves running multiple Java application servers that are run on two or more separate physical machines. Horizontal scaling is more reliable than vertical scaling, since there are multiple machines involved in the cluster environment, as compared to only one machine.

vartical Clustering: however, consists of multiple java applications servers on a single physical machine. with vertical scaling, the machine's processing power, cpu usage, JVM heap memory configurations are the main factors in deciding how many servers instances should be run on one machine.

Difference between horizontal and vertical cluster?
Horizontal clustering involves running multiple Java application servers that are run on two or more separate physical machines.
Vertical clustering, however, consists of multiple Java application servers on a single physical machine.

How does Cluster Communication Happen?

Members of the Cluster communicate over the Cluster Multicast IP and Port by sending  periodic heart beat messages.

How do stubs work in a WebLogic Server cluster?

Clients that connect to a WebLogic Server cluster and look up a clustered object obtain a replica-aware stub for the object. This stub contains the list of available server instances that host implementations of the object. The stub also contains the load balancing logic for distributing the load among its host servers.

Stubs perform the process of removing the failed instance from the list whenever there is a failure. The stub usually makes use of DNS for finding the running server and also for obtaining the list of the instances that are currently available with the application.

What are the  States of the Weblogic Server?

There are 9 states of server which are given below

Shutdown
Starting
Standby
Resuming
Running
Suspending
Shutting down
Failed
Unknown

What is Thread?
A thread is a single sequential flow of control within a program.

What is Thread dump?

Thread Dump is a textual dump of all active threads and monitors of Java apps running in a Virtual Machine.

A Java thread dump is a way of finding out what every thread in the JVM is doing at a particular point in time. This is especially useful if your Java application sometimes seems to hang when running under load, as an analysis of the dump will show where the threads are stuck.

We take thread dump as below:

1) Linux : kill -3 <ps_id>
2) Windows (console mode) : crtl+break
3) Windows (service) : beasvc -dump -svcname:mydomain_myserver.

examples when we take thread dump:
1.  when server is hang Position, i.e. that time server will not respond to coming requests.
2. While sever is taking more time to restart
3. When we are Getting exception like “java.lang.OutOfMemoryException”
4. Process running out of File descriptors. Server cannot accept further  requests because sockets cannot be created
5. Infinite Looping in the code

What is Stuck Thread?
A Stuck Thread is a thread which is processing a request for more than maximum time that you configured in admin console.

How to deal with Stuck Thread?
Take multiple thread dumps immediately.
Review thread dumps or from console (managed server > monitoring > threads).
See how many threads got stuck?
If the stuck thread count is increasing or constant?
If constant then if got stuck on same area (application code etc ) or at different places ?
If getting increase then there would be some serious problem and you have to do a quick health check of you application server, database and other integrated technologies wherever your application reaching like ldap server for authentication, some other API’s or web services etc, and in parallel review thread dumps for stuck threads and share same with your developers to analyze quickly.
If you have one, two or few constant stuck  threads and it’s not increasing then you can monitor it for some more time to check if they get clear or not, if not then to clear them you have only option to restart your managed server(s), and its better to restart and clear them before they make further any impact.

What is Hogging Thread? 
Hogging threads are candidates for stuck threads.
Threads that “might” get stuck. These threads will be declared “stuck” after StuckThreadMaxTimeout seconds which usually is 600secs.

If the thread gets released by the request before this timeout, it wont be called hogging thread anymore and will be released to the thread pool.

Hogging Threads that have taken the too much time and we can assume that they are never going to come back.
Hogging threads help us take some decisions, lets say many threads are hogging, we may take a decision to create new threads for next cycle.
A hogging thread is a thread which is taking more than usual time to complete the request and can be declared as Stuck .

How Weblogic determine a thread to declare as hogging?

A thread declared as Stuck if it runs over 600 secs (default configuration which you can increase or decrease from admin console).
There is an internal WebLogic polar which runs every 2 secs (by default 2 secs and can be alter)
It checks for the number of requests completed in last two minutes
Then it check how much times each took to complete
Then it takes the average time of all completed request (completed in last 2 sec)
Then multiply average time with 7, and the value came consider as “usual time to complete the request”
Now weblogic check each current executed thread in last 2 secs and compare with above average time, if for any of the thread it’s above this value then that thread will declare as Hogged thread.
For example –
At a particular moment, total number of completed requests in last two seconds – 4
Total time took by all 4 requests – 16 secs
Req1 took – 5 secs, Req2 took – 3 secs, Req3 took – 7 secs, Req4 took – 1 sec
Average time = 16/4 = 4 secs
7*4 = 28 secs
Now weblogic check all executed threads to see which taking more than 28 secs, if any then that thread(s) declared as Hogged Thread.
Only the thing you can change with respect to hogging threads configuration is Polar time (Stuck Thread Timer Interval parameter)which is 2 secs by default. You can change this polar value to some different value like 4 secs if you want polar to run in every 4 secs instead of 2 secs.

Version:

1) Go to Fusion middleware Home directory with run or patch edition
cd $MW_HOME
grep -I version registry.xml
2) we can check the version from console also
Login to http://<server_name>:<port>/console
Weblogic version is there in the bottom

Data source:
A data source is an object that enables a Java Database Connectivity (JDBC) client to obtain a database connection. The data source has a collection of database connections called a connection pool. An application can request a connection from the data source, use the connection to access the database, and then close the connection. Rather than actually close the connection, however, the data source places it back in the connection pool to be used again. Data sources can be created as part of the configuration of an instance of WebLogic Server. When that server comes up, (or the data source is deployed), the server creates database connections to fill the connection pool of its data source.


Saturday, April 28, 2018

start/stop Node manager and Managed server in Weblogic:

How to start/stop Node manager and Managed server in Weblogic

We will be discussing Node manager and manage server start/stop in this article.

Weblogic Administration Console:

Weblogic Administration Console:
Web browser-based, graphical user interface to manage an Oracle WebLogic Server domain.

Use the Administration Console to:
Create /Clone/ Cluster/Start/Stop Managed servers.
Manage Datasources and connection pools
Deploy java EE applications
Change passwords.
View and Manage Log files related to WLS
Change Ports for Weblogic server and Java components.
Configure, start, and stop WebLogic Server instances
Configure WebLogic Server clusters
Configure WebLogic Server services, such as database connectivity (JDBC) and messaging (JMS)
Configure security parameters, including managing users, groups, and roles
Configure and deploy your applications
Monitor server and application performance
View server and domain log files
View application deployment descriptors
Edit selected run-time application deployment descriptor elements

How to access the Weblogic Administration Console:
Once administration server started, we can access the weblogic administration console using

http://<hostname:<port>/console

Incase of https

https://<hostname:<secure -port>/console, will get the login page:


Pass username and password which  set while created the weblogic domain, wll get the main page:


To make changes in configuration Via Weblogic console:

Before saving and activating any changes we have to obtain the domain configuration lock.This protection is to avoid changes from any other users during the edit session.
(i)Click on Lock & Edit and obtain lock configuration.












(ii)Make required changes and save it.
(iii)If any configuration level changes are in pending,Lock & Edit would be as Activate changes. Click Activate Changes to confirm the made changes.









Before activating all made changes are saved in directory $DOMAIN_HOME/pending and contains a new version of the configuration files (config.xml).AS Activate Changes button clicked,Administration Server issues a command to all the WebLogic Managed Server to update the configuration.If changed configuration is not accepted by any of the Managed Servers will be rolled back and error will be shown by Console. If configuration is accepted,changes will be activated and the configuration files will be updated.

Creating  managed server: This can be done vie console as well as backend(with command):
with console:
(i)Click Lock and Edit.
(ii) select Environment > Servers>Servers table>click new
(iii)On the Create a New Server: Server Properties page.
a.Enter the name of the server in the Name field.
b.In Listen Address,if you want to limit the valid addresses for a server instance, enter an IP address or DNS name.For example, enter 12.34.5.67 or mymachine, respectively. Otherwise, URLs to the server can specify any of the host computer's IP address, any DNS name that maps to one of the IP addresses, or the localhost string. The value you specify for the listen address is not the URL to the host machine and it does not include the communication protocol, listen port, or channel.

c.In Listen Port,enter the port number from which you want to access the server instance.
d.Specify whether or not this server will be a stand alone server or will belong to a cluster.
If this server is part of a cluster:

    Select an existing cluster from the drop down list.
    Select the button to create a new cluster

e.Click Next.
f.Review the configuration options you have chosen.
g.Click Finish.

(iv)To activate these changes, in the Change Center of the Administration Console, click Activate Changes. Not all changes take effect immediately—some require a restart.

Creating a cluster:

(i)click Lock & Edit
(ii)In the left pane of the Console, select Environment > Clusters.
(iii)Click New and select Cluster.
(iv)On the Cluster Properties page:

a.In Name, enter a unique name for your new cluster.
b.In Messaging Mode, select the messaging mode you want to use for this cluster.
If you are creating a cluster within an existing WebLogic Server environment or you need to ensure backward compatibility with older versions of WebLogic Server, you must use the multicast messaging type. IP multicast is a simple broadcast technology that enables multiple applications to subscribe to a given IP address and port number and listen for messages.

c.If you are using the unicast messaging mode, enter the unicast broadcast channel in Unicast Broadcast Channel. This channel is used to transmit messages with the cluster. If you do not specify a channel, the default channel is used.
d.If you are using the multicast messaging mode:
Enter the multicast address of the new cluster in Multicast Address. A multicast address is an IP address in the range from 224.0.0.0 to 239.255.255.255. The valid range is from 224.0.0.0 to 239.255.255.255. The default value used by WebLogic Server is 239.192.0.0. You should avoid using multicast addresses in the range x.0.0.1. This address must be unique to this cluster and should not be shared by other applications.

Enter the multicast port for the new cluster in Multicast Port. The multicast port is used by cluster members to communicate with each other. Valid values are between 1 and 65535.

(v)Click OK.
(vi)To activate these changes, in the Change Center of the Administration Console, click Activate Changes.Not all changes take effect immediately—some require a restart


Command to get console url:

echo "http://"$(cat $CONTEXT_FILE | grep s_webhost | cut -d '>' -f2 | cut -d '<' -f1)"."$(cat $CONTEXT_FILE | grep s_wls_admin_domain | cut -d '>' -f2 | cut -d '<' -f1)":"$(cat $CONTEXT_FILE | grep s_wls_adminport | cut -d '>' -f2 | cut -d '<' -f1)"/console"


Rollback the patch once failed in cutover phase in R12.2:

There may be scenario  when  cutover phase failed . It is possible to go back to previous state of cutover(rollback the patch) ,if flashback database is either enabled in the database or we have taken full backup prior to cutover.

with respect to Database Flashback to rollback the patch:

SQL>select FLASHBACK_ON from v$database;
FLASHBACK_ON
————
Yes

Case 1:

You are running an Online Patching cycle:

$ adop phase=prepare

$ adop phase=apply patches=12345

$ adop phase=finalize

$ adop phase=cutover

Cutover fails, and you need to go back to the state of the system before you ran the cutover phase.

If you had not run the cutover phase, you would have been able to roll back the patch application process by running the adop abort phase. However, this is not possible once cutover has been run.

two main parts to rollback the patch:

1) Database Restore : Flashback database is the fastest method to rollback the database changes and go back to point in time. We can use database restore technique also but that is very time consuming

Flashing Back the Database
a).First, shut down the database, then start it up in mount state:
SQL>shutdown immediate
Database closed.
Database dismounted.
ORACLE instance shut down.
SQL>startup mount
ORACLE instance started.

b).Restore the flashback to the specified time.
SQL>flashback database to time to_data(<time before teh cutover>;
Flashback complete.
c).Start the database in read-only mode:
SQL>alter database open read only;
Database altered.
Check all looks as expected.

d).Shut down the database, start it up in mount state, then open it with the resetlogs option:
SQL>shutdown immediate
Database closed.
Database dismounted.
ORACLE instance shut down.
SQL>startup mount
ORACLE instance started.
Database mounted.
SQL>alter database open resetlogs;
Database altered.

2) Filesystem restore: Depending on when cutover failed, you may also need to restore the application tier file systems

Restoring the File Systems

Whether you need to perform this step is conditional, depending on whether cutover failed before the file systems were switched. You can identify which of these cases applies by referring to the cutover logs in $NE_BASE/EBSapps/log/adop/<current_session_id>/cutover_<timestamp>/ for your current session id.

Case 1 – If the log messages indicate that cutover failed before the file systems were switched, do a clean shutdown of any services that are running. Then restart all the services using the normal startup script,

Case 2 – If the log messages indicate that cutover failed after the file systems were switched, then follow below step to switch the file systems back.
1.Source the environment on the new run file system.

2.From $ADMIN_SCRIPTS_HOME, shut down all the services (using adstpall.sh on UNIX).
b)In a multi-node environment, repeat the preceding two steps on all nodes, leaving the admin node until after all the slave nodes.

c) Switch file systems back
1.On all nodes where file systems have been switched, run the following command to switch the file systems back:
$ perl $AD_TOP/patch/115/bin/txkADOPCutOverPhaseCtrlScript.pl \
-action=ctxupdate \
-contextfile=<full path to new run context file> \
-patchcontextfile=<full path to new patch file system context file> \
-outdir=<full path to out directory>

d)Start up all services from the old run file system (using adstrtal.sh on UNIX).
e)In a multi-node environment, repeat the preceding two steps on all nodes, starting with the admin node and then proceeding to the slave nodes

Conclusion

After the restore is complete, you have two basic options for proceeding:
a) Abort the current patching cycle, if the issue that required you to restore was caused by the patches you were attempting to apply.

Here are steps to Abort an Online Patching Cycle

If a patching cycle is failing and the issue cannot be resolved quickly, it is possible to abort the patching cycle and return to normal runtime operation. The patch edition will be dropped.

You can abandon a patching cycle (without applying any patches) by running the command:
$ adop phase=abort

Aborting a patching cycle will drop the patch edition, but you must then run the cleanup and fs_clone phases before starting a new patching cycle. The cleanup must be a full cleanup.

For example:
$ adop phase=prepare
$ adop phase=apply patches=9999999
$ adop phase=abort
$ adop phase=cleanup cleanup_mode=full
$ adop phase=fs_clone

Optionally, you can combine the abort and cleanup commands as follows:
$ $ adop phase=abort,cleanup cleanup_mode=full
b) Identify and fix any other issues in the current patching cycle, and proceed with patching.

Friday, April 27, 2018

Adop(Ad online patching utility) utility:

Adop is the patch utility introduced in R12.2 to apply the patches online without any downtime.It is perl script and call internally adpatch . Uncompounded to apply patches directly using adpatch in R12.2.

Online patching make use of 11g database feature called as “Edition Based Redefinition” and applications side, there will be two file systems,patch and run(both are identical to each other and services will be running out of run file system). Patches are applied to patch file system, which does not require any downtime. For the patch changes to come to effect, patch file system is switched as run file system and vice versa, and services are restarted. In essence, Downtime will be only for services are restarted.

Adop Phases: 



Prepare:This sync current file system with run file system, to make sure patch file is up to date to run file system and ready for patch to be applied. This also create a new patch edition in database.

$ adop phase=prepare



Apply: In this phase, patches are applied to patch file system,many patches can be applied during this phase. Changed code objects are from patch edition of database. Changes to tables are stored in new columns which are only visible from patch edition of the database.

$ adop phase=apply patches=<patch no>


Finalize: This phase involves compiling of invalid objects etc.
$ adop phase=finalize


Cutover: In this patch file system will become as new run file system and patch edition of the database as new run edition. Restart application services from new run file system. This is where basically downtime is required.

$ adop phase=cutover


Cleanup: Deletes obsolete code objects and columns from earlier patch edition are removed at database.

$ adop phase=cleanup


Need to know:

adop -help
Enabling maintenance mode is not required in R12.2.
Special phases:
abort – Abort the current patching cycle.
actualize_all – Create new copies of all code objects in the patch
edition.
fs_clone – Copy the run file system to the patch file system.

Log location:
s_ne_base/EBSapps/log/adop//__/ext_name>/log
example, if s_ne_base was /u01/apps/R122/fs_ne, the session ID was 50, and the
was xxx_test, the path to the adop log files from 1th
July 2013 would resemble this:
/u01/apps/R122/fs_ne/EBSapps/log/adop/50/apply_20130701_112226/xxx_test/log

brief report for the current patching session.
display information of completed phases and time taken

Phases in ADOP in details:

Prepare: This syncs the current patch file system with run file system, to make sure patch file system is up to date compared to run file system and ready for the patches to get applied. This also created a new patch edition in the database. This syncing process is incremental process to be efficient.

1. Verifying data dictionary
2. Validating system is ready to prepare.
3. Detecting configuration changes
4. Detecting ad/txk codelevel bumpup
5. Checking for pending cleanup actions
6. Validating database is ready to prepare.
7. Starting Apps listener -- will be already running
8. Submitting ADZDPATCH concurrent program

This request is waiting to be processed by the Conflict Resolution
Manager.  This request cannot yet begin execution because other requests
may conflict with it. The Conflict Resolution Manager will determine when
this request may begin execution. User SYSADMIN submitted this request on
09-FEB-2019

9. Creating database patch edition
10. Synchronizing patch filesystem with run filesystem.

[1] SINGLE PATCH (DRV=u27984039.drv):  27984039
SYNCHRONIZATION is in progress

11. Stopping services on patch file system
 - Stopping admin server -- already shutdown
 - Stopping node manager -- already shutdown

12. The prepare phase completed successfully.

$ adop phase=prepare

  • Prepares the system for patching cycle.
  • Creates the Database Patch Edition
  • Validates system configuration
  • Check & Submit Concurrent Request 'Online Patching In Progress'(ADZDPATCH)
  • Prepare is run on all nodes in a mute-node configuration
  • Synchronizes the Run and Patch File System
  • If cleanup was not executed in previous adop cycle it will also run Cleanup.

sequence of activities in prepare phase:
1.Checks whether to perform a cleanup, which will be needed if the user failed to invoke cleanup after the cutover phase of a previous online patching cycle.

2.Validates system configuration to ensure that the system is ready to start an online patching cycle.
3.Checks to see if the database is prepared for online patching:
a)Checks if the database user is edition-enabled. If not, adop immediately exits with an error.

b) Checks to see if the patch service has been created. adop requires that a special database service exists for the purpose of connecting to the patch edition. This service is created automatically, but its continued existence is validated on each prepare.

c) Checks to see if logon trigger exists and is enabled. If the logon trigger is missing or the patch service has not been created, adop will automatically try to fix the issue so that it can proceed. If it cannot do so, it will exit with an error message.

d)Checks the integrity of the database data dictionary. If any corruption is found, adop exits with an errorease 12.2.

e) Checks that the E-Business Suite Technology Codelevel Checker (ETCC) has has been run, to verify that all required patches have been applied to the database.

4.Checks system configuration on each application tier node. A number of critical settings are validated to ensure that each application tier node is correctly registered, configured, and ready for patching.
Checks the file system, using the TXK script $AD_TOP/patch/115/bin/txkADOPPreparePhaseSanityCheck.pl. This script checks for the file system space, database connections, Apps/System/Weblogic Passwords, Contextfile Validation and so on
And it also Produces a report showing information about the most important tablespaces is generated. This report is created in $APPL_TOP/admin/$TWO_TASK/out.

5.Checks for the existence of the “Online Patching In Progress” (ADZDPATCH) concurrent program. This program prevents certain predefined concurrent programs from being started, and as such needs to be active while a patching cycle is in progress (that is, while a database patch edition exists).

The flow of process is

a.If the ADZDPATCH program has not yet been requested to run, a request is submitted.If it does not exists ,below error is reported
ERROR at line 1:

ORA-20008: No Concurrent Manager is defined that can run concurrent program

ADZDPATCH

b.The status of ADZDPATCH is determined. If it is pending, it may be waiting for an incompatible program to finish. After the incompatibility is clear, its status will change to running, and it will allow the prepare phase to proceed. A message to this effect is displayed to the user.
c.The next stage depends on whether the concurrent managers are running:

i.If the concurrent managers are all down, the prepare phase continues, with ADZDPATCH entering a status of pending (with the highest priority) until the managers are started.
ii.If the concurrent managers are partially up, but there is no manager defined that can run ADZDPATCH, then the prepare phase will exit with an error.
iii.If the concurrent managers are up, and there is one defined that can run ADZDPATCH, processing will loop until ADZDPATCH changes status from pending to running . The prepare phase then continues.
ADZDPATCH is cancelled when the cutover phase is complete.

If you want any custom program not to run in patching cycle ,you will have to make it incompatible with this program.
6.Invokes the TXK script $AD_TOP/patch/115/bin/txkADOPPreparePhaseSynchronize.pl to synchronize the patches which have been applied to the run APPL_TOP, but not the patch APPL_TOP. The script depends on the adop repository for patches that have been applied on the run APPL_TOP but not the patch APPL_TOP.

it Identify the patches that were applied to the run APPL_TOP and apply them to the patch APPL_TOP. The following steps are performed automatically:
a.The patches that need to be applied to the patch APPL_TOP are identified from the database.
b.The merged patches are applied by the adop utility.
The adop utility identifies the delta patches to be applied, and applies them silently to the current patch APPL_TOP. As this procedure only requires the application of delta patches, it requires less time

In some circumstances, the delta-style (incremental) synchronization method may fail when applying a series of patches to the patch edition. This can happen if the previous patching cycle included patches that failed to apply correctly, and was followed by subsequent patches that corrected the issue.

The skipsyncerror parameter enables you to specify that you expect any synchronization errors in the prepare phase to be fixed automatically in the synchronization that takes place with subsequent patches.

If the value of the parameter is passed as ‘yes’, the first patch to be synchronized will be done with the ‘autoskip’ flag set.
Important: It is your responsibility to check the log files and correct any errors in the subsequent apply phase, or to confirm that synchronization with subsequent patches resolved the issue.
An example of using this parameter would be as follows.

a.You run adop phase=prepare.
b.The phase fails with an error when trying to synchronize the run and patch file systems. That is, the attempt to synchronize a patch fails, but it is known that a subsequent patch will correct the problem.
c.You examine the log files and conclude that the synchronization errors will be fixed automatically in the synchronization that takes place with subsequent patches.
d.You run the command adop phase=prepare skipsyncerror=yes to restart the prepare phase. This time, application of the patch that failed in the previous prepare will be retried with the ‘autoskip’ flag set.
Synchronizing Customizations

The default delta-style (incremental) method of file system synchronization handles official patches but will not synchronize any manually applied customization. Examples of patching actions that are not synchronized by default include:

Compiling user-defined JSPs

Copying some third-party libraries

Copying and compiling user-defined concurrent programs

Copying and generating user-defined forms
To include custom patching actions in the default file system synchronization, you must include the required commands in the Custom Synchronization Driver, $APPL_TOP_NE/ad/custom/adop_sync.drv. You will add your customizations to the following section of the file:
#Begin Customization

#End Customization

All the actions defined in this file will be performed by adop automatically during the prepare phase. Be aware that there are two categories of custom command in adop_sync.drv: those that are run one time only, and those that are run at each file system synchronization (during the adop prepare phase).
Important: The adop_sync.drv file is not currently reset to its template file at any point. Consequently, after cutover (and before the next prepare phase), you should review the contents of adop_sync.drv and ensure the requirementns for your custom commands continue to be met.
7.Checks the database for the existence of a patch edition, and creates one if it does not find one.

a)A patch edition is created in the database.
b)All code objects in the patch edition begin as pointers to code objects in the run edition. Code objects in the patch edition begin as lightweight “stub objects” that point to the actual object definitions, which are inherited from earlier editions. Stub objects consume minimal space, so the database patch edition is initially very small in size.
c) As patches are applied to the patch edition, code objects are actualized (have a new definition created) in that edition.

8.Calls the $AD_TOP/patch/115/bin/txkADOPPreparePhaseSanityCheck.pl script again to confirm that the database connection to the patch edition is working.

Apply phase:

1. It will check for Run Edition context file :
2. It will check for Patch edition context file:
3. It will check for Patch file system free space.
4. Shut down the application tier services
5. Run the following commands in the order shown
        $AD_TOP/patch/115/sql/ADFIXUSER.sql
        @ $AD_TOP/sql/adutlrcmp.sql
6. Checking for existing adop sessions.
    Continuing with existing session
7. Applying patch

  • Patches are applied in this phase.
  • Adop internally calls adpatch to apply the patches, but we cannot run adpatch utility as standalone in R12.2 .
  • Patches are applied in the Patch Edition.
  • Application user are connected to RUN edition and they are not impacted by patching cycle.
  • We can apply multiple patches in a patching cycle.

Multiple patches can be applied, including customizations
Patches are applied to the copy (Patch Edition)
The production application is unaffected by the changes
Users are connected to the Application and can perform their work unaffected by the patch
Changes are made in the isolation of an Edition
The running application is unaffected by these changes.

Finalize Phase:

1. It compile invalid objects
2. It generate derived objects
3. We can run Finalize phase in ‘quick’ mode or ‘full’ mode where ‘quick’ mode is the default mode.
4. It is recommended to be done explicitly in order to reduce cutover (downtime) phase time
5. In this phase adop patching cycle can be pause as long as required, because after this finalize phase we need to run cutover where downtime required.
  • Ready the system for Cutover.
  • Compile Invalid objects.
  • If we don't run finalize, then the cutover phase will call finalize automatically before doing the actual cutover. But that will increase the downtime window for the cutover.
  • Computes any DDL to be executed before the cutover.

Cutover phase: This is downtime phase.This ensure ,ready to commit to application of the selected patches.Once cutover is complete, it is not possible to revert to the previous edition.

1. First it performs the ADOP validations on all of the application nodes
2. When we run the Cutover Phase all the users will be logged off the system including the third party applications.
3. First it will bring down the Application Services and then will start up the Application Services.
4. After the start up the Patch file system and Run file system will be switched over like patch file will be Run fs and Run fs will be Patch fs.
5. Even the Patch database edition will be promoted to be the Run database edition
6. It will terminates ‘ADZDPATCH’ concurrent program which was started in prepare phase.

  • Switches to the patch edition of database and file system.
  • In the phase, the system actually goes down.
  • All application tier services are stopped and starts after the cutover.

Following steps in this phase:

1) Shut down internal concurrent manager: The adop utility signals the internal concurrent manager to shut down, but will wait for any existing concurrent requests to finish before it proceeds with cutover actions. The system is still available to users during this waiting period.
If you do not wish to wait indefinitely for concurrent requests to finish, specify the option cm_wait=<maximum_minutes_to_wait> with a number of minutes that reflects your operational needs
On production systems, do not specify cm_wait, but monitor progress of concurrent tasks and take manual action on them if needed.It is good to schedule cutover during the time where least jobs are running.
On non-production systems, we specify cm_wait to limit the waiting time before cutover proceeds as it is development and we can tolerate abort of Concurrenyt Manager
Note: Cutover will take longer if it has to wait for long-running concurrent processes to complete. In such a case, you can expect to see an informational message of the form:
[STATEMENT] [END ] Waiting for ICM to go down
If you do not want to wait for in-progress concurrent requests to finish normally, you can terminate the internal concurrent manager by executing the adcmctl.sh abort command from a different shell

4) Cutover file system: Promote patch file system to become the new run file system, switching the $FILE_EDITION values in the patch and run enviroments. The current patch APPL_TOP becomes the new run APPL_TOP, and the current run APPL_TOP becomes the new patch APPL_TOP.
This task is completed by Autoconfig
5) Terminate old database sessions: Terminate any database connections to the old run edition of the database.
This task is performed by $FND_TOP/bin/txkADOPCutOverPhaseCtrlScript.pl script

6) Start application tier services: Application tier services are restarted, on the new run edition. The system is now available again to users.
This task is performed by $FND_TOP/bin/txkADOPCutOverPhaseCtrlScript.pl script

Example .
$ adop phase=cutover
This will promote the patch edition to be the new run edition, as well as switching the patch and run labels on the file systems (and thereby, as noted above, changing the patch file system to be the new run file system and the run file system to be the new patch file system).
Deferring Application Tier Restart at Cutover
Many times you may need to perform additional manual steps after cutover but before restarting the application tier services. If this is the case, you can supply an additional parameter to the cutover command that causes the application services to remain shut down:
$ adop phase=cutover mtrestart=no
With this parameter, cutover will complete without restarting the application tier services. You can perform any additional steps that require the services to be shut down, and then start the application tier services manually using the adstrtal.sh script.
JAR Files and Cutover
In an online patching cycle, the requisite JAR files are initially stored in the $APPL_TOP/admin/<SID>/out directory, and then uploaded into the database during the cutover phase. Therefore, the out directory must not be deleted at least until cutover is complete.

This task is performed by $FND_TOP/bin/txkADOPCutOverPhaseCtrlScript.pl script
2) Shut down application tier services: All application tier services are brought down. During this period, the system is unavailable to users.
This task is performed by $FND_TOP/bin/txkADOPCutOverPhaseCtrlScript.pl script

3) Cutover database: Promote patch database edition to become the new run database edition, using adzdpmgr.pl script.
This task is performed by $FND_TOP/bin/txkADOPCutOverPhaseCtrlScript.pl script

what happenes during cutover?
All the patching is done in online way. We just need to small downtime which is called cutover to complete the whole patching
During cutover ,these things happens
– Users are logged off the system
– The Patch file system is promoted to be the Run file system
– The Patch database edition is promoted to be the Run database edition
– Perform final maintenance operations
– Users are brought back online on the patched system.

Cleanup Phase:
  • Cleans up old edition and objects.
  • Recovers space.

Monday, April 23, 2018

Patch in Oracle Apps R12.2(ADOP(AD Online Patching):

Oracle has introduced online patching functionality called ADOP(AD Online Patching) in version R12.2.0. Online patching uses the latest feature of the Oracle database 11gR2 which is called “Edition Based Redefinition” and also uses multiple file systems on the application side.During online patching, business application users continue using the Oracle application and simultaneously a patch or a sequence of patches can be applied to another edition of the same database and application. Another edition here means another exact copy of database and application. And once the patching is complete the users are switched over to a patched file system/database in cutover phase by just bouncing middle tier services.

adop online patching utility doesn't require downtime. It involves minimal downtime during cutover phase where switching of filesystems happens.We can apply lot of patches and do cutover any time to minimize downtime.where as in  adpatch we just apply patch most of the times by bringing down applications or in hot patch mode.

There are five phases or life cycles of ADOP which are:
1) PREPARE                                                           
2) APPLY
3) FINALIZE
4) CUTOVER
5) CLEANUP


Prepare :-  Prepares the environment for patching. It involves synchronization of the filesystems fs1 (run ,filesystem) and fs2(patch filesystem), filesystems are inter changeable.


Apply :- Applies the specified patches to the environment. In this phase we can apply all the patches related to application. (Note: These patches need to be copied in fs_ne (non editioned filesystems))

Finalize :-  Performs any final steps required to make the system ready for cutover.In this phase we are getting ready for cutover phase.

Cutover :- Shuts down application tier services, makes the patch edition the new run edition, and then restarts application tier services. This is the only phase the involves a brief downtime. In cutover phase the filesystem switchover takes place. Previously the filesystem which was patch filesystem will now become run filesystem.

Cleanup :-  Removes obsolete code and data from old editions. In cleanup phase all the obsolute objects gets compiled .

abort - Aborts the online patching cycle by dropping the database patch edition. This phase cannot be specified with any other phase.

fs_clone - Recreates the patch edition file system as an exact copy of the run edition file system. This phase cannot be specified with any other phase. Use of fs_clone is normally not required. Situations that do require fs_clone are will explicitly document that requirement. If running this phase, ensure that your current working directory is not within the patch edition file system.

Steps:-
1. Download the patch and unzip on patch_top.
2. Must go through Readme.html or readme.txt.
3. Prepare the system for patching

     a.  we can run the .env file . or the adop utility sets its own environment.
     b. Then run the below command for prepare the System for patching:
Note: The adop utility sets its own environment. There is therefore no need to source the environment before running it.

$ adop phase=prepare

4. Wait for the prepare phase to be complete.
Please check the status from below points:

$ adop -status

Enter the APPS username: apps
Enter the APPS password:

5. After completion Prepare phase, go to next phase called Apply phase:

$adop phase=apply patches=1234567
adop phase=apply patches=, workers=
Enter the APPS password:
Enter the SYSTEM password:
Enter the WLSADMIN password:
Please wait. Validating credentials...

6 . After completion of apply phase, goto next phase called Finalize phase:
i.e like Ready the instance for cutover.
Used to perform the final patching operations that can be executed while the application is still online:

$ adop phase=finalize
adop phase=finalize workers= (called automatically)

Enter the APPS password:
Enter the SYSTEM password:
Enter the WLSADMIN password:
Please wait. Validating credentials...

7. After completion of Finalize phase, goto next phase called Cutover:
 Used to perform the transition to the patched environment:
$ adop phase=cutover
adop phase=cutover workers=
Enter the APPS password:
Enter the SYSTEM password:
Enter the WLSADMIN password:
Please wait. Validating credentials...

Note: Keep checking Status after every Phases.

8. After completion of Cutover phase, run next Cleanup Phase
We can do this step after instance is released to Customer.
 Used to remove old objects that are no longer needed:
$adop phase=cleanup
adop phase=cleanup (called automatically)
Enter the APPS password:
Enter the SYSTEM password:
Enter the WLSADMIN password:

Now synchronise the technology level between patch and run fc_clone.
7. adop phase=fs_clone

Note:
If there are some reason either the prepare or apply phase failed or any problems then you can abort this patching cycle.
Used only if the online patching cycle needs to be terminated before completion:
The abort phase can only be run after the prepare phase and before completion of the cutover phase. After the cutover phase completes, it is no longer possible to abort the online patching cycle.
$adop phase=abort

But always remember, after running abort, you must always run a full cleanup as:

$adop phase=cleanup cleanup_mode=full

So, this will remove all columns that were added by the patches but are no longer needed because of the abort.

And If that columns are not removed, then that may create problems in a later patching cycle.

Last two commands for adop
a. adop -help
b. adop -examples

Patch Log Files:-
To review the relevant log files after any patching operation. The adop log files are located on the non-editioned file system (fs_ne), under:

$NE_BASE/EBSapps/log/adop//__//log

Saturday, April 21, 2018

R12.2 ARCHITECTURE

Oracle Application have three tier architecture:

  • The Client (Desktop) Tier
  • The Application Tier
  • The Database Tier

All major services are started out of the Fusion Middleware ORACLE_HOME.


Client Tier:

Application Tier:

There are two oracle home:

FMW ORACLE_HOME (sometimes referred to as the Web or Java ORACLE_HOME)(replaced OracleAS 10.1.3 home).
10.1.2 ORACLE_HOME (sometimes referred to as the Tools, C, or Developer ORACLE_HOME).

  • Manage communication b/w database tier and client tier.
  • Consist of two oracle home(FMW Oracle Home and oracle As 10.1.2)
  • Consist of HTTP,Java, Forms and Concurrent processing manager.
  • Oracle E-Business Suite modules (packaged in the file formsapp.ear) are deployed out of the OracleAS 10.1.2 ORACLE_HOME, and the frmweb executable is also invoked out of this ORACLE_HOME.
  • fmwweb, FOrm executable, invoked out of FMW.
  • formsapp.ear,collection of form based module.
  • Http/web listener for all client request.
  • oacore> Core functionality.
  • Two architecture to support form based application(form servlet and socket mode),servlet is default mode and communicate with database using oracle net.
  • A server is an instance of WebLogic Server that runs in its own Java Virtual Machine (JVM) and has its own configuration. And there five WebLogic instances these are ;
  1. Admin(Server)
  2. oacore (also cluster)
  3. oafm (also cluster)
  4. forms (also cluster)
  5. forms–c4ws  (also cluster)



Another View:


Another View:




Client request :

Requests received from the desktop clients over n/w is processed by Web services component which includes the following major components (Web Listener (Oracle HTTP Server powered by Apache)  and Java Servlet Engine (Oracle WebLogic Server, WLS)).

The Web listener component of the Oracle HTTP server accepts incoming HTTP requests (for particular URLs) from  client browsers, and routes the requests to WLS.

If possible, the Web server services the requests itself, for example by returning the HTML to construct a simple Web page. If the page referenced by the URL needs advanced processing, the listener passes the request on to the servlet engine, which contacts the database server as needed.

HTTP and Oracle WebLogic Server configuration files are not stored under the $INST_TOP.

When compare 12.2 with previous R12 versions, we can’t not see any changes for Forms Services .No changes in Form Listener Servlet .They are still connecting database with Oracle Net networking infrastructure.But! Weblogic manages this processes.

Weblogic services are replaced with OC4J (Oracle Container for Java) as new Servlet Engine. All HTML-Based Applications and Oracle Application Framework (OAF) is still using Java servlets provided by Apache Jserv module and BC4J patterns (Business Components for Java) but Oracle Weblogic provides connection to database and application-specific functionality such as flexfields via WLS service. Weblogic is integrated with this tier planning to create hybrid applications using OAF-ADF and optimizing clustring performances . I think possible further requirements in the future for EBS to Fusion evolution is also key reason for Oracle’s decision on this change.

In 12.2 , OA Framework Architecture includes Weblogic services such as WLS,UIX,Java Controller, Metadata UI. And it looks like this:




Oracle Applications 12.2 have two file system:

FS1 : Production File System used by users
FS2 : Copy of production File System used by patching tools (adop : More on adop later)
Both  fs1 and fs2 which alternates between ‘run’ and ‘patch’ with every patching cycle at the end, means after cutover.
FS1 and FS2 can switch role i.e. FS2 will become production File system and FS1 will become File System for Patching at Cutover Phase of online patching (ADOP).
fs_ne non-editioned filesystem which is used to store files needed across all filesystems like log files, patch files etc.fs_ne stores data on filesystem like log, out, data import/exportfiles.

Dual File system
Two application tier ORACLE_HOMEs:

An OracleAS 10.1.2 ORACLE_HOME that was used in previous 12.x releases. 
An Oracle Fusion Middleware (FMW) ORACLE_HOME that supports Oracle WebLogic Server (WLS) and supersedes the Java (OracleAS 10.1.3)ORACLE_HOME that was used in previous releases.Oracle E-Business Suite R12 (12.2) uses Oracle Fusion Middleware 11g R1 PS3 (11.1.1.4) including WebLogic 10.3.4 as its application server (Note: In previous version of R12 i.e. 12.0.X and 12.1.X this is 10g R3 i.e. 10.1.3.X). Oracle HTTP Server (OHS) in R12 (12.2) is 11.1.1.4
The use of these two ORACLE_HOMEs enables Oracle E-Business Suite to take advantage of the latest Oracle technologies.


Application tier manage communication between client and database and consist of HTTP,java,form and concurrent processing.
HTTP/web listener for client request.
formsapp.ear:Collection form based modue.
frmweb:Executable for form. Invoked out of FMW.
The Oracle E-Business Suite modules (packaged in the file formsapp.ear) are deployed out of the OracleAS 10.1.2 ORACLE_HOME, and the frmweb executable is also invoked out of this ORACLE_HOME.
Oacore:core functionality.


Two architecture to support form based application.
  • form listener servlet
  • form soket mode


form listener servlet id default one and communicate with database using oracle net.







The Web services component of Oracle Application Server processes requests received over the network from the desktop clients, and includes the following major components:
• Web Listener (Oracle HTTP Server powered by Apache)
• Java Servlet Engine (Oracle WebLogic Server, WLS)
The Web listener component of the Oracle HTTP server accepts incoming HTTP requests (for particular URLs) from  client browsers, and routes the requests to WLS.
If possible, the Web server services the requests itself, for example by returning the HTML to construct a simple Web page. If the page referenced by the URL needs advanced processing, the listener passes the request on to the servlet engine, which contacts the database server as needed.



===============
Rapid Install deploys one WebLogic domain for Oracle E-Business Suite, with four different application types being provisioned out of the box:

oacore - Used to provide core functionality in Oracle E-Business Suite application tier Java code, including OAF-based functionality for Oracle E-Business Suite products.

oafm - Used for web services, Secure Enterprise Search, Oracle Transport Agent, and other components.

forms - Serves all Oracle Forms functionality.

forms-c4ws - Exposes Oracle Forms-based functionality as web services.

An additional application type, which is not deployed out-of-the-box, may be provisioned if additional Oracle applications are installed:

oaea: Used when installing additional Oracle applications such as Oracle E-Business Suite AccessGate, eKanban, and Spatial.

https://docs.oracle.com/cd/E26401_01/doc.122/e22953/T174296T589913.htm#:~:text=oacore%20-%20Used%20to%20provide%20core,Transport%20Agent%2C%20and%20other%20components.