Hi everyone,

Hope you’re doing good.

A client contacted us asking to update their Oracle Enterprise Manager and its agents from 13.5.0.16 to latest Release Update available.

First of all, we need to understand from which note we can find the latest OEM RU available. Here we go:

Enterprise Manager 13.5 Main Release Update List (Includes Plug-ins) (Doc ID 2760230.2)

From this note, we have a lot of informations, I will show below the two sections I’ve got the latest RU for OEM and OEM Agent.

Below we have the info for latest RU for OMS (Oracle Management Service), the OEM server:

Below we have the info for latest RU for OEM Agent:

OK, so, we know that we need at least two patches:

  • 36005341: RU 20 for OMS
  • 36005346: RU 20 for OEM Agent

Now, let’s check the note for the patch:

13.5.0 Enterprise Manager Cloud Control Base Platform Monthly Release Update (RU) 20 (Doc ID 3006198.1)

On this note, we can find this useful info:

So, we have basically three main instructions here:

  • Mandatory steps to execute after we apply the RU;
  • Prerequisite patches to apply before we apply the RU;
  • If your repository is running on 19.18, you also have mandatory patch(es). In our case here, we don’t have to apply it because repository is running on 19.21.

Please, keep in mind that before proceed updating OMS and OEM Agent, we also need to update the tools that we will use to perform this update. As we need to apply some “one-off” patches in OEM, we also need to update the tool to apply those one-off patches into OEM.

If you are a DBA, you know very well about OPatch. OK, for OEM/OEM Agent, the tools are a little bit different:

  • omspatcher: to apply patches into OMS
  • opatch: to apply one-off patches into OMS
  • agentpatcher: to apply patches into OEM Agent

Great, so, we need to update omspatcher, opatch and agentpatcher, you need to follow three notes for that:

13.5: How To Upgrade Enterprise Manager 13.5 Cloud Control OMSPatcher Utility to the Latest Version (Doc ID 2809842.1)

13.5: How to Upgrade AgentPatcher to the Latest Version (Doc ID 2810322.1)

PATCH 28186730 – OPATCH 13.9.4.2.14 FOR EM 13.4, 13.5 and FMW/WLS 12.2.1.3.0, 12.2.1.4.0 AND 14.1.1.0.0

So, to summarize, in order to update omspatcher, opatch and agentpatcher to latest versions the following patches must be applied:

Patch 19999993: OMSPatcher patch of version 13.9.5.18.0 for Enterprise Manager Cloud Control 13.5.0.0.0

Patch 33355570: AgentPatcher release of version 13.9.5.6.0 for Enterprise Manager Cloud Control Agent 13.5.0.0.0

Patch 28186730: OPatch 13.9.4.2.14 for EM 13.4, EM 13.5 and FMW/WLS 12.2.1.3.0, 12.2.1.4.0 and 14.1.1.0.0

Perfect, so, we need to download 8 files in total:

  • ————————————————–
  • Server Side:
    • 36005341: RU 16 for OMS
    • 19999993: OMSPatcher version 13.9.5.18.0
    • 35430934: MERGE REQUEST ON TOP OF 12.2.1.4.0 FOR BUGS 32720458 33607709
    • 34153238: HTTPS PROXY CONFIGURATION IS NOT USED WHEN PROTOCOL IS CONFIGURED TO TCP
    • 31657681: THREADS CONTEND FOR LOCK IN LOADFILEBASEDKEYSTORE WHEN OPENING TLS/SSL ENABLED JDBC CONNECTIONS
    • 28186730: OPatch 13.9.4.2.14 for EM 13.4, EM 13.5 and FMW/WLS 12.2.1.3.0, 12.2.1.4.0 and 14.1.1.0.0
  • ————————————————–
  • Agent Side
    • 36005346: RU 20 for OEM Agent
    • 33355570: AgentPatcher version 13.9.5.6.0
  • ————————————————–

Perfect!

Let’s proceed updating OMSPatcher and OPatch.

The ORACLE_HOME for OEM is: /u04/app/oracle/product/OEM_135.

The user in operating system whom is the owner of OEM is: oraoem.

PATH environment variable is defined as: $ORACLE_HOME/bin:$ORACLE_HOME/OMSPatcher:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:

Basically we need to:

  • Take a backup of current omspatcher folder;
  • Unzip the patch 19999993 into $ORACLE_HOME to “install” new omspatcher;
  • Check if omspatcher has been updated.
  • Unzip the patch 28186730 (to update OPatch);
  • Apply the patch 28186730;
  • Check if OPatch has been updated.

With user which is the owner of OEM in OS (Linux):

cd $ORACLE_HOME

mv OMSPatcher OMSPatcher.bkp.20230908

cp /sapcd/OEM_13.5.0.20/p19999993_135000_Generic.zip .

unzip -q -o p19999993_135000_Generic.zip

omspatcher version

OMSPatcher Version: 13.9.5.18.0
OPlan Version: 12.2.0.1.16
OsysModel build: Tue Apr 28 18:16:31 PDT 2020

OMSPatcher succeeded.

Now, let’s update OPatch utility:

cd /sapcd/OEM_13.5.0.20

unzip -q -o p28186730_1394214_Generic.zip

$ORACLE_HOME/oracle_common/jdk/jre/bin/java -jar /sapcd/OEM_13.5.0.20/6880880/opatch_generic.jar -silent oracle_home=$ORACLE_HOME

$ORACLE_HOME/OPatch/opatch version

OPatch Version: 13.9.4.2.14

OPatch succeeded.

OK, before we start applying the patches, it’s good to have the information about our environment. Let’s get the status about OMS with flag details. With oraoem user:

emctl status oms -details

Oracle Enterprise Manager Cloud Control 13c Release 5
Copyright (c) 1996, 2021 Oracle Corporation. All rights reserved.
Console Server Host : OEM-01.domain.com
HTTP Console Port : 7788
HTTPS Console Port : 7802
HTTP Upload Port : 4889
HTTPS Upload Port : 4903
EM Instance Home : /u04/app/oracle/product/gc_inst2/em/EMGC_OMS1
OMS Log Directory Location : /u04/app/oracle/product/gc_inst2/em/EMGC_OMS1/sysman/log
OMS is not configured with SLB or virtual hostname
Agent Upload is locked.
OMS Console is locked.
Active CA ID: 2
Console URL: https://OEM-01.domain.com:7802/em
Upload URL: https://OEM-01.domain.com:4903/empbs/upload

WLS Domain Information
Domain Name : GCDomain
Admin Server Host : OEM-01.domain.com
Admin Server HTTPS Port: 7102
Admin Server is RUNNING

Oracle Management Server Information
Managed Server Instance Name: EMGC_OMS1
Oracle Management Server Instance Host: OEM-01.domain.com
WebTier is Up
Oracle Management Server is Up
JVMD Engine is Up

Now, we need to apply the one-off patches, as they are listed as prerequisites BEFORE we apply OEM 13.5 RU 20.

The patches to be applied now are the following exhibited below:

  • 35430934: MERGE REQUEST ON TOP OF 12.2.1.4.0 FOR BUGS 32720458 33607709
  • 34153238: HTTPS PROXY CONFIGURATION IS NOT USED WHEN PROTOCOL IS CONFIGURED TO TCP
  • 31657681: THREADS CONTEND FOR LOCK IN LOADFILEBASEDKEYSTORE WHEN OPENING TLS/SSL ENABLED JDBC CONNECTIONS

All of those three patches listed above have basically the same Pre-Installation Instructions:

If you see the step 2, it’s clear: Stop all Servers (Admin Server and all Managed Server(s))

So, we need to stop OEM. Please keep in mind that the command to stop OEM will stop OEM itself, but not the Admin Server. To stop everything, you need to use the flag “-all”. So, let’s do it:

emctl stop oms -all -force

Next step is to apply the patches. I will not show all the output because the blog post will be huge. So, I’ll focus on main steps.

So. I’m going to perform the following steps:

  • Go to stage directory where we have the patches;
  • Unzip all three patch files;
  • Apply the patches one by one.

When opatch asks for your confirmation, just press “y” to move forward.

cd /sapcd/OEM_13.5.0.20

unzip -q -o p35430934_122140_Generic.zip

unzip -q -o p34153238_122140_Generic.zip

unzip -q -o p31657681_191000_Generic.zip

cd 35430934

opatch apply

cd ..

cd 34153238

opatch apply

cd ..

cd 31657681

opatch apply

cd ..

As example for the opatch apply, let’s check a piece of one of the outputs below:

Oracle Interim Patch Installer version 13.9.4.2.14
Copyright (c) 2024, Oracle Corporation. All rights reserved.


Oracle Home : /u04/app/oracle/product/OEM_135
Central Inventory : /u01/app/oraInventory
from : /u04/app/oracle/product/OEM_135/oraInst.loc
OPatch version : 13.9.4.2.14
OUI version : 13.9.4.0.0
Log file location : /u04/app/oracle/product/OEM_135/cfgtoollogs/opatch/opatch2024-03-26_12-36-41PM_1.log


OPatch detects the Middleware Home as “/u04/app/oracle/product/OEM_135”

Verifying environment and performing prerequisite checks…
OPatch continues with these patches: 31657681

Do you want to proceed? [y|n]
y
User Responded with: Y
All checks passed.

Please shutdown Oracle instances running out of this ORACLE_HOME on the local system.
(Oracle Home = ‘/u04/app/oracle/product/OEM_135’)


Is the local system ready for patching? [y|n]
y
User Responded with: Y
Backing up files…
Applying interim patch ‘31657681’ to OH ‘/u04/app/oracle/product/OEM_135’

Patching component oracle.javavm.jrf, 19.3.0.0.0…
Patch 31657681 successfully applied.

Great! All three mandatory patches to apply into OEM have been applied. Now, in order to apply RU 20, we need to keep OEM in shutdown state, but we need to start Admin Server. Easy-peasy.

You can find the syntax for emctl command here:

Oracle Enterprise Manager – Cloud Control Administrator’s Guide – 13c Release 5 – EMCTL Commands for OMS

Let’s go:

emctl start oms -admin_only

Oracle Enterprise Manager Cloud Control 13c Release 5
Copyright (c) 1996, 2021 Oracle Corporation. All rights reserved.
Starting Admin Server only…
Admin Server Successfully Started

Next step is run the analyze to check if everything is fine before we apply the patch. But before run the analyze we need to unzip the patch file:

cd /sapcd/OEM_13.5.0.20

unzip -q -o p36005341_135000_Generic.zip

cd 36005341

Now, let’s run the analyze:

omspatcher apply -analyze

You’ll be asked to enter some information related to Weblogic username, password for this user, a DB username and password as well:


Please enter OMS weblogic admin server URL(t3s://OEM-01.domain.com:7102):>
Please enter OMS weblogic admin server username(weblogic):>
Please enter OMS weblogic admin server password:>

Enter DB user name : sys
Enter ‘sys’ password :

The output is truncated:

Complete Summary
——————————-
The following sub-patch(es) are applicable:
oracle.sysman.top.oms:36003265,36003227,36003223,36003273,36003108,36003254

The following sub-patches are incompatible with components installed in the OMS system:
35582217,34430509,34706773,36003235,35854914,36003245,35582170,36003258,36003270

OMSPatcher succeeded.

The analyze session finished successfully, we can proceed with patching:

omspatcher apply

You’ll be asked to enter some information related to Weblogic username, password for this user, a DB username and password as well:


Please enter OMS weblogic admin server URL(t3s://OEM-01.domain.com:7102):>
Please enter OMS weblogic admin server username(weblogic):>
Please enter OMS weblogic admin server password:>

Enter DB user name : sys
Enter ‘sys’ password :

The output is truncated:

Patching summary:
—————–

Binaries of the following sub-patch(es) have been applied successfully:

Featureset Sub-patches Log file
———- ———– ——–
oracle.sysman.top.oms_13.5.0.0.0 36003108,36003223,36003227,36003254,36003265,36003273 opatch2024-03-26_13-33-20PM_1.log


The following sub-patches are incompatible with components installed in the OMS system:
35582217,34430509,34706773,36003235,35854914,36003245,35582170,36003258,36003270


Deployment summary:
——————-

The following artifact(s) have been successfully deployed:

Artifacts Log file
——— ——–
SQL rcu_applypatch_original_patch_2024-03-26_13-08-13PM.log
SQL rcu_applypatch_original_patch_2024-03-26_13-12-06PM.log
SQL rcu_applypatch_original_patch_2024-03-26_13-12-31PM.log



MRS-commands emctl_register_commands_2024-03-26_13-14-53PM.log

MRS-derivedAssocs emctl_register_derivedAssocs_2024-03-26_13-28-25PM.log


Log file location: /u04/app/oracle/product/OEM_135/cfgtoollogs/omspatcher/36005341/omspatcher_2024-03-26_12-54-00PM_apply.log



OMSPatcher succeeded.

Patch is applied. Let’s check into OEM.

In main OEM page, click on SYSMAN / About Enterprise Manager:

The below screen will be exhibited:

Perfect! OEM now is running on 13.5 Release Update 20

OK, if you remember, we have a step to run AFTER we apply the RU:

Let’s go to the recommended note:

Release of Fixes to Support OAuth move from OAM to IDCS (Doc ID 2998693.1)

This client is not using the “Online mode” for patching on OEM, this is when OEM can connect to My Oracle Support (Metalink). So, as you can see below, the steps mentioned in the above note are not required:

Perfect, so, we completed OMS part of update to RU 20.

Next we need to apply the patch into OEM Agents, first step is to update AgentPatcher utility in all servers running the agent. To identify all servers that we have agents running, you can perform the following steps in OEM server. You’ll need to login as SYSMAN and then list the targets where the type are oracle_emd (agents):

emcli login -username=SYSMAN

Enter password

emcli get_targets | grep oracle_emd

1 Up oracle_emd SERVER01.domain.com
1 Up oracle_emd SERVER02.domain.com
1 Up oracle_emd SERVER03.domain.com
1 Up oracle_emd SERVER04.domain.com
1 Up oracle_emd SERVER05.domain.com
1 Up oracle_emd SERVER06.domain.com
1 Up oracle_emd SERVER07.domain.com
1 Up oracle_emd OEM-01.domain.com
1 Up oracle_emd SERVER08.domain.com
1 Up oracle_emd SERVER09.domain.com
1 Up oracle_emd SERVER10.domain.com
1 Up oracle_emd SERVER11.domain.com
1 Up oracle_emd SERVER12.domain.com
1 Up oracle_emd SERVER13.domain.com
1 Up oracle_emd SERVER14.domain.com
1 Up oracle_emd SERVER15.domain.com
1 Up oracle_emd SERVER16.domain.com
1 Up oracle_emd SERVER17.domain.com
1 Up oracle_emd SERVER18.domain.com
1 Up oracle_emd SERVER19.domain.com
1 Up oracle_emd SERVER20.domain.com
1 Up oracle_emd SERVER21.domain.com
1 Up oracle_emd SERVER22.domain.com
1 Up oracle_emd SERVER23.domain.com
1 Up oracle_emd SERVER24.domain.com
1 Up oracle_emd SERVER25.domain.com
1 Up oracle_emd SERVER26.domain.com
1 Up oracle_emd SERVER27.domain.com
1 Up oracle_emd SERVER28.domain.com

Now we have the list of servers where we have the OEM Agent running, you’ll need to perform the following steps as the user which is owner of agent running on each server:

ORACLE_HOME=$(ps -ef |grep agent_13 | grep -v grep | grep emw | awk ‘{print $8}’ | sed ‘s/\/perl\/bin\/perl//g’)

cd ${ORACLE_HOME}

$ORACLE_HOME/AgentPatcher/agentpatcher version

mv AgentPatcher AgentPatcher.bkp.20230908

cp /sapcd/OEM_13.5.0.20/p33355570_135000_Generic.zip .

unzip -q -o p33355570_135000_Generic.zip

$ORACLE_HOME/AgentPatcher/agentpatcher version

AgentPatcher Automation Tool
Copyright (c) 2021, Oracle Corporation.  All rights reserved.

AgentPatcher version : 13.9.5.6.0
OUI version        : 13.9.4.0.0

AgentPatcher is updated, next step, we need to upload the patch 36005346 (Oracle Enterprise Manager 13c Release 5 Update 20 (13.5.0.20) for Oracle Management Agent) to OEM, as this OEM environment is not connected to Internet and we need to use the Offline approach for patching. To be able to upload the patch to OEM, we need to upload two files:

  • The patch file itself: p36005346_135000_Generic.zip
  • The patch file metadata: p36005346_135000_American English_M.xml

You can download the patch file metadata from same window you download the patch from Metalink / My Oracle Support:

Let’s upload the patch, click on Enterprise / Provisioning and Patching / Saved Patches:

Click on Upload:

Now choose the files for patch (zip file) and the patch metadata, which were downloaded before:

After some time, patch will be uploaded to OEM:

We are ready to apply the patch into Agents, for this, first click on Enterprise / Provisioning and Patching / Patches & Updates:

Next step, search for the patch number 36005346 and once the patch is displayed, click on Create a Plan. This will proceed with Patch Plan.

I recommend you do for only one server to test the procedure and the patch, then, you can repeat all the steps below for the rest of servers:

Next screen, we need to define a name for the Patch Plan and click on Add to add the Targets (Hosts) we want to apply the patch:

With the hosts specified (as I told, I will only use one target to validate the procedure – and please note that the real hostname has been obfuscated for security reasons), click on Create Plan:

Now that the patch plan is created. Click on patch plan:

In the first screen of Patch Plan, we have some few information. Just click on Next:

In the next screen, we have all the patches that will be applied on this patch plan. In this case, we have only patch 36005346, which is the OEM 13c RU 20 for Agent. Click on Next again:

In the next screen, we have the Deployment Options, also is listed the Oracle Home, Host and Agent where the patch will be applied. Click on Next again:

In the next screen, Validation is required for this patch to check if there is any conflict. Click on Analyze:

The Validation/Analysis will start. If you want, you can click on Show Detailed Results:

Once you click on Show Detailed Results, a new Tab will be opened in your browser. My advice to you is to change the Auto Refresh to 15 seconds. Then you just need to monitor the Status for the task:

We can see in above screen that Analysis completed with no issues. Now, our plan is Ready for Deployment. You can click on Next:

Once we are Ready for Deployment, just click on Deploy button:

You need to choose if you want to Deploy immediately or if you would like to schedule it to a later moment. In my case, I choose the option to Deploy immediately:

Once you click on Submit, the Deployment for the patch will start. If you want, you can also check the Details (I will put both screenshots, but as you can see, the Deployment has been successful implemented):

That’s great! Now, the agent on this specific server has been updated to Oracle Enterprise Manager 13c Release 5 Update 20 (13.5.0.20).

Now, you can create a new Patch Plan and add the rest of your servers (yes, you can add all servers to be updated at same time). Once you add all the remaining servers and follow the steps, the Oracle Enterprise Manager 13c Release 5 Update 20 (13.5.0.20) will be applied in all OEM Agents.

Useful notes I’ve used for this activity:

OMS:

Enterprise Manager 13.5 Main Release Update List (Includes Plug-ins) (Doc ID 2760230.2)
13.5.0 Enterprise Manager Cloud Control Base Platform Monthly Release Update (RU) 20 (Doc ID 3006198.1)
13.5: How To Upgrade Enterprise Manager 13.5 Cloud Control OMSPatcher Utility to the Latest Version (Doc ID 2809842.1)
Patch 19999993: OMSPatcher patch of version 13.9.5.18.0 for Enterprise Manager Cloud Control 13.5.0.0.0
Patch 36005341: Oracle Enterprise Manager 13c Release 5 Update 20 (13.5.0.20) for Oracle Management Service
PATCH 28186730 – OPATCH 13.9.4.2.14 FOR EM 13.4, 13.5 and FMW/WLS 12.2.1.3.0, 12.2.1.4.0 AND 14.1.1.0.0
Patch 35430934: MERGE REQUEST ON TOP OF 12.2.1.4.0 FOR BUGS 32720458 33607709
Patch 34153238: HTTPS PROXY CONFIGURATION IS NOT USED WHEN PROTOCOL IS CONFIGURED TO TCP
Patch 31657681: THREADS CONTEND FOR LOCK IN LOADFILEBASEDKEYSTORE WHEN OPENING TLS/SSL ENABLED JDBC CONNECTIONS

Agent:

13.5: How to Upgrade AgentPatcher to the Latest Version (Doc ID 2810322.1)
Patch 36005346: Oracle Enterprise Manager 13c Release 5 Update 20 (13.5.0.20) for Oracle Management Agent
Patch 33355570: AgentPatcher release of version 13.9.5.5.0 for Enterprise Manager Cloud Control Agent 13.5.0.0.0

Hope this helps.

Peace!

Vinicius

PS: last year, I created a post to show a simple Ansible playbook to update AgentPatcher in all servers we have the OEM Agent running. This time, I will create a post soon to show to check the RU version on OEM Agent using Ansible playbook.

You can see the posts for last year here:

How to Update Oracle Enterprise Manager from 13.5.0.9 to 13.5.0.16

Ansible Playbook to Update AgentPatcher on OEM Agents