Hey guys!

How are you doing?

Oracle launch a new version for Oracle Enterprise Manager a few weeks ago: 13cR3.

In this blog post we’ll se how to upgrade to EM13cR3 from EM13cR2.

This virtual environment have just one virtual machine, the VM is called OEM.

In this VM, we have the following products installed:

  • OEM13cR2;
  • Database 12.1.0.2 (OEM Repository).

Well, we are considering that we already have an OEM 13cR2 environment installed. So, in this post we’ll check the steps to upgrade EM13cR2 to EM13cR3. So, in this post I won’t show how to install and configure an OEM environment.

So let’s go!

The first step is download all the installation files. My environment is Linux 64-bit, so, I’ve downloaded the files exhibited below:

Link to download this files:

https://www.oracle.com/technetwork/oem/grid-control/downloads/oem-linux64-4956012.html

Make sure that you’ve downloaded all the files and transfer all of them to the server. Use some software that uses SCP protocolo, for example.

Pre-Reqs Checking:

You need to ensure that all the pre-reqs are OK before continue to upgrade.

In the documentation Enterprise Manager Cloud Control Basic Installation Guide (click here), we can check it.

I won’t list all the pre-reqs, but, pay attention to the following, to the server:

1.1 CPU, RAM, Heap Size, and Hard Disk Space Requirements for Oracle Management Service – (click here)

2.1.3 Package Requirements for Oracle Management Service – (click here)

2.2.1 Kernel Parameter Requirements for Oracle Management Service – (click here) – even in 64-bit installations we need to install 32-bit packages.

2.3.1 Library Requirements for Oracle Management Service – (click here)

Do not forget to check all the pre-reqs, for example: users, groups, etc. Is supposed to all the pre-reqs to be ok, since we already have an installed environment.

When we finish the pre-reqs check noted above, we need to check for some specific upgrade pre-reqs. You can do this using the document: Enterprise Manager Cloud Control Upgrade Guide, in the chapter 3 Prerequisites for Upgrading to Enterprise Manager Cloud Control 13c Release 3 – (click here)

In this document, we have some items:

3.1 Hardware, Software, and Platform Requirements – we already talked about this document few lines above.

3.2 Supported Database Release Requirements – this is a point that we need to check carefully. If we want to Upgrade OEM to EM13cR3, we need to check if database version used for the repository is supported. EM13cR3 uses only databases in versions 12.2.0.1+ or 12.1.0.2 as repository. To check the database versions supported, please check tab Certifications from My Oracle Support:

3.3 Database Patch Requirements – Is mandatory that we are running database with the latest PSU applied. This blog post won’t show how to do this.

3.4 Optimizer Adaptive Feature Disabling Requirements – parameter optimizer_adaptive_features needs to be defined to FALSE.

3.5 Database Initialization Parameter in Oracle Database 12.2.0.1.0/18.0.0.0.0 – If repository is running in those versions, you need to set parameter _allow_insert_with_update_check to TRUE.

3.6 Management Agent Patch Requirements – If you still have some agent running in version 12.1.0.4 on AIX you will need to apply patch 20282974 before upgrade OMS.

3.7 Port Requirements – If the actual OMS installation use some port below port 1024, the Upgrade will fail.

3.8 Customization Removal Requirements – If you have some datasource with customized parameters in Weblogic, remove them and use the default parameters while you are doing upgrade. If you use smart card authentication, remove this configuration as well. Both configurations can be recreated after upgrade.

3.9 Oracle BI Publisher Shutdown Requirements – Ensure that BI Publisher is down.

3.10 Database Service Instance Creation Request Verification Requirements – If you was creating a Database Cloud Service, wait this operation finishes.

3.11 Repository Table Snapshot Verification Requirements – Ensure that you don’t have any materialized view log (snapshot logs) in EM’s owner repository tables. Usually is SYSMAN.

SQL> select master,log_table 
from all_mview_logs where log_owner='SYSMAN';

If  above query return any results. Drop it. There is an example script to generate drop commands:

SQL> select 'drop materialized view log on '||table_name||';' 
from all_mview_logs where log_owner='SYSMAN';

3.12 Logon and Logoff Trigger Setting Verification Requirements – If you have any LOGON or LOGOFF triggers, disable them:

SQL> SELECT owner,trigger_name
 FROM sys.dba_triggers WHERE 
(TRIGGERING_EVENT LIKE 'LOGON%' 
or TRIGGERING_EVENT LIKE 'LOGOFF%') 
AND status='ENABLED'; 

OWNER                TRIGGER_NAME
-------------------- ------------------------------ 
GSMADMIN_INTERNAL    GSMLOGOFF 

alter trigger GSMADMIN_INTERNAL.GSMLOGOFF disable;

3.13 Delete Target Operation Auditing Requirements –  You need to enable audit operations for DELETE_TARGET:

[oracle@oem ~]$ emcli login -username=sysman
Enter password : *********

Login successful

emcli update_audit_settings -operations_to_enable="DELETE_TARGET" -audit_switch="ENABLE" -file_prefix="aud"

3.14 Selectively Skipping Some Job Type Updates for Reduced Downtime of Your Enterprise Manager System – While the upgrade is doing, some types of jobs are registered. As part of the register of this job, all active executions will be automatically updated to this new type of job registered. We can skip this step so we’ll reduce downtime while Upgrade is doing. So, if we skip, this jobs will be updated after OEM upgrade and with OEM up and running, without downtime.

To identify the job types that we want to skip during upgrade, connect to database with SYSMAN and execute the following SQL:

SELECT job_type, COUNT(1) as n_execs FROM MGMT_JOB_EXEC_SUMMARY JOIN MGMT_JOB_TYPE_INFO USING (job_type_id) WHERE status NOT IN (3,4,5,8,18,19,23) GROUP BY job_type HAVING COUNT(1) > 5000 ORDER BY COUNT(1) DESC;

If the query return any rows, pick the value from JOB_TYPE column and do the following INSERT:

INSERT INTO MGMT_PARAMETERS(parameter_name, parameter_value) VALUES ('mgmt_job_skip_job_type_upg.1', '');
COMMIT;

3.15 EMKEY Copy Requirements – You need to copy emkey from OMS to repository, set the environment variables to OMS Home and execute the following command:

[oracle@oem ~]$ emctl config emkey -copy_to_repos -sysman_pwd oracle123
Oracle Enterprise Manager Cloud Control 13c Release 2
Copyright (c) 1996, 2016 Oracle Corporation. All rights reserved.
The EMKey has been copied to the Management Repository. This operation will cause the EMKey to become unsecure.
After the required operation has been completed, secure the EMKey by running "emctl config emkey -remove_from_repos".

In the above example, the password for SYSMAN is oracle123.\

To check emkey status:

emctl status emkey

If emkey was copied, the following message will be exhibit:

The EMKey is configured properly, but is not secure.
Secure the EMKey by running "emctl config emkey -remove_from_repos".

3.16 Certificate Key Strength Requirements – You need to ensure that certificate used by EM should match at least 1024 bits. If you is using a certificate with lower security level (for example: in EM 11gR1), you need to create a new one with at least 1024 bits. You can follow the instructions at Metalink’s Note (MOS – My Oracle Support) to fix this issue: 1611578.1.

3.17 Out-of-Box Memory Settings Backup Requirements – If you have changed the default values for JVM memory, you need to save this values for reference to not lose the during Upgrade process. 

3.18 Prerequisite Check and Environment Validation Requirements – You need to execute EM Prerequisit Kit to check if all pre-reqs are OK to start Upgrade.

Go to the directory where the installation files were copied and run the following command (you will need X11 – Display):

./em13300_linux64.bin EMPREREQ_KIT=true

[oracle@oem oem]$ ./em13300_linux64.bin EMPREREQ_KIT=true
0%...........................................................................100%
Launcher log file is /tmp/OraInstall2018-07-06_10-38-10PM/launcher2018-07-06_10-38-10PM.log.
Starting Oracle Universal Installer

Checking if CPU speed is above 300 MHz. Actual 2793.532 MHz Passed
Checking monitor: must be configured to display at least 256 colors. Actual 16777216 Passed
Checking swap space: must be greater than 512 MB. Actual 3967 MB Passed
Checking if this platform requires a 64-bit JVM. Actual 64 Passed (64-bit not required)

Preparing to launch the Oracle Universal Installer from /tmp/OraInstall2018-07-06_10-38-10PM
====Prereq Config Location main===
/tmp/OraInstall2018-07-06_10-38-10PM/stage/prereq
EMGCInstaller args -scratchPath
EMGCInstaller args /tmp/OraInstall2018-07-06_10-38-10PM
EMGCInstaller args -sourceType
EMGCInstaller args network
EMGCInstaller args -timestamp
EMGCInstaller args 2018-07-06_10-38-10PM
EMGCInstaller args -paramFile
EMGCInstaller args /tmp/sfx_TVTdCx/Disk1/install/linux64/oraparam.ini
EMGCInstaller args EMPREREQ_KIT=true
EMGCInstaller args -nocleanUpOnExit
DiskLoc inside SourceLoc/home/oracle/oem
EMFileLoc:/tmp/OraInstall2018-07-06_10-38-10PM/oui/em/
ScratchPathValue :/tmp/OraInstall2018-07-06_10-38-10PM

The following screen will be exhibited:

Click on “Upgrade an Existing Enteprise Manager System” and then select the actual installation. Then click on [Next].

The following screen will be exhibited:

Input SYS and SYSMAN passwords for the database used as repository and then click on [Execute].

The pre-reqs check will be executed:

The following items were failed:

Recommendation :

The pga_aggregate_target instance parameter should be set to at least 1000000000 bytes.

CorrectionType : Not Required

Log Location : /tmp/OraInstall2018-07-06_10-38-10PM/emdbprereqs/LATEST/componentLog

Recommendation :

The sga_target instance parameter should be set to at least 3000000000 bytes.

CorrectionType : Not Required

Log Location : /tmp/OraInstall2018-07-06_10-38-10PM/emdbprereqs/LATEST/componentLog

Recommendation :

The job_queue_processes instance parameter should be zero.

CorrectionType : Auto

Log Location : /tmp/OraInstall2018-07-06_10-38-10PM/emdbprereqs/LATEST/componentLog

Recommendation :

Temporarily disabling the gather stats job. It will be automatically re-enabled at the end of this process.

CorrectionType : Auto

Log Location : /tmp/OraInstall2018-07-06_10-38-10PM/emdbprereqs/LATEST/componentLog

Recommendation :

Temporarily disabling the gather stats job. It will be automatically re-enabled at the end of this process.

CorrectionType : Auto

Log Location : /tmp/OraInstall2018-07-06_10-38-10PM/emdbprereqs/LATEST/componentLog

 

According to the EM13cR3 documentation, those are the description for CorrectionType category:

Not Required – It is a warning message and you can ignore it and move ahead.

Manual – It is an error message and you must manually fix it before moving ahead.

Auto – It is an error message and you can ignore it since EM has predefined corrective actions to fix it.

 

3.19 OMS Backup Requirements – Make sure you are a good DBA and take a backup of your environment. 🙂

3.20 OMS Shutdown Requirements – Stop the OMS components

emctl stop oms -all

3.22 Management Agent Proxy Removal Requirements – If you have some proxy defined for some agents, remove them. If you not remove, the agent’s upgrade will fail.

3.23 Additional Preparation and Steps Required for Performing an Upgrade and Transition to DR Readiness – This step is useful only you have DR configured for EM.

3.24 Undeploy Obsolete Plug-ins – You need to undeploy some obsolete plugins. There is a list of them:

  • Oracle Audit Vault (oracle.em.soav)
  • Oracle Virtual Networking (oracle.em.sovn)
  • Oracle Engineered System Healthchecks (oracle.em.sehc)
  • Oracle Ops Center Infrastructure stack (oracle.em.sooc).

Now you already did a check on pre-reqs, you can start the upgrade process.

Let’s follow to the chapter 4 Upgrading Oracle Management Service and Oracle Management Repository to 13c Release 3 – (click here)

To make the rollback step (if needed) more easier, you should create a restore point in the database:

col name for a15
col time for a35

SQL> create restore point before_upgrade guarantee flashback database;

Restore point created.

SQL> select name,time,guarantee_flashback_database,storage_size from v$restore_point;

NAME TIME GUA STORAGE_SIZE
--------------- ----------------------------------- --- ------------
BEFORE_UPGRADE 11-JUL-18 07.59.57.000000000 PM YES 304087040

I choose the Upgrade in graphical mode:

Go to the directory where installation files are located:

[oracle@oem ~]$ cd ~/oem
[oracle@oem oem]$ ./em13300_linux64.bin
0%...........................................................................100%
Launcher log file is /tmp/OraInstall2018-07-07_12-26-02AM/launcher2018-07-07_12-26-02AM.log.
Starting Oracle Universal Installer

Checking if CPU speed is above 300 MHz. Actual 2793.532 MHz Passed
Checking monitor: must be configured to display at least 256 colors. Actual 16777216 Passed
Checking swap space: must be greater than 512 MB. Actual 3967 MB Passed
Checking if this platform requires a 64-bit JVM. Actual 64 Passed (64-bit not required)

Preparing to launch the Oracle Universal Installer from /tmp/OraInstall2018-07-07_12-26-02AM
====Prereq Config Location main===
/tmp/OraInstall2018-07-07_12-26-02AM/stage/prereq
EMGCInstaller args -scratchPath
EMGCInstaller args /tmp/OraInstall2018-07-07_12-26-02AM
EMGCInstaller args -sourceType
EMGCInstaller args network
EMGCInstaller args -timestamp
EMGCInstaller args 2018-07-07_12-26-02AM
EMGCInstaller args -paramFile
EMGCInstaller args /tmp/sfx_Pkro6J/Disk1/install/linux64/oraparam.ini
EMGCInstaller args -nocleanUpOnExit
DiskLoc inside SourceLoc/home/oracle/oem
EMFileLoc:/tmp/OraInstall2018-07-07_12-26-02AM/oui/em/
ScratchPathValue :/tmp/OraInstall2018-07-07_12-26-02AM

In the first screen, we have the option to input My Oracle Support account data to initiate Oracle Configuration Manager:

If you choose to not configure this, uncheck the option [I wish to receive security updates via My Oracle Support] and then click on [Next] button. You should receive a warning, click on [Yes] button.

 

In the next screen is possible to check if there are some critical patches to be installed before the upgrade process. This is valid when your environment aren’t updated with latest patches. If you are OK to continue, click on [Skip] and then on [Next] buttons:

A pre-reqs check will be executed. If you ran the EM Prerequisit Kit, the warnings and failures should be the same:

There is a pre-req failure for the kernel parameter kernel ip_local_port_range. This can be ignored since the value expected is the same to the value configured. Click on [Next].

A warning message should be exhibited because some failures were found and it’s supposed that those failures were not fixed. You can move on, click on [Yes] to continue:

In the next screen you have the option to install a fresh one EM. We will upgrade the actual installation. Choose [Upgrade an existing Enterprise Manager System] and then select the actual installation. Then click on [Next] button:

A new Oracle Home should be defined. Input a valid path. Click on [Next]:

As reference:

Actual path: /u01/app/oracle/em13c/middleware
New path: /u01/app/oracle/13.3/middleware

In the next screen you need to input some data to the database:

Input SYS and SYSMAN password:

You are not able to continue without a backup (or to take OUI aware that you have one :-)) . Click on [Confirm that you have backed up the Management Repository] option.

It’s a best practice click on option [Disable DDMP jobs]. DDMP Jobs means Deferred Data Migration Jobs. As the official documentation (click here) cita:

Deferred Data Migration is a post-upgrade activity to migrate the format of the data stored in an earlier release of Enterprise Manager to the format compatible with the upgraded Enterprise Manager system. The migration activity is essentially a job in Enterprise Manager that is submitted when the Oracle Management Repository gets upgraded and is scheduled to run in the background when the upgraded Enterprise Manager system starts functioning.

The format of the data stored in Enterprise Manager Cloud Control is different from the format in which the data was stored in any earlier release of Enterprise Manager.

When you upgrade from an earlier release of Enterprise Manager to Enterprise Manager Cloud Control, the data format gets automatically converted or migrated to the format that is compatible with Enterprise Manager Cloud ontrol.

However, the time taken to migrate the data format depends on the volume of data available in your earlier release of Enterprise Manager. Therefore, if you have a large amount of data, then it takes longer to migrate, and as a result, the upgrade process takes more time to complete. Unfortunately, until the upgrade process completes, your existing Enterprise Manager system might be unavailable, particularly when you use a 1-System upgrade approach (either on the local host or on a different, remote host).

Considering this, Oracle has fine-tuned its upgrade process to migrate the data format in two distinct phases.

In the first phase, when the Enterprise Manager system is shut down and upgraded, the most critical data relevant to the functioning of Enterprise Manager Cloud Control is migrated within a short time so that the new system can start functioning without much downtime. At this point, only some historical data is unavailable, but you can start monitoring the targets in the upgraded Enterprise Manager system, and see new alerts generated by the upgraded Oracle Management Agent.

In the second phase, after the upgraded Enterprise Manager system starts functioning, the remaining data is migrated.

The data whose format is migrated in the second phase, after Enterprise Manager Cloud Control starts functioning, is called the Deferred Data, and this process of migrating from old to new format is called the Deferred Data Migration.

With both options selected, click on [Next] button:

This warning is exhibited because you have MD5 certificates. This certificates still work because this is an Upgrade, but, is recommended that the targets use SHA-based certificates. Click on [OK] button.

A information message is exhibited asking you to apply the mandatory patches in the database used as repository. You can read more about this patches in the documentation. The message tell you that if you don’t apply the patches, the repository’s upgrade will fail.

To check the patches: (click here)

Or below:

3.3 Database Patch Requirements
Ensure that you apply the latest PSU on the supported database.

It is mandatory to apply the latest DB PSU on the supported database before upgrading to Enterprise Manager Cloud Control 13.3.0.0.

So, ensure that you have the latest PSU applied on database!

Clique em [OK].

Uma mensagem informando que temos JVM Diagnostics Engines configurados no nosso ambiente. A mensagem pede para verificar se os Engines estão Up. Sabemos que não estão pois realizamos um Stop do OMS durante os pré-requisitos. Click on [OK] button.

Another information message is exhibited because of the Agent that is running in the server is monitoring OMS and Repository. It’s recommended to stop the agent and continue with the upgrade process. Click on [OK] button.

A warning is exhibited due some pre-reqs that aren’t OK:

  • Set the parameter job_queue_processes to zero (0);
  • Temporarily disable CBO stats gathering job in the database.

Click on [Yes]  and the installer will automatically execute this tasks. After the upgrade process completed, the original values will be restored automatically by the installer.

There is another warning message due some pre-reqs not OK:

– SGA size lower than expected value;
– PGA size lower than expected value;
– Grant explicity execute on DBMS_RANDOM to DBSNMP.

In this case, is a virtual machine environment, so, the memory components are lower than expected because I did this. We can continue. Click on [OK] buton.

A list of plugins that will be upgraded will be exhibited. As you also can see a list of plugins that needs to be migrated after the upgrade. Click on [Next].

In the next screen is possible to upgrade another plugins during Upgrade process. If you want to, select the plugins clicking on each box related to plugin. Click on [Next] button to continue:

In the next screen exhibited, you need to input a password for Weblogic user. In my example I simply input as Oracle 123. Click on [Next] button to continue:

Is possible to configure a shared location for Oracle BI Publisher. I unchecked this item for now. Click on [Next] button to continue:

In the next screen exhibited you can see the ports that will be used by Oracle HTTP Server. Click on [Next] button to continue:

Well, we are all set  to continue with Upgrade. Click on [Upgrade] button:

A screen with Upgrade Process will be exhibited. You now can take a coffee and wait for the completion! 🙂

In my VM environment, Upgrade process has completed in 1 hour and 40 minutes.

The expected result is that all the process will run without any type of issue. In the end, setup information will be exhibited. Click on [Close] button, but not forget to execute as root:

Reproducing the information above:

This information is also available at:

/u01/app/oracle/em13.3/middleware/install/setupinfo.txt

See the following for information pertaining to your Enterprise Manager installation:

Use the following URL to access:

1. Enterprise Manager Cloud Control URL: https://oem.oracle.com:7802/em
2. Admin Server URL: https://oem.oracle.com:7102/console
3. BI Publisher URL: https://oem.oracle.com:9803/xmlpserver/servlet/home

The following details need to be provided while installing an additional OMS:

1. Admin Server Host Name: oem.oracle.com
2. Admin Server Port: 7102

You can find the details on ports used by this deployment at : /u01/app/oracle/em13.3/middleware/install/portlist.ini

NOTE:
An encryption key has been generated to encrypt sensitive data in the Management Repository. If this key is lost, all encrypted data in the Repository becomes unusable.

A backup of the OMS configuration is available in /u01/app/oracle/em13.3/gc_inst/em/EMGC_OMS1/sysman/backup on host oem.oracle.com. See Cloud Control Administrators Guide for details on how to back up and recover an OMS.

NOTE: This backup is valid only for the initial OMS configuration. For example, it will not reflect plug-ins installed later, topology changes like the addition of a load balancer, or changes to other properties made using emctl or emcli. Backups should be created on a regular basis to ensure they capture the current OMS configuration. Use the following command to backup the OMS configuration:
/u01/app/oracle/em13.3/middleware/bin/emctl exportconfig oms -dir

[root@oem ~]# /u01/app/oracle/em13.3/middleware/allroot.sh

Starting to execute allroot.sh .........

Starting to execute /u01/app/oracle/em13.3/middleware/root.sh ......
/etc exist
/u01/app/oracle/em13.3/middleware
Finished execution of /u01/app/oracle/em13.3/middleware/root.sh .....

 

Now you can go to OEM URL, in my environment is: https://oem.oracle.com:7802/em

Login screen will be exhibited:

 

After logon on OEM, we can see the home page:

In the next blog post we’ll execute the post-upgrade tasks and also upgrade the plugins.

All the best!

Vinicius