opatch rollback cleaning up content of $GRID_HOME/jdk folder 

Hi everyone,

Hope you’re doing good!

A few days ago, a client reached out to me because they had some issues during the rollback of a JDK patch that was installed in $GRID_HOME.

Let’s better understand about the scenario:

Client was running on RU 19.17 and need to apply RU 19.19. So, before they apply, they ran the analyze:

==Following patches FAILED in analysis for apply:

Patch: /sapcd/19C/AprilDBPatches/35037840/35042068
Log: /oracle/app/193/grid/cfgtoollogs/opatchauto/core/opatch/opatch2023-08-03_08-31-44AM_1.log
Reason: Failed during Analysis: CheckConflictAgainstOracleHome Failed, [ Prerequisite Status: FAILED, Prerequisite output:
The details are:
Reason -
Superset Patch 35042068 has
Subset Patch 34419443 which has overlay patches [35004974] and these overlay patches conflict with Superset Patch
Subset Patch 34419443 which has overlay patches [35004974] and these overlay patches conflict with Superset Patch

OPatch recommends any one of the following actions
- Please rebuild the superset patch to make su ...

Then client opened a SR with Oracle and Oracle Support said:

1. Rollback JDK patch 35004974 JDK BUNDLE PATCH 19.0.0.0.230418

2. Apply GI Release Update 35037840 19.19.0.0.0

3. Apply Release Update patch 35042068 DATABASE RELEASE UPDATE 19.19.0.0.0

A few months ago, the IT Security team executed the scan to check about vulnerabilities and they found that the binary under $GRID_HOME/jdk/bin/java must be updated. So, client proceed with installation of patch 35004974 – JDK BUNDLE PATCH 19.0.0.0.230418.

And, in order to install GI Release Update 19.19, they need to rollback the patch 35004974 from GRID_HOME. Well, they tried to rollback the patch:

Let’s see if rollback completed fine:

[Aug 3, 2023 3:10:13 AM] [INFO]     Command call is : /oracle/app/193/grid/bin/patchgen commit -rb 35004974
[Aug 3, 2023 3:10:13 AM] [INFO]     /oracle/app/193/grid/bin/patchgen: line 138: /oracle/app/193/grid/jdk/bin/java: No such file or directory
[Aug 3, 2023 3:10:13 AM] [SEVERE]   OUI-67073:UtilSession failed: RollbackSession failed in system modification phase... '/oracle/app/193/grid/bin/patchgen: line 138: /oracle/app/193/grid/jdk/bin/java: No such file or directory
                                    '
[Aug 3, 2023 3:10:13 AM] [INFO]     Finishing UtilSession at Thu Aug 03 03:10:13 UTC 2023
[Aug 3, 2023 3:10:13 AM] [INFO]     Log file location: /oracle/app/193/grid/cfgtoollogs/opatch/opatch2023-08-03_03-09-39AM_1.log
[Aug 3, 2023 3:10:13 AM] [INFO]     Stack Description: java.lang.RuntimeException: RollbackSession failed in system modification phase... '/oracle/app/193/grid/bin/patchgen: line 138: /oracle/app/193/grid/jdk/bin/java: No such file or directory
                                    '
                                    	at oracle.opatch.OPatchSession.runPatchgen(OPatchSession.java:932)
                                    	at oracle.opatch.RollbackSession.rollback(RollbackSession.java:1993)
                                    	at oracle.opatch.RollbackSession.autoRollback(RollbackSession.java:903)
                                    	at oracle.opatch.RollbackSession.processLocal(RollbackSession.java:1627)
                                    	at oracle.opatch.RollbackSession.process(RollbackSession.java:2528)
                                    	at oracle.opatch.RollbackSession.process(RollbackSession.java:2639)
                                    	at oracle.opatch.OPatchACL.processRollback(OPatchACL.java:281)
                                    	at oracle.opatch.opatchutil.NRollback.legacy_process(NRollback.java:1048)
                                    	at oracle.opatch.opatchutil.NRollback.process(NRollback.java:242)
                                    	at oracle.opatch.opatchutil.OUSession.nrollback(OUSession.java:1150)
                                    	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
                                    	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
                                    	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
                                    	at java.lang.reflect.Method.invoke(Method.java:498)
                                    	at oracle.opatch.UtilSession.process(UtilSession.java:355)
                                    	at oracle.opatch.OPatchSession.process(OPatchSession.java:2643)
                                    	at oracle.opatch.OPatch.process(OPatch.java:873)
                                    	at oracle.opatch.OPatch.main(OPatch.java:930)
                                    Caused by: java.lang.RuntimeException: RollbackSession failed in system modification phase... '/oracle/app/193/grid/bin/patchgen: line 138: /oracle/app/193/grid/jdk/bin/java: No such file or directory
                                    '
                                    	... 18 more

Let’s pay attention on this piece of the message:

RollbackSession failed in system modification phase… ‘/oracle/app/193/grid/bin/patchgen: line 138: /oracle/app/193/grid/jdk/bin/java: No such file or directory

Well, can you believe that after this failed operation, the content of $GRID_HOME/jdk was totally erased? The $GRID_HOME/jdk directory was empty!

As DBA’s, of course we took a backup of $GRID_HOME before the rollback, so, we restored the $GRID_HOME/jdk content and then we can continue with troubleshooting.

Next step is to activate the DEBUG for opatch, so we can try to understand better what’s going on here. With grid user:

Let’s analyze the debug (output truncated to make easier to read it):

[SystemCall.ExecReturn: systemCommand= ("/oracle/app/193/grid/bin/patchgen commit -rb 35004974"), errMsg= mv: cannot move ‘/oracle/app/193/grid/lib/libasmclntsh19.so’ to ‘/oracle/app/193/grid/lib/libasmclntsh19.so.bak’: Permission denied
make: *** [update_patchrep] Error 1
, normalMsg= 19
loading the appropriate library for linux
patchgensh19.so loaded succesfully.

Note: Successfully commited, created .s file with apply and recover patches
mv -f /oracle/app/193/grid/lib/libasmclntsh19.so /oracle/app/193/grid/lib/libasmclntsh19.so.bak
make libasmclntsh19.so returned code 2
, return code= 99, exec. ok= false]
RollbackSession failed in system modification phase... 'mv: cannot move ‘/oracle/app/193/grid/lib/libasmclntsh19.so’ to ‘/oracle/app/193/grid/lib/libasmclntsh19.so.bak’: Permission denied
make: *** [update_patchrep] Error 1
'
UtilSession failed: RollbackSession failed in system modification phase... 'mv: cannot move ‘/oracle/app/193/grid/lib/libasmclntsh19.so’ to ‘/oracle/app/193/grid/lib/libasmclntsh19.so.bak’: Permission denied
make: *** [update_patchrep] Error 1

Let’s again pay attention here to the message:

mv: cannot move ‘/oracle/app/193/grid/lib/libasmclntsh19.so’ to ‘/oracle/app/193/grid/lib/libasmclntsh19.so.bak’: Permission denied

Well, user has no permission to move the file.

After a check on $GRID_HOME/lib, I noticed that some files were with wrong ownership (that’s a good question! How it was changed? Probably someone with a “heavy” finger did it!)

In Linux, is “easy” to fix that, you only need a server where the files have the right permission/ownership, which is our case here:

Let’s save all permissions/ownership in a file. As root user:

This will save the permissions and ownership for all files and subdirectories (this is why we used -R, for recursive use) inside permissions.txt file.

Let’s check the contente of this file, just a small piece:

# file: libasmclntsh19.a
# owner: grid
# group: oinstall
user::rw-
group::r--
other::r--

# file: libasmclntsh19.so
# owner: grid
# group: oinstall
user::rwx
group::r-x
other::r-x

Perfect! Now, let’s restore the permissions/ownership in the server where we have the wrong ownership. As root user:

This will restore, also in a recursive way, all the ownership/permissions.

Now we can test the rollback again, with grid user:

Let’s check the output of this rollback operation:

Oracle Interim Patch Installer version 12.2.0.1.36
Copyright (c) 2023, Oracle Corporation.  All rights reserved.


Oracle Home       : /oracle/app/193/grid
Central Inventory : /oracle/app/oraInventory
   from           : /oracle/app/193/grid/oraInst.loc
OPatch version    : 12.2.0.1.36
OUI version       : 12.2.0.7.0
Log file location : /oracle/app/193/grid/cfgtoollogs/opatch/opatch2023-08-05_23-00-09PM_1.log


Patches will be rolled back in the following order: 
   35004974
The following patch(es) will be rolled back: 35004974  

Rolling back patch 35004974...
RollbackSession rolling back interim patch '35004974' from OH '/oracle/app/193/grid'

Patching component oracle.jdk, 1.8.0.201.0...
RollbackSession removing interim patch '35004974' from inventory

Log file location: /oracle/app/193/grid/cfgtoollogs/opatch/opatch2023-08-05_23-00-09PM_1.log

OPatch succeeded.

Perfect!

Hope this helps!

Peace,

Vinicius