Hi everyone,
Hope you’re doing good!
A few days ago, a client reached out to me because they had some issues during the rollback of a JDK patch that was installed in $GRID_HOME.
Let’s better understand about the scenario:
Client was running on RU 19.17 and need to apply RU 19.19. So, before they apply, they ran the analyze:
/oracle/app/193/grid/OPatch/opatchauto apply /sapcd/19C/AprilDBPatches/35037840 -analyze
==Following patches FAILED in analysis for apply: Patch: /sapcd/19C/AprilDBPatches/35037840/35042068 Log: /oracle/app/193/grid/cfgtoollogs/opatchauto/core/opatch/opatch2023-08-03_08-31-44AM_1.log Reason: Failed during Analysis: CheckConflictAgainstOracleHome Failed, [ Prerequisite Status: FAILED, Prerequisite output: The details are: Reason - Superset Patch 35042068 has Subset Patch 34419443 which has overlay patches [35004974] and these overlay patches conflict with Superset Patch Subset Patch 34419443 which has overlay patches [35004974] and these overlay patches conflict with Superset Patch OPatch recommends any one of the following actions - Please rebuild the superset patch to make su ...
Then client opened a SR with Oracle and Oracle Support said:
1. Rollback JDK patch 35004974 JDK BUNDLE PATCH 19.0.0.0.230418
2. Apply GI Release Update 35037840 19.19.0.0.0
3. Apply Release Update patch 35042068 DATABASE RELEASE UPDATE 19.19.0.0.0
A few months ago, the IT Security team executed the scan to check about vulnerabilities and they found that the binary under $GRID_HOME/jdk/bin/java must be updated. So, client proceed with installation of patch 35004974 – JDK BUNDLE PATCH 19.0.0.0.230418.
And, in order to install GI Release Update 19.19, they need to rollback the patch 35004974 from GRID_HOME. Well, they tried to rollback the patch:
opatch rollback -id 35004974
Let’s see if rollback completed fine:
[Aug 3, 2023 3:10:13 AM] [INFO] Command call is : /oracle/app/193/grid/bin/patchgen commit -rb 35004974 [Aug 3, 2023 3:10:13 AM] [INFO] /oracle/app/193/grid/bin/patchgen: line 138: /oracle/app/193/grid/jdk/bin/java: No such file or directory [Aug 3, 2023 3:10:13 AM] [SEVERE] OUI-67073:UtilSession failed: RollbackSession failed in system modification phase... '/oracle/app/193/grid/bin/patchgen: line 138: /oracle/app/193/grid/jdk/bin/java: No such file or directory ' [Aug 3, 2023 3:10:13 AM] [INFO] Finishing UtilSession at Thu Aug 03 03:10:13 UTC 2023 [Aug 3, 2023 3:10:13 AM] [INFO] Log file location: /oracle/app/193/grid/cfgtoollogs/opatch/opatch2023-08-03_03-09-39AM_1.log [Aug 3, 2023 3:10:13 AM] [INFO] Stack Description: java.lang.RuntimeException: RollbackSession failed in system modification phase... '/oracle/app/193/grid/bin/patchgen: line 138: /oracle/app/193/grid/jdk/bin/java: No such file or directory ' at oracle.opatch.OPatchSession.runPatchgen(OPatchSession.java:932) at oracle.opatch.RollbackSession.rollback(RollbackSession.java:1993) at oracle.opatch.RollbackSession.autoRollback(RollbackSession.java:903) at oracle.opatch.RollbackSession.processLocal(RollbackSession.java:1627) at oracle.opatch.RollbackSession.process(RollbackSession.java:2528) at oracle.opatch.RollbackSession.process(RollbackSession.java:2639) at oracle.opatch.OPatchACL.processRollback(OPatchACL.java:281) at oracle.opatch.opatchutil.NRollback.legacy_process(NRollback.java:1048) at oracle.opatch.opatchutil.NRollback.process(NRollback.java:242) at oracle.opatch.opatchutil.OUSession.nrollback(OUSession.java:1150) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at oracle.opatch.UtilSession.process(UtilSession.java:355) at oracle.opatch.OPatchSession.process(OPatchSession.java:2643) at oracle.opatch.OPatch.process(OPatch.java:873) at oracle.opatch.OPatch.main(OPatch.java:930) Caused by: java.lang.RuntimeException: RollbackSession failed in system modification phase... '/oracle/app/193/grid/bin/patchgen: line 138: /oracle/app/193/grid/jdk/bin/java: No such file or directory ' ... 18 more
Let’s pay attention on this piece of the message:
RollbackSession failed in system modification phase… ‘/oracle/app/193/grid/bin/patchgen: line 138: /oracle/app/193/grid/jdk/bin/java: No such file or directory
Well, can you believe that after this failed operation, the content of $GRID_HOME/jdk was totally erased? The $GRID_HOME/jdk directory was empty!
As DBA’s, of course we took a backup of $GRID_HOME before the rollback, so, we restored the $GRID_HOME/jdk content and then we can continue with troubleshooting.
Next step is to activate the DEBUG for opatch, so we can try to understand better what’s going on here. With grid user:
export OPATCH_DEBUG=true
opatch rollback -id 35004974
Let’s analyze the debug (output truncated to make easier to read it):
[SystemCall.ExecReturn: systemCommand= ("/oracle/app/193/grid/bin/patchgen commit -rb 35004974"), errMsg= mv: cannot move ‘/oracle/app/193/grid/lib/libasmclntsh19.so’ to ‘/oracle/app/193/grid/lib/libasmclntsh19.so.bak’: Permission denied make: *** [update_patchrep] Error 1 , normalMsg= 19 loading the appropriate library for linux patchgensh19.so loaded succesfully. Note: Successfully commited, created .s file with apply and recover patches mv -f /oracle/app/193/grid/lib/libasmclntsh19.so /oracle/app/193/grid/lib/libasmclntsh19.so.bak make libasmclntsh19.so returned code 2 , return code= 99, exec. ok= false] RollbackSession failed in system modification phase... 'mv: cannot move ‘/oracle/app/193/grid/lib/libasmclntsh19.so’ to ‘/oracle/app/193/grid/lib/libasmclntsh19.so.bak’: Permission denied make: *** [update_patchrep] Error 1 ' UtilSession failed: RollbackSession failed in system modification phase... 'mv: cannot move ‘/oracle/app/193/grid/lib/libasmclntsh19.so’ to ‘/oracle/app/193/grid/lib/libasmclntsh19.so.bak’: Permission denied make: *** [update_patchrep] Error 1
Let’s again pay attention here to the message:
mv: cannot move ‘/oracle/app/193/grid/lib/libasmclntsh19.so’ to ‘/oracle/app/193/grid/lib/libasmclntsh19.so.bak’: Permission denied
Well, user has no permission to move the file.
After a check on $GRID_HOME/lib, I noticed that some files were with wrong ownership (that’s a good question! How it was changed? Probably someone with a “heavy” finger did it!)
In Linux, is “easy” to fix that, you only need a server where the files have the right permission/ownership, which is our case here:
Let’s save all permissions/ownership in a file. As root user:
cd /oracle/app/193/grid
getfacl -R * >> /sapcd/enkitec/19c/permissions.txt
This will save the permissions and ownership for all files and subdirectories (this is why we used -R, for recursive use) inside permissions.txt file.
Let’s check the contente of this file, just a small piece:
# file: libasmclntsh19.a # owner: grid # group: oinstall user::rw- group::r-- other::r-- # file: libasmclntsh19.so # owner: grid # group: oinstall user::rwx group::r-x other::r-x
Perfect! Now, let’s restore the permissions/ownership in the server where we have the wrong ownership. As root user:
cd /oracle/app/193/grid
setfacl -R –restore=/sapcd/enkitec/19c/permissions.txt
This will restore, also in a recursive way, all the ownership/permissions.
Now we can test the rollback again, with grid user:
opatch rollback -id 35004974
Let’s check the output of this rollback operation:
Oracle Interim Patch Installer version 12.2.0.1.36 Copyright (c) 2023, Oracle Corporation. All rights reserved. Oracle Home : /oracle/app/193/grid Central Inventory : /oracle/app/oraInventory from : /oracle/app/193/grid/oraInst.loc OPatch version : 12.2.0.1.36 OUI version : 12.2.0.7.0 Log file location : /oracle/app/193/grid/cfgtoollogs/opatch/opatch2023-08-05_23-00-09PM_1.log Patches will be rolled back in the following order: 35004974 The following patch(es) will be rolled back: 35004974 Rolling back patch 35004974... RollbackSession rolling back interim patch '35004974' from OH '/oracle/app/193/grid' Patching component oracle.jdk, 1.8.0.201.0... RollbackSession removing interim patch '35004974' from inventory Log file location: /oracle/app/193/grid/cfgtoollogs/opatch/opatch2023-08-05_23-00-09PM_1.log OPatch succeeded.
Perfect!
Hope this helps!
Peace,
Vinicius
Related posts
Disclaimer
My postings reflect my own views and do not necessarily represent the views of my employer, Accenture.