CLSRSC-180: An error occurred while executing the command ‘/bin/rpm -qf /sbin/init’ during postpatch 

Hello everyone,

Hope you’re doing good.

Over the last weeks I published two blog posts regarding upgrade DB Server Operating System from RHEL 7 to RHEL 8.

In my last post, I am describing how to fix a bug/issue that is related to ACFS/AFD kernel modules if you are upgrading to RHEL 8.9.

You can see my last blog post here:

ACFS-9459: ADVM/ACFS is not supported on this OS version: ‘4.18.0-513.11.1.el8_9.x86_64’ after upgrade to RHEL/OEL 8.9

When you are applying a patch using opatchauto, one of the tasks that are executed/managed by opatchauto is the postpatch step. This is the last step when you’re applying a patch.

During postpatch, we received an error as exhibited below:

—————————Patching Failed———————————

Command execution failed during patching in home: /oracle/app/193/grid, host: dbl01.

Command failed: bash -c “LD_LIBRARY_PATH=/oracle/app/193/grid/lib:/oracle/app/193/grid/srvm/lib:/oracle/app/193/grid/lib:/oracle/app/193/grid/lib:/oracle/app/193/grid/srvm/lib /oracle/app/193/grid/perl/bin/perl -I/oracle/app/193/grid/perl/lib -I/oracle/app/193/grid/opatchautocfg/db/dbtmp/bootstrap_dbl01/patchwork/crs/install -I/oracle/app/193/grid/opatchautocfg/db/dbtmp/bootstrap_dbl01/patchwork/xag /oracle/app/193/grid/opatchautocfg/db/dbtmp/bootstrap_dbl01/patchwork/crs/install/rootcrs.pl -postpatch”

Command failure output:

Using configuration parameter file: /oracle/app/193/grid/opatchautocfg/db/dbtmp/bootstrap_dbl01/patchwork/crs/install/crsconfig_params

The log of current session can be found at:

/oracle/app/grid/crsdata/dbl01/crsconfig/crs_postpatch_apply_inplace_dbl01_2024-03-13_02-13-54AM.log

2024/03/13 02:14:07 CLSRSC-180: An error occurred while executing the command ‘/bin/rpm -qf /sbin/init’

After fixing the cause of failure Run opatchauto resume

If you noticed, the error is highlighted here:

2024/03/13 02:14:07 CLSRSC-180: An error occurred while executing the command ‘/bin/rpm -qf /sbin/init’

Let’s check the mentioned log, the default path for the log is $ORACLE_BASE/crsdata/<hostname>/crsconfig/crs_postpatch_apply_inplace_<hostname>_YYYY-MM-DD_HH-MI-SSAM.log.

So, in my case, log is /oracle/app/grid/crsdata/dbl01/crsconfig/crs_postpatch_apply_inplace_dbl01_2024-03-13_02-13-54AM.log.

Let’s check the log:

2024-03-13 02:14:07: Command output:

> error: rpmdb: BDB0113 Thread/process 4954/140508599121984 failed: BDB1507 Thread died in Berkeley DB library

> error: db5 error(-30973) from dbenv->failchk: BDB0087 DB_RUNRECOVERY: Fatal error, run database recovery

> error: cannot open Packages index using db5 – (-30973)

> error: cannot open Packages database in /var/lib/rpm

> error: rpmdb: BDB0113 Thread/process 4954/140508599121984 failed: BDB1507 Thread died in Berkeley DB library

> error: db5 error(-30973) from dbenv->failchk: BDB0087 DB_RUNRECOVERY: Fatal error, run database recovery

> error: cannot open Packages database in /var/lib/rpm

> error: rpmdb: BDB0113 Thread/process 4954/140508599121984 failed: BDB1507 Thread died in Berkeley DB library

> error: db5 error(-30973) from dbenv->failchk: BDB0087 DB_RUNRECOVERY: Fatal error, run database recovery

> error: cannot open Packages database in /var/lib/rpm

> file /sbin/init is not owned by any package

OK, we are able to see that issue is with Berkeley DB library, and if you remember, the error happened during execution of the following command:

/bin/rpm -qf /sbin/init

Let’s see what happens when we execute this command:

Humm, this means that Operating System RPM DB has been corrupted. The root-cause for this can be due several reasons and I’ll not focus on find the root cause here, as, this looks like is an isolated issue and it is not a recurring issue.

So, to fix this issue, we need to rebuild RPM DB. As root user, execute the following commands:

Now, let’s check if the command has been executed successfully, the expected output is 0:

Let’s check if we’ll receive the error again:

OK, sounds good!

If you noticed the patching log, you must now execute opatchauto resume to continue with the remaining steps.

This issue is also mentioned in the following support note:

Postpatch Command Fails on Grid Infrastructure Home Due To rpm Database Corruption At The OS Level (Doc ID 2365433.1)

Hope this helps!

Peace,

Vinicius