Incident Report: OpenSSL Upgrade Issue on CentOS Stream 9

Incident Summary:

During the process of upgrading OpenSSL from version 3.2.2 to version 3.4.0 on a CentOS Stream 9 system, the upgrade could not be performed using the yum upgrade command. The decision was made to install OpenSSL from source code. After successfully downloading, extracting, configuring, compiling, and installing the new version, errors were encountered when testing the installation using the openssl version command.

Error Messages:

# openssl -v
openssl: /lib64/libssl.so.3: version `OPENSSL_3.4.0' not found (required by openssl)
openssl: /lib64/libcrypto.so.3: version `OPENSSL_3.3.0' not found (required by openssl)
openssl: /lib64/libcrypto.so.3: version `OPENSSL_3.4.0' not found (required by openssl)

Root Cause Analysis:

1. Incorrect LDLIBRARYPATH: The environment variable LD_LIBRARY_PATH was incorrectly set to /usr/local/openssl-3.4/lib instead of /usr/local/openssl-3.4/lib64.
2. Path Conflicts: The old OpenSSL version was still visible in the system path, causing conflicts.

Steps Taken to Resolve the Issue:

1. Identified the Incorrect LDLIBRARYPATH:

   • The LD_LIBRARY_PATH was initially set as follows:

     export LD_LIBRARY_PATH=/usr/local/openssl-3.4/lib:$LD_LIBRARY_PATH
   

   • This was corrected to:

     export LD_LIBRARY_PATH=/usr/local/openssl-3.4/lib64:$LD_LIBRARY_PATH
   

2. Resolved Path Conflicts:

   • The system path was checked and found to contain multiple entries for the new OpenSSL version:

     # echo $PATH
     /usr/local/openssl-3.4/bin:/root/.local/bin:/root/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/usr/local/openssl-3.4/bin:/usr/local/openssl-3.4/bin
   

   • The /etc/bashrc file was edited to correctly set the path:

     vi /etc/bashrc
   

   • The path was rearranged as follows:

     PATH=/usr/local/openssl-3.4/bin:$PATH
     LD_LIBRARY_PATH=/usr/local/openssl-3.4/lib64:$LD_LIBRARY_PATH
     export PATH LD_LIBRARY_PATH
   

   • The changes were applied by sourcing the bashrc file:

     # source /etc/bashrc
   

3. Verified the New Path:

   • The new path was confirmed to be:

     # echo $PATH
     /root/.local/bin:/root/bin:/usr/local/openssl-3.4/bin:/usr/local/openssl-3.4/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin
   

4. Tested the OpenSSL Installation:

   • The openssl version command was run to verify the installation: bash # openssl version OpenSSL 3.4.0 22 Oct 2024 (Library: OpenSSL 3.4.0 22 Oct 2024)

Key Points and Additional Details:

• Environment Variables: Ensure that environment variables such as PATH and LD_LIBRARY_PATH are correctly set to avoid conflicts and errors.
• Path Management: Regularly check and clean up the system path to prevent multiple entries and conflicts.
• Documentation: Document each step of the installation and troubleshooting process for future reference and knowledge sharing.

By following these steps, the OpenSSL upgrade issue was successfully resolved, and the new version was correctly installed and verified.