Check for booting issues after XEN-based Rocky Linux 8 update

Prev Next

Available in VPC

After the update for all packages using dnf(yum) update on an Xen-based Rocky Linux 8 VM, the system fails to boot and enters the GRUB prompt upon rebooting.

Note

This issue occurs only in a XEN-based Rocky Linux 8 environment, especially when the latest grub2 package of the 8.10 version is installed.
However, the issue doesn't occur in a KVM hypervisor-based Rocky Linux 8 and Rocky Linux 9 environments.

Caution

NAVER Cloud Platform doesn't recommend performing updates for all OS packages.
Before a kernel patch or package update, create and thoroughly validate My server image to apply the kernel patch or package update to actual services.
Any issues caused by OS updates fall under the responsibility of the customer, and technical support for these issues is not provided.
This guide is not meant to offer provide technical support, but it aims to help you troubleshoot issues yourself.

How to reproduce the symptoms of an issue

  1. Remotely access the server.

  2. Update all OS packages.

    Note

    On Rocky Linux 8, dnf update is a command totally different from apt update of Ubuntu OS.
    apt update used on Ubuntu is a command to update the OS package repository's meta information. You must run the command before you run apt related commands.
    dnf (or yum) update is a command to update all Rocky Linux OS packages. Note that if you run the command, restoration (rollback) is not available.

    [root@xen-rockylinux810 ~]# dnf update
    Rocky Linux 8 - AppStream                                                                                                                                          97 MB/s |  18 MB     00:00
    Rocky Linux 8 - BaseOS                                                                                                                                             80 MB/s |  24 MB     00:00
    Rocky Linux 8 - Extras                                                                                                                                            1.6 MB/s |  15 kB     00:00
    Dependencies resolved.
    ==================================================================================================================================================================================================
     Package                                               Architecture                    Version                                                           Repository                          Size
    ==================================================================================================================================================================================================
    Installing:
     kernel                                                x86_64                          4.18.0-553.53.1.el8_10                                            baseos                              10 M
     kernel-core                                           x86_64                          4.18.0-553.53.1.el8_10                                            baseos                              43 M
     kernel-devel                                          x86_64                          4.18.0-553.53.1.el8_10                                            baseos                              24 M
     kernel-modules                                        x86_64                          4.18.0-553.53.1.el8_10                                            baseos                              36 M
     kernel-modules-extra                                  x86_64                          4.18.0-553.53.1.el8_10                                            baseos                              11 M
    Upgrading:
    ... Omitted ...
     grub2-common                                          noarch                          1:2.02-165.el8_10.rocky.0.1                                       baseos                             897 k
     grub2-pc                                              x86_64                          1:2.02-165.el8_10.rocky.0.1                                       baseos                              47 k
     grub2-pc-modules                                      noarch                          1:2.02-165.el8_10.rocky.0.1                                       baseos                             929 k
     grub2-tools                                           x86_64                          1:2.02-165.el8_10.rocky.0.1                                       baseos                             2.0 M
     grub2-tools-extra                                     x86_64                          1:2.02-165.el8_10.rocky.0.1                                       baseos                             1.1 M
     grub2-tools-minimal                                   x86_64                          1:2.02-165.el8_10.rocky.0.1                                       baseos                             215 k
    
    ... ... (rest omitted)
    
  3. After update completion, reboot the VM.

    [root@xen-rockylinux810 ~]# sync;reboot
    
  4. If the server does not boot properly, check the server access console on the MC console.
    xen_rockylinux8_01

    xen_rockylinux8_02

How to mitigate GRUB prompt drops

  1. Do not run the commands to update all OS packages (dnf update).
  2. If the grub2 package is reinstalled, reinstall GRUB using the grub2-install command.
    [root@xen-rockylinux810 ~]# grub2-install /dev/xvda
    

How to boot OS if the system is already in the GRUB prompt

  1. Check if the file system is normal.
    xen_rockylinux8_03

  2. Inspect the kernel version you want to boot and related files.
    xen_rockylinux8_04

  3. Attempt to boot by entering the following GRUB commands:

    Note

    The kernel used in the following example is 4.18.0-553.53.1.el8_10.x86_64.
    You can use a bootable kernel other than the version.

    grub> set root=(hd0,msdos1)
    grub> linux /vmlinuz-4.18.0-553.53.1.el8_10.x86_64 root=/dev/xvda2 ro
    grub> initrd /initramfs-4.18.0-553.53.1.el8_10.x86_64.img
    grub> boot
    
  4. If booting completes properly, log in to the server and reinstall the GRUB file.

    # grub2-install /dev/xvda
    

    To return to the In operation status, click the Force stop button on the MC console to end the VM and then restart it.

  5. If booting fails despite the above method, create a snapshot of the corresponding VM, mount the snapshot to another VM, and copy the data.
    When the server cannot be recovered, see How to recover data through snapshot.