Bug 1619994

I am unable to reproduce the same condition on RHEL 8 after multiple attempts. Though a side-by-side comparison with a RHEL 7 system shows that it is reproducible after a few iterations. That being said, when Red Hat shipped 7.7 on Aug 6, 2019 Red Hat Enterprise Linux 7 entered Maintenance Support 1 Phase.

    https://access.redhat.com/support/policy/updates/errata#Maintenance_Support_1_Phase

That means only "Critical and Important Security errata advisories (RHSAs) and Urgent Priority Bug Fix errata advisories (RHBAs) may be released". This BZ does not appear to meet Maintenance Support 1 Phase criteria so is being closed NEXTRELEASE. If this is critical for your environment please open a case in the Red Hat Customer Portal, https://access.redhat.com, provide a thorough business justification and ask that the BZ be re-opened for consideration in the next minor release.

I would like to reopen this.

The justification is the admin must have a way to know if the reboot was due to a crash or clean reboot happened.

Digging there, it looks like a solution is to remove the "Requisite" dependency in systemd-update-utmp-runlevel.service:

/etc/systemd/system/systemd-update-utmp-runlevel.service:
-------- 8< ---------------- 8< ---------------- 8< ---------------- 8< --------
[Unit]
Description=Update UTMP about System Runlevel Changes
Documentation=man:systemd-update-utmp.service(8) man:utmp(5)
DefaultDependencies=no
RequiresMountsFor=/var/log/wtmp
Conflicts=shutdown.target
#Requisite=systemd-update-utmp.service
After=systemd-update-utmp.service
After=runlevel1.target runlevel2.target runlevel3.target runlevel4.target runlevel5.target
Before=shutdown.target

[Service]
Type=oneshot
ExecStart=/usr/lib/systemd/systemd-update-utmp runlevel
-------- 8< ---------------- 8< ---------------- 8< ---------------- 8< --------

This prevents the "systemd-update-utmp/verify-active" job to run and create the conflict.

Testing this, after 48 reboots, the service *all* ran successfully:
-------- 8< ---------------- 8< ---------------- 8< ---------------- 8< --------
# ausearch -m SYSTEM_BOOT | grep -c SYSTEM_BOOT
48
# ausearch -m SYSTEM_SHUTDOWN | grep -c SYSTEM_SHUTDOWN
48
-------- 8< ---------------- 8< ---------------- 8< ---------------- 8< --------


Reproducer:
-------- 8< ---------------- 8< ---------------- 8< ---------------- 8< --------
for i in $(seq 1 50); do
  echo "Reboot $i"
  while ! timeout 5 ssh -o ConnectTimeout=5s vm-shutdown7 true; do
    sleep 5
  done
  ssh vm-shutdown7 reboot
  sleep 10
done
-------- 8< ---------------- 8< ---------------- 8< ---------------- 8< --------

Hi Team,

I found the same issue while working on case 02691482

Hence attaching the same.

I did not have a assured reproducer, but I was able to hit
the same problem many a times while testing.

[1] when shutdown event was noted.


1127.8.2.el7 ---> 327.el7


# last -x reboot shutdown|head
reboot   system boot  3.10.0-327.el7.x Sat Jul 18 17:31 - 17:31  (00:00)    
shutdown system down  3.10.0-1127.8.2. Sat Jul 18 17:31 - 17:31  (00:00)    

-------------------
[  OK  ] Stopped target System Initialization.
[  OK  ] Stopped target Local Encrypted Volumes.
[  OK  ] Stopped Read and set NIS domainname from /etc/sysconfig/network.
         Stopping Update UTMP about System Boot/Shutdown...
         Stopping Load/Save Random Seed...
[  OK  ] Stopped Apply Kernel Variables.
[  OK  ] Stopped Setup Virtual Console.
[  OK  ] Stopped Load/Save Random Seed.
[  OK  ] Stopped Update UTMP about System Boot/Shutdown.                       <=====---------------------<-----
         Stopping Security Auditing Service...
-------------------

As you can see above the service is stopped and hence 'shutdown' event was logged

[2] When shutdown event did *NOT* get logged


1127.8.2.el7 ---> 1062.1.2.el7

# last -x reboot shutdown|head
reboot   system boot  3.10.0-1062.1.2. Sat Jul 18 17:37 - 17:37  (00:00)    
reboot   system boot  3.10.0-1127.8.2. Sat Jul 18 17:36 - 17:37  (00:01)    

We do not see shutdown entry here

---------------
[  OK  ] Stopped target System Initialization.
[  OK  ] Stopped target Swap.
         Deactivating swap /dev/disk/by-uuid...0-4879-4e19-be36-1f4e3b112f90...
         Stopping Security Auditing Service...
[  OK  ] Stopped Setup Virtual Console.
[  OK  ] Stopped Read and set NIS domainname from /etc/sysconfig/network.
         Stopping Load/Save Random Seed...
[  OK  ] Stopped Apply Kernel Variables.
[  OK  ] Stopped target Local Encrypted Volumes.
---------------

NOTE:
======

-> Do not think that this is issue within kernel versions.

-> I used 'reboot' command only during both the times.

Red Hat Enterprise Linux 7 shipped it's final minor release on September 29th, 2020. 7.9 was the last minor releases scheduled for RHEL 7.
From intial triage it does not appear the remaining Bugzillas meet the inclusion criteria for Maintenance Phase 2 and will now be closed. 

From the RHEL life cycle page:
https://access.redhat.com/support/policy/updates/errata#Maintenance_Support_2_Phase
"During Maintenance Support 2 Phase for Red Hat Enterprise Linux version 7,Red Hat defined Critical and Important impact Security Advisories (RHSAs) and selected (at Red Hat discretion) Urgent Priority Bug Fix Advisories (RHBAs) may be released as they become available."

If this BZ was closed in error and meets the above criteria please re-open it flag for 7.9.z, provide suitable business and technical justifications, and follow the process for Accelerated Fixes:
https://source.redhat.com/groups/public/pnt-cxno/pnt_customer_experience_and_operations_wiki/support_delivery_accelerated_fix_release_handbook  

Feature Requests can re-opened and moved to RHEL 8 if the desired functionality is not already present in the product. 

Please reach out to the applicable Product Experience Engineer[0] if you have any questions or concerns.  

[0] https://bugzilla.redhat.com/page.cgi?id=agile_component_mapping.html&product=Red+Hat+Enterprise+Linux+7