Note: This bug is displayed in read-only format because
the product is no longer active in Red Hat Bugzilla.
RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Created attachment 1922984[details]
cloud-init.log
Description of problem:
Start an instance running RHEL-9.1 on aws t4g.large system, after reboot system via sysrq 'b', the system is not accessible via ssh and cloudinit service failed to start.
$ cat cloud-init.service.log
× cloud-init.service - Initial cloud-init job (metadata service crawler)
Loaded: loaded (/usr/lib/systemd/system/cloud-init.service; enabled; vendor preset: disabled)
Active: failed (Result: exit-code) since Tue 2022-11-08 02:33:59 UTC; 19min ago
Process: 703 ExecStart=/usr/bin/cloud-init init (code=exited, status=1/FAILURE)
Main PID: 703 (code=exited, status=1/FAILURE)
CPU: 424ms
Nov 08 02:33:59 ip-10-22-1-50.us-west-2.compute.internal cloud-init[801]: return _default_decoder.decode(s)
Nov 08 02:33:59 ip-10-22-1-50.us-west-2.compute.internal cloud-init[801]: File "/usr/lib64/python3.9/json/decoder.py", line 337, in decode
Nov 08 02:33:59 ip-10-22-1-50.us-west-2.compute.internal cloud-init[801]: obj, end = self.raw_decode(s, idx=_w(s, 0).end())
Nov 08 02:33:59 ip-10-22-1-50.us-west-2.compute.internal cloud-init[801]: File "/usr/lib64/python3.9/json/decoder.py", line 355, in raw_decode
Nov 08 02:33:59 ip-10-22-1-50.us-west-2.compute.internal cloud-init[801]: raise JSONDecodeError("Expecting value", s, err.value) from None
Nov 08 02:33:59 ip-10-22-1-50.us-west-2.compute.internal cloud-init[801]: json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
Nov 08 02:33:59 ip-10-22-1-50.us-west-2.compute.internal cloud-init[801]: ------------------------------------------------------------
Nov 08 02:33:59 ip-10-22-1-50.us-west-2.compute.internal systemd[1]: cloud-init.service: Main process exited, code=exited, status=1/FAILURE
Nov 08 02:33:59 ip-10-22-1-50.us-west-2.compute.internal systemd[1]: cloud-init.service: Failed with result 'exit-code'.
Nov 08 02:33:59 ip-10-22-1-50.us-west-2.compute.internal systemd[1]: Failed to start Initial cloud-init job (metadata service crawler).
$ cat journal.log|grep sshd
Nov 08 02:33:57 ip-10-22-1-50.us-west-2.compute.internal systemd[1]: Created slice Slice /system/sshd-keygen.
Nov 08 02:33:57 ip-10-22-1-50.us-west-2.compute.internal systemd[1]: Reached target sshd-keygen.target.
Nov 08 02:33:59 ip-10-22-1-50.us-west-2.compute.internal sshd[949]: sshd: no hostkeys available -- exiting.
Nov 08 02:33:59 ip-10-22-1-50.us-west-2.compute.internal systemd[1]: sshd.service: Main process exited, code=exited, status=1/FAILURE
Nov 08 02:33:59 ip-10-22-1-50.us-west-2.compute.internal systemd[1]: sshd.service: Failed with result 'exit-code'.
Nov 08 02:34:41 ip-10-22-1-50.us-west-2.compute.internal systemd[1]: sshd.service: Scheduled restart job, restart counter is at 1.
RHEL Version:
RHEL-9.1(5.14.0-162.6.1.el9_1.x86_64)
How reproducible:
50%
Steps to Reproduce:
1. Create an aws t4g.large instance using RHEL-9.1.0_HVM-20221101
2. Trigger system reboot('echo b > /proc/sysrq-trigger & echo b > /proc/sysrq-trigger')
3. Repeat step1~2 if cannot reproduce it.
4. option, reproduce in auto
$ os-tests --user ec2-user --keyfile /home/virtqe_s1.pem --platform_profile /home/aws.yaml -p test_reboot_simultaneous
Actual results:
cannot access system via ssh after boot up
Expected results:
system can boot up and access normally
Additional info:
- N/A
(In reply to Emanuele Giuseppe Esposito from comment #14)
> Frank can correct me, but it seems to affect 9.1 so also 9.2/8.7/8.8.
Yes, we can trigger this exception by manually empty "/var/lib/cloud/data/set-hostname".
Comment 19Emanuele Giuseppe Esposito
2023-01-17 08:37:40 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.
For information on the advisory (cloud-init bug fix and enhancement update), and where to find the updated
files, follow the link below.
If the solution does not work for you, open a new bug report.
https://access.redhat.com/errata/RHBA-2023:2183
Created attachment 1922984 [details] cloud-init.log Description of problem: Start an instance running RHEL-9.1 on aws t4g.large system, after reboot system via sysrq 'b', the system is not accessible via ssh and cloudinit service failed to start. $ cat cloud-init.service.log × cloud-init.service - Initial cloud-init job (metadata service crawler) Loaded: loaded (/usr/lib/systemd/system/cloud-init.service; enabled; vendor preset: disabled) Active: failed (Result: exit-code) since Tue 2022-11-08 02:33:59 UTC; 19min ago Process: 703 ExecStart=/usr/bin/cloud-init init (code=exited, status=1/FAILURE) Main PID: 703 (code=exited, status=1/FAILURE) CPU: 424ms Nov 08 02:33:59 ip-10-22-1-50.us-west-2.compute.internal cloud-init[801]: return _default_decoder.decode(s) Nov 08 02:33:59 ip-10-22-1-50.us-west-2.compute.internal cloud-init[801]: File "/usr/lib64/python3.9/json/decoder.py", line 337, in decode Nov 08 02:33:59 ip-10-22-1-50.us-west-2.compute.internal cloud-init[801]: obj, end = self.raw_decode(s, idx=_w(s, 0).end()) Nov 08 02:33:59 ip-10-22-1-50.us-west-2.compute.internal cloud-init[801]: File "/usr/lib64/python3.9/json/decoder.py", line 355, in raw_decode Nov 08 02:33:59 ip-10-22-1-50.us-west-2.compute.internal cloud-init[801]: raise JSONDecodeError("Expecting value", s, err.value) from None Nov 08 02:33:59 ip-10-22-1-50.us-west-2.compute.internal cloud-init[801]: json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0) Nov 08 02:33:59 ip-10-22-1-50.us-west-2.compute.internal cloud-init[801]: ------------------------------------------------------------ Nov 08 02:33:59 ip-10-22-1-50.us-west-2.compute.internal systemd[1]: cloud-init.service: Main process exited, code=exited, status=1/FAILURE Nov 08 02:33:59 ip-10-22-1-50.us-west-2.compute.internal systemd[1]: cloud-init.service: Failed with result 'exit-code'. Nov 08 02:33:59 ip-10-22-1-50.us-west-2.compute.internal systemd[1]: Failed to start Initial cloud-init job (metadata service crawler). $ cat journal.log|grep sshd Nov 08 02:33:57 ip-10-22-1-50.us-west-2.compute.internal systemd[1]: Created slice Slice /system/sshd-keygen. Nov 08 02:33:57 ip-10-22-1-50.us-west-2.compute.internal systemd[1]: Reached target sshd-keygen.target. Nov 08 02:33:59 ip-10-22-1-50.us-west-2.compute.internal sshd[949]: sshd: no hostkeys available -- exiting. Nov 08 02:33:59 ip-10-22-1-50.us-west-2.compute.internal systemd[1]: sshd.service: Main process exited, code=exited, status=1/FAILURE Nov 08 02:33:59 ip-10-22-1-50.us-west-2.compute.internal systemd[1]: sshd.service: Failed with result 'exit-code'. Nov 08 02:34:41 ip-10-22-1-50.us-west-2.compute.internal systemd[1]: sshd.service: Scheduled restart job, restart counter is at 1. RHEL Version: RHEL-9.1(5.14.0-162.6.1.el9_1.x86_64) How reproducible: 50% Steps to Reproduce: 1. Create an aws t4g.large instance using RHEL-9.1.0_HVM-20221101 2. Trigger system reboot('echo b > /proc/sysrq-trigger & echo b > /proc/sysrq-trigger') 3. Repeat step1~2 if cannot reproduce it. 4. option, reproduce in auto $ os-tests --user ec2-user --keyfile /home/virtqe_s1.pem --platform_profile /home/aws.yaml -p test_reboot_simultaneous Actual results: cannot access system via ssh after boot up Expected results: system can boot up and access normally Additional info: - N/A