Note: This bug is displayed in read-only format because
the product is no longer active in Red Hat Bugzilla.
RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Description of problem
Cloud-init prevents one from utilizing the new feature of AWS EC2: copying amis between regions. Especially, second and further copies of a snapshot-based ami will fail to be reached via ssh due to an error in /etc/ssh/sshd_config file.
Version-Release number of selected component (if applicable):
Cloud-init v. 0.7.1
How reproducible:
Always
Steps to Reproduce:
1. instantiate an ami#0
2. having stopped the instance, create a new ami#1 out of it
3. copy the ami to a new region -> ami#2
4. instantiate the copied ami#2
5. having stopped the instance of ami#2, create a new ami#3 out of it
6. copy ami#3 to another region -> ami#4
7. instantiate ami#4; sshd not starting anymore -> instance not reachable
Actual results:
instances of copied amis can't be reached because of an sshd config issue originated in cloud init
Expected results:
copied amis instances should be always reachable via ssh
Additional info:
Apr 24 10:16:23 ip-10-121-14-26 [CLOUDINIT] util.py[WARNING]: Restarting of the ssh daemon failed
Apr 24 10:16:23 ip-10-121-14-26 [CLOUDINIT] util.py[DEBUG]: Restarting of the ssh daemon failed#012Traceback (most recent call last):#012 File "/usr/lib/python2.6/site-packages/cloudinit/config/cc_set_passwords.py", line 142, in handle#012 util.subp(cmd)#012 File "/usr/lib/python2.6/site-packages/cloudinit/util.py", line 1429, in subp#012 cmd=args)#012ProcessExecutionError: Unexpected error while running command.#012Command: ['service', 'sshd', 'restart']#012Exit code: 255#012Reason: -#012Stdout: 'Stopping sshd: [FAILED]\r\nStarting sshd: [FAILED]\r\n'#012Stderr: '/etc/ssh/sshd_config line 139: Bad yes/without-password/forced-commands-only/no argument: without-passwordUseDNS\r\n'
[root@ip-10-252-160-224 log]# sed -e '/^#/d' -e '/^$/d' /etc/ssh/sshd_config
Protocol 2
SyslogFacility AUTHPRIV
PasswordAuthentication no
ChallengeResponseAuthentication no
GSSAPIAuthentication yes
GSSAPICleanupCredentials yes
UsePAM yes
AcceptEnv LANG LC_CTYPE LC_NUMERIC LC_TIME LC_COLLATE LC_MONETARY LC_MESSAGES
AcceptEnv LC_PAPER LC_NAME LC_ADDRESS LC_TELEPHONE LC_MEASUREMENT
AcceptEnv LC_IDENTIFICATION LC_ALL LANGUAGE
AcceptEnv XMODIFIERS
X11Forwarding yes
Subsystem sftp /usr/libexec/openssh/sftp-server
PermitRootLogin without-passwordUseDNS no
PermitRootLogin without-passwordUseDNS no
PermitRootLogin without-password
[root@ip-10-252-160-224 log]#
I believe we see reincarnation of https://bugzilla.redhat.com/show_bug.cgi?id=923996
We have /etc/rc.d/rc.local script in images which screws up sshd config by adding multiple "UseDNS no
PermitRootLogin without-password" sections. Cloud-init does its own manipulations with the config and (possibly) removes last newline character.
Vitaly is right; removing the /etc/rc.local's portion tampering with sshd_config the file is no more a mess. Still, cloud init doesn't care about last newline character when modifying /etc/ssh/sshd_config:
[root@ip-10-151-56-248 ~]# sed -e '/^#/d' -e '/^$/d' /etc/ssh/sshd_config
Protocol 2
SyslogFacility AUTHPRIV
PasswordAuthentication no
ChallengeResponseAuthentication no
GSSAPIAuthentication yes
GSSAPICleanupCredentials yes
UsePAM yes
AcceptEnv LANG LC_CTYPE LC_NUMERIC LC_TIME LC_COLLATE LC_MONETARY LC_MESSAGES
AcceptEnv LC_PAPER LC_NAME LC_ADDRESS LC_TELEPHONE LC_MEASUREMENT
AcceptEnv LC_IDENTIFICATION LC_ALL LANGUAGE
AcceptEnv XMODIFIERS
X11Forwarding yes
Subsystem sftp /usr/libexec/openssh/sftp-server
PermitRootLogin without-password[root@ip-10-151-56-248 ~]#
Ref comment #5, are we agreed that this is an issue with the image rc.local, and not a bug caused by cloud-init?
If so propose we reassign this to ec2-images?
Looks like the amazon RHEL servers have this patch applied incorrectly. After my initial reboot, I see the following in my /etc/sshd_config. Notice the lack of a newline in front of "UseDNS no" which breaks the next line of PermitRootLogin:
# Example of overriding settings on a per-user basis
#Match User anoncvs
# X11Forwarding no
# AllowTcpForwarding no
# ForceCommand cvs serverUseDNS no
PermitRootLogin without-passwordUseDNS no
PermitRootLogin without-password
This is a mostly stock RHEL 6.4 (Santiago) server with just apache httpd booted from the AWS / RHEL AMI.
I've fixed this by booting the affected machines, commenting out the patch in rc.local (why do we have it running at EVERY boot, and only AFTER all of the other init scripts run) and then fixing the etc/sshd_config by hand.
Let me know if you need anything else to reproduce this.