RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 956531 - EC2 Implement support for copying amis
Summary: EC2 Implement support for copying amis
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: Red Hat Enterprise Linux 6
Classification: Red Hat
Component: ec2-images
Version: 6.4
Hardware: x86_64
OS: Linux
unspecified
unspecified
Target Milestone: rc
: ---
Assignee: wes hayutin
QA Contact: mkovacik
URL:
Whiteboard:
Depends On:
Blocks: 921116 1068715
TreeView+ depends on / blocked
 
Reported: 2013-04-25 07:32 UTC by mkovacik
Modified: 2019-06-13 07:53 UTC (History)
13 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Clone Of:
: 1068715 (view as bug list)
Environment:
Last Closed: 2013-12-05 16:06:18 UTC
Target Upstream Version:
Embargoed:

Attachments (Terms of Use)
cloud init log (209.36 KB, text/plain)
2013-04-25 07:34 UTC, mkovacik 		no flags Details
boot log (7.06 KB, text/plain)
2013-04-25 07:35 UTC, mkovacik 		no flags Details
/var/log/messages (74.96 KB, text/plain)
2013-04-25 07:35 UTC, mkovacik 		no flags Details
rc.local.patch (285 bytes, patch)
2013-04-25 15:43 UTC, mkovacik 		no flags Details | Diff
View All


Links
System ID Private Priority Status Summary Last Updated
Red Hat Knowledge Base (Solution) 369523 0 None None None Never

Description mkovacik 2013-04-25 07:32:08 UTC 
Description of problem
 Cloud-init prevents one from utilizing the new feature of AWS EC2: copying amis between regions. Especially, second and further copies of a snapshot-based ami will fail to be reached via ssh due to an error in /etc/ssh/sshd_config file.

Version-Release number of selected component (if applicable):
 Cloud-init v. 0.7.1 

How reproducible:
 Always

Steps to Reproduce:
 1. instantiate an ami#0
 2. having stopped the instance, create a new ami#1 out of it
 3. copy the ami to a new region -> ami#2
 4. instantiate the copied ami#2
 5. having stopped the instance of ami#2, create a new ami#3 out of it
 6. copy ami#3 to another region -> ami#4
 7. instantiate ami#4; sshd not starting anymore -> instance not reachable

  
Actual results:
 instances of copied amis can't be reached because of an sshd config issue originated in cloud init

Expected results:
 copied amis instances should be always reachable via ssh


Additional info:
Apr 24 10:16:23 ip-10-121-14-26 [CLOUDINIT] util.py[WARNING]: Restarting of the ssh daemon failed
Apr 24 10:16:23 ip-10-121-14-26 [CLOUDINIT] util.py[DEBUG]: Restarting of the ssh daemon failed#012Traceback (most recent call last):#012  File "/usr/lib/python2.6/site-packages/cloudinit/config/cc_set_passwords.py", line 142, in handle#012    util.subp(cmd)#012  File "/usr/lib/python2.6/site-packages/cloudinit/util.py", line 1429, in subp#012    cmd=args)#012ProcessExecutionError: Unexpected error while running command.#012Command: ['service', 'sshd', 'restart']#012Exit code: 255#012Reason: -#012Stdout: 'Stopping sshd: [FAILED]\r\nStarting sshd: [FAILED]\r\n'#012Stderr: '/etc/ssh/sshd_config line 139: Bad yes/without-password/forced-commands-only/no argument: without-passwordUseDNS\r\n'

[root@ip-10-252-160-224 log]# sed -e '/^#/d' -e '/^$/d' /etc/ssh/sshd_config                                                                                                                                                                 
Protocol 2
SyslogFacility AUTHPRIV
PasswordAuthentication no
ChallengeResponseAuthentication no
GSSAPIAuthentication yes
GSSAPICleanupCredentials yes
UsePAM yes
AcceptEnv LANG LC_CTYPE LC_NUMERIC LC_TIME LC_COLLATE LC_MONETARY LC_MESSAGES
AcceptEnv LC_PAPER LC_NAME LC_ADDRESS LC_TELEPHONE LC_MEASUREMENT
AcceptEnv LC_IDENTIFICATION LC_ALL LANGUAGE
AcceptEnv XMODIFIERS
X11Forwarding yes
Subsystem sftp  /usr/libexec/openssh/sftp-server
PermitRootLogin without-passwordUseDNS no
PermitRootLogin without-passwordUseDNS no
PermitRootLogin without-password
[root@ip-10-252-160-224 log]#
Comment 1 mkovacik 2013-04-25 07:34:38 UTC 
Created attachment 739695 [details]
cloud init log
Comment 2 mkovacik 2013-04-25 07:35:02 UTC 
Created attachment 739696 [details]
boot log
Comment 3 mkovacik 2013-04-25 07:35:36 UTC 
Created attachment 739697 [details]
/var/log/messages
Comment 4 mkovacik 2013-04-25 08:02:53 UTC 
Feature description: http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/CopyingAMIs.html
Comment 5 Vitaly Kuznetsov 2013-04-25 11:24:31 UTC 
I believe we see reincarnation of https://bugzilla.redhat.com/show_bug.cgi?id=923996 
We have /etc/rc.d/rc.local script in images which screws up sshd config by adding multiple "UseDNS no
PermitRootLogin without-password" sections. Cloud-init does its own manipulations with the config and (possibly) removes last newline character.
Comment 6 mkovacik 2013-04-25 14:47:59 UTC 
Vitaly is right; removing the /etc/rc.local's portion tampering with sshd_config the file is no more a mess. Still, cloud init doesn't care about last newline character when modifying /etc/ssh/sshd_config:

[root@ip-10-151-56-248 ~]# sed -e '/^#/d' -e '/^$/d' /etc/ssh/sshd_config 
Protocol 2
SyslogFacility AUTHPRIV
PasswordAuthentication no
ChallengeResponseAuthentication no
GSSAPIAuthentication yes
GSSAPICleanupCredentials yes
UsePAM yes
AcceptEnv LANG LC_CTYPE LC_NUMERIC LC_TIME LC_COLLATE LC_MONETARY LC_MESSAGES
AcceptEnv LC_PAPER LC_NAME LC_ADDRESS LC_TELEPHONE LC_MEASUREMENT
AcceptEnv LC_IDENTIFICATION LC_ALL LANGUAGE
AcceptEnv XMODIFIERS
X11Forwarding yes
Subsystem sftp  /usr/libexec/openssh/sftp-server
PermitRootLogin without-password[root@ip-10-151-56-248 ~]#
Comment 7 mkovacik 2013-04-25 15:43:47 UTC 
Created attachment 739964 [details]
rc.local.patch

This patch of /etc/rc.local solves the issue
Comment 8 wes hayutin 2013-04-29 14:14:14 UTC 
agreed.. this needs to be fixed.. for 6.5 and hopefully a fix for 6.4 or respin
Comment 9 Steven Hardy 2013-04-29 15:04:44 UTC 
Ref comment #5, are we agreed that this is an issue with the image rc.local, and not a bug caused by cloud-init?

If so propose we reassign this to ec2-images?
Comment 10 wes hayutin 2013-04-29 18:53:57 UTC 
yup! switching to ec2-images
Comment 13 wes hayutin 2013-07-12 12:51:29 UTC 
spin-kickstarts git
238e0401763ef8721e6633472f453b7865a3e562

cloude
commit dca168ea330976e6c8b9a2b15d1f3faa69a0b92a
Comment 14 Vitaly Kuznetsov 2013-07-12 13:00:07 UTC 
Verified with ami-044d326d (stage), rh-amazon-rhui-client-2.2.85-1
Comment 16 Ina Panova 2013-10-17 10:44:48 UTC 
This fix needs to be propagated in all amis in 6.5. Lubos, will you do it please?
Comment 17 Mark McCorkle 2013-11-05 01:14:43 UTC 
Looks like the amazon RHEL servers have this patch applied incorrectly.  After my initial reboot, I see the following in my /etc/sshd_config.  Notice the lack of a newline in front of "UseDNS no" which breaks the next line of PermitRootLogin:

# Example of overriding settings on a per-user basis
#Match User anoncvs
#	X11Forwarding no
#	AllowTcpForwarding no
#	ForceCommand cvs serverUseDNS no
PermitRootLogin without-passwordUseDNS no
PermitRootLogin without-password

This is a mostly stock RHEL 6.4 (Santiago) server with just apache httpd booted from the AWS / RHEL AMI.  

I've fixed this by booting the affected machines, commenting out the patch in rc.local (why do we have it running at EVERY boot, and only AFTER all of the other init scripts run) and then fixing the etc/sshd_config by hand.

Let me know if you need anything else to reproduce this.
Note You need to log in before you can comment on or make changes to this bug.