Bug 956531 - EC2 Implement support for copying amis
Summary: EC2 Implement support for copying amis
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: Red Hat Enterprise Linux 6
Classification: Red Hat
Component: ec2-images
Version: 6.4
Hardware: x86_64
OS: Linux
unspecified
unspecified
Target Milestone: rc
: ---
Assignee: wes hayutin
QA Contact: mkovacik
URL:
Whiteboard:
Depends On:
Blocks: 921116 1068715
TreeView+ depends on / blocked
 
Reported: 2013-04-25 07:32 UTC by mkovacik
Modified: 2019-06-13 07:53 UTC (History)
13 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Clone Of:
: 1068715 (view as bug list)
Environment:
Last Closed: 2013-12-05 16:06:18 UTC
Target Upstream Version:


Attachments (Terms of Use)
cloud init log (209.36 KB, text/plain)
2013-04-25 07:34 UTC, mkovacik
no flags Details
boot log (7.06 KB, text/plain)
2013-04-25 07:35 UTC, mkovacik
no flags Details
/var/log/messages (74.96 KB, text/plain)
2013-04-25 07:35 UTC, mkovacik
no flags Details
rc.local.patch (285 bytes, patch)
2013-04-25 15:43 UTC, mkovacik
no flags Details | Diff


Links
System ID Priority Status Summary Last Updated
Red Hat Knowledge Base (Solution) 369523 None None None Never

Description mkovacik 2013-04-25 07:32:08 UTC
Description of problem
 Cloud-init prevents one from utilizing the new feature of AWS EC2: copying amis between regions. Especially, second and further copies of a snapshot-based ami will fail to be reached via ssh due to an error in /etc/ssh/sshd_config file.

Version-Release number of selected component (if applicable):
 Cloud-init v. 0.7.1 

How reproducible:
 Always

Steps to Reproduce:
 1. instantiate an ami#0
 2. having stopped the instance, create a new ami#1 out of it
 3. copy the ami to a new region -> ami#2
 4. instantiate the copied ami#2
 5. having stopped the instance of ami#2, create a new ami#3 out of it
 6. copy ami#3 to another region -> ami#4
 7. instantiate ami#4; sshd not starting anymore -> instance not reachable

  
Actual results:
 instances of copied amis can't be reached because of an sshd config issue originated in cloud init

Expected results:
 copied amis instances should be always reachable via ssh


Additional info:
Apr 24 10:16:23 ip-10-121-14-26 [CLOUDINIT] util.py[WARNING]: Restarting of the ssh daemon failed
Apr 24 10:16:23 ip-10-121-14-26 [CLOUDINIT] util.py[DEBUG]: Restarting of the ssh daemon failed#012Traceback (most recent call last):#012  File "/usr/lib/python2.6/site-packages/cloudinit/config/cc_set_passwords.py", line 142, in handle#012    util.subp(cmd)#012  File "/usr/lib/python2.6/site-packages/cloudinit/util.py", line 1429, in subp#012    cmd=args)#012ProcessExecutionError: Unexpected error while running command.#012Command: ['service', 'sshd', 'restart']#012Exit code: 255#012Reason: -#012Stdout: 'Stopping sshd: [FAILED]\r\nStarting sshd: [FAILED]\r\n'#012Stderr: '/etc/ssh/sshd_config line 139: Bad yes/without-password/forced-commands-only/no argument: without-passwordUseDNS\r\n'

[root@ip-10-252-160-224 log]# sed -e '/^#/d' -e '/^$/d' /etc/ssh/sshd_config                                                                                                                                                                 
Protocol 2
SyslogFacility AUTHPRIV
PasswordAuthentication no
ChallengeResponseAuthentication no
GSSAPIAuthentication yes
GSSAPICleanupCredentials yes
UsePAM yes
AcceptEnv LANG LC_CTYPE LC_NUMERIC LC_TIME LC_COLLATE LC_MONETARY LC_MESSAGES
AcceptEnv LC_PAPER LC_NAME LC_ADDRESS LC_TELEPHONE LC_MEASUREMENT
AcceptEnv LC_IDENTIFICATION LC_ALL LANGUAGE
AcceptEnv XMODIFIERS
X11Forwarding yes
Subsystem sftp  /usr/libexec/openssh/sftp-server
PermitRootLogin without-passwordUseDNS no
PermitRootLogin without-passwordUseDNS no
PermitRootLogin without-password
[root@ip-10-252-160-224 log]#

Comment 1 mkovacik 2013-04-25 07:34:38 UTC
Created attachment 739695 [details]
cloud init log

Comment 2 mkovacik 2013-04-25 07:35:02 UTC
Created attachment 739696 [details]
boot log

Comment 3 mkovacik 2013-04-25 07:35:36 UTC
Created attachment 739697 [details]
/var/log/messages

Comment 4 mkovacik 2013-04-25 08:02:53 UTC
Feature description: http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/CopyingAMIs.html

Comment 5 Vitaly Kuznetsov 2013-04-25 11:24:31 UTC
I believe we see reincarnation of https://bugzilla.redhat.com/show_bug.cgi?id=923996 
We have /etc/rc.d/rc.local script in images which screws up sshd config by adding multiple "UseDNS no
PermitRootLogin without-password" sections. Cloud-init does its own manipulations with the config and (possibly) removes last newline character.

Comment 6 mkovacik 2013-04-25 14:47:59 UTC
Vitaly is right; removing the /etc/rc.local's portion tampering with sshd_config the file is no more a mess. Still, cloud init doesn't care about last newline character when modifying /etc/ssh/sshd_config:

[root@ip-10-151-56-248 ~]# sed -e '/^#/d' -e '/^$/d' /etc/ssh/sshd_config 
Protocol 2
SyslogFacility AUTHPRIV
PasswordAuthentication no
ChallengeResponseAuthentication no
GSSAPIAuthentication yes
GSSAPICleanupCredentials yes
UsePAM yes
AcceptEnv LANG LC_CTYPE LC_NUMERIC LC_TIME LC_COLLATE LC_MONETARY LC_MESSAGES
AcceptEnv LC_PAPER LC_NAME LC_ADDRESS LC_TELEPHONE LC_MEASUREMENT
AcceptEnv LC_IDENTIFICATION LC_ALL LANGUAGE
AcceptEnv XMODIFIERS
X11Forwarding yes
Subsystem sftp  /usr/libexec/openssh/sftp-server
PermitRootLogin without-password[root@ip-10-151-56-248 ~]#

Comment 7 mkovacik 2013-04-25 15:43:47 UTC
Created attachment 739964 [details]
rc.local.patch

This patch of /etc/rc.local solves the issue

Comment 8 wes hayutin 2013-04-29 14:14:14 UTC
agreed.. this needs to be fixed.. for 6.5 and hopefully a fix for 6.4 or respin

Comment 9 Steven Hardy 2013-04-29 15:04:44 UTC
Ref comment #5, are we agreed that this is an issue with the image rc.local, and not a bug caused by cloud-init?

If so propose we reassign this to ec2-images?

Comment 10 wes hayutin 2013-04-29 18:53:57 UTC
yup! switching to ec2-images

Comment 13 wes hayutin 2013-07-12 12:51:29 UTC
spin-kickstarts git
238e0401763ef8721e6633472f453b7865a3e562

cloude
commit dca168ea330976e6c8b9a2b15d1f3faa69a0b92a

Comment 14 Vitaly Kuznetsov 2013-07-12 13:00:07 UTC
Verified with ami-044d326d (stage), rh-amazon-rhui-client-2.2.85-1

Comment 16 Ina Panova 2013-10-17 10:44:48 UTC
This fix needs to be propagated in all amis in 6.5. Lubos, will you do it please?

Comment 17 Mark McCorkle 2013-11-05 01:14:43 UTC
Looks like the amazon RHEL servers have this patch applied incorrectly.  After my initial reboot, I see the following in my /etc/sshd_config.  Notice the lack of a newline in front of "UseDNS no" which breaks the next line of PermitRootLogin:

# Example of overriding settings on a per-user basis
#Match User anoncvs
#	X11Forwarding no
#	AllowTcpForwarding no
#	ForceCommand cvs serverUseDNS no
PermitRootLogin without-passwordUseDNS no
PermitRootLogin without-password

This is a mostly stock RHEL 6.4 (Santiago) server with just apache httpd booted from the AWS / RHEL AMI.  

I've fixed this by booting the affected machines, commenting out the patch in rc.local (why do we have it running at EVERY boot, and only AFTER all of the other init scripts run) and then fixing the etc/sshd_config by hand.

Let me know if you need anything else to reproduce this.


Note You need to log in before you can comment on or make changes to this bug.