Bug 956531

Summary:

EC2 Implement support for copying amis

Product:

Red Hat Enterprise Linux 6

Reporter:

mkovacik

Component:

ec2-images

Assignee:

wes hayutin <whayutin>

Status:

CLOSED CURRENTRELEASE

QA Contact:

mkovacik

Severity:

unspecified

Docs Contact:

Priority:

unspecified

Version:

6.4

CC:

dgregor, greg.martyn, ipanova, jboutaud, jcircle, jgreguske, jmatthew, jraju, lkocman, mmccorkle, tsanders, vkuznets, whayutin

Target Milestone:

Keywords:

EC2

Target Release:

---

Hardware:

x86_64

OS:

Linux

Whiteboard:

Fixed In Version:

Doc Type:

Bug Fix

Doc Text:

Story Points:

---

Clone Of:

Clones:

1068715 (view as bug list)

Environment:

Last Closed:

2013-12-05 16:06:18 UTC

Type:

Bug

Regression:

---

Mount Type:

---

Documentation:

---

CRM:

Verified Versions:

Category:

---

oVirt Team:

---

RHEL 7.3 requirements from Atomic Host:

Cloudforms Team:

---

Target Upstream Version:

Embargoed:

Bug Depends On:

Bug Blocks:

921116, 1068715

Attachments:

Description	Flags
cloud init log	none
boot log	none
/var/log/messages	none
rc.local.patch	none

Description mkovacik 2013-04-25 07:32:08 UTC

Description of problem
 Cloud-init prevents one from utilizing the new feature of AWS EC2: copying amis between regions. Especially, second and further copies of a snapshot-based ami will fail to be reached via ssh due to an error in /etc/ssh/sshd_config file.

Version-Release number of selected component (if applicable):
 Cloud-init v. 0.7.1 

How reproducible:
 Always

Steps to Reproduce:
 1. instantiate an ami#0
 2. having stopped the instance, create a new ami#1 out of it
 3. copy the ami to a new region -> ami#2
 4. instantiate the copied ami#2
 5. having stopped the instance of ami#2, create a new ami#3 out of it
 6. copy ami#3 to another region -> ami#4
 7. instantiate ami#4; sshd not starting anymore -> instance not reachable

  
Actual results:
 instances of copied amis can't be reached because of an sshd config issue originated in cloud init

Expected results:
 copied amis instances should be always reachable via ssh


Additional info:
Apr 24 10:16:23 ip-10-121-14-26 [CLOUDINIT] util.py[WARNING]: Restarting of the ssh daemon failed
Apr 24 10:16:23 ip-10-121-14-26 [CLOUDINIT] util.py[DEBUG]: Restarting of the ssh daemon failed#012Traceback (most recent call last):#012  File "/usr/lib/python2.6/site-packages/cloudinit/config/cc_set_passwords.py", line 142, in handle#012    util.subp(cmd)#012  File "/usr/lib/python2.6/site-packages/cloudinit/util.py", line 1429, in subp#012    cmd=args)#012ProcessExecutionError: Unexpected error while running command.#012Command: ['service', 'sshd', 'restart']#012Exit code: 255#012Reason: -#012Stdout: 'Stopping sshd: [FAILED]\r\nStarting sshd: [FAILED]\r\n'#012Stderr: '/etc/ssh/sshd_config line 139: Bad yes/without-password/forced-commands-only/no argument: without-passwordUseDNS\r\n'

[root@ip-10-252-160-224 log]# sed -e '/^#/d' -e '/^$/d' /etc/ssh/sshd_config                                                                                                                                                                 
Protocol 2
SyslogFacility AUTHPRIV
PasswordAuthentication no
ChallengeResponseAuthentication no
GSSAPIAuthentication yes
GSSAPICleanupCredentials yes
UsePAM yes
AcceptEnv LANG LC_CTYPE LC_NUMERIC LC_TIME LC_COLLATE LC_MONETARY LC_MESSAGES
AcceptEnv LC_PAPER LC_NAME LC_ADDRESS LC_TELEPHONE LC_MEASUREMENT
AcceptEnv LC_IDENTIFICATION LC_ALL LANGUAGE
AcceptEnv XMODIFIERS
X11Forwarding yes
Subsystem sftp  /usr/libexec/openssh/sftp-server
PermitRootLogin without-passwordUseDNS no
PermitRootLogin without-passwordUseDNS no
PermitRootLogin without-password
[root@ip-10-252-160-224 log]#

Comment 1 mkovacik 2013-04-25 07:34:38 UTC

Created attachment 739695 [details]
cloud init log

Comment 2 mkovacik 2013-04-25 07:35:02 UTC

Created attachment 739696 [details]
boot log

Comment 3 mkovacik 2013-04-25 07:35:36 UTC

Created attachment 739697 [details]
/var/log/messages

Comment 4 mkovacik 2013-04-25 08:02:53 UTC

Feature description: http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/CopyingAMIs.html

Comment 5 Vitaly Kuznetsov 2013-04-25 11:24:31 UTC

I believe we see reincarnation of https://bugzilla.redhat.com/show_bug.cgi?id=923996 
We have /etc/rc.d/rc.local script in images which screws up sshd config by adding multiple "UseDNS no
PermitRootLogin without-password" sections. Cloud-init does its own manipulations with the config and (possibly) removes last newline character.

Comment 6 mkovacik 2013-04-25 14:47:59 UTC

Vitaly is right; removing the /etc/rc.local's portion tampering with sshd_config the file is no more a mess. Still, cloud init doesn't care about last newline character when modifying /etc/ssh/sshd_config:

[root@ip-10-151-56-248 ~]# sed -e '/^#/d' -e '/^$/d' /etc/ssh/sshd_config 
Protocol 2
SyslogFacility AUTHPRIV
PasswordAuthentication no
ChallengeResponseAuthentication no
GSSAPIAuthentication yes
GSSAPICleanupCredentials yes
UsePAM yes
AcceptEnv LANG LC_CTYPE LC_NUMERIC LC_TIME LC_COLLATE LC_MONETARY LC_MESSAGES
AcceptEnv LC_PAPER LC_NAME LC_ADDRESS LC_TELEPHONE LC_MEASUREMENT
AcceptEnv LC_IDENTIFICATION LC_ALL LANGUAGE
AcceptEnv XMODIFIERS
X11Forwarding yes
Subsystem sftp  /usr/libexec/openssh/sftp-server
PermitRootLogin without-password[root@ip-10-151-56-248 ~]#

Comment 7 mkovacik 2013-04-25 15:43:47 UTC

Created attachment 739964 [details]
rc.local.patch

This patch of /etc/rc.local solves the issue

Comment 8 wes hayutin 2013-04-29 14:14:14 UTC

agreed.. this needs to be fixed.. for 6.5 and hopefully a fix for 6.4 or respin

Comment 9 Steven Hardy 2013-04-29 15:04:44 UTC

Ref comment #5, are we agreed that this is an issue with the image rc.local, and not a bug caused by cloud-init?

If so propose we reassign this to ec2-images?

Comment 10 wes hayutin 2013-04-29 18:53:57 UTC

yup! switching to ec2-images

Comment 13 wes hayutin 2013-07-12 12:51:29 UTC

spin-kickstarts git
238e0401763ef8721e6633472f453b7865a3e562

cloude
commit dca168ea330976e6c8b9a2b15d1f3faa69a0b92a

Comment 14 Vitaly Kuznetsov 2013-07-12 13:00:07 UTC

Verified with ami-044d326d (stage), rh-amazon-rhui-client-2.2.85-1

Comment 16 Ina Panova 2013-10-17 10:44:48 UTC

This fix needs to be propagated in all amis in 6.5. Lubos, will you do it please?

Comment 17 Mark McCorkle 2013-11-05 01:14:43 UTC

Looks like the amazon RHEL servers have this patch applied incorrectly.  After my initial reboot, I see the following in my /etc/sshd_config.  Notice the lack of a newline in front of "UseDNS no" which breaks the next line of PermitRootLogin:

# Example of overriding settings on a per-user basis
#Match User anoncvs
#	X11Forwarding no
#	AllowTcpForwarding no
#	ForceCommand cvs serverUseDNS no
PermitRootLogin without-passwordUseDNS no
PermitRootLogin without-password

This is a mostly stock RHEL 6.4 (Santiago) server with just apache httpd booted from the AWS / RHEL AMI.  

I've fixed this by booting the affected machines, commenting out the patch in rc.local (why do we have it running at EVERY boot, and only AFTER all of the other init scripts run) and then fixing the etc/sshd_config by hand.

Let me know if you need anything else to reproduce this.