This service will be undergoing maintenance at 00:00 UTC, 2016-08-01. It is expected to last about 1 hours
Bug 956531 - EC2 Implement support for copying amis
EC2 Implement support for copying amis
Status: CLOSED CURRENTRELEASE
Product: Red Hat Enterprise Linux 6
Classification: Red Hat
Component: ec2-images (Show other bugs)
6.4
x86_64 Linux
unspecified Severity unspecified
: rc
: ---
Assigned To: wes hayutin
mkovacik
: EC2
Depends On:
Blocks: 921116 1068715
  Show dependency treegraph
 
Reported: 2013-04-25 03:32 EDT by mkovacik
Modified: 2015-02-13 09:53 EST (History)
13 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
: 1068715 (view as bug list)
Environment:
Last Closed: 2013-12-05 11:06:18 EST
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:


Attachments (Terms of Use)
cloud init log (209.36 KB, text/plain)
2013-04-25 03:34 EDT, mkovacik
no flags Details
boot log (7.06 KB, text/plain)
2013-04-25 03:35 EDT, mkovacik
no flags Details
/var/log/messages (74.96 KB, text/plain)
2013-04-25 03:35 EDT, mkovacik
no flags Details
rc.local.patch (285 bytes, patch)
2013-04-25 11:43 EDT, mkovacik
no flags Details | Diff


External Trackers
Tracker ID Priority Status Summary Last Updated
Red Hat Knowledge Base (Solution) 369523 None None None Never

  None (edit)
Description mkovacik 2013-04-25 03:32:08 EDT
Description of problem
 Cloud-init prevents one from utilizing the new feature of AWS EC2: copying amis between regions. Especially, second and further copies of a snapshot-based ami will fail to be reached via ssh due to an error in /etc/ssh/sshd_config file.

Version-Release number of selected component (if applicable):
 Cloud-init v. 0.7.1 

How reproducible:
 Always

Steps to Reproduce:
 1. instantiate an ami#0
 2. having stopped the instance, create a new ami#1 out of it
 3. copy the ami to a new region -> ami#2
 4. instantiate the copied ami#2
 5. having stopped the instance of ami#2, create a new ami#3 out of it
 6. copy ami#3 to another region -> ami#4
 7. instantiate ami#4; sshd not starting anymore -> instance not reachable

  
Actual results:
 instances of copied amis can't be reached because of an sshd config issue originated in cloud init

Expected results:
 copied amis instances should be always reachable via ssh


Additional info:
Apr 24 10:16:23 ip-10-121-14-26 [CLOUDINIT] util.py[WARNING]: Restarting of the ssh daemon failed
Apr 24 10:16:23 ip-10-121-14-26 [CLOUDINIT] util.py[DEBUG]: Restarting of the ssh daemon failed#012Traceback (most recent call last):#012  File "/usr/lib/python2.6/site-packages/cloudinit/config/cc_set_passwords.py", line 142, in handle#012    util.subp(cmd)#012  File "/usr/lib/python2.6/site-packages/cloudinit/util.py", line 1429, in subp#012    cmd=args)#012ProcessExecutionError: Unexpected error while running command.#012Command: ['service', 'sshd', 'restart']#012Exit code: 255#012Reason: -#012Stdout: 'Stopping sshd: [FAILED]\r\nStarting sshd: [FAILED]\r\n'#012Stderr: '/etc/ssh/sshd_config line 139: Bad yes/without-password/forced-commands-only/no argument: without-passwordUseDNS\r\n'

[root@ip-10-252-160-224 log]# sed -e '/^#/d' -e '/^$/d' /etc/ssh/sshd_config                                                                                                                                                                 
Protocol 2
SyslogFacility AUTHPRIV
PasswordAuthentication no
ChallengeResponseAuthentication no
GSSAPIAuthentication yes
GSSAPICleanupCredentials yes
UsePAM yes
AcceptEnv LANG LC_CTYPE LC_NUMERIC LC_TIME LC_COLLATE LC_MONETARY LC_MESSAGES
AcceptEnv LC_PAPER LC_NAME LC_ADDRESS LC_TELEPHONE LC_MEASUREMENT
AcceptEnv LC_IDENTIFICATION LC_ALL LANGUAGE
AcceptEnv XMODIFIERS
X11Forwarding yes
Subsystem sftp  /usr/libexec/openssh/sftp-server
PermitRootLogin without-passwordUseDNS no
PermitRootLogin without-passwordUseDNS no
PermitRootLogin without-password
[root@ip-10-252-160-224 log]#
Comment 1 mkovacik 2013-04-25 03:34:38 EDT
Created attachment 739695 [details]
cloud init log
Comment 2 mkovacik 2013-04-25 03:35:02 EDT
Created attachment 739696 [details]
boot log
Comment 3 mkovacik 2013-04-25 03:35:36 EDT
Created attachment 739697 [details]
/var/log/messages
Comment 4 mkovacik 2013-04-25 04:02:53 EDT
Feature description: http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/CopyingAMIs.html
Comment 5 Vitaly Kuznetsov 2013-04-25 07:24:31 EDT
I believe we see reincarnation of https://bugzilla.redhat.com/show_bug.cgi?id=923996 
We have /etc/rc.d/rc.local script in images which screws up sshd config by adding multiple "UseDNS no
PermitRootLogin without-password" sections. Cloud-init does its own manipulations with the config and (possibly) removes last newline character.
Comment 6 mkovacik 2013-04-25 10:47:59 EDT
Vitaly is right; removing the /etc/rc.local's portion tampering with sshd_config the file is no more a mess. Still, cloud init doesn't care about last newline character when modifying /etc/ssh/sshd_config:

[root@ip-10-151-56-248 ~]# sed -e '/^#/d' -e '/^$/d' /etc/ssh/sshd_config 
Protocol 2
SyslogFacility AUTHPRIV
PasswordAuthentication no
ChallengeResponseAuthentication no
GSSAPIAuthentication yes
GSSAPICleanupCredentials yes
UsePAM yes
AcceptEnv LANG LC_CTYPE LC_NUMERIC LC_TIME LC_COLLATE LC_MONETARY LC_MESSAGES
AcceptEnv LC_PAPER LC_NAME LC_ADDRESS LC_TELEPHONE LC_MEASUREMENT
AcceptEnv LC_IDENTIFICATION LC_ALL LANGUAGE
AcceptEnv XMODIFIERS
X11Forwarding yes
Subsystem sftp  /usr/libexec/openssh/sftp-server
PermitRootLogin without-password[root@ip-10-151-56-248 ~]#
Comment 7 mkovacik 2013-04-25 11:43:47 EDT
Created attachment 739964 [details]
rc.local.patch

This patch of /etc/rc.local solves the issue
Comment 8 wes hayutin 2013-04-29 10:14:14 EDT
agreed.. this needs to be fixed.. for 6.5 and hopefully a fix for 6.4 or respin
Comment 9 Steven Hardy 2013-04-29 11:04:44 EDT
Ref comment #5, are we agreed that this is an issue with the image rc.local, and not a bug caused by cloud-init?

If so propose we reassign this to ec2-images?
Comment 10 wes hayutin 2013-04-29 14:53:57 EDT
yup! switching to ec2-images
Comment 13 wes hayutin 2013-07-12 08:51:29 EDT
spin-kickstarts git
238e0401763ef8721e6633472f453b7865a3e562

cloude
commit dca168ea330976e6c8b9a2b15d1f3faa69a0b92a
Comment 14 Vitaly Kuznetsov 2013-07-12 09:00:07 EDT
Verified with ami-044d326d (stage), rh-amazon-rhui-client-2.2.85-1
Comment 16 Ina Panova 2013-10-17 06:44:48 EDT
This fix needs to be propagated in all amis in 6.5. Lubos, will you do it please?
Comment 17 Mark McCorkle 2013-11-04 20:14:43 EST
Looks like the amazon RHEL servers have this patch applied incorrectly.  After my initial reboot, I see the following in my /etc/sshd_config.  Notice the lack of a newline in front of "UseDNS no" which breaks the next line of PermitRootLogin:

# Example of overriding settings on a per-user basis
#Match User anoncvs
#	X11Forwarding no
#	AllowTcpForwarding no
#	ForceCommand cvs serverUseDNS no
PermitRootLogin without-passwordUseDNS no
PermitRootLogin without-password

This is a mostly stock RHEL 6.4 (Santiago) server with just apache httpd booted from the AWS / RHEL AMI.  

I've fixed this by booting the affected machines, commenting out the patch in rc.local (why do we have it running at EVERY boot, and only AFTER all of the other init scripts run) and then fixing the etc/sshd_config by hand.

Let me know if you need anything else to reproduce this.

Note You need to log in before you can comment on or make changes to this bug.