RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 688608 - /etc/rc.local is executed after ssh started, making some changes useless
Summary: /etc/rc.local is executed after ssh started, making some changes useless
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux 6
Classification: Red Hat
Component: releng
Version: 6.1
Hardware: All
OS: Linux
urgent
urgent
Target Milestone: rc
: ---
Assignee: Jay Greguske
QA Contact: wes hayutin
URL:
Whiteboard:
Depends On:
Blocks: 700583
TreeView+ depends on / blocked
 
Reported: 2011-03-17 13:43 UTC by Pierre Carrier
Modified: 2018-11-14 13:50 UTC (History)
7 users (show)

Fixed In Version: GA
Doc Type: Bug Fix
Doc Text:
Clone Of:
: 700583 (view as bug list)
Environment:
Last Closed: 2011-06-09 19:07:07 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)
Proposed patch (2.15 KB, application/octet-stream)
2011-03-17 13:44 UTC, Pierre Carrier
no flags Details


Links
System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHEA-2011:0540 0 normal SHIPPED_LIVE redhat-release enhancement update for Red Hat Enterprise Linux 6.1 2011-05-18 17:44:50 UTC

Description Pierre Carrier 2011-03-17 13:43:27 UTC
Description of problem:
- Changes to the ssh config and deployment of the EC2 ssh key is done through /etc/rc.local
- rc.local is ran in /etc/rc.d/rc3.d/S99local
- That's after /etc/rc.d/rc3.d/S55sshd

Conclusion:
- ssh client return after the connection was refused during boot time
- Then sshd starts and they prompt for a password even if the private key is provided
- Then the key is made available and following connection attempts succeed directly
- The sshd_config changes are not taken into account by sshd

Version-Release number of selected component (if applicable):
- http://cvs.devel.redhat.com/cgi-bin/cvsweb.cgi/utility-scripts/cloud/kickstarts/RHEL-5.5-starter-ec2.ks?rev=1.14;content-type=text%2Fplain;cvsroot=RH-RelEng;f=h
- Apparently that would be the currently released RHEL5.5 image

How reproducible:
100%

Steps to Reproduce:
1. Boot an instance
2. Try to ssh every few seconds, with -i $key
  
Actual results:
- Failure for the first attempts
- Then password prompt
- Finally, success without a password prompt

Expected results:
- Failure for the first attempts
- Then success without a password prompt

Additional info:
See find a patch for this ks file attached. Please note it was not tested.

Comment 1 Pierre Carrier 2011-03-17 13:44:27 UTC
Created attachment 486015 [details]
Proposed patch

Comment 3 wes hayutin 2011-03-21 15:09:04 UTC
Adding Jay to the bug

Comment 4 Jay Greguske 2011-03-28 16:15:41 UTC
Patch applied to the 5.5, 5.6, 6.0, and 6.1 kickstart files.

For 6.1 beta the fixes should be evident in snapshot 2.

Comment 5 wes hayutin 2011-03-29 00:35:07 UTC
Jay, whats a good way to test for the patch?

Comment 6 Jay Greguske 2011-03-29 12:32:00 UTC
Tough question. As soon as the instance comes back as "running" from AWS and not "pending" you have to slam it with ssh connections hoping for a password prompt even if you supply it with the private key.

I don't think you can test this easily from within the instance.

Comment 7 Jay Greguske 2011-04-01 19:26:22 UTC
After some review I'm not a fan of this patch. There is still a race condition: if sshd completes before the ec2config "service" completes you'll still have this problem. If you put '-o "PreferredAuthentications publickey"' in the ssh command you won't get the password prompt.

Comment 8 Mike Bonnet 2011-04-01 19:33:14 UTC
Also, your comment about the sshd_config changes not taking effect isn't correct.  sshd_config and firstboot changes are being executed during the kickstart %post, which happens at image creation time.  So when the image boots for the first time, those changes are already there.  Moving those changes into a service script is not correct.

Comment 9 Pierre Carrier 2011-04-03 21:24:11 UTC
(In reply to comment #7)
> After some review I'm not a fan of this patch. There is still a race condition:
> if sshd completes before the ec2config "service" completes you'll still have
> this problem.

A race condition between initscripts? I *really* don't see how this could happen...

ec2config is S54 and sshd is S55.
AFAIK we do not parallelize anything.

> If you put '-o "PreferredAuthentications publickey"' in the ssh
> command you won't get the password prompt.

Yes, this workaround was already provided to the original bug reporter.

(In reply to comment #8)
> Also, your comment about the sshd_config changes not taking effect isn't
> correct.  sshd_config and firstboot changes are being executed during the
> kickstart %post, which happens at image creation time.  So when the image boots
> for the first time, those changes are already there.  Moving those changes into
> a service script is not correct.

I didn't know about initscripts being executed then, in which case you can effectively drop the remark.

However, this makes me wonder: if this script is executed during the kickstart and not the first boot, how come we have this ssh key issue?

-- 
Pierre

Comment 10 Mike Bonnet 2011-04-04 14:21:39 UTC
(In reply to comment #9)
> (In reply to comment #8)
> I didn't know about initscripts being executed then, in which case you can
> effectively drop the remark.

The initscripts are not being executed in %post, only the script in the kickstart.

Take a look at the original kickstart linked above.  Everything between %post and %end is a script that gets executed at image creation time.  It edits grub.conf, sets up fstab, appends to sshd_config, etc.  All of these changes are made to the image when it's created, before it has ever been booted.  The script also appends *another* script to /etc/rc.local.  That second script is executed at boot time, and is responsible downloading the ssh key and installing it to /root/.ssh/authorized_keys.

> However, this makes me wonder: if this script is executed during the kickstart
> and not the first boot, how come we have this ssh key issue?

The script that downloads the ssh keys needs to be executed at boot time, not image creation time, because the key is coming from the metadata associated with that particular instance.  Each instance booted from the same disk image will have its own metadata, including its own ssh key.  EC2 users can set the ssh key associated with an instance via the web UI, and that user-specific key will be downloaded and installed into the instance when it is first booted, by the script in /etc/rc.local.  If we installed a ssh key at image creation time, then everyone would need to use the same ssh private key to access all Red Hat images, which is obviously not appropriate.

All that being said, an answer to this issue is to set "PasswordAuthentication no" in sshd_config at image creation time.

Comment 11 Pierre Carrier 2011-04-04 20:02:04 UTC
(In reply to comment #10)
> Take a look at the original kickstart linked above.  Everything between %post
> and %end is a script that gets executed at image creation time.  It edits
> grub.conf, sets up fstab, appends to sshd_config, etc.  All of these changes
> are made to the image when it's created, before it has ever been booted.  The
> script also appends *another* script to /etc/rc.local.  That second script is
> executed at boot time, and is responsible downloading the ssh key and
> installing it to /root/.ssh/authorized_keys.

Sorry, I got completely confused by my patch: I didn't realise it was merging the scriptlets with the initscript. We definitely should only keep the ssh key operation in the initscript.

> The script that downloads the ssh keys needs to be executed at boot time, not
> image creation time, because the key is coming from the metadata associated
> with that particular instance. [...]

Back to my original understanding on this :)

> All that being said, an answer to this issue is to set "PasswordAuthentication
> no" in sshd_config at image creation time.

It would have more effects than simply grabbing the key before starting ssh.
Sure "PasswordAuthentication no" is a good policy, but can we change it?
And given that some admins might want to have "PasswordAuthentication yes", shouldn't we also fix the boot order anyway?

-- 
Pierre

Comment 12 Mike Bonnet 2011-04-04 20:15:06 UTC
(In reply to comment #11)
> (In reply to comment #10)
> It would have more effects than simply grabbing the key before starting ssh.
> Sure "PasswordAuthentication no" is a good policy, but can we change it?
> And given that some admins might want to have "PasswordAuthentication yes",
> shouldn't we also fix the boot order anyway?

This really only affects how you access the instance the very first time it boots.  At that point there are no local users created, and root has no password set, so ssh keys are the only possible way to access the instance.  Setting "PasswordAuthentication no" would prevent ssh clients from hanging at password prompts, which I believe was the original complaint.  Once the admin is able to log in as root, they can change the ssh config in whatever way they'd like.  Of course, this change to the default would have to be documented to avoid surprising admins who create new users and expect them to be able to login with their password.

If you'd really rather pursue the initscript approach I'm not going to object, but it does seem more complicated, for only minor improvement in functionality.

Comment 13 Jay Greguske 2011-04-20 16:13:52 UTC
I'm going to work towards the "PasswordAuthentication no" approach. Any objections? If a customer wants it to be yes, they can log in and change it in their instance easily. Our images do not have valid passwords in the first place anyway.

Comment 14 Pierre Carrier 2011-04-22 14:01:28 UTC
No objection from me.

I would still prefer putting the keys in place before we start sshd, makes more sense to me not to offer a service until it's actually usable, but I've given my opinion more times than enough so I'll keep quiet now.


Thanks!

-- 
Pierre Carrier

Comment 16 RHEL Program Management 2011-04-29 06:00:15 UTC
Since RHEL 6.1 External Beta has begun, and this bug remains
unresolved, it has been rejected as it is not proposed as
exception or blocker.

Red Hat invites you to ask your support representative to
propose this request, if appropriate and relevant, in the
next release of Red Hat Enterprise Linux.

Comment 18 wes hayutin 2011-05-25 15:20:42 UTC
Connection to ec2-122-248-207-169.ap-southeast-1.compute.amazonaws.com closed.
[whayutin@minidoe ~]$ ssh -i ~/.ec2/WESHAYUTIN/asia-cloudekey.pem root.compute.amazonaws.com
Last login: Wed May 25 10:53:06 2011 from 99.39.212.236
[root@ip-10-130-43-97 ~]# cat /etc/ssh/sshd_config  | grep -i password
# To disable tunneled clear text passwords, change to no here!
#PasswordAuthentication yes
#PermitEmptyPasswords no
PasswordAuthentication no
# Change to no to disable s/key passwords
# PasswordAuthentication.  Depending on your PAM configuration,
# the setting of "PermitRootLogin without-password".
# PAM authentication, then enable this but set PasswordAuthentication
PermitRootLogin without-password
[root@ip-10-130-43-97 ~]# 



adding test to test scripts..

Comment 19 errata-xmlrpc 2011-06-09 19:07:07 UTC
An advisory has been issued which should help the problem
described in this bug report. This report is therefore being
closed with a resolution of ERRATA. For more information
on therefore solution and/or where to find the updated files,
please follow the link below. You may reopen this bug report
if the solution does not work for you.

http://rhn.redhat.com/errata/RHEA-2011-0540.html


Note You need to log in before you can comment on or make changes to this bug.