Bug 513363 - Not suitable error message when ssh public key changed
Not suitable error message when ssh public key changed
Status: CLOSED WONTFIX
Product: Red Hat Enterprise Linux 6
Classification: Red Hat
Component: libvirt (Show other bugs)
6.2
All Linux
low Severity medium
: rc
: 6.3
Assigned To: Peter Krempa
Virtualization Bugs
:
Depends On: 749873
Blocks:
  Show dependency treegraph
 
Reported: 2009-07-23 06:32 EDT by Mark Xie
Modified: 2013-01-29 10:47 EST (History)
19 users (show)

See Also:
Fixed In Version:
Doc Type: Enhancement
Doc Text:
Feature: Libvirt now supports a internal ssh connection mode using the libssh2 library. Reason: The use of externally spawned "ssh" client processes didn't allow for proper error reporting and didn't allow to use password authentication to a remote host when using graphical tools. Result (if any): The libssh2 connection driver avoids the external process and uses libvirt's authentication callbacks that when used with graphical management tools allow to use all kind of authentication and report sensible errors if the connection fails.
Story Points: ---
Clone Of:
Environment:
Last Closed: 2013-01-29 10:40:09 EST
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description Mark Xie 2009-07-23 06:32:06 EDT
Description of problem:
Connect to a remote host by ssh tunnel, when the public key of the remote ssh daemon have changed, virt-manager will popup a error message tells the end user to check libvirtd.

Version-Release number of selected component (if applicable):
virt-manager-0.6.1-7.el5
libvirt-0.6.3-15.el5
openssh-server-4.3p2-36.el5
openssh-clients-4.3p2-36.el5
openssh-askpass-4.3p2-36.el5
openssh-4.3p2-36.el5


How reproducible:
100%

Steps to Reproduce:
1.Launch virt-manager, create a connect to a remote machine by ssh tunnel("File -> Add Connection -> Remote tunnel over SSH")
2.Connect to the remote host (choose the connect -> right click it -> click "Connect" )
3.disconnect the remote host
4.change ssh public key on the remote machine(remove "/etc/ssh/ssh_host_rsa_key" and "/etc/ssh/ssh_host_rsa_key.pub", then restart sshd daemon)
5.Try to step2 again.
6.remove "~/.ssh/known_hosts" of the machine virt-manager located.
7.try step2 again
  
Actual results:
It will success at step2 and step7.
At step 4, there is a pop up dialogue with error message:

Libvirt URI is: qemu+ssh://root@dhcp-66-70-58/system
Verify that:
 - The 'libvirtd' daemon has been started
Details:
Unable to open connection to hypervisor URI 'qemu+ssh://root@dhcp-66-70-58/system':
libvirt.libvirtError cannot recv data: Connection reset by peer
None


Expected results:
It's better to remind end user it's an issue of ssh key changed, like ssh client says:


Additional info:

1, when same conditions happens to ssh client, it will give out warning like:

@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@    WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED!     @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
Someone could be eavesdropping on you right now (man-in-the-middle attack)!
It is also possible that the RSA host key has just been changed.
The fingerprint for the RSA key sent by the remote host is
88:ce:24:aa:d0:e5:56:a2:b3:b4:9b:fd:3b:68:46:26.
Please contact your system administrator.
Add correct host key in /root/.ssh/known_hosts to get rid of this message.
Offending key in /root/.ssh/known_hosts:1
RSA host key for 10.66.70.58 has changed and you have requested strict checking.
Host key verification failed.

2,this issue will most probably happens when the remote machine have been re-install.

3, When this issue occur, most probably, most end user will check libvirtd, but that still not work, they maybe treat it as a fault of virt-manager.
Comment 2 Cole Robinson 2009-12-13 22:05:26 EST
This is really something that should be improved at the libvirt level. Reassigning.
Comment 4 Daniel Berrange 2009-12-16 09:56:28 EST
We can't really do any good handling of failures like this when we're forking & exec'ing ssh. To do a good job, we'd need to re-write the remote driver to use libssh2 instead. Such a change is not really material for a stable RHEL update, so I'd recommend this be pushed to RHEL-6 or upstream.
Comment 7 Daniel Veillard 2010-09-15 10:32:01 EDT
As suggested in comment #4 we won't change to use libssh2 in libvirt, even
after the rebase for 5.6

Daniel
Comment 11 Peter Krempa 2011-10-13 04:58:37 EDT
I don't think that this is a bug. Lessening the importance of the fact that a host key of a machine has changed might result in serious security vulnerabilities. Reinstalls of machines are a common operation, but admins should notify about the host key change, or copy the old ones. This is in my opinion an administrative "bug" and not something we shoud fix or lessen importance of.

I'd bet on that if we lessen/change the meaning of this error message, someone will report, what a security vulnerability we've created.

On the other hand, I agree that rewriting ssh connection support using library functions instead of spawning the ssh client process would be a better solution, but in that case I'd also prefer to leave the message complaining about changed host keys to stay as-is.

I'd like to hear your opinions on this case.

Peter
Comment 12 Peter Krempa 2011-10-13 05:03:17 EDT
Oh, I shoud've read the bug report more closely. I missed the fact, that virt-manager and others don't report the error correctly. Sorry for the confusion. :/
Comment 19 Peter Krempa 2012-08-21 09:22:51 EDT
Libvirt now supports a native ssh transport driver that reports errors to the management applications gracefuly. The new transport driver was added upstream with commits:

commit dfbb2d31208deeb68570a8242a3fae08b1bd27c7
Author: Peter Krempa <pkrempa@redhat.com>
Date:   Tue Jul 17 15:48:04 2012 +0200

    libssh2_transport: Use libssh2 driver code in remote driver
    
commit 25f2c8b4b04807f5c24fa87dc54d929ba1331421
Author: Peter Krempa <pkrempa@redhat.com>
Date:   Mon Nov 14 16:12:53 2011 +0100

    libssh2_transport: Add libssh2 session support to net client code
    
commit 637ea5427442d76c9499fb3922da3d81f9f0ad89
Author: Peter Krempa <pkrempa@redhat.com>
Date:   Mon Nov 14 15:50:02 2011 +0100

    libssh2_transport: add ssh context support to virNetSocket
    
commit 1193fc5f4465a622b22c10b16ab8081e98c2e622
Author: Peter Krempa <pkrempa@redhat.com>
Date:   Mon Nov 14 15:30:23 2011 +0100

    libssh2_transport: add main libssh2 transport implementation

Moving to POST.
Comment 21 Huang Wenlong 2012-08-23 04:07:43 EDT
Hi,

I try to verify this bug , but can not get expected result  
2 hosts pkg version : 
libvirt-0.10.0-0rc1.el6.x86_64
qemu-kvm-rhev-0.12.1.2-2.303.el6.x86_64
virt-manager-0.9.0-14.el6.x86_64


1) 
When I test it via virt-manager I get error message : 


Unable to open a connection to the libvirt management daemon.

Libvirt URI is: qemu+ssh://root@10.66.5.12/system

Verify that:
 - The 'libvirtd' daemon has been started

Cannot recv data: Value too large for defined data type

Traceback (most recent call last):
  File "/usr/share/virt-manager/virtManager/connection.py", line 1154, in _open_thread
    self.vmm = self._try_open()
  File "/usr/share/virt-manager/virtManager/connection.py", line 1138, in _try_open
    flags)
  File "/usr/lib64/python2.6/site-packages/libvirt.py", line 102, in openAuth
    if ret is None:raise libvirtError('virConnectOpenAuth() failed')
libvirtError: Cannot recv data: Value too large for defined data type


2) 
when I test it via virsh  I get error:
virsh -c qemu+ssh://10.66.5.12/system list
error: Cannot recv data: @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@    WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED!     @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
Someone could be eavesdropping on you right now (man-in-the-middle attack)!
It is also possible that the RSA host key has just been changed.
The fingerprint for the RSA key sent by the remote host is
56:eb:86:62:26:5e:11:40:c6:88:83:3a:7f:8b:84:e1.
Please contact your system administrator.
Add correct host key in /root/.ssh/known_hosts to get rid of this message.
Offending key in /root/.ssh/known_hosts:2
RSA host key for 10.66.5.12 has changed and you have requested strict checking.
Host key verification failed.: Connection reset by peer
error: failed to connect to the hypervisor
Comment 22 Peter Krempa 2012-08-23 04:31:13 EDT
You should test this with the new transport scheme as the old can't be fixed. Your URIs should look like: qemu+libssh2://10.66.5.12/system . Please note that this approach uses a different known_hosts file that should be located in ~/.config/libvirt/known_hosts. 

Also it might be possible that the libssh2 transport is not available if the libssh2 package wasn't updated to a compatible version yet (1.3 and later)
Comment 23 Huang Wenlong 2012-08-23 05:10:27 EDT
Hi, Peter Krempa
host env : 
libssh2-1.2.2-7.el6_2.3.x86_64 


I test it via libssh2 but get error :

#virsh -c qemu+libssh2://10.66.5.12/system list
error: libssh2 transport support was not enabled: Function not implemented
error: failed to connect to the hypervisor

I think maybe it did not add libssh2 support when build this version of libvirt.
Comment 24 Peter Krempa 2012-08-23 05:18:03 EDT
As I stated in the second paragraph in my comment libssh2 1.3 and later is required for the libssh2 transport to work. Unfortunately the new version is not available yet. The library should be updated in time for the next release. You'll be able to verify this bug after libvirt is built against the newer version.
Comment 25 Huang Wenlong 2012-08-23 05:19:17 EDT
(In reply to comment #24)
> As I stated in the second paragraph in my comment libssh2 1.3 and later is
> required for the libssh2 transport to work. Unfortunately the new version is
> not available yet. The library should be updated in time for the next
> release. You'll be able to verify this bug after libvirt is built against
> the newer version.

Thanks very much
Comment 27 Huang Wenlong 2012-10-12 02:04:54 EDT
Hi,Peter Krempa 

Is there a valid libvirt pkg to test libssh2 ? 
my env : 

libvirt-0.10.2-2.el6.x86_64
libssh2-1.4.2-1.el6.x86_64

it still can not verify this bug 


Wenlong
Comment 28 Peter Krempa 2012-10-12 17:09:18 EDT
Moving back to ASSIGNED as the spec file needs to be fixed to actually build libvirt with libssh2 support in Red Hat environments.
Comment 29 Peter Krempa 2012-10-12 17:41:47 EDT
Fixed upstream:

commit 1e25c54f66b46223e8e31843cec9a825fd02c735
Author: Peter Krempa <pkrempa@redhat.com>
Date:   Fri Oct 12 16:50:42 2012 +0200

    spec: Add support for libssh2 transport
    
    Libssh2 transport support was enabled lately but the spec file wasn't
    updated to take this into account. This caused libvirt to be built
    without libssh2 support in Red Hat based OSes.
Comment 32 Huang Wenlong 2012-10-16 02:56:57 EDT
Hi, Peter 

I have the new libvirt rebuild with new libssh2 , but 
there is an error when I test libssh2 , I do not know what I missed 
please help me ,thanks very much .


libvirt-0.10.2-3.el6.x86_64
libssh2-1.4.2-1.el6.x86_64


# virsh -c qemu+libssh2://127.0.0.1/system list 
Accept SSH host key with hash 'D5:0F:B6:94:55:74:1D:BE:7E:BB:D2:D9:89:B7:26:CD' for host '127.0.0.1:22' (y/n)?: y
error: authentication failed: All provided authentication methods with credentials were rejected by the server
error: failed to connect to the hypervisor



ssh still works

# virsh -c qemu+ssh://127.0.0.1/system list 
The authenticity of host '127.0.0.1 (127.0.0.1)' can't be established.
RSA key fingerprint is d5:0f:b6:94:55:74:1d:be:7e:bb:d2:d9:89:b7:26:cd.
Are you sure you want to continue connecting (yes/no)? yes
root@127.0.0.1's password: 
 Id    Name                           State
----------------------------------------------------
 14    ga                             running
 16    q64                            running
Comment 33 Peter Krempa 2012-11-27 06:01:41 EST
sshd in RHEL-6.4 does not support keyboard-interactive authentication in the default configuration.

debug1: Authentications that can continue: publickey,gssapi-keyex,gssapi-with-mic,password

The libssh2 driver uses this authentication method to to the password authentication. A workaround is to use publickey authentication until either sshd or libvirt get fixed to support passwords.
Comment 34 Peter Krempa 2012-11-27 09:39:50 EST
Fix proposed upstream: http://www.redhat.com/archives/libvir-list/2012-November/msg01152.html
Comment 35 Peter Krempa 2012-11-29 04:53:20 EST
As there is a workaround, I filed bug https://bugzilla.redhat.com/show_bug.cgi?id=881663 to track the issue of password authentication not working when the "keyboard-interactive" method is disabled on the ssh server.

The workaround is to enable option "ChallengeResponseAuthentication" in /etc/ssh/sshd_config and restart the sshd service.
Comment 36 Huang Wenlong 2012-11-29 04:58:19 EST
(In reply to comment #35)
> As there is a workaround, I filed bug
> https://bugzilla.redhat.com/show_bug.cgi?id=881663 to track the issue of
> password authentication not working when the "keyboard-interactive" method
> is disabled on the ssh server.
> 
> The workaround is to enable option "ChallengeResponseAuthentication" in
> /etc/ssh/sshd_config and restart the sshd service.


Thanks very much Peter 

Verify this bug :
libvirt-0.10.2-10.el6.x86_64

enable ChallengeResponseAuthentication in sshd config 
#cat /etc/ssh/sshd_config |grep ChallengeResponseAuthentication
ChallengeResponseAuthentication yes

then restart sshd 
service sshd restart

#  virsh -c qemu+libssh2://127.0.0.1/system list 
Accept SSH host key with hash '9E:3F:64:96:C7:47:E5:28:3B:6C:72:2C:76:D9:18:CB' for host '127.0.0.1:22' (y/n)?: y
Password: 
 Id    Name                           State
----------------------------------------------------
Comment 41 Dave Allan 2013-01-29 10:40:09 EST
Unfortunately libssh2 turned out to be insufficiently stable and I must close this BZ as WONTFIX.

Note You need to log in before you can comment on or make changes to this bug.