Bug 1010942 - Libvirt fail to connect to esx server during virt-v2v conversion
Summary: Libvirt fail to connect to esx server during virt-v2v conversion
Keywords:
Status: CLOSED WORKSFORME
Alias: None
Product: Red Hat Enterprise Linux 6
Classification: Red Hat
Component: libvirt
Version: 6.5
Hardware: Unspecified
OS: Unspecified
high
high
Target Milestone: rc
: ---
Assignee: Jiri Denemark
QA Contact: Virtualization Bugs
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2013-09-23 10:39 UTC by tingting zheng
Modified: 2013-10-01 14:49 UTC (History)
10 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2013-10-01 14:49:50 UTC
Target Upstream Version:


Attachments (Terms of Use)
Log info on fresh system (88.86 KB, text/plain)
2013-09-25 03:01 UTC, tingting zheng
no flags Details
Log info on upgrade system (32.25 KB, text/plain)
2013-09-25 03:02 UTC, tingting zheng
no flags Details

Description tingting zheng 2013-09-23 10:39:37 UTC
Description
Libvirt fail to connect to esx server during virt-v2v conversion

Version:
kernel-2.6.32-419.el6.x86_64
qemu-kvm-0.12.1.2-2.402.el6.x86_64
libvirt-0.10.2-24.el6.x86_64
virt-v2v-0.9.1-3.el6.x86_64


How reproducible:
100%

Steps to Reproduce:
1. Create /root/.netrc,add the below items:
machine 10.66.72.39 login root password $password

2. # chmod 600 /root/.netrc

3. Use virt-v2v to convert a esx guest.
# virt-v2v  -ic esx://10.66.72.39/?no_verify=1 -os default  esx3.5-rhel6.4-x86_64
virt-v2v: Failed to connect to esx://10.66.72.39/?no_verify=1: libvirt error code: 1, message: internal error curl_easy_perform() returned an error: 
Problem with the SSL CA cert (path? access rights?) (77) : Problem with the SSL CA cert (path? access rights?)

4.I can use virsh -c to connect esx server.
# virsh -c esx://10.66.72.39/?no_verify=1
Enter username for 10.66.72.39 [root]: 
Enter root's password for 10.66.72.39: 
Welcome to virsh, the virtualization interactive terminal.

Type:  'help' for help with commands
       'quit' to quit

virsh # list --all
 Id    Name                           State
----------------------------------------------------
 -     esx3.5-rhel6.4-i386            shut off
 -     esx3.5-rhel6.4-x86_64          shut off


Actual results:
As description.

Expected results:
virt-v2v can convert guests successfully.

Additional info:
1.The system is newly installed,using tree RHEL6.5-20130913.0,if I use old system and installed the packages to the latest,the error will not show.
2.Downgrade virt-v2v and libvirt to: virt-v2v-0.8.9-2.el6.x86_64  libvirt-0.10.2-18.el6.x86_64,the error still shows.

Comment 2 Jiri Denemark 2013-09-24 09:42:54 UTC
Since you can't reproduce when upgrading old system and even downgrading the package in a freshed system doesn't make the issue go away, this looks like a configuration issue. However, I'm not sure where to look. Matthew, do you have any idea?

Comment 3 Matthew Booth 2013-09-24 10:04:09 UTC
The libvirt connection is made like this:

    my $vmm;
    eval {
        $vmm = Sys::Virt->new(
            uri => $uri,
            auth => 1,
            credlist => [
                Sys::Virt::CRED_AUTHNAME,
                Sys::Virt::CRED_PASSPHRASE
            ],
            callback => sub {
                my $creds = shift;

                foreach my $cred (@$creds) {
                    if ($cred->{type} == Sys::Virt::CRED_AUTHNAME) {
                        $cred->{result} = $self->{username};
                    }

                    elsif ($cred->{type} == Sys::Virt::CRED_PASSPHRASE) {
                        $cred->{result} = $self->{password};
                    }

                    else { die($cred->{type}, "\n"); }
                }
            }
        );
    };
    v2vdie __x('Failed to connect to {uri}: {error}',
               uri => $uri, error => $@->stringify()) if $@;

Comment 4 Matthew Booth 2013-09-24 10:08:00 UTC
I'm going to go out on a limb and guess this is an authentication issue, which causes libvirt to fall through to attempting SSL cert authentication. Is that feasible?

Tingting, are you sure that the password in .netrc is correct? How about that leading '$'.

Comment 5 tingting zheng 2013-09-24 10:15:49 UTC
(In reply to Matthew Booth from comment #4)
> I'm going to go out on a limb and guess this is an authentication issue,
> which causes libvirt to fall through to attempting SSL cert authentication.
> Is that feasible?
> 
> Tingting, are you sure that the password in .netrc is correct? How about
> that leading '$'.

Yes,I am sure the password is correct.We have tried several newly installed machine and can always reproduce this issue.

Comment 6 Jiri Denemark 2013-09-24 12:49:03 UTC
Could you turn on libvirt debug messages

# export LIBVIRT_DEBUG=1
# export LIBVIRT_LOG_OUTPUTS="1:file:libvirt-client.log

and run the virt-v2v command on both the fresh system and the upgraded system and also run the virsh command on the fresh system. Once done, please attach the three libvirt logs.

Comment 7 tingting zheng 2013-09-25 03:01:21 UTC
Created attachment 802537 [details]
Log info on fresh system

Comment 8 tingting zheng 2013-09-25 03:02:04 UTC
Created attachment 802538 [details]
Log info on upgrade system

Comment 9 tingting zheng 2013-09-25 03:02:59 UTC
(In reply to Jiri Denemark from comment #6)
> Could you turn on libvirt debug messages
> 
> # export LIBVIRT_DEBUG=1
> # export LIBVIRT_LOG_OUTPUTS="1:file:libvirt-client.log
> 
> and run the virt-v2v command on both the fresh system and the upgraded
> system and also run the virsh command on the fresh system. Once done, please
> attach the three libvirt logs.

Related logs are attached refer to comment 7 and comment 8.

Comment 10 Jiri Denemark 2013-09-26 12:08:11 UTC
I tried running both virt-v2v and virsh commands in gdb and esxVI_Context_Connect() is called with exactly the same parameters in both cases (i.e., the same URI, username, and password). However, connecting to esx fails from virt-v2v in esxVI_RetrieveServiceContent() while from virsh the function successfully connects to esx and retrieves the desired data. The point where the two commands differ in execution is in NSS_InitContext called from curl's nss.c:892 with cerpath = "sql:/etc/pki/nssdb". I don't really know what that means but I assume virt-v2v must be changing the environment in some way...

Comment 11 tingting zheng 2013-09-29 08:54:46 UTC
I installed a new system with tree RHEL6.5-20130925.2,related version:
qemu-kvm-0.12.1.2-2.406.el6.x86_64
kernel-2.6.32-420.el6.x86_64
libvirt-0.10.2-27.el6.x86_64
virt-v2v-0.9.1-3.el6.x86_64

Retested the issue and the bug can not be reproduced anymore.

Comment 12 Jiri Denemark 2013-10-01 14:49:50 UTC
OK, I'm closing this bug. Feel free to reopen if it ever appears again.


Note You need to log in before you can comment on or make changes to this bug.