Bug 1010942

Summary: Libvirt fail to connect to esx server during virt-v2v conversion
Product: Red Hat Enterprise Linux 6 Reporter: tingting zheng <tzheng>
Component: libvirtAssignee: Jiri Denemark <jdenemar>
Status: CLOSED WORKSFORME QA Contact: Virtualization Bugs <virt-bugs>
Severity: high Docs Contact:
Priority: high    
Version: 6.5CC: acathrow, bfan, codong, dyuan, hyao, jdenemar, lcui, mbooth, tzheng, zsong
Target Milestone: rc   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2013-10-01 14:49:50 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Attachments:
Description Flags
Log info on fresh system
none
Log info on upgrade system none

Description tingting zheng 2013-09-23 10:39:37 UTC
Description
Libvirt fail to connect to esx server during virt-v2v conversion

Version:
kernel-2.6.32-419.el6.x86_64
qemu-kvm-0.12.1.2-2.402.el6.x86_64
libvirt-0.10.2-24.el6.x86_64
virt-v2v-0.9.1-3.el6.x86_64


How reproducible:
100%

Steps to Reproduce:
1. Create /root/.netrc,add the below items:
machine 10.66.72.39 login root password $password

2. # chmod 600 /root/.netrc

3. Use virt-v2v to convert a esx guest.
# virt-v2v  -ic esx://10.66.72.39/?no_verify=1 -os default  esx3.5-rhel6.4-x86_64
virt-v2v: Failed to connect to esx://10.66.72.39/?no_verify=1: libvirt error code: 1, message: internal error curl_easy_perform() returned an error: 
Problem with the SSL CA cert (path? access rights?) (77) : Problem with the SSL CA cert (path? access rights?)

4.I can use virsh -c to connect esx server.
# virsh -c esx://10.66.72.39/?no_verify=1
Enter username for 10.66.72.39 [root]: 
Enter root's password for 10.66.72.39: 
Welcome to virsh, the virtualization interactive terminal.

Type:  'help' for help with commands
       'quit' to quit

virsh # list --all
 Id    Name                           State
----------------------------------------------------
 -     esx3.5-rhel6.4-i386            shut off
 -     esx3.5-rhel6.4-x86_64          shut off


Actual results:
As description.

Expected results:
virt-v2v can convert guests successfully.

Additional info:
1.The system is newly installed,using tree RHEL6.5-20130913.0,if I use old system and installed the packages to the latest,the error will not show.
2.Downgrade virt-v2v and libvirt to: virt-v2v-0.8.9-2.el6.x86_64  libvirt-0.10.2-18.el6.x86_64,the error still shows.

Comment 2 Jiri Denemark 2013-09-24 09:42:54 UTC
Since you can't reproduce when upgrading old system and even downgrading the package in a freshed system doesn't make the issue go away, this looks like a configuration issue. However, I'm not sure where to look. Matthew, do you have any idea?

Comment 3 Matthew Booth 2013-09-24 10:04:09 UTC
The libvirt connection is made like this:

    my $vmm;
    eval {
        $vmm = Sys::Virt->new(
            uri => $uri,
            auth => 1,
            credlist => [
                Sys::Virt::CRED_AUTHNAME,
                Sys::Virt::CRED_PASSPHRASE
            ],
            callback => sub {
                my $creds = shift;

                foreach my $cred (@$creds) {
                    if ($cred->{type} == Sys::Virt::CRED_AUTHNAME) {
                        $cred->{result} = $self->{username};
                    }

                    elsif ($cred->{type} == Sys::Virt::CRED_PASSPHRASE) {
                        $cred->{result} = $self->{password};
                    }

                    else { die($cred->{type}, "\n"); }
                }
            }
        );
    };
    v2vdie __x('Failed to connect to {uri}: {error}',
               uri => $uri, error => $@->stringify()) if $@;

Comment 4 Matthew Booth 2013-09-24 10:08:00 UTC
I'm going to go out on a limb and guess this is an authentication issue, which causes libvirt to fall through to attempting SSL cert authentication. Is that feasible?

Tingting, are you sure that the password in .netrc is correct? How about that leading '$'.

Comment 5 tingting zheng 2013-09-24 10:15:49 UTC
(In reply to Matthew Booth from comment #4)
> I'm going to go out on a limb and guess this is an authentication issue,
> which causes libvirt to fall through to attempting SSL cert authentication.
> Is that feasible?
> 
> Tingting, are you sure that the password in .netrc is correct? How about
> that leading '$'.

Yes,I am sure the password is correct.We have tried several newly installed machine and can always reproduce this issue.

Comment 6 Jiri Denemark 2013-09-24 12:49:03 UTC
Could you turn on libvirt debug messages

# export LIBVIRT_DEBUG=1
# export LIBVIRT_LOG_OUTPUTS="1:file:libvirt-client.log

and run the virt-v2v command on both the fresh system and the upgraded system and also run the virsh command on the fresh system. Once done, please attach the three libvirt logs.

Comment 7 tingting zheng 2013-09-25 03:01:21 UTC
Created attachment 802537 [details]
Log info on fresh system

Comment 8 tingting zheng 2013-09-25 03:02:04 UTC
Created attachment 802538 [details]
Log info on upgrade system

Comment 9 tingting zheng 2013-09-25 03:02:59 UTC
(In reply to Jiri Denemark from comment #6)
> Could you turn on libvirt debug messages
> 
> # export LIBVIRT_DEBUG=1
> # export LIBVIRT_LOG_OUTPUTS="1:file:libvirt-client.log
> 
> and run the virt-v2v command on both the fresh system and the upgraded
> system and also run the virsh command on the fresh system. Once done, please
> attach the three libvirt logs.

Related logs are attached refer to comment 7 and comment 8.

Comment 10 Jiri Denemark 2013-09-26 12:08:11 UTC
I tried running both virt-v2v and virsh commands in gdb and esxVI_Context_Connect() is called with exactly the same parameters in both cases (i.e., the same URI, username, and password). However, connecting to esx fails from virt-v2v in esxVI_RetrieveServiceContent() while from virsh the function successfully connects to esx and retrieves the desired data. The point where the two commands differ in execution is in NSS_InitContext called from curl's nss.c:892 with cerpath = "sql:/etc/pki/nssdb". I don't really know what that means but I assume virt-v2v must be changing the environment in some way...

Comment 11 tingting zheng 2013-09-29 08:54:46 UTC
I installed a new system with tree RHEL6.5-20130925.2,related version:
qemu-kvm-0.12.1.2-2.406.el6.x86_64
kernel-2.6.32-420.el6.x86_64
libvirt-0.10.2-27.el6.x86_64
virt-v2v-0.9.1-3.el6.x86_64

Retested the issue and the bug can not be reproduced anymore.

Comment 12 Jiri Denemark 2013-10-01 14:49:50 UTC
OK, I'm closing this bug. Feel free to reopen if it ever appears again.