Bug 1253623

Summary: virt-customize tool corrupts Library paths for Neutron server.
Product: Red Hat OpenStack Reporter: Chandra Shekar Rangavajjula <chandra.s.rangavajjula>
Component: libguestfsAssignee: Pino Toscano <ptoscano>
Status: CLOSED NOTABUG QA Contact: yeylon <yeylon>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: DirectorCC: apevec, chandra.s.rangavajjula, lhh, ptoscano, rjones, srevivo, vcojot
Target Milestone: ---   
Target Release: 8.0 (Liberty)   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2015-10-13 08:00:29 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
neutron server.log
none
repo used for virt-customize
none
builder.log
none
virt-customize.log
none
virt-diff.log
none
logs Version-2 none

Description Chandra Shekar Rangavajjula 2015-08-14 09:36:04 UTC
Created attachment 1062954 [details]
neutron server.log

Description of problem:
Modifying overcloud image (Installing vim package) using virt-customize, corrupts overcloud image. When deployed overcloud with this image Neutron server fails to start with below error.

Version-Release number of selected component (if applicable):


How reproducible:
100%

Steps to Reproduce:
1. Download Overcloud images.
https://access.redhat.com/downloads/content/191/ver=7.0/rhel---7/7.0/x86_64/product-downloads
2. Follow below instructions to install director:
https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux_OpenStack_Platform/7/html/Director_Installation_and_Usage/index.html
3. modify overcloud image with below "virt-customize" command:
# virt-customize -a overcloud-full.qcow2 --upload <Repo for installing VIM>:/etc/yum.repos.d/ 
# virt-customize -a overcloud-full.qcow2 --install vim 
# virt-customize -a overcloud-full.qcow2 --run-command 'rm -rf /etc/yum.repos.d/<repo file>'
4. Continue with below instructions to install Overcloud:
https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux_OpenStack_Platform/7/html/Director_Installation_and_Usage/index.html


Actual results:
Neutron server on overcloud controller fails with below error. Overcloud deployment also fails as a result:

2015-08-14 04:24:52.099 51280 TRACE neutron.common.config   File "/usr/lib64/python2.7/site-packages/MySQLdb/__init__.py", line 19, in <module>
2015-08-14 04:24:52.099 51280 TRACE neutron.common.config     import _mysql
2015-08-14 04:24:52.099 51280 TRACE neutron.common.config ImportError: libmysqlclient.so.18: cannot open shared object file: No such file or directory

Expected results:
Neutron server should come up healthy.

Additional info:

Comment 3 Pino Toscano 2015-08-24 08:20:35 UTC
Can you please attach the .repo being uploaded? Please note that you might need to register the image to RHN (using subscription-manager), in case whatever you are installing from your own .repo needs to install dependencies from system packages.

Also, we need better logs of what is being actually done by virt-customize/libguestfs. Please run the following steps on a fresh copy of the overcloud image (we'll create an overlay, so it is not modified):

  $ qemu-img create -f qcow2 -o backing_file=overcloud-full.qcow2 overcloud-full-copy.qcow2
  $ virt-customize -v -x -a overcloud-full-copy.qcow2 --upload <Repo for installing VIM>:/etc/yum.repos.d/ --install vim --run-command 'rm -rf /etc/yum.repos.d/<repo file>' &> virt-customize.log
  $ virt-cat -a overcloud-full-copy.qcow2 /tmp/builder.log > builder.log
  $ virt-diff -a overcloud-full.qcow2 -A overcloud-full-copy.qcow2 > virt-diff.log

then please attach the resulting three virt-customize.log, builder.log, and virt-diff.log.

Comment 4 Chandra Shekar Rangavajjula 2015-08-25 16:19:19 UTC
Created attachment 1066941 [details]
repo used for virt-customize

This file contains local mirrored repos of RHEL OSP-7

Comment 5 Chandra Shekar Rangavajjula 2015-08-25 16:30:06 UTC
Created attachment 1066943 [details]
builder.log

Comment 6 Chandra Shekar Rangavajjula 2015-08-25 16:30:51 UTC
Created attachment 1066944 [details]
virt-customize.log

Comment 7 Chandra Shekar Rangavajjula 2015-08-25 16:31:24 UTC
Created attachment 1066945 [details]
virt-diff.log

Comment 8 Chandra Shekar Rangavajjula 2015-08-25 16:32:12 UTC
(In reply to Pino Toscano from comment #3)
> Can you please attach the .repo being uploaded? Please note that you might
> need to register the image to RHN (using subscription-manager), in case
> whatever you are installing from your own .repo needs to install
> dependencies from system packages.
> 
> Also, we need better logs of what is being actually done by
> virt-customize/libguestfs. Please run the following steps on a fresh copy of
> the overcloud image (we'll create an overlay, so it is not modified):
> 
>   $ qemu-img create -f qcow2 -o backing_file=overcloud-full.qcow2
> overcloud-full-copy.qcow2
>   $ virt-customize -v -x -a overcloud-full-copy.qcow2 --upload <Repo for
> installing VIM>:/etc/yum.repos.d/ --install vim --run-command 'rm -rf
> /etc/yum.repos.d/<repo file>' &> virt-customize.log
>   $ virt-cat -a overcloud-full-copy.qcow2 /tmp/builder.log > builder.log
>   $ virt-diff -a overcloud-full.qcow2 -A overcloud-full-copy.qcow2 >
> virt-diff.log
> 
> then please attach the resulting three virt-customize.log, builder.log, and
> virt-diff.log.


I have run the commands suggested by you. Uploaded all the files requested.

Comment 9 Richard W.M. Jones 2015-08-25 16:52:06 UTC
Bit confused by the bug report.  The reported error is:

  Neutron server on overcloud controller fails with below error. Overcloud deployment also fails as a result:

  2015-08-14 04:24:52.099 51280 TRACE neutron.common.config   File "/usr/lib64/python2.7/site-packages/MySQLdb/__init__.py", line 19, in <module>
  2015-08-14 04:24:52.099 51280 TRACE neutron.common.config     import _mysql
  2015-08-14 04:24:52.099 51280 TRACE neutron.common.config ImportError: libmysqlclient.so.18: cannot open shared object file: No such file or directory

Is this error inside the guest or on the host?

It seems as if this is a bug in MySQL, rather than anything
to do with virt-customize.

Comment 10 Chandra Shekar Rangavajjula 2015-08-25 18:12:47 UTC
(In reply to Richard W.M. Jones from comment #9)
> Bit confused by the bug report.  The reported error is:
> 
>   Neutron server on overcloud controller fails with below error. Overcloud
> deployment also fails as a result:
> 
>   2015-08-14 04:24:52.099 51280 TRACE neutron.common.config   File
> "/usr/lib64/python2.7/site-packages/MySQLdb/__init__.py", line 19, in
> <module>
>   2015-08-14 04:24:52.099 51280 TRACE neutron.common.config     import _mysql
>   2015-08-14 04:24:52.099 51280 TRACE neutron.common.config ImportError:
> libmysqlclient.so.18: cannot open shared object file: No such file or
> directory
> 
> Is this error inside the guest or on the host?
> 
> It seems as if this is a bug in MySQL, rather than anything
> to do with virt-customize.

This issue happens on the Overcloud overcloud controller deployed with the modified (by virt-customize) qcow2 image (not on undercloud director node). 

When an unmodified image is used to deploy overcloud controller, no such errors are seen. 

Can it be that virt-customize's VIM install has played with ldconfig? Did the virt-diff.log give any traces of it?

Comment 11 Richard W.M. Jones 2015-08-25 18:27:06 UTC
Apparently yes, /etc/ld.so.cache was deleted.

Note that virt-customize doesn't delete this file.  It must have
been deleted by something in one of the installed packages (likely
a %post script):

Installing:
 vim-enhanced        x86_64      2:7.4.160-1.el7        RH7.0-Server      1.0 M
Installing for dependencies:
 gpm-libs            x86_64      1.20.7-5.el7           RH7.0-Server       32 k
 vim-common          x86_64      2:7.4.160-1.el7        RH7.0-Server      5.9 M
 vim-filesystem      x86_64      2:7.4.160-1.el7        RH7.0-Server      9.6 k

It's remotely possible that a %trigger script from some other
RPM already on the system deleted it.

It's also very remotely possible that the %post script tried to
run ldconfig, but for some reason ldconfig wasn't able to run
properly and ended up deleting the ld.so.cache file (instead of
leaving it alone or printing an error).

You should start by looking at the scripts (rpm -q --scripts) of
all packages that are installed.  'virt-rescue' can help with doing
this.

Comment 12 Richard W.M. Jones 2015-08-25 18:29:27 UTC
(In reply to Richard W.M. Jones from comment #11)
> It's also very remotely possible that the %post script tried to
> run ldconfig, but for some reason ldconfig wasn't able to run
> properly and ended up deleting the ld.so.cache file (instead of
> leaving it alone or printing an error).

I mention this one because it's plausible that this might be
caused by the chroot / appliance environment that libguestfs uses
when running yum, in which case it would be a bug in libguestfs.
However I've never seen this happen before.

Comment 13 Chandra Shekar Rangavajjula 2015-08-25 18:42:25 UTC
(In reply to Richard W.M. Jones from comment #12)
> (In reply to Richard W.M. Jones from comment #11)
> > It's also very remotely possible that the %post script tried to
> > run ldconfig, but for some reason ldconfig wasn't able to run
> > properly and ended up deleting the ld.so.cache file (instead of
> > leaving it alone or printing an error).
> 
> I mention this one because it's plausible that this might be
> caused by the chroot / appliance environment that libguestfs uses
> when running yum, in which case it would be a bug in libguestfs.
> However I've never seen this happen before.

[stack@manager ticket]$ vim virt-customize.log
[stack@manager ticket]$ rpm -q --scripts vim-enhanced-7.4.160-1.el7.x86_64
[stack@manager ticket]$ rpm -q --scripts vim-common-7.4.160-1.el7.x86_64
[stack@manager ticket]$ rpm -q --scripts vim-filesystem-7.4.160-1.el7.x86_64
[stack@manager ticket]$ rpm -q --scripts vim-minimal-7.4.160-1.el7.x86_64
(reverse-i-search)`g': vim virt-customize.lo^C
[stack@manager ticket]$ rpm -q --scripts gpm-libs-1.20.7-5.el7.x86_64
postinstall program: /sbin/ldconfig
postuninstall program: /sbin/ldconfig

I think gpm-libs, is running ldconfig in the post installation.
Does this virt-customize need to be run as root?

Comment 14 Richard W.M. Jones 2015-08-25 19:51:43 UTC
(In reply to Chandra Shekar Rangavajjula from comment #13)
> Does this virt-customize need to be run as root?

virt-customize does not need to be run as root, but the yum (hence
rpmlib) commands will be run as root inside the appliance.  See the
architecture diagram here:

http://libguestfs.org/guestfs.3.html#architecture

> I think gpm-libs, is running ldconfig in the post installation.

Even if ldconfig fails, it wouldn't expect it to delete the
/etc/ld.so.conf file.  Somehow that file got deleted by something.

Does it work if you add this to the end of the virt-customize
command line?

  --run-command "ldconfig"

-------

Also, starting with the initial image again, can you run virt-rescue
on it and find out what process touches /etc/ld.so.conf:

  $ virt-rescue --suggest -a overcloud-full.qcow2

  $ virt-rescue --network --ro -a overcloud-full.qcow2
  ><rescue> # type the suggested mount commands from above
  ><rescue> chroot /sysroot
  ><rescue> ls -l /etc/ld*
  ><rescue> yum install vim-enhanced
  ><rescue> ls -l /etc/ld*

Does /etc/ld.so.conf exist before the yum command?  Does it exist
after?  There may be some systemtap magic you can use to find out
what deletes the file.

Comment 15 Chandra Shekar Rangavajjula 2015-08-25 19:54:36 UTC
(In reply to Richard W.M. Jones from comment #14)
> (In reply to Chandra Shekar Rangavajjula from comment #13)
> > Does this virt-customize need to be run as root?
> 
> virt-customize does not need to be run as root, but the yum (hence
> rpmlib) commands will be run as root inside the appliance.  See the
> architecture diagram here:
> 
> http://libguestfs.org/guestfs.3.html#architecture
> 
> > I think gpm-libs, is running ldconfig in the post installation.
> 
> Even if ldconfig fails, it wouldn't expect it to delete the
> /etc/ld.so.conf file.  Somehow that file got deleted by something.
> 
> Does it work if you add this to the end of the virt-customize
> command line?
> 
>   --run-command "ldconfig"
> 
> -------
> 
> Also, starting with the initial image again, can you run virt-rescue
> on it and find out what process touches /etc/ld.so.conf:
> 
>   $ virt-rescue --suggest -a overcloud-full.qcow2
> 
>   $ virt-rescue --network --ro -a overcloud-full.qcow2
>   ><rescue> # type the suggested mount commands from above
>   ><rescue> chroot /sysroot
>   ><rescue> ls -l /etc/ld*
>   ><rescue> yum install vim-enhanced
>   ><rescue> ls -l /etc/ld*
> 
> Does /etc/ld.so.conf exist before the yum command?  Does it exist
> after?  There may be some systemtap magic you can use to find out
> what deletes the file.


Were you able to reproduce this?
If so can you try them yourself?

Comment 16 Richard W.M. Jones 2015-08-25 20:02:21 UTC
I may have the time at some point, but not soon.

Comment 17 Pino Toscano 2015-08-26 10:02:54 UTC
(In reply to Richard W.M. Jones from comment #11)
> Apparently yes, /etc/ld.so.cache was deleted.

It seems like it was just changed ('=' in virt-diff output, and empty diff since it is a binary file).

Also, the provided virt-customize.log seems broken, and that was a bug (*). /tmp/builder.log; because of this, can you please run my instructions from comment #3, but just with a differerence:

  $ qemu-img create -f qcow2 -o backing_file=overcloud-full.qcow2 overcloud-full-copy.qcow2
  $ virt-customize -v -x -a overcloud-full-copy.qcow2 --upload <Repo for installing VIM>:/etc/yum.repos.d/ --install vim --run-command 'rm -rf /etc/yum.repos.d/<repo file>' --run-command 'mv /tmp/builder.log /tmp/builder-safe.log' &> virt-customize.log
  $ virt-cat -a overcloud-full-copy.qcow2 /tmp/builder-safe.log > builder.log
  $ virt-diff -a overcloud-full.qcow2 -A overcloud-full-copy.qcow2 > virt-diff.log

Now the resulting virt-customize.log should contain all the actual virt-customize log.

(*) fixed in 1.30 (>= 1.29.44) with commit c919f6f75c8eb44098e6329d57cbecb3e697cf1c

Comment 18 Chandra Shekar Rangavajjula 2015-08-26 10:29:08 UTC
(In reply to Pino Toscano from comment #17)
> (In reply to Richard W.M. Jones from comment #11)
> > Apparently yes, /etc/ld.so.cache was deleted.
> 
> It seems like it was just changed ('=' in virt-diff output, and empty diff
> since it is a binary file).
> 
> Also, the provided virt-customize.log seems broken, and that was a bug (*).
> /tmp/builder.log; because of this, can you please run my instructions from
> comment #3, but just with a differerence:
> 
>   $ qemu-img create -f qcow2 -o backing_file=overcloud-full.qcow2
> overcloud-full-copy.qcow2
>   $ virt-customize -v -x -a overcloud-full-copy.qcow2 --upload <Repo for
> installing VIM>:/etc/yum.repos.d/ --install vim --run-command 'rm -rf
> /etc/yum.repos.d/<repo file>' --run-command 'mv /tmp/builder.log
> /tmp/builder-safe.log' &> virt-customize.log
>   $ virt-cat -a overcloud-full-copy.qcow2 /tmp/builder-safe.log > builder.log
>   $ virt-diff -a overcloud-full.qcow2 -A overcloud-full-copy.qcow2 >
> virt-diff.log
> 
> Now the resulting virt-customize.log should contain all the actual
> virt-customize log.
> 
> (*) fixed in 1.30 (>= 1.29.44) with commit
> c919f6f75c8eb44098e6329d57cbecb3e697cf1c


Hi,
Here are the commands I executed. I am attaching the respective logs.

[stack@manager ticket]$ qemu-img create -f qcow2 -o backing_file=overcloud-full.qcow2 overcloud-full-copy.qcow2
[stack@manager ticket]$ virt-customize -v -x -a overcloud-full-copy.qcow2 --upload /etc/yum.repos.d/luxem.repo:/etc/yum.repos.d/ --install vim --run-command 'rm -rf /etc/yum.repos.d/luxem.repo' --run-command 'mv /tmp/builder.log /tmp/builder-safe.log' &> virt-customize.log
[stack@manager ticket]$ virt-cat -a overcloud-full-copy.qcow2 /tmp/builder-safe.log > builder.log
[stack@manager ticket]$ virt-diff -a overcloud-full.qcow2 -A overcloud-full-copy.qcow2 > virt-diff.log

Comment 19 Chandra Shekar Rangavajjula 2015-08-26 10:32:05 UTC
Created attachment 1067217 [details]
logs Version-2

Comment 20 Chandra Shekar Rangavajjula 2015-10-12 11:03:55 UTC
Any update in this case?

Comment 21 Pino Toscano 2015-10-12 11:53:00 UTC
(In reply to Chandra Shekar Rangavajjula from comment #20)
> Any update in this case?

Ah, sorry about it.

From the new set of logs, it seems like the whole process was fine -- repository added, and things installed as requested.

What comes to my mind now is about SELinux: virt-customize works with no SELinux enabled in its appliance, so the guest does not get any change in SELinux labels/attributes.  Since we use SELinux in RHEL, you should make sure the guest has the correct SELinux attributes for the new files as well: this can be done by passing --selinux-relabel as additional argument to virt-customize.

Can you please try adding it in your virt-customize steps, and see whether the overcloud works?

Comment 22 Chandra Shekar Rangavajjula 2015-10-12 15:22:39 UTC
Yes.
This works for me. Thanks for the support. We can close this ticket now.

Comment 23 Richard W.M. Jones 2015-10-13 08:00:29 UTC
Closing per comment 22.