Bug 1253623
Summary: | virt-customize tool corrupts Library paths for Neutron server. | ||||||||||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
Product: | Red Hat OpenStack | Reporter: | Chandra Shekar Rangavajjula <chandra.s.rangavajjula> | ||||||||||||||
Component: | libguestfs | Assignee: | Pino Toscano <ptoscano> | ||||||||||||||
Status: | CLOSED NOTABUG | QA Contact: | yeylon <yeylon> | ||||||||||||||
Severity: | unspecified | Docs Contact: | |||||||||||||||
Priority: | unspecified | ||||||||||||||||
Version: | Director | CC: | apevec, chandra.s.rangavajjula, lhh, ptoscano, rjones, srevivo, vcojot | ||||||||||||||
Target Milestone: | --- | ||||||||||||||||
Target Release: | 8.0 (Liberty) | ||||||||||||||||
Hardware: | Unspecified | ||||||||||||||||
OS: | Unspecified | ||||||||||||||||
Whiteboard: | |||||||||||||||||
Fixed In Version: | Doc Type: | Bug Fix | |||||||||||||||
Doc Text: | Story Points: | --- | |||||||||||||||
Clone Of: | Environment: | ||||||||||||||||
Last Closed: | 2015-10-13 08:00:29 UTC | Type: | Bug | ||||||||||||||
Regression: | --- | Mount Type: | --- | ||||||||||||||
Documentation: | --- | CRM: | |||||||||||||||
Verified Versions: | Category: | --- | |||||||||||||||
oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||||||||||||
Cloudforms Team: | --- | Target Upstream Version: | |||||||||||||||
Embargoed: | |||||||||||||||||
Attachments: |
|
Description
Chandra Shekar Rangavajjula
2015-08-14 09:36:04 UTC
Can you please attach the .repo being uploaded? Please note that you might need to register the image to RHN (using subscription-manager), in case whatever you are installing from your own .repo needs to install dependencies from system packages. Also, we need better logs of what is being actually done by virt-customize/libguestfs. Please run the following steps on a fresh copy of the overcloud image (we'll create an overlay, so it is not modified): $ qemu-img create -f qcow2 -o backing_file=overcloud-full.qcow2 overcloud-full-copy.qcow2 $ virt-customize -v -x -a overcloud-full-copy.qcow2 --upload <Repo for installing VIM>:/etc/yum.repos.d/ --install vim --run-command 'rm -rf /etc/yum.repos.d/<repo file>' &> virt-customize.log $ virt-cat -a overcloud-full-copy.qcow2 /tmp/builder.log > builder.log $ virt-diff -a overcloud-full.qcow2 -A overcloud-full-copy.qcow2 > virt-diff.log then please attach the resulting three virt-customize.log, builder.log, and virt-diff.log. Created attachment 1066941 [details]
repo used for virt-customize
This file contains local mirrored repos of RHEL OSP-7
Created attachment 1066943 [details]
builder.log
Created attachment 1066944 [details]
virt-customize.log
Created attachment 1066945 [details]
virt-diff.log
(In reply to Pino Toscano from comment #3) > Can you please attach the .repo being uploaded? Please note that you might > need to register the image to RHN (using subscription-manager), in case > whatever you are installing from your own .repo needs to install > dependencies from system packages. > > Also, we need better logs of what is being actually done by > virt-customize/libguestfs. Please run the following steps on a fresh copy of > the overcloud image (we'll create an overlay, so it is not modified): > > $ qemu-img create -f qcow2 -o backing_file=overcloud-full.qcow2 > overcloud-full-copy.qcow2 > $ virt-customize -v -x -a overcloud-full-copy.qcow2 --upload <Repo for > installing VIM>:/etc/yum.repos.d/ --install vim --run-command 'rm -rf > /etc/yum.repos.d/<repo file>' &> virt-customize.log > $ virt-cat -a overcloud-full-copy.qcow2 /tmp/builder.log > builder.log > $ virt-diff -a overcloud-full.qcow2 -A overcloud-full-copy.qcow2 > > virt-diff.log > > then please attach the resulting three virt-customize.log, builder.log, and > virt-diff.log. I have run the commands suggested by you. Uploaded all the files requested. Bit confused by the bug report. The reported error is: Neutron server on overcloud controller fails with below error. Overcloud deployment also fails as a result: 2015-08-14 04:24:52.099 51280 TRACE neutron.common.config File "/usr/lib64/python2.7/site-packages/MySQLdb/__init__.py", line 19, in <module> 2015-08-14 04:24:52.099 51280 TRACE neutron.common.config import _mysql 2015-08-14 04:24:52.099 51280 TRACE neutron.common.config ImportError: libmysqlclient.so.18: cannot open shared object file: No such file or directory Is this error inside the guest or on the host? It seems as if this is a bug in MySQL, rather than anything to do with virt-customize. (In reply to Richard W.M. Jones from comment #9) > Bit confused by the bug report. The reported error is: > > Neutron server on overcloud controller fails with below error. Overcloud > deployment also fails as a result: > > 2015-08-14 04:24:52.099 51280 TRACE neutron.common.config File > "/usr/lib64/python2.7/site-packages/MySQLdb/__init__.py", line 19, in > <module> > 2015-08-14 04:24:52.099 51280 TRACE neutron.common.config import _mysql > 2015-08-14 04:24:52.099 51280 TRACE neutron.common.config ImportError: > libmysqlclient.so.18: cannot open shared object file: No such file or > directory > > Is this error inside the guest or on the host? > > It seems as if this is a bug in MySQL, rather than anything > to do with virt-customize. This issue happens on the Overcloud overcloud controller deployed with the modified (by virt-customize) qcow2 image (not on undercloud director node). When an unmodified image is used to deploy overcloud controller, no such errors are seen. Can it be that virt-customize's VIM install has played with ldconfig? Did the virt-diff.log give any traces of it? Apparently yes, /etc/ld.so.cache was deleted. Note that virt-customize doesn't delete this file. It must have been deleted by something in one of the installed packages (likely a %post script): Installing: vim-enhanced x86_64 2:7.4.160-1.el7 RH7.0-Server 1.0 M Installing for dependencies: gpm-libs x86_64 1.20.7-5.el7 RH7.0-Server 32 k vim-common x86_64 2:7.4.160-1.el7 RH7.0-Server 5.9 M vim-filesystem x86_64 2:7.4.160-1.el7 RH7.0-Server 9.6 k It's remotely possible that a %trigger script from some other RPM already on the system deleted it. It's also very remotely possible that the %post script tried to run ldconfig, but for some reason ldconfig wasn't able to run properly and ended up deleting the ld.so.cache file (instead of leaving it alone or printing an error). You should start by looking at the scripts (rpm -q --scripts) of all packages that are installed. 'virt-rescue' can help with doing this. (In reply to Richard W.M. Jones from comment #11) > It's also very remotely possible that the %post script tried to > run ldconfig, but for some reason ldconfig wasn't able to run > properly and ended up deleting the ld.so.cache file (instead of > leaving it alone or printing an error). I mention this one because it's plausible that this might be caused by the chroot / appliance environment that libguestfs uses when running yum, in which case it would be a bug in libguestfs. However I've never seen this happen before. (In reply to Richard W.M. Jones from comment #12) > (In reply to Richard W.M. Jones from comment #11) > > It's also very remotely possible that the %post script tried to > > run ldconfig, but for some reason ldconfig wasn't able to run > > properly and ended up deleting the ld.so.cache file (instead of > > leaving it alone or printing an error). > > I mention this one because it's plausible that this might be > caused by the chroot / appliance environment that libguestfs uses > when running yum, in which case it would be a bug in libguestfs. > However I've never seen this happen before. [stack@manager ticket]$ vim virt-customize.log [stack@manager ticket]$ rpm -q --scripts vim-enhanced-7.4.160-1.el7.x86_64 [stack@manager ticket]$ rpm -q --scripts vim-common-7.4.160-1.el7.x86_64 [stack@manager ticket]$ rpm -q --scripts vim-filesystem-7.4.160-1.el7.x86_64 [stack@manager ticket]$ rpm -q --scripts vim-minimal-7.4.160-1.el7.x86_64 (reverse-i-search)`g': vim virt-customize.lo^C [stack@manager ticket]$ rpm -q --scripts gpm-libs-1.20.7-5.el7.x86_64 postinstall program: /sbin/ldconfig postuninstall program: /sbin/ldconfig I think gpm-libs, is running ldconfig in the post installation. Does this virt-customize need to be run as root? (In reply to Chandra Shekar Rangavajjula from comment #13) > Does this virt-customize need to be run as root? virt-customize does not need to be run as root, but the yum (hence rpmlib) commands will be run as root inside the appliance. See the architecture diagram here: http://libguestfs.org/guestfs.3.html#architecture > I think gpm-libs, is running ldconfig in the post installation. Even if ldconfig fails, it wouldn't expect it to delete the /etc/ld.so.conf file. Somehow that file got deleted by something. Does it work if you add this to the end of the virt-customize command line? --run-command "ldconfig" ------- Also, starting with the initial image again, can you run virt-rescue on it and find out what process touches /etc/ld.so.conf: $ virt-rescue --suggest -a overcloud-full.qcow2 $ virt-rescue --network --ro -a overcloud-full.qcow2 ><rescue> # type the suggested mount commands from above ><rescue> chroot /sysroot ><rescue> ls -l /etc/ld* ><rescue> yum install vim-enhanced ><rescue> ls -l /etc/ld* Does /etc/ld.so.conf exist before the yum command? Does it exist after? There may be some systemtap magic you can use to find out what deletes the file. (In reply to Richard W.M. Jones from comment #14) > (In reply to Chandra Shekar Rangavajjula from comment #13) > > Does this virt-customize need to be run as root? > > virt-customize does not need to be run as root, but the yum (hence > rpmlib) commands will be run as root inside the appliance. See the > architecture diagram here: > > http://libguestfs.org/guestfs.3.html#architecture > > > I think gpm-libs, is running ldconfig in the post installation. > > Even if ldconfig fails, it wouldn't expect it to delete the > /etc/ld.so.conf file. Somehow that file got deleted by something. > > Does it work if you add this to the end of the virt-customize > command line? > > --run-command "ldconfig" > > ------- > > Also, starting with the initial image again, can you run virt-rescue > on it and find out what process touches /etc/ld.so.conf: > > $ virt-rescue --suggest -a overcloud-full.qcow2 > > $ virt-rescue --network --ro -a overcloud-full.qcow2 > ><rescue> # type the suggested mount commands from above > ><rescue> chroot /sysroot > ><rescue> ls -l /etc/ld* > ><rescue> yum install vim-enhanced > ><rescue> ls -l /etc/ld* > > Does /etc/ld.so.conf exist before the yum command? Does it exist > after? There may be some systemtap magic you can use to find out > what deletes the file. Were you able to reproduce this? If so can you try them yourself? I may have the time at some point, but not soon. (In reply to Richard W.M. Jones from comment #11) > Apparently yes, /etc/ld.so.cache was deleted. It seems like it was just changed ('=' in virt-diff output, and empty diff since it is a binary file). Also, the provided virt-customize.log seems broken, and that was a bug (*). /tmp/builder.log; because of this, can you please run my instructions from comment #3, but just with a differerence: $ qemu-img create -f qcow2 -o backing_file=overcloud-full.qcow2 overcloud-full-copy.qcow2 $ virt-customize -v -x -a overcloud-full-copy.qcow2 --upload <Repo for installing VIM>:/etc/yum.repos.d/ --install vim --run-command 'rm -rf /etc/yum.repos.d/<repo file>' --run-command 'mv /tmp/builder.log /tmp/builder-safe.log' &> virt-customize.log $ virt-cat -a overcloud-full-copy.qcow2 /tmp/builder-safe.log > builder.log $ virt-diff -a overcloud-full.qcow2 -A overcloud-full-copy.qcow2 > virt-diff.log Now the resulting virt-customize.log should contain all the actual virt-customize log. (*) fixed in 1.30 (>= 1.29.44) with commit c919f6f75c8eb44098e6329d57cbecb3e697cf1c (In reply to Pino Toscano from comment #17) > (In reply to Richard W.M. Jones from comment #11) > > Apparently yes, /etc/ld.so.cache was deleted. > > It seems like it was just changed ('=' in virt-diff output, and empty diff > since it is a binary file). > > Also, the provided virt-customize.log seems broken, and that was a bug (*). > /tmp/builder.log; because of this, can you please run my instructions from > comment #3, but just with a differerence: > > $ qemu-img create -f qcow2 -o backing_file=overcloud-full.qcow2 > overcloud-full-copy.qcow2 > $ virt-customize -v -x -a overcloud-full-copy.qcow2 --upload <Repo for > installing VIM>:/etc/yum.repos.d/ --install vim --run-command 'rm -rf > /etc/yum.repos.d/<repo file>' --run-command 'mv /tmp/builder.log > /tmp/builder-safe.log' &> virt-customize.log > $ virt-cat -a overcloud-full-copy.qcow2 /tmp/builder-safe.log > builder.log > $ virt-diff -a overcloud-full.qcow2 -A overcloud-full-copy.qcow2 > > virt-diff.log > > Now the resulting virt-customize.log should contain all the actual > virt-customize log. > > (*) fixed in 1.30 (>= 1.29.44) with commit > c919f6f75c8eb44098e6329d57cbecb3e697cf1c Hi, Here are the commands I executed. I am attaching the respective logs. [stack@manager ticket]$ qemu-img create -f qcow2 -o backing_file=overcloud-full.qcow2 overcloud-full-copy.qcow2 [stack@manager ticket]$ virt-customize -v -x -a overcloud-full-copy.qcow2 --upload /etc/yum.repos.d/luxem.repo:/etc/yum.repos.d/ --install vim --run-command 'rm -rf /etc/yum.repos.d/luxem.repo' --run-command 'mv /tmp/builder.log /tmp/builder-safe.log' &> virt-customize.log [stack@manager ticket]$ virt-cat -a overcloud-full-copy.qcow2 /tmp/builder-safe.log > builder.log [stack@manager ticket]$ virt-diff -a overcloud-full.qcow2 -A overcloud-full-copy.qcow2 > virt-diff.log Created attachment 1067217 [details]
logs Version-2
Any update in this case? (In reply to Chandra Shekar Rangavajjula from comment #20) > Any update in this case? Ah, sorry about it. From the new set of logs, it seems like the whole process was fine -- repository added, and things installed as requested. What comes to my mind now is about SELinux: virt-customize works with no SELinux enabled in its appliance, so the guest does not get any change in SELinux labels/attributes. Since we use SELinux in RHEL, you should make sure the guest has the correct SELinux attributes for the new files as well: this can be done by passing --selinux-relabel as additional argument to virt-customize. Can you please try adding it in your virt-customize steps, and see whether the overcloud works? Yes. This works for me. Thanks for the support. We can close this ticket now. Closing per comment 22. |