Bug 259421 - yum: Slow transaction test/transaction execution
yum: Slow transaction test/transaction execution
Status: CLOSED ERRATA
Product: Fedora
Classification: Fedora
Component: rpm (Show other bugs)
7
i686 Linux
medium Severity low
: ---
: ---
Assigned To: Panu Matilainen
Fedora Extras Quality Assurance
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2007-08-28 05:36 EDT by Christian Mandery
Modified: 2014-01-21 17:59 EST (History)
6 users (show)

See Also:
Fixed In Version: 4.4.2.2-2.fc7
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2007-10-24 03:14:00 EDT
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
Output from strace -fFtt (20 MB unpacked!) (1.58 MB, application/x-7z-compressed)
2007-08-30 11:43 EDT, Christian Mandery
no flags Details
Strace output with -fFtt of a rpm run with the freeze in it (311.79 KB, application/octet-stream)
2007-08-31 05:13 EDT, Christian Mandery
no flags Details

  None (edit)
Description Christian Mandery 2007-08-28 05:36:44 EDT
Description of problem:
Since three weeks now, I am experiencing a strange problem, i.e. yum is hanging
around three to five minutes on both "Running Transaction Test" and "Running
Transaction".

During this hangs, "/usr/bin/python /usr/bin/yum install foobar" uses 100% CPU
and does not show anything in strace or ltrace.

This is a low priority issue because as far as I can tell, yum works just fine
if you have the patience to wait the total ten minutes every time.

Version-Release number of selected component (if applicable):
Fedora 7 stable.

# rpm -qa | grep yum
yum-3.2.2-1.fc7
yum-updatesd-3.2.2-1.fc7
yum-metadata-parser-1.1.0-2.fc7

How reproducible:
Install or erase any package. yum update also shows the same problem if packages
to update are available.

For example:

--

# yum erase apachetop
Setting up Remove Process
Resolving Dependencies
--> Running transaction check
---> Package apachetop.i386 0:0.12.6-2.fc6 set to be erased

Dependencies Resolved

=============================================================================
 Package                 Arch       Version          Repository        Size
=============================================================================
Removing:
 apachetop               i386       0.12.6-2.fc6     installed          55 k

Transaction Summary
=============================================================================
Install      0 Package(s)
Update       0 Package(s)
Remove       1 Package(s)

Is this ok [y/N]: y
Downloading Packages:
Running rpm_check_debug
--> Populating transaction set with selected packages. Please wait.
---> Package apachetop.i386 0:0.12.6-2.fc6 set to be erased
Running Transaction Test
[hang]
Finished Transaction Test
Transaction Test Succeeded
Running Transaction
[hang]
  Removing  : apachetop                    ######################### [1/1]

Removed: apachetop.i386 0:0.12.6-2.fc6
Complete!

--

# yum install apachetop
Setting up Install Process
Parsing package install arguments
Resolving Dependencies
--> Running transaction check
---> Package apachetop.i386 0:0.12.6-2.fc6 set to be updated

Dependencies Resolved

=============================================================================
 Package                 Arch       Version          Repository        Size
=============================================================================
Installing:
 apachetop               i386       0.12.6-2.fc6     fedora             32 k

Transaction Summary
=============================================================================
Install      1 Package(s)
Update       0 Package(s)
Remove       0 Package(s)

Total download size: 32 k
Is this ok [y/N]: y
Downloading Packages:
(1/1): apachetop-0.12.6-2 100% |=========================|  32 kB    00:00
Running rpm_check_debug
--> Populating transaction set with selected packages. Please wait.
---> Package apachetop.i386 0:0.12.6-2.fc6 set to be updated
Running Transaction Test
[hang]
Finished Transaction Test
Transaction Test Succeeded
Running Transaction
[hang]
  Installing: apachetop                    ######################### [1/1]

Installed: apachetop.i386 0:0.12.6-2.fc6
Complete!

--
  
Actual results:
On two points, yum hangs consuming 100% CPU. I marked this points with [hang] in
the dump.

Expected results:
Both the erase and the installation should complete in around 10-20 seconds as
they did before.
Comment 1 Seth Vidal 2007-08-29 15:06:22 EDT
Try something for me

shut down all processes related to yum or rpm.

then as root run:

rpm --rebuilddb

then re-run yum and see if you can make it hang like that.

Comment 2 Christian Mandery 2007-08-30 06:28:09 EDT
I stopped yum-updatesd and did not see anything in ps aux | grep yum and
executed rpm --rebuilddb but it did not solve the issue.

It still hangs some minutes.
Comment 3 Seth Vidal 2007-08-30 08:28:13 EDT
okay. a couple more things.

could you run yum -d 5 install something

and annotate where it hangs?
Comment 4 Christian Mandery 2007-08-30 08:50:36 EDT
Here's to output:

[root@ls3163 ~]# yum -d 5 install apachetop
Yum Version: 3.2.2
COMMAND: yum -d 5 install apachetop
Installroot: /
Ext Commands:

   apachetop
Setting up Install Process
Parsing package install arguments
Setting up Package Sacks
fedora                    100% |=========================| 2.1 kB    00:00
updates                   100% |=========================| 1.9 kB    00:00
Reading Local RPMDB
No other apachetop installed, adding to list for potential install
reduced installs :
   apachetop.i386 0:0.12.6-2.fc6
Building updates object
Resolving Dependencies
1188477705.93
--> Running transaction check
---> Package apachetop.i386 0:0.12.6-2.fc6 set to be updated
Checking deps for apachetop.i386 0-0.12.6-2.fc6 - u
looking for (u'libc.so.6(GLIBC_2.2)', None, (None, None, None)) as a requirement
of apachetop.i386 0-0.12.6-2.fc6 - u
looking for (u'libc.so.6(GLIBC_2.4)', None, (None, None, None)) as a requirement
of apachetop.i386 0-0.12.6-2.fc6 - u
looking for (u'libgcc_s.so.1', None, (None, None, None)) as a requirement of
apachetop.i386 0-0.12.6-2.fc6 - u
looking for (u'libc.so.6(GLIBC_2.1)', None, (None, None, None)) as a requirement
of apachetop.i386 0-0.12.6-2.fc6 - u
looking for (u'rtld(GNU_HASH)', None, (None, None, None)) as a requirement of
apachetop.i386 0-0.12.6-2.fc6 - u
looking for (u'libstdc++.so.6(CXXABI_1.3)', None, (None, None, None)) as a
requirement of apachetop.i386 0-0.12.6-2.fc6 - u
looking for (u'libncurses.so.5', None, (None, None, None)) as a requirement of
apachetop.i386 0-0.12.6-2.fc6 - u
looking for (u'libc.so.6', None, (None, None, None)) as a requirement of
apachetop.i386 0-0.12.6-2.fc6 - u
looking for (u'libgcc_s.so.1(GCC_3.0)', None, (None, None, None)) as a
requirement of apachetop.i386 0-0.12.6-2.fc6 - u
looking for (u'libc.so.6(GLIBC_2.0)', None, (None, None, None)) as a requirement
of apachetop.i386 0-0.12.6-2.fc6 - u
looking for (u'libstdc++.so.6', None, (None, None, None)) as a requirement of
apachetop.i386 0-0.12.6-2.fc6 - u
looking for (u'libc.so.6(GLIBC_2.3)', None, (None, None, None)) as a requirement
of apachetop.i386 0-0.12.6-2.fc6 - u
looking for (u'libreadline.so.5', None, (None, None, None)) as a requirement of
apachetop.i386 0-0.12.6-2.fc6 - u
looking for (u'libstdc++.so.6(GLIBCXX_3.4)', None, (None, None, None)) as a
requirement of apachetop.i386 0-0.12.6-2.fc6 - u
looking for (u'libm.so.6', None, (None, None, None)) as a requirement of
apachetop.i386 0-0.12.6-2.fc6 - u

Dependencies Resolved
1188477705.98

=============================================================================
 Package                 Arch       Version          Repository        Size
=============================================================================
Installing:
 apachetop               i386       0.12.6-2.fc6     fedora             32 k

Transaction Summary
=============================================================================
Install      1 Package(s)
Update       0 Package(s)
Remove       0 Package(s)

Total download size: 32 k
Is this ok [y/N]: y
Downloading Packages:
(1/1): apachetop-0.12.6-2 100% |=========================|  32 kB    00:00
Running rpm_check_debug
--> Populating transaction set with selected packages. Please wait.
Adding Package apachetop - 0.12.6-2.fc6.i386 in mode u
---> Package apachetop.i386 0:0.12.6-2.fc6 set to be updated
Running Transaction Test
Adding Package apachetop - 0.12.6-2.fc6.i386 in mode u
*** hang ***
Finished Transaction Test
Transaction Test Succeeded
Adding Package apachetop - 0.12.6-2.fc6.i386 in mode u
Running Transaction
*** hang ***
  Installing: apachetop                    ######################### [1/1]

Installed: apachetop.i386 0:0.12.6-2.fc6
Complete!


Additionally, I have taken a look into the execution with strace. As I already
said, during the hangs, nothing is shown by strace or ltrace but this are the
last syscalls made before the first hang (transaction test):

read(10, "lashplayer\\.so.*\t--\tuser_u:objec"..., 4096) = 4096
brk(0x1030d000)                         = 0x1030d000
brk(0x1032e000)                         = 0x1032e000
read(10, "em_u:object_r:home_root_t:s0\n/us"..., 4096) = 4096
brk(0x1034f000)                         = 0x1034f000
brk(0x10370000)                         = 0x10370000
read(10, "ns/nprhapengine\\.so.*\t--\tuser_u:"..., 4096) = 4096
brk(0x10391000)                         = 0x10391000
brk(0x103b2000)                         = 0x103b2000
read(10, "\n/usr/sap/Z96/[^/]*/\\.mozilla(/."..., 4096) = 4096
brk(0x103d3000)                         = 0x103d3000
read(10, "ZAW/lost\\+found/.*\t<<none>>\n/usr"..., 4096) = 4096
brk(0x103f4000)                         = 0x103f4000
brk(0x10415000)                         = 0x10415000
read(10, "ect_r:user_home_t:s0\n/usr/sap/ZD"..., 4096) = 4096
brk(0x10436000)                         = 0x10436000
brk(0x10457000)                         = 0x10457000
read(10, "))(/.+)?\tuser_u:object_r:httpd_u"..., 4096) = 4096
brk(0x10478000)                         = 0x10478000
read(10, "\t-d\tuser_u:object_r:user_home_di"..., 4096) = 4096
brk(0x10499000)                         = 0x10499000
brk(0x104ba000)                         = 0x104ba000
read(10, " user user_u\n#\n\n/usr/sap/ZYB/[^/"..., 4096) = 4096
brk(0x104db000)                         = 0x104db000
brk(0x104fc000)                         = 0x104fc000
read(10, "/]*/((www)|(web)|(public_html))("..., 4096) = 3208
brk(0x1051d000)                         = 0x1051d000
mmap2(NULL, 2072576, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) =
0xb72c1000
munmap(0xb74bb000, 2072576)             = 0

These system calls are exactly the same before the second "freeze".

Another information: Yum does not respond to Strg + C when hanging but Strg + Z
works.
Comment 5 Seth Vidal 2007-08-30 08:59:59 EDT
Do you have any extremely slow drives or network disks mounted on this system?
right now it looks like all the hang time is when yum hands off things to rpm.

also - ctrl-c not being able to help is correct when rpm is running the
transaction - we don't want it aborting in the middle of the transaction and
leaving you with a questionably useful system.

Comment 6 Christian Mandery 2007-08-30 10:24:47 EDT
I don't have slow drives but a really huge amount of NFS mounts that are not all
mounted but available via automount. I just tested and stopped the automounter
while all NFS dirs where unmounted but that didn't speed it up.
Comment 7 Seth Vidal 2007-08-30 10:34:49 EDT
Just b/c I'm out of other ideas:

1. go to /var/lib/rpm and remove any __db* files
2. what ver of rpm and what kernel ver is this?

thanks
Comment 8 Panu Matilainen 2007-08-30 10:52:27 EDT
Huge amount of NFS mounts (note that rpm will try to mount all configured mounts
when it runs) are one possibility, also things like having user accounts in huge
LDAP directory can cause significant "hangs".
Comment 9 Panu Matilainen 2007-08-30 10:54:05 EDT
Oh btw.. can you attach the full strace up to the point it hangs, including
forks and timestamps?
Comment 10 Christian Mandery 2007-08-30 11:06:24 EDT
I'm afraid deleting the __db* files didn't help.

rpm-4.4.2.1-1.fc7
rpm-libs-4.4.2.1-1.fc7

Linux ls3163.wdf.sap.corp 2.6.22.1-41.fc7 #1 SMP Fri Jul 27 18:10:34 EDT 2007
i686 i686 i386 GNU/Linux

And yes, we have a huge NIS with ~ 5k - 10k users but as I already stated above,
the hangups in yum only happen since two or three weeks ago while the NIS setup
didn't change over the whole server lifetime (around two month now).

I'll attach a full strace with forks and timestamps.
Comment 11 Christian Mandery 2007-08-30 11:43:12 EDT
Created attachment 181321 [details]
Output from strace -fFtt (20 MB unpacked!)
Comment 12 Panu Matilainen 2007-08-30 15:31:43 EDT
Ok, thanks. The pauses with no syscall activity are clearly there, now we just
need to figure out what it exactly is. I have my suspicions, but can you try one
more thing: try installing it with rpm itself with extra verbosity (rpm -Uvvh
apachetop*.rpm) and see if it hangs with that too and if so, where in the debug
spew the pause is.
Comment 13 Christian Mandery 2007-08-31 05:11:38 EDT
When running rpm -Uvvh [package], the problem occurs before any output is shown.
The duration of the pause seems the same one as one of those when running yum
(around two-three minutes). After this initial freeze, the whole rest of the rpm
installation (the complete output) works like a charm with normal speed in less
than one second.

I'll attach the strace output of the rpm run.






Comment 14 Christian Mandery 2007-08-31 05:13:55 EDT
Created attachment 182961 [details]
Strace output with -fFtt of a rpm run with the freeze in it
Comment 15 Panu Matilainen 2007-08-31 07:03:51 EDT
Thanks. That removes one suspect (fingerprinting) from the list. It's got
something to do with SELinux file context initialisation, but what exactly...
Comment 16 Christian Mandery 2007-08-31 08:07:37 EDT
SELinux is disabled on this system as far as possible.

If it helps, some weeks before we had problems with our setup here (because we
have so many NIS users) with the SELinux tool genhomedircon.

These problems are documented here:
https://bugzilla.redhat.com/show_bug.cgi?id=250233

But it cannot be exactly the same problem here because if it would do anything
with our NIS or NFS shares, we should see it in the strace...
Comment 17 Panu Matilainen 2007-08-31 08:59:51 EDT
Hmm.. there are a few suspicious spots in the selinux-patch that fedora
carries... I'll try to have a look next week, todays time is running out.
As a temporary workaround, --nocontexts switch might help with rpm itself but
there doesn't seem to be a way to set that from yum.

In any case not a yum bug so reassigning. 
Comment 18 Christian Mandery 2007-08-31 09:11:05 EDT
Yes, calling rpm with --nocontexts fixes the problem for me, so I'm glad we have
at least identified where the error stems from.

Perhaps you should consider that- independend from this bug- all SELinux-related
code shouldn't be run in yum/rpm if SELinux is disabled on the system.

Anyway, thanks for your help.
Comment 19 Christian Mandery 2007-09-04 04:29:40 EDT
Strange things are happening here. I made an identical setup of the system with
the problem as a backup machine. With identical I mean really identical: Same
packages installed, Winbind joined the same domain and same NIS server.

Anyway, on this second machine, the problem described above does not occur. So I
guess it cannot be NIS or the Active Directory. Maybe that helps you to track
down the bug.
Comment 20 Panu Matilainen 2007-09-11 04:34:56 EDT
My guess is that it's really just a missing SELinux enabled/disabled check in a
spot or two. Should be fixed in rpm.org HEAD, just needs backporting to 4.4.2.x
for fixing in Fedora.
Comment 21 Fedora Update System 2007-10-12 16:02:26 EDT
rpm-4.4.2.2-2.fc7 has been pushed to the Fedora 7 testing repository.  If problems still persist, please make note of it in this bug report.
 If you want to test the update, you can install it with 
 su -c 'yum --enablerepo=updates-testing update rpm'
Comment 22 Helge Deller 2007-10-15 08:13:19 EDT
As suggested, I just updated to rpm-4.4.2.2-2.fc7. Installing this updated RPM 
packages took around 5 minutes (so the bug was existent at that stage).

Directly aferwards I did a "yum update" and installed some more updates (incl. 
eclipse-cdt.i386 1:3.1.2-8.fc7 hplip.i386 0:1.7.4a-6.fc7 libsane-hpaio.i386 
0:1.7.4a-6.fc7). This update process finished in a few seconds.

So it seems this updated rpm package fixed this problem.
I think you can close this bug report.
THANKS!
Helge
Comment 23 Fedora Update System 2007-10-24 03:13:46 EDT
rpm-4.4.2.2-2.fc7 has been pushed to the Fedora 7 stable repository.  If problems still persist, please make note of it in this bug report.

Note You need to log in before you can comment on or make changes to this bug.