Red Hat Bugzilla – Bug 155730
(2.6.11-1.14_FC3 sparse file?) x86_64 shadow-utils fc3-update install broken
Last modified: 2007-11-30 17:11:04 EST
From Bugzilla Helper:
User-Agent: Mozilla/5.0 (X11; U; Linux x86_64; rv:1.7.3) Gecko/20041020 Firefox/0.10.1
Description of problem:
When trying to update shadow-utils-4.0.3-37 to shadow-utils-4.0.3-56 system eats up processing time and memory until the oom killer destroys pid. This is only a problem on our x86_64 machines. When using yum this appers to happen during the transaction tests. Up2date's progress dialog states "Testing package set / solving RPM inter-dependencies." An rpm install reports an out of memory condition if one has the patience to let it run for a looong while. /var/log/messages reports an out of memory condition for all theree methods.
Version-Release number of selected component (if applicable):
Steps to Reproduce:
1.yum update shadow-utils
3.rmp -ivh shadow-utils-4.0.3-56.x86_64.rpm
Actual Results: oom killer kills process, update not completed.
Expected Results: Update performed.
The package is somehow broken and nobody noticed since December? Odd. Similar
complaints in this thread.
Odd, that's the right word. I looked into it, here are the things I found:
- I had a old FC3 x86_64 install that was keept up2date with yum. shadow-utils
was at 4.0.3-56 and everything was working fine. I installed several new
packages in the last days since installing 2.6.11-1.14_FC3 and everything worked
- After downgrading to the release package 4.0.3-38 I tried to reupdate to
4.0.3-56 -- the problem described in this bugreport showed up
- Killed rpm; Tried to update from 4.0.3-38 to 4.0.3-40 -- worked fine
- Tried to update from 4.0.3-40 to 4.0.3-56 -- same problem as in this bug report
- Started with an older kernel (2.6.10-1.770_FC3) -- updateing shadow-utils to
4.0.3-56 works fine now
- Booted with 2.6.11-1.14_FC3 again and downgraded to 4.0.3-38; Tried updateing
to 4.0.3-56 while running strace. Last lines showed something with
/var/log/lastlog; Doing a $ ls -lh /var/log/lastlog showed
-r-------- 1 root root 1,2T 23. Apr 13:28 /var/log/lastlog
Whats that? Search, search, ahh: Sparse file, See Bug 138676 and Bug 134803;
Okay, I still find that odd. Did a : > /var/log/lastlog and rebooted; Now it's
19M. Tried updateing to shadow-utils (still running with kernel 2.6.11-1.14_FC3)
to 4.0.3-56 and it worked fine now.
Looking closer this seems to be part of the problem:
$ rpm -qp shadow-utils-4.0.3-56.x86_64.rpm --scripts
postinstall scriptlet (using /bin/sh):
chmod 0400 /var/log/lastlog
shadow-utils-4.0.3-40.x86_64.rpm has no such %pre, seems the reason why that
My current suspect: Maybe kernel 2.6.11-1.14_FC3 on x86_64 has problems with
*** Bug 155944 has been marked as a duplicate of this bug. ***
Why would this be a kernel problem? It's the rpm process that's trying to
allocate 1068Gb: the kernel does its best to appease it.
Changing component to rpm.
Created attachment 114094 [details]
an output from 'rpm -Fvvv shadow-utils*'
I bumped into the same problem. It actually happens with various kernels
(2.6.11-1.14_FC3 and some rawhide kernels, both SMP and UP). A weird thing
that in the meantime we run the same update on a number of x86_64 machines
and this is the first time I am aware the issue showed up. At least some
of these had what appears a very similar software configuration to the box
in question so why this showed up now is a mystery.
Attaching strace to a stuck process does not show anything. Not even
an unfished system call.
Attached are also results of 'SysRq-T' with 2.6.11-1.1284_FC4smp kernel
and results of running an update command under strace. Terminating
'killed by SIGKILL' comes from 'pkill -9 rpm' as this is the only way to
terminate that process.
Created attachment 114095 [details]
results of 'strace rpm -Fvvv shadow-utils-4.0.3-56.x86_64.rpm'
/me wonders if a update from and fresh FC3 (not updated) to FC4t3/FC4 will fail
due to this error... I'll try to check next week
Created attachment 114096 [details]
sysrq showing tasks with stuck rpm
I forgot to add. Booting with 'selinux=0' does not change anything.
(In reply to comment #7)
> /me wonders if a update from and fresh FC3 (not updated) to FC4t3/FC4 will fail
> due to this error... I'll try to check next week
I have just installed a fresh FC3 (x86_64) and ran into this problem.
I've just hit the same problem with 2.6.11-1.27_FC3. I installed a fresh FC3 and
then applied all the updates in small batches until I was left with perl and
shadow-utils. perl gave in to a --force but shadow-utils is buggered.
Someone else seems to have found the culprit: from speedyr6:
"Finally fixed by manually editing their Makefiles to point to the right mkinstalldirs that was in the root
of the source tree. I can't figure out for the life of me what these folks were trying to do with the logic
they were using to point to all of these different mkinstalldir files. It kept getting ../.././mkinstalldirs
and I just had to manually point it to $srcdir/mkinstalldirs."
Can someone send me the revised version. I don't know how to amend an rpm. Thanks.
Detailed instructions on diagnosis and workarounds at bugzilla #160125.
A test rpm will be available from updates-testing:
Is FC4 (4.4.1-21) affected also? I found no mention of dealing with sparse files
or lastlog in either CHANGES or %changelog.
4.4.x has this patch.
You have a simple upgrade that will crash your FC3 system.
How is this not a high severity bug?
You have a simple solution for a reproducible problem. Why does severity matter?
re comment #25 - I'm still awaiting feedback from the testing update I pushed -
so far I have recieved none. Positive feedback from testing will help expediate
the push to final updates.
yum --enablerepo=updates-testing update rpm
Nope, yum --enablerepo=updates-testing update rpm didn't do it. Still the same
hang at yum update or rpm -Uvh shadow-utils-4.0.3-56.x86_64.rpm. Looks like rpm
4.3.3-3.0.fc3.1 does not solve it.
Man, I thought I go crazy, brand new 64-bit server, new FC3 x86_64 installation
and doing the first set of updates crashes that thing!
rpm -q rpm
rpm -q rpm-libs
The fix really should work as it's what we are using in FC4 also. Can you
confirm with strace where it is hanging using rpm -Uvvh.
*** Bug 160125 has been marked as a duplicate of this bug. ***
Since there are insufficient details provided in this report for us to
investigate the issue further, and we have not received the feedback we
requested, we will assume the problem was not reproduceable or has been fixed in
a later update for this product. If you have further details, feel free to
reopen the report with the additional details attached.
Installing 'update-testing' rpm-4.3.3 does not help me, either.
BTW, syncing rpm and rpm-libs both to 4.3.3 breaks 'yum' completely :-(
Traceback (most recent call last):
File "/usr/bin/yum", line 6, in ?
File "/usr/share/yum-cli/yummain.py", line 23, in ?
File "__init__.py", line 21, in ?
ImportError: /usr/lib64/librpm-4.3.so: undefined symbol: rpm_execcon
My setup is:
Fresh FC3 install on a dual-Opteron machine plus kernel pre-updated (with
dependecies) to latest 2.6.12-1.1381_FC3, yum pre-updated to latest 2.2.2-0.fc3.
Now, "yum update" goes all the way through downloading headers and packages and
hangs at Running Transaction Test
'top' shows following line:
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
4153 root 25 0 1168g 3.5g 3.4g R 95.6 45.7 3:06.94 yum
I am not sure how the system allows a process to grow to 1168g on a 8GB memory +
16GB swap machine but maybe it is only the RESident 3.5g which matters.
Zeroing /var/log/lastlog helped but was described as "ugly workaround" so I
guess this bug is still unresolved.
Actually, it helped only in that the process did not hang. But it failed with
plenty of following messages:
file /..../.... from install of perl-5.8.5-20.FC3 conflicts with file from
I had to downgrade rpm-* to 4.3.2 (to make 'yum' work again), do 'yum remove
perl.i386' and retry.