From Bugzilla Helper: User-Agent: Mozilla/5.0 (X11; U; Linux x86_64; rv:1.7.3) Gecko/20041020 Firefox/0.10.1 Description of problem: When trying to update shadow-utils-4.0.3-37 to shadow-utils-4.0.3-56 system eats up processing time and memory until the oom killer destroys pid. This is only a problem on our x86_64 machines. When using yum this appers to happen during the transaction tests. Up2date's progress dialog states "Testing package set / solving RPM inter-dependencies." An rpm install reports an out of memory condition if one has the patience to let it run for a looong while. /var/log/messages reports an out of memory condition for all theree methods. Version-Release number of selected component (if applicable): 4.0.3-56 How reproducible: Always Steps to Reproduce: 1.yum update shadow-utils 2.up2date 3.rmp -ivh shadow-utils-4.0.3-56.x86_64.rpm Actual Results: oom killer kills process, update not completed. Expected Results: Update performed. Additional info:
http://forums.fedoraforum.org/showthread.php?t=51783 The package is somehow broken and nobody noticed since December? Odd. Similar complaints in this thread.
Odd, that's the right word. I looked into it, here are the things I found: - I had a old FC3 x86_64 install that was keept up2date with yum. shadow-utils was at 4.0.3-56 and everything was working fine. I installed several new packages in the last days since installing 2.6.11-1.14_FC3 and everything worked fine - After downgrading to the release package 4.0.3-38 I tried to reupdate to 4.0.3-56 -- the problem described in this bugreport showed up - Killed rpm; Tried to update from 4.0.3-38 to 4.0.3-40 -- worked fine - Tried to update from 4.0.3-40 to 4.0.3-56 -- same problem as in this bug report - Started with an older kernel (2.6.10-1.770_FC3) -- updateing shadow-utils to 4.0.3-56 works fine now - Booted with 2.6.11-1.14_FC3 again and downgraded to 4.0.3-38; Tried updateing to 4.0.3-56 while running strace. Last lines showed something with /var/log/lastlog; Doing a $ ls -lh /var/log/lastlog showed -r-------- 1 root root 1,2T 23. Apr 13:28 /var/log/lastlog Whats that? Search, search, ahh: Sparse file, See Bug 138676 and Bug 134803; Okay, I still find that odd. Did a : > /var/log/lastlog and rebooted; Now it's 19M. Tried updateing to shadow-utils (still running with kernel 2.6.11-1.14_FC3) to 4.0.3-56 and it worked fine now. Looking closer this seems to be part of the problem: $ rpm -qp shadow-utils-4.0.3-56.x86_64.rpm --scripts postinstall scriptlet (using /bin/sh): touch /var/log/lastlog chmod 0400 /var/log/lastlog shadow-utils-4.0.3-40.x86_64.rpm has no such %pre, seems the reason why that update worked. My current suspect: Maybe kernel 2.6.11-1.14_FC3 on x86_64 has problems with sparse files?
*** Bug 155944 has been marked as a duplicate of this bug. ***
Why would this be a kernel problem? It's the rpm process that's trying to allocate 1068Gb: the kernel does its best to appease it. Changing component to rpm.
Created attachment 114094 [details] an output from 'rpm -Fvvv shadow-utils*' I bumped into the same problem. It actually happens with various kernels (2.6.11-1.14_FC3 and some rawhide kernels, both SMP and UP). A weird thing that in the meantime we run the same update on a number of x86_64 machines and this is the first time I am aware the issue showed up. At least some of these had what appears a very similar software configuration to the box in question so why this showed up now is a mystery. Attaching strace to a stuck process does not show anything. Not even an unfished system call. Attached are also results of 'SysRq-T' with 2.6.11-1.1284_FC4smp kernel and results of running an update command under strace. Terminating 'killed by SIGKILL' comes from 'pkill -9 rpm' as this is the only way to terminate that process.
Created attachment 114095 [details] results of 'strace rpm -Fvvv shadow-utils-4.0.3-56.x86_64.rpm'
/me wonders if a update from and fresh FC3 (not updated) to FC4t3/FC4 will fail due to this error... I'll try to check next week
Created attachment 114096 [details] sysrq showing tasks with stuck rpm
I forgot to add. Booting with 'selinux=0' does not change anything.
(In reply to comment #7) > /me wonders if a update from and fresh FC3 (not updated) to FC4t3/FC4 will fail > due to this error... I'll try to check next week I have just installed a fresh FC3 (x86_64) and ran into this problem.
I've just hit the same problem with 2.6.11-1.27_FC3. I installed a fresh FC3 and then applied all the updates in small batches until I was left with perl and shadow-utils. perl gave in to a --force but shadow-utils is buggered.
Someone else seems to have found the culprit: from speedyr6: "Finally fixed by manually editing their Makefiles to point to the right mkinstalldirs that was in the root of the source tree. I can't figure out for the life of me what these folks were trying to do with the logic they were using to point to all of these different mkinstalldir files. It kept getting ../.././mkinstalldirs and I just had to manually point it to $srcdir/mkinstalldirs." -Robert
Can someone send me the revised version. I don't know how to amend an rpm. Thanks.
Detailed instructions on diagnosis and workarounds at bugzilla #160125.
A test rpm will be available from updates-testing: rpm-4.3.3-3.0.fc3.
Is FC4 (4.4.1-21) affected also? I found no mention of dealing with sparse files or lastlog in either CHANGES or %changelog.
4.4.x has this patch.
You have a simple upgrade that will crash your FC3 system. How is this not a high severity bug?
You have a simple solution for a reproducible problem. Why does severity matter?
re comment #25 - I'm still awaiting feedback from the testing update I pushed - so far I have recieved none. Positive feedback from testing will help expediate the push to final updates. yum --enablerepo=updates-testing update rpm
Nope, yum --enablerepo=updates-testing update rpm didn't do it. Still the same hang at yum update or rpm -Uvh shadow-utils-4.0.3-56.x86_64.rpm. Looks like rpm 4.3.3-3.0.fc3.1 does not solve it. Man, I thought I go crazy, brand new 64-bit server, new FC3 x86_64 installation and doing the first set of updates crashes that thing!
rpm -q rpm rpm -q rpm-libs The fix really should work as it's what we are using in FC4 also. Can you confirm with strace where it is hanging using rpm -Uvvh.
*** Bug 160125 has been marked as a duplicate of this bug. ***
Since there are insufficient details provided in this report for us to investigate the issue further, and we have not received the feedback we requested, we will assume the problem was not reproduceable or has been fixed in a later update for this product. If you have further details, feel free to reopen the report with the additional details attached.
Installing 'update-testing' rpm-4.3.3 does not help me, either. BTW, syncing rpm and rpm-libs both to 4.3.3 breaks 'yum' completely :-( >yum update Traceback (most recent call last): File "/usr/bin/yum", line 6, in ? import yummain File "/usr/share/yum-cli/yummain.py", line 23, in ? import yum File "__init__.py", line 21, in ? ImportError: /usr/lib64/librpm-4.3.so: undefined symbol: rpm_execcon My setup is: Fresh FC3 install on a dual-Opteron machine plus kernel pre-updated (with dependecies) to latest 2.6.12-1.1381_FC3, yum pre-updated to latest 2.2.2-0.fc3. Now, "yum update" goes all the way through downloading headers and packages and hangs at Running Transaction Test 'top' shows following line: PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 4153 root 25 0 1168g 3.5g 3.4g R 95.6 45.7 3:06.94 yum I am not sure how the system allows a process to grow to 1168g on a 8GB memory + 16GB swap machine but maybe it is only the RESident 3.5g which matters. Zeroing /var/log/lastlog helped but was described as "ugly workaround" so I guess this bug is still unresolved. Actually, it helped only in that the process did not hang. But it failed with plenty of following messages: file /..../.... from install of perl-5.8.5-20.FC3 conflicts with file from package perl-5.8.5-9 I had to downgrade rpm-* to 4.3.2 (to make 'yum' work again), do 'yum remove perl.i386' and retry. regards, Michal.