Bug 191841

I suspect this may be limited to the i386 platform, I could not verify the issue with x86_64 ...

gathering additional information... stay tuned.

0- install U7
1- rpm -q pvm
pvm-3.4.4-22
2- run up2date to upgrade from U7 to U8
3- observe the cpio error described in comment #1 when up2date tries to upgrade
the pvm package
4- rpm -q pvm
package pvm is not instaled
5- rpm -qf /usr/share/pvm3/*
file /usr/share/pvm3/bin is not owned by any package
file /usr/share/pvm3/conf is not owned by any package
file /usr/share/pvm3/console is not owned by any package
...

Similar problem found when using up2date.  Interrupted updates.  
I think because updates did not complete entirely, the ssh daemon did not
restart.   

(In reply to comment #4)
> Similar problem found when using up2date.  Interrupted updates.  
> I think because updates did not complete entirely, the ssh daemon did not
> restart.   

Ditto, only NFS did not restart either. 
So with NFS and SSH not restarting it ummm took away the jobs of the network. 

(In reply to comment #5)
> (In reply to comment #4)
> > Similar problem found when using up2date.  Interrupted updates.  
> > I think because updates did not complete entirely, the ssh daemon did not
> > restart.   
> 
> Ditto, only NFS did not restart either. 
> So with NFS and SSH not restarting it ummm took away the jobs of the network. 
> 
> 

I should restate, when pvm failed up2date stops, leaving incomplete packages and
then rendering things useless. Such as NFS not restarting, SSH not restarting,
possibly due to the initscripts package never installing after the failure. 

This bugreport is known since 2006-05-15.

RHEL3 U8 comes with this problem. I do not understand Red Hat.

I install U8 with up2date and LANG=de_DE. It reports in shell:

sshd stoppen:[  OK  ]
sshd starten:[  OK  ]
error: unpacking of archive failed on file /usr/share/pvm3/bin/LINUXI386: cpio:
rename
Fehler beim Lesen der Informationen über den Dienst keytable: Datei oder
Verzeichnis nicht gefunden
error: %trigger(kbd-1.08-10.2) scriptlet failed, exit status 1
Shutting down smartd: [FEHLGESCHLAGEN]
NFS mountd herunterfahren:[  OK  ]
NFS-Daemon herunterfahren: [  OK  ]
Shutting down NFS quotas: [  OK  ]
NFS-Dienste herunterfahren:  [  OK  ]
NFS statd stoppen: [  OK  ]
groupdel : Gruppe rpm existiert nicht

SSH was restarted but NFS not if it was activ before.
The same situation over rhnsd.

I've totally fucked two remote systems applying update 8 via up2date. 
Obviously, Red Hat can't be bothered to test their updates before releasing them
on their paying customers.  This may be the final straw for me and my clients
with RHEL.  Once I assess the total costs of the damage caused, I'll carefully
consider a lawsuit.

At reboot I was seen the result of the problem with NFS in comment #7.
There was not longer service nfs and nfslock available.
I fixed it with:
chkconfig --add nfs
chkconfig --add nfslock

I can not tell you, if there also other problems with the system I do not see
in this moment.

If you up2date your system without the pvm errata then there are no problems.

Just a "me too". This bug has affected several (100s?) of our systems all over
the university campus. It'll take time and effort to fix, since is has to be
done manually via a local console. Also we experienced that not only sshd and
nfs stops, but crond dies as well, and the RPM db states that vixie-cron isn't
installed. With that in mind, this bug should be elevated to a security priority,
as it clearly has security implications, but that depends on how dependent you
are of crond. YMMV.

*** Bug 199789 has been marked as a duplicate of this bug. ***

What is present in this case is less of an issue with the PVM failing to unpack
and more of an issue with up2date and what happens if an update fails and halts
the up2date process, leaving packages uninstalled. 

I had reported a similiar case in August of 2004 with other packages that failed
causing up2date to halt its progress and the result was similar, packages not
being installed and services being affected. 

This was reported as bug 129294

In that case I was told that I would be better off going through RedHat support. 

What would be nice would be for RH to take responsibility for the problem and
display some common courtesy to their customers and work on the root problem
itself (the up2date issue.)

It is relatively trivial. If PVM fails to unpack (or any package for that
matter) continue with the up2date process and finish what was started. 

Unfortunately as was the case this time around, with NFS and SSH being affected,
remote machines become useless and productivity suffers. 

When shelling out the dollars for the updates it is expected for them to work
properly. If there is a problem and the problem is addressed this can be
accepted. When there is a problem and it sits idle for months and in the end
affects customers machines then it is completely unacceptable. 

Created attachment 132934 [details]
44 key rpm's lost when up2date crashed

I'll heartily endorse the problem of bug handling in up2date.  I'm attaching
the
list of the 44 rpm's which were left uninstalled when up2date crashed applying
update 8 due to this pvm bug.  With rpm, openssh, samba, vsftp, vixie-cron,
XFree86, sysreport, nfs-utils, bind all gone, I think you will agree that
the situation is Very Bad.  Particuarly since it takes console access to
fix it.

Profuse apologies for this mess .

This is now fixed with pvm-3.4.5-8_EL3, which I'll try to get pushed through
FastTrack ASAP.

Can you advise how we could pursue having up2date fixed to allow for
error-checking rather than having it just shut down and fail to install
remaining packages. 
Who do we contact, how do we go about contacting them. 

I think that given this prime example, the up2date service NEEDS to have the
error checking improved to prevent further similar issues. 

Unfortunately this fix is a bit late and for the most part the damage has been
done. 

(In reply to comment #15)
> This is now fixed with pvm-3.4.5-8_EL3, which I'll try to get pushed through
> FastTrack ASAP.

We need pvm-3.4.5-8_EL3 as fix for RHBA-2006:0320-5 in the base channels.
And please remove pvm-3.4.5-6_EL3 as one of available version of pvm from
Red Hat Network. pvm-3.4.5-6_EL3 was enough to do damage.

The new installations CDs which comes with U8 also contain the pvm-3.4.5-6_EL3.
I can not tell you if the pvm-3.4.5-6_EL3 also fails on a fresh installation.
It would be better to have also new installations CDs with pvm-3.4.5-8_EL3.

I am not Red Hat but this would be the task for me in the next hours today.

pvm-3.4.5-6_EL3 does not fail on a fresh install, just the update.  It's obvious
that Red Hat couldn't be bothered to do the obvious "no-brainer" check and
actually attempt to install update 8 with up2date on at least one "everything
installed" update 7 system before releasing it on a trusting, paying clientele.
 The iso images are dated the 12th and the update released on the 20th, so it
wasn't like it would have taken an inordinant amount of time or effort for a
lowly RH engineer to perform the obvious test update before releasing it!! 
There wasn't even an update 8 beta like there was for update 7!!!

Ubuntu 6.06 LTS is looking mighty attractive for the price...

Ditto on these problems. I have various and sundry screwups on my systems that
tried to install this patch in scheduled updates--sshd chkconfiged off, services
not running. I will never be certain things are completely back to normal on all
the systems, since I don't know everything that was broken.

This is really inexcusable.

An advisory has been issued which should help the problem
described in this bug report. This report is therefore being
closed with a resolution of ERRATA. For more information
on the solution and/or where to find the updated files,
please follow the link below. You may reopen this bug report
if the solution does not work for you.

http://rhn.redhat.com/errata/RHBA-2006-0607.html

IMHO the most important thing to fix here is rpm. Just yesterday I rebuilt my
own postfix package and had a similar problem ( cpio: rename failed.. because
the new package had a symlink to a directory instead of the directory itself).
While up2date should be fixed for such situations, fixing the real root of the
problem should be considered too.

*** Bug 200267 has been marked as a duplicate of this bug. ***

This message is in reference to Red Hat Bugzilla 191841. The synopsis of this
bug is that a package on RHN failed to install properly via
up2date. Depending on how users run up2date, this can leave systems in
an inconsistent state.

In the spirit of transparency, included in this post are the following
items:

1) An overview of what our current test plan calls for relative to
up2date package testing.
2) What went wrong in this case.
3) What we are doing to keep the problem from occurring again.



1- Current test plan

a) Each package is tested individually, to verify that it can be updated
from the previous RHEL update.
b) We test that a virgin RHEL X.uY release can be completely updated to
a RHEL X.uY+1 via up2date.
c) Developers must execute a static analysis tool on each package that
is going to be shipped via RHN. This tool often uncovers and helps us
resolve problems with packages that cause up2date to fail.

I have been at Red Hat over 18 months. The above test plan steps have
caught numerous upgrade issue like this bug before they could ever get
close to making it out of the building. In addition, as far as I know
this is the  first problem of this nature that has made it out of the
building in recent years.

2 - What went wrong in this case.

-We got conflicting test results from items 1a and 1b.

-What created some confusion for the testing organization was that the pvm
Errata had shipped as a FastTrack errata in early May and no external parties
were reporting this bug and we had positive up2date testing results for this
Errata. We made a judgment call that the tests from step 1a were the correct
results and the tests from step 1b were incorrect.

-The conclusion that the test results from step 1a were correct and step
1b were not correct was a failure in judgment on our part. I understand
that it appears obvious that we should have drilled down on the failures
as  result of executing  step 1b. We are tightening up our release
criteria and test plans to make sure this does not happen again.


3 - Here is what we are doing to fix things going forward:

-We are putting a check in the tool mentioned in Step 1c that will catch the
specific issue with this rpm.
-No package will be released with out positive results from both steps
1a and 1b.

-We are fixing the issues in step 1b that resulted in the false positive
result. Specifically that the system was not a completely virgin U7
system when the test was performed. It had some additional packages on it.



This message is in reference to Red Hat Bugzilla 191841. The synopsis of this
bug is that a package on RHN failed to install properly via up2date. Depending
on how users run up2date, this can leave systems in an inconsistent state.

In the spirit of transparency, included in this post are the following
items:

1) An overview of what our current test plan calls for relative to up2date
package testing.
2) What went wrong in this case.
3) What we are doing to keep the problem from occurring again.


1- Current test plan

a) Each package is tested individually, to verify that it can be updated
from the previous RHEL update.
b) We test that a virgin RHEL X.uY release can be completely updated to a RHEL
X.uY+1 via up2date.
c) Developers must execute a static analysis tool on each package that is going
to be shipped via RHN. This tool often uncovers and helps us resolve problems
with packages that cause up2date to fail.

I have been at Red Hat over 18 months. The above test plan steps have caught
numerous upgrade issue like this bug before they could ever get close to making
it out of the building. In addition, as far as I know this is the  first problem
of this nature that has made it out of the building in recent years.

2 - What went wrong in this case.


-We got conflicting test results from items 1a and 1b.

-What created some confusion for the testing organization was that the
pvm errata shipped as a FastTrack errata in early May and we had
positive up2date testing results for this Errata. We made a judgment call that
the tests from step 1a were the correct results and the tests from step 1b were
 incorrect.

-The conclusion that the test results from step 1a were correct and step
1b were not correct was a failure in judgment on our part. I understand
that it appears obvious that we should have drilled down on the failures
as  result of executing  step 1b. We are tightening up our release
criteria and test plans to make sure this does not happen again.


3 - Here is what we are doing to fix things going forward:

-We are putting a check in the tool mentioned in Step 1c that will catch
the specific issue with this rpm.
-No package will be released with out positive results from both steps
1a and 1b.
-We are fixing the issues in step 1b that resulted in the false positive
result. Specifically that the system was not a completely virgin U7
system when the test was performed. It had some additional packages on it from a
prior test.

(In reply to comment #31)

> 3 - Here is what we are doing to fix things going forward:
> 
> -We are putting a check in the tool mentioned in Step 1c that will catch the
> specific issue with this rpm.
> -No package will be released with out positive results from both steps
> 1a and 1b.
> 
> -We are fixing the issues in step 1b that resulted in the false positive
> result. Specifically that the system was not a completely virgin U7
> system when the test was performed. It had some additional packages on it.

Will the systems damaged by this bug be fixed via up2date?

Will fault-tolerance be incorporated into up2date? An RPM can
be fixed, and more tests can be done, but if there's a power
failure or a connectivity problem while up2date is running
it would be nice for it to recover gracefully (or at least it
should be able to protect itself from faulty RPMs by rolling
back to a former, working state instead of just leaving a
crippled system).

Internal Status set to 'Resolved'
Status set to: Closed by Client
Resolution set to: 'Closed by Client'

This event sent from IssueTracker by Uwe Beck 
 issue 98473