Bug 128622 - bi-arch packages cause unnecessary .rpmnew and .rpmsave files
bi-arch packages cause unnecessary .rpmnew and .rpmsave files
Status: CLOSED RAWHIDE
Product: Fedora
Classification: Fedora
Component: rpm (Show other bugs)
rawhide
x86_64 Linux
medium Severity medium
: ---
: ---
Assigned To: Panu Matilainen
: Reopened
: 144747 246236 254721 (view as bug list)
Depends On:
Blocks: 29470 426258
  Show dependency treegraph
 
Reported: 2004-07-27 06:36 EDT by Thomas Zehetbauer
Modified: 2009-02-19 06:04 EST (History)
15 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2007-08-09 09:34:34 EDT
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
perl script to remove duplicates (1.20 KB, application/octet-stream)
2005-10-14 07:43 EDT, Thomas Zehetbauer
no flags Details
Proposed patch (3.21 KB, patch)
2007-02-13 09:50 EST, Tomas Mraz
no flags Details | Diff

  None (edit)
Description Thomas Zehetbauer 2004-07-27 06:36:02 EDT
When upgrading packages which are installed for both the 32- and
64-bit architecture, rpm creates unnecessary .rpmsave and .rpmnew files:

rpm -q --queryformat '%{NAME}-%{VERSION}-%{RELEASE}.%{ARCH}\n' pam
pam-0.77-50.x86_64
pam-0.77-50.i386

warning: /etc/pam.d/other created as /etc/pam.d/other.rpmnew
warning: /etc/security/access.conf created as
/etc/security/access.conf.rpmnew
warning: /etc/security/chroot.conf created as
/etc/security/chroot.conf.rpmnew
warning: /etc/security/console.perms created as
/etc/security/console.perms.rpmnew
warning: /etc/security/group.conf created as
/etc/security/group.conf.rpmnew
warning: /etc/security/limits.conf created as
/etc/security/limits.conf.rpmnew
warning: /etc/security/pam_env.conf created as
/etc/security/pam_env.conf.rpmnew
warning: /etc/security/time.conf created as /etc/security/time.conf.rpmnew

All these files were identical.
Comment 1 Jeff Johnson 2004-08-03 10:02:21 EDT
rpm does exactly what it has always done, and must continue to do
so. That is not "unnecessary".

The only difference with elf32 and elf64 pkgs installed is
that path collisions are more likely, and so you are seeing
*.rpmnew files created more often.
Comment 2 Thomas Zehetbauer 2004-08-06 06:30:35 EDT
You are almost right, this is what rpm has always done and what was
appropiate for single architecture systems. But now that we have
x86_64 support this behaviour has become an unnecessary annoyance. The
alternative to fixing this in rpm would be to change every i386
package built for x86_64 to not include these files, but do we really
want this?
Comment 3 Jeff Johnson 2004-08-06 09:01:49 EDT
I cannot change the existing behavior in rpm without violating
some expectations.

Nor is there any clearly defined algotithm to improve the
existing behavior when given identical packages
for elf32/elf64 to be installed.

So, WONTFIX, but that does not mean that I disagree
with "unnecessary annoyance".
Comment 4 Thomas Zehetbauer 2004-08-06 14:59:28 EDT
Fine, so I guess this needs to be brought before the steering
committee, how do we proceed?
Comment 5 Tomas Mraz 2005-08-05 08:51:00 EDT
Is it really necessary to create .rpmnew files when only timestamp differs?
Comment 6 Tomas Mraz 2005-08-05 08:54:47 EDT
*** Bug 144747 has been marked as a duplicate of this bug. ***
Comment 7 Dan Hollis 2005-10-14 07:37:46 EDT
this is silly.

/etc/cron.daily/yum.cron:

warning: /etc/pki/tls/certs/ca-bundle.crt created as
/etc/pki/tls/certs/ca-bundle.crt.rpmnew
warning: /etc/pki/tls/openssl.cnf created as /etc/pki/tls/openssl.cnf.rpmnew

# ls -la
total 20
drwxr-xr-x  5 root root  248 Oct 14 04:09 .
drwxr-xr-x  6 root root  144 Sep 14 23:13 ..
lrwxrwxrwx  1 root root   19 Oct 14 04:09 cert.pem -> certs/ca-bundle.crt
drwxr-xr-x  2 root root  272 Oct 14 04:09 certs
drwxr-xr-x  2 root root  192 Oct 14 04:10 misc
-rw-r--r--  1 root root  660 Sep 20 20:26 new.cert.csr
-rw-r--r--  1 root root 7965 Oct 12 03:22 openssl.cnf
-rw-r--r--  1 root root 7965 Oct 12 03:22 openssl.cnf.rpmnew

# md5sum openssl.cnf*
ef0fc8ad519bb9ed9eb7456cb25fa486  openssl.cnf
ef0fc8ad519bb9ed9eb7456cb25fa486  openssl.cnf.rpmnew

# ls -al
total 860
drwxr-xr-x  2 root root    272 Oct 14 04:09 .
drwxr-xr-x  5 root root    208 Oct 14 04:31 ..
-rw-r--r--  1 root root 427833 Oct 12 03:22 ca-bundle.crt
-rw-r--r--  1 root root 427833 Oct 12 03:22 ca-bundle.crt.rpmnew

# md5sum ca-bundle.crt*
7c14b8d716016ef958cebe56c1264b3f  ca-bundle.crt
7c14b8d716016ef958cebe56c1264b3f  ca-bundle.crt.rpmnew
Comment 8 Thomas Zehetbauer 2005-10-14 07:43:38 EDT
Created attachment 119969 [details]
perl script to remove duplicates

I have created a small perl script to erase unnecessary (identical)
*.rpm(new|save) files I have yet encountered.
Comment 9 Jeff Johnson 2006-04-05 19:17:44 EDT
work around script WORKSFORME
Comment 10 Orion Poplawski 2007-02-13 00:15:58 EST
Reopening this hoping that with new RPM maintainer and direction that this will
be addressed.
Comment 11 Tomas Mraz 2007-02-13 09:50:25 EST
Created attachment 147987 [details]
Proposed patch

This patch will prevent RPM to create .rpmnew and .rpmsave files when
file/symlink on disk differs just by timestamp.
Comment 12 Axel Thimm 2007-02-13 10:32:43 EST
This would still leave rpm -V output non-quiet.

How about instead of modifying runtime behaviour to implicitely assume a
%verify(not mtime) at rpmbuild time when encountering %config tags?

No conflicts, rpm -V would be silent, too and even an unpatched rpm would play
along. The latter is interesting for more conservative distributions like RHEL,
which could live with patching rpmbuild, but less with touching rpm/rpmlibs that
are being (or already have been) shipped.
Comment 13 Tomas Mraz 2007-02-13 10:40:17 EST
%verify(not mtime) is AFAIK not consulted in case of rpm package install.
So the .rpmnew .rpmsave files would be created if the patch above is not applied.
Comment 14 Aleksey Nogin 2007-02-13 12:07:27 EST
According to Tomas, his patch (attachment 147987 [details]) should also fix bug 29470,
marking it as a dependency.
Comment 15 Jeff Johnson 2007-02-13 12:22:57 EST
%verify(not mtime) controls what parameters are to be verified and should not be abused
for %config(...) handling. A different mechanism should be used, and it should be end-user,
not package builder, configurable.
Comment 16 Tomas Mraz 2007-02-13 12:27:57 EST
I agree with comment #15.
Comment 17 Axel Thimm 2007-02-13 13:18:09 EST
(In reply to comment #13)
> %verify(not mtime) is AFAIK not consulted in case of rpm package install.
> So the .rpmnew .rpmsave files would be created if the patch above is not applied.

It is, I just tried it and it works:



Name:           test
Version:        1
Release:        1
Summary:        test
Group:          somegroup
License:        none
#BuildRoot:     %{_tmppath}/%{name}-%{version}-%{release}-root
BuildRoot:      %(mktemp -d %{_tmppath}/%{name}-%{version}-XXXX)
%description
test

%prep
echo %{buildroot}

%build
echo %{buildroot}

%install
rm -rf %{buildroot}
mkdir -p %{buildroot}/etc/test-config
echo Test > %{buildroot}/etc/test-config/configfile

%clean
rm -rf %{buildroot}


%files
%defattr(-,root,root,-)
%dir /etc/test-config
%config %verify(not mtime) /etc/test-config/configfile
~
Comment 18 Orion Poplawski 2007-02-13 13:33:14 EST
(In reply to comment #17)
> %files
> %defattr(-,root,root,-)
> %dir /etc/test-config
> %config %verify(not mtime) /etc/test-config/configfile

Really?  More cruft we need to add to every spec file?  Let's get rid of this
kind of stuff, not add more.
Comment 19 Jeff Johnson 2007-02-13 13:40:49 EST
No, all files are going to need a %dist attribute so that file conflicts between, say, fc6 and fc7
can be automagically resolved to prefer "fc7".

So more gunk, not less, is what is needed!
Comment 20 Axel Thimm 2007-02-13 13:48:26 EST
(In reply to comment #15)
> %verify(not mtime) controls what parameters are to be verified and should not
be abused
> for %config(...) handling. A different mechanism should be used, and it should
be end-user,
> not package builder, configurable.

I think %config files should default to "not mtime" regardless of multilib. If
the packager knows that the timestamp matters he can turn the verify bit on
again for this config file.

Also the decision whether a timestamp of a config file should be part of
verification/conflict checking should be primarily at the packager's hands and
thus part of the resulting build - especially if there are any config files
where the timestamp does matter and the packager chose to turn timestamp
verification on.

That is not to say that having an option for the user to override this behaviour
is bad.
Comment 21 Axel Thimm 2007-02-13 13:53:55 EST
(In reply to comment #18)
> (In reply to comment #17)
> > %files
> > %defattr(-,root,root,-)
> > %dir /etc/test-config
> > %config %verify(not mtime) /etc/test-config/configfile
> 
> Really?  More cruft we need to add to every spec file?  Let's get rid of this
> kind of stuff, not add more.

I wasn't suggesting to add this to the specfiles, at all, you misread the above.
Check comment #12 (especially the word "implicitely" ;).

Comment #17 was there only to demonstrate that conflict resolution in rpm does
take into account the verify flags set at rpmbuild time, nothing more. Next step
would be to have %config imply a default of %verify(not mtime) so no specfile
needs to be touched.
Comment 22 Jeff Johnson 2007-02-13 14:06:46 EST
Make sure that %config implies %{dist} as well. File conflicts need to be resolved, not just detected.
Comment 23 D. Hugh Redelmeier 2007-02-13 16:23:36 EST
We can (and should) debate what the right fix is.  Maybe it would be useful to
mitigate the problem while we wait for the utlimate solution.  I've had to live
with this problem for two and a half years already.

The current diagnostic could be refined to report what differences were
encountered, perhaps in the way verify does.  If I could read in the message
that only the timestamp differed, I could decide to ignore the message.  As it
is now, I feel I should diff each file that is reported.


The fundamental problem arises when two binary RPMs are really the same package,
just for different architectures.  RPM has no convenient way of modelling the
sharing involved.  There is an inconvenient way: add a noarch package for the
shared portion.  Perhaps there is a way to automate the inconvenient way so as
to make it convenient.

If we don't address the fundamental problem, there is a good chance that the fix
is just a cover-up.  It may turn out to suppress diagnostics for a real case
that deserves diagnostics.  For example, if I manually changed a config file,
and the result had the same text as the config file from the next version, I'd
like a diagnostic.
Comment 24 D. Hugh Redelmeier 2007-02-13 16:27:07 EST
Another bug that reflects bad modelling of sharing between packages for
different architectures: https://bugzilla.redhat.com/bugzilla/show_bug.cgi?id=209306
Comment 25 Jeff Johnson 2007-02-13 17:07:33 EST
The definintion of "sharing" within rpm is quite clear, the items that are compared
have not changed for years.

What is different is that some of the items, like mtime, can and will differ from builds
on different machines. Even content can differ on different builds.

But distributing configuration, the issue w %config, from two packages that are supposedly,
but not actually, the same leads to *.rpmnew litter unsurprisingly.

This is a design flaw, not otherwise. You are quite correct.
Comment 26 Philip Prindeville 2007-02-13 20:13:11 EST
(In reply to comment #23)
> We can (and should) debate what the right fix is.  Maybe it would be useful to
> mitigate the problem while we wait for the utlimate solution.  I've had to live
> with this problem for two and a half years already.
> 
> [...]
> The fundamental problem arises when two binary RPMs are really the same package,
> just for different architectures.  RPM has no convenient way of modelling the
> sharing involved.

That's a valid concern.  I think that the issue of "fat binaries" is perhaps the
most glaring instance of this problem:  it might be possible in the future to
have a single binary that is .i586, .i686, .x86_64, and .emt64 (or whatever the
64-bit Intel architecture is called).  In such instances, it will often be the
degenerative case of all 4 architectures (or compiler settings) sharing a common
config file.
Comment 27 Jeff Johnson 2007-02-17 17:13:07 EST
FAT binaries already exist on Mac OS X, so does rpm.

Meanwhile, arch is unlikely the correct attribute to model config files.

ELF32/ELF64 executables/libraries within rpm is already ok, its the "other"" files that are causing 
problems.
Comment 28 Philip Prindeville 2007-02-17 21:48:29 EST
(In reply to comment #27)
> FAT binaries already exist on Mac OS X, so does rpm.
> 
> Meanwhile, arch is unlikely the correct attribute to model config files.
> 
> ELF32/ELF64 executables/libraries within rpm is already ok, its the "other""
files that are causing 
> problems.

That wasn't really my point.  I'm suggesting that if you had a single RPM with
all x86 binaries, then at install time you could specify that you wanted i386
and x86_64 libraries... and still have a single config file be installed because
that's all the RPM would need to contain... thus indirectly eliminating the case
of having two identical config files with different timestamps.

Comment 29 Jeff Johnson 2007-02-18 00:24:53 EST
There are practical difficulties compiling elf32 and elf64 for inclusion
in the same package with shared config.

The likelier solution is splitting config out into another common package, leaving
elf32 and elf64 as is.
Comment 30 Jeff Johnson 2007-02-19 19:54:56 EST
FWIW, the patch in comment #11 is in rpm-4.4.8-1.
Comment 31 Jeff Johnson 2007-02-19 20:10:52 EST
Eeek, too many %config bugs.

Ignoring mtime as end-user configurableI haven't done yet.

What has been added to rpm-4.4.8 is to scope %config more narrowly, so that
%config files are compered only against files using a digest, while %config links are
compared only against links comparing symlink end-points.

That removes a source of (perhaps unnecessary) .rpmnew creation when a
symlink is followed to compare a file digest.

Paramaterizing mtime (and digest and symlink endpoint and ...) comparisons
very soon. All of the %config .rpmnew creation needs to be carefully simplified and
eliminated in rpm imho.

Hint: Saving virgin %config files in a shadow tree and adding --diff to rpmquery is
likely much more useful to most than adding a .rpmnew prefix.
Comment 32 Aleksey Nogin 2007-02-19 22:25:38 EST
(In reply to comment #31)

> Hint: Saving virgin %config files in a shadow tree and adding --diff to
rpmquery is
> likely much more useful to most than adding a .rpmnew prefix.

I would like to disagree. Even if there was an "rpmquery --diff" (which, I
agree, would be very nice), I would still want some sort of indicator "the
default values for this config have changed, you need to review this config" -
this is what "locate .rpmnew" (e.g. following an upgrade) currently provides
nicely, while "rpmquery --diff" would be of no help for that.
Comment 33 Jeff Johnson 2007-02-20 07:20:12 EST
Everyone wants to disagree.

Let's see:

 1) You disagree.
 2) You agree that --diff would be nice.
 3) You want an indicator like .rpmnew.
 4) You don't want an indicator if mtime (or other values changed).
 5) You seem to want a dialog or report (you need to review ...)

all from a misposted comment in this bug report about vaporware.
Comment 34 Jeff Johnson 2007-04-29 08:59:39 EDT
Comment added so that I can this bug in spite of bugzilla normal -> medium churn.
Comment 35 D. Hugh Redelmeier 2007-05-28 17:32:35 EDT
Here is a shell script to help with this problem.  It assumes that the locate
database has been updated recently enough.

For each .rpmnew file (found by locate), the file is compared with the
non-rpmnew version.  If the files have identical content (ignoring
permissions and timestamps) the .rpmnew is deleted; otherwise, a diff
is done.

# remove .rpmnew files that have content that is the same as the base file

set -u -e

locate '*.rpmnew' |
    while read FS
    do
        F=`expr match "$FS" '\(.*\)\.rpmnew$'`
        if cmp "$F" "$FS"
        then
            echo remove "$FS"
            rm "$FS"
        else
            [ "$?" == 1 ] && diff -u "$F" "$FS" || :
        fi
    done
Comment 36 Panu Matilainen 2007-07-04 09:49:42 EDT
*** Bug 246236 has been marked as a duplicate of this bug. ***
Comment 37 Panu Matilainen 2007-08-09 09:34:34 EDT
The original issue fixed upstream at rpm.org by patch from c#11, thanks Tomas.
Also will be in next rawhide build (rpm-4.4.2.1-5.fc8).

Open new bugs for the other issues discussed here if necessary, this one's
closing now.
Comment 38 D. Wagner 2007-08-26 14:30:17 EDT
*** Bug 254721 has been marked as a duplicate of this bug. ***

Note You need to log in before you can comment on or make changes to this bug.