Bug 240697

Oh, I'm upgrading from glibc-2.5.90-21.

Can't reproduce.  Can you try upgrading again and run the whole
command under strace -f ?
/usr/lib/locale/locale-archive.tmpl is one of the files in glibc-common
which rpm is supposed to unpack before the %post script is run.
The %post script then processes this file together with any preexisting
subdirectories in /usr/lib/locale/ as source and creates
/usr/lib/locale/locale-archive
file from it (and wipes the template).
Running /usr/sbin/build-locale-archive by hand is expected to fail, because
it is already missing, but rpm -Uvh shouldn't fail (and doesn't fail for me).

It looks like something else is broken, because I'm no longer able to install 
older glibc having build-locale-archive too. Removing from F7 blocker list so 
far until I really know, what's up.

Hmmm, here's the immediate problem, /usr/lib/locale/locale-archive.tmpl not being installed
because
    
    D: /usr/lib/locale/locale-archive.tmpl skipped due to missingok flag
    ...
    D: fini      100644  1 (   0,   0)    65179424 /usr/lib/locale/locale-archive.tmpl;46522417 skip

Now to figger why the missingok flag is set in rpm-4.4.9 ...

And here's the next piece:

D: Conflicts: glibc-common <= 2.3.2-63                      NO  
==16368== 
==16368== Invalid read of size 1
==16368==    at 0x480635E: strcmp (mc_replace_strmem.c:341)
==16368==    by 0x482C5C7: checkPackageDeps (depends.c:1180)
==16368==    by 0x482C91A: checkPackageSet (depends.c:1308)
==16368==    by 0x482CD0E: rpmtsCheck (depends.c:1365)
==16368==    by 0x485EDDD: rpmInstall (rpminstall.c:659)
==16368==    by 0x4CD6: main (rpmqv.c:749)
==16368==  Address 0xCB86970 is 0 bytes inside a block of size 13 free'd
==16368==    at 0x48050FF: free (vg_replace_malloc.c:233)
==16368==    by 0x484E2EF: rpmdsNext (rpmlib.h:66)
==16368==    by 0x48489D5: rpmalAllSatisfiesDepend (rpmal.c:857)
==16368==    by 0x4848AF1: rpmalSatisfiesDepend (rpmal.c:891)
==16368==    by 0x482AD28: unsatisfiedDepend (depends.c:959)
==16368==    by 0x482C56F: checkPackageDeps (depends.c:1156)
==16368==    by 0x482C91A: checkPackageSet (depends.c:1308)
==16368==    by 0x482CD0E: rpmtsCheck (depends.c:1365)
==16368==    by 0x485EDDD: rpmInstall (rpminstall.c:659)
==16368==    by 0x4CD6: main (rpmqv.c:749)

The above problem is now fixed, nothing to do with this bug.

Here's the data in the pkg:
$ rpm -Kvv glibc-common-2.6-1.i386.rpm 
D: Expected size:     18628944 = lead(96)+sigs(180)+pad(4)+data(18628664)
D:   Actual size:     18628944
glibc-common-2.6-1.i386.rpm:
    Header SHA1 digest: OK (9ccf73fd3c6d70da0dec460f69ce5ab505a7ebb7)
    MD5 digest: OK (ff557d28ede0a5d5c4875f4a97b428f1)
$ rpm -qp --qf '[%{filenames} %{fileflags}\n]'  glibc-common-2.6-1.i386.rpm | grep tmpl
/usr/lib/locale/locale-archive.tmpl 9

So the flags are truly
    RPMFILE_CONFIG|RPMFILE_MISSINGOK
as can be seen in FC/devel/glibc/glibc.spec CVS:

1.301        (jakub    17-Apr-07): %attr(0644,root,root) %config(missingok) %{_prefix}/lib/locale/locale-
archive.tmpl

I have no idea what is being attempted, so its difficult to guess what should be done.

Lemme try a glibc build without missingok ...

FWIW, rpm-4.4.2 will behave identically, here's lib/rpmfi.c code in CVS:

2.44         (jbj      09-Jul-03):     if (lstat(fn, &sb)) {
2.44         (jbj      09-Jul-03):      /*
2.44         (jbj      09-Jul-03):       * The file doesn't exist on the disk. Create it unless the new
2.44         (jbj      09-Jul-03):       * package has marked it as missingok, or allfiles is requested.
2.44         (jbj      09-Jul-03):       */
2.44         (jbj      09-Jul-03):      if (skipMissing && (newFlags & RPMFILE_MISSINGOK)) {
2.44         (jbj      09-Jul-03):          rpmMessage(RPMMESS_DEBUG, _("%s skipped due to missingok flag
\n"),
2.44         (jbj      09-Jul-03):                      fn);
2.44         (jbj      09-Jul-03):          return FA_SKIP;
2.44         (jbj      09-Jul-03):      } else {
2.44         (jbj      09-Jul-03):          return FA_CREATE;
2.44         (jbj      09-Jul-03):      }
2.44         (jbj      09-Jul-03):     }

The difficulty in reproducing is likely due to whether the file exists (or not) ...

Note that rpm -Uvv --force has the behavior of erase-before-install which may also help explain why 
the reproducer is tricky.

All my comments are from examining rpm -Uvv --force glibc-common installs.

And I appear to have multiple glibc-common packages installed for whatever yum/rawhide/rpm reason:

$ rpm -q glibc-common
glibc-common-2.5.90-21.i386
glibc-common-2.5.90-22.i386
glibc-common-2.6-1.i386

 Let's clean up the mess ... same behavior after removing the 2 older glibc-common packages.

So this works with the distro rpm (4.4.2) and doesn't with jbj's rpm?

The file really exist in the package, but has to be marked missingok,
because it is only used during %post and then unlinked.
locale-archive.tmpl together with /usr/lib/locale/*_*/ directories are sources
for build-locale-archive created locale-archive.  locale-archive.tmpl is
then unlinked because it is huge (~70MB) and not needed afterwards.

Too early to say Buggy!

Erasing a file contained in a package payload and marking config(missingok)
is unusual packaging that has not been attempted in any package I'm aware of
until now.

Feel free to propose other packaging of this.
The needs:
1) the locale files are packed together, rpm really doesn't handle thousands of
files well
2) no duplication of the locale data files, when glibc-common is installed,
locales should only occupy roughly 70MB of disk space, not 140MB or more as they
used to before
3) it must be flexible with user supplied locales
4) programs should mmap locale-archive, not individual files in locale
subdirectories.

Until recently glibc-common contained almost 5000 files in ~ 1500 directories
and during %post created a ~ 70MB locale archive from it, so it ate 70+80MB of
diskspace, where the 80MB in the subdirectories were completely unused after
install (resp. glibc upgrades).

Re-installing w rpm-4.4.9 using --allfiles
    rpm -Uvv --force --allfiles glibc-common-2.6-1.i386.rpm
runs the %post scriptlet without errors.

I see no reason why rpm-4.4.2 should not behave identically to 4.4.9. (not looked).

Yes, it's a hard packaging problem. Lemme think a bit about adding a transient oneshot
install, like a self-erasing task sub-package.

No matter what glibc-common is tricky because no /bin/sh in chroot's.

FWIW, doing %post -p <lua> is likely easier to maintain than the current static helpers.

Again FYI: rpm has always had two glibc-common problems:
    1) partial hardlink sets to accomodate a locale install policy
    2) a busted algorithm (and combinatorial memory explosion) with
    lots of identical basenames like LC_MESSAGES.

1) appears to be a non-feature any more. I will start phasing out the mechanism in 4.4.9

2) was fixed on 4/1/2007 in 4.4.9 by adding a unique identifier to the join key to
prevent the combinatorially large memory explosion with lots of identical basenames.

With those flaws fixed, rpm performance should scale more linearally with lots of directories and files.

Jakub, do you really consider this to be a FC7Blocker? PERSONALLY, I would 
revert the changes in glibc-common regarding build-local-archive and fix up
the 1) and 2) from comment #13 within Fedora RPM - just my 2 ct. Yes, I know 
we're 8 days before F7 goes gold. In fact, I'm able to reproduce this problem 
with Fedora RPM 4.4.2 e.g. by playing with --force but the reproducing seems 
not to be that easy like with RPM 4.4.9 - and I don't know why. You also could
only do the first thing of my proposal and fix the Fedora RPM for F8 (or even 
F7-post) to have better performing.

Try
    cp /dev/null /usr/lib/locale/locale-archive.tmpl
instead of removing. Then lose the tricky %config(missingok)
which was never intended for what you are trying to do.

And replace with %verify(not md5 size mtime) ?
If that works, that would be great.  I was already considering bzip2ing
the locale-archive.tmpl and not removing it at all, but that means wasted 12MB
on the filesystem permanently.

Summary: Test tempfile
Name: testtempfile
Version: 2.0
Release: 1
License: GPL
Group: Development/Libraries
BuildRoot: %{_tmppath}/%{name}-%{version}-buildroot
BuildArch: noarch
Prefix: /usr/opt/%{name}
%description
Other.
%prep
%build
%install
install -d -m 755 $RPM_BUILD_ROOT/usr/opt/%{name}
echo "%{name} %{version}" > $RPM_BUILD_ROOT/usr/opt/%{name}/%{name}
%clean
%post
if [ "$1" -ge 2 ]; then
    if echo "%{name} %{version}" | cmp -s - /usr/opt/%{name}/%{name}; then
        :
    else
        echo Failure
    fi
fi
> /usr/opt/%{name}/%{name} || :
%files
%defattr(0644,root,root,0755)
%dir /usr/opt/%{name}
%verify(not md5 size mtime)     /usr/opt/%{name}/%{name}

worked in my quick testing, can I rely on this?  If yes, I'll build new glibc.

For me it worked in my testing, too. But as Jeff is the RPM god, I really would 
wait for a reply/short feedback by him.

You'll need to worry about empty chroo't's not having shell (I still suggest lua
so that others can see what glibc* helpers are doing), but basically yes, the
mechanism is simpler than %config(foo), and that means likelier to just "work".

Jakub, Jesse - can we use LUA?

Is the fix in comment #17 in glibc-2.6-3? If so, I'd like to close this bug.

Re: comment #21: Yes, it is.
As for lua, it wouldn't help in any way to glibc-common's %post,
as merging of the locale archives with locale directories isn't something
I'd like to do in interpreted languages directly in the %spec file,
whether it is shell (using some statically linked shell and utilities), lua,
busybox or anything similar (and perl/python are neither desirable nor a good
idea to do it in).