Description of problem: There is no /var/spool/at/spool directory after upgrading from 3.1.12-11.fc16 to 3.1.13-1.fc16 Version-Release number of selected component (if applicable): 3.1.13-1.fc16 How reproducible: always Steps to Reproduce: 1.make sure you installed at-3.1.12-11.fc16 2.run yum upgrade 3.check /var/spool/at directory Actual results: there is no /var/spool/at/spool Expected results: /var/spool/at/spool should exist Additional info: Re-install(yum reinstall) at package fix the issue.
RPM bug? # rpm -e at ; yum -y install at ... # rpm -q at ; rpm -V at at-3.1.13-1.fc16.x86_64 # rpmls at|grep ^d drwxr-xr-x /usr/share/doc/at-3.1.13 drwx------ /var/spool/at drwx------ /var/spool/at/spool # rpm -e at ; yum --disablerepo=updates-testing install at ... # rpm -q at ; rpm -V at at-3.1.12-11.fc16.x86_64 # rpmls at|grep ^d drwxr-xr-x /usr/share/doc/at-3.1.12 drwx------ /var/spool/at/ drwx------ /var/spool/at/spool/ As one can see, both packages do include that directory. However, updating to the newer at removes the directory: # yum -y update at ... # rpm -V at missing /var/spool/at/spool And starting from scratch, installing at-3.1.13-1.fc16 and _downgrading_ at also removes the directory: # rpm -e at ; yum -y install at ... # rpm -V at # yum downgrade at ... # rpm -V at missing /var/spool/at/spool/
Strange: rpm -ql at /var/spool/at /var/spool/at/.SEQ /var/spool/at/spool but: rpm -V at missing /var/spool/at/spool And section files correctly owns it: %files %doc docs/* %attr(0644,root,root) %config(noreplace) %{_sysconfdir}/at.deny %attr(0644,root,root) %config(noreplace) %{_sysconfdir}/sysconfig/atd %attr(0700,daemon,daemon) %dir %{_localstatedir}/spool/at %attr(0600,daemon,daemon) %verify(not md5 size mtime) %ghost %{_localstatedir}/spool/at/.SEQ %attr(0700,daemon,daemon) %dir %{_localstatedir}/spool/at/spool rpm -q rpm rpm-4.9.1.1-1.fc17.x86_64
Not limited to "at": # rpm -q rpm rpm-4.9.0-10.fc16.x86_64 # grep ^missing rpm-Va.txt missing /var/run/wpa_supplicant missing /var/run/openvpn missing /etc/openldap/cacerts/ missing /var/lib/AccountsService/icons/ missing /var/lib/initramfs/ missing /etc/multipath/ missing /usr/lib64/nss/saved/ missing /var/lib/ntp/ missing /var/log/ntpstats/ ... Not the full list, because check hasn't completed yet.
It sounds like this is affecting quite a few packages, then. Are the missing directories re-created during normal use? Other than the fact that the directories are missing, what is the user-noticeable effect of this?
> affecting quite a few packages Perhaps even more because this is not an "Everything" install. Under which circumstances are directories lost? There are many more which creates empty dirs with special uid/gid settings. > what is the user-noticeable effect of this? "at" for example only queues a job but doesn't execute it. It logs "Cannot chdir to /var/spool/at/spool: No such file or directory" without telling the user. "ntpd" fails to create/reuse its "drift" file. To create a dir in /var/lib, it would need to do that as root user prior to becoming ntp user. Dirs in /var/run -> /run tmpfs can be ignored.
this is obviously a tricky bug, and I confirm it on my everyday f16 system: [adamw@adam tmp]$ grep issing rpmva.txt missing /etc/openldap/cacerts/ missing /var/lib/AccountsService/icons/ missing /etc/cups/ssl missing /usr/lib64/nss/saved/ missing /var/lib/initramfs/ missing /var/run/wpa_supplicant missing /etc/multipath/ missing /var/cvs/ missing /etc/gcrypt/ missing /etc/pki/CA missing /etc/pki/CA/certs missing /etc/pki/CA/crl missing /etc/pki/CA/newcerts missing /etc/pki/CA/private missing /etc/pki/CA missing /etc/pki/CA/certs missing /etc/pki/CA/crl missing /etc/pki/CA/newcerts missing /etc/pki/CA/private missing /etc/openldap/cacerts/ missing /etc/subversion missing /var/spool/at/spool missing /usr/lib/nss/saved/ missing /var/run/gdm/greeter missing /etc/gcrypt/ missing /var/run/openvpn missing /var/log/ntpstats/ missing /var/run/vpnc/ missing /var/run/NetworkManager missing /var/run/libvirt/lxc missing /var/run/libvirt/uml missing /etc/security/console.perms.d missing /etc/security/namespace.d missing /lib/security missing /usr/lib/ConsoleKit/run-seat.d missing /etc/lvm/cache I'm not sure it makes a great Alpha blocker, though, as it's something that happens outside of installation and first use, and could easily be fixed outsid of the process of generating and shipping images.
Discussed in the 2011-08-10 Fedora 16 go/no-go meeting. Accepted as a Fedora 16 alpha blocker as it violates the following alpha release criterion [1]: The installed system must be able to download and install updates with yum and the default graphical package manager in all release-blocking desktops. The ability to correctly apply updates is implied in the criterion. [1] https://fedoraproject.org/wiki/Fedora_16_Alpha_Release_Criteria
(In reply to comment #1) > RPM bug? > > > # rpm -e at ; yum -y install at > ... > # rpm -q at ; rpm -V at > at-3.1.13-1.fc16.x86_64 > # rpmls at|grep ^d > drwxr-xr-x /usr/share/doc/at-3.1.13 > drwx------ /var/spool/at > drwx------ /var/spool/at/spool > > > # rpm -e at ; yum --disablerepo=updates-testing install at > ... > # rpm -q at ; rpm -V at > at-3.1.12-11.fc16.x86_64 > # rpmls at|grep ^d > drwxr-xr-x /usr/share/doc/at-3.1.12 > drwx------ /var/spool/at/ > drwx------ /var/spool/at/spool/ > Yes, this seems to be related to 722474 where rpmbuild complains about unpackaged files. Note the trailing slash in the second listing. Not sure what to do if the trailing slashes are already in rpmdb on installed F16 system. Fresh F16 install + testing the upgrade will reveal if it is fixed by rpm-4.9.1-2 and newer so that new F16 installs won't see this bug any more.
A duplicate of Bug 725137?
Thinking from a high level here, ... Wouldn't a 'proper' fix to be to determine (by stat, or otherwise), if a given entry in the RPMDB is a DIRECTORY, (and it may be that a LINK needs to considered here), and 'normalize' the database entry to reflect that all directories to CARRY the trailing end slash (to permit quick disambiguation from a FILE) This would be possible on an existing RPMDB in a --rebuild process, an a backward-compatible extension to the present code, I would think, and does not necessitate database changes, as it relates to the content housed within it, rather than the structures storing the information The other possible secondary effect would be to make sure that the VERIFY function silently accepted either the version WITH or WITHOUT the trailing slash as 'unmodified' in the case of directories -- Russ herrold
I see mention in Bug 725137 of rpmbuild also complaining ... obviously that code, handling identification of 'files' (here in the broader sense including directories to be created) needs to be aware of the 'identity' of a directory with and without the trailing slash in its textual specification, when doing 'unpackaged' file enumeration tar, cpio, and other friends all cope with (and do the 'right thing') this issue silently at present and have for years ... something like 'dirname' comes to mind. ... and it is not improper to add the trailing slash as to RPM's internal representation of database content, as that convention is being done to perform a 'flattening' function for disambiguation of a file from a directory without carrying around a 'file attribute' entry in the RPMDB, and is contained within a namespace that RPM alone controls (and according can manage as it sees fit) -- Russ herrold
Trailing slash on directory entries in %files lists has been in use for many years and improves spec file readability a lot.
Interestingly I get some missings right after a DVD install of Alpha RC3: /var/run/vpnc/ /var/run/openvpn /var/run/setroubleshoot /var/run/gdm/greeter /var/run/wpa_supplicant /var/run/NetworkManager After doing a 'yum update', the list changes to: /var/spool/at/spool /var/run/vpnc/ /var/run/openvpn /var/run/setroubleshoot /var/lib/initramfs /var/run/NetworkManager so at and initramfs are added, but gdm and wpa_supplicant drop out. I'll do some more testing with rpm-4.9.1-3 to see if it fixes the issue.
Well, it seems not... I tested with at. I removed it to get a clean start, then: * install at-3.1.12-11.fc16 from DVD * 'rpm -V at' reports clean * 'yum update at' with rpm-4.9.1-3 installed * updates to at-3.1.13-1.fc16 * 'rpm -V at' reports 'missing /var/spool/at/spool' So there's clearly still an issue here. Panu, Jindrich, can you look at this urgently? It's blocking F16 Alpha and we really want to compose RC4 today. Thanks.
The line in the 'at' spec reads: %attr(0700,daemon,daemon) %dir %{_localstatedir}/spool/at/spool this doesn't seem to have changed between 3.1.12-11 and 3.1.13-1, so I'm not sure this has anything to do with trailing slashes.
Panu explains the problem in this mail to fedora-devel-list: http://lists.fedoraproject.org/pipermail/devel/2011-July/154658.html For a while rawhide had a build of rpm in buildroots that added trailing slashes to some directories in rpm filelists. It was an unintended change and the offending build was untagged. However, rpm thinks that directories with a trailing slash and directories without one are different and this is causing the bug. For example, if package bar-1 owns the directory /foo/bar and bar-2 changes it to /foo/bar/, then when bar-1 is uninstalled at the end of an upgrade transaction, /foo/bar gets removed. The same happens when the directory changes from /foo/bar/ to /foo/bar. This particular bug might have been fixed in rpm 4.9.1.1, I am not sure. In my eyes, the fix is twofold. First, rpm should consider /foo/bar/ and /foo/bar the same, so that the directory wouldn't be removed on upgrade. Second, we also need to consider the fact that there are going to be users who upgrade from an older distro that has an older rpm build which doesn't have the fix. For the sake of these users, I would recommend rebuilding all packages that currently have directories with a trailing slash in file lists, so that upgrades from e.g. F15->F16 alpha wouldn't start removing directories.
Kalev: that can't be the whole problem. As I mentioned in comment #15, at has not changed the format of the line for that directory. In neither of the two versions used in the test does the line read %dir %{_localstatedir}/spool/at/spool/ . In both versions, it reads %dir %{_localstatedir}/spool/at/spool .
oh, unless it was added during build...i see.
Yes, I was talking about file lists in binary rpms; the %files section in the spec file didn't have to get changed to trigger the bug.
I came up with a script to figure out what packages are affected. As mentioned earlier, updating an affected package to a fixed one will still result in empty directories being removed. The goal here is to provide a working upgrade path from older releases to F16, and from F16 alpha to F16 final. To ensure that, I'd prefer if none of the affected builds were in F16 alpha. # $ koji list-tag-history --package=rpm --tag=f16 # Tue Jul 19 11:35:20 2011: rpm-4.9.1-2.fc16 tagged into f16 by pmatilai # Wed Jul 20 19:40:53 2011: rpm-4.9.1-2.fc16 untagged from f16 by rdieter # Wed Jul 20 19:49:19 2011: rpm-4.9.1-2.fc16 tagged into f16 by rdieter # Thu Jul 21 20:34:46 2011: rpm-4.9.1-2.fc16 untagged from f16 by rdieter BUILDTIME1=$(date --date='Tue Jul 19 11:35:20 2011 EEST' '+%s') BUILDTIME2=$(date --date='Thu Jul 21 20:34:46 2011 EEST' '+%s') # add an hour to BUILDTIME2 to compensate for buildroot regeneration time BUILDTIME2=$(($BUILDTIME2+60*60)) REPO_OPTS="--disablerepo='*' --enablerepo=fedora --releasever=16" # Get the list of the packages that were build during that time PKGLIST=$(repoquery -qa --qf "%{buildtime} %{NEVRA}" $REPO_OPTS \ | awk "\$1 > $BUILDTIME1 && \$1 < $BUILDTIME2" \ | cut -f 2 -d ' ' | sort | uniq) # Download the packages (yum filelists database has trailing slashes stripped, # so we can't use repoquery.) DESTDIR=/tmp/rpm-rebuild-query mkdir -p $DESTDIR for pkg in $PKGLIST ; do yumdownloader --destdir $DESTDIR --noplugins --cacheonly $REPO_OPTS $pkg done empty_dir() { local rpm=$1 local dir=$2 if rpm -qlp $rpm | grep -E "${dir}.+" ; then return 1 else return 0 fi } # Iterate over all the downloaded rpms and check if any of these has # a directory with a trailing slash in its file list for pkg in $DESTDIR/* ; do dirs_with_slash=$(rpm -qlp $pkg | grep '/$') if [ $? -eq 0 ] ; then for dir in $dirs_with_slash ; do # The bug only manifests with empty directories if empty_dir $pkg $dir ; then build=$(rpm -qp --qf "%{sourcerpm}" $pkg \ | sed 's/\.src\.rpm$//') AFFECTED_BUILDS="$AFFECTED_BUILDS $build" break 1 fi done fi done echo "DONE! Affected builds are:" for build in $(echo $AFFECTED_BUILDS | tr ' ' '\n' | sort | uniq) ; do echo "$build" done
And the list of affected builds. Note that some of these might already be fixed by subsequent builds in updates-testing; I only checked the stable F16 repo as of today. accountsservice-0.6.13-1.fc16 akonadi-1.6.0-3.fc16 at-3.1.12-11.fc16 BackupPC-3.2.1-1.fc16 chkconfig-1.3.54-1.fc16 couchdb-1.0.3-1.fc16 cvs-1.11.23-20.fc16 device-mapper-multipath-0.4.9-17.fc16 dracut-011-15.git20110720 isdn4k-utils-3.2-78.fc16 libgcrypt-1.5.0-1.fc16 libibverbs-1.1.5-4.fc16 matahari-0.4.2-1.fc16 ntp-4.2.6p3-5.fc16 openldap-2.4.26-1.fc16 openpts-0.2.5-1.fc16 opensm-3.3.9-1.fc16 qpid-cpp-0.10-4.fc16 qt3-3.3.8b-36.fc16 thunderbird-5.0-2.fc16 transmission-2.33-1.fc16 xen-4.1.1-2.fc16 xine-ui-0.99.6-27.fc16
So, at a minimum we ought to rebuild all those items from comment #21? If so, I can help make that happen.
Yes, lets do that. Rebuilding the packages from comment #21 will ensure that F15 -> F16 Alpha upgrade works and also that F16 Alpha -> F16 Beta would work. This won't fix F16 pre-Alpha -> F16 Alpha upgrades, but that shouldn't block the Alpha compose any more; what's most important is that those people who install Alpha are able to properly install updates. These two already have rebuilds available: at-3.1.12-11.fc16 -> at-3.1.13-1.fc16, https://admin.fedoraproject.org/updates/at-3.1.13-1.fc16 dracut-011-15.git20110720 -> dracut-013-1.fc16, https://admin.fedoraproject.org/updates/dracut-013-1.fc16 So, I propose we split the remaining of the list in two. I'll rebuild these: accountsservice-0.6.13-1.fc16 akonadi-1.6.0-3.fc16 BackupPC-3.2.1-1.fc16 chkconfig-1.3.54-1.fc16 couchdb-1.0.3-1.fc16 cvs-1.11.23-20.fc16 device-mapper-multipath-0.4.9-17.fc16 isdn4k-utils-3.2-78.fc16 libgcrypt-1.5.0-1.fc16 libibverbs-1.1.5-4.fc16 Rex, can you do these? matahari-0.4.2-1.fc16 (F17 has a newer build available, but F16 doesn't) ntp-4.2.6p3-5.fc16 openldap-2.4.26-1.fc16 openpts-0.2.5-1.fc16 opensm-3.3.9-1.fc16 qpid-cpp-0.10-4.fc16 qt3-3.3.8b-36.fc16 transmission-2.33-1.fc16 xen-4.1.1-2.fc16 xine-ui-0.99.6-27.fc16 That leaves one package where provenpackagers don't have commit access: thunderbird-5.0-2.fc16. If we don't manage to rebuild that one, I think it's OK, because with TB the issue is mostly cosmetic and doesn't stop the package from working: $ rpm -V thunderbird missing /usr/lib64/mozilla/extensions/{3550f703-e582-4d05-9a08-453d09bdfdc6}/ missing /usr/share/mozilla/extensions/{3550f703-e582-4d05-9a08-453d09bdfdc6}/
ok
You don't need to do xen-4.1.1-2.fc16 , there was a security alert on Friday, and I have now built xen-4.1.1-3.fc16 with the security patch applied.
the way I'm seeing this, we would still need to include the rebuilds of any of those packages that are on the DVD or live media in the Alpha compose. it really seems like the right way to fix this is just to fix rpm...
accountsservice-0.6.13-2.fc16,akonadi-1.6.0-4.fc16,chkconfig-1.3.54-2.fc16,couchdb-1.0.3-2.fc16,cvs-1.11.23-21.fc16,device-mapper-multipath-0.4.9-18.fc16,isdn4k-utils-3.2-79.fc16,libgcrypt-1.5.0-2.fc16,libibverbs-1.1.5-5.fc16,openpts-0.2.5-2.fc16,opensm-3.3.9-2.fc16,matahari-0.4.2-1.fc16.1,ntp-4.2.6p3-5.fc16.1,openldap-2.4.26-1.fc16.1,qpid-cpp-0.10-4.fc16.1,qt3-3.3.8b-36.fc16.1,transmission-2.33-1.fc16.1,xine-ui-0.99.6-27.fc16.1,BackupPC-3.2.1-1.fc16.1 has been submitted as an update for Fedora 16. https://admin.fedoraproject.org/updates/accountsservice-0.6.13-2.fc16,akonadi-1.6.0-4.fc16,chkconfig-1.3.54-2.fc16,couchdb-1.0.3-2.fc16,cvs-1.11.23-21.fc16,device-mapper-multipath-0.4.9-18.fc16,isdn4k-utils-3.2-79.fc16,libgcrypt-1.5.0-2.fc16,libibverbs-1.1.5-5.fc16,openpts-0.2.5-2.fc16,opensm-3.3.9-2.fc16,matahari-0.4.2-1.fc16.1,ntp-4.2.6p3-5.fc16.1,openldap-2.4.26-1.fc16.1,qpid-cpp-0.10-4.fc16.1,qt3-3.3.8b-36.fc16.1,transmission-2.33-1.fc16.1,xine-ui-0.99.6-27.fc16.1,BackupPC-3.2.1-1.fc16.1
Rex Dieter and I rebuilt all the affected packages, except for the following three that have separate updates, and thunderbird, where we don't have commit access. The large update linked in comment #27 should be relatively safe to include in Alpha as it only contains straight rebuilds; the following three contain additional changes and could use some more testing. https://admin.fedoraproject.org/updates/at-3.1.13-1.fc16 https://admin.fedoraproject.org/updates/dracut-013-1.fc16 https://admin.fedoraproject.org/updates/xen-4.1.1-3.fc16
Back from vacation... The right thing to do is indeed to rebuild the broken packages, thank you Kalev & Rex for all your work on dealing with this!
*** Bug 725137 has been marked as a duplicate of this bug. ***
I rebuilt at https://admin.fedoraproject.org/updates/at-3.1.13-2.fc16
panu: is there no fix required to rpm's behaviour here? it still doesn't seem right for it to start wiping directories when the format of a directory's entry in the file list changes - whatever the cause of the change in the file list. -- Fedora Bugzappers volunteer triage team https://fedoraproject.org/wiki/BugZappers
Rebuilding "at" wasn't necessary. The trailing slash is only in the already released pre-Alpha build at-3.1.12-11.fc16: $ rpmls at-3.1.13-2.fc16.x86_64.rpm | grep ^d drwxr-xr-x /usr/share/doc/at-3.1.13 drwx------ /var/spool/at drwx------ /var/spool/at/spool $ rpmls at-3.1.13-1.fc16.x86_64.rpm | grep ^d drwxr-xr-x /usr/share/doc/at-3.1.13 drwx------ /var/spool/at drwx------ /var/spool/at/spool $ rpmls at-3.1.12-11.fc16.x86_64.rpm | grep ^d drwxr-xr-x /usr/share/doc/at-3.1.12 drwx------ /var/spool/at/ drwx------ /var/spool/at/spool/ ^ |--- this old one
(In reply to comment #32) > panu: is there no fix required to rpm's behaviour here? it still doesn't seem > right for it to start wiping directories when the format of a directory's entry > in the file list changes - whatever the cause of the change in the file list. There are *never* supposed to be any trailing slashes in the header filelist, no matter what the spec %files section says. The bug in rpm-4.9.1 is that it built packages which in some cases did have that trailing slash which should've never been there. No change of rpm behavior is needed, we just need to get rid of the buggy packages (which is what is being done here).
ah, gotcha - so even if the spec file has a trailing slash, for readability, the file list generated in the package isn't supposed to have one? makes sense, thanks. -- Fedora Bugzappers volunteer triage team https://fedoraproject.org/wiki/BugZappers
after installing RC5, updating, 'rpm -Va | grep missing' shows: /var/run/vpnc/ /var/run/wpa_supplicant /var/run/NetworkManager /var/run/openvpn /var/run/gdm/greeter /var/run/setroubleshoot so, still something going on, but it may not be this issue. Given that rpm itself is not going to be causing any issues going forwards, I think I'm not too worried about this for Alpha; we can close or drop it from the blocker list as appropriate. The specific rebuilds done by Rex and co seem to have done their jobs, so thanks for those.
This is an unrelated issue. /var/run is on tmpfs and the directories probably weren't just created when you booted up.
Yes, the /var/run missing bits can be seen on F15 as well. However unless it's a copy-paste error of some kind, there appears to be (at least) one more package left with the trailing slash issue: /var/run/vpnc/
My script only flagged packages where _empty_ directories had a trailing slash; there are probably hundreds of packages more where a non-empty directory has a trailing slash. In case of vpnc rpm, there are several subdirectories/files in /var/run/vpnc/ [1] and unless a future package update removes all the subdirectories/files, I believe the trailing slash shouldn't cause problems for upgrades. At least I don't think it's worth rebuilding hundreds of more packages for Alpha. [1] $ rpm -ql vpnc ... /var/run/vpnc/ /var/run/vpnc/defaultroute /var/run/vpnc/pid /var/run/vpnc/resolv.conf-backup
I think with that info we can call this VERIFIED then: we've fixed what we intended to fix.
(In reply to comment #39) > My script only flagged packages where _empty_ directories had a trailing slash; > there are probably hundreds of packages more where a non-empty directory has a > trailing slash. Ah, sorry I missed that part. > In case of vpnc rpm, there are several subdirectories/files in /var/run/vpnc/ > [1] and unless a future package update removes all the subdirectories/files, I > believe the trailing slash shouldn't cause problems for upgrades. The thing is, we dont know for sure what additional problems might be lurking there due to those unexpected trailing slashes. > At least I don't think it's worth rebuilding hundreds of more packages for > Alpha. Yeh, for Alpha resolving this particular issue probably suffices. But we need to get rid of ALL the trailing slashes before F16 final, painful as it might be (fortunately its not hundreds of packages, "just" ~100): http://lists.fedoraproject.org/pipermail/devel/2011-August/155641.html
accountsservice-0.6.13-2.fc16, akonadi-1.6.0-4.fc16, chkconfig-1.3.54-2.fc16, couchdb-1.0.3-2.fc16, cvs-1.11.23-21.fc16, device-mapper-multipath-0.4.9-18.fc16, isdn4k-utils-3.2-79.fc16, libgcrypt-1.5.0-2.fc16, libibverbs-1.1.5-5.fc16, openpts-0.2.5-2.fc16, opensm-3.3.9-2.fc16, matahari-0.4.2-1.fc16.1, ntp-4.2.6p3-5.fc16.1, openldap-2.4.26-1.fc16.1, qpid-cpp-0.10-4.fc16.1, qt3-3.3.8b-36.fc16.1, transmission-2.33-1.fc16.1, xine-ui-0.99.6-27.fc16.1, BackupPC-3.2.1-1.fc16.1, thunderbird-5.0-3.fc16 has been pushed to the Fedora 16 stable repository. If problems still persist, please make note of it in this bug report.
I think we can consider this case closed by now...