Proposed as a Blocker for 19-final by Fedora user pschindl using the blocker tracking app because:

 Bug violates the criterion: "The installed system must be able to download and install updates with yum and with the default graphical package manager in all release-blocking desktops." The option to do offline updates is offered by default in GNOME.

Created attachment 754837 [details]
journalctl -a

Reproduced in KVM with Fedora 19 Beta RC4, adding logs from clean DVD installation.

Created attachment 754838 [details]
rpm -qa before starting offline updates

Created attachment 754840 [details]
packages about to be installed (retrieved by calling yum update and cancelling it)

We've also tried to set SElinux to permissive, but it didn't help.

Discussed at 2013-05-30 blocker review meeting: http://meetbot.fedoraproject.org/fedora-blocker-review/2013-05-30/f19final-blocker-review-1.1.2013-05-30-16.02.log.txt . Accepted as a blocker per criterion "The installed system must be able to download and install updates with yum and with the default graphical package manager in all release-blocking desktops", on the understanding it really means 'default update method for release-blocking desktops'.

What's the 'default' update method for GNOME is kind of an open question: this is right there on the user menu, but the 'update available' notifications still run gpk-update-viewer, the online update method (I've had a bug report in on that for several months). So arguably GNOME is currently providing *two* update methods, and both ought to work.

I have a theory that this fails if the Fedora GPG key hasn't been imported yet - so only on DVD installs and only until you do some operation with yum or pk interactively. But we need to test that.

Actually, this seems to be failing for me on a fresh DVD Beta install every try, even after I install a package via yum to get the GPG key imported. Looks like the function is just flat broken somehow.

I will fix the crash (just downloading the DVD now to work out the exact place it's crashing) but the bigger problem is if the user hasn't accepted the Fedora GPG key before doing the offline update, it's always going to fail as the offline update by definition can't stop and ask the user questions.

Of course, the obvious thing would be to import the Fedora keys on the DVD like we do on the Live media on the assumption you either trust the media or you don't, and if you don't then the lack of a GPG key isn't your primary concern.

If we don't have the keys installed, what should the update client do? The GNOME designers don't want me to have a random GPG pop-up in the deskop asking for some random thing when the user is idle (the updates are downloaded by default on idle time) and it seems quite antisocial to just refuse the offline update with a message to the console like "You need to run the GUI updater manually first".

Ideas welcome.

This could be a good opportunity to finally solve bug 748320.

Hmm, I downloaded the beta DVD and tried to do an offline update of a single package and of the whole system, and it worked without a hitch, both with a package that existed in @fedora and also one that only existed in @updates. I did notice a race between the offline updates and the packagekitd daemon, which I've fixed in http://koji.fedoraproject.org/koji/buildinfo?buildID=424924 but that would only produce a scary warning, not a segfault. Can you describe *exactly* how to reproduce this bug please? Include things like what options you chose in anaconda, thanks. I'll continue to debug here.

Okay, I've fixed a few bugs in PK, but nothing that precisely matches this bug. I'd appreciate it if you could download the PackageKit packages from http://people.freedesktop.org/~hughsient/fedora/19/x86_64/ and then do something like "sudo rpm -Fvh PackageKit*.rpm" -- if this fixes this, or doesn't fix this I'd love to know. Thanks!

hughsie: my reproducer was simply to do a default install from the DVD - only setting the Installation Destination and root password spokes, not touching any of the others - set up a user in g-i-s, and then trigger an offline update via the user menu. I'll check if I can still reproduce with and without your updated PK shortly.

kparal: it seems de rigeur to mention https://bugzilla.redhat.com/show_bug.cgi?id=998 at this point :P

Reproduced on clean DVD install of F19-TC1. The update fails even after the action suggested in comment #11. The output of  `# journalctl -u packagekit-offline-update.service -a` where the part after Reboot is with the updated PackageKit.

-- Logs begin at Fri 2013-06-07 06:49:17 EDT, end at Fri 2013-06-07 07:08:21 EDT. --
Jun 07 06:53:54 localhost.localdomain systemd[1]: packagekit-offline-update.service: main process exited, code=killed, status=11/SEGV
Jun 07 06:53:54 localhost.localdomain systemd[1]: Unit packagekit-offline-update.service entered failed state.
Jun 07 06:53:54 localhost.localdomain systemd[1]: Triggering OnFailure= dependencies of packagekit-offline-update.service.
^[[1;39m-- Reboot --^[[0m
Jun 07 07:05:20 localhost.localdomain pk-offline-update[361]: 0: /lib64/libc.so.6  (__restore_rt+0x0) [0x7fad8b43f88f]
Jun 07 07:05:20 localhost.localdomain systemd[1]: packagekit-offline-update.service: main process exited, code=killed, status=5/TRAP
Jun 07 07:05:20 localhost.localdomain systemd[1]: Unit packagekit-offline-update.service entered failed state.
Jun 07 07:05:20 localhost.localdomain systemd[1]: Triggering OnFailure= dependencies of packagekit-offline-update.service.

So, from further testing it seems doing a small number of updates works, but doing over 300 segfaults for some reason. I'd appreciate people trying my test builds, as those should have the backtrace in the journalctl output.

Okay, progress:

commit ad1e920ba588fc8da8a86090ba196597cbfa6135
Author: Richard Hughes <richard>
Date:   Fri Jun 7 16:29:31 2013 +0100

    lib: Actually return the error if any PkClient methods failed
    
    Giovanni changed the g_new0() to a local helper object in commit
    771f87943d3fc0a8579eee42ade373052208c8e2, but nothing actually set the return
    value to NULL for failure. This meant we returned random memory for the
    PkResults on error, and blew up with a segfault as soon as we tried accessing
    the private structure.
    
    It seems we could rely on the structure being zeroed by coincidence for small
    numbers of packages, but for larger number of packages we got junk.

With that commit I get the more friendly (but equally buggy):

Failed To Update
Network access was required but not available.

Richard.

(In reply to Richard Hughes from comment #15)

Just to make it clear - the part of logs after "REBOOT" is with the updated packaeges. Are there any other sources I should look at for the backtrace?

richard: "Failed To Update
Network access was required but not available."

*that* sounds like the error I was getting in the 'missing GPG key' case. Does getting the GPG key installed fix it?

Richard, can we at least arrange things so we show a

System update failed
You need to install the repository GPG key first.
                                        [Install]

Notification after the failed update ?

(In reply to Adam Williamson from comment #18)
> Does getting the GPG key installed fix it?

No, after investigating this is a yum issue. Basically, the desktop client does the equivalent of:

pkcon --only-download update foo

This makes sure that all the repodata is up to date (including filelists), depsolves the update transaction and downloads foo and any additional packages the transaction needs to complete. 'foo' is written to the prepared-updates file, and the user is rebooted into the offline updates mode.

In this mode, the prepared-updates file is read, and a transaction is setup so that 'foo' is updated. Remember all the metadata and packages are downloaded. When in offline updates mode, there is no network, and so this is set in the yumBackend.py file:

            self.yumbase.conf.cache = 1
            for repo in self.yumbase.repos.listEnabled():
                repo.metadata_expire = -1  # never refresh

So, basically, work from cache and never expire the metadata. We setup the transaction and call yb.processTransaction() which depsolves again, and throws: yum.Errors.YumDownloadError

The standard out of the yum command looks like this:

Failed to download prestodelta for repository updates-testing: [Errno 256] No more mirrors to try.
error	package-download-failed	Errors were encountered while downloading packages.;libgfortran-4.8.1-1.fc19.x86_64: [Errno 256] No more mirrors to try.

Note: /var/cache/yum/x86_64/19/updates-testing/packages/libgfortran-4.8.1-1.fc19.x86_64.rpm really *does* exist, is the correct size and has the correct checksum.

Does the yum backend also have to explicitly disable the presto functionality even though the rpm file exists in entirety? This used to work in F18, so this is definitely a behaviour change in yum. Presumably the presto downloading functionality should with check if the whole rpm is already downloaded or just key off yb.conf.cache which is already set.

You can test this without dropping to the update-system state just by disabling all network connectivity and trying to execute a prepared transaction.

Richard.

Note: it might be the download is being requested as the GPG key does not exist and the package is unverified. If this is the case, a better exception or error message is certainly required, so that I could implement something like #c19 -- but also note manually installing the GPG key for fedora did not make the downloading error go away.

(In reply to Richard Hughes from comment #8)
> If we don't have the keys installed, what should the update client do? The
> GNOME designers don't want me to have a random GPG pop-up in the deskop
> asking for some random thing when the user is idle (the updates are
> downloaded by default on idle time) and it seems quite antisocial to just
> refuse the offline update with a message to the console like "You need to
> run the GUI updater manually first".

You trigger the offline updates process by consciously clicking "Install updates & restart". So that would be the obvious time to ask the user to accept the key.

1. Click "Install updates & restart"
2. Box pops up, offering options of installing the key(s), rebooting without attempting the update, or cancelling the reboot.

James

Well, that seems sensible. It'd be best to write it such that anything else which ever triggers the offline update process automatically picks it up too, though, as other ways of triggering the process are likely to come into existence (like, you know, when someone gets around to fixing https://bugzilla.redhat.com/show_bug.cgi?id=863592, HINT HINT :>)

(In reply to James Heather from comment #22)
> So that would be the obvious time to ask the user to accept the key.

Agreed, but I still don't see the point in asking the user to basically do:

---- Random Dialog ----

Do you trust distribution-that-you-just-downloaded with the key:

ACBDEF123456789 <random>

[ just do what I wanted ] [ I don't know what this button does]

-----------------------

The only sensible thing from a user experience point of view is for the DVD to automatically import the keys. That's not this bug tho. This bug is about yum needing network access when it shouldn't do.

(In reply to Richard Hughes from comment #24)
> The only sensible thing from a user experience point of view is for the DVD
> to automatically import the keys.

Yes, indeed so. Same goes for fedup. I upgraded to F19 using fedup, and it didn't import the keys for me, which is why my first offline update attempt failed (despite having network access). The keys should have been imported during the upgrade.

> That's not this bug tho. This bug is about
> yum needing network access when it shouldn't do.

Yes. And I have a feeling I've just mentioned another one...

> Failed to download prestodelta for repository updates-testing: [Errno 256] No more mirrors to try.

I've checked yum's DRPM code.  The only place we request prestodelta is DeltaInfo() constructor, and the "makecache" command.  It definitely should NOT use prestodelta metadata if all RPMs could be verified (implying pkgs == []).

yum/__init__:  downloadPkgs(pkglist)
        pkgs = []
        for po in pkglist:
            ..
            if verify_local(po):
                continue
            ..
            pkgs.append(po)

        # download presto metadata and use drpms
        presto = DeltaInfo(self, pkgs, adderror)

If Yum is trying to use prestodelta MD, then verify_local() must have failed for some reason (checksum, probably).  Note, with debug output on, Yum should also emit  "using local copy of <package>" for each cached pkg.

This seems to be back on Richard per c#26, setting needinfo on him.

commit cac9936e0950831905039c25f4b8e25ee4db3ce1
Author: Richard Hughes <richard>
Date:   Thu Jun 13 13:05:07 2013 +0100

    yum: Use yb.downloadPkgs() to download updates
    
    There were two bugs here:
    
    - Using repo.getPackage() did not check the package checksum, only the size,
      so it was possible to download a corrupt package and then not be able to
      apply the updates
    
    - By restricting to TS_UPDATE and TS_INSTALL we were ignoring any package that
      was obsoleting another which could miss out packages.
    
    Many thanks to Zdenek Pavlas for all the help in finding these issues.
    
    Resolves: https://bugzilla.redhat.com/show_bug.cgi?id=968936

New builds are here: http://koji.fedoraproject.org/koji/taskinfo?taskID=5498906

PackageKit-0.8.9-4.fc19 has been submitted as an update for Fedora 19.
https://admin.fedoraproject.org/updates/PackageKit-0.8.9-4.fc19

(In reply to Fedora Update System from comment #29)
> PackageKit-0.8.9-4.fc19 has been submitted as an update for Fedora 19.

This fixed the problem. I clean installed F19 TC3 DVD, tried to do offline updates (it crashed), then updated PK without installing gpg keys, performed offline updates again, and this time the system was updated.

One note though, PK no longer requires password for installing new applications, just for removal. I don't know whether it's intended.

Kamil: https://fedorahosted.org/fesco/ticket/1115

PackageKit-0.8.9-4.fc19 appears to fix this problem if it's applied before running the "Install Updates & Restart" from user menu option.

PackageKit-0.8.9-4.fc19 has been pushed to the Fedora 19 stable repository.  If problems still persist, please make note of it in this bug report.