Bug 968936 - network access is always required for offline update
network access is always required for offline update
Status: CLOSED ERRATA
Product: Fedora
Classification: Fedora
Component: yum (Show other bugs)
19
Unspecified Unspecified
unspecified Severity unspecified
: ---
: ---
Assigned To: packaging-team-maint
Fedora Extras Quality Assurance
AcceptedBlocker
:
Depends On:
Blocks: F19Blocker/F19FinalBlocker
  Show dependency treegraph
 
Reported: 2013-05-30 06:14 EDT by Petr Schindler
Modified: 2013-06-18 02:20 EDT (History)
21 users (show)

See Also:
Fixed In Version: PackageKit-0.8.9-4.fc19
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2013-06-18 02:20:45 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:


Attachments (Terms of Use)
journalctl -a (297.22 KB, text/plain)
2013-05-30 08:51 EDT, Vojtěch Boček
no flags Details
rpm -qa before starting offline updates (38.86 KB, text/plain)
2013-05-30 08:51 EDT, Vojtěch Boček
no flags Details
packages about to be installed (retrieved by calling yum update and cancelling it) (71.85 KB, text/plain)
2013-05-30 08:52 EDT, Vojtěch Boček
no flags Details

  None (edit)
Description Petr Schindler 2013-05-30 06:14:58 EDT
Description of problem:
Offline updates are killed (with SIGSEGV) during updating. After booting to 
system again there is a pop up message which says "Failed To Update".

Output of 'journalctl -u packagekit-offline-update.service -a':

May 30 11:04:45 hostname systemd[1]: packagekit-offline-update.service: main process exited, code=killed, status=11/SEGV
May 30 11:04:45 hostname systemd[1]: Unit packagekit-offline-update.service entered failed state.
May 30 11:04:45 hostname systemd[1]: Triggering OnFailure= dependencies of packagekit-offline-update.service.

Version-Release number of selected component (if applicable):
PackageKit-0.8.9-1.fc19.x86_64

How reproducible:
tested few times on bare metal and in KVM

Steps to Reproduce:
1. install system from Beta DVD
2. after boot choose 'Install Updates & Restart" from user menu (up right corner)

Actual results:
Update fails

Expected results:
system is updated

Additional info:
Comment 1 Fedora Blocker Bugs Application 2013-05-30 06:21:59 EDT
Proposed as a Blocker for 19-final by Fedora user pschindl using the blocker tracking app because:

 Bug violates the criterion: "The installed system must be able to download and install updates with yum and with the default graphical package manager in all release-blocking desktops." The option to do offline updates is offered by default in GNOME.
Comment 2 Vojtěch Boček 2013-05-30 08:51:01 EDT
Created attachment 754837 [details]
journalctl -a

Reproduced in KVM with Fedora 19 Beta RC4, adding logs from clean DVD installation.
Comment 3 Vojtěch Boček 2013-05-30 08:51:28 EDT
Created attachment 754838 [details]
rpm -qa before starting offline updates
Comment 4 Vojtěch Boček 2013-05-30 08:52:07 EDT
Created attachment 754840 [details]
packages about to be installed (retrieved by calling yum update and cancelling it)
Comment 5 Vojtěch Boček 2013-05-30 10:12:41 EDT
We've also tried to set SElinux to permissive, but it didn't help.
Comment 6 Adam Williamson 2013-05-30 13:43:39 EDT
Discussed at 2013-05-30 blocker review meeting: http://meetbot.fedoraproject.org/fedora-blocker-review/2013-05-30/f19final-blocker-review-1.1.2013-05-30-16.02.log.txt . Accepted as a blocker per criterion "The installed system must be able to download and install updates with yum and with the default graphical package manager in all release-blocking desktops", on the understanding it really means 'default update method for release-blocking desktops'.

What's the 'default' update method for GNOME is kind of an open question: this is right there on the user menu, but the 'update available' notifications still run gpk-update-viewer, the online update method (I've had a bug report in on that for several months). So arguably GNOME is currently providing *two* update methods, and both ought to work.

I have a theory that this fails if the Fedora GPG key hasn't been imported yet - so only on DVD installs and only until you do some operation with yum or pk interactively. But we need to test that.
Comment 7 Adam Williamson 2013-05-30 14:43:37 EDT
Actually, this seems to be failing for me on a fresh DVD Beta install every try, even after I install a package via yum to get the GPG key imported. Looks like the function is just flat broken somehow.
Comment 8 Richard Hughes 2013-06-06 04:33:54 EDT
I will fix the crash (just downloading the DVD now to work out the exact place it's crashing) but the bigger problem is if the user hasn't accepted the Fedora GPG key before doing the offline update, it's always going to fail as the offline update by definition can't stop and ask the user questions.

Of course, the obvious thing would be to import the Fedora keys on the DVD like we do on the Live media on the assumption you either trust the media or you don't, and if you don't then the lack of a GPG key isn't your primary concern.

If we don't have the keys installed, what should the update client do? The GNOME designers don't want me to have a random GPG pop-up in the deskop asking for some random thing when the user is idle (the updates are downloaded by default on idle time) and it seems quite antisocial to just refuse the offline update with a message to the console like "You need to run the GUI updater manually first".

Ideas welcome.
Comment 9 Kamil Páral 2013-06-06 06:34:59 EDT
This could be a good opportunity to finally solve bug 748320.
Comment 10 Richard Hughes 2013-06-06 09:38:37 EDT
Hmm, I downloaded the beta DVD and tried to do an offline update of a single package and of the whole system, and it worked without a hitch, both with a package that existed in @fedora and also one that only existed in @updates. I did notice a race between the offline updates and the packagekitd daemon, which I've fixed in http://koji.fedoraproject.org/koji/buildinfo?buildID=424924 but that would only produce a scary warning, not a segfault. Can you describe *exactly* how to reproduce this bug please? Include things like what options you chose in anaconda, thanks. I'll continue to debug here.
Comment 11 Richard Hughes 2013-06-06 11:59:28 EDT
Okay, I've fixed a few bugs in PK, but nothing that precisely matches this bug. I'd appreciate it if you could download the PackageKit packages from http://people.freedesktop.org/~hughsient/fedora/19/x86_64/ and then do something like "sudo rpm -Fvh PackageKit*.rpm" -- if this fixes this, or doesn't fix this I'd love to know. Thanks!
Comment 12 Adam Williamson 2013-06-06 15:38:57 EDT
hughsie: my reproducer was simply to do a default install from the DVD - only setting the Installation Destination and root password spokes, not touching any of the others - set up a user in g-i-s, and then trigger an offline update via the user menu. I'll check if I can still reproduce with and without your updated PK shortly.
Comment 13 Adam Williamson 2013-06-06 15:45:25 EDT
kparal: it seems de rigeur to mention https://bugzilla.redhat.com/show_bug.cgi?id=998 at this point :P
Comment 14 Josef Skladanka 2013-06-07 07:12:13 EDT
Reproduced on clean DVD install of F19-TC1. The update fails even after the action suggested in comment #11. The output of  `# journalctl -u packagekit-offline-update.service -a` where the part after Reboot is with the updated PackageKit.

-- Logs begin at Fri 2013-06-07 06:49:17 EDT, end at Fri 2013-06-07 07:08:21 EDT. --
Jun 07 06:53:54 localhost.localdomain systemd[1]: packagekit-offline-update.service: main process exited, code=killed, status=11/SEGV
Jun 07 06:53:54 localhost.localdomain systemd[1]: Unit packagekit-offline-update.service entered failed state.
Jun 07 06:53:54 localhost.localdomain systemd[1]: Triggering OnFailure= dependencies of packagekit-offline-update.service.
^[[1;39m-- Reboot --^[[0m
Jun 07 07:05:20 localhost.localdomain pk-offline-update[361]: 0: /lib64/libc.so.6  (__restore_rt+0x0) [0x7fad8b43f88f]
Jun 07 07:05:20 localhost.localdomain systemd[1]: packagekit-offline-update.service: main process exited, code=killed, status=5/TRAP
Jun 07 07:05:20 localhost.localdomain systemd[1]: Unit packagekit-offline-update.service entered failed state.
Jun 07 07:05:20 localhost.localdomain systemd[1]: Triggering OnFailure= dependencies of packagekit-offline-update.service.
Comment 15 Richard Hughes 2013-06-07 09:49:23 EDT
So, from further testing it seems doing a small number of updates works, but doing over 300 segfaults for some reason. I'd appreciate people trying my test builds, as those should have the backtrace in the journalctl output.
Comment 16 Richard Hughes 2013-06-07 11:32:27 EDT
Okay, progress:

commit ad1e920ba588fc8da8a86090ba196597cbfa6135
Author: Richard Hughes <richard@hughsie.com>
Date:   Fri Jun 7 16:29:31 2013 +0100

    lib: Actually return the error if any PkClient methods failed
    
    Giovanni changed the g_new0() to a local helper object in commit
    771f87943d3fc0a8579eee42ade373052208c8e2, but nothing actually set the return
    value to NULL for failure. This meant we returned random memory for the
    PkResults on error, and blew up with a segfault as soon as we tried accessing
    the private structure.
    
    It seems we could rely on the structure being zeroed by coincidence for small
    numbers of packages, but for larger number of packages we got junk.

With that commit I get the more friendly (but equally buggy):

Failed To Update
Network access was required but not available.

Richard.
Comment 17 Josef Skladanka 2013-06-07 13:52:39 EDT
(In reply to Richard Hughes from comment #15)

Just to make it clear - the part of logs after "REBOOT" is with the updated packaeges. Are there any other sources I should look at for the backtrace?
Comment 18 Adam Williamson 2013-06-07 20:14:26 EDT
richard: "Failed To Update
Network access was required but not available."

*that* sounds like the error I was getting in the 'missing GPG key' case. Does getting the GPG key installed fix it?
Comment 19 Matthias Clasen 2013-06-09 10:26:44 EDT
Richard, can we at least arrange things so we show a

System update failed
You need to install the repository GPG key first.
                                        [Install]

Notification after the failed update ?
Comment 20 Richard Hughes 2013-06-10 06:59:20 EDT
(In reply to Adam Williamson from comment #18)
> Does getting the GPG key installed fix it?

No, after investigating this is a yum issue. Basically, the desktop client does the equivalent of:

pkcon --only-download update foo

This makes sure that all the repodata is up to date (including filelists), depsolves the update transaction and downloads foo and any additional packages the transaction needs to complete. 'foo' is written to the prepared-updates file, and the user is rebooted into the offline updates mode.

In this mode, the prepared-updates file is read, and a transaction is setup so that 'foo' is updated. Remember all the metadata and packages are downloaded. When in offline updates mode, there is no network, and so this is set in the yumBackend.py file:

            self.yumbase.conf.cache = 1
            for repo in self.yumbase.repos.listEnabled():
                repo.metadata_expire = -1  # never refresh

So, basically, work from cache and never expire the metadata. We setup the transaction and call yb.processTransaction() which depsolves again, and throws: yum.Errors.YumDownloadError

The standard out of the yum command looks like this:

Failed to download prestodelta for repository updates-testing: [Errno 256] No more mirrors to try.
error	package-download-failed	Errors were encountered while downloading packages.;libgfortran-4.8.1-1.fc19.x86_64: [Errno 256] No more mirrors to try.

Note: /var/cache/yum/x86_64/19/updates-testing/packages/libgfortran-4.8.1-1.fc19.x86_64.rpm really *does* exist, is the correct size and has the correct checksum.

Does the yum backend also have to explicitly disable the presto functionality even though the rpm file exists in entirety? This used to work in F18, so this is definitely a behaviour change in yum. Presumably the presto downloading functionality should with check if the whole rpm is already downloaded or just key off yb.conf.cache which is already set.

You can test this without dropping to the update-system state just by disabling all network connectivity and trying to execute a prepared transaction.

Richard.
Comment 21 Richard Hughes 2013-06-10 07:08:05 EDT
Note: it might be the download is being requested as the GPG key does not exist and the package is unverified. If this is the case, a better exception or error message is certainly required, so that I could implement something like #c19 -- but also note manually installing the GPG key for fedora did not make the downloading error go away.
Comment 22 James Heather 2013-06-10 12:54:02 EDT
(In reply to Richard Hughes from comment #8)
> If we don't have the keys installed, what should the update client do? The
> GNOME designers don't want me to have a random GPG pop-up in the deskop
> asking for some random thing when the user is idle (the updates are
> downloaded by default on idle time) and it seems quite antisocial to just
> refuse the offline update with a message to the console like "You need to
> run the GUI updater manually first".

You trigger the offline updates process by consciously clicking "Install updates & restart". So that would be the obvious time to ask the user to accept the key.

1. Click "Install updates & restart"
2. Box pops up, offering options of installing the key(s), rebooting without attempting the update, or cancelling the reboot.

James
Comment 23 Adam Williamson 2013-06-10 20:03:39 EDT
Well, that seems sensible. It'd be best to write it such that anything else which ever triggers the offline update process automatically picks it up too, though, as other ways of triggering the process are likely to come into existence (like, you know, when someone gets around to fixing https://bugzilla.redhat.com/show_bug.cgi?id=863592, HINT HINT :>)
Comment 24 Richard Hughes 2013-06-11 04:19:26 EDT
(In reply to James Heather from comment #22)
> So that would be the obvious time to ask the user to accept the key.

Agreed, but I still don't see the point in asking the user to basically do:

---- Random Dialog ----

Do you trust distribution-that-you-just-downloaded with the key:

ACBDEF123456789 <random@person.com>

[ just do what I wanted ] [ I don't know what this button does]

-----------------------

The only sensible thing from a user experience point of view is for the DVD to automatically import the keys. That's not this bug tho. This bug is about yum needing network access when it shouldn't do.
Comment 25 James Heather 2013-06-11 04:26:04 EDT
(In reply to Richard Hughes from comment #24)
> The only sensible thing from a user experience point of view is for the DVD
> to automatically import the keys.

Yes, indeed so. Same goes for fedup. I upgraded to F19 using fedup, and it didn't import the keys for me, which is why my first offline update attempt failed (despite having network access). The keys should have been imported during the upgrade.

> That's not this bug tho. This bug is about
> yum needing network access when it shouldn't do.

Yes. And I have a feeling I've just mentioned another one...
Comment 26 Zdeněk Pavlas 2013-06-11 10:11:08 EDT
> Failed to download prestodelta for repository updates-testing: [Errno 256] No more mirrors to try.

I've checked yum's DRPM code.  The only place we request prestodelta is DeltaInfo() constructor, and the "makecache" command.  It definitely should NOT use prestodelta metadata if all RPMs could be verified (implying pkgs == []).

yum/__init__:  downloadPkgs(pkglist)
        pkgs = []
        for po in pkglist:
            ..
            if verify_local(po):
                continue
            ..
            pkgs.append(po)

        # download presto metadata and use drpms
        presto = DeltaInfo(self, pkgs, adderror)

If Yum is trying to use prestodelta MD, then verify_local() must have failed for some reason (checksum, probably).  Note, with debug output on, Yum should also emit  "using local copy of <package>" for each cached pkg.
Comment 27 Adam Williamson 2013-06-12 14:03:43 EDT
This seems to be back on Richard per c#26, setting needinfo on him.
Comment 28 Richard Hughes 2013-06-13 08:19:09 EDT
commit cac9936e0950831905039c25f4b8e25ee4db3ce1
Author: Richard Hughes <richard@hughsie.com>
Date:   Thu Jun 13 13:05:07 2013 +0100

    yum: Use yb.downloadPkgs() to download updates
    
    There were two bugs here:
    
    - Using repo.getPackage() did not check the package checksum, only the size,
      so it was possible to download a corrupt package and then not be able to
      apply the updates
    
    - By restricting to TS_UPDATE and TS_INSTALL we were ignoring any package that
      was obsoleting another which could miss out packages.
    
    Many thanks to Zdenek Pavlas for all the help in finding these issues.
    
    Resolves: https://bugzilla.redhat.com/show_bug.cgi?id=968936

New builds are here: http://koji.fedoraproject.org/koji/taskinfo?taskID=5498906
Comment 29 Fedora Update System 2013-06-14 08:57:28 EDT
PackageKit-0.8.9-4.fc19 has been submitted as an update for Fedora 19.
https://admin.fedoraproject.org/updates/PackageKit-0.8.9-4.fc19
Comment 30 Kamil Páral 2013-06-14 09:53:52 EDT
(In reply to Fedora Update System from comment #29)
> PackageKit-0.8.9-4.fc19 has been submitted as an update for Fedora 19.

This fixed the problem. I clean installed F19 TC3 DVD, tried to do offline updates (it crashed), then updated PK without installing gpg keys, performed offline updates again, and this time the system was updated.

One note though, PK no longer requires password for installing new applications, just for removal. I don't know whether it's intended.
Comment 31 Adam Williamson 2013-06-14 13:18:02 EDT
Kamil: https://fedorahosted.org/fesco/ticket/1115
Comment 32 Chris Murphy 2013-06-17 23:17:35 EDT
PackageKit-0.8.9-4.fc19 appears to fix this problem if it's applied before running the "Install Updates & Restart" from user menu option.
Comment 33 Fedora Update System 2013-06-18 02:20:45 EDT
PackageKit-0.8.9-4.fc19 has been pushed to the Fedora 19 stable repository.  If problems still persist, please make note of it in this bug report.

Note You need to log in before you can comment on or make changes to this bug.