Bug 1513111 - dnf system-upgrade reboot fails: unable to upgrade from f26 to f27
Summary: dnf system-upgrade reboot fails: unable to upgrade from f26 to f27
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: Fedora
Classification: Fedora
Component: dnf-plugins-extras
Version: 26
Hardware: x86_64
OS: Linux
unspecified
unspecified
Target Milestone: ---
Assignee: rpm-software-management
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2017-11-14 19:05 UTC by aannoaanno
Modified: 2018-03-06 14:23 UTC (History)
24 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2017-12-05 08:37:32 UTC
Type: Bug
Embargoed:


Attachments (Terms of Use)

Description aannoaanno 2017-11-14 19:05:34 UTC
This is what I find in `/var/log/dnf.log`:

```
2017-11-14T17:56:17Z DEBUG Kann nicht herunterladen 'https://mirrors.fedoraproject.org/metalink?repo=updates-released-f27
&arch=x86_64': Cannot prepare internal mirrorlist: Curl error (6): Couldn't resolve host name for https://mirrors.fedorap
roject.org/metalink?repo=updates-released-f27&arch=x86_64 [Could not resolve host: mirrors.fedoraproject.org].
2017-11-14T17:56:17Z DDEBUG Cleaning up.
2017-11-14T17:56:17Z SUBDEBUG
Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/dnf/repo.py", line 229, in _perform
    return super(_Handle, self).perform(result)
  File "/usr/lib64/python3.6/site-packages/librepo/__init__.py", line 1522, in perform
    _librepo.Handle.perform(self, result)
librepo.LibrepoException: (8, "Cannot prepare internal mirrorlist: Curl error (6): Couldn't resolve host name for https:/
/mirrors.fedoraproject.org/metalink?repo=updates-released-f27&arch=x86_64 [Could not resolve host: mirrors.fedoraproject.
org]", 'An Curl handle error')
```

However, on my command line, curl works:
```
$ curl https://mirrors.fedoraproject.org/metalink?repo=updates-released-f27&arch=x86_6
<?xml version="1.0" encoding="utf-8"?>
<metalink version="3.0" xmlns="http://www.metalinker.org/" type="dynamic" pubdate="Tue, 14 Nov 2017 18:51:33 GMT" generator="mirrormanager" xmlns:mm0="http://fedorahosted.org/mirrormanager">
<!--
# either path=, or repo= and arch= must be specified
-->
</metalink>
```
I tried if this a IPv6 problem in the line of https://ask.fedoraproject.org/en/question/32319/could-not-resolve-host-mirrorsfedoraprojectorg/ but:

```
$ curl -6 https://mirrors.fedoraproject.org/metalink?repo=updates-released-f27&arch=x86_6
<?xml version="1.0" encoding="utf-8"?>
<metalink version="3.0" xmlns="http://www.metalinker.org/" type="dynamic" pubdate="Tue, 14 Nov 2017 18:53:36 GMT" generator="mirrormanager" xmlns:mm0="http://fedorahosted.org/mirrormanager">
<!--
# either path=, or repo= and arch= must be specified
-->
</metalink>
```

Comment 2 Sam Varshavchik 2017-11-16 15:45:17 UTC
Is this happening during "dnf system-upgrade reboot"?

It appears that during the reboot to install the upgrade, something is attempting to resync with all the repositories being upgraded.

In my case, I have a local repository on the same server being upgraded. Normally, apache serves the repository via http, nothing exotic. But, "dnf system-upgrade reboot" does not, of course, start apache before it attempts to upgrade all the packages, so I end up with:

Nov 16 08:10:51 octopus.email-scan.com dnf[910]: Failed to synchronize cache for repo 'my', disabling.
Nov 16 08:10:51 octopus.email-scan.com dnf[910]: Last metadata expiration check: 0:54:41 ago on Thu 16 Nov 2017 07:16:08 AM EST.
Nov 16 08:10:51 octopus.email-scan.com system-python[910]: Starting system upgrade. This will take a while.
Nov 16 08:11:01 octopus.email-scan.com dnf[910]: Unable to match package: courier-0.78.1-1.fc27.x86_64 my
Nov 16 08:11:01 octopus.email-scan.com dnf[910]: Unable to match package: courier-authlib-0.68.0.20170908-1.fc27.x86_64 my
Nov 16 08:11:01 octopus.email-scan.com dnf[910]: Unable to match package: courier-authlib-devel-0.68.0.20170908-1.fc27.x86_64 my
Nov 16 08:11:01 octopus.email-scan.com dnf[910]: Unable to match package: courier-authlib-userdb-0.68.0.20170908-1.fc27.x86_64 my
Nov 16 08:11:01 octopus.email-scan.com dnf[910]: Unable to match package: courier-imapd-0.78.1-1.fc27.x86_64 my
Nov 16 08:11:01 octopus.email-scan.com dnf[910]: Unable to match package: courier-maildrop-0.78.1-1.fc27.x86_64 my
Nov 16 08:11:01 octopus.email-scan.com dnf[910]: Unable to match package: courier-mlm-0.78.1-1.fc27.x86_64 my
Nov 16 08:11:01 octopus.email-scan.com dnf[910]: Unable to match package: courier-mlm-web-0.78.1-1.fc27.x86_64 my
Nov 16 08:11:01 octopus.email-scan.com dnf[910]: Unable to match package: courier-mysql-0.78.1-1.fc27.x86_64 my
Nov 16 08:11:01 octopus.email-scan.com dnf[910]: Unable to match package: courier-pop3d-0.78.1-1.fc27.x86_64 my
Nov 16 08:11:02 octopus.email-scan.com dnf[910]: Unable to match package: courier-unicode-2.0-5.fc27.x86_64 my
Nov 16 08:11:02 octopus.email-scan.com dnf[910]: Unable to match package: courier-unicode-devel-2.0-5.fc27.x86_64 my
Nov 16 08:11:02 octopus.email-scan.com dnf[910]: Unable to match package: courier-webadmin-0.78.1-1.fc27.x86_64 my
Nov 16 08:11:02 octopus.email-scan.com dnf[910]: Unable to match package: libcxxbase-0.9-1.x86_64 my
Nov 16 08:11:02 octopus.email-scan.com dnf[910]: Unable to match package: libcxxbase-devel-0.9-1.x86_64 my
Nov 16 08:11:02 octopus.email-scan.com dnf[910]: Unable to match package: libcxxbase-httpd-0.9-1.x86_64 my
Nov 16 08:11:02 octopus.email-scan.com dnf[910]: Unable to match package: libcxxbase-ltdl-0.9-1.x86_64 my
Nov 16 08:11:02 octopus.email-scan.com dnf[910]: Unable to match package: stasher-0.06-1.x86_64 my
Nov 16 08:11:02 octopus.email-scan.com dnf[910]: Unable to match package: stasher-devel-0.06-1.x86_64 my
Nov 16 08:11:02 octopus.email-scan.com dnf[910]: Unable to match package: stasher-distreboot-0.05-1.x86_64 my
Nov 16 08:11:02 octopus.email-scan.com dnf[910]: Unable to match package: stasher-perl-0.06-1.x86_64 my
Nov 16 08:11:02 octopus.email-scan.com dnf[910]: Error: Unable to match some of packages
Nov 16 08:11:02 octopus.email-scan.com systemd[1]: dnf-system-upgrade.service: Main process exited, code=exited, status=1/FAILURE

It appears that it is impossible to upgrade a server if it contains one of the repositories being used on the same server, because "dnf system-upgrade reboot" now apparently requires everything to be online. It seems that it is no longer sufficient for the server being upgraded to be online and reachable only for "dnf system-upgrade download", but it must be completely online and all repositories must be reachable for the reboot as well.

This doesn't sound right to me.

Comment 3 Rahul 2017-11-16 17:42:23 UTC
While upgrading from F25 to F27, I am also getting the same issue ie. while the download of the packages happens properly, the reboot (viz. "dnf system-upgrade reboot") does not trigger the upgrade and instead, just boots the system into the existing F25 installation. Have tried stepping through the log (/var/log/dnf.log) but could not spot any issues. Let me know if this needs to be shared.

Comment 4 Gordon Messmer 2017-11-16 23:43:08 UTC
The bug appears to be that dnf processes the last metadata update time while in the offline system-upgrade mode.  I was able to work around the problem by re-running "dnf system-upgrade download --releasever 27" immediately before "dnf system-upgrade reboot"

Comment 5 Sam Varshavchik 2017-11-17 03:28:14 UTC
That does seem to be the case. Even though I did this, it's not much use if you set metadata_expire to 60 seconds, or so. I push out packages to my local LAN repo all the time, and then install them right away, so I need short expirations.

I bumped up the metadata expire and refreshed, this allowed the system to update.

Comment 6 Rahul 2017-11-17 16:51:44 UTC
Still no luck, I am afraid, with what Gordon has suggested above (ie. run the "dnf system-upgrade download --refresh --allowerasing --releasever=27" and "dnf system-upgrade reboot" consecutively). What is happening is that during the reboot, a message is shown to the effect that the system will be upgraded and this may take a while but after a few minutes, it just goes ahead and boots into the existing instance without any changes. 

Also checked the "metadata_expire" parameter - for me it is set to 7d so do not thing this is an issue, at least in my case.

Comment 7 Sam Varshavchik 2017-11-17 18:40:24 UTC
Use the log parameter to 'dnf system-upgrade' (man dnf.plugin.system-upgrade for the details) to retrieve the logs from the reboot. Could be other issues.

Comment 8 Alexander Korsunsky 2017-11-17 19:04:48 UTC
I have the same problem, DNF tries to update the cache in the upgrade system, which obviously fails because there is no network connection.

Running `dnf system-upgrade download --refresh --allowerasing --releasever=27` twice did NOT help, neither did setting `metadata_expire=-1` in dnf.conf.


The real question is: why does DNF even try to update during the live system?

Comment 9 Thomas Meyer 2017-11-17 20:25:14 UTC
another fedora release, anoher system-upgrade bug...
Does somebody actually test this?

I fixed the problem by commenting the raise exception in dnf/repo.py:931

Comment 10 Andreas Schneider 2017-11-18 08:59:06 UTC
I really don't understand why Fedora just uses zypper. Instead it needs to rewrite it everything in python.

Comment 11 Wolfram Wagner 2017-11-18 18:20:06 UTC
Same issue here. Most annoying is, that the cached files are deleted and need to be downloaded again.
I wonder if I made something dangerous when I simply

dnf update --releasever=27 --best --allowerasing
followed by reboot
dnf distro-sync
rpmconf -a
dnf repoquery --unsatisfied
dnf repoquery --duplicated
dnf list extras 
touch /.autorelabel

I am on 27 now, but wonder, if this process destroys something or is dangerous.

Comment 12 Lionel F. Lebeau 2017-11-21 16:25:07 UTC
With Thomas Meyer solution I was able to enter the update sequence.
I just hope that the restart will be OK.

Comment 13 Lionel F. Lebeau 2017-11-21 18:31:07 UTC
(In reply to Lionel F. Lebeau from comment #12)
> With Thomas Meyer solution I was able to enter the update sequence.
> I just hope that the restart will be OK.

I confirm that this solution resolved my problem.
I am now on 27.

What is strange is that I had that problem on only one computer. The two others were upgraded wwithout any trouble.

Comment 14 Michael Lipp 2017-11-21 19:02:11 UTC
Same problem here. "fedora-upgrade" (manual, online) worked for me.

Comment 15 Alex Villacís Lasso 2017-11-21 19:11:29 UTC
(In reply to Lionel F. Lebeau from comment #13)
> (In reply to Lionel F. Lebeau from comment #12)
> > With Thomas Meyer solution I was able to enter the update sequence.
> > I just hope that the restart will be OK.
> 
> I confirm that this solution resolved my problem.
> I am now on 27.
> 
> What is strange is that I had that problem on only one computer. The two
> others were upgraded wwithout any trouble.

I have not yet applied the solution by Thomas Meyer in any of my computers.

In my case, one computer upgraded correctly and two failed to upgrade. It so happens that the one that upgraded correctly has only a wired (Ethernet) connection, and the two that failed have only WiFi as their main connection. Is this the same in your case?

Comment 16 Lionel F. Lebeau 2017-11-21 20:26:37 UTC
(In reply to Alex Villacís Lasso from comment #15)
> (In reply to Lionel F. Lebeau from comment #13)
> > (In reply to Lionel F. Lebeau from comment #12)
> > > With Thomas Meyer solution I was able to enter the update sequence.
> > > I just hope that the restart will be OK.
> > 
> > I confirm that this solution resolved my problem.
> > I am now on 27.
> > 
> > What is strange is that I had that problem on only one computer. The two
> > others were upgraded wwithout any trouble.
> 
> I have not yet applied the solution by Thomas Meyer in any of my computers.
> 
> In my case, one computer upgraded correctly and two failed to upgrade. It so
> happens that the one that upgraded correctly has only a wired (Ethernet)
> connection, and the two that failed have only WiFi as their main connection.
> Is this the same in your case?

No. They all have a wired connection. On that was OK even goes through a Win 7 relay.
I don't think the connection is important, except if thre is a timeout, but it shouldn't happend every time.

Comment 17 Rahul 2017-11-25 06:36:11 UTC
Here is the snapshot from the end of my dnf log (/var/log/dnf.log) :

"21180 Nov 25 10:15:37 DEBUG Completion plugin: Generating completion cache...
  21181 Nov 25 10:15:37 INFO Complete!
  21182 Nov 25 10:15:37 INFO Download complete! Use 'dnf system-upgrade reboot' to start the upgrade.
  21183 Nov 25 10:15:37 DDEBUG Cleaning up.
  21184 Nov 25 10:15:37 INFO The downloaded packages were saved in cache until the next successful transaction.
  21185 Nov 25 10:15:37 INFO You can remove cached packages by executing 'dnf clean packages'.
  21186 Nov 25 11:08:23 INFO --- logging initialized ---
  21187 Nov 25 11:08:23 DDEBUG timer: config: 3 ms
  21188 Nov 25 11:08:23 DEBUG cachedir: /var/cache/dnf
  21189 Nov 25 11:08:23 DEBUG Loaded plugins: config-manager, generate_completion_cache, noroot, copr, playground, debuginfo-install, reposync, system-upgrade, builddep, protected_packa        needs-restarting, Query, download
  21190 Nov 25 11:08:23 DEBUG DNF version: 1.1.10
  21191 Nov 25 11:08:23 DDEBUG Command: dnf makecache timer
  21192 Nov 25 11:08:23 DDEBUG Installroot: /
  21193 Nov 25 11:08:23 DDEBUG Releasever: 25
  21194 Nov 25 11:08:23 DDEBUG Base command: makecache
  21195 Nov 25 11:08:23 DDEBUG Extra commands: ['timer']
  21196 Nov 25 11:08:23 DEBUG Making cache files for all metadata files.
  21197 Nov 25 11:08:23 INFO Metadata cache refreshed recently.
  21198 Nov 25 11:08:23 DDEBUG Cleaning up.
"
The reboot, unfortunately, still did not start the upgrade process from (F25 to F27). 

Any suggestions pl. ?

Comment 18 Marek Blaha 2017-11-30 07:59:09 UTC
Please try version 2.0.4 of system-upgrade plugin (should be in fedora updates). There were changes fixing issues with dnf trying to access network during system-upgrade reboot phase.

Comment 19 Alexander Korsunsky 2017-11-30 11:34:18 UTC
(In reply to Marek Blaha from comment #18)
> Please try version 2.0.4 of system-upgrade plugin (should be in fedora
> updates). There were changes fixing issues with dnf trying to access network
> during system-upgrade reboot phase.

Nope. Still the same error:

2017-11-30T11:06:11Z DEBUG Cannot download 'https://negativo17.org/repos/nvidia/fedora-27/x86_64/': Cannot download repomd.xml: Cannot download repodata/repomd.xml: All mirrors were tried.
2017-11-30T11:06:11Z DEBUG Cannot download 'https://mirrors.fedoraproject.org/metalink?repo=updates-released-f27&arch=x86_64': Cannot prepare internal mirrorlist: Curl error (6): Couldn't resolve host name for h
ttps://mirrors.fedoraproject.org/metalink?repo=updates-released-f27&arch=x86_64 [Could not resolve host: mirrors.fedoraproject.org].
2017-11-30T11:06:11Z DDEBUG Cleaning up.
2017-11-30T11:06:11Z SUBDEBUG 
Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/dnf/repo.py", line 229, in _perform
    return super(_Handle, self).perform(result)
  File "/usr/lib64/python3.6/site-packages/librepo/__init__.py", line 1522, in perform
    _librepo.Handle.perform(self, result)
librepo.LibrepoException: (8, "Cannot prepare internal mirrorlist: Curl error (6): Couldn't resolve host name for https://mirrors.fedoraproject.org/metalink?repo=updates-released-f27&arch=x86_64 [Could not resol
ve host: mirrors.fedoraproject.org]", 'An Curl handle error')

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/dnf/repo.py", line 909, in load
    if self._try_revive():
  File "/usr/lib/python3.6/site-packages/dnf/repo.py", line 857, in _try_revive
    return self._try_revive_by_metalink()
  File "/usr/lib/python3.6/site-packages/dnf/repo.py", line 810, in _try_revive_by_metalink
    handle._perform()
  File "/usr/lib/python3.6/site-packages/dnf/repo.py", line 233, in _perform
    raise _DetailedLibrepoError(exc, source)
dnf.repo._DetailedLibrepoError

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/dnf/cli/main.py", line 64, in main
    return _main(base, args, cli_class, option_parser_class)
  File "/usr/lib/python3.6/site-packages/dnf/cli/main.py", line 99, in _main
    return cli_run(cli, base)
  File "/usr/lib/python3.6/site-packages/dnf/cli/main.py", line 115, in cli_run
    cli.run()
  File "/usr/lib/python3.6/site-packages/dnf/cli/cli.py", line 1010, in run
    self._process_demands()
  File "/usr/lib/python3.6/site-packages/dnf/cli/cli.py", line 763, in _process_demands
    load_available_repos=self.demands.available_repos)
  File "/usr/lib/python3.6/site-packages/dnf/base.py", line 353, in fill_sack
    self._add_repo_to_sack(r)
  File "/usr/lib/python3.6/site-packages/dnf/base.py", line 118, in _add_repo_to_sack
    repo.load()
  File "/usr/lib/python3.6/site-packages/dnf/repo.py", line 931, in load
    raise dnf.exceptions.RepoError(msg)
dnf.exceptions.RepoError: Failed to synchronize cache for repo 'updates'
2017-11-30T11:06:11Z CRITICAL Error: Failed to synchronize cache for repo 'updates'

python3-dnf-plugin-system-upgrade-2.0.4-1.fc26.noarch


> Component: dnf-plugin-system-upgrade → dnf-plugins-extras

Why did you reassign the component? This seems like an error that is quite specific to  dnf-plugin-system-upgrade.

ps.: I will not be available for testing anymore, I will just upgrade using Thomas Meyer's workaround from Comment #9.

Comment 20 Marek Blaha 2017-11-30 14:12:13 UTC
Oh, sorry. You will need also this patch for dnf which fixes cacheonly behavior (it is now being released):

https://github.com/rpm-software-management/dnf/pull/990

And component is changed just because system-upgrade plugin is now part of dnf-plugins-extras.

Comment 21 Ivan Cormeau 2017-11-30 15:32:00 UTC
I just had the same problem on a wire-connected x86_64  using repos fedora, fedora-updates, rpmfusion-free, rpmfusion-free-updates, rpmfusion-nonfree and rpmfusion-nonfree-updates. Using Thomas Meyer's workaround from Comment #9 worked like a charm. I only noticed some timeouts on mirrors while making the final dnf upgrade on FC26 before launching dnf system-upgrade download --releasever=27.

Comment 22 Kay Schubert 2017-12-04 21:38:07 UTC
I had same error. For me it went away when I removed file
/var/lib/dnf/system-upgrade/expired_repos.json
It contained several entries, even though I had done a "dnf upgrade --refresh" immediately before without any error messages.

Comment 23 Jaroslav Mracek 2017-12-05 08:37:32 UTC
I believe that dnf-2.7.5-2 with dnf-plugins-extras-2.0.4-1 should solve the issue or issues. Please don't hesitate to reopen the bug report if the problem persists.

Comment 24 Alexander Korsunsky 2017-12-06 13:58:57 UTC
Worked for me with dnf-2.7.5-2 and dnf-plugins-extras-2.0.4-1 from updates-testing.

Comment 25 Rahul 2017-12-31 19:45:34 UTC
Apologies but could you kindly give the exact sequence of commands for doing the above ? 

Tried :
# dnf --enablerepo=updates-testing install dnf.noarch 
but did not work :(

Also, I am trying to move from F25 to F27 and that is why the note in Comment #9 above does not seem to work for me since I have Python 3.5 installed. Tried replicating the same in my version at line no. 848 but not it did not work either.

Comment 26 Jaroslav Mracek 2018-01-02 07:59:55 UTC
The problem in the bug report only affect dnf-plugin-system-upgrade-2.0.0 or higher that is only available for Fedora 26+. Rahul I would normally ask you to open new bug report for system upgrade for Fedora 25, but it is already End Of Life, therefore we cannot even release a new bug fix for it.

I could recommend to you for f25-27 upgrade 25 to 26 then 26 to 27 upgrade.

Comment 27 Rahul 2018-02-10 09:41:23 UTC
Tried moving from F25 to F26 but that is also failing with the same error :(

Comment 28 Jaroslav Mracek 2018-03-06 14:23:51 UTC
Rahul, please can you provide log /var/log/dnf.log with both download and reboot transactions?


Note You need to log in before you can comment on or make changes to this bug.