Created attachment 1324826 [details] snippet of dnf.log I'm not able to upgrade Fedora 26 to Fedora 27 using dnf system-upgrade. After running 'dnf system-upgrade reboot', system restarts twice and boots back to Fedora 26. There is traceback at the end of dnf.log: Traceback (most recent call last): File "/usr/lib/python3.6/site-packages/dnf/cli/main.py", line 64, in main return _main(base, args, cli_class, option_parser_class) File "/usr/lib/python3.6/site-packages/dnf/cli/main.py", line 99, in _main return cli_run(cli, base) File "/usr/lib/python3.6/site-packages/dnf/cli/main.py", line 115, in cli_run cli.run() File "/usr/lib/python3.6/site-packages/dnf/cli/cli.py", line 1009, in run return self.command.run() File "/usr/lib/python3.6/site-packages/dnf-plugins/system_upgrade.py", line 313, in run self._call_sub("run") File "/usr/lib/python3.6/site-packages/dnf-plugins/system_upgrade.py", line 321, in _call_sub subfunc() File "/usr/lib/python3.6/site-packages/dnf-plugins/system_upgrade.py", line 464, in run_upgrade self.base.install(pkgspec, reponame=repo_id) File "/usr/lib/python3.6/site-packages/dnf/base.py", line 1658, in install _('no package matched'), pkg_spec) dnf.exceptions.MarkingError: no package matched Version-Release number of selected component (if applicable): python3-dnf-plugin-system-upgrade-2.0.2-1.fc26.noarch dnf-2.6.3-1.fc26.noarch Steps to Reproduce: 1. dnf system-upgrade download --releasever=27 2. dnf system-upgrade reboot
Created attachment 1324827 [details] whole dnf.log
Created attachment 1324828 [details] journalctl -b -1
Proposed as a Blocker for 27-beta by Fedora user lbrabec using the blocker tracking app because: This bug violates beta blocker criterion: Upgrade requirements - Recommended upgrade mechanisms - Upgrading with DNF system upgrade plugin
I tried to reproduce this and I succeeded once (same traceback), but didn't see it in other attempts. Maybe a race condition?
I was able to reproduce this on another bare metal and in virtual machine.
I think openQA may be hitting this intermittently as well, I'd have to check through the results carefully to be sure. Sometimes it looks like the test basically fails to upgrade and boots back to the original release, which could well be this bug.
Do we know how common the failure case is?
Can someone seeing this attach /var/log/dnf.rpm.log and /var/log/dnf.log?
I experienced this same exact bug, and found a solution. Tentative solution (please try it out): 1) Enable the updates-testing repo at the configuration level by running 'dnf config-manager --set-enable updates-testing' 2) Run 'dnf system-upgrade reboot' (or dnf system-upgrade download --release-ver=27 first, if you need to) I think this falls under the "Is it a bug, or a feature?" category. Background: Due to rhbz #1487867 (Wrong version on legacy variant (e.g. grub2-pc) Obsoletes:), system-upgrade requires the updates-testing repo to be enabled (until the grub2 changes get pushed to stable). This ~can~ be accomplished by adding '--enablerepo=updates-testing' to the command, which I used, and, per the attached dnf.log, so did Lukas: 2017-09-12T09:59:03Z DDEBUG Command: dnf system-upgrade download --releasever=27 --enablerepo=updates-testing However, updates-testing is not enabled on the 'dnf system-upgrade upgrade' command, and a quick look through the log shows that the list of repos does not include it. My initial attempt at a solution was to run: dnf system-upgrade reboot --enablerepo=updates-testing (plus a debug option I feel a little silly about) but this does not pass --enablerepo=updates-testing to the upgrade command. Here is the pertinent snippet of the log file. 2017-09-14T18:56:54Z DDEBUG Command: dnf system-upgrade reboot --enablerepo=updates-testing --debuglevel=10 2017-09-14T18:56:54Z DDEBUG Installroot: / 2017-09-14T18:56:54Z DDEBUG Releasever: 26 2017-09-14T18:56:54Z DEBUG cachedir: /var/cache/dnf 2017-09-14T18:56:54Z DDEBUG Base command: system-upgrade 2017-09-14T18:56:54Z DDEBUG Extra commands: ['system-upgrade', 'reboot', '--enablerepo=updates-testing', '--debuglevel=10'] 2017-09-14T18:56:54Z DDEBUG Cleaning up. 2017-09-14T18:57:30Z INFO --- logging initialized --- 2017-09-14T18:57:30Z DDEBUG timer: config: 12 ms 2017-09-14T18:57:30Z DEBUG Loaded plugins: builddep, config-manager, copr, debug, debuginfo-install, download, generate_completion_cache, needs-restar ting, playground, repoclosure, repograph, repomanage, reposync, system-upgrade 2017-09-14T18:57:30Z DEBUG DNF version: 2.6.3 2017-09-14T18:57:30Z DDEBUG Command: dnf --releasever=27 system-upgrade upgrade 2017-09-14T18:57:30Z DDEBUG Installroot: / 2017-09-14T18:57:30Z DDEBUG Releasever: 27 2017-09-14T18:57:30Z DEBUG cachedir: /var/cache/dnf 2017-09-14T18:57:31Z DDEBUG Base command: system-upgrade 2017-09-14T18:57:31Z DDEBUG Extra commands: ['--releasever=27', 'system-upgrade', 'upgrade'] Whether these parameters should or should not be passed is the bug/feature question. Also, sorry for rambling.
I don't think that has anything to do with this bug. If you upgrade without updates-testing enabled you might run into some problems with the upgraded system due to the bugs in grub2 and shim-signed, but they don't explain why the upgrade would simply not run *at all*.
Discussed at 2017-09-14 Beta Go/No-Go meeting, acting as a blocker review meeting: https://meetbot-raw.fedoraproject.org/fedora-meeting-2/2017-09-14/f27-beta-go-no-go-meeting.2017-09-14-17.00.html . We agreed to delay the decision on blocker status for this bug while we look into it more closely and figure some things out (like whether you can just re-run the 'dnf system-upgrade reboot' step to make it work, and how common this is).
The system-upgrade plugin actually comes from dnf-plugins-extras these days.
(In reply to Adam Williamson from comment #10) > I don't think that has anything to do with this bug. If you upgrade without > updates-testing enabled you might run into some problems with the upgraded > system due to the bugs in grub2 and shim-signed, but they don't explain why > the upgrade would simply not run *at all*. Pardon the confusion. This has nothing to do with grub2 and shim. In both documented cases (mine and Lukas's), the download step was performed with a repo enabled (the f27 updates-testing repo). This repo was disabled prior to running the update step. Packages were pulled from this repo for the update. From the debug info, we see that, during the update step, dnf accesses the repos (see the following example): 2017-09-12T10:20:41Z DEBUG repo: using cache for: fedora 2017-09-12T10:20:41Z DEBUG not found deltainfo for: Fedora 27 - x86_64 2017-09-12T10:20:41Z DEBUG not found updateinfo for: Fedora 27 - x86_64 2017-09-12T10:20:41Z DEBUG fedora: using metadata from Sun 10 Sep 2017 03:06:09 PM CEST. so we know that dnf wants these repos for something. The code throws the exception here: sltrs = subj._get_best_selectors(self.sack, forms=forms, obsoletes=self.conf.obsoletes, reponame=reponame, reports=True) if not any((s.matches() for s in sltrs)): raise dnf.exceptions.MarkingError( _('no package matched'), pkg_spec) With some additional digging, it looks like dnf: 1) builds itself an internal state which includes repos (I have nooooo desire to dig through the libhif(hawkey) & libsolv stuff) 2) Requires that the repos be available during the update state (see code snippet from system_upgrade.py, below): def configure_upgrade(self): # same as the download, but offline and non-interactive. so... self.cli.demands.root_user = True self.cli.demands.resolving = True self.cli.demands.available_repos = True self.cli.demands.sack_activation = True ... I think this could be exactly why it does not run at all.
(In reply to Kevin Fenzi from comment #8) > Can someone seeing this attach /var/log/dnf.rpm.log and /var/log/dnf.log? dnf.log is in comment 1. dnf.rpm.log is not there, I can attach it if needed, but if will be from a different upgrade attempt. (In reply to Dennis Gilmore from comment #7) > Do we know how common the failure case is? Lukas reproduced this on 2 different bare metal machines and 1 VM on the first try. I either hit this or bug 1491320. IOW, we haven't managed to perform a single successful system upgrade yet. (In reply to Adam Williamson from comment #11) > whether you can just re-run the 'dnf system-upgrade > reboot' step to make it work Nope.
Bug reproduced on bare metal.
Created attachment 1326482 [details] dnf.librepo.log
Created attachment 1326483 [details] dnf.log
Created attachment 1326484 [details] dnf.rpm.log
Created attachment 1326485 [details] journal
Created attachment 1326486 [details] List of installed packages
Created attachment 1326531 [details] dnf.log with successful f26->f27 upgrade at 2017-09-14T19:10:54Z
Created attachment 1326540 [details] dnf.rpm.log with successful f26->f27 upgrade at 2017-09-14T19:10:54Z
(In reply to Kamil Páral from comment #14) > Lukas reproduced this on 2 different bare metal machines and 1 VM on the > first try. I either hit this or bug 1491320. IOW, we haven't managed to > perform a single successful system upgrade yet. I have attached logs including both unsuccessful upgrade attempts and my successful upgrade.
*** Bug 1492601 has been marked as a duplicate of this bug. ***
I've been able to reproduce this on two different machines, several attempts on each. So nothing that occurs only from time to time. A clear beta blocker to me.
Discussed at blocker review meeting [1]: AcceptedBlocker (beta) - This bug violates the beta criterion: "For each one of the release-blocking package sets, it must be possible to successfully complete a direct upgrade from fully updated installations of the last two stable Fedora releases with that package set installed." [1] https://meetbot-raw.fedoraproject.org/fedora-blocker-review/2017-09-18
@kwkaess, how did you manage to perform a successful upgrade? You just tried repeatedly and it simply worked after some time, or was it something else?
(In reply to Kamil Páral from comment #27) > @kwkaess, how did you manage to perform a successful upgrade? You just tried > repeatedly and it simply worked after some time, or was it something else? As per my original comment, I ran as an intermediate step: 'dnf config-manager --set-enable updates-testing' This solved the problem. Previously, I had tried repeatedly and unsuccessfully.
Tomorrow I will try to investigate the issue. But do you run the system with in dnf.conf multilib_policy=all ?
I create a patch that should allow to investigate the issue (https://github.com/rpm-software-management/dnf-plugins-extras/pull/105). Please can anyone try to run the new version at condition that previously triggered the issue, and then report outputs? The patch in case of problem will fail but it will provide name of argument. Also an output in this command from system-upgrade --download would be helpful.
Also file /var/lib/dnf/system-upgrade.json from system that failed to upgrade could help.
I did an upgrade of vm fc26 to fc27 (boxes on my fc26) and no problem. But now I see why it happen. The option --enablerepo is incompatible with present implementation. I will create a patch...
I have a patch that should solve the issue if is used "--enablerepo" with "dnf system-upgrade download" (https://github.com/rpm-software-management/dnf-plugins-extras/pull/105). Please if anyone can confirm that it solves the issue, I would be very happy.
Can you please provide a test build?
The patch didn't work for me: Sep 21 11:15:28 f26 dnf[541]: Traceback (most recent call last): Sep 21 11:15:28 f26 dnf[541]: File "/usr/bin/dnf", line 58, in <module> Sep 21 11:15:28 f26 dnf[541]: main.user_main(sys.argv[1:], exit_code=True) Sep 21 11:15:28 f26 dnf[541]: File "/usr/lib/python3.6/site-packages/dnf/cli/main.py", line 179, in user_main Sep 21 11:15:28 f26 dnf[541]: errcode = main(args) Sep 21 11:15:28 f26 dnf[541]: File "/usr/lib/python3.6/site-packages/dnf/cli/main.py", line 64, in main Sep 21 11:15:28 f26 dnf[541]: return _main(base, args, cli_class, option_parser_class) Sep 21 11:15:28 f26 dnf[541]: File "/usr/lib/python3.6/site-packages/dnf/cli/main.py", line 95, in _main Sep 21 11:15:28 f26 dnf[541]: cli.configure(list(map(ucd, args)), option_parser()) Sep 21 11:15:28 f26 dnf[541]: File "/usr/lib/python3.6/site-packages/dnf/cli/cli.py", line 891, in configure Sep 21 11:15:28 f26 dnf[541]: self.command.configure() Sep 21 11:15:28 f26 dnf[541]: File "/usr/lib/python3.6/site-packages/dnf-plugins/system_upgrade.py", line 309, in co Sep 21 11:15:28 f26 dnf[541]: self._call_sub("configure") Sep 21 11:15:28 f26 dnf[541]: File "/usr/lib/python3.6/site-packages/dnf-plugins/system_upgrade.py", line 321, in _c Sep 21 11:15:28 f26 dnf[541]: subfunc() Sep 21 11:15:28 f26 dnf[541]: File "/usr/lib/python3.6/site-packages/dnf-plugins/system_upgrade.py", line 357, in co Sep 21 11:15:28 f26 dnf[541]: self.opts.repos_ed = self.state.enable_disable_repos Sep 21 11:15:28 f26 dnf[541]: AttributeError: 'State' object has no attribute 'enable_disable_repos' Sep 21 11:15:28 f26 systemd[1]: dnf-system-upgrade.service: Main process exited, code=exited, status=1/FAILURE Sep 21 11:15:28 f26 rm[552]: removed '/system-update' Sep 21 11:15:28 f26 systemd[1]: Failed to start System Upgrade using DNF. Sep 21 11:15:28 f26 systemd[1]: dnf-system-upgrade.service: Unit entered failed state. Sep 21 11:15:28 f26 systemd[1]: dnf-system-upgrade.service: Failed with result 'exit-code'. Sep 21 11:15:28 f26 systemd[1]: Rebooting: service failed However, now I understand better when it fails and when it does not. This bug only occurs when "--enablerepo" option is used with "dnf system-upgrade download". Otherwise system-upgrade works fine. I tested the following scenarios: 1. "dnf system-upgrade download --releasever=27 --allowerasing" with fedora+updates repo enabled. "--allowerasing" is needed until fix for bug 1493090 is fully pushed (or for any other similar package users might have on their system). That works without issues. 2. "dnf config-manager --set-enabled updates-testing && dnf system-upgrade download --releasever=27". That also works without problems. So the only problem here is really temporarily enabling a repo using --enablerepo during download. I think that can be documented and easily worked around (scenario 2), so this bug doesn't necessarily need to be a Beta blocker (rather a Final one instead). Moving to a proposed blocker once again.
Created attachment 1328880 [details] system-upgrade.json
Thanks a lot Kamil for additional information. I update the patch, that should now work. But it requires a patched dnf also. I created a copr repo for fc25 and fc26 (dnf copr enable jmracek/System-upgrade) that provides patches plus missing dependencies. dnf-plugins-extras-2.0.2-1.git.326.5e859d1.fc26, dnf-2.7.1-1.git.60.e8f15bf.fc26
I tested it and it works for me.
Discussed during blocker review [1]: RejectedBlocker (beta) - there is an easy workaround - enable repo before running update. No data are lost. We will repropose as Final blocker [1] https://meetbot-raw.fedoraproject.org/fedora-meeting-1/2017-09-21/
dnf-plugins-extras-2.0.3-1.fc27 dnf-plugins-core-2.1.4-1.fc27 dnf-2.7.2-1.fc27 libdnf-0.10.1-1.fc27 has been submitted as an update to Fedora 27. https://bodhi.fedoraproject.org/updates/FEDORA-2017-faf235c683
dnf-plugins-extras-2.0.3-1.fc26 dnf-plugins-core-2.1.4-1.fc26 dnf-2.7.2-1.fc26 libdnf-0.10.1-1.fc26 has been submitted as an update to Fedora 26. https://bodhi.fedoraproject.org/updates/FEDORA-2017-70a8618065
Discussed at blocker review meeting [1]: AcceptedBlocker (Final) - This violates "For each one of the release-blocking package sets, it must be possible to successfully complete a direct upgrade from fully updated installations of the last two stable Fedora releases with that package set installed." when --enablerepo is used. [1] https://meetbot-raw.fedoraproject.org/fedora-blocker-review/2017-10-02/
dnf-2.7.2-1.fc26, dnf-plugins-core-2.1.4-1.fc26, dnf-plugins-extras-2.0.3-1.fc26, libdnf-0.10.1-1.fc26 has been pushed to the Fedora 26 testing repository. If problems still persist, please make note of it in this bug report. See https://fedoraproject.org/wiki/QA:Updates_Testing for instructions on how to install test updates. You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2017-70a8618065
dnf-2.7.2-1.fc27, dnf-plugins-core-2.1.4-1.fc27, dnf-plugins-extras-2.0.3-1.fc27, libdnf-0.10.1-1.fc27 has been pushed to the Fedora 27 testing repository. If problems still persist, please make note of it in this bug report. See https://fedoraproject.org/wiki/QA:Updates_Testing for instructions on how to install test updates. You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2017-faf235c683
This issue still persists in dnf-2.7.2-1.fc26, dnf-plugins-core-2.1.4-1.fc26, dnf-plugins-extras-2.0.3-1.fc26, libdnf-0.10.1-1.fc26.
František, is the traceback the same? If not, please attach journal from upgrade and /var/log/dnf.log (cut it to just contain the upgrade download and run).
The issue is fixed after all, just forgot to update the system-upgrade plugin. Sorry for the noise.
dnf-2.7.2-1.fc27, dnf-plugins-core-2.1.4-1.fc27, dnf-plugins-extras-2.0.3-1.fc27, libdnf-0.10.1-1.fc27 has been pushed to the Fedora 27 stable repository. If problems still persist, please make note of it in this bug report.
dnf-2.7.2-1.fc26, dnf-plugins-core-2.1.4-1.fc26, dnf-plugins-extras-2.0.3-1.fc26, libdnf-0.10.1-1.fc26 has been pushed to the Fedora 26 stable repository. If problems still persist, please make note of it in this bug report.
Created attachment 1335074 [details] dnf.log I'm wondering why it didn't work on my computer My command 604 sudo dnf config-manager --set-enabled updates-testing 605 cd 606 sudo dnf system-upgrade download --releasever=27 607 sudo dnf system-upgrade reboot $ tail /var/log/dnf.log 2017-10-06T04:42:31Z DEBUG DNF version: 2.7.2 2017-10-06T04:42:31Z DDEBUG Command: dnf makecache timer 2017-10-06T04:42:31Z DDEBUG Installroot: / 2017-10-06T04:42:31Z DDEBUG Releasever: 26 2017-10-06T04:42:31Z DEBUG cachedir: /var/cache/dnf 2017-10-06T04:42:31Z DDEBUG Base command: makecache 2017-10-06T04:42:31Z DDEBUG Extra commands: ['makecache', 'timer'] 2017-10-06T04:42:31Z DEBUG Making cache files for all metadata files. 2017-10-06T04:42:31Z INFO Metadata cache refreshed recently. 2017-10-06T04:42:31Z DDEBUG Cleaning up. $ dnf --version 2.7.2 Installed: dnf-0:2.7.2-1.fc26.noarch at Fri 06 Oct 2017 04:03:35 AM GMT Built : Fedora Project at Mon 02 Oct 2017 09:31:46 AM GMT
I guest that problem in Comment 50 is another issue. Probably this is same problem like https://bugzilla.redhat.com/show_bug.cgi?id=1498207 . From log I can say only, that the upgrade died suddenly. Journalctrl should provide additional information.
(In reply to Jaroslav Mracek from comment #51) > I guest that problem in Comment 50 is another issue. Probably this is same > problem like https://bugzilla.redhat.com/show_bug.cgi?id=1498207 . > From log I can say only, that the upgrade died suddenly. Journalctrl should > provide additional information. Thank you. My report belongs to another issue: Bug 1498875