Bug 1690915 - dnf: Can't handle being passed 35+ file names as input for downgrade operation
Summary: dnf: Can't handle being passed 35+ file names as input for downgrade operation
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Fedora
Classification: Fedora
Component: dnf
Version: 30
Hardware: x86_64
OS: Linux
unspecified
unspecified
Target Milestone: ---
Assignee: Jaroslav Mracek
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard:
Depends On:
Blocks: 1725213
TreeView+ depends on / blocked
 
Reported: 2019-03-20 13:10 UTC by Andrea Bolognani
Modified: 2019-09-06 12:31 UTC (History)
9 users (show)

Fixed In Version: dnf-4.2.8-1.fc30
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
: 1725213 (view as bug list)
Environment:
Last Closed: 2019-09-06 12:31:05 UTC


Attachments (Terms of Use)
dnf.log (220.72 KB, application/x-xz)
2019-03-22 11:12 UTC, Andrea Bolognani
no flags Details
valgrind output (18.03 KB, text/plain)
2019-06-28 10:39 UTC, Andrea Bolognani
no flags Details
gdb output (9.08 KB, text/plain)
2019-06-28 10:39 UTC, Andrea Bolognani
no flags Details
rpm -qa output (83.21 KB, text/plain)
2019-06-28 10:40 UTC, Andrea Bolognani
no flags Details
yum repos (17.81 KB, text/plain)
2019-06-28 10:40 UTC, Andrea Bolognani
no flags Details

Description Andrea Bolognani 2019-03-20 13:10:57 UTC
I have a subset of libvirt 5.2.0 installed on my machine

  $ rpm -qa | grep -E 'libvirt-.*5.2.0' | sort
  libvirt-5.2.0-1.fc29.x86_64
  libvirt-bash-completion-5.2.0-1.fc29.x86_64
  libvirt-client-5.2.0-1.fc29.x86_64
  libvirt-daemon-5.2.0-1.fc29.x86_64
  libvirt-daemon-config-network-5.2.0-1.fc29.x86_64
  libvirt-daemon-config-nwfilter-5.2.0-1.fc29.x86_64
  libvirt-daemon-driver-interface-5.2.0-1.fc29.x86_64
  libvirt-daemon-driver-libxl-5.2.0-1.fc29.x86_64
  libvirt-daemon-driver-lxc-5.2.0-1.fc29.x86_64
  libvirt-daemon-driver-network-5.2.0-1.fc29.x86_64
  libvirt-daemon-driver-nodedev-5.2.0-1.fc29.x86_64
  libvirt-daemon-driver-nwfilter-5.2.0-1.fc29.x86_64
  libvirt-daemon-driver-qemu-5.2.0-1.fc29.x86_64
  libvirt-daemon-driver-secret-5.2.0-1.fc29.x86_64
  libvirt-daemon-driver-storage-5.2.0-1.fc29.x86_64
  libvirt-daemon-driver-storage-core-5.2.0-1.fc29.x86_64
  libvirt-daemon-driver-storage-disk-5.2.0-1.fc29.x86_64
  libvirt-daemon-driver-storage-gluster-5.2.0-1.fc29.x86_64
  libvirt-daemon-driver-storage-iscsi-5.2.0-1.fc29.x86_64
  libvirt-daemon-driver-storage-iscsi-direct-5.2.0-1.fc29.x86_64
  libvirt-daemon-driver-storage-logical-5.2.0-1.fc29.x86_64
  libvirt-daemon-driver-storage-mpath-5.2.0-1.fc29.x86_64
  libvirt-daemon-driver-storage-rbd-5.2.0-1.fc29.x86_64
  libvirt-daemon-driver-storage-scsi-5.2.0-1.fc29.x86_64
  libvirt-daemon-driver-storage-sheepdog-5.2.0-1.fc29.x86_64
  libvirt-daemon-driver-storage-zfs-5.2.0-1.fc29.x86_64
  libvirt-daemon-driver-vbox-5.2.0-1.fc29.x86_64
  libvirt-daemon-kvm-5.2.0-1.fc29.x86_64
  libvirt-libs-5.2.0-1.fc29.x86_64
  libvirt-nss-5.2.0-1.fc29.x86_64
  $

and a full set of libvirt 5.1.0 RPMs in a directory

  $ ls | grep -E 'libvirt-.*5.1.0' | sort
  libvirt-5.1.0-1.fc29.x86_64.rpm
  libvirt-admin-5.1.0-1.fc29.x86_64.rpm
  libvirt-bash-completion-5.1.0-1.fc29.x86_64.rpm
  libvirt-client-5.1.0-1.fc29.x86_64.rpm
  libvirt-daemon-5.1.0-1.fc29.x86_64.rpm
  libvirt-daemon-config-network-5.1.0-1.fc29.x86_64.rpm
  libvirt-daemon-config-nwfilter-5.1.0-1.fc29.x86_64.rpm
  libvirt-daemon-driver-interface-5.1.0-1.fc29.x86_64.rpm
  libvirt-daemon-driver-libxl-5.1.0-1.fc29.x86_64.rpm
  libvirt-daemon-driver-lxc-5.1.0-1.fc29.x86_64.rpm
  libvirt-daemon-driver-network-5.1.0-1.fc29.x86_64.rpm
  libvirt-daemon-driver-nodedev-5.1.0-1.fc29.x86_64.rpm
  libvirt-daemon-driver-nwfilter-5.1.0-1.fc29.x86_64.rpm
  libvirt-daemon-driver-qemu-5.1.0-1.fc29.x86_64.rpm
  libvirt-daemon-driver-secret-5.1.0-1.fc29.x86_64.rpm
  libvirt-daemon-driver-storage-5.1.0-1.fc29.x86_64.rpm
  libvirt-daemon-driver-storage-core-5.1.0-1.fc29.x86_64.rpm
  libvirt-daemon-driver-storage-disk-5.1.0-1.fc29.x86_64.rpm
  libvirt-daemon-driver-storage-gluster-5.1.0-1.fc29.x86_64.rpm
  libvirt-daemon-driver-storage-iscsi-5.1.0-1.fc29.x86_64.rpm
  libvirt-daemon-driver-storage-iscsi-direct-5.1.0-1.fc29.x86_64.rpm
  libvirt-daemon-driver-storage-logical-5.1.0-1.fc29.x86_64.rpm
  libvirt-daemon-driver-storage-mpath-5.1.0-1.fc29.x86_64.rpm
  libvirt-daemon-driver-storage-rbd-5.1.0-1.fc29.x86_64.rpm
  libvirt-daemon-driver-storage-scsi-5.1.0-1.fc29.x86_64.rpm
  libvirt-daemon-driver-storage-sheepdog-5.1.0-1.fc29.x86_64.rpm
  libvirt-daemon-driver-storage-zfs-5.1.0-1.fc29.x86_64.rpm
  libvirt-daemon-driver-vbox-5.1.0-1.fc29.x86_64.rpm
  libvirt-daemon-kvm-5.1.0-1.fc29.x86_64.rpm
  libvirt-daemon-lxc-5.1.0-1.fc29.x86_64.rpm
  libvirt-daemon-qemu-5.1.0-1.fc29.x86_64.rpm
  libvirt-daemon-vbox-5.1.0-1.fc29.x86_64.rpm
  libvirt-daemon-xen-5.1.0-1.fc29.x86_64.rpm
  libvirt-debugsource-5.1.0-1.fc29.x86_64.rpm
  libvirt-devel-5.1.0-1.fc29.x86_64.rpm
  libvirt-docs-5.1.0-1.fc29.x86_64.rpm
  libvirt-libs-5.1.0-1.fc29.x86_64.rpm
  libvirt-lock-sanlock-5.1.0-1.fc29.x86_64.rpm
  libvirt-login-shell-5.1.0-1.fc29.x86_64.rpm
  libvirt-nss-5.1.0-1.fc29.x86_64.rpm
  libvirt-wireshark-5.1.0-1.fc29.x86_64.rpm
  $

Because of $REASONS, I need to downgrade from 5.2.0 to 5.1.0, so
from the directory mentioned above I ran

  $ sudo dnf downgrade ./libvirt-*5.1.0*.rpm
  Last metadata expiration check: 0:00:42 ago on Wed 20 Mar 2019 01:34:27 PM CET.
  malloc(): invalid next size (unsorted)
  Aborted
  $

Not exactly what I expected :) Note that a more surgical approach to
the problem succeeds:

  $ sudo dnf downgrade $(rpm -qa | grep -E 'libvirt-.*5.2.0.*' | sed 's,^,./,; s,5.2.0,5.1.0,; s,$,.rpm,')
  Last metadata expiration check: 0:05:01 ago on Wed 20 Mar 2019 01:34:27 PM CET.
  Dependencies resolved.
  ========================================================================================
   Package                                     Arch    Version        Repository     Size
  ========================================================================================
  Downgrading:
   libvirt-client                              x86_64  5.1.0-1.fc29   @commandline  306 k
   libvirt-daemon-driver-nodedev               x86_64  5.1.0-1.fc29   @commandline   81 k
   libvirt-libs                                x86_64  5.1.0-1.fc29   @commandline  4.2 M
   libvirt-daemon-driver-storage-gluster       x86_64  5.1.0-1.fc29   @commandline   81 k
   libvirt-daemon-driver-interface             x86_64  5.1.0-1.fc29   @commandline   81 k
   libvirt-daemon-driver-storage-scsi          x86_64  5.1.0-1.fc29   @commandline   70 k
   libvirt-daemon-driver-storage-disk          x86_64  5.1.0-1.fc29   @commandline   72 k
   libvirt-daemon-kvm                          x86_64  5.1.0-1.fc29   @commandline  6.4 k
   libvirt-daemon-config-nwfilter              x86_64  5.1.0-1.fc29   @commandline   13 k
   libvirt-bash-completion                     x86_64  5.1.0-1.fc29   @commandline  7.5 k
   libvirt-daemon-driver-storage-core          x86_64  5.1.0-1.fc29   @commandline  299 k
   libvirt-daemon-driver-storage-rbd           x86_64  5.1.0-1.fc29   @commandline   77 k
   libvirt-daemon-driver-storage-iscsi-direct  x86_64  5.1.0-1.fc29   @commandline   72 k
   libvirt-daemon-driver-secret                x86_64  5.1.0-1.fc29   @commandline   70 k
   libvirt-daemon-driver-vbox                  x86_64  5.1.0-1.fc29   @commandline  163 k
   libvirt-daemon-driver-qemu                  x86_64  5.1.0-1.fc29   @commandline  634 k
   libvirt-daemon-driver-storage-zfs           x86_64  5.1.0-1.fc29   @commandline   70 k
   libvirt-daemon-driver-storage-iscsi         x86_64  5.1.0-1.fc29   @commandline   70 k
   libvirt-daemon-driver-lxc                   x86_64  5.1.0-1.fc29   @commandline  179 k
   libvirt                                     x86_64  5.1.0-1.fc29   @commandline  7.4 k
   libvirt-daemon-driver-nwfilter              x86_64  5.1.0-1.fc29   @commandline  104 k
   libvirt-daemon-driver-storage-sheepdog      x86_64  5.1.0-1.fc29   @commandline   69 k
   libvirt-daemon                              x86_64  5.1.0-1.fc29   @commandline  721 k
   libvirt-daemon-driver-storage-mpath         x86_64  5.1.0-1.fc29   @commandline   68 k
   libvirt-daemon-config-network               x86_64  5.1.0-1.fc29   @commandline  8.5 k
   libvirt-daemon-driver-libxl                 x86_64  5.1.0-1.fc29   @commandline  184 k
   libvirt-daemon-driver-storage-logical       x86_64  5.1.0-1.fc29   @commandline   74 k
   libvirt-nss                                 x86_64  5.1.0-1.fc29   @commandline  186 k
   libvirt-daemon-driver-storage               x86_64  5.1.0-1.fc29   @commandline  6.7 k
   libvirt-daemon-driver-network               x86_64  5.1.0-1.fc29   @commandline  268 k

  Transaction Summary
  ========================================================================================
  Downgrade  30 Packages

  Total size: 8.1 M
  Is this ok [y/N]: n
  Operation aborted.
  $

Whether or not the issue manifests itself seems to be connected to
the number of packages provided on the command line rather than
anything more specific: using a simple test script, I chose a set
number of random packages from those in the directory and tried
using them as input for a downgrade repeatedly, like

  $ for i in $(seq 0 99)
  > do
  >   pkgs=$(ls ./libvirt-*5.1.0* | shuf | head -40)
  >   sudo dnf downgrade $pkgs 2>&1 | tail -2
  >   echo =====
  > done
  Last metadata expiration check: 0:14:04 ago on Wed 20 Mar 2019 01:34:27 PM CET.
  corrupted size vs. prev_size
  =====
  Last metadata expiration check: 0:14:07 ago on Wed 20 Mar 2019 01:34:27 PM CET.
  malloc(): invalid next size (unsorted)
  =====
  Last metadata expiration check: 0:14:09 ago on Wed 20 Mar 2019 01:34:27 PM CET.
  corrupted size vs. prev_size
  =====
    - conflicting requests
  (try to add '--allowerasing' to command line to replace conflicting packages or '--skip-broken' to skip uninstallable packages)
  =====
  Last metadata expiration check: 0:14:13 ago on Wed 20 Mar 2019 01:34:27 PM CET.
  corrupted size vs. prev_size
  =====
  ^C
  $

The cutoff point seems to be around 35 packages: for 36 packages, out
of 100 iterations, I recorded the following split

  90 corrupted size vs. prev_size
   7 Failed transaction due to dependency issues (expected)
   4 malloc(): invalid next size (unsorted)
   3 munmap_chunk(): invalid pointer

whereas with 35 packages the split looks like

  91 Failed transaction due to dependency issues (expected)
   9 munmap_chunk(): invalid pointer

and with 30 packages I was unable to trigger a single failure over
the same 100 iterations.

I'm running a fully up to date Fedora 29 installation:

  $ rpm -q dnf
  dnf-4.1.0-1.fc29.noarch
  $

Comment 1 Jaroslav Mracek 2019-03-22 09:55:28 UTC
Please could you provide "/var/log/dnf.log" with entries of failed command?

Comment 2 Jaroslav Mracek 2019-03-22 10:13:58 UTC
Please could you try to reproduce the issue with dnf-0:4.2.1-1.fc29.noarch, libdnf-0:0.28.0-1.fc29.x86_64, and libsolv-0:0.7.2-2.fc29.x86_64?

Comment 3 Andrea Bolognani 2019-03-22 11:12:23 UTC
Created attachment 1546822 [details]
dnf.log

(In reply to Jaroslav Mracek from comment #1)
> Please could you provide "/var/log/dnf.log" with entries of failed command?

The full log is attached. It's fairly big, as it contains logs for
all the attempts made using the scripts defined above, plus some
more manual testing...

The following snippet should be representative:

  2019-03-20T12:48:35Z INFO --- logging initialized ---
  2019-03-20T12:48:35Z DDEBUG timer: config: 2 ms
  2019-03-20T12:48:35Z DEBUG Loaded plugins: builddep, changelog, config-manager, copr, debug, debuginfo-install, download, generate_completion_cache, needs-restarting, playground, repoclosure, repodiff, repograph, repomanage, reposync, system-upgrade
  2019-03-20T12:48:35Z DEBUG DNF version: 4.1.0
  2019-03-20T12:48:35Z DDEBUG Command: dnf downgrade ./libvirt-daemon-driver-storage-iscsi-5.1.0-1.fc29.x86_64.rpm ./libvirt-nss-5.1.0-1.fc29.x86_64.rpm ./libvirt-daemon-driver-storage-scsi-5.1.0-1.fc29.x86_64.rpm ./libvirt-admin-5.1.0-1.fc29.x86_64.rpm ./libvirt-lock-sanlock-5.1.0-1.fc29.x86_64.rpm ./libvirt-daemon-driver-storage-sheepdog-5.1.0-1.fc29.x86_64.rpm ./libvirt-daemon-driver-interface-5.1.0-1.fc29.x86_64.rpm ./libvirt-libs-5.1.0-1.fc29.x86_64.rpm ./libvirt-daemon-driver-storage-gluster-5.1.0-1.fc29.x86_64.rpm ./libvirt-client-5.1.0-1.fc29.x86_64.rpm ./libvirt-5.1.0-1.fc29.x86_64.rpm ./libvirt-daemon-driver-vbox-5.1.0-1.fc29.x86_64.rpm ./libvirt-daemon-driver-qemu-5.1.0-1.fc29.x86_64.rpm ./libvirt-daemon-kvm-5.1.0-1.fc29.x86_64.rpm ./libvirt-daemon-driver-storage-rbd-5.1.0-1.fc29.x86_64.rpm ./libvirt-daemon-lxc-5.1.0-1.fc29.x86_64.rpm ./libvirt-bash-completion-5.1.0-1.fc29.x86_64.rpm ./libvirt-daemon-5.1.0-1.fc29.x86_64.rpm ./libvirt-daemon-driver-nwfilter-5.1.0-1.fc29.x86_64.rpm ./libvirt-daemon-driver-storage-mpath-5.1.0-1.fc29.x86_64.rpm ./libvirt-daemon-vbox-5.1.0-1.fc29.x86_64.rpm ./libvirt-debugsource-5.1.0-1.fc29.x86_64.rpm ./libvirt-daemon-qemu-5.1.0-1.fc29.x86_64.rpm ./libvirt-daemon-driver-storage-zfs-5.1.0-1.fc29.x86_64.rpm ./libvirt-docs-5.1.0-1.fc29.x86_64.rpm ./libvirt-daemon-config-nwfilter-5.1.0-1.fc29.x86_64.rpm ./libvirt-daemon-driver-lxc-5.1.0-1.fc29.x86_64.rpm ./libvirt-daemon-driver-storage-5.1.0-1.fc29.x86_64.rpm ./libvirt-daemon-driver-storage-iscsi-direct-5.1.0-1.fc29.x86_64.rpm ./libvirt-daemon-driver-secret-5.1.0-1.fc29.x86_64.rpm ./libvirt-daemon-driver-nodedev-5.1.0-1.fc29.x86_64.rpm ./libvirt-daemon-config-network-5.1.0-1.fc29.x86_64.rpm ./libvirt-wireshark-5.1.0-1.fc29.x86_64.rpm ./libvirt-daemon-driver-storage-core-5.1.0-1.fc29.x86_64.rpm ./libvirt-daemon-driver-storage-disk-5.1.0-1.fc29.x86_64.rpm ./libvirt-daemon-driver-libxl-5.1.0-1.fc29.x86_64.rpm ./libvirt-devel-5.1.0-1.fc29.x86_64.rpm ./libvirt-login-shell-5.1.0-1.fc29.x86_64.rpm ./libvirt-daemon-driver-storage-logical-5.1.0-1.fc29.x86_64.rpm ./libvirt-daemon-xen-5.1.0-1.fc29.x86_64.rpm 
  2019-03-20T12:48:35Z DDEBUG Installroot: /
  2019-03-20T12:48:35Z DDEBUG Releasever: 29
  2019-03-20T12:48:35Z DEBUG cachedir: /var/cache/dnf
  2019-03-20T12:48:35Z DDEBUG Base command: downgrade
  2019-03-20T12:48:35Z DDEBUG Extra commands: ['downgrade', './libvirt-daemon-driver-storage-iscsi-5.1.0-1.fc29.x86_64.rpm', './libvirt-nss-5.1.0-1.fc29.x86_64.rpm', './libvirt-daemon-driver-storage-scsi-5.1.0-1.fc29.x86_64.rpm', './libvirt-admin-5.1.0-1.fc29.x86_64.rpm', './libvirt-lock-sanlock-5.1.0-1.fc29.x86_64.rpm', './libvirt-daemon-driver-storage-sheepdog-5.1.0-1.fc29.x86_64.rpm', './libvirt-daemon-driver-interface-5.1.0-1.fc29.x86_64.rpm', './libvirt-libs-5.1.0-1.fc29.x86_64.rpm', './libvirt-daemon-driver-storage-gluster-5.1.0-1.fc29.x86_64.rpm', './libvirt-client-5.1.0-1.fc29.x86_64.rpm', './libvirt-5.1.0-1.fc29.x86_64.rpm', './libvirt-daemon-driver-vbox-5.1.0-1.fc29.x86_64.rpm', './libvirt-daemon-driver-qemu-5.1.0-1.fc29.x86_64.rpm', './libvirt-daemon-kvm-5.1.0-1.fc29.x86_64.rpm', './libvirt-daemon-driver-storage-rbd-5.1.0-1.fc29.x86_64.rpm', './libvirt-daemon-lxc-5.1.0-1.fc29.x86_64.rpm', './libvirt-bash-completion-5.1.0-1.fc29.x86_64.rpm', './libvirt-daemon-5.1.0-1.fc29.x86_64.rpm', './libvirt-daemon-driver-nwfilter-5.1.0-1.fc29.x86_64.rpm', './libvirt-daemon-driver-storage-mpath-5.1.0-1.fc29.x86_64.rpm', './libvirt-daemon-vbox-5.1.0-1.fc29.x86_64.rpm', './libvirt-debugsource-5.1.0-1.fc29.x86_64.rpm', './libvirt-daemon-qemu-5.1.0-1.fc29.x86_64.rpm', './libvirt-daemon-driver-storage-zfs-5.1.0-1.fc29.x86_64.rpm', './libvirt-docs-5.1.0-1.fc29.x86_64.rpm', './libvirt-daemon-config-nwfilter-5.1.0-1.fc29.x86_64.rpm', './libvirt-daemon-driver-lxc-5.1.0-1.fc29.x86_64.rpm', './libvirt-daemon-driver-storage-5.1.0-1.fc29.x86_64.rpm', './libvirt-daemon-driver-storage-iscsi-direct-5.1.0-1.fc29.x86_64.rpm', './libvirt-daemon-driver-secret-5.1.0-1.fc29.x86_64.rpm', './libvirt-daemon-driver-nodedev-5.1.0-1.fc29.x86_64.rpm', './libvirt-daemon-config-network-5.1.0-1.fc29.x86_64.rpm', './libvirt-wireshark-5.1.0-1.fc29.x86_64.rpm', './libvirt-daemon-driver-storage-core-5.1.0-1.fc29.x86_64.rpm', './libvirt-daemon-driver-storage-disk-5.1.0-1.fc29.x86_64.rpm', './libvirt-daemon-driver-libxl-5.1.0-1.fc29.x86_64.rpm', './libvirt-devel-5.1.0-1.fc29.x86_64.rpm', './libvirt-login-shell-5.1.0-1.fc29.x86_64.rpm', './libvirt-daemon-driver-storage-logical-5.1.0-1.fc29.x86_64.rpm', './libvirt-daemon-xen-5.1.0-1.fc29.x86_64.rpm']
  2019-03-20T12:48:35Z DEBUG Unknown configuration value: failovermethod=priority in /etc/yum.repos.d/fedora-cisco-openh264.repo; Configuration: OptionBinding with id "failovermethod" does not exist
  2019-03-20T12:48:35Z DEBUG Unknown configuration value: failovermethod=priority in /etc/yum.repos.d/fedora-modular.repo; Configuration: OptionBinding with id "failovermethod" does not exist
  2019-03-20T12:48:35Z DEBUG Unknown configuration value: failovermethod=priority in /etc/yum.repos.d/fedora-modular.repo; Configuration: OptionBinding with id "failovermethod" does not exist
  2019-03-20T12:48:35Z DEBUG Unknown configuration value: failovermethod=priority in /etc/yum.repos.d/fedora-modular.repo; Configuration: OptionBinding with id "failovermethod" does not exist
  2019-03-20T12:48:35Z DEBUG Unknown configuration value: failovermethod=priority in /etc/yum.repos.d/fedora-updates-modular.repo; Configuration: OptionBinding with id "failovermethod" does not exist
  2019-03-20T12:48:35Z DEBUG Unknown configuration value: failovermethod=priority in /etc/yum.repos.d/fedora-updates-modular.repo; Configuration: OptionBinding with id "failovermethod" does not exist
  2019-03-20T12:48:35Z DEBUG Unknown configuration value: failovermethod=priority in /etc/yum.repos.d/fedora-updates-modular.repo; Configuration: OptionBinding with id "failovermethod" does not exist
  2019-03-20T12:48:35Z DEBUG Unknown configuration value: failovermethod=priority in /etc/yum.repos.d/fedora-updates-testing-modular.repo; Configuration: OptionBinding with id "failovermethod" does not exist
  2019-03-20T12:48:35Z DEBUG Unknown configuration value: failovermethod=priority in /etc/yum.repos.d/fedora-updates-testing-modular.repo; Configuration: OptionBinding with id "failovermethod" does not exist
  2019-03-20T12:48:35Z DEBUG Unknown configuration value: failovermethod=priority in /etc/yum.repos.d/fedora-updates-testing-modular.repo; Configuration: OptionBinding with id "failovermethod" does not exist
  2019-03-20T12:48:35Z DEBUG Unknown configuration value: failovermethod=priority in /etc/yum.repos.d/fedora-updates-testing.repo; Configuration: OptionBinding with id "failovermethod" does not exist
  2019-03-20T12:48:35Z DEBUG Unknown configuration value: failovermethod=priority in /etc/yum.repos.d/fedora-updates-testing.repo; Configuration: OptionBinding with id "failovermethod" does not exist
  2019-03-20T12:48:35Z DEBUG Unknown configuration value: failovermethod=priority in /etc/yum.repos.d/fedora-updates-testing.repo; Configuration: OptionBinding with id "failovermethod" does not exist
  2019-03-20T12:48:35Z DEBUG Unknown configuration value: failovermethod=priority in /etc/yum.repos.d/fedora-updates.repo; Configuration: OptionBinding with id "failovermethod" does not exist
  2019-03-20T12:48:35Z DEBUG Unknown configuration value: failovermethod=priority in /etc/yum.repos.d/fedora-updates.repo; Configuration: OptionBinding with id "failovermethod" does not exist
  2019-03-20T12:48:35Z DEBUG Unknown configuration value: failovermethod=priority in /etc/yum.repos.d/fedora-updates.repo; Configuration: OptionBinding with id "failovermethod" does not exist
  2019-03-20T12:48:35Z DEBUG Unknown configuration value: failovermethod=priority in /etc/yum.repos.d/fedora.repo; Configuration: OptionBinding with id "failovermethod" does not exist
  2019-03-20T12:48:35Z DEBUG Unknown configuration value: failovermethod=priority in /etc/yum.repos.d/fedora.repo; Configuration: OptionBinding with id "failovermethod" does not exist
  2019-03-20T12:48:35Z DEBUG Unknown configuration value: failovermethod=priority in /etc/yum.repos.d/fedora.repo; Configuration: OptionBinding with id "failovermethod" does not exist
  2019-03-20T12:48:35Z DEBUG repo: using cache for: group_virtmaint-sig-virt-preview
  2019-03-20T12:48:35Z DEBUG not found other for: Copr repo for virt-preview owned by @virtmaint-sig
  2019-03-20T12:48:35Z DEBUG not found modules for: Copr repo for virt-preview owned by @virtmaint-sig
  2019-03-20T12:48:35Z DEBUG not found deltainfo for: Copr repo for virt-preview owned by @virtmaint-sig
  2019-03-20T12:48:35Z DEBUG not found updateinfo for: Copr repo for virt-preview owned by @virtmaint-sig
  2019-03-20T12:48:35Z DEBUG group_virtmaint-sig-virt-preview: using metadata from Wed 20 Mar 2019 06:14:54 AM CET.
  2019-03-20T12:48:35Z DEBUG repo: using cache for: fedora-modular
  2019-03-20T12:48:35Z DEBUG not found other for: Fedora Modular 29 - x86_64
  2019-03-20T12:48:35Z DEBUG not found deltainfo for: Fedora Modular 29 - x86_64
  2019-03-20T12:48:35Z DEBUG not found updateinfo for: Fedora Modular 29 - x86_64
  2019-03-20T12:48:35Z DEBUG fedora-modular: using metadata from Thu 25 Oct 2018 12:22:05 AM CEST.
  2019-03-20T12:48:35Z DEBUG repo: using cache for: updates-modular
  2019-03-20T12:48:35Z DEBUG not found other for: Fedora Modular 29 - x86_64 - Updates
  2019-03-20T12:48:35Z DEBUG not found deltainfo for: Fedora Modular 29 - x86_64 - Updates
  2019-03-20T12:48:35Z DEBUG updates-modular: using metadata from Fri 15 Mar 2019 06:39:11 AM CET.
  2019-03-20T12:48:35Z DEBUG repo: using cache for: updates
  2019-03-20T12:48:35Z DEBUG not found other for: Fedora 29 - x86_64 - Updates
  2019-03-20T12:48:35Z DEBUG not found modules for: Fedora 29 - x86_64 - Updates
  2019-03-20T12:48:35Z DEBUG updates: using metadata from Tue 19 Mar 2019 05:43:55 AM CET.
  2019-03-20T12:48:35Z DEBUG repo: using cache for: fedora
  2019-03-20T12:48:35Z DEBUG not found other for: Fedora 29 - x86_64
  2019-03-20T12:48:35Z DEBUG not found modules for: Fedora 29 - x86_64
  2019-03-20T12:48:35Z DEBUG not found deltainfo for: Fedora 29 - x86_64
  2019-03-20T12:48:35Z DEBUG not found updateinfo for: Fedora 29 - x86_64
  2019-03-20T12:48:36Z DEBUG fedora: using metadata from Thu 25 Oct 2018 12:20:15 AM CEST.
  2019-03-20T12:48:36Z DEBUG repo: using cache for: rcm-tools-fedora-rpms
  2019-03-20T12:48:36Z DEBUG not found other for: RCM Tools for Fedora 29 (RPMs)
  2019-03-20T12:48:36Z DEBUG not found modules for: RCM Tools for Fedora 29 (RPMs)
  2019-03-20T12:48:36Z DEBUG not found deltainfo for: RCM Tools for Fedora 29 (RPMs)
  2019-03-20T12:48:36Z DEBUG not found updateinfo for: RCM Tools for Fedora 29 (RPMs)
  2019-03-20T12:48:36Z DEBUG rcm-tools-fedora-rpms: using metadata from Thu 14 Feb 2019 02:11:48 PM CET.
  2019-03-20T12:48:36Z DEBUG repo: using cache for: rhel7-csb
  2019-03-20T12:48:36Z DEBUG not found other for: RHEL7 CSB packages
  2019-03-20T12:48:36Z DEBUG not found modules for: RHEL7 CSB packages
  2019-03-20T12:48:36Z DEBUG not found deltainfo for: RHEL7 CSB packages
  2019-03-20T12:48:36Z DEBUG not found updateinfo for: RHEL7 CSB packages
  2019-03-20T12:48:36Z DEBUG rhel7-csb: using metadata from Wed 20 Mar 2019 11:26:25 AM CET.
  2019-03-20T12:48:36Z DEBUG repo: using cache for: rpmfusion-free-updates
  2019-03-20T12:48:36Z DEBUG not found other for: RPM Fusion for Fedora 29 - Free - Updates
  2019-03-20T12:48:36Z DEBUG not found modules for: RPM Fusion for Fedora 29 - Free - Updates
  2019-03-20T12:48:36Z DEBUG not found deltainfo for: RPM Fusion for Fedora 29 - Free - Updates
  2019-03-20T12:48:36Z DEBUG not found updateinfo for: RPM Fusion for Fedora 29 - Free - Updates
  2019-03-20T12:48:36Z DEBUG rpmfusion-free-updates: using metadata from Wed 20 Mar 2019 08:57:23 AM CET.
  2019-03-20T12:48:36Z DEBUG repo: using cache for: rpmfusion-free
  2019-03-20T12:48:36Z DEBUG not found other for: RPM Fusion for Fedora 29 - Free
  2019-03-20T12:48:36Z DEBUG not found modules for: RPM Fusion for Fedora 29 - Free
  2019-03-20T12:48:36Z DEBUG not found deltainfo for: RPM Fusion for Fedora 29 - Free
  2019-03-20T12:48:36Z DEBUG not found updateinfo for: RPM Fusion for Fedora 29 - Free
  2019-03-20T12:48:36Z DEBUG rpmfusion-free: using metadata from Tue 23 Oct 2018 01:05:19 PM CEST.
  2019-03-20T12:48:36Z DEBUG repo: using cache for: rpmfusion-nonfree-updates
  2019-03-20T12:48:36Z DEBUG not found other for: RPM Fusion for Fedora 29 - Nonfree - Updates
  2019-03-20T12:48:36Z DEBUG not found modules for: RPM Fusion for Fedora 29 - Nonfree - Updates
  2019-03-20T12:48:36Z DEBUG not found deltainfo for: RPM Fusion for Fedora 29 - Nonfree - Updates
  2019-03-20T12:48:36Z DEBUG not found updateinfo for: RPM Fusion for Fedora 29 - Nonfree - Updates
  2019-03-20T12:48:36Z DEBUG rpmfusion-nonfree-updates: using metadata from Wed 20 Mar 2019 09:46:22 AM CET.
  2019-03-20T12:48:36Z DEBUG repo: using cache for: rpmfusion-nonfree
  2019-03-20T12:48:36Z DEBUG not found other for: RPM Fusion for Fedora 29 - Nonfree
  2019-03-20T12:48:36Z DEBUG not found modules for: RPM Fusion for Fedora 29 - Nonfree
  2019-03-20T12:48:36Z DEBUG not found deltainfo for: RPM Fusion for Fedora 29 - Nonfree
  2019-03-20T12:48:36Z DEBUG not found updateinfo for: RPM Fusion for Fedora 29 - Nonfree
  2019-03-20T12:48:36Z DEBUG rpmfusion-nonfree: using metadata from Tue 23 Oct 2018 01:34:17 PM CEST.
  2019-03-20T12:48:36Z INFO Last metadata expiration check: 0:14:09 ago on Wed 20 Mar 2019 01:34:27 PM CET.
  2019-03-20T12:48:36Z DDEBUG timer: sack setup: 1128 ms
  2019-03-20T12:48:36Z DEBUG Completion plugin: Generating completion cache...
  2019-03-20T12:48:36Z DEBUG Includes in repo rhel7-csb: redhat-internal-*
  2019-03-20T12:48:37Z INFO --- logging initialized ---
  2019-03-20T12:48:37Z DDEBUG timer: config: 2 ms
  2019-03-20T12:48:37Z DEBUG Loaded plugins: builddep, changelog, config-manager, copr, debug, debuginfo-install, download, generate_completion_cache, needs-restarting, playground, repoclosure, repodiff, repograph, repomanage, reposync, system-upgrade
  2019-03-20T12:48:37Z DEBUG DNF version: 4.1.0
  2019-03-20T12:48:37Z DDEBUG Command: dnf downgrade ./libvirt-devel-5.1.0-1.fc29.x86_64.rpm ./libvirt-daemon-driver-lxc-5.1.0-1.fc29.x86_64.rpm ./libvirt-admin-5.1.0-1.fc29.x86_64.rpm ./libvirt-daemon-driver-storage-gluster-5.1.0-1.fc29.x86_64.rpm ./libvirt-client-5.1.0-1.fc29.x86_64.rpm ./libvirt-daemon-5.1.0-1.fc29.x86_64.rpm ./libvirt-daemon-driver-storage-logical-5.1.0-1.fc29.x86_64.rpm ./libvirt-nss-5.1.0-1.fc29.x86_64.rpm ./libvirt-login-shell-5.1.0-1.fc29.x86_64.rpm ./libvirt-daemon-config-nwfilter-5.1.0-1.fc29.x86_64.rpm ./libvirt-daemon-lxc-5.1.0-1.fc29.x86_64.rpm ./libvirt-daemon-kvm-5.1.0-1.fc29.x86_64.rpm ./libvirt-daemon-driver-storage-core-5.1.0-1.fc29.x86_64.rpm ./libvirt-daemon-xen-5.1.0-1.fc29.x86_64.rpm ./libvirt-daemon-driver-storage-5.1.0-1.fc29.x86_64.rpm ./libvirt-wireshark-5.1.0-1.fc29.x86_64.rpm ./libvirt-daemon-driver-storage-zfs-5.1.0-1.fc29.x86_64.rpm ./libvirt-daemon-driver-network-5.1.0-1.fc29.x86_64.rpm ./libvirt-lock-sanlock-5.1.0-1.fc29.x86_64.rpm ./libvirt-daemon-driver-nodedev-5.1.0-1.fc29.x86_64.rpm ./libvirt-daemon-vbox-5.1.0-1.fc29.x86_64.rpm ./libvirt-daemon-driver-storage-iscsi-direct-5.1.0-1.fc29.x86_64.rpm ./libvirt-daemon-driver-storage-mpath-5.1.0-1.fc29.x86_64.rpm ./libvirt-docs-5.1.0-1.fc29.x86_64.rpm ./libvirt-daemon-qemu-5.1.0-1.fc29.x86_64.rpm ./libvirt-daemon-driver-qemu-5.1.0-1.fc29.x86_64.rpm ./libvirt-libs-5.1.0-1.fc29.x86_64.rpm ./libvirt-daemon-driver-storage-rbd-5.1.0-1.fc29.x86_64.rpm ./libvirt-daemon-driver-storage-sheepdog-5.1.0-1.fc29.x86_64.rpm ./libvirt-daemon-driver-libxl-5.1.0-1.fc29.x86_64.rpm ./libvirt-daemon-driver-storage-iscsi-5.1.0-1.fc29.x86_64.rpm ./libvirt-daemon-driver-nwfilter-5.1.0-1.fc29.x86_64.rpm ./libvirt-daemon-config-network-5.1.0-1.fc29.x86_64.rpm ./libvirt-daemon-driver-secret-5.1.0-1.fc29.x86_64.rpm ./libvirt-daemon-driver-storage-scsi-5.1.0-1.fc29.x86_64.rpm ./libvirt-daemon-driver-storage-disk-5.1.0-1.fc29.x86_64.rpm ./libvirt-bash-completion-5.1.0-1.fc29.x86_64.rpm ./libvirt-daemon-driver-interface-5.1.0-1.fc29.x86_64.rpm ./libvirt-daemon-driver-vbox-5.1.0-1.fc29.x86_64.rpm ./libvirt-debugsource-5.1.0-1.fc29.x86_64.rpm 
  2019-03-20T12:48:37Z DDEBUG Installroot: /
  2019-03-20T12:48:37Z DDEBUG Releasever: 29
  2019-03-20T12:48:37Z DEBUG cachedir: /var/cache/dnf
  2019-03-20T12:48:37Z DDEBUG Base command: downgrade
  2019-03-20T12:48:37Z DDEBUG Extra commands: ['downgrade', './libvirt-devel-5.1.0-1.fc29.x86_64.rpm', './libvirt-daemon-driver-lxc-5.1.0-1.fc29.x86_64.rpm', './libvirt-admin-5.1.0-1.fc29.x86_64.rpm', './libvirt-daemon-driver-storage-gluster-5.1.0-1.fc29.x86_64.rpm', './libvirt-client-5.1.0-1.fc29.x86_64.rpm', './libvirt-daemon-5.1.0-1.fc29.x86_64.rpm', './libvirt-daemon-driver-storage-logical-5.1.0-1.fc29.x86_64.rpm', './libvirt-nss-5.1.0-1.fc29.x86_64.rpm', './libvirt-login-shell-5.1.0-1.fc29.x86_64.rpm', './libvirt-daemon-config-nwfilter-5.1.0-1.fc29.x86_64.rpm', './libvirt-daemon-lxc-5.1.0-1.fc29.x86_64.rpm', './libvirt-daemon-kvm-5.1.0-1.fc29.x86_64.rpm', './libvirt-daemon-driver-storage-core-5.1.0-1.fc29.x86_64.rpm', './libvirt-daemon-xen-5.1.0-1.fc29.x86_64.rpm', './libvirt-daemon-driver-storage-5.1.0-1.fc29.x86_64.rpm', './libvirt-wireshark-5.1.0-1.fc29.x86_64.rpm', './libvirt-daemon-driver-storage-zfs-5.1.0-1.fc29.x86_64.rpm', './libvirt-daemon-driver-network-5.1.0-1.fc29.x86_64.rpm', './libvirt-lock-sanlock-5.1.0-1.fc29.x86_64.rpm', './libvirt-daemon-driver-nodedev-5.1.0-1.fc29.x86_64.rpm', './libvirt-daemon-vbox-5.1.0-1.fc29.x86_64.rpm', './libvirt-daemon-driver-storage-iscsi-direct-5.1.0-1.fc29.x86_64.rpm', './libvirt-daemon-driver-storage-mpath-5.1.0-1.fc29.x86_64.rpm', './libvirt-docs-5.1.0-1.fc29.x86_64.rpm', './libvirt-daemon-qemu-5.1.0-1.fc29.x86_64.rpm', './libvirt-daemon-driver-qemu-5.1.0-1.fc29.x86_64.rpm', './libvirt-libs-5.1.0-1.fc29.x86_64.rpm', './libvirt-daemon-driver-storage-rbd-5.1.0-1.fc29.x86_64.rpm', './libvirt-daemon-driver-storage-sheepdog-5.1.0-1.fc29.x86_64.rpm', './libvirt-daemon-driver-libxl-5.1.0-1.fc29.x86_64.rpm', './libvirt-daemon-driver-storage-iscsi-5.1.0-1.fc29.x86_64.rpm', './libvirt-daemon-driver-nwfilter-5.1.0-1.fc29.x86_64.rpm', './libvirt-daemon-config-network-5.1.0-1.fc29.x86_64.rpm', './libvirt-daemon-driver-secret-5.1.0-1.fc29.x86_64.rpm', './libvirt-daemon-driver-storage-scsi-5.1.0-1.fc29.x86_64.rpm', './libvirt-daemon-driver-storage-disk-5.1.0-1.fc29.x86_64.rpm', './libvirt-bash-completion-5.1.0-1.fc29.x86_64.rpm', './libvirt-daemon-driver-interface-5.1.0-1.fc29.x86_64.rpm', './libvirt-daemon-driver-vbox-5.1.0-1.fc29.x86_64.rpm', './libvirt-debugsource-5.1.0-1.fc29.x86_64.rpm']
  2019-03-20T12:48:37Z DEBUG Unknown configuration value: failovermethod=priority in /etc/yum.repos.d/fedora-cisco-openh264.repo; Configuration: OptionBinding with id "failovermethod" does not exist
  2019-03-20T12:48:37Z DEBUG Unknown configuration value: failovermethod=priority in /etc/yum.repos.d/fedora-modular.repo; Configuration: OptionBinding with id "failovermethod" does not exist
  2019-03-20T12:48:37Z DEBUG Unknown configuration value: failovermethod=priority in /etc/yum.repos.d/fedora-modular.repo; Configuration: OptionBinding with id "failovermethod" does not exist
  2019-03-20T12:48:37Z DEBUG Unknown configuration value: failovermethod=priority in /etc/yum.repos.d/fedora-modular.repo; Configuration: OptionBinding with id "failovermethod" does not exist
  2019-03-20T12:48:37Z DEBUG Unknown configuration value: failovermethod=priority in /etc/yum.repos.d/fedora-updates-modular.repo; Configuration: OptionBinding with id "failovermethod" does not exist
  2019-03-20T12:48:37Z DEBUG Unknown configuration value: failovermethod=priority in /etc/yum.repos.d/fedora-updates-modular.repo; Configuration: OptionBinding with id "failovermethod" does not exist
  2019-03-20T12:48:37Z DEBUG Unknown configuration value: failovermethod=priority in /etc/yum.repos.d/fedora-updates-modular.repo; Configuration: OptionBinding with id "failovermethod" does not exist
  2019-03-20T12:48:37Z DEBUG Unknown configuration value: failovermethod=priority in /etc/yum.repos.d/fedora-updates-testing-modular.repo; Configuration: OptionBinding with id "failovermethod" does not exist
  2019-03-20T12:48:37Z DEBUG Unknown configuration value: failovermethod=priority in /etc/yum.repos.d/fedora-updates-testing-modular.repo; Configuration: OptionBinding with id "failovermethod" does not exist
  2019-03-20T12:48:37Z DEBUG Unknown configuration value: failovermethod=priority in /etc/yum.repos.d/fedora-updates-testing-modular.repo; Configuration: OptionBinding with id "failovermethod" does not exist
  2019-03-20T12:48:37Z DEBUG Unknown configuration value: failovermethod=priority in /etc/yum.repos.d/fedora-updates-testing.repo; Configuration: OptionBinding with id "failovermethod" does not exist
  2019-03-20T12:48:37Z DEBUG Unknown configuration value: failovermethod=priority in /etc/yum.repos.d/fedora-updates-testing.repo; Configuration: OptionBinding with id "failovermethod" does not exist
  2019-03-20T12:48:37Z DEBUG Unknown configuration value: failovermethod=priority in /etc/yum.repos.d/fedora-updates-testing.repo; Configuration: OptionBinding with id "failovermethod" does not exist
  2019-03-20T12:48:37Z DEBUG Unknown configuration value: failovermethod=priority in /etc/yum.repos.d/fedora-updates.repo; Configuration: OptionBinding with id "failovermethod" does not exist
  2019-03-20T12:48:37Z DEBUG Unknown configuration value: failovermethod=priority in /etc/yum.repos.d/fedora-updates.repo; Configuration: OptionBinding with id "failovermethod" does not exist
  2019-03-20T12:48:37Z DEBUG Unknown configuration value: failovermethod=priority in /etc/yum.repos.d/fedora-updates.repo; Configuration: OptionBinding with id "failovermethod" does not exist
  2019-03-20T12:48:37Z DEBUG Unknown configuration value: failovermethod=priority in /etc/yum.repos.d/fedora.repo; Configuration: OptionBinding with id "failovermethod" does not exist
  2019-03-20T12:48:37Z DEBUG Unknown configuration value: failovermethod=priority in /etc/yum.repos.d/fedora.repo; Configuration: OptionBinding with id "failovermethod" does not exist
  2019-03-20T12:48:37Z DEBUG Unknown configuration value: failovermethod=priority in /etc/yum.repos.d/fedora.repo; Configuration: OptionBinding with id "failovermethod" does not exist
  2019-03-20T12:48:37Z DEBUG repo: using cache for: group_virtmaint-sig-virt-preview
  2019-03-20T12:48:37Z DEBUG not found other for: Copr repo for virt-preview owned by @virtmaint-sig
  2019-03-20T12:48:37Z DEBUG not found modules for: Copr repo for virt-preview owned by @virtmaint-sig
  2019-03-20T12:48:37Z DEBUG not found deltainfo for: Copr repo for virt-preview owned by @virtmaint-sig
  2019-03-20T12:48:37Z DEBUG not found updateinfo for: Copr repo for virt-preview owned by @virtmaint-sig
  2019-03-20T12:48:37Z DEBUG group_virtmaint-sig-virt-preview: using metadata from Wed 20 Mar 2019 06:14:54 AM CET.
  2019-03-20T12:48:37Z DEBUG repo: using cache for: fedora-modular
  2019-03-20T12:48:37Z DEBUG not found other for: Fedora Modular 29 - x86_64
  2019-03-20T12:48:37Z DEBUG not found deltainfo for: Fedora Modular 29 - x86_64
  2019-03-20T12:48:37Z DEBUG not found updateinfo for: Fedora Modular 29 - x86_64
  2019-03-20T12:48:37Z DEBUG fedora-modular: using metadata from Thu 25 Oct 2018 12:22:05 AM CEST.
  2019-03-20T12:48:37Z DEBUG repo: using cache for: updates-modular
  2019-03-20T12:48:37Z DEBUG not found other for: Fedora Modular 29 - x86_64 - Updates
  2019-03-20T12:48:37Z DEBUG not found deltainfo for: Fedora Modular 29 - x86_64 - Updates
  2019-03-20T12:48:37Z DEBUG updates-modular: using metadata from Fri 15 Mar 2019 06:39:11 AM CET.
  2019-03-20T12:48:37Z DEBUG repo: using cache for: updates
  2019-03-20T12:48:37Z DEBUG not found other for: Fedora 29 - x86_64 - Updates
  2019-03-20T12:48:37Z DEBUG not found modules for: Fedora 29 - x86_64 - Updates
  2019-03-20T12:48:37Z DEBUG updates: using metadata from Tue 19 Mar 2019 05:43:55 AM CET.
  2019-03-20T12:48:37Z DEBUG repo: using cache for: fedora
  2019-03-20T12:48:37Z DEBUG not found other for: Fedora 29 - x86_64
  2019-03-20T12:48:37Z DEBUG not found modules for: Fedora 29 - x86_64
  2019-03-20T12:48:37Z DEBUG not found deltainfo for: Fedora 29 - x86_64
  2019-03-20T12:48:37Z DEBUG not found updateinfo for: Fedora 29 - x86_64
  2019-03-20T12:48:38Z DEBUG fedora: using metadata from Thu 25 Oct 2018 12:20:15 AM CEST.
  2019-03-20T12:48:38Z DEBUG repo: using cache for: rcm-tools-fedora-rpms
  2019-03-20T12:48:38Z DEBUG not found other for: RCM Tools for Fedora 29 (RPMs)
  2019-03-20T12:48:38Z DEBUG not found modules for: RCM Tools for Fedora 29 (RPMs)
  2019-03-20T12:48:38Z DEBUG not found deltainfo for: RCM Tools for Fedora 29 (RPMs)
  2019-03-20T12:48:38Z DEBUG not found updateinfo for: RCM Tools for Fedora 29 (RPMs)
  2019-03-20T12:48:38Z DEBUG rcm-tools-fedora-rpms: using metadata from Thu 14 Feb 2019 02:11:48 PM CET.
  2019-03-20T12:48:38Z DEBUG repo: using cache for: rhel7-csb
  2019-03-20T12:48:38Z DEBUG not found other for: RHEL7 CSB packages
  2019-03-20T12:48:38Z DEBUG not found modules for: RHEL7 CSB packages
  2019-03-20T12:48:38Z DEBUG not found deltainfo for: RHEL7 CSB packages
  2019-03-20T12:48:38Z DEBUG not found updateinfo for: RHEL7 CSB packages
  2019-03-20T12:48:38Z DEBUG rhel7-csb: using metadata from Wed 20 Mar 2019 11:26:25 AM CET.
  2019-03-20T12:48:38Z DEBUG repo: using cache for: rpmfusion-free-updates
  2019-03-20T12:48:38Z DEBUG not found other for: RPM Fusion for Fedora 29 - Free - Updates
  2019-03-20T12:48:38Z DEBUG not found modules for: RPM Fusion for Fedora 29 - Free - Updates
  2019-03-20T12:48:38Z DEBUG not found deltainfo for: RPM Fusion for Fedora 29 - Free - Updates
  2019-03-20T12:48:38Z DEBUG not found updateinfo for: RPM Fusion for Fedora 29 - Free - Updates
  2019-03-20T12:48:38Z DEBUG rpmfusion-free-updates: using metadata from Wed 20 Mar 2019 08:57:23 AM CET.
  2019-03-20T12:48:38Z DEBUG repo: using cache for: rpmfusion-free
  2019-03-20T12:48:38Z DEBUG not found other for: RPM Fusion for Fedora 29 - Free
  2019-03-20T12:48:38Z DEBUG not found modules for: RPM Fusion for Fedora 29 - Free
  2019-03-20T12:48:38Z DEBUG not found deltainfo for: RPM Fusion for Fedora 29 - Free
  2019-03-20T12:48:38Z DEBUG not found updateinfo for: RPM Fusion for Fedora 29 - Free
  2019-03-20T12:48:38Z DEBUG rpmfusion-free: using metadata from Tue 23 Oct 2018 01:05:19 PM CEST.
  2019-03-20T12:48:38Z DEBUG repo: using cache for: rpmfusion-nonfree-updates
  2019-03-20T12:48:38Z DEBUG not found other for: RPM Fusion for Fedora 29 - Nonfree - Updates
  2019-03-20T12:48:38Z DEBUG not found modules for: RPM Fusion for Fedora 29 - Nonfree - Updates
  2019-03-20T12:48:38Z DEBUG not found deltainfo for: RPM Fusion for Fedora 29 - Nonfree - Updates
  2019-03-20T12:48:38Z DEBUG not found updateinfo for: RPM Fusion for Fedora 29 - Nonfree - Updates
  2019-03-20T12:48:38Z DEBUG rpmfusion-nonfree-updates: using metadata from Wed 20 Mar 2019 09:46:22 AM CET.
  2019-03-20T12:48:38Z DEBUG repo: using cache for: rpmfusion-nonfree
  2019-03-20T12:48:38Z DEBUG not found other for: RPM Fusion for Fedora 29 - Nonfree
  2019-03-20T12:48:38Z DEBUG not found modules for: RPM Fusion for Fedora 29 - Nonfree
  2019-03-20T12:48:38Z DEBUG not found deltainfo for: RPM Fusion for Fedora 29 - Nonfree
  2019-03-20T12:48:38Z DEBUG not found updateinfo for: RPM Fusion for Fedora 29 - Nonfree
  2019-03-20T12:48:38Z DEBUG rpmfusion-nonfree: using metadata from Tue 23 Oct 2018 01:34:17 PM CEST.
  2019-03-20T12:48:38Z INFO Last metadata expiration check: 0:14:11 ago on Wed 20 Mar 2019 01:34:27 PM CET.
  2019-03-20T12:48:38Z DDEBUG timer: sack setup: 1141 ms
  2019-03-20T12:48:38Z DEBUG Completion plugin: Generating completion cache...
  2019-03-20T12:48:39Z DEBUG Includes in repo rhel7-csb: redhat-internal-*
  2019-03-20T12:48:39Z WARNING Package libvirt-devel not installed, cannot downgrade it.
  2019-03-20T12:48:39Z INFO No match for argument: libvirt-devel-5.1.0-1.fc29.x86_64.rpm
  2019-03-20T12:48:39Z WARNING Package libvirt-admin not installed, cannot downgrade it.
  2019-03-20T12:48:39Z INFO No match for argument: libvirt-admin-5.1.0-1.fc29.x86_64.rpm
  2019-03-20T12:48:39Z WARNING Package libvirt-login-shell not installed, cannot downgrade it.
  2019-03-20T12:48:39Z INFO No match for argument: libvirt-login-shell-5.1.0-1.fc29.x86_64.rpm
  2019-03-20T12:48:39Z WARNING Package libvirt-daemon-lxc not installed, cannot downgrade it.
  2019-03-20T12:48:39Z INFO No match for argument: libvirt-daemon-lxc-5.1.0-1.fc29.x86_64.rpm
  2019-03-20T12:48:39Z WARNING Package libvirt-daemon-xen not installed, cannot downgrade it.
  2019-03-20T12:48:39Z INFO No match for argument: libvirt-daemon-xen-5.1.0-1.fc29.x86_64.rpm
  2019-03-20T12:48:39Z WARNING Package libvirt-wireshark not installed, cannot downgrade it.
  2019-03-20T12:48:39Z INFO No match for argument: libvirt-wireshark-5.1.0-1.fc29.x86_64.rpm
  2019-03-20T12:48:39Z WARNING Package libvirt-lock-sanlock not installed, cannot downgrade it.
  2019-03-20T12:48:39Z INFO No match for argument: libvirt-lock-sanlock-5.1.0-1.fc29.x86_64.rpm
  2019-03-20T12:48:39Z WARNING Package libvirt-daemon-vbox not installed, cannot downgrade it.
  2019-03-20T12:48:39Z INFO No match for argument: libvirt-daemon-vbox-5.1.0-1.fc29.x86_64.rpm
  2019-03-20T12:48:39Z WARNING Package libvirt-docs not installed, cannot downgrade it.
  2019-03-20T12:48:39Z INFO No match for argument: libvirt-docs-5.1.0-1.fc29.x86_64.rpm
  2019-03-20T12:48:39Z WARNING Package libvirt-daemon-qemu not installed, cannot downgrade it.
  2019-03-20T12:48:39Z INFO No match for argument: libvirt-daemon-qemu-5.1.0-1.fc29.x86_64.rpm
  2019-03-20T12:48:39Z WARNING Package libvirt-debugsource not installed, cannot downgrade it.
  2019-03-20T12:48:39Z INFO No match for argument: libvirt-debugsource-5.1.0-1.fc29.x86_64.rpm
  2019-03-20T12:48:39Z DEBUG --> Starting dependency resolution
  2019-03-20T12:48:39Z DEBUG --> Finished dependency resolution
  2019-03-20T12:48:39Z DDEBUG timer: depsolve: 597 ms
  2019-03-20T12:48:39Z SUBDEBUG 
  Traceback (most recent call last):
    File "/usr/lib/python3.7/site-packages/dnf/cli/main.py", line 123, in cli_run
      ret = resolving(cli, base)
    File "/usr/lib/python3.7/site-packages/dnf/cli/main.py", line 160, in resolving
      base.resolve(cli.demands.allow_erasing)
    File "/usr/lib/python3.7/site-packages/dnf/base.py", line 783, in resolve
      raise exc
  dnf.exceptions.DepsolveError: 
   Problem: problem with installed package libvirt-5.2.0-1.fc29.x86_64
    - package libvirt-5.2.0-1.fc29.x86_64 requires libvirt-daemon-driver-lxc = 5.2.0-1.fc29, but none of the providers can be installed
    - package libvirt-5.1.0-2.fc29.x86_64 requires libvirt-daemon-driver-lxc = 5.1.0-2.fc29, but none of the providers can be installed
    - package libvirt-4.7.0-1.fc29.x86_64 requires libvirt-daemon-driver-lxc = 4.7.0-1.fc29, but none of the providers can be installed
    - cannot install both libvirt-daemon-driver-lxc-5.1.0-1.fc29.x86_64 and libvirt-daemon-driver-lxc-5.2.0-1.fc29.x86_64
    - cannot install both libvirt-daemon-driver-lxc-5.1.0-1.fc29.x86_64 and libvirt-daemon-driver-lxc-5.1.0-2.fc29.x86_64
    - cannot install both libvirt-daemon-driver-lxc-5.1.0-1.fc29.x86_64 and libvirt-daemon-driver-lxc-4.7.0-1.fc29.x86_64
    - conflicting requests
  2019-03-20T12:48:39Z CRITICAL Error: 
   Problem: problem with installed package libvirt-5.2.0-1.fc29.x86_64
    - package libvirt-5.2.0-1.fc29.x86_64 requires libvirt-daemon-driver-lxc = 5.2.0-1.fc29, but none of the providers can be installed
    - package libvirt-5.1.0-2.fc29.x86_64 requires libvirt-daemon-driver-lxc = 5.1.0-2.fc29, but none of the providers can be installed
    - package libvirt-4.7.0-1.fc29.x86_64 requires libvirt-daemon-driver-lxc = 4.7.0-1.fc29, but none of the providers can be installed
    - cannot install both libvirt-daemon-driver-lxc-5.1.0-1.fc29.x86_64 and libvirt-daemon-driver-lxc-5.2.0-1.fc29.x86_64
    - cannot install both libvirt-daemon-driver-lxc-5.1.0-1.fc29.x86_64 and libvirt-daemon-driver-lxc-5.1.0-2.fc29.x86_64
    - cannot install both libvirt-daemon-driver-lxc-5.1.0-1.fc29.x86_64 and libvirt-daemon-driver-lxc-4.7.0-1.fc29.x86_64
    - conflicting requests
  2019-03-20T12:48:39Z INFO (try to add '--allowerasing' to command line to replace conflicting packages or '--skip-broken' to skip uninstallable packages)
  2019-03-20T12:48:39Z DDEBUG Cleaning up.

As you can see, both times 'dnf downgrade' is called with 41 local
files as arguments, but while the second time it manages to report
an error to the user and clean up after itself, the first time it
just kinda... Dies after a while.

Comment 4 Andrea Bolognani 2019-03-22 11:17:34 UTC
(In reply to Jaroslav Mracek from comment #2)
> Please could you try to reproduce the issue with dnf-0:4.2.1-1.fc29.noarch,
> libdnf-0:0.28.0-1.fc29.x86_64, and libsolv-0:0.7.2-2.fc29.x86_64?

Unfortunately I can't seem to be able to reproduce the issue today at
all... The libvirt 5.1.0 RPMs in the directory are the very same as
yesterday, but I had to downgrade and subsequently upgrade libvirt a
couple times since reporting the bug, and now I'm no longer hitting
the crashes. I'm sorry, I know this is not very useful to you :(

Comment 5 Jaroslav Mracek 2019-03-22 12:53:32 UTC
I am also unable to reproduce the bug. I even tried 900 packages to downgrade without any issue. We really need a reproducer, otherwise we cannot do anything here. Please reopen the bug with additional data, reproducer or even if you feel unhappy about closing the bug.

Comment 6 Andrea Bolognani 2019-05-24 10:31:36 UTC
(In reply to Jaroslav Mracek from comment #5)
> I am also unable to reproduce the bug. I even tried 900 packages to
> downgrade without any issue. We really need a reproducer, otherwise we
> cannot do anything here. Please reopen the bug with additional data,
> reproducer or even if you feel unhappy about closing the bug.

Okay, it showed up again today:

  $ sudo dnf reinstall ./*.rpm
  Last metadata expiration check: 0:15:09 ago on Fri 24 May 2019 09:26:50 AM CEST.
  free(): invalid next size (normal)
  Aborted
  $

This time I can get it to reproduce (though not consistently) with a
smaller number of packages than last time:

  $ rpm -qa | grep -E 'libvirt.*5.4' | wc -l
  27
  $ sudo dnf reinstall $(rpm -qa | grep -E 'libvirt.*5.4' | sed 's,^,./,g;s,$,.rpm,g')
  Last metadata expiration check: 0:22:57 ago on Fri 24 May 2019 09:26:50 AM CEST.
  corrupted size vs. prev_size
  Aborted
  $

Looking at /var/log/dnf.log, I see

  2019-05-24T07:55:22Z INFO --- logging initialized ---
  2019-05-24T07:55:22Z DDEBUG timer: config: 2 ms
  2019-05-24T07:55:22Z DEBUG Loaded plugins: builddep, changelog, config-manager, copr, debug, debuginfo-install, download, generate_completion_cache, needs-restarting, playground, repoclosure, repodiff, repograph, repomanage, reposync, system-upgrade
  2019-05-24T07:55:22Z DEBUG DNF version: 4.2.5
  2019-05-24T07:55:22Z DDEBUG Command: dnf reinstall ./libvirt-5.4.0-1.fc30.x86_64.rpm ./libvirt-admin-5.4.0-1.fc30.x86_64.rpm ./libvirt-admin-debuginfo-5.4.0-1.fc30.x86_64.rpm ./libvirt-bash-completion-5.4.0-1.fc30.x86_64.rpm ./libvirt-client-5.4.0-1.fc30.x86_64.rpm ./libvirt-client-debuginfo-5.4.0-1.fc30.x86_64.rpm ./libvirt-daemon-5.4.0-1.fc30.x86_64.rpm ./libvirt-daemon-config-network-5.4.0-1.fc30.x86_64.rpm ./libvirt-daemon-config-nwfilter-5.4.0-1.fc30.x86_64.rpm ./libvirt-daemon-debuginfo-5.4.0-1.fc30.x86_64.rpm ./libvirt-daemon-driver-interface-5.4.0-1.fc30.x86_64.rpm ./libvirt-daemon-driver-interface-debuginfo-5.4.0-1.fc30.x86_64.rpm ./libvirt-daemon-driver-libxl-5.4.0-1.fc30.x86_64.rpm ./libvirt-daemon-driver-libxl-debuginfo-5.4.0-1.fc30.x86_64.rpm ./libvirt-daemon-driver-lxc-5.4.0-1.fc30.x86_64.rpm ./libvirt-daemon-driver-lxc-debuginfo-5.4.0-1.fc30.x86_64.rpm ./libvirt-daemon-driver-network-5.4.0-1.fc30.x86_64.rpm ./libvirt-daemon-driver-network-debuginfo-5.4.0-1.fc30.x86_64.rpm ./libvirt-daemon-driver-nodedev-5.4.0-1.fc30.x86_64.rpm ./libvirt-daemon-driver-nodedev-debuginfo-5.4.0-1.fc30.x86_64.rpm ./libvirt-daemon-driver-nwfilter-5.4.0-1.fc30.x86_64.rpm ./libvirt-daemon-driver-nwfilter-debuginfo-5.4.0-1.fc30.x86_64.rpm ./libvirt-daemon-driver-qemu-5.4.0-1.fc30.x86_64.rpm ./libvirt-daemon-driver-qemu-debuginfo-5.4.0-1.fc30.x86_64.rpm ./libvirt-daemon-driver-secret-5.4.0-1.fc30.x86_64.rpm ./libvirt-daemon-driver-secret-debuginfo-5.4.0-1.fc30.x86_64.rpm ./libvirt-daemon-driver-storage-5.4.0-1.fc30.x86_64.rpm ./libvirt-daemon-driver-storage-core-5.4.0-1.fc30.x86_64.rpm ./libvirt-daemon-driver-storage-core-debuginfo-5.4.0-1.fc30.x86_64.rpm ./libvirt-daemon-driver-storage-disk-5.4.0-1.fc30.x86_64.rpm ./libvirt-daemon-driver-storage-disk-debuginfo-5.4.0-1.fc30.x86_64.rpm ./libvirt-daemon-driver-storage-gluster-5.4.0-1.fc30.x86_64.rpm ./libvirt-daemon-driver-storage-gluster-debuginfo-5.4.0-1.fc30.x86_64.rpm ./libvirt-daemon-driver-storage-iscsi-5.4.0-1.fc30.x86_64.rpm ./libvirt-daemon-driver-storage-iscsi-debuginfo-5.4.0-1.fc30.x86_64.rpm ./libvirt-daemon-driver-storage-iscsi-direct-5.4.0-1.fc30.x86_64.rpm ./libvirt-daemon-driver-storage-iscsi-direct-debuginfo-5.4.0-1.fc30.x86_64.rpm ./libvirt-daemon-driver-storage-logical-5.4.0-1.fc30.x86_64.rpm ./libvirt-daemon-driver-storage-logical-debuginfo-5.4.0-1.fc30.x86_64.rpm ./libvirt-daemon-driver-storage-mpath-5.4.0-1.fc30.x86_64.rpm ./libvirt-daemon-driver-storage-mpath-debuginfo-5.4.0-1.fc30.x86_64.rpm ./libvirt-daemon-driver-storage-rbd-5.4.0-1.fc30.x86_64.rpm ./libvirt-daemon-driver-storage-rbd-debuginfo-5.4.0-1.fc30.x86_64.rpm ./libvirt-daemon-driver-storage-scsi-5.4.0-1.fc30.x86_64.rpm ./libvirt-daemon-driver-storage-scsi-debuginfo-5.4.0-1.fc30.x86_64.rpm ./libvirt-daemon-driver-storage-sheepdog-5.4.0-1.fc30.x86_64.rpm ./libvirt-daemon-driver-storage-sheepdog-debuginfo-5.4.0-1.fc30.x86_64.rpm ./libvirt-daemon-driver-storage-zfs-5.4.0-1.fc30.x86_64.rpm ./libvirt-daemon-driver-storage-zfs-debuginfo-5.4.0-1.fc30.x86_64.rpm ./libvirt-daemon-driver-vbox-5.4.0-1.fc30.x86_64.rpm ./libvirt-daemon-driver-vbox-debuginfo-5.4.0-1.fc30.x86_64.rpm ./libvirt-daemon-kvm-5.4.0-1.fc30.x86_64.rpm ./libvirt-daemon-lxc-5.4.0-1.fc30.x86_64.rpm ./libvirt-daemon-qemu-5.4.0-1.fc30.x86_64.rpm ./libvirt-daemon-vbox-5.4.0-1.fc30.x86_64.rpm ./libvirt-daemon-xen-5.4.0-1.fc30.x86_64.rpm ./libvirt-debuginfo-5.4.0-1.fc30.x86_64.rpm ./libvirt-debugsource-5.4.0-1.fc30.x86_64.rpm ./libvirt-devel-5.4.0-1.fc30.x86_64.rpm ./libvirt-docs-5.4.0-1.fc30.x86_64.rpm ./libvirt-libs-5.4.0-1.fc30.x86_64.rpm ./libvirt-libs-debuginfo-5.4.0-1.fc30.x86_64.rpm ./libvirt-lock-sanlock-5.4.0-1.fc30.x86_64.rpm ./libvirt-lock-sanlock-debuginfo-5.4.0-1.fc30.x86_64.rpm ./libvirt-login-shell-5.4.0-1.fc30.x86_64.rpm ./libvirt-login-shell-debuginfo-5.4.0-1.fc30.x86_64.rpm ./libvirt-nss-5.4.0-1.fc30.x86_64.rpm ./libvirt-nss-debuginfo-5.4.0-1.fc30.x86_64.rpm ./libvirt-wireshark-5.4.0-1.fc30.x86_64.rpm ./libvirt-wireshark-debuginfo-5.4.0-1.fc30.x86_64.rpm
  2019-05-24T07:55:22Z DDEBUG Installroot: /
  2019-05-24T07:55:22Z DDEBUG Releasever: 30
  2019-05-24T07:55:22Z DEBUG cachedir: /var/cache/dnf
  2019-05-24T07:55:22Z DDEBUG Base command: reinstall
  2019-05-24T07:55:22Z DDEBUG Extra commands: ['reinstall', './libvirt-5.4.0-1.fc30.x86_64.rpm', './libvirt-admin-5.4.0-1.fc30.x86_64.rpm', './libvirt-admin-debuginfo-5.4.0-1.fc30.x86_64.rpm', './libvirt-bash-completion-5.4.0-1.fc30.x86_64.rpm', './libvirt-client-5.4.0-1.fc30.x86_64.rpm', './libvirt-client-debuginfo-5.4.0-1.fc30.x86_64.rpm', './libvirt-daemon-5.4.0-1.fc30.x86_64.rpm', './libvirt-daemon-config-network-5.4.0-1.fc30.x86_64.rpm', './libvirt-daemon-config-nwfilter-5.4.0-1.fc30.x86_64.rpm', './libvirt-daemon-debuginfo-5.4.0-1.fc30.x86_64.rpm', './libvirt-daemon-driver-interface-5.4.0-1.fc30.x86_64.rpm', './libvirt-daemon-driver-interface-debuginfo-5.4.0-1.fc30.x86_64.rpm', './libvirt-daemon-driver-libxl-5.4.0-1.fc30.x86_64.rpm', './libvirt-daemon-driver-libxl-debuginfo-5.4.0-1.fc30.x86_64.rpm', './libvirt-daemon-driver-lxc-5.4.0-1.fc30.x86_64.rpm', './libvirt-daemon-driver-lxc-debuginfo-5.4.0-1.fc30.x86_64.rpm', './libvirt-daemon-driver-network-5.4.0-1.fc30.x86_64.rpm', './libvirt-daemon-driver-network-debuginfo-5.4.0-1.fc30.x86_64.rpm', './libvirt-daemon-driver-nodedev-5.4.0-1.fc30.x86_64.rpm', './libvirt-daemon-driver-nodedev-debuginfo-5.4.0-1.fc30.x86_64.rpm', './libvirt-daemon-driver-nwfilter-5.4.0-1.fc30.x86_64.rpm', './libvirt-daemon-driver-nwfilter-debuginfo-5.4.0-1.fc30.x86_64.rpm', './libvirt-daemon-driver-qemu-5.4.0-1.fc30.x86_64.rpm', './libvirt-daemon-driver-qemu-debuginfo-5.4.0-1.fc30.x86_64.rpm', './libvirt-daemon-driver-secret-5.4.0-1.fc30.x86_64.rpm', './libvirt-daemon-driver-secret-debuginfo-5.4.0-1.fc30.x86_64.rpm', './libvirt-daemon-driver-storage-5.4.0-1.fc30.x86_64.rpm', './libvirt-daemon-driver-storage-core-5.4.0-1.fc30.x86_64.rpm', './libvirt-daemon-driver-storage-core-debuginfo-5.4.0-1.fc30.x86_64.rpm', './libvirt-daemon-driver-storage-disk-5.4.0-1.fc30.x86_64.rpm', './libvirt-daemon-driver-storage-disk-debuginfo-5.4.0-1.fc30.x86_64.rpm', './libvirt-daemon-driver-storage-gluster-5.4.0-1.fc30.x86_64.rpm', './libvirt-daemon-driver-storage-gluster-debuginfo-5.4.0-1.fc30.x86_64.rpm', './libvirt-daemon-driver-storage-iscsi-5.4.0-1.fc30.x86_64.rpm', './libvirt-daemon-driver-storage-iscsi-debuginfo-5.4.0-1.fc30.x86_64.rpm', './libvirt-daemon-driver-storage-iscsi-direct-5.4.0-1.fc30.x86_64.rpm', './libvirt-daemon-driver-storage-iscsi-direct-debuginfo-5.4.0-1.fc30.x86_64.rpm', './libvirt-daemon-driver-storage-logical-5.4.0-1.fc30.x86_64.rpm', './libvirt-daemon-driver-storage-logical-debuginfo-5.4.0-1.fc30.x86_64.rpm', './libvirt-daemon-driver-storage-mpath-5.4.0-1.fc30.x86_64.rpm', './libvirt-daemon-driver-storage-mpath-debuginfo-5.4.0-1.fc30.x86_64.rpm', './libvirt-daemon-driver-storage-rbd-5.4.0-1.fc30.x86_64.rpm', './libvirt-daemon-driver-storage-rbd-debuginfo-5.4.0-1.fc30.x86_64.rpm', './libvirt-daemon-driver-storage-scsi-5.4.0-1.fc30.x86_64.rpm', './libvirt-daemon-driver-storage-scsi-debuginfo-5.4.0-1.fc30.x86_64.rpm', './libvirt-daemon-driver-storage-sheepdog-5.4.0-1.fc30.x86_64.rpm', './libvirt-daemon-driver-storage-sheepdog-debuginfo-5.4.0-1.fc30.x86_64.rpm', './libvirt-daemon-driver-storage-zfs-5.4.0-1.fc30.x86_64.rpm', './libvirt-daemon-driver-storage-zfs-debuginfo-5.4.0-1.fc30.x86_64.rpm', './libvirt-daemon-driver-vbox-5.4.0-1.fc30.x86_64.rpm', './libvirt-daemon-driver-vbox-debuginfo-5.4.0-1.fc30.x86_64.rpm', './libvirt-daemon-kvm-5.4.0-1.fc30.x86_64.rpm', './libvirt-daemon-lxc-5.4.0-1.fc30.x86_64.rpm', './libvirt-daemon-qemu-5.4.0-1.fc30.x86_64.rpm', './libvirt-daemon-vbox-5.4.0-1.fc30.x86_64.rpm', './libvirt-daemon-xen-5.4.0-1.fc30.x86_64.rpm', './libvirt-debuginfo-5.4.0-1.fc30.x86_64.rpm', './libvirt-debugsource-5.4.0-1.fc30.x86_64.rpm', './libvirt-devel-5.4.0-1.fc30.x86_64.rpm', './libvirt-docs-5.4.0-1.fc30.x86_64.rpm', './libvirt-libs-5.4.0-1.fc30.x86_64.rpm', './libvirt-libs-debuginfo-5.4.0-1.fc30.x86_64.rpm', './libvirt-lock-sanlock-5.4.0-1.fc30.x86_64.rpm', './libvirt-lock-sanlock-debuginfo-5.4.0-1.fc30.x86_64.rpm', './libvirt-login-shell-5.4.0-1.fc30.x86_64.rpm', './libvirt-login-shell-debuginfo-5.4.0-1.fc30.x86_64.rpm', './libvirt-nss-5.4.0-1.fc30.x86_64.rpm', './libvirt-nss-debuginfo-5.4.0-1.fc30.x86_64.rpm', './libvirt-wireshark-5.4.0-1.fc30.x86_64.rpm', './libvirt-wireshark-debuginfo-5.4.0-1.fc30.x86_64.rpm']
  2019-05-24T07:55:22Z DEBUG Unknown configuration value: failovermethod=priority in /etc/yum.repos.d/fedora-updates-modular.repo; Configuration: OptionBinding with id "failovermethod" does not exist
  2019-05-24T07:55:22Z DEBUG Unknown configuration value: failovermethod=priority in /etc/yum.repos.d/fedora-updates-modular.repo; Configuration: OptionBinding with id "failovermethod" does not exist
  2019-05-24T07:55:22Z DEBUG Unknown configuration value: failovermethod=priority in /etc/yum.repos.d/fedora-updates-modular.repo; Configuration: OptionBinding with id "failovermethod" does not exist
  2019-05-24T07:55:22Z DEBUG repo: using cache for: group_virtmaint-sig-virt-preview
  2019-05-24T07:55:22Z DEBUG group_virtmaint-sig-virt-preview: using metadata from Thu 23 May 2019 01:21:14 PM CEST.
  2019-05-24T07:55:22Z DEBUG repo: using cache for: fedora-modular
  2019-05-24T07:55:22Z DEBUG fedora-modular: using metadata from Fri 26 Apr 2019 01:47:30 AM CEST.
  2019-05-24T07:55:22Z DEBUG repo: using cache for: updates-modular
  2019-05-24T07:55:22Z DEBUG updates-modular: using metadata from Thu 23 May 2019 04:17:20 PM CEST.
  2019-05-24T07:55:22Z DEBUG repo: using cache for: updates
  2019-05-24T07:55:22Z DEBUG updates: using metadata from Thu 23 May 2019 07:37:23 PM CEST.
  2019-05-24T07:55:23Z DEBUG repo: using cache for: fedora
  2019-05-24T07:55:23Z DEBUG fedora: using metadata from Fri 26 Apr 2019 01:49:41 AM CEST.
  2019-05-24T07:55:23Z DEBUG repo: using cache for: rcm-tools-fedora-rpms
  2019-05-24T07:55:23Z DEBUG rcm-tools-fedora-rpms: using metadata from Thu 23 May 2019 05:26:43 PM CEST.
  2019-05-24T07:55:23Z DEBUG repo: using cache for: rhel7-csb
  2019-05-24T07:55:23Z DEBUG rhel7-csb: using metadata from Thu 23 May 2019 10:13:42 AM CEST.
  2019-05-24T07:55:23Z DEBUG repo: using cache for: rpmfusion-free-updates
  2019-05-24T07:55:23Z DEBUG rpmfusion-free-updates: using metadata from Wed 22 May 2019 12:30:52 PM CEST.
  2019-05-24T07:55:23Z DEBUG repo: using cache for: rpmfusion-free
  2019-05-24T07:55:23Z DEBUG rpmfusion-free: using metadata from Tue 16 Apr 2019 10:46:20 PM CEST.
  2019-05-24T07:55:23Z DEBUG repo: using cache for: rpmfusion-nonfree-updates
  2019-05-24T07:55:23Z DEBUG rpmfusion-nonfree-updates: using metadata from Wed 22 May 2019 12:51:42 PM CEST.
  2019-05-24T07:55:23Z DEBUG repo: using cache for: rpmfusion-nonfree
  2019-05-24T07:55:23Z DEBUG rpmfusion-nonfree: using metadata from Tue 16 Apr 2019 11:53:39 PM CEST.
  2019-05-24T07:55:23Z INFO Last metadata expiration check: 0:28:33 ago on Fri 24 May 2019 09:26:50 AM CEST.
  2019-05-24T07:55:24Z DDEBUG timer: sack setup: 1537 ms
  2019-05-24T07:55:24Z DEBUG Completion plugin: Generating completion cache...
  2019-05-24T07:55:24Z DEBUG Includes in repo rhel7-csb: redhat-internal-*

The log ends abruptly after that.

So I tried passing --disablerepo=rhel7-csb and sure enough, I could
attempt the reinstall operation 100 times without encountering a
single crash.

So it looks like dnf is somehow choking on that specific repository.
The repository configuration looks like:

  $ cat /etc/yum.repos.d/rhel7-csb.repo
  [rhel7-csb]
  name=RHEL7 CSB packages
  baseurl=http://...
  enabled=1
  gpgcheck=0
  includepkgs=redhat-internal-*

That's mostly very standard stuff, except for that last line which
I added to make sure only a small subset of the packages included in
the repository would actually be considered during eg. updates, since
it also contains a bunch of packages clashing with the ones shipped
in Fedora.

If I remove that line, I can once again attempt 100 downgrades
without dnf crashing even once! But I need the line to be there, for
the reasons explained above, so I can't just get rid of it; and I've
had it there for a long time, yet somehow it only caused trouble for
dnf three months ago and today, which is pretty weird.

I'll gladly provide more information such as a stack trace or any
information dnf might be caching locally about the repository, but
I need exact instructions and I'm afraid changes to the remote side
of the repository, which of course I have no control over, might
once again cause the issue to disappear in a few days, just like it
happened last time.

Comment 7 Andrea Bolognani 2019-05-24 10:51:35 UTC
By the way, I'm using Fedora 30 now.

  $ rpm -q dnf libdnf libsolv
  dnf-4.2.5-1.fc30.noarch
  libdnf-0.31.0-3.fc30.x86_64
  libsolv-0.7.4-2.fc30.x86_64

Comment 8 Andrea Bolognani 2019-05-27 07:36:06 UTC
(In reply to Andrea Bolognani from comment #6)
> I'll gladly provide more information such as a stack trace or any
> information dnf might be caching locally about the repository, but
> I need exact instructions and I'm afraid changes to the remote side
> of the repository, which of course I have no control over, might
> once again cause the issue to disappear in a few days, just like it
> happened last time.

And that's exactly what happened: without any change having been
made to the system (outside of DNF automatically refreshing
repository information), I can no longer reproduce the issue :(

Fearing this might happen, on Friday I took a snapshot of everything
that looked like it might be relevant:

  /etc/dnf/
  /etc/yum.repos.d/
  /var/cache/dnf/
  /var/lib/dnf/
  /var/lib/rpm/

I'll take another snapshot right now so that we have a point of
comparison; hopefully there's something in there that we can use to
track down the issue.

Comment 9 Andrea Bolognani 2019-06-27 14:42:48 UTC
FYI it happened again today... Same error messages, same workaround :)

Comment 10 Jaroslav Mracek 2019-06-27 15:27:21 UTC
Please could you provide information about your hardware? Also please could you look at usage of your system during the fail?

Thanks a lot

Comment 11 Jaroslav Mracek 2019-06-27 15:40:15 UTC
Also are you able to reproduce the issue without playground plugin or without all plugins? (--disableplugin=*)

Please could you run the failing command wit valgrind?

Comment 12 Andrea Bolognani 2019-06-27 15:46:52 UTC
(In reply to Jaroslav Mracek from comment #10)
> Please could you provide information about your hardware?

It's a ThinkPad T480s with i7-8650U and 16 GiB of RAM.

> Also please could
> you look at usage of your system during the fail?

Do you mean CPU usage, memory usage, both?

> Also are you able to reproduce the issue without playground plugin or
> without all plugins? (--disableplugin=*)
> 
> Please could you run the failing command wit valgrind?

Can you give me the exact command line that'll convince dnf to run
under Valgrind and report information you'll find useful to track
down the issue? Anything else you'd like me to collect?


As mentioned earlier I have already worked around the issue, but
I'll try to downgrade libvirt and upgrade it again, see if it shows
up; and even if that doesn't work, at least I'll know what to do
next time! ;)

Comment 13 Jaroslav Mracek 2019-06-27 17:15:46 UTC
Ok, we have same hardware that has a plenty of resources.

Please find an argument that fails with dnf command like ./libvirt-*5.1.0*.rpm, then:

sudo valgrind dnf downgrade ./libvirt-*5.1.0*.rpm --assumeno


Also please could you provide a coredump?
su -c 'echo "core.%e.%p" > /proc/sys/kernel/core_pattern'
Then run dnf command that fails terribly and it will create file like core.dnf.9299 - but according to your output the coredump could be not created.

Also output from rpm -qa could help - please provide it as attachment


What about list of repositories? Please could you provide all .repo files in /etc/yum.repos.d/?

It is really depressing that I am unable to reproduce your problem. And you are the only one with the issue.

Comment 14 Andrea Bolognani 2019-06-28 10:39:21 UTC
Created attachment 1585526 [details]
valgrind output

Comment 15 Andrea Bolognani 2019-06-28 10:39:52 UTC
Created attachment 1585527 [details]
gdb output

Comment 16 Andrea Bolognani 2019-06-28 10:40:15 UTC
Created attachment 1585528 [details]
rpm -qa output

Comment 17 Andrea Bolognani 2019-06-28 10:40:40 UTC
Created attachment 1585529 [details]
yum repos

Comment 18 Andrea Bolognani 2019-06-28 10:41:59 UTC
(In reply to Jaroslav Mracek from comment #13)
> Please find an argument that fails with dnf command like
> ./libvirt-*5.1.0*.rpm, then:
> 
> sudo valgrind dnf downgrade ./libvirt-*5.1.0*.rpm --assumeno

Done (attachment 1585526 [details]).

I also ran the command under gdb using some trickery, and the stack
trace points to the same function being the culprit. I've included
that as well (attachment 1585527 [details]).

> Also please could you provide a coredump?
> su -c 'echo "core.%e.%p" > /proc/sys/kernel/core_pattern'
> Then run dnf command that fails terribly and it will create file like
> core.dnf.9299 - but according to your output the coredump could be not
> created.

I have created it but it's fairly large and I'm not sure how to share
it with you... I'll keep it around locally for the time being, maybe
the other information is enough to track down the issue; if not,
we'll have to figure out a way for you to get hold of it.

> Also output from rpm -qa could help - please provide it as attachment

Done (attachment 1585528 [details]).

> What about list of repositories? Please could you provide all .repo files in
> /etc/yum.repos.d/?

Done (attachment 1585529 [details]).

> It is really depressing that I am unable to reproduce your problem. And you
> are the only one with the issue.

Let's just be happy that there aren't many people affected by it :)

And thanks for trying to help out. I'm sure we'll get to the bottom
of it eventually.

Comment 20 Jaroslav Mracek 2019-06-28 14:35:20 UTC
I have additional question

Please could you also investigate your coredump more. What you provide from gdp is exactly what I need, with one minor difference. It reports
: Missing separate debuginfos, use: dnf debuginfo-install brotli-1.0.7-3.fc30.x86_64 bzip2-libs-1.0.6-29.fc30.x86_64 cyrus-sasl-lib-2.1.27-0.6rc7.fc30.x86_64 elfutils-libelf-0.176-3.fc30.x86_64 elfutils-libs-0.176-3.fc30.x86_64 expat-2.2.6-2.fc30.x86_64 file-libs-5.36-3.fc30.x86_64 glib2-2.60.4-1.fc30.x86_64 glibc-2.29-15.fc30.x86_64 gpgme-1.12.0-1.fc30.x86_64 ima-evm-utils-1.1-5.fc30.x86_64 json-c-0.13.1-4.fc30.x86_64 keyutils-libs-1.6-2.fc30.x86_64 krb5-libs-1.17-14.fc30.x86_64 libacl-2.2.53-3.fc30.x86_64 libassuan-2.5.2-2.fc30.x86_64 libcap-2.26-5.fc30.x86_64 libcurl-7.64.0-7.fc30.x86_64 libdb-5.3.28-37.fc30.x86_64 libffi-3.1-19.fc30.x86_64 libgcc-9.1.1-1.fc30.x86_64 libgcrypt-1.8.4-3.fc30.x86_64 libgpg-error-1.33-2.fc30.x86_64 libidn2-2.2.0-1.fc30.x86_64 libmodulemd1-1.8.11-2.fc30.x86_64 libmount-2.33.2-1.fc30.x86_64 libnghttp2-1.38.0-1.fc30.x86_64 librepo-1.10.2-2.fc30.x86_64 libselinux-2.9-1.fc30.x86_64 libsmartcols-2.33.2-1.fc30.x86_64 libstdc++-9.1.1-1.fc30.x86_64 libunistring-0.9.10-5.fc30.x86_64 libuuid-2.33.2-1.fc30.x86_64 libxcrypt-4.4.6-1.fc30.x86_64 libxml2-2.9.9-2.fc30.x86_64 libyaml-0.2.1-5.fc30.x86_64 libzstd-1.4.0-1.fc30.x86_64 lz4-libs-1.8.3-2.fc30.x86_64 openldap-2.4.47-1.fc30.x86_64 openssl-libs-1.1.1c-2.fc30.x86_64 pcre-8.43-2.fc30.x86_64 pcre2-10.33-2.fc30.x86_64 popt-1.16-17.fc30.x86_64 python3-libcomps-0.1.11-1.fc30.x86_64 python3-rpm-4.14.2.1-4.fc30.1.x86_64 python3-systemd-234-8.fc30.x86_64 rpm-build-libs-4.14.2.1-4.fc30.1.x86_64 rpm-libs-4.14.2.1-4.fc30.1.x86_64 rpm-sign-libs-4.14.2.1-4.fc30.1.x86_64 sqlite-libs-3.26.0-5.fc30.x86_64 sssd-client-2.2.0-1.fc30.x86_64 xz-libs-5.2.4-5.fc30.x86_64 zchunk-libs-1.1.2-2.fc30.x86_64 zlib-1.2.11-15.fc30.x86_64

Please could you run above dnf debuginfo-install command command from report and open gdp again and probably it will complain again with shorter set of packages, then again install them please. Without a proper versions of debuginfo the output could be completely misleading.

 please could you reproduce it with latest dnf from our nightly repository? To enable it just run "dnf copr enable rpmsoftwaremanagement/dnf-nightly" and then "dnf upgrade dnf".

From valgrind I can see one invalid read that wasn't in my report.

Thanks a lot

Comment 21 Jaroslav Mracek 2019-06-28 15:49:12 UTC
Ok, I have a clue. I am still unable to reproduce the issue, but I am able to reproduce a problem in valgrind. It requires option includepkgs=something in .repo file like you have. Without information that you provided, I will be completely lost.

Comment 22 Jaroslav Mracek 2019-06-28 16:27:51 UTC
I am able to reproduce the issue, finally. Your rhel7-csb repo is a key to reproduce it.

Comment 23 Jaroslav Mracek 2019-06-28 17:37:52 UTC
Here is the patch https://github.com/rpm-software-management/libdnf/pull/756. Just for QE - it fixes the crash in allocation of memory.

Comment 24 Fedora Update System 2019-08-14 12:11:18 UTC
FEDORA-2019-40235845dc has been submitted as an update to Fedora 30. https://bodhi.fedoraproject.org/updates/FEDORA-2019-40235845dc

Comment 25 Fedora Update System 2019-08-16 19:50:24 UTC
dnf-4.2.8-1.fc30, dnf-plugins-core-4.0.9-1.fc30, dnf-plugins-extras-4.0.5-1.fc30, libdnf-0.35.2-1.fc30 has been pushed to the Fedora 30 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2019-40235845dc

Comment 26 Fedora Update System 2019-08-26 13:26:32 UTC
FEDORA-2019-40235845dc has been submitted as an update to Fedora 30. https://bodhi.fedoraproject.org/updates/FEDORA-2019-40235845dc

Comment 27 Fedora Update System 2019-08-27 18:21:29 UTC
dnf-4.2.8-1.fc30, dnf-plugins-core-4.0.9-1.fc30, dnf-plugins-extras-4.0.5-1.fc30, libdnf-0.35.2-1.fc30, swid-tools-0.8.6-1.fc30 has been pushed to the Fedora 30 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2019-40235845dc

Comment 28 Fedora Update System 2019-08-29 05:05:27 UTC
FEDORA-2019-40235845dc has been submitted as an update to Fedora 30. https://bodhi.fedoraproject.org/updates/FEDORA-2019-40235845dc

Comment 29 Fedora Update System 2019-08-30 00:03:41 UTC
dnf-4.2.8-1.fc30, dnf-plugins-core-4.0.9-1.fc30, libdnf-0.35.2-1.fc30, swid-tools-0.8.6-1.fc30 has been pushed to the Fedora 30 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2019-40235845dc

Comment 30 Fedora Update System 2019-09-06 12:31:05 UTC
dnf-4.2.8-1.fc30, dnf-plugins-core-4.0.9-1.fc30, libdnf-0.35.2-1.fc30, swid-tools-0.8.6-1.fc30 has been pushed to the Fedora 30 stable repository. If problems still persist, please make note of it in this bug report.


Note You need to log in before you can comment on or make changes to this bug.