Bug 1498207 - DNF crash during upgrade installation F26 -> F27
DNF crash during upgrade installation F26 -> F27
Status: CLOSED ERRATA
Product: Fedora
Classification: Fedora
Component: libdnf (Show other bugs)
26
Unspecified Unspecified
unspecified Severity high
: ---
: ---
Assigned To: Jaroslav Mracek
Fedora Extras Quality Assurance
AcceptedPreviousRelease
: Reopened, Triaged
: 1498875 1499101 1499131 1499514 1500329 1500530 1500940 (view as bug list)
Depends On:
Blocks: F27FinalBlocker
  Show dependency treegraph
 
Reported: 2017-10-03 14:12 EDT by Couret Charles-Antoine
Modified: 2017-11-13 20:57 EST (History)
43 users (show)

See Also:
Fixed In Version: libdnf-0.11.0-1.fc26 libdnf-0.11.0-1.fc27 libdnf-0.11.1-1.fc26
Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2017-10-20 12:36:59 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
Journalctl output (82.85 KB, text/plain)
2017-10-03 14:12 EDT, Couret Charles-Antoine
no flags Details
Journalctl output before dnf process (133.67 KB, text/plain)
2017-10-04 07:17 EDT, Couret Charles-Antoine
no flags Details
journalctl output for comment 7 (199.18 KB, text/plain)
2017-10-06 10:39 EDT, Zamir SUN
no flags Details
journalctl -b-17 > t1.txt (217.29 KB, text/plain)
2017-10-06 13:28 EDT, kartochka22
no flags Details
journalctl -b-7 > t2.txt (136.87 KB, text/plain)
2017-10-06 13:41 EDT, kartochka22
no flags Details
journalctl -b -1 after failed upgrade (228.43 KB, text/x-vhdl)
2017-10-08 08:26 EDT, Gilles Duboscq
no flags Details

  None (edit)
Description Couret Charles-Antoine 2017-10-03 14:12:31 EDT
Created attachment 1333868 [details]
Journalctl output

I am trying to upgrade my F26 to F27.
Downloading part, no problem. I rebooted my system to install.

Then, dnf crashed at the beginning of this process and reboot to F26 automatically.

Packages release (F26 is up to date currently):
dnf.noarch                                                            2.7.2-1.fc26                                                    @updates-testing
dnf-conf.noarch                                                       2.7.2-1.fc26                                                    @updates-testing
dnf-plugins-core.noarch                                               2.1.4-1.fc26                                                    @updates-testing
dnf-yum.noarch                                                        2.7.2-1.fc26                                                    @updates-testing
python3-dnf-plugin-system-upgrade.noarch                                       2.0.3-1.fc26                                           @updates-testing

F26 is working fine, but no way to get F27 over dnf upgrade for the moment.
Comment 1 Igor Gnatenko 2017-10-04 01:40:37 EDT
That seems like bug in python... I don't see any other libraries in backtrace.
Comment 2 Zbigniew Jędrzejewski-Szmek 2017-10-04 04:15:45 EDT
Memory corruption... this will be very hard to debug. Can you maybe attach the full log from that boot with failed upgrade? It's possible that the earlier messages will tell us what went wrong.
Comment 3 Couret Charles-Antoine 2017-10-04 07:17 EDT
Created attachment 1334194 [details]
Journalctl output before dnf process
Comment 4 Zbigniew Jędrzejewski-Szmek 2017-10-06 03:15:51 EDT
*** Bug 1499101 has been marked as a duplicate of this bug. ***
Comment 5 Zbigniew Jędrzejewski-Szmek 2017-10-06 03:18:31 EDT
"Start the upgrade process. This may take some time." (translation) and then the crash. Nothing interesting in the logs.
Comment 6 Zbigniew Jędrzejewski-Szmek 2017-10-06 03:42:02 EDT
*** Bug 1499131 has been marked as a duplicate of this bug. ***
Comment 7 Zamir SUN 2017-10-06 10:39 EDT
Created attachment 1335350 [details]
journalctl output for comment 7

I also failed to upgrade with similar output. With the journalctl attached.
Comment 8 Charalampos Stratakis 2017-10-06 11:02:59 EDT
I am currently building Python 3.6.3 for F26:

https://koji.fedoraproject.org/koji/taskinfo?taskID=22290082

Please try the build as soon as it is ready to see, if that fixes the issue.

Also maaaybe relevant (but it will require some more digging): https://bugs.python.org/issue31532
Comment 9 Zamir SUN 2017-10-06 12:22:29 EDT
(In reply to Charalampos Stratakis from comment #8)
> I am currently building Python 3.6.3 for F26:
> 
> https://koji.fedoraproject.org/koji/taskinfo?taskID=22290082
> 
> Please try the build as soon as it is ready to see, if that fixes the issue.
> 
> Also maaaybe relevant (but it will require some more digging):
> https://bugs.python.org/issue31532

Unfortunately, this build did not solve my issue.
Comment 10 kartochka22 2017-10-06 13:00:03 EDT
Same here. First attempt was upgrade from 26-testing, second one I revert main system using dnf dist-sync w/o testing repo, it crashed again.

Beside, no more attempts as dnf go full reload 5gb updates and I pissed off. Why it cannot keep local cache ? Same was in 24-26 transitions and maybe previous. That insane. I wait for at least RC1 to try again.
Comment 11 kartochka22 2017-10-06 13:28 EDT
Created attachment 1335436 [details]
journalctl -b-17 > t1.txt

my log of first attempt to sys-upgrade with dnf crash
Comment 12 Charalampos Stratakis 2017-10-06 13:39:30 EDT
Unfortunately I was not able to reproduce the issue on a VM.

I've made an experimental scratch build for people to test to see if that fixes the issues: https://koji.fedoraproject.org/koji/taskinfo?taskID=22293517

(Note that the scratch build will be garbage collected soon so if required I will provide a copr repo as well.)
Comment 13 kartochka22 2017-10-06 13:41 EDT
Created attachment 1335437 [details]
journalctl -b-7 > t2.txt

Log of last attempt to upgrade, this one from f26 w/o testing, removing steam, nvidia akmod crap, etc. Looks like no crash, but that line

окт 03 21:17:40 fast25 dnf[789]: Ошибка: Не удается синхронизировать кэш для репозитория «updates»

"cannot sync cache for updated repo"

Say it looks like known bug listed in Wiki, i had to "dnf makecache" before "dnf system-upgrade reboot", but I stopped here as it will download 5gb third time and I tired. So, it seems that f26 testing only crashing.
Comment 14 Hein-Pieter van Braam 2017-10-06 13:47:46 EDT
I'm trying the scratch build now.
Comment 15 Hein-Pieter van Braam 2017-10-06 13:50:04 EDT
After installing the scratch build dnf can no longer successfully calculate the upgrade.

  file /usr/lib64/python3.6/collections/__init__.py from install of python3-libs-3.6.2-13.fc27.x86_64 conflicts with file from package system-python-libs-3.6.3-2.fc26.x86_64

etc. I can't test with these RPMs it seems.
Comment 16 Charalampos Stratakis 2017-10-06 15:43:46 EDT
(In reply to Hein-Pieter van Braam from comment #15)
> After installing the scratch build dnf can no longer successfully calculate
> the upgrade.
> 
>   file /usr/lib64/python3.6/collections/__init__.py from install of
> python3-libs-3.6.2-13.fc27.x86_64 conflicts with file from package
> system-python-libs-3.6.3-2.fc26.x86_64
> 
> etc. I can't test with these RPMs it seems.

Please install first https://koji.fedoraproject.org/koji/taskinfo?taskID=22290082 which will land in updates-testing soon and then try to install the scratch build on top.
Comment 17 Charalampos Stratakis 2017-10-06 15:48:58 EDT
(In reply to Hein-Pieter van Braam from comment #15)
> After installing the scratch build dnf can no longer successfully calculate
> the upgrade.
> 
>   file /usr/lib64/python3.6/collections/__init__.py from install of
> python3-libs-3.6.2-13.fc27.x86_64 conflicts with file from package
> system-python-libs-3.6.3-2.fc26.x86_64
> 
> etc. I can't test with these RPMs it seems.

That might also be a packaging bug in the python3 rpm for the F27 branch (aka not correctly obsoleting system-python-libs which was removed).
Comment 18 Hein-Pieter van Braam 2017-10-06 15:56:49 EDT
Yeah, seems to be the case. The RPMs installed cleanly this time, but the upgrade still failed:

  file /usr/lib64/python3.6/collections/__init__.py from install of python3-libs-3.6.2-13.fc27.x86_64 conflicts with file from package system-python-libs-3.6.3-2.fc26.x86_64

I'll try upgrading system-python from F27 maybe?
Comment 19 Hein-Pieter van Braam 2017-10-06 16:04:24 EDT
OK, 'pre-upgrading' python to the f27 packages leads to the exact same result. So the bug is at least still present in those packages also.

As for the F27 packages failing to obsolete system-python, this conflicting files failure does not happen on a 'stock' F26 box (right after a distro-sync)
Comment 20 Christian Stadelmann 2017-10-06 16:15:40 EDT
Same issue here (original bug report).

With your 3.6.3-2 build installed, `dnf system-upgrade --releasever=27 download" fails for me too.
It fails during transaction check (i.e. after download but before reboot) with hundreds of collissions between python3-libs-3.6.2-13.fc27.x86_64 and system-python-libs-3.6.3-2.fc26.x86_64. This includes many __pycache__ folders under /usr/lib64/python3.6 subfolders. It does not matter whether I upgraded python from 3.6.2-7.fc26 -> 3.6.3-2.fc26 or 3.6.2-7.fc26 -> 3.6.3-1.fc26 -> 3.6.3-2.fc26.
Comment 21 Gilles Duboscq 2017-10-08 08:26 EDT
Created attachment 1335855 [details]
journalctl -b -1 after failed upgrade

I can see that issue as well. Attaching the logs of the failed upgrade.
Comment 22 Kamil Páral 2017-10-09 04:00:04 EDT
*** Bug 1499514 has been marked as a duplicate of this bug. ***
Comment 23 Charalampos Stratakis 2017-10-09 06:14:26 EDT
(In reply to Hein-Pieter van Braam from comment #19)

> As for the F27 packages failing to obsolete system-python, this conflicting
> files failure does not happen on a 'stock' F26 box (right after a
> distro-sync)

This is really confusing. What is the %{version}-%{release} of python3 after a distro-sync?
Comment 24 Hein-Pieter van Braam 2017-10-09 06:19:06 EDT
I'm sorry if I was not clear. These are the python3 versions I have now:

$ rpm -qa python3 system-python
system-python-3.6.2-7.fc26.x86_64
python3-3.6.2-7.fc26.x86_64

With these packages doing the dnf system upgrade computes the dependencies correctly and it appears all deprecations work as expected. With the scratch builds installed system upgrade fails with the conflicting files. 

Does that clarify what I intended to say?
Comment 25 Charalampos Stratakis 2017-10-09 06:52:11 EDT
(In reply to Hein-Pieter van Braam from comment #24)
> I'm sorry if I was not clear. These are the python3 versions I have now:
> 
> $ rpm -qa python3 system-python
> system-python-3.6.2-7.fc26.x86_64
> python3-3.6.2-7.fc26.x86_64
> 
> With these packages doing the dnf system upgrade computes the dependencies
> correctly and it appears all deprecations work as expected. With the scratch
> builds installed system upgrade fails with the conflicting files. 
> 
> Does that clarify what I intended to say?

Yep that helps a lot actually. Will investigate further to see what might be causing that.

On another note, you did not get the memory corruption issue?
Comment 26 Charalampos Stratakis 2017-10-09 06:55:06 EDT
Well yes it makes sense as the NVR is now bigger for F26 if the scratch builds are installed which doesn't make it possible to obsolete it.
Comment 27 Charalampos Stratakis 2017-10-09 07:02:58 EDT
My understanding from the comments is that https://bugs.python.org/issue31532 seems to fix the issue as people only get transaction error issues after installing the scratch builds (so we are past the invalid pointer issue).

Then the update will fail as the NVR of the F27 build is lower from that of the scratch build for F26 which is expected at that point.

Will push an update for all the branches.
Comment 28 Hein-Pieter van Braam 2017-10-09 10:01:35 EDT
Charalampos, I did also get the memory corruption issue. I filed #1499131 which was a duplicate of this bug.
Comment 29 Fedora Update System 2017-10-09 10:18:48 EDT
python3-3.6.3-2.fc26 has been submitted as an update to Fedora 26. https://bodhi.fedoraproject.org/updates/FEDORA-2017-8e91b32f31
Comment 30 Fedora Update System 2017-10-09 10:19:29 EDT
python3-3.5.4-2.fc25 has been submitted as an update to Fedora 25. https://bodhi.fedoraproject.org/updates/FEDORA-2017-48f0da57ca
Comment 31 Charalampos Stratakis 2017-10-09 10:29:44 EDT
Updates have been submitted for all the active fedora branches.

(Also the F27 is here since the fedora update system did not add it to the bugzilla: https://bodhi.fedoraproject.org/updates/FEDORA-2017-a20cf0e9b8 )

Do note that in order to properly test the upgrade path, as soon as the F27 update hit the updates-testing repos, you will need to issue the 'dnf system upgrade' command like this:

sudo dnf system-upgrade --enablerepo=updates-testing download --refresh --releasever=27
Comment 32 Hein-Pieter van Braam 2017-10-09 10:35:28 EDT
OK, I'll test it as soon as it hits F27 updates-testing
Comment 33 Jaroslav Mracek 2017-10-09 11:31:44 EDT
I was able to reproduce the similar problem. In libdnf/nevra-py.c there is a function has_just_name(). If I call it 16x and it returns Py_True, during closing it can create an error. 

*** Error in `/usr/bin/python3': free(): invalid pointer: 0x00007f7414001bc0 ***
======= Backtrace: =========
/lib64/libc.so.6(+0x7c8dc)[0x7f7412e418dc]
/lib64/libc.so.6(+0x87789)[0x7f7412e4c789]
/lib64/libc.so.6(cfree+0x16e)[0x7f7412e520ee]
/lib64/libpython3.6m.so.1.0(+0x109a48)[0x7f7413bdba48]
/lib64/libpython3.6m.so.1.0(PyInterpreterState_Clear+0x10d)[0x7f7413cab7bd]
/lib64/libpython3.6m.so.1.0(Py_FinalizeEx+0x92)[0x7f7413cf4f42]
/lib64/libpython3.6m.so.1.0(Py_Main+0x392)[0x7f7413cf6232]
/usr/bin/python3(main+0x1f5)[0x560238732cf5]
/lib64/libc.so.6(__libc_start_main+0xea)[0x7f7412de550a]
/usr/bin/python3(_start+0x2a)[0x560238732e6a]
======= Memory map: ========


Error cannot be reachable with only 15x repeat. If the function returns PyLong_FromLong(1) or Py_False, no problem appears.

My python version: python3-3.6.1-8.fc26.x86_64

I can provide dnf and libdnf packages that can simulate the problem.
Comment 34 Christian Stadelmann 2017-10-09 11:34:55 EDT
(In reply to Jaroslav Mracek from comment #33)
> My python version: python3-3.6.1-8.fc26.x86_64
> 
> I can provide dnf and libdnf packages that can simulate the problem.

Can you still reproduce this with python3-3.6.3-2.fc26?
Comment 35 Jaroslav Mracek 2017-10-09 11:40:22 EDT
Unfortunately the issue from Comment 33 can be also reproduced with python3-3.6.3-2.fc26.
Comment 36 Jaroslav Mracek 2017-10-09 12:08:09 EDT
Here is the reproducer. I created a copr repo jmracek/python-problem with libdnf(2 version) and dnf. To reproduce the problem install libdnf-0.10.1-1.git.1566.45ded29.fc2* and dnf-2.7.2-1.git.7957.08323fe.fc2*
and run following script.
```
#!/usr/bin/python3

import dnf

def get_base():
    base = dnf.Base()
    base.conf.assumeyes = True
    base.read_all_repos()
    base.fill_sack(load_system_repo='auto')
    base.install("acpi")
    base.install("acpi")
    base.install("acpi")
    base.install("acpi")
    base.install("acpi")
    base.install("acpi")
    base.install("acpi")
    base.install("acpi")
    base.install("acpi")
    base.install("acpi")
    base.install("acpi")
    base.install("acpi")
    base.install("acpi")
    base.install("acpi")
    base.install("acpi")
    base.install("acpi")
    base.close()

a = 1

while a:
    print(a)
    get_base()
    a -= 1

```

If you will install libdnf-0.10.1-1.git.1567.2d692dd.fc2*, the problem disappear. Also removal of single base.install("acpi") step restore functionality.

Differences between two libdnf versions:

libdnf-0.10.1-1.git.1566.45ded29.fc2* Can return Py_True
```
has_just_name(_NevraObject *self, PyObject *unused)
{
    return hy_nevra_has_just_name(self->nevra) ? Py_True : Py_False;
}
```

libdnf-0.10.1-1.git.1567.2d692dd.fc2* can return PyLong_FromLong(1)
```
has_just_name(_NevraObject *self, PyObject *unused)
{
    return hy_nevra_has_just_name(self->nevra) ? PyLong_FromLong(1) : Py_False;
}
```
Comment 37 Jaroslav Mracek 2017-10-09 12:08:42 EDT
Here is the reproducer. I created a copr repo jmracek/python-problem with libdnf(2 version) and dnf. To reproduce the problem install libdnf-0.10.1-1.git.1566.45ded29.fc2* and dnf-2.7.2-1.git.7957.08323fe.fc2*
and run following script.
```
#!/usr/bin/python3

import dnf

def get_base():
    base = dnf.Base()
    base.conf.assumeyes = True
    base.read_all_repos()
    base.fill_sack(load_system_repo='auto')
    base.install("acpi")
    base.install("acpi")
    base.install("acpi")
    base.install("acpi")
    base.install("acpi")
    base.install("acpi")
    base.install("acpi")
    base.install("acpi")
    base.install("acpi")
    base.install("acpi")
    base.install("acpi")
    base.install("acpi")
    base.install("acpi")
    base.install("acpi")
    base.install("acpi")
    base.install("acpi")
    base.close()

a = 1

while a:
    print(a)
    get_base()
    a -= 1

```

If you will install libdnf-0.10.1-1.git.1567.2d692dd.fc2*, the problem disappear. Also removal of single base.install("acpi") step restores functionality.

Differences between two libdnf versions:

libdnf-0.10.1-1.git.1566.45ded29.fc2* Can return Py_True
```
has_just_name(_NevraObject *self, PyObject *unused)
{
    return hy_nevra_has_just_name(self->nevra) ? Py_True : Py_False;
}
```

libdnf-0.10.1-1.git.1567.2d692dd.fc2* can return PyLong_FromLong(1)
```
has_just_name(_NevraObject *self, PyObject *unused)
{
    return hy_nevra_has_just_name(self->nevra) ? PyLong_FromLong(1) : Py_False;
}
```
Comment 38 Jaroslav Mracek 2017-10-09 12:08:49 EDT
Here is the reproducer. I created a copr repo jmracek/python-problem with libdnf(2 version) and dnf. To reproduce the problem install libdnf-0.10.1-1.git.1566.45ded29.fc2* and dnf-2.7.2-1.git.7957.08323fe.fc2*
and run following script.
```
#!/usr/bin/python3

import dnf

def get_base():
    base = dnf.Base()
    base.conf.assumeyes = True
    base.read_all_repos()
    base.fill_sack(load_system_repo='auto')
    base.install("acpi")
    base.install("acpi")
    base.install("acpi")
    base.install("acpi")
    base.install("acpi")
    base.install("acpi")
    base.install("acpi")
    base.install("acpi")
    base.install("acpi")
    base.install("acpi")
    base.install("acpi")
    base.install("acpi")
    base.install("acpi")
    base.install("acpi")
    base.install("acpi")
    base.install("acpi")
    base.close()

a = 1

while a:
    print(a)
    get_base()
    a -= 1

```

If you will install libdnf-0.10.1-1.git.1567.2d692dd.fc2*, the problem disappear. Also removal of single base.install("acpi") step restores functionality.

Differences between two libdnf versions:

libdnf-0.10.1-1.git.1566.45ded29.fc2* Can return Py_True
```
has_just_name(_NevraObject *self, PyObject *unused)
{
    return hy_nevra_has_just_name(self->nevra) ? Py_True : Py_False;
}
```

libdnf-0.10.1-1.git.1567.2d692dd.fc2* can return PyLong_FromLong(1)
```
has_just_name(_NevraObject *self, PyObject *unused)
{
    return hy_nevra_has_just_name(self->nevra) ? PyLong_FromLong(1) : Py_False;
}
```
Comment 39 Charalampos Stratakis 2017-10-09 12:37:27 EDT
Changing it to the appropriate component.
Comment 40 Kamil Páral 2017-10-09 12:40:54 EDT
Discussed at blocker review meeting [1]:

AcceptedPreviousRelease - this bug violates the criterion: "For each one of the release-blocking package sets, it must be possible to successfully complete a direct upgrade from fully updated installations of the last two stable Fedora releases with that package set installed."

[1] https://meetbot-raw.fedoraproject.org/fedora-blocker-review/2017-10-09/
Comment 41 Jaroslav Mracek 2017-10-09 13:12:01 EDT
I am really sorry but this is not a problem of libdnf but Python. Or is there any limitation why Py_True cannot be return more that 15x? 
If it is like that it means that any python binding in any application that returns Py_True must be changed to return something else because Py_True do not allocate memory correctly and is not safe anymore?


Changing it to the appropriate component.

PS: Py_True is used in libdnf about 17x. It means that we have here about 17x hidden bombs and they are ticking.
Comment 42 Jaroslav Mracek 2017-10-09 14:29:48 EDT
Here is more simple reproducer for dnf libdnf versions in updates:
dnf-2.7.3-1.fc26.noarch
libdnf-0.10.1-1.fc26.x86_64
python3-3.6.3-2.fc26.x86_64

```
#!/usr/bin/python3

import hawkey

nevra = hawkey.NEVRA(name='names', epoch=0)
nevra.epoch = None
nevra.has_just_name()
nevra.has_just_name()
nevra.has_just_name()
nevra.has_just_name()
nevra.has_just_name()
nevra.has_just_name()
nevra.has_just_name()
nevra.has_just_name()
nevra.has_just_name()
nevra.has_just_name()
nevra.has_just_name()
nevra.has_just_name()
nevra.has_just_name()
nevra.has_just_name()
nevra.has_just_name()
nevra.has_just_name()
```

Notes:
With Python2 no problem.
If function returns False, no problem.
Even fails if:
``
has_just_name(_NevraObject *self, PyObject *unused)
{
    return Py_True;
}
```
Comment 43 Steve Barnsley 2017-10-09 15:08:53 EDT
Tried upgrade from 26 to 27 and had the same problem as the OP.
Noticed that dnf would be downgraded as part of the upgrade.
Made second attempt - hit escape to see details during upgrade reboot, saw a bunch of dnf and python errors flash past.
Remembered that there was a recent upgrade to dnf on 26 & figured that upgrade is not compatible with the older dnf on 27.

Did:
dnf --refresh upgrade
dnf downgrade dnf (this also downgraded some python stuff - don't recall details)

Tried upgrade again and it is working - currently at 3K of 8K operations.

Figured this workaround might help other get the upgrade done & might help with diagnosing the cause.
Comment 44 Steve Barnsley 2017-10-09 17:07:36 EDT
(In reply to Steve Barnsley from comment #43)
> Tried upgrade from 26 to 27 and had the same problem as the OP.
> Noticed that dnf would be downgraded as part of the upgrade.
> Made second attempt - hit escape to see details during upgrade reboot, saw a
> bunch of dnf and python errors flash past.
> Remembered that there was a recent upgrade to dnf on 26 & figured that
> upgrade is not compatible with the older dnf on 27.
> 
> Did:
> dnf --refresh upgrade
> dnf downgrade dnf (this also downgraded some python stuff - don't recall
> details)
> 
> Tried upgrade again and it is working - currently at 3K of 8K operations.
> 
> Figured this workaround might help other get the upgrade done & might help
> with diagnosing the cause.

Update: The upgrade to 27 worked fine after downgrading dnf on 26.  However, as soon as the upgrade was completed there were 316 updates available for 27 including updates to dnf and python ..... the above workaround may no longer be needed or may no longer work.
Comment 45 Felix Kaechele 2017-10-09 17:34:23 EDT
I can confirm that with first downgrading dnf the upgrade from F26 to F27 works.

I did, however, enable the updates-testing repo in the dnf system-upgrade download step, so I did not have any updates available after the upgrade, as they had already been installed when performing the upgrade.
Comment 46 Hein-Pieter van Braam 2017-10-10 02:51:53 EDT
I'll leave my box at F26 so I can test any potential fixes for this.
Comment 47 Jaroslav Mracek 2017-10-10 03:57:30 EDT
I am really sorry, the problem was really in libdnf. I create a patch that should solve the issue (https://github.com/rpm-software-management/libdnf/pull/337).
Comment 48 Petr Viktorin 2017-10-10 05:27:48 EDT
For anyone following along –

By convention, returning an object from a function means the caller receives a reference to that object. So, before returning a pre-existing object (such as Py_True), its reference count must be increased.

There are helpers for this:
https://docs.python.org/3/c-api/bool.html
Comment 49 Charalampos Stratakis 2017-10-10 05:30:56 EDT
Changing it back to libdnf again. Thanks for figuring out the root cause Jaroslav.
Comment 50 Fedora Update System 2017-10-10 06:52:23 EDT
libdnf-0.11.0-1.fc26 has been submitted as an update to Fedora 26. https://bodhi.fedoraproject.org/updates/FEDORA-2017-925d78e821
Comment 51 Fedora Update System 2017-10-10 06:52:52 EDT
libdnf-0.11.0-1.fc27 has been submitted as an update to Fedora 27. https://bodhi.fedoraproject.org/updates/FEDORA-2017-b1cb6f7720
Comment 52 Gwendal 2017-10-10 09:13:11 EDT
Thanks for the fix! Is there a way to test with dnf system-upgrade? I have tried this:

sudo dnf system-upgrade download --refresh --releasever=27 --enablerepo=updates-testing  

but it shows me that libdnf version 0.10.1-1.fc27 will be installed, and not libdnf-0.11.0-1.fc27
Comment 53 Kamil Páral 2017-10-10 09:28:13 EDT
Gwendal:

$ sudo dnf install bodhi-client
$ bodhi updates download --updateid FEDORA-2017-925d78e821
$ sudo dnf update ./*.rpm

Also, you want to install the F26 version on F26, obviously, if you want to test upgrading to F27.
Comment 54 LiZhenbo 2017-10-10 12:34:08 EDT
Thanks, Kamil

(In reply to Kamil Páral from comment #53)
> Gwendal:
> 
> $ sudo dnf install bodhi-client
> $ bodhi updates download --updateid FEDORA-2017-925d78e821
> $ sudo dnf update ./*.rpm
> 

What should I do after running these commands?
Running
$ sudo dnf system-upgrade download --refresh --releasever=27 --enablerepo=updates-testing  
after them would downgrade libdnf-0.11.0-1.fc26.x86_64 to  0.10.1-1.fc27
Comment 55 Charalampos Stratakis 2017-10-10 12:39:12 EDT
(In reply to LiZhenbo from comment #54)
> Thanks, Kamil
> 
> (In reply to Kamil Páral from comment #53)
> > Gwendal:
> > 
> > $ sudo dnf install bodhi-client
> > $ bodhi updates download --updateid FEDORA-2017-925d78e821
> > $ sudo dnf update ./*.rpm
> > 
> 
> What should I do after running these commands?
> Running
> $ sudo dnf system-upgrade download --refresh --releasever=27
> --enablerepo=updates-testing  
> after them would downgrade libdnf-0.11.0-1.fc26.x86_64 to  0.10.1-1.fc27

You should wait a bit.

The update hasn't yet reached the updates-testing repo.

https://bodhi.fedoraproject.org/updates/FEDORA-2017-b1cb6f7720
Comment 56 Adam Goode 2017-10-10 22:20:46 EDT
*** Bug 1500329 has been marked as a duplicate of this bug. ***
Comment 57 Adam Goode 2017-10-10 22:22:07 EDT
*** Bug 1498875 has been marked as a duplicate of this bug. ***
Comment 58 Fedora Update System 2017-10-10 22:52:43 EDT
python3-3.6.3-2.fc26 has been pushed to the Fedora 26 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2017-8e91b32f31
Comment 59 Fedora Update System 2017-10-10 22:54:09 EDT
libdnf-0.11.0-1.fc26 has been pushed to the Fedora 26 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2017-925d78e821
Comment 60 Fedora Update System 2017-10-11 00:20:48 EDT
python3-3.5.4-2.fc25 has been pushed to the Fedora 25 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2017-48f0da57ca
Comment 61 Fedora Update System 2017-10-11 02:26:19 EDT
python3-3.6.3-2.fc27 has been pushed to the Fedora 27 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2017-a20cf0e9b8
Comment 62 Fedora Update System 2017-10-11 02:29:07 EDT
libdnf-0.11.0-1.fc27 has been pushed to the Fedora 27 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2017-b1cb6f7720
Comment 63 hugotrip 2017-10-11 03:12:15 EDT
Coming from bug 1498875, I can confirm updating libdnf to libdnf-0.11.0-1.fc26 solved this bug for me
Comment 64 Igor Gnatenko 2017-10-11 03:40:35 EDT
*** Bug 1500530 has been marked as a duplicate of this bug. ***
Comment 65 Kamil Páral 2017-10-11 07:33:38 EDT
(In reply to LiZhenbo from comment #54)
> > $ sudo dnf install bodhi-client
> > $ bodhi updates download --updateid FEDORA-2017-925d78e821
> > $ sudo dnf update ./*.rpm
> > 
> 
> What should I do after running these commands?
> Running
> $ sudo dnf system-upgrade download --refresh --releasever=27
> --enablerepo=updates-testing  
> after them would downgrade libdnf-0.11.0-1.fc26.x86_64 to  0.10.1-1.fc27

Just run the upgrade, yes. The downgrade doesn't matter, because the upgrade will performed with libdnf-0.11.0-1.fc26, and that's all that matters.
Comment 66 LiZhenbo 2017-10-11 12:01:11 EDT
(In reply to Kamil Páral from comment #65)
> Just run the upgrade, yes. The downgrade doesn't matter, because the upgrade
> will performed with libdnf-0.11.0-1.fc26, and that's all that matters.

Thanks Kamil
My procedure

$ sudo dnf install bodhi-client
$ bodhi updates download --updateid FEDORA-2017-925d78e821
$ sudo dnf update ./*.rpm
$ sudo dnf system-upgrade download --refresh --releasever=27 --enablerepo=updates-testing

Now I've upgraded to f27 successfully
Comment 67 Couret Charles-Antoine 2017-10-11 16:35:38 EDT
This issue vanished on my computer but another happened: https://paste.fedoraproject.org/paste/ghuuAhAc8Drko~kE-5IHHg/raw
Comment 68 Christian Stadelmann 2017-10-11 17:27:51 EDT
(In reply to Couret Charles-Antoine from comment #67)
> This issue vanished on my computer but another happened:
> https://paste.fedoraproject.org/paste/ghuuAhAc8Drko~kE-5IHHg/raw

As a first guess that looks like bug #1492036 to me.
Comment 69 Igor Gnatenko 2017-10-12 03:27:22 EDT
*** Bug 1500940 has been marked as a duplicate of this bug. ***
Comment 70 Fedora Update System 2017-10-12 13:20:44 EDT
libdnf-0.11.0-1.fc26 has been pushed to the Fedora 26 stable repository. If problems still persist, please make note of it in this bug report.
Comment 71 Hein-Pieter van Braam 2017-10-13 20:33:24 EDT
I can confirm that the problem is solved, I just successfully upgraded to F27, thanks!
Comment 72 Stefan Becker 2017-10-15 16:46:43 EDT
I can also confirm that updating from latest F26 to F27 Beta works now.
Comment 73 Fedora Update System 2017-10-15 17:32:38 EDT
libdnf-0.11.0-1.fc27 has been pushed to the Fedora 27 stable repository. If problems still persist, please make note of it in this bug report.
Comment 74 Fedora Update System 2017-10-16 05:06:19 EDT
dnf-2.7.4-1.fc26 libdnf-0.11.1-1.fc26 has been submitted as an update to Fedora 26. https://bodhi.fedoraproject.org/updates/FEDORA-2017-9f04c2c90f
Comment 75 Fedora Update System 2017-10-16 05:07:04 EDT
dnf-2.7.4-1.fc27 libdnf-0.11.1-1.fc27 has been submitted as an update to Fedora 27. https://bodhi.fedoraproject.org/updates/FEDORA-2017-113a221a3d
Comment 76 Fedora Update System 2017-10-16 14:23:12 EDT
dnf-2.7.4-1.fc26, libdnf-0.11.1-1.fc26 has been pushed to the Fedora 26 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2017-9f04c2c90f
Comment 77 Fedora Update System 2017-10-16 22:24:44 EDT
dnf-2.7.4-1.fc27, libdnf-0.11.1-1.fc27 has been pushed to the Fedora 27 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2017-113a221a3d
Comment 78 Adam Williamson 2017-10-19 20:29:30 EDT
Two confirmations above that this works, so setting VERIFIED.

I am curious: what does the dnf-2.7.4-1.fc26 libdnf-0.11.1-1.fc26 update bring to the party that the libdnf-0.11.0-1.fc26 update, which has already been pushed stable, did not? Do we really need to push the newer update to fully resolve this bug?
Comment 79 Igor Gnatenko 2017-10-20 02:38:43 EDT
(In reply to Adam Williamson from comment #78)
> Two confirmations above that this works, so setting VERIFIED.
> 
> I am curious: what does the dnf-2.7.4-1.fc26 libdnf-0.11.1-1.fc26 update
> bring to the party that the libdnf-0.11.0-1.fc26 update, which has already
> been pushed stable, did not? Do we really need to push the newer update to
> fully resolve this bug?

Absolutely nothing, it is just Jaroslav who copied all bug references from upstream.
Comment 80 Adam Williamson 2017-10-20 12:36:59 EDT
OK, so I'll edit the update to remove the reference to this bug, and close the bug again. Thanks.
Comment 81 Fedora Update System 2017-10-25 19:11:39 EDT
dnf-2.7.4-1.fc26, libdnf-0.11.1-1.fc26 has been pushed to the Fedora 26 stable repository. If problems still persist, please make note of it in this bug report.
Comment 82 Pavel Alexeev 2017-11-06 09:12:53 EST
# dnf upgrade --refresh --enablerepo=updates-testing
# dnf distro-sync --allowerasing --enablerepo=updates-testing
# dnf system-upgrade download --refresh --releasever=27 --allowerasing --best

And it leads to mass errors like on stage of transaction testing:

  file /usr/lib64/python3.6/xml/parsers/__pycache__/expat.cpython-36.opt-1.pyc from install of python3-libs-3.6.2-13.fc27.x86_64 conflicts with file from package system-python-libs-3.6.3-2.fc26.x86_64
  file /usr/lib64/python3.6/xml/parsers/__pycache__/expat.cpython-36.opt-2.pyc from install of python3-libs-3.6.2-13.fc27.x86_64 conflicts with file from package system-python-libs-3.6.3-2.fc26.x86_64
  file /usr/lib64/python3.6/xml/parsers/__pycache__/expat.cpython-36.pyc from install of python3-libs-3.6.2-13.fc27.x86_64 conflicts with file from package system-python-libs-3.6.3-2.fc26.x86_64
  file /usr/lib64/python3.6/xml/sax/__pycache__/_exceptions.cpython-36.opt-1.pyc from install of python3-libs-3.6.2-13.fc27.x86_64 conflicts with file from package system-python-libs-3.6.3-2.fc26.x86_64
  file /usr/lib64/python3.6/xml/sax/__pycache__/_exceptions.cpython-36.opt-2.pyc from install of python3-libs-3.6.2-13.fc27.x86_64 conflicts with file from package system-python-libs-3.6.3-2.fc26.x86_64
  file /usr/lib64/python3.6/xml/sax/__pycache__/_exceptions.cpython-36.pyc from install of python3-libs-3.6.2-13.fc27.x86_64 conflicts with file from package system-python-libs-3.6.3-2.fc26.x86_64
  file /usr/lib64/python3.6/xml/sax/__pycache__/expatreader.cpython-36.opt-1.pyc from install of python3-libs-3.6.2-13.fc27.x86_64 conflicts with file from package system-python-libs-3.6.3-2.fc26.x86_64
  file /usr/lib64/python3.6/xml/sax/__pycache__/expatreader.cpython-36.opt-2.pyc from install of python3-libs-3.6.2-13.fc27.x86_64 conflicts with file from package system-python-libs-3.6.3-2.fc26.x86_64
  file /usr/lib64/python3.6/xml/sax/__pycache__/expatreader.cpython-36.pyc from install of python3-libs-3.6.2-13.fc27.x86_64 conflicts with file from package system-python-libs-3.6.3-2.fc26.x86_64
  file /usr/lib64/python3.6/xml/sax/__pycache__/handler.cpython-36.opt-1.pyc from install of python3-libs-3.6.2-13.fc27.x86_64 conflicts with file from package system-python-libs-3.6.3-2.fc26.x86_64
  file /usr/lib64/python3.6/xml/sax/__pycache__/handler.cpython-36.opt-2.pyc from install of python3-libs-3.6.2-13.fc27.x86_64 conflicts with file from package system-python-libs-3.6.3-2.fc26.x86_64
  file /usr/lib64/python3.6/xml/sax/__pycache__/handler.cpython-36.pyc from install of python3-libs-3.6.2-13.fc27.x86_64 conflicts with file from package system-python-libs-3.6.3-2.fc26.x86_64
  file /usr/lib64/python3.6/xml/sax/__pycache__/saxutils.cpython-36.opt-1.pyc from install of python3-libs-3.6.2-13.fc27.x86_64 conflicts with file from package system-python-libs-3.6.3-2.fc26.x86_64
  file /usr/lib64/python3.6/xml/sax/__pycache__/saxutils.cpython-36.opt-2.pyc from install of python3-libs-3.6.2-13.fc27.x86_64 conflicts with file from package system-python-libs-3.6.3-2.fc26.x86_64
  file /usr/lib64/python3.6/xml/sax/__pycache__/saxutils.cpython-36.pyc from install of python3-libs-3.6.2-13.fc27.x86_64 conflicts with file from package system-python-libs-3.6.3-2.fc26.x86_64
  file /usr/lib64/python3.6/xml/sax/__pycache__/xmlreader.cpython-36.opt-1.pyc from install of python3-libs-3.6.2-13.fc27.x86_64 conflicts with file from package system-python-libs-3.6.3-2.fc26.x86_64
  file /usr/lib64/python3.6/xml/sax/__pycache__/xmlreader.cpython-36.opt-2.pyc from install of python3-libs-3.6.2-13.fc27.x86_64 conflicts with file from package system-python-libs-3.6.3-2.fc26.x86_64
  file /usr/lib64/python3.6/xml/sax/__pycache__/xmlreader.cpython-36.pyc from install of python3-libs-3.6.2-13.fc27.x86_64 conflicts with file from package system-python-libs-3.6.3-2.fc26.x86_64


I have updated packages on Fedora 26 box:
# rpm -q dnf libdnf
dnf-2.7.5-1.fc26.noarch
libdnf-0.11.1-1.fc26.x86_64


Should I reopen this bug or report new one? To which component? It looks like python3-libs-3.6.2-13.fc27.x86_64 does not properly obsoletes system-python-libs.
Comment 83 Charalampos Stratakis 2017-11-06 09:34:12 EST
(In reply to Pavel Alexeev from comment #82)
> # dnf upgrade --refresh --enablerepo=updates-testing
> # dnf distro-sync --allowerasing --enablerepo=updates-testing
> # dnf system-upgrade download --refresh --releasever=27 --allowerasing --best
> 
> And it leads to mass errors like on stage of transaction testing:
> 
>   file
> /usr/lib64/python3.6/xml/parsers/__pycache__/expat.cpython-36.opt-1.pyc from
> install of python3-libs-3.6.2-13.fc27.x86_64 conflicts with file from
> package system-python-libs-3.6.3-2.fc26.x86_64
>   file
> /usr/lib64/python3.6/xml/parsers/__pycache__/expat.cpython-36.opt-2.pyc from
> install of python3-libs-3.6.2-13.fc27.x86_64 conflicts with file from
> package system-python-libs-3.6.3-2.fc26.x86_64
>   file /usr/lib64/python3.6/xml/parsers/__pycache__/expat.cpython-36.pyc
> from install of python3-libs-3.6.2-13.fc27.x86_64 conflicts with file from
> package system-python-libs-3.6.3-2.fc26.x86_64
>   file
> /usr/lib64/python3.6/xml/sax/__pycache__/_exceptions.cpython-36.opt-1.pyc
> from install of python3-libs-3.6.2-13.fc27.x86_64 conflicts with file from
> package system-python-libs-3.6.3-2.fc26.x86_64
>   file
> /usr/lib64/python3.6/xml/sax/__pycache__/_exceptions.cpython-36.opt-2.pyc
> from install of python3-libs-3.6.2-13.fc27.x86_64 conflicts with file from
> package system-python-libs-3.6.3-2.fc26.x86_64
>   file /usr/lib64/python3.6/xml/sax/__pycache__/_exceptions.cpython-36.pyc
> from install of python3-libs-3.6.2-13.fc27.x86_64 conflicts with file from
> package system-python-libs-3.6.3-2.fc26.x86_64
>   file
> /usr/lib64/python3.6/xml/sax/__pycache__/expatreader.cpython-36.opt-1.pyc
> from install of python3-libs-3.6.2-13.fc27.x86_64 conflicts with file from
> package system-python-libs-3.6.3-2.fc26.x86_64
>   file
> /usr/lib64/python3.6/xml/sax/__pycache__/expatreader.cpython-36.opt-2.pyc
> from install of python3-libs-3.6.2-13.fc27.x86_64 conflicts with file from
> package system-python-libs-3.6.3-2.fc26.x86_64
>   file /usr/lib64/python3.6/xml/sax/__pycache__/expatreader.cpython-36.pyc
> from install of python3-libs-3.6.2-13.fc27.x86_64 conflicts with file from
> package system-python-libs-3.6.3-2.fc26.x86_64
>   file /usr/lib64/python3.6/xml/sax/__pycache__/handler.cpython-36.opt-1.pyc
> from install of python3-libs-3.6.2-13.fc27.x86_64 conflicts with file from
> package system-python-libs-3.6.3-2.fc26.x86_64
>   file /usr/lib64/python3.6/xml/sax/__pycache__/handler.cpython-36.opt-2.pyc
> from install of python3-libs-3.6.2-13.fc27.x86_64 conflicts with file from
> package system-python-libs-3.6.3-2.fc26.x86_64
>   file /usr/lib64/python3.6/xml/sax/__pycache__/handler.cpython-36.pyc from
> install of python3-libs-3.6.2-13.fc27.x86_64 conflicts with file from
> package system-python-libs-3.6.3-2.fc26.x86_64
>   file
> /usr/lib64/python3.6/xml/sax/__pycache__/saxutils.cpython-36.opt-1.pyc from
> install of python3-libs-3.6.2-13.fc27.x86_64 conflicts with file from
> package system-python-libs-3.6.3-2.fc26.x86_64
>   file
> /usr/lib64/python3.6/xml/sax/__pycache__/saxutils.cpython-36.opt-2.pyc from
> install of python3-libs-3.6.2-13.fc27.x86_64 conflicts with file from
> package system-python-libs-3.6.3-2.fc26.x86_64
>   file /usr/lib64/python3.6/xml/sax/__pycache__/saxutils.cpython-36.pyc from
> install of python3-libs-3.6.2-13.fc27.x86_64 conflicts with file from
> package system-python-libs-3.6.3-2.fc26.x86_64
>   file
> /usr/lib64/python3.6/xml/sax/__pycache__/xmlreader.cpython-36.opt-1.pyc from
> install of python3-libs-3.6.2-13.fc27.x86_64 conflicts with file from
> package system-python-libs-3.6.3-2.fc26.x86_64
>   file
> /usr/lib64/python3.6/xml/sax/__pycache__/xmlreader.cpython-36.opt-2.pyc from
> install of python3-libs-3.6.2-13.fc27.x86_64 conflicts with file from
> package system-python-libs-3.6.3-2.fc26.x86_64
>   file /usr/lib64/python3.6/xml/sax/__pycache__/xmlreader.cpython-36.pyc
> from install of python3-libs-3.6.2-13.fc27.x86_64 conflicts with file from
> package system-python-libs-3.6.3-2.fc26.x86_64
> 
> 
> I have updated packages on Fedora 26 box:
> # rpm -q dnf libdnf
> dnf-2.7.5-1.fc26.noarch
> libdnf-0.11.1-1.fc26.x86_64
> 
> 
> Should I reopen this bug or report new one? To which component? It looks
> like python3-libs-3.6.2-13.fc27.x86_64 does not properly obsoletes
> system-python-libs.

This happens because you enabled the updates-testing repository during the distro-sync which installs python3-3.6.3-2.fc26, but you did not on the system-upgrade, so it tries to upgrade to python3-3.6.2-13.fc27.x86_64 which is not correct.

I have intentionally left the python3-3.6.3-2.fc26 build in updates-testing for this case to not happen (however it will happen if you enable the updates-testing repo and update your system before the distro upgrade).

In order to remedy this you should either downgrade python3 first, disable the updates-testing repo during distro-sync or enable it during system-upgrade.
Comment 84 Pavel Alexeev 2017-11-06 10:31:48 EST
Thank you.

# dnf system-upgrade download --refresh --releasever=27 --allowerasing --best --enablerepo=updates-testing

works.
Comment 85 Adam Williamson 2017-11-13 20:57:55 EST
Fixed, no longer needs documenting.

Note You need to log in before you can comment on or make changes to this bug.