Bug 2080519

Thanks for the report. When you upgrade back to python3-libs-3.9.12-1.fc34, does the problem happen again? Is your s390x emulated or bare metal?

We got a similar report for RHEL 9 in bz2073195 but we could not reproduce it.

I cannot reproduce this:




$ podman run --arch s390x --rm -ti fedora:34 /usr/bin/bash

[root@cfc4cbf3e0b3 /]# rpm -q python3-libs
python3-libs-3.9.10-1.fc34.s390x

[root@cfc4cbf3e0b3 /]# dnf upgrade python3-libs
...

[root@cfc4cbf3e0b3 /]# rpm -q python3-libs
python3-libs-3.9.12-1.fc34.s390x

[root@d6736d92a448 /]# python3
Python 3.9.12 (main, Mar 25 2022, 00:00:00) 
[GCC 11.2.1 20220127 (Red Hat 11.2.1-9)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> sys.path
['', '/usr/lib64/python39.zip', '/usr/lib64/python3.9', '/usr/lib64/python3.9/lib-dynload', '/usr/lib64/python3.9/site-packages', '/usr/lib/python3.9/site-packages']


[root@d6736d92a448 /]# python3 -m site
sys.path = [
    '/',
    '/usr/lib64/python39.zip',
    '/usr/lib64/python3.9',
    '/usr/lib64/python3.9/lib-dynload',
    '/usr/lib64/python3.9/site-packages',
    '/usr/lib/python3.9/site-packages',
]
USER_BASE: '/root/.local' (doesn't exist)
USER_SITE: '/root/.local/lib/python3.9/site-packages' (doesn't exist)
ENABLE_USER_SITE: True

> Is your s390x emulated or bare metal?

And if emulated, please tell me exactly how. What is the host operating system (e.g. is it Fedora 34 itself on x86_64?) and what is the virtualization tool.

Yes, the s390x machine is emulated:
host: Ubuntu 22.04 x86_64 on top of Windows Subsystem for Linux (WSL) on Windows 10 Pro
s390x system emulation: QEMU 6.2.0

(In reply to Miro Hrončok from comment #1)
> Thanks for the report. When you upgrade back to python3-libs-3.9.12-1.fc34,
> does the problem happen again? Is your s390x emulated or bare metal?
> 
> We got a similar report for RHEL 9 in bz2073195 but we could not reproduce
> it.

Yes, I tried upgrading after the successful downgrade and it introduced the same problem again

(In reply to Kenneth Salerno from comment #4)
> Yes, the s390x machine is emulated:
> host: Ubuntu 22.04 x86_64 on top of Windows Subsystem for Linux (WSL) on
> Windows 10 Pro
> s390x system emulation: QEMU 6.2.0

I also tried upgrading to WSL2, same issue:

C:\Users\KenSalerno>wsl -l -v
  NAME            STATE           VERSION
* Ubuntu-22.04    Running         2

python3-3.9.12-1.fc34.s390x works just fine on real hw (tested with z13, z14, z15), so this is very unlikely a python issue, but most likely an emulation issue. Please move under qemu.

Reassigned to qemu in Fedora 36, where we have 6.2.0

In BZ 2073195 it has been mentioned that it should work again when using the new QEMU v7.0.0 ... so it might be worth a try to bisect the issue if you need the fix in 6.2 as well.

I can reproduce! On Fedora 36 with qemu-system-s390x-core-6.2.0-6.fc36.x86_64

I downloaded https://download.fedoraproject.org/pub/fedora-secondary/releases/34/Cloud/s390x/images/Fedora-Cloud-Base-34-1.2.s390x.qcow2


$ virt-customize -v -a Fedora-Cloud-Base-34-1.2.s390x.qcow2 --root-password password:toor
$ qemu-system-s390x -machine s390-ccw-virtio -device virtio-rng -cpu max,zpci=on -m 4096 -smp 2 -serial mon:stdio -display none -boot menu=on -device virtio-blk-ccw,id=dev-disk0,drive=disk0 -drive id=disk0,file=Fedora-Cloud-Base-34-1.2.s390x.qcow2,if=none -device virtio-scsi
...
After "[  OK  ] Reached target Cloud-init target" I logged in as root:toor (the login prompt is intermingled with the most recent logs)

[root@fedora ~]# rpm -q python3
python3-3.9.2-1.fc34.s390x
[root@fedora ~]# dnf --repo=updates update python3
...
[root@fedora ~]# rpm -q python3
python3-3.9.12-1.fc34.s390x
[root@fedora ~]# python3
Could not find platform independent libraries <prefix>
Could not find platform dependent libraries <exec_prefix>
Consider setting $PYTHONHOME to <prefix>[:<exec_prefix>]
Python path configuration:
  PYTHONHOME = (not set)
  PYTHONPATH = (not set)
  program name = 'python3'
  isolated = 0
  environment = 1
  user site = 1
  import site = 1
  sys._base_executable = '/usr/bin/python3'
  sys.base_prefix = '/usr'
  sys.base_exec_prefix = '/usr'
  sys.platlibdir = 'lib64'
  sys.executable = '/usr/bin/python3'
  sys.prefix = '/usr'
  sys.exec_prefix = '/usr'
  sys.path = [
    '/usr/lib64/python39.zip',
    '/usr/lib64python3.9',
    '/usr/lib64/lib-dynload',
  ]
Fatal Python error: init_fs_encoding: failed to get the Python codec of the filesystem encoding
Python runtime state: core initialized
ModuleNotFoundError: No module named 'encodings'

Current thread 0x000003ff84478710 (most recent call first):
<no Python frame>

Tip: add ,snapshot=on to place a snapshot over the drive.
Also: virt-customize --remove cloud-init

I can also reproduce this with v6.2.0, can confirm it is fixed
in current git head (44f28df24767c), and am attempting the bisection.

(In reply to Richard W.M. Jones from comment #11)
> I can also reproduce this with v6.2.0, can confirm it is fixed
> in current git head (44f28df24767c), and am attempting the bisection.
                       ^^^^

Sorry, copied the wrong hash.  The good hash is 1fba9dc71a17

> Also: virt-customize --remove cloud-init

Nice tip!

------

FTR I've tried to run the exact same commands on Fedora 35 with qemu 6.1.0-14.fc35 and it doe snot happen there.

Bisected the fix to:

commit 46697cb96e1cc6c3f1edbe572cee1ce9ac97cc58
Author: Richard Henderson <richard.henderson>
Date:   Mon Mar 14 17:25:06 2022 -0700

    accel/tcg: Fix cpu_ldq_be_mmu typo
    
    In the conversion to cpu_ld_*_mmu, the retaddr parameter
    was corrupted in the one case of cpu_ldq_be_mmu.
    
    Fixes: f83bcecb1 ("accel/tcg: Add cpu_{ld,st}*_mmu interfaces")
    Resolves: https://gitlab.com/qemu-project/qemu/-/issues/902
    Signed-off-by: Richard Henderson <richard.henderson>
    Message-Id: <20220315002506.152030-1-richard.henderson>
    Reviewed-by: Philippe Mathieu-Daudé <f4bug>
    Tested-by: Thomas Huth <thuth>
    Signed-off-by: Thomas Huth <thuth>

 accel/tcg/cputlb.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

Do we want to backport this?  This is already fixed in Rawhide / F37+
but is broken in Fedora 36.  It is OK in Fedora 35.

Is qemu doing upstream 6.2.x maintanance releases? I'd like to get that there if possible, so people won't assume Python is broken on RHEL9/s390x.

If not, a Fedora backport is the next best thing.

(In reply to Miro Hrončok from comment #15)
> Is qemu doing upstream 6.2.x maintanance releases? I'd like to get that
> there if possible, so people won't assume Python is broken on RHEL9/s390x.

In theory, although I do not see any stable-6.2 branch upstream so far.

> If not, a Fedora backport is the next best thing.

Let's try this one for now.

NB I had to modify the upstream patch to make it apply.  Compare:

https://gitlab.com/qemu/qemu/-/commit/46697cb96e1cc6c3f1edbe572cee1ce9ac97cc58
https://src.fedoraproject.org/rpms/qemu/c/a9028c06adb5e2973796d57ab983dea3fef41efd?branch=f36

FEDORA-2022-eed7d1e529 has been submitted as an update to Fedora 36. https://bodhi.fedoraproject.org/updates/FEDORA-2022-eed7d1e529

*** Bug 2073195 has been marked as a duplicate of this bug. ***

FEDORA-2022-eed7d1e529 has been pushed to the Fedora 36 testing repository.
Soon you'll be able to install the update with the following command:
`sudo dnf upgrade --enablerepo=updates-testing --advisory=FEDORA-2022-eed7d1e529`
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2022-eed7d1e529

See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.

FEDORA-2022-eed7d1e529 has been pushed to the Fedora 36 stable repository.
If problem still persists, please make note of it in this bug report.