Bug 2073195

Seems the issue might be missing "/" in sys.path ? ("'/usr/lib64python3.9'," seems suspicious...)

The missing / in '/usr/lib64python3.9' would indeed explain the ModuleNotFoundError. Does this happen with isolated mode? python3 -I

(In reply to Miro Hrončok from comment #3)
> The missing / in '/usr/lib64python3.9' would indeed explain the
> ModuleNotFoundError. Does this happen with isolated mode? python3 -I

The result is the same:

# python3 -I
Could not find platform independent libraries <prefix>
Could not find platform dependent libraries <exec_prefix>
Consider setting $PYTHONHOME to <prefix>[:<exec_prefix>]
Python path configuration:
  PYTHONHOME = (not set)
  PYTHONPATH = (not set)
  program name = 'python3'
  isolated = 1
  environment = 0
  user site = 0
  import site = 1
  sys._base_executable = '/usr/bin/python3'
  sys.base_prefix = '/usr'
  sys.base_exec_prefix = '/usr'
  sys.platlibdir = 'lib64'
  sys.executable = '/usr/bin/python3'
  sys.prefix = '/usr'
  sys.exec_prefix = '/usr'
  sys.path = [
    '/usr/lib64/python39.zip',
    '/usr/lib64python3.9',
    '/usr/lib64/lib-dynload',
  ]
Fatal Python error: init_fs_encoding: failed to get the Python codec of the filesystem encoding
Python runtime state: core initialized
ModuleNotFoundError: No module named 'encodings'

Current thread 0x000003ff8a576720 (most recent call first):
<no Python frame>

Hi Frantisek,
we are not able to reproduce the issue. Can you still reproduce it consistently?

Would you be able to reproduce it on some network machine (e.g. beaker) and give us access?

Alternatively, could you verify the RPMs in question are installed correctly on the affected system? The command is to do this is:

$ rpm -V python3 python3-libs

I cannot reproduce the issue on a RHEL9 s390x machine provided by Beaker.

Which python3 and python3-libs packages are you used? Where do these packages come from in the cloud image?

# python -m site
sys.path = [
    '/root',
    '/usr/lib64/python39.zip',
    '/usr/lib64/python3.9',
    '/usr/lib64/python3.9/lib-dynload',
    '/usr/lib64/python3.9/site-packages',
    '/usr/lib/python3.9/site-packages',
]
USER_BASE: '/root/.local' (doesn't exist)
USER_SITE: '/root/.local/lib/python3.9/site-packages' (doesn't exist)
ENABLE_USER_SITE: True

# env python -I -EsS
Python 3.9.10 (main, Feb  9 2022, 00:00:00) 
[GCC 11.2.1 20220127 (Red Hat 11.2.1-9)] on linux
>>> 

# virt-what 
ibm_systemz
ibm_systemz-zvm

# uname -m
s390x

# cat /etc/redhat-release 
Red Hat Enterprise Linux release 9.0 (Plow)

# rpm -q python3 python3-libs
python3-3.9.10-2.el9.s390x
python3-libs-3.9.10-2.el9.s390x

# rpm -Vv python3|grep /usr/bin/python
.........    /usr/bin/python3
.........    /usr/bin/python3.9

# rpm -Vv python3-libs|grep libpython
.........    /usr/lib64/libpython3.9.so.1.0
.........    /usr/lib64/libpython3.so


---


If I set PYTHONHOME env var to the wrong patch, well, I get trigger the "Python path configuration" fatal error:

# PYTHONHOME=/bug python3
Python path configuration:
  PYTHONHOME = '/bug'
  PYTHONPATH = (not set)
  (...)
  sys.path = [
    '/bug/lib64/python39.zip',
    '/bug/lib64/python3.9',
    '/bug/lib64/python3.9/lib-dynload',
  ]
Fatal Python error: init_fs_encoding: failed to get the Python codec of the filesystem encoding
(...)

But your output says "PYTHONHOME = (not set)" and "PYTHONPATH = (not set)". So your issue is not caused by an environment variable. Moreover, you still reproduce the issue with -I option (isolate) which ignores PYTHON environment variables like PYTHONHOME.

We have the same bugreport for AlmaLinux 9 Beta x390x:
https://bugs.almalinux.org/view.php?id=216

Copying a reporducer from there:

qemu-system-s390x \
    -machine s390-ccw-virtio \
    -device virtio-rng \
    -cpu max,zpci=on \
    -m 4096 -smp 2 \
    -serial mon:stdio -display none \
    -boot menu=on \
    -net nic,macaddr=52:54:00:12:34:56,model=virtio \
    -net tap,ifname=tap0,script=no,downscript=no \
    -device virtio-blk-ccw,id=dev-disk0,drive=disk0 \
    -drive id=disk0,file=dasd/almalinux.qcow2,if=none \
    -device virtio-scsi \
    -device scsi-cd,drive=c1,bootindex=1 \
    -drive file=AlmaLinux-9.0-beta-1-s390x-minimal.iso,format=raw,if=none,id=c1

I still cannot reproduce this, sorry. I run Fedora 35 if it matters.

I've downloaded and sha256sum-verified http://almalinux-mirror.dal1.hivelocity.net/9.0-beta/isos/s390x/AlmaLinux-9.0-beta-1-s390x-minimal.iso

I've run:

$ sudo dnf install /usr/bin/tunctl
$ sudo tunctl -t tap0 -u <my_usrname>

$ qemu-img create -f qcow2 almalinux.qcow2 4G

$ qemu-system-s390x \
    -machine s390-ccw-virtio \
    -device virtio-rng \
    -cpu max,zpci=on \
    -m 4096 -smp 2 \
    -serial mon:stdio -display none \
    -boot menu=on \
    -net nic,model=virtio \
    -net tap,ifname=tap0,script=no,downscript=no \
    -device virtio-blk-ccw,id=dev-disk0,drive=disk0 \
    -drive id=disk0,file=almalinux.qcow2,if=none \
    -device virtio-scsi \
    -device scsi-cd,drive=c1,bootindex=1 \
    -drive file=AlmaLinux-9.0-beta-1-s390x-minimal.iso,format=raw,if=none,id=c1

Booting to Anaconda proceeded and I could continue the installation. I could even switch to Shell and run Python just fine.

I've run the same command with CentOS-Stream-9-latest-s390x-boot.iso from https://www.centos.org/centos-stream/ -- still no issue.

Could somebody who can reproduce this issue with CentOS 9 Stream, Alma, RHEL, or anywhere provide a complete reproducer, step by step, command by command? Don't assume I know how you run qemu (for example, even the current reproducer provided in the Alma bug ended with dasd/almalinux.qcow2 file not found first and I had to quess how to create the file -- include the command I need to run to create it please).

> Have el9 loud image s390x image virtualized on x86_64 

Frnatišek,
on what operating system do you virtualize? Is it Fedora or RHEL or something else entirely? What version?

I've been told by another channel that this happened on Fedora 36 (has qemu 6.2.0). The AlamaLinux thing also happened on qemu 6.2.0 (on Gentoo).


"""I think this is because Almalinux 9 s390x binaries are compiled with -march z14
(see perl -V output), but versions of Qemu less than 7.0 emulate a z13 model
2975 CPU.  From version 7.0, qemu s390x emulates a z15 model 8561."""


I was not able to reproduce on Fedora 35 with qemu 6.1.0, but I can reproduce this on Fedora 36. This is how:

$ qemu-img create -f qcow2 almalinux.qcow2 4G
$ qemu-system-s390x \
    -machine s390-ccw-virtio \
    -device virtio-rng \
    -cpu max,zpci=on \
    -m 4096 -smp 2 \
    -serial mon:stdio -display none \
    -boot menu=on \
    -net nic,model=virtio \
    -device virtio-blk-ccw,id=dev-disk0,drive=disk0 \
    -drive id=disk0,file=almalinux.qcow2,if=none \
    -device virtio-scsi \
    -device scsi-cd,drive=c1,bootindex=1 \
    -drive file=AlmaLinux-9.0-beta-1-s390x-minimal.iso,format=raw,if=none,id=c1

(I removed the -net tap... argument thing as it complicated things and was not needed to reproduce.)


The bug reproduces, with AlmaLinux-9.0-beta-1-s390x-minimal.iso as well as with CentOS-Stream-9-latest-s390x-boot.iso.


I do not really understand why an older version of qemu works, 6.2.0 doesn't and 7.0.0 does, if this is about Z14, but at least I pinpointed it down to qemu version.

What worries me is bz2080519 which says this also happens on Fedora 34 which was not compiled with -march z14.

As there is no such problem with python3 when running on real hw, this is most likely an emulation issue, please move under qemu. cross-arch emulation is definitely not a supported scenario.

Hi John,
could you or somebody else from your SST please confirm if it is correct that RHEL 9 s390x images are not supported on qemu versions < 7?

Thank you!

Passing question along to Thomas Huth who knows more about s390x than I do.

(In reply to Tomas Orsava from comment #13)
> could you or somebody else from your SST please confirm if it is correct
> that RHEL 9 s390x images are not supported on qemu versions < 7?

If you ask for the support status: Guest emulation (TCG) is not supported at all in RHEL, neither on x86, nor on s390x, we only support KVM in downstream RHEL.

Having said that, do you still need a fix for QEMU 6.2.0, or are you fine with QEMU 7.0.0 ? In case you need it, it could be worth a try to bisect the issue to find out which patch fixed it in QEMU 7.0.0.

> do you still need a fix for QEMU 6.2.0

We do not. Closing in favor of bz2080519, as this is not a RHEL issue.

Thank you, Thomas.

Whether or not s390x Z14 is supported in qemu 6.2, this particular problem is fixed by a qemu fix, so I am marking this as duplicate instead.

*** This bug has been marked as a duplicate of bug 2080519 ***