Description of problem: Running a reservesys job on s390 using Fedora30 fails during installation in the 'beaker setup' phase. Sample output https://beaker.engineering.redhat.com/jobs/3824447 A conversation with Dan H., the Fedora s390 maintainer, indicated this problem is a Beaker install problem and not an anaconda one. Version-Release number of selected component (if applicable): How reproducible: 100% Steps to Reproduce: 1. run a reservesys job 2. select s390x 3. select Fedora30 Actual results: Fails to install Expected results: Succeeds. Additional info:
Console log output of error: http://beaker-archive.host.prod.eng.bos.redhat.com/beaker-logs/2019/10/38244/3824447/7433105/console.log [ 40.848771] anaconda[1992]: 14:59:18 Running pre-installation scripts + command -v curl + grep -q '.*--retry' + curl --help + fetch - http://lab-02.rhts.eng.bos.redhat.com:8000/install_start/7433105 + curl -L --retry 20 --remote-time -o - http://lab-02.rhts.eng.bos.redhat.com:80 00/install_start/7433105 % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 4 100 4 0 0 35 0 --:--:-- --:--:-- --:--:-- 35 1 00 4 100 4 0 0 35 0 --:--:-- --:--:-- --:--:-- 35 True+ command -v python3 + fetch /tmp/anamon http://lab-02.rhts.eng.bos.redhat.com/beaker/anamon3 + curl -L --retry 20 --remote-time -o /tmp/anamon http://lab-02.rhts.eng.bos.red hat.com/beaker/anamon3 % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 9522 100 9522 0 0 232k 0 --:--:-- --:--:-- --:--:-- 232k + python_command=python3 + python3 /tmp/anamon --recipe-id 7433105 --xmlrpc-url http://lab-02.rhts.eng.bo s.redhat.com:8000/RPC2 [ 44.007583] anaconda[1992]: 14:59:21 Not asking for VNC because of an automat ed install Exception in thread Thread-1: Traceback (most recent call last): File "/usr/lib64/python3.7/threading.py", line 917, in _bootstrap_inner self.run() File "/usr/lib64/python3.7/threading.py", line 865, in run self._target(*self._args, **self._kwargs) File "/tmp/anamon", line 70, in watch self.upload() File "/tmp/anamon", line 75, in upload super(WatchedCommand, self).upload() File "/tmp/anamon", line 40, in upload self.size, '', self.last_length, self.data): File "/usr/lib64/python3.7/xmlrpc/client.py", line 1112, in __call__ return self.__send(self.__name, args) File "/usr/lib64/python3.7/xmlrpc/client.py", line 1452, in __request verbose=self.__verbose File "/usr/lib64/python3.7/xmlrpc/client.py", line 1154, in request return self.single_request(host, handler, request_body, verbose) File "/usr/lib64/python3.7/xmlrpc/client.py", line 1166, in single_request http_conn = self.send_request(host, handler, request_body, verbose) File "/usr/lib64/python3.7/xmlrpc/client.py", line 1279, in send_request self.send_content(connection, request_body) File "/usr/lib64/python3.7/xmlrpc/client.py", line 1309, in send_content connection.endheaders(request_body) File "/usr/lib64/python3.7/http/client.py", line 1224, in endheaders self._send_output(message_body, encode_chunked=encode_chunked) File "/usr/lib64/python3.7/http/client.py", line 1016, in _send_output self.send(msg) File "/usr/lib64/python3.7/http/client.py", line 956, in send self.connect() File "/usr/lib64/python3.7/http/client.py", line 928, in connect (self.host,self.port), self.timeout, self.source_address) File "/usr/lib64/python3.7/socket.py", line 707, in create_connection for res in getaddrinfo(host, port, 0, SOCK_STREAM): File "/usr/lib64/python3.7/socket.py", line 748, in getaddrinfo for res in _socket.getaddrinfo(host, port, family, type, proto, flags): socket.gaierror: [Errno -2] Name or service not known Traceback (most recent call last): File "/tmp/anamon", line 294, in <module> anamon_loop() File "/tmp/anamon", line 249, in anamon_loop wf.update() File "/tmp/anamon", line 115, in update self.upload() File "/tmp/anamon", line 107, in upload status = super(WatchedFile, self).upload() File "/tmp/anamon", line 40, in upload self.size, '', self.last_length, self.data): File "/usr/lib64/python3.7/xmlrpc/client.py", line 1112, in __call__ return self.__send(self.__name, args) File "/usr/lib64/python3.7/xmlrpc/client.py", line 1452, in __request verbose=self.__verbose File "/usr/lib64/python3.7/xmlrpc/client.py", line 1154, in request return self.single_request(host, handler, request_body, verbose) File "/usr/lib64/python3.7/xmlrpc/client.py", line 1166, in single_request http_conn = self.send_request(host, handler, request_body, verbose) File "/usr/lib64/python3.7/xmlrpc/client.py", line 1279, in send_request self.send_content(connection, request_body) File "/usr/lib64/python3.7/xmlrpc/client.py", line 1309, in send_content connection.endheaders(request_body) File "/usr/lib64/python3.7/http/client.py", line 1224, in endheaders self._send_output(message_body, encode_chunked=encode_chunked) File "/usr/lib64/python3.7/http/client.py", line 1016, in _send_output self.send(msg) File "/usr/lib64/python3.7/http/client.py", line 956, in send self.connect() File "/usr/lib64/python3.7/http/client.py", line 928, in connect (self.host,self.port), self.timeout, self.source_address) File "/usr/lib64/python3.7/socket.py", line 707, in create_connection for res in getaddrinfo(host, port, 0, SOCK_STREAM): File "/usr/lib64/python3.7/socket.py", line 748, in getaddrinfo for res in _socket.getaddrinfo(host, port, family, type, proto, flags): socket.gaierror: [Errno -2] Name or service not known
for the record, I've reproduced this on "my" openshift-5.s390.bos.redhat.com too
(In reply to Dan Horák from comment #2) > for the record, I've reproduced this on "my" openshift-5.s390.bos.redhat.com > too Can you please provide me all params included in netboot configuration? socket.gaierror: [Errno -2] Name or service not known <-- Casual DNS resolve problem
https://beaker.engineering.redhat.com/recipes/7471416#installation is the beaker job [ 0.366198] Kernel command line: cio_ignore=all,!condev inst.sshd ip=10.16.1 6.212::10.16.111.254:255.255.248.0:openshift-5.s390.bos.redhat.com:enc600:none k s=http://beaker.engineering.redhat.com/kickstart/6024063 nameserver=10.11.5.19 n etbootloader= ramdisk_size=40000 rd.dasd=0.0.541f rd.dasd=0.0.551f rd.dasd=0.0.5 61f rd.dasd=0.0.571f rd.znet=qeth,0.0.0600,0.0.0601,0.0.0602,layer2=1,portno=0 r o netboot_method=zpxe For my test I've used "ip=10.16.106.212::10.16.111.254:255.255.248.0:openshift-5.s390.bos.redhat.com:enc600:none inst.sshd " as "Kernel options" string when Reserving the system using F-30. Interestingly the network works fine (checks in with the lab controller, downloads the anamon script) until "/tmp/anamon" is being started ... And it's true that later the machine isn't reachable with ping. I wonder if "network --bootproto=dhcp --hostname=..." in the kickstart can't break it.
I'm 99% sure it's the "network --bootproto=dhcp --hostname=..." in the kickstart. I have tested a F-31 installation with a kickstart, that worked previous time, with the "network" setting and it behaves the same, it loses network connectivity during the installation process (right after anaconda starts). In my opinion it's a beaker bug, it shouldn't override the networking setup with dhcp when the network is set up statically using dracut options.
There were some changes recently in anaconda to use NetworkManager for managing networking instead of their own code, which is probably why it worked until now.
Created attachment 1626025 [details] console log from successfully provisioned F-29
You can see the same anamon crash in the F-29 log too, but at a different place. I wonder if such crash occurs also in RHEL-8 installs ...
I doubt this ^.
no crash in RHEL-8.0.0 console/install log, same for F-28
We now have F31 GA imported into beaker so I tried again to see if it's still reproducible, however this time it's failing for a different reason ? Starting [0;1;39miSCSI UserSpace I/O driver [0m... [ 6.990582] audit: type=1130 audit(1574257472.777:12): pid=1 uid=0 auid=42949 67295 ses=4294967295 subj=kernel msg='unit=iscsiuio comm="systemd" exe="/usr/lib /systemd/systemd" hostname=? addr=? terminal=? res=success' [ [0;32m OK [0m] Started [0;1;39miSCSI UserSpace I/O driver [0m. Starting [0;1;39mOpen-iSCSI [0m... [ [0;32m OK [0m] Started [0;1;39mOpen-iSCSI [0m. [-- MARK -- Wed Nov 20 13:45:00 2019] [ 49.508707] random: crng init done [ 49.508712] random: 7 urandom warning(s) missed due to ratelimiting Starting [0;1;39mSetup Virtual Console [0m... [ [0;32m OK [0m] Started [0;1;39mSetup Virtual Console [0m. [ 240.060300] kauditd_printk_skb: 1 callbacks suppressed [ 240.060302] audit: type=1130 audit(1574257705.847:14): pid=1 uid=0 auid=42949 67295 ses=4294967295 subj=kernel msg='unit=systemd-vconsole-setup comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success' [ 240.060374] audit: type=1131 audit(1574257705.847:15): pid=1 uid=0 auid=42949 67295 ses=4294967295 subj=kernel msg='unit=systemd-vconsole-setup comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success' Starting [0;1;39mDracut Emergency Shell [0m... [ 240.094097] audit: type=1131 audit(1574257705.877:16): pid=1 uid=0 auid=42949 67295 ses=4294967295 subj=kernel msg='unit=plymouth-start comm="systemd" exe="/u sr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success' Warning: /dev/ram0 does not exist Generating "/run/initramfs/rdsosreport.txt" Entering emergency mode. Exit the shell to continue. Type "journalctl" to view system logs. You might want to save "/run/initramfs/rdsosreport.txt" to a USB stick or /boot after mounting them and attach it to a bug report. dracut:/# All five jobs failed failed for the same reason: https://beaker.engineering.redhat.com/jobs/3906964
I'm not able to find a single successful provision of F31 on s390x. Can anybody provide us with the correct kickstart for F31? We cannot help with fixing this unless we can get kickstart that actually works for latest Fedora.
regarding kickstarts - it should be sufficient to drop the "network" command, anaconda will ignore it for pre-configured networking in F-32 again (I saw some commits recently) for F-31 - it has own specifics, unfortunately, when configuring the network on the kernel command line, ip= must precede rd.znet=, again F-32 will behave normally, this is a result of network setup changes in the installer initrd (bug 1753975 and https://gitlab.freedesktop.org/NetworkManager/NetworkManager/merge_requests/317)
And for Rachel's jobs - they used the old style for setting up the devices (RHEL<=6), when using RHEL-8 options it should work fine, with the exception mentioned above
Thanks Dan! I tested that but I'm afraid this didn't help - https://beaker.engineering.redhat.com/recipes/7593101#installation You can see in the kickstart that the network line is missing yet it still fails.
Tomas, can you retry with rhel8 kernel parameters instead of the old style "HOSTNAME", "DNS" ...?
Hi just following up, was there any more progress on this ?
Hi Dan, I can schedule a meeting and we can try to figure out how to run Fedora on F30. I can modify the whole KS.. But I will need some advice regarding kernel cmdline
(In reply to Martin Styk 🤦♂️ from comment #18) > Hi Dan, > > I can schedule a meeting and we can try to figure out how to run Fedora on > F30. > I can modify the whole KS.. But I will need some advice regarding kernel > cmdline Hi Martin, sure, I will help, let's schedule a call.
So fix depends on 2 things. 1 => Proper order of arguments for kernel opts 2 => Put no_networks in ks_meta.
Hi Martin, is there a separate ticket we can reference to track the beaker install option changes needed for this ?
Fedora 30 https://beaker.engineering.redhat.com/jobs/4057749 1. added no_networks into distro ks_meta Running fine Fedora 31 https://beaker.engineering.redhat.com/jobs/4057914 1. added no_networks into distro ks_meta 2. Added S390x harness (Beaker currently missing F31 and F32 Restraint on s390x due to lack of builders in Red Hat) Furthermore, to make this work you have to make sure that Anaconda can read all parameters correctly. `ip=` has to be in front of `rd.znet` You can see it in my jobs: cio_ignore=all,!condev inst.sshd ip=10.16.106.212::10.16.111.254:255.255.248.0:openshift-5.s390.bos.redhat.com:enc600:none ks=http://beaker.engineering.redhat.com/kickstart/6455559 ksdevice=bootif nameserver=10.11.5.19 netbootloader= ramdisk_size=40000 rd.dasd=0.0.541f rd.dasd=0.0.551f rd.dasd=0.0.561f rd.dasd=0.0.571f rd.znet=qeth,0.0.0600,0.0.0601,0.0.0602,layer2=1,portno=0 ro