Bug 2116221
| Summary: | Hang at "start aardvark-dns" when run container with podman | ||||||
|---|---|---|---|---|---|---|---|
| Product: | Red Hat Enterprise Linux 9 | Reporter: | Xiaofeng Wang <xiaofwan> | ||||
| Component: | netavark | Assignee: | Jindrich Novy <jnovy> | ||||
| Status: | CLOSED ERRATA | QA Contact: | Joy Pu <ypu> | ||||
| Severity: | high | Docs Contact: | |||||
| Priority: | high | ||||||
| Version: | 9.1 | CC: | arajan, atodorov, bbaude, dwalsh, elpereir, jnovy, jrusz, lsm5, mboddu, mheon, obudai, perobins, pthomas, tech, tsweeney, umohnani, ypu | ||||
| Target Milestone: | rc | Keywords: | TestBlocker, Triaged | ||||
| Target Release: | --- | Flags: | pm-rhel:
mirror+
|
||||
| Hardware: | Unspecified | ||||||
| OS: | Unspecified | ||||||
| Whiteboard: | |||||||
| Fixed In Version: | aardvark-dns-1.1.0-3.el9 | Doc Type: | If docs needed, set a value | ||||
| Doc Text: | Story Points: | --- | |||||
| Clone Of: | Environment: | ||||||
| Last Closed: | 2022-11-15 10:39:46 UTC | Type: | Bug | ||||
| Regression: | --- | Mount Type: | --- | ||||
| Documentation: | --- | CRM: | |||||
| Verified Versions: | Category: | --- | |||||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||
| Cloudforms Team: | --- | Target Upstream Version: | |||||
| Embargoed: | |||||||
| Attachments: |
|
||||||
Cannot reproduce. Can you provide more details - the configuration of the Edge network is particularly relevant. I add edge network with command sudo podman network create --driver=bridge --subnet=192.168.200.0/24 --gateway=192.168.200.254 edge Thanks. Is the test run by a script? Are there any more details on the system used to test with? It's run by script "ostree-simplified-installer.sh" from repo https://github.com/virt-s1/rhel-edge. Test runs on RHEL 9.1 VM (latest nightly image) on PSI openstack. If you need a test environment, I can setup one for you. I have a libvirt network configured for my VM. <network xmlns:dnsmasq='http://libvirt.org/schemas/network/dnsmasq/1.0'> <name>integration</name> <uuid>1c8fe98c-b53a-4ca4-bbdb-deb0f26b3579</uuid> <forward mode='nat'> <nat> <port start='1024' end='65535'/> </nat> </forward> <bridge name='integration' zone='trusted' stp='on' delay='0'/> <mac address='52:54:00:36:46:ef'/> <ip address='192.168.100.1' netmask='255.255.255.0'> <dhcp> <range start='192.168.100.2' end='192.168.100.254'/> <host mac='34:49:22:B0:83:30' name='vm-1' ip='192.168.100.50'/> <host mac='34:49:22:B0:83:31' name='vm-2' ip='192.168.100.51'/> <host mac='34:49:22:B0:83:32' name='vm-3' ip='192.168.100.52'/> </dhcp> </ip> <dnsmasq:options> <dnsmasq:option value='dhcp-vendorclass=set:efi-http,HTTPClient:Arch:00016'/> <dnsmasq:option value='dhcp-option-force=tag:efi-http,60,HTTPClient'/> <dnsmasq:option value='dhcp-boot=tag:efi-http,"http://192.168.100.1/httpboot/EFI/BOOT/BOOTX64.EFI"'/> </dnsmasq:options> </network> I checked the packages on the system and it seems to have started happening when the netavark package got updated to "netavark 1.1.0-6.el9" It was working fine with: "netavark 1.0.1-40.el9" I have prepared a reproducer. 1) Provision rhel-9.1 nightly VM 2) Use this as system repos (it's just a snapshot of nightly trees at the given date): [baseos] name=baseos baseurl=https://rpmrepo.osbuild.org/v2/mirror/rhvpn/el9/el9-x86_64-baseos-n9.1-20220715 enabled=1 gpgcheck=0 [appstream] name=appstream baseurl=https://rpmrepo.osbuild.org/v2/mirror/rhvpn/el9/el9-x86_64-appstream-n9.1-20220715 enabled=1 gpgcheck=0 3) install podman This should give you podman-4.1.1-3.el9.x86_64 netavark-1.0.1-39.el9.x86_64 4) create a network with based on Comment #3 (sudo podman network create --driver=bridge --subnet=192.168.200.0/24 --gateway=192.168.200.254 edge) 5) sudo podman run -d --name rhel-test --network edge1 --ip 192.168.200.10 quay.io/jrusz/rhbz-2116221-reproducer This should work and run the container 6) Go back to system sources and change the snapshots to 20220815 at least on appstream so your url will look like: baseurl=https://rpmrepo.osbuild.org/v2/mirror/rhvpn/el9/el9-x86_64-appstream-n9.1-20220815 7) sudo dnf update netavark This should give you: netavark-1.1.0-6.el9.x86_64 8) run step 5 gain but change Ip and name: podman run -d --name rhel-test1 --network edge1 --ip 192.168.200.11 quay.io/jrusz/rhbz-2116221-reproducer This gets stuck exactly like in the description. I hope this helps, I'm switching the component to netavark as this seems it's related to that. I've made a slight mistake in steps 5 and 8, the network name should be 'edge' not 'edge1'. So - from your notes, Netavark v1.1.1 *does not* show the issue? I think we're planning on shipping 1.1.1 in 9.1 already - is that satisfactory? I have not tried Netavark v1.1.1 so I don't know, I can try if you send me link to rpm though. What I am saying is I see the issue with Netavark v1.1.0 but don't see it with v1.0.1 According to the Netavark Github issue https://github.com/containers/netavark/issues/391, the problem is caused by a mismatch between the netavark and aardvark-dns versions. I see the same problem with CentOS Stream 9 and downgrading netavark solved it. Assigning to Jindrich to tend to the packaging changes. I added conflict from aardvark-dns to netavark older than aarvark-dns's version. Checked with the package aardvark-dns-1.1.0-4.el9.x86_64.rpm: # rpm -q --conflicts aardvark-dns-1.1.0-4.el9.x86_64.rpm netavark < 2:1.1.0 It is already effect. So set the Tested flag. Verified with aardvark-dns-1.1.0-4.el9.x86_64.rpm and netavark-1.1.0-6.el9.x86_64.rpm. Move to verified directly as no new build since aardvark-dns-1.1.0-4.el9.x86_64.rpm Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (netavark bug fix and enhancement update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2022:8231 |
Created attachment 1904221 [details] podman debug log Description of problem: Run a container with command: sudo podman run --log-level=debug -d -v /home/admin/fdo-containers/ownership_vouchers:/etc/fdo/ownership_vouchers:z -v /home/admin/fdo-containers/config/manufacturing-server.yml:/etc/fdo/manufacturing-server.conf.d/00-default.yml:z -v /home/admin/fdo-containers/keys:/etc/fdo/keys:z --ip 192.168.200.2 --name fdo-manufacturing-server --network edge quay.io/fido-fdo/fdo-manufacturing-server:0.4.0 Hang at "start aardvark-dns" with debug log: [DEBUG netavark::dns::aardvark] Spawning aardvark server [DEBUG netavark::dns::aardvark] start aardvark-dns: ["systemd-run", "-q", "--scope", "/usr/libexec/podman/aardvark-dns", "--config", "/run/containers/networks/aardvark-dns", "-p", "53", "run"] At the same time, in another tty session, run "sudo podman ps -a" hang as well. Version-Release number of selected component (if applicable): Client: Podman Engine Version: 4.1.1 API Version: 4.1.1 Go Version: go1.18.1 Built: Thu Jul 28 11:01:30 2022 OS/Arch: linux/amd64 How reproducible: Steps to Reproduce: 1. Run container with command: sudo podman run --log-level=debug -d -v /home/admin/fdo-containers/ownership_vouchers:/etc/fdo/ownership_vouchers:z -v /home/admin/fdo-containers/config/manufacturing-server.yml:/etc/fdo/manufacturing-server.conf.d/00-default.yml:z -v /home/admin/fdo-containers/keys:/etc/fdo/keys:z --ip 192.168.200.2 --name fdo-manufacturing-server --network edge quay.io/fido-fdo/fdo-manufacturing-server:0.4.0 Actual results: Hang there Expected results: Run container without error Additional info: podman debug log attached.