RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 2116221 - Hang at "start aardvark-dns" when run container with podman
Summary: Hang at "start aardvark-dns" when run container with podman
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Enterprise Linux 9
Classification: Red Hat
Component: netavark
Version: 9.1
Hardware: Unspecified
OS: Unspecified
high
high
Target Milestone: rc
: ---
Assignee: Jindrich Novy
QA Contact: Joy Pu
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2022-08-08 01:44 UTC by Xiaofeng Wang
Modified: 2022-11-15 11:46 UTC (History)
17 users (show)

Fixed In Version: aardvark-dns-1.1.0-3.el9
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2022-11-15 10:39:46 UTC
Type: Bug
Target Upstream Version:
Embargoed:
pm-rhel: mirror+


Attachments (Terms of Use)
podman debug log (20.62 KB, text/plain)
2022-08-08 01:44 UTC, Xiaofeng Wang
no flags Details


Links
System ID Private Priority Status Summary Last Updated
Red Hat Issue Tracker RHELPLAN-130450 0 None None None 2022-08-08 01:50:03 UTC
Red Hat Product Errata RHBA-2022:8231 0 None None None 2022-11-15 10:39:49 UTC

Description Xiaofeng Wang 2022-08-08 01:44:38 UTC
Created attachment 1904221 [details]
podman debug log

Description of problem:
Run a container with command:
sudo podman run --log-level=debug -d -v /home/admin/fdo-containers/ownership_vouchers:/etc/fdo/ownership_vouchers:z -v /home/admin/fdo-containers/config/manufacturing-server.yml:/etc/fdo/manufacturing-server.conf.d/00-default.yml:z -v /home/admin/fdo-containers/keys:/etc/fdo/keys:z --ip 192.168.200.2 --name fdo-manufacturing-server --network edge quay.io/fido-fdo/fdo-manufacturing-server:0.4.0

Hang at "start aardvark-dns" with debug log:
[DEBUG netavark::dns::aardvark] Spawning aardvark server
[DEBUG netavark::dns::aardvark] start aardvark-dns: ["systemd-run", "-q", "--scope", "/usr/libexec/podman/aardvark-dns", "--config", "/run/containers/networks/aardvark-dns", "-p", "53", "run"]

At the same time, in another tty session, run "sudo podman ps -a" hang as well.

Version-Release number of selected component (if applicable):
Client:       Podman Engine
Version:      4.1.1
API Version:  4.1.1
Go Version:   go1.18.1
Built:        Thu Jul 28 11:01:30 2022
OS/Arch:      linux/amd64

How reproducible:

Steps to Reproduce:
1. Run container with command:
sudo podman run --log-level=debug -d -v /home/admin/fdo-containers/ownership_vouchers:/etc/fdo/ownership_vouchers:z -v /home/admin/fdo-containers/config/manufacturing-server.yml:/etc/fdo/manufacturing-server.conf.d/00-default.yml:z -v /home/admin/fdo-containers/keys:/etc/fdo/keys:z --ip 192.168.200.2 --name fdo-manufacturing-server --network edge quay.io/fido-fdo/fdo-manufacturing-server:0.4.0


Actual results:
Hang there

Expected results:
Run container without error

Additional info:
podman debug log attached.

Comment 2 Matthew Heon 2022-08-11 14:10:41 UTC
Cannot reproduce. Can you provide more details - the configuration of the Edge network is particularly relevant.

Comment 3 Xiaofeng Wang 2022-08-11 15:23:27 UTC
I add edge network with command
sudo podman network create --driver=bridge --subnet=192.168.200.0/24 --gateway=192.168.200.254 edge

Thanks.

Comment 4 Matthew Heon 2022-08-11 17:15:18 UTC
Is the test run by a script? Are there any more details on the system used to test with?

Comment 5 Xiaofeng Wang 2022-08-11 17:20:50 UTC
It's run by script "ostree-simplified-installer.sh" from repo https://github.com/virt-s1/rhel-edge.
Test runs on RHEL 9.1 VM (latest nightly image) on PSI openstack.
If you need a test environment, I can setup one for you.

Comment 6 Xiaofeng Wang 2022-08-11 17:24:47 UTC
I have a libvirt network configured for my VM.
<network xmlns:dnsmasq='http://libvirt.org/schemas/network/dnsmasq/1.0'>
  <name>integration</name>
  <uuid>1c8fe98c-b53a-4ca4-bbdb-deb0f26b3579</uuid>
  <forward mode='nat'>
    <nat>
      <port start='1024' end='65535'/>
    </nat>
  </forward>
  <bridge name='integration' zone='trusted' stp='on' delay='0'/>
  <mac address='52:54:00:36:46:ef'/>
  <ip address='192.168.100.1' netmask='255.255.255.0'>
    <dhcp>
      <range start='192.168.100.2' end='192.168.100.254'/>
      <host mac='34:49:22:B0:83:30' name='vm-1' ip='192.168.100.50'/>
      <host mac='34:49:22:B0:83:31' name='vm-2' ip='192.168.100.51'/>
      <host mac='34:49:22:B0:83:32' name='vm-3' ip='192.168.100.52'/>
    </dhcp>
  </ip>
  <dnsmasq:options>
    <dnsmasq:option value='dhcp-vendorclass=set:efi-http,HTTPClient:Arch:00016'/>
    <dnsmasq:option value='dhcp-option-force=tag:efi-http,60,HTTPClient'/>
    <dnsmasq:option value='dhcp-boot=tag:efi-http,&quot;http://192.168.100.1/httpboot/EFI/BOOT/BOOTX64.EFI&quot;'/>
  </dnsmasq:options>
</network>

Comment 7 Jakub Rusz 2022-08-18 13:28:41 UTC
I checked the packages on the system and it seems to have started happening when the netavark package got updated to "netavark 1.1.0-6.el9" 
It was working fine with: "netavark 1.0.1-40.el9"

Comment 8 Jakub Rusz 2022-08-19 12:56:39 UTC
I have prepared a reproducer.

1) Provision rhel-9.1 nightly VM
2) Use this as system repos (it's just a snapshot of nightly trees at the given date):
[baseos]
name=baseos
baseurl=https://rpmrepo.osbuild.org/v2/mirror/rhvpn/el9/el9-x86_64-baseos-n9.1-20220715
enabled=1
gpgcheck=0

[appstream]
name=appstream
baseurl=https://rpmrepo.osbuild.org/v2/mirror/rhvpn/el9/el9-x86_64-appstream-n9.1-20220715
enabled=1
gpgcheck=0

3) install podman
This should give you
podman-4.1.1-3.el9.x86_64
netavark-1.0.1-39.el9.x86_64

4) create a network with based on Comment #3 (sudo podman network create --driver=bridge --subnet=192.168.200.0/24 --gateway=192.168.200.254 edge)

5) sudo podman run -d --name rhel-test --network edge1 --ip 192.168.200.10 quay.io/jrusz/rhbz-2116221-reproducer
This should work and run the container

6) Go back to system sources and change the snapshots to 20220815 at least on appstream so your url will look like:
baseurl=https://rpmrepo.osbuild.org/v2/mirror/rhvpn/el9/el9-x86_64-appstream-n9.1-20220815

7) sudo dnf update netavark
This should give you:
netavark-1.1.0-6.el9.x86_64

8) run step 5 gain but change Ip and name:
podman run -d --name rhel-test1 --network edge1 --ip 192.168.200.11 quay.io/jrusz/rhbz-2116221-reproducer

This gets stuck exactly like in the description. I hope this helps, I'm switching the component to netavark as this seems it's related to that.

Comment 9 Jakub Rusz 2022-08-19 13:05:44 UTC
I've made a slight mistake in steps 5 and 8, the network name should be 'edge' not 'edge1'.

Comment 10 Matthew Heon 2022-08-19 13:24:45 UTC
So - from your notes, Netavark v1.1.1 *does not* show the issue?

I think we're planning on shipping 1.1.1 in 9.1 already - is that satisfactory?

Comment 11 Jakub Rusz 2022-08-19 13:31:13 UTC
I have not tried Netavark v1.1.1 so I don't know, I can try if you send me link to rpm though. 

What I am saying is I see the issue with Netavark v1.1.0 but don't see it with v1.0.1

Comment 20 Scott Hansen 2022-08-22 19:41:28 UTC
According to the Netavark Github issue https://github.com/containers/netavark/issues/391, the problem is caused by a mismatch between the netavark and aardvark-dns versions. I see the same problem with CentOS Stream 9 and downgrading netavark solved it.

Comment 21 Tom Sweeney 2022-08-22 21:57:52 UTC
Assigning to Jindrich to tend to the packaging changes.

Comment 22 Jindrich Novy 2022-08-23 07:19:22 UTC
I added conflict from aardvark-dns to netavark older than aarvark-dns's version.

Comment 23 Joy Pu 2022-08-31 03:18:41 UTC
Checked with the package aardvark-dns-1.1.0-4.el9.x86_64.rpm:
# rpm -q --conflicts aardvark-dns-1.1.0-4.el9.x86_64.rpm 
netavark < 2:1.1.0

It is already effect. 
So set the Tested flag.

Comment 24 Xiaofeng Wang 2022-08-31 03:28:15 UTC
Verified with aardvark-dns-1.1.0-4.el9.x86_64.rpm and netavark-1.1.0-6.el9.x86_64.rpm.

Comment 28 Joy Pu 2022-09-05 15:17:22 UTC
Move to verified directly as no new build since aardvark-dns-1.1.0-4.el9.x86_64.rpm

Comment 30 errata-xmlrpc 2022-11-15 10:39:46 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (netavark bug fix and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2022:8231


Note You need to log in before you can comment on or make changes to this bug.