RHEL Engineering is moving the tracking of its product development work on RHEL 6 through RHEL 9 to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "RHEL project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs in the statuses "NEW", "ASSIGNED", and "POST" are being migrated throughout September 2023. Bugs of Red Hat partners with an assigned Engineering Partner Manager (EPM) are migrated in late September as per pre-agreed dates. Bugs against components "kernel", "kernel-rt", and "kpatch" are only migrated if still in "NEW" or "ASSIGNED". If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "RHEL project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/RHEL-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 1915950 - IPv6 errors after exiting podman
Summary: IPv6 errors after exiting podman
Keywords:
Status: CLOSED DUPLICATE of bug 1918417
Alias: None
Product: Red Hat Enterprise Linux 8
Classification: Red Hat
Component: podman
Version: 8.4
Hardware: All
OS: Linux
medium
medium
Target Milestone: rc
: 8.4
Assignee: Matthew Heon
QA Contact: atomic-bugs@redhat.com
URL:
Whiteboard:
: 1954844 (view as bug list)
Depends On:
Blocks: 1918417 1932399 1942665
TreeView+ depends on / blocked
 
Reported: 2021-01-13 19:24 UTC by Qian Cai
Modified: 2021-08-02 08:26 UTC (History)
12 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
: 1918417 (view as bug list)
Environment:
Last Closed: 2021-02-24 18:25:46 UTC
Type: Bug
Target Upstream Version:
Embargoed:

Attachments (Terms of Use)

Description Qian Cai 2021-01-13 19:24:46 UTC 
Description of problem:
After setting up the ipv4/ipv6 dual-stack for podman using the diff below.

# diff -u /etc/cni/net.d/87-podman-bridge.conflist.orig /etc/cni/net.d/87-podman-bridge.conflist
--- /etc/cni/net.d/87-podman-bridge.conflist.orig	2021-01-13 10:44:39.787742958 -0500
+++ /etc/cni/net.d/87-podman-bridge.conflist	2021-01-13 10:59:12.012980806 -0500
@@ -7,16 +7,21 @@
       "bridge": "cni-podman0",
       "isGateway": true,
       "ipMasq": true,
-      "hairpinMode": true,
       "ipam": {
         "type": "host-local",
-        "routes": [{ "dst": "0.0.0.0/0" }],
+        "routes": [{ "dst": "0.0.0.0/0" }, {"dst": "2000::/3" }],
         "ranges": [
           [
             {
               "subnet": "10.88.0.0/16",
               "gateway": "10.88.0.1"
             }
+          ],
+          [
+            {
+              "subnet": "2601:601:9f80:3c4f::/64",
+              "gateway": "2601:601:9f80:3c4f::1"
+            }
           ]
         ]
       }

# podman run --rm --runtime=kata -it alpine sh
/ # exit
ERRO[0011] Error deleting network: running [/usr/sbin/ip6tables -t nat -D POSTROUTING -s 2601:601:9f80:3c4f::4/64 -j CNI-b0b16c6135b11805f014bb70 -m comment --comment name: "podman" id: "e4b7fa1550ac6c98dfbfb5a44e2928852d789bc172868645c80376142ea405df" --wait]: exit status 1: iptables: Bad rule (does a matching rule exist in that chain?). 
ERRO[0011] Error while removing pod from CNI network "podman": running [/usr/sbin/ip6tables -t nat -D POSTROUTING -s 2601:601:9f80:3c4f::4/64 -j CNI-b0b16c6135b11805f014bb70 -m comment --comment name: "podman" id: "e4b7fa1550ac6c98dfbfb5a44e2928852d789bc172868645c80376142ea405df" --wait]: exit status 1: iptables: Bad rule (does a matching rule exist in that chain?). 
ERRO[0011] unable to cleanup network for container e4b7fa1550ac6c98dfbfb5a44e2928852d789bc172868645c80376142ea405df: "error tearing down CNI namespace configuration for container e4b7fa1550ac6c98dfbfb5a44e2928852d789bc172868645c80376142ea405df: running [/usr/sbin/ip6tables -t nat -D POSTROUTING -s 2601:601:9f80:3c4f::4/64 -j CNI-b0b16c6135b11805f014bb70 -m comment --comment name: \"podman\" id: \"e4b7fa1550ac6c98dfbfb5a44e2928852d789bc172868645c80376142ea405df\" --wait]: exit status 1: iptables: Bad rule (does a matching rule exist in that chain?).\n"

Version-Release number of selected component (if applicable):
containernetworking-plugins-0.8.6-2.module+el8.3.0+8221+97165c3f.x86_64
podman-2.0.5-5.module+el8.3.0+8221+97165c3f.x86_64
iptables-1.8.4-15.el8_3.3.x86_64

How reproducible:
always
Comment 1 Qian Cai 2021-01-13 19:26:19 UTC 
Same thing with runc, i.e., # podman run --rm -it alpine sh
Comment 2 Tom Sweeney 2021-01-13 21:54:52 UTC 
Qian,

Can you try this test with a more recent version of Podman?  I think this may have been addressed in later versions of Podman.
Comment 3 Qian Cai 2021-01-13 22:34:46 UTC 
I don't have an environment set up for building the git master yet, so it might take a while. I'll look to see if there is a newer version in brew that can be installed on RHEL-8. Then, that would be easiest.
Comment 4 Qian Cai 2021-01-14 14:16:58 UTC 
Same thing with podman-3.0.0-6.module+el8.4.0+9088+ac8dc0c6.x86_64

ERRO[0016] Error deleting network: running [/usr/sbin/ip6tables -t nat -D POSTROUTING -s 2601:601:9f80:3c4f::2/64 -j CNI-2a33fd25b62f81040709553e -m comment --comment name: "podman" id: "dcaaf5d43d08a2a78c74e37c15cc4a398dfde2d8f8414f5f62ebc41977418721" --wait]: exit status 1: iptables: Bad rule (does a matching rule exist in that chain?). 
ERRO[0016] Error while removing pod from CNI network "podman": running [/usr/sbin/ip6tables -t nat -D POSTROUTING -s 2601:601:9f80:3c4f::2/64 -j CNI-2a33fd25b62f81040709553e -m comment --comment name: "podman" id: "dcaaf5d43d08a2a78c74e37c15cc4a398dfde2d8f8414f5f62ebc41977418721" --wait]: exit status 1: iptables: Bad rule (does a matching rule exist in that chain?). 
ERRO[0016] unable to cleanup network for container dcaaf5d43d08a2a78c74e37c15cc4a398dfde2d8f8414f5f62ebc41977418721: "error tearing down CNI namespace configuration for container dcaaf5d43d08a2a78c74e37c15cc4a398dfde2d8f8414f5f62ebc41977418721: running [/usr/sbin/ip6tables -t nat -D POSTROUTING -s 2601:601:9f80:3c4f::2/64 -j CNI-2a33fd25b62f81040709553e -m comment --comment name: \"podman\" id: \"dcaaf5d43d08a2a78c74e37c15cc4a398dfde2d8f8414f5f62ebc41977418721\" --wait]: exit status 1: iptables: Bad rule (does a matching rule exist in that chain?).\n"
Comment 5 Qian Cai 2021-01-14 21:11:36 UTC 
crictl looks like the same thing.

# crictl stopp a382307204515
FATA[0000] stopping the pod sandbox "a382307204515": rpc error: code = Unknown desc = failed to destroy network for pod sandbox k8s_ubi8-sandbox_default_hdishd83djaidwnduwk28bcsb_1(a38230720451583ade7c0b84fa7d08663e3417b52ad2ffadca9b553fb36d82d3): running [/usr/sbin/ip6tables -t nat -D POSTROUTING -s 1100:200::5/24 -j CNI-7860ea922d1a46225b9c13c1 -m comment --comment name: "crio" id: "a38230720451583ade7c0b84fa7d08663e3417b52ad2ffadca9b553fb36d82d3" --wait]: exit status 1: iptables: Bad rule (does a matching rule exist in that chain?).

cri-o-1.20.0-0.rhaos4.7.git845747f.el8.40.x86_64
cri-tools-1.20.0-1.el8.x86_64
Comment 6 Daniel Walsh 2021-01-28 21:25:46 UTC 
Michael, any ideas?
Comment 7 mcambria@redhat.com 2021-01-28 22:45:56 UTC 
Is the cni firewall plugin missing from /etc/cni/net.d/87-podman-bridge.conflist ?

I only see a diff.

Does the firewall plugin supply the right backend for the installed OS?   IIRC this was needed at one point.


Is --runtime=kata sending CNI the firewall plugin?  

Look at what I ran below (fedora 32) for differences in e.g. configuration.  Suggest identifying diffs in config syntax etc. getting things to work without --runtime==kata.


[mcc@snark ebpf-stuff]$ sudo podman run --rm --network=ipv6test  -it alpine sh
/ # ip addr show
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
3: eth0@if25: <BROADCAST,MULTICAST,UP,LOWER_UP,M-DOWN> mtu 1500 qdisc noqueue state UP 
    link/ether aa:2d:81:7a:a5:9a brd ff:ff:ff:ff:ff:ff
    inet 10.66.0.6/16 brd 10.66.255.255 scope global eth0
       valid_lft forever preferred_lft forever
    inet6 fd00::1:8:9/112 scope global 
       valid_lft forever preferred_lft forever
    inet6 fe80::a82d:81ff:fe7a:a59a/64 scope link 
       valid_lft forever preferred_lft forever
/ # exit
[mcc@snark ebpf-stuff]$ 
[mcc@snark ebpf-stuff]$ sudo cat /etc/cni/net.d/666-podman-ipv6test.conflist
{
    "cniVersion": "0.4.0",
    "name": "ipv6test",
    "plugins": [
        {
            "type": "bridge",
            "bridge": "v6test0",
            "isGateway": true,
            "ipMasq": true,
            "hairpinMode": true,
            "ipam": {
                "type": "host-local",
                "routes": [
                    {
                        "dst": "0.0.0.0/0"
                    },
                    {
                        "dst": "::/0"
                    }
                ],
                "ranges": [
                    [
                        {
                            "subnet": "10.66.0.0/16",
                            "gateway": "10.66.0.1"
                        }
                    ],
                    [
                        {
                            "subnet": "fd00::1:8:0/112",
                            "gateway": "fd00::1:8:1" 
                        }
                    ]
                ]
            }
        },
        {
            "type": "portmap",
            "capabilities": {
                "portMappings": true
            }
        },
        {
            "type": "firewall",
            "backend": ""
        },
        {
            "type": "tuning"
        }
    ]
}
[mcc@snark ebpf-stuff]$
Comment 8 Qian Cai 2021-02-24 18:25:46 UTC 
I believe this is the same CNI plugin issue.

*** This bug has been marked as a duplicate of bug 1918417 ***
Comment 9 Matthew Heon 2021-04-29 14:01:29 UTC 
*** Bug 1954844 has been marked as a duplicate of this bug. ***
Note You need to log in before you can comment on or make changes to this bug.