Bug 2214112
| Summary: | ironic_pxe_tftp container is not starting after disabling ipv6 | ||
|---|---|---|---|
| Product: | Red Hat OpenStack | Reporter: | Jeremy <jmelvin> |
| Component: | openstack-ironic | Assignee: | OSP Team <rhos-maint> |
| Status: | CLOSED NOTABUG | QA Contact: | |
| Severity: | medium | Docs Contact: | |
| Priority: | unspecified | ||
| Version: | 16.2 (Train) | CC: | hjensas, pweeks, sbaker, tkajinam |
| Target Milestone: | --- | Keywords: | Triaged |
| Target Release: | --- | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | Doc Type: | If docs needed, set a value | |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2023-07-24 19:42:47 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
I reproduced this, and when I disable ipv6 using the kernel cmdline I can see the tftp is not running. In additon to tftp not running many container health check services are also failing - for example: Jun 14 14:19:29 undercloud.lab.example.com healthcheck_nova_conductor[26334]: awk: cmd. line:1: fatal: cannot open file `/proc/net/tcp6' for reading (No such file or directory) If I remove "ipv6.disable=1" from kernel cmdline all the issues are resolved. When disabling IPv6 using sysctl setting i.e option #1 in https://access.redhat.com/solutions/8709 I don't see any issues. The ExtraSysctlSettings THT parameter should be used set the sysctl options to ensure OSP director does not override them again. Thanks, the customer said they tried to disable IPv6 using different method and it is not affecting tftp container. Working now.
Updated the /etc/sysctl.d/99-tripleo.conf file, the below parameters
net.ipv6.conf.all.disable_ipv6=1
net.ipv6.conf.default.disable_ipv6=1
net.ipv6.conf.lo.disable_ipv6=1
I assume we set ExtraSysctlSettings in the undercloud.conf customer environment like [1] , do you know the syntax?
Like this maybe?
##custom-undercloud-params.yaml
parameter_defaults:
ExtraSysctlSettings:
net.ipv6.conf.all.disable_ipv6=1
net.ipv6.conf.default.disable_ipv6=1
net.ipv6.conf.lo.disable_ipv6=1
[1]
https://access.redhat.com/documentation/en-us/red_hat_openstack_platform/16.2/html-single/director_installation_and_usage/index#configuring-the-undercloud-with-environment-files
(In reply to Jeremy from comment #4) > Thanks, the customer said they tried to disable IPv6 using different method > and it is not affecting tftp container. Working now. > > Updated the /etc/sysctl.d/99-tripleo.conf file, the below parameters > net.ipv6.conf.all.disable_ipv6=1 > net.ipv6.conf.default.disable_ipv6=1 > net.ipv6.conf.lo.disable_ipv6=1 > > > I assume we set ExtraSysctlSettings in the undercloud.conf customer > environment like [1] , do you know the syntax? > Yes [1] is correct, the custom_env_files paramter in undercloud.conf point to a heat env file. > Like this maybe? > > ##custom-undercloud-params.yaml > parameter_defaults: > ExtraSysctlSettings: > net.ipv6.conf.all.disable_ipv6=1 > net.ipv6.conf.default.disable_ipv6=1 > net.ipv6.conf.lo.disable_ipv6=1 > I believe it needs to be like this: parameter_defaults: ExtraSysctlSettings: net.ipv6.conf.all.disable_ipv6: value: 1 net.ipv6.conf.default.disable_ipv6: value: 1 net.ipv6.conf.lo.disable_ipv6: value: 1 I.e the same format used for the RHOSP defaults see: https://opendev.org/openstack/tripleo-heat-templates/src/branch/master/deployment/kernel/kernel-baremetal-ansible.yaml#L166-L206 > [1] > https://access.redhat.com/documentation/en-us/red_hat_openstack_platform/16. > 2/html-single/director_installation_and_usage/index#configuring-the- > undercloud-with-environment-files I remember that we faced some problems caused by missing ::1 (eg. bz 1590602) and we intentionally enabled IPv6 address for lo device, so you might need net.ipv6.conf.lo.disable_ipv6=0 instead of 1. There is a tht parameter to inject these settings (KernelDisableIPv6: 1) so you can use it instead of ExtraSysctlSettings if enabling IPv6 for lo is acceptable. As a side note, the failure in healtcheck looks legit, but probably does not occur in case kernel parameters instead of kernel args are used. In case it reproduces with kernel parameters then I'd suggest opening a separate bug. No action currently required from us, so closing this for now. It can be reopened if necessary. |
Description of problem: Customer disabled ipv6 in the environment for security concerns. Now ironic-pxe will not start. Would like to get this working with ipv4 so we can disable ipv6 Jun 9 13:44:34 osdirwlkzl101 in.tftpd[11]: cannot open IPv6 socket, disable IPv6: Address family not supported by protocol Jun 9 13:44:34 osdirwlkzl101 in.tftpd[11]: Cannot set nonblock flag on socket: Bad file descriptor Version-Release number of selected component (if applicable): 16.2 How reproducible: 100% Steps to Reproduce: 1. disable ipv6 2. notice ironic-pxe not working 3. Actual results: ironic-pxe container not working Expected results: ironic-pxe container works without ipv6 Additional info: Jun 9 13:44:34 osdirwlkzl101 podman[4076]: 2023-06-09 13:44:34.124952198 -0400 EDT m=+0.348997075 container init d778ef1aed79607e72f7b98cfdf130715b6e5203fa7d6314108b4609c6e4a120 (image=10.214.14.10:8787/rhosp-rhel8/openstack-ironic-pxe:16.2.1, name=ironic_pxe_tftp, name=rhosp16/openstack-ironic-pxe, vendor=Red Hat, Inc., config_data={"command": ["/bin/bash", "-c", "BIND_HOST=$(hiera ironic::pxe::tftp_bind_host -c /etc/puppet/hiera.yaml); /usr/sbin/in.tftpd --foreground --user root --address $BIND_HOST:69 --map-file /var/lib/ironic/tftpboot/map-file /var/lib/ironic/tftpboot"], "environment": {"KOLLA_CONFIG_STRATEGY": "COPY_ALWAYS", "TRIPLEO_CONFIG_HASH": "405d90670b4105d6081825946f043cff"}, "healthcheck": {"test": "/openstack/healthcheck"}, "image": "10.214.14.10:8787/rhosp-rhel8/openstack-ironic-pxe:16.2.1", "net": "host", "privileged": false, "restart": "always", "start_order": 90, "volumes": ["/etc/hosts:/etc/hosts:ro", "/etc/localtime:/etc/localtime:ro", "/etc/pki/ca-trust/extracted:/etc/pki/ca-trust/extracted:ro", "/etc/pki/ca-trust/source/anchors:/etc/pki/ca-trust/source/anchors:ro", "/etc/pki/tls/certs/ca-bundle.crt:/etc/pki/tls/certs/ca-bundle.crt:ro", "/etc/pki/tls/certs/ca-bundle.trust.crt:/etc/pki/tls/certs/ca-bundle.trust.crt:ro", "/etc/pki/tls/cert.pem:/etc/pki/tls/cert.pem:ro", "/dev/log:/dev/log", "/etc/puppet:/etc/puppet:ro", "/var/lib/kolla/config_files/ironic_pxe_tftp.json:/var/lib/kolla/config_files/config.json:ro", "/var/lib/config-data/puppet-generated/ironic:/var/lib/kolla/config_files/src:ro", "/var/lib/ironic:/var/lib/ironic/:shared,z", "/var/log/containers/ironic:/var/log/ironic:z", "/var/log/containers/httpd/ironic-pxe:/var/log/httpd:z"]}, io.k8s.description=Red Hat OpenStack Platform 16.2 ironic-pxe, maintainer=OpenStack TripleO team, url=https://access.redhat.com/containers/#/registry.access.redhat.com/rhosp16/openstack-ironic-pxe/images/16.2.1-6.1645706354, batch=16.2_20211202.1, version=16.2.1, managed_by=tripleo-Undercloud, tcib_managed=true, container_name=ironic_pxe_tftp, build-date=2022-02-24T12:55:56.202429, vcs-ref=9604fc10484224856767f687d3dd48b0f127d64f, vcs-type=git, release=6.1645706354, com.redhat.license_terms=https://www.redhat.com/agreements, com.redhat.component=openstack-ironic-pxe-container, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, config_id=tripleo_step4, description=Red Hat OpenStack Platform 16.2 ironic-pxe, distribution-scope=public, architecture=x86_64, summary=Red Hat OpenStack Platform 16.2 ironic-pxe, io.k8s.display-name=Red Hat OpenStack Platform 16.2 ironic-pxe, io.openshift.tags=rhosp osp openstack osp-16.2, io.openshift.expose-services=) Jun 9 13:44:34 osdirwlkzl101 podman[4076]: 2023-06-09 13:44:34.147424638 -0400 EDT m=+0.371469501 container start d778ef1aed79607e72f7b98cfdf130715b6e5203fa7d6314108b4609c6e4a120 (image=10.214.14.10:8787/rhosp-rhel8/openstack-ironic-pxe:16.2.1, name=ironic_pxe_tftp, com.redhat.component=openstack-ironic-pxe-container, io.k8s.description=Red Hat OpenStack Platform 16.2 ironic-pxe, io.openshift.tags=rhosp osp openstack osp-16.2, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, config_id=tripleo_step4, vcs-type=git, architecture=x86_64, com.redhat.license_terms=https://www.redhat.com/agreements, version=16.2.1, config_data={"command": ["/bin/bash", "-c", "BIND_HOST=$(hiera ironic::pxe::tftp_bind_host -c /etc/puppet/hiera.yaml); /usr/sbin/in.tftpd --foreground --user root --address $BIND_HOST:69 --map-file /var/lib/ironic/tftpboot/map-file /var/lib/ironic/tftpboot"], "environment": {"KOLLA_CONFIG_STRATEGY": "COPY_ALWAYS", "TRIPLEO_CONFIG_HASH": "405d90670b4105d6081825946f043cff"}, "healthcheck": {"test": "/openstack/healthcheck"}, "image": "10.214.14.10:8787/rhosp-rhel8/openstack-ironic-pxe:16.2.1", "net": "host", "privileged": false, "restart": "always", "start_order": 90, "volumes": ["/etc/hosts:/etc/hosts:ro", "/etc/localtime:/etc/localtime:ro", "/etc/pki/ca-trust/extracted:/etc/pki/ca-trust/extracted:ro", "/etc/pki/ca-trust/source/anchors:/etc/pki/ca-trust/source/anchors:ro", "/etc/pki/tls/certs/ca-bundle.crt:/etc/pki/tls/certs/ca-bundle.crt:ro", "/etc/pki/tls/certs/ca-bundle.trust.crt:/etc/pki/tls/certs/ca-bundle.trust.crt:ro", "/etc/pki/tls/cert.pem:/etc/pki/tls/cert.pem:ro", "/dev/log:/dev/log", "/etc/puppet:/etc/puppet:ro", "/var/lib/kolla/config_files/ironic_pxe_tftp.json:/var/lib/kolla/config_files/config.json:ro", "/var/lib/config-data/puppet-generated/ironic:/var/lib/kolla/config_files/src:ro", "/var/lib/ironic:/var/lib/ironic/:shared,z", "/var/log/containers/ironic:/var/log/ironic:z", "/var/log/containers/httpd/ironic-pxe:/var/log/httpd:z"]}, build-date=2022-02-24T12:55:56.202429, container_name=ironic_pxe_tftp, description=Red Hat OpenStack Platform 16.2 ironic-pxe, distribution-scope=public, batch=16.2_20211202.1, name=rhosp16/openstack-ironic-pxe, io.k8s.display-name=Red Hat OpenStack Platform 16.2 ironic-pxe, tcib_managed=true, url=https://access.redhat.com/containers/#/registry.access.redhat.com/rhosp16/openstack-ironic-pxe/images/16.2.1-6.1645706354, release=6.1645706354, managed_by=tripleo-Undercloud, summary=Red Hat OpenStack Platform 16.2 ironic-pxe, maintainer=OpenStack TripleO team, vcs-ref=9604fc10484224856767f687d3dd48b0f127d64f, io.openshift.expose-services=, vendor=Red Hat, Inc.) Jun 9 13:44:34 osdirwlkzl101 in.tftpd[11]: cannot open IPv6 socket, disable IPv6: Address family not supported by protocol Jun 9 13:44:34 osdirwlkzl101 in.tftpd[11]: Cannot set nonblock flag on socket: Bad file descriptor ###looks like it should use ipv4 bind_host ? but doesn't [root@osdirwlkzl101 ~]# hiera ironic::pxe::tftp_bind_host 10.214.14.10