I found out that this require specific conntrack helper module to be loaded and configured in the router.
And Neutron has got API for that already. See RFE https://bugs.launchpad.net/neutron/+bug/1823633 and API ref https://docs.openstack.org/api-ref/network/v2/?expanded=create-conntrack-helper-detail#routers-conntrack-helper-ct-target-rules

So it has to be enabled in the neutron server (service_plugin conntrack_helper) and in the L3 agent (extension conntrack_helper). With those change new API should be available and You can enable such helper with API request like:

curl -X POST http://10.0.0.115:9696/v2.0/routers/26284f88-daf0-4dbb-9564-2263edd4445a/conntrack_helpers -H "Content-Type: application/json" -H "User-Agent: openstacksdk/0.36.5 keystoneauth1/3.17.3 python-requests/2.20.0 CPython/3.6.8" -H                              
 "X-Auth-Token: $token" -d '{"conntrack_helper": {"protocol": "udp", "port": 69, "helper": "tftp"}}'


Unfortunatelly we don't have OSC support for that API yet. I will propose it u/s and will track it in that BZ.

Ahh, one more thing - module "nf_nat_tftp" has to be loaded on the nodes where router is running to make that helper working. And that will not be done by Neutron so You need to load it on Your own.

Hi Slawek,

Thanks for taking a look at this issue.  If I understand you correctly, the following manual steps should enable conntrack to handle tftp traffic?

- Enable the service plugin in neutron.conf (neutron server & l3 agent)

# crudini --get /var/lib/config-data/puppet-generated/neutron/etc/neutron/neutron.conf DEFAULT service_plugins
router,qos,segments,trunk,port_forwarding,conntrack_helper

- Restart neutron api and l3 agents.

podman restart neutron_l3_agent
podman restart neutron_api

- With debug enabled we see the service plugins loaded?

# grep service_plugins /var/log/containers/neutron/server.log
2021-03-25 15:13:44.089 6 DEBUG oslo_service.service [-] service_plugins                = ['router', 'qos', 'segments', 'trunk', 'port_forwarding', 'conntrack_helper'] log_opt_values /usr/lib/python3.6/site-packages/oslo_config/cfg.py:2581

# grep service_plugins /var/log/containers/neutron/l3-agent.log
2021-03-25 15:52:53.022 461782 DEBUG neutron.wsgi [-] service_plugins                = ['router', 'qos', 'segments', 'trunk', 'port_forwarding', 'conntrack_helper'] log_opt_values /usr/lib/python3.6/site-packages/oslo_config/cfg.py:2581
2021-03-25 15:52:54.639 461782 DEBUG oslo_service.service [req-32a211f7-a108-4aff-b5c2-12e39272dbe0 - - - - -] service_plugins                = ['router', 'qos', 'segments', 'trunk', 'port_forwarding', 'conntrack_helper'] log_opt_values /usr/lib/python3.6/site-packages/oslo_config/cfg.py:2581


- enable the nf_nat_tftp kernel module on the l3 agents.

# modprobe nf_nat_tftp
# lsmod |grep tftp
nf_nat_tftp            16384  0
nf_conntrack_tftp      16384  1 nf_nat_tftp
nf_nat                 36864  6 nf_nat_ipv6,nf_nat_ipv4,xt_nat,nf_nat_tftp,openvswitch,xt_REDIRECT
nf_conntrack          155648  16 xt_conntrack,nf_conntrack_ipv6,nf_conntrack_ipv4,nf_nat,nf_conntrack_tftp,xt_state,nfnetlink_cttimeout,nf_nat_ipv6,nf_nat_ipv4,xt_nat,nf_nat_tftp,openvswitch,nf_conntrack_netlink,xt_connmark,nf_conncount,xt_REDIRECT


- enable tftp helper for router


$ curl -X POST http://192.168.2.226:9696/v2.0/routers/dc476d0e-62b2-46c6-b141-3b112ebfbf26/conntrack_helpers -H "Content-Type: application/json" -H "Use
r-Agent: openstacksdk/0.36.5 keystoneauth1/3.17.3 python-requests/2.20.0 CPython/3.6.8" -H "X-Auth-Token: $token" -d '{"conntrack_helper": {"protocol": "udp", "port": 69, "helper": "tftp
"}}'
{"conntrack_helper": {"id": "032ea860-959d-46ea-9163-3f42b9935b36", "protocol": "udp", "port": 69, "helper": "tftp"}}


- in the l3 agent log we see the helper config for the router:

/var/log/containers/neutron/l3-agent.log
2021-03-25 15:53:19.199 461782 DEBUG neutron.agent.l3.agent [req-9aca361d-ba68-4e6a-8d62-29ae07deb99f - - - - -] Processing :[{'id': 'dc476d0e-62b2-46c6-b141-3b112ebfbf26', 'name': 'router1', 'tenant_id': '03beefe4d1cc439fa64fae5d99c1f258', 'admin_state_up': True, 'status': 'ACTIVE', 'external_gateway_info': {'network_id': '6c2dfa55-0253-4559-805d-735fcdae24f3', 'external_fixed_ips': [{'subnet_id': '229e54c7-58fa-439d-8a9f-3c41bdbf29be', 'ip_address': '192.168.2.160'}], 'enable_snat': True}, 'gw_port_id': '6bc64863-8abf-4615-a4e8-40a703d5e429', 'description': '', 'availability_zones': ['nova'], 'distributed': False, 'ha': True, 'ha_vr_id': 222, 'availability_zone_hints': [], 'routes': [], 'flavor_id': None, 'conntrack_helpers': [{'protocol': 'udp', 'port': 69, 'helper': 'tftp'}], 'tags': [], 'created_at': '2021-03-08T18:18:18Z', 'updated_at': '2021-03-25T15:22:56Z', 'revision_number': 10, 'project_id': '03beefe4d1cc4
39fa64fae5d99c1f258', 'gw_port': {'id': '6bc64863-8abf-4615-a4e8-40a703d5e429', 'name': '', 'network_id': '6c2dfa55-0253-4559-805d-735fcdae24f3', 'tenant_id': '', 'mac_address': 'fa:16:3e:00:ae:e8', 'admin_state_up': True, 'status': 'ACTIVE', 'device_id': 'dc476d0e-62b2-46c6-b141-3b112ebfbf26', 'device_owner': 'network:router_gateway', 'fixed_ips': [{'subnet_id': '229e54c7-58fa-439d-8a9f-3c41bdbf29be', 'ip_add 

However tftp still doesn't seem to work through the router.

Is there more to the config?  Any advise to troubleshoot this would be great.

Thanks Slawek,

After setting the following:

# crudini --get /var/lib/config-data/puppet-generated/neutron/etc/neutron/l3_agent.ini  agent extensions
port_forwarding,conntrack_helper

I get the PREROUTING entry for tftp.

# ip netns exec qrouter-42057c7f-c5dc-449d-b86d-dbf02b2058f7 iptables -nvL -t raw

[...snip...]

Chain neutron-l3-agent-cth-4809eb7 (1 references)
 pkts bytes target     prot opt in     out     source               destination
    0     0 CT         udp  --  *      *       0.0.0.0/0            0.0.0.0/0            udp dpt:69 CT helper tftp
        

This works for tftp clients with security groups disabled which is odd.

Is there a way to allow this helper for all new routers by default?

No, there is no way to set it by default for all (new) routers. You need to add it for each router.

This config seems to work for an OSP 16.1 deployment.

Overcloud deployment parameters:

parameter_defaults:
  NeutronServicePlugins: 'router,qos,segments,trunk,port_forwarding,conntrack_helper'
  ControllerExtraConfig:
    neutron::agents::l3::extensions: conntrack_helper,port_forwarding
  ControllerParameters:
    ExtraKernelModules:
      nf_nat_tftp: {}


Post deployment, enable tftp conntrack helper for a router example:

$ openstack router list
+--------------------------------------+---------+--------+-------+----------------------------------+-------+
| ID                                   | Name    | Status | State | Project                          | HA    |
+--------------------------------------+---------+--------+-------+----------------------------------+-------+
| ad908033-eece-4c88-9b59-f9588657fc1c | router1 | ACTIVE | UP    | e1e667d5c61e4f7eb75731696b5b5648 | False |
+--------------------------------------+---------+--------+-------+----------------------------------+-------+


$ ROUTER=ad908033-eece-4c88-9b59-f9588657fc1c
$ token=$(openstack token issue -c id -f value)
$ neutronUrl=$(openstack catalog show neutron  -c endpoints -f json |jq -r '.endpoints[]  | select(.interface == "public") | .url')


$ curl -X POST $neutronUrl/v2.0/routers/$ROUTER/conntrack_helpers -H "Content-Type: application/json" -H "User-Agent: openstacksdk/0.36.5 keystoneauth1/3.17.3 python-requests/2.20.0 CPython/3.6.8" -H "X-Auth-Token: $token" -d '{"conntrack_helper": {"protocol": "udp", "port": 69, "helper": "tftp"}}'

{"conntrack_helper": {"id": "650d140f-a058-460f-b6d4-2fb0af6b7352", "protocol": "udp", "port": 69, "helper": "tftp"}}