Hide Forgot
When setting rp_filter and log martian on Openshift 3.2 Node via sysctl.conf: net.ipv4.conf.all.log_martians = 1 net.ipv4.conf.all.rp_filter = 1 Then in system log we see: .779186] ll header: 00000000: 44 1e a1 3e 8a 0a 8c 60 4f 31 45 42 08 00 D..>...`O1EB.. [1308931.779191] ll header: 00000000: 44 1e a1 3e 8a 0a 8c 60 4f 31 44 42 08 00 D..>...`O1DB.. [1308931.784013] IPv4: martian source 44.47.102.107 from 10.121.242.6, on dev enp3s0f0 [1308931.784034] ll header: 00000000: 44 1e a1 3e 8a 0a 8c 60 4f 31 45 42 08 00 D..>...`O1EB.. [1308931.784043] IPv4: martian source 44.47.102.107 from 10.121.242.6, on dev enp3s0f0 [1308931.784051] ll header: 00000000: 44 1e a1 3e 8a 0a 8c 60 4f 31 45 42 08 00 D..>...`O1EB.. [1308931.784101] IPv4: martian source 44.47.102.107 from 10.121.242.6, on dev enp3s0f0 [1308931.784105] ll header: 00000000: 44 1e a1 3e 8a 0a 8c 60 4f 31 45 42 08 00 D..>...`O1EB.. [1308936.757623] IPv4: martian source 44.47.102.107 from 10.121.242.6, on dev enp3s0f0 [1308936.757675] ll header: 00000000: 44 1e a1 3e 8a 0a 8c 60 4f 31 45 42 08 00 D..>...`O1EB.. [1308936.757681] IPv4: martian source 44.47.102.107 from 10.121.242.6, on dev enp3s0f0 [1308936.757684] ll header: 00000000: 44 1e a1 3e 8a 0a 8c 60 4f 31 45 42 08 00 D..>...`O1EB.. [1308941.749595] IPv4: martian source 44.47.102.107 from 10.121.242.6, on dev enp3s0f0 [1308941.749605] ll header: 00000000: 44 1e a1 3e 8a 0a 8c 60 4f 31 45 42 08 00 D..>...`O1EB.. [1308941.749612] IPv4: martian source 44.47.102.107 from 10.121.242.6, on dev enp3s0f0 [1308941.749615] ll header: 00000000: 44 1e a1 3e 8a 0a 8c 60 4f 31 45 42 08 00 D..>...`O1EB.. [1308941.767807] IPv4: martian source 44.47.102.107 from 10.121.242.6, on dev enp3s0f0 [1308941.767811] ll header: 00000000: 44 1e a1 3e 8a 0a 8c 60 4f 31 45 42 08 00 D..>...`O1EB.. [1308941.767816] IPv4: martian source 44.47.102.107 from 10.121.242.6, on dev enp3s0f0 [1308941.767817] ll header: 00000000: 44 1e a1 3e 8a 0a 8c 60 4f 31 45 42 08 00 D..>...`O1EB.. [1308941.767977] IPv4: martian source 44.47.102.107 from 10.121.242.6, on dev enp3s0f0 [1308941.768009] ll header: 00000000: 44 1e a1 3e 8a 0a 8c 60 4f 31 45 42 08 00 D..>...`O1EB.. [1308941.768016] IPv4: martian source 44.47.102.107 from 10.121.242.6, on dev enp3s0f0 [1308941.768019] ll header: 00000000: 44 1e a1 3e 8a 0a 8c 60 4f 31 45 42 08 00 D..>...`O1EB.. [1308951.764442] IPv4: martian source 44.47.102.107 from 10.121.242.6, on dev enp3s0f0 [1308951.764467] ll header: 00000000: 44 1e a1 3e 8a 0a 8c 60 4f 31 44 42 08 00 D..>...`O1DB.. [1308956.731573] IPv4: martian source 44.47.102.107 from 10.121.242.6, on dev enp3s0f0 [1308956.731601] ll header: 00000000: 44 1e a1 3e 8a 0a 8c 60 4f 31 45 42 08 00 D..>...`O1EB.. [1308956.739746] IPv4: martian source 44.47.102.107 from 10.121.242.6, on dev enp3s0f0 [1308956.739766] ll header: 00000000: 44 1e a1 3e 8a 0a 8c 60 4f 31 44 42 08 00 D..>...`O1DB.. [1308959.569266] SELinux: initialized (dev tmpfs, type tmpfs), uses transition SIDs [1308966.729187] IPv4: martian source 44.47.102.107 from 10.121.242.6, on dev enp3s0f0 [1308966.729196] ll header: 00000000: 44 1e a1 3e 8a 0a 8c 60 4f 31 45 42 08 00 D..>...`O1EB.. [1308976.737304] IPv4: martian source 44.47.102.107 from 10.121.242.6, on dev enp3s0f0 [1308976.737318] ll header: 00000000: 44 1e a1 3e 8a 0a 8c 60 4f 31 44 42 08 00 D..>...`O1DB.. [1308976.737326] IPv4: martian source 44.47.102.107 from 10.121.242.6, on dev enp3s0f0 [1308976.737329] ll header: 00000000: 44 1e a1 3e 8a 0a 8c 60 4f 31 44 42 08 00 D..>...`O1DB.. This all seems to come from heapster. May be Heapster is accessing nodes on external interface instead of using internal cluster address?
Ok, sross managed to reproduce this with the heapster container. Since the SDN IPs are Matrian (from the reserved private IP ranges) they set off the alert when they are targetted to the host public address. We are still investigating to see if this message is avoidable, or if it will always happen when private IP ranges are used for the SDN.
Log_martian setting only log them but It also means those packets are dropped by the rp_filter.... This is the default on RHEL6 and RHEL7: https://access.redhat.com/solutions/53031 https://access.redhat.com/solutions/32261 What is actually the effect on heapster? Let supposes you have 100 nodes.... It will at least make heapster busy as I guess it retry..... I see around 3 packets / second rejected on each node.
I'd advise you disable the martian logging for now. Given that we are using the private IP ranges deliberately, then it is just going to cause trouble. We'll work on getting more concrete advice.
We need to work out what in particular is triggering this. And see if we can change the rules so that it doesn't, or see if a different subnet choice makes a difference. Then we need to document our needs (probably in a new hardening doc that refers to the RHEL one, but adds this as an exception if needed).
When net.ipv4.conf.all.log_martians = 1 and net.ipv4.conf.all.rp_filter = 1 setup, when update clusterNetworkCIDR and serviceNetworkCIDR to the new ip address range, the log messages start to show up martian source information, after deploy Heapster pod, more martian source information show up. I can duplicate this issue in openshift v3.2 and v3.3, no martian source information show up from v3.4 in my setup.
Based on Weibin's findings I'm closing this since it works in 3.4.
*** Bug 1393815 has been marked as a duplicate of this bug. ***