Describe the issue: Customer wants to add a similar statement to the following in our doc; "The user must specify --node-ip in kubelet.service using MCO if they want to select other than first nic for OCP internal communications." Additional information: Related to https://bugzilla.redhat.com/show_bug.cgi?id=1920282#c16
@fshaikh is there any obstacle to recording the information in the KCS? Regarding product doc: * The information that Alex provided is for 4.5 only. * A single sentence some place in the product doc that mentions a kubelet command-line argument seems out of place for the majority of our customers. It seems to me that while customers have been confused more than once--and that is regrettable--the solution also seems customer-specific. I'm reluctant to find a location to squeeze in a single sentence because it does not seem likely to help the majority of customers. I feel like customers had a specific need, contacted RH support and received the support they needed. I don't see how a doc update helps most customers.
Dear Mike, The requests was raised from us, so maybe we should comment a concrete background of it. Our customer is using their OpenShift on their vSphere cluster, and many other VMs are also running there. They want to keep the first nic for managing/monitoring VMs from vSphere cluster's admin. They don't want to use the first nic for other purposes. There is no technical hurdle, because kubelet itself can specify a nic with --node-ip parameter. (And we already know how to do that as Comment 8 of Bug 1920282) So the MOST and ONLY important thing for us is whether Red Hat will support it or not. > I feel like customers had a specific need, contacted RH support and received the support they needed. Red Hat support team always says that Red Hat will support only things written in OpenShift manual. So, we asked Red Hat to write it in OpenShift manual as an evidence which indicates that Red Hat really supports it. We believe that to specify other than first nic is not a rare case. We could do such thing in OpenShift3. And kublet has --node-ip parameter for that. Best Regards, Masaki Hatada
We do not support overwriting the kubelet.service systemd unit generated by MCO. But on platforms where kubelet.service starts kubelet with --node-ip=$KUBELET_NODE_IP we do support overriding the autodetected KUBELET_NODE_IP value. OCP writes its value out to /etc/systemd/system/kubelet.service.d/20-nodenet.conf, so you can override it by writing a new value to a file with a higher number (eg /etc/systemd/system/kubelet.service.d/80-node-ip-local.conf) We do not support overwriting or deleting the existing "20-nodenet.conf" file; you have to write out a new file, which has a name that will cause it to be read after the OCP-generated file, and then run "systemctl daemon-reload" to ensure systemd sees the new file. In 4.6 and later, this should work on all metal/on-prem platforms (ie, anything but public clouds). I think in 4.5 it's not quite as widely available. (You can check by looking at /etc/systemd/system/kubelet.service to see if it has the "--node-ip=$KUBELET_NODE_IP" line.) Note that OCP does not support using different network interfaces for different OCP-internal purposes (eg using one interface for node-to-master traffic and a different interface for VXLAN traffic between nodes). This is only for picking the single network interface that will be used for all OCP communication, not for trying to separate out some OCP communication from others.
Huge thanks to Dan Winship and my patient support colleagues on this BZ. Question: If a different NIC is selected and prevents OCP communications, is it even possible to deliver a systemd unit file with MCO? Does that need to be acknowledged? The content might get moved around, but the information is reviewable at https://github.com/openshift/openshift-docs/pull/32693.
(In reply to Mike McKiernan from comment #6) > Question: If a different NIC is selected and prevents OCP communications, is > it even possible to deliver a systemd unit file with MCO? Does that need to > be acknowledged? If the admin does something like tell a worker node to use an IP on a subnet that the masters aren't connected to, then yes, that would totally break them, but I feel like that's kind of obvious and shouldn't need to be said. (There are lots of things a sufficiently perverse admin could do to break things, and we couldn't possibly list all of them...) The "trying to use multiple NICs" thing is mostly in the context of people wanting to separate out either pod-to-pod or pod-to-external traffic (or both) onto a different interface from OCP-internal node-to-node traffic, but the worst you can do there is make it so that pod-to-pod traffic to that node would no longer work, which wouldn't block traffic to the machine-config-daemon since that would be host-network.
Dear Dan, Thank you for your reply. > we do support overriding the autodetected KUBELET_NODE_IP value. OCP writes its value out to /etc/systemd/system/kubelet.service.d/20-nodenet.conf, so you can override it by writing a new value to a file with a higher number (eg /etc/systemd/system/kubelet.service.d/80-node-ip-local.conf) We do not support overwriting or deleting the existing "20-nodenet.conf" file; you have to write out a new file, which has a name that will cause it to be read after the OCP-generated file, and then run "systemctl daemon-reload" to ensure systemd sees the new file. > > In 4.6 and later, this should work on all metal/on-prem platforms (ie, anything but public clouds). I think in 4.5 it's not quite as widely available. (You can check by looking at /etc/systemd/system/kubelet.service to see if it has the "--node-ip=$KUBELET_NODE_IP" line.) First we will upgrade our customer's cluster to 4.6 before doing that. > Note that OCP does not support using different network interfaces for different OCP-internal purposes (eg using one interface for node-to-master traffic and a different interface for VXLAN traffic between nodes). This is only for picking the single network interface that will be used for all OCP communication, not for trying to separate out some OCP communication from others. Ok. We have never received such a request. Our customers want to separate OCP-internal communication from communications of: - Managing VMs (Login from outside for trouble shooting, Healthcheck from an external monitoring system, etc) - Storage data (Read/write from/to external storage) Best Regards, Masaki Hatada
Dear Mark, Thank you for your information. > The content might get moved around, but the information is reviewable at https://github.com/openshift/openshift-docs/pull/32693. The above is good for us. Yes, it would cause some trouble if customer did some mistake when installing a systemd unit file with MCO. It would be very good if Red Hat could introduce some example of MC definition in the manual, but the most important thing for us is that Red Hat clearly describes the supportability of overwriting KUBELET_NODE_IP in the manual. Maybe Red Hat can describe in the manual that user must be careful when overwriting KUBELET_NODE_IP and must evaluate it in test env before moving it to production env. Then we will follow the description. Best Regards, Masaki Hatada
@zzhao, PTAL at the PR and let me know if it looks good to you or if you would like me to revise: https://github.com/openshift/openshift-docs/pull/32693
Dear Dan, I got a similar issue where Customer has upgraded the cluster from 4.5 to 4.6 and it doesn't have any "--node-ip=$KUBELET_NODE_IP" line in kubelet.service file. I tried editing the same manually but it got reverted back as the render config doesn't have that parameter and gives config mismatch error. Although nodes get the required IP if I do all steps manually but I am afraid it will again go degraded mode if any node got a restart or cluster is upgraded in near future. Is there any other solution for 4.5 upgraded cluster? Best Regards. Pawan Kumar
(In reply to pawankum from comment #11) > Dear Dan, > > I got a similar issue where Customer has upgraded the cluster from 4.5 to > 4.6 and it doesn't have any "--node-ip=$KUBELET_NODE_IP" line in > kubelet.service file. If there is no "--node-ip=$KUBELET_NODE_IP" in the kubelet.service then either (a) they are on a public cloud platform and there is no way to override the node IP because the cloud will tell us what IP to use, or (b) they are on ovirt, and there's a bug somewhere about making it possible to override the node IP on ovirt in 4.6. (It's already possible in 4.7.)
(In reply to Dan Winship from comment #12) > (In reply to pawankum from comment #11) > > Dear Dan, > > > > I got a similar issue where Customer has upgraded the cluster from 4.5 to > > 4.6 and it doesn't have any "--node-ip=$KUBELET_NODE_IP" line in > > kubelet.service file. > > If there is no "--node-ip=$KUBELET_NODE_IP" in the kubelet.service then > either (a) they are on a public cloud platform and there is no way to > override the node IP because the cloud will tell us what IP to use, or (b) > they are on ovirt, and there's a bug somewhere about making it possible to > override the node IP on ovirt in 4.6. (It's already possible in 4.7.) Hello Dan, Customer is not using any cloud platform, they are using Vsphere. BR, Pawan
Product documentation is updated: https://docs.openshift.com/container-platform/4.6/support/troubleshooting/troubleshooting-network-issues.html
The needinfo request[s] on this closed bug have been removed as they have been unresolved for 500 days