Created attachment 987496 [details] Updating not bonded interface with IP 10.0.16.32 If an interface of a host, just assigned to a RHEL OSP deployment with Foreman (rhel-osp-installer/staypuft) is a bond device and contains one of the subnets of the deployment (e.g. Storage), Foreman won't save the new IP when changing it. The steps to reproduce are: 1. Add a host to a deployment which has multiple subnets (e.g. Storage, Management, Cluster Management...) 2. Change the network configuration of the host and: a) create a bond between 2 interfaces b) assign the subnets to the bond0 device just created 3. Go to the host then Edit->Network and then change the IP in question, for example the one in the Management subnet Expected result: The IP is saved and stored Actual result: Going back to editing the host the IP hasn't change. Reproducing the above steps on a host with an interface that's not bonded works as expected. We can see this in the production.log file when the interface is not bonded when updating the IP to 10.0.16.32 (0.3ms) UPDATE "nics" SET "ip" = '10.0.16.32', "updated_at" = '2015-02-03 10:22:50.518729', "attrs" = '--- {} We can't see the update if the interface is bonded. Attached both, logs for a bonded interface and logs for a non bonded interface. This is rhel-osp-installer A2: rhel-osp-installer-0.4.7-1.el6ost.noarch ruby193-rubygem-staypuft-0.4.14-1.el6ost.noarch foreman-1.6.0.44-6.el6ost.noarch
Created attachment 987497 [details] Updating bonded interface with IP 10.0.16.32
Marek, Any ideas here? This seems to reproduce on both OSP 5 and OSP 6, but the workflow being described seems to be purely in foreman rather than in staypuft. Thanks
According to log, it seems that interfaces attributes are not valid. Unfortunately Foreman ignores invalid interfaces resulting in not saving them at all. I see that there's one bond defined but no MAC was specified for it (should be one of MAC of assigned interfaces so either eno33557248 or eno50336512). Then I see other interfaces (e.g. 10.0.16.32) without MAC. Even VLAN interface should have MAC (same as the interface to which it's attached).
Marek, I have proposed a fix in Foreman for this. Thoughts? https://github.com/theforeman/foreman/pull/2142
I added comments in that PR.
On site at a partner (Fujitsu), hit the same bug. Confirmed workaround: setting the bond0.x MAC to one of the slave interfaces allows IP change.
This is fixed in https://github.com/theforeman/staypuft/pull/428
I tried it on A2 but puppet didnt changed it. I used my setup, tried to change an ip address of one hosts using "edit" window,I changed the ip on the bond interface and checked that the value is kept when I leave the window. after that i ran puppt but it couldn't manage to change it. puppet: Debug: /Stage[main]/Quickstack::Pacemaker::Rsync::Galera/Quickstack::Pacemaker::Rsync::Get[/etc/pki/galera]/Exec[rsync /etc/pki/galera]/returns: Sleeping for 10.0 seconds between tries Debug: /Stage[main]/Quickstack::Pacemaker::Rsync::Galera/Quickstack::Pacemaker::Rsync::Get[/etc/pki/galera]/Exec[rsync /etc/pki/galera]/returns: Exec try 244/360 Debug: Exec[rsync /etc/pki/galera](provider=posix): Executing 'rsync -q -aIX --delete rsync://10.35.174.188/galera/ /etc/pki/galera' ifconfig: before and after puppet bond0.189: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500 inet 10.35.174.34 netmask 255.255.255.0 broadcast 10.35.174.255 inet6 fe80::29c:2ff:feb0:bf70 prefixlen 64 scopeid 0x20<link> ether 00:9c:02:b0:bf:70 txqueuelen 0 (Ethernet) RX packets 5125281 bytes 1656195229 (1.5 GiB) RX errors 0 dropped 0 overruns 0 frame 0 TX packets 4983982 bytes 2064910670 (1.9 GiB) TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
This is not something controlled by puppet. It's purely a setting in Foreman. The ask here is to be able to change the value *before* deployment on the bond, not after deployment. Test here should be: Create the bond in the NIC config screen in foreman. Go to the Host networking tab change the ip address of the bond save If that works, then it's verified. A slightly more in depth test would be to do all of the above before deployment, then click deploy and see that it deploys with the right ip address. Moving back to ON_QA
Failed to deployed after changed the ip-address of the public-api of one controller. I changed the ip and then deployed, the installation paused without any indication of any error. two of the controllers got the same ip address from the default network subnet(192.168.0.2) and after reboot for one of them the interface enp5s0f1 got an ip:192.168.0.21. the deployment is stuck and I cant reach two of my controllers and the deplyment stuck for more then 3 hours. attaching logs.
Created attachment 1005070 [details] grep dhcp of /var/log/messages from staypuft server
Created attachment 1005071 [details] sosreport 1
Created attachment 1005072 [details] sosreport 2
Can you verify that you are not setting the IP for the bond to an IP that is coming from a DHCP pool?
the ip i changed is part of "public-api" subnet that is internal-db/none. the "default" subnet is dhcp/dhcp(default).
I do not believe what you are experiencing is a part of this bug. This bug is about whether the ip address can be set and saved on a bond. It appears that it does that. Please verify that, and open a new bug if there is a separate issue here.
Created attachment 1005717 [details] production log from staypuft and controller var log
I tried to reproduced it again as the user did. I used the tenant subnet that is set on the bond. The tenant subnet use dhcp server that is in our lab. I assigned the networks and before I deployed I edited one of the controllers. I entered a new ip address(Ip address was blank) "10.35.160.77" (The ip I entered is a fixed ip for that server) and I unchecked the "managed" check-box. I submitted and re-entered the "edit" window and it saved the address. The deployment didn't pass, the specific host where I changed the ip didn't configured any interface related to the bond(but I'm not sure if it's related to this bug or is it a new one) In the production log I only saw: [root@staypuft ~]# cat /var/log/foreman/production.log |grep -i 10.35.160.77 Parameters: {"utf8"=>"✓", "authenticity_token"=>"7GkZ8XAFImfJpqwxhPyRe/i8qmwyFVeC9XYDuDFeBHk=", "host"=>{"name"=>"mac441ea173366b", "hostgroup_id"=>"33", "environment_id"=>"2", "puppet_ca_proxy_id"=>"1", "puppet_proxy_id"=>"1", "puppetclass_ids"=>[""], "managed"=>"t", "progress_report_id"=>"[FILTERED]", "domain_id"=>"1", "realm_id"=>"", "mac"=>"44:1e:a1:73:36:6b", "subnet_id"=>"1", "ip"=>"192.168.0.4", "interfaces_attributes"=>{"0"=>{"_destroy"=>"false", "mac"=>"00:9c:02:b0:bf:70", "name"=>"", "domain_id"=>"", "subnet_id"=>"2", "ip"=>"10.35.160.77", "managed"=>"0", "mode"=>"802.3ad", "attached_devices"=>"enp4s0f0,enp4s0f1", "bond_options"=>"miimon=100", "virtual"=>"true", "id"=>"572"}, "1"=>{"_destroy"=>"false", "mac"=>"00:9c:02:b0:bf:70", "name"=>"", "domain_id"=>"", "subnet_id"=>"4", "ip"=>"10.35.174.14", "managed"=>"1", "tag"=>"", "attached_to"=>"bond0", "id"=>"575"}, "2"=>{"_destroy"=>"false", "mac"=>"00:9c:02:b0:bf:70", "name"=>"", "domain_id"=>"", "subnet_id"=>"3", "ip"=>"10.35.180.1", "managed"=>"1", "tag"=>"", "attached_to"=>"bond0", "id"=>"578"}, "3"=>{"_destroy"=>"false", "mac"=>"00:9c:02:b0:bf:70", "name"=>"", "domain_id"=>"", "subnet_id"=>"", "ip"=>"", "managed"=>"1", "tag"=>"", "attached_to"=>"", "id"=>"561"}, "4"=>{"_destroy"=>"false", "mac"=>"00:9c:02:b0:bf:74", "name"=>"", "domain_id"=>"", "subnet_id"=>"", "ip"=>"", "managed"=>"1", "tag"=>"", "attached_to"=>"", "id"=>"562"}, "5"=>{"_destroy"=>"false", "mac"=>"44:1e:a1:73:36:6a", "name"=>"", "domain_id"=>"", "subnet_id"=>"", "ip"=>"10.35.160.205", "managed"=>"1", "tag"=>"", "attached_to"=>"", "id"=>"563"}, "new_interfaces"=>{"_destroy"=>"false", "type"=>"Nic::Managed", "mac"=>"", "identifier"=>"", "name"=>"", "domain_id"=>"", "subnet_id"=>"", "ip"=>"", "managed"=>"1", "virtual"=>"0", "tag"=>"", "attached_to"=>""}}, "fencing"=>{"fencing_enabled"=>"0", "fencing_type"=>"", "fence_ipmilan_address"=>"", "fence_ipmilan_username"=>"", "fence_ipmilan_password"=>"[FILTERED]", "fence_ipmilan_expose_lanplus"=>"0", "fence_ipmilan_lanplus_options"=>""}, "architecture_id"=>"1", "operatingsystem_id"=>"1", "provision_method"=>"build", "medium_id"=>"7", "ptable_id"=>"12", "disk"=>"", "root_pass"=>"[FILTERED]", "is_owned_by"=>"3-Users", "enabled"=>"1", "model_id"=>"1", "comment"=>"", "overwrite"=>"false"}, "id"=>"mac441ea173366b.example.com"} [root@mac441ea173366b ~]# ifconfig enp5s0f1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500 inet 192.168.0.4 netmask 255.255.255.0 broadcast 192.168.0.255 inet6 fe80::461e:a1ff:fe73:366b prefixlen 64 scopeid 0x20<link> ether 44:1e:a1:73:36:6b txqueuelen 1000 (Ethernet) RX packets 380 bytes 403101 (393.6 KiB) RX errors 0 dropped 0 overruns 0 frame 0 TX packets 163 bytes 41579 (40.6 KiB) TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 device memory 0xfbee0000-fbefffff lo: flags=73<UP,LOOPBACK,RUNNING> mtu 65536 inet 127.0.0.1 netmask 255.0.0.0 inet6 ::1 prefixlen 128 scopeid 0x10<host> loop txqueuelen 0 (Local Loopback) RX packets 0 bytes 0 (0.0 B) RX errors 0 dropped 0 overruns 0 frame 0 TX packets 0 bytes 0 (0.0 B) TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0 i created an attachment of production log from the staypuft server and a /var/log/messages from the controller.
A couple things here. 1. this change is *only* in foreman to save the ip address. You reported in comment 46 that this works, so this bug is verified. 2. if you're using a DHCP network (whether foreman dhcp or external), saving the ip address will work, but functionally do nothing on the host itself.
verified based on comment 46. related bug: https://bugzilla.redhat.com/show_bug.cgi?id=1205241
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://rhn.redhat.com/errata/RHSA-2015-0791.html