Rubygem-Staypuft: Unable to assign some hosts to deployment after finishing deploying and cleaning/re-discovering. Environment: rhel-osp-installer-0.1.1-1.el6ost.noarch foreman-proxy-1.6.0.8-1.el6sat.noarch foreman-installer-1.5.0-0.4.RC2.el6ost.noarch ruby193-rubygem-foreman-tasks-0.6.4-2.el6sat.noarch rubygem-foreman_api-0.1.11-4.el6sat.noarch openstack-foreman-installer-2.0.15-1.el6ost.noarch foreman-discovery-image-6.5-20140620.2.el6sat.noarch ruby193-rubygem-foreman_openstack_simplify-0.0.6-8.el6ost.noarch openstack-puppet-modules-2014.1-19.1.el6ost.noarch Steps to reproduce: 1. Install rhel-osp-installer. 2. Create/Run a deployment of nova network + 2 compute nodes (3 hosts in total). 3. After the deployment finishes, clean the environment by: a. removing all hosts from UI b. cleaning the remaining entries under /var/lib/tftpboot/pxelinux.cfg/ (due to https://bugzilla.redhat.com/show_bug.cgi?id=1107681) c. removing the file /var/lib/dhcpd/dhcpd.leases + restarting the dhcpd service. 4. reboot the nodes comprising the openstack setup and wait until they appear in the "discovered hosts" page. 5. Attempt to assign the hosts to the deployment. Result: some hosts can't be assigned. if the issue wasn't re-produced, repeat steps 3-5. Expected results: should be able to assign all hosts to the deployment.
Created attachment 919993 [details] /var/log/messages file from the staypuft machine.
Created attachment 919994 [details] production.log from the staypuft host.
Created attachment 919995 [details] /var/log/forman-proxy/proxy.log from the staypuft host.
This workaround made it possible to assign hosts without reinstalling rhel-osp-installer: 1. clear the dns zones files from all dynamic entries 2. remove the dhcpd.aliases file 3. remove remained entries under tftproot/pxelinux.cfg/ 4. restart the named service 5. restert the dhcpd service 6. restart the foreman-proxy service 7. restart the foreman service
From the production log it seems that following happened: There was a host with MAC 04:7d:7b:62:7d:5c which was discovered by Foreman. User tried to associate it within some deployment. This action failed because there was conflicting Reverse DNS record for 047d7b627d5c.example.com. From proxy log (line ~326) we can see that during related nsupdate we got an error '5.0.168.192.in-addr.arpa is already used by 047d7b685a04.example.com' but I see no log about creating this PTR record. Logs are probably not complete but I suppose that this record existed before cleaning/re-discovering and either Foreman did not clean it or there's some cache. Either access to a machine or full logs would greatly help to debug this.
Happens on rhel-osp-installer 6,latest poodle.
This does not reproduce in latest builds.