Bug 1122180 - Rubygem-Staypuft: Unable to assign some hosts to deployment after finishing deploying and cleaning/re-discovering.
Summary: Rubygem-Staypuft: Unable to assign some hosts to deployment after finishing d...
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: foreman-proxy
Version: unspecified
Hardware: x86_64
OS: Linux
high
high
Target Milestone: ---
: Installer
Assignee: Dominic Cleal
QA Contact: Omri Hochman
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2014-07-22 17:31 UTC by Alexander Chuzhoy
Modified: 2015-04-30 16:23 UTC (History)
7 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2015-04-30 16:23:00 UTC


Attachments (Terms of Use)
/var/log/messages file from the staypuft machine. (1.08 MB, text/plain)
2014-07-22 17:34 UTC, Alexander Chuzhoy
no flags Details
production.log from the staypuft host. (514.37 KB, text/x-log)
2014-07-22 17:34 UTC, Alexander Chuzhoy
no flags Details
/var/log/forman-proxy/proxy.log from the staypuft host. (165.28 KB, text/x-log)
2014-07-22 17:35 UTC, Alexander Chuzhoy
no flags Details

Description Alexander Chuzhoy 2014-07-22 17:31:48 UTC
Rubygem-Staypuft: Unable to assign some hosts to deployment after finishing deploying and cleaning/re-discovering.


Environment: 
rhel-osp-installer-0.1.1-1.el6ost.noarch
foreman-proxy-1.6.0.8-1.el6sat.noarch
foreman-installer-1.5.0-0.4.RC2.el6ost.noarch
ruby193-rubygem-foreman-tasks-0.6.4-2.el6sat.noarch
rubygem-foreman_api-0.1.11-4.el6sat.noarch
openstack-foreman-installer-2.0.15-1.el6ost.noarch
foreman-discovery-image-6.5-20140620.2.el6sat.noarch
ruby193-rubygem-foreman_openstack_simplify-0.0.6-8.el6ost.noarch
openstack-puppet-modules-2014.1-19.1.el6ost.noarch

Steps to reproduce:
1. Install rhel-osp-installer.
2. Create/Run a deployment of nova network + 2 compute nodes (3 hosts in total).
3. After the deployment finishes, clean the environment by:
    a. removing all hosts from UI
    b. cleaning the remaining entries under /var/lib/tftpboot/pxelinux.cfg/ (due to https://bugzilla.redhat.com/show_bug.cgi?id=1107681)
    c. removing the file /var/lib/dhcpd/dhcpd.leases  + restarting the dhcpd service.
4. reboot the nodes comprising the openstack setup and wait until they appear in the "discovered hosts" page.
5. Attempt to assign the hosts to the deployment.  


Result: some hosts can't be assigned.
if the issue wasn't re-produced, repeat steps 3-5.

Expected results: should be able to assign all hosts to the deployment.

Comment 1 Alexander Chuzhoy 2014-07-22 17:34:25 UTC
Created attachment 919993 [details]
/var/log/messages file from the staypuft machine.

Comment 2 Alexander Chuzhoy 2014-07-22 17:34:58 UTC
Created attachment 919994 [details]
production.log from the staypuft host.

Comment 3 Alexander Chuzhoy 2014-07-22 17:35:37 UTC
Created attachment 919995 [details]
/var/log/forman-proxy/proxy.log from the staypuft host.

Comment 5 Alexander Chuzhoy 2014-07-22 18:48:25 UTC
This workaround made it possible to assign hosts without reinstalling rhel-osp-installer:
1. clear the dns zones files from all dynamic entries
2. remove the dhcpd.aliases file
3. remove remained entries under tftproot/pxelinux.cfg/
4. restart the named service
5. restert the dhcpd service
6. restart the foreman-proxy service
7. restart the foreman service

Comment 7 Marek Hulan 2014-08-14 07:03:38 UTC
From the production log it seems that following happened:
There was a host with MAC 04:7d:7b:62:7d:5c which was discovered by Foreman. User tried to associate it within some deployment. This action failed because there was conflicting Reverse DNS record for 047d7b627d5c.example.com. From proxy log (line ~326) we can see that during related nsupdate we got an error '5.0.168.192.in-addr.arpa is already used by 047d7b685a04.example.com' but I see no log about creating this PTR record.

Logs are probably not complete but I suppose that this record existed before cleaning/re-discovering and either Foreman did not clean it or there's some cache. Either access to a machine or full logs would greatly help to debug this.

Comment 8 Leonid Natapov 2014-12-09 15:51:11 UTC
Happens on rhel-osp-installer 6,latest poodle.

Comment 9 Mike Burns 2015-04-30 16:23:00 UTC
This does not reproduce in latest builds.


Note You need to log in before you can comment on or make changes to this bug.