Description of problem: Trying a distributed neutron deployment I was stuck in error state due to early problems in the deployment. I decided to start over, delete the vms to deploy to and recreate them. The vms that where previously in a error state were able to pick up the foreman discovery correctly, the ones that were in active or "not changed" state were not. I had to change the mac address to let the hosts pick again the foreman discovery image. Version-Release number of selected component (if applicable): Version : 0.1.1 Release : 1.el6ost How reproducible: Not sure Steps to Reproduce: 1. induce error state on hosts 2. disassociate and delete these hosts 3. recreate VMS (with same mac) and reboot 4. look at the discovered hosts Actual results: half of hosts discovered Expected results: all hosts discovered again Additional info:
Did you delete the hosts from foreman? or did you just remove the vms? Workaround -- remove 01-<mac-address> files from /var/lib/tftpboot/boot.
I was able to replicate the problem one more time. I simply deleted (Delete hosts) all the hosts from Hosts->All Hosts and tried to reboot the vms. Some were able to pick up the foreman discovery image, some were not. As you suggested, I looked for files named as the mac address in /var/lib/tftpboot/pxelinux.cfg and found that the file associated with the mac address of the machine that is not working exists and contains outdated data. /var/lib/tftpboot/pxelinux.cfg/01-52-54-00-36-2f-87 DEFAULT menu PROMPT 0 MENU TITLE PXE Menu TIMEOUT 200 TOTALTIMEOUT 6000 ONTIMEOUT local LABEL local MENU LABEL (local) MENU DEFAULT LOCALBOOT 0 So every time the host boots, the vm is presented with a PXE menu that contains only the useless entry (local), and eventually boots the previously installed OS. Removing the file fixed the issue.
Moving to foreman. I suspect this is more core foreman than staypuft specific.
Apparently removing the pxe file was not enough. Host were discovered, but I am unable to reassign the host to other hostgroups due (seeing foreman/production.log) to a conflict in DHCP reservation. I checked /etc/dhcp/dhcpd.{conf,hosts} but there are no entries. Where foreman keeps those static declarations ?
Please do a "yum install foreman-console && foreman-rake console" and then at the prompt get the following debug details (replace the example hostname with one of your own problematic hosts): > h = Host.find_by_name("host.example.com") > h.dhcp? > h.tftp? > h.subnet > h.operatingsystem > h.managed? > h.pxe_build? > h.provision_method (alternatively, if you have a system I can log into to reproduce the above, that's good too) Regarding DHCP reservations, dhcpd usually writes them to /var/lib/dhcpd/dhcpd.leases.
I had the same issue, -- > delete hosts --> re-discovered -> and then I couldn't assigned back the hosts on hosts- groups --> looking at the production.log , it seems a DHCP conflict. So I cleared dhcp.lease file and restart dhcpd then I tried again --> delete hosts --> re-discovered -> and still I couldn't - assigne back the hosts on hosts- groups with the same Error. environment: ------------ foreman-installer-staypuft-0.0.18-1.el6ost.noarch ruby193-rubygem-staypuft-0.1.4-1.el6ost.noarch foreman-1.6.0.15-1.el6sat.noarch openstack-puppet-modules-2014.1-14.6.el7ost.noarch production.log: ---------------- Started POST "/deployments/1/associate_host" for 10.36.5.19 at 2014-06-19 11:19:31 -0400 Processing by Staypuft::DeploymentsController#associate_host as HTML Parameters: {"utf8"=>"✓", "authenticity_token"=>"QydD3nncnWP5sGEiJqxoC94DXY62VHiTtlUashE2w9o=", "commit"=>"Assign / Unassign", "hostgroup_id"=>"6", "host_ids"=>["9"], "id"=>"1"} Create DHCP reservation for 525400868093.lab.eng.rdu2.redhat.com-52:54:00:86:80:93/192.168.0.7 Create DHCP Settings for 525400868093.lab.eng.rdu2.redhat.com task failed with the following error: ERF12-6899 [ProxyAPI::ProxyException]: Unable to set DHCP entry ([RestClient::Conflict]: 409 Conflict) for prox y https://staypuft.lab.eng.rdu2.redhat.com:8443/dhcp/usr/share/foreman/lib/proxy_api/dhcp.rb:66:in `rescue in set' /usr/share/foreman/lib/proxy_api/dhcp.rb:62:in `set' /usr/share/foreman/lib/net/dhcp/record.rb:28:in `create' /usr/share/foreman/app/models/concerns/orchestration/dhcp.rb:22:in `set_dhcp' /usr/share/foreman/app/models/concerns/orchestration.rb:137:in `execute' /usr/share/foreman/app/models/concerns/orchestration.rb:85:in `block in process' /usr/share/foreman/app/models/concerns/orchestration.rb:77:in `each' /usr/share/foreman/app/models/concerns/orchestration.rb:77:in `process' /usr/share/foreman/app/models/concerns/orchestration.rb:18:in `on_save' /opt/rh/ruby193/root/usr/share/gems/gems/activesupport-3.2.8/lib/active_support/callbacks.rb:649:in `_run__138396949222223340__save__759825608298643462__callbacks' /opt/rh/ruby193/root/usr/share/gems/gems/activesupport-3.2.8/lib/active_support/callbacks.rb:405:in `__run_callback' /opt/rh/ruby193/root/usr/share/gems/gems/activesupport-3.2.8/lib/active_support/callbacks.rb:385:in `_run_save_callbacks' /opt/rh/ruby193/root/usr/share/gems/gems/activesupport-3.2.8/lib/active_support/callbacks.rb:81:in `run_callbacks' /opt/rh/ruby193/root/usr/share/gems/gems/activerecord-3.2.8/lib/active_record/callbacks.rb:264:in `create_or_update' .. .. .. .. /opt/rh/ruby193/root/usr/share/gems/gems/rack-cache-1.2/lib/rack/cache/context.rb:143:in `pass' /opt/rh/ruby193/root/usr/share/gems/gems/rack-cache-1.2/lib/rack/cache/context.rb:155:in `invalidate' /opt/rh/ruby193/root/usr/share/gems/gems/rack-cache-1.2/lib/rack/cache/context.rb:71:in `call!' /opt/rh/ruby193/root/usr/share/gems/gems/rack-cache-1.2/lib/rack/cache/context.rb:51:in `call' /opt/rh/ruby193/root/usr/share/gems/gems/railties-3.2.8/lib/rails/engine.rb:479:in `call' /opt/rh/ruby193/root/usr/share/gems/gems/railties-3.2.8/lib/rails/application.rb:223:in `call' /opt/rh/ruby193/root/usr/share/gems/gems/railties-3.2.8/lib/rails/railtie/configurable.rb:30:in `method_missing' /opt/rh/ruby193/root/usr/share/gems/gems/rack-1.4.1/lib/rack/builder.rb:134:in `call' /opt/rh/ruby193/root/usr/share/gems/gems/rack-1.4.1/lib/rack/urlmap.rb:64:in `block in call' /opt/rh/ruby193/root/usr/share/gems/gems/rack-1.4.1/lib/rack/urlmap.rb:49:in `each' /opt/rh/ruby193/root/usr/share/gems/gems/rack-1.4.1/lib/rack/urlmap.rb:49:in `call' /usr/lib/ruby/gems/1.8/gems/passenger-4.0.18/lib/phusion_passenger/rack/thread_handler_extension.rb:77:in `process_request' /usr/lib/ruby/gems/1.8/gems/passenger-4.0.18/lib/phusion_passenger/request_handler/thread_handler.rb:140:in `accept_and_process_next_request' /usr/lib/ruby/gems/1.8/gems/passenger-4.0.18/lib/phusion_passenger/request_handler/thread_handler.rb:108:in `main_loop' /usr/lib/ruby/gems/1.8/gems/passenger-4.0.18/lib/phusion_passenger/request_handler.rb:441:in `block (3 levels) in start_threads' Rolling back due to a problem: [Create DHCP Settings for 525400868093.lab.eng.rdu2.redhat.com 9 failed [#<Host::Managed id: 9, name: "525400868093.lab.eng.rdu2.redhat.com", ip: "192.168.0.7", last_compile: nil, last_freshcheck: nil, last_report: "2014-06-19 14:51:42", updated_at: "2014-06-19 14:51:43", source_file_id: nil, created_at: "2014-06-19 14:51:42", mac: "52:54:00:86:80:93", root_pass: "$1$5fP0De+Y$/U06KVVNQUZuX5OACrOb4.", serial: nil, puppet_status: 0, domain_id: 1, architecture_id: 1, operatingsystem_id: 2, environment_id: 2, subnet_id: 1, ptable_id: 7, medium_id: 7, build: true, comment: nil, disk: nil, installed_at: nil, model_id: 2, hostgroup_id: 6, owner_id: 1, owner_type: "User", enabled: true, puppet_ca_proxy_id: 1, managed: true, use_image: nil, image_file: nil, uuid: nil, compute_resource_id: nil, puppet_proxy_id: 1, certname: nil, image_id: nil, organization_id: nil, location_id: nil, type: "Host::Managed", otp: nil, realm_id: nil, compute_profile_id: nil, provision_method: nil>, :set_dhcp]] Unassigned hosts: 525400868093 (Create DHCP Settings for 525400868093.lab.eng.rdu2.redhat.com task failed with the following error: ERF12-6899 [ProxyAPI::ProxyException]: Unable to set DHCP entry ([RestClient::Conflict]: 409 Conflict) for proxy https://staypuft.lab.eng.rdu2.redhat.com:8443/dhcp) Redirected to https://10.8.30.234/deployments/1/hostgroup/6-base_redhat_7-omri-neutron-01-neutron-networker Completed 302 Found in 1115ms (ActiveRecord: 42.3ms)
see comment #6
Installed ruby193-rubygem-minitest instead to get the console. Host.find_by_name("525400868092.example.com") returns Host::Managed Load (2.1ms) SELECT "hosts".* FROM "hosts" WHERE "hosts"."type" IN ('Host::Managed') AND "hosts"."name" = '525400868092.example.com' LIMIT 1 => nil If I try to find a Host::discovered by raw SQL I get {"id"=>"3", "name"=>"525400654fdd", "ip"=>"192.168.100.201", "last_compile"=>nil, "last_freshcheck"=>nil, "last_report"=>"2014-07-07 20:25:34.165999", "updated_at"=>"2014-07-07 20:25:34.697861", "source_file_id"=>nil, "created_at"=>"2014-07-07 20:25:34.120484", "mac"=>"52:54:00:65:4f:dd", "root_pass"=>nil, "serial"=>nil, "puppet_status"=>"0", "domain_id"=>nil, "architecture_id"=>nil, "operatingsystem_id"=>nil, "environment_id"=>nil, "subnet_id"=>"1", "ptable_id"=>nil, "medium_id"=>nil, "build"=>"f", "comment"=>nil, "disk"=>nil, "installed_at"=>nil, "model_id"=>"2", "hostgroup_id"=>nil, "owner_id"=>nil, "owner_type"=>nil, "enabled"=>"t", "puppet_ca_proxy_id"=>nil, "managed"=>"f", "use_image"=>nil, "image_file"=>nil, "uuid"=>nil, "compute_resource_id"=>nil, "puppet_proxy_id"=>nil, "certname"=>nil, "image_id"=>nil, "organization_id"=>nil, "location_id"=>nil, "type"=>"Host::Discovered", "otp"=>nil, "realm_id"=>nil, "compute_profile_id"=>nil, "provision_method"=>nil} but this does not have dhcp properties.
This should work in current versions
The needinfo request[s] on this closed bug have been removed as they have been unresolved for 1000 days