Bug 1272238 - Setting a host build state to true facts are lost
Setting a host build state to true facts are lost
Status: CLOSED EOL
Product: Red Hat OpenStack
Classification: Red Hat
Component: foreman (Show other bugs)
Foreman (RHEL 6)
Unspecified Unspecified
high Severity high
: async
: Installer
Assigned To: Mike Burns
Shai Revivo
: ZStream
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2015-10-15 16:35 EDT by Dave Sullivan
Modified: 2016-09-29 09:39 EDT (History)
14 users (show)

See Also:
Fixed In Version: foreman-1.6.0.44-7.el6ost
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2016-09-29 09:39:42 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description Dave Sullivan 2015-10-15 16:35:15 EDT
Description of problem:

First make sure a host has fact values

[root@gss-rhos-3 api_scripts]# psql -h localhost -U foreman -d foreman -c "select * from hosts where id=95"
Password for user foreman: 
 id |                   name                   |      ip      |    last_compile     | last_freshcheck |     last_report     |         updated_at         | source_file_id |         created_at         |        mac
        |             root_pass              | serial | puppet_status | domain_id | architecture_id | operatingsystem_id | environment_id | subnet_id | ptable_id | medium_id | build | comment | disk |        ins
talled_at        | model_id | hostgroup_id | owner_id | owner_type | enabled | puppet_ca_proxy_id | managed | use_image | image_file | uuid | compute_resource_id | puppet_proxy_id |                 certname     
            | image_id | organization_id | location_id |     type      | otp | realm_id | compute_profile_id | provision_method | primary_interface 
----+------------------------------------------+--------------+---------------------+-----------------+---------------------+----------------------------+----------------+----------------------------+-----------
--------+------------------------------------+--------+---------------+-----------+-----------------+--------------------+----------------+-----------+-----------+-----------+-------+---------+------+-----------
-----------------+----------+--------------+----------+------------+---------+--------------------+---------+-----------+------------+------+---------------------+-----------------+------------------------------
------------+----------+-----------------+-------------+---------------+-----+----------+--------------------+------------------+-------------------
 95 | rhosp-6.openstack.gsslab.rdu2.redhat.com | 10.10.72.104 | 2015-10-15 19:36:47 |                 | 2015-10-15 19:36:18 | 2015-10-15 19:37:03.029948 |                | 2015-07-17 07:54:54.842362 | 40:f2:e9:8
5:82:82 | $1$T8aNAgKO$Sn.KLDP2hpD/rDow3QKzD. |        |             0 |         1 |               1 |                  1 |              1 |         1 |        12 |         7 | f     |         |      | 2015-07-17
 09:32:45.798942 |        2 |           19 |        3 | User       | t       |                  1 | t       |           |            |      |                     |               1 | rhosp-6.openstack.gsslab.rdu2
.redhat.com |          |                 |             | Host::Managed |     |          |                    | build            | eth2
(1 row)

[root@gss-rhos-3 api_scripts]# psql -h localhost -U foreman -d foreman -c "select * from fact_values where host_id=95"Password for user foreman: 
  id   |                                                                                                                                                                                        value              
                                                                                                                                                                           | fact_name_id | host_id |         updat
ed_at         |         created_at         
-------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------+--------------+---------+--------------
--------------+----------------------------
 17706 | be:fd:fe:81:b1:96                                                                                                                                                                                         
                                                                                                                                                                           |          378 |      95 | 2015-08-11 19
:37:50.518444 | 2015-08-11 19:37:50.518444
 17707 | be:fd:fe:81:b1:96                                                                                                                                                                                         
                                                                                                                                                                           |          379 |      95 | 2015-08-11 19
:37:50.525294 | 2015-08-11 19:37:50.525294
 17708 | 96:0c:48:88:3e:cb                                                                                                                                                                                         
                                                                                                                                                                           |          380 |      95 | 2015-08-11 19
:37:50.52995  | 2015-08-11 19:37:50.52995

turn on debugging.

Edit ~foreman/config/environments/production.rb and uncomment this line:

#apparently this is needed to reload foreman nicely
touch ~foreman/tmp/restart.txt

then I tail /var/log/foreman/production.log

Then switch build to true/1


#!/bin/sh
curl -s -H "Accept:application/json" -X PUT -u admin:vq8bD5S9DJamGaJa \
    --cacert /var/lib/puppet/ssl/certs/gss-rhos-3.openstack.gsslab.rdu2.redhat.com.pem --insecure \
    : -d "host[build]=1" https://gss-rhos-3.openstack.gsslab.rdu2.redhat.com/api/hosts/rhosp-6.openstack.gsslab.rdu2.redhat.com

[root@gss-rhos-3 api_scripts]# sh addbuildfact_rhosp-6.sh

Rerun query on fact_values

[root@gss-rhos-3 api_scripts]# psql -h localhost -U foreman -d foreman -c "select * from fact_values where host_id=95
> "
Password for user foreman: 
 id | value | fact_name_id | host_id | updated_at | created_at 
----+-------+--------------+---------+------------+------------
(0 rows)

[root@gss-rhos-3 api_scripts]#

So they have been wiped

The log files show

SQL (1.5ms)  DELETE FROM "fact_values" WHERE (host_id = 95)
  SQL (24.7ms)  DELETE FROM "reports" WHERE (host_id = 95)

So why delete the fact values, might be another problem with deleting host from reports too.

Now further problems noted on facts query as you see the error where we try to reference WHERE (hosts.id = 'rhosp-6.openstack.gsslab.rdu2.redhat.com') where that is really the hosts.name.

[root@gss-rhos-3 api_scripts]# cat gethostfacts.sh 
#!/bin/sh
curl -X GET -u admin:vq8bD5S9DJamGaJa \
    -H "Accept:application/json" \
    --cacert /var/lib/puppet/ssl/certs/gss-rhos-3.openstack.gsslab.rdu2.redhat.com.pem --insecure \
    https://gss-rhos-3.openstack.gsslab.rdu2.redhat.com/api/v2/hosts/rhosp-6.openstack.gsslab.rdu2.redhat.com/facts

 FactValue Load (0.9ms)  SELECT fact_values.id FROM "fact_values" INNER JOIN "hosts" ON "hosts"."id" = "fact_values"."host_id" AND "hosts"."type" IN ('Host::Managed') WHERE (hosts.name = 'rhosp-6.openstack.gsslab.rdu2.redhat.com')
  FactValue Load (0.5ms)  SELECT fact_values.id FROM "fact_values" INNER JOIN "hosts" ON "hosts"."id" = "fact_values"."host_id" AND "hosts"."type" IN ('Host::Managed') WHERE (hosts.id = 'rhosp-6.openstack.gsslab.rdu2.redhat.com')
PG::InvalidTextRepresentation: ERROR:  invalid input syntax for integer: "rhosp-6.openstack.gsslab.rdu2.redhat.com"
LINE 1: ...ts"."type" IN ('Host::Managed') WHERE (hosts.id = 'rhosp-6.o...
                                                             ^
: SELECT fact_values.id FROM "fact_values" INNER JOIN "hosts" ON "hosts"."id" = "fact_values"."host_id" AND "hosts"."type" IN ('Host::Managed') WHERE (hosts.id = 'rhosp-6.openstack.gsslab.rdu2.redhat.com')
  SQL (38.4ms)  SELECT "fact_values"."id" AS t0_r0, "fact_values"."value" AS t0_r1, "fact_values"."fact_name_id" AS t0_r2, "fact_values"."host_id" AS t0_r3, "fact_values"."updated_at" AS t0_r4, "fact_values"."created_at" AS t0_r5, "fact_names"."id" AS t1_r0, "fact_names"."name" AS t1_r1, "fact_names"."updated_at" AS t1_r2, "fact_names"."created_at" AS t1_r3, "fact_names"."compose" AS t1_r4, "fact_names"."short_name" AS t1_r5, "fact_names"."type" AS t1_r6, "fact_names"."ancestry" AS t1_r7, "hosts"."id" AS t2_r0, "hosts"."name" AS t2_r1, "hosts"."ip" AS t2_r2, "hosts"."last_compile" AS t2_r3, "hosts"."last_freshcheck" AS t2_r4, "hosts"."last_report" AS t2_r5, "hosts"."updated_at" AS t2_r6, "hosts"."source_file_id" AS t2_r7, "hosts"."created_at" AS t2_r8, "hosts"."mac" AS t2_r9, "hosts"."root_pass" AS t2_r10, "hosts"."serial" AS t2_r11, "hosts"."puppet_status" AS t2_r12, "hosts"."domain_id" AS t2_r13, "hosts"."architecture_id" AS t2_r14, "hosts"."operatingsystem_id" AS t2_r15, "hosts"."environment_id" AS t2_r16, "hosts"."subnet_id" AS t2_r17, "hosts"."ptable_id" AS t2_r18, "hosts"."medium_id" AS t2_r19, "hosts"."build" AS t2_r20, "hosts"."comment" AS t2_r21, "hosts"."disk" AS t2_r22, "hosts"."installed_at" AS t2_r23, "hosts"."model_id" AS t2_r24, "hosts"."hostgroup_id" AS t2_r25, "hosts"."owner_id" AS t2_r26, "hosts"."owner_type" AS t2_r27, "hosts"."enabled" AS t2_r28, "hosts"."puppet_ca_proxy_id" AS t2_r29, "hosts"."managed" AS t2_r30, "hosts"."use_image" AS t2_r31, "hosts"."image_file" AS t2_r32, "hosts"."uuid" AS t2_r33, "hosts"."compute_resource_id" AS t2_r34, "hosts"."puppet_proxy_id" AS t2_r35, "hosts"."certname" AS t2_r36, "hosts"."image_id" AS t2_r37, "hosts"."organization_id" AS t2_r38, "hosts"."location_id" AS t2_r39, "hosts"."type" AS t2_r40, "hosts"."otp" AS t2_r41, "hosts"."realm_id" AS t2_r42, "hosts"."compute_profile_id" AS t2_r43, "hosts"."provision_method" AS t2_r44, "hosts"."primary_interface" AS t2_r45 FROM "fact_values" LEFT OUTER JOIN "fact_names" ON "fact_names"."id" = "fact_values"."fact_name_id" LEFT OUTER JOIN "hosts" ON "hosts"."id" = "fact_values"."host_id" AND "hosts"."type" IN ('Host::Managed') WHERE (fact_names.name <> '_timestamp') ORDER BY "fact_values"."value" ASC NULLS FIRST LIMIT 9999 OFFSET 0
  Rendered api/v2/fact_values/index.json.rabl within api/v2/layouts/index_layout (3.3ms)
   (1.8ms)  SELECT COUNT(DISTINCT "fact_values"."id") FROM "fact_values" LEFT OUTER JOIN "fact_names" ON "fact_names"."id" = "fact_values"."fact_name_id" WHERE (fact_names.name <> '_timestamp')
Body: {
  "total": 1685,
  "subtotal": 10,
  "page": 1,
  "per_page": 9999,
  "search": " host = rhosp-6.openstack.gsslab.rdu2.redhat.com",
  "sort": {
    "by": null,
    "order": null
  },
  "results": {"rhosp-2.openstack.gsslab.rdu2.redhat.com":{"swapfree_mb":"0.00","swapfree":"0.00 MB","boardproductname":"00FG986","bios_release_date":"05/14/2014","uniqueid":"0a0a6548","network_eth2":"10.10.72.0","gluster_host_ip":"10.10.72.101","ipaddress_eth2":"10.10.72.101","ipa_host_ip":"10.10.72.101","network_br_ex":"10.10.77.0","ipaddress_br_ex":"10.10.77.107","vlans":"101,102,100","swapsize_mb":"1024.00","swapsize":"1024.00 MB","augeasversion":"1.1.0","network_lo":"127.0.0.0","ipaddress_lo":"127.0.0.1","iptables_version":"1.4.21","ip6tables_version":"1.4.21","memorysize":"15.49 GB","memorytotal":"15.49 GB","memorysize_mb":"158

The results return are actually bogus, not sure why we'd return results back for fact_values based on timestamp anyway.

In customers environment no results are returned.


Version-Release number of selected component (if applicable):

My testing was done on

[root@gss-rhos-3 api_scripts]# rpm -qa | grep foreman
foreman-installer-1.6.0-0.4.RC1.el7ost.noarch
foreman-discovery-image-7.0-20150227.0.el7ost.noarch
foreman-proxy-1.6.0.30-6.el7ost.noarch
rubygem-foreman_api-0.1.11-6.el7sat.noarch
foreman-postgresql-1.6.0.49-6.el7ost.noarch
foreman-selinux-1.6.0.14-1.el7sat.noarch
ruby193-rubygem-foreman_discovery-1.3.0-2.3.el7ost.noarch
rubygem-hammer_cli_foreman-0.1.1-16.el7sat.noarch
ruby193-rubygem-foreman_openstack_simplify-0.0.6-8.el7ost.noarch
openstack-foreman-installer-3.0.24-1.el7ost.noarch
ruby193-rubygem-foreman-tasks-0.6.9-1.1.1.el7ost.noarch
foreman-1.6.0.49-6.el7ost.noarch

Customer's environment is more important though in regards to needing a fix first.

# rpm -qa |grep -i foreman
rubygem-hammer_cli_foreman-0.1.1-16.el6sat.noarch
ruby193-rubygem-foreman_column_view-0.2.0-1.el6.noarch
ruby193-rubygem-foreman_openstack_simplify-0.0.6-8.el6ost.noarch
rubygem-hammer_cli_foreman-doc-0.1.1-16.el6sat.noarch
ruby193-rubygem-foreman_hooks-0.3.8-1.el6.noarch
ruby193-rubygem-foreman_column_view-doc-0.2.0-1.el6.noarch
bac-RHEL6-ost5-foreman-1.0-1.noarch
foreman-1.6.0.44-6.el6ost.noarch
ruby193-rubygem-foreman_discovery-1.3.0-2.2.el6ost.noarch
foreman-discovery-image-7.0-20140905.0.2.el7sat.noarch
openstack-foreman-installer-2.0.34-1.el6ost.noarch
foreman-proxy-1.6.0.33-2.el6ost.noarch
ruby193-rubygem-foreman_hooks-doc-0.3.8-1.el6.noarch
foreman-installer-1.6.0-0.2.RC1.el6ost.noarch
foreman-mysql2-1.6.0.44-6.el6ost.noarch
foreman-selinux-1.6.0.14-1.el6sat.noarch

How reproducible:

See above


Actual results:

facts still be available after setting hosts build to 1/true


Expected results:


Additional info:

Also concerned about hosts being deleted from reports table
Comment 5 Mike Burns 2015-10-16 16:00:23 EDT
Adding bkearney as well.

Bryan,  any thoughts on this?
Comment 6 Ohad Levy 2015-10-21 06:47:19 EDT
deleting facts (and reports) on host build was done on purpose, Could you please provide information on why you would need it to stay when rebuilding a machine? (granted that when the machine is re provisioned, its out of date / irrelevant facts are purged anyway).
Comment 7 Dave Sullivan 2015-10-21 09:40:34 EDT
per customer....

Current workflow is:
1. Boot servers into discovered_hosts
2. Host added to "hosts" table from discovered_hosts (build=0) and assigned to proper groups,hostname,ip,etc..
3. Server build is executed "build=1" and foreman-hooks (after_build) are executed.   

If any hooks were relying on facts_hash (ie: checking for processorcount,processor0,memorytotal,manufactuer,modelname,etc...) those hooks will fail to find those entries.

Since most facts collected during discovered_hosts is hardware related, we expected those facts to remain (as hardware isn't changing), at least until the provision process is finished (ie: before_provision hook is executed).

Hope this helps?
Comment 8 Lukas Zapletal 2015-10-21 10:36:19 EDT
In latest version of Discovery (upstream) we actually prevent deletion of some facts (prefixed with "discovery_"):

https://github.com/theforeman/foreman_discovery/blob/develop/app/services/foreman_discovery/host_converter.rb#L22-L25

You can use similar code as a workaround for now, maybe it makes sense to keep *all* discovered facts during provisioning. Or at least some (configurable by an option) if that breaks provisioning (network detection might fail perhaps).

Ohad, if it makes sense we can keep them all: http://projects.theforeman.org/issues/12244
Comment 10 Mike Burns 2015-10-21 12:21:59 EDT
This is on the foreman server side, right?  not the discovery image?
Comment 11 Dave Sullivan 2015-10-21 17:13:19 EDT
Right it's foreman server side.

[root@gss-rhos-3 foreman]# grep -B 5 -A 5 setBuild app/models/host/managed.rb
  # Called by build link in the list
  # Build is set
  # The boot link and autosign entry are created
  # Any existing puppet certificates are deleted
  # Any facts are discarded
  def setBuild
    self.build = true
    self.save
    errors.empty?
  end

So looks like self.save calls a before_save hook that calls clear_data_on_build

/usr/share/foreman/app/models/host/managed.rb

# Custom hooks will be executed after_commit
  after_commit :build_hooks
  before_save :clear_data_on_build

...

 def clear_data_on_build
    return unless respond_to?(:old) && old && build? && !old.build?
    clearFacts
    clearReports
  end


If we know on provision/re-provision the facts are regenerated/overwritten.  Why can't we just comment out clearFacts, not sure the implications of this though.
Comment 12 Ohad Levy 2015-10-22 03:39:13 EDT
Dave, I believe your fix would work as well (for all usage cases and not just discovered hosts) - would that be sufficient?
Comment 24 Mike McCune 2016-03-28 19:03:07 EDT
This bug was accidentally moved from POST to MODIFIED via an error in automation, please see mmccune@redhat.com with any questions
Comment 26 Jaromir Coufal 2016-09-29 09:39:42 EDT
Closing list of bugs for RHEL OSP Installer since its support cycle has already ended [0]. If there is some bug closed by mistake, feel free to re-open.

For new deployments, please, use RHOSP director (starting with version 7).

-- Jaromir Coufal
-- Sr. Product Manager
-- Red Hat OpenStack Platform

[0] https://access.redhat.com/support/policy/updates/openstack/platform

Note You need to log in before you can comment on or make changes to this bug.