Bug 1553263

Summary: [RFE] add fact name filtering during fact import
Product: Red Hat Satellite Reporter: Chris Duryee <cduryee>
Component: FactAssignee: Shimon Shtein <sshtein>
Status: CLOSED ERRATA QA Contact: Ranjan Kumar <rankumar>
Severity: high Docs Contact:
Priority: unspecified    
Version: 6.2.14CC: andrew.schofield, bkearney, cduryee, kabbott, lpramuk, mhulan, mmccune, pcreech, rankumar, sshtein, tstrachota
Target Milestone: UnspecifiedKeywords: FutureFeature, Triaged
Target Release: Unused   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: foreman-1.15.6.39-1 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of:
: 1572305 (view as bug list) Environment:
Last Closed: 2018-06-19 20:17:00 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Chris Duryee 2018-03-08 15:24:31 UTC
Description of problem:

Satellite currently pulls in all facts reported by clients. This can lead to an extremely large (>50K) fact_names table, which can make fact import slow down. Eventually, this will lead to transaction errors logged to production.log during fact import.

For example:

 [app] [E] Fact dmi::system::uuid could not be imported because of PG::Error: ERROR:  duplicate key value violates unique constraint "index_fact_values_on_fact_name_id_and_host_id"

Version-Release number of selected component (if applicable): 6.2.14/6.3.0


How reproducible: not every time


Steps to Reproduce:
1. pull in a customer DB with a lot of fact_names
2. perform simultaneous registrations and checkins (perhaps 5 concurrent, maybe more)
3. watch for errors in production.log

Actual results: postgres errors, failed registrations


Expected results: no errors


Additional info: There are a couple of similar errors you may get. I usually see "[app] [E] Fact dmi::system::uuid could not be imported because of PG::Error: ERROR:  duplicate key value violates unique constraint "index_fact_values_on_fact_name_id_and_host_id"".

Comment 5 Satellite Program 2018-03-08 17:17:43 UTC
Upstream bug assigned to sshtein

Comment 6 Satellite Program 2018-03-08 17:17:46 UTC
Moving this bug to POST for triage into Satellite 6 since the upstream issue http://projects.theforeman.org/issues/20929 has been resolved.

Comment 17 Ranjan Kumar 2018-06-06 04:04:48 UTC
@chris

Can you please help me in 

1. Steps how to reproduce this issue? I tried to register 500 system at a time to Satellite 6.3.2 1.0 but could not found any similar error in log

2. Do i need to follow comment 7 to test this fix?

Comment 18 Chris Duryee 2018-06-08 16:38:16 UTC
(In reply to Ranjan Kumar from comment #17)
> @chris
> 
> Can you please help me in 
> 
> 1. Steps how to reproduce this issue? I tried to register 500 system at a
> time to Satellite 6.3.2 1.0 but could not found any similar error in log
> 
> 2. Do i need to follow comment 7 to test this fix?

I think comment #7 is an OK test, since it will show that the interface is filtered out.

Comment 19 Ranjan Kumar 2018-06-12 12:25:32 UTC
@chris

I tried registering 30 clients (10 in parallel) to satellite server 6.3.2.1.0 with each client having 10k facts. 

Here is script to create facts
-------
echo { >> /tmp/a.facts
for i in {1..10000}; do echo \"fact$i\": \"value$i\",; done >> /tmp/a.facts
echo } >> /tmp/a.facts
sed -zr 's/,([^,]*$)/\1/' /tmp/a.facts  > /etc/rhsm/facts/custom.facts

However, I didn't see any error mentioned in the Bugzilla.

Also, Tried comment 7 to test the fix. But i can see all the facts are listed under "net -> interface" after registration.

Here is output of hammer command
---------------------------------|----------------------------------|------
HOST                             | FACT                             | VALUE                                                                           
---------------------------------|----------------------------------|------
rhel7u4-5.gsslab.pek2.redhat.com | net::interface::dummy0                                                                                                           
rhel7u4-5.gsslab.pek2.redhat.com | net::interface::foo-veth-foo2                                                                                                    
rhel7u4-5.gsslab.pek2.redhat.com | net::interface::foo-veth-foo                                                                                                     
rhel7u4-5.gsslab.pek2.redhat.com | net::interface::foo-foo4                                                                                                         
rhel7u4-5.gsslab.pek2.redhat.com | net::interface::foo-veth-foo4                                                                                                    
rhel7u4-5.gsslab.pek2.redhat.com | net::interface::foo-foo2                                                                                                         
rhel7u4-5.gsslab.pek2.redhat.com | net::interface::foo-foo5                                                                                                         
rhel7u4-5.gsslab.pek2.redhat.com | net::interface::foo-veth-foo1                                                                                                    
rhel7u4-5.gsslab.pek2.redhat.com | net::interface::foo-veth-foo3                                                                                                    
rhel7u4-5.gsslab.pek2.redhat.com | net::interface::foo-foo3                                                                                                         
rhel7u4-5.gsslab.pek2.redhat.com | net::interface::foo-foo                                                                                                          
rhel7u4-5.gsslab.pek2.redhat.com | net::interface::foo-veth-foo5                                                                                                    
rhel7u4-5.gsslab.pek2.redhat.com | net::interface::foo-foo1   

Result truncated

Foreman version:
# rpm -qa | grep foreman
foreman-1.15.6.45-1.el7sat.noarch

I am unable to verify this bug using comment 7, let me know what to do next?

Comment 20 Chris Duryee 2018-06-12 14:09:24 UTC
re comment 19: I forgot to mention to filter for foo* in "Ignore interfaces with matching identifier" setting. I would give that a shot and register a new host to see if the interfaces show up in the fact list.

Comment 21 Mike McCune 2018-06-12 19:57:17 UTC
To filter net::interface facts coming in from subscription-manager you have to use a different notation because there is a bit of translation. The facts in subscription-manager are with dot notation, net.interface.foo, eg:

net.interface.foo-foo.mac_address: CE:44:54:F9:B8:6A

to filter these you need to use the value in the Settings as:

net.interface.foo*

eg:

[ lo, usb*, vnet*, macvtap*, _vdsmdummy_, veth*, docker*, tap*, qbr*, qvb*, qvo*, qr-*, qg-*, vlinuxbr*, vovsbr*, dummy*, net.interface.foo*]

I tested this and it worked well if you use the dot notation.

Comment 22 Ranjan Kumar 2018-06-13 06:02:12 UTC
Thaks Chris and McCune for the quick reply.

Verified! on Satellite 6.3.2 Snap 2.0

Verification Steps:
1. Built Satellite 6.3.2 Snap 2
2. Created dummy network interface on the client
   #modprobe dummy
   #ip link add name foo-veth-foo type dummy
   #ip link add name foo-foo type dummy
3. Added "net.interface.foo*" in the setting Settings -> Provisioning -> "Exclude pattern for facts stored in satellite"
4. Registerd client to satellite 
5. hammer fact list --search="host = rhel7u4-5.gsslab.pek2.redhat.com" | grep foo
   No facts found matching foo*

Detailed steps are attached

Comment 25 errata-xmlrpc 2018-06-19 20:17:00 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2018:1950