Bug 1390237

Summary:	Foreman api failing after upgrade to 6.2.3
Product:	Red Hat Satellite	Reporter:	Michele <michelefrettoli>
Component:	Content Management	Assignee:	satellite6-bugs <satellite6-bugs>
Status:	CLOSED NEXTRELEASE	QA Contact:	Katello QA List <katello-qa-list>
Severity:	medium	Docs Contact:
Priority:	high
Version:	6.2.3	CC:	bbuckingham, bkearney, jsherril, michelefrettoli
Target Milestone:	Unspecified	Keywords:	PrioBumpField, PrioBumpGSS, Triaged
Target Release:	Unused
Hardware:	x86_64
OS:	Linux
Whiteboard:
Fixed In Version:		Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2017-09-22 17:51:44 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:

Description Michele 2016-10-31 14:37:21 UTC

Description of problem:
After update to 6.2.3 I noticed a number of pending (stuck) sync tasks and, while investigating the problem, I've found hammer hanging and not working.
I re-ran satellite-installer to make sure everything is sane and two errors showed up:

/Stage[main]/Foreman::Database/Foreman::Rake[db:seed]/Exec[foreman-rake-db:seed]: Failed to call refresh: /usr/sbin/foreman-rake d
b:seed returned 1 instead of one of [0]
/Stage[main]/Foreman::Database/Foreman::Rake[db:seed]/Exec[foreman-rake-db:seed]: /usr/sbin/foreman-rake db:seed returned 1 instea
d of one of [0]

/Stage[main]/Foreman_proxy::Register/Foreman_smartproxy[redhat-wb-p2.systems.private]: Could not evaluate: Proxy redhat-wb-p2.syst
ems.private cannot be registered (SSL_connect SYSCALL returned=5 errno=0 state=SSLv2/v3 read server hello A): N/A
Report processor failed: Could not send report to Foreman at https://redhat-wb-p2.systems.private/api/reports: SSL_connect SYSCALL
returned=5 errno=0 state=SSLv2/v3 read server hello A

The first, I've seen it before, and sometimes it's there sometime not, not causing issues: if I run foreman-rake db:seed --trace, it finishes with "All seed files executed".

The second, seems foreman api is unavailable. Web UI is working, as also the repos and systems registration.

Version-Release number of selected component (if applicable):


How reproducible:

I don't have a snapshot before the upgrade and a spare environment to reproduce it.

Steps to Reproduce:
1. Upgrade to 6.2.3
2.
3.

Actual results:

Foreman API unavailable.

Expected results:

Foreman API back to work after calling satellite-install.

Additional info:

Comment 1 Michele 2016-10-31 15:49:44 UTC

hammer hangs, and after 20-30 seconds:

bash-4.2# hammer task list --page 1 --per-page 10
[Foreman] Password for admin: 
Error: Request Timeout

Comment 2 Bryan Kearney 2016-10-31 17:52:39 UTC

Given the urgency, can you please open a support case on this? Thanks!

Comment 3 Marek Hulan 2016-11-01 07:41:21 UTC

Also, could you verify few more things (please attach answers to the case too if you'll open it)

1) Is hammer ran on the same host where Satellite runs?
2) Double check firewall configuration on Satellite server, timeout can mean that incoming communication packets are simply dropped
3) Do you see any new lines in `tail -f ~foreman/production.log` while running a hammer command?
4) Could you share output of `hammer -d task list --page 1 --per-page 10` or other hammer command, e.g. `hammer -d user list`, feel free to clear any sensitive information like URLs

Comment 4 Michele 2016-11-01 17:05:11 UTC

Thank you for coming back soon and sorry for the severity. hammer problems where caused by a completely unrelated thing:
A game called 0ad has been pushed to EPEL 7Server yesterday (two rpms totalling 1.1GB), and the repo update caused spikes in CPU usage and memory usage to nearly 100%, also causing a couple of OOM kills to ruby. Do you think it makes sense to open a different bug report?

The error reported from satellite-installer is there, but actually I don't have any side effect for now. I leave the bug open and change the severity.

Comment 5 Brad Buckingham 2016-11-01 19:42:29 UTC

Hi Michele,

Based on the feedback in comment 4, a separate bug report would be recommended.  If it is urgent, please be sure to open a support case.  Thanks!

Comment 6 Marek Hulan 2016-11-02 09:26:27 UTC

And this could be closed perhaps?

Comment 7 Michele 2016-11-28 12:36:52 UTC

If no one needs to look into this - the fact that very large package downloads can brick satellite 6 for a while - yes it can be closed

Comment 8 Marek Hulan 2016-11-28 13:32:23 UTC

Moving to content component so it can be reviewed there. Maybe there are some possible optimizations for large packages. Also tuning background worker process might help.

Comment 12 Brad Buckingham 2017-09-22 17:51:44 UTC

Thanks again for the bugzilla report.  We've tested this upstream (which will be the basis for Satellite 6.3) and were unable to reproduce; therefore, we are going to close the bugzilla.  If the issue reoccurs, please let us know along with the appropriate details (e.g. release version, steps used, specific RPMs...etc).