Hide Forgot
Description of problem: After update to 6.2.3 I noticed a number of pending (stuck) sync tasks and, while investigating the problem, I've found hammer hanging and not working. I re-ran satellite-installer to make sure everything is sane and two errors showed up: /Stage[main]/Foreman::Database/Foreman::Rake[db:seed]/Exec[foreman-rake-db:seed]: Failed to call refresh: /usr/sbin/foreman-rake d b:seed returned 1 instead of one of [0] /Stage[main]/Foreman::Database/Foreman::Rake[db:seed]/Exec[foreman-rake-db:seed]: /usr/sbin/foreman-rake db:seed returned 1 instea d of one of [0] /Stage[main]/Foreman_proxy::Register/Foreman_smartproxy[redhat-wb-p2.systems.private]: Could not evaluate: Proxy redhat-wb-p2.syst ems.private cannot be registered (SSL_connect SYSCALL returned=5 errno=0 state=SSLv2/v3 read server hello A): N/A Report processor failed: Could not send report to Foreman at https://redhat-wb-p2.systems.private/api/reports: SSL_connect SYSCALL returned=5 errno=0 state=SSLv2/v3 read server hello A The first, I've seen it before, and sometimes it's there sometime not, not causing issues: if I run foreman-rake db:seed --trace, it finishes with "All seed files executed". The second, seems foreman api is unavailable. Web UI is working, as also the repos and systems registration. Version-Release number of selected component (if applicable): How reproducible: I don't have a snapshot before the upgrade and a spare environment to reproduce it. Steps to Reproduce: 1. Upgrade to 6.2.3 2. 3. Actual results: Foreman API unavailable. Expected results: Foreman API back to work after calling satellite-install. Additional info:
hammer hangs, and after 20-30 seconds: bash-4.2# hammer task list --page 1 --per-page 10 [Foreman] Password for admin: Error: Request Timeout
Given the urgency, can you please open a support case on this? Thanks!
Also, could you verify few more things (please attach answers to the case too if you'll open it) 1) Is hammer ran on the same host where Satellite runs? 2) Double check firewall configuration on Satellite server, timeout can mean that incoming communication packets are simply dropped 3) Do you see any new lines in `tail -f ~foreman/production.log` while running a hammer command? 4) Could you share output of `hammer -d task list --page 1 --per-page 10` or other hammer command, e.g. `hammer -d user list`, feel free to clear any sensitive information like URLs
Thank you for coming back soon and sorry for the severity. hammer problems where caused by a completely unrelated thing: A game called 0ad has been pushed to EPEL 7Server yesterday (two rpms totalling 1.1GB), and the repo update caused spikes in CPU usage and memory usage to nearly 100%, also causing a couple of OOM kills to ruby. Do you think it makes sense to open a different bug report? The error reported from satellite-installer is there, but actually I don't have any side effect for now. I leave the bug open and change the severity.
Hi Michele, Based on the feedback in comment 4, a separate bug report would be recommended. If it is urgent, please be sure to open a support case. Thanks!
And this could be closed perhaps?
If no one needs to look into this - the fact that very large package downloads can brick satellite 6 for a while - yes it can be closed
Moving to content component so it can be reviewed there. Maybe there are some possible optimizations for large packages. Also tuning background worker process might help.
Thanks again for the bugzilla report. We've tested this upstream (which will be the basis for Satellite 6.3) and were unable to reproduce; therefore, we are going to close the bugzilla. If the issue reoccurs, please let us know along with the appropriate details (e.g. release version, steps used, specific RPMs...etc).