Bug 1465942 - unable to connect to database pod because the database.yaml isn't configured correctly
unable to connect to database pod because the database.yaml isn't configured ...
Status: NEW
Product: Red Hat CloudForms Management Engine
Classification: Red Hat
Component: Documentation (Show other bugs)
5.8.0
All All
high Severity medium
: GA
: cfme-future
Assigned To: Red Hat CloudForms Documentation
Red Hat CloudForms Documentation
container:pod:database:doc
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2017-06-28 09:44 EDT by Felix Dewaleyne
Modified: 2018-05-04 15:39 EDT (History)
8 users (show)

See Also:
Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed:
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)
cfme-on-ocp-failed-to-start.txt (15.23 KB, text/plain)
2017-06-28 09:44 EDT, Felix Dewaleyne
no flags Details
evm.log (44.29 KB, text/plain)
2017-06-28 10:53 EDT, Felix Dewaleyne
no flags Details

  None (edit)
Description Felix Dewaleyne 2017-06-28 09:44:45 EDT
Created attachment 1292654 [details]
cfme-on-ocp-failed-to-start.txt

Description of problem:
When deploying Cloudforms 4.5 in OCP it 

Version-Release number of selected component (if applicable):
cfme-5.8.0.17

How reproducible:
all the time

Steps to Reproduce:
1.deploy cloudforms as an OCP container
2.
3.

Actual results:
Jun 28 11:13:53 cloudforms-4-nfs70 sh[752]: Starting EVM...
Jun 28 11:13:53 cloudforms-4-nfs70 sh[752]: rake aborted!
Jun 28 11:13:53 cloudforms-4-nfs70 sh[752]: PG::ConnectionBad: could not connect to server: No such file or directory
Jun 28 11:13:53 cloudforms-4-nfs70 sh[752]: Is the server running locally and accepting
Jun 28 11:13:53 cloudforms-4-nfs70 sh[752]: connections on Unix domain socket "/var/run/postgresql/.s.PGSQL.5432"?
Jun 28 11:13:53 cloudforms-4-nfs70 sh[752]: /opt/rh/rh-ruby23/root/usr/share/gems/gems/pg-0.18.2/lib/pg.rb:45:in `initialize'
Jun 28 11:13:53 cloudforms-4-nfs70 sh[752]: /opt/rh/rh-ruby23/root/usr/share/gems/gems/pg-0.18.2/lib/pg.rb:45:in `new'
Jun 28 11:13:53 cloudforms-4-nfs70 sh[752]: /opt/rh/rh-ruby23/root/usr/share/gems/gems/pg-0.18.2/lib/pg.rb:45:in `connect'
Jun 28 11:13:53 cloudforms-4-nfs70 sh[752]: /opt/rh/cfme-gemset/gems/activerecord-5.0.3/lib/active_record/connection_adapters/postgresql_adapter.rb:671:in `connect'
Jun 28 11:13:53 cloudforms-4-nfs70 sh[752]: /opt/rh/cfme-gemset/gems/activerecord-5.0.3/lib/active_record/connection_adapters/postgresql_adapter.rb:217:in `initialize'
Jun 28 11:13:53 cloudforms-4-nfs70 sh[752]: /opt/rh/cfme-gemset/gems/activerecord-5.0.3/lib/active_record/connection_adapters/postgresql_adapter.rb:37:in `new'
Jun 28 11:13:53 cloudforms-4-nfs70 sh[752]: /opt/rh/cfme-gemset/gems/activerecord-5.0.3/lib/active_record/connection_adapters/postgresql_adapter.rb:37:in `postgresql_connection'
Jun 28 11:13:53 cloudforms-4-nfs70 sh[752]: /opt/rh/cfme-gemset/gems/activerecord-5.0.3/lib/active_record/connection_adapters/abstract/connection_pool.rb:729:in `new_connection'
Jun 28 11:13:53 cloudforms-4-nfs70 sh[752]: /opt/rh/cfme-gemset/gems/activerecord-5.0.3/lib/active_record/connection_adapters/abstract/connection_pool.rb:773:in `checkout_new_connection'
Jun 28 11:13:53 cloudforms-4-nfs70 sh[752]: /opt/rh/cfme-gemset/gems/activerecord-5.0.3/lib/active_record/connection_adapters/abstract/connection_pool.rb:752:in `try_to_checkout_new_connection'
Jun 28 11:13:53 cloudforms-4-nfs70 sh[752]: /opt/rh/cfme-gemset/gems/activerecord-5.0.3/lib/active_record/connection_adapters/abstract/connection_pool.rb:713:in `acquire_connection'
Jun 28 11:13:53 cloudforms-4-nfs70 sh[752]: /opt/rh/cfme-gemset/gems/activerecord-5.0.3/lib/active_record/connection_adapters/abstract/connection_pool.rb:490:in `checkout'
Jun 28 11:13:53 cloudforms-4-nfs70 sh[752]: /opt/rh/cfme-gemset/gems/activerecord-5.0.3/lib/active_record/connection_adapters/abstract/connection_pool.rb:364:in `connection'
Jun 28 11:13:53 cloudforms-4-nfs70 sh[752]: /opt/rh/cfme-gemset/gems/activerecord-5.0.3/lib/active_record/connection_adapters/abstract/connection_pool.rb:883:in `retrieve_connection'
Jun 28 11:13:53 cloudforms-4-nfs70 sh[752]: /opt/rh/cfme-gemset/gems/activerecord-5.0.3/lib/active_record/connection_handling.rb:128:in `retrieve_connection'
Jun 28 11:13:53 cloudforms-4-nfs70 sh[752]: /opt/rh/cfme-gemset/gems/activerecord-5.0.3/lib/active_record/connection_handling.rb:91:in `connection'
Jun 28 11:13:53 cloudforms-4-nfs70 sh[752]: /opt/rh/cfme-gemset/gems/activerecord-5.0.3/lib/active_record/model_schema.rb:442:in `load_schema!'
Jun 28 11:13:53 cloudforms-4-nfs70 sh[752]: /opt/rh/cfme-gemset/gems/activerecord-5.0.3/lib/active_record/attributes.rb:233:in `load_schema!'
Jun 28 11:13:53 cloudforms-4-nfs70 sh[752]: /opt/rh/cfme-gemset/gems/activerecord-5.0.3/lib/active_record/attribute_decorators.rb:28:in `load_schema!'
Jun 28 11:13:53 cloudforms-4-nfs70 sh[752]: /var/www/miq/vmdb/lib/extensions/ar_virtual.rb:400:in `load_schema!'
Jun 28 11:13:53 cloudforms-4-nfs70 sh[752]: /opt/rh/cfme-gemset/gems/activerecord-5.0.3/lib/active_record/model_schema.rb:437:in `load_schema'
Jun 28 11:13:53 cloudforms-4-nfs70 sh[752]: /opt/rh/cfme-gemset/gems/activerecord-5.0.3/lib/active_record/model_schema.rb:339:in `columns_hash'
Jun 28 11:13:53 cloudforms-4-nfs70 sh[752]: /opt/rh/cfme-gemset/gems/activerecord-5.0.3/lib/active_record/core.rb:192:in `block in find_by'
Jun 28 11:13:53 cloudforms-4-nfs70 sh[752]: /opt/rh/cfme-gemset/gems/activerecord-5.0.3/lib/active_record/core.rb:192:in `each'
Jun 28 11:13:53 cloudforms-4-nfs70 sh[752]: /opt/rh/cfme-gemset/gems/activerecord-5.0.3/lib/active_record/core.rb:192:in `all?'
Jun 28 11:13:53 cloudforms-4-nfs70 sh[752]: /opt/rh/cfme-gemset/gems/activerecord-5.0.3/lib/active_record/core.rb:192:in `find_by'
Jun 28 11:13:53 cloudforms-4-nfs70 sh[752]: /var/www/miq/vmdb/app/models/miq_server.rb:531:in `block in <class:MiqServer>'
Jun 28 11:13:53 cloudforms-4-nfs70 sh[752]: /opt/rh/cfme-gemset/bundler/gems/manageiq-gems-pending-e0f3ea8755bf/lib/gems/pending/util/extensions/miq-module.rb:37:in `block (2 levels) in cache_with_timeout
Jun 28 11:13:53 cloudforms-4-nfs70 sh[752]: /opt/rh/cfme-gemset/bundler/gems/manageiq-gems-pending-e0f3ea8755bf/lib/gems/pending/util/extensions/miq-module.rb:25:in `block in cache_with_timeout'
Jun 28 11:13:53 cloudforms-4-nfs70 sh[752]: /var/www/miq/vmdb/app/models/miq_server/worker_management.rb:14:in `kill_all_workers'
Jun 28 11:13:53 cloudforms-4-nfs70 sh[752]: /var/www/miq/vmdb/lib/tasks/evm_application.rb:21:in `start'
Jun 28 11:13:53 cloudforms-4-nfs70 sh[752]: /var/www/miq/vmdb/lib/tasks/evm.rake:8:in `block (2 levels) in <top (required)>'
Jun 28 11:13:53 cloudforms-4-nfs70 sh[752]: /opt/rh/cfme-gemset/gems/rake-12.0.0/exe/rake:27:in `<top (required)>'
Jun 28 11:13:53 cloudforms-4-nfs70 sh[752]: Tasks: TOP => evm:start
Jun 28 11:13:53 cloudforms-4-nfs70 sh[752]: (See full trace by running task with --trace)
Jun 28 11:13:53 cloudforms-4-nfs70 systemd[1]: evmserverd.service: control process exited, code=exited status=1
Jun 28 11:13:53 cloudforms-4-nfs70 systemd[1]: Failed to start EVM server daemon.


Expected results:
able to connect to the database

Additional info:

the database.yaml production section does not have a "host" and a "paassword" set so evmserverd tries to use sockets, but the database isn't running in the pod.

the database.yaml is also not in the persistent storage
Comment 2 Felix Dewaleyne 2017-06-28 09:52:44 EDT
we run into this if following https://access.redhat.com/documentation/en-us/red_hat_cloudforms/4.5/html/installing_red_hat_cloudforms_on_openshift_container_platform/installing-cloudforms#prerequisites

but using https://github.com/ManageIQ/manageiq-pods allows to avoid the issue

changing to documentation issue - can otherwise be resolved by rebuilding the container
Comment 3 Felix Dewaleyne 2017-06-28 10:02:00 EDT
unsure if this is really a documentation issue or a problem with how deployments are done after further cross-reading of the two documentations.
Comment 5 Felix Dewaleyne 2017-06-28 10:29:36 EDT
using template from v1.5, fixing the cloudforms to cloudforms45 (see https://bugzilla.redhat.com/show_bug.cgi?id=1465828) , pulling the latest image we ran into this issue on the 28/06/2017
Comment 6 Felix Dewaleyne 2017-06-28 10:36:59 EDT
used to reproduce :

 oc create -f /usr/share/ansible/openshift-ansible/roles/openshift_examples/files/examples/v1.5/cfme-templates/cfme-template.yaml
 oc get templates
 oc process --parameters -n cfme58-in-ocp cloudforms
 oc new-app --template=cloudforms -p DATABASE_REGION=667 -p DATABASE_VOLUME_CAPACITY=14G -p APPLICATION_DOMAIN=cfme.cfhack.coe.muc.redhat.com -p APPLICATION_VOLUME_CAPACITY=14G -p MEMORY_APPLICATION_MIN=12G -p MEMORY_POSTGRESQL_LIMIT=4G
Comment 7 Felix Dewaleyne 2017-06-28 10:53 EDT
Created attachment 1292669 [details]
evm.log

fixing the database to not require authentication & injecting the database.yml with a host and password value, the next issue involves the v2 key.

the evm.log is attached - at this point we could not continue
Comment 8 Nick Carboni 2017-06-28 11:14:08 EDT
The CF 4.5 documentation should mostly match the upstream fine branch documentation (https://github.com/ManageIQ/manageiq-pods/blob/fine/README.md)

Looking as the master branch docs will cause some confusion as we are in the process of making some fairly significant changes.

The main issue I see with the current 4.5 docs is the line here:
> As the admin user, make two persistent volumes: one to host the Red Hat CloudForms database, and one to host the application data.

We require at least 3 volumes by default. One for the database, one shared across the region (which contains database.yml) and one per server.

This also does not take into account additional volumes needed when scaling to multiple server instances.
Comment 9 Felix Dewaleyne 2017-06-29 03:28:26 EDT
(In reply to Nick Carboni from comment #8)
> The CF 4.5 documentation should mostly match the upstream fine branch
> documentation (https://github.com/ManageIQ/manageiq-pods/blob/fine/README.md)
> 
> Looking as the master branch docs will cause some confusion as we are in the
> process of making some fairly significant changes.
> 
> The main issue I see with the current 4.5 docs is the line here:
> > As the admin user, make two persistent volumes: one to host the Red Hat CloudForms database, and one to host the application data.
> 
> We require at least 3 volumes by default. One for the database, one shared
> across the region (which contains database.yml) and one per server.
> 
> This also does not take into account additional volumes needed when scaling
> to multiple server instances.

we looked at the fine branch the other day, but unfortunately the templates didn't seem like they were making use of the persistent storage used. I would be more than happy to give this another try when we have a new template
Comment 10 Nick Carboni 2017-06-29 09:04:52 EDT
It sounds like the issue was hit because the only change made to the template was to update the repository from cloudforms to cloudforms45 when really all the changes in this patch [1] needed to be made as well. These changes include adding the region PV which would explain the issue you are seeing.

I'll leave a needinfo here for you until you can test this with the complete new template.

[1] https://github.com/openshift/openshift-ansible/pull/4334/files
Comment 11 Felix Dewaleyne 2017-07-11 05:59:16 EDT
(In reply to Nick Carboni from comment #10)
> It sounds like the issue was hit because the only change made to the
> template was to update the repository from cloudforms to cloudforms45 when
> really all the changes in this patch [1] needed to be made as well. These
> changes include adding the region PV which would explain the issue you are
> seeing.
> 
> I'll leave a needinfo here for you until you can test this with the complete
> new template.
> 
> [1] https://github.com/openshift/openshift-ansible/pull/4334/files

I'll pass that to bz#1465828
Comment 12 Andrew Dahms 2017-09-19 02:18:40 EDT
Moving back to the default assignee before reviewing this bug and triaging.

Note You need to log in before you can comment on or make changes to this bug.