Bug 1167261 - Nightly installation failing: foreman-rake-db:migrate: cannot load such file -- v8
Summary: Nightly installation failing: foreman-rake-db:migrate: cannot load such file ...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Satellite
Classification: Red Hat
Component: Packaging
Version: Nightly
Hardware: Unspecified
OS: Unspecified
high
high
Target Milestone: Unspecified
Assignee: Katello Bug Bin
QA Contact: Elyézer Rezende
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2014-11-24 10:33 UTC by Sachin Ghai
Modified: 2017-02-23 20:46 UTC (History)
6 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2015-08-12 05:19:50 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)
on browsing the URL, UI says: Web application could not be started (116.24 KB, image/png)
2014-11-24 10:33 UTC, Sachin Ghai
no flags Details
foreman-debug logs (197.51 KB, application/x-xz)
2014-11-24 11:14 UTC, Sachin Ghai
no flags Details
complete installer logs are attached. (1.25 MB, text/plain)
2014-11-24 11:30 UTC, Sachin Ghai
no flags Details


Links
System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHSA-2015:1592 0 normal SHIPPED_LIVE Important: Red Hat Satellite 6.1.1 on RHEL 6 2015-08-12 09:04:35 UTC

Description Sachin Ghai 2014-11-24 10:33:03 UTC
Created attachment 960651 [details]
on browsing the URL, UI says: Web application could not be started

Description of problem:
foreman nightly install failed with below errors as per production.log:

---
Creating scope :completer_scope. Overwriting existing method Location.completer_scope.
Creating scope :completer_scope. Overwriting existing method Organization.completer_scope.
Database was not initialized yet: skipping smart proxy katello extension
PGError: ERROR:  relation "template_kinds" does not exist
LINE 4:              WHERE a.attrelid = '"template_kinds"'::regclass
                                        ^
:             SELECT a.attname, format_type(a.atttypid, a.atttypmod), d.adsrc, a.attnotnull
              FROM pg_attribute a LEFT JOIN pg_attrdef d
                ON a.attrelid = d.adrelid AND a.attnum = d.adnum
             WHERE a.attrelid = '"template_kinds"'::regclass
               AND a.attnum > 0 AND NOT a.attisdropped
             ORDER BY a.attnum

Connecting to database specified by database.yml
ENCRYPTION_KEY is not defined, so encryption is turned off for ComputeResource.
PGError: ERROR:  relation "users" does not exist
LINE 1: SELECT id FROM "users"
                       ^
: SELECT id FROM "users"
----


Version-Release number of selected component (if applicable):
recent nightly
* apr-util-ldap-1.3.9-3.el6_0.1.x86_64
* candlepin-0.9.34-1.el6.noarch
* candlepin-common-1.0.14-1.el6.noarch
* candlepin-selinux-0.9.34-1.el6.noarch
* candlepin-tomcat6-0.9.34-1.el6.noarch
* elasticsearch-0.90.10-7.el6.noarch
* foreman-1.8.0-0.develop.201411211432giteaefbec.el6.noarch
* foreman-compute-1.8.0-0.develop.201411211432giteaefbec.el6.noarch
* foreman-gce-1.8.0-0.develop.201411211432giteaefbec.el6.noarch
* foreman-libvirt-1.8.0-0.develop.201411211432giteaefbec.el6.noarch
* foreman-ovirt-1.8.0-0.develop.201411211432giteaefbec.el6.noarch
* foreman-postgresql-1.8.0-0.develop.201411211432giteaefbec.el6.noarch
* foreman-proxy-1.8.0-0.develop.201411211342git98b0dd3.el6.noarch
* foreman-release-1.8.0-0.develop.201411211432giteaefbec.el6.noarch
* foreman-selinux-1.8.0-0.develop.201410280941git10de1c5.el6.noarch
* foreman-vmware-1.8.0-0.develop.201411211432giteaefbec.el6.noarch
* katello-2.1.0-1.201411061509gitb0b8f43.el6.noarch
* katello-certs-tools-2.0.1-1.el6.noarch
* katello-default-ca-1.0-1.noarch
* katello-installer-2.1.0-1.201411202327gitb8eaaba.el6.noarch
* katello-repos-2.1.1-1.el6.noarch
* katello-server-ca-1.0-1.noarch
* openldap-2.4.39-8.el6.x86_64
* pulp-docker-plugins-0.2.1-0.2.beta.el6.noarch
* pulp-katello-0.3-3.el6.noarch
* pulp-nodes-common-2.5.0-0.7.beta.el6.noarch
* pulp-nodes-parent-2.5.0-0.7.beta.el6.noarch
* pulp-puppet-plugins-2.5.0-0.7.beta.el6.noarch
* pulp-puppet-tools-2.5.0-0.7.beta.el6.noarch
* pulp-rpm-plugins-2.5.0-0.7.beta.el6.noarch
* pulp-selinux-2.5.0-0.7.beta.el6.noarch
* pulp-server-2.5.0-0.7.beta.el6.noarch
* python-ldap-2.3.10-1.el6.x86_64
* ruby193-rubygem-ldap_fluff-0.3.3-1.el6.noarch
* ruby193-rubygem-net-ldap-0.3.1-2.el6.noarch
* ruby193-rubygem-runcible-1.3.0-1.el6.noarch
* rubygem-hammer_cli-0.1.3-1.201411181453git0c80831.el6.noarch
* rubygem-hammer_cli_foreman-0.1.3-1.201411121216git9381fc5.el6.noarch
* rubygem-hammer_cli_foreman_tasks-0.0.3-2.201409091410gitc96619d.git.0.37f3704.el6.noarch
* rubygem-hammer_cli_import-0.10.4-1.el6.noarch
* rubygem-hammer_cli_katello-0.0.6-1.201411211442git7cfe40c.git.0.0d94acd.el6.noarch



How reproducible:
always

Comment 1 RHEL Program Management 2014-11-24 10:33:41 UTC
Since this issue was entered in Red Hat Bugzilla, the release flag has been
set to ? to ensure that it is properly evaluated for this release.

Comment 3 Dominic Cleal 2014-11-24 10:57:35 UTC
Did the installation complete successfully?  Any installer errors?  Please attach installer logs and foreman-debug.

Comment 4 Sachin Ghai 2014-11-24 11:14:40 UTC
Created attachment 960661 [details]
foreman-debug logs

Comment 5 Dominic Cleal 2014-11-24 11:19:33 UTC
Logs appear to show an error, so I presume the installer also showed an error when you ran it?

[ WARN 2014-11-24 03:11:25 main]  /Stage[main]/Foreman::Database/Foreman::Rake[db:migrate]/Exec[foreman-rake-db:migrate]/returns: [WARNING] Please install gem 'therubyracer' to use Less.
[ WARN 2014-11-24 03:11:25 main]  /Stage[main]/Foreman::Database/Foreman::Rake[db:migrate]/Exec[foreman-rake-db:migrate]/returns: rake aborted!
[ WARN 2014-11-24 03:11:25 main]  /Stage[main]/Foreman::Database/Foreman::Rake[db:migrate]/Exec[foreman-rake-db:migrate]/returns: Gem loading error: cannot load such file -- v8
[ WARN 2014-11-24 03:11:25 main]  /Stage[main]/Foreman::Database/Foreman::Rake[db:migrate]/Exec[foreman-rake-db:migrate]/returns: 
[ WARN 2014-11-24 03:11:25 main]  /Stage[main]/Foreman::Database/Foreman::Rake[db:migrate]/Exec[foreman-rake-db:migrate]/returns: (See full trace by running task with --trace)

Comment 6 Sachin Ghai 2014-11-24 11:25:11 UTC
Yeah.. installer failed with below errors. thanks for quick ack.

Installer log errors:

---
[ERROR 2014-11-24 03:19:40 main] Repeating errors encountered during run:
[ERROR 2014-11-24 03:19:40 main]  /Stage[main]/Foreman::Database/Foreman::Rake[db:migrate]/Exec[foreman-rake-db:migrate]: Failed to call refresh: /usr/sbin/foreman-rake db:migrate returned 1 instead of one of [0]
[ERROR 2014-11-24 03:19:40 main]  /Stage[main]/Foreman::Database/Foreman::Rake[db:migrate]/Exec[foreman-rake-db:migrate]: /usr/sbin/foreman-rake db:migrate returned 1 instead of one of [0]
[ERROR 2014-11-24 03:19:40 main]  /Stage[main]/Foreman::Database/Foreman::Rake[db:seed]/Exec[foreman-rake-db:seed]: Failed to call refresh: /usr/sbin/foreman-rake db:seed returned 1 instead of one of [0]
[ERROR 2014-11-24 03:19:40 main]  /Stage[main]/Foreman::Database/Foreman::Rake[db:seed]/Exec[foreman-rake-db:seed]: /usr/sbin/foreman-rake db:seed returned 1 instead of one of [0]
[ERROR 2014-11-24 03:19:40 main]  /Stage[main]/Foreman::Database/Foreman::Rake[apipie:cache]/Exec[foreman-rake-apipie:cache]: Failed to call refresh: /usr/sbin/foreman-rake apipie:cache returned 1 instead of one of [0]
[ERROR 2014-11-24 03:19:40 main]  /Stage[main]/Foreman::Database/Foreman::Rake[apipie:cache]/Exec[foreman-rake-apipie:cache]: /usr/sbin/foreman-rake apipie:cache returned 1 instead of one of [0]
[ERROR 2014-11-24 03:19:40 main]  /usr/share/foreman/script/foreman-config      -k ssl_ca_file -v '/etc/foreman/proxy_ca.pem'      -k ssl_certificate -v '/etc/foreman/client_cert.pem'      -k ssl_priv_key -v '/etc/foreman/client_key.pem' returned 1 instead of one of [0]
[ERROR 2014-11-24 03:19:40 main]  /Stage[main]/Certs::Foreman/Exec[foreman_certs_config]/returns: change from notrun to 0 failed: /usr/share/foreman/script/foreman-config      -k ssl_ca_file -v '/etc/foreman/proxy_ca.pem'      -k ssl_certificate -v '/etc/foreman/client_cert.pem'      -k ssl_priv_key -v '/etc/foreman/client_key.pem' returned 1 instead of one of [0]
[ERROR 2014-11-24 03:19:40 main]  /Stage[main]/Foreman_proxy::Register/Foreman_smartproxy[qe-foreman-rhel65.usersys.redhat.com]: Could not evaluate: Could not load data from https://qe-foreman-rhel65.usersys.redhat.com
[ERROR 2014-11-24 03:19:40 main]  /Stage[main]/Foreman_proxy::Register/Foreman_smartproxy[qe-foreman-rhel65.usersys.redhat.com]: Failed to call refresh: Could not load data from https://qe-foreman-rhel65.usersys.redhat.com
[ERROR 2014-11-24 03:19:40 main]  /Stage[main]/Foreman_proxy::Register/Foreman_smartproxy[qe-foreman-rhel65.usersys.redhat.com]: Could not load data from https://qe-foreman-rhel65.usersys.redhat.com
---

will attach the complete installer log file shortly

Comment 7 Sachin Ghai 2014-11-24 11:30:34 UTC
Created attachment 960684 [details]
complete installer logs are attached.

Comment 9 Justin Sherrill 2014-11-24 15:00:47 UTC
First error i see in the logs:

Exec[foreman-rake-db:migrate]/returns: [WARNING] Please install gem 'therubyracer' to use Less.
Exec[foreman-rake-db:migrate]/returns: rake aborted!
Exec[foreman-rake-db:migrate]/returns: Gem loading error: cannot load such file -- v8
Exec[foreman-rake-db:migrate]/returns: 
Exec[foreman-rake-db:migrate]/returns: (See full trace by running task with --trace)
Exec[foreman-rake-db:migrate]: Failed to call refresh: /usr/sbin/foreman-rake db:migrate returned 1 instead of one of [0]
Exec[foreman-rake-db:migrate]: /usr/sbin/foreman-rake db:migrate returned 1 instead of one of [0]

Comment 10 Justin Sherrill 2014-11-24 15:04:17 UTC
I don't see v8314-v8  installed, which seems odd, nor  ruby193-rubygem-therubyracer

yet foreman-1.8.0-0.develop.201411211432giteaefbec.el6.noarch  is installed....

Comment 11 Dominic Cleal 2014-11-24 16:12:44 UTC
(In reply to Justin Sherrill from comment #10)
> I don't see v8314-v8  installed, which seems odd, nor 
> ruby193-rubygem-therubyracer
> 
> yet foreman-1.8.0-0.develop.201411211432giteaefbec.el6.noarch  is
> installed....

I removed it from Foreman's dependencies in an effort to minimise our package set, it isn't a runtime dep for the core app, only built time.  If Katello needs it for something at runtime, best add it again there, or check which dep needs it and add it to that package.

Comment 12 Justin Sherrill 2014-11-24 16:56:53 UTC
fixed here: 

https://github.com/Katello/katello/pull/4830

Comment 13 Dominic Cleal 2014-11-25 09:55:11 UTC
Please confirm this is working now, and we can close it as UPSTREAM.

Comment 14 Sachin Ghai 2014-11-26 07:49:45 UTC
Its still broken with same issue:

--
[10.8.30.192] out: [ WARN 2014-11-26 02:35:52 verbose]  /Stage[main]/Foreman::Database/Foreman::Rake[db:migrate]/Exec[foreman-rake-db:migrate]/returns: Please install the 'sass' gem if you want to use Sass.
[10.8.30.192] out: [ WARN 2014-11-26 02:35:52 verbose]  /Stage[main]/Foreman::Database/Foreman::Rake[db:migrate]/Exec[foreman-rake-db:migrate]/returns: 
[10.8.30.192] out: [ WARN 2014-11-26 02:35:52 verbose]  /Stage[main]/Foreman::Database/Foreman::Rake[db:migrate]/Exec[foreman-rake-db:migrate]/returns: [WARNING] Please install gem 'therubyracer' to use Less.
[10.8.30.192] out: [ WARN 2014-11-26 02:35:52 verbose]  /Stage[main]/Foreman::Database/Foreman::Rake[db:migrate]/Exec[foreman-rake-db:migrate]/returns: rake aborted!
[10.8.30.192] out: [ WARN 2014-11-26 02:35:52 verbose]  /Stage[main]/Foreman::Database/Foreman::Rake[db:migrate]/Exec[foreman-rake-db:migrate]/returns: Gem loading error: cannot load such file -- v8
[10.8.30.192] out: [ WARN 2014-11-26 02:35:52 verbose]  /Stage[main]/Foreman::Database/Foreman::Rake[db:migrate]/Exec[foreman-rake-db:migrate]/returns: 
[10.8.30.192] out: [ WARN 2014-11-26 02:35:52 verbose]  /Stage[main]/Foreman::Database/Foreman::Rake[db:migrate]/Exec[foreman-rake-db:migrate]/returns: (See full trace by running task with --trace)
[10.8.30.192] out: [ERROR 2014-11-26 02:35:52 verbose]  /Stage[main]/Foreman::Database/Foreman::Rake[db:migrate]/Exec[foreman-rake-db:migrate]: Failed to call refresh: /usr/sbin/foreman-rake db:migrate returned 1 instead of one of [0]
[10.8.30.192] out: [ERROR 2014-11-26 02:35:52 verbose]  /Stage[main]/Foreman::Database/Foreman::Rake[db:migrate]/Exec[foreman-rake-db:migrate]: /usr/sbin/foreman-rake db:migrate returned 1 instead of one of [0]
---

Comment 15 Sachin Ghai 2014-11-26 07:51:45 UTC
These are the logs from latest nightly:

--
[10.8.30.192] out: [ERROR 2014-11-26 02:45:03 verbose] Repeating errors encountered during run:
[10.8.30.192] out: [ERROR 2014-11-26 02:45:03 verbose]  /Stage[main]/Foreman::Database/Foreman::Rake[db:migrate]/Exec[foreman-rake-db:migrate]: Failed to call refresh: /usr/sbin/foreman-rake db:migrate returned 1 instead of one of [0]
[10.8.30.192] out: [ERROR 2014-11-26 02:45:03 verbose]  /Stage[main]/Foreman::Database/Foreman::Rake[db:migrate]/Exec[foreman-rake-db:migrate]: /usr/sbin/foreman-rake db:migrate returned 1 instead of one of [0]
[10.8.30.192] out: [ERROR 2014-11-26 02:45:03 verbose]  /Stage[main]/Foreman::Database/Foreman::Rake[db:seed]/Exec[foreman-rake-db:seed]: Failed to call refresh: /usr/sbin/foreman-rake db:seed returned 1 instead of one of [0]
[10.8.30.192] out: [ERROR 2014-11-26 02:45:03 verbose]  /Stage[main]/Foreman::Database/Foreman::Rake[db:seed]/Exec[foreman-rake-db:seed]: /usr/sbin/foreman-rake db:seed returned 1 instead of one of [0]
[10.8.30.192] out: [ERROR 2014-11-26 02:45:03 verbose]  /Stage[main]/Foreman::Database/Foreman::Rake[apipie:cache]/Exec[foreman-rake-apipie:cache]: Failed to call refresh: /usr/sbin/foreman-rake apipie:cache returned 1 instead of one of [0]
[10.8.30.192] out: [ERROR 2014-11-26 02:45:03 verbose]  /Stage[main]/Foreman::Database/Foreman::Rake[apipie:cache]/Exec[foreman-rake-apipie:cache]: /usr/sbin/foreman-rake apipie:cache returned 1 instead of one of [0]
[10.8.30.192] out: [ERROR 2014-11-26 02:45:03 verbose]  /usr/share/foreman/script/foreman-config      -k ssl_ca_file -v '/etc/foreman/proxy_ca.pem'      -k ssl_certificate -v '/etc/foreman/client_cert.pem'      -k ssl_priv_key -v '/etc/foreman/client_key.pem' returned 1 instead of one of [0]
[10.8.30.192] out: [ERROR 2014-11-26 02:45:03 verbose]  /Stage[main]/Certs::Foreman/Exec[foreman_certs_config]/returns: change from notrun to 0 failed: /usr/share/foreman/script/foreman-config      -k ssl_ca_file -v '/etc/foreman/proxy_ca.pem'      -k ssl_certificate -v '/etc/foreman/client_cert.pem'      -k ssl_priv_key -v '/etc/foreman/client_key.pem' returned 1 instead of one of [0]
[10.8.30.192] out: [ERROR 2014-11-26 02:45:03 verbose]  /Stage[main]/Foreman_proxy::Register/Foreman_smartproxy[qe-foreman-rhel66.usersys.redhat.com]: Could not evaluate: Could not load data from https://qe-foreman-rhel66.usersys.redhat.com
[10.8.30.192] out: [ERROR 2014-11-26 02:45:03 verbose]  /Stage[main]/Foreman_proxy::Register/Foreman_smartproxy[qe-foreman-rhel66.usersys.redhat.com]: Failed to call refresh: Could not load data from https://qe-foreman-rhel66.usersys.redhat.com
[10.8.30.192] out: [ERROR 2014-11-26 02:45:03 verbose]  /Stage[main]/Foreman_proxy::Register/Foreman_smartproxy[qe-foreman-rhel66.usersys.redhat.com]: Could not load data from https://qe-foreman-rhel66.usersys.redhat.com
--

Comment 16 Sachin Ghai 2014-11-26 08:45:21 UTC
Comment15 has logs from nightly installed on rhel66..

And I found similar errors with nightly installed on rhel65. Please see the logs below from rhel65 install

--
[ WARN 2014-11-26 03:15:02 main]  /Stage[main]/Foreman::Database/Foreman::Rake[apipie:cache]/Exec[foreman-rake-apipie:cache]/returns: Gem loading error: cannot load such file -- v8
[ERROR 2014-11-26 03:15:02 main]  /Stage[main]/Foreman::Database/Foreman::Rake[apipie:cache]/Exec[foreman-rake-apipie:cache]: Failed to call refresh: /usr/sbin/foreman-rake apipie:cache returned 1 instead of one of [0]
[ERROR 2014-11-26 03:15:02 main]  /Stage[main]/Foreman::Database/Foreman::Rake[apipie:cache]/Exec[foreman-rake-apipie:cache]: /usr/sbin/foreman-rake apipie:cache returned 1 instead of one of [0]
--

Comment 17 Sachin Ghai 2014-12-02 08:02:35 UTC
The latest nightly is broken but with a bit different error:

--
[10.8.29.46] out: [ WARN 2014-12-02 02:54:20 verbose]  /Stage[main]/Foreman::Database/Foreman::Rake[db:migrate]/Exec[foreman-rake-db:migrate]/returns: rake aborted!
[10.8.29.46] out: [ WARN 2014-12-02 02:54:20 verbose]  /Stage[main]/Foreman::Database/Foreman::Rake[db:migrate]/Exec[foreman-rake-db:migrate]/returns: uninitialized constant Haml::Util::Sass
[10.8.29.46] out: [ WARN 2014-12-02 02:54:20 verbose]  /Stage[main]/Foreman::Database/Foreman::Rake[db:migrate]/Exec[foreman-rake-db:migrate]/returns: 
[10.8.29.46] out: [ WARN 2014-12-02 02:54:20 verbose]  /Stage[main]/Foreman::Database/Foreman::Rake[db:migrate]/Exec[foreman-rake-db:migrate]/returns: (See full trace by running task with --trace)
[10.8.29.46] out: [ERROR 2014-12-02 02:54:20 verbose]  /Stage[main]/Foreman::Database/Foreman::Rake[db:migrate]/Exec[foreman-rake-db:migrate]: Failed to call refresh: /usr/sbin/foreman-rake db:migrate returned 1 instead of one of [0]
[10.8.29.46] out: [ERROR 2014-12-02 02:54:20 verbose]  /Stage[main]/Foreman::Database/Foreman::Rake[db:migrate]/Exec[foreman-rake-db:migrate]: /usr/sbin/foreman-rake db:migrate returned 1 instead of one of [0]
--


If these errors are totally different from the earlier one..then I'll file a separate bz..?

Comment 20 Elyézer Rezende 2015-03-06 21:16:53 UTC
Verified on: Nightly build from 2015-03-06

Steps to verify:

Installed the nightly build on both RHEL 6.6 and RHEL 7.1. The installer completed successfully.

Was checked in the provisioning jobs too and the nightly installer and also compose, installer is working for at least the past 5 builds.

Comment 21 Bryan Kearney 2015-08-11 13:29:01 UTC
This bug is slated to be released with Satellite 6.1.

Comment 22 errata-xmlrpc 2015-08-12 05:19:50 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2015:1592


Note You need to log in before you can comment on or make changes to this bug.