Created attachment 915203 [details] Comment (This comment was longer than 65,535 characters and has been moved to an attachment by Red Hat Bugzilla).
== LOGS == [root@dell-pe1950-1 log]# cat aeolus-conductor/ aeolus-connector.log condor_refreshd.log dbomatic.log mongrel.log rails.log thin.log warehouse_sync.log [root@dell-pe1950-1 log]# cat aeolus-conductor/thin.log >> Writing PID to /var/run/aeolus-conductor/thin.pid >> Changing process privilege to aeolus:aeolus >> Using rails adapter >> Thin web server (v1.2.5 codename This Is Not A Web Server) >> Maximum connections set to 1024 >> Listening on 127.0.0.1:3000, CTRL+C to stop [root@dell-pe1950-1 log]# cat aeolus-conductor/rails.log Processing Resources::DeploymentsController#index (for 127.0.0.1 at 2011-04-04 09:05:25) [GET] Parameters: {"action"=>"index", "controller"=>"resources/deployments"} Redirected to http://127.0.0.1:3000/conductor/login Filter chain halted as [:require_user] rendered_or_redirected. Completed in 13ms (DB: 2) | 302 Found [http://127.0.0.1/conductor] Processing Resources::DeploymentsController#index (for 10.11.228.65 at 2011-04-04 09:08:40) [GET] Parameters: {"action"=>"index", "controller"=>"resources/deployments"} Redirected to http://dell-pe1950-1.rhts.eng.rdu.redhat.com/conductor/login Filter chain halted as [:require_user] rendered_or_redirected. Completed in 6ms (DB: 8) | 302 Found [http://dell-pe1950-1.rhts.eng.rdu.redhat.com/conductor] Processing UserSessionsController#new (for 10.11.228.65 at 2011-04-04 09:08:40) [GET] Parameters: {"action"=>"new", "controller"=>"user_sessions"} Rendering template within layouts/application Rendering user_sessions/new Completed in 46ms (View: 35, DB: 11) | 200 OK [http://dell-pe1950-1.rhts.eng.rdu.redhat.com/conductor/login] Processing UserSessionsController#create (for 10.11.228.65 at 2011-04-04 09:09:39) [POST] Parameters: {"commit"=>"Login", "user_session"=>{"remember_me"=>"0", "password"=>"[FILTERED]", "login"=>"admin"}, "action"=>"create", "controller"=>"user_sessions"} Redirected to http://dell-pe1950-1.rhts.eng.rdu.redhat.com/conductor/ Completed in 701ms (DB: 22) | 302 Found [http://dell-pe1950-1.rhts.eng.rdu.redhat.com/conductor/user_session] Processing Resources::DeploymentsController#index (for 10.11.228.65 at 2011-04-04 09:09:39) [GET] Parameters: {"action"=>"index", "controller"=>"resources/deployments"} Rendering template within layouts/application Rendering resources/deployments/index Completed in 519ms (View: 188, DB: 11) | 200 OK [http://dell-pe1950-1.rhts.eng.rdu.redhat.com/conductor/] Processing Admin::UsersController#index (for 10.11.228.65 at 2011-04-04 09:09:43) [GET] Parameters: {"action"=>"index", "controller"=>"admin/users"} Rendering template within layouts/application Rendering admin/users/index Completed in 446ms (View: 55, DB: 26) | 200 OK [http://dell-pe1950-1.rhts.eng.rdu.redhat.com/conductor/admin/users] Processing Admin::ProviderAccountsController#index (for 10.11.228.65 at 2011-04-04 09:09:45) [GET] Parameters: {"action"=>"index", "controller"=>"admin/provider_accounts"} Rendering template within layouts/application Rendering admin/provider_accounts/index Completed in 395ms (View: 43, DB: 18) | 200 OK [http://dell-pe1950-1.rhts.eng.rdu.redhat.com/conductor/admin/provider_accounts] Processing Admin::ProviderAccountsController#new (for 10.11.228.65 at 2011-04-04 09:09:47) [GET] Parameters: {"action"=>"new", "controller"=>"admin/provider_accounts"} Rendering template within layouts/application Rendering admin/provider_accounts/new Completed in 491ms (View: 33, DB: 19) | 200 OK [http://dell-pe1950-1.rhts.eng.rdu.redhat.com/conductor/admin/provider_accounts/new] Processing Admin::ProvidersController#index (for 10.11.228.65 at 2011-04-04 09:09:50) [GET] Parameters: {"action"=>"index", "controller"=>"admin/providers"} Rendering template within layouts/application Rendering admin/providers/index Completed in 402ms (View: 44, DB: 23) | 200 OK [http://dell-pe1950-1.rhts.eng.rdu.redhat.com/conductor/admin/providers] [root@dell-pe1950-1 log]#
trying to recreate now
recreated.. [root@dell-pe2950-01 yum.repos.d]# rpm -q aeolus package aeolus is not installed [root@dell-pe2950-01 yum.repos.d]# rpm -qa | grep aeolus aeolus-conductor-0.0.3-0.fc14.20110401225256git6800ed8.x86_64 aeolus-conductor-doc-0.0.3-0.fc14.20110401225256git6800ed8.x86_64 aeolus-configure-2.0.0-5.fc14.20110401173241git4d62feb.noarch aeolus-conductor-daemons-0.0.3-0.fc14.20110401225256git6800ed8.x86_64 [root@dell-pe2950-01 yum.repos.d]# psql -t -U aeolus -d conductor -c 'select name from providers' | grep us-east ec2-us-east-1 [root@dell-pe2950-01 yum.repos.d]# psql -t -U aeolus -d conductor -c 'select name from providers' | grep us-west [root@dell-pe2950-01 yum.repos.d]# psql -t -U aeolus -d conductor -c 'select name from providers' | grep mock [root@dell-pe2950-01 yum.repos.d]#
Created attachment 489823 [details] configure and process, db checks
Created attachment 489824 [details] sos report
recreated on my local desktop [root@localhost ~]# rpm -qa | grep aeolus aeolus-conductor-doc-0.0.3-0.fc14.20110401225256git6800ed8.x86_64 aeolus-configure-2.0.0-5.fc14.20110401173241git4d62feb.noarch aeolus-conductor-daemons-0.0.3-0.fc14.20110401225256git6800ed8.x86_64 aeolus-conductor-0.0.3-0.fc14.20110401225256git6800ed8.x86_64 [root@localhost ~]# 0;36mnotice: /Stage[main]/Aeolus::Image-factory/Service[imagefactory]/ensure: ensure changed 'stopped' to 'running'[0m [0;36mnotice: /Stage[main]//Aeolus::Site_admin[admin]/Exec[create_site_admin_user]/returns: (in /usr/share/aeolus-conductor)[0m [0;36mnotice: /Stage[main]//Aeolus::Site_admin[admin]/Exec[create_site_admin_user]/returns: User admin registered[0m [0;36mnotice: /Stage[main]//Aeolus::Site_admin[admin]/Exec[create_site_admin_user]/returns: executed successfully[0m [0;36mnotice: /Stage[main]//Aeolus::Site_admin[admin]/Exec[grant_site_admin_privs]/returns: (in /usr/share/aeolus-conductor)[0m [0;36mnotice: /Stage[main]//Aeolus::Site_admin[admin]/Exec[grant_site_admin_privs]/returns: Granting administrator privileges for admin...[0m [0;36mnotice: /Stage[main]//Aeolus::Site_admin[admin]/Exec[grant_site_admin_privs]/returns: executed successfully[0m [0;36mnotice: /File[/var/log/deltacloud-ec2-us-east-1]/ensure: created[0m [0;36mnotice: /File[/etc/init.d/deltacloud-ec2-us-west-1]/ensure: defined content as '{md5}9df53610081811c33820aacfc9393a19'[0m [0;36mnotice: /File[/var/log/deltacloud-mock]/ensure: created[0m [0;36mnotice: /File[/etc/init.d/deltacloud-mock]/ensure: defined content as '{md5}604861023be9be9710cedeaf5bc442c5'[0m [0;36mnotice: /Stage[main]//Aeolus::Provider[mock]/Aeolus::Deltacloud[mock]/Service[deltacloud-mock]/ensure: ensure changed 'stopped' to 'running'[0m [0;36mnotice: /Stage[main]//Aeolus::Provider[mock]/Aeolus::Conductor::Provider[mock]/Aeolus::Conductor::Login[provider-mock]/Exec[conductor-login-for-provider-mock]/returns: executed successfully[0m [0;36mnotice: /Stage[main]//Aeolus::Provider[mock]/Aeolus::Conductor::Provider[mock]/Exec[add-conductor-provider-mock]/returns: % Total % Received % Xferd Average Speed Time Time Time Current[0m [0;36mnotice: /Stage[main]//Aeolus::Provider[mock]/Aeolus::Conductor::Provider[mock]/Exec[add-conductor-provider-mock]/returns: Dload Upload Total Spent Left Speed[0m [0;36mnotice: /Stage[main]//Aeolus::Provider[mock]/Aeolus::Conductor::Provider[mock]/Exec[add-conductor-provider-mock]/returns: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 124 398 100 398 0 97 49675 12106 --:--:-- --:--:-- --:--:-- 194k[0m [0;36mnotice: /Stage[main]//Aeolus::Provider[mock]/Aeolus::Conductor::Provider[mock]/Exec[add-conductor-provider-mock]/returns: <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">[0m [0;36mnotice: /Stage[main]//Aeolus::Provider[mock]/Aeolus::Conductor::Provider[mock]/Exec[add-conductor-provider-mock]/returns: <html><head>[0m [0;36mnotice: /Stage[main]//Aeolus::Provider[mock]/Aeolus::Conductor::Provider[mock]/Exec[add-conductor-provider-mock]/returns: <title>503 Service Temporarily Unavailable</title>[0m [0;36mnotice: /Stage[main]//Aeolus::Provider[mock]/Aeolus::Conductor::Provider[mock]/Exec[add-conductor-provider-mock]/returns: </head><body>[0m [0;36mnotice: /Stage[main]//Aeolus::Provider[mock]/Aeolus::Conductor::Provider[mock]/Exec[add-conductor-provider-mock]/returns: <h1>Service Temporarily Unavailable</h1>[0m [0;36mnotice: /Stage[main]//Aeolus::Provider[mock]/Aeolus::Conductor::Provider[mock]/Exec[add-conductor-provider-mock]/returns: <p>The server is temporarily unable to service your[0m [0;36mnotice: /Stage[main]//Aeolus::Provider[mock]/Aeolus::Conductor::Provider[mock]/Exec[add-conductor-provider-mock]/returns: request due to maintenance downtime or capacity[0m [0;36mnotice: /Stage[main]//Aeolus::Provider[mock]/Aeolus::Conductor::Provider[mock]/Exec[add-conductor-provider-mock]/returns: problems. Please try again later.</p>[0m [0;36mnotice: /Stage[main]//Aeolus::Provider[mock]/Aeolus::Conductor::Provider[mock]/Exec[add-conductor-provider-mock]/returns: <hr>[0m [0;36mnotice: /Stage[main]//Aeolus::Provider[mock]/Aeolus::Conductor::Provider[mock]/Exec[add-conductor-provider-mock]/returns: <address>Apache/2.2.17 (Fedora) Server at localhost Port 80</address>[0m [0;36mnotice: /Stage[main]//Aeolus::Provider[mock]/Aeolus::Conductor::Provider[mock]/Exec[add-conductor-provider-mock]/returns: </body></html>[0m [0;36mnotice: /Stage[main]//Aeolus::Provider[mock]/Aeolus::Conductor::Provider[mock]/Exec[add-conductor-provider-mock]/returns: executed successfully[0m [0;36mnotice: /File[/var/log/deltacloud-ec2-us-west-1]/ensure: created[0m [0;36mnotice: /Stage[main]//Aeolus::Provider[ec2-us-west-1]/Aeolus::Deltacloud[ec2-us-west-1]/Service[deltacloud-ec2-us-west-1]/ensure: ensure changed 'stopped' to 'running'[0m [0;36mnotice: /Stage[main]//Aeolus::Provider[ec2-us-west-1]/Aeolus::Conductor::Provider[ec2-us-west-1]/Aeolus::Conductor::Login[provider-ec2-us-west-1]/Exec[conductor-login-for-provider-ec2-us-west-1]/returns: executed successfully[0m [0;36mnotice: /Stage[main]//Aeolus::Provider[ec2-us-west-1]/Aeolus::Conductor::Provider[ec2-us-west-1]/Exec[add-conductor-provider-ec2-us-west-1]/returns: % Total % Received % Xferd Average Speed Time Time Time Current[0m [0;36mnotice: /Stage[main]//Aeolus::Provider[ec2-us-west-1]/Aeolus::Conductor::Provider[ec2-us-west-1]/Exec[add-conductor-provider-ec2-us-west-1]/returns: Dload Upload Total Spent Left Speed[0m [0;36mnotice: /Stage[main]//Aeolus::Provider[ec2-us-west-1]/Aeolus::Conductor::Provider[ec2-us-west-1]/Exec[add-conductor-provider-ec2-us-west-1]/returns: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 126 398 100 398 0 105 123k 33238 --:--:-- --:--:-- --:--:-- 194k[0m [0;36mnotice: /Stage[main]//Aeolus::Provider[ec2-us-west-1]/Aeolus::Conductor::Provider[ec2-us-west-1]/Exec[add-conductor-provider-ec2-us-west-1]/returns: <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">[0m [0;36mnotice: /Stage[main]//Aeolus::Provider[ec2-us-west-1]/Aeolus::Conductor::Provider[ec2-us-west-1]/Exec[add-conductor-provider-ec2-us-west-1]/returns: <html><head>[0m [0;36mnotice: /Stage[main]//Aeolus::Provider[ec2-us-west-1]/Aeolus::Conductor::Provider[ec2-us-west-1]/Exec[add-conductor-provider-ec2-us-west-1]/returns: <title>503 Service Temporarily Unavailable</title>[0m [0;36mnotice: /Stage[main]//Aeolus::Provider[ec2-us-west-1]/Aeolus::Conductor::Provider[ec2-us-west-1]/Exec[add-conductor-provider-ec2-us-west-1]/returns: </head><body>[0m [0;36mnotice: /Stage[main]//Aeolus::Provider[ec2-us-west-1]/Aeolus::Conductor::Provider[ec2-us-west-1]/Exec[add-conductor-provider-ec2-us-west-1]/returns: <h1>Service Temporarily Unavailable</h1>[0m [0;36mnotice: /Stage[main]//Aeolus::Provider[ec2-us-west-1]/Aeolus::Conductor::Provider[ec2-us-west-1]/Exec[add-conductor-provider-ec2-us-west-1]/returns: <p>The server is temporarily unable to service your[0m [0;36mnotice: /Stage[main]//Aeolus::Provider[ec2-us-west-1]/Aeolus::Conductor::Provider[ec2-us-west-1]/Exec[add-conductor-provider-ec2-us-west-1]/returns: request due to maintenance downtime or capacity[0m [0;36mnotice: /Stage[main]//Aeolus::Provider[ec2-us-west-1]/Aeolus::Conductor::Provider[ec2-us-west-1]/Exec[add-conductor-provider-ec2-us-west-1]/returns: problems. Please try again later.</p>[0m [0;36mnotice: /Stage[main]//Aeolus::Provider[ec2-us-west-1]/Aeolus::Conductor::Provider[ec2-us-west-1]/Exec[add-conductor-provider-ec2-us-west-1]/returns: <hr>[0m [0;36mnotice: /Stage[main]//Aeolus::Provider[ec2-us-west-1]/Aeolus::Conductor::Provider[ec2-us-west-1]/Exec[add-conductor-provider-ec2-us-west-1]/returns: <address>Apache/2.2.17 (Fedora) Server at localhost Port 80</address>[0m [0;36mnotice: /Stage[main]//Aeolus::Provider[ec2-us-west-1]/Aeolus::Conductor::Provider[ec2-us-west-1]/Exec[add-conductor-provider-ec2-us-west-1]/returns: </body></html>[0m [0;36mnotice: /Stage[main]//Aeolus::Provider[ec2-us-west-1]/Aeolus::Conductor::Provider[ec2-us-west-1]/Exec[add-conductor-provider-ec2-us-west-1]/returns: executed successfully[0m [0;36mnotice: /Stage[main]//Aeolus::Conductor::Hwp[hwp1]/Aeolus::Conductor::Login[hwp-hwp1]/Exec[conductor-login-for-hwp-hwp1]/returns: executed successfully[0m [0;36mnotice: /Stage[main]//Aeolus::Conductor::Hwp[hwp1]/Exec[add-conductor-hwp-hwp1]/returns: % Total % Received % Xferd Average Speed Time Time Time Current[0m [0;36mnotice: /Stage[main]//Aeolus::Conductor::Hwp[hwp1]/Exec[add-conductor-hwp-hwp1]/returns: Dload Upload Total Spent Left Speed[0m [0;36mnotice: /Stage[main]//Aeolus::Conductor::Hwp[hwp1]/Exec[add-conductor-hwp-hwp1]/returns: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 256 398 100 398 0 622 127k 198k --:--:-- --:--:-- --:--:-- 607k[0m [0;36mnotice: /Stage[main]//Aeolus::Conductor::Hwp[hwp1]/Exec[add-conductor-hwp-hwp1]/returns: <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">[0m [0;36mnotice: /Stage[main]//Aeolus::Conductor::Hwp[hwp1]/Exec[add-conductor-hwp-hwp1]/returns: <html><head>[0m [0;36mnotice: /Stage[main]//Aeolus::Conductor::Hwp[hwp1]/Exec[add-conductor-hwp-hwp1]/returns: <title>503 Service Temporarily Unavailable</title>[0m [0;36mnotice: /Stage[main]//Aeolus::Conductor::Hwp[hwp1]/Exec[add-conductor-hwp-hwp1]/returns: </head><body>[0m [0;36mnotice: /Stage[main]//Aeolus::Conductor::Hwp[hwp1]/Exec[add-conductor-hwp-hwp1]/returns: <h1>Service Temporarily Unavailable</h1>[0m [0;36mnotice: /Stage[main]//Aeolus::Conductor::Hwp[hwp1]/Exec[add-conductor-hwp-hwp1]/returns: <p>The server is temporarily unable to service your[0m [0;36mnotice: /Stage[main]//Aeolus::Conductor::Hwp[hwp1]/Exec[add-conductor-hwp-hwp1]/returns: request due to maintenance downtime or capacity[0m [0;36mnotice: /Stage[main]//Aeolus::Conductor::Hwp[hwp1]/Exec[add-conductor-hwp-hwp1]/returns: problems. Please try again later.</p>[0m [0;36mnotice: /Stage[main]//Aeolus::Conductor::Hwp[hwp1]/Exec[add-conductor-hwp-hwp1]/returns: <hr>[0m [0;36mnotice: /Stage[main]//Aeolus::Conductor::Hwp[hwp1]/Exec[add-conductor-hwp-hwp1]/returns: <address>Apache/2.2.17 (Fedora) Server at localhost Port 80</address>[0m [0;36mnotice: /Stage[main]//Aeolus::Conductor::Hwp[hwp1]/Exec[add-conductor-hwp-hwp1]/returns: </body></html>[0m [0;36mnotice: /Stage[main]//Aeolus::Conductor::Hwp[hwp1]/Exec[add-conductor-hwp-hwp1]/returns: executed successfully[0m [0;36mnotice: /File[/etc/init.d/deltacloud-ec2-us-east-1]/ensure: defined content as '{md5}260290bf5f6257d8ed894752f2fb6bab'[0m [0;36mnotice: /Stage[main]//Aeolus::Provider[ec2-us-east-1]/Aeolus::Deltacloud[ec2-us-east-1]/Service[deltacloud-ec2-us-east-1]/ensure: ensure changed 'stopped' to 'running'[0m [0;36mnotice: /Stage[main]//Aeolus::Provider[ec2-us-east-1]/Aeolus::Conductor::Provider[ec2-us-east-1]/Aeolus::Conductor::Login[provider-ec2-us-east-1]/Exec[conductor-login-for-provider-ec2-us-east-1]/returns: executed successfully[0m [0;36mnotice: /Stage[main]//Aeolus::Provider[ec2-us-east-1]/Aeolus::Conductor::Provider[ec2-us-east-1]/Exec[add-conductor-provider-ec2-us-east-1]/returns: % Total % Received % Xferd Average Speed Time Time Time Current[0m [0;36mnotice: /Stage[main]//Aeolus::Provider[ec2-us-east-1]/Aeolus::Conductor::Provider[ec2-us-east-1]/Exec[add-conductor-provider-ec2-us-east-1]/returns: Dload Upload Total Spent Left Speed[0m [0;36mnotice: /Stage[main]//Aeolus::Provider[ec2-us-east-1]/Aeolus::Conductor::Provider[ec2-us-east-1]/Exec[add-conductor-provider-ec2-us-east-1]/returns: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 126 398 100 398 0 105 127k 34313 --:--:-- --:--:-- --:--:-- 388k[0m [0;36mnotice: /Stage[main]//Aeolus::Provider[ec2-us-east-1]/Aeolus::Conductor::Provider[ec2-us-east-1]/Exec[add-conductor-provider-ec2-us-east-1]/returns: <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">[0m [0;36mnotice: /Stage[main]//Aeolus::Provider[ec2-us-east-1]/Aeolus::Conductor::Provider[ec2-us-east-1]/Exec[add-conductor-provider-ec2-us-east-1]/returns: <html><head>[0m [0;36mnotice: /Stage[main]//Aeolus::Provider[ec2-us-east-1]/Aeolus::Conductor::Provider[ec2-us-east-1]/Exec[add-conductor-provider-ec2-us-east-1]/returns: <title>503 Service Temporarily Unavailable</title>[0m [0;36mnotice: /Stage[main]//Aeolus::Provider[ec2-us-east-1]/Aeolus::Conductor::Provider[ec2-us-east-1]/Exec[add-conductor-provider-ec2-us-east-1]/returns: </head><body>[0m [0;36mnotice: /Stage[main]//Aeolus::Provider[ec2-us-east-1]/Aeolus::Conductor::Provider[ec2-us-east-1]/Exec[add-conductor-provider-ec2-us-east-1]/returns: <h1>Service Temporarily Unavailable</h1>[0m [0;36mnotice: /Stage[main]//Aeolus::Provider[ec2-us-east-1]/Aeolus::Conductor::Provider[ec2-us-east-1]/Exec[add-conductor-provider-ec2-us-east-1]/returns: <p>The server is temporarily unable to service your[0m [0;36mnotice: /Stage[main]//Aeolus::Provider[ec2-us-east-1]/Aeolus::Conductor::Provider[ec2-us-east-1]/Exec[add-conductor-provider-ec2-us-east-1]/returns: request due to maintenance downtime or capacity[0m [0;36mnotice: /Stage[main]//Aeolus::Provider[ec2-us-east-1]/Aeolus::Conductor::Provider[ec2-us-east-1]/Exec[add-conductor-provider-ec2-us-east-1]/returns: problems. Please try again later.</p>[0m [0;36mnotice: /Stage[main]//Aeolus::Provider[ec2-us-east-1]/Aeolus::Conductor::Provider[ec2-us-east-1]/Exec[add-conductor-provider-ec2-us-east-1]/returns: <hr>[0m [0;36mnotice: /Stage[main]//Aeolus::Provider[ec2-us-east-1]/Aeolus::Conductor::Provider[ec2-us-east-1]/Exec[add-conductor-provider-ec2-us-east-1]/returns: <address>Apache/2.2.17 (Fedora) Server at localhost Port 80</address>[0m [0;36mnotice: /Stage[main]//Aeolus::Provider[ec2-us-east-1]/Aeolus::Conductor::Provider[ec2-us-east-1]/Exec[add-conductor-provider-ec2-us-east-1]/returns: </body></html>[0m [0;36mnotice: /Stage[main]//Aeolus::Provider[ec2-us-east-1]/Aeolus::Conductor::Provider[ec2-us-east-1]/Exec[add-conductor-provider-ec2-us-east-1]/returns: executed successfully[0m [0;36mnotice: /Stage[main]/Aeolus::Conductor/Service[conductor-warehouse_sync]/ensure: ensure changed 'stopped' to 'running'[0m [0;36mnotice: /Stage[main]/Aeolus::Conductor/Service[conductor-delayed_job]/ensure: ensure changed 'stopped' to 'running'[0m [0;36mnotice: /Stage[main]/Aeolus::Conductor/Selinux::Mode[permissive]/Exec[set_selinux_permissive]/returns: executed successfully[0m [0;36mnotice: Finished catalog run in 425.14 seconds[0m (com.redhat.qe.tools.SSHCommandRunner.runCommandAndWait) [root@localhost ~]# psql -t -U aeolus -d conductor -c 'select name from providers' | grep us-east [root@localhost ~]# psql -t -U aeolus -d conductor -c 'select name from providers' | grep us-west [root@localhost ~]# psql -t -U aeolus -d conductor -c 'select name from providers' | grep mock [root@localhost ~]#
Created attachment 489833 [details] sos report from local workstation
mmorsi found this is a selinux issue. Sometime selinux is the first thing disabled.. other times its the last.. BOOO!!!
To elaborate, there was no dependency for selinux to be set to permissive before httpd was run, thus it was possible for selinux to interfere with mod_proxy as was manifesting in the apache error log [Mon Apr 04 14:54:02 2011] [notice] Apache/2.2.17 (Unix) DAV/2 configured -- resuming normal operations [Mon Apr 04 14:56:53 2011] [error] (13)Permission denied: proxy: HTTP: attempt to connect to 127.0.0.1:3000 (localhost) failed [Mon Apr 04 14:56:53 2011] [error] ap_proxy_connect_backend disabling worker for (localhost) [Mon Apr 04 14:56:53 2011] [error] proxy: HTTP: disabled connection for (localhost) Submitted a patch permitting the necessary operations over selinux and ensuring that the http service is up before the provider creation commands are run: https://fedorahosted.org/pipermail/aeolus-devel/2011-April/001000.html
The providers not getting added properly still seems to be an issue. My bet is that selinux is not only cause here. Waiting to see if this bug has been integrated into the build
As of right now my patch has not been ack'd or pushed to the configure repo yet and will need to be in before it can be pulled into the build. I'm pretty sure that the original issue at least was caused due to selinux. The errors in the log when the providers were attempted to be created were indicating apache didn't have permissions to make network connections, and thus after we enable that ability, those errors go away.
Ok.. good to know its not integrated. Agreed about what the original issue was. I understand selinux would be set to permissive sometimes at the end of the script. Once selinux is set to permissive the providers should be added correctly at any subsequent run. Even though selinux is permissive I'm seeing inconsistent results with the providers. I'll have to dig in and get you more details, obviously I have nothing for you to go after yet. Thanks for updating
This fails in the most recent ami.. ami-18fa0671 The started w/ only the us-west provider present. Will attach a recreate and logs.
Created attachment 493220 [details] configure output, instance1 spun up two instances of the AMI (ami-18fa0671) this morning and both do not have pre-populated providers, here is the output from the first
After some debugging, we believe this issue is caused by the core init script returning before the core service is actually accessible. Sent a patch the list, to change the init script so as to wait until the service is actually up before returning https://fedorahosted.org/pipermail/aeolus-devel/2011-April/001241.html
As far as I can tell this is fixed!!! Tried three time to cleanup/configure all came back OK [root@hp-xw8600-01 ~]# rpm -qa | grep aeolus aeolus-conductor-0.2.0-1.el6.x86_64 aeolus-conductor-doc-0.2.0-1.el6.x86_64 aeolus-conductor-daemons-0.2.0-1.el6.x86_64 aeolus-configure-2.0.0-9.el6.noarch [root@hp-xw8600-01 ~]# Thank you!
release pending...
release pending 3
release pending.. 2
perm close
closing out old bugs