Bug 972287 - [v1cart]Failed to embed mysql-5.1 to application due to selinux avc denial
[v1cart]Failed to embed mysql-5.1 to application due to selinux avc denial
Status: CLOSED WONTFIX
Product: OpenShift Container Platform
Classification: Red Hat
Component: Containers (Show other bugs)
1.2.0
Unspecified Unspecified
high Severity high
: ---
: ---
Assigned To: Brenton Leanhardt
libra bugs
:
Depends On:
Blocks: 975056 1005049
  Show dependency treegraph
 
Reported: 2013-06-08 03:12 EDT by Gaoyun Pei
Modified: 2013-09-06 02:34 EDT (History)
3 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
: 975056 (view as bug list)
Environment:
Last Closed: 2013-08-15 11:03:24 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description Gaoyun Pei 2013-06-08 03:12:54 EDT
Description of problem:
After installing an OSE env in v1 mode, failed to embed mysql-5.1 cartridge to an app.

Version-Release number of selected component (if applicable):


How reproducible:
Always

Steps to Reproduce:
1. Set up an OSE v1 env using the installation script
2. Create an app, then add mysql-5.1 cartridge to it

Actual results:
[root@dhcp-8-133 workspace]# rhc cartridge add mysql -a app6
Using mysql-5.1 (MySQL Database 5.1) for 'mysql'
Adding mysql-5.1 to application 'app6' ... 
The server did not respond correctly. This may be an issue with the server configuration or with your connection to the server (such as a Web proxy or
firewall). Please verify that you can access the OpenShift server https://broker.ose0605v1.com/broker/rest/domains/uu/applications/app6/cartridges


Expected results:
Mysql-5.1 should be added successfully

Additional info:
The mcollective.log on node during the embed action:
...
I, [2013-06-07T22:56:31.733025 #1602]  INFO -- : openshift.rb:51:in `cartridge_do_action' cartridge_do_action call / action: cartridge_do, agent=openshift, data={:cartridge=>"mysql-5.1",
 :action=>"configure",
 :args=>
  {"--with-app-uuid"=>"51b2c0d8d8db736a6600005b",
   "--with-app-name"=>"app6",
   "--with-container-uuid"=>"51b2c0d8d8db736a6600005b",
   "--with-container-name"=>"app6",
   "--with-namespace"=>"uu",
   "--with-request-id"=>"3df17de5879230b4e4117cf2c3e39094",
   "--cart-name"=>"mysql-5.1",
   "--component-name"=>"mysql-5.1",
   "--with-software-version"=>"5.1",
   "--cartridge-vendor"=>"redhat"},
 :process_results=>true}

I, [2013-06-07T22:56:31.733152 #1602]  INFO -- : openshift.rb:52:in `cartridge_do_action' cartridge_do_action validation = mysql-5.1 configure {"--with-app-uuid"=>"51b2c0d8d8db736a6600005b", "--with-app-name"=>"app6", "--with-container-uuid"=>"51b2c0d8d8db736a6600005b", "--with-container-name"=>"app6", "--with-namespace"=>"uu", "--with-request-id"=>"3df17de5879230b4e4117cf2c3e39094", "--cart-name"=>"mysql-5.1", "--component-name"=>"mysql-5.1", "--with-software-version"=>"5.1", "--cartridge-vendor"=>"redhat"}
I, [2013-06-07T22:56:31.733444 #1602]  INFO -- : openshift.rb:91:in `execute_action' Executing action [configure] using method oo_configure with args [{"--with-app-uuid"=>"51b2c0d8d8db736a6600005b", "--with-app-name"=>"app6", "--with-container-uuid"=>"51b2c0d8d8db736a6600005b", "--with-container-name"=>"app6", "--with-namespace"=>"uu", "--with-request-id"=>"3df17de5879230b4e4117cf2c3e39094", "--cart-name"=>"mysql-5.1", "--component-name"=>"mysql-5.1", "--with-software-version"=>"5.1", "--cartridge-vendor"=>"redhat"}]
D, [2013-06-07T22:56:31.731240 #1602] DEBUG -- : cache.rb:105:in `read' Cache hit on 'ddl' key 'agent/openshift'
D, [2013-06-07T22:56:31.731715 #1602] DEBUG -- : openshift.rb:35:in `before_processing_hook' Changing working directory to /tmp
I, [2013-06-07T22:56:31.733025 #1602]  INFO -- : openshift.rb:51:in `cartridge_do_action' cartridge_do_action call / action: cartridge_do, agent=openshift, data={:cartridge=>"mysql-5.1",
 :action=>"configure",
 :args=>
  {"--with-app-uuid"=>"51b2c0d8d8db736a6600005b",
   "--with-app-name"=>"app6",
   "--with-container-uuid"=>"51b2c0d8d8db736a6600005b",
   "--with-container-name"=>"app6",
   "--with-namespace"=>"uu",
   "--with-request-id"=>"3df17de5879230b4e4117cf2c3e39094",
   "--cart-name"=>"mysql-5.1",
   "--component-name"=>"mysql-5.1",
   "--with-software-version"=>"5.1",
   "--cartridge-vendor"=>"redhat"},
 :process_results=>true}

I, [2013-06-07T22:56:31.733152 #1602]  INFO -- : openshift.rb:52:in `cartridge_do_action' cartridge_do_action validation = mysql-5.1 configure {"--with-app-uuid"=>"51b2c0d8d8db736a6600005b", "--with-app-name"=>"app6", "--with-container-uuid"=>"51b2c0d8d8db736a6600005b", "--with-container-name"=>"app6", "--with-namespace"=>"uu", "--with-request-id"=>"3df17de5879230b4e4117cf2c3e39094", "--cart-name"=>"mysql-5.1", "--component-name"=>"mysql-5.1", "--with-software-version"=>"5.1", "--cartridge-vendor"=>"redhat"}
I, [2013-06-07T22:56:31.733444 #1602]  INFO -- : openshift.rb:91:in `execute_action' Executing action [configure] using method oo_configure with args [{"--with-app-uuid"=>"51b2c0d8d8db736a6600005b", "--with-app-name"=>"app6", "--with-container-uuid"=>"51b2c0d8d8db736a6600005b", "--with-container-name"=>"app6", "--with-namespace"=>"uu", "--with-request-id"=>"3df17de5879230b4e4117cf2c3e39094", "--cart-name"=>"mysql-5.1", "--component-name"=>"mysql-5.1", "--with-software-version"=>"5.1", "--cartridge-vendor"=>"redhat"}]
I, [2013-06-07T22:56:32.677521 #1602]  INFO -- : openshift.rb:100:in `execute_action' Finished executing action [configure] (119)
I, [2013-06-07T22:56:32.677884 #1602]  INFO -- : openshift.rb:73:in `cartridge_do_action' cartridge_do_action failed (119)
------
Control action 'configure' returned an error. rc=119
Failed to create mysqldb


------)
D, [2013-06-07T22:56:32.678307 #1602] DEBUG -- : pluginmanager.rb:83:in `[]' Returning cached plugin security_plugin with class MCollective::Security::Psk
D, [2013-06-07T22:56:32.678533 #1602] DEBUG -- : pluginmanager.rb:83:in `[]' Returning cached plugin security_plugin with class MCollective::Security::Psk
D, [2013-06-07T22:56:32.678851 #1602] DEBUG -- : base.rb:168:in `create_reply' Encoded a message for request af4f93d921085d1a9f139114e9582cbf
D, [2013-06-07T22:56:32.679318 #1602] DEBUG -- : pluginmanager.rb:83:in `[]' Returning cached plugin connector_plugin with class MCollective::Connector::Activemq
D, [2013-06-07T22:56:32.679434 #1602] DEBUG -- : activemq.rb:266:in `publish' Sending a broadcast message to ActiveMQ target '/queue/mcollective.reply.broker.ose0605v1.com_24776' with headers '{}'
D, [2013-06-07T22:56:32.685995 #1602] DEBUG -- : runnerstats.rb:56:in `block in sent' Incrementing replies stat
D, [2013-06-07T22:56:32.798773 #1602] DEBUG -- : runnerstats.rb:49:in `received' Incrementing total stat
D, [2013-06-07T22:56:32.798972 #1602] DEBUG -- : pluginmanager.rb:83:in `[]' Returning cached plugin security_plugin with class MCollective::Security::Psk
D, [2013-06-07T22:56:32.799206 #1602] DEBUG -- : runnerstats.rb:38:in `validated' Incrementing validated stat
D, [2013-06-07T22:56:32.799363 #1602] DEBUG -- : pluginmanager.rb:83:in `[]' Returning cached plugin security_plugin with class MCollective::Security::Psk
D, [2013-06-07T22:56:32.799519 #1602] DEBUG -- : pluginmanager.rb:83:in `[]' Returning cached plugin security_plugin with class MCollective::Security::Psk
D, [2013-06-07T22:56:32.799698 #1602] DEBUG -- : base.rb:117:in `block (2 levels) in validate_filter?' Passing based on agent openshift
D, [2013-06-07T22:56:32.799851 #1602] DEBUG -- : base.rb:153:in `validate_filter?' Message passed the filter checks
D, [2013-06-07T22:56:32.799959 #1602] DEBUG -- : runnerstats.rb:26:in `passed' Incrementing passed stat
D, [2013-06-07T22:56:32.800070 #1602] DEBUG -- : runner.rb:80:in `agentmsg' Handling message for agent 'openshift' on collective 'mcollective'
D, [2013-06-07T22:56:32.800176 #1602] DEBUG -- : agents.rb:119:in `dispatch' Dispatching a message to agent openshift
D, [2013-06-07T22:56:32.800337 #1602] DEBUG -- : activemq.rb:233:in `receive' Waiting for a message from ActiveMQ
D, [2013-06-07T22:56:32.803064 #1602] DEBUG -- : pluginmanager.rb:88:in `[]' Returning new plugin openshift_agent with class MCollective::Agent::Openshift
D, [2013-06-07T22:56:32.804792 #1602] DEBUG -- : cache.rb:105:in `read' Cache hit on 'ddl' key 'agent/openshift'
D, [2013-06-07T22:56:32.805388 #1602] DEBUG -- : openshift.rb:35:in `before_processing_hook' Changing working directory to /tmp
I, [2013-06-07T22:56:32.807983 #1602]  INFO -- : openshift.rb:51:in `cartridge_do_action' cartridge_do_action call / action: cartridge_do, agent=openshift, data={:cartridge=>"mysql-5.1",
 :action=>"deconfigure",
 :args=>
  {"--with-app-uuid"=>"51b2c0d8d8db736a6600005b",
   "--with-app-name"=>"app6",
   "--with-container-uuid"=>"51b2c0d8d8db736a6600005b",
   "--with-container-name"=>"app6",
   "--with-namespace"=>"uu",
   "--with-request-id"=>"3df17de5879230b4e4117cf2c3e39094",
   "--cart-name"=>"mysql-5.1",
   "--component-name"=>"mysql-5.1",
   "--with-software-version"=>"5.1",
   "--cartridge-vendor"=>"redhat"},
 :process_results=>true}

I, [2013-06-07T22:56:32.808198 #1602]  INFO -- : openshift.rb:52:in `cartridge_do_action' cartridge_do_action validation = mysql-5.1 deconfigure {"--with-app-uuid"=>"51b2c0d8d8db736a6600005b", "--with-app-name"=>"app6", "--with-container-uuid"=>"51b2c0d8d8db736a6600005b", "--with-container-name"=>"app6", "--with-namespace"=>"uu", "--with-request-id"=>"3df17de5879230b4e4117cf2c3e39094", "--cart-name"=>"mysql-5.1", "--component-name"=>"mysql-5.1", "--with-software-version"=>"5.1", "--cartridge-vendor"=>"redhat"}
I, [2013-06-07T22:56:32.808574 #1602]  INFO -- : openshift.rb:91:in `execute_action' Executing action [deconfigure] using method oo_deconfigure with args [{"--with-app-uuid"=>"51b2c0d8d8db736a6600005b", "--with-app-name"=>"app6", "--with-container-uuid"=>"51b2c0d8d8db736a6600005b", "--with-container-name"=>"app6", "--with-namespace"=>"uu", "--with-request-id"=>"3df17de5879230b4e4117cf2c3e39094", "--cart-name"=>"mysql-5.1", "--component-name"=>"mysql-5.1", "--with-software-version"=>"5.1", "--cartridge-vendor"=>"redhat"}]
I, [2013-06-07T22:56:35.902085 #1602]  INFO -- : openshift.rb:100:in `execute_action' Finished executing action [deconfigure] (0)
I, [2013-06-07T22:56:35.902395 #1602]  INFO -- : openshift.rb:71:in `cartridge_do_action' cartridge_do_action reply (0):
------
No IP specified
/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.9.12/lib/openshift-origin-node/model/frontend_proxy.rb:203:in `find_mapped_proxy_port'
/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.9.12/lib/openshift-origin-node/model/application_container.rb:259:in `block in delete_public_endpoints'
/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.9.12/lib/openshift-origin-node/model/application_container.rb:253:in `each'
/opt/rh/ruby193/root/usr/share/gems/gems/openshift-origin-node-1.9.12/lib/openshift-origin-node/model/application_container.rb:253:in `delete_public_endpoints'
/usr/bin/oo-delete-endpoints:71:in `<main>'
MySQL already stopped

------)
Comment 2 Brenton Leanhardt 2013-06-10 13:23:51 EDT
Looking into this shows that the directories under /var/lib/openshift/[uuid]/mysql-5.1 are all owned by root instead of the gear user.  In this case the database can't be created under /var/lib/openshift/[uuid]/mysql-5.1/data
Comment 3 Brenton Leanhardt 2013-06-10 13:48:18 EDT
Definitely seems related to the v2 cartridge format.  I'm guessing the configure script is called differently in 1.2 and the 'secure_cart_instance_dir' would need to be called sooner or an explicit chown would have to be added similar to the postgres cartridge.

That said:

* it's only a bug in the mysql v1 cartridge (other DBs work fine)
* it would be simple to fix if we need to.  Right now the plan is not to update the v1 cartridges if we don't have to.

I'm going to leave this bug against 1.2 until I verify whether or not we have to ship the v1 cartridges for 1.2.
Comment 5 Johnny Liu 2013-06-17 05:32:50 EDT
Actually this issue is introduced by selinux packages - selinux-policy-3.7.19-195.el6_4.10.noarch and selinux-policy-targeted-3.7.19-195.el6_4.10.noarch.

When embed mysql to app, the following avc denial message is generated.

type=AVC msg=audit(1371461211.708:59326): avc:  denied  { search } for  pid=11574 comm="mysqld" name="openshift" dev=dm-0 ino=25212 scontext=system_u:system_r:mysqld_t:s0-s0:c0.c1023 tcontext=system_u:object_r:openshift_var_lib_t:s0 tclass=dir
type=AVC msg=audit(1371461211.711:59327): avc:  denied  { getattr } for  pid=11574 comm="mysqld" path="/var/lib/openshift" dev=dm-0 ino=25212 scontext=system_u:system_r:mysqld_t:s0-s0:c0.c1023 tcontext=system_u:object_r:openshift_var_lib_t:s0 tclass=dir
type=AVC msg=audit(1371461211.732:59328): avc:  denied  { search } for  pid=11574 comm="mysqld" name="openshift" dev=dm-0 ino=25212 scontext=system_u:system_r:mysqld_t:s0-s0:c0.c1023 tcontext=system_u:object_r:openshift_var_lib_t:s0 tclass=dir
type=AVC msg=audit(1371461211.732:59329): avc:  denied  { search } for  pid=11574 comm="mysqld" name="openshift" dev=dm-0 ino=25212 scontext=system_u:system_r:mysqld_t:s0-s0:c0.c1023 tcontext=system_u:object_r:openshift_var_lib_t:s0 tclass=dir
type=AVC msg=audit(1371461211.733:59330): avc:  denied  { search } for  pid=11574 comm="mysqld" name="openshift" dev=dm-0 ino=25212 scontext=system_u:system_r:mysqld_t:s0-s0:c0.c1023 tcontext=system_u:object_r:openshift_var_lib_t:s0 tclass=dir
type=AVC msg=audit(1371461211.733:59331): avc:  denied  { search } for  pid=11574 comm="mysqld" name="openshift" dev=dm-0 ino=25212 scontext=system_u:system_r:mysqld_t:s0-s0:c0.c1023 tcontext=system_u:object_r:openshift_var_lib_t:s0 tclass=dir
type=AVC msg=audit(1371461211.734:59332): avc:  denied  { search } for  pid=11574 comm="mysqld" name="openshift" dev=dm-0 ino=25212 scontext=system_u:system_r:mysqld_t:s0-s0:c0.c1023 tcontext=system_u:object_r:openshift_var_lib_t:s0 tclass=dir


1). Installed selinux-policy-3.7.19-195.el6_4.10.noarch and selinux-policy-targeted-3.7.19-195.el6_4.10.noarch, and set selinux to permissive, embed mysql successfully.
or
2). Downgrade selinux to selinux-policy-3.7.19-195.el6_4.6.noarch and selinux-policy-targeted-3.7.19-195.el6_4.6.noarch, and set selinux to enforcing, embed mysql successfully.

This is a regression bug introduced by new selinux packages.

The same issue also exists for 1.1.z release, once 1.1.z user install the latest selinux packages, they will also encounter this issue.
Comment 6 Brenton Leanhardt 2013-06-17 05:43:32 EDT
Yikes. Miroslav, do you have any idea what is happening here?
Comment 7 Miroslav Grepl 2013-06-17 07:00:59 EDT
Well I don't see it as regression. We don't allow this. 

The problem is the mysqld tries to search mountpoints. Are you really sure you downgraded only the policy to make this working?
Comment 8 Johnny Liu 2013-06-17 08:26:27 EDT
(In reply to Miroslav Grepl from comment #7)
> Well I don't see it as regression. We don't allow this. 
> 
> The problem is the mysqld tries to search mountpoints. Are you really sure
> you downgraded only the policy to make this working?

Yes, I really sure about that, I already tried that, after downgrade, it indeed works.
Comment 9 Miroslav Grepl 2013-06-17 08:55:44 EDT
Are you getting these AVC msgs also with the downgraded policy?
Comment 10 Johnny Liu 2013-06-17 09:16:46 EDT
(In reply to Miroslav Grepl from comment #9)
> Are you getting these AVC msgs also with the downgraded policy?

No, did not get these AVC msgs also with the downgraded policy
Comment 12 Miroslav Grepl 2013-06-17 09:29:19 EDT
Ok, maybe I know where the problem is.

Could you add me output of

# ps -eZ |grep mysqld

for both policies during a test.
Comment 13 Johnny Liu 2013-06-17 10:09:25 EDT
(In reply to Miroslav Grepl from comment #12)
> Ok, maybe I know where the problem is.
> 
> Could you add me output of
> 
> # ps -eZ |grep mysqld
> 
> for both policies during a test.

Test with old selinux package (selinux-policy-3.7.19-195.el6_4.6.noarch and selinux-policy-targeted-3.7.19-195.el6_4.6.noarch), "ps -eZ |grep mysqld" get the following output.
system_u:system_r:openshift_initrc_t:s0-s0:c0.c1023 23425 ? 00:00:00 mysqld
system_u:system_r:openshift_t:s0:c0,c500 23886 ? 00:00:00 mysqld_safe

Test with new selinux package (selinux-policy-3.7.19-195.el6_4.10.noarch and selinux-policy-targeted-3.7.19-195.el6_4.10.noarch), "ps -eZ |grep mysqld" get nothing.

Chat with mgrepl via IRC, get the following details.
<jialiu> mgrepl, when i use new package, i can not capture any mysqld process
<jialiu> mgrepl, when i use old selinux package, get the following output - http://pastebin.test.redhat.com/147524
<mgrepl> ok
<mgrepl> so yes, please open a new bug for selinux-policy
<mgrepl> basically this is caused by another fix 
<mgrepl> jialiu:^
<mgrepl> system_u:system_r:openshift_initrc_t:s0-s0:c0.c1023 23400 ? 00:00:00 mysqld
* mpg has quit (Quit: Leaving)
<mgrepl> is wrong .. with updated packages it runs as mysqld_t
<mgrepl> and this is a reason why you are getting these AVC msgs

So clone this bug to selinux-policy component for tracking it.
Comment 14 Miroslav Grepl 2013-06-21 10:24:33 EDT
Could you open a new SELinux bug. Thank you.
Comment 15 Brenton Leanhardt 2013-06-21 10:55:35 EDT
Johnny opened Bug #975056
Comment 16 Brenton Leanhardt 2013-08-15 11:03:24 EDT
Closing this since the v1 cart is no longer supported.

Note You need to log in before you can comment on or make changes to this bug.