Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 1244232

Summary: [linearstore] Symlink creation fails when running broker as a service
Product: Red Hat Enterprise MRG Reporter: Petr Matousek <pematous>
Component: qpid-cppAssignee: Irina Boverman <iboverma>
Status: CLOSED ERRATA QA Contact: Zdenek Kraus <zkraus>
Severity: high Docs Contact:
Priority: high    
Version: DevelopmentCC: iboverma, jross, pematous, zkraus
Target Milestone: 3.2Keywords: Regression, Tracking
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
: 1247278 1247279 (view as bug list) Environment:
Last Closed: 2015-11-04 15:50:46 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 1247278, 1247279, 1261026    
Bug Blocks:    

Description Petr Matousek 2015-07-17 14:03:37 UTC
Description of problem:

An attempt to create durable queue reports a misleading/general error (Value for replyText is too large), moreover the queue is not properly created and the messages are not delivered to that queue.

NOTE: This has to be probably some permission issue, because it appears only when running qpidd as a service. When running the broker from command line everything works as expected.

This effectively blocks the EA testing because multiple CI tests are failing due to this issue. 

Version-Release number of selected component (if applicable):
qpid-cpp-server-linearstore-0.34-1
qpid-cpp-*-0.34-1

How reproducible:
100% (on all supported platforms rhel6[x86_64, i386], rhel7)

Steps to Reproduce:
1. try to create a durable node
# ./spout "q;{create:sender, node:{durable:true}}"
2. following error is reported
2015-07-17 14:52:59 [Client] warning Broker closed connection: 320, illegal-argument: Value for replyText is too large
connection-forced: illegal-argument: Value for replyText is too large
3. By running management tools see that the queue is visible, but the message was not delivered to the queue
# qpid-stat -q
Queues
  queue   dur  autoDel  excl  msg   msgIn  msgOut  bytes  bytesIn  bytesOut  cons  bind
  ================================================================================
  q       Y                      0     0      0       0      0        0         0     0
4. An attempt to send another message to that queue reports that the queue doesn't exists
# ./spout q
2015-07-17 15:53:29 [Client] warning Exception received from broker: not-found: Queue not found: q (/builddir/build/BUILD/qpid-cpp-0.34/src/qpid/broker/QueueRegistry.cpp:127) [caused by 2 \x08:\x01]
Queue q does not exist

Actual results:
Unable to use durable queues when broker is running as a service.

Expected results:
No errors reported
Durable queues can be created.
Messages can be sent towards the durable queues.

Additional info:

Broker log:

2015-07-17 15:27:18 [Broker] error Connection exception: framing-error: Queue test_node_durable_property-durable_q: create() failed: jexception 0x010c EmptyFilePool::createSymLink() threw JERR__SYMLINK: Symbolic link operation failed (file="/var/lib/qpidd/qls/p001/efp/2048k/in_use/508a1da0-ce5a-4787-b08c-7d5a05dd5014.jrnl" symlink="/var/lib/qpidd/qls/jrnl2/test_node_durable_property-durable_q/508a1da0-ce5a-4787-b08c-7d5a05dd5014.jrnl" errno=13 (Permission denied)) (/builddir/build/BUILD/qpid-cpp-0.34/src/qpid/linearstore/MessageStoreImpl.cpp:425)
2015-07-17 15:27:18 [Protocol] error Connection qpid.[::1]:5672-[::1]:44593 closed by error: Queue test_node_durable_property-durable_q: create() failed: jexception 0x010c EmptyFilePool::createSymLink() threw JERR__SYMLINK: Symbolic link operation failed (file="/var/lib/qpidd/qls/p001/efp/2048k/in_use/508a1da0-ce5a-4787-b08c-7d5a05dd5014.jrnl" symlink="/var/lib/qpidd/qls/jrnl2/test_node_durable_property-durable_q/508a1da0-ce5a-4787-b08c-7d5a05dd5014.jrnl" errno=13 (Permission denied)) (/builddir/build/BUILD/qpid-cpp-0.34/src/qpid/linearstore/MessageStoreImpl.cpp:425)(501)
2015-07-17 15:27:18 [Protocol] error Connection qpid.[::1]:5672-[::1]:44593 closed by error: illegal-argument: Value for replyText is too large(320)

Comment 1 Petr Matousek 2015-07-20 11:35:15 UTC
SElinux is preventing the symlink creation:

# ./spout "q;{create:sender, node:{durable:true}}"
2015-07-20 13:30:03 [Client] warning Broker closed connection: 320, illegal-argument: Value for replyText is too large
connection-forced: illegal-argument: Value for replyText is too large
# setenforce 0
# ./spout "q;{create:sender, node:{durable:true}}"
Message(properties={spout-id:9e4bafec-239b-48b1-95af-f3741e56a5c4:0}')

audit.log:
type=AVC msg=audit(1437391939.977:4192): avc:  denied  { create } for  pid=26365 comm="qpidd" name="a30f0971-cf2f-4b7c-916c-43c499ea50a4.jrnl" scontext=unconfined_u:system_r:qpidd_t:s0 tcontext=unconfined_u:object_r:qpidd_var_lib_t:s0 tclass=lnk_file
type=SYSCALL msg=audit(1437391939.977:4192): arch=c000003e syscall=88 success=no exit=-13 a0=7f3304009d68 a1=7f3304009ca8 a2=1 a3=653939346333342d items=0 ppid=1 pid=26365 auid=0 uid=498 gid=498 euid=498 suid=498 fsuid=498 egid=498 sgid=498 fsgid=498 tty=(none) ses=654 comm="qpidd" exe="/usr/sbin/qpidd" subj=unconfined_u:system_r:qpidd_t:s0 key=(null)

Comment 3 Petr Matousek 2015-07-22 12:13:12 UTC
Retested with selinux-policy-3.7.19-279.el6 which shall contain the fix for BZ171275. The problem still persist, updated selinux-policy packages do _not_ solve this issue. 

* el7: packages not available yet

Comment 4 Irina Boverman 2015-07-27 16:34:02 UTC
(In reply to Petr Matousek from comment #3)
> Retested with selinux-policy-3.7.19-279.el6 which shall contain the fix for
> BZ171275. The problem still persist, updated selinux-policy packages do
> _not_ solve this issue. 
> 
> * el7: packages not available yet

There is RHEL 7/selinux-policy-3.13.1-30.el7, it should be available in brew.

Comment 5 Petr Matousek 2015-07-28 08:29:54 UTC
(In reply to Irina Boverman from comment #4)
> (In reply to Petr Matousek from comment #3)
> > Retested with selinux-policy-3.7.19-279.el6 which shall contain the fix for
> > BZ171275. The problem still persist, updated selinux-policy packages do
> > _not_ solve this issue. 
> > 
> > * el7: packages not available yet
> 
> There is RHEL 7/selinux-policy-3.13.1-30.el7, it should be available in brew.

Retested with selinux-policy-3.13.1-30.el7 and the latest available selinux-policy-3.13.1-35.el7, but the issue still persists (updated selinux-policy packages doesn't solve this issue).

Comment 6 Irina Boverman 2015-08-03 15:08:58 UTC
Please test with selinux-policy-3.13.1-37.el7, https://bugzilla.redhat.com/show_bug.cgi?id=1247279 claims it was fixed there.

Comment 7 Petr Matousek 2015-08-03 15:14:12 UTC
(In reply to Irina Boverman from comment #6)
> Please test with selinux-policy-3.13.1-37.el7,
> https://bugzilla.redhat.com/show_bug.cgi?id=1247279 claims it was fixed
> there.

It was already retested, selinux-policy update solves the issue for rhel7, please see bug 1247279, comment 5 for details.

RHEL6 selinux-policy is still waiting for the update.

Comment 9 Zdenek Kraus 2015-09-14 09:09:14 UTC
MRG 3.1 is fine all along.

MRG 3.2:
qpid-cpp-server-0.34-3

current:
selinux-policy-3.7.19-279.el6_7.5.noarch
.. FAIL
type=AVC msg=audit(1441900907.537:72): avc:  denied  { create } for  pid=1909 comm="qpidd" name="a6c3a43d-288b-470c-9f44-9f65d396ee3d.jrnl" scontext=unconfined_u:system_r:qpidd_t:s0 tcontext=unconfined_u:object_r:qpidd_var_lib_t:s0 tclass=lnk_file


selinux-policy-3.13.1-23.el7_1.17.noarch
.. FAIL
type=AVC msg=audit(1441900911.268:395): avc:  denied  { create } for  pid=2867 comm="qpidd" name="d71f6a98-cada-4f2e-9448-94684a24fd8a.jrnl" scontext=system_u:system_r:qpidd_t:s0 tcontext=system_u:object_r:qpidd_var_lib_t:s0 tclass=lnk_file


new packages:
selinux-policy-3.7.19-279.el6_7.6.noarch
.. PASS

selinux-policy-3.13.1-23.el7_1.18.noarch
.. FAIL
type=AVC msg=audit(1441901424.970:396): avc:  denied  { create } for  pid=2641 comm="qpidd" name="094fbf28-1188-44a0-8a89-f13b299d848d.jrnl" scontext=system_u:system_r:qpidd_t:s0 tcontext=system_u:object_r:qpidd_var_lib_t:s0 tclass=lnk_file

Comment 12 Zdenek Kraus 2015-11-04 15:45:50 UTC
resolved by selinux-policy-3.13.1-23.el7_1.21.noarch

-> VERIFIED

Comment 13 Zdenek Kraus 2015-11-04 15:50:46 UTC
above package is live moving to close

-> CLOSED ERRATA