Bug 735443

Summary: Incorrect path length check on sanlock lockspace limits directory name length to 47 characters
Product: Red Hat Enterprise Linux 6 Reporter: Daniel Berrangé <berrange>
Component: libvirtAssignee: Libvirt Maintainers <libvirt-maint>
Status: CLOSED ERRATA QA Contact: Virtualization Bugs <virt-bugs>
Severity: unspecified Docs Contact:
Priority: unspecified    
Version: 6.2CC: ajia, dyuan, mzhan, rwu, veillard, weizhan
Target Milestone: rc   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: libvirt-0.9.4-8.el6 Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2011-12-06 11:28:06 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Daniel Berrangé 2011-09-02 16:11:28 UTC
Description of problem:
The libvirt sanlock plugin code uses the incorrect constant for validating the directory name length. The result is that the directory name cannot exceed 47 characters, instead of the intended 1024.

Version-Release number of selected component (if applicable):
0.9.4-1.el6

How reproducible:
Always

Steps to Reproduce:
1. Edit /etc/libvirt/qemu-sanlock.conf
2. Change the sanlock directory to a path > 48 characters long 
3. Restart libvirtd
4. Attempt to start a guest
  
Actual results:


Expected results:


Additional info:


commit 2223b1f71ff52c8b4e73b4a0c5d02f81b317e923
Author: Daniel P. Berrange <berrange>
Date:   Tue Aug 30 12:24:06 2011 -0400

    Fix incorrect path length check in sanlock lockspace setup
    
    The code for creating a sanlock lockspace accidentally used
    SANLK_NAME_LEN instead of SANLK_PATH_LEN for a size check.
    This meant disk paths were limited to 48 bytes !
    
    * src/locking/lock_driver_sanlock.c: Fix disk path length
      check

Comment 3 Alex Jia 2011-09-09 02:30:10 UTC
It has a little wired, I can't reproduce this issue on 0.9.4-1.el6, the following is my test steps, please correct me if it's wrong:

# grep "^disk_lease_dir" /etc/libvirt/qemu-sanlock.conf
disk_lease_dir = "/home/ajia/Workspace/virsh-rail/cases/common/functional/migration"

Notes: string is 65 > 48 characters.

# service libvirtd restart
Stopping libvirtd daemon:                                  [  OK  ]
Starting libvirtd daemon:                                  [  OK  ]

# virsh start vr-rhel5u4-x86_64-kvm
error: Failed to start domain vr-rhel5u4-x86_64-kvm
error: internal error unsupported configuration: Read/write, exclusive access, disks were present, but no leases specified

# grep -i avc /var/log/audit/audit.log

Notes: without any "avc:  denied" error.


Alex

Comment 4 Alex Jia 2011-09-09 02:37:56 UTC
(In reply to comment #3)
> It has a little wired, I can't reproduce this issue on 0.9.4-1.el6, the
> following is my test steps, please correct me if it's wrong:
> 
> # grep "^disk_lease_dir" /etc/libvirt/qemu-sanlock.conf
> disk_lease_dir =
> "/home/ajia/Workspace/virsh-rail/cases/common/functional/migration"
> 
> Notes: string is 65 > 48 characters.
> 
> # service libvirtd restart
> Stopping libvirtd daemon:                                  [  OK  ]
> Starting libvirtd daemon:                                  [  OK  ]
> 
> # virsh start vr-rhel5u4-x86_64-kvm
> error: Failed to start domain vr-rhel5u4-x86_64-kvm
> error: internal error unsupported configuration: Read/write, exclusive access,
> disks were present, but no leases specified
> 
> # grep -i avc /var/log/audit/audit.log
> 
> Notes: without any "avc:  denied" error.
Ah, the above step isn't related to this bug, so please ignore 'avc' check in here.

It should be check whether the following error information is raised by libvirt:
_(Lockspace path '%s' exceeded %d characters, path, SANLK_PATH_LEN)

However, I haven't seen this error based on above steps, I will research it again.


Alex
> 
> 
> Alex

Comment 5 Alex Jia 2011-09-09 02:55:28 UTC
(In reply to comment #4)
> (In reply to comment #3)
> > It has a little wired, I can't reproduce this issue on 0.9.4-1.el6, the
> > following is my test steps, please correct me if it's wrong:
> > 
> > # grep "^disk_lease_dir" /etc/libvirt/qemu-sanlock.conf
> > disk_lease_dir =
> > "/home/ajia/Workspace/virsh-rail/cases/common/functional/migration"
> > 
> > Notes: string is 65 > 48 characters.
> > 
> > # service libvirtd restart
> > Stopping libvirtd daemon:                                  [  OK  ]
> > Starting libvirtd daemon:                                  [  OK  ]
> > 
> > # virsh start vr-rhel5u4-x86_64-kvm
> > error: Failed to start domain vr-rhel5u4-x86_64-kvm
> > error: internal error unsupported configuration: Read/write, exclusive access,
> > disks were present, but no leases specified
I forgot to disable 'sanlock' in qemu.conf, so I met the above error, now I disable 'sanlock' then restart libvirtd service and can successfully start guest without any error:
# virsh start vr-rhel5u4-x86_64-kvm
Domain vr-rhel5u4-x86_64-kvm started

Alex

Comment 6 Daniel Berrangé 2011-09-12 16:01:32 UTC
> # virsh start vr-rhel5u4-x86_64-kvm
> error: Failed to start domain vr-rhel5u4-x86_64-kvm
> error: internal error unsupported configuration: Read/write, exclusive access,
> disks were present, but no leases specified

This indicates you need

  auto_disk_leases = 1


enabled in /etc/libvirt/qemu-sanlock.conf

Comment 7 Alex Jia 2011-09-13 02:35:37 UTC
(In reply to comment #6)
Hi Daniel,

Thanks for you comment, although I enable auto_disk_leases = 1 and let disk_lease_dir = "/home/ajia/Workspace/virsh-rail/cases/common/functional/migration" in /etc/libvirt/qemu-sanlock.conf then restart libvirtd service,
I still can't reproduce the bug based on libvirt-0.9.4-1.el6.x86_64 when I start a guest (guest can be successfully started), I haven't seen any error like this "_(Lockspace path '%s' exceeded %d characters, path, SANLK_PATH_LEN)".

Maybe, my environment is still different from you, in addition, I try to enable other items in /etc/libvirt/qemu-sanlock.conf and turn off/on sanlock service, the result is the same to previous, so I need your help to reproduce this issue firstly.

Thanks in advance,
Alex

Comment 8 Daniel Berrangé 2011-09-13 11:09:07 UTC
Here is a better set of steps to reproduce the problem, I have re-verified with 0.9.4-7.el6

$ service libvirtd stop
$ mkdir /var/lib/libvirt/images/sanlock
$ yum install /usr/bin/augtool
$ augtool
augtool> set /files/etc/libvirt/qemu.conf/lock_manager "sanlock"
augtool> set /files/etc/libvirt/qemu-sanlock.conf/host_id 1
augtool> set /files/etc/libvirt/qemu-sanlock.conf/auto_disk_leases 1
augtool> set /files/etc/libvirt/qemu-sanlock.conf/disk_lease_dir "/var/lib/libvirt/images/sanlock"
augtool> save
Saved 1 file(s)
augtool> quit

$ /usr/sbin/libvirtd 
12:06:34.482: 3726: info : libvirt version: 0.9.4, package: 7.el6 (Red Hat, Inc. <http://bugzilla.redhat.com/bugzilla>, 2011-08-30-23:09:11, x86-008.build.bos.redhat.com)
12:06:34.482: 3726: error : virLockManagerSanlockSetupLockspace:165 : internal error Lockspace path '/var/lib/libvirt/images/sanlock/__LIBVIRT__DISKS__' exceeded 48 characters
12:06:34.483: 3726: error : qemudLoadDriverConfig:457 : Failed to load lock manager sanlock
12:06:34.484: 3726: error : qemudStartup:556 : Missing lock manager implementation
12:06:34.485: 3726: error : virStateInitialize:849 : Initialization of QEMU state driver failed
12:06:34.534: 3726: error : daemonRunStateInit:1175 : Driver state initialization failed

Comment 9 Alex Jia 2011-09-14 03:23:40 UTC
I can reproduce this issue on 6.2 Beta with libvirt-0.9.4-1.el6.x86_64 based on the above steps:

10:49:46.651: 28684: error : virLockManagerSanlockSetupLockspace:165 : internal error Lockspace path '/var/lib/libvirt/images/sanlock/__LIBVIRT__DISKS__' exceeded 48 characters
10:49:46.651: 28684: error : qemudLoadDriverConfig:457 : Failed to load lock manager sanlock
10:49:46.651: 28684: error : qemudStartup:540 : Missing lock manager implementation
10:49:46.651: 28684: error : virStateInitialize:846 : Initialization of QEMU state driver failed
10:49:46.675: 28684: error : daemonRunStateInit:1162 : Driver state initialization failed

Although I haven't met the same issue to previous with libvirt-0.9.4-11.el6.x86_64. However, there is another issue when run /usr/sbin/libvirtd:

11:15:16.608: 29754: error : virLockManagerSanlockSetupLockspace:242 : Unable to add lockspace /var/lib/libvirt/images/sanlock/__LIBVIRT__DISKS__: Invalid argument
11:15:16.608: 29754: error : qemudLoadDriverConfig:457 : Failed to load lock manager sanlock
11:15:16.608: 29754: error : qemudStartup:566 : Missing lock manager implementation
11:15:16.608: 29754: error : virStateInitialize:849 : Initialization of QEMU state driver failed
11:15:16.634: 29754: error : daemonRunStateInit:1181 : Driver state initialization failed

Is this a expected result? or should file a new bug to trace this issue.


Thanks,
Alex

Comment 10 Daniel Berrangé 2011-09-14 08:19:28 UTC
Try shutting down sanlock and libvirtd, and deleting the __LIBVIRT_DISKS__ file and any other files in that same directory. Then start sanlock again and start libvirtd

Comment 11 Alex Jia 2011-09-14 09:32:36 UTC
(In reply to comment #10)
> Try shutting down sanlock and libvirtd, and deleting the __LIBVIRT_DISKS__ file
> and any other files in that same directory. Then start sanlock again and start
> libvirtd

Daniel, thanks for your comments, libvirt works well for me now after trying the above steps, I haven't met "internal error Lockspace path '/var/lib/libvirt/images/sanlock/__LIBVIRT__DISKS__' exceeded 48 characters" again, so move the bug to VERIFIED status.

Alex

Comment 12 errata-xmlrpc 2011-12-06 11:28:06 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

http://rhn.redhat.com/errata/RHBA-2011-1513.html