Bug 434546 - MySQL inside Xen VM refuses to start on dedicated partition
MySQL inside Xen VM refuses to start on dedicated partition
Status: CLOSED NOTABUG
Product: Red Hat Enterprise Linux 5
Classification: Red Hat
Component: mysql (Show other bugs)
5.1
i386 Linux
low Severity high
: rc
: ---
Assigned To: Tom Lane
:
Depends On:
Blocks:
  Show dependency treegraph
 
Reported: 2008-02-22 12:19 EST by Dmitry S. Makovey
Modified: 2013-07-02 23:17 EDT (History)
2 users (show)

See Also:
Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
Environment:
Last Closed: 2008-03-05 15:21:30 EST
Type: ---
Regression: ---
Mount Type: ---
Documentation: ---
CRM:
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---


Attachments (Terms of Use)

  None (edit)
Description Dmitry S. Makovey 2008-02-22 12:19:02 EST
Description of problem:

MySQL inside Xen VM refuses to start on dedicated partition. What it means that
after creating VM with MySQL and attaching extra storage to that VM to be used
for partition mounted under /var/lib/mysql /etc/init.d/mysqld complains about
"Timeout". However if run either as:

bash -x /etc/init.d/mysqld start

or

mysqld_safe

it starts just fine.


Version-Release number of selected component (if applicable):

redhat-release-5Server-5.1.0.2
mysql-server-5.0.22-2.2.el5_1.1
mysql-5.0.22-2.2.el5_1.1

How reproducible:

always

Steps to Reproduce:
1. create a fresh VM with minimal install of RHEL5 inside and start it up
2. install MySQL inside VM
3. start up MySQL

  domU # service mysqld start 

4. create LVM logical volume inside dom0: /dev/xen/data.partition
5. attach extra disk to VM:
xm block-attach del phy:/dev/xen/data.partition /dev/xvdb w

6. inside domU create new partition (it doesn't matter if you use entire
attached disk or partition it first with fdisk, so if you want you can run fdisk
/dev/xvdb first and create /dev/xvdb1 and use it instead of /dev/xvdb throughout).

 domU # pvcreate /dev/xvdb
 domU # vgcreate data /dev/xvdb
 domU # vgscan
 domU # vgchange -a y
 domU # lvcreate -n datalv -L 20G /dev/data
 domU # mke2fs /dev/data/datalv
 domU # tune2fs -j /dev/data/datalv
 
7. migrate mysqld data over to new partition

 domU # service mysqld stop
 domU # mount /dev/data/datalv /mnt
 domU # tar -cf - -C /var/lib . | tar -xpf - -C /mnt
 domU # umount /mnt
 domU # mv /var/lib /var/lib/old ; mkdir /var/lib
 domU # mount /dev/data/datalv /var/lib

8. try to start up mysqld now

 domU # service mysqld start

Actual results:

1. with mounted partition
 domU # service mysqld start
   Timeout error occurred trying to start MySQL Daemon.
   Starting MySQL:                                            [FAILED]
 domU # bash -x /etc/init.d/mysqld start
  ........
 domU # ps ax | grep mysql > /dev/null && echo "MySQL is running"
  MySQL is running
 domU # service mysqld stop
   Stopping MySQL:                                            [  OK  ]
 domU # service mysqld start
   Timeout error occurred trying to start MySQL Daemon.
   Starting MySQL:                                            [FAILED]
 domU #

 domU # mysqld_safe
  Starting mysqld daemon with databases from /var/lib/mysql
  <Ctrl-Z>
  [1]+  Stopped                 mysqld_safe
 domU # bg
  [1]+ mysqld_safe &
 domU # mysql -p
  Enter password:
  Welcome to the MySQL monitor.  Commands end with ; or \g.
  Your MySQL connection id is 1 to server version: 5.0.22

  Type 'help;' or '\h' for help. Type '\c' to clear the buffer.

  mysql> show databases;
  +--------------------+
  | Database           |
  +--------------------+
  | information_schema |
  | mysql              |
  | test               |
  +--------------------+
  4 rows in set (0.00 sec)

  mysql> quit
 domU # mysqladmin shutdown -p
  Enter password:
  STOPPING server from pid file /var/run/mysqld/mysqld.pid
  080222 10:05:02  mysqld ended

  [1]+  Done                    mysqld_safe
 domU # cat /var/log/mysql.log
  ....
  080222 10:07:02  mysqld started
  080222 10:07:02 [Warning] Can't create test file /var/lib/mysql/domU.lower-test
  ^G/usr/libexec/mysqld: Can't change dir to '/var/lib/mysql/' (Errcode: 13)
  080222 10:07:02 [ERROR] Aborting
  ....

  domU # rm -f /var/lib/mysql/domU.lower-test; su mysql -c 'touch
/var/lib/mysql/domU.lower-test' && ls -1 /var/lib/mysql/domU.lower-test
/var/lib/mysql/domU.lower-test

Expected results:

 domU # service mysqld start 
  Starting MySQL:                                            [  OK  ]

 In other words - I expected all 3 scenarios to work the same - obviously - they
don't. Maybe it's a bug in startup script or maybe it's a bug in bash either way
- startup script starts to work all of sudden with debugging on, which is
very-very suspicious. However it does work with "native" disk inside VM - /dev/xvda.

Additional info:

 I will try to play around some more but as it is this bug is very-very annoying
and is a big showstopper with our plan of rolling out Xen environments.

 I've done it several times now changing steps slightly with the same result -
as soon as I mount /var/lib or /var/lib/mysql, /etc/init.d/mysqld stops working.
Comment 1 Dmitry S. Makovey 2008-02-22 12:33:02 EST
just performed very same scenario creating logical volume on volume group
located on /dev/xvda (which is the initial drive for the minimal VM) - I've got
same results.

kernel-xen-2.6.18-53.1.13.el5 on both dom[0,U]
Comment 2 Tom Lane 2008-03-03 20:45:28 EST
errno 13 is "Permission denied", which doesn't sound like mysql's fault.  I'm thinking that you tripped over 
a SELinux restriction on where the database directory can be.  This would be enforced only against the 
service-started daemon, not against execution from an interactive command, which explains the varying 
results.

Try using star, or manually setting the right security context on the copied files.
Comment 3 Dmitry S. Makovey 2008-03-05 10:48:02 EST
sorry for noise - that's what it was - I thought SELinux was disabled so I 
didn't even check. You can close it.

Note You need to log in before you can comment on or make changes to this bug.