Description of problem:

MySQL inside Xen VM refuses to start on dedicated partition. What it means that
after creating VM with MySQL and attaching extra storage to that VM to be used
for partition mounted under /var/lib/mysql /etc/init.d/mysqld complains about
"Timeout". However if run either as:

bash -x /etc/init.d/mysqld start

or

mysqld_safe

it starts just fine.


Version-Release number of selected component (if applicable):

redhat-release-5Server-5.1.0.2
mysql-server-5.0.22-2.2.el5_1.1
mysql-5.0.22-2.2.el5_1.1

How reproducible:

always

Steps to Reproduce:
1. create a fresh VM with minimal install of RHEL5 inside and start it up
2. install MySQL inside VM
3. start up MySQL

  domU # service mysqld start 

4. create LVM logical volume inside dom0: /dev/xen/data.partition
5. attach extra disk to VM:
xm block-attach del phy:/dev/xen/data.partition /dev/xvdb w

6. inside domU create new partition (it doesn't matter if you use entire
attached disk or partition it first with fdisk, so if you want you can run fdisk
/dev/xvdb first and create /dev/xvdb1 and use it instead of /dev/xvdb throughout).

 domU # pvcreate /dev/xvdb
 domU # vgcreate data /dev/xvdb
 domU # vgscan
 domU # vgchange -a y
 domU # lvcreate -n datalv -L 20G /dev/data
 domU # mke2fs /dev/data/datalv
 domU # tune2fs -j /dev/data/datalv
 
7. migrate mysqld data over to new partition

 domU # service mysqld stop
 domU # mount /dev/data/datalv /mnt
 domU # tar -cf - -C /var/lib . | tar -xpf - -C /mnt
 domU # umount /mnt
 domU # mv /var/lib /var/lib/old ; mkdir /var/lib
 domU # mount /dev/data/datalv /var/lib

8. try to start up mysqld now

 domU # service mysqld start

Actual results:

1. with mounted partition
 domU # service mysqld start
   Timeout error occurred trying to start MySQL Daemon.
   Starting MySQL:                                            [FAILED]
 domU # bash -x /etc/init.d/mysqld start
  ........
 domU # ps ax | grep mysql > /dev/null && echo "MySQL is running"
  MySQL is running
 domU # service mysqld stop
   Stopping MySQL:                                            [  OK  ]
 domU # service mysqld start
   Timeout error occurred trying to start MySQL Daemon.
   Starting MySQL:                                            [FAILED]
 domU #

 domU # mysqld_safe
  Starting mysqld daemon with databases from /var/lib/mysql
  <Ctrl-Z>
  [1]+  Stopped                 mysqld_safe
 domU # bg
  [1]+ mysqld_safe &
 domU # mysql -p
  Enter password:
  Welcome to the MySQL monitor.  Commands end with ; or \g.
  Your MySQL connection id is 1 to server version: 5.0.22

  Type 'help;' or '\h' for help. Type '\c' to clear the buffer.

  mysql> show databases;
  +--------------------+
  | Database           |
  +--------------------+
  | information_schema |
  | mysql              |
  | test               |
  +--------------------+
  4 rows in set (0.00 sec)

  mysql> quit
 domU # mysqladmin shutdown -p
  Enter password:
  STOPPING server from pid file /var/run/mysqld/mysqld.pid
  080222 10:05:02  mysqld ended

  [1]+  Done                    mysqld_safe
 domU # cat /var/log/mysql.log
  ....
  080222 10:07:02  mysqld started
  080222 10:07:02 [Warning] Can't create test file /var/lib/mysql/domU.lower-test
  ^G/usr/libexec/mysqld: Can't change dir to '/var/lib/mysql/' (Errcode: 13)
  080222 10:07:02 [ERROR] Aborting
  ....

  domU # rm -f /var/lib/mysql/domU.lower-test; su mysql -c 'touch
/var/lib/mysql/domU.lower-test' && ls -1 /var/lib/mysql/domU.lower-test
/var/lib/mysql/domU.lower-test

Expected results:

 domU # service mysqld start 
  Starting MySQL:                                            [  OK  ]

 In other words - I expected all 3 scenarios to work the same - obviously - they
don't. Maybe it's a bug in startup script or maybe it's a bug in bash either way
- startup script starts to work all of sudden with debugging on, which is
very-very suspicious. However it does work with "native" disk inside VM - /dev/xvda.

Additional info:

 I will try to play around some more but as it is this bug is very-very annoying
and is a big showstopper with our plan of rolling out Xen environments.

 I've done it several times now changing steps slightly with the same result -
as soon as I mount /var/lib or /var/lib/mysql, /etc/init.d/mysqld stops working.