Bug 145307

I'd say ghost is not compatible with ext3. Indeed, this is a ghost bug.

It might be a solution to drop the journal and to use these partitions as ext2
partitions.

Closing as not a bug.

It's not an ext3 bug, but the incompatibility is ineed likely to be
due to the addition of the online resize inode.  That addition has
already been made upstream --- ghost needs to be modified to
understand it if it is going to work on newer filesystems, and RHEL-4
will have the same feature so it's something they might want to work
on now.

Disabling online resize by deleting the resize inode manually,
clearing the feature flag and running fsck may be a workaround for now.

Can you tell us how to "delete the resize inode manually" as the
"tune2fs -O ^resize_inode ..." didn't work.

Thanks.

I downloaded e2progs-1.36-rc3 from Sourceforge, compiled and 
installed the new utilities. On all filesystems, I used debugfs to 
remove "resize-inode" feature and ran e2fsck. Everything was done in 
single-user mode with all filesystems unmounted, including /. Still 
unabled to ghost partitions.

Added myself to the CC list.

re comment #4:

I've just been checking the latest e2fsprogs from bitkeeper, and using
debugfs to remove the resize_inode feature then fscking does seem to
correctly remove all traces of the resize inode from the superblock.  

After that, there's basically only one sign left that there was ever a
resize inode: the bitmaps and inode blocks won't be 100% at the start
of the block group for any block group with a superblock.

But that's legal for ext2/3, and can happen for other reasons such as
the presence of bad blocks, or if the mke2fs -R stride=$N option is
used.  If ghost cannot cope with that, it's a ghost bug.

re comment #3:

The way to delete the resize inode using FC3 e2fsprogs is:

  # debugfs -w /dev/$foo
  debugfs 1.35 (28-Feb-2004)
  debugfs:  feature ^resize_inode
  Filesystem features: has_journal filetype sparse_super
  debugfs:  kill_file <7>
   
  debugfs:  quit
  # e2fsck -fy /dev/$foo

This clears the feature flag, kills the resize inode (which is always
inode number 7), then forces a fsck to cleanup the disk blocks that
were previously owned by the resize inode.

re comment #7

The debugfs solution works (it clears the feature flag).

But after, ghost 7.5 is still unable to logical dump the partition.
Ghost 9.0 automaticaly do a physical dump.

So i will continue to tar/mke2fs/untar the partitions before using
ghost and wait for a patch from symantec.

Thanks for the time used on this problem.

I think that the resize_inode feature is not the only one that Ghost 
does not recognize. There's also the ext_attr feature that is new in 
FC3 ext2/ext3 filesystems and it's not and easy job to remove it.

I tried to remove this feature from the root filesystem using debugfs 
and e2fsck in single-user mode. It took a long time for e2fsck to 
completly check the filesystem because it had to fix the i_blocks and 
the i_file_acl of every inode on the filesystem. The result was an 
unusable filesystem.

Stephen, do you have any idea how to safely remove the ext_attr 
feature from an ext3 filesystem so I can try to Ghost it?

e2fsck ought to do it.

But FC3 includes SELinux, and installs a minimal, targetted SELinux
policy by default.  That simply won't work if you have deleted all the
extended attributes, as you'll have destroyed all of the filesystem
labels that SELinux requires.  To recover you'd need to disable
SELinux entirely (boot from a rescue disk and set SELINUX=disabled in 
/etc/sysconfig/selinux.)

If e2fsck's xattr clearing renders a box unbootable due to SELinux,
that's not a bug --- you don't expect an SELinux box to survive the
loss of its security labels.  

But if e2fsck was doing something else that actually corrupted the
filesystem, then please open that as a separate bug.

I already tried to perform a fresh install of FC3 with SELinux option 
disabled but the ext_attr feature is still present on the root 
filesystem. Then I tried to remove the feature following the same 
procedure that I described in comment #9. The result was also an 
unusable filesystem.

What do you mean by an "unusable filesystem"?  What was the failure mode?

After booting in single-user mode and umounting / (still mounted but 
read-only) so I can run e2fsck, I run the following commands:

sh-3.00# debugfs -w /dev/sda2 -R "features ^ext_attr"

sh-3.00# e2fsck -y -f /dev/sda2

.
.
.
.

/: ***** FILE SYSTEM WAS MODIFIED *****
/: ***** REBOOT LINUX *****
/: 133033/1310720 files(0.5% non-contiguous...

sh-3.00# reboot
sh: reboot: command not found
sh-3.00# ls
ls: error loading shared librairies: librt.so.1: cannot open shared 
object 
file: no such file or directory
sh-3.00# init 0

After a few INIT messages, system hangs

Then I poweroff and poweron the system:

After the kernel is uncompressed and booted, I get the following 
messages:

exec of init (/sbin/init) failed!!!: 2
umount /initrd/dev failed: 2
Kernel panic - not syncing: Attempted to kill init!

The system hangs.

It is extremely dangerous to do this --- you should always boot from a
rescue CD when attempting such surgery on a root filesystem.  Does the
same occur if you do that?

Yes it does and e2fsck tells you that you should not run e2fsck on a 
mounted filesystem, even if I selected the read-only option when I 
booted from the rescue CD. It is not possible to umounted it because 
it's busy.

If a run e2fsck anyway, it takes about two hours to complete and the 
result is the same.

In single-user mode, you can umount / and in stays mounted in read-
only mode so you can run e2fsck on it. That's the reaseon why I ran 
debugfs and e2fsck in single-user mode.

You should avoid mounting it at all from the rescue CD.  The rescue CD
should give you a minimal environment including fsck within which you
can remove the xattrs, without ever mounting it in the first place.

You're right, it's the skip option when you boot from the rescue CD. 
Selecting this option does not mount any filesystem on your disk. 
Unfortunately, it didn't change anything. I still get a corrupted 
root filesystem after running e2fsck.

This morning, I will try with e2progs-1.36-rc3 to see if the problem 
has been fixed. I know there's a lot of bugs in version 1.35. But I 
will have to build my own rescue CD to include the updated version of 
e2progs.

I didn't find a way to create a rescue disk which would include. I 
did everything in single-user mode with e2progs-1.36-rc3.

This should not be a problem considering the fact that, when you 
answer Y when you are asked to run a consistency check after booting 
a system that was not properly shutdowned and there are too many 
errors on the filesystem, you are asked to run fsck manually. You are 
then brought to a shell in single-user mode with the root filesystem 
mounted read-only.

I still get a corrupted root filesystem with e2progs-1.36-rc3. I 
submitted a bug report to Theodore Ts'o.

Good news.

I tried to ghost only the /boot partition on a fresh FC3 installation 
with SELinux disabled. I got the same error message from Ghost. Then 
I removed the ext_attr feature from /boot using the procedure 
described above. Unlike the root filesystem, /boot didn't get 
corrupted. I was then able to Ghost /boot succesfully.

Conclusion: Ghost does not like the ext_attr feature of FC3 ext2/ext3 
filesystems. The resize_inode feature does not affect Ghost. This 
applies to / and /boot only; /home is not affected even if the 
ext_attr feature is present.

The only question that remains is "why does the root filesystem gets 
corrupted after removing the ext_attr feature". If we can find an 
answer to this question, then we will have a workaround for the Ghost 
problem.

Can you please open a separate bug report for the xattr corruptions on
e2fsck?  I'm currently trying to reproduce that (using Xen
virtualisation from rawhide, wish me luck.)

After doing
- fresh FC3 installation (SElinux disable)
- boot on FC2 rescue CD
- tar cf xxx /
- mke2fs -j -L / -O dir_index /dev/hda
- tar xf xxx
nb ; ext_attr not set
- boot FC3 cause an fsck ( only on / ) which set the ext_attr feature on.

Ghost 7.5 works well for this partition, so i think it's only a
"resize_inode" problem. (and ext_attr was already present on FC2)

Remi, the fact that ext_attr feature is on does not necessarily means 
that Ghost will not work. Try to Ghost by partition instead and 
select /home and you will see that it works even if both ext_attr and 
resize_inode are set.

On / and /boot filesystems, all the files also have their extended 
ACL entries set (extended attributes), which is not the case 
on /home. That's the reason why /home can be Ghosted even if ext_attr 
is set.

When you remove the ext_attr with debugfs and the run e2fsck, e2fsck 
clears the extended ACL entries on all files. The filesystem can then 
be Ghosted even if resize_inode is still set. Moreover, if you use 
debugfs to set the ext_attr on again, you're still able to Ghost the 
filesystem because the extented ACL entries are not set on the files.

In your procedure, when you boot FC3, fsck sets the ext_attr feature 
on but does not set the extended ACL entries on the files. That 
explains why you're able to Ghost the partition.

In FC2, ext_attr was already present but the extented ACL entries on 
the files were not set. FC3 installation sets the extended ACL 
entries of the files on / and /boot. The extended ACL entries are 
necessary for the implementation of SELinux which was nonfunctional 
in FC2 due to kernel and policy incompatibilities.

If you want to see the extended attributes of a file on a fresh FC3 
installation, say /etc/passwd, type the following command:

getfattr -m . -d /etc/passwd

I opened a separate bug report for the xattr corruptions on
e2fsck. Bug report #146284.

Adding a link to bug report #146284:

https://bugzilla.redhat.com/bugzilla/show_bug.cgi?id=146284

Here's the procedure to remove resize_inode and ext_attr features 
from filesystems on an FC3 installation. Everything must be done from 
the root account.

Step 1

If you installed FC3 with SELinux enabled, which is the default, it 
you must disabled. Log on the system and edit /etc/selinux/config. 
Replace SELINUX=enforced by SELINUX=disabled. If you don't disable 
SELinux, the ext_attr will be set again on the root filesystem the 
next time you boot and every file created or modified will have its 
extended ACL entries set.

Step 2

Download e2fsprogs-1.36-rc5 from SourceForge. Click on the following 
link:

    http://sourceforge.net/projects/e2fsprogs

Step 3

Untar, compile and install the new utilities:

   tar -zxf e2fsprogs-1.36-rc5.tar.gz
   cd e2fsprogs-1.36-rc5
   ./configure
   make
   make install

Step 4

Reboot the system in single-user mode:

   type reboot

   at the GRUB splash screen, press enter then press the 'a' and
   add the word single at the end of the command line (there's a space
   between quiet and single)

Step 5

Unmount all filesystems, including /, and remove both resize_inode 
and ext_attr features. Run the following commands on all filesystems:

   debugfs -w /dev/XXXX -R "features ^resize_inode ^ext_attr"
   e2fsck -y -f /dev/XXXX

Example:

   /dev/sda1 is mounted on /boot
   /dev/sda2 is mounted on /
   /dev/sda5 is mounted on /home

   umount /boot
   umount /
   umount /home

   debugfs -w /dev/sda1 -R "features ^resize_inode ^ext_attr"
   debugfs -w /dev/sda2 -R "features ^resize_inode ^ext_attr"
   debugfs -w /dev/sda5 -R "features ^resize_inode ^ext_attr"

   e2fsck -y -f /dev/sda1
   e2fsck -y -f /dev/sda2
   e2fsck -y -f /dev/sda5

Running e2fsck on the root filesystem takes a while.

Step 6

Reboot the system.

During shutdown, you will get error messages telling you that the 
root filesystem cannot be written. That's normal because the root 
filesystem is still mounted read-only. It must remain mounted read-
only during shutdown.

During boot, you will get a message telling you that the system was 
not properly shutdown and ask you if you want to verify the 
filesystem integrity. You can skip this step. It didn't find anything 
wrong on my system.

Your system is now ready to be Ghosted.

For people who want to keep the online resizing capability and have 
SELinux enabled after Ghosting a system or after restoring FC3 on an 
another system (cloning), the features can be readded to the 
filesystems and the extended ACL entries can be restored from a 
backup file. You must create the backup file just before step 4 in 
the above procedure. Here's how to create it:

   find / -exec getfattr -d -h -m . --absolute-names {} \; 
> /home/attr.backup

The filename and it's location can be anything you want.

Here's the procedure to restore the filesystem features and the 
extended ACL entries:

Step1

Reboot the system in single-user mode:

   type reboot

   at the GRUB splash screen, press enter then press the 'a' and
   add the word single at the end of the command line (there's a space
   between quiet and single)

Step 2

Unmount all filesystems, including /, and add both resize_inode 
and ext_attr features. Run the following commands on all filesystems:

   debugfs -w /dev/XXXX -R "features resize_inode ext_attr"
   e2fsck -y -f /dev/XXXX

Example:

   /dev/sda1 is mounted on /boot
   /dev/sda2 is mounted on /
   /dev/sda5 is mounted on /home

   umount /boot
   umount /
   umount /home

   debugfs -w /dev/sda1 -R "features resize_inode ext_attr"
   debugfs -w /dev/sda2 -R "features resize_inode ext_attr"
   debugfs -w /dev/sda5 -R "features resize_inode ext_attr"

   e2fsck -y -f /dev/sda1
   e2fsck -y -f /dev/sda2
   e2fsck -y -f /dev/sda5

Step3

Reboot the system, logon as root, restore the extended ACL entries 
from the backup file and enable SELinux:

   setfattr -h --restore=/home/attr.backup

Replace SELINUX=disabled by SELINUX=enforced in /etc/selinux/config.

Reboot system.

I'm the developer of Symantec Ghost support for Linux. Ghost is unable to
intelligently clone the default ext2 and ext3 filesystems configured by Fedora
Core 3 for three reasons: it doesn't understand resize_inode, acls or LVM. This
is a Ghost problem, not a Fedora problem.

The next version of Ghost (which will be part of Ghost Solution Suite 1.x) will
support ACLs as implemented in FC3, and will partially support the resize_inode
by disabling this feature in any filesystem it creates. It's too early to say
whether it will support drives in logical volumes.

A suggested workaround at present is to use the -IAL (Image All Linux) switch to
Ghost, which will make a sector copy of Linux partitions while still
intelligently cloning FAT and NTFS partitions. This will result in much larger
image files, longer cloning times, inability to resize Linux partitions, and on
some systems (those that have a diagnostic partition before any Linux partition,
for example), may leave GRUB or LILO unable to boot.

Thanks for the update.  If there's any information we can provide to help with
this work, please feel free to ask me.