Bug 1120964 - Windows NTFS volume corrupted beyond repair during installation
Summary: Windows NTFS volume corrupted beyond repair during installation
Status: CLOSED ERRATA
Alias: None
Product: Fedora
Classification: Fedora
Component: python-blivet
Version: 21
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: ---
Assignee: David Lehman
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard: https://fedoraproject.org/wiki/Common...
Keywords: CommonBugs, Reopened
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2014-07-18 04:22 UTC by Chris Murphy
Modified: 2014-11-25 03:06 UTC (History)
10 users (show)

(edit)
Clone Of:
(edit)
Last Closed: 2014-11-15 09:16:02 UTC


Attachments (Terms of Use)
storage.log c17 (237.95 KB, text/plain)
2014-10-21 02:05 UTC, Chris Murphy
no flags Details
program.log c17 (79.85 KB, text/plain)
2014-10-21 02:05 UTC, Chris Murphy
no flags Details
storage.log c23 (237.97 KB, text/plain)
2014-10-22 16:34 UTC, Chris Murphy
no flags Details
program.log (79.85 KB, text/plain)
2014-10-22 16:34 UTC, Chris Murphy
no flags Details
program.log c23 (new) (79.85 KB, text/plain)
2014-10-22 16:49 UTC, Chris Murphy
no flags Details
storage.log c23 (new) (237.97 KB, text/plain)
2014-10-22 16:49 UTC, Chris Murphy
no flags Details
program.log c27 (79.85 KB, text/plain)
2014-10-22 16:58 UTC, Chris Murphy
no flags Details
storage.log c27 (237.91 KB, text/plain)
2014-10-22 16:59 UTC, Chris Murphy
no flags Details

Description Chris Murphy 2014-07-18 04:22:45 UTC
Description of problem: As a result of bug 1120947, the Windows 7 system that was resized has been irreparably corrupted. I don't know if this is an ntfsprogs bug with resize, or some mistake in the computation of either the new NTFS volume size vs its partition.



Version-Release number of selected component (if applicable):
python-blivet-0.61-1
anaconda-21.48-1

How reproducible:
Uncertain, not yet attempted.

Steps to Reproduce:
1.
2.
3.

Actual results:
Windows 7 corrupted, can't be repaired, appears to be unrecoverable.

Expected results:
Not this.

Additional info:

Windows diskpart and chkdsk consider the volume not to be NTFS, but RAW. It won't even attempt to repair it.

blkid sees it as ntfs.

root@f20v ~]# ntfsfix -n /dev/sdb2
Mounting volume... Failed to read last sector (270509760): Invalid argument
HINTS: Either the volume is a RAID/LDM but it wasn't setup yet,
   or it was not setup correctly (e.g. by not using mdadm --build ...),
   or a wrong device is tried to be mounted,
   or the partition table is corrupt (partition is smaller than NTFS),
   or the NTFS boot sector is corrupt (NTFS size is not valid).
FAILED
Attempting to correct errors... Failed to read last sector (270509760): Invalid argument
HINTS: Either the volume is a RAID/LDM but it wasn't setup yet,
   or it was not setup correctly (e.g. by not using mdadm --build ...),
   or a wrong device is tried to be mounted,
   or the partition table is corrupt (partition is smaller than NTFS),
   or the NTFS boot sector is corrupt (NTFS size is not valid).
FAILED
Failed to startup volume: Invalid argument
Failed to read last sector (270509760): Invalid argument
HINTS: Either the volume is a RAID/LDM but it wasn't setup yet,
   or it was not setup correctly (e.g. by not using mdadm --build ...),
   or a wrong device is tried to be mounted,
   or the partition table is corrupt (partition is smaller than NTFS),
   or the NTFS boot sector is corrupt (NTFS size is not valid).
Trying the alternate boot sector
The alternate bootsector is usable
Set sector count to 270508031 instead of 270509760
The startup data can be fixed, but no change was requested
Volume is corrupt. You should run chkdsk.
No change made


[root@f20v ~]# fdisk -l /dev/sdb

Disk /dev/sdb: 250 GiB, 268435456000 bytes, 524288000 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0x1c66d6fa

Device    Boot     Start       End    Blocks  Id System
/dev/sdb1 *         2048    206847    102400   7 HPFS/NTFS/exFAT
/dev/sdb2         206848 270714879 135254016   7 HPFS/NTFS/exFAT
/dev/sdb3      270714880 271738879    512000  83 Linux
/dev/sdb4      271738880 524287999 126274560   5 Extended
/dev/sdb5      271740928 524287999 126273536  8e Linux LVM


So the sector ntfsfix wants 270509760 is inside the sdb2 partition.

Comment 1 David Lehman 2014-07-18 14:13:29 UTC
I see that when we align the end sector for a partition resize we always align down. This may be a mistake. In this case, we align the end sector down and that causes it to end up smaller than the filesystem.

21:18:35,978 INFO program: Running... ntfsresize -ff -s 138501M /dev/sda2

>>> end_sector = 270714879
>>> start_sector = 206848
>>> sector_size = 512
>>> length = end_sector - start_sector + 1
>>> mb = 1000.0 ** 2
>>> size_in_mb = (length * sector_size) / mb
>>> size_in_mb
138500.112384

So we did indeed corrupt your ntfs. Sorry about that.


I'm not sure there is going to be a simple solution to this, unfortunately. It involves some back and forth between device size (alignment), filesystem size (min size), and the rest of the system (new free space on disk).

Comment 2 Chris Murphy 2014-07-18 15:16:52 UTC
The Windows VM is a throw away...

Ick. Looks like ntfsresize uses MB not MiB, so it doesn't even sector align. 

The ntfsresize man page suggests under -f option it's possible to do multiple resizes without using chkdsk in between, so what about a shrink, followed by a rounded up and aligned partition change, then an ntfsresize -ff -x to expand it into that final space?

Comment 3 Chris Murphy 2014-07-18 15:44:11 UTC
Nominating as a beta blocker even though the applicable release criteria is for final. I think a beta release causing this kind of corruption isn't acceptable when users are actively encouraged to test it by virtue of it being offered on the home page for broad usage.

"All known bugs that can cause corruption of user data must be fixed or documented at Common F21 bugs."

Comment 4 Chris Murphy 2014-08-24 22:12:15 UTC
Another rationalization for beta blocking from the beta release criteria page:

A bug in a Critical Path package that:
-Cannot be fixed with a future stable update
-Has a severity rating of high or greater and no reasonable workaround 
Bug hinders execution of required Beta test plans or dramatically reduces test coverage

All of these are true: anaconda is crit path package, once released in beta it can't be fixed for beta, severity is either high or urgent (not everyone will trigger this bug), we'd expect much reduced testing for Windows NTFS resize based installs since users should avoid doing them.

This bug is not always triggered, but it looks like it's about a 50/50 shot whether the installer over or under estimates the partition size relative to the volume size.

Comment 5 David Lehman 2014-09-25 22:32:09 UTC
I think I have a pretty good handle on this one. Chris, I can get you an updates image sometime tomorrow if you are in a position to test.

Comment 6 Chris Murphy 2014-09-25 23:24:32 UTC
I can early next week.

Comment 7 David Lehman 2014-09-29 17:09:43 UTC
Let me know what version of blivet you're going to test against.

Comment 8 Adam Williamson 2014-10-03 17:15:48 UTC
Discussed at 2014-10-03 blocker review meeting: http://meetbot.fedoraproject.org/fedora-blocker-review/2014-10-03/f21-blocker-review.2014-10-03-15.58.log.txt . Accepted as a Beta blocker per the justification in c#4 (been a while since we used the 'backstop' blocker definition, but it seems justified here).

Comment 9 Adam Williamson 2014-10-08 17:16:58 UTC
Discussed at 2014-10-08 blocker review meeting: http://meetbot.fedoraproject.org/fedora-blocker-review/2014-10-08/f21-blocker-review.2014-10-08-15.58.log.txt . dlehman: could you post an updates.img against the blivet that's in TC2, that is, 0.61.3-1.fc21 ? TC2 should be reliably available for cmurf to test against when he appears.

Comment 10 David Lehman 2014-10-09 01:14:58 UTC
Updates image available for testing against blivet-0.61.3-1:

  https://dlehman.fedorapeople.org/updates/updates-1120964.0.img

Comment 11 David Lehman 2014-10-09 20:33:28 UTC
Expanded (and, hopefully, improved) udpates image:

  https://dlehman.fedorapeople.org/updates/updates-1120964.1.img

Comment 12 Chris Murphy 2014-10-15 16:18:39 UTC
I'll be using python-blivet-0.61.4-1

Comment 13 Chris Murphy 2014-10-15 16:28:52 UTC
When I use the c11 updates image against python-blivet-0.61.4-1 I get a crash during resizing that points to parent bug 1021507. Maybe expected since the image isn't meant for this version of blivet.

Comment 14 Adam Williamson 2014-10-15 19:50:35 UTC
dlehman: does c#13 tell you anything useful, or do we need Chris to try with a new updates.img or the current one against an older blivet, or something? Thanks!

This along with the udev issue in fedup is one of only two outstanding Beta blockers at this point (though we need to do more testing).

Comment 15 David Lehman 2014-10-20 15:27:07 UTC
Try this:

  https://dlehman.fedorapeople.org/updates/updates-1120964.2.img


If you hit a traceback, attach the anaconda-tb-* file to this bug.

Comment 16 Adam Williamson 2014-10-20 23:39:44 UTC
Chris, if you could test this ASAP that'd be great - we're up against go/no-go as always, it's this week.

Comment 17 Chris Murphy 2014-10-21 02:04:25 UTC
(In reply to David Lehman from comment #15)
>   https://dlehman.fedorapeople.org/updates/updates-1120964.2.img

Doesn't crash, but doesn't fix the problem either. When Windows 8.1 is started, it finds problems, attempts repairs, and is unsuccessful.

Updates image applied to Fedora-Live-Workstation-x86_64-21_Beta-TC4.iso, python-blivet-0.61.5-1.fc21, will attach program and storage logs.

Comment 18 Chris Murphy 2014-10-21 02:05:16 UTC
Created attachment 948768 [details]
storage.log c17

Comment 19 Chris Murphy 2014-10-21 02:05:29 UTC
Created attachment 948769 [details]
program.log c17

Comment 20 Chris Murphy 2014-10-21 02:19:27 UTC
From program log:
02:02:51,479 INFO program: New volume size    : 54416998912 bytes (54417 MB)
From parted u s p:
 2      718848s     107001855s  106283008s  primary   ntfs

106283008*512=54416900096
54416900096 bytes in partition
54416998912 bytes in volume
-98816 shortfall in partition size, so the end of the fs was stomped on again.

I don't know if it helps or is more complicated, but I really see no point in letting the user specify fractions of MB for resizes. If it were in whole MB, maybe this whole problem goes away?

Comment 21 Adam Williamson 2014-10-21 21:28:02 UTC
For the record, the basis on which this is accepted as a blocker is not that resize must work. It is a beta blocker under the fairly rarely-used 'backup criteria':

"A bug in a Critical Path package that:

    Cannot be fixed with a future stable update
    Has a severity rating of high or greater and no reasonable workaround (see definition of severity and priority) "

Therefore, we don't actually need to fix NTFS resizing to unblock Beta. Other viable options are to disable the functionality or simply fail in such a way that the volume is not damaged, which would make the bug lower than 'high' severity.

There is no criterion requiring resize functionality to work or even exist at Beta (or, in fact, at all) - the only resize criterion per se is for Final and states only that any resize mechanisms present must attempt resize operations sanely.

Comment 22 David Lehman 2014-10-22 15:51:21 UTC
New image:

  https://dlehman.fedorapeople.org/updates/updates-1120964.3.img


This one has several changes, the most important being:

 1. account for partition alignment in partition min/max sizes
 2. ensure resize targets are valid in terms of alignment
 3. specify resize target to ntfsresize in bytes (not MB)

The image is for use with Beta_TC4. Please provide logs if there are issues. If this image works, I'll want to try one with only the essential changes and make sure it still fixes the immediate problem.

Comment 23 Chris Murphy 2014-10-22 16:34:07 UTC
(In reply to David Lehman from comment #22)
> New image:
>   https://dlehman.fedorapeople.org/updates/updates-1120964.3.img
Still breaks the fs, will attach logs.

Comment 24 Chris Murphy 2014-10-22 16:34:32 UTC
Created attachment 949493 [details]
storage.log c23

Comment 25 Chris Murphy 2014-10-22 16:34:43 UTC
Created attachment 949494 [details]
program.log

Comment 26 Chris Murphy 2014-10-22 16:49:11 UTC
Created attachment 949495 [details]
program.log c23 (new)

Comment 27 Chris Murphy 2014-10-22 16:49:26 UTC
Created attachment 949496 [details]
storage.log c23 (new)

Comment 28 Chris Murphy 2014-10-22 16:58:09 UTC
Reproduced again with updates-1120964.3.img
parted /dev/sda u s p
 2      718848s     107323391s  106604544s  primary   ntfs
So the partition is 54581526528 bytes.
program.log
16:54:13,261 INFO program: New volume size    : 54581998080 bytes (54582 MB)
-471552
Volume is larger than partition, so the end of the volume is being stepped on still. Will attached these logs also.

Comment 29 Chris Murphy 2014-10-22 16:58:42 UTC
Created attachment 949498 [details]
program.log c27

Comment 30 Chris Murphy 2014-10-22 16:59:13 UTC
Created attachment 949499 [details]
storage.log c27

Comment 31 Chris Murphy 2014-10-22 17:59:11 UTC
updates-1120964.4.img appears to fix this, continuing to test

Comment 32 Fedora Update System 2014-10-23 14:17:00 UTC
anaconda-21.48.12-1.fc21, python-blivet-0.61.7-1.fc21 has been submitted as an update for Fedora 21.
https://admin.fedoraproject.org/updates/python-blivet-0.61.7-1.fc21,anaconda-21.48.12-1.fc21

Comment 33 Fedora Update System 2014-10-23 16:21:56 UTC
Package anaconda-21.48.12-1.fc21, python-blivet-0.61.7-1.fc21:
* should fix your issue,
* was pushed to the Fedora 21 testing repository,
* should be available at your local mirror within two days.
Update it with:
# su -c 'yum update --enablerepo=updates-testing anaconda-21.48.12-1.fc21 python-blivet-0.61.7-1.fc21'
as soon as you are able to.
Please go to the following url:
https://admin.fedoraproject.org/updates/FEDORA-2014-13542/python-blivet-0.61.7-1.fc21,anaconda-21.48.12-1.fc21
then log in and leave karma (feedback).

Comment 34 Chris Murphy 2014-10-24 21:25:14 UTC
CommonBugs suggested text: NTFS resizing support is disabled in both Automatic and Manual partitioning paths for Fedora 21 Beta. Dual boot installations are still possible by first resizing the Windows volume from within Windows, and then using the Fedora installer to install into free space.

Tested with Fedora-Live-Workstation-x86_64-21_Beta-1.iso which includes anaconda-21.48.12-1.fc21 python-blivet-0.61.7-1.fc21; against Windows 8.1 Enterprise (BIOS+vbox) default installation.

Comment 35 Adam Williamson 2014-10-25 00:34:14 UTC
We do have the option now, since we've slipped, to consider taking the 'real' fix, if David thinks there's enough time to review and test it. To give a time frame, I'd very much want to get the 'expected final' compose done by Tuesday at the latest. If that's not enough time, let's stick with the disablement.

Comment 36 Adam Williamson 2014-10-27 18:14:52 UTC
Confirmed that RC1 reports ntfs as 'Not resizable', so for blocker purposes this is VERIFIED. (We probably should keep the bug open after the update goes stable, though, as the thing in RC1 is a short term bodge/workaround, not a fix for the bug).

Comment 37 Fedora Update System 2014-10-28 18:17:56 UTC
anaconda-21.48.13-1.fc21, python-blivet-0.61.8-1.fc21 has been submitted as an update for Fedora 21.
https://admin.fedoraproject.org/updates/python-blivet-0.61.8-1.fc21,anaconda-21.48.13-1.fc21

Comment 38 Fedora Update System 2014-10-28 23:07:08 UTC
anaconda-21.48.12-1.fc21, python-blivet-0.61.7-1.fc21 has been pushed to the Fedora 21 stable repository.  If problems still persist, please make note of it in this bug report.

Comment 39 Adam Williamson 2014-10-28 23:44:54 UTC
Re-opening as noted in c#36, as what we did in .12 was not to fix this bug but to disable NTFS resizing as a workaround for it. I'll check that NTFS resizing is disabled in RC2 and drop the blocker status of the bug if it is.

Comment 40 Adam Williamson 2014-10-30 17:45:01 UTC
I confirmed that Beta RC4 does not allow NTFS resizing, which is sufficient to clear the blocker status of this bug. You cannot resize NTFS partitions through guided or custom partitioning in Beta RC4.

(for the record, we plan to fix this bug properly and re-allow NTFS resizing for Final, the disabling is a temporary measure for Beta.)

Comment 41 Fedora Update System 2014-10-31 02:42:54 UTC
anaconda-21.48.13-1.fc21, python-blivet-0.61.8-1.fc21 has been pushed to the Fedora 21 stable repository.  If problems still persist, please make note of it in this bug report.

Comment 42 Adam Williamson 2014-11-04 00:13:41 UTC
Setting back to assigned, as the update did not fix this, only work around it by disabling NTFS resize.

Comment 43 Fedora Update System 2014-11-11 21:02:45 UTC
anaconda-21.48.14-1.fc21 has been submitted as an update for Fedora 21.
https://admin.fedoraproject.org/updates/anaconda-21.48.14-1.fc21

Comment 44 Fedora Update System 2014-11-13 18:16:56 UTC
Package anaconda-21.48.14-1.fc21, python-blivet-0.61.9-1.fc21:
* should fix your issue,
* was pushed to the Fedora 21 testing repository,
* should be available at your local mirror within two days.
Update it with:
# su -c 'yum update --enablerepo=updates-testing anaconda-21.48.14-1.fc21 python-blivet-0.61.9-1.fc21'
as soon as you are able to.
Please go to the following url:
https://admin.fedoraproject.org/updates/FEDORA-2014-14928/python-blivet-0.61.9-1.fc21,anaconda-21.48.14-1.fc21
then log in and leave karma (feedback).

Comment 45 Fedora Update System 2014-11-15 09:16:02 UTC
anaconda-21.48.14-1.fc21, python-blivet-0.61.9-1.fc21 has been pushed to the Fedora 21 stable repository.  If problems still persist, please make note of it in this bug report.

Comment 46 Fedora Update System 2014-11-19 01:53:36 UTC
anaconda-21.48.15-1.fc21 has been submitted as an update for Fedora 21.
https://admin.fedoraproject.org/updates/anaconda-21.48.15-1.fc21

Comment 47 Fedora Update System 2014-11-25 03:06:11 UTC
python-blivet-0.61.10-1.fc21, anaconda-21.48.16-1.fc21 has been pushed to the Fedora 21 stable repository.  If problems still persist, please make note of it in this bug report.


Note You need to log in before you can comment on or make changes to this bug.