Bug 1527231 - Linux cannot mount NTFS filesystem from Windows 10 creators update 1709
Summary: Linux cannot mount NTFS filesystem from Windows 10 creators update 1709
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Fedora
Classification: Fedora
Component: ntfs-3g
Version: 27
Hardware: x86_64
OS: Linux
unspecified
high
Target Milestone: ---
Assignee: Tom "spot" Callaway
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2017-12-18 22:40 UTC by Winfrid Tschiedel
Modified: 2018-01-10 02:07 UTC (History)
2 users (show)

Fixed In Version: ntfs-3g-2017.3.23-4.fc26 ntfs-3g-2017.3.23-4.fc27
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2018-01-09 16:49:51 UTC
Type: Bug


Attachments (Terms of Use)
BR for working NTFS file system /dev/sdf1 (512 bytes, application/octet-stream)
2017-12-19 14:08 UTC, Winfrid Tschiedel
no flags Details
BR for working NTFS file system /dev/sdf (512 bytes, application/octet-stream)
2017-12-19 14:11 UTC, Winfrid Tschiedel
no flags Details
BR for working NTFS file system /dev/sdg1 (512 bytes, application/octet-stream)
2017-12-19 14:14 UTC, Winfrid Tschiedel
no flags Details
BR for working NTFS file system /dev/sdg (512 bytes, application/octet-stream)
2017-12-19 14:15 UTC, Winfrid Tschiedel
no flags Details
ntfsinfo output for NTFS device /dev/sdg (1.12 KB, text/plain)
2017-12-19 14:28 UTC, Winfrid Tschiedel
no flags Details
requested mft (64.00 KB, application/octet-stream)
2017-12-19 18:59 UTC, Winfrid Tschiedel
no flags Details
requested mftmirr (64.00 KB, application/octet-stream)
2017-12-19 19:00 UTC, Winfrid Tschiedel
no flags Details
requested mft (updated version) (64.00 KB, application/octet-stream)
2017-12-20 06:55 UTC, Winfrid Tschiedel
no flags Details
requested mftmirr (updated version) (64.00 KB, application/octet-stream)
2017-12-20 06:56 UTC, Winfrid Tschiedel
no flags Details
Proposed patch for fixing the $MFT/$MFTMirr mismatch (550 bytes, patch)
2017-12-20 07:29 UTC, Jean-Pierre André
no flags Details | Diff

Description Winfrid Tschiedel 2017-12-18 22:40:20 UTC
Description of problem:

Mounting an NTFS filesystem which was open by the newest creators update of Microsoft Windows 10 fails :
mount -o ro -t ntfs /dev/sde1 /mnt1
$MFTMirr does not match $MFT (record 28).
Failed to mount '/dev/sde1': Input/output error
NTFS is either inconsistent, or there is a hardware fault, or it's a
SoftRAID/FakeRAID hardware. In the first case run chkdsk /f on Windows
then reboot into Windows twice. The usage of the /f parameter is very
important! If the device is a SoftRAID/FakeRAID then first activate
it and mount a different device under the /dev/mapper/ directory, (e.g.
/dev/mapper/nvidia_eahaabcc1). Please see the 'dmraid' documentation
for more details.

A check of the filesystem under Windows 10 x64 does not report any errors !

A NTFS filesystem opened by Windows 10 creators update Spring 2017 is opened without any problems :

df -kT
Filesystem     Type      1K-blocks     Used  Available Use% Mounted on
devtmpfs       devtmpfs    4012800        0    4012800   0% /dev
tmpfs          tmpfs       4020332    32048    3988284   1% /dev/shm
tmpfs          tmpfs       4020332     2504    4017828   1% /run
tmpfs          tmpfs       4020332        0    4020332   0% /sys/fs/cgroup
/dev/sdc8      xfs        36160256 16991080   19169176  47% /
tmpfs          tmpfs        804068       20     804048   1% /run/user/0
/dev/sdj1      fuseblk  3906982848   187776 3906795072   1% /run/media/root/Passport-2 4TB


Version-Release number of selected component (if applicable):

It is independent of the linux distribution -
reproduced also on openSUSE Leap 42.3 or openSUSE 13.2


How reproducible: always


Steps to Reproduce:
1. Have a NTFS filesystem which was opened with Windows 10 creators update fall 2017
2. try to mount this NTFS filesystem

Actual results:
any linux (fedora 27, openSUSE Leap 42.3 / 13.2,  ...) fails to mount the NTFS filesystem


Expected results:
linux should be able mount NTFS, like it was before installing Windows 10 creators update Fall 2017


Additional info:

Comment 1 Jean-Pierre André 2017-12-19 07:36:08 UTC
> $MFTMirr does not match $MFT (record 28).

You are using an unusual formatting, probably with big clusters.

Please post the metadata of the rejected partition (probably /dev/sdj1). As root, with the partition unmounted do :

ntfsclone -mst --full-logfile --ignore-fs-check -o - /dev/sdj1 | gzip > sdj1-metadata.gz

This might be big an incompatible with this bugzilla, so use another public storage. You may get a smaller size by using "xz -T0" instead of gzip.

The metadata file contains no user data though the file names are visible.

Comment 2 Jean-Pierre André 2017-12-19 07:43:06 UTC
edit : Your ntfs partition is probably sde1 ("df" obviously can only show mounted partitions)

Comment 3 Jean-Pierre André 2017-12-19 08:09:17 UTC
edit 2 : I see you will the $MFT/$MFTMirr mismatch will prevent you from running ntfsclone, so I have to go though a few steps to get the formatting parameters.

First, please post the boot sector :

dd if=/dev/sde1 of=bootsector bs=512 count=1

The bootsector file will point me to further needed data.

Comment 4 Winfrid Tschiedel 2017-12-19 14:08:33 UTC
Created attachment 1370044 [details]
BR for working NTFS file system /dev/sdf1

Comment 5 Winfrid Tschiedel 2017-12-19 14:11:47 UTC
Created attachment 1370045 [details]
BR for working NTFS file system /dev/sdf

I also attached the first 512 byte of /def/sdf

/dev/sdf was not used under windows after installing creators update Fall 2017

Comment 6 Winfrid Tschiedel 2017-12-19 14:14:16 UTC
Created attachment 1370046 [details]
BR for working NTFS file system /dev/sdg1

same as for /dev/sdf1 - but this file system was used also after installation of creators update Fall 2017

Comment 7 Winfrid Tschiedel 2017-12-19 14:15:10 UTC
Created attachment 1370047 [details]
BR for working NTFS file system /dev/sdg

same as for /dev/sdf - but this file system was used also after installation of creators update Fall 2017

Comment 8 Winfrid Tschiedel 2017-12-19 14:20:02 UTC
The problem is not restricted to linux,
also with my DVB-S2 receiver (AliChip Set) I have similar problems,
but these are restricted to USB disks - USB Sticks with NTFS work.

Comment 9 Winfrid Tschiedel 2017-12-19 14:28:29 UTC
Created attachment 1370053 [details]
ntfsinfo output for NTFS device /dev/sdg

Comment 10 Winfrid Tschiedel 2017-12-19 14:36:40 UTC
Hi Jean-Pierre,

first many thanks for your quick response.

Just to make it clear - all my NTFS file systems were created before installing Windows 10 x64 creators update Fall 2017. 
Until I installed Windows 10 x64 creators update Fall 2017 I was able to access to any of my NTFS file systems from any off my linux distributions.

Both disks are WD Passport 4TB 2.5" with NTFS and clustersize 64k -
the only difference /dev/sdf was not used under windows 10 creators update Fall 2017

Take care,

Winfrid

Comment 11 Jean-Pierre André 2017-12-19 17:00:34 UTC
Thank you for your posts. I was asking for the partitions boot sectors (like sdf1, sdg1), not the devices ones (sdf, sdg). Also in your initial post, the partition which you could not mount was sde1, is it the same partition which you now show as sdf1 or sdg1 ? I am of course mostly interested in the partition which cannot be mounted.

Fortunately, the partitions sdf1 and sdg1 have the same formatting parameters (3.5GB and 512-byte sectors), but their clustering is unusual (65536 bytes), which means you lose a lot of space in small files. These partitions have not been formatted by mkntfs, I suspect they were formatted by an earlier version of Windows. Do you know which formatter was used ? Was the cluster size intentional ?

So, now, please post the first clusters of $MFT and of $MFTMirr, if possible from a partition which cannot be mounted (if you have no unmountable partition, use sdf1 or sdg1). If this is from sdf1 or sdg1, the locations are the same, and the commands are :

dd if=/dev/sdf1 of=mft skip=1 bs=512 count=128
dd if=/dev/sdf1 of=mftmirr skip=3072 bs=512 count=128

Note : apparently Microsoft has changed the rules for copying $MFT into $MFTMirr, and I will have to do accordingly when I understand the new rules. Will you be able to compile ntfs-3g and test a patch ?

In the meantime, you can probably fix the partitions by using :
ntfsfix -d /dev/sdf1
I however suggest I have a look at your $MFTMirr before you try.

Comment 12 Winfrid Tschiedel 2017-12-19 18:59:00 UTC
Thanks for your quick response.
My 4TB WD disks where all formatted under Windows 10 x64 (GUI).

The reason I am using the 64kb clustersize is very simple -
the disks are archives for pictures (size between 6 mb and 30 mb) 
and video DVD9 or video BD. 

Don't be confused by the different letters in the device name, it is always a disk, which cannot be mounted by linux.

I will not try to fix the NTFS under linux, because after mounting the file system under my windows 10 and going back to linux, the trouble starts again.

If it is a simple make, I should be able to recompile ntfs-3g and test the fix.

parted /dev/sdf
GNU Parted 3.2
/dev/sdf wird verwendet
Willkommen zu GNU Parted! Rufen Sie »help« auf, um eine Liste der verfügbaren
Befehle zu erhalten.
(parted) print                                                            
Modell: WD My Passport 25E2 (scsi)
Festplatte  /dev/sdf:  4001GB
Sektorgröße (logisch/physisch): 512B/4096B
Partitionstabelle: gpt
Disk-Flags: 

Nummer  Anfang  Ende    Größe   Dateisystem  Name         Flags
 1      1049kB  4001GB  4001GB  ntfs         My Passport  msftdata

Comment 13 Winfrid Tschiedel 2017-12-19 18:59:39 UTC
Created attachment 1370129 [details]
requested mft

Comment 14 Winfrid Tschiedel 2017-12-19 19:00:24 UTC
Created attachment 1370130 [details]
requested mftmirr

Comment 15 Jean-Pierre André 2017-12-19 21:07:46 UTC
Hmm. Both mft and mftmirr are wrong...

My fault, the skip parameter represents clusters, not sectors. The correct commands are (after swapping mft and mftmirr) :

dd if=/dev/sdf1 of=mftmirr skip=1 bs=65536 count=1
dd if=/dev/sdf1 of=mft skip=3072 bs=65536 count=1

I will post a patch as soon as I can see the mft and mftmirr. Compiling only implies the usual "./configure ; make", and as root "make install"

Comment 16 Jean-Pierre André 2017-12-19 21:12:12 UTC
Sorry, still wrong, for mft this should be
dd if=/dev/sdf1 of=mft skip=49152 bs=65536 count=1

Comment 17 Tom "spot" Callaway 2017-12-19 21:32:13 UTC
(In reply to Jean-Pierre André from comment #15)

> I will post a patch as soon as I can see the mft and mftmirr. Compiling only
> implies the usual "./configure ; make", and as root "make install"

When the patch is posted, I'll make a temporary package for Fedora so that it is easy to test.

Comment 18 Winfrid Tschiedel 2017-12-20 06:55:49 UTC
Created attachment 1370312 [details]
requested mft (updated version)

Comment 19 Winfrid Tschiedel 2017-12-20 06:56:27 UTC
Created attachment 1370313 [details]
requested mftmirr (updated version)

Comment 20 Jean-Pierre André 2017-12-20 07:29:07 UTC
Created attachment 1370318 [details]
Proposed patch for fixing the $MFT/$MFTMirr mismatch

Thank you, Tom, for your assistance for testing the proposed fix.

Winfrid : better unmount all your ntfs partitions before installing the updated package.

Comment 21 Tom "spot" Callaway 2017-12-20 14:55:28 UTC
This is a scratch build with the patch applied (for Fedora 27):

https://koji.fedoraproject.org/koji/taskinfo?taskID=23808013

Comment 22 Jean-Pierre André 2017-12-21 08:12:28 UTC
A bit of information for using the package prepared by Tom (Fedora 27 on x86_64 being among the distributions you have access to) :

1) download the driver and tools :
wget https://kojipkgs.fedoraproject.org//work/tasks/8014/23808014/ntfsprogs-2017.3.23-3.fc27.1.x86_64.rpm
wget https://kojipkgs.fedoraproject.org//work/tasks/8014/23808014/ntfs-3g-2017.3.23-3.fc27.1.x86_64.rpm

2) install them :
rpm -ivhU ntfs-3g-2017.3.23-3.fc27.1.x86_64.rpm ntfsprogs-2017.3.23-3.fc27.1.x86_64.rpm

Comment 23 Winfrid Tschiedel 2017-12-21 09:00:09 UTC
many thanks, everything seems to work perfect.

I did tests for NTFS on 3 different USB-devices + on a SATA disk.

Comment 24 Jean-Pierre André 2017-12-21 11:09:44 UTC
Tom : you can probably put the patch on the track for an update, though I am noticing :
1) you need not package CVE-2015-3202.patch which is not used any more
2) you need not swap ntfsprogs/boot.c any more
Both issues were fixed in the original ntfs-3g_ntfsprogs-2017.3.23.tgz from Tuxera

Comment 25 Fedora Update System 2018-01-04 19:36:13 UTC
ntfs-3g-2017.3.23-4.fc26 has been submitted as an update to Fedora 26. https://bodhi.fedoraproject.org/updates/FEDORA-2018-5d0b0fde58

Comment 26 Fedora Update System 2018-01-04 19:36:21 UTC
ntfs-3g-2017.3.23-4.fc27 has been submitted as an update to Fedora 27. https://bodhi.fedoraproject.org/updates/FEDORA-2018-321744ed93

Comment 27 Fedora Update System 2018-01-05 11:37:25 UTC
ntfs-3g-2017.3.23-4.fc26 has been pushed to the Fedora 26 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2018-5d0b0fde58

Comment 28 Fedora Update System 2018-01-05 11:59:04 UTC
ntfs-3g-2017.3.23-4.fc27 has been pushed to the Fedora 27 testing repository. If problems still persist, please make note of it in this bug report.
See https://fedoraproject.org/wiki/QA:Updates_Testing for
instructions on how to install test updates.
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2018-321744ed93

Comment 29 Fedora Update System 2018-01-09 16:49:51 UTC
ntfs-3g-2017.3.23-4.fc26 has been pushed to the Fedora 26 stable repository. If problems still persist, please make note of it in this bug report.

Comment 30 Fedora Update System 2018-01-10 02:07:49 UTC
ntfs-3g-2017.3.23-4.fc27 has been pushed to the Fedora 27 stable repository. If problems still persist, please make note of it in this bug report.


Note You need to log in before you can comment on or make changes to this bug.