Created attachment 1431856 [details] $MFTMirr Description of problem: fedora cannot mount NTFS filesystem with cluster size bigger 64kb. With the creator's update 1709 MS did some major changes in the NTFS filesystem. Beside the already fixed problem (https://bugzilla.redhat.com/show_bug.cgi?id=1527231) the cluster size was extended up to 2mb. Version-Release number of selected component (if applicable): ntfs-3g-2017.3.23-4.fc27.x86_64 ntfsprogs-2017.3.23-4.fc27.x86_64 How reproducible: Have a fedora 27 with actual ntfs installed. Format a HDD on Windows 10 x64 creator's update fall 2017 or higher with a cluster size bigger than 64kb My test was with 2mb Steps to Reproduce: 1. try to mount the filesystem described above 2. 3. Actual results: mount -t ntfs /dev/sdi1 /mnt1 Unexpected sectors per cluster value (244). Failed to mount '/dev/sdi1': Invalid argument The device '/dev/sdi1' doesn't seem to have a valid NTFS. Maybe the wrong device is used? Or the whole disk instead of a partition (e.g. /dev/sda, not /dev/sda1)? Or the other way around? Expected results: clean mount Additional info: parted /dev/sdi GNU Parted 3.2 /dev/sdi wird verwendet Willkommen zu GNU Parted! Rufen Sie »help« auf, um eine Liste der verfügbaren Befehle zu erhalten. (parted) print Modell: WD Elements 25A1 (scsi) Festplatte /dev/sdi: 4001GB Sektorgröße (logisch/physisch): 512B/4096B Partitionstabelle: gpt Disk-Flags: Nummer Anfang Ende Größe Dateisystem Name Flags 1 1049kB 4001GB 4001GB ntfs Elements msftdata (parted) quit [root@p900 ~]# dd if=/dev/sdi1 of=mftmirr skip=1 bs=2M count=1 1+0 Datensätze ein 1+0 Datensätze aus 2097152 Bytes (2,1 MB, 2,0 MiB) kopiert, 0,833257 s, 2,5 MB/s [root@p900 ~]# gzip mftmirr hope I got the correct $MFTMirr please tell me if you need something else.
Created attachment 1431857 [details] format menu for NTFS (Windows 10 Creator's Update 1709)
I have a patch available to enable cluster size up to 524K, only with 4K sectors, but currently the number of sectors per cluster is stored as a byte in the boot sector and cannot be greater than 128. Microsoft has had to make some hack to go beyond that, and I have no appropriate disk to check. If you were able to use 2MB clusters, please post your format data, which you should be able to collect by : dd if=/dev/PARTITION count=1 | od -t x1 | head -n 16 Where PARTITION is the relevant partition (apparently sdi1) Note : if this is a disk with no significant data on it, could you also please post the data for 1MB and 524K clusters.
Hi Jean-Pierre, don't be confused, after reboot the disk is now /dev/sdh Here are the requested data : parted /dev/sdh GNU Parted 3.2 /dev/sdh wird verwendet Willkommen zu GNU Parted! Rufen Sie »help« auf, um eine Liste der verfügbaren Befehle zu erhalten. (parted) print Modell: WD Elements 25A1 (scsi) Festplatte /dev/sdh: 4001GB Sektorgröße (logisch/physisch): 512B/4096B Partitionstabelle: gpt Disk-Flags: Nummer Anfang Ende Größe Dateisystem Name Flags 1 1049kB 4001GB 4001GB ntfs Elements msftdata (parted) quit [root@p900 ~]# dd if=/dev/sdh1 count=1 | od -t x1 | head -n 16 1+0 Datensätze ein 1+0 Datensätze aus 512 Bytes kopiert, 0,000450209 s, 1,1 MB/s 0000000 eb 52 90 4e 54 46 53 20 20 20 20 00 02 f4 00 00 0000020 00 00 00 00 00 f8 00 00 3f 00 ff 00 00 08 00 00 0000040 00 00 00 00 80 00 80 00 ff a7 bf d1 01 00 00 00 0000060 00 06 00 00 00 00 00 00 01 00 00 00 00 00 00 00 0000100 f6 00 00 00 f4 00 00 00 3d 5a 31 e2 60 31 e2 c2 0000120 00 00 00 00 fa 33 c0 8e d0 bc 00 7c fb 68 c0 07 0000140 1f 1e 68 66 00 cb 88 16 0e 00 66 81 3e 03 00 4e 0000160 54 46 53 75 15 b4 41 bb aa 55 cd 13 72 0c 81 fb 0000200 55 aa 75 06 f7 c1 01 00 75 03 e9 dd 00 1e 83 ec 0000220 18 68 1a 00 b4 48 8a 16 0e 00 8b f4 16 1f cd 13 0000240 9f 83 c4 18 9e 58 1f 72 e1 3b 06 0b 00 75 db a3 0000260 0f 00 c1 2e 0f 00 04 1e 5a 33 db b9 00 20 2b c8 0000300 66 ff 06 11 00 03 16 0f 00 8e c2 ff 06 16 00 e8 0000320 4b 00 2b c8 77 ef b8 00 bb cd 1a 66 23 c0 75 2d 0000340 66 81 fb 54 43 50 41 75 24 81 f9 02 01 72 1e 16 0000360 68 07 bb 16 68 52 11 16 68 09 00 66 53 66 53 66 [root@p900 ~]#
Created attachment 1432411 [details] Patch for allowing big clusters This proposed patch is based on what I can tell from the data you posted. Though I have no real device to check it, I can apply chkdsk from Windows 10 on a device image rebuilt from your data, and it finds no errors. I recommend you first mount your device read-only and make sure you can read existing files. If this is fine, try creating a few files, then run chkdsk on Windows to check for any errors. From the tests I could make, I am reasonably confident this will run fine.
Hi Jean-Pierre, could you please prepare the patch in a different format - rpm or just the modules I need. Thank you, Winfrid
Here's a test build with that patch applied: https://koji.fedoraproject.org/koji/taskinfo?taskID=26839862
@winfrid : with the rpms provided by Tom, you probably only need to download two of them : wget https://kojipkgs.fedoraproject.org//work/tasks/9863/26839863/ntfs-3g-2017.3.23-5.fc27.x86_64.rpm wget https://kojipkgs.fedoraproject.org//work/tasks/9863/26839863/ntfsprogs-2017.3.23-5.fc27.x86_64.rpm and install them : rpm -ivh --upgrade ntfs-3g-2017.3.23-5.fc27.x86_64.rpm ntfsprogs-2017.3.23-5.fc27.x86_64.rpm @Tom : thank you
Thanks to everybody - the problem seems to be solved. I did the following tests on fedora : mount NTFS read-only mount NTFS read-write copy a directory (approx. 32 gb, 17 files) to the file system chkdsk on Windows 10 : chkdsk /f p: Der Typ des Dateisystems ist NTFS. Die Volumebezeichnung lautet WD Passport 4TB. Phase 1: Die Basisdatei-Systemstruktur wird untersucht... 12288 Datensätze verarbeitet. Dateiüberprüfung beendet. 0 große Datensätze verarbeitet. 0 ungültige Datensätze verarbeitet. Phase 2: Die Dateinamenverknüpfung wird untersucht... 24 Analysedatensätze verarbeitet. 13670 Indexeinträge verarbeitet. Indexüberprüfung beendet. 0 nicht indizierte Dateien überprüft. 0 nicht indizierte Dateien wiederhergestellt. 24 Analysedatensätze verarbeitet. Phase 3: Sicherheitsbeschreibungen werden untersucht... Überprüfung der Sicherheitsbeschreibungen beendet. 691 Datendateien verarbeitet. Dateisystem wurde überprüft, keine Probleme festgestellt. Keine weiteren Aktionen erforderlich. 3815412 MB Speicherplatz auf dem Datenträger insgesamt. 3702006 MB in 11505 Dateien 569344 KB in 693 Indizes 0 KB in fehlerhaften Sektoren 93183 KB vom System benutzt 65536 KB von der Protokolldatei belegt 115466240 KB auf dem Datenträger verfügbar 2097152 Bytes in jeder Zuordnungseinheit 1907706 Zuordnungseinheiten auf dem Datenträger insgesamt 56380 Zuordnungseinheiten auf dem Datenträger verfügbar Just a remark : copying the directory (both disks connected with USB3) was quite slow : it took 77 minutes to copy the 17 files - just to compare - it took on Windows 10 (cygwin64) 11m04s to compare the directories.
> the problem seems to be solved Good news. > copying the directory (both disks connected with USB3) was quite slow What you probably experience is a mismatch between the cluster size and the fuse buffer size. Each time ntfs-3g receives a buffer from fuse, a read-modify-write cycle has to be done, and the performance of doing that is related to the efficiency of the write cache and the ability to modify a cluster before it is actually written to disk. You should first try mounting with option "big_writes" so that fuse sends bigger chunks (up to 131K) to ntfs-3g, but this may now be the default, and the option will have no effect. Even so, the chunks size will be limited to the size of the buffers of the application doing the write(), and in the case of "cp" this is 65K. With "dd" you can define the chunk size, but you will still be limited to 131K. For bigger values you have to recompile the kernel (or at least the fuse kernel module).
Created attachment 1433620 [details] Update to the big clusters patch This to fix a bug in ntfsresize in the initial patch. Also, the patch has been rebased on ntfs-3g-2017.3.23-4.fc27
ntfs-3g-2017.3.23-6.el7 has been submitted as an update to Fedora EPEL 7. https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2018-8f414877c8
ntfs-3g-2017.3.23-6.fc26 has been submitted as an update to Fedora 26. https://bodhi.fedoraproject.org/updates/FEDORA-2018-eb2435dd6a
ntfs-3g-2017.3.23-6.el6 has been submitted as an update to Fedora EPEL 6. https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2018-fb6db31b8a
ntfs-3g-2017.3.23-6.fc27 has been submitted as an update to Fedora 27. https://bodhi.fedoraproject.org/updates/FEDORA-2018-419559236b
ntfs-3g-2017.3.23-6.fc28 has been submitted as an update to Fedora 28. https://bodhi.fedoraproject.org/updates/FEDORA-2018-81f69b5fa4
ntfs-3g-2017.3.23-6.fc27 has been pushed to the Fedora 27 testing repository. If problems still persist, please make note of it in this bug report. See https://fedoraproject.org/wiki/QA:Updates_Testing for instructions on how to install test updates. You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2018-419559236b
ntfs-3g-2017.3.23-6.el6 has been pushed to the Fedora EPEL 6 testing repository. If problems still persist, please make note of it in this bug report. See https://fedoraproject.org/wiki/QA:Updates_Testing for instructions on how to install test updates. You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2018-fb6db31b8a
ntfs-3g-2017.3.23-6.el7 has been pushed to the Fedora EPEL 7 testing repository. If problems still persist, please make note of it in this bug report. See https://fedoraproject.org/wiki/QA:Updates_Testing for instructions on how to install test updates. You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2018-8f414877c8
ntfs-3g-2017.3.23-6.fc26 has been pushed to the Fedora 26 testing repository. If problems still persist, please make note of it in this bug report. See https://fedoraproject.org/wiki/QA:Updates_Testing for instructions on how to install test updates. You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2018-eb2435dd6a
ntfs-3g-2017.3.23-6.fc28 has been pushed to the Fedora 28 testing repository. If problems still persist, please make note of it in this bug report. See https://fedoraproject.org/wiki/QA:Updates_Testing for instructions on how to install test updates. You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2018-81f69b5fa4
(In reply to Fedora Update System from comment #20) > ntfs-3g-2017.3.23-6.fc28 has been pushed to the Fedora 28 testing > repository. If problems still persist, please make note of it in this bug > report. > See https://fedoraproject.org/wiki/QA:Updates_Testing for > instructions on how to install test updates. > You can provide feedback for this update here: > https://bodhi.fedoraproject.org/updates/FEDORA-2018-81f69b5fa4 I have installed ntfs-3g-2017.3.23-6.fc28.x86_64 and ntfsprogs-2017.3.23-6.fc28.x86_64 The problem with NTFS file systems from Windows 10 x64 Creators Update Fall 2017 and cluster size 64kb is solved - but a file system with 2048kb cluster size cannot be mounted : parted /dev/sdj GNU Parted 3.2 /dev/sdj wird verwendet Willkommen zu GNU Parted! Rufen Sie »help« auf, um eine Liste der verfügbaren Befehle zu erhalten. (parted) print Modell: WD Elements 25A1 (scsi) Festplatte /dev/sdj: 4001GB Sektorgröße (logisch/physisch): 512B/4096B Partitionstabelle: gpt Disk-Flags: Nummer Anfang Ende Größe Dateisystem Name Flags 1 1049kB 4001GB 4001GB ntfs Elements msftdata (parted) quit [root@p900 ~]# mount -o ro /dev/sdj1 /mnt2 mount: /mnt2: Falscher Dateisystemtyp, ungültige Optionen, der Superblock von /dev/sdj1 ist beschädigt, fehlende Kodierungsseite oder ein anderer Fehler.
I do not think this behavior is to be associated with ntfs-3g. This is probably in some code which tries to guess the file system type to be mounted. If you did not reboot after the install, this may be due to some program linked to the former libntfs-3g before updating it. When I try mounting a file system rebuilt from the data you posted, without telling this is ntfs, I get an error similar to yours : [root@dimension ntfslowprof]# mount -o ro /shared/ntfs/images/big-rebuilt.ntfs disk mount: wrong fs type, bad option, bad superblock on /dev/loop0, missing codepage or helper program, or other error But if I state the file system type, it mounts correctly as expected (using the rpm from update-testing for fc28). [root@dimension ntfslowprof]# mount -t ntfs -o ro /shared/ntfs/images/big-rebuilt.ntfs disk Note : if the patch for big clusters had not been applied, you would have got a different error : Unexpected sectors per cluster value (244). Failed to mount '/dev/loop0': Invalid argument The device '/dev/loop0' doesn't seem to have a valid NTFS. I suggest rebooting and retrying to mount. If it fails, retry with stating "-t ntfs". If it still fails, please post the outputs of : md5sum /usr/lib64/libntfs-3g.so.88.0.0 ntfsinfo -fm /dev/PARTITION HTH
I was not quite sure about the reboot - the simple command "mount -t ntfs -o ro /dev/sdi1 /mnt1" works but not mount -t ntfs -o ro /shared/ntfs/images/big-rebuilt.ntfs /dev/sdj1 mount: /dev/sdj1: Spezialgerät /shared/ntfs/images/big-rebuilt.ntfs ist nicht vorhanden. parted /dev/sdj GNU Parted 3.2 /dev/sdj wird verwendet Willkommen zu GNU Parted! Rufen Sie »help« auf, um eine Liste der verfügbaren Befehle zu erhalten. (parted) print Modell: WD Elements 25A1 (scsi) Festplatte /dev/sdj: 4001GB Sektorgröße (logisch/physisch): 512B/4096B Partitionstabelle: gpt Disk-Flags: Nummer Anfang Ende Größe Dateisystem Name Flags 1 1049kB 4001GB 4001GB ntfs Elements msftdata (parted) quit [root@p900 ~]# md5sum /usr/lib64/libntfs-3g.so.88.0.0 d4daeca5790d47dd43620934965c46d7 /usr/lib64/libntfs-3g.so.88.0.0 [root@p900 ~]# ntfsinfo -fm /dev/sdj1 Volume Information Name of device: /dev/sdj1 Device state: 11 Volume Name: DVD+BD (Sicherungskopie) Volume State: 91 Volume Flags: 0x0080 Volume Version: 3.1 Sector Size: 512 Cluster Size: 2097152 Index Block Size: 4096 Volume Size in Clusters: 1907706 MFT Information MFT Record Size: 1024 MFT Zone Multiplier: 0 MFT Data Position: 24 MFT Zone Start: 1536 MFT Zone End: 239999 MFT Zone Position: 1536 Current Position in First Data Zone: 239999 Current Position in Second Data Zone: 0 Allocated clusters 7 (0,0%) LCN of Data Attribute for FILE_MFT: 1536 FILE_MFTMirr Size: 2048 LCN of Data Attribute for File_MFTMirr: 1 Size of Attribute Definition Table: 2560 Number of Attached Extent Inodes: 0 FILE_Bitmap Information FILE_Bitmap MFT Record Number: 6 State of FILE_Bitmap Inode: 80 Length of Attribute List: 0 Number of Attached Extent Inodes: 0 FILE_Bitmap Data Attribute Information Decompressed Runlist: not done yet Base Inode: 6 Attribute Types: not done yet Attribute Name Length: 0 Attribute State: 3 Attribute Allocated Size: 2097152 Attribute Data Size: 238464 Attribute Initialized Size: 238464 Attribute Compressed Size: 0 Compression Block Size: 0 Compression Block Size Bits: 0 Compression Block Clusters: 0 Free Clusters: 105094 (5,5%)
(In reply to Fedora Update System from comment #18) > ntfs-3g-2017.3.23-6.el7 has been pushed to the Fedora EPEL 7 testing > repository. If problems still persist, please make note of it in this bug > report. > See https://fedoraproject.org/wiki/QA:Updates_Testing for > instructions on how to install test updates. > You can provide feedback for this update here: > https://bodhi.fedoraproject.org/updates/FEDORA-EPEL-2018-8f414877c8 On Centos 7 with EPEL 7 even the mount for cluster size 64kb fails : mount -t ntfs -o ro /dev/sdf1 /mnt1 Unexpected sectors per cluster value (244). Failed to mount '/dev/sdf1': Das Argument ist ungültig The device '/dev/sdf1' doesn't seem to have a valid NTFS. Maybe the wrong device is used? Or the whole disk instead of a partition (e.g. /dev/sda, not /dev/sda1)? Or the other way around? parted /dev/sdf GNU Parted 3.1 Verwende /dev/sdf Willkommen zu GNU Parted! Geben Sie 'help' ein, um eine Liste der verfügbaren Kommados zu erhalten. (parted) print Modell: WD My Passport 25E2 (scsi) Festplatte /dev/sdf: 4001GB Sektorgröße (logisch/physisch): 512B/4096B Partitionstabelle: gpt Disk Flags: Nummer Anfang Ende Größe Dateisystem Name Flags 1 1049kB 4001GB 4001GB ntfs My Passport md5sum /usr/lib64/libntfs-3g.so.88.0.0 45a914ef3ae9e8bb9390c319b7206a84 /usr/lib64/libntfs-3g.so.88.0.0 [root@p900 ~]# ntfsinfo -fm /dev/sdf1 Unexpected sectors per cluster value (244). Failed to mount '/dev/sdf1': Das Argument ist ungültig The device '/dev/sdf1' doesn't have a valid NTFS. Maybe you selected the wrong device? Or the whole disk instead of a partition (e.g. /dev/hda, not /dev/hda1)? Or the other way around? Failed to open '/dev/sdf1'. uname -a Linux p900.at.home 3.10.0-862.3.2.el7.x86_64 #1 SMP Mon May 21 23:36:36 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux PS.: This time I did not forget the reboot.
> the simple command "mount -t ntfs -o ro /dev/sdi1 /mnt1" works Ok. So the patch is effective. > but not mount -t ntfs -o ro /shared/ntfs/images/big-rebuilt.ntfs /dev/sdj1 Of course it fails (/shared/ntfs/images is only meaningful on my own computer...) The expected test was like the one you posted in Comment 21, but after being sure you rebooted after the update. "mount -o ro /dev/sdi1 /mnt1" If it still fails, I have no idea why. I see nothing wrong in your ntfsinfo data. Did you observe this behavior with the former test build ? (No significant change since).
> On Centos 7 with EPEL 7 even the mount for cluster size 64kb fails : > > md5sum /usr/lib64/libntfs-3g.so.88.0.0 > 45a914ef3ae9e8bb9390c319b7206a84 /usr/lib64/libntfs-3g.so.88.0.0 You have probably not installed the update, From ntfs-3g-2017.3.23-6.el7.x86_64.rpm I get : [linux@dimension lib64]$ md5 libntfs-3g.so.88.0.0 8630991addd7b0257a8ba26e790977ca libntfs-3g.so.88.0.0
About mount failing when the file system type is not stated : The mount manual has : "If no -t option is given, or if the auto type is specified, mount will try to guess the desired type. Mount uses the blkid library for guessing the filesystem type; etc." The blkid manual has : "Note that blkid reads information directly from devices" When "stracing" blkid, I do not see any calls to libntfs-3g, so blkid has its own algorithm for determining the file system type, and it probably needs updating.
Ok, I have located the issue in blkid, namely in "libblkid/src/superblocks/ntfs.c" there is the traditional limitation on the number of sectors per cluster. I will post a patch to the blkid maintainer.
ntfs-3g-2017.3.23-6.fc28 has been pushed to the Fedora 28 stable repository. If problems still persist, please make note of it in this bug report.
(In reply to Jean-Pierre André from comment #26) > > On Centos 7 with EPEL 7 even the mount for cluster size 64kb fails : > > > > md5sum /usr/lib64/libntfs-3g.so.88.0.0 > > 45a914ef3ae9e8bb9390c319b7206a84 /usr/lib64/libntfs-3g.so.88.0.0 > > You have probably not installed the update, > > From ntfs-3g-2017.3.23-6.el7.x86_64.rpm I get : > > [linux@dimension lib64]$ md5 lintfs-3gbntfs-3g.so.88.0.0 > 8630991addd7b0257a8ba26e790977ca libntfs-3g.so.88.0.0 Yes, you are right, I installed now ntfs-3g from testing and everything works like on fedora 28
ntfs-3g-2017.3.23-6.el6 has been pushed to the Fedora EPEL 6 stable repository. If problems still persist, please make note of it in this bug report.
ntfs-3g-2017.3.23-6.el7 has been pushed to the Fedora EPEL 7 stable repository. If problems still persist, please make note of it in this bug report.
ntfs-3g-2017.3.23-6.fc27 has been pushed to the Fedora 27 stable repository. If problems still persist, please make note of it in this bug report.