Bug 1341126
| Summary: | Cumulative xfsrestore does not restore files and folders in a directory which was renamed | ||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Product: | Red Hat Enterprise Linux 7 | Reporter: | Damien Gombault <desintegr> | ||||||||||
| Component: | xfsdump | Assignee: | Eric Sandeen <esandeen> | ||||||||||
| Status: | CLOSED WONTFIX | QA Contact: | Zorro Lang <zlang> | ||||||||||
| Severity: | unspecified | Docs Contact: | |||||||||||
| Priority: | unspecified | ||||||||||||
| Version: | 7.2 | CC: | billodo, desintegr, xzhou | ||||||||||
| Target Milestone: | rc | ||||||||||||
| Target Release: | --- | ||||||||||||
| Hardware: | Unspecified | ||||||||||||
| OS: | Unspecified | ||||||||||||
| Whiteboard: | |||||||||||||
| Fixed In Version: | Doc Type: | If docs needed, set a value | |||||||||||
| Doc Text: | Story Points: | --- | |||||||||||
| Clone Of: | Environment: | ||||||||||||
| Last Closed: | 2020-12-15 07:41:50 UTC | Type: | Bug | ||||||||||
| Regression: | --- | Mount Type: | --- | ||||||||||
| Documentation: | --- | CRM: | |||||||||||
| Verified Versions: | Category: | --- | |||||||||||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||||||||
| Cloudforms Team: | --- | Target Upstream Version: | |||||||||||
| Embargoed: | |||||||||||||
| Attachments: |
|
||||||||||||
|
Description
Damien Gombault
2016-05-31 10:24:38 UTC
In general, it would helpful to provide the error messages after setting LANG=en_US - because your English is far better than my French. :) A couple questions: > Some folders are not restored at the right place. What does that mean? Were they restored to the wrong place, or were they not restored at all? Also, the "Aucun fichier ou dossier de ce type" errors indicate that files could not be moved into a directory hierarchy, presumably because some path component did not exist. The common portions of those paths which failed are: > var/www/html/owncloud/data/F1000buv/files/Dossier personnel et source de partage/E-Administration/ and > var/www/html/owncloud/data/F1000buv/files_versions/Dossier personnel et source de partage/E-Administration Do those paths still exist on the original filesystem? Is it possible that they were removed while the dump was occurring? It seems most likely that the filesystem changed significantly during the dump processes, and this is what led to the errors. This may or may not be useful, but just in case: If you still have the original filesystem available, could you make an xfs_metadump image of it, with the "-o" option if the filenames are not considered sensitive information? Then I can recreate your xfsdump/xfsrestore steps, it might provide a reproducer for the problem, although I think this probably has more to do with the filesystem being modified while the dump was occurring, and so the dump/restore of the metadata image may not be useful. Thanks, -Eric Hi. Thank you for you answer. I'll try to use the C or en_US locale next time :o > Some folders are not restored at the right place. Between the level 0 dump and level 1 dump, the user renamed and moved a lot of subfolders and files in the "E-Administration" folder. Some folders and files stayed at the place they were after the level 0 restoration, they should be moved by the level1 xfsrestore in another folder. Some other files were not restored because the folder owning them was not recreated during the level1 restoration. An exemple : The real content after level 0 + level 1 restoration should be : dir c/ file aa file bb dir d/ file dd After level 0 xfsrestore, the folder contains : dir a/ file aa dir b/ file bb After level 0 + level 1 xfsrestore, the folder contains : dir b/ file bb (stayed at level 0 step, not moved to dir c/) dir c/ file aa (file dd was not restored because folder d was not recreated) > It seems most likely that the filesystem changed significantly during the dump processes, and this is what led to the errors. I think that too, but the user said he does not modified this "E-Administration" folder during the dump. I have restored the lost files with another method (another backup with tar tool). I will try to go deeper with the debugging. > I think that too, but the user said he does not modified this "E-Administration" folder during the dump.
Well, easy enough to check - do those paths I mentioned still exist in the user's filesystem?
(or more specifically, do full paths like
var/www/html/owncloud/data/F1000buv/files/Dossier personnel et source de partage/E-Administration/09 Questionnaires communes solære/Dossiers des collectivités/37003 - Amboise/02-Compte rendu de réunions/
still exist?)
Even if the user didn't modify it, perhaps owncloud moved something?
I will focus on the "Gestion des mots de passe - Keepass/" folder.
level 0 dump content (with xfsrestore -i), folder E-Administration/ : may 7
-> cd E-Administration/
-> ls
1049063261 @ctes/
2332116702 Certificats Solaere/
1365634914 Démonstrations et maquettes/
1677721775 Gestion des mots de passe - Keepass/
860054100 Formations/
2566914232 Plateforme e-collectivités/
1644167337 Communication/
1566551222 Certificat Serveur/
1593835688 Pré projet/
1660944566 Mise en oeuvre/
1627390136 Infos communes région Centre/
1610612923 Conventions/
186255747 Questionnaires communes solaære/
3036693132 Documents administratifs de référence/
371821106 Documentation E-administration/
355157532 Catalogue de service E-Administration/
318899452 Processus/
-> cd "Gestion des mots de passe - Keepass/"
-> ls
1677721776 Mots de passe SOLAERE.kdbx
1677721786 Mots de passe Escolan.kdbx
E-Administration folder content after level 0 restoration :
# ls -i1
2013274150 Catalogue de service E-Administration
268445730 Certificat Serveur
1207971935 Certificats Solaere
16421 Communication
1342191651 Conventions
1074364511 @ctes
1342191647 Démonstrations et maquettes
1879066660 Documentation E-administration
1610682474 Documents administratifs de référence
1610682470 Formations
1476409439 Gestion des mots de passe - Keepass
1207971939 Infos communes région Centre
671100962 Mise en oeuvre
1476409441 Plateforme e-collectivités
536881250 Pré projet
16425 Processus
1476409443 Questionnaires communes solaære
# cd "Gestion des mots de passe - Keepass"/
ls -i1
1514453057 Mots de passe Escolan.kdbx
1514453055 Mots de passe SOLAERE.kdbx
It looks OK, the folder contains the 2 kbdx files.
level 1 content (with xfsrestore -i), folder E-Administration/ : may 29
-> cd E-Administration/
-> ls
3036693132 00 Documents administratifs de référence/
1627390136 10 Infos communes région Centre/
186255747 09 Questionnaires communes solære/
654325809 08 Sécurité/
1644167337 07 Communication/
371821106 06 Documentation E-administration/
860054100 05 Formations/
318899452 04 Processus/
355157532 03 Catalogue de service E-Administration/
1660944566 02 Mise en oeuvre sOlære/
1593835688 01 Pré projet sOlære/
1365634914 Démonstrations et maquettes
-> cd "08 Sécurité"
-> ls
1677721775 08c Gestion des mots de passe - Keepass/
2332116702 08b Certificats Solaere/
1566551222 08a Certificat Serveur/
-> cd "08c Gestion des mots de passe - Keepass"
-> ls
1677795044 Mots de passe SOLAERE.kdbx
1677721786 Mots de passe Escolan.kdbx
Days before level1 dump, the user renamed the "Gestion des mots de passe - Keepass" to "08c Gestion des mots de passe - Keepass".
He moves it under a new "08 Sécurité" folder.
The folder contains the 2 kdbx files (the SOLAERE one has a new inode number, file updated or rewritten I think).
E-Administration folder content after level 1 cumulative restoration :
# ls -l1
1610682474 00 Documents administratifs de référence
536881250 01 Pré projet sOlære
671100962 02 Mise en oeuvre sOlære
2013274150 03 Catalogue de service E-Administration
16425 04 Processus
1610682470 05 Formations
1879066660 06 Documentation E-administration
16421 07 Communication
134875138 08 Sécurité
1476409443 09 Questionnaires communes solære
1207971939 10 Infos communes région Centre
1342191647 Démonstrations et maquettes
1476409439 Gestion des mots de passe - Keepass
# cd "Gestion des mots de passe - Keepass"/
# ls -i1
1514453057 Mots de passe Escolan.kdbx
# cd "../08 Sécurité"
# ls -i1
268445730 08a Certificat Serveur
1207971935 08b Certificats Solaere
The "08 Sécurité" folder is created.
The folder "Gestion des mots de passe - Keepass" was not renamed and not moved under "08 Sécurité" folder.
This is related to this xfsrestore trace log :
xfsrestore: unlink var/www/html/owncloud/data/F1000buv/files/Dossier personnel et source de partage/E-Administration/Gestion des mots de passe - Keepass/Mots de passe SOLAERE.kdbx
xfsrestore: rename dir var/www/html/owncloud/data/F1000buv/files/Dossier personnel et source de partage/E-Administration/Gestion des mots de passe - Keepass to var/www/html/owncloud/data/F1000buv/files/Dossier personnel et source de partage/E-Administration/Gestion des mots de passe - Keepass
xfsrestore: rename dir orphanage/1677721775.4173752252 to var/www/html/owncloud/data/F1000buv/files/Dossier personnel et source de partage/E-Administration/08 Sécurité/08c Gestion des mots de passe - Keepass
xfsrestore: WARNING: unable to rename dir orphanage/1677721775.4173752252 to dir var/www/html/owncloud/data/F1000buv/files/Dossier personnel et source de partage/E-Administration/08 Sécurité/08c Gestion des mots de passe - Keepass: Aucun fichier ou dossier de ce type
xfsrestore tries to rename a non non-renamed orphanage folder.
Why was the "Gestion des mots de passe" folder renamed to the same name instead of orphanage/1677721775.4173752252 ?
Then the file "Mots de passe SOLAERE.kdbx" is not restored because orphanage/1677721775.4173752252 folder is missing ("Gestion des mots de passe" not renamed to orphanage/1677721775.4173752252).
Related log :
xfsrestore: unlink var/www/html/owncloud/data/F1000buv/files/Dossier personnel et source de partage/E-Administration/Gestion des mots de passe - Keepass/Mots de passe SOLAERE.kdbx
xfsrestore: restoring orphanage/1677721775.4173752252/Mots de passe SOLAERE.kdbx (1677795044 23718485)
xfsrestore: restoring regular file ino 1677795044 orphanage/1677721775.4173752252/Mots de passe SOLAERE.kdbx
xfsrestore: WARNING: open of orphanage/1677721775.4173752252/Mots de passe SOLAERE.kdbx failed: Aucun fichier ou dossier de ce type: discarding ino 1677795044
This "E-Administration" directory structure was modified earlier, I can found the same in another previous dump :
Another dump, folder E-Administration/ : may 20
-> cd E-Administration/
-> ls
3036693132 00 Documents administratifs de référence/
1627390136 10 Infos communes région Centre/
186255747 09 Questionnaires communes solære/
654325809 08 Sécurité/
1644167337 07 Communication/
371821106 06 Documentation E-administration/
860054100 05 Formations/
318899452 04 Processus/
355157532 03 Catalogue de service E-Administration/
1660944566 02 Mise en oeuvre sOlære/
1593835688 01 Pré projet sOlære/
1365634914 Démonstrations et maquettes
-> cd "08 Sécurité"
-> ls
1677721775 08c Gestion des mots de passe - Keepass/
2332116702 08b Certificats Solaere/
1566551222 08a Certificat Serveur
-> ls
1677721787 Mots de passe SOLAERE.kdbx
1677721786 Mots de passe Escolan.kdbx
I think this folders and files were not moved during the may 29 level 1 dump.
I also checked the owncloud logs during dump time, there was no write activity.
What could cause this ?
Hi.
I am able to reproduce the problem with a minimal test case.
Here is the test case :
Create a new XFS filesystem, mount it as "/mnt/test"
Create a directory "dira/"
Create a file "filea" en directory "dira/"
You should get :
.
└── dira
└── filea
Make a level 0 dump :
xfsdump -l 0 -f /root/test0.dump /mnt/test
Rename "dira/" to "dirA/"
Create a directory "dirb/" in "dirA/"
Create a file "fileb" in "dirb/"
You should get :
.
└── dirA
├── dirb
│ └── fileb
└── filea
Make a level 1 dump :
xfsdump -l 1 -f /root/test1.dump /mnt/test
Create a new XFS filesystem, mount it as "/mnt/restore"
Restore (cumulative mode) level 0 dump :
xfsrestore -r -f /root/test0.dump /mnt/restore/
Restore (cumulative mode) level 1 dump :
xfsrestore -r -f /root/test1.dump /mnt/restore/
You get a WARNING :
xfsrestore: WARNING: open of dirA/dirb/fileb failed: Aucun fichier ou dossier de ce type: discarding ino 526337 (No such file or directory in english)
Directory "dirb" and fileb is not restored :
.
├── dirA
│ └── filea
└── xfsrestorehousekeepingdir
├── dirattr
├── dirextattr
├── namreg
├── state
└── tree
I attached test0.dump (level 0 dump), test1.dump (level 1 dump), debug0.txt (level 0 debug trace), debug1.txt (level 1 debug trace).
Created attachment 1170674 [details]
Level 0 dump
Created attachment 1170675 [details]
Level 1 dump
Created attachment 1170676 [details]
Debug Trace Level 0 Dump
Created attachment 1170677 [details]
Debug Trace Level 1 Dump
Great, thanks for the testcase. I'll try to dig into this one further. Thank you for your help :) I can also reproduce this on Fedora 24 (with recent XFS tools) : xfsprogs-4.5.0-1.fc24.x86_64 xfsdump-3.1.6-2.fc24.x86_64 kernel-4.5.7-300.fc24.x86_64 I've verified the problem on rhel7 and latest xfsdump v3.16 using reproducer in comment 6. Still looking at it... -Bill Dave Chinner have posted a patch on XFS mailing list for this bug : http://oss.sgi.com/pipermail/xfs/2016-June/049845.html I will test it tomorrow. Using reproducer (comment 6), I tested Dave's patch (comment 14) on a RHEL7 target and verified that it fixes the issue. Dave's patch fixes the simple reproducer.
I have tested the patch on my real data, it fixes some but not all warnings.
Here is another testcase which fails (with the patch) :
mkdir dira
mkdir dira/dirc
touch dira/dirc/filea
mkdir dirb
.
├── dira
│ └── dirc
│ └── filea
└── dirb
Make a level 0 dump.
mv dirb dira/dirB
mv dira/dirc/ dira/dirB/dirC
touch dira/dirB/dirC/fileb
.
└── dira
└── dirB
└── dirC
├── filea
└── fileb
Make a level 1 dump.
Restore level 0 then level 1 dump, you will get :
xfsrestore: directory post-processing
xfsrestore: WARNING: unable to set secure extended attribute for dira/dirB: No such file or directory (2)
xfsrestore: restoring non-directory files
xfsrestore: WARNING: open of dira/dirB/dirC/fileb failed: No such file or directory: discarding ino 524386
xfsrestore: WARNING: unable to set secure extended attribute for dira/dirB/dirC/fileb: No such file or directory (2)
xfsrestore: WARNING: path_to_handle of dira/dirB/dirC failed:No such file or directory
xfsrestore: could not set access and modification times of dira/dirB/dirC: No such file or directory
xfsrestore: chown (uid=0, gid=0) dira/dirB/dirC failed: No such file or directory
xfsrestore: chmod dira/dirB/dirC failed: No such file or directory
xfsrestore: WARNING: attempt to set extended attributes (xflags 0x80000000, extsize = 0x0, projid = 0x0) of dira/dirB/dirC failed: Bad file descriptor
xfsrestore: WARNING: path_to_handle of dira/dirB failed:No such file or directory
xfsrestore: could not set access and modification times of dira/dirB: No such file or directory
xfsrestore: chown (uid=0, gid=0) dira/dirB failed: No such file or directory
xfsrestore: chmod dira/dirB failed: No such file or directory
xfsrestore: WARNING: attempt to set extended attributes (xflags 0x80000000, extsize = 0x0, projid = 0x0) of dira/dirB failed: Bad file descriptor
xfsrestore: WARNING: unable to rmdir /mnt/test//orphanage: Directory not empty
Directory dirB is placed in orphanage folder and fileb is not restored :
.
├── dira
├── orphanage
│ └── 1069152.734839917
│ └── dirC
│ └── filea
└── xfsrestorehousekeepingdir
├── dirattr
├── dirextattr
├── namreg
├── state
└── tree
Unfortunately this bug surfaced too late to consider for the next RHEL7 release; I will move it to the subsequent release in the hopes that we can address it then. Still reproducible on upstream kernel 5.0-rc8 and xfsdump. A report to xfs upstream may help. Still reproducible on upstream kernel 5.7-rc4+ and xfsdump: version 3.1.8 (dump format 3.0) After evaluating this issue, there are no plans to address it further or fix it in an upcoming release. Therefore, it is being closed. If plans change such that this issue will be fixed in an upcoming release, then the bug can be reopened. |