Bug 1381502

Summary: fstrim doesn't remove content of deleted files on XFS
Product: [Fedora] Fedora Reporter: Piotr Muchowski <piotr.muchowski.ext>
Component: kernelAssignee: Kernel Maintainer List <kernel-maint>
Status: CLOSED NOTABUG QA Contact: Fedora Extras Quality Assurance <extras-qa>
Severity: medium Docs Contact:
Priority: unspecified    
Version: 25CC: harshula, piotr.muchowski.ext, rjones, security-response-team, wmealing
Target Milestone: ---Keywords: Security
Target Release: ---   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2016-10-18 00:49:45 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:

Description Piotr Muchowski 2016-10-04 10:15:38 UTC
Summary: virt-sparsify doesn't remove content of deleted files on XFS

Description of problem: I'm using commands below to prepare customized (hardened) versions of RedHat / CentOS cloud images using commands below.

#!/bin/bash

$ virt-sysprep -v --delete /etc/openldap/certs/password --truncate /etc/chrony.keys  --selinux-relabel -a "${1}"

$ virt-sparsify -v "${1}" "cloudimage-${1}"




Version-Release number of selected component (if applicable): virt-sysprep and virt-sparsify 1.32.2 ( from standard repositories in Ubuntu 16.04.1 LTS )


How reproducible: Always


Steps to Reproduce:
1. A customized virtual instance (kvm qcow2 type) is shutted down and a snapshot is prepared.
2. bash script with commands above is ran on snapshot.
3. grepping the produced cloudimage-*qcow2 files using the ntp key string from /etc/chrony.keys says these images still CONTAIN this string.

Actual results: some sensitive information on cloud image are not removed, and content can be extracted from image. ( I suppose this may affect also the RedHat certificates needed for downloading updates )


Expected results: the garbage sensitive data on disk image should be completely removed from customized cloud image.


Additional info:
filesystem used by cloud images is XFS
tools runs on Ubuntu system

( The real keys has been removed )

ubuntu@jumpbox:~$ grep -c -U 'AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA' * ; grep -c -U 'BBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBBB' * ; grep -c -U 'CCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCC' *
CentOS-6-HARDENED-v3-CLEAN_2016-09-26_153751.qcow2:0
CentOS-7-HARDENED-v3-CLEAN_2016-09-26_153446.qcow2:1
cloudimage-CentOS-6-HARDENED-v3-CLEAN_2016-09-26_153751.qcow2:0
cloudimage-CentOS-7-HARDENED-v3-CLEAN_2016-09-26_153446.qcow2:1
image-prepare.sh:0
mount-vms-101-106.sh:0
CentOS-6-HARDENED-v3-CLEAN_2016-09-26_153751.qcow2:1
CentOS-7-HARDENED-v3-CLEAN_2016-09-26_153446.qcow2:0
cloudimage-CentOS-6-HARDENED-v3-CLEAN_2016-09-26_153751.qcow2:1
cloudimage-CentOS-7-HARDENED-v3-CLEAN_2016-09-26_153446.qcow2:0
image-prepare.sh:0
mount-vms-101-106.sh:0
CentOS-6-HARDENED-v3-CLEAN_2016-09-26_153751.qcow2:0
CentOS-7-HARDENED-v3-CLEAN_2016-09-26_153446.qcow2:0
cloudimage-CentOS-6-HARDENED-v3-CLEAN_2016-09-26_153751.qcow2:0
cloudimage-CentOS-7-HARDENED-v3-CLEAN_2016-09-26_153446.qcow2:0
image-prepare.sh:0
mount-vms-101-106.sh:0
ubuntu@jumpbox:~$ ls -l
total 6371744
-rw-r--r-- 1 ubuntu ubuntu 1456406528 Oct  4 09:53 CentOS-6-HARDENED-v3-CLEAN_2016-09-26_153751.qcow2
-rw-r--r-- 1 ubuntu ubuntu 2518614016 Oct  4 10:01 CentOS-7-HARDENED-v3-CLEAN_2016-09-26_153446.qcow2
-rw-r--r-- 1 ubuntu ubuntu 1105985536 Oct  4 10:00 cloudimage-CentOS-6-HARDENED-v3-CLEAN_2016-09-26_153751.qcow2
-rw-r--r-- 1 ubuntu ubuntu 1443758080 Oct  4 10:11 cloudimage-CentOS-7-HARDENED-v3-CLEAN_2016-09-26_153446.qcow2
-rwxr-xr-x 1 ubuntu ubuntu        167 Oct  4 09:45 image-prepare.sh
-rwxr-xr-x 1 ubuntu ubuntu        317 Sep 20 14:35 mount-vms-101-106.sh
ubuntu@jumpbox:~$

Comment 1 Richard W.M. Jones 2016-10-04 10:22:23 UTC
I'll just note if you want to securely delete a file, you should
use the virt-sysprep --scrub option.

For this bug, we just run fstrim, so if the file is still there
it's because either the kernel or qemu is not removing it.
Therefore reassigning to the kernel.

Comment 2 Harshula Jayasuriya 2016-10-12 00:42:46 UTC
Hi Piotr,

Can you please clarify whether you can still see the actual file? Or do you only see the known content of the file in the raw filesystem image?

Thanks,
Harshula

Comment 3 Piotr Muchowski 2016-10-12 08:37:52 UTC
Hi Harshula,

The deleted files are removed and don't appear in operating system.
Their content however is still on virtual hard disk image file and therefore is recoverable.

I'm preparing a process to automatically generate hardened cloud images and one of the tasks is to completely erase sensitive data from image to prevent end users "trash diving" and trying to recover something from raw sectors. 

Thanks, Piotr Muchowski

Comment 4 Harshula Jayasuriya 2016-10-12 13:53:16 UTC
Hi Piotr,

1) Thanks for the clarification confirming that the deleted files are NOT still there.

2) Did you try using "virt-sysprep --scrub"? Did that ensure that the known content of the file is NOT present in the raw filesystem image?

Thanks,
Harshula

Comment 5 Piotr Muchowski 2016-10-13 13:51:56 UTC
Hi Harshula,

I have tested this today on CentOS 7 cloud version, and not on Fedora yet.
Anyway it seems this is not kernel or qemu issue.

I tested with scrub option and yes, the scrub removes the content as expected. But because there is no global virt-sysprep's "scrubmode=always-on" option, the "--scrub" can be used only with files provided by command line, the other virt-sysprep cleanup modules do ordinary file deletions and files therefore are recoverable. 

Virt-sparsify could remediate here but it doesn't recognize areas occupied by deleted files as unused and doesn't trim it. Perhaps it rather works on disk sectors instead specific bytes and maybe the affected sector cannot be trimmed as it contains other data as well.

virt-sysprep man page says i can use virt-sparsify to securely remove data. I thought it works reliably and discovered this issue only by accident.

Regards, Piotr

Comment 6 Richard W.M. Jones 2016-10-13 14:33:53 UTC
virt-sysprep man page actually says:

       Although virt-sysprep removes some sensitive information from the
       guest, it does not pretend to remove all of it.  You should examine the
       "OPERATIONS" above and the guest afterwards.

       Sensitive files are simply removed.  The data they contained may still
       exist on the disk, easily recovered with a hex editor or undelete tool.
       The --scrub option can be used to scrub files instead of just deleting
       them.  virt-sparsify(1) is another way to remove this content.  See
       also the scrub(1) command to get rid of deleted content in directory
       entries and inodes.

virt-sysprep --inplace simply performs an fstrim operation, so
it relies on the kernel and qemu doing the right thing for fstrim.

fstrim has its own limitations of course.  It only operates on blocks
of a particular size (blocksize depends on many factors but can be
read from /sys/block/*/queue/discard_*), and it is possible that file
boundaries might not align with block boundaries.

You can also look at the guestfish 'scrub-device' command, which is
also an interface to the scrub(1) command, but should write patterns
all over the free space in the filesystem.  It'll probably be quite slow.

Comment 10 Piotr Muchowski 2016-10-14 07:59:27 UTC
OK, I will do research on this topic later, thanks for the advices.
It seems that such advanced image sanitization needs just more steps than ordinary cloud image preparation (at the cost of increased execution time).

Comment 12 Wade Mealing 2016-10-18 00:49:45 UTC
Gday Poitr,

Thanks for the report, at this time we don't consider this a security issue.  If you have any further information please don't hesitate to add a comment and re-open this bug and we can continue this conversation.

Thanks,

Wade Mealing
Red Hat Product Security