Bug 2109742

Description of problem:

Many disconnected/air-gap Satellite users also have a strict organization policies that all files being transferred to the air-gapped network pass a virus/malware/security scan. 

When breaking up a hammer content-export with the --chunk-size option for data transfer, the virus scanner accepts only the first file and the rest fail. I know why.

The gzipped tarball is piped into the split command to create the chunks. Every chunk ( or split ) after the first one does not have file header data to indicate the type of file, so the virus scanner sees it as random data.


I checked out the Pulpcore source code and did some tests with tar to confirm my suspicions;

Seen in line 261 onwards of pulpcore/app/tasks/export.py in Pulpcore source code ( https://github.com/pulp/pulpcore );

        if the_export.validated_chunk_size:
            # write it into chunks
            with subprocess.Popen(
                [
                    "split",
                    "-a",
                    "4",
                    "-b",
                    str(the_export.validated_chunk_size),
                    "-d",
                    "-",
                    tarfile_fp + ".",
                ],
                stdin=subprocess.PIPE,
            ) as split_process:
                try:
                    with tarfile.open(tarfile_fp, "w|gz", fileobj=split_process.stdin) as tar:
                        _do_export(pulp_exporter, tar, the_export)


When creating a chunked tar archive on my local machine and piping it through "split", only the first chunk is recognised with the file command:

woody@woody-arch-thinkpad ~/Documents $ tar -cvzf - file1.mp4 file2.mp4 | split -b 2MB - test.tar.gz.
woody@woody-arch-thinkpad ~/Documents $ file test.tar.gz.aa 
test.tar.gz.aa: gzip compressed data, from Unix, original size modulo 2^32 842674519
woody@woody-arch-thinkpad ~/Documents $ file test.tar.gz.ab
test.tar.gz.ab: data


If I use the "--tape-length" option, I can use the "file" command to identify each chunk as a tarball. I assume the virus scanning tool would be using "file" or similar methods.

woody@woody-arch-thinkpad ~/Documents $ tar -cv --tape-length=2000 --file=test-archive-{0..100}.tar file1.mp4 file2.mp4
woody@woody-arch-thinkpad ~/Documents $ file test-archive-0.tar 
test-archive-0.tar: POSIX tar archive (GNU)
woody@woody-arch-thinkpad ~/Documents $ file test-archive-1.tar 
test-archive-1.tar: tar archive

A drawback is that those "--tape-length"  archives can't be gzipped from that single tar command.  It gives an error
"tar: Cannot use multi-volume compressed archives"

Can we request that the exports are chunked with "--tape-length" instead of piping into the "split" command?
Many Satellite users in Banking, Government etc. will have similar security scanning requirements. 


Version-Release number of selected component (if applicable):
Problem appears in latest upstream Pulp code and in the version that ships with Red Hat Satellite 6.11

How reproducible:
Fully reproducible
 

Steps to Reproduce:
1. Create a chunked export from Satellite or Pulp
2. Run the "file" command against any chunk after first 

Actual results:
Chunks after the first are not recognized as GZ or tarball files, but instead random data because there is no header information in the file.


Expected results:
Chunks need to be identifiable for security scanning methods in high security disconnected scenarios.


Additional info:
Many Satellite users in Banking, Government etc. will have similar security scanning requirements. Dividing the archive with the "--tape-length" option instead of piping tar output to the to the split command will make the scanning tools work with Satellite content exports.