Description of problem:

During migration from Pulp2 to Pulp3, The migration of /var/lib/pulp/content/units/docker_blob requires additional free disk space in /var/lib/pulp.

i.e. if we have 

50 GB consumed for /var/lib/pulp/published by Pulp2 

60 GB consumed for /var/lib/pulp/content/units/docker_blob by Pulp2

Then just having 50*2=100 GB free space in /var/lib/pulp is not good enough. We will require the 60 GB additional disk space to get the docker blobs migrated to pulp 3 as well.

Simply assume the same happenng for a satellite where 100 GB is the data consumed by /var/lib/pulp/published , 1.1 TB by /var/lib/pulp/content/units/docker_blob and free disk space in pulp is only 383 gb.


Version-Release number of selected component (if applicable):

Satellite 6.9.9 ( with GBs of docker blobs )


How reproducible:

In customer's environment 


Steps to Reproduce: [ Not exactly the reproducer steps but these are only ones I could think of ]

1. Install Satellite 6.9 with /var mounted seperately on a XFS LV [ of 150 or 200 GB ]
2. Sync 10 or 20 rpm\yum repos
3. Sync 20+ docker type repos without limiting the tags during sync.
4. Add them in several CVs , Publish + Promote 
5. Ensure that /var has just enough free space equals to "2x of space consumed by /var/lib/pulp/published + space consumed by /var/lib/mongodb" + 20 more GB

    ( without taking the disk space utilized by /var/lib/pulp/content/units/docker_blob )


6. Perform pulp2 to Pulp3 migration while keeping an eye on the usage of /var and /var/lib/pulp/media

Actual results:

/var/lib/pulp/media gets filled up and eventually failed to complete the migration of docker type content :

                "error": {
                        "traceback": "  File \"/usr/lib/python3.6/site-packages/rq/worker.py\", line 936, in perform_job\n    rv = job.perform()\n  File \"/usr/lib/python3.6/site-packages/rq/job.py\", line 684, in perform\n    self._result = self._execute()\n  File \"/usr/lib/python3.6/site-packages/rq/job.py\", line 690, in _execute\n    return self.func(*self.args, **self.kwargs)\n  File \"/usr/lib/python3.6/site-packages/pulp_2to3_migration/app/tasks/migrate.py\", line 81, in migrate_from_pulp2\n    migrate_content(plan, skip_corrupted=skip_corrupted)\n  File \"/usr/lib/python3.6/site-packages/pulp_2to3_migration/app/migration.py\", line 55, in migrate_content\n    plugin.migrator.migrate_content_to_pulp3(skip_corrupted=skip_corrupted)\n  File \"/usr/lib/python3.6/site-packages/pulp_2to3_migration/app/plugin/docker/migrator.py\", line 106, in migrate_content_to_pulp3\n    loop.run_until_complete(dm.create())\n  File \"/usr/lib64/python3.6/asyncio/base_events.py\", line 484, in run_until_complete\n    return future.result()\n  File \"/usr/lib/python3.6/site-packages/pulp_2to3_migration/app/plugin/content.py\", line 89, in create\n    await pipeline\n  File \"/usr/lib/python3.6/site-packages/pulpcore/plugin/stages/api.py\", line 225, in create_pipeline\n    await asyncio.gather(*futures)\n  File \"/usr/lib/python3.6/site-packages/pulpcore/plugin/stages/api.py\", line 43, in __call__\n    await self.run()\n  File \"/usr/lib/python3.6/site-packages/pulp_2to3_migration/app/plugin/content.py\", line 181, in run\n    self.migrate_to_pulp3(cmodel, ctype)\n  File \"/usr/lib/python3.6/site-packages/pulp_2to3_migration/app/plugin/content.py\", line 420, in migrate_to_pulp3\n    downloaded=pulp2content.downloaded\n  File \"/usr/lib/python3.6/site-packages/pulp_2to3_migration/app/plugin/content.py\", line 150, in create_artifact\n    shutil.copy2(pulp2_storage_path, pulp3_storage_path)\n  File \"/usr/lib64/python3.6/shutil.py\", line 263, in copy2\n    copyfile(src, dst, follow_symlinks=follow_symlinks)\n  File \"/usr/lib64/python3.6/shutil.py\", line 122, in copyfile\n    copyfileobj(fsrc, fdst)\n  File \"/usr/lib64/python3.6/shutil.py\", line 82, in copyfileobj\n    fdst.write(buf)\n",
                        "description": "[Errno 28] No space left on device"
                },

[ Data collected by querying the failed pulp3 task ]                


Expected results:

All contents including docker\rpm\files should be hardlinked from /var/lib/pulp/content/units/ to /var/lib/pulp/media/artifacts. And there should not be any additional space required by docker content migration. 


Additional info:

NA