Description of problem: During migration from Pulp2 to Pulp3, The migration of /var/lib/pulp/content/units/docker_blob requires additional free disk space in /var/lib/pulp. i.e. if we have 50 GB consumed for /var/lib/pulp/published by Pulp2 60 GB consumed for /var/lib/pulp/content/units/docker_blob by Pulp2 Then just having 50*2=100 GB free space in /var/lib/pulp is not good enough. We will require the 60 GB additional disk space to get the docker blobs migrated to pulp 3 as well. Simply assume the same happenng for a satellite where 100 GB is the data consumed by /var/lib/pulp/published , 1.1 TB by /var/lib/pulp/content/units/docker_blob and free disk space in pulp is only 383 gb. Version-Release number of selected component (if applicable): Satellite 6.9.9 ( with GBs of docker blobs ) How reproducible: In customer's environment Steps to Reproduce: [ Not exactly the reproducer steps but these are only ones I could think of ] 1. Install Satellite 6.9 with /var mounted seperately on a XFS LV [ of 150 or 200 GB ] 2. Sync 10 or 20 rpm\yum repos 3. Sync 20+ docker type repos without limiting the tags during sync. 4. Add them in several CVs , Publish + Promote 5. Ensure that /var has just enough free space equals to "2x of space consumed by /var/lib/pulp/published + space consumed by /var/lib/mongodb" + 20 more GB ( without taking the disk space utilized by /var/lib/pulp/content/units/docker_blob ) 6. Perform pulp2 to Pulp3 migration while keeping an eye on the usage of /var and /var/lib/pulp/media Actual results: /var/lib/pulp/media gets filled up and eventually failed to complete the migration of docker type content : "error": { "traceback": " File \"/usr/lib/python3.6/site-packages/rq/worker.py\", line 936, in perform_job\n rv = job.perform()\n File \"/usr/lib/python3.6/site-packages/rq/job.py\", line 684, in perform\n self._result = self._execute()\n File \"/usr/lib/python3.6/site-packages/rq/job.py\", line 690, in _execute\n return self.func(*self.args, **self.kwargs)\n File \"/usr/lib/python3.6/site-packages/pulp_2to3_migration/app/tasks/migrate.py\", line 81, in migrate_from_pulp2\n migrate_content(plan, skip_corrupted=skip_corrupted)\n File \"/usr/lib/python3.6/site-packages/pulp_2to3_migration/app/migration.py\", line 55, in migrate_content\n plugin.migrator.migrate_content_to_pulp3(skip_corrupted=skip_corrupted)\n File \"/usr/lib/python3.6/site-packages/pulp_2to3_migration/app/plugin/docker/migrator.py\", line 106, in migrate_content_to_pulp3\n loop.run_until_complete(dm.create())\n File \"/usr/lib64/python3.6/asyncio/base_events.py\", line 484, in run_until_complete\n return future.result()\n File \"/usr/lib/python3.6/site-packages/pulp_2to3_migration/app/plugin/content.py\", line 89, in create\n await pipeline\n File \"/usr/lib/python3.6/site-packages/pulpcore/plugin/stages/api.py\", line 225, in create_pipeline\n await asyncio.gather(*futures)\n File \"/usr/lib/python3.6/site-packages/pulpcore/plugin/stages/api.py\", line 43, in __call__\n await self.run()\n File \"/usr/lib/python3.6/site-packages/pulp_2to3_migration/app/plugin/content.py\", line 181, in run\n self.migrate_to_pulp3(cmodel, ctype)\n File \"/usr/lib/python3.6/site-packages/pulp_2to3_migration/app/plugin/content.py\", line 420, in migrate_to_pulp3\n downloaded=pulp2content.downloaded\n File \"/usr/lib/python3.6/site-packages/pulp_2to3_migration/app/plugin/content.py\", line 150, in create_artifact\n shutil.copy2(pulp2_storage_path, pulp3_storage_path)\n File \"/usr/lib64/python3.6/shutil.py\", line 263, in copy2\n copyfile(src, dst, follow_symlinks=follow_symlinks)\n File \"/usr/lib64/python3.6/shutil.py\", line 122, in copyfile\n copyfileobj(fsrc, fdst)\n File \"/usr/lib64/python3.6/shutil.py\", line 82, in copyfileobj\n fdst.write(buf)\n", "description": "[Errno 28] No space left on device" }, [ Data collected by querying the failed pulp3 task ] Expected results: All contents including docker\rpm\files should be hardlinked from /var/lib/pulp/content/units/ to /var/lib/pulp/media/artifacts. And there should not be any additional space required by docker content migration. Additional info: NA
There is nothing special about docker blobs, they should be handled in the same way as other content with artifacts. Looking at the traceback `shutil.copy2(pulp2_storage_path, pulp3_storage_path)`, it seems that Pulp does go into the copy mode. That's a failover behaviour when it is not possible to create hard links https://github.com/pulp/pulp-2to3-migration/blob/0.11/pulp_2to3_migration/app/plugin/content.py#L144-L150 Could you or customer check that it is possible to create a hard link from pulp3 storage to a docker blob in pulp2 storage? cc @ggainey @ipanova
(In reply to Tanya Tereshchenko from comment #4) > There is nothing special about docker blobs, they should be handled in the > same way as other content with artifacts. > Looking at the traceback `shutil.copy2(pulp2_storage_path, > pulp3_storage_path)`, it seems that Pulp does go into the copy mode. > That's a failover behaviour when it is not possible to create hard links > https://github.com/pulp/pulp-2to3-migration/blob/0.11/pulp_2to3_migration/ > app/plugin/content.py#L144-L150 > > Could you or customer check that it is possible to create a hard link from > pulp3 storage to a docker blob in pulp2 storage? > > cc @ggainey @ipanova Hello Tanya, So my doubt was correct for this CU i.e. the docker_blobs were indeed being copied instead of hard-linked. They use XFS file system for /var LV and that is the only LV that contains all satellite+pulp stuff. So I am not sure why hard linking will fail and fall back to failover code. Anyway, I am testing migration on a small scale and have requested the customer to do the following as well i.e. * Take Any one docker_blob file from inside /var/lib/pulp/content/units/docker_blob/*/ directory and try to hardlink it by a different name inside /var/lib/pulp/media/artifacts. * Once done, verify the inode number of both and share that data with me. If the customer gets back to me, I will share the details here.