Bug 2106530 - Migration to pulp3 requires additional disk space to migrate the docker blobs ( /var/lib/pulp/content/units/docker_blob ) in Satellite 6.9
Summary: Migration to pulp3 requires additional disk space to migrate the docker blobs...
Keywords:
Status: NEW
Alias: None
Product: Red Hat Satellite
Classification: Red Hat
Component: Pulp
Version: 6.9.9
Hardware: x86_64
OS: Unspecified
high
high vote
Target Milestone: Unspecified
Assignee: satellite6-bugs
QA Contact: Lai
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2022-07-12 21:50 UTC by Sayan Das
Modified: 2022-07-29 13:15 UTC (History)
3 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed:
Target Upstream Version:


Attachments (Terms of Use)

Description Sayan Das 2022-07-12 21:50:34 UTC
Description of problem:

During migration from Pulp2 to Pulp3, The migration of /var/lib/pulp/content/units/docker_blob requires additional free disk space in /var/lib/pulp.

i.e. if we have 

50 GB consumed for /var/lib/pulp/published by Pulp2 

60 GB consumed for /var/lib/pulp/content/units/docker_blob by Pulp2

Then just having 50*2=100 GB free space in /var/lib/pulp is not good enough. We will require the 60 GB additional disk space to get the docker blobs migrated to pulp 3 as well.

Simply assume the same happenng for a satellite where 100 GB is the data consumed by /var/lib/pulp/published , 1.1 TB by /var/lib/pulp/content/units/docker_blob and free disk space in pulp is only 383 gb.


Version-Release number of selected component (if applicable):

Satellite 6.9.9 ( with GBs of docker blobs )


How reproducible:

In customer's environment 


Steps to Reproduce: [ Not exactly the reproducer steps but these are only ones I could think of ]

1. Install Satellite 6.9 with /var mounted seperately on a XFS LV [ of 150 or 200 GB ]
2. Sync 10 or 20 rpm\yum repos
3. Sync 20+ docker type repos without limiting the tags during sync.
4. Add them in several CVs , Publish + Promote 
5. Ensure that /var has just enough free space equals to "2x of space consumed by /var/lib/pulp/published + space consumed by /var/lib/mongodb" + 20 more GB

    ( without taking the disk space utilized by /var/lib/pulp/content/units/docker_blob )


6. Perform pulp2 to Pulp3 migration while keeping an eye on the usage of /var and /var/lib/pulp/media

Actual results:

/var/lib/pulp/media gets filled up and eventually failed to complete the migration of docker type content :

                "error": {
                        "traceback": "  File \"/usr/lib/python3.6/site-packages/rq/worker.py\", line 936, in perform_job\n    rv = job.perform()\n  File \"/usr/lib/python3.6/site-packages/rq/job.py\", line 684, in perform\n    self._result = self._execute()\n  File \"/usr/lib/python3.6/site-packages/rq/job.py\", line 690, in _execute\n    return self.func(*self.args, **self.kwargs)\n  File \"/usr/lib/python3.6/site-packages/pulp_2to3_migration/app/tasks/migrate.py\", line 81, in migrate_from_pulp2\n    migrate_content(plan, skip_corrupted=skip_corrupted)\n  File \"/usr/lib/python3.6/site-packages/pulp_2to3_migration/app/migration.py\", line 55, in migrate_content\n    plugin.migrator.migrate_content_to_pulp3(skip_corrupted=skip_corrupted)\n  File \"/usr/lib/python3.6/site-packages/pulp_2to3_migration/app/plugin/docker/migrator.py\", line 106, in migrate_content_to_pulp3\n    loop.run_until_complete(dm.create())\n  File \"/usr/lib64/python3.6/asyncio/base_events.py\", line 484, in run_until_complete\n    return future.result()\n  File \"/usr/lib/python3.6/site-packages/pulp_2to3_migration/app/plugin/content.py\", line 89, in create\n    await pipeline\n  File \"/usr/lib/python3.6/site-packages/pulpcore/plugin/stages/api.py\", line 225, in create_pipeline\n    await asyncio.gather(*futures)\n  File \"/usr/lib/python3.6/site-packages/pulpcore/plugin/stages/api.py\", line 43, in __call__\n    await self.run()\n  File \"/usr/lib/python3.6/site-packages/pulp_2to3_migration/app/plugin/content.py\", line 181, in run\n    self.migrate_to_pulp3(cmodel, ctype)\n  File \"/usr/lib/python3.6/site-packages/pulp_2to3_migration/app/plugin/content.py\", line 420, in migrate_to_pulp3\n    downloaded=pulp2content.downloaded\n  File \"/usr/lib/python3.6/site-packages/pulp_2to3_migration/app/plugin/content.py\", line 150, in create_artifact\n    shutil.copy2(pulp2_storage_path, pulp3_storage_path)\n  File \"/usr/lib64/python3.6/shutil.py\", line 263, in copy2\n    copyfile(src, dst, follow_symlinks=follow_symlinks)\n  File \"/usr/lib64/python3.6/shutil.py\", line 122, in copyfile\n    copyfileobj(fsrc, fdst)\n  File \"/usr/lib64/python3.6/shutil.py\", line 82, in copyfileobj\n    fdst.write(buf)\n",
                        "description": "[Errno 28] No space left on device"
                },

[ Data collected by querying the failed pulp3 task ]                


Expected results:

All contents including docker\rpm\files should be hardlinked from /var/lib/pulp/content/units/ to /var/lib/pulp/media/artifacts. And there should not be any additional space required by docker content migration. 


Additional info:

NA

Comment 4 Tanya Tereshchenko 2022-07-13 09:48:42 UTC
There is nothing special about docker blobs, they should be handled in the same way as other content with artifacts.
Looking at the traceback `shutil.copy2(pulp2_storage_path, pulp3_storage_path)`, it seems that Pulp does go into the copy mode.
That's a failover behaviour when it is not possible to create hard links https://github.com/pulp/pulp-2to3-migration/blob/0.11/pulp_2to3_migration/app/plugin/content.py#L144-L150

Could you or customer check that it is possible to create a hard link from pulp3 storage to a docker blob in pulp2 storage?

cc @ggainey @ipanova

Comment 5 Sayan Das 2022-07-13 12:58:35 UTC
(In reply to Tanya Tereshchenko from comment #4)
> There is nothing special about docker blobs, they should be handled in the
> same way as other content with artifacts.
> Looking at the traceback `shutil.copy2(pulp2_storage_path,
> pulp3_storage_path)`, it seems that Pulp does go into the copy mode.
> That's a failover behaviour when it is not possible to create hard links
> https://github.com/pulp/pulp-2to3-migration/blob/0.11/pulp_2to3_migration/
> app/plugin/content.py#L144-L150
> 
> Could you or customer check that it is possible to create a hard link from
> pulp3 storage to a docker blob in pulp2 storage?
> 
> cc @ggainey @ipanova


Hello Tanya,

So my doubt was correct for this CU i.e. the docker_blobs were indeed being copied instead of hard-linked.

They use XFS file system for /var LV and that is the only LV that contains all satellite+pulp stuff. So I am not sure why hard linking will fail and fall back to failover code. 

Anyway, I am testing migration on a small scale and have requested the customer to do the following as well i.e. 

* Take Any one docker_blob file from inside /var/lib/pulp/content/units/docker_blob/*/ directory and try to hardlink it by a different name inside /var/lib/pulp/media/artifacts.

* Once done, verify the inode number of both and share that data with me.

If the customer gets back to me, I will share the details here.


Note You need to log in before you can comment on or make changes to this bug.