Bug 2000135

Possible upstream fix from 20.3.2: https://github.com/pypa/pip/pull/9226

I have patches for two possible approaches to this problem. They both share the first commit implementing the parsing of the `yanked_reason` attribute from HTML anchors.

1. Just print a warning with some help:

Patch: https://github.com/frenzymadness/pip/pull/1

Result:

$ python -m pip install "setuptools_scm<=6.3.0"
Collecting setuptools_scm<=6.3.0
  The best candidate for 'setuptools-scm' has been yanked! The reason is 'regression due to toml dependencies'.
  The old pip does not fully support handling of yanked releases
  so you should either update pip to the latest version and then
  install the package again or install a different version manually.
  Using cached https://files.pythonhosted.org/packages/5b/00/c89e466c8d439b6ca44275174bfbbcda2b3d65a9db65732f85649704cfe4/setuptools_scm-6.3.0-py3-none-any.whl
Requirement already satisfied: setuptools in /home/lbalhar/.virtualenvs/pip/lib/python3.9/site-packages (from setuptools_scm<=6.3.0)
Requirement already satisfied: packaging>=20.0 in /home/lbalhar/.virtualenvs/pip/lib/python3.9/site-packages (from setuptools_scm<=6.3.0)
Requirement already satisfied: pyparsing>=2.0.2 in /home/lbalhar/.virtualenvs/pip/lib/python3.9/site-packages (from packaging>=20.0->setuptools_scm<=6.3.0)
Installing collected packages: setuptools-scm
Successfully installed setuptools-scm-6.3.0
You are using pip version 9.0.3, however version 21.2.4 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.

2. Ignore yanked releases entirely:

Patch: https://github.com/frenzymadness/pip/pull/2

Result:

$ python -m pip install "setuptools_scm<=6.3.0"
Collecting setuptools_scm<=6.3.0
  Using cached https://files.pythonhosted.org/packages/c4/d5/e50358c82026f44cd8810c8165002746cd3f8b78865f6bcf5d7f0fe4f652/setuptools_scm-6.0.1-py3-none-any.whl
Requirement already satisfied: setuptools>=45 in /home/lbalhar/.virtualenvs/pip/lib/python3.9/site-packages (from setuptools_scm<=6.3.0)
Installing collected packages: setuptools-scm
Successfully installed setuptools-scm-6.0.1
You are using pip version 9.0.3, however version 21.2.4 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.

Thanks! I like (2) more.

One thing to test: pip install setuptools_scm==6.3.0 should still work.

I have a feeling that the `==` won't work. Implemeting all of PEP 592 will
be a bit harder, and simply disabling installation of yanked packages entirely
isn't appropriate.
Before we solve this properly (if ever), the message could be a bit more robust.
How about this:

  The best candidate for 'setuptools-scm', version '6.3.0', has been
  yanked (marked as not suitable)!
  The reason is 'regression due to toml dependencies'.
  Proceeding to install setuptools-scm==6.3.0 even though it is yanked.

  This version of pip does not fully support handling of yanked releases.
  When installing third-party packages from a repository that supports
  yanking (PEP 592), either always explicitly specify a known-good version,
  or try updating pip itself to the latest version from the third-party
  repository:
     python -m pip install --update pip
  Note that this newer version of pip, and any other software installed
  by pip, are generally not supported by Red Hat.

I believe that the latest implementation of method 2 in the PR correctly implements PEP 592.

The logic is as follows:

if the best candidate is yanked:
 - if there is only one candidate
   - install that yanked candidate and warn about it
 - else:
   - try to find new best candidate and warn about ignored yanked candidates - this may fail if all possible candidates
     are yanked but I think it's correct behavior.
else:
 - continue as usual and install the best candidate without any warning

The described behavior seems correct. Great!

I've played with your implementation on Fedora a bit. Uninstalls in between every installation omitted here.



(__venv__) $ pip install -U https://github.com/frenzymadness/pip/archive/refs/heads/skip_yanked.zip
...
Successfully installed pip-9.0.3

(__venv__) [tmp]$ pip install "setuptools_scm<=6.3.0"
Collecting setuptools_scm<=6.3.0
  Some of the 'setuptools-scm' releases have been yanked and therefore ignored now.
  Please, consider using a different version of 'setuptools-scm'.
  Downloading https://files.pythonhosted.org/packages/c4/d5/e50358c82026f44cd8810c8165002746cd3f8b78865f6bcf5d7f0fe4f652/setuptools_scm-6.0.1-py3-none-any.whl
Requirement already satisfied: setuptools>=45 in ./__venv__/lib/python3.6/site-packages (from setuptools_scm<=6.3.0)
Installing collected packages: setuptools-scm
Successfully installed setuptools-scm-6.0.1
You are using pip version 9.0.3, however version 21.2.4 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.

(__venv__) [tmp]$ pip install "setuptools_scm==6.2.0"
Collecting setuptools_scm==6.2.0
  The best candidate for 'setuptools-scm' has been yanked! The reason is 'regression due to a discovered setuptools bug'.
  The installation will continue because this is the only candidate.
  Please, consider using a different version of 'setuptools-scm'.
  Downloading https://files.pythonhosted.org/packages/6d/85/4436669ef4d143cdd77426cd72f62e456cd51cfe8b55fd246503d3cbc5d9/setuptools_scm-6.2.0-py3-none-any.whl
Requirement already satisfied: setuptools>=45 in ./__venv__/lib/python3.6/site-packages (from setuptools_scm==6.2.0)
Collecting tomli>=1.0 (from setuptools_scm==6.2.0)
  Downloading https://files.pythonhosted.org/packages/18/47/f7dab5b63b97efa7a715e389291d46246a5999c7b4705c2d147fc879e3b5/tomli-1.2.1-py3-none-any.whl
Installing collected packages: tomli, setuptools-scm
Successfully installed setuptools-scm-6.2.0 tomli-1.2.1
You are using pip version 9.0.3, however version 21.2.4 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.

(__venv__) [tmp]$ pip install "setuptools_scm<6.3,>=6.2"
Collecting setuptools_scm<6.3,>=6.2
  The best candidate for 'setuptools-scm' has been yanked! The reason is 'regression due to a discovered setuptools bug'.
  The installation will continue because this is the only candidate.
  Please, consider using a different version of 'setuptools-scm'.
  Using cached https://files.pythonhosted.org/packages/6d/85/4436669ef4d143cdd77426cd72f62e456cd51cfe8b55fd246503d3cbc5d9/setuptools_scm-6.2.0-py3-none-any.whl
Requirement already satisfied: tomli>=1.0 in ./__venv__/lib/python3.6/site-packages (from setuptools_scm<6.3,>=6.2)
Requirement already satisfied: setuptools>=45 in ./__venv__/lib/python3.6/site-packages (from setuptools_scm<6.3,>=6.2)
Installing collected packages: setuptools-scm
Successfully installed setuptools-scm-6.2.0
You are using pip version 9.0.3, however version 21.2.4 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.




New pip behaves like this:

(__venv__) $ pip install "setuptools_scm<=6.3.0"
Collecting setuptools_scm<=6.3.0
  Using cached setuptools_scm-6.0.1-py3-none-any.whl (27 kB)
Requirement already satisfied: setuptools>=45 in ./__venv__/lib/python3.6/site-packages (from setuptools_scm<=6.3.0) (49.1.3)
Installing collected packages: setuptools-scm
Successfully installed setuptools-scm-6.0.1



(__venv__) $ pip install "setuptools_scm<6.3,>=6.2"
Collecting setuptools_scm<6.3,>=6.2
  Using cached setuptools_scm-6.2.0-py3-none-any.whl (32 kB)
Requirement already satisfied: setuptools>=45 in ./__venv__/lib/python3.6/site-packages (from setuptools_scm<6.3,>=6.2) (49.1.3)
Requirement already satisfied: tomli>=1.0 in ./__venv__/lib/python3.6/site-packages (from setuptools_scm<6.3,>=6.2) (1.2.1)
WARNING: The candidate selected for download or install is a yanked version: 'setuptools-scm' candidate (version 6.2.0 at https://files.pythonhosted.org/packages/6d/85/4436669ef4d143cdd77426cd72f62e456cd51cfe8b55fd246503d3cbc5d9/setuptools_scm-6.2.0-py3-none-any.whl#sha256=3ff3c48a44fcd16b4354a1f82db64daff7434240fe913c51537dca016af466a7 (from https://pypi.org/simple/setuptools-scm/) (requires-python:>=3.6))
Reason for being yanked: regression due to a discovered setuptools bug
Installing collected packages: setuptools-scm
Successfully installed setuptools-scm-6.2.0



Maybe we could try to match the warning message and presence (e.g. the yanked versions ignored case does not warn)?

Thanks for the review. I've reimplemented it slightly and make the warning the same as it is in the new pip. See the fixup commit in my repository.

Testing:

--- non-yanked candidates available

-- My implementation

$ pip install "setuptools_scm<=6.3.0"
Collecting setuptools_scm<=6.3.0
  Using cached https://files.pythonhosted.org/packages/c4/d5/e50358c82026f44cd8810c8165002746cd3f8b78865f6bcf5d7f0fe4f652/setuptools_scm-6.0.1-py3-none-any.whl
Requirement already satisfied: setuptools>=45 in /home/lbalhar/.virtualenvs/pip/lib/python3.9/site-packages (from setuptools_scm<=6.3.0)
Installing collected packages: setuptools-scm
Successfully installed setuptools-scm-6.0.1
You are using pip version 9.0.3, however version 21.2.4 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.

-- Latest pip

$ pip install "setuptools_scm<=6.3.0"
Collecting setuptools_scm<=6.3.0
  Using cached setuptools_scm-6.0.1-py3-none-any.whl (27 kB)
Requirement already satisfied: setuptools>=45 in ./lib/python3.9/site-packages (from setuptools_scm<=6.3.0) (57.4.0)
Installing collected packages: setuptools-scm
Successfully installed setuptools-scm-6.0.1

--- specific version, yanked

-- My implementation

$ pip install "setuptools_scm==6.2.0"
Collecting setuptools_scm==6.2.0
  Cache entry deserialization failed, entry ignored
  WARNING: The candidate selected for download or install is a yanked version: 'setuptools-scm' candidate (version 6.2.0 at https://files.pythonhosted.org/packages/6d/85/4436669ef4d143cdd77426cd72f62e456cd51cfe8b55fd246503d3cbc5d9/setuptools_scm-6.2.0-py3-none-any.whl#sha256=3ff3c48a44fcd16b4354a1f82db64daff7434240fe913c51537dca016af466a7 (from https://pypi.org/simple/setuptools-scm/) (requires-python:>=3.6))
  Reason for being yanked: regression due to a discovered setuptools bug
  Using cached https://files.pythonhosted.org/packages/6d/85/4436669ef4d143cdd77426cd72f62e456cd51cfe8b55fd246503d3cbc5d9/setuptools_scm-6.2.0-py3-none-any.whl
Requirement already satisfied: tomli>=1.0 in /home/lbalhar/.virtualenvs/pip/lib/python3.9/site-packages (from setuptools_scm==6.2.0)
Requirement already satisfied: setuptools>=45 in /home/lbalhar/.virtualenvs/pip/lib/python3.9/site-packages (from setuptools_scm==6.2.0)
Installing collected packages: setuptools-scm
Successfully installed setuptools-scm-6.2.0
You are using pip version 9.0.3, however version 21.2.4 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.

-- Latest pip

$ pip install "setuptools_scm==6.2.0"
Collecting setuptools_scm==6.2.0
  Using cached setuptools_scm-6.2.0-py3-none-any.whl (32 kB)
Requirement already satisfied: setuptools>=45 in ./lib/python3.9/site-packages (from setuptools_scm==6.2.0) (57.4.0)
Collecting tomli>=1.0
  Using cached tomli-1.2.1-py3-none-any.whl (11 kB)
WARNING: The candidate selected for download or install is a yanked version: 'setuptools-scm' candidate (version 6.2.0 at https://files.pythonhosted.org/packages/6d/85/4436669ef4d143cdd77426cd72f62e456cd51cfe8b55fd246503d3cbc5d9/setuptools_scm-6.2.0-py3-none-any.whl#sha256=3ff3c48a44fcd16b4354a1f82db64daff7434240fe913c51537dca016af466a7 (from https://pypi.org/simple/setuptools-scm/) (requires-python:>=3.6))
Reason for being yanked: regression due to a discovered setuptools bug
Installing collected packages: tomli, setuptools-scm
Successfully installed setuptools-scm-6.2.0 tomli-1.2.1

--- multiple candidates, but all of them yanked

-- My implementation

$ pip install "setuptools_scm<6.3,>=6.1"
Collecting setuptools_scm<6.3,>=6.1
  WARNING: The candidate selected for download or install is a yanked version: 'setuptools-scm' candidate (version 6.2.0 at https://files.pythonhosted.org/packages/6d/85/4436669ef4d143cdd77426cd72f62e456cd51cfe8b55fd246503d3cbc5d9/setuptools_scm-6.2.0-py3-none-any.whl#sha256=3ff3c48a44fcd16b4354a1f82db64daff7434240fe913c51537dca016af466a7 (from https://pypi.org/simple/setuptools-scm/) (requires-python:>=3.6))
  Reason for being yanked: regression due to a discovered setuptools bug
  Using cached https://files.pythonhosted.org/packages/6d/85/4436669ef4d143cdd77426cd72f62e456cd51cfe8b55fd246503d3cbc5d9/setuptools_scm-6.2.0-py3-none-any.whl
Requirement already satisfied: setuptools>=45 in /home/lbalhar/.virtualenvs/pip/lib/python3.9/site-packages (from setuptools_scm<6.3,>=6.1)
Requirement already satisfied: tomli>=1.0 in /home/lbalhar/.virtualenvs/pip/lib/python3.9/site-packages (from setuptools_scm<6.3,>=6.1)
Installing collected packages: setuptools-scm
Successfully installed setuptools-scm-6.2.0
You are using pip version 9.0.3, however version 21.2.4 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.

-- Latest pip

$ pip install "setuptools_scm<6.3,>=6.1"
Collecting setuptools_scm<6.3,>=6.1
  Using cached setuptools_scm-6.2.0-py3-none-any.whl (32 kB)
Requirement already satisfied: setuptools>=45 in ./lib/python3.9/site-packages (from setuptools_scm<6.3,>=6.1) (57.4.0)
Requirement already satisfied: tomli>=1.0 in ./lib/python3.9/site-packages (from setuptools_scm<6.3,>=6.1) (1.2.1)
WARNING: The candidate selected for download or install is a yanked version: 'setuptools-scm' candidate (version 6.2.0 at https://files.pythonhosted.org/packages/6d/85/4436669ef4d143cdd77426cd72f62e456cd51cfe8b55fd246503d3cbc5d9/setuptools_scm-6.2.0-py3-none-any.whl#sha256=3ff3c48a44fcd16b4354a1f82db64daff7434240fe913c51537dca016af466a7 (from https://pypi.org/simple/setuptools-scm/) (requires-python:>=3.6))
Reason for being yanked: regression due to a discovered setuptools bug
Installing collected packages: setuptools-scm
Successfully installed setuptools-scm-6.2.0


I cannot think about other possible tests because the new logic is involved only if the best candidate is yanked. Therefore we can test only version ranges where the highest version is yanked and I think all three possible cases are covered by the tests above:

- there is only one candidate and it's yanked - use it and warn about it
- there are multiple candidates but all of them are yanked - use the best one and warn about it
- there are multiple candidates and only some of them yanked - use the best (newest) non-yanked candidate without any warning

Looks perfect!

Let's be careful and keep it open for Petr and after his review, I'll prepare a PR and some downstream tests.

Petr pointed out that the reason might be empty - in that case, the line "Reason for being yanked:…" doesn't show. See the PR mentioned before for the latest info.

Scratch build: https://brewweb.engineering.redhat.com/brew/taskinfo?taskID=40136027
PR: https://src.osci.redhat.com/rpms/python-pip/pull-request/32

Verification:

script/commands:
# old installs 6.3.0, new installs 6.0.1, no warning in both
pip3.6 install "setuptools_scm<=6.3.0"
pip3.6 uninstall -y "setuptools_scm" > /dev/null 2>&1
# both installs 6.2.0 but new version shows a warning
pip3.6 install "setuptools_scm==6.2.0"
pip3.6 uninstall -y "setuptools_scm" > /dev/null 2>&1
# both installs 6.2.0 but new version shows a warning
pip3.6 install "setuptools_scm<6.3,>=6.1"
pip3.6 uninstall -y "setuptools_scm" > /dev/null 2>&1

# rpm -qv python3-pip
python3-pip-9.0.3-19.el8.noarch

WARNING: Running pip install with root privileges is generally not a good idea. Try `pip3.6 install --user` instead.
Collecting setuptools_scm<=6.3.0
  Using cached https://files.pythonhosted.org/packages/5b/00/c89e466c8d439b6ca44275174bfbbcda2b3d65a9db65732f85649704cfe4/setuptools_scm-6.3.0-py3-none-any.whl
Requirement already satisfied: setuptools in /usr/local/lib/python3.6/site-packages (from setuptools_scm<=6.3.0)
Requirement already satisfied: packaging>=20.0 in /usr/local/lib/python3.6/site-packages (from setuptools_scm<=6.3.0)
Requirement already satisfied: pyparsing>=2.0.2 in /usr/local/lib/python3.6/site-packages (from packaging>=20.0->setuptools_scm<=6.3.0)
Installing collected packages: setuptools-scm
Successfully installed setuptools-scm-6.3.0


WARNING: Running pip install with root privileges is generally not a good idea. Try `pip3.6 install --user` instead.
Collecting setuptools_scm==6.2.0
  Using cached https://files.pythonhosted.org/packages/6d/85/4436669ef4d143cdd77426cd72f62e456cd51cfe8b55fd246503d3cbc5d9/setuptools_scm-6.2.0-py3-none-any.whl
Requirement already satisfied: setuptools>=45 in /usr/local/lib/python3.6/site-packages (from setuptools_scm==6.2.0)
Requirement already satisfied: tomli>=1.0 in /usr/local/lib/python3.6/site-packages (from setuptools_scm==6.2.0)
Installing collected packages: setuptools-scm
Successfully installed setuptools-scm-6.2.0


WARNING: Running pip install with root privileges is generally not a good idea. Try `pip3.6 install --user` instead.
Collecting setuptools_scm<6.3,>=6.1
  Using cached https://files.pythonhosted.org/packages/6d/85/4436669ef4d143cdd77426cd72f62e456cd51cfe8b55fd246503d3cbc5d9/setuptools_scm-6.2.0-py3-none-any.whl
Requirement already satisfied: setuptools>=45 in /usr/local/lib/python3.6/site-packages (from setuptools_scm<6.3,>=6.1)
Requirement already satisfied: tomli>=1.0 in /usr/local/lib/python3.6/site-packages (from setuptools_scm<6.3,>=6.1)
Installing collected packages: setuptools-scm
Successfully installed setuptools-scm-6.2.0

# rpm -qv python3-pip
python3-pip-9.0.3-21.el8.noarch

WARNING: Running pip install with root privileges is generally not a good idea. Try `pip3.6 install --user` instead.
Collecting setuptools_scm<=6.3.0
  Downloading https://files.pythonhosted.org/packages/c4/d5/e50358c82026f44cd8810c8165002746cd3f8b78865f6bcf5d7f0fe4f652/setuptools_scm-6.0.1-py3-none-any.whl
Requirement already satisfied: setuptools>=45 in /usr/local/lib/python3.6/site-packages (from setuptools_scm<=6.3.0)
Installing collected packages: setuptools-scm
Successfully installed setuptools-scm-6.0.1

WARNING: Running pip install with root privileges is generally not a good idea. Try `pip3.6 install --user` instead.
Collecting setuptools_scm==6.2.0
  WARNING: The candidate selected for download or install is a yanked version: 'setuptools-scm' candidate (version 6.2.0 at https://files.pythonhosted.org/packages/6d/85/4436669ef4d143cdd77426cd72f62e456cd51cfe8b55fd246503d3cbc5d9/setuptools_scm-6.2.0-py3-none-any.whl#sha256=3ff3c48a44fcd16b4354a1f82db64daff7434240fe913c51537dca016af466a7 (from https://pypi.org/simple/setuptools-scm/) (requires-python:>=3.6))
  Reason for being yanked: regression due to a discovered setuptools bug
  Using cached https://files.pythonhosted.org/packages/6d/85/4436669ef4d143cdd77426cd72f62e456cd51cfe8b55fd246503d3cbc5d9/setuptools_scm-6.2.0-py3-none-any.whl
Requirement already satisfied: tomli>=1.0 in /usr/local/lib/python3.6/site-packages (from setuptools_scm==6.2.0)
Requirement already satisfied: setuptools>=45 in /usr/local/lib/python3.6/site-packages (from setuptools_scm==6.2.0)
Installing collected packages: setuptools-scm
Successfully installed setuptools-scm-6.2.0

WARNING: Running pip install with root privileges is generally not a good idea. Try `pip3.6 install --user` instead.
Collecting setuptools_scm<6.3,>=6.1
  WARNING: The candidate selected for download or install is a yanked version: 'setuptools-scm' candidate (version 6.2.0 at https://files.pythonhosted.org/packages/6d/85/4436669ef4d143cdd77426cd72f62e456cd51cfe8b55fd246503d3cbc5d9/setuptools_scm-6.2.0-py3-none-any.whl#sha256=3ff3c48a44fcd16b4354a1f82db64daff7434240fe913c51537dca016af466a7 (from https://pypi.org/simple/setuptools-scm/) (requires-python:>=3.6))
  Reason for being yanked: regression due to a discovered setuptools bug
  Using cached https://files.pythonhosted.org/packages/6d/85/4436669ef4d143cdd77426cd72f62e456cd51cfe8b55fd246503d3cbc5d9/setuptools_scm-6.2.0-py3-none-any.whl
Requirement already satisfied: setuptools>=45 in /usr/local/lib/python3.6/site-packages (from setuptools_scm<6.3,>=6.1)
Requirement already satisfied: tomli>=1.0 in /usr/local/lib/python3.6/site-packages (from setuptools_scm<6.3,>=6.1)
Installing collected packages: setuptools-scm
Successfully installed setuptools-scm-6.2.0

FTR: there is no need for fixing this in other pip versions in RHEL 8. PEP 592 (yanked releases support) has been implemented in 19.2 (2019-07-22) for the old resolver and in 20.3.2 (2020-12-15) for the new one. In RHEL 8 in alternative Python stacks we have:

python38-pip-19.3.1-4.module+el8.5.0+12205+a865257a.noarch
python39-pip-20.2.4-6.module+el8.5.0+12204+54860423.noarch

which are both higher than 19.2 and both still use the old resolver. I've manually verified that they behave correctly - basically the same way as the new pip and the fixed version of pip3.6.

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (python-pip bug fix and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2022:2006