Bug 2030621 - Python 3.10 ignores rewritten __init__() in subinterpreters
Summary: Python 3.10 ignores rewritten __init__() in subinterpreters
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Fedora
Classification: Fedora
Component: python3.10
Version: 35
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: ---
Assignee: Miro Hrončok
QA Contact: Fedora Extras Quality Assurance
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2021-12-09 09:59 UTC by Aurelien Bompard
Modified: 2022-01-16 01:18 UTC (History)
8 users (show)

Fixed In Version: python3.10-3.10.1-3.fc36 python3.10-3.10.1-3.fc35
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2022-01-16 01:18:43 UTC
Type: Bug


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Python 46006 0 None None None 2021-12-10 12:34:07 UTC
Python 46034 0 None None None 2021-12-10 12:01:24 UTC

Description Aurelien Bompard 2021-12-09 09:59:29 UTC
Description of problem:
I don't know if it actually comes from Python, but here's what's going wrong. In some situations the decorators that rewrite __init__() seem to be ignored. Here's a reproducer:

import attr

@attr.s
class TestClass:
    foo = attr.ib()

TestClass("debug")


This works when run on the command line. However, if you try to run it in mod_wsgi:

def application(environ, start_response):
    response = "Debugging.\n".encode()
    headers = [
        ('Content-Length', str(len(response))),
        ('Content-Type', "text/plain"),
    ]
    start_response('200 OK', headers)
    TestClass("debug")  # Here's the instanciation
    return [response]

It will fail with the following error in the logs:

Traceback (most recent call last):
  File "/var/www/cgi-bin/testing.py", line 18, in application
    TestClass("debug")
TypeError: TestClass() takes no arguments

We're not the only ones to have this issue:
- https://github.com/GrahamDumpleton/mod_wsgi/issues/729
- https://github.com/poljar/weechat-matrix/issues/293

Version-Release number of selected component:
python3-3.10.0-1.fc35.x86_64
python3-mod_wsgi-4.9.0-1.fc35.x86_64

How reproducible: Always

Comment 1 Aurelien Bompard 2021-12-09 10:03:19 UTC
FYI it prevents us from upgrading the Koji builders to F35, because Koji runs on mod_wsgi, and it has a fedora-messaging plugin which uses Twisted which uses attrs.

Comment 2 Miro Hrončok 2021-12-09 10:34:37 UTC
> However, if you try to run it in mod_wsgi...

Thanks for the code snippet, but *how* do you run it exactly?

Comment 3 Miro Hrončok 2021-12-09 10:48:48 UTC
From our conversation on IRC:

$ cat /etc/httpd/conf.d/essai.conf 
Alias /essai /var/www/cgi-bin/essai/essai.py
 
<Directory "/var/www/cgi-bin/essai">
    Options ExecCGI
    SetHandler wsgi-script
    Require all granted
</Directory>


$ cat /var/www/cgi-bin/essai/essai.py 
import attr
 
@attr.s
class TestClass:
    foo = attr.ib()
 
 
print(attr.s, repr(attr.s), attr.define, repr(attr.define))
 
def application(environ, start_response):
    response = "Debugging.\n"
    response = response.encode()
    headers = [
        ('Content-Length', str(len(response))),
        ('Content-Type', "text/plain"),
    ]
    start_response('200 OK', headers)
    TestClass("debug")
    return [response]


$ sudo systemctl restart httpd

Go to http://localhost/essai and observe the Internal Server Error.

$ sudo cat /var/log/httpd/error_log
...
[Thu Dec 09 11:46:22.823867 2021] [wsgi:error] [pid 667962:tid 668079] <function attrs at 0x7f61fc29e3b0> <function attrs at 0x7f61fc29e3b0> <function define at 0x7f61fc0c5240> <function define at 0x7f61fc0c5240>
[Thu Dec 09 11:46:22.823946 2021] [wsgi:error] [pid 667962:tid 668079] [client 127.0.0.1:33668] mod_wsgi (pid=667962): Exception occurred processing WSGI script '/var/www/cgi-bin/essai/essai.py'.
[Thu Dec 09 11:46:22.824269 2021] [wsgi:error] [pid 667962:tid 668079] [client 127.0.0.1:33668] Traceback (most recent call last):
[Thu Dec 09 11:46:22.824284 2021] [wsgi:error] [pid 667962:tid 668079] [client 127.0.0.1:33668]   File "/var/www/cgi-bin/essai/essai.py", line 18, in application
[Thu Dec 09 11:46:22.824287 2021] [wsgi:error] [pid 667962:tid 668079] [client 127.0.0.1:33668]     TestClass("debug")
[Thu Dec 09 11:46:22.824295 2021] [wsgi:error] [pid 667962:tid 668079] [client 127.0.0.1:33668] TypeError: TestClass() takes no arguments

Comment 4 Aurelien Bompard 2021-12-09 11:16:57 UTC
Here's a simpler reproducer that does not involve attrs:

def tweak_init(cls):
    def new_init(inst, name):
        inst.text = f"Hello, {name}\n"

    cls.__init__ = new_init
    return cls

@tweak_init
class Greeting:
    pass


def application(environ, start_response):
    response = "Debugging.\n"
    response = response.encode()
    headers = [
        ('Content-Length', str(len(response))),
        ('Content-Type', "text/plain"),
    ]
    start_response('200 OK', headers)
    Greeting("debug")
    return [response]

Comment 5 Miro Hrončok 2021-12-09 11:27:20 UTC
Interesting:

(Pdb) Greeting
<class '_mod_wsgi_bd7b5f024fe2438c2a653d79710eba24.Greeting'>
(Pdb) Greeting.__init__
<function tweak_init.<locals>.new_init at 0x7ff79c0d7640>
(Pdb) Greeting.__init__(object(), "a")
*** AttributeError: 'object' object has no attribute 'text'
(Pdb) Greeting("a")
*** TypeError: Greeting() takes no arguments
(Pdb) Greeting.__call__("a")
*** TypeError: Greeting() takes no arguments


Seems like the __init__ attribute is set, but __call__ keep calling the previous one.

Comment 6 Miro Hrončok 2021-12-09 11:58:46 UTC
This behaves similarly:


class Greeting:
    pass

def new_init(inst, name):
    inst.text = f"Hello, {name}\n"
 
Greeting.__init__ = new_init



(Pdb) Greeting("a")
*** TypeError: Greeting() takes no arguments

(Pdb) Greeting.__init__(object(), "a")
*** AttributeError: 'object' object has no attribute 'text'

Comment 7 Miro Hrončok 2021-12-09 12:32:53 UTC
Inspect things it takes the arg:

(Pdb) import inspect
(Pdb) inspect.signature(Greeting)
<Signature (name)>
(Pdb) Greeting("a")
*** TypeError: Greeting() takes no arguments



Setting any __init__ before replacing it works:

class Greeting():
    __init__ = None  # <- define whatever __init__ here and it works

def new_init(inst, name):
    inst.text = f"Hello, {name}\n"

Greeting.__init__ = new_init

Comment 9 Victor Stinner 2021-12-09 13:17:25 UTC
I don't understand how mod_wsgi is involved the issue. Moreover, the following code works properly with Python 3.9 and 3.10, and it doesn't seem to be a Python 3.10 regression:
---
def new_init(inst, name):
    inst.text = f"Hello, {name}\n"
    print("new_init")

def tweak_init(cls):
    cls.__init__ = new_init
    return cls

@tweak_init
class Greeting:
    pass

class Greeting2:
    pass
Greeting2.__init__ = new_init

print(Greeting("name"))
print(Greeting2("name"))
---

Python 3.9 and 3.10 output:
---
new_init
<__main__.Greeting object at 0x7fd5db4b4eb0>
new_init
<__main__.Greeting2 object at 0x7fd5db4b4eb0>
---

Comment 10 Miro Hrončok 2021-12-09 13:21:41 UTC
It breaks in mod_wsgi only.

Comment 11 Miro Hrončok 2021-12-09 14:07:06 UTC
For the record, Python 3.10.1 does not change this.

Comment 12 Miro Hrončok 2021-12-10 11:49:19 UTC
This is sub-interpreter related:


>>> import _testcapi
>>> code = r"""class Greeting():
...     pass
... 
... def new_init(inst, name):
...     inst.text = f"Hello, {name}\n"
... 
... Greeting.__init__ = new_init
... 
... Greeting("Miro")"""
>>> _testcapi.run_in_subinterp(code)
Traceback (most recent call last):
  File "<string>", line 9, in <module>
TypeError: Greeting() takes no arguments
-1

Comment 13 Miro Hrončok 2021-12-10 12:01:24 UTC
Reported upstream as https://bugs.python.org/issue46034

Comment 14 Victor Stinner 2021-12-10 12:42:08 UTC
> Reported upstream as https://bugs.python.org/issue46034

I confirm that it's a Python 3.10 regression, I marked issue46034 as a duplicate of: https://bugs.python.org/issue46006

For the mod_wsgi case, a workaround is to avoid subinterpreters by changing the mod_wsgi configuration. Sorry, I'm not sure when and how mod_wsgi decides to use subinterpreters or not. It seems like "WSGIApplicationGroup %{GLOBAL}" directive should be used somehow to run all code in the same interpreter.

See for example:

* https://modwsgi.readthedocs.io/en/develop/user-guides/processes-and-threading.html#python-sub-interpreters
* https://modwsgi.readthedocs.io/en/develop/user-guides/checking-your-installation.html#sub-interpreter-being-used

Comment 15 Miro Hrončok 2022-01-10 14:12:12 UTC
If I provide you a build with this fix applied https://github.com/python/cpython/pull/30425 -- will you be able to test it?

Comment 16 Miro Hrončok 2022-01-10 14:28:07 UTC
Rawhide pull request: https://src.fedoraproject.org/rpms/python3.10/pull-request/90
Fedora 35 pull request: https://src.fedoraproject.org/rpms/python3.10/pull-request/91

If the scratch build succeeds there, you can download it and test it. Thanks

Comment 17 Kevin Fenzi 2022-01-10 21:30:03 UTC
I guess I never added myself to cc here. ;) 

So, I pulled the scratch build and it seems to work in stg koji. Many thanks!

Comment 18 Fedora Update System 2022-01-12 19:55:11 UTC
FEDORA-2022-23b8acee83 has been submitted as an update to Fedora 36. https://bodhi.fedoraproject.org/updates/FEDORA-2022-23b8acee83

Comment 19 Fedora Update System 2022-01-12 19:58:19 UTC
FEDORA-2022-23b8acee83 has been pushed to the Fedora 36 stable repository.
If problem still persists, please make note of it in this bug report.

Comment 20 Miro Hrončok 2022-01-12 20:02:12 UTC
Reopening for F35.

Comment 21 Fedora Update System 2022-01-12 20:03:37 UTC
FEDORA-2022-2303439072 has been submitted as an update to Fedora 35. https://bodhi.fedoraproject.org/updates/FEDORA-2022-2303439072

Comment 22 Fedora Update System 2022-01-13 01:11:59 UTC
FEDORA-2022-2303439072 has been pushed to the Fedora 35 testing repository.
Soon you'll be able to install the update with the following command:
`sudo dnf upgrade --enablerepo=updates-testing --advisory=FEDORA-2022-2303439072`
You can provide feedback for this update here: https://bodhi.fedoraproject.org/updates/FEDORA-2022-2303439072

See also https://fedoraproject.org/wiki/QA:Updates_Testing for more information on how to test updates.

Comment 23 Fedora Update System 2022-01-16 01:18:43 UTC
FEDORA-2022-2303439072 has been pushed to the Fedora 35 stable repository.
If problem still persists, please make note of it in this bug report.


Note You need to log in before you can comment on or make changes to this bug.