Add support for Python 3.11.
Review Request #12498 — Created Aug. 3, 2022 and submitted
Python 3.11 introduced some internal changes to how functions with
closures and exceptions worked. This led to segfaults when calling our
forwarding functions, as our closure state didn't match what Python
internally expected; and failures representing stack traces, because
internal cached state was never rebuilt.There's a lot more knowledge internally now in Python 3.11 when it comes
to closures. If aFunctionType
andCodeType
isn't set up properly,
that internal Python state crashes. The state that matters can't
actually be set by us after defining the function, so it needs to be
properly prepared up-front.We conditionally do this now by generating the forwarding function code
string in such a way where the forwarding function is inside of another
function. The outer function provides all the variables the inner
function needs in its scope, and the inner forwarding function
references them just enough to satisfy Python.The generated outer function is then called, returning the inner
forwarding function, which is properly set up for Python 3.11's closer
semantics.Exception behavior has also changed. Python 3.11 has improved exception
support, and to facilitate this, theCodeType
now keeps track of state
that helps map exceptions to lines, as well as marking ranges of lines
and columns to bytecode. The former is just another argument that goes
intoCodeType
, but the latter is more internal (accessible, but not
modifiable, bycode_type.co_positions()
) It seems we can't correctly
set it when setting up theCodeType
.We now use
code_type.replace(...)
on the forwarding function's
__code__
, instead of constructing aCodeType()
. This has been
available since Python 3.8, but we've never used it. It's historically
been equivalent to constructing aCodeType()
with a function's
defaults, allowing the caller to selectively replace only what's needed.
It now seems to properly rebuild some state needed to ensure the cde
positions all map correctly.Given the increased complexity in building the forwarding function and
theCodeType
, these blocks of logic have been moved into new utility
methods.With these fixes, all unit tests (including a couple new ones) pass on
Python 3.11. This is now listed in our documentation and in our
packaging metadata.
Unit tests pass on Python 2.7 through 3.11.
Performed complete runs of the Review Board and Djblets test suites
using Python 2.7, 3.6, 3.10, and 3.11.
Summary | ID |
---|---|
29800655fe6e8c91ced11c7dda7ba75a35fb2acd |
Description | From | Last Updated |
---|---|---|
too many blank lines (2) Column: 9 Error code: E303 |
reviewbot | |
local variable 'ctx' is assigned to but never used Column: 37 Error code: F841 |
reviewbot | |
local variable 'ctx' is assigned to but never used Column: 37 Error code: F841 |
reviewbot | |
local variable 'c' is assigned to but never used Column: 25 Error code: F841 |
reviewbot | |
local variable 'b' is assigned to but never used Column: 21 Error code: F841 |
reviewbot |
- Change Summary:
-
Fixed flake8 issues.
- Commits:
-
Summary ID 10e9a5940a77793243023570bf85d9cf3f57cab3 19ff0f01fb4c89b2f116228356d017e117b12465 - Diff:
-
Revision 2 (+698 -114)
Checks run (2 succeeded)
- Change Summary:
-
-
Removed some leftover code from debugging and development.
-
Fixed typos.
-
- Commits:
-
Summary ID 19ff0f01fb4c89b2f116228356d017e117b12465 29800655fe6e8c91ced11c7dda7ba75a35fb2acd - Diff:
-
Revision 3 (+692 -112)