• 
      

    Add support for Python 3.11.

    Review Request #12498 — Created Aug. 3, 2022 and submitted

    Information

    kgb
    master

    Reviewers

    kgb

    Python 3.11 introduced some internal changes to how functions with
    closures and exceptions worked. This led to segfaults when calling our
    forwarding functions, as our closure state didn't match what Python
    internally expected; and failures representing stack traces, because
    internal cached state was never rebuilt.

    There's a lot more knowledge internally now in Python 3.11 when it comes
    to closures. If a FunctionType and CodeType isn't set up properly,
    that internal Python state crashes. The state that matters can't
    actually be set by us after defining the function, so it needs to be
    properly prepared up-front.

    We conditionally do this now by generating the forwarding function code
    string in such a way where the forwarding function is inside of another
    function. The outer function provides all the variables the inner
    function needs in its scope, and the inner forwarding function
    references them just enough to satisfy Python.

    The generated outer function is then called, returning the inner
    forwarding function, which is properly set up for Python 3.11's closer
    semantics.

    Exception behavior has also changed. Python 3.11 has improved exception
    support, and to facilitate this, the CodeType now keeps track of state
    that helps map exceptions to lines, as well as marking ranges of lines
    and columns to bytecode. The former is just another argument that goes
    into CodeType, but the latter is more internal (accessible, but not
    modifiable, by code_type.co_positions()) It seems we can't correctly
    set it when setting up the CodeType.

    We now use code_type.replace(...) on the forwarding function's
    __code__, instead of constructing a CodeType(). This has been
    available since Python 3.8, but we've never used it. It's historically
    been equivalent to constructing a CodeType() with a function's
    defaults, allowing the caller to selectively replace only what's needed.
    It now seems to properly rebuild some state needed to ensure the cde
    positions all map correctly.

    Given the increased complexity in building the forwarding function and
    the CodeType, these blocks of logic have been moved into new utility
    methods.

    With these fixes, all unit tests (including a couple new ones) pass on
    Python 3.11. This is now listed in our documentation and in our
    packaging metadata.

    Unit tests pass on Python 2.7 through 3.11.

    Performed complete runs of the Review Board and Djblets test suites
    using Python 2.7, 3.6, 3.10, and 3.11.

    Summary ID
    Add support for Python 3.11.
    Python 3.11 introduced some internal changes to how functions with closures and exceptions worked. This led to segfaults when calling our forwarding functions, as our closure state didn't match what Python internally expected; and failures representing stack traces, because internal cached state was never rebuilt. There's a lot more knowledge internally now in Python 3.11 when it comes to closures. If a `FunctionType` and `CodeType` isn't set up properly, that internal Python state crashes. The state that matters can't actually be set by us after defining the function, so it needs to be properly prepared up-front. We conditionally do this now by generating the forwarding function code string in such a way where the forwarding function is inside of another function. The outer function provides all the variables the inner function needs in its scope, and the inner forwarding function references them just enough to satisfy Python. The generated outer function is then called, returning the inner forwarding function, which is properly set up for Python 3.11's closer semantics. Exception behavior has also changed. Python 3.11 has improved exception support, and to facilitate this, the `CodeType` now keeps track of state that helps map exceptions to lines, as well as marking ranges of lines and columns to bytecode. The former is just another argument that goes into `CodeType`, but the latter is more internal (accessible, but not modifiable, by `code_type.co_positions()`) It seems we can't correctly set it when setting up the `CodeType`. We now use `code_type.replace(...)` on the forwarding function's `__code__`, instead of constructing a `CodeType()`. This has been available since Python 3.8, but we've never used it. It's historically been equivalent to constructing a `CodeType()` with a function's defaults, allowing the caller to selectively replace only what's needed. It now seems to properly rebuild some state needed to ensure the cde positions all map correctly. Given the increased complexity in building the forwarding function and the `CodeType`, these blocks of logic have been moved into new utility methods. With these fixes, all unit tests (including a couple new ones) pass on Python 3.11. This is now listed in our documentation and in our packaging metadata.
    29800655fe6e8c91ced11c7dda7ba75a35fb2acd
    Description From Last Updated

    too many blank lines (2) Column: 9 Error code: E303

    reviewbotreviewbot

    local variable 'ctx' is assigned to but never used Column: 37 Error code: F841

    reviewbotreviewbot

    local variable 'ctx' is assigned to but never used Column: 37 Error code: F841

    reviewbotreviewbot

    local variable 'c' is assigned to but never used Column: 25 Error code: F841

    reviewbotreviewbot

    local variable 'b' is assigned to but never used Column: 21 Error code: F841

    reviewbotreviewbot
    Checks run (1 failed, 1 succeeded)
    flake8 failed.
    JSHint passed.

    flake8

    chipx86
    chipx86
    david
    1. Ship It!
    2. 
        
    chipx86
    Review request changed
    Status:
    Completed
    Change Summary:
    Pushed to master (57947cc)