• 
      

    Improve pyflakes reporting and fix processing all types of output.

    Review Request #11508 — Created March 10, 2021 and submitted

    Information

    ReviewBot
    release-3.0.x

    Reviewers

    The pyflakes tool wasn't handling all types of output correctly. Syntax
    errors and unknown errors reported by pyflakes would trigger a crash,
    which we would occasionally see in some uploaded code. There were also
    issues where source code could manifest as errors that couldn't be
    resolved.

    This change redoes the processing to handle the various forms of output.
    Pyflakes sends lintian errors to stdout and syntax/unknown errors to
    stderr, in different formats.

    Lintian errors are one line of content (the filename, line number, and
    message).

    Unknown errors are one line of content (the filename and message).

    Syntax errors will show an info line (filename, line number, possible
    offset, and message), a line of code, and a possible offset indicator
    line (a ^ under the offending character).

    Now that we can detect each type, we can make smarter assumptions about
    what we're looking at. The syntax error case is the most complicated,
    but it's parsable now with the new code.

    To separate out stdout and stderr, execute() had to be updated to take
    a return_errors option. This logic exactly matches the equivalent in
    RBTools (which we should probably just wrap in a future change).

    To ensure everything works as expected, there's now unit tests for
    pyflakes that throws the various forms of output at the parser.

    Unit tests pass.

    This needs some manual testing in production.

    Summary ID
    Improve pyflakes reporting and fix processing all types of output.
    The pyflakes tool wasn't handling all types of output correctly. Syntax errors and unknown errors reported by pyflakes would trigger a crash, which we would occasionally see in some uploaded code. There were also issues where source code could manifest as errors that couldn't be resolved. This change redoes the processing to handle the various forms of output. Pyflakes sends lintian errors to stdout and syntax/unknown errors to stderr, in different formats. Lintian errors are one line of content (the filename, line number, and message). Unknown errors are one line of content (the filename and message). Syntax errors will show an info line (filename, line number, possible offset, and message), a line of code, and a possible offset indicator line (a `^` under the offending character). Now that we can detect each type, we can make smarter assumptions about what we're looking at. The syntax error case is the most complicated, but it's parsable now with the new code. To separate out stdout and stderr, `execute()` had to be updated to take a `return_errors` option. This logic exactly matches the equivalent in RBTools (which we should probably just wrap in a future change). To ensure everything works as expected, there's now unit tests for pyflakes that throws the various forms of output at the parser.
    63e6e285a87d1debed91ba5c1745cde0c81e89cb
    Description From Last Updated

    E501 line too long (80 > 79 characters)

    reviewbotreviewbot
    Checks run (1 failed, 1 succeeded)
    flake8 failed.
    JSHint passed.

    flake8

    david
    1. Ship It!
    2. 
        
    chipx86
    Review request changed
    Status:
    Completed
    Change Summary:
    Pushed to release-3.0.x (45a758d)