Improve pyflakes reporting and fix processing all types of output.

Review Request #11508 — Created March 10, 2021 and submitted

Information

ReviewBot
release-3.0.x

Reviewers

The pyflakes tool wasn't handling all types of output correctly. Syntax
errors and unknown errors reported by pyflakes would trigger a crash,
which we would occasionally see in some uploaded code. There were also
issues where source code could manifest as errors that couldn't be
resolved.

This change redoes the processing to handle the various forms of output.
Pyflakes sends lintian errors to stdout and syntax/unknown errors to
stderr, in different formats.

Lintian errors are one line of content (the filename, line number, and
message).

Unknown errors are one line of content (the filename and message).

Syntax errors will show an info line (filename, line number, possible
offset, and message), a line of code, and a possible offset indicator
line (a ^ under the offending character).

Now that we can detect each type, we can make smarter assumptions about
what we're looking at. The syntax error case is the most complicated,
but it's parsable now with the new code.

To separate out stdout and stderr, execute() had to be updated to take
a return_errors option. This logic exactly matches the equivalent in
RBTools (which we should probably just wrap in a future change).

To ensure everything works as expected, there's now unit tests for
pyflakes that throws the various forms of output at the parser.

Unit tests pass.

This needs some manual testing in production.

Summary ID
Improve pyflakes reporting and fix processing all types of output.
The pyflakes tool wasn't handling all types of output correctly. Syntax errors and unknown errors reported by pyflakes would trigger a crash, which we would occasionally see in some uploaded code. There were also issues where source code could manifest as errors that couldn't be resolved. This change redoes the processing to handle the various forms of output. Pyflakes sends lintian errors to stdout and syntax/unknown errors to stderr, in different formats. Lintian errors are one line of content (the filename, line number, and message). Unknown errors are one line of content (the filename and message). Syntax errors will show an info line (filename, line number, possible offset, and message), a line of code, and a possible offset indicator line (a `^` under the offending character). Now that we can detect each type, we can make smarter assumptions about what we're looking at. The syntax error case is the most complicated, but it's parsable now with the new code. To separate out stdout and stderr, `execute()` had to be updated to take a `return_errors` option. This logic exactly matches the equivalent in RBTools (which we should probably just wrap in a future change). To ensure everything works as expected, there's now unit tests for pyflakes that throws the various forms of output at the parser.
63e6e285a87d1debed91ba5c1745cde0c81e89cb
Description From Last Updated

E501 line too long (80 > 79 characters)

reviewbotreviewbot
Checks run (1 failed, 1 succeeded)
flake8 failed.
JSHint passed.

flake8

david
  1. Ship It!
  2. 
      
chipx86
Review request changed
Status:
Completed
Change Summary:
Pushed to release-3.0.x (45a758d)