Fix applying Pygments syntax highlighting to diffs on Python 3.

Review Request #10708 — Created Sept. 10, 2019 and submitted


Review Board


A string type compatibility issue on Python 3 resulted in a failure to
apply syntax highlighting using Pygments. After generating the
syntax-highlighted HTML (which is returned as a Unicode string), we
attempted to split the lines using split_line_endings() (which
required a byte string), and the type discrepency resulted in an
exception that was caught and ignored, causing the diff viewer to fall
back on a non-highlighted view.

To address this, split_line_endings() has been made compatible with
both byte and Unicode strings, and we no longer blindly ignore all
exceptions. Now, the only exception that is ignored is Pygments'
ClassNotFound, which is used when it can't determine a lexer for the
file type.

As part of the fix for exception handling, the code has been reworked to
both handle exceptions and to look up files in the highlighting
blacklist entirely within _apply_pygments(), to simplify the
requirements for the caller and to ensure that if the left-hand side of
the diff fails to syntax-highlight, it won't prevent the right-hand side
from trying.

Unit tests have been added to ensure that both split_line_endings()
and _apply_pygments() will do their jobs correctly, and that the final
chunk results from RawDiffChunkGenerator contain the resulting lines
we expect.

All unit tests passed on Python 2.7, 3.5, 3.6, and 3.7.

Verified that syntax highlighting was functioning properly.

  1. Ship It!
Review request changed
Change Summary:
Pushed to release-4.0.x (4a20674)