Require byte strings for diff chunk generation and use Unicode for differs.

Review Request #10500 — Created April 2, 2019 and submitted — Latest diff uploaded

Information

Review Board
release-4.0.x

Reviewers

The process of generating diffs requires different types of strings at
different stages. The differ itself can technically work with either
byte or Unicode strings and doesn't really care, but when bringing the
"interesting lines" regexes into the process, the string types suddenly
matter. Our code expects the strings to be normalized to Unicode at this
stage, so that there's a consistent format to diff (without worrying
about mismatched encodings). However, we were passing byte strings in
some unit tests, which wasn't consistent with normal usage and caused
problems on Python 3. Those have been fixed to be Unicode.

The diff chunk generator, on the other hand, expects byte strings. It
takes these, normalizes them, converts to Unicode, and then hands them
off to the differ. To ensure it's getting the format it requires, it now
checks the types coming in during construction so that there's no
accidental cases of Unicode strings coming in.

Unit tests pass on Python 2.7 and 3.7 (with other in-progress changes).

Tested viewing a handful of diffs with Emoji and other non-ASCII content,
with and without a primed cache.

Commits

Files

    Loading...