Require byte strings for diff chunk generation and use Unicode for differs.

Review Request #10500 — Created April 2, 2019 and submitted — Latest diff uploaded

Information

Review Board
release-4.0.x

Reviewers

The process of generating diffs requires different types of strings at
different stages. The differ itself can technically work with either
byte or Unicode strings and doesn't really care, but when bringing the
"interesting lines" regexes into the process, the string types suddenly
matter. Our code expects the strings to be normalized to Unicode at this
stage, so that there's a consistent format to diff (without worrying
about mismatched encodings). However, we were passing byte strings in
some unit tests, which wasn't consistent with normal usage and caused
problems on Python 3. Those have been fixed to be Unicode.

The diff chunk generator, on the other hand, expects byte strings. It
takes these, normalizes them, converts to Unicode, and then hands them
off to the differ. To ensure it's getting the format it requires, it now
checks the types coming in during construction so that there's no
accidental cases of Unicode strings coming in.

Unit tests pass on Python 2.7 and 3.7 (with other in-progress changes).

Tested viewing a handful of diffs with Emoji and other non-ASCII content,
with and without a primed cache.

Diff Revision 2

This is not the most recent revision of the diff. The latest diff is revision 3. See what's changed.

orig
1
2
3

Commits

First Last Summary ID Author
Require byte strings for diff chunk generation and use Unicode for differs.
The process of generating diffs requires different types of strings at different stages. The differ itself can technically work with either byte or Unicode strings and doesn't really care, but when bringing the "interesting lines" regexes into the process, the string types suddenly matter. Our code expects the strings to be normalized to Unicode at this stage, so that there's a consistent format to diff (without worrying about mismatched encodings). However, we were passing byte strings in some unit tests, which wasn't consistent with normal usage and caused problems on Python 3. Those have been fixed to be Unicode. The diff chunk generator, on the other hand, expects byte strings. It takes these, normalizes them, converts to Unicode, and then hands them off to the differ. To ensure it's getting the format it requires, it now checks the types coming in during construction so that there's no accidental cases of Unicode strings coming in.
7533a005ec1514ac0f6ac388bdcd857fcb531a55 Christian Hammond
reviewboard/diffviewer/chunk_generator.py
reviewboard/diffviewer/diffutils.py
reviewboard/diffviewer/tests/test_diff_opcode_generator.py
reviewboard/diffviewer/tests/test_interesting_lines.py
reviewboard/diffviewer/tests/test_raw_diff_chunk_generator.py
Loading...