• 
      

    Require byte strings for diff chunk generation and use Unicode for differs.

    Review Request #10500 — Created April 2, 2019 and submitted — Latest diff uploaded

    Information

    Review Board
    release-4.0.x

    Reviewers

    The process of generating diffs requires different types of strings at
    different stages. The differ itself can technically work with either
    byte or Unicode strings and doesn't really care, but when bringing the
    "interesting lines" regexes into the process, the string types suddenly
    matter. Our code expects the strings to be normalized to Unicode at this
    stage, so that there's a consistent format to diff (without worrying
    about mismatched encodings). However, we were passing byte strings in
    some unit tests, which wasn't consistent with normal usage and caused
    problems on Python 3. Those have been fixed to be Unicode.

    The diff chunk generator, on the other hand, expects byte strings. It
    takes these, normalizes them, converts to Unicode, and then hands them
    off to the differ. To ensure it's getting the format it requires, it now
    checks the types coming in during construction so that there's no
    accidental cases of Unicode strings coming in.

    Unit tests pass on Python 2.7 and 3.7 (with other in-progress changes).

    Tested viewing a handful of diffs with Emoji and other non-ASCII content,
    with and without a primed cache.

    Commits

    Files