• 
      

    Fix issues with multi-byte encodings in content sections.

    Review Request #11714 — Created July 10, 2021 and submitted — Latest diff uploaded

    Information

    DiffX
    master

    Reviewers

    The original implementations of the reader and writer didn't properly
    handle multi-byte encodings, such as UTF-16 or UTF-32 strings.

    The reason for this came down to newlines. We were looking for and
    adding newlines based on their 8-bit representation, assuming that all
    content would end in those, or that we could add them to the end of each
    line while splitting. This didn't actually work, and would result in
    failures in multi-byte encodings.

    A few things had to be done to address this:

    1. Some design flaws in the text utilities, which made assumptions on
      lengths, offsets, and the presence of newline characters, had to be
      fixed.

    2. We no longer store our newline constants as byte strings, but rather
      Unicode strings. They're then encoded as needed to ensure that
      they're in a compatible format. BOMs are stripped from these, to
      avoid further corruption of content.

    3. The order in which we encode, decode, and process strings for content
      sections has changed a bit. We add on a missing newline before
      encoding, rather than after. We encode or decode newlines at the same
      time as the content strings.

    Unit tests were updated to check different multi-byte encodings for all
    the content sections, making sure they are written and parsed correctly.

    All unit tests pass on Python 2 and 3.

    Commits

    Files