Fix issues with multi-byte encodings in content sections.

Review Request #11714 — Created July 10, 2021 and submitted — Latest diff uploaded

Information

DiffX
master

Reviewers

The original implementations of the reader and writer didn't properly
handle multi-byte encodings, such as UTF-16 or UTF-32 strings.

The reason for this came down to newlines. We were looking for and
adding newlines based on their 8-bit representation, assuming that all
content would end in those, or that we could add them to the end of each
line while splitting. This didn't actually work, and would result in
failures in multi-byte encodings.

A few things had to be done to address this:

  1. Some design flaws in the text utilities, which made assumptions on
    lengths, offsets, and the presence of newline characters, had to be
    fixed.

  2. We no longer store our newline constants as byte strings, but rather
    Unicode strings. They're then encoded as needed to ensure that
    they're in a compatible format. BOMs are stripped from these, to
    avoid further corruption of content.

  3. The order in which we encode, decode, and process strings for content
    sections has changed a bit. We add on a missing newline before
    encoding, rather than after. We encode or decode newlines at the same
    time as the content strings.

Unit tests were updated to check different multi-byte encodings for all
the content sections, making sure they are written and parsed correctly.

All unit tests pass on Python 2 and 3.

Commits

Files