The original implementations of the reader and writer didn't properly
handle multi-byte encodings, such as UTF-16 or UTF-32 strings.
The reason for this came down to newlines. We were looking for and
adding newlines based on their 8-bit representation, assuming that all
content would end in those, or that we could add them to the end of each
line while splitting. This didn't actually work, and would result in
failures in multi-byte encodings.
A few things had to be done to address this:
-
Some design flaws in the text utilities, which made assumptions on
lengths, offsets, and the presence of newline characters, had to be
fixed.
-
We no longer store our newline constants as byte strings, but rather
Unicode strings. They're then encoded as needed to ensure that
they're in a compatible format. BOMs are stripped from these, to
avoid further corruption of content.
-
The order in which we encode, decode, and process strings for content
sections has changed a bit. We add on a missing newline before
encoding, rather than after. We encode or decode newlines at the same
time as the content strings.
Unit tests were updated to check different multi-byte encodings for all
the content sections, making sure they are written and parsed correctly.
