The problem has to do when working with large amounts of data. File attachment uploads, for instance, will take a lot more memory with string concatenation, resulting in multiple copies of the file attachment data in memory at one time, just due to the way that Python works.

Say we upload two files that's 5MB and 10MB in size, respectively. During the loop in which we set the first file, we'll end up with a resulting string containing all the previous form-data content and the 5MB file. The next string concatenation will allocate another string containing the size of the original string + the new content, meaning another ~5MB string in memory, alongside the original. The reference then gets set to the new one, and the old one is flagged for garbage collection. We now have ~10MB in memory. The next bit of data that gets added will allocate another ~5MB string, giving us ~15MB in memory. And then ~20MB. When we start building the payload for the new file, that begins to grow further.

At some point during this, garbage collection is likely to kick in and start freeing up older strings, but given the sizes, this isn't exactly instantaneous, and it still means the process has allocated much more memory than needed.

Going the BytesIO route, we have an efficient way of storing this data without all the extra allocations. It's faster and more memory-efficient.

Because of that, the original mechanism for building these payloads must remain.

Also, while not as big a deal, it's probably not worth defining a new function inline. This has to be generated every time we call encode_multipart_formdata, and I don't think it's needed.

I'd really like to use the original code for this function, converted for BytesIO, instead of the new mechanism. The old method is well-tested.

Refactor encode_multipart_formdata and tests to be easier to read and maintain

Information

Reviewers