Better manage memory usage of condensediffs.
Review Request #5870 — Created May 24, 2014 and submitted — Latest diff uploaded
condensediffs had some memory issues. Even though we were using
QuerySet.iterator, it seems memory was still increasing far too much.To work around this, we're using a couple tricks that we used in
loaddb and dumpdb. We're only operating on batches of 200 diffs at a
time, and are resetting queries and garbage-collecting after each batch.The query resets shouldn't impact production installs, since DEBUG
should be False, but it's a precaution. We're also forcing DEBUG to be
False in the management command as well.With these changes, memory still increases over time, to a degree, but
seems to stabilize. This is with a sample set of over 8200 diffs.
Added some debug information to check process memory usage before and after.
Before, memory was steadily rising in usage (even with DEBUG = False).
After, memory rose for a bit and then stayed pretty steady, without growing
unexpectedly large.