Better manage memory usage of condensediffs.

Review Request #5870 — Created May 24, 2014 and submitted — Latest diff uploaded

Information

Review Board
release-2.0.x
eb69fb2...

Reviewers

condensediffs had some memory issues. Even though we were using
QuerySet.iterator, it seems memory was still increasing far too much.

To work around this, we're using a couple tricks that we used in
loaddb and dumpdb. We're only operating on batches of 200 diffs at a
time, and are resetting queries and garbage-collecting after each batch.

The query resets shouldn't impact production installs, since DEBUG
should be False, but it's a precaution. We're also forcing DEBUG to be
False in the management command as well.

With these changes, memory still increases over time, to a degree, but
seems to stabilize. This is with a sample set of over 8200 diffs.

Added some debug information to check process memory usage before and after.

Before, memory was steadily rising in usage (even with DEBUG = False).

After, memory rose for a bit and then stayed pretty steady, without growing
unexpectedly large.