Better manage memory usage of condensediffs.
Review Request #5870 — Created May 24, 2014 and submitted
condensediffs had some memory issues. Even though we were using
QuerySet.iterator, it seems memory was still increasing far too much.To work around this, we're using a couple tricks that we used in
loaddb and dumpdb. We're only operating on batches of 200 diffs at a
time, and are resetting queries and garbage-collecting after each batch.The query resets shouldn't impact production installs, since DEBUG
should be False, but it's a precaution. We're also forcing DEBUG to be
False in the management command as well.With these changes, memory still increases over time, to a degree, but
seems to stabilize. This is with a sample set of over 8200 diffs.
Added some debug information to check process memory usage before and after.
Before, memory was steadily rising in usage (even with DEBUG = False).
After, memory rose for a bit and then stayed pretty steady, without growing
unexpectedly large.
Description | From | Last Updated |
---|---|---|
I don't think this does the right thing (it seems to iterate over items 0-199 over and over). |
david |
-
This is a review from Review Bot. Tool: Pyflakes Processed Files: reviewboard/diffviewer/management/commands/condensediffs.py reviewboard/diffviewer/managers.py Ignored Files:
- Change Summary:
-
- Added some doc comments.
- Renamed
filediffs
tounmigrated_filediffs
. - Switch the
while
loop to afor
loop.
- Commit:
-
05a5f75b56ec6b51fd8b114ac60539390b934606eb69fb2ed2be744dbb8ce33082894f9284055808
-
This is a review from Review Bot. Tool: PEP8 Style Checker Processed Files: reviewboard/diffviewer/management/commands/condensediffs.py reviewboard/diffviewer/managers.py Ignored Files: