Enhance our cache_memoize support to allow for use with generators.
Review Request #7162 — Created April 2, 2015 and submitted
A new cache_memoize_iter function has been added that allows the data generation function to be a generator. Unlike with cache_memoize, the results won't need to be gathered up into a list before being handed down to the caller, and instead, cache_memoize_iter will simply store the cached data as items are yielded. When data is already cached, it still functions as a generator, as efficiently as possible. All the chunk data is still loaded and optionally decompressed up-front, as before, since it's measurably faster to do it then rather than on-demand. The unpickling, however, is on-demand. We only unpickle when we're ready to parse out the next item from cached generator. All data passed to cache_memoize_iter is assumed to be "large data," unlike with cache_memoize, where that is optional and up to the caller. Since that function now exclusively handles large data, cache_memoize now uses it for all large data, passing in a single-item list with the data to cache. There are also some optimizations in here for making better use of StringIO instead of combining/splitting byte strings.
Djblets and Review Board unit tests pass.
Browsed around a bunch of review requests and diffs, looking to see if anything
was broken. Everything worked fine. Verified in a different browser that the
data was properly cached in memory.
Description | From | Last Updated |
---|---|---|
local variable 'remaining_len' is assigned to but never used |
reviewbot | |
Col: 9 E265 block comment should start with '# ' |
reviewbot | |
local variable 'result' is assigned to but never used |
reviewbot | |
Col: 9 E265 block comment should start with '# ' |
reviewbot | |
Since this is just a debug log message, I don't think it's worth handling pluralization. I'd just keep the ', … |
david | |
If you leave off the [] it will be more efficient (generator expression vs. list comprehension). |
david | |
Can you update the quotes on this string too? |
david |
- Change Summary:
-
Fixed some silly mistakes that Review Bot was kind enough to inform me about.
- Commit:
-
caac7e9a21478068414b7b31b9bcd3fb9a5e0e504d258fa06b1d0952a2a08724e933c3ab26b62889
- Diff:
-
Revision 2 (+514 -66)
-
Tool: PEP8 Style Checker Processed Files: djblets/cache/tests.py djblets/cache/backend.py Tool: Pyflakes Processed Files: djblets/cache/tests.py djblets/cache/backend.py