Enhance our cache_memoize support to allow for use with generators.

Review Request #7162 — Created April 2, 2015 and submitted

Information

Djblets
release-0.9.x
4d258fa...

Reviewers

A new cache_memoize_iter function has been added that allows the
data generation function to be a generator. Unlike with cache_memoize,
the results won't need to be gathered up into a list before being handed
down to the caller, and instead, cache_memoize_iter will simply
store the cached data as items are yielded.

When data is already cached, it still functions as a generator, as
efficiently as possible. All the chunk data is still loaded and
optionally decompressed up-front, as before, since it's measurably
faster to do it then rather than on-demand. The unpickling, however, is
on-demand. We only unpickle when we're ready to parse out the next item
from cached generator.

All data passed to cache_memoize_iter is assumed to be "large
data," unlike with cache_memoize, where that is optional and up to the
caller.

Since that function now exclusively handles large data, cache_memoize
now uses it for all large data, passing in a single-item list with the
data to cache.

There are also some optimizations in here for making better use of
StringIO instead of combining/splitting byte strings.

Djblets and Review Board unit tests pass.

Browsed around a bunch of review requests and diffs, looking to see if anything
was broken. Everything worked fine. Verified in a different browser that the
data was properly cached in memory.

Description From Last Updated

local variable 'remaining_len' is assigned to but never used

reviewbotreviewbot

Col: 9 E265 block comment should start with '# '

reviewbotreviewbot

local variable 'result' is assigned to but never used

reviewbotreviewbot

Col: 9 E265 block comment should start with '# '

reviewbotreviewbot

Since this is just a debug log message, I don't think it's worth handling pluralization. I'd just keep the ', …

daviddavid

If you leave off the [] it will be more efficient (generator expression vs. list comprehension).

daviddavid

Can you update the quotes on this string too?

daviddavid
reviewbot
  1. Tool: Pyflakes
    Processed Files:
        djblets/cache/tests.py
        djblets/cache/backend.py
    
    
    
    Tool: PEP8 Style Checker
    Processed Files:
        djblets/cache/tests.py
        djblets/cache/backend.py
    
    
  2. djblets/cache/backend.py (Diff revision 1)
     
     
    Show all issues
     local variable 'remaining_len' is assigned to but never used
    
  3. djblets/cache/tests.py (Diff revision 1)
     
     
    Show all issues
    Col: 9
     E265 block comment should start with '# '
    
  4. djblets/cache/tests.py (Diff revision 1)
     
     
    Show all issues
     local variable 'result' is assigned to but never used
    
  5. djblets/cache/tests.py (Diff revision 1)
     
     
    Show all issues
    Col: 9
     E265 block comment should start with '# '
    
  6. 
      
chipx86
reviewbot
  1. Tool: PEP8 Style Checker
    Processed Files:
        djblets/cache/tests.py
        djblets/cache/backend.py
    
    
    
    Tool: Pyflakes
    Processed Files:
        djblets/cache/tests.py
        djblets/cache/backend.py
    
    
  2. 
      
david
  1. 
      
  2. djblets/cache/backend.py (Diff revision 2)
     
     
     
     
     
    Show all issues

    Since this is just a debug log message, I don't think it's worth handling pluralization. I'd just keep the ', '.join() form and drop the singular.

  3. djblets/cache/backend.py (Diff revision 2)
     
     
     
     
     
    Show all issues

    If you leave off the [] it will be more efficient (generator expression vs. list comprehension).

  4. djblets/cache/tests.py (Diff revision 2)
     
     
    Show all issues

    Can you update the quotes on this string too?

  5. 
      
chipx86
Review request changed
Status:
Completed
Change Summary:
Pushed to release-0.9.x (d42ede6)