• 
      

    Enhance our cache_memoize support to allow for use with generators.

    Review Request #7162 — Created April 2, 2015 and submitted

    Information

    Djblets
    release-0.9.x
    4d258fa...

    Reviewers

    A new cache_memoize_iter function has been added that allows the
    data generation function to be a generator. Unlike with cache_memoize,
    the results won't need to be gathered up into a list before being handed
    down to the caller, and instead, cache_memoize_iter will simply
    store the cached data as items are yielded.
    
    When data is already cached, it still functions as a generator, as
    efficiently as possible. All the chunk data is still loaded and
    optionally decompressed up-front, as before, since it's measurably
    faster to do it then rather than on-demand. The unpickling, however, is
    on-demand. We only unpickle when we're ready to parse out the next item
    from cached generator.
    
    All data passed to cache_memoize_iter is assumed to be "large
    data," unlike with cache_memoize, where that is optional and up to the
    caller.
    
    Since that function now exclusively handles large data, cache_memoize
    now uses it for all large data, passing in a single-item list with the
    data to cache.
    
    There are also some optimizations in here for making better use of
    StringIO instead of combining/splitting byte strings.

    Djblets and Review Board unit tests pass.

    Browsed around a bunch of review requests and diffs, looking to see if anything
    was broken. Everything worked fine. Verified in a different browser that the
    data was properly cached in memory.

    Description From Last Updated

    local variable 'remaining_len' is assigned to but never used

    reviewbotreviewbot

    Col: 9 E265 block comment should start with '# '

    reviewbotreviewbot

    local variable 'result' is assigned to but never used

    reviewbotreviewbot

    Col: 9 E265 block comment should start with '# '

    reviewbotreviewbot

    Since this is just a debug log message, I don't think it's worth handling pluralization. I'd just keep the ', …

    daviddavid

    If you leave off the [] it will be more efficient (generator expression vs. list comprehension).

    daviddavid

    Can you update the quotes on this string too?

    daviddavid
    reviewbot
    1. Tool: Pyflakes
      Processed Files:
          djblets/cache/tests.py
          djblets/cache/backend.py
      
      
      
      Tool: PEP8 Style Checker
      Processed Files:
          djblets/cache/tests.py
          djblets/cache/backend.py
      
      
    2. djblets/cache/backend.py (Diff revision 1)
       
       
      Show all issues
       local variable 'remaining_len' is assigned to but never used
      
    3. djblets/cache/tests.py (Diff revision 1)
       
       
      Show all issues
      Col: 9
       E265 block comment should start with '# '
      
    4. djblets/cache/tests.py (Diff revision 1)
       
       
      Show all issues
       local variable 'result' is assigned to but never used
      
    5. djblets/cache/tests.py (Diff revision 1)
       
       
      Show all issues
      Col: 9
       E265 block comment should start with '# '
      
    6. 
        
    chipx86
    reviewbot
    1. Tool: PEP8 Style Checker
      Processed Files:
          djblets/cache/tests.py
          djblets/cache/backend.py
      
      
      
      Tool: Pyflakes
      Processed Files:
          djblets/cache/tests.py
          djblets/cache/backend.py
      
      
    2. 
        
    david
    1. 
        
    2. djblets/cache/backend.py (Diff revision 2)
       
       
       
       
       
      Show all issues

      Since this is just a debug log message, I don't think it's worth handling pluralization. I'd just keep the ', '.join() form and drop the singular.

    3. djblets/cache/backend.py (Diff revision 2)
       
       
       
       
       
      Show all issues

      If you leave off the [] it will be more efficient (generator expression vs. list comprehension).

    4. djblets/cache/tests.py (Diff revision 2)
       
       
      Show all issues

      Can you update the quotes on this string too?

    5. 
        
    chipx86
    Review request changed
    Status:
    Completed
    Change Summary:
    Pushed to release-0.9.x (d42ede6)