Add API caching support to rbtools.

Review Request #6504 — Created Oct. 25, 2014 and submitted

brennie
RBTools
master
6608, 6584
89b50a0...
rbtools, students

Add the rbtools.api.cache module. This module includes the
APICache class, which allows the results of HTTP GET requests to
be cached locally to achieve faster lookup. The APICache understands
the following headers:

  • Cache-Control
  • Pragma
  • Vary
  • Last-Modified
  • ETag
  • Expires

The APICache is backed by a SQLite database in the user's cache
directory. If the cache cannot be accessed, the APICache will pass
all requests to the specified urlopen (defaulting to
urllib2.urlopen) method.

The APICache allows for a custom urlopen function so that it can
be tested without making actual HTTP requests.

The APICache stores the version of its schema in the database. If
the APICache is instantiated and the versions of the schema in the
class doesn't match the version of the schema in the database, then
the current cache will be invalidted and the schema will be updated.

Add the ActiveState appdirs module (at version 1.4.0) as
rbtools.utils.appdirs. This module is used to find the user cache
directory in platform-dependant ways.

Add the rbtools.api.cache.Response class which emulates the
behaviour of a urllib2 response. This class allows the read
method to be called more than once.

Add the rbtools.api.cache.CachedResponse class, which is the result
of a cach entry being found the the APICache. It also emulates the
behaviour of a urllib2 response.

Add the clear-cache command. This command manually clears the API
cache.

Added unit tests for the APICache.

Unit tests pass.

The following commands were tested and were successful:

  • rbt status
  • rbt post
  • rbt diff
  • rbt patch
  • rbt close
  • rbt clear-cache

Upon clearing the cache with clear-cache, a new cache is created.
The first run of a command always creates cache entries (where
appropriate, i.e., not for PUT or POST requests).

The /api/ endpoint is successfully cached and the cached version is
used.

When using the patch command for the second time on the same review
request, if the diff has not been updated, then the cached version of
the /api/review-requests/<id>/diff/<diff_revision>/ endpoint is
used.

This review request was posted with the APICache enabled.

  • 0
  • 0
  • 82
  • 684
  • 766
Description From Last Updated
reviewbot
  1. Tool: Pyflakes
    Processed Files:
        rbtools/api/transport/sync.py
        rbtools/api/request.py
        rbtools/utils/appdirs.py
        rbtools/api/cache.py
    
    Ignored Files:
        rbtools/utils/COPYING.appdirs
        rbtools/utils/README.appdirs
        docs/rbtools/index.rst
    
    
    
    Tool: PEP8 Style Checker
    Processed Files:
        rbtools/api/transport/sync.py
        rbtools/api/request.py
        rbtools/utils/appdirs.py
        rbtools/api/cache.py
    
    Ignored Files:
        rbtools/utils/COPYING.appdirs
        rbtools/utils/README.appdirs
        docs/rbtools/index.rst
    
    
    WARNING: Number of comments exceeded maximum, showing 30 of 42.
  2. rbtools/api/cache.py (Diff revision 1)
     
     
    Col: 1
     E303 too many blank lines (3)
    
  3. rbtools/api/cache.py (Diff revision 1)
     
     
    Col: 34
     E128 continuation line under-indented for visual indent
    
  4. rbtools/api/cache.py (Diff revision 1)
     
     
    Col: 5
     E303 too many blank lines (2)
    
  5. rbtools/api/cache.py (Diff revision 1)
     
     
    Col: 13
     E303 too many blank lines (3)
    
  6. rbtools/api/cache.py (Diff revision 1)
     
     
    Col: 9
     E303 too many blank lines (2)
    
  7. rbtools/api/cache.py (Diff revision 1)
     
     
    Col: 5
     E303 too many blank lines (2)
    
  8. rbtools/api/cache.py (Diff revision 1)
     
     
    Col: 80
     E501 line too long (88 > 79 characters)
    
  9. rbtools/api/cache.py (Diff revision 1)
     
     
    Col: 32
     E261 at least two spaces before inline comment
    
  10. rbtools/api/cache.py (Diff revision 1)
     
     
    Col: 5
     E303 too many blank lines (2)
    
  11. rbtools/api/request.py (Diff revision 1)
     
     
    Col: 1
     W391 blank line at end of file
    
  12. rbtools/utils/appdirs.py (Diff revision 1)
     
     
    Col: 80
     E501 line too long (96 > 79 characters)
    
  13. rbtools/utils/appdirs.py (Diff revision 1)
     
     
    Col: 80
     E501 line too long (93 > 79 characters)
    
  14. rbtools/utils/appdirs.py (Diff revision 1)
     
     
    Col: 38
     E261 at least two spaces before inline comment
    
  15. rbtools/utils/appdirs.py (Diff revision 1)
     
     
    Col: 36
     E261 at least two spaces before inline comment
    
  16. rbtools/utils/appdirs.py (Diff revision 1)
     
     
    Col: 10
     E261 at least two spaces before inline comment
    
  17. rbtools/utils/appdirs.py (Diff revision 1)
     
     
    Col: 1
     E303 too many blank lines (3)
    
  18. rbtools/utils/appdirs.py (Diff revision 1)
     
     
    Col: 80
     E501 line too long (94 > 79 characters)
    
  19. rbtools/utils/appdirs.py (Diff revision 1)
     
     
    Col: 80
     E501 line too long (107 > 79 characters)
    
  20. rbtools/utils/appdirs.py (Diff revision 1)
     
     
    Col: 80
     E501 line too long (122 > 79 characters)
    
  21. rbtools/utils/appdirs.py (Diff revision 1)
     
     
    Col: 80
     E501 line too long (87 > 79 characters)
    
  22. rbtools/utils/appdirs.py (Diff revision 1)
     
     
    Col: 80
     E501 line too long (89 > 79 characters)
    
  23. rbtools/utils/appdirs.py (Diff revision 1)
     
     
    Col: 80
     E501 line too long (94 > 79 characters)
    
  24. rbtools/utils/appdirs.py (Diff revision 1)
     
     
    Col: 80
     E501 line too long (85 > 79 characters)
    
  25. rbtools/utils/appdirs.py (Diff revision 1)
     
     
    Col: 80
     E501 line too long (92 > 79 characters)
    
  26. rbtools/utils/appdirs.py (Diff revision 1)
     
     
    Col: 80
     E501 line too long (89 > 79 characters)
    
  27. rbtools/utils/appdirs.py (Diff revision 1)
     
     
    Col: 80
     E501 line too long (92 > 79 characters)
    
  28. rbtools/utils/appdirs.py (Diff revision 1)
     
     
    Col: 80
     E501 line too long (81 > 79 characters)
    
  29. rbtools/utils/appdirs.py (Diff revision 1)
     
     
    Col: 80
     E501 line too long (89 > 79 characters)
    
  30. rbtools/utils/appdirs.py (Diff revision 1)
     
     
    Col: 80
     E501 line too long (85 > 79 characters)
    
  31. rbtools/utils/appdirs.py (Diff revision 1)
     
     
    Col: 80
     E501 line too long (89 > 79 characters)
    
  32. rbtools/utils/appdirs.py (Diff revision 1)
     
     
     local variable 'kernel' is assigned to but never used
    
  33. rbtools/utils/appdirs.py (Diff revision 1)
     
     
     undefined name 'kernal'
    
  34. rbtools/utils/appdirs.py (Diff revision 1)
     
     
     'win32com' imported but unused
    
  35. rbtools/utils/appdirs.py (Diff revision 1)
     
     
     'windll' imported but unused
    
  36. rbtools/utils/appdirs.py (Diff revision 1)
     
     
     'com' imported but unused
    
  37. 
      
brennie
reviewbot
  1. Tool: Pyflakes
    Processed Files:
        rbtools/api/transport/sync.py
        rbtools/api/request.py
        rbtools/utils/appdirs.py
        rbtools/api/cache.py
    
    Ignored Files:
        rbtools/utils/COPYING.appdirs
        rbtools/utils/README.appdirs
        docs/rbtools/index.rst
    
    
    
    Tool: PEP8 Style Checker
    Processed Files:
        rbtools/api/transport/sync.py
        rbtools/api/request.py
        rbtools/utils/appdirs.py
        rbtools/api/cache.py
    
    Ignored Files:
        rbtools/utils/COPYING.appdirs
        rbtools/utils/README.appdirs
        docs/rbtools/index.rst
    
    
    WARNING: Number of comments exceeded maximum, showing 30 of 32.
  2. rbtools/utils/appdirs.py (Diff revision 2)
     
     
    Col: 80
     E501 line too long (96 > 79 characters)
    
  3. rbtools/utils/appdirs.py (Diff revision 2)
     
     
    Col: 80
     E501 line too long (93 > 79 characters)
    
  4. rbtools/utils/appdirs.py (Diff revision 2)
     
     
    Col: 38
     E261 at least two spaces before inline comment
    
  5. rbtools/utils/appdirs.py (Diff revision 2)
     
     
    Col: 36
     E261 at least two spaces before inline comment
    
  6. rbtools/utils/appdirs.py (Diff revision 2)
     
     
    Col: 10
     E261 at least two spaces before inline comment
    
  7. rbtools/utils/appdirs.py (Diff revision 2)
     
     
    Col: 1
     E303 too many blank lines (3)
    
  8. rbtools/utils/appdirs.py (Diff revision 2)
     
     
    Col: 80
     E501 line too long (94 > 79 characters)
    
  9. rbtools/utils/appdirs.py (Diff revision 2)
     
     
    Col: 80
     E501 line too long (107 > 79 characters)
    
  10. rbtools/utils/appdirs.py (Diff revision 2)
     
     
    Col: 80
     E501 line too long (122 > 79 characters)
    
  11. rbtools/utils/appdirs.py (Diff revision 2)
     
     
    Col: 80
     E501 line too long (87 > 79 characters)
    
  12. rbtools/utils/appdirs.py (Diff revision 2)
     
     
    Col: 80
     E501 line too long (89 > 79 characters)
    
  13. rbtools/utils/appdirs.py (Diff revision 2)
     
     
    Col: 80
     E501 line too long (94 > 79 characters)
    
  14. rbtools/utils/appdirs.py (Diff revision 2)
     
     
    Col: 80
     E501 line too long (85 > 79 characters)
    
  15. rbtools/utils/appdirs.py (Diff revision 2)
     
     
    Col: 80
     E501 line too long (92 > 79 characters)
    
  16. rbtools/utils/appdirs.py (Diff revision 2)
     
     
    Col: 80
     E501 line too long (89 > 79 characters)
    
  17. rbtools/utils/appdirs.py (Diff revision 2)
     
     
    Col: 80
     E501 line too long (92 > 79 characters)
    
  18. rbtools/utils/appdirs.py (Diff revision 2)
     
     
    Col: 80
     E501 line too long (81 > 79 characters)
    
  19. rbtools/utils/appdirs.py (Diff revision 2)
     
     
    Col: 80
     E501 line too long (89 > 79 characters)
    
  20. rbtools/utils/appdirs.py (Diff revision 2)
     
     
    Col: 80
     E501 line too long (85 > 79 characters)
    
  21. rbtools/utils/appdirs.py (Diff revision 2)
     
     
    Col: 80
     E501 line too long (89 > 79 characters)
    
  22. rbtools/utils/appdirs.py (Diff revision 2)
     
     
    Col: 80
     E501 line too long (116 > 79 characters)
    
  23. rbtools/utils/appdirs.py (Diff revision 2)
     
     
    Col: 80
     E501 line too long (81 > 79 characters)
    
  24. rbtools/utils/appdirs.py (Diff revision 2)
     
     
    Col: 80
     E501 line too long (80 > 79 characters)
    
  25. rbtools/utils/appdirs.py (Diff revision 2)
     
     
    Col: 80
     E501 line too long (80 > 79 characters)
    
  26. rbtools/utils/appdirs.py (Diff revision 2)
     
     
    Col: 80
     E501 line too long (81 > 79 characters)
    
  27. rbtools/utils/appdirs.py (Diff revision 2)
     
     
    Col: 80
     E501 line too long (115 > 79 characters)
    
  28. rbtools/utils/appdirs.py (Diff revision 2)
     
     
    Col: 80
     E501 line too long (80 > 79 characters)
    
  29. rbtools/utils/appdirs.py (Diff revision 2)
     
     
    Col: 30
     E128 continuation line under-indented for visual indent
    
  30. rbtools/utils/appdirs.py (Diff revision 2)
     
     
    Col: 1
     E265 block comment should start with '# '
    
  31. rbtools/utils/appdirs.py (Diff revision 2)
     
     
    Col: 1
     E302 expected 2 blank lines, found 1
    
  32. rbtools/utils/appdirs.py (Diff revision 2)
     
     
     'win32com' imported but unused
    
  33. rbtools/utils/appdirs.py (Diff revision 2)
     
     
     'windll' imported but unused
    
  34. rbtools/utils/appdirs.py (Diff revision 2)
     
     
     'com' imported but unused
    
  35. 
      
brennie
reviewbot
  1. Tool: Pyflakes
    Processed Files:
        rbtools/api/tests.py
        rbtools/utils/appdirs.py
        rbtools/api/request.py
        rbtools/api/cache.py
    
    Ignored Files:
        rbtools/utils/COPYING.appdirs
        rbtools/utils/README.appdirs
        docs/rbtools/index.rst
    
    
    
    Tool: PEP8 Style Checker
    Processed Files:
        rbtools/api/tests.py
        rbtools/utils/appdirs.py
        rbtools/api/request.py
        rbtools/api/cache.py
    
    Ignored Files:
        rbtools/utils/COPYING.appdirs
        rbtools/utils/README.appdirs
        docs/rbtools/index.rst
    
    
    WARNING: Number of comments exceeded maximum, showing 30 of 35.
  2. rbtools/api/cache.py (Diff revision 3)
     
     
    Col: 1
     E302 expected 2 blank lines, found 1
    
  3. rbtools/api/cache.py (Diff revision 3)
     
     
    Col: 1
     W293 blank line contains whitespace
    
  4. rbtools/api/cache.py (Diff revision 3)
     
     
    Col: 1
     W293 blank line contains whitespace
    
  5. rbtools/utils/appdirs.py (Diff revision 3)
     
     
    Col: 80
     E501 line too long (96 > 79 characters)
    
  6. rbtools/utils/appdirs.py (Diff revision 3)
     
     
    Col: 80
     E501 line too long (93 > 79 characters)
    
  7. rbtools/utils/appdirs.py (Diff revision 3)
     
     
    Col: 38
     E261 at least two spaces before inline comment