• 
      

    Speed up page load times when using our LessCompiler.

    Review Request #11406 — Created Jan. 25, 2021 and submitted

    Information

    Djblets
    release-2.x

    Reviewers

    We have a specialization of Pipeline's LessCompiler that looks for all
    dependencies recursively in a LessCSS file, making use of an in-house
    script. If any dependency changes, we consider the file we want to
    compile to be outdated.

    This operation can be pretty slow, particularly for larger codebases.
    Parsing all the LessCSS files takes some time, and we perform a whole
    lot of redundant stat calls to get modification times for files. On
    a fairly modern computer, this can take around 4 or 5 seconds for a
    sizable codebase. For older computers, this can take dozens or more.

    To improve performance, this change implements a couple new caches: one
    for the list of imported files, and one for modification times.

    Any time we want to get a list of files, we check the import files
    cache. If not found, we build the new list and stick it in the cache.

    Any time we want an mtime, we check the mtime cache, populating that
    from a stat call if needed. The way we do this is faster than the
    default way that Pipeline's compiler does things, as we're performing
    one os.path.getmtime() and not an os.path.exists() +
    os.path.getmtime().

    If any file (a file we're compiling or a dependency) is outdated, we
    invalidate all import file caches that reference it, ensuring the list
    of imports will be recalculated.

    The list of imported files are always checked individually for outdated
    timestamps, whether or not we ran the script to get that list. The mtime
    cache expires after 10 seconds, which is just enough to cut back on
    lookups during compilation without much risk of actually missing
    modifications.

    This can help a little with an initial page load, since we're cuttingA
    back on the stats, but really improves subsequent page loads, since
    there's very little to do until something changes.

    Unit tests pass on all supported versions of Python.

    Manually tested. Inserted timing code in Pipeline to verify how long
    each bundle's outdated checks were taking. Verified that the initial
    page load took about the same as before (slight reduction on my system)
    but that subsequent page loads were very noticeably faster.

    Summary ID
    Speed up page load times when using our LessCompiler.
    We have a specialization of Pipeline's `LessCompiler` that looks for all dependencies recursively in a LessCSS file, making use of an in-house script. If any dependency changes, we consider the file we want to compile to be outdated. This operation can be pretty slow, particularly for larger codebases. Parsing all the LessCSS files takes some time, and we perform a whole lot of redundant stat calls to get modification times for files. On a fairly modern computer, this can take around 4 or 5 seconds for a sizable codebase. For older computers, this can take dozens or more. To improve performance, this change implements a couple new caches: one for the list of imported files, and one for modification times. Any time we want to get a list of files, we check the import files cache. If not found, we build the new list and stick it in the cache. Any time we want an mtime, we check the mtime cache, populating that from a stat call if needed. The way we do this is faster than the default way that Pipeline's compiler does things, as we're performing one `os.path.getmtime()` and not an `os.path.exists()` + `os.path.getmtime()`. If any file (a file we're compiling or a dependency) is outdated, we invalidate all import file caches that reference it, ensuring the list of imports will be recalculated. The list of imported files are always checked individually for outdated timestamps, whether or not we ran the script to get that list. The mtime cache expires after 10 seconds, which is just enough to cut back on lookups during compilation without much risk of actually missing modifications. This can help a little with an initial page load, since we're cuttingA back on the stats, but really improves subsequent page loads, since there's very little to do until something changes.
    ec88ebce7c67309aa056bf646538989258c5ab5c
    Description From Last Updated

    Typo in description: cuttingA

    daviddavid

    F401 'time' imported but unused

    reviewbotreviewbot

    F401 'django.utils.six' imported but unused

    reviewbotreviewbot

    This ought to be wrapped in a try so that failures don't cause the test runner to bail.

    daviddavid
    Checks run (1 failed, 1 succeeded)
    flake8 failed.
    JSHint passed.

    flake8

    david
    1. 
        
    2. Show all issues

      Typo in description: cuttingA

    3. Show all issues

      This ought to be wrapped in a try so that failures don't cause the test runner to bail.

    4. 
        
    chipx86
    Review request changed
    Status:
    Completed
    Change Summary:
    Pushed to release-2.x (aa4217a)