Speed up page load times when using our LessCompiler.

Review Request #11406 — Created Jan. 25, 2021 and submitted

Information

Djblets
release-2.x

Reviewers

We have a specialization of Pipeline's LessCompiler that looks for all
dependencies recursively in a LessCSS file, making use of an in-house
script. If any dependency changes, we consider the file we want to
compile to be outdated.

This operation can be pretty slow, particularly for larger codebases.
Parsing all the LessCSS files takes some time, and we perform a whole
lot of redundant stat calls to get modification times for files. On
a fairly modern computer, this can take around 4 or 5 seconds for a
sizable codebase. For older computers, this can take dozens or more.

To improve performance, this change implements a couple new caches: one
for the list of imported files, and one for modification times.

Any time we want to get a list of files, we check the import files
cache. If not found, we build the new list and stick it in the cache.

Any time we want an mtime, we check the mtime cache, populating that
from a stat call if needed. The way we do this is faster than the
default way that Pipeline's compiler does things, as we're performing
one os.path.getmtime() and not an os.path.exists() +
os.path.getmtime().

If any file (a file we're compiling or a dependency) is outdated, we
invalidate all import file caches that reference it, ensuring the list
of imports will be recalculated.

The list of imported files are always checked individually for outdated
timestamps, whether or not we ran the script to get that list. The mtime
cache expires after 10 seconds, which is just enough to cut back on
lookups during compilation without much risk of actually missing
modifications.

This can help a little with an initial page load, since we're cuttingA
back on the stats, but really improves subsequent page loads, since
there's very little to do until something changes.

Unit tests pass on all supported versions of Python.

Manually tested. Inserted timing code in Pipeline to verify how long
each bundle's outdated checks were taking. Verified that the initial
page load took about the same as before (slight reduction on my system)
but that subsequent page loads were very noticeably faster.

Summary ID
Speed up page load times when using our LessCompiler.
We have a specialization of Pipeline's `LessCompiler` that looks for all dependencies recursively in a LessCSS file, making use of an in-house script. If any dependency changes, we consider the file we want to compile to be outdated. This operation can be pretty slow, particularly for larger codebases. Parsing all the LessCSS files takes some time, and we perform a whole lot of redundant stat calls to get modification times for files. On a fairly modern computer, this can take around 4 or 5 seconds for a sizable codebase. For older computers, this can take dozens or more. To improve performance, this change implements a couple new caches: one for the list of imported files, and one for modification times. Any time we want to get a list of files, we check the import files cache. If not found, we build the new list and stick it in the cache. Any time we want an mtime, we check the mtime cache, populating that from a stat call if needed. The way we do this is faster than the default way that Pipeline's compiler does things, as we're performing one `os.path.getmtime()` and not an `os.path.exists()` + `os.path.getmtime()`. If any file (a file we're compiling or a dependency) is outdated, we invalidate all import file caches that reference it, ensuring the list of imports will be recalculated. The list of imported files are always checked individually for outdated timestamps, whether or not we ran the script to get that list. The mtime cache expires after 10 seconds, which is just enough to cut back on lookups during compilation without much risk of actually missing modifications. This can help a little with an initial page load, since we're cuttingA back on the stats, but really improves subsequent page loads, since there's very little to do until something changes.
ec88ebce7c67309aa056bf646538989258c5ab5c
Description From Last Updated

Typo in description: cuttingA

daviddavid

F401 'time' imported but unused

reviewbotreviewbot

F401 'django.utils.six' imported but unused

reviewbotreviewbot

This ought to be wrapped in a try so that failures don't cause the test runner to bail.

daviddavid
Checks run (1 failed, 1 succeeded)
flake8 failed.
JSHint passed.

flake8

david
  1. 
      
  2. Show all issues

    Typo in description: cuttingA

  3. Show all issues

    This ought to be wrapped in a try so that failures don't cause the test runner to bail.

  4. 
      
chipx86
Review request changed
Status:
Completed
Change Summary:
Pushed to release-2.x (aa4217a)