Speed up page load times when using our LessCompiler.

Review Request #11406 — Created Jan. 25, 2021 and submitted — Latest diff uploaded

Information

Djblets
release-2.x

Reviewers

We have a specialization of Pipeline's LessCompiler that looks for all
dependencies recursively in a LessCSS file, making use of an in-house
script. If any dependency changes, we consider the file we want to
compile to be outdated.

This operation can be pretty slow, particularly for larger codebases.
Parsing all the LessCSS files takes some time, and we perform a whole
lot of redundant stat calls to get modification times for files. On
a fairly modern computer, this can take around 4 or 5 seconds for a
sizable codebase. For older computers, this can take dozens or more.

To improve performance, this change implements a couple new caches: one
for the list of imported files, and one for modification times.

Any time we want to get a list of files, we check the import files
cache. If not found, we build the new list and stick it in the cache.

Any time we want an mtime, we check the mtime cache, populating that
from a stat call if needed. The way we do this is faster than the
default way that Pipeline's compiler does things, as we're performing
one os.path.getmtime() and not an os.path.exists() +
os.path.getmtime().

If any file (a file we're compiling or a dependency) is outdated, we
invalidate all import file caches that reference it, ensuring the list
of imports will be recalculated.

The list of imported files are always checked individually for outdated
timestamps, whether or not we ran the script to get that list. The mtime
cache expires after 10 seconds, which is just enough to cut back on
lookups during compilation without much risk of actually missing
modifications.

This can help a little with an initial page load, since we're cuttingA
back on the stats, but really improves subsequent page loads, since
there's very little to do until something changes.

Unit tests pass on all supported versions of Python.

Manually tested. Inserted timing code in Pipeline to verify how long
each bundle's outdated checks were taking. Verified that the initial
page load took about the same as before (slight reduction on my system)
but that subsequent page loads were very noticeably faster.

Commits

Files

    Loading...