• 
      

    Fix the diff viewer to handle very large files.

    Review Request #3402 — Created Oct. 7, 2012 and submitted — Latest diff uploaded

    Information

    Review Board

    Reviewers

    Fix the diff viewer to handle very large files.
    
    Previously, very large files (10,000+ lines) would greatly slow down
    Review Board and take forever to render. This slowdown was entirely in
    the rendering step, rather than being in the diffing or patching steps.
    
    The reason for the slowdown is that we executed a fragment of a Django
    template (consisting of variable resolution, template tags, and filters)
    per line of the diff.
    
    This, it turns out, is very slow. Variable lookups (and possible
    escaping) is slow. Template tag and filter execution is slow. Copying of
    contexts is slow. To make matters worse, the context we supply for diffs
    contains data that can be quite large (chunk, file of chunks, etc.),
    which ends up getting copied a bunch for different tags. While we may
    want to audit this for other templates, fixing it for the diff lines
    is still slow due to how templates are processed.
    
    So what we're now doing is defining printf-style templates for the lines
    (and conditional parts of lines), processing some data in code, and then
    piecing them together for output.
    
    This creates a massive speedup. a 44,000 line change I tested with took
    around a minute to render before, bringing the server to its knees at
    the same time. Now, the total render time for the page (including all
    the rest of the diff viewer page) is now only a few seconds.
    Tested with my usual scattering of diffs. Verified that line numbers
    were correct, as were the coloring, expansion, and collapsing.
    
    Tested with a nearly 2MB patch containing over 44,000 lines of changes
    in the diff viewer. Saw it speed by (not that the browser was very
    happy about it).