• 
      

    Tree Sitter: Add highlighting implementation.

    Review Request #14522 — Created July 28, 2025 and updated — Latest diff uploaded

    Information

    Review Board
    master

    Reviewers

    This change adds the main implementation for syntax highlighting using
    Tree Sitter. The top-level highlight function takes in the file
    content both as a utf8-encoded bytestring and a decoded list of lines,
    the parsed tree sitter Tree object, and the language name to highlight
    for.

    The basic procedure for highlighting involves several steps:

    1. Get highlighted nodes for the parsed tree with the given language.
    2. Get injections for the parsed tree for the given language. If there
      are any injections present, reparse the file with the injected
      language and get highlighted nodes for those injections. This currently
      only does one level of injections (so JS inside of HTML inside of
      markdown would not be highlighted).
    3. Process the highlighted nodes into a new list organized by line.
      Prior to this step, nodes can span multiple lines (for example, an
      entire Python docstring will result in one comment.documentation
      node). After, there would be separate nodes covering each line of the
      docstring.
    4. Turn highlighted nodes into a sequence of events. These events have a
      starting position (within the line) and HTML opening or closing tags
      to insert at that position.
    5. Scan through each line character by character, applying the events
      and escaping any HTML special characters, resulting in a list of HTML
      for each line in the file.

    Unit tests include tests for the basic functionality of those steps, as
    well as a set of sample files in common languages so we can compare
    full-file highlighting results. These tests may require updating when we
    update to new versions of tree-sitter-language-pack or update queries
    files from nvim-treesitter or grammars. The recompute-test-data.py
    script will take the sample files and generate the expected output.

    This commit includes the source files for all the full-file highlighting
    tests, but for reviewability does not include the *.expected files
    that they get compared to. Those are in a separate commit, since they're
    not really suitable for review.

    • Ran unit tests.
    • Verified the appearance of syntax highlighting across a range of file
      types, including ones with various injected languages.

    Commits

    Files