Tree Sitter: Add highlighting implementation.

Review Request #14522 — Created July 28, 2025 and updated — Latest diff uploaded

Information

Review Board
master

Reviewers

This change adds the main implementation for syntax highlighting using
Tree Sitter. The top-level highlight function takes in the file
content both as a utf8-encoded bytestring and a decoded list of lines,
the parsed tree sitter Tree object, and the language name to highlight
for.

The basic procedure for highlighting involves several steps:

  1. Get highlighted nodes for the parsed tree with the given language.
  2. Get injections for the parsed tree for the given language. If there
    are any injections present, reparse the file with the injected
    language and get highlighted nodes for those injections. This currently
    only does one level of injections (so JS inside of HTML inside of
    markdown would not be highlighted).
  3. Process the highlighted nodes into a new list organized by line.
    Prior to this step, nodes can span multiple lines (for example, an
    entire Python docstring will result in one comment.documentation
    node). After, there would be separate nodes covering each line of the
    docstring.
  4. Turn highlighted nodes into a sequence of events. These events have a
    starting position (within the line) and HTML opening or closing tags
    to insert at that position.
  5. Scan through each line character by character, applying the events
    and escaping any HTML special characters, resulting in a list of HTML
    for each line in the file.

Unit tests include tests for the basic functionality of those steps, as
well as a set of sample files in common languages so we can compare
full-file highlighting results. These tests may require updating when we
update to new versions of tree-sitter-language-pack or update queries
files from nvim-treesitter or grammars. The recompute-test-data.py
script will take the sample files and generate the expected output.

This commit includes the source files for all the full-file highlighting
tests, but for reviewability does not include the *.expected files
that they get compared to. Those are in a separate commit, since they're
not really suitable for review.

  • Ran unit tests.
  • Verified the appearance of syntax highlighting across a range of file
    types, including ones with various injected languages.

Commits

Files