Tree Sitter: Add highlighting implementation.

 






 

 


  Review Request #14522
  —
  Created July 28, 2025 and updated 

 




















 

  

   Information

  





 
 

  david
 









 
 

  Review Board
 









 
 

  master
 









 
 

  
 









 
 

  
 






 




 

  

   Reviewers

  





 
 

  reviewboard
 









 
 

  
 






 








 





  


 
 

  
This change adds the main implementation for syntax highlighting using

Tree Sitter. The top-level highlight function takes in the file

content both as a utf8-encoded bytestring and a decoded list of lines,

the parsed tree sitter Tree object, and the language name to highlight

for.


The basic procedure for highlighting involves several steps:


    

  1. Get highlighted nodes for the parsed tree with the given language.
    2. 

  2. Get injections for the parsed tree for the given language. If there
    
   are any injections present, reparse the file with the injected
    
   language and get highlighted nodes for those injections. This currently
    
   only does one level of injections (so JS inside of HTML inside of
    
   markdown would not be highlighted).
    3. 

  3. Process the highlighted nodes into a new list organized by line.
    
   Prior to this step, nodes can span multiple lines (for example, an
    
   entire Python docstring will result in one comment.documentation
    
   node). After, there would be separate nodes covering each line of the
    
   docstring.
    4. 

  4. Turn highlighted nodes into a sequence of events. These events have a
    
   starting position (within the line) and HTML opening or closing tags
    
   to insert at that position.
    5. 

  5. Scan through each line character by character, applying the events
    
   and escaping any HTML special characters, resulting in a list of HTML
    
   for each line in the file.
    6. 



Unit tests include tests for the basic functionality of those steps, as

well as a set of sample files in common languages so we can compare

full-file highlighting results. These tests may require updating when we

update to new versions of tree-sitter-language-pack or update queries

files from nvim-treesitter or grammars. The recompute-test-data.py

script will take the sample files and generate the expected output.


This commit includes the source files for all the full-file highlighting

tests, but for reviewability does not include the *.expected files

that they get compared to. Those are in a separate commit, since they're

not really suitable for review.

 








  


 
 

  
    

  • Ran unit tests.
    • 

  • Verified the appearance of syntax highlighting across a range of file
    
  types, including ones with various injected languages.
    • 


 







 









 





  

 


 
 

  



 

  
   


    Summary
    ID

   

  
  

   


    

     

      Tree Sitter: Add highlighting implementation.
      
This change adds the main implementation for syntax highlighting using
Tree Sitter. The top-level `highlight` function takes in the file
content both as a utf8-encoded bytestring and a decoded list of lines,
the parsed tree sitter `Tree` object, and the language name to highlight
for.

The basic procedure for highlighting involves several steps:

1. Get highlighted nodes for the parsed tree with the given language.
2. Get injections for the parsed tree for the given language. If there
   are any injections present, reparse the file with the injected
   language and get highlighted nodes for those injections. This currently
   only does one level of injections (so JS inside of HTML inside of
   markdown would not be highlighted).
3. Process the highlighted nodes into a new list organized by line.
   Prior to this step, nodes can span multiple lines (for example, an
   entire Python docstring will result in one `comment.documentation`
   node). After, there would be separate nodes covering each line of the
   docstring.
4. Turn highlighted nodes into a sequence of events. These events have a
   starting position (within the line) and HTML opening or closing tags
   to insert at that position.
5. Scan through each line character by character, applying the events
   and escaping any HTML special characters, resulting in a list of HTML
   for each line in the file.

Unit tests include tests for the basic functionality of those steps, as
well as a set of sample files in common languages so we can compare
full-file highlighting results. These tests may require updating when we
update to new versions of tree-sitter-language-pack or update queries
files from nvim-treesitter or grammars. The `recompute-test-data.py`
script will take the sample files and generate the expected output.

This commit includes the source files for all the full-file highlighting
tests, but for reviewability does not include the `*.expected` files
that they get compared to. Those will be in the next change.

Testing Done:
- Ran unit tests.
- Verified the appearance of syntax highlighting across a range of file
  types, including ones with various injected languages.

     


    		
    qxkmonknlypwrsrsulutpwoxlqvxtnrv

   



  
 




 













 






 
 
 

  

   

    

     
    

    
    
     
  • 
      
     
    • 
     
  • 
      
     
    • 
     
  • 
      
     
    • 
     
  • 
      
     
    • 
     
  • 
      
     
    • 
    

   

   

    
     

      Description
      From
      Last Updated
     

    
    



     

      
       
       
line too long (85 > 79 characters) Column: 80 Error code: E501

      		
      
       reviewbotreviewbot
      
      
       
      
     




     

      
       
       
'DEFAULT_TIMEOUT' is defined but never used. Column: 7 Error code: W098

      		
      
       reviewbotreviewbot
      
      
       
      
     




     

      
       
       
'os' imported but unused Column: 1 Error code: F401

      		
      
       reviewbotreviewbot
      
      
       
      
     




     

      
       
       
'DEFAULT_TIMEOUT' is defined but never used. Column: 7 Error code: W098

      		
      
       reviewbotreviewbot
      
      
       
      
     




     

      
       
       
'os' imported but unused Column: 1 Error code: F401

      		
      
       reviewbotreviewbot
      
      
       
      
     




     

      
       
       
'functools.lru_cache' imported but unused Column: 1 Error code: F401

      		
      
       reviewbotreviewbot
      
      
       
      
     




     

      
       
       
line break before binary operator Column: 5 Error code: W503

      		
      
       reviewbotreviewbot
      
      
       
      
     




     

      
       
       
'DEFAULT_TIMEOUT' is defined but never used. Column: 7 Error code: W098

      		
      
       reviewbotreviewbot
      
      
       
      
     




     

      
       
       
'os' imported but unused Column: 1 Error code: F401

      		
      
       reviewbotreviewbot
      
      
       
      
     




     

      
       
       
'DEFAULT_TIMEOUT' is defined but never used. Column: 7 Error code: W098

      		
      
       reviewbotreviewbot
      
      
       
      
     




     

      
       
       
'os' imported but unused Column: 1 Error code: F401

      		
      
       reviewbotreviewbot
      
      
       
      
     




     

      
       
       
'DEFAULT_TIMEOUT' is defined but never used. Column: 7 Error code: W098

      		
      
       reviewbotreviewbot
      
      
       
      
     




     

      
       
       
'os' imported but unused Column: 1 Error code: F401

      		
      
       reviewbotreviewbot
      
      
       
      
     


    
   

  

 







 






  

 



















 




 

  


   

   

    
 
Checks run (2 failed)


    
   


  

  

  





 

 flake8
 failed.
 





 

 JSHint
 failed.
 













 
flake8

 


 










 
JSHint

 


 







  

 





 














 



 

  

   




david




   

   

  

 



 

  


   

   

    
 Review request changed

    
   


  

  

  









  

   

    Change Summary:
   

   

    
Use tuples for pytest parametrize arg names.

   

  








  

   

    Commits:
   

   

    





 

  
   


    

    Summary
    ID

   

  
  

   


    

    

     

      Tree Sitter: Add highlighting implementation.
      
This change adds the main implementation for syntax highlighting using
Tree Sitter. The top-level `highlight` function takes in the file
content both as a utf8-encoded bytestring and a decoded list of lines,
the parsed tree sitter `Tree` object, and the language name to highlight
for.

The basic procedure for highlighting involves several steps:

1. Get highlighted nodes for the parsed tree with the given language.
2. Get injections for the parsed tree for the given language. If there
   are any injections present, reparse the file with the injected
   language and get highlighted nodes for those injections. This currently
   only does one level of injections (so JS inside of HTML inside of
   markdown would not be highlighted).
3. Process the highlighted nodes into a new list organized by line.
   Prior to this step, nodes can span multiple lines (for example, an
   entire Python docstring will result in one `comment.documentation`
   node). After, there would be separate nodes covering each line of the
   docstring.
4. Turn highlighted nodes into a sequence of events. These events have a
   starting position (within the line) and HTML opening or closing tags
   to insert at that position.
5. Scan through each line character by character, applying the events
   and escaping any HTML special characters, resulting in a list of HTML
   for each line in the file.

Unit tests include tests for the basic functionality of those steps, as
well as a set of sample files in common languages so we can compare
full-file highlighting results. These tests may require updating when we
update to new versions of tree-sitter-language-pack or update queries
files from nvim-treesitter or grammars. The `recompute-test-data.py`
script will take the sample files and generate the expected output.

This commit includes the source files for all the full-file highlighting
tests, but for reviewability does not include the `*.expected` files
that they get compared to. Those will be in the next change.

Testing Done:
- Ran unit tests.
- Verified the appearance of syntax highlighting across a range of file
  types, including ones with various injected languages.

     


    		
    22677924f64bc345de4094d290b766e9ec0311fe

   




   


    

    

     

      Tree Sitter: Add highlighting implementation.
      
This change adds the main implementation for syntax highlighting using
Tree Sitter. The top-level `highlight` function takes in the file
content both as a utf8-encoded bytestring and a decoded list of lines,
the parsed tree sitter `Tree` object, and the language name to highlight
for.

The basic procedure for highlighting involves several steps:

1. Get highlighted nodes for the parsed tree with the given language.
2. Get injections for the parsed tree for the given language. If there
   are any injections present, reparse the file with the injected
   language and get highlighted nodes for those injections. This currently
   only does one level of injections (so JS inside of HTML inside of
   markdown would not be highlighted).
3. Process the highlighted nodes into a new list organized by line.
   Prior to this step, nodes can span multiple lines (for example, an
   entire Python docstring will result in one `comment.documentation`
   node). After, there would be separate nodes covering each line of the
   docstring.
4. Turn highlighted nodes into a sequence of events. These events have a
   starting position (within the line) and HTML opening or closing tags
   to insert at that position.
5. Scan through each line character by character, applying the events
   and escaping any HTML special characters, resulting in a list of HTML
   for each line in the file.

Unit tests include tests for the basic functionality of those steps, as
well as a set of sample files in common languages so we can compare
full-file highlighting results. These tests may require updating when we
update to new versions of tree-sitter-language-pack or update queries
files from nvim-treesitter or grammars. The `recompute-test-data.py`
script will take the sample files and generate the expected output.

This commit includes the source files for all the full-file highlighting
tests, but for reviewability does not include the `*.expected` files
that they get compared to. Those will be in the next change.

Testing Done:
- Ran unit tests.
- Verified the appearance of syntax highlighting across a range of file
  types, including ones with various injected languages.

     


    		
    aea443cd841183b369d76b54005cc3cf91e6d9fb

   



  
 







   

  





  

   

    Diff:
   

   

    
 Revision 2 (+7674)
Show changes
 
    contrib/internal/treesitter/recompute-test-data.py 
    reviewboard/treesitter/highlight.py 
    reviewboard/treesitter/tests/test_highlight.py 
    reviewboard/treesitter/tests/testdata/sample.go 
    reviewboard/treesitter/tests/testdata/sample.html 
    reviewboard/treesitter/tests/testdata/sample.java 
    reviewboard/treesitter/tests/testdata/sample.js 
    reviewboard/treesitter/tests/testdata/sample.json 
  9 more
 

   

  











 

  
Checks run (2 failed)



  

 flake8
 failed.
 





  

 JSHint
 failed.
 





 







 
flake8

 


 










 
JSHint

 


 












  

 
















 



 

  

   




david




   

   

  

 



 

  


   

   

    
 Review request changed

    
   


  

  

  









  

   

    Change Summary:
   

   

    
    

  • Update to use reviewboard.treesitter.core utilities.
    • 

  • Update some type hints for Python 3.10+
    • 


   

  








  

   

    Commits:
   

   

    





 

  
   


    

    Summary
    ID

   

  
  

   


    

    

     

      Tree Sitter: Add highlighting implementation.
      
This change adds the main implementation for syntax highlighting using
Tree Sitter. The top-level `highlight` function takes in the file
content both as a utf8-encoded bytestring and a decoded list of lines,
the parsed tree sitter `Tree` object, and the language name to highlight
for.

The basic procedure for highlighting involves several steps:

1. Get highlighted nodes for the parsed tree with the given language.
2. Get injections for the parsed tree for the given language. If there
   are any injections present, reparse the file with the injected
   language and get highlighted nodes for those injections. This currently
   only does one level of injections (so JS inside of HTML inside of
   markdown would not be highlighted).
3. Process the highlighted nodes into a new list organized by line.
   Prior to this step, nodes can span multiple lines (for example, an
   entire Python docstring will result in one `comment.documentation`
   node). After, there would be separate nodes covering each line of the
   docstring.
4. Turn highlighted nodes into a sequence of events. These events have a
   starting position (within the line) and HTML opening or closing tags
   to insert at that position.
5. Scan through each line character by character, applying the events
   and escaping any HTML special characters, resulting in a list of HTML
   for each line in the file.

Unit tests include tests for the basic functionality of those steps, as
well as a set of sample files in common languages so we can compare
full-file highlighting results. These tests may require updating when we
update to new versions of tree-sitter-language-pack or update queries
files from nvim-treesitter or grammars. The `recompute-test-data.py`
script will take the sample files and generate the expected output.

This commit includes the source files for all the full-file highlighting
tests, but for reviewability does not include the `*.expected` files
that they get compared to. Those will be in the next change.

Testing Done:
- Ran unit tests.
- Verified the appearance of syntax highlighting across a range of file
  types, including ones with various injected languages.

     


    		
    aea443cd841183b369d76b54005cc3cf91e6d9fb

   




   


    

    

     

      Tree Sitter: Add highlighting implementation.
      
This change adds the main implementation for syntax highlighting using
Tree Sitter. The top-level `highlight` function takes in the file
content both as a utf8-encoded bytestring and a decoded list of lines,
the parsed tree sitter `Tree` object, and the language name to highlight
for.

The basic procedure for highlighting involves several steps:

1. Get highlighted nodes for the parsed tree with the given language.
2. Get injections for the parsed tree for the given language. If there
   are any injections present, reparse the file with the injected
   language and get highlighted nodes for those injections. This currently
   only does one level of injections (so JS inside of HTML inside of
   markdown would not be highlighted).
3. Process the highlighted nodes into a new list organized by line.
   Prior to this step, nodes can span multiple lines (for example, an
   entire Python docstring will result in one `comment.documentation`
   node). After, there would be separate nodes covering each line of the
   docstring.
4. Turn highlighted nodes into a sequence of events. These events have a
   starting position (within the line) and HTML opening or closing tags
   to insert at that position.
5. Scan through each line character by character, applying the events
   and escaping any HTML special characters, resulting in a list of HTML
   for each line in the file.

Unit tests include tests for the basic functionality of those steps, as
well as a set of sample files in common languages so we can compare
full-file highlighting results. These tests may require updating when we
update to new versions of tree-sitter-language-pack or update queries
files from nvim-treesitter or grammars. The `recompute-test-data.py`
script will take the sample files and generate the expected output.

This commit includes the source files for all the full-file highlighting
tests, but for reviewability does not include the `*.expected` files
that they get compared to. Those will be in the next change.

Testing Done:
- Ran unit tests.
- Verified the appearance of syntax highlighting across a range of file
  types, including ones with various injected languages.

     


    		
    qxkmonknlypwrsrsulutpwoxlqvxtnrv

   



  
 







   

  





  

   

    Diff:
   

   

    
 Revision 3 (+7582)
Show changes
 
    pyproject.toml 
    contrib/internal/treesitter/recompute-test-data.py 
    reviewboard/treesitter/highlight.py 
    reviewboard/treesitter/tests/test_highlight.py 
    reviewboard/treesitter/tests/testdata/sample.go 
    reviewboard/treesitter/tests/testdata/sample.html 
    reviewboard/treesitter/tests/testdata/sample.java 
    reviewboard/treesitter/tests/testdata/sample.js 
  10 more
 

   

  











 

  
Checks run (2 failed)



  

 flake8
 failed.
 





  

 JSHint
 failed.
 





 







 
flake8

 


 










 
JSHint

 


 












  

 
















 



 

  

   




david




   

   

  

 



 

  


   

   

    
 Review request changed

    
   


  

  

  















  

   

    Commits:
   

   

    





 

  
   


    

    Summary
    ID

   

  
  

   


    

    

     

      Tree Sitter: Add highlighting implementation.
      
This change adds the main implementation for syntax highlighting using
Tree Sitter. The top-level `highlight` function takes in the file
content both as a utf8-encoded bytestring and a decoded list of lines,
the parsed tree sitter `Tree` object, and the language name to highlight
for.

The basic procedure for highlighting involves several steps:

1. Get highlighted nodes for the parsed tree with the given language.
2. Get injections for the parsed tree for the given language. If there
   are any injections present, reparse the file with the injected
   language and get highlighted nodes for those injections. This currently
   only does one level of injections (so JS inside of HTML inside of
   markdown would not be highlighted).
3. Process the highlighted nodes into a new list organized by line.
   Prior to this step, nodes can span multiple lines (for example, an
   entire Python docstring will result in one `comment.documentation`
   node). After, there would be separate nodes covering each line of the
   docstring.
4. Turn highlighted nodes into a sequence of events. These events have a
   starting position (within the line) and HTML opening or closing tags
   to insert at that position.
5. Scan through each line character by character, applying the events
   and escaping any HTML special characters, resulting in a list of HTML
   for each line in the file.

Unit tests include tests for the basic functionality of those steps, as
well as a set of sample files in common languages so we can compare
full-file highlighting results. These tests may require updating when we
update to new versions of tree-sitter-language-pack or update queries
files from nvim-treesitter or grammars. The `recompute-test-data.py`
script will take the sample files and generate the expected output.

This commit includes the source files for all the full-file highlighting
tests, but for reviewability does not include the `*.expected` files
that they get compared to. Those will be in the next change.

Testing Done:
- Ran unit tests.
- Verified the appearance of syntax highlighting across a range of file
  types, including ones with various injected languages.

     


    		
    qxkmonknlypwrsrsulutpwoxlqvxtnrv

   




   


    

    

     

      Tree Sitter: Add highlighting implementation.
      
This change adds the main implementation for syntax highlighting using
Tree Sitter. The top-level `highlight` function takes in the file
content both as a utf8-encoded bytestring and a decoded list of lines,
the parsed tree sitter `Tree` object, and the language name to highlight
for.

The basic procedure for highlighting involves several steps:

1. Get highlighted nodes for the parsed tree with the given language.
2. Get injections for the parsed tree for the given language. If there
   are any injections present, reparse the file with the injected
   language and get highlighted nodes for those injections. This currently
   only does one level of injections (so JS inside of HTML inside of
   markdown would not be highlighted).
3. Process the highlighted nodes into a new list organized by line.
   Prior to this step, nodes can span multiple lines (for example, an
   entire Python docstring will result in one `comment.documentation`
   node). After, there would be separate nodes covering each line of the
   docstring.
4. Turn highlighted nodes into a sequence of events. These events have a
   starting position (within the line) and HTML opening or closing tags
   to insert at that position.
5. Scan through each line character by character, applying the events
   and escaping any HTML special characters, resulting in a list of HTML
   for each line in the file.

Unit tests include tests for the basic functionality of those steps, as
well as a set of sample files in common languages so we can compare
full-file highlighting results. These tests may require updating when we
update to new versions of tree-sitter-language-pack or update queries
files from nvim-treesitter or grammars. The `recompute-test-data.py`
script will take the sample files and generate the expected output.

This commit includes the source files for all the full-file highlighting
tests, but for reviewability does not include the `*.expected` files
that they get compared to. Those will be in the next change.

Testing Done:
- Ran unit tests.
- Verified the appearance of syntax highlighting across a range of file
  types, including ones with various injected languages.

     


    		
    qxkmonknlypwrsrsulutpwoxlqvxtnrv

   



  
 







   

  





  

   

    Diff:
   

   

    
 Revision 4 (+7578)
Show changes
 
    pyproject.toml 
    setup.cfg 
    contrib/internal/treesitter/recompute-test-data.py 
    reviewboard/treesitter/highlight.py 
    reviewboard/treesitter/tests/test_highlight.py 
    reviewboard/treesitter/tests/testdata/sample.go 
    reviewboard/treesitter/tests/testdata/sample.html 
    reviewboard/treesitter/tests/testdata/sample.java 
  11 more
 

   

  











 

  
Checks run (2 failed)



  

 flake8
 failed.
 





  

 JSHint
 failed.
 





 







 
flake8

 


 










 
JSHint

 


 












  

 
















 



 

  

   




david




   

   

  

 



 

  


   

   

    
 Review request changed

    
   


  

  

  















  

   

    Commits:
   

   

    





 

  
   


    

    Summary
    ID

   

  
  

   


    

    

     

      Tree Sitter: Add highlighting implementation.
      
This change adds the main implementation for syntax highlighting using
Tree Sitter. The top-level `highlight` function takes in the file
content both as a utf8-encoded bytestring and a decoded list of lines,
the parsed tree sitter `Tree` object, and the language name to highlight
for.

The basic procedure for highlighting involves several steps:

1. Get highlighted nodes for the parsed tree with the given language.
2. Get injections for the parsed tree for the given language. If there
   are any injections present, reparse the file with the injected
   language and get highlighted nodes for those injections. This currently
   only does one level of injections (so JS inside of HTML inside of
   markdown would not be highlighted).
3. Process the highlighted nodes into a new list organized by line.
   Prior to this step, nodes can span multiple lines (for example, an
   entire Python docstring will result in one `comment.documentation`
   node). After, there would be separate nodes covering each line of the
   docstring.
4. Turn highlighted nodes into a sequence of events. These events have a
   starting position (within the line) and HTML opening or closing tags
   to insert at that position.
5. Scan through each line character by character, applying the events
   and escaping any HTML special characters, resulting in a list of HTML
   for each line in the file.

Unit tests include tests for the basic functionality of those steps, as
well as a set of sample files in common languages so we can compare
full-file highlighting results. These tests may require updating when we
update to new versions of tree-sitter-language-pack or update queries
files from nvim-treesitter or grammars. The `recompute-test-data.py`
script will take the sample files and generate the expected output.

This commit includes the source files for all the full-file highlighting
tests, but for reviewability does not include the `*.expected` files
that they get compared to. Those will be in the next change.

Testing Done:
- Ran unit tests.
- Verified the appearance of syntax highlighting across a range of file
  types, including ones with various injected languages.

     


    		
    qxkmonknlypwrsrsulutpwoxlqvxtnrv

   




   


    

    

     

      Tree Sitter: Add highlighting implementation.
      
This change adds the main implementation for syntax highlighting using
Tree Sitter. The top-level `highlight` function takes in the file
content both as a utf8-encoded bytestring and a decoded list of lines,
the parsed tree sitter `Tree` object, and the language name to highlight
for.

The basic procedure for highlighting involves several steps:

1. Get highlighted nodes for the parsed tree with the given language.
2. Get injections for the parsed tree for the given language. If there
   are any injections present, reparse the file with the injected
   language and get highlighted nodes for those injections. This currently
   only does one level of injections (so JS inside of HTML inside of
   markdown would not be highlighted).
3. Process the highlighted nodes into a new list organized by line.
   Prior to this step, nodes can span multiple lines (for example, an
   entire Python docstring will result in one `comment.documentation`
   node). After, there would be separate nodes covering each line of the
   docstring.
4. Turn highlighted nodes into a sequence of events. These events have a
   starting position (within the line) and HTML opening or closing tags
   to insert at that position.
5. Scan through each line character by character, applying the events
   and escaping any HTML special characters, resulting in a list of HTML
   for each line in the file.

Unit tests include tests for the basic functionality of those steps, as
well as a set of sample files in common languages so we can compare
full-file highlighting results. These tests may require updating when we
update to new versions of tree-sitter-language-pack or update queries
files from nvim-treesitter or grammars. The `recompute-test-data.py`
script will take the sample files and generate the expected output.

This commit includes the source files for all the full-file highlighting
tests, but for reviewability does not include the `*.expected` files
that they get compared to. Those will be in the next change.

Testing Done:
- Ran unit tests.
- Verified the appearance of syntax highlighting across a range of file
  types, including ones with various injected languages.

     


    		
    qxkmonknlypwrsrsulutpwoxlqvxtnrv

   



  
 







   

  





  

   

    Diff:
   

   

    
 Revision 5 (+7578)
Show changes
 
    pyproject.toml 
    setup.cfg 
    contrib/internal/treesitter/recompute-test-data.py 
    reviewboard/treesitter/highlight.py 
    reviewboard/treesitter/tests/test_highlight.py 
    reviewboard/treesitter/tests/testdata/sample.go 
    reviewboard/treesitter/tests/testdata/sample.html 
    reviewboard/treesitter/tests/testdata/sample.java 
  11 more
 

   

  











 

  
Checks run (2 failed)



  

 flake8
 failed.
 





  

 JSHint
 failed.
 





 







 
flake8

 


 










 
JSHint