Tree Sitter: Add query utils.

 






 

 


  Review Request #14517
  —
  Created July 26, 2025 and updated 

 




















 

  

   Information

  





 
 

  david
 









 
 

  Review Board
 









 
 

  master
 









 
 

  
 









 
 

  
 






 




 

  

   Reviewers

  





 
 

  reviewboard
 









 
 

  
 






 








 





  


 
 

  
This change adds some utilities for manipulating Tree Sitter queries.

This new module has two main public-facing functions:


get_all_predicate_names is a method that simply returns a list of all

the predicate names in a query file. This is used in the update-queries

script (coming in a later change) to print a warning if any of the

queries include predicates which we do not have implemented in Review

Board.


apply_standard_query_edits is used to make edits to query files to

make them compatible with Review Board.


For doing syntax highlighting with tree sitter, we're going to be

shipping many of the highlights files from the nvim-treesitter package.

These highlights files are often much, much better than the ones that

are shipped with the grammars themselves, but they do have a few things

that need to be worked around.


Our "standard edits" include:


    

  • Removing any queries that use a set! directive that takes three
    
  arguments. This is something entirely custom inside neovim's
    
  implementation, and because set! is implemented in the
    
  py-tree-sitter C code, there's no way for us to do it. This is used in
    
  very few places in the actual queries files, and doesn't seem to
    
  impact syntax highlighting in any meaningful way, so we just get rid
    
  of them.
    • 

  • Rewriting lua-match? and not-lua-match? predicates to use Python
    
  regex syntax and change the predicate name to match? or not-match?
    
  as appropriate.
    • 

  • Rewriting gsub! directives to use Python regex syntax.
    • 



The accompanying tests include unit tests for each of the edits, as well

as two parameterized tests for loading highlights and injections queries

for each language. These include a few xfail marks for languages where

the bundled queries are currently known to fail; as I work through

those, hopefully those lists will become completely empty.

 








  


 
 

  
    

  • Ran unit tests.
    • 

  • Used these from the update-queries script to rewrite queries files
    
  from nvim-treesitter to make them compatible with Review Board.
    • 

  • Verified that rewritten queries worked correctly, especially with
    
  rewritten match queries.
    • 


 







 









 





  

 


 
 

  



 

  
   


    Summary
    ID

   

  
  

   


    

     

      Tree Sitter: Add query utils.
      
This change adds some utilities for manipulating Tree Sitter queries.
This new module has two main public-facing functions:

`get_all_predicate_names` is a method that simply returns a list of all
the predicate names in a query file. This is used in the update-queries
script (coming in a later change) to print a warning if any of the
queries include predicates which we do not have implemented in Review
Board.

`apply_standard_query_edits` is used to make edits to query files to
make them compatible with Review Board.

For doing syntax highlighting with tree sitter, we're going to be
shipping many of the highlights files from the nvim-treesitter package.
These highlights files are often much, much better than the ones that
are shipped with the grammars themselves, but they do have a few things
that need to be worked around.

Our "standard edits" include:

- Removing any queries that use a `set!` directive that takes three
  arguments. This is something entirely custom inside neovim's
  implementation, and because `set!` is implemented in the
  py-tree-sitter C code, there's no way for us to do it. This is used in
  very few places in the actual queries files, and doesn't seem to
  impact syntax highlighting in any meaningful way, so we just get rid
  of them.
- Rewriting `lua-match?` and `not-lua-match?` predicates to use Python
  regex syntax and change the predicate name to `match?` or `not-match?`
  as appropriate.
- Rewriting `gsub!` directives to use Python regex syntax.

The accompanying tests include unit tests for each of the edits, as well
as two parameterized tests for loading highlights and injections queries
for each language. These include a few xfail marks for languages where
the bundled queries are currently known to fail; as I work through
those, hopefully those lists will become completely empty.

Testing Done:
- Ran unit tests.
- Used these from the update-queries script to rewrite queries files
  from nvim-treesitter to make them compatible with Review Board.
- Verified that rewritten queries worked correctly, especially with
  rewritten match queries.

     


    		
    qvvsmvtyozynwrzswmwktwvyvvsluyzu

   



  
 




 













 









 






  

 



















 




 




 














 



 

  

   




david




   

   

  

 



 

  


   

   

    
 Review request changed

    
   


  

  

  









  

   

    Change Summary:
   

   

    
    

  • Moved reviewboard.treesitter.core module into this change (from a much later one)
    • 

  • Sort parametrize params to make this compatible with pytest-xdist
    • 

  • Get TypeAlias from typing since RB10 will be Python 3.10+
    • 


   

  








  

   

    Commits:
   

   

    





 

  
   


    

    Summary
    ID

   

  
  

   


    

    

     

      Tree Sitter: Add query utils.
      
This change adds some utilities for manipulating Tree Sitter queries.
This new module has two main public-facing functions:

`get_all_predicate_names` is a method that simply returns a list of all
the predicate names in a query file. This is used in the update-queries
script (coming in a later change) to print a warning if any of the
queries include predicates which we do not have implemented in Review
Board.

`apply_standard_query_edits` is used to make edits to query files to
make them compatible with Review Board.

For doing syntax highlighting with tree sitter, we're going to be
shipping many of the highlights files from the nvim-treesitter package.
These highlights files are often much, much better than the ones that
are shipped with the grammars themselves, but they do have a few things
that need to be worked around.

Our "standard edits" include:

- Removing any queries that use a `set!` directive that takes three
  arguments. This is something entirely custom inside neovim's
  implementation, and because `set!` is implemented in the
  py-tree-sitter C code, there's no way for us to do it. This is used in
  very few places in the actual queries files, and doesn't seem to
  impact syntax highlighting in any meaningful way, so we just get rid
  of them.
- Rewriting `lua-match?` and `not-lua-match?` predicates to use Python
  regex syntax and change the predicate name to `match?` or `not-match?`
  as appropriate.
- Rewriting `gsub!` directives to use Python regex syntax.

The accompanying tests include unit tests for each of the edits, as well
as two parameterized tests for loading highlights and injections queries
for each language. These include a few xfail marks for languages where
the bundled queries are currently known to fail; as I work through
those, hopefully those lists will become completely empty.

Testing Done:
- Ran unit tests.
- Used these from the update-queries script to rewrite queries files
  from nvim-treesitter to make them compatible with Review Board.
- Verified that rewritten queries worked correctly, especially with
  rewritten match queries.

     


    		
    fa4d586f852aec460396f1f0647e906c911686fe

   




   


    

    

     

      Tree Sitter: Add query utils.
      
This change adds some utilities for manipulating Tree Sitter queries.
This new module has two main public-facing functions:

`get_all_predicate_names` is a method that simply returns a list of all
the predicate names in a query file. This is used in the update-queries
script (coming in a later change) to print a warning if any of the
queries include predicates which we do not have implemented in Review
Board.

`apply_standard_query_edits` is used to make edits to query files to
make them compatible with Review Board.

For doing syntax highlighting with tree sitter, we're going to be
shipping many of the highlights files from the nvim-treesitter package.
These highlights files are often much, much better than the ones that
are shipped with the grammars themselves, but they do have a few things
that need to be worked around.

Our "standard edits" include:

- Removing any queries that use a `set!` directive that takes three
  arguments. This is something entirely custom inside neovim's
  implementation, and because `set!` is implemented in the
  py-tree-sitter C code, there's no way for us to do it. This is used in
  very few places in the actual queries files, and doesn't seem to
  impact syntax highlighting in any meaningful way, so we just get rid
  of them.
- Rewriting `lua-match?` and `not-lua-match?` predicates to use Python
  regex syntax and change the predicate name to `match?` or `not-match?`
  as appropriate.
- Rewriting `gsub!` directives to use Python regex syntax.

The accompanying tests include unit tests for each of the edits, as well
as two parameterized tests for loading highlights and injections queries
for each language. These include a few xfail marks for languages where
the bundled queries are currently known to fail; as I work through
those, hopefully those lists will become completely empty.

Testing Done:
- Ran unit tests.
- Used these from the update-queries script to rewrite queries files
  from nvim-treesitter to make them compatible with Review Board.
- Verified that rewritten queries worked correctly, especially with
  rewritten match queries.

     


    		
    qvvsmvtyozynwrzswmwktwvyvvsluyzu

   



  
 







   

  





  

   

    Diff:
   

   

    
 Revision 2 (+1756)
Show changes
 
    reviewboard/treesitter/core.py 
    reviewboard/treesitter/query_utils.py 
    reviewboard/treesitter/tests/test_queries.py 
 

   

  











 

  
Checks run (2 succeeded)



  

 flake8
 passed.
 





  

 JSHint
 passed.