Summary

Tree Sitter: Add update-language-info script and resulting data.

Review Request #14514 — Created July 26, 2025 and submitted March 24, 2026, 11:54 a.m.

Information

Owner

david

Repository

Review Board

Branch

master

Bugs

Depends On

Reviewers

Groups

reviewboard

People

Description

This change adds a script that loads information about supported
languages from a tree-sitter-language-pack checkout. This creates a
file, reviewboard/treesitter/_languages.py, which has two mappings,
MIME_TYPE_TO_LANGUAGE and FILE_SUFFIX_TO_LANGUAGES, which are used
for determining the tree sitter language name to use for a given file.

The compiled grammars that are shipped in tree-sitter-language-pack
don't include any of the grammar metadata, so this script will go into a
a checkout and grab the filename suffixes. This also fetches the
"first_line_regex" key, although that is not currently used for
detection.

Testing Done

Ran script to create the _languages.py file.

Commits

Summary	ID
Tree Sitter: Add update-language-info script and resulting data. This change adds a script that loads information about supported languages from a tree-sitter-language-pack checkout. This creates a file, `reviewboard/treesitter/_languages.py`, which has two mappings, `MIME_TYPE_TO_LANGUAGE` and `FILE_SUFFIX_TO_LANGUAGES`, which are used for determining the tree sitter language name to use for a given file. The compiled grammars that are shipped in tree-sitter-language-pack don't include any of the grammar metadata, so this script will go into a a checkout and grab the filename suffixes. This also fetches the "first_line_regex" key, although that is not currently used for detection. Testing Done: Ran script to create the _languages.py file.	nnsksqorxrnnmnuxklvrymslmwqmpxxt

Summary

Tree Sitter: Add update-language-info script and resulting data.

This change adds a script that loads information about supported languages from a tree-sitter-language-pack checkout. This creates a file, `reviewboard/treesitter/_languages.py`, which has two mappings, `MIME_TYPE_TO_LANGUAGE` and `FILE_SUFFIX_TO_LANGUAGES`, which are used for determining the tree sitter language name to use for a given file. The compiled grammars that are shipped in tree-sitter-language-pack don't include any of the grammar metadata, so this script will go into a a checkout and grab the filename suffixes. This also fetches the "first_line_regex" key, although that is not currently used for detection. Testing Done: Ran script to create the _languages.py file.

nnsksqorxrnnmnuxklvrymslmwqmpxxt

Issues

Description	From	Last Updated
continuation line over-indented for visual indent Column: 10 Error code: E127	reviewbot	July 26, 2025, 10:53 a.m.
continuation line under-indented for visual indent Column: 33 Error code: E128	reviewbot	July 26, 2025, 10:53 a.m.
line too long (83 > 79 characters) Column: 80 Error code: E501	reviewbot	July 26, 2025, 10:53 a.m.
This and the rest of the variables/classes/functions in this file are missing Version Addeds in their docstrings.	maubin	March 14, 2026, 10:45 a.m.
We should show what the invalid file_suffixes data is in this warning.	maubin	March 14, 2026, 10:46 a.m.
Same comment here to show what the invalid first_line_regex is.	maubin	March 14, 2026, 10:47 a.m.
Should we log some warning/debug statement when we encounter OSErrors in this function?	maubin	March 14, 2026, 10:48 a.m.
line too long (83 > 79 characters) Column: 80 Error code: E501	reviewbot	July 26, 2025, 10:53 a.m.
line too long (83 > 79 characters) Column: 80 Error code: E501	reviewbot	March 14, 2026, 10:50 a.m.

flake8 failed.

JSHint passed.

flake8

contrib/internal/treesitter/update-language-info.py (Diff revision 1)
The issue has been dropped. Show all issues
```
continuation line over-indented for visual indent

Column: 10
Error code: E127
```
contrib/internal/treesitter/update-language-info.py (Diff revision 1)
The issue has been dropped. Show all issues
```
continuation line under-indented for visual indent

Column: 33
Error code: E128
```
reviewboard/treesitter/_languages.py (Diff revision 1)
The issue has been dropped. Show all issues
```
line too long (83 > 79 characters)

Column: 80
Error code: E501
```

Commits:

	Summary	ID
	Tree Sitter: Add update-language-info script and resulting data. This change adds a script that loads information about supported languages from a tree-sitter-language-pack checkout. This creates a file, `reviewboard/treesitter/_languages.py`, which has two mappings, `MIME_TYPE_TO_LANGUAGE` and `FILE_SUFFIX_TO_LANGUAGES`, which are used for determining the tree sitter language name to use for a given file. The compiled grammars that are shipped in tree-sitter-language-pack don't include any of the grammar metadata, so this script will go into a a checkout and grab the filename suffixes. This also fetches the "first_line_regex" key, although that is not currently used for detection. Testing Done: Ran script to create the _languages.py file.	43fd40d0beb3c46b3edf86d5aa7a87e8cdc58486
	Tree Sitter: Add update-language-info script and resulting data. This change adds a script that loads information about supported languages from a tree-sitter-language-pack checkout. This creates a file, `reviewboard/treesitter/_languages.py`, which has two mappings, `MIME_TYPE_TO_LANGUAGE` and `FILE_SUFFIX_TO_LANGUAGES`, which are used for determining the tree sitter language name to use for a given file. The compiled grammars that are shipped in tree-sitter-language-pack don't include any of the grammar metadata, so this script will go into a a checkout and grab the filename suffixes. This also fetches the "first_line_regex" key, although that is not currently used for detection. Testing Done: Ran script to create the _languages.py file.	d79179711d6e9d6845a3bbf7b28b15c7e5b2ef56

Diff:

Revision 2 (+2142)

Show changes

	contrib/internal/treesitter/update-language-info.py
	reviewboard/treesitter/_languages.py

Checks run (1 failed, 1 succeeded)

flake8 failed.

JSHint passed.

flake8

reviewboard/treesitter/_languages.py (Diff revision 2)
The issue has been dropped. Show all issues
```
line too long (83 > 79 characters)

Column: 80
Error code: E501
```

contrib/internal/treesitter/update-language-info.py (Diff revision 2)

The issue has been resolved. Show all issues

This and the rest of the variables/classes/functions in this file are missing Version Addeds in their docstrings.

contrib/internal/treesitter/update-language-info.py (Diff revision 2)
The issue has been resolved. Show all issues
```
We should show what the invalid file_suffixes data is in this warning.
```
contrib/internal/treesitter/update-language-info.py (Diff revision 2)
The issue has been resolved. Show all issues
```
Same comment here to show what the invalid first_line_regex is.
```

contrib/internal/treesitter/update-language-info.py (Diff revision 2)

The issue has been resolved. Show all issues

Should we log some warning/debug statement when we encounter OSErrors in this function?

david March 14, 2026, 10:50 a.m.

No, it's expected that one or both of these will fail. I'll add comments.

Commits:

	Summary	ID
	Tree Sitter: Add update-language-info script and resulting data. This change adds a script that loads information about supported languages from a tree-sitter-language-pack checkout. This creates a file, `reviewboard/treesitter/_languages.py`, which has two mappings, `MIME_TYPE_TO_LANGUAGE` and `FILE_SUFFIX_TO_LANGUAGES`, which are used for determining the tree sitter language name to use for a given file. The compiled grammars that are shipped in tree-sitter-language-pack don't include any of the grammar metadata, so this script will go into a a checkout and grab the filename suffixes. This also fetches the "first_line_regex" key, although that is not currently used for detection. Testing Done: Ran script to create the _languages.py file.	d79179711d6e9d6845a3bbf7b28b15c7e5b2ef56
	Tree Sitter: Add update-language-info script and resulting data. This change adds a script that loads information about supported languages from a tree-sitter-language-pack checkout. This creates a file, `reviewboard/treesitter/_languages.py`, which has two mappings, `MIME_TYPE_TO_LANGUAGE` and `FILE_SUFFIX_TO_LANGUAGES`, which are used for determining the tree sitter language name to use for a given file. The compiled grammars that are shipped in tree-sitter-language-pack don't include any of the grammar metadata, so this script will go into a a checkout and grab the filename suffixes. This also fetches the "first_line_regex" key, although that is not currently used for detection. Testing Done: Ran script to create the _languages.py file.	nnsksqorxrnnmnuxklvrymslmwqmpxxt

Diff:

Revision 3 (+2188)

Show changes

	contrib/internal/treesitter/update-language-info.py
	reviewboard/treesitter/_languages.py

Checks run (1 failed, 1 succeeded)

flake8 failed.

JSHint passed.

flake8

reviewboard/treesitter/_languages.py (Diff revision 3)
The issue has been dropped. Show all issues
```
line too long (83 > 79 characters)

Column: 80
Error code: E501
```

Ship it!

```
Ship It!
```

Status:: Completed
Change Summary:: Pushed to master (13610bb)