Summary

Add support in the Trojan Source Checker to disable confusable checks.

Review Request #12784 — Created Jan. 11, 2023 and submitted Jan. 31, 2023, 1:23 a.m.

Information

Owner

chipx86

Repository

Review Board

Branch

release-5.0.x

Bugs

Depends On

Reviewers

Groups

reviewboard

People

Description

This updates the Trojan Source code safety checker to take arguments to
turn off Unicode confusable checks as a whole, or to disable them for
specific Unicode aliases.

To allow this to work, the build-confusables.py script now provides a
mapping of confusable alias names to numeric indexes, and a reverse of
that map. The number indexes are then referenced in the confusable
character map. The checker can then use this to quickly determine which
alias a particular character is part of, and to then exclude that if
provided in the call.

There's no user-level customization at this stage. This is purely an
update to the safety checker code. Additional work is still needed to
allow customization of these settings and to provide it at all the right
stages of diff generation.

There's also a fix that was encountered when posting this change, which
caused a crash when Python failed to identify the name of a Unicode
character.

Testing Done

Unit tests pass on all supported versions of Python.

Tested this along with in-progress changes to allow customization of
which confusables are checked.

Commits

Summary	ID
Add support in the Trojan Source Checker to disable confusable checks. This updates the Trojan Source code safety checker to take arguments to turn off Unicode confusable checks as a whole, or to disable them for specific Unicode aliases. To allow this to work, the `build-confusables.py` script now provides a mapping of confusable alias names to numeric indexes, and a reverse of that map. The number indexes are then referenced in the confusable character map. The checker can then use this to quickly determine which alias a particular character is part of, and to then exclude that if provided in the call. There's no user-level customization at this stage. This is purely an update to the safety checker code. Additional work is still needed to allow customization of these settings and to provide it at all the right stages of diff generation. There's also a fix that was encountered when posting this change, which caused a crash when Python failed to identify the name of a Unicode character.	58190502fe5bbf9b0ac27c58a298e99f948923b6

Summary

Add support in the Trojan Source Checker to disable confusable checks.

This updates the Trojan Source code safety checker to take arguments to turn off Unicode confusable checks as a whole, or to disable them for specific Unicode aliases. To allow this to work, the `build-confusables.py` script now provides a mapping of confusable alias names to numeric indexes, and a reverse of that map. The number indexes are then referenced in the confusable character map. The checker can then use this to quickly determine which alias a particular character is part of, and to then exclude that if provided in the call. There's no user-level customization at this stage. This is purely an update to the safety checker code. Additional work is still needed to allow customization of these settings and to provide it at all the right stages of diff generation. There's also a fix that was encountered when posting this change, which caused a crash when Python failed to identify the name of a Unicode character.

58190502fe5bbf9b0ac27c58a298e99f948923b6

Issues

Description	From	Last Updated
Looks like there's a bug displaying the diff for reviewboard/codesafety/_unicode_confusables.py	david	Jan. 13, 2023, 5:52 p.m.
< 3.9	david	Jan. 13, 2023, 5:53 p.m.
Need to add check_confusables and confusable_aliases_allowed to the args section.	maubin	Jan. 13, 2023, 5:54 p.m.
Should probably add check_confusables and confusable_aliases_allowed to the args section here too.	maubin	Jan. 13, 2023, 5:54 p.m.

flake8 passed.

JSHint passed.

The issue has been resolved. Show all issues

Looks like there's a bug displaying the diff for reviewboard/codesafety/_unicode_confusables.py

chipx86 Jan. 13, 2023, 5:52 p.m.

Yep, I mentioned that in the description. The fix is included in this change.

It's just generated output. Nothing that really needs to be reviewed itself.

contrib/internal/build-confusables.py (Diff revision 1)
The issue has been resolved. Show all issues
```
< 3.9
```

reviewboard/codesafety/checkers/trojan_source.py (Diff revision 1)

The issue has been resolved. Show all issues

Need to add check_confusables and confusable_aliases_allowed to the args section.

reviewboard/codesafety/checkers/trojan_source.py (Diff revision 1)

The issue has been resolved. Show all issues

Should probably add check_confusables and confusable_aliases_allowed to the args section here too.

Change Summary:


Fixed a Python version range in docs.
Added missing argument docs.

Commits:

	Summary	ID
	Add support in the Trojan Source Checker to disable confusable checks. This updates the Trojan Source code safety checker to take arguments to turn off Unicode confusable checks as a whole, or to disable them for specific Unicode aliases. To allow this to work, the `build-confusables.py` script now provides a mapping of confusable alias names to numeric indexes, and a reverse of that map. The number indexes are then referenced in the confusable character map. The checker can then use this to quickly determine which alias a particular character is part of, and to then exclude that if provided in the call. There's no user-level customization at this stage. This is purely an update to the safety checker code. Additional work is still needed to allow customization of these settings and to provide it at all the right stages of diff generation. There's also a fix that was encountered when posting this change, which caused a crash when Python failed to identify the name of a Unicode character.	227f12e5d0240146a100ea7722aebf72586bea87
	Add support in the Trojan Source Checker to disable confusable checks. This updates the Trojan Source code safety checker to take arguments to turn off Unicode confusable checks as a whole, or to disable them for specific Unicode aliases. To allow this to work, the `build-confusables.py` script now provides a mapping of confusable alias names to numeric indexes, and a reverse of that map. The number indexes are then referenced in the confusable character map. The checker can then use this to quickly determine which alias a particular character is part of, and to then exclude that if provided in the call. There's no user-level customization at this stage. This is purely an update to the safety checker code. Additional work is still needed to allow customization of these settings and to provide it at all the right stages of diff generation. There's also a fix that was encountered when posting this change, which caused a crash when Python failed to identify the name of a Unicode character.	58190502fe5bbf9b0ac27c58a298e99f948923b6

Diff:

Revision 2 (+3776 -3140)

Show changes

	contrib/internal/build-confusables.py
	reviewboard/codesafety/_unicode_confusables.py
	reviewboard/codesafety/checkers/trojan_source.py
	reviewboard/codesafety/tests/test_trojan_source_code_safety_checker.py

Checks run (2 succeeded)

flake8 passed.

JSHint passed.

Ship it!

```
Ship It!
```

Ship it!

```
Ship It!
```

Status:: Completed
Change Summary:: Pushed to release-5.0.x (ed18f57)