Update CredentialChecker for new tool support and parsing improvements.

Review Request #11579 — Created April 6, 2021 and submitted

Information

ReviewBot
release-3.0.x

Reviewers

This updates CredentialChckerTool to subclass the newer BaseTool.

The original implementation had some regex patterns that didn't survive
the later stages of development, when combining into a single verbose
regex. Any pattern that expected a space (such as PGP private key
blocks) failed to match. This is now fixed.

The GitHub token checks have been greatly improved. As of April 2021,
GitHub introduced a new token format that can more easily be scanned,
and contains a checksum as the final 6 characters of the token, allowing
a scanner to detect real vs. fake tokens (though not active vs.
inactive). We now support enhanced validation for these GitHub tokens,
and have logic in place to allow for enhanced validation for future
tokens as well.

Most previous token types have been updated for modern token formats and
to reduce false-positives by preventing the tokens from being embedded
in other strings.

Token support has been added for:

  • Asana
  • Discord
  • Dropbox
  • Google Cloud Platform
  • JSON Web Tokens
  • Mailchimp
  • Mailgun
  • NPM
  • PyPI
  • Slack (WebHooks)
  • SSL Certificates
  • Stripe
  • Twilio

Unit tests pass on Python 2.7 and 3.x.

Extensively tested the new GitHub token validation logic. I've tested this
with a number of real-world tokens, along with garbage tokens, ensuring that
the real ones always validated and matched.

Summary ID
Update CredentialChecker for new tool support and parsing improvements.
This updates `CredentialChckerTool` to subclass the newer `BaseTool`. The original implementation had some regex patterns that didn't survive the later stages of development, when combining into a single verbose regex. Any pattern that expected a space (such as PGP private key blocks) failed to match. This is now fixed. The GitHub token checks have been greatly improved. As of April 2021, GitHub introduced a new token format that can more easily be scanned, and contains a checksum as the final 6 characters of the token, allowing a scanner to detect real vs. fake tokens (though not active vs. inactive). We now support enhanced validation for these GitHub tokens, and have logic in place to allow for enhanced validation for future tokens as well. Most previous token types have been updated for modern token formats and to reduce false-positives by preventing the tokens from being embedded in other strings. New token have been added for: * Asana * Discord * Dropbox * Google Cloud Platform * JSON Web Tokens * Mailchimp * Mailgun * NPM * PyPI * Slack (WebHooks) * SSL Certificates * Stripe * Twilio
95d6da275c2637656ff80ceb6fb41d78f220b8b2
Description From Last Updated

E501 line too long (93 > 79 characters)

reviewbotreviewbot

E501 line too long (80 > 79 characters)

reviewbotreviewbot
david
  1. Ship It!
  2. 
      
chipx86
Review request changed
Change Summary:
  • Fixed up many of the previously-implemented tokens to reduce false-positives and match current token formats for services.
  • Grouped tokens by sections (Vendor-identifiable, vendor-likely, and general patterns), for priority matching.
  • Added "as of" dates for the tokens, to ease future maintenance.
  • Added a blurb to the comment stating that the user may want to revoke the existing token.
  • Added a bunch of new supported token types.
Description:
   

This updates CredentialChckerTool to subclass the newer BaseTool.

   
   

The original implementation had some regex patterns that didn't survive

    the later stages of development, when combining into a single verbose
    regex. Any pattern that expected a space (such as PGP private key
    blocks) failed to match. This is now fixed.

   
   

The GitHub token checks have been greatly improved. As of April 2021,

    GitHub introduced a new token format that can more easily be scanned,
    and contains a checksum as the final 6 characters of the token, allowing
    a scanner to detect real vs. fake tokens (though not active vs.
    inactive). We now support enhanced validation for these GitHub tokens,
    and have logic in place to allow for enhanced validation for future
    tokens as well.

  +
  +

Most previous token types have been updated for modern token formats and

  + to reduce false-positives by preventing the tokens from being embedded
  + in other strings.

  +
  +

Token support has been added for:

  +
  +
  • Asana
  +
  • Discord
  +
  • Dropbox
  +
  • Google Cloud Platform
  +
  • JSON Web Tokens
  +
  • Mailchimp
  +
  • Mailgun
  +
  • NPM
  +
  • PyPI
  +
  • Slack (WebHooks)
  +
  • SSL Certificates
  +
  • Stripe
  +
  • Twilio
Commits:
Summary ID
Update CredentialChecker for new tool support and parsing improvements.
This updates `CredentialChckerTool` to subclass the newer `BaseTool`. The original implementation had some regex patterns that didn't survive the later stages of development, when combining into a single verbose regex. Any pattern that expected a space (such as PGP private key blocks) failed to match. This is now fixed. The GitHub token checks have been greatly improved. As of April 2021, GitHub introduced a new token format that can more easily be scanned, and contains a checksum as the final 6 characters of the token, allowing a scanner to detect real vs. fake tokens (though not active vs. inactive). We now support enhanced validation for these GitHub tokens, and have logic in place to allow for enhanced validation for future tokens as well.
f21c19da7d3af256ce67f8f48f53a38fd68b90b8
Update CredentialChecker for new tool support and parsing improvements.
This updates `CredentialChckerTool` to subclass the newer `BaseTool`. The original implementation had some regex patterns that didn't survive the later stages of development, when combining into a single verbose regex. Any pattern that expected a space (such as PGP private key blocks) failed to match. This is now fixed. The GitHub token checks have been greatly improved. As of April 2021, GitHub introduced a new token format that can more easily be scanned, and contains a checksum as the final 6 characters of the token, allowing a scanner to detect real vs. fake tokens (though not active vs. inactive). We now support enhanced validation for these GitHub tokens, and have logic in place to allow for enhanced validation for future tokens as well. Most previous token types have been updated for modern token formats and to reduce false-positives by preventing the tokens from being embedded in other strings. New token have been added for: * Asana * Discord * Dropbox * Google Cloud Platform * JSON Web Tokens * Mailchimp * Mailgun * NPM * PyPI * Slack (WebHooks) * SSL Certificates * Stripe * Twilio
95d6da275c2637656ff80ceb6fb41d78f220b8b2

Checks run (1 failed, 1 succeeded)

flake8 failed.
JSHint passed.

flake8

chipx86
Review request changed
Status:
Completed
Change Summary:
Pushed to release-3.0.x (ea1f4a7)