Update CredentialChecker for new tool support and parsing improvements.

 






 

 


  Review Request #11579
  —
  Created April 6, 2021 and submitted 

 




















 

  

   Information

  





 
 

  chipx86
 









 
 

  ReviewBot
 









 
 

  release-3.0.x
 









 
 

  
 









 
 

  
 






 




 

  

   Reviewers

  





 
 

  reviewbot
 









 
 

  
 






 








 





  


 
 

  
This updates CredentialChckerTool to subclass the newer BaseTool.


The original implementation had some regex patterns that didn't survive

the later stages of development, when combining into a single verbose

regex. Any pattern that expected a space (such as PGP private key

blocks) failed to match. This is now fixed.


The GitHub token checks have been greatly improved. As of April 2021,

GitHub introduced a new token format that can more easily be scanned,

and contains a checksum as the final 6 characters of the token, allowing

a scanner to detect real vs. fake tokens (though not active vs.

inactive). We now support enhanced validation for these GitHub tokens,

and have logic in place to allow for enhanced validation for future

tokens as well.


Most previous token types have been updated for modern token formats and

to reduce false-positives by preventing the tokens from being embedded

in other strings.


Token support has been added for:


    

  • Asana
    • 

  • Discord
    • 

  • Dropbox
    • 

  • Google Cloud Platform
    • 

  • JSON Web Tokens
    • 

  • Mailchimp
    • 

  • Mailgun
    • 

  • NPM
    • 

  • PyPI
    • 

  • Slack (WebHooks)
    • 

  • SSL Certificates
    • 

  • Stripe
    • 

  • Twilio
    • 


 








  


 
 

  
Unit tests pass on Python 2.7 and 3.x.


Extensively tested the new GitHub token validation logic. I've tested this

with a number of real-world tokens, along with garbage tokens, ensuring that

the real ones always validated and matched.

 







 









 





  

 


 
 

  



 

  
   


    Summary
    ID

   

  
  

   


    

     

      Update CredentialChecker for new tool support and parsing improvements.
      
This updates `CredentialChckerTool` to subclass the newer `BaseTool`.

The original implementation had some regex patterns that didn't survive
the later stages of development, when combining into a single verbose
regex. Any pattern that expected a space (such as PGP private key
blocks) failed to match. This is now fixed.

The GitHub token checks have been greatly improved. As of April 2021,
GitHub introduced a new token format that can more easily be scanned,
and contains a checksum as the final 6 characters of the token, allowing
a scanner to detect real vs. fake tokens (though not active vs.
inactive). We now support enhanced validation for these GitHub tokens,
and have logic in place to allow for enhanced validation for future
tokens as well.

Most previous token types have been updated for modern token formats and
to reduce false-positives by preventing the tokens from being embedded
in other strings.

New token have been added for:

* Asana
* Discord
* Dropbox
* Google Cloud Platform
* JSON Web Tokens
* Mailchimp
* Mailgun
* NPM
* PyPI
* Slack (WebHooks)
* SSL Certificates
* Stripe
* Twilio

     


    		
    95d6da275c2637656ff80ceb6fb41d78f220b8b2

   



  
 




 













 






 
 
 

  

   

    

     
    

    
    
     
  • 
      
     
    • 
     
  • 
      
     
    • 
     
  • 
      
     
    • 
     
  • 
      
     
    • 
     
  • 
      
     
    • 
    

   

   

    
     

      Description
      From
      Last Updated
     

    
    



     

      
       
       
E501 line too long (93 > 79 characters)

      		
      
       reviewbotreviewbot
      
      
       
      
     




     

      
       
       
E501 line too long (80 > 79 characters)

      		
      
       reviewbotreviewbot
      
      
       
      
     


    
   

  

 







 






  

 



















 




 




 














 





 

  

   




david




   

   







  

 



 

  


   

   

    
 
david


    
   


  

  

  



    




  1. 
 
    
  
    
   
    Ship It!
    
  
    
 
    

 
    
  


    
 
    
 
      

  

 
    


    

 
    

    2. 















  2. 
 
    
  
    
   
    
  
    
 
    

 
    
  


    
 
    
 
      

  

 
    


    

 
    

    3. 




  

 
















 



 

  

   




chipx86




   

   

  

 



 

  


   

   

    
 Review request changed

    
   


  

  

  









  

   

    Change Summary:
   

   

    
    

  • Fixed up many of the previously-implemented tokens to reduce false-positives and match current token formats for services.
    • 

  • Grouped tokens by sections (Vendor-identifiable, vendor-likely, and general patterns), for priority matching.
    • 

  • Added "as of" dates for the tokens, to ease future maintenance.
    • 

  • Added a blurb to the comment stating that the user may want to revoke the existing token.
    • 

  • Added a bunch of new supported token types.
    • 


   

  








  

   

    Description:
   

   

    
     
This updates CredentialChckerTool to subclass the newer BaseTool.
     

     
The original implementation had some regex patterns that didn't survive
     the later stages of development, when combining into a single verbose
     regex. Any pattern that expected a space (such as PGP private key
     blocks) failed to match. This is now fixed.
     

     
The GitHub token checks have been greatly improved. As of April 2021,
     GitHub introduced a new token format that can more easily be scanned,
     and contains a checksum as the final 6 characters of the token, allowing
     a scanner to detect real vs. fake tokens (though not active vs.
     inactive). We now support enhanced validation for these GitHub tokens,
     and have logic in place to allow for enhanced validation for future
     tokens as well.
   + 

   + 
Most previous token types have been updated for modern token formats and
   + to reduce false-positives by preventing the tokens from being embedded
   + in other strings.
   + 

   + 
Token support has been added for:
   + 

   + 
  • Asana
   + 
  • Discord
   + 
  • Dropbox
   + 
  • Google Cloud Platform
   + 
  • JSON Web Tokens
   + 
  • Mailchimp
   + 
  • Mailgun
   + 
  • NPM
   + 
  • PyPI
   + 
  • Slack (WebHooks)
   + 
  • SSL Certificates
   + 
  • Stripe
   + 
  • Twilio

   

  





  

   

    Commits:
   

   

    





 

  
   


    

    Summary
    ID

   

  
  

   


    

    

     

      Update CredentialChecker for new tool support and parsing improvements.
      
This updates `CredentialChckerTool` to subclass the newer `BaseTool`.

The original implementation had some regex patterns that didn't survive
the later stages of development, when combining into a single verbose
regex. Any pattern that expected a space (such as PGP private key
blocks) failed to match. This is now fixed.

The GitHub token checks have been greatly improved. As of April 2021,
GitHub introduced a new token format that can more easily be scanned,
and contains a checksum as the final 6 characters of the token, allowing
a scanner to detect real vs. fake tokens (though not active vs.
inactive). We now support enhanced validation for these GitHub tokens,
and have logic in place to allow for enhanced validation for future
tokens as well.

     


    		
    f21c19da7d3af256ce67f8f48f53a38fd68b90b8

   




   


    

    

     

      Update CredentialChecker for new tool support and parsing improvements.
      
This updates `CredentialChckerTool` to subclass the newer `BaseTool`.

The original implementation had some regex patterns that didn't survive
the later stages of development, when combining into a single verbose
regex. Any pattern that expected a space (such as PGP private key
blocks) failed to match. This is now fixed.

The GitHub token checks have been greatly improved. As of April 2021,
GitHub introduced a new token format that can more easily be scanned,
and contains a checksum as the final 6 characters of the token, allowing
a scanner to detect real vs. fake tokens (though not active vs.
inactive). We now support enhanced validation for these GitHub tokens,
and have logic in place to allow for enhanced validation for future
tokens as well.

Most previous token types have been updated for modern token formats and
to reduce false-positives by preventing the tokens from being embedded
in other strings.

New token have been added for:

* Asana
* Discord
* Dropbox
* Google Cloud Platform
* JSON Web Tokens
* Mailchimp
* Mailgun
* NPM
* PyPI
* Slack (WebHooks)
* SSL Certificates
* Stripe
* Twilio

     


    		
    95d6da275c2637656ff80ceb6fb41d78f220b8b2

   



  
 







   

  





  

   

    Diff:
   

   

    
 Revision 2 (+1954 -80)
Show changes
 
    bot/reviewbot/tools/rbcredentialchecker.py 
    bot/reviewbot/tools/testing/testcases.py 
    bot/reviewbot/tools/tests/test_rbcredentialchecker.py 
    bot/reviewbot/utils/text.py 
    bot/reviewbot/utils/tests/test_text.py 
 

   

  











 

  
Checks run (1 failed, 1 succeeded)



  

 flake8
 failed.
 





  

 JSHint
 passed.
 





 







 
flake8

 


 















  

 
















 



 

  

   




chipx86




   

   

  

 



 

  


   

   

    
 Review request changed

    
   


  

  

  







  

   

    Status:
   

   


    Completed

   

  




  

   

    Change Summary:
   

   

    
Pushed to release-3.0.x (ea1f4a7)