Enable detection of variable rename on Diff Viewer.

Review Request #1701 — Created July 3, 2010 and discarded — Latest diff uploaded

Information

Review Board

Reviewers

Detection is only active if the user also decides to enable syntax
highlight. The rationale is that if he wants to pay for a time
consuming feature, he can also pay for the other and both features
depends of Pygments for tokenization.

That being said, the most time consuming part is the tokenization.
I did write a tokenizer by hand (based on regular expressions). but
I got caught up in several corner cases (such as the use of $ as a valid
token in javascript). Latter on, I just figured that it would be better to
use a more robust, albeit heavier, tokenizer to do that job.

As a guideline, renames should be one to one, AKA one token changing to another.
If there are multiple renames on the same line, it's a special case that shouldn't be
overlooked by the reviewer.

Also refactor lexer guessing out of apply_pygments so we have to guess it only once.
I'll be adding a test case here, for future automatic testing, but I did test it on several types of diffs.