[TIP] About testing Pygments lexers

Olemis Lang olemis at gmail.com
Wed Aug 25 13:05:01 PDT 2010

On Wed, Aug 25, 2010 at 1:46 PM, Tim Hatch <tim at timhatch.com> wrote:
> On 8/25/10 9:47 AM, Olemis Lang wrote:
>> Hello , I don't recall if I already asked this but I'm looking for a
>> test lib able to detect if a Pygments lexer (e.g. using Regex lexer
>> rules) will be stuck
> By "stuck" do you mean an infinite loop, or an error token?

I have experienced this recently . Sometimes when the lexer finds no
match inside a state (i.e. there's no empty regex ;o) it doesn't
consume chars and it stays in that state until forever .

> Off the top
> of my head, the only case where an infinite loop can happen is if a
> regex can match the empty string [although empty string to enter a new
> state is used by a couple of lexers, and is probably safe].

Yes I use empty regex quite often to abort an state without evident
end marker (e.g. comma-delimited list of identifiers ;o)

>> and also able to generate state & rule coverage
>> by running tests
> The best place to do this would be inside RegexLexer's
> get_tokens_unprocessed method.  Although the code hasn't changed since
> 2007, IMO the best way to add coverage would be to replace the match
> function ('rexmatch' there, self._tokens[...][...][0] more generally)
> with your own function using a subclass:

Thnx , I'll analyze your example to figure out if it can help ;o)



Blog ES: http://simelo-es.blogspot.com/
Blog EN: http://simelo-en.blogspot.com/

Featured article:

More information about the testing-in-python mailing list