Ambiguity In Parsing Comma As A Operator Using Ply Python

I have following tokens and many more, but I want to keep my question short that's why not including the whole code. tokens = ( 'COMMA', 'OP', 'FUNC1', 'FUNC2' ) def t_OP(t):

Solution 1:

Ply has no way to know whether you want a given , to be the lexeme COMMA, or the lexeme OP. Or, rather, it has a way, but it will always choose the same one: OP. That's because patterns in token functions are tested before tokens in pattern variables.

I'm assuming you have t_COMMA = r',' somewhere in the part of your program you did not provide. It is also possible that you have a token function to recognise COMMA, in which case whichever function comes first will win. But however you do it, the order the regexes are tested is fixed, so either , is always COMMA or it is always OP. This is well explained in the Ply manual section on Specification of Tokens.

Personally, I'd suggest removing the comma from OP and modifying the grammar to use COMMA in the definition of expression. As you observed, you will get shift-reduce conflicts so you must include it in your precedence declaration (which you have also chosen to omit from your question). In fact, it seems likely that you would want to have different precedences for different operators, so you will probably want to separate the different operators into different tokens, since that is precedence is by the token. See the explanation in the Ply manual section on precedence declarations.

Solution 2:

Adding one more rule like solved my problem :

expression:expression COMMA expression

added because as @rici told, in expression like FUNC2("hello",FUNC1("ghost")) the first comma is always taken as operator.

and adding precedence thing removed 4shift/reduce conflicts.

precedence = (
    ('left','COMMA'),
    ('left','OP')
)