Regular Expression To Confirm Whether A String Is A Valid Python Identifier?

July 25, 2024 Post a Comment

I have the following definition for an Identifier: Identifier --> letter{ letter| digit} Basically I have an identifier function that gets a string from a file and tests it t

Solution 1:

Question was made 10 years ago, when Python 2 was still dominant. As many comments in the last decade demonstrated, my answer needed a serious update, starting with a big heads up:

No single regex will properly match all (and only) valid Python identifiers. It didn't for Python 2, it doesn't for Python 3.

The reasons are:

As @JoeCondron pointed out, Python reserved keywords such as True, if, return, are not valid identifiers, and regexes alone are unable to handle this, so additional filtering is required.
Python 3 allows non-ascii letters and numbers in an identifier, but the Unicode categories of letters and numbers accepted by the lexical parser for a valid identifier do not match the same categories of \d, \w, \W in the re module, as demonstrated in @martineau's counter-example and explained in great detail by @Hatshepsut's amazing research.

While we could try to solve the first issue using keyword.iskeyword(), as @Alexander Huszagh suggested, and workaround the other by limiting to ascii-only identifiers, why bother using a regex at all?

As Hatshepsut said:

str.isidentifier() works

Just use it, problem solved.

As requested by the question, my original 2012 answer presents a regular expression based on the Python's 2 official definition of an identifier:

identifier ::=  (letter|"_") (letter | digit | "_")*

Which can be expressed by the regular expression:

^[^\d\W]\w*\Z

Example:

import re
identifier = re.compile(r"^[^\d\W]\w*\Z", re.UNICODE)

tests = [ "a", "a1", "_a1", "1a", "aa$%@%", "aa bb", "aa_bb", "aa\n" ]
for test in tests:
    result = re.match(identifier, test)
    print("%r\t= %s" % (test, (result isnotNone)))

Result:

'a'      = True'a1'     = True'_a1'    = True'1a'     = False'aa$%@%' = False'aa bb'  = False'aa_bb'  = True'aa\n'   = False

Solution 2:

str.isidentifier() works. The regex answers incorrectly fail to match some valid python identifiers and incorrectly match some invalid ones.

str.isidentifier() Return true if the string is a valid identifier according to the language definition, section Identifiers and keywords.
Use keyword.iskeyword() to test for reserved identifiers such as def and class.

@martineau's comment gives the example of '℘᧚' where the regex solutions fail.

>>>'℘᧚'.isidentifier()
True
>>>import re>>>bool(re.search(r'^[^\d\W]\w*\Z', '℘᧚'))
False

Why does this happen?

Lets define the sets of code points that match the given regular expression, and the set that match str.isidentifier.

import re
import unicodedata

chars = {chr(i) for i inrange(0x10ffff) if re.fullmatch(r'^[^\d\W]\w*\Z', chr(i))}
identifiers = {chr(i) for i inrange(0x10ffff) ifchr(i).isidentifier()}

How many regex matches are not identifiers?

In [26]: len(chars - identifiers)                                                                                                               
Out[26]: 698

How many identifiers are not regex matches?

In [27]: len(identifiers - chars)                                                                                                               
Out[27]: 4

Interesting -- which ones?

In [37]: {(c, unicodedata.name(c), unicodedata.category(c)) for c in identifiers - chars}                                                       
Out[37]: 
set([
    ('\u1885', 'MONGOLIAN LETTER ALI GALI BALUDA', 'Mn'),
    ('\u1886', 'MONGOLIAN LETTER ALI GALI THREE BALUDA', 'Mn'),
    ('℘', 'SCRIPT CAPITAL P', 'Sm'),
    ('℮', 'ESTIMATED SYMBOL', 'So'),
])

What's different about these two sets?

They have different Unicode "General Category" values.

In [31]: {unicodedata.category(c) for c in chars - identifiers}                                                                                 
Out[31]: set(['Lm', 'Lo', 'No'])

From wikipedia, that's Letter, modifier; Letter, other; Number, other. This is consistent with the re docs, since \d is only decimal digits:

\d Matches any Unicode decimal digit (that is, any character in Unicode character category [Nd])

What about the other way?

In [32]: {unicodedata.category(c) for c in identifiers - chars}                                                                                 
Out[32]: set(['Mn', 'Sm', 'So'])

That's Mark, nonspacing; Symbol, math; Symbol, other.

Where is this all documented?

Where is it implemented?

https://github.com/python/cpython/commit/47383403a0a11259acb640406a8efc38981d2255

I still want a regular expression

Look at the regex module on PyPI.

This regex implementation is backwards-compatible with the standard ‘re’ module, but offers additional functionality.

It includes filters for "General Category".

Solution 3:

For Python 3, you need to handle Unicode letters and digits. So if that's a concern, you should get along with this:

re_ident = re.compile(r"^[^\d\W]\w*$", re.UNICODE)

[^\d\W] matches a character that is not a digit and not "not alphanumeric" which translates to "a character that is a letter or underscore".

Solution 4:

\w matches digits and characters. Try ^[_a-zA-Z]\w*$

Solution 5:

Works like a charm: r'[^\d\W][\w\d]+'

lacucinadiadine