… in which we look at one or two ways to make life easier
when working with Python regular expressions.

tl;dr: You can compose verbose regular expressions using f‍-‍strings.

Here’s a real-world example – instead of this:

pattern = r"((?:(s*)?[A-Z]*Hd+[a-z]*(?:s*+s*[A-Z]*Hd+[a-z]*)*(?:s*[):+])?)(.*?)(?=(?:(s*)?[A-Z]*Hd+[a-z]*(?:s*+s*[A-Z]*Hd+[a-z]*)*(?:s*[):+])?(?![^ws])|$)"

… do this:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22

code = r"""
[A-Z]*H  # prefix
d+      # digits
[a-z]*   # suffix
"""

multicode = fr"""
(?: ( s* )?               # maybe open paren and maybe space
{code}                      # one code
(?: s* + s* {code} )*    # maybe followed by other codes, plus-separated
(?: s* [):+] )?           # maybe space and maybe close paren or colon or plus
"""

pattern = fr"""
( {multicode} )             # code (capture)
( .*? )                     # message (capture): everything ...
(?=                         # ... up to (but excluding) ...
    {multicode}             # ... the next code
        (?! [^ws] )       # (but not when followed by punctuation)
    | $                     # ... or the end
)
"""

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60

pattern = r"""
(                       # code (capture)
    # BEGIN multicode

    (?: ( s* )?       # maybe open paren and maybe space

    # code
    [A-Z]*H  # prefix
    d+      # digits
    [a-z]*   # suffix

    (?:                 # maybe followed by other codes,
        s* + s*      # ... plus-separated

        # code
        [A-Z]*H  # prefix
        d+      # digits
        [a-z]*   # suffix
    )*

    (?: s* [):+] )?   # maybe space and maybe close paren or colon or plus

    # END multicode
)

( .*? )                 # message (capture): everything ...

(?=                     # ... up to (but excluding) ...
    # ... the next code

    # BEGIN multicode

    (?: ( s* )?       # maybe open paren and maybe space

    # code
    [A-Z]*H  # prefix
    d+      # digits
    [a-z]*   # suffix

    (?:                 # maybe followed by other codes,
        s* + s*      # ... plus-separated

        # code
        [A-Z]*H  # prefix
        d+      # digits
        [a-z]*   # suffix
    )*

    (?: s* [):+] )?   # maybe space and maybe close paren or colon or plus

    # END multicode

        # (but not when followed by punctuation)
        (?! [^ws] )

    # ... or the end
    | $
)

"""

Read on for details and some caveats.

Prerequisites

Formatted string literals (f‍-‍strings) were added in Python 3.6,
and provide a way to embed expressions inside string literals,
using a syntax similar to that of str.format():

>>> name = "world"
>>>
>>> "Hello, {name}!".format(name=name)
'Hello, world!'
>>>
>>> f"Hello, {name}!"
'Hello, world!'

Verbose regular expressions (re.VERBOSE) have been around since forever,
and allow writing regular expressions
with non-significant whitespace and comments:

>>> text = "H1 code (AH2b+EUH3) fancy code"
>>>
>>> code = r"[A-Z]*Hd+[a-z]*"
>>> re.findall(code, text)
['H1', 'AH2b', 'EUH3']
>>>
>>> code = r"""
... [A-Z]*H  # prefix
... d+      # digits
... [a-z]*   # suffix
... """
>>> re.findall(code, text, re.VERBOSE)
['H1', 'AH2b', 'EUH3']

The “one weird trick”

Once you see it, it’s obvious
– you can use f‍-‍strings to compose regular expressions:

>>> multicode = fr"""
... (?: ( )?         # maybe open paren
... {code}            # one code
... (?: + {code} )*  # maybe other codes, plus-separated
... (?: ) )?         # maybe close paren
... """
>>> re.findall(multicode, text, re.VERBOSE)
['H1', '(AH2b+EUH3)']

It’s so obvious, it only took me three years to do it
after I started using Python 3.6+,
despite using both features during all that time.

Of course, there’s any number of libraries
for building regular expressions;
the benefit of this is that it has zero dependencies,
and zero extra things you need to learn.

Caveats

Hashes and spaces need to be escaped

Because a hash is used to mark the start of a comment,
and spaces are mostly ignored,
you have to represent them in some other way.

The documentation of re.VERBOSE is quite helpful:

When a line contains a # that is not in a character class and is not preceded by an unescaped backslash, all characters from the leftmost such # through the end of the line are ignored.

That is, this won’t work as the non-verbose version:

>>> re.findall("d+#d+", "1#23a")
['1#23']
>>> re.findall("d+ # d+", "1#23a", re.VERBOSE)
['1', '23']

… but these will:

>>> re.findall("d+ [#] d+", "1#23a", re.VERBOSE)
['1#23']
>>> re.findall("d+ # d+", "1#23a", re.VERBOSE)
['1#23']

The same is true for spaces:

>>> re.findall("d+ [ ] d+", "1 23a", re.VERBOSE)
['1 23']
>>> re.findall("d+   d+", "1 23a", re.VERBOSE)
['1 23']

When composing regexes,
ending a pattern on the same line as a comment
might accidentally comment the following line in the enclosing pattern:

>>> one = "1 # comment"
>>> onetwo = f"{one} 2"
>>> re.findall(onetwo, '0123', re.VERBOSE)
['1']
>>> print(onetwo)
1 # comment 2

This can be avoided by always ending the pattern on a new line:

>>> one = """
... 1 # comment
... """
>>> onetwo = f"""
... {one} 2
... """
>>> re.findall(onetwo, '0123', re.VERBOSE)
['12']

While a bit cumbersome,
in real life most patterns would span multiple lines anyway,
so it’s not really an issue.

(Note that this is only needed if you use comments.)

Brace quantifiers need to be escaped

Because f‍-‍strings already use braces for replacements,
to represent brace quantifiers you must double the braces:

>>> re.findall("m{2}", "entire mm but only two of mmm")
['mm', 'mm']
>>> letter = "m"
>>> pattern = f"{letter}{{2}}"
>>> re.findall(pattern, "entire mm but only two of mmm")
['mm', 'mm']

I don’t control the flags

Maybe you’d like to use verbose regexes,
but don’t control the flags passed to the re functions
(for example, because you’re passing the regex to an API).

Worry not! The regular ex