Regex Pattern Explainer and Tester

---
name: regex-pattern-explainer-and-tester
description: Explains Python regular expressions token by token, tests them on must-match and must-not-match samples with spans and named groups, and flags risky patterns: nested quantifiers that backtrack catastrophically, [A-z], unescaped dots, empty matches, and missing anchors, then proposes a safer rewrite. Use when a user asks "what does this regex do?", "why doesn't it match?", or "is this regex safe?". Includes a tested stdlib Python checker.
---

# Regex Pattern Explainer and Tester

You help people understand, fix, and trust regular expressions. You explain a pattern in plain words, prove what it does with real test cases, and find the traps that make regexes fail in production: inputs that should not pass, inputs that should pass but do not, and patterns that hang on long input. You return a corrected pattern with a test table, not just an opinion.

## Files in this skill

- `scripts/regex_check.py` - explains a pattern from Python's own regex parser, runs sample inputs with "+" (must match) and "-" (must not match) expectations, and reports HIGH, WARN, and INFO risks; `--probe` times the pattern on generated worst-case input in a subprocess (Python 3 standard library only)
- `references/regex-pitfalls.md` - every risk the checker reports, why it matters, and the safe pattern to use instead
- `references/flavor-differences.md` - how Python `re` differs from JavaScript, PCRE, Go RE2, .NET, Java, and POSIX grep, so a fix works where the user runs it
- `templates/regex-review.md` - the review report to return
- `examples/example-email-validation-review.md` - a worked review of an email check that hangs on long input

## Workflow

### 1. Collect the facts
Ask, or assume and say so:
- The pattern exactly as written in the code (raw string or escaped string literal) and the language or tool that runs it.
- What it must accept and what it must reject: 4 to 10 real examples of each, including edge cases (empty input, spaces, Unicode, very long input).
- How it is used: validate a whole value (fullmatch), find the first match (search), extract all matches (findall), or replace.
- Where the input comes from. Untrusted input (web forms, logs, API payloads) makes catastrophic backtracking a security issue.

### 2. Run the checker
```bash
python3 scripts/regex_check.py 'PATTERN' -t 'sample one' -t 'sample two'
python3 scripts/regex_check.py 'PATTERN' --cases cases.txt --mode fullmatch --flags a
python3 scripts/regex_check.py 'PATTERN' --probe --no-explain         # confirm a backtracking risk
python3 scripts/regex_check.py 'PATTERN' --cases - --json < cases.txt
```
A cases file has one sample per line: `+ text` must match, `- text` must not match, `# ...` is a comment. Always quote the pattern in single quotes in the shell. The checker uses Python's `re`; for other flavors, read `references/flavor-differences.md` and translate the result. If you cannot run the script, explain the pattern by hand with the same structure and apply `references/regex-pitfalls.md`.

### 3. Explain, then fix
1. Explain the pattern top to bottom in plain words, one line per token or group, using the checker's explanation as the base.
2. Show the test table: each sample, expected result, actual result, and the matched text or groups.
3. Fix in this order: patterns that can hang (HIGH), wrong accepts and wrong rejects (failed expectations), then clarity and the INFO notes.
4. Prefer simple, readable rewrites: one repeat instead of nested ones, negated classes like `[^,]*` instead of `.*`, explicit anchors `\A...\Z` or `fullmatch`, and `re.VERBOSE` with comments for long patterns.
5. Rerun the checker on the corrected pattern with the same cases plus any new edge cases until every expectation passes and there is no HIGH risk. Run `--probe` on the fix if the original had a HIGH risk.

### 4. Deliver
Fill in `templates/regex-review.md`: the plain-language explanation, the test table before and after, the risks found, the corrected pattern in the user's language with correct escaping, and a short list of test cases to keep in their test suite. Follow the style of `examples/example-email-validation-review.md`.

## Rules
- Never claim a pattern is correct without showing it on test cases; add the edge cases the user forgot.
- Do not over-validate: for emails, URLs, and names, recommend a simple shape check plus a real verification step (sending an email, parsing the URL) instead of a giant pattern.
- Keep the user's flavor: do not suggest lookbehind, possessive quantifiers, or atomic groups where their engine does not support them.
- Escape correctly for the target language (raw strings in Python, double backslashes in Java and C# strings, slashes in JavaScript literals).
- Treat catastrophic backtracking on untrusted input as a security bug (ReDoS) and say so plainly; suggest input length limits as a second defense.
FILE:references/regex-pitfalls.md
# Regex pitfalls checked by the script

Each rule: what is detected, why it matters, and the pattern to use instead. Levels: HIGH can hang or be exploited, WARN usually accepts or rejects the wrong input, INFO is a habit worth fixing.

## HIGH (catastrophic backtracking, ReDoS)

### nested-quantifier
Detected: a repeat inside another repeat where nothing else in the outer repeat is required, such as `(a+)+`, `(\w+\s?)+`, `([a-z0-9]+\.?)+`, `(\w+|\d+)+`.
Why: on input that almost matches (a long run of letters followed by a char the pattern rejects), the engine tries every way to split the run between the inner and outer repeat: 2^n ways for n chars. 30 chars can take minutes. On a web form or API this is a denial-of-service bug.
Use instead: one repeat that describes the whole run, with the separator required.
```
(\w+\s?)+$          ->  \w+(?:\s\w+)*$
([a-z0-9]+\.?)+@    ->  [a-z0-9]+(?:\.[a-z0-9]+)*@
```
Python 3.11+ also supports atomic groups `(?>...)` and possessive quantifiers `a++`, which never give back what they matched.

### ambiguous-repeat
Detected: a repeat whose body has variable length and whose required parts can also be eaten by the variable part, such as `(x+x+)+`, `(aa|a)+`, `(foo|f\w+)+`.
Why: same as above, the iterations can be cut in many ways. Growth may be slower (Fibonacci instead of 2^n) but is still exponential.
Use instead: make each iteration start with a char nothing else in the body can match, or merge the parts into one class.

Confirm both with `--probe`: the script runs the pattern on `c * n + tail` for n = 12 to 28 in a subprocess with a 2 second limit and reports how fast the time grows. A growth of more than 3x per 4 extra chars, or a timeout, confirms the risk.

## WARN

### nested-overlap
A repeat whose body has a separator that the inner part also accepts, such as `([\w.]+\.)+`. Make the inner class exclude the separator: `(?:[\w-]+\.)+`.

### adjacent-overlap
Two unbounded single-char repeats in a row that accept the same chars, such as `\d+\d+` or `.*.*`. The split point is ambiguous and a failing match backtracks polynomially (n^2, n^3). Keep one of them or make the boundary explicit.

### range-A-z
`[A-z]` covers ASCII 65 to 122, which includes `[ \ ] ^ _` and the backtick. Use `[A-Za-z]`, or `[^\W\d_]` for any Unicode letter.

### unescaped-dot
`.` between letters or before a word such as `com` or `json` matches any char, so `example-com` passes a check for `example.com`. Write `\.`.

### matches-empty
The pattern accepts the empty string. A validation then accepts blank input, and `findall` returns empty strings between real matches. Use `+` instead of `*` where at least one char is required.

## INFO

### not-anchored
`re.search(r"\d+", "abc123xyz")` succeeds. For validating a whole value, use `re.fullmatch` or anchor with `\A ... \Z`.

### dollar-newline
`$` matches at the end of the string and also just before a final newline, so `^abc$` accepts `"abc\n"` with `re.search` or `re.match`. Use `\Z` or `re.fullmatch` for strict checks.

### unicode-classes
In Python 3, `\d` matches any Unicode digit (for example Arabic-Indic digits) and `\w` matches letters in every script. Good for names and free text; for IDs, card numbers, and codes use `re.ASCII` or `[0-9]`.

### greedy-dotstar and many-dotstar
`.*` takes everything and then gives back one char at a time. Three or more `.*` in one pattern make failures slow on long lines. Describe each field with a negated class: `"[^"]*"` for a quoted string, `[^,]*` for a CSV field.

## Not detected by the script (check by hand)

- Wrong escaping in the host language (a Python string without `r""`, a Java string with single backslashes).
- Business rules a regex cannot express well: valid dates (February 30), checksums (IBAN, card numbers), real email deliverability.
- Locale issues: case-insensitive matching of Turkish dotted and dotless i, German sharp s.
- Performance of huge alternations (thousands of words): use a set lookup or a trie instead.
FILE:references/flavor-differences.md
# Regex flavor differences that break fixes

The checker runs Python `re`. Before handing a pattern to another engine, check this table.

| Feature | Python `re` | JavaScript | PCRE / PHP | Go (RE2) | .NET | Java | POSIX grep -E |
|---|---|---|---|---|---|---|---|
| Lookahead `(?=...)` | yes | yes | yes | no | yes | yes | no |
| Lookbehind `(?<=...)` | fixed width only | yes (ES2018) | fixed or bounded | no | any width | bounded | no |
| Named group | `(?P<name>...)` only | `(?<name>...)` | both | `(?P<name>...)` (also `(?<name>...)` in Go 1.22+) | `(?<name>...)` | `(?<name>...)` | no |
| Atomic group `(?>...)` | 3.11+ | no | yes | no | yes | yes | no |
| Possessive `a++` | 3.11+ | no | yes | no | no | yes | no |
| Backreference `\1` | yes | yes | yes | no | yes | yes | yes (basic) |
| `\d` matches non-ASCII digits | yes (unless re.ASCII) | no | no (unless UCP) | no | yes | no (unless UNICODE_CHARACTER_CLASS) | locale |
| `$` before final newline | yes | no (without m flag) | yes | no | yes | yes | n/a (line based) |
| Inline flags `(?i)` | yes, at start or scoped `(?i:...)` | flags after the literal (`/i`); scoped `(?i:...)` only in the newest engines | yes | yes | yes | yes | no |
| Catastrophic backtracking | possible | possible | possible (has limits) | impossible (linear time) | possible (timeout option) | possible | engine dependent |
| Full match helper | `re.fullmatch` | anchor with `^...$` | anchor | anchor | anchor | `String.matches()` | `-x` flag |

## Translation notes

- JavaScript: write the literal as `/pattern/flags`; escape `/` inside it. Use the `u` flag for Unicode and `v` for set operations in newer engines. `\Z` does not exist: use `$` without the `m` flag.
- Go RE2: no lookarounds or backreferences, but every pattern runs in linear time, so ReDoS is not a concern there. Rewrite lookarounds as a second check in code.
- Java and C#: every backslash doubles in a normal string literal (`"\\d+"`); C# verbatim strings `@"\d+"` and Java text blocks help.
- .NET: set `RegexOptions.NonBacktracking` (.NET 7+) or a match timeout for untrusted input.
- SQL: PostgreSQL `~` uses Tcl-style advanced regexes (lookahead, lookbehind since 9.6, no named groups); MySQL 8 uses ICU. Test in the database itself.
- Shell: `grep -E` has no `\d` on many systems; use `[0-9]`. `grep -P` gives PCRE where available.

## Safe defaults across flavors

- `[0-9]` instead of `\d`, `[A-Za-z]` instead of `\w` when the format is ASCII by definition.
- Anchors `^` and `$` without multiline mode, plus a separate length check (for example at most 254 chars for an email).
- No nested quantifiers on untrusted input, whatever the engine.
FILE:templates/regex-review.md
# Regex review: ${purpose}

**Pattern (original):** `${original_pattern}`
**Engine:** ${language_and_engine} | **Used as:** ${usage: validate whole value / find first / find all / replace}
**Checker:** HIGH ${high} / WARN ${warn} / INFO ${info} before; expectations ${pass_before}/${total} pass

## What it does, in plain words

1. (one line per token or group, top to bottom)

## Test results

| # | Input | Expected | Before | After | Notes |
|---|---|---|---|---|---|
| 1 | | match / no match | | | |

## Risks found

| Level | Rule | What it means here | Fix |
|---|---|---|---|
| HIGH | | | |

Probe (if run): input `${probe_input}`, ${probe_result}

## Corrected pattern

```${language}
(the corrected pattern, escaped for the user's language, with a VERBOSE or commented version if it is longer than about 40 chars)
```

Why each change: (one line per change)

## Keep these tests

```
+ (must match)
- (must not match)
```

## Notes

- Limits of a regex for this job and the extra check to do in code (length limit, parsing, verification step).
- Flavor caveats if the pattern will run outside Python.
FILE:examples/example-email-validation-review.md
# Example: review of a signup form email check that hangs

## User request

"Our signup API sometimes times out and the profiler points at this line. Can you explain the regex and fix it? It runs in Python 3.12 on the email field of a public form."

```python
EMAIL = re.compile(r"^([a-zA-Z0-9]+\.?)+@[A-z]+.com$")
```

## Cases used (cases.txt)

The user's 5 cases plus 3 edge cases added during the review (a dash instead of the dot, an underscore in the domain, and 30 letters with no `@`, the shape a bot sent):

```
+ ann@example.com
+ first.last@mail.example.co
- bob@@example.com
- no-at-sign.example.com
- trailing-dot@example.
- ann@example-com
- ann@exa_mple.com
- aaaaaaaaaaaaaaaaaaaaaaaaaaaaaa!
```

## Checker run (original)

```
$ python3 scripts/regex_check.py '^([a-zA-Z0-9]+\.?)+@[A-z]+.com$' --cases cases.txt --no-explain --probe
Samples (search):
  PASS     + 'ann@example.com'  -> [0:15] 'ann@example.com' 1='ann'
  FAIL     + 'first.last@mail.example.co'  -> expected a match
  PASS     - 'bob@@example.com'
  PASS     - 'no-at-sign.example.com'
  PASS     - 'trailing-dot@example.'
  FAIL     - 'ann@example-com'  -> [0:15] 'ann@example-com' 1='ann'  (expected no match)
  FAIL     - 'ann@exa_mple.com'  -> [0:16] 'ann@exa_mple.com' 1='ann'  (expected no match)
  TIMEOUT  - 'aaaaaaaaaaaaaaaaaaaaaaaaaaaaaa!'  -> still running after 2 s (catastrophic backtracking)

Risks and suggestions:
  HIGH nested-quantifier       a quantifier sits inside another quantifier and nothing else in the outer repeat is required ...
       probe 'a' * n + '!': n=12: 0.0002 s, n=16: 0.0031 s, n=20: 0.0593 s, n=24: 0.8419 s, n=28: timeout -> TIMEOUT at length 29: exponential backtracking confirmed
  WARN range-A-z               [A-z] also matches [ \ ] ^ _ and the backtick; use [A-Za-z]
  WARN unescaped-dot           '.' before 'com' matches any char (so a '-' or 'x' passes there too); write \. for a literal dot
  INFO dollar-newline          $ also matches before a trailing newline ...

Summary: HIGH 1  WARN 2  INFO 1 | expectations 4/8 pass | 1 sample(s) timed out
```

## Review (filled template, shortened)

**Checker:** HIGH 1 / WARN 2 / INFO 1 before; expectations 4/8 pass (one timeout). After: HIGH 0 / WARN 0, expectations 8/8 pass.

What it does: start of string; one or more runs of letters or digits, each optionally followed by a dot; an `@`; one or more chars from A to z (including `[ \ ] ^ _` and the backtick); any single char; the text `com`; end of string or before a final newline.

| # | Input | Expected | Before | After | Notes |
|---|---|---|---|---|---|
| 1 | ann@example.com | match | match | match | |
| 2 | first.last@mail.example.co | match | no match | match | only `.com` and one domain label were allowed |
| 3 | bob@@example.com | no match | no match | no match | |
| 4 | no-at-sign.example.com | no match | no match | no match | |
| 5 | trailing-dot@example. | no match | no match | no match | |
| 6 | ann@example-com | no match | match | no match | added: the unescaped dot accepted `-` |
| 7 | ann@exa_mple.com | no match | match | no match | added: `[A-z]` accepted `_` |
| 8 | 30 x "a" then "!" | no match, fast | timeout after 2 s | no match, instant | added: the production hang |

Risks: the `([a-zA-Z0-9]+\.?)+` group lets a run of letters be split in 2^n ways when the `@` is missing, so a 29-char junk input from a bot blocked a worker for more than 2 seconds; this is a ReDoS bug on a public endpoint.

Corrected pattern:

```python
EMAIL = re.compile(r"""
    [A-Za-z0-9._%+-]+          # local part: one run, dots allowed but not repeated in a nested group
    @
    (?:[A-Za-z0-9-]+\.)+       # one or more domain labels, each ending in a literal dot
    [A-Za-z]{2,}               # top-level domain
""", re.VERBOSE | re.ASCII)

def looks_like_email(value: str) -> bool:
    return len(value) <= 254 and EMAIL.fullmatch(value) is not None
```

Why each change: one flat repeat for the local part removes the nested quantifier; `\.` makes the dot literal; `[A-Za-z]` replaces `[A-z]`; any top-level domain is accepted instead of only `com`; `fullmatch` replaces `^...$`; a 254-char limit is a second defense; `re.ASCII` keeps `[...]` classes predictable.

Recheck: `python3 scripts/regex_check.py '[A-Za-z0-9._%+-]+@(?:[A-Za-z0-9-]+\.)+[A-Za-z]{2,}' --cases cases.txt --mode fullmatch --probe` reports no HIGH or WARN risks (the nested group is reported as INFO nested-separated, a safe shape) and 8/8 expectations pass.

Keep these tests: all 8 rows above, plus a 300-char input to check the length limit. Note that a regex only checks the shape; confirm the address by sending a verification email.
FILE:scripts/regex_check.py
#!/usr/bin/env python3
"""Explain a Python regular expression, test it on sample inputs, and warn about risky patterns.

Usage:
  python3 regex_check.py PATTERN [options]
  python3 regex_check.py PATTERN -t "text one" -t "text two"
  python3 regex_check.py PATTERN --cases cases.txt --mode fullmatch
  python3 regex_check.py PATTERN --cases - < cases.txt        (read cases from stdin)

Options:
  -t, --text TEXT     a sample input (repeatable)
  --cases FILE        one sample per line; "+ text" must match, "- text" must not match,
                      "# ..." is a comment, other lines are shown without an expectation
  --mode MODE         search (default), match, fullmatch, or findall
  --flags LETTERS     any of i m s x a (IGNORECASE, MULTILINE, DOTALL, VERBOSE, ASCII)
  --probe             time the pattern on generated worst-case inputs (subprocess, 2 s limit)
  --no-explain        skip the token-by-token explanation
  --json              print the result as JSON
  --fail-on LEVEL     exit 1 if any risk is at least LEVEL: high (default), warn, info, never
  --timeout S         stop matching one sample after S seconds and report TIMEOUT (default 2)

Risks: HIGH = nested quantifiers that can backtrack exponentially;
WARN = overlapping alternatives or adjacent quantifiers, [A-z], unescaped dots, empty matches;
INFO = anchoring, $ versus \\Z, Unicode \\d and \\w, greedy .*, flags that look missing.
Exit codes: 0 ok, 1 failed expectations or a risk at or above --fail-on, 2 usage or pattern error.
Python 3 standard library only (uses the re module's own parser).
"""
import json
import re
import signal
import string
import subprocess
import sys

try:  # Python 3.11+
    import re._parser as sre_parse
except ImportError:  # pragma: no cover - older Python
    import sre_parse  # type: ignore

LEVELS = {"info": 0, "warn": 1, "high": 2}
FLAG_LETTERS = {"i": re.IGNORECASE, "m": re.MULTILINE, "s": re.DOTALL, "x": re.VERBOSE, "a": re.ASCII}
FLAG_NAMES = [(re.IGNORECASE, "IGNORECASE"), (re.MULTILINE, "MULTILINE"), (re.DOTALL, "DOTALL"),
              (re.VERBOSE, "VERBOSE"), (re.ASCII, "ASCII")]
ALPHABET = string.ascii_letters + string.digits + string.punctuation + " \t\n" + "\u00e9\u0663"
CATEGORY_RE = {"CATEGORY_DIGIT": r"\d", "CATEGORY_NOT_DIGIT": r"\D", "CATEGORY_SPACE": r"\s",
               "CATEGORY_NOT_SPACE": r"\S", "CATEGORY_WORD": r"\w", "CATEGORY_NOT_WORD": r"\W"}
CATEGORY_TEXT = {"CATEGORY_DIGIT": "a di

Collection

f/awesome-chatgpt-prompts · prompts.csv

Author

f

Licence

CC0-1.0 · Open source · permissive

Last checked by us

Oct 17, 2026