Skip to content

Commit 92b3133

Browse files
gh-109638: Fix exponential time in csv.Sniffer for doubled quotes
The regular expression which looks for a doubled quote character let its runs match the quote character too, so every quote could be taken either as a part of a run or as one of the matched quotes. Exclude the quote character from the runs and match them possessively. The runs no longer exclude the delimiter and the line break either, so a doubled quote is now also found in a field which contains them.
1 parent 1088266 commit 92b3133

3 files changed

Lines changed: 16 additions & 4 deletions

File tree

Lib/csv.py

Lines changed: 6 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -331,10 +331,12 @@ def _guess_quote_and_delimiter(self, data, delimiters):
331331
# if we see an extra quote between delimiters, we've got a
332332
# double quoted format
333333
dq_regexp = re.compile(
334-
r"((%(delim)s)|^)\W*%(quote)s[^%(delim)s\n]*%(quote)s[^%(delim)s\n]*%(quote)s\W*((%(delim)s)|$)" % \
335-
{'delim':re.escape(delim), 'quote':quotechar}, re.MULTILINE)
336-
337-
334+
r"(?:%(delim)s|^) *+%(quote)s" # ,"
335+
r"[^%(quote)s]*+%(quote)s%(quote)s" # the doubled quote
336+
r"(?:%(quote)s%(quote)s|[^%(quote)s]++)*+" # the rest of the field
337+
r"%(quote)s(?:%(delim)s|$)" # ",
338+
% {'delim': re.escape(delim), 'quote': quotechar},
339+
re.MULTILINE)
338340

339341
if dq_regexp.search(data):
340342
doublequote = True

Lib/test/test_csv.py

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1564,6 +1564,13 @@ def test_zero_mode_tie_order_colon_first(self):
15641564
sniffer.sniff(sample)
15651565

15661566

1567+
def test_sniff_regex_backtracking(self):
1568+
# gh-109638: this artificial sample used to take minutes.
1569+
sniffer = csv.Sniffer()
1570+
sample = '"",' * 100 + '"' * 100 + '0' + '"' * 100 + '0'
1571+
self.assertEqual(sniffer.sniff(sample).delimiter, ',')
1572+
1573+
15671574
class NUL:
15681575
def write(s, *args):
15691576
pass
Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,3 @@
1+
Fix exponential time in :meth:`csv.Sniffer.sniff` for a sample which contains
2+
many quote characters. A doubled quote character is now also detected in
3+
a field which contains the delimiter or a line break.

0 commit comments

Comments
 (0)