Skip to content

MDEV-39933 JSON_NORMALIZE returns wrong result for invalid JSON with … #5242

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
akshatnehra wants to merge 1 commit into
base: main
Choose a base branch
from
+78 −3
Open

MDEV-39933 JSON_NORMALIZE returns wrong result for invalid JSON with … #5242

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
43 changes: 43 additions & 0 deletions mysql-test/main/mdev_39933.result
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,43 @@
#
# MDEV-39933: Incorrect result of JSON_NORMALIZE on invalid json data
#
# JSON with trailing literal junk (no NUL) - should return NULL
SELECT JSON_NORMALIZE('{"a":1}0junk');

@gkodinov gkodinov Jun 16, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

is this really reproducing anything? I suppose not.
In my unpatched server:

MariaDB [(none)]> SELECT JSON_NORMALIZE('{"a":1}0junk');
+--------------------------------+
| JSON_NORMALIZE('{"a":1}0junk') |
+--------------------------------+
| NULL                           |
+--------------------------------+
1 row in set (0.002 sec)

JSON_NORMALIZE('{"a":1}0junk')
NULL
# JSON_VALID correctly rejects JSON with embedded NUL byte + trailing junk
SELECT JSON_VALID(CONVERT(CONCAT('{"a":1}', CHAR(0), 'junk') USING latin1));

@gkodinov gkodinov Jun 16, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this one too:

MariaDB [(none)]> SELECT JSON_VALID(CONVERT(CONCAT('{"a":1}', CHAR(0), 'junk') USING latin1));
+----------------------------------------------------------------------+
| JSON_VALID(CONVERT(CONCAT('{"a":1}', CHAR(0), 'junk') USING latin1)) |
+----------------------------------------------------------------------+
|                                                                    0 |
+----------------------------------------------------------------------+
1 row in set (0.003 sec)

JSON_VALID(CONVERT(CONCAT('{"a":1}', CHAR(0), 'junk') USING latin1))
0
# JSON_NORMALIZE should also return NULL for invalid JSON with embedded NUL
SELECT JSON_NORMALIZE(CONVERT(CONCAT('{"a":1}', CHAR(0), 'junk') USING latin1));
JSON_NORMALIZE(CONVERT(CONCAT('{"a":1}', CHAR(0), 'junk') USING latin1))
NULL
# Additional case: just a NUL after valid JSON
SELECT JSON_NORMALIZE(CONVERT(CONCAT('{"a":1}', CHAR(0)) USING latin1));
JSON_NORMALIZE(CONVERT(CONCAT('{"a":1}', CHAR(0)) USING latin1))
NULL
# Valid JSON through conversion should still work
SELECT JSON_NORMALIZE(CONVERT('{"a":1}' USING latin1));

@gkodinov gkodinov Jun 16, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

no need to re-test that IMHO

JSON_NORMALIZE(CONVERT('{"a":1}' USING latin1))
{"a":1.0E0}
# Multi-byte source charset (utf16)
SELECT JSON_NORMALIZE(CONVERT(CONCAT('{"a":1}', CHAR(0), 'x') USING utf16));

@gkodinov gkodinov Jun 16, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this is a bit too much and is not touching on any of the changed lines.

JSON_NORMALIZE(CONVERT(CONCAT('{"a":1}', CHAR(0), 'x') USING utf16))
NULL
# NUL embedded inside a JSON string value
SELECT JSON_NORMALIZE(CONVERT(CONCAT('{"a":"b', CHAR(0), 'c"}') USING latin1));

@gkodinov gkodinov Jun 16, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

also this. too much IMHO

JSON_NORMALIZE(CONVERT(CONCAT('{"a":"b', CHAR(0), 'c"}') USING latin1))
NULL
# JSON array with trailing NUL
SELECT JSON_NORMALIZE(CONVERT(CONCAT('[1,2,3]', CHAR(0), 'x') USING latin1));
JSON_NORMALIZE(CONVERT(CONCAT('[1,2,3]', CHAR(0), 'x') USING latin1))
NULL
# Nested object - valid through conversion
SELECT JSON_NORMALIZE(CONVERT('{"a":{"b":1}}' USING latin1));

@gkodinov gkodinov Jun 16, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

what is this testing?

JSON_NORMALIZE(CONVERT('{"a":{"b":1}}' USING latin1))
{"a":{"b":1.0E0}}
# Empty string
SELECT JSON_NORMALIZE(CONVERT('' USING latin1));

@gkodinov gkodinov Jun 16, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

what is this testing?

JSON_NORMALIZE(CONVERT('' USING latin1))
NULL
33 changes: 33 additions & 0 deletions mysql-test/main/mdev_39933.test
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,33 @@
--echo #
--echo # MDEV-39933: Incorrect result of JSON_NORMALIZE on invalid json data
--echo #

--echo # JSON with trailing literal junk (no NUL) - should return NULL
SELECT JSON_NORMALIZE('{"a":1}0junk');

--echo # JSON_VALID correctly rejects JSON with embedded NUL byte + trailing junk
SELECT JSON_VALID(CONVERT(CONCAT('{"a":1}', CHAR(0), 'junk') USING latin1));

--echo # JSON_NORMALIZE should also return NULL for invalid JSON with embedded NUL
SELECT JSON_NORMALIZE(CONVERT(CONCAT('{"a":1}', CHAR(0), 'junk') USING latin1));

--echo # Additional case: just a NUL after valid JSON
SELECT JSON_NORMALIZE(CONVERT(CONCAT('{"a":1}', CHAR(0)) USING latin1));

--echo # Valid JSON through conversion should still work
SELECT JSON_NORMALIZE(CONVERT('{"a":1}' USING latin1));

--echo # Multi-byte source charset (utf16)
SELECT JSON_NORMALIZE(CONVERT(CONCAT('{"a":1}', CHAR(0), 'x') USING utf16));

--echo # NUL embedded inside a JSON string value
SELECT JSON_NORMALIZE(CONVERT(CONCAT('{"a":"b', CHAR(0), 'c"}') USING latin1));

--echo # JSON array with trailing NUL
SELECT JSON_NORMALIZE(CONVERT(CONCAT('[1,2,3]', CHAR(0), 'x') USING latin1));

--echo # Nested object - valid through conversion
SELECT JSON_NORMALIZE(CONVERT('{"a":{"b":1}}' USING latin1));

--echo # Empty string
SELECT JSON_NORMALIZE(CONVERT('' USING latin1));

@gkodinov gkodinov Jun 16, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we typically end files with :

--echo End of 10.11 tests.

5 changes: 2 additions & 3 deletions strings/json_normalize.c
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1030,15 +1030,14 @@ json_normalize(DYNAMIC_STRING *result,
if (!s_utf8)
return 1;
memset(s_utf8, 0x00, in_size);

@gkodinov gkodinov Jun 16, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

while you're at it, can you please remove this memset and replace it with a s_utf8[in_size] = 0; after the my_convert call please?

my_convert(s_utf8, (uint32)in_size, &my_charset_utf8mb4_bin,
s, (uint32)size, cs, &convert_err);
in_size= my_convert(s_utf8, (uint32)in_size, &my_charset_utf8mb4_bin,
s, (uint32)size, cs, &convert_err);
if (convert_err)
{
my_free(s_utf8);
return 1;
}
in= s_utf8;
in_size= strlen(s_utf8);
}


Expand Down
Loading