Custom Response Body Rule Not Blocking for Responses Larger Than 1KB

[Dr-Lazarus-V2]

Describe the bug

The custom rule below is triggered correctly when the response body contains the word "jolly." However, if the response size exceeds 1KB, the rule no longer blocks the request, even though the rule is triggered.

Custom Rule:

SecRule RESPONSE_BODY "@rx (?i)(\n|\''|\:|\W*)jolly" \
    "id:102, \
    phase:4, \
    deny, \
    log, \
    t:none, \
    msg:'UNAUTHORIZED DATA ACCESS'"

This issue occurs despite the configurations set for response body access limits and processing, indicating that the rule might not be evaluated correctly for larger response sizes.

---

Logs and dumps

1. Debug Logs (Level 9):
   Modsecurity-Response-Debug.txt

2. Audit Logs:
   modsec_audit_waf.theviscousweb.com.txt

3. Error Logs:
   [Attach relevant error logs]

4. Core Dump (if applicable):
   [Attach core dump if there was a crash]

---

To Reproduce

1. Curl Request:

   curl -X GET "https://your-server-endpoint" -H "Content-Type: application/json"

2. Ensure that the response body contains:

   ...some data...\n jolly
   

3. Ensure the response body size is greater than 1KB.

4. Verify that the request goes through when it should be denied based on the rule.

---

Expected behavior

The custom rule should deny the request and log the corresponding message whenever the word "jolly" appears in the response body, regardless of the response size.

---

Server (please complete the following information):

• ModSecurity version (and connector):
  ModSecurity v3.0.7 with nginx-connector v1.0.1

• WebServer:
  nginx-1.24.0

• OS (and distro):
  Linux, Ubuntu 24

---

Rule Set (please complete the following information):

• Running any public or commercial rule set?
  OWASP CRS

• Version number:
  4.7.0-nginx-202410030910 (OWASP CRS Docker image)

---

[airween]

I'm not sure if the old version is causing the problem (v3.0.7 is really outdated), but I tested this with v3.0.13 and I couldn't reproduce your issue.

I tried with this request:

curl -v -H "Content-Type: application/json" -X POST -d '{"a": "11111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111", "b": "jolly"}' http://localhost/post

I tried it with CRS regression test backed - see this documantation so I got this response:

* upload completely sent off: 2071 bytes
< HTTP/1.1 200 OK
< Server: nginx/1.26.0
< Date: Fri, 18 Oct 2024 14:21:34 GMT
< Content-Type: application/json
< Content-Length: 4482
< Connection: keep-alive
< Vary: Accept-Encoding
< Access-Control-Allow-Origin: *
< Access-Control-Allow-Credentials: true
< 
{
  "args": {},
  "data": "{\"a\": \"1111111...111\", \"b\": \"jolly\"}",
  "files": {},
...

As you can see the response body length is 4482 bytes and it contains the world jolly. And as I expected your rule was triggered:

ModSecurity: Access denied with code 403 (phase 4). Matched "Operator `Rx' with parameter `(?i)(\n|\''|\:|\W*)jolly' against variable `RESPONSE_BODY' (Value: `{\x0a  "args": {},\x0a  "data": "{\"a\": \"111111111111111111111111111111111111111111111111111111111 (4442 characters omitted)' ) [file "/home/airween/src/test.conf"] [line "461"] [id "102"] ...

Or may be there is some misconfiguration in your settings.

[Dr-Lazarus-V2]

Hey @airween, thank you for your quick response.

I tested it out with modsecurity V3.0.13. I noticed that the following:

Case 1: Response Size > 1.5KB
Behavior: The custom rule is triggered, but the request is not blocked. I can still see the valid response (200) on my frontend. I can see the following log, however the request still seems to go through and not get blocked by a 403, even though the log says otherwise.

ModSecurity: Access denied with code 403 (phase 4). Matched "Operator `Rx' with parameter `(?i)(\n|\''|\:|\W*)jolly' against variable `RESPONSE_BODY' (Value: `{\x0a  "args": {},\x0a  "data": "{\"a\": \"111111111111111111111111111111111111111111111111111111111 (4442 characters omitted)' )....

However the request still reaches the FE with a 200 response.

Case 2: Response Size < 1KB
Behavior: The custom rule triggers, and the request is blocked with a 403 Forbidden error. I can see that the request is blocked and it works as intended.

[cadeath]

I checked your logs and you are using LLM on this, right?
I have the same problem but I haven't checked the Request Size.

You are telling us that the rule will deny if the Request Size is < 1KB. I will get back on my project again.
Thanks

[Dr-Lazarus-V2]

Hey @cadeath, I meant response size, so the if the size of the output from my BE server is greater than 1KB, it seems to log the rule but not block/drop the request.

[Dr-Lazarus-V2]

It seems that the headers are already sent to the client, not sure if this is relevant:

2024/10/22 05:43:34 [error] 90#90: *320 [client 3.237.**.**] ModSecurity: Access denied with code 403 (phase 4). Matched "Operator `StrEq' with parameter `1' against variable `TX:**' (Value: `1' ) [file "/var/opt/modsecurity.d/owasp-crs/**/**/**.conf"] [line "6"] [id "20000002"] [rev ""] [msg "** ** detected in the response."] [data ""] [severity "0"] [ver ""] [maturity "0"] [accuracy "0"] [hostname "172.27.0.8"] [uri "/api/v1/**/**"] [unique_id "8c46729a8957f1fef93552689ee6d191"] [ref ""] while sending to client, client: 3.237.**.**, server: waf.****.com, request: "POST /api/v1/**/** HTTP/1.1", upstream: "http://****/api/v1/**/**", host: "waf.***.com"

2024/10/22 05:43:34 [alert] 90#90: *320 header already sent while sending to client, client: 3.237.**.**, server: waf.***.com, request: "POST /api/v1/**/** HTTP/1.1", upstream: "http://***/api/v1/**/**", host: "waf.***.com"

Modsecurity Configuration

SecRuleEngine On

SecRequestBodyAccess on
SecRule REQUEST_HEADERS:Content-Type "^(?:application(?:/soap\+|/)|text/)xml" \
     "id:'200000',phase:1,t:none,t:lowercase,pass,nolog,ctl:requestBodyProcessor=XML"
SecRule REQUEST_HEADERS:Content-Type "^application/json" \
     "id:'200001',phase:1,t:none,t:lowercase,pass,nolog,ctl:requestBodyProcessor=JSON"
SecRequestBodyLimit 10485760
SecRequestBodyNoFilesLimit 1048576
SecRequestBodyLimitAction Reject
SecRequestBodyJsonDepthLimit 512
SecArgumentsLimit 1000

SecRule &ARGS "@ge 1000" \
"id:'200007', phase:2,t:none,log,deny,status:400,msg:'Failed to fully parse request body due to large argument count',severity:2"

SecRule REQBODY_ERROR "!@eq 0" \
"id:'200002', phase:2,t:none,log,deny,status:400,msg:'Failed to parse request body.',logdata:'%{reqbody_error_msg}',severity:2"

SecRule MULTIPART_STRICT_ERROR "!@eq 0" \
"id:'200003',phase:2,t:none,log,deny,status:400, \
msg:'Multipart request body failed strict validation: \
PE %{REQBODY_PROCESSOR_ERROR}, \
BQ %{MULTIPART_BOUNDARY_QUOTED}, \
BW %{MULTIPART_BOUNDARY_WHITESPACE}, \
DB %{MULTIPART_DATA_BEFORE}, \
DA %{MULTIPART_DATA_AFTER}, \
HF %{MULTIPART_HEADER_FOLDING}, \
LF %{MULTIPART_LF_LINE}, \
SM %{MULTIPART_MISSING_SEMICOLON}, \
IQ %{MULTIPART_INVALID_QUOTING}, \
IP %{MULTIPART_INVALID_PART}, \
IH %{MULTIPART_INVALID_HEADER_FOLDING}, \
FL %{MULTIPART_FILE_LIMIT_EXCEEDED}'"

SecPcreMatchLimit 100000
SecPcreMatchLimitRecursion 100000
SecRule TX:/^MSC_/ "!@streq 0" \
        "id:'200005',phase:2,t:none,deny,msg:'ModSecurity internal error flagged: %{MATCHED_VAR_NAME}'"

SecResponseBodyAccess on
SecResponseBodyMimeType text/plain text/html text/xml application/json
SecResponseBodyLimit 537600
SecResponseBodyLimitAction Reject

SecTmpDir /tmp/
SecDataDir /tmp/

SecAuditEngine On
SecAuditLogType Serial
SecAuditLogRelevantStatus "^(?:5|4(?!04))"
SecAuditLog /var/log/modsec_logs/modsec_audit_waf.***.com.log
SecAuditLogFormat JSON
SecAuditLogParts ABFHZ

SecArgumentSeparator &
SecCookieFormat 0
SecUnicodeMapFile unicode.mapping 20127
SecStatusEngine Off

SecGeoLookupDb /var/GeoLite2-Country/GeoLite2-Country.mmdb

SecDebugLog /var/log/debug/debug.log
SecDebugLogLevel 9

[Dr-Lazarus-V2]

I believe this issue has got to do with the Modsecurity-Nginx connector. A PR has been raised to handle this as well: owasp-modsecurity/ModSecurity-nginx#326