Fix 408 timeout caused by ignoring DATA frames after GOAWAY

[yknoya]

Issue

When multiple POST requests are sent over a single HTTP/2 connection, 408/504 (timeout) responses may occur if the HTTP/2 stream error rate exceeds the configured threshold.
When the stream error rate exceeds the threshold, ATS sends a GOAWAY frame in the following code path. However, a timeout occurs between sending the GOAWAY frame and closing the connection with the client.

trafficserver/src/proxy/http2/Http2CommonSession.cc

Lines 371 to 393 in 00604e1

	Http2ErrorCode err = Http2ErrorCode::HTTP2_ERROR_NO_ERROR;
	if (this->connection_state.get_stream_error_rate() > std::min(1.0, Http2::stream_error_rate_threshold * 2.0)) {
	ip_port_text_buffer ipb;
	const char *peer_ip = ats_ip_ntop(this->get_proxy_session()->get_remote_addr(), ipb, sizeof(ipb));
	SiteThrottledWarning("HTTP/2 session error peer_ip=%s session_id=%" PRId64
	" closing a connection, because its stream error rate (%f) exceeded the threshold (%f)",
	peer_ip, this->get_connection_id(), this->connection_state.get_stream_error_rate(),
	Http2::stream_error_rate_threshold);
	err = Http2ErrorCode::HTTP2_ERROR_ENHANCE_YOUR_CALM;
	}
	// Return if there was an error
	if (err > Http2ErrorCode::HTTP2_ERROR_NO_ERROR || do_start_frame_read(err) < 0) {
	// send an error if specified. Otherwise, just go away
	this->connection_state.restart_receiving(nullptr);
	if (err > Http2ErrorCode::HTTP2_ERROR_NO_ERROR) {
	if (!this->connection_state.is_state_closed()) {
	this->connection_state.send_goaway_frame(this->connection_state.get_latest_stream_id_in(), err);
	this->set_half_close_local_flag(true);
	}
	}
	return 0;
	}

Root Cause

The issue is caused by ATS canceling HTTP/2 frame reception processing after sending the GOAWAY frame.

trafficserver/src/proxy/http2/Http2CommonSession.cc

Lines 363 to 369 in 00604e1

	while (this->_read_buffer_reader->read_avail() >= static_cast<int64_t>(HTTP2_FRAME_HEADER_LEN)) {
	// Cancel reading if there was an error or connection is closed
	if (connection_state.tx_error_code.code != static_cast<uint32_t>(Http2ErrorCode::HTTP2_ERROR_NO_ERROR) ||
	connection_state.is_state_closed()) {
	Http2SsnDebug("reading a frame has been canceled (%u)", connection_state.tx_error_code.code);
	break;
	}

Specifically, while handling multiple POST requests, the timeout occurs in the following sequence:

1. The client sends a HEADERS frame.
2. ATS receives the HEADERS frame.
3. ATS forwards the request headers to the origin.
4. The client sends a DATA frame.
5. The stream error rate exceeds the threshold, and ATS sends a GOAWAY frame to the client.
6. ATS receives the DATA frame, but cancels processing because the GOAWAY frame has already been sent.
7. ATS creates an HttpTunnel to forward the request body to the origin.
8. ATS waits indefinitely for the request body to be sent to the origin, resulting in a timeout.

I reviewed RFC 9113 regarding the handling of frames received after sending a GOAWAY frame. The following description indicates that canceling the reception of all frames violates the RFC:

After sending a GOAWAY frame, the sender can discard frames for streams initiated by the receiver with identifiers higher than the identified last stream. However, any frames that alter connection state cannot be completely ignored. For instance, HEADERS, PUSH_PROMISE, and CONTINUATION frames MUST be minimally processed to ensure that the state maintained for field section compression is consistent (see Section 4.3); similarly, DATA frames MUST be counted toward the connection flow-control window.

https://datatracker.ietf.org/doc/html/rfc9113#name-goaway

Fix

ATS has been modified to continue receiving frames even after the HTTP/2 stream error rate exceeds the threshold and a GOAWAY frame is sent.
However, if the stream ID of a received frame is greater than the Last-Stream-ID specified at the time the GOAWAY frame was sent, frame processing is limited to HEADERS, PUSH_PROMISE, CONTINUATION, and DATA frames.
In such cases, ATS performs only the minimal processing required by RFC 9113 and sends an RST_STREAM frame.