From a2dabad9ae120e5b2c129248370d2b056c827acf Mon Sep 17 00:00:00 2001 From: Samir Solanki Date: Fri, 13 Mar 2026 19:07:10 -0700 Subject: [PATCH] Log ScheduledEnqueueTimeUtc for timer messages in SentMessageLog MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit When DurableTask creates a timer (CreateTimer orchestrator action), the ScheduledEnqueueTimeUtc is set on the Service Bus message but never logged to telemetry. This makes it difficult to investigate orchestration stalls caused by lost/delayed scheduled messages — investigators must infer the expected fire time from orchestration parameters. This change adds the ScheduledEnqueueTimeUtc to the existing SentMessageLog trace event for timer messages. Non-timer messages are unaffected (their ScheduledEnqueueTimeUtc is DateTime.MinValue and is omitted from the log). Before: 1 messages queued for Timer Message: abc123 <-1> After: 1 messages queued for Timer Message: abc123 <-1> ScheduledEnqueueTimeUtc:2026-03-12T03:04:00.0000000Z Context: Two Sev2 incidents (IcM 761467738, 758301760) involved Service Bus scheduled messages that were committed but never delivered, causing billing orchestrations to stall for 16-24 hours. Investigation required multiple rounds of Kusto queries to infer the expected fire time. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> --- src/DurableTask.ServiceBus/ServiceBusOrchestrationService.cs | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/DurableTask.ServiceBus/ServiceBusOrchestrationService.cs b/src/DurableTask.ServiceBus/ServiceBusOrchestrationService.cs index 3d49bb749..9afba0c39 100644 --- a/src/DurableTask.ServiceBus/ServiceBusOrchestrationService.cs +++ b/src/DurableTask.ServiceBus/ServiceBusOrchestrationService.cs @@ -1580,7 +1580,7 @@ void LogSentMessages(IMessageSession session, string messageType, IList $"{m.Message.MessageId} <{m.Action?.Event.EventId.ToString()}>"))}")); + string.Join(",", messages.Select(m => $"{m.Message.MessageId} <{m.Action?.Event.EventId.ToString()}>{(m.Message.ScheduledEnqueueTimeUtc > DateTime.MinValue ? $" ScheduledEnqueueTimeUtc:{m.Message.ScheduledEnqueueTimeUtc:O}" : "")}"))}")); } async Task GetSessionStateAsync(IMessageSession session, IOrchestrationServiceBlobStore orchestrationServiceBlobStore)