Login failed for user '<token-identified principal>'.

[mircotoepfer]

We are running Hangfire with Azure SQL Database in General Purpose – Serverless configuration.
Under normal operation everything works fine — however:

During scaling down of Azure SQL compute

Hangfire starts producing a massive number of SQL exceptions

Hangfire reports jobs as successfully executed, but their actual code does not run

→ This leads to silently skipped job execution logic, which is a critical reliability issue.

Errors appear in Application Insights, with no immediate application crash.
The behavior suggests Hangfire gets into an inconsistent state while SQL resources are throttled or transitioning.

We suspect Hangfire does not reliably handle transient outage/throttling during scale-down.

---

Hangfire:

1.8.18 SQL Server (custom) 1.8.18 .NET 9.0

Azure SQL Config:

Service Tier: General Purpose
Computer tier: Serverless
Hardware: Standard-series (Gen5)
MaxVCores: 6
MinVCores: 3

Stack Trace:

[
{
"method": "Microsoft.Data.SqlClient.TdsParser.ThrowExceptionAndWarning",
"level": 0,
"line": 0,
"assembly": "Microsoft.Data.SqlClient, Version=6.0.0.0, Culture=neutral, PublicKeyToken=23ec7fc2d6eaa4a5"
},
{
"method": "Microsoft.Data.SqlClient.TdsParser.TryRun",
"level": 1,
"line": 0,
"assembly": "Microsoft.Data.SqlClient, Version=6.0.0.0, Culture=neutral, PublicKeyToken=23ec7fc2d6eaa4a5"
},
{
"method": "Microsoft.Data.SqlClient.TdsParser.Run",
"level": 2,
"line": 0,
"assembly": "Microsoft.Data.SqlClient, Version=6.0.0.0, Culture=neutral, PublicKeyToken=23ec7fc2d6eaa4a5"
},
{
"method": "Microsoft.Data.SqlClient.SqlInternalConnectionTds.CompleteLogin",
"level": 3,
"line": 0,
"assembly": "Microsoft.Data.SqlClient, Version=6.0.0.0, Culture=neutral, PublicKeyToken=23ec7fc2d6eaa4a5"
},
{
"method": "Microsoft.Data.SqlClient.SqlInternalConnectionTds.LoginNoFailover",
"level": 4,
"line": 0,
"assembly": "Microsoft.Data.SqlClient, Version=6.0.0.0, Culture=neutral, PublicKeyToken=23ec7fc2d6eaa4a5"
},
{
"method": "Microsoft.Data.SqlClient.SqlInternalConnectionTds.OpenLoginEnlist",
"level": 5,
"line": 0,
"assembly": "Microsoft.Data.SqlClient, Version=6.0.0.0, Culture=neutral, PublicKeyToken=23ec7fc2d6eaa4a5"
},
{
"method": "Microsoft.Data.SqlClient.SqlInternalConnectionTds..ctor",
"level": 6,
"line": 0,
"assembly": "Microsoft.Data.SqlClient, Version=6.0.0.0, Culture=neutral, PublicKeyToken=23ec7fc2d6eaa4a5"
},
{
"method": "Microsoft.Data.SqlClient.SqlConnectionFactory.CreateConnection",
"level": 7,
"line": 0,
"assembly": "Microsoft.Data.SqlClient, Version=6.0.0.0, Culture=neutral, PublicKeyToken=23ec7fc2d6eaa4a5"
},
{
"method": "Microsoft.Data.ProviderBase.DbConnectionPool.CreateObject",
"level": 8,
"line": 0,
"assembly": "Microsoft.Data.SqlClient, Version=6.0.0.0, Culture=neutral, PublicKeyToken=23ec7fc2d6eaa4a5"
},
{
"method": "Microsoft.Data.ProviderBase.DbConnectionPool.UserCreateRequest",
"level": 9,
"line": 0,
"assembly": "Microsoft.Data.SqlClient, Version=6.0.0.0, Culture=neutral, PublicKeyToken=23ec7fc2d6eaa4a5"
},
{
"method": "Microsoft.Data.ProviderBase.DbConnectionPool.TryGetConnection",
"level": 10,
"line": 0,
"assembly": "Microsoft.Data.SqlClient, Version=6.0.0.0, Culture=neutral, PublicKeyToken=23ec7fc2d6eaa4a5"
},
{
"method": "Microsoft.Data.ProviderBase.DbConnectionFactory.TryGetConnection",
"level": 11,
"line": 0,
"assembly": "Microsoft.Data.SqlClient, Version=6.0.0.0, Culture=neutral, PublicKeyToken=23ec7fc2d6eaa4a5"
},
{
"method": "Microsoft.Data.ProviderBase.DbConnectionInternal.TryOpenConnectionInternal",
"level": 12,
"line": 0,
"assembly": "Microsoft.Data.SqlClient, Version=6.0.0.0, Culture=neutral, PublicKeyToken=23ec7fc2d6eaa4a5"
},
{
"method": "Microsoft.Data.SqlClient.SqlConnection.TryOpen",
"level": 13,
"line": 0,
"assembly": "Microsoft.Data.SqlClient, Version=6.0.0.0, Culture=neutral, PublicKeyToken=23ec7fc2d6eaa4a5"
},
{
"method": "Microsoft.Data.SqlClient.SqlConnection.Open",
"level": 14,
"line": 0,
"assembly": "Microsoft.Data.SqlClient, Version=6.0.0.0, Culture=neutral, PublicKeyToken=23ec7fc2d6eaa4a5"
},
{
"method": "Hangfire.SqlServer.SqlServerStorage.CreateAndOpenConnection",
"level": 15,
"line": 393,
"assembly": "Hangfire.SqlServer, Version=1.8.18.0, Culture=neutral, PublicKeyToken=null",
"fileName": "C:\projects\hangfire-525\src\Hangfire.SqlServer\SqlServerStorage.cs"
},
{
"method": "Hangfire.SqlServer.SqlServerConnection.AcquireLock",
"level": 16,
"line": 800,
"assembly": "Hangfire.SqlServer, Version=1.8.18.0, Culture=neutral, PublicKeyToken=null",
"fileName": "C:\projects\hangfire-525\src\Hangfire.SqlServer\SqlServerConnection.cs"
},
{
"method": "Hangfire.States.BackgroundJobStateChanger.ChangeState",
"level": 17,
"line": 61,
"assembly": "Hangfire.Core, Version=1.8.18.0, Culture=neutral, PublicKeyToken=null",
"fileName": "C:\projects\hangfire-525\src\Hangfire.Core\States\BackgroundJobStateChanger.cs"
},
{
"method": "Hangfire.Server.Worker.TryChangeState",
"level": 18,
"line": 235,
"assembly": "Hangfire.Core, Version=1.8.18.0, Culture=neutral, PublicKeyToken=null",
"fileName": "C:\projects\hangfire-525\src\Hangfire.Core\Server\Worker.cs"
}
]

[odinserj]

Could you answer the following questions, please:

Hangfire starts producing a massive number of SQL exceptions

How many of them per second or minute?

This leads to silently skipped job execution logic, which is a critical reliability issue.

Could you provide any evidence of this behavior? Skipped job execution logic is indeed a critical issue, however, Hangfire can not mark a background job as succeeded in case of an exception.

[mircotoepfer]

Well, it was a strange and unclear situation. In the dashboard, I saw that the jobs were being processed as usual, but nothing was happening. What I also saw was that the entire display had doubled in the Server view, so everything was there twice. Under Failed Jobs, I could see failed jobs that were several hours old, so I restarted them using Requeue, and then the node resolved itself. As I said, this happens when downscaling from 24 VCores to 6VCores and may be a rare exception. What interests me more is the general behavior during dynamic scaling of the DB. In the logging, I find the message

Microsoft.Data.SqlClient.SqlException at Hangfire.SqlServer.SqlServerStorage.CreateAndOpenConnection

with the message Login failed for user ‘’. This probably has to do with the downscaling.

The exception occurs regularly in our system. Now that I had provoked the downscaling and, as a result, the exceptions occurred at a much higher than average rate at the same time, I think the two are related.

[odinserj]

Such exception indeed may happen, when a database is not ready to receive connections, please see a similar issue dotnet/SqlClient#3707. What we should understand is what to do in Hangfire itself:

1. Amount of logging depends on the configured level, TRACE and DEBUG levels will log any exception Hangfire gets, so can be noisy when there are problems. INFO and above levels should not be so noisy, logging only when Hangfire gives up after a bunch of retries, but it might be difficult to understand what's going on in some cases. What logging levels are configured in your application for Hangfire?
2. The TryChangeState method highlighted in your stack trace will try to perform 10 retry attempts with increasing intervals, where for each attempt number there would be a corresponding delay in seconds, so 5th failed attempt will cause 5 second delay before the next one. During these attempts, background job will be in the Processing state (and we can't change it, because database is inaccessible), but Hangfire will not give up before all the retries are made. After this, Hangfire will attempt to move background job to the Failed State without any retries. If this action fails, an exception will be thrown and everything will depend on the sliding invisibility timeout setting.