Windows HPC fails to start after update (Server 2019)

[hob63]

Hi all,

When looking at the logs after the HPC console add-in fails to start I get the following (HPC scheduler):


Exception: System.Data.SqlClient.SqlException (0x80131904): The datediff function resulted in an overflow. The number of dateparts separating two date/time instances is too large. Try to use datediff with a less precise datepart.
at System.Data.SqlClient.SqlConnection.OnError(SqlException exception, Boolean breakConnection, Action`1 wrapCloseInAction)
at System.Data.SqlClient.TdsParser.ThrowExceptionAndWarning(TdsParserStateObject stateObj, Boolean callerHasConnectionLock, Boolean asyncClose)
at System.Data.SqlClient.TdsParser.TryRun(RunBehavior runBehavior, SqlCommand cmdHandler, SqlDataReader dataStream, BulkCopySimpleResultSet bulkCopyHandler, TdsParserStateObject stateObj, Boolean& dataReady)
at System.Data.SqlClient.SqlDataReader.TryConsumeMetaData()
at System.Data.SqlClient.SqlDataReader.get_MetaData()
at System.Data.SqlClient.SqlCommand.FinishExecuteReader(SqlDataReader ds, RunBehavior runBehavior, String resetOptionsString, Boolean isInternal, Boolean forDescribeParameterEncryption, Boolean shouldCacheForAlwaysEncrypted)
at System.Data.SqlClient.SqlCommand.CompleteAsyncExecuteReader(Boolean isInternal, Boolean forDescribeParameterEncryption)
at System.Data.SqlClient.SqlCommand.InternalEndExecuteReader(IAsyncResult asyncResult, String endMethod, Boolean isInternal)
at System.Data.SqlClient.SqlCommand.EndExecuteReaderInternal(IAsyncResult asyncResult)
at System.Data.SqlClient.SqlCommand.EndExecuteReaderAsync(IAsyncResult asyncResult)
at System.Threading.Tasks.TaskFactory`1.FromAsyncCoreLogic(IAsyncResult iar, Func`2 endFunction, Action`1 endAction, Task`1 promise, Boolean requiresSynchronization)
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Microsoft.Hpc.HighAvailabilityModule.Client.SQL.SQLMembershipClient.<GetHeartBeatEntryAsync>d__28.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Microsoft.Hpc.SqlClientContext.<>c__DisplayClass13_0.<<ResolveSingletonServicePrimaryAsync>b__0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Microsoft.Hpc.RetryManager.<InvokeWithRetryAsync>d__33`1.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at Microsoft.Hpc.RetryManager.<InvokeWithRetryAsync>d__33`1.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Microsoft.Hpc.SqlClientContext.<InvokeWithRetryAsync>d__22`1.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Microsoft.Hpc.SqlClientContext.<ResolveSingletonServicePrimaryAsync>d__13.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Microsoft.Hpc.ServiceResolverExtension.<ResolveMonitoringNodeAsync>d__20.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Microsoft.Hpc.Monitoring.CounterDataSender.OpenConnection()
at Microsoft.Hpc.Monitoring.MonitoringClientController.InitializeService()
at Microsoft.Hpc.Monitoring.MonitoringClientController.InitializeServiceCallBack(Object sender, ElapsedEventArgs e)
ClientConnectionId:0a159359-04ec-484d-bb4f-22cecebec65d
Error Number:535,State:0,Class:16


Everything was working fine and this happened after a few windows updates.

 

[ChatGPT]

The error message you are encountering suggests that there is an issue with a datediff function in SQL which is part of the HPC (High-Performance Computing) Scheduler. This error could be related to a date calculation that results in an overflow, indicating that the difference between two date/time values is too large. Here are some steps you can take to address this issue: 1. Check for Updates: - Make sure all software components, including the HPC Scheduler and related tools, are up to date. Check for any available updates or patches that may address this issue. 2. Database Maintenance: - Consider running maintenance tasks on the database used by the HPC Scheduler. This could involve rebuilding indexes, updating statistics, or performing a general database health check. 3. Review Date Calculations: - If you have control over the SQL queries or stored procedures involved in this process, review the datediff calculations to ensure they are functioning as expected and not generating overly large values. 4. Database Backup: - Before making any significant changes or updates, ensure you have a recent and valid backup of the database and any critical system configurations. 5. Audit Windows Updates: - If the issue started after Windows updates, review the recent updates that were applied to the system. It's possible that a specific update may have affected the behavior of the HPC Scheduler. 6. Reach Out to Support: - If the issue persists and you are unable to resolve it using the above steps, consider reaching out to HPC Scheduler support or the vendor for further assistance. They may have encountered similar issues and could provide specific guidance. By following these steps and possibly working with the support team for the HPC Scheduler, you can troubleshoot and hopefully resolve the issue with the datediff function overflow in your HPC environment.