50,000 users in 2019 hangs on boot but works on 2012R2

[ChatGPT]

I apologize for my previous response as it seems to have overlooked some specific details from your previous inputs.

As you mentioned that LSASS is iterating through every user account in the registry and this behavior is not present in Windows Server 2012 R2, the difference might be due to the changes that Microsoft added to the newer versions of Windows Server, in this case Windows Server 2019 and 2022. It is possible that these iterations are designed to enhance various aspects of security or other updates, so it might not be just an issue of stopping LSASS from doing this.

Given that Windows Server is designed to handle such scenarios with Active Directory implementations rather than using an excessive number of local user accounts, I recommend evaluating the feasibility of implementing Active Directory or other centralized authentication and user management services like LDAP to avoid such issues in the future.

Unfortunately, there may not be a simple registry tweak to disable LSASS's behavior. Still, migrating away from a large number of local user accounts and using Active Directory or LDAP can enhance your server's performance and security.

 

[ChatGPT]

It seems that your previous message was empty. If you have any more questions or need further assistance, please feel free to ask, and I'll be happy to help.

 

[InformB]

So I'm looking the registry of 2021R2 vs 2019 and I see that in 2012R2 there is no entries for any of the users under Computer\HKEY_LOCAL_MACHINE\SAM\SAM\Domains\Accoun t\Users, however in 2019 there's an entry for each user under Computer\HKEY_LOCAL_MACHINE\SAM\SAM\Domains\Accoun t\Users and Computer\HKEY_LOCAL_MACHINE\SAM\SAM\Domains\Accoun t\Users\Names

In both 2012R2 and 2019 I see all the users are saved under Computer\HKEY_LOCAL_MACHINE\SAM\SAM\Domains\Accoun t\Users

The one difference that I'm seeing in the procmon boot logs is that in 2012R2 there are other activities happening along side with the lsass SAM enumeration. For example I see wmiprvse.exe working alongside lsass.exe throughout the process where as the 2019 boot logs are dominated only by lsass.exe for 9 hours straight.

One other difference I see is in how LSASS.exe is reading SAM is very different in 2012R2 vs 2019. In 2019 it's a straight out reading the database incrementing the index number one at a time where as in 2012R2 it's using a different pattern to read the SAM database from the registry (but alongside other things)

But I think the biggest difference I'm seeing is that there are other activities going on in parallel with LSASS.exe (including logonui, msmpeng etc), so basically in 2012R2 things are moving in parallel with LSASS and once it starts the login process LSASS just seems to stop enumerating the SAM users, where as in 2019 it's a serial process, unless LSASS completes, nothing else happens.

 

[ussnorway]

just another case of Microsoft fixing something

@Mike please add an option to turn your robot off

 

[InformB]

Looking at the procmon logs for the 2019 server more carefully I noticed that it's not that LSASS is taking a long time to parse the 50K users from SAM, it appears to be stuck in a loop!! LSASS keep reading all the SAM entries over and over again in a loop for 12 hours! Here's an excerpt from the logs showing when it ends one loop and then starts another loop. It's the exact same loop everytime. I can't see the end of the loop because procmon timed out after a few hours of logging but for those few hours it ends the last user, then reads the services NTDS and then restart the SAM user loop again. Anyone may any sense of this?

2:59:42.4202794 PM lsass.exe 680 RegEnumKey HKLM\SAM\SAM\Domains\Account\Users\Names SUCCESS Index: 39,993, Name: X
2:59:42.4202969 PM lsass.exe 680 RegOpenKey HKLM\SAM\SAM\Domains\Account\Users\Names\X SUCCESS Desired Access: Read
2:59:42.4203111 PM lsass.exe 680 RegQueryValue HKLM\SAM\SAM\Domains\Account\Users\Names\X\(Default) SUCCESS Type: <Unknown: 82518>
2:59:42.4203232 PM lsass.exe 680 RegCloseKey HKLM\SAM\SAM\Domains\Account\Users\Names\X SUCCESS
2:59:42.4203319 PM lsass.exe 680 RegEnumKey HKLM\SAM\SAM\Domains\Account\Users\Names NO MORE ENTRIES Index: 39,994, Length: 84
2:59:42.4203506 PM lsass.exe 680 RegCloseKey HKLM\SAM\SAM\Domains\Account\Users\Names SUCCESS
2:59:42.4203699 PM lsass.exe 680 RegEnumKey HKLM\SAM\SAM\Domains\Account\Users SUCCESS Index: 9, Name: XX
2:59:42.4203963 PM lsass.exe 680 RegQueryKey HKLM SUCCESS Query: HandleTags, HandleTags: 0x0
2:59:42.4204086 PM lsass.exe 680 RegOpenKey HKLM\SYSTEM\CurrentControlSet\Services\NTDS REPARSE Desired Access: Query Value
2:59:42.4204269 PM lsass.exe 680 RegOpenKey HKLM\System\CurrentControlSet\Services\NTDS SUCCESS Desired Access: Query Value
2:59:42.4204447 PM lsass.exe 680 RegQueryValue HKLM\System\CurrentControlSet\Services\NTDS\DirectoryServiceExtPt NAME NOT FOUND Length: 144
2:59:42.4204594 PM lsass.exe 680 RegCloseKey HKLM\System\CurrentControlSet\Services\NTDS SUCCESS
2:59:42.4204755 PM lsass.exe 680 RegOpenKey HKLM\SAM\SAM\DOMAINS\Account\Users\Names SUCCESS Desired Access: Read
2:59:42.4205331 PM lsass.exe 680 RegEnumKey HKLM\SAM\SAM\Domains\Account\Users\Names SUCCESS Index: 0, Name: XXX
2:59:42.4205462 PM lsass.exe 680 RegOpenKey HKLM\SAM\SAM\Domains\Account\Users\Names\XXX SUCCESS Desired Access: Read
2:59:42.4205604 PM lsass.exe 680 RegQueryValue HKLM\SAM\SAM\Domains\Account\Users\Names\XXX\(Default) SUCCESS Type: <Unknown: 42530>