50,000 users in 2019 hangs on boot but works on 2012R2

InformB

InformB

New Member
Joined
Mar 27, 2023
Messages
11
I've been using Windows server 2012R2 Standard for almost a decade now. I have apps running on the server which have created about 50,000 local users. These are local users created using WMIC scripts and eq. API's. The purpose is to allow for easy integration with IIS authentication. The app creates a local user account and IIS uses the built in Windows authentication to authenticate the user.

It's been running great on 2021R2 but I recently decided to upgrade to a 2019 Datacenter edition so I created a new 2019 server and used the same scripts to recreate the 50,000 users on the 2019 server.

However when I reboot the 2019 server it just hangs for hours at the spinning circle and takes up 100% CPU. It never get the login screen. I've never had this issues with 2012R2, to test it I spun up a new 2012R2 server and recreated the same 50,000 local users with the scripts and it took a few minutes to reboot but that was it.

Is there something different about 2019 server datacenter vs 2012r2 standard which causes the 2019 server to hang on boot when there are 50,000 local users? Is it trying to process all the users before presenting the login (I've set it to login automatically to an admin account but it never seems to get there). I'm at my wits end now and I'm open to suggestions on how to debug this further or what settings should I look at? I've checked out the group security and logon policies and they're both the same for the 2012 and 2019 servers. What am I overlooking?

They're both running on AWS with 1GB RAM. Thanks in advance and please don't hesitate to throw out whatever ideas you may have. I cannot change the architecture at this time and the easiest way to integrate IIS with authentication is to create a local user account. They don't need to have local login privileges only network login (for IIS), so these accounts are added to a custom Group on the windows server and removed from the standard Users group.
 
Solution
ChatGPT
It's possible that the issue is related to the number of local users on the 2019 server. While there is no hard limit on the number of local users that can be created on a Windows server, excessive numbers of users can cause performance issues.
You may want to try some troubleshooting steps to determine the cause of the issue:
  • Check the event logs on the 2019 server to see if there are any errors or warnings that might indicate what is causing the hang.
  • Try booting the server in safe mode to see if it can successfully boot without all of the installed services and drivers. If it does boot successfully, you can begin to narrow down the cause by selectively enabling services and drivers until the issue returns.
  • Check the...
It's possible that the issue is related to the number of local users on the 2019 server. While there is no hard limit on the number of local users that can be created on a Windows server, excessive numbers of users can cause performance issues.
You may want to try some troubleshooting steps to determine the cause of the issue:
  • Check the event logs on the 2019 server to see if there are any errors or warnings that might indicate what is causing the hang.
  • Try booting the server in safe mode to see if it can successfully boot without all of the installed services and drivers. If it does boot successfully, you can begin to narrow down the cause by selectively enabling services and drivers until the issue returns.
  • Check the system resources (CPU, memory, disk usage) during the boot process to see if there are any bottlenecks that might be causing the hang.
  • Consider temporarily disabling some of the startup programs to see if any of them might be causing the issue.
  • Run the System File Checker (SFC) tool to check for any system file corruption that might be contributing to the issue.
    It's also worth noting that running a Windows server on 1GB of RAM is quite low and may be contributing to the performance issues. It might be worth considering increasing the RAM allocation for the server to see if that helps.
 
Solution
I would highly like to avoid a chatgpt response as it's absolutely useless. My first comment was it does NOT boot, which means I can't check logs, I can't get to the login screen, I've checked the bootlog and the last thing it shows was a successful loading of a driver. I can't even boot into safe mode as it just hangs.

I start with a clean 2019 image, run the scripts to create the users and it's working great, the moment I reboot it just keeps spinning on the circle with 100% CPU utilization. When I do the exact same process with 2012R2 it works fine.
 
There must be some kind of processing going on that did not exist in Server 2012 that would prohibit that many accounts. Are you able to see the PID or descriptor of the CPU process taking up all the resources? And yeah the bot really can't solve this kind of problem without a real investigation of whatever is going on with the server. There are some more advanced security features in Server 19 that could be getting triggered when this many local accounts are created.
 
@Mike thanks for the suggestion.

Just for kicks, I tried a new AWS 2019 server with 4 and 8GB RAM and also more processors, same result. The scripts runs, add 40K users and everything is working fine up, it's responsive and working after the scripts are done (takes about 30 min to complete), the CPU utils are under 1%. When I restart the machine and I end up with the spinning dots (on AWS I can't see it because RDP isn't up or isn't responding) but when tested with a local VM I can see it there, so it's not the size of the RAM/hardware which appears to be the issue here.

I can't see any PID since it's still booting (the spinning dots). I tried using safe mode and it's the exact same issue, ends up with the spinning dots or a blank screen (depending on what version of safe mode I pick).

Can you think of anything else? Is there to export all the configuration/settings of a windows server? That way I can export the 2021R2 and do a diff against the 2019 and see what settings are different.
 
I have one more insight, I created a new 2019 server, added the 50k new account but then just left the machine without rebooting. I logged out of the RDP session, wait for a few hours and tried to RDP back in and now it's hung on "Please wait for Local Session Manager" and it just times out, the console shows the CPU usage spikes to 100%. So it seems like something with Local Session Manager is causing the issue. I did the same with a 2022 server also and the same results (so it's not specific to 2019). Any ideas?
 
have you tested just making one or two users by hand and restarting?
 
ussnorway said:
have you tested just making one or two users by hand and restarting?
Click to expand...
Thanks, yes I've done that. I've tried testing with increasing numbers, under a 100 it's fine (scripting or by hand). When I run it up to about 3000 then it really slows down, takes 15 minutes to start. So as the numbers keep increasing it's slowing down more, but I don't see that with my older 2012R2 instance. It takes a consistent 3-4 minutes to boot and login to the automatic user login configured, even at 50K users.
 
For this specifically, the actual service might be timing out on start-up. There is a registry edit to try to correct this:

A service does not start, and events 7000 and 7011 are logged in Windows Server - Windows Server

Describes a problem in which a service times out before it starts. Explains how to work around this problem by increasing the value of the ServicesPipeTimeout registry entry.
learn.microsoft.com
 
One more possibility here:

Long delay to log in, 'Please wait for the Local Session Manager'

social.technet.microsoft.com
 
Mike said:
One more possibility here:

Long delay to log in, 'Please wait for the Local Session Manager'

social.technet.microsoft.com
Click to expand...
Thanks, I tried this and it didn't work but it's given me a few leads to chase up by looking at differences between the registry settings between the 2012R2 and 2019 servers. I'll also try the other link you sent me, this is very helpful - please send me any other ideas you may think of. I suspect it's something to do with LASS or SAM. When it did eventually boot up (after 12 hours), I looked the system logs and there was a 3 hour gap between the kernel boot and next system message and the next message something about the SAM being unavailable.
 
I don't think the SAM database was meant to handle that many local accounts (that's what LDAP or a sql server is meant for). I would suspect the entire SAM db is being loaded or profiles are being pre-fetched.
 
Any ideas on what I can do? Can I move the users to LDAP on the same machine and then use that for integrated Windows authentication via IIS? That's pretty much the only requirement (authenticate via IIS and be able to create the user via a batch/command prompt command).
 
I'm almost convinced it's a SAM issue. When I look at the server system event logs (without any users), the Directory-Services-SAM there are 3 SAM event logs that complete within about 1 second. On the server with the 50k users, those same 3 events are 12 hours apart, the first one right after Wininit and the the next one 12 hours later (when login takes place)

So the first one after Wininit:
> Remote calls to the SAM database are being restricted using the default security descriptor: O:SYG:SYDA;;RC;;;BA).
For more information please see Network access - Restrict clients allowed to make remote calls to SAM.

The second one after 12 hours:
> The domain is configured with the following minimum password length-related settings.
MinimumPasswordLength: 0
MinimumPasswordLengthAudit: -1
For more information see Minimum Password Length auditing and enforcement on certain versions of Windows - Microsoft Support.

And the third one immediately thereafter:
> The security account manager is now logging periodic summary events for remote clients that call legacy password change or set RPC methods.
For more information please see KB5004605: Update adds AES encryption protections to the MS-SAMR protocol for CVE-2021-33757 - Microsoft Support.

Any thoughts on how to get SAM to speed up or not load all profiles at boot up?
 
I found the culprit, it's LSASS (using ProcMon), it's iterating through every user account in the registry HKLM\SAM\Domains\Account\Users and for every user it's making 4 calls to the registry





So the question is how do I stop LSASS from doing this? For some reason 2012R2 isn't doing it but 2019 and 2022 servers are doing this. Is there a registry setting to control this?
 
Sorry, I was testing something. I apologize for the notification blast you may have just received from it. I'm not sure there is a solution yet.
 
There is no tweaking of LSASS with regards to SAM performance that I am aware of. I would strongly recommend using LDAP or a SQL based authentication system as these are intended for large user bases.
 
So I'm looking the registry of 2021R2 vs 2019 and I see that in 2012R2 there is no entries for any of the users under Computer\HKEY_LOCAL_MACHINE\SAM\SAM\Domains\Accoun t\Users, however in 2019 there's an entry for each user under Computer\HKEY_LOCAL_MACHINE\SAM\SAM\Domains\Accoun t\Users and Computer\HKEY_LOCAL_MACHINE\SAM\SAM\Domains\Accoun t\Users\Names

In both 2012R2 and 2019 I see all the users are saved under Computer\HKEY_LOCAL_MACHINE\SAM\SAM\Domains\Accoun t\Users

The one difference that I'm seeing in the procmon boot logs is that in 2012R2 there are other activities happening along side with the lsass SAM enumeration. For example I see wmiprvse.exe working alongside lsass.exe throughout the process where as the 2019 boot logs are dominated only by lsass.exe for 9 hours straight.

One other difference I see is in how LSASS.exe is reading SAM is very different in 2012R2 vs 2019. In 2019 it's a straight out reading the database incrementing the index number one at a time where as in 2012R2 it's using a different pattern to read the SAM database from the registry (but alongside other things)

But I think the biggest difference I'm seeing is that there are other activities going on in parallel with LSASS.exe (including logonui, msmpeng etc), so basically in 2012R2 things are moving in parallel with LSASS and once it starts the login process LSASS just seems to stop enumerating the SAM users, where as in 2019 it's a serial process, unless LSASS completes, nothing else happens.
 
just another case of Microsoft fixing something

@Mike please add an option to turn your robot off
 
You must log in or register to reply here.

Similar threads

  • Featured
  • Article Article
Resolving SQL Server 2019 Service Account Password Errors in Home Labs
Replies
0
Views
91
ChatGPT
G
  • Solved
RDP Connection Authentication Error
Replies
1
Views
1K
ChatGPT
  • Featured
  • Article Article
How to Fix and Prevent the 'Account Locked Out' Error on Windows 10/11
Replies
0
Views
100
ChatGPT
  • Featured
  • Article Article
Why Using a Local Windows Account Is Becoming More Challenging in the Cloud Era
Replies
0
Views
67
ChatGPT
  • Featured
  • Article Article
How to Install and Use Active Directory Tools on Windows 11 for Seamless Directory Management
Replies
1
Views
359
ChatGPT