Problem logging into Windows 2022 OpenSSH server using public and private key authentication

K

kaplan71

New Member
Joined
Dec 5, 2025
Messages
5
Hello.

We have set up an OpenSSH server on our Windows 2022 Server with Desktop Experience configuration. The version of the software installed is the following:

OpenSSH_for_Windows_10.0p2 Win32-OpenSSH-GitHub, LibreSSL 4.2.0

Testing has confirmed that username/password authentication is working without issue. We would like to have public/private key authentication, but we are running into difficulties.

We have gone into the C:\ProgramData\ssh folder of the server, and have made the following modifications to the sshd_config file:
  1. PubkeyAuthentication yes
  2. PasswordAuthentication yes
The OpenSSH SSH server service has been restarted, but access using the public and private authentication is not successful.

When connecting to the server from a client system, the MobaXterm utility was used to create the public and private keys. The names of the keys are as follows:

  1. id_rsa <- public key
  2. id_rsa.ppk <- private key
The connection is done using the -v parameter to get the connection readout. The output file has been attached to this posting. The following text was spotted towards the end of the readout:

Load key "/home/mobaxterm/.ssh/id_rsa": invalid format

Can someone provide assistance on this?
 
Solution
Good progress — your client is now successfully offering the private key (the client-side debug shows the client signed and sent a publickey packet) but the server is refusing it and falling back to password. That means the server either didn’t find/accept the matching public key or refused it because of format/permission/location issues. Below are the most likely causes (priority order) and concrete checks/fixes to try next.
Key causes and what to check (quick)
  • Public key in the server’s authorized_keys does not exactly match the client’s public key (copy/paste error, extra whitespace, line breaks, wrong key).
  • authorized_keys is in the wrong location or sshd_config’s AuthorizedKeysFile has been changed.
  • File/folder...
Hi kaplan71 — welcome. The “Load key ... invalid format” you’re seeing is almost always a client-side key-format issue (OpenSSH expecting an OpenSSH-format private key but getting something else, commonly a PuTTY .ppk or the wrong file). I didn’t receive the verbose output file you mentioned — please attach it if you want me to inspect the exact log — but here’s a focused checklist and step‑by‑step fixes that will get key auth working.
Summary (short)
  • Cause: your client is trying to load /home/mobaxterm/.ssh/id_rsa but that file is in the wrong format (likely a PuTTY .ppk or a public-key file) so OpenSSH reports “invalid format.”
  • Fix: supply a proper OpenSSH private key file on the client and the matching OpenSSH-format public key on the Windows server, and ensure Windows SSH file/ACLs are correct. Windows OpenSSH also has known ACL/permission gotchas you should verify.
Step-by-step troubleshooting and fixes
1) Confirm what the client private key actually is
  • Open the key file in a text editor or peek at the first line:
    • PPK (PuTTY) header looks like: PuTTY-User-Key-File-2: ssh-rsa
    • OpenSSH private key looks like: -----BEGIN OPENSSH PRIVATE KEY----- or -----BEGIN RSA PRIVATE KEY-----
  • If the file is a .ppk or the header is PuTTY, OpenSSH will reject it with “invalid format.”
2) Convert the PuTTY / MobaXterm key to OpenSSH format (pick one)
  • With PuTTYgen (GUI):
    • Load id_rsa.ppk → Conversions → Export OpenSSH key → save as id_rsa (no extension).
  • With puttygen CLI (if installed):
    • puttygen id_rsa.ppk -O private-openssh -o ~/.ssh/id_rsa
  • With MobaXterm (MobaKeyGen): Load .ppk → Export OpenSSH → save to ~/.ssh/id_rsa
  • After saving, set permissions (in MobaXterm / WSL / Linux client): chmod 600 ~/.ssh/id_rsa
  • Note: the public key file should be id_rsa.pub (OpenSSH format, single-line starting with ssh-rsa ....
3) Make sure the server has the correct public key in the right place
  • On the Windows 2022 server put the public key (the contents of id_rsa.pub) into:
    • C:\Users\<USERNAME>.ssh\authorized_keys
    • One key per line, OpenSSH public-key format (ssh-rsa AAAA... comment)
  • Confirm sshd_config’s AuthorizedKeysFile setting if you customized it. Default on Windows typically uses the user .ssh\authorized_keys or an administrators_authorized_keys location for admin logins.
4) Fix Windows/OpenSSH permissions (very common problem)
  • Windows OpenSSH and recent updates can change ACLs and block key auth or even service start. Microsoft/community workarounds recommend adjusting ACLs on SSH folders (C:\ProgramData\ssh and each user .ssh folder). Example quick steps (run as Administrator; back up ACLs first):
    • Backup (optional): icacls "C:\ProgramData\ssh" /save ssh_permissions_backup.txt
    • Grant proper rights (example):
      icacls "C:\ProgramData\ssh" /grant "Administrators:F" /grant "SYSTEM:F" /grant "Authenticated Users:R" /T
      icacls "C:\Users\<USERNAME>.ssh" /grant "<USERNAME>:F" /grant "SYSTEM:F" /grant "Administrators:F" /T
    • Or use the SDDL-based PowerShell fix recommended in community docs. These ACL/workaround details and guidance are documented for Windows OpenSSH.
  • Important: avoid giving Everyone full control; only grant the minimal required rights. Back up before changing ACLs.
5) Enable verbose/debug on the server if still failing
  • In sshd_config set LogLevel DEBUG3, restart sshd, then inspect:
    • Event Viewer -> Applications and Services Logs -> OpenSSH/Operational
    • C:\ProgramData\ssh\logs (if enabled)
  • The logs will show whether the server accepted the key and why it rejected it.
6) Client session configuration (MobaXterm)
  • In the MobaXterm session settings ensure you point the session at the OpenSSH-format private key (no .ppk) or use Pageant if you want to continue using the .ppk. If you converted the key, point MobaXterm to the new id_rsa file.
  • When testing from a shell you can force which key to use: ssh -i ~/.ssh/id_rsa -v user@server and watch the verbose output.
Common mistakes I see from your description
  • You named the public key file id_rsa (missing the .pub) and the private key id_rsa.ppk — OpenSSH expects private key to be id_rsa (no extension) and public id_rsa.pub. If you copied the .ppk into ~/.ssh/id_rsa, OpenSSH will fail. Convert the .ppk to OpenSSH format or generate an OpenSSH keypair. (This is the most likely cause of "invalid format".
What I need from you to go further
  • Please attach the MobaXterm verbose log you mentioned (I don’t see it attached) or paste the final ~20 lines around the error. I can then confirm whether the client is presenting a PPK or another problem.
  • If you prefer, paste the first line of your client private key (sanitize if you prefer) so I can identify the format quickly.
  • Tell me the exact location of the authorized_keys file you used on the Windows server (C:\Users\<user>.ssh\authorized_keys or elsewhere).
If you want, I can:
  • Give exact puttygen/ssh-keygen commands to convert and set permissions for your environment.
  • Provide the PowerShell/icacls commands tailored to your user account so you can apply the ACL fixes safely.
Quick reference links in this thread: Windows/OpenSSH setup and known ACL issues are discussed in the OpenSSH-on-Windows guidance and community posts here.
Tell me which step you want to try first or attach the verbose log and I’ll walk through the next steps.
 
I have some progress. I was able to get the correct format of the public and private keys, and I was able to upload the public key to the SSH server. The 'invalid format' error is not present. However, I am still being prompted for the password. The excerpt shown below lists what is occurring:

What are my next steps?
 
Good progress — your client is now successfully offering the private key (the client-side debug shows the client signed and sent a publickey packet) but the server is refusing it and falling back to password. That means the server either didn’t find/accept the matching public key or refused it because of format/permission/location issues. Below are the most likely causes (priority order) and concrete checks/fixes to try next.
Key causes and what to check (quick)
  • Public key in the server’s authorized_keys does not exactly match the client’s public key (copy/paste error, extra whitespace, line breaks, wrong key).
  • authorized_keys is in the wrong location or sshd_config’s AuthorizedKeysFile has been changed.
  • File/folder permissions or ACLs prevent sshd from reading the user’s .ssh/authorized_keys (Windows OpenSSH ACL issues are common).
  • Key file line endings / formatting (must be one OpenSSH-format line: ssh-rsa AAAA... comment).
  • The user you authenticate as is not the same user whose authorized_keys you edited (typo, different UPN vs sAMAccountName, connecting as Administrator vs domain user).
Step‑by‑step checks and fixes (do these on the server; PowerShell examples)
1) Confirm exact public key on server matches the client
  • On the client run (where private key lives): ssh-keygen -y -f ~/.ssh/id_rsa — this prints the public key derived from the private key. Copy that output.
  • On the server show the authorized_keys contents:
    PowerShell: Get-Content "C:\Users\<username>\.ssh\authorized_keys"
    Make sure the single-line output exactly equals the client’s public key (no extra leading/trailing spaces, no CRLF inserted mid-line). If you used MobaXterm to upload, re-open the file in a text editor and ensure it's a single line beginning with ssh-rsa (or ssh-ed25519 etc). The server expects the standard OpenSSH public-key one‑line format.
2) Confirm you put the key in the right place / check sshd_config
  • Default expected file is the user’s %USERPROFILE%.ssh\authorized_keys (C:\Users\<username>.ssh\authorized_keys) unless AuthorizedKeysFile was changed in sshd_config (C:\ProgramData\ssh\sshd_config).
  • Open sshd_config and search for AuthorizedKeysFile to confirm the path(s) sshd expects; if changed, either copy the key to that location or restore the default.
3) Check and fix file/folder permissions / ACLs (very common on Windows)
  • Show current ACLs:
    • icacls "C:\Users\<username>\.ssh\authorized_keys"
    • icacls "C:\Users\<username>\.ssh"
    • icacls "C:\ProgramData\ssh"
  • Minimal safe ACLs: the user must be able to read their authorized_keys, and sshd (SYSTEM) must have access to the folder chain. A practical fix many admins use (run as Administrator; replace <username> with the actual account name):
    • icacls "C:\Users\<username>\.ssh" /grant "<username>:(F)" /grant "SYSTEM:(F)" /grant "Administrators:(F)" /T
    • icacls "C:\Users\<username>\.ssh\authorized_keys" /grant "<username>:(R)" /grant "SYSTEM:(R)" /grant "Administrators:(R)"
    • Also ensure C:\ProgramData\ssh and its logs are accessible to SYSTEM and Administrators (Microsoft/community workarounds use similar icacls/PowerShell SDDL fixes).
  • After changing ACLs restart the sshd service: Restart-Service sshd
4) Turn up server logging to see exact rejection reason
  • Edit C:\ProgramData\ssh\sshd_config and set LogLevel DEBUG3 (uncomment if needed). Restart sshd. Then inspect logs:
    • Look in Event Viewer under Applications/Services Logs → OpenSSH, and/or check C:\ProgramData\ssh\logs (if present). The server log will show lines like “Authentication refused: ...” or “user key mismatch” and exactly why the key was rejected. This is the clearest diagnostic.
5) Test again (client)
  • From your client run with maximum verbosity and keep the server logs open to correlate:
    ssh -vvv -i ~/.ssh/id_rsa <username>@<server>
  • Copy the relevant server-side log lines (the lines when the client sends the publickey and the server’s response). Paste those here if you want help interpreting them.
Extra checks / gotchas
  • If you put the key into a machine-wide file (C:\ProgramData\ssh\administrators_authorized_keys) or changed AuthorizedKeysFile, confirm which accounts use which file. If you edited the Administrator’s file but you’re connecting as DOMAIN\user (or vice versa), the server won’t match.
  • Ensure there are no duplicate keys or two different keys with similar comments; compare the base64 blob (the long AAAA... part).
  • If the public key file was edited on Windows, some editors can insert a BOM or change line endings — re-save as plain UTF-8 without BOM or re-create the file by appending the key with echo/Out-File to avoid hidden characters.
What to send me if you want me to diagnose further
  • The server-side sshd log lines (LogLevel DEBUG3) covering the failed authentication attempt; paste the 10–20 lines around the failure. This is the most helpful single item.
  • Output of icacls for the user’s .ssh folder and authorized_keys file.
  • The exact content of the public key as returned by ssh-keygen -y -f on the client (or the first ~40 characters of the base64 blob) so I can confirm it’s the same as on the server.
If you want, I can give exact PowerShell/ICACLS commands tailored to your username and show how to safely back up current ACLs before making changes. Which would you prefer — run the ACL checks yourself and paste the results, or shall I give the ready-to-run commands for you to apply?
 
Solution
I changed the debug level to 3 on the ssh server, and the Event Log had the following entry:

The description for Event ID 4 from source OpenSSH cannot be found. Either the component that raises this event is not installed on your local computer or the installation is corrupted. You can install or repair the component on the local computer.

If the event originated on another computer, the display information had to be saved with the event.

The following information was included with the event:

sshd-session
Failed publickey for <username> from <ip address> port 50342 ssh2: RSA SHA256:FEde5f/e/t3sF5FGxmWHUH8b7mFXi6i8IaFR2skbYhY

The message resource is present but the message was not found in the message table
 
Thanks — the Event Log entry is helpful: it shows the server received and rejected the offered key (Failed publickey for … RSA SHA256:FEde5f/…), which means the client is presenting the correct key but the server is refusing it. That narrows the causes to: (A) the server’s authorized_keys doesn’t match exactly, (B) sshd can’t read the file because of location/format/ACLs, or (C) sshd configuration is pointing somewhere else or enforcing checks (StrictModes). Below are concrete checks and commands (copy/paste) you can run to find the exact reason and fix it.
1) Verify the public key on the client and the server match
  • On the client (MobaXterm / Linux shell), print the public key derived from the private key:
    • ssh-keygen -y -f ~/.ssh/id_rsa > /tmp/mykey.pub
    • cat /tmp/mykey.pub
  • On the server (PowerShell, run as Administrator) print what’s in authorized_keys:
    • Get-Content "C:\Users\<username>.ssh\authorized_keys"
  • Compare them byte-for-byte. The public key line must be a single line beginning with ssh-rsa (or ssh-ed25519) and the long base64 blob must match exactly. If you prefer fingerprints, create a temp file on the server and compare fingerprints:
    • PS> $ak = Get-Content "C:\Users\<username>.ssh\authorized_keys"
    • PS> Set-Content -Path C:\Temp\authkey.pub -Value $ak
    • PS> & 'C:\Windows\System32\OpenSSH\ssh-keygen.exe' -lf C:\Temp\authkey.pub
  • On the client get fingerprint of your private key:
    • ssh-keygen -y -f ~/.ssh/id_rsa | ssh-keygen -lf -
  • If the fingerprints differ, re-copy the exact public key output from the client into the server’s authorized_keys (overwrite and save as one line).
2) Check for bad formatting / line endings / BOM
  • Edit authorized_keys with an editor that can show hidden characters (Notepad++ or use PowerShell). If the file contains CRLF, BOM, or extra line breaks it can fail. Save as plain UTF-8 without BOM, Unix line endings (LF) if possible.
  • Quick PowerShell fix to rewrite as plain text:
    • (Get-Content "C:\Users\<username>.ssh\authorized_keys") | Out-File -FilePath "C:\Users\<username>.ssh\authorized_keys" -Encoding ascii
3) Confirm sshd is looking in the file you edited
  • Open C:\ProgramData\ssh\sshd_config and check AuthorizedKeysFile (if uncommented). Default expects the user’s %USERPROFILE%.ssh\authorized_keys. If you changed it, copy the key to the configured location. Also confirm PubkeyAuthentication yes is present.
4) Check and fix permissions / ACLs (very common on Windows)
  • Show ACLs:
    • icacls "C:\Users\<username>.ssh"
    • icacls "C:\Users\<username>.ssh\authorized_keys"
  • A practical permissive-but-safe set (run as Administrator; replace <username>):
    • icacls "C:\Users\<username>.ssh" /grant "<usernameF)" /grant "SYSTEMF)" /grant "AdministratorsF)" /T
    • icacls "C:\Users\<username>.ssh\authorized_keys" /grant "<usernameR)" /grant "SYSTEMR)" /grant "AdministratorsR)"
  • Also ensure C:\ProgramData\ssh is accessible by SYSTEM and Administrators; Windows OpenSSH ACL issues have been a recurring cause and Microsoft/community guidance shows similar icacls or SDDL fixes.
  • After changing ACLs restart sshd:
    • Restart-Service sshd
5) Temporarily disable StrictModes to test (if you want a quick diagnostic)
  • In C:\ProgramData\ssh\sshd_config set:
    • StrictModes no
  • Restart sshd and try connecting. If key auth then works, the failure is definitely permissions/ownership/format. Do not leave StrictModes off in production — it’s a diagnostic step only.
6) Use DEBUG3 logs and check the detailed server log
  • You already set LogLevel DEBUG3; with that enabled, find the detailed messages in:
    • Event Viewer → Applications and Services Logs → OpenSSH (or check C:\ProgramData\ssh\logs\sshd.log if present)
  • Look for lines immediately after the “Failed publickey” that explain why (e.g., “bad ownership or modes for directory”, “no matching key found in authorized_keys”, “signature invalid”). Paste 15–30 lines around that failure and I’ll interpret them if you want.
7) Common account/location mistakes
  • Make sure you’re authenticating as the same Windows account whose C:\Users\<username>.ssh\authorized_keys you edited.
  • For domain accounts, confirm the username used in the ssh command matches the user profile folder on the server (sometimes UPN vs sAMAccountName differences confuse which profile is used).
8) If still failing, gather these items and post them here
  • The DEBUG3 server log lines around the failure (10–20 lines).
  • Output of icacls for the user’s .ssh and authorized_keys.
  • The public-key fingerprint from the client (ssh-keygen commands above) and the fingerprint printed from the server temporary authkey.pub file.
If you’d like, paste the server DEBUG3 lines and the icacls outputs here and I’ll read them and tell you exactly which ACL or key mismatch is causing the rejection and the precise command to fix it.
 
I checked if the public key on the client and server match. To that end, I ran the ssh-keygen to generate a mykey.pub file on the client workstation. The file was then copied to the server. The mykey.pub file is the same size as the authorized_keys file. The Winmerge utility was used to compare the files, and the following error message appeared on-screen:

Files use different EOL.
Treat all EOL as equivalent?
Set 'Ignore EOL differences in Compare section of Options
 
Good — the WinMerge message confirms the files differ only by EOL, which is a very likely reason the server rejected the key. On Windows the authorized_keys must be the exact OpenSSH-format public-key line (no CRLF/BOM/extra characters) and readable by sshd; converting the file to Unix line endings (LF) and removing any BOM normally fixes this. Below are safe, concrete steps (with copy/paste commands) to fix EOL/encoding, verify the key matches, and confirm ACLs — followed by what to send me if it still fails. I’ve included the same troubleshooting checks we discussed earlier.
Quick plan
1) Back up the current authorized_keys.
2) Replace/rewrite authorized_keys with the exact public-key output from the client, saved with LF-only line endings and no BOM.
3) Verify fingerprints match between client key and server file.
4) Confirm ACLs so sshd can read the file.
5) Retry and read the DEBUG3 log if it still fails.
Commands — do these on the server (run PowerShell as Administrator)
1) Back up existing file
Replace <username> with the Windows account name.
PowerShell:
Copy-Item -Path "C:\Users\<username>.ssh\authorized_keys" -Destination "C:\Users\<username>.ssh\authorized_keys.bak"
2) On the client produce the exact public key line
(on MobaXterm / client shell)
ssh-keygen -y -f ~/.ssh/id_rsa > /tmp/mykey.pub
cat /tmp/mykey.pub
Copy the single line it prints (starts with ssh-rsa or ssh-ed25519 ....
3) Recreate authorized_keys on the server as a plain, single-line file with LF and no BOM
On the server (PowerShell), paste the one-line public key into the command below (replace <paste-key-here> with the copied single line exactly):
Set-Content -Path "C:\Users\<username>.ssh\authorized_keys" -Value '<paste-key-here>' -Encoding ascii
Notes:
  • Using -Encoding ascii (or -Encoding utf8NoBOM on newer PowerShell) ensures no BOM is added; ascii is safe because the key line is plain ASCII. This removes CRLF/BOM issues. If you prefer GUI, open the file in Notepad++ and: Encoding → UTF-8 without BOM, Edit → EOL Conversion → UNIX (LF), then Save.
4) Verify fingerprints match (client vs server)
  • Client fingerprint (run on the client):
    ssh-keygen -y -f ~/.ssh/id_rsa | ssh-keygen -lf -
  • Server fingerprint (on the server; creates temp file from authorized_keys and checks it with the OpenSSH ssh-keygen that ships with Windows):
    $ak = Get-Content "C:\Users\<username>.ssh\authorized_keys"
    Set-Content -Path C:\Temp\authkey.pub -Value $ak -Encoding ascii
    & 'C:\Windows\System32\OpenSSH\ssh-keygen.exe' -lf C:\Temp\authkey.pub
Compare the two fingerprints; they must be identical. If they differ, the public key on the server is not the same as the private key on the client and you need to re-copy the client output exactly.
5) Check and set safe ACLs so sshd can read the file
Show current ACLs:
icacls "C:\Users\<username>.ssh"
icacls "C:\Users\<username>.ssh\authorized_keys"
A practical (safe-enough) fix to apply (run as Administrator; replace <username>):
icacls "C:\Users\<username>.ssh" /grant "<usernameF)" /grant "SYSTEMF)" /grant "AdministratorsF)" /T
icacls "C:\Users\<username>.ssh\authorized_keys" /grant "<usernameR)" /grant "SYSTEMR)" /grant "AdministratorsR)"
After changing ACLs restart the sshd service:
Restart-Service sshd
Windows OpenSSH has known ACL/permission gotchas after some updates — fixing ACLs is a common resolution.
6) (Optional short test) Temporarily set StrictModes no to rule out permission problems quickly
  • Edit C:\ProgramData\ssh\sshd_config and set StrictModes no, Restart-Service sshd, then test. If key auth works only when StrictModes is off, the issue is definitely permissions/ownership/line-ending related. (Do not leave StrictModes disabled in production.
7) Re-test from the client with max verbosity and, if it fails, collect the server debug lines
Client:
ssh -vvv -i ~/.ssh/id_rsa username>@<server
If auth still fails, on the server collect the DEBUG3 lines from Event Viewer / OpenSSH log around the failure and paste them here (about 10–20 lines). That log will show the precise reason (e.g., “no matching key found”, “bad ownership or modes”, “signature invalid”).
If you want, I can give you a one-line pipeline to copy the client public key into the server authorized_keys over an SSH/password session (so you avoid manual paste), or exact PowerShell commands that write the key and set ACLs automatically — tell me whether you prefer to (A) paste the key manually or (B) use an automated command and I’ll provide it.
Recap / next steps
  • Convert authorized_keys to LF/no BOM and overwrite it with the client’s ssh-keygen -y output; verify fingerprints as shown.
  • Ensure ACLs permit SYSTEM and the user to read the file; restart sshd.
  • If it still fails, paste: (a) the server DEBUG3 log lines around the Failed publickey, and (b) the outputs of the two icacls commands above. I’ll read them and tell you the exact fix.
Which do you want to do now — paste the public-key line and I’ll give the exact Set-Content command with it, or shall I provide the automated copy command you can run from the client?
 
You must log in or register to reply here.