An AI model that answered a test prompt one way last month and another way today may have changed—but a different answer alone does not prove it. Generative models can vary between runs, and changes to prompts, settings, or the surrounding application can also affect results.

A small Windows PowerShell test can make those differences easier to spot. The following walkthrough saves responses to fixed prompts, reruns them later, and flags exact text changes for review. It is a smoke test, not a safety certification or a guarantee that model drift has been detected.

A desktop monitor displays an AI smoke test comparing baseline and new JSON responses, with two small differences highlighted. What you will need​

  • Windows PowerShell 5.1 on a Windows PC, or PowerShell 7 on Windows.
  • An OpenAI API account, an API key, and access to the model used below.
  • A network connection and permission to send the test prompts to the API.
  • A local folder where you can keep test results.

API requests may incur charges. Use fictional or otherwise approved test data: both prompts and responses will be saved locally. The example uses OpenAI’s API directly, not an Azure OpenAI endpoint; authentication and request details for other providers may differ.

OpenAI documents that model output is nondeterministic and can differ between model snapshots and families. This example pins a specific GPT-5.4 mini snapshot, which makes the test configuration more stable than using a moving model alias. A pinned snapshot still does not guarantee identical answers.

1. Create the test folder and script​

Open PowerShell and create a working folder:

Code:
New-Item -ItemType Directory -Path "$HOME\AI-Drift-Test" -Force
Set-Location "$HOME\AI-Drift-Test"
notepad.\Test-AIOutput.ps1

If Notepad asks to create the file, accept. Paste the script below and save it as Test-AIOutput.ps1.

The script sends three fixed prompts. Each run makes three API requests, saves a JSON record, and—when run in comparison mode—checks each answer against the baseline using a case-sensitive exact comparison.

Code:
param(
 [Parameter(Mandatory = $true)]
 [ValidateSet('Baseline', 'Compare')]
 [string]$Mode
)

$ErrorActionPreference = 'Stop'

if ([string]::IsNullOrWhiteSpace($env:OPENAI_API_KEY)) {
 throw 'Set OPENAI_API_KEY in this PowerShell session first.'
}

$folder = Join-Path $HOME 'AI-Drift-Test'
$baselinePath = Join-Path $folder 'baseline.json'
New-Item -ItemType Directory -Path $folder -Force | Out-Null

if ($Mode -eq 'Baseline' -and (Test-Path -LiteralPath $baselinePath)) {
 throw 'baseline.json already exists. Archive it before creating a new baseline.'
}

if ($Mode -eq 'Compare' -and -not (Test-Path -LiteralPath $baselinePath)) {
 throw 'baseline.json was not found. Run Baseline mode first.'
}

# Keep these settings and prompts unchanged between comparable runs.
$model = 'gpt-5.4-mini-2026-03-17'
$maxOutputTokens = 512

$tests = @(
 [pscustomobject]@{
 Id = 'summary'
 Prompt = 'Summarize this sentence in one short sentence: A workstation was restarted after its security updates were installed.'
 },
 [pscustomobject]@{
 Id = 'classification'
 Prompt = 'Classify this message as either urgent or routine. Reply with only the label: Please reset my password when convenient.'
 },
 [pscustomobject]@{
 Id = 'calculation'
 Prompt = 'What is 17 plus 25? Reply with only the number.'
 }
)

$headers = @{
 Authorization = "Bearer $env:OPENAI_API_KEY"
}

$results = @()

foreach ($test in $tests) {
 Write-Host "Running test: $($test.Id)"

 $body = @{
 model = $model
 input = $test.Prompt
 max_output_tokens = $maxOutputTokens
 store = $false
 } | ConvertTo-Json -Depth 10

 try {
 $response = Invoke-RestMethod `
 -Method Post `
 -Uri '[url]https://api.openai.com/v1/responses[/url]' `
 -Headers $headers `
 -ContentType 'application/json; charset=utf-8' `
 -Body $body `
 -TimeoutSec 120 `
 -ErrorAction Stop
 }
 catch {
 throw "API request failed for test '$($test.Id)'. No run file was saved. $($_.Exception.Message)"
 }

 if ($response.status -ne 'completed') {
 throw "Test '$($test.Id)' returned status '$($response.status)'. No run file was saved."
 }

 # The API output can contain more than one item.
 $textParts = @(
 foreach ($item in $response.output) {
 if ($item.type -eq 'message') {
 foreach ($part in $item.content) {
 if ($part.type -eq 'output_text') {
 $part.text
 }
 }
 }
 }
 )

 if ($textParts.Count -eq 0) {
 throw "Test '$($test.Id)' returned no output text. No run file was saved; inspect the request or response."
 }

 $results += [pscustomobject]@{
 Id = $test.Id
 Prompt = $test.Prompt
 Response = ($textParts -join "`n")
 ResponseId = $response.id
 ReturnedModel = $response.model
 }
}

$run = [pscustomobject]@{
 CreatedUtc = [DateTime]::UtcNow.ToString('o')
 RequestedModel = $model
 MaxOutputTokens = $maxOutputTokens
 Results = $results
}

if ($Mode -eq 'Baseline') {
 $run | ConvertTo-Json -Depth 10 |
 Set-Content -LiteralPath $baselinePath -Encoding UTF8

 Write-Host "Baseline saved: $baselinePath"
 Write-Host 'Review the saved responses for correctness before relying on them.'
 return
}

$runName = 'run-{0}-{1}.json' -f `
 [DateTime]::UtcNow.ToString('yyyyMMdd-HHmmss'), `
 [Guid]::NewGuid.ToString('N').Substring(0, 8)

$runPath = Join-Path $folder $runName
$run | ConvertTo-Json -Depth 10 |
 Set-Content -LiteralPath $runPath -Encoding UTF8

$baseline = Get-Content -LiteralPath $baselinePath -Raw |
 ConvertFrom-Json

if ($baseline.RequestedModel -cne $run.RequestedModel -or
 $baseline.MaxOutputTokens -ne $run.MaxOutputTokens) {
 Write-Warning 'Model or output-token setting differs from the baseline. Treat this as a configuration comparison.'
}

$comparison = foreach ($current in $run.Results) {
 $original = @($baseline.Results | Where-Object { $_.Id -ceq $current.Id })

 if ($original.Count -ne 1) {
 [pscustomobject]@{
 Test = $current.Id
 Result = 'Missing or duplicate baseline ID'
 }
 continue
 }

 if ($original[0].Prompt -cne $current.Prompt) {
 [pscustomobject]@{
 Test = $current.Id
 Result = 'Prompt changed; not directly comparable'
 }
 continue
 }

 $match = [string]::Equals(
 [string]$original[0].Response,
 [string]$current.Response
 )

 [pscustomobject]@{
 Test = $current.Id
 Result = if ($match) { 'Exact match' } else { 'Different - review' }
 }
}

$comparison | Format-Table -AutoSize
Write-Host "Current run saved: $runPath"
Write-Host "Baseline: $baselinePath"

The script deliberately walks the API’s output items rather than assuming the answer is always in the first item. OpenAI’s documentation warns that the output array can contain multiple types of items. Microsoft documents that Invoke-RestMethod converts JSON responses into PowerShell objects.

2. Set your API key for this session​

In the same PowerShell window, run:

Code:
$secureKey = Read-Host 'OpenAI API key' -AsSecureString
$pointer = [Runtime.InteropServices.Marshal]::SecureStringToBSTR($secureKey)

try {
 $env:OPENAI_API_KEY = [Runtime.InteropServices.Marshal]::PtrToStringBSTR($pointer)
}
finally {

 $secureKey.Dispose
}

Paste the key at the prompt. It will not be displayed as you type.

The environment variable is available to the current PowerShell process and can be inherited by child processes; it is not a substitute for an enterprise secrets manager. Do not put the key in the script, a test prompt, or a shared transcript. Microsoft notes that session-level environment-variable changes apply to the current session.

3. Capture and review a baseline​

Run:

.\Test-AIOutput.ps1 -Mode Baseline

A successful run reports the path to baseline.json. Inspect it:

Code:
Get-Content.\baseline.json -Raw | ConvertFrom-Json |
 Select-Object -ExpandProperty Results |
 Format-List Id, Prompt, Response, ReturnedModel

Check the answers before treating them as a reference. For example, the arithmetic answer should be 42, and the password-reset message should be classified as routine. A saved answer is not automatically a correct answer.

The script refuses to overwrite an existing baseline. If you intentionally want a new one, archive baseline.json first—and record why you replaced it. Otherwise, you risk making an unwanted change look normal.

4. Rerun and compare​

With the same script, prompts, model, and settings, run:

.\Test-AIOutput.ps1 -Mode Compare

The table will show one of these outcomes for each test:

  • Exact match: The new text is identical to the saved baseline text.
  • Different - review: At least one character differs.
  • Prompt changed; not directly comparable: The input no longer matches the baseline.
  • Missing or duplicate baseline ID: The saved baseline needs investigation.

The new responses are saved in a separate timestamped run-...json file. Open that file and baseline.json to examine a flagged result side by side. Whitespace, capitalization, or punctuation alone can trigger a difference, so a flag is an invitation to investigate—not a verdict.

The script records the returned model for each response. If you change the requested model or output-token limit, it warns that you are comparing different configurations. Keep the original baseline when testing a proposed model migration; the differences may be exactly what you want to evaluate.

Troubleshooting and interpretation​

  • Script execution is blocked: Check your organization’s PowerShell execution policy and approved procedures. Do not bypass a centrally managed restriction without authorization.
  • Authentication or access error: Check that the key belongs to the intended API project and that the project can access the selected model. Set the key again if you opened a new PowerShell window.
  • Rate-limit or quota error: Check the API account’s limits and billing status, then retry after resolving the cause. A failed run is not evidence of drift.
  • Incomplete or empty response: The script stops rather than recording it as a valid answer. The output-token limit may need adjustment. OpenAI notes that an output limit can produce an incomplete response; if you change the limit, document the configuration change before comparing runs.
  • Local files expose sensitive information: Restrict folder access and apply your organization’s retention rules. The script requests store = false, but that is not a blanket promise of zero provider-side retention; review OpenAI’s data controls and your account configuration.

To remove the session variable when finished:

Remove-Item Env:OPENAI_API_KEY

What a difference can—and cannot—tell you​

Repeated changes across representative tests may justify a closer evaluation. Run the same suite multiple times, compare the rate and kind of differences, and review whether answers remain correct and useful. For production monitoring, add task-specific checks: valid JSON, required fields, classification accuracy, or human-reviewed expected answers.

Also test the application around the model. A changed system instruction, retrieved document, tool result, or user-input pattern can alter the final answer even if the model snapshot stays the same. The sample script does not reproduce those components.

Finally, identical responses to three prompts do not establish that an AI system is safe, secure, unbiased, or reliable on other inputs. Baseline testing gives administrators a useful early-warning instrument; it does not remove the need for broader evaluations, access controls, human oversight, and incident response.


Correction (November 21, 2026): The published script had two syntax errors that stopped it from running at all: a bare @ where an empty array @() was intended, and a trailing comma in the string-equality call. Both are now fixed, and the script parses cleanly in PowerShell.

 

Last edited: