That also means you do not need to process a CSV just to obtain the published comparison. The reason to build a pipeline is more useful: reproducibility, respondent overlap, role-level analysis, and an audit trail explaining exactly what each percentage means.
Below is a 12-step, Windows-friendly workflow for doing that. Allow roughly 80 minutes as a planning estimate, not a benchmark. Downloading the data and checking the questionnaire may take longer than running the analysis.
First, understand the comparison
| Tool | Respondents reporting past-year use | Share of the question’s 12,255 respondents |
|---|---|---|
| GitHub Copilot | 7,196 | 58.7% |
| Cursor | 3,127 | 25.5% |
Stack Overflow separately measures interest in using these tools over the next six months. Keep that measure apart from past-year use; multiple tool selections also mean the percentages are not slices of one exclusive market-share pie.
The denominator problem is not hypothetical. In 2025, Stack Overflow’s out-of-the-box assistant question was explicitly addressed to respondents who had indicated they used or developed AI agents. Its accompanying summary put GitHub Copilot at 68%. That screened question is not interchangeable with an editor-use question asked of another population.
The working rule: compare products within the same question, time window, respondent filter, and denominator. A spreadsheet cannot rescue a comparison that started with different definitions.
Steps 1–3: Establish the question and environment
1. Record what you intend to measure
Create a project note containing:
- Survey edition and download filename.
- Exact question wording.
- Whether the question permits multiple selections.
- Any eligibility or routing conditions.
- Whether you are measuring past use or future interest.
- The response-validity filter.
- The denominator you will report.
For this exercise, select the past-year coding-agent or assistant question, not the future-interest field. The official 2026 page presents those as distinct measures.
Do not assume a blank means “does not use AI.” Leave it as unanswered unless the questionnaire or export documentation establishes something more specific.
2. Create an isolated Python environment
Use an installed, stable Python environment compatible with pandas and pytest. The commands below use Windows PowerShell and invoke the virtual environment’s Python directly, avoiding activation-script complications.
mkdir survey-share
cd survey-share
mkdir data
mkdir out
py -m venv.venv
.\.venv\Scripts\python.exe -m pip install pandas pytest
.\.venv\Scripts\python.exe -m pip freeze > requirements-lock.txt
.\.venv\Scripts\python.exe --version
Retain the Python version output alongside requirements-lock.txt. This records the environment you actually installed rather than relying on an unverified claim about the “latest” release.
3. Obtain the response file and verify its scope
Obtain the edition’s public response export through Stack Overflow’s survey resources, retaining its accompanying schema or questionnaire. Save the response CSV as:
data/survey.csv
Check whether it is already restricted to valid responses. Stack Overflow’s 2026 methodology distinguishes 39,266 recorded rows from 30,903 valid responses, collected between June 23 and August 5, 2026. It says section exports use valid_response == 1. Counting every row of an unfiltered extract can therefore produce a different result from a published chart.
The script below applies that filter when a valid_response column is present. If it is absent, you must establish from the export documentation whether filtering has already occurred.
Steps 4–6: Build a deliberately narrow pipeline
4. Preserve blanks and discover candidate columns
Pandas documents that specifying a text dtype does not, by itself, disable missing-value interpretation. The implementation below explicitly preserves text tokens while treating empty CSV fields as missing; its tokenizer also treats whitespace-only cells as unanswered. Check any additional missing-value codes against the export schema before proceeding.
Discovery is only an aid. Finding “Cursor” in a column does not establish whether that column measures use, interest, or free-text commentary.
5. Use one question per run
Do not combine separate tool questions with a logical OR merely because both mention Copilot.
For this workflow:
- The numerator is respondents selecting the exact tool name.
- The denominator is respondents with a nonempty answer to the selected question.
- An explicit answer such as “None of the above” remains an answer.
- Missing answers are excluded from the answered-question denominator.
Microsoft Copilotdoes not matchGitHub Copilot.
This single-column design makes an invalid cross-question union harder to create accidentally.
6. Save the analysis script
Save the following as analyse.py. It produces counts, model-based Wilson intervals, overlap buckets, optional segments, nonresponse bounds, and a source-file checksum.
The CSV-loading choices follow pandas’ documented read_csv controls. The statistical intervals require the limitations discussed in Step 10; they are not population-wide margins of error.
import argparse
import hashlib
import json
import math
from pathlib import Path
import pandas as pd
TOOLS = {
"Cursor": "cursor",
"GitHub Copilot": "github copilot",
}
def tokens(value):
if pd.isna(value):
return set()
return {
part.strip().casefold()
for part in str(value).split(";")
if part.strip()
}
def wilson(k, n):
if n == 0:
return None, None
z = 1.959963984540054
p = k / n
denominator = 1 + z * z / n
centre = (p + z * z / (2 * n)) / denominator
half = z * math.sqrt(
p * (1 - p) / n + z * z / (4 * n * n)
) / denominator
return max(0.0, centre - half), min(1.0, centre + half)
def flags_for(series):
parsed = series.map(tokens)
result = pd.DataFrame({
name: parsed.map(lambda values: token in values)
for name, token in TOOLS.items()
})
result["answered"] = parsed.map(bool)
return result
def summarize(flags):
n = int(flags["answered"].sum())
total = len(flags)
rows = []
for tool in TOOLS:
k = int(flags.loc[flags["answered"], tool].sum())
low, high = wilson(k, n)
rows.append({
"tool": tool,
"users": k,
"answered": n,
"eligible_rows": total,
"share_answered": k / n if n else None,
"wilson_low": low,
"wilson_high": high,
"recorded_selection_fraction": k / total if total else None,
"nonresponse_upper": (
(k + total - n) / total if total else None
),
})
return rows
def overlap(flags):
answered = flags.loc[flags["answered"]]
cursor = answered["Cursor"]
copilot = answered["GitHub Copilot"]
groups = {
"both": cursor & copilot,
"Cursor only": cursor & ~copilot,
"GitHub Copilot only": ~cursor & copilot,
"neither": ~cursor & ~copilot,
}
return {name: int(mask.sum()) for name, mask in groups.items()}
def main():
parser = argparse.ArgumentParser()
parser.add_argument("csv")
parser.add_argument("--discover", action="store_true")
parser.add_argument("--col")
parser.add_argument("--segment")
parser.add_argument("--min-n", type=int, default=100)
parser.add_argument("--out", default="out")
args = parser.parse_args()
if args.min_n < 1:
parser.error("--min-n must be positive")
path = Path(args.csv)
digest = hashlib.sha256(path.read_bytes()).hexdigest()
df = pd.read_csv(
path,
dtype="string",
encoding="utf-8-sig",
keep_default_na=False,
na_values=[""],
)
df.columns = df.columns.str.strip()
if df.columns.duplicated().any():
parser.error("Duplicate headers after trimming whitespace")
original_rows = len(df)
validity_rule = "No automatic filter; confirm export eligibility"
if "valid_response" in df.columns:
valid = df["valid_response"].str.strip()
unexpected = set(valid.dropna().unique()) - {"0", "1"}
if unexpected:
parser.error("Unrecognized valid_response encoding")
df = df.loc[valid.eq("1").fillna(False)].copy()
validity_rule = "valid_response == 1"
if args.discover:
for column in df.columns:
parsed = df[column].map(tokens)
seen = set().union(*parsed.tolist()) if len(parsed) else set()
matches = sorted(
token for token in seen
if "cursor" in token or "copilot" in token
)
if matches:
print(column)
print(" answered:", int(parsed.map(bool).sum()))
print(" tokens:", " | ".join(matches))
return
if not args.col or args.col not in df.columns:
parser.error("Provide --col using an exact discovered header")
if args.segment and args.segment not in df.columns:
parser.error("Segment column not found")
flags = flags_for(df[args.col])
if not flags["answered"].any():
parser.error("No nonempty answers in selected column")
segments = []
if args.segment:
labels = df[args.segment].fillna("").str.strip()
for value in sorted(set(labels) - {""}):
sub = flags.loc[labels.eq(value)]
if int(sub["answered"].sum()) >= args.min_n:
for row in summarize(sub):
segments.append({"segment": value, **row})
result = {
"source_file": path.name,
"sha256": digest,
"pandas_version": pd.__version__,
"original_rows": original_rows,
"eligible_rows": len(df),
"validity_rule": validity_rule,
"question_column": args.col,
"segment_column": args.segment,
"missing_segment_rows": (
int(labels.eq("").sum()) if args.segment else None
),
"shares": summarize(flags),
"overlap_counts": overlap(flags),
"segments": segments,
}
out = Path(args.out)
out.mkdir(parents=True, exist_ok=True)
(out / "report.json").write_text(
json.dumps(result, indent=2, allow_nan=False) + "\n",
encoding="utf-8",
)
pd.DataFrame(result["shares"]).to_csv(
out / "shares.csv", index=False
)
print(json.dumps(result, indent=2, allow_nan=False))
if __name__ == "__main__":
main()
Boundary: this parser expects semicolon-separated, named choices in one CSV column. If the export instead contains numeric codes, separate checkbox columns, or another delimiter, stop and adapt it using the schema. Guessing is not data cleaning.
Steps 7–9: Run, reconcile and segment
7. Discover the field, then calculate shares
First run:
.\.venv\Scripts\python.exe analyse.py data/survey.csv --discover
Review the matching tokens and consult the questionnaire. Then substitute the verified past-use header:
.\.venv\Scripts\python.exe analyse.py data/survey.csv --col "EXACT PAST-USE HEADER"
Success means:
out/report.jsoncontains the filename, checksum and selected column.out/shares.csvcontains each tool’s numerator and denominator.- The selected question and filters reconcile with the official table.
For the corresponding 2026 published comparison, the reconciliation targets are 12,255 answers, 7,196 GitHub Copilot selections and 3,127 Cursor selections. If your output differs, investigate the export, question, routing and filters before quoting it.
The script writes proportions: 0.255 means 25.5%, not 0.255%.
8. Read the overlap correctly
The report divides answered respondents into four exclusive buckets:
- Both products.
- Cursor only.
- GitHub Copilot only.
- Neither product.
Those counts should sum to the answered-question denominator. “Neither” means neither of these two products; it does not necessarily mean no coding assistant.
An overlap result establishes reported use of both tools within the question’s time window. It does not establish simultaneous use, preference, employer payment, or whether the tools are complements rather than substitutes.
9. Add a documented segment
After checking an appropriate role or experience field, run:
.\.venv\Scripts\python.exe analyse.py data/survey.csv --col "EXACT PAST-USE HEADER" --segment "EXACT SEGMENT HEADER" --min-n 100
The report suppresses groups with fewer than 100 answered tool-question responses. That is an editorial guardrail, not a certificate of reliability.
Use a single-category field where possible. If a role column itself permits multiple selections, this script groups the complete answer combination—it does not independently count every role token. Treating that output as individual-role adoption would mislabel the result.
Steps 10–11: Handle uncertainty without overselling it
10. Separate descriptive results from inference
Stack Overflow recruited through its websites and opted-in community email channels. Consequently, these results should be described as respondent findings, not a random sample of every developer.
The script’s Wilson intervals describe binomial uncertainty under a model. They do not repair self-selection, inaccurate recall, or question nonresponse. AAPOR cautions against presenting conventional sampling-error measures as though opt-in samples had known selection probabilities.
There is another important distinction: Cursor and Copilot answers from the same people are paired observations. An independent two-proportion test is therefore not the right default for comparing those two selections. McNemar’s test is designed for paired nominal data; its documentation describes using the discordant cells of a two-by-two table.
For a straightforward public report, publishing counts, denominators and overlap is preferable to adding a poorly justified p-value.
11. Use bounds before inventing weights
The script reports two useful sensitivity quantities:
recorded_selection_fraction: observed tool selections divided by all eligible rows.nonresponse_upper: the fraction if every unanswered eligible row also selected that tool.
These are arithmetic extremes within the eligible file—not estimates of the global developer population.
Post-stratification is a separate, optional exercise. Only introduce weights when you have a defensible target population and documented target proportions. Present weighted estimates beside unweighted results, explain the assumptions, and do not attach the script’s unweighted Wilson intervals to weighted percentages. AAPOR’s nonprobability-sampling guidance emphasizes that inference from such samples requires explicit assumptions; weighting is not an automatic cure.
Step 12: Test the counting rules and preserve the report
Save this as test_analyse.py:
import pandas as pd
from analyse import flags_for, overlap, summarize, wilson
def fixture():
return pd.Series([
"Cursor;GitHub Copilot",
"GitHub Copilot",
"Microsoft Copilot",
None,
"Cursor",
" ",
], dtype="string")
def test_exact_names_and_denominator():
rows = {
row["tool"]: row
for row in summarize(flags_for(fixture()))
}
assert rows["Cursor"]["users"] == 2
assert rows["GitHub Copilot"]["users"] == 2
assert rows["Cursor"]["answered"] == 4
assert rows["Cursor"]["share_answered"] == 0.5
def test_overlap():
assert overlap(flags_for(fixture())) == {
"both": 1,
"Cursor only": 1,
"GitHub Copilot only": 1,
"neither": 1,
}
def test_empty_question():
rows = summarize(
flags_for(pd.Series([None, ""], dtype="string"))
)
assert all(row["share_answered"] is None for row in rows)
def test_wilson():
low, high = wilson(50, 100)
assert 0.40 < low < 0.41
assert 0.59 < high < 0.60
assert wilson(0, 0) == (None, None)
Run:
.\.venv\Scripts\python.exe -m pytest -q
The expected outcome is four passing tests. These fixtures check the implementation’s counting rules; they do not prove that you chose the correct survey question.
Preserve the script, tests, dependency lock, questionnaire notes and generated report together. When publishing derived data, retain Stack Overflow attribution and review its stated ODbL 1.0 data and DbCL 1.0 contents licensing terms.
Troubleshooting the common failures
| Symptom | Check or fix |
|---|---|
py is not recognized | Use the command for your installed Python interpreter to create the environment. |
| Discovery finds nothing | Confirm you downloaded responses rather than the schema; inspect whether choices are coded or split across columns. |
| Column-not-found error | Copy the complete header from discovery output, including spaces. |
| Copilot count seems inflated | Inspect tokens; Microsoft Copilot must remain separate from GitHub Copilot. |
| Published percentages do not match | Check edition, past-use versus future-interest fields, validity filters and question routing. |
| Segment output is empty | Check the field and whether any group reaches --min-n. |
| Unicode decoding fails | Verify the export’s documented encoding; do not silently discard undecodable characters. |
| Tests pass but results look wrong | Tests protect known counting behavior, not questionnaire interpretation. |
What this tells a development team
The strongest conclusion is also the narrowest: the official 2026 question records a higher past-year-use share for GitHub Copilot than Cursor among its respondents. It does not rank code quality, security, value for money or productivity.
For a purchasing decision, use this survey comparison as context. Pair it with your organization’s actual usage records, governance requirements and a controlled evaluation on representative repositories. Keep those measures separate rather than blending them into a mysterious “winner” score.
The useful outcome of an 80-minute analysis is not a louder percentage. It is a percentage whose question, denominator, limitations and provenance remain attached—even after somebody pastes it into a slide.
References
- Cursor vs Copilot Survey Data: 12 Steps, 80 Min (2026) shattered.io · 2026-11-16T03:57:46+00:00
- AI data 2026 | Stack Overflow Developer Survey survey.stackoverflow.co
- AI 2026 | Stack Overflow Developer Survey survey.stackoverflow.co