PadChest-GR: Grounded, Bilingual Dataset Revolutionizing AI in Radiology

The modern intersection of artificial intelligence and radiology is experiencing a profound shift, with transformative advancements not only in algorithmic prowess but in the very data that underpin model development and clinical translation. One of the most significant recent innovations comes in the form of a unique benchmark dataset: PadChest-GR, a bilingual, multimodal, sentence-level radiology report corpus co-developed by the University of Alicante, Microsoft Research, University Hospital Sant Joan d’Alacant, and MedBravo. This collaboration signifies a new era in machine learning for healthcare—one where the richness, structure, and interoperability of datasets open doors to interpretability, reliability, and, ultimately, better patient outcomes.

The Evolving Landscape of AI in Medical Imaging​

Radiology has long stood at the confluence of technical innovation and clinical practice. With over half of all hospital patients requiring some form of radiological imaging—from chest X-rays to MRI—there exists immense demand for both timely and accurate interpretation. Traditionally, radiology reporting has relied on free-text, unstructured narratives, often condensing several findings into complex, sometimes ambiguous summaries. These conventional reports challenge not only clinical clarity but also the comprehensibility and robustness of artificial intelligence models tasked with assisting medical professionals.
In this context, “grounded radiology reporting” represents a methodological leap forward. Here, each individual radiological finding is described and localized independently, creating a granular, interpretable trail between visual data, text, and clinical meaning. This approach not only mitigates the risk of erroneous or “hallucinated” AI outputs but provides a pathway for interactive decision-support tools that can both inform and be audited by clinicians.

PadChest-GR: Origins and Purpose​

Recognizing a critical gap in existing resources, the PadChest-GR dataset was conceived and engineered to serve as the world’s first public bilingual benchmark for grounded chest X-ray report generation. The impetus for this initiative came from the pioneering work on the original PadChest dataset, published in 2020 by a team led by Dr. Aurelia Bustos at MedBravo and Dr. Antonio Pertusa at the University of Alicante, in partnership with other Spanish clinical centers. Impressed by its scale and diversity, Microsoft Research, in collaboration with these original architects, sought to elevate the dataset into a resource purpose-built for the next chapter of radiology AI.
PadChest-GR encompasses 4,555 meticulously annotated chest X-ray studies. Each study contains:

• Sentence-level descriptions of clinical findings in both Spanish and English
• Precise spatial annotations via bounding boxes for both positive and negative findings, allowing exact localization on imagery
• Translation and standardization to harmonize diverse reporting styles into a unified structure

This bilingual approach is critical, expanding dataset utility far beyond English-centric research and facilitating development and validation of AI models in multilingual and multicultural healthcare environments.

The Data Annotation Pipeline: Combining AI and Human Expertise​

The curation of PadChest-GR stands as a testament to the symbiosis between advanced machine learning and careful expert oversight. The annotation process consisted of several major phases:

1. Automated Data Extraction and Processing​

Utilizing Microsoft’s Azure OpenAI Service and the latest generation of large language models (LLMs) such as GPT-4, the team programmatically extracted sentences identifying individual radiological findings. This was no simple translation task: the raw reports often included both positive and negative findings, and the models were tasked with precisely distinguishing and mapping these, linking them directly to the expert-validated PadChest ontology.
The extracted texts were then translated from Spanish into high-fidelity English, a non-trivial step given the nuanced clinical vocabulary and the necessity of sentence-level, context-aware translation.

2. Manual Quality Control and Deep Annotation​

While AI automation streamlined initial data extraction, manual scrutiny was indispensable. Overseeing this process was the radiology team at University Hospital Sant Joan d'Alacant, coordinated via the HIPAA-compliant Centaur Labs platform. Each finding was reviewed for accuracy, and spatial attributes were annotated via bounding boxes—defining not only what was seen, but exactly where it appeared on the chest X-rays.
Consistency and inter-rater reliability were emphasized through a rigorous protocol, ensuring that both normal (negative) and abnormal (positive) findings were represented with equal care in the dataset.

3. Standardization and Integration​

The final stage comprised harmonizing annotations and translations into coherent, “grounded” reports. Unlike prior datasets that leave interpretation to subsequent users, PadChest-GR packages reports in a consistent, richly structured format, explicitly linking sentences to locations and clinical meanings. This structure makes it uniquely valuable for both training and evaluative benchmarks in language-image AI research.

Why Grounded and Bilingual Data Matter​

Several challenges have historically limited the impact and safety of AI in radiology:

• Hallucination Risk: Generative models can, when prompted, fabricate findings not supported by imagery—a critical risk in clinical settings.
• Ambiguity in Free-Text: Unstructured narratives can obscure clinically actionable information, impede natural-language processing, and limit reproducibility.
• Translation Gaps: The vast majority of publicly available radiology datasets exist only in English, overlooking much of the world’s clinical practice and population.
• Lack of Precise Localization: Sentence-only or weakly labeled datasets prevent rigorous validation and limit the transparency of AI “reasoning.”

PadChest-GR addresses each of these gaps, setting a new bar for dataset quality and utility in grounded radiology reporting.

Empowering State-of-the-Art AI Models with PadChest-GR​

One direct beneficiary of the new dataset is Microsoft’s MAIRA-2, a cutting-edge multimodal report generation model. By training and validating on PadChest-GR, MAIRA-2 demonstrates several emergent capabilities:

• Highly interpretable outputs: Each generated report sentence is anchored to a specific region on the X-ray, increasing clinician trust and facilitating error checking.
• Multilingual performance: The bilingual dataset enables evaluation and deployment in diverse healthcare systems.
• Reduced fabrication risk: By constraining generation to grounded findings, the model minimizes unsupported or spurious claims.

Internal user research with clinicians, as reported on Microsoft’s Azure AI Foundry, has indicated positive feedback regarding both the clinical relevance and the usability of MAIRA-2 outputs—a promising step for the responsible, real-world integration of such technologies.

Collaboration at Scale: Universities, Hospitals, and Industry​

A recurring theme in the development of PadChest-GR is interdisciplinary collaboration. While Microsoft brought machine learning and infrastructure strength, the University of Alicante and MedBravo contributed deep clinical insight, robust data stewardship, and access to real-world patient populations. The role of Hospital Sant Joan d’Alacant’s radiologists, particularly under the coordination of Joaquin Galant, underscored the criticality of multi-level expert review.
The importance of such partnerships cannot be overstated. Healthcare data projects, especially those touching clinical care, demand both technical innovation and unwavering ethical rigor. PadChest-GR’s high annotation quality reflects the value of combining institutional strengths for maximal real-world impact.

Evaluating PadChest-GR: Notable Strengths​

PadChest-GR brings several exceptional features to the research and clinical AI community:

Unprecedented Granularity​

Unlike other public datasets, PadChest-GR pairs every reportable finding—no matter how subtle or complex—with both a textual description and a precise bounding box. This enables:

• Fine-grained error analysis of AI models, helping uncover not only “what” went wrong but “where” and “how.”
• Richer training data for models aiming to achieve “grounded” generation, supporting research in explainable AI.

Bilingual and Multicultural Breadth​

PadChest-GR is one of the few datasets that support both Spanish and English at a granular sentence level. This is a major asset for:

• Democratizing access to state-of-the-art AI tools across non-English-speaking health systems
• Supporting global research collaborations and multi-center trials
• Enabling natural translation, transfer learning, and evaluation of language-agnostic or multilingual models

Dual Positive and Negative Labeling​

By including explicit annotations for normal (negative) findings—a rarity in the public data landscape—PadChest-GR supports balanced evaluation and guards against “positive finding bias” common in many AI benchmarks.

Open Access and Community Engagement​

Microsoft and its partners have made PadChest-GR openly available via the BIMCV PadChest-GR Project, with comprehensive documentation and benchmarks. This fosters transparency, encourages reuse, and sets the stage for broader, more rigorous benchmarks in radiology AI.

Potential Risks and Areas for Caution​

While PadChest-GR represents a major step forward, it is also important to critically assess limitations and potential pitfalls.

Dataset Size and Diversity​

With 4,555 annotated studies, PadChest-GR is among the larger public, fully grounded radiology datasets. However, this is still modest compared to the scale of data typically held within single large hospitals or international consortia. Researchers should be aware of:

• Sampling bias: The dataset originates largely from one health system in Spain, possibly limiting generalizability to other patient populations, imaging devices, and clinical practices.
• Disease prevalence: Some rare pathologies or presentation types may be underrepresented, impacting model robustness for edge cases.

Annotation Limitations​

Though the annotation process was rigorous, all manual labeling efforts face the perennial challenges of:

• Inter-rater variability: Even expert radiologists may slightly differ in interpretations, affecting bounding box precision and report content.
• Evolving ontologies: Medical knowledge constantly advances; future ontological standards might require updates or reinterpretation of labels.

Model Overfitting​

By providing highly detailed sentence-to-location pairings, PadChest-GR risks enabling overfitting to dataset-specific patterns—models may inadvertently “memorize” dataset quirks rather than generalize. This risk can be mitigated through multi-center validation and further dataset expansion.

Risks in Clinical Deployment​

Grounded datasets lower the risk of hallucination, but even state-of-the-art models like MAIRA-2 are not immune to failure. Overreliance on AI-generated or AI-interpreted reports—absent continuous expert review—poses real dangers, particularly in high-stakes clinical decisions.

The Road Ahead: Open Collaboration and Responsible Innovation​

PadChest-GR’s release signals a new trajectory for the global research community. Several papers have already cited or utilized the dataset, using it as the gold standard for benchmarking grounded report generation and explainable AI. The Microsoft-initiated models MAIRA-2 and RadFact, available in Azure AI Foundry, exemplify the next wave of clinical AI: interpretable, localizable, and multilingual.
Importantly, the PadChest-GR team encourages broad collaboration, inviting researchers worldwide to build upon this resource, submit new models for benchmark evaluation, and propose dataset extensions for other modalities or body regions.
For those seeking further details or interested in direct download and experimentation, more information is available via the BIMCV PadChest-GR Project. Technical documentation and frequent updates are maintained to ensure open, reproducible research.

Conclusion: A New Standard for Radiology AI​

The release of PadChest-GR is more than a technical milestone; it is a call to action for the global medical and AI communities. As research pivots toward interpretability, safety, and global relevance, high-fidelity, richly annotated, and multilingual datasets are indispensable.
PadChest-GR’s grounded, sentence-by-sentence, bilingual labeling, coupled with precise localization, offers an unprecedented foundation for developing trustworthy, transparent, and clinically meaningful AI models. Yet, as with all innovations, its full value will be realized only through community engagement, critical scrutiny, and continual iteration.
Researchers, developers, and clinicians are now afforded a powerful tool—one that bridges languages, disciplines, and the perennial gulf between technology and practical care. The future of radiology AI will, in no small part, be shaped by resources and collaborations of this caliber. As models grow ever more capable, and the datasets that shape them ever richer, the ultimate beneficiaries will be the practitioners on the front lines—and, most importantly, the patients for whom each finding, grounded and precise, can make all the difference.

---

Source: Microsoft New dataset built to help experts and AI interpret medical images more effectively