CVE-2025-22010: Linux RDMA HNS soft lockup fix with cond_resched

A subtle but consequential Linux-kernel fix landed upstream this spring: CVE-2025-22010 closes a soft‑lockup hazard in the RDMA hns driver that could let a large memory‑region registration (MR) stall CPU cores for tens of seconds, producing real-world denial‑of‑service symptoms on RDMA‑enabled hosts.

Background / Overview​

The vulnerability is tracked as CVE‑2025‑22010 and was disclosed publicly on April 8, 2025. In short, the hns RDMA driver’s bt‑page allocation loop could iterate long enough while mapping very large buffer regions (examples reported around and above 100 GiB) that a CPU watchdog would flag a soft lockup — the kernel’s way of signalling a CPU has been busy in kernel mode too long without yielding. The upstream remedy inserted voluntary scheduling points (cond_resched()) inside the allocation loop so the scheduler can preempt the task and avoid sustained CPU starvation. This fix is documented in the public vulnerability records and has been picked up in distribution patch sets.
Why this matters: RDMA drivers operate in high‑performance stacks (HPC, distributed databases, AI training clusters) where very large memory registrations and low‑latency transfers are common. A kernel thread looping without yielding on behalf of a userland RDMA registration can turn into a broad availability problem — particularly in multi‑tenant systems or when untrusted workloads can issue large MR requests. Multiple distro advisories and vulnerability databases classify the flaw as a medium severity availability issue with a CVSS v3.1 base score of 5.5.

---

Technical deep dive​

What went wrong: the bt pages loop​

At the heart of the issue is a long-running for‑loop in the hns RoCE driver where the driver:

• computes the number of bt pages required to back a requested MR,
• iterates to allocate and map each bt page (a granularity used by this hardware’s scatter/gather translation tables),
• performs the mapping work synchronously in the caller’s context.

When the MR size is enormous, the loop’s iteration count skyrockets. Without an internal yield point the kernel watchdog detects a CPU for which the scheduler cannot regain control and reports a soft lockup (e.g., "watchdog: BUG: soft lockup - CPU#27 stuck for 22s!"). The trace logged by the watchdog illustrates the call chain through hns_roce_mtr_create(), alloc_mr_pbl(), hns_roce_reg_user_mr() and the helper routines that allocate the BT pages. Several independent advisories reproduce that call trace in their writeups.

The upstream fix: voluntary yields​

The upstream patch inserted conditional scheduler yields (cond_resched()) at a measured frequency inside the loop. The intent is narrow and surgical:

• avoid introducing a heavy performance penalty for normal MR sizes,
• but ensure that extreme allocation loops do not prevent the scheduler from running,
• therefore eliminate prolonged soft lockups while preserving throughput for typical workloads.

Multiple distribution advisories and stable‑kernel backports reference these commits and list the kernel versions that include the fix (notably in kernel stable trees and backported releases used by mainstream distros). The Debian kernel changelog and other vendor trackers list the hns items among 6.12.21 and similar stable updates.

Why cond_resched() is the pragmatic choice​

In kernelland, cond_resched() is a well‑understood mechanism to let long‑running kernel paths yield when it’s safe. It is preferable to coarse-grained work‑offloading unless the operation is latency‑sensitive and must be completed in atomic context. Here, the allocation and mapping of page tables and the attendant bookkeeping are safe to yield around, so adding a few reschedule checks is conservative and effective.
That said, where and how often to call cond_resched() is a tradeoff: too infrequent and the soft lockup may still occur; too frequent and you cripple throughput for legitimate, high‑performance workloads. The upstream approach uses a threshold and periodic check, designed to only engage for very large MRs. Multiple vulnerability trackers summarize this behavior.

---

Who is affected​

• Systems using the hns_roce_hw_v2 driver (the Hisilicon/Huawei RDMA RoCE driver) are directly implicated.
• Affected operating systems include Linux kernels that carry the vulnerable driver code; vendors have published advisories and backports across stable kernels and distribution kernels. Several vendor trackers (Ubuntu, Debian, Oracle Linux, Amazon Linux) list the CVE and the kernel packages carrying the fix.
• Multi‑tenant cloud nodes, HPC clusters, and virtualized/containarized hosts that provide RDMA capabilities to untrusted workloads are higher risk because an untrusted party could intentionally trigger large MR registrations.

It is important to note: this is not a remote arbitrary code execution or information‑disclosure hole — it is primarily an availability risk (denial‑of‑service via soft lockup). That makes the operational response (patching / restricting RDMA access) straightforward but urgent for affected environments.

---

Patch verification and cross‑checking​

I validated the key technical claim (cond_resched() inserted in bt allocation loops) across multiple, independent sources:

• The NVD entry for CVE‑2025‑22010 documents the exact call traces and summarizes the fix as "Add a cond_resched() to fix soft lockup during these loops."
• Vendor/distros: Ubuntu, Debian and Oracle Linux advisories list the CVE and reference the hns fixes and stable‑kernel backports. Debian’s kernel changelog that shipped 6.12.21 lists multiple RDMA/hns fixes including "Fix soft lockup during bt pages loop."
• Open vulnerability trackers (OSV, Debian security tracker, Tenable) independently index the CVE and carry the same technical summary and impact assessment. These independent entries converge on the same root cause and the same remediation (patch in upstream plus distro packages/backports).

A set of stable‑tree commit hashes and patch IDs is referenced in vendor and national advisories; where possible, administrators should inspect the kernel tree or vendor patch set that ships with their distribution to confirm the presence of the cond_resched() addition before declaring a host remediated. If you maintain your own kernels, apply the upstream commit or follow vendor backports. The Japanese national vulnerability DB (JVN) collected the upstream commit references for auditing purposes.
Caveat: a direct file‑level fetch of git.kernel.org commits may be blocked by tooling or network constraints for some readers; when that occurs, rely on the distribution changelog entries and vendor advisories that explicitly map the CVE to shipped kernel package versions. Multiple distros have shipped packages that incorporate the upstream correction, so package updates are the recommended route.

---

Operational risk: what administrators must consider​

While the code change is small, the operational implications are important:

• Availability impact: The vulnerability is an availability concern — repeated triggering can cause persistent disruption (sustained while the attacker runs the workload) and can be exploited by untrusted tenants to create noisy‑neighbor conditions in shared environments. The vendor documentation and NVD classify the availability impact accordingly.
• Attack surface: Access vectors depend on local privileges — a local user or container that can invoke RDMA verbs and request very large MRs can trigger the condition. In multi‑tenant clouds that expose RDMA to guest tenants (or to containerized workloads), this operation may be reachable from untrusted code unless proper isolation controls exist. Several advisories flag exactly this tenancy risk.
• Performance tradeoffs: The inserted cond_resched() calls are intentional to be infrequent for normal-sized MRs. However, operators should test performance‑sensitive RDMA workloads after the update to ensure there is no unexpected regression, particularly in latency‑sensitive paths where every microsecond matters.

---

Detection and indicators​

There are practical indicators to detect attempts that trigger the bug or the bug itself:

• Kernel logs: look for repeated watchdog soft‑lockup messages mentioning hns_roce functions, e.g., call traces containing hem_list_alloc_mid_bt, hns_roce_mtr_create, alloc_mr_pbl, or hns_roce_reg_user_mr. Those traces were recorded in the initial reports and are a reliable sign.
• Elevated kernel warnings / perf anomalies: spikes in CPU stuck reports, elevated latency or task scheduling anomalies during RDMA registration events.
• Audit RDMA usage: instrument and log IB/RDMA verbs (ib_uverbs) activity and large MR creation from user space. If you see repeated large MR registrations from an untrusted process, treat it as suspicious.

Suggested first response steps:

• If you observe the soft‑lockup trace, isolate the host from production traffic and verify kernel package versions.
• If you cannot immediately patch, restrict RDMA access to trusted workloads (see mitigations below).
• Collect kernel logs and a reproduction trace (if safe) for later forensic analysis and vendor support.

---

Mitigations and recommended remediation steps​

Short term (while you patch):

• Restrict RDMA privileges: Only allow RDMA verb operations to trusted users/containers. Remove RDMA device access from untrusted namespaces. This reduces the local attack surface. Several advisories explicitly recommend controlling access to RDMA resources pending patching.
• Throttle or cap MR sizes: If your stack allows user-level limits on MR sizes (via configuration or cgroup‑like controls), enforce conservative limits for tenants that do not require huge MRs.
• Monitor logs aggressively: watch for the specific soft lockup call traces and for abnormally frequent MR creations.

Medium/long term:

• Apply vendor kernel updates: Install distributor patches that include the hns fix. Distribution trackers list which kernel versions include the changes (for example, Debian/Ubuntu and vendor kernel updates referenced this patch in the 6.12.21 series and vendor advisories).
• If you run custom kernels, apply the upstream hns patch (or backport it) and rebuild. Verify the insertion of cond_resched() or equivalent scheduling yield. Consult vendor documentation or commit messages to ensure you’ve applied the correct changes.
• Test performance: after applying fixes, validate RDMA workloads (throughput, latency, CPU usage) in a staging environment to exclude regressions in noise‑sensitive workloads.

---

Vendor and distribution response (summary)​

• Debian and Ubuntu have tracked the CVE in their security trackers and applied the upstream backports into their stable kernel packages. Check your vendor advisory for the exact kernel package version mapping.
• Major vulnerability databases (NVD, OSV, Tenable, Amazon Linux advisory) have recorded the CVE and its technical summary; many of these databases list the fix as adding cond_resched() in the allocation loop and provide the CVSS scoring and mitigation guidance. Use vendor packages (not database copy) as the authoritative fix source.
• National and vendor trackers (e.g., JVN) captured upstream commit references for auditing, useful for teams that build their own kernels.

---

Critical analysis: strengths of the fix and residual risks​

Strengths

• The upstream remedy is small, targeted, and low‑risk: inserting voluntary scheduling points in a long loop is a conservative, well‑understood mitigation for soft‑lockup conditions that does not require a redesign of the RDMA memory registration code. Multiple vendors accepted the fixes and backported them into stable kernel releases quickly, which is evidence the change is operationally safe.
• The fix addresses the root operational pain — CPU starvation — directly and keeps the rest of the driver logic intact, preserving performance for normal workloads.

Residual and follow‑on risks

• Access controls remain the primary operational control: if RDMA resources remain exposed to untrusted tenants, an attacker can still cause high load or create other RNIC exhaustion vectors. The cond_resched() change prevents soft lockups, but it does not introduce per-tenant RNIC resource accounting or hardware quotas. For multi‑tenant cloud providers, that means deeper isolation and throttling strategies are still necessary. Recent research into RDMA resource exhaustion (noisy‑neighbor attacks) underscores this ongoing risk.
• Backport fragmentation: not every vendor releases patches on the same cadence. Some older or vendor‑modified kernel branches may lag behind upstream; operators who do not or cannot easily upgrade kernels must backport the patch themselves or rely on vendor backports. The distribution changelogs help, but patch mapping requires careful package/version verification.
• Performance corner cases: the inserted yields are deliberate and minimal, but any scheduling change in a performance‑critical hot path requires careful validation on production workloads. Organizations that depend on micro‑latency RDMA must validate the update in a staging environment.

---

Practical checklist for sysadmins (quick actionable items)​

• Inventory: locate hosts with the hns driver loaded (check lsmod / dmesg for hns_roce entries) and identify kernel package versions.
• Patch: install vendor kernel updates that include the hns fix; where vendor updates are unavailable, plan a controlled kernel rebuild with the upstream patch backported.
• Restrict: enforce RDMA access control — limit device access to trusted VMs/containers and enforce MR size quotas where possible.
• Monitor: watch kernel logs for soft‑lockup traces and audit ibverbs activity for unusually large MR registrations.
• Test: run representative RDMA workloads after patching and confirm throughput/latency meet SLAs.

---

Conclusion​

CVE‑2025‑22010 is a representative example of how availability bugs in kernel drivers — not just raw memory corruption or privilege‑escalation flaws — can cause serious operational impact. The fix is surgical: adding scheduler yields inside a chokepoint loop removes the soft‑lockup without prescriptively altering driver behavior for normal workloads. Multiple independent sources (vendor advisories, NVD, distro trackers) converge on the same diagnosis and remediation path — apply vendor patches, or backport and test the upstream change, and restrict RDMA operations from untrusted workloads until that work is complete.
For teams running RDMA in production, the CVE is a timely reminder to treat kernel driver updates as first‑class operational risks: test, stage, patch quickly, and pair kernel updates with access controls and monitoring so a single untrusted workload cannot take down critical services. Internal postmortems from similar RDMA and HNS fixes show this is an operational problem as much as a coding bug; mitigation requires both the patch and controls around who can request large memory registrations.

---

---

Source: MSRC Security Update Guide - Microsoft Security Response Center