Security Disclosure: Addressing Tenant Isolation Vulnerability in Workers AI

On September 8, 2026, Cloudflare was alerted to a critical security vulnerability involving tenant isolation within its Workers AI platform. The issue, reported by security researcher Gadi Evron of Knostic via the company’s HackerOne bug bounty program, affected the GLM-5.3 model (@cf/zai-org/glm-5.3). Between August 27 and September 9, 2026, a configuration mismatch and a software bug in the underlying inference engine resulted in a limited number of instances where customer data was leaked between sessions.

Cloudflare confirmed that the issue was isolated exclusively to the GLM-5.3 model. No other AI models hosted on the Workers AI platform were impacted. Following an immediate investigation, the model was taken offline, patched, and restored within 24 hours of the vulnerability being confirmed.

Understanding the Incident: The Technical "Perfect Storm"

The vulnerability arose from a combination of two distinct technical failures: a configuration oversight and a logic error in the SGLang inference engine, which Cloudflare utilizes for its disaggregated prefill/decode architecture.

The Configuration Mismatch

Cloudflare deployed the GLM-5.3 model with an advertised context window of 1,048,576 tokens. However, the Key-Value (KV) cache pool—the GPU memory allocated to store the state of the conversation—was inadvertently limited to 681,280 tokens on the prefill nodes. While Cloudflare engineers were aware of this discrepancy, initial pre-launch testing relied on monitoring HTTP status codes rather than verifying the actual output coherence. Because the system returned successful responses during testing, the team erroneously concluded that the model was handling the context window limit gracefully.

The SGLang Logic Bug

The second component of the failure was a bug in how SGLang managed "over-length" requests. When a prompt exceeded the 681,280-token limit, the engine’s scheduler correctly identified the request as rejected. However, rather than aborting the request and returning a clear client-side error, the system substituted the long prompt with a one-token placeholder.

Crucially, the prefill node generated a KV cache for this placeholder and transmitted it to the decode node. The decode node, receiving a "transfer complete" signal, erroneously assumed the prefill node had overwritten the previous cache state with the full prompt data. In reality, the cache contained a mixture of new placeholder data and "stale" cache pages from previous, unrelated user sessions. Consequently, the decode node generated responses based on this residual data, leading to the cross-tenant exposure.

Chronology of the Event

The timeline of the incident reflects a swift response from both the Cloudflare security team and the SGLang maintainers.

• August 27, 2026: GLM-5.3 launches on Workers AI with the misconfigured KV cache pool.
• September 8, 2026 (23:11 UTC): Gadi Evron submits a report via HackerOne providing a reproduction script.
• September 9, 2026 (08:00 – 13:22 UTC): Cloudflare begins verification. Product Security reproduces the leak using two distinct test accounts within 25 minutes of investigation.
• September 9, 2026 (14:00 UTC): Incident declared. GLM-5.3 is set to private, and the model is taken offline.
• September 9, 2026 (14:52 – 19:40 UTC): The team identifies the root cause, reconfigures the KV cache pool to 2.04 million tokens (well above the context window), and patches the engine to correctly handle rejected requests. The model is restored to public access.
• September 10, 2026: A high-level meeting is held between Cloudflare and the SGLang maintainers (RadixArk) to coordinate long-term upstream fixes.
• September 11 – October 26, 2026: A series of hardening updates are deployed, including per-transfer checksums and improved abort-notification protocols, culminating in the coordinated disclosure of these findings.

Analysis of Affected Data

Cloudflare’s forensic analysis indicates that the scope of the exposure was minimal. A total of 155 responses were affected across the entire period. Each of these instances involved prompts exceeding the 681,280-token threshold.

After accounting for internal testing and the researcher’s verification, the company concluded that the timing and volume of requests were consistent with legitimate long-context use cases. There was no evidence of malicious actors discovering and repeatedly exploiting the bug. Cloudflare’s internal monitoring confirms that the data leakage was a byproduct of standard, high-volume processing rather than a targeted attack.

Immediate Remediation and System Hardening

Upon identifying the vulnerability, Cloudflare implemented a two-fold fix to ensure immediate safety and long-term stability.

1. Configuration Correction: The team adjusted the GLM-5.3 configuration to ensure the prefill KV pool could accommodate the full context window. By matching the configuration to the established GLM-5.2 standards, they increased the available cache to approximately 2.04 million tokens. This ensures that no valid request will ever reach the "rejection" state that triggered the bug.
2. Engine-Level Patching: Recognizing that relying on pool size is a fragile defense, Cloudflare authored a patch for the SGLang engine. This patch forces the system to retire any rejected request at the admission queue level—before any memory allocation or KV transfer occurs. This ensures that if a prompt is too large, the process is killed instantly, preventing the possibility of the decode node ever seeing "stale" data.

Implications for AI Infrastructure

This incident serves as a significant case study in the challenges of "disaggregated" AI inference. In modern high-performance LLM deployments, separating the prefill and decode tasks across different GPU pools is standard practice for efficiency. However, this separation introduces complex state-synchronization requirements.

Memory Safety and KV Cache

The KV cache is a shared pool of fixed-size pages. In many inference engines, pages are not zeroed out between requests for performance reasons. While this is acceptable in single-node environments, it introduces a reliance on the engine to perfectly track memory ownership in distributed architectures. If the prefill node and the decode node become desynchronized—as they did here—the decode node can inadvertently "read" memory that has not been properly cleared or allocated to the current request.

The Move Toward "Integrity by Construction"

Cloudflare has emphasized that configuration checks are only a stopgap. The company is now advocating for "integrity by construction." This involves a system where every physical KV page carries a unique tag that changes upon reallocation. Before a decode node reads a cache page, it would verify that the tag matches the expected state. If a mismatch is detected, the request would fail, preventing the leak of sensitive data.

Commitment to Open Source and Future Security

The partnership between Cloudflare and the SGLang maintainers (RadixArk) highlights the importance of coordinated disclosure in the AI ecosystem. Since the incident, the SGLang project has incorporated several security-focused pull requests from Cloudflare, including improved observability metrics and more robust error handling for failed transfers.

Cloudflare has also committed to applying its extensive experience in formal verification—previously used to secure DNS infrastructure—to the domain of AI inference. By modeling the distributed KV cache system, the company intends to prove the mathematical safety of these transfer protocols.

Recommendations for Operators

For other operators utilizing SGLang or similar inference engines in disaggregated mode, Cloudflare offers the following recommendations:

• Validate Pool Sizing: Ensure the KV cache pool size is explicitly verified against the model’s context length at startup.
• Implement Integrity Checks: Enable checksums for KV transfers to ensure the data arriving at the decode node is exactly what the prefill node intended to send.
• Update Dependencies: All operators should ensure they are running SGLang v0.5.22 or later, which contains the critical patches for these race conditions and memory-safety issues.

Conclusion

The security of large-scale AI platforms relies on the integrity of the underlying inference engines. While the vulnerability in GLM-5.3 was a critical failure of state management, the rapid response of both the vendor and the open-source community demonstrates the efficacy of modern security disclosure processes. As AI models move toward larger context windows and more complex distributed architectures, the industry must prioritize memory-safe design and formal verification to ensure that data isolation remains a foundational guarantee of AI service delivery.

Cloudflare continues to encourage the reporting of security vulnerabilities through its HackerOne program. Further technical details regarding the upstream fixes can be found in the official SGLang disclosure post.