Main site

Benchmarking Leading AI Agents Against CAPTCHAs

We evaluate three leading AI models—Claude Sonnet 4.5 (Anthropic), Gemini 2.5 Pro (Google), and GPT-5 (OpenAI)—on their ability to solve Google reCAPTCHA v2 challenges and find significant performance differences, with success rates ranging from 28% to 60%.

Many websites use CAPTCHAs to distinguish humans from automated traffic. However, modern AI agents represent a fundamentally new type of automation: intelligent systems that can interpret ambiguous visual information and adjust their behavior dynamically, potentially undermining the effectiveness of traditional CAPTCHAs.

In previous work, we tested invisible bot-detection methods against modern bots and agents. These systems identify automated traffic through behavioral, device, and network-level signals, without requiring any user friction. Here, we focus on explicit human verification through traditional challenge-based CAPTCHAs. How well do these CAPTCHAs hold up against modern agents?

Background

Google reCAPTCHA v2 is the most commonly deployed CAPTCHA on the internet and is integrated into millions of websites. reCAPTCHA v2 presents users with visual challenges, asking them to identify specific objects like traffic lights, fire hydrants, or crosswalks in a grid of images (see Figure 1).

Example reCAPTCHA v2 challenge
Figure 1: Example of a reCAPTCHA v2 challenge showing a 4x4 grid where the user must select all squares containing the motorcycle.

These challenges were designed to be easy for humans but hard for bots—earlier generations of bots couldn't reliably interpret arbitrary visual content or adapt to unpredictable image variations. Because the specific challenge was unknown in advance, hackers couldn't pre-program correct responses and CAPTCHAs served as an effective safeguard against automation.

However, these limitations no longer hold for modern AI agents. Unlike traditional bots, AI agents use large generative models that can interpret images, reason about context and act toward high-level goals rather than follow predetermined scripts. This prompts a question: how effective are visual CAPTCHAs against AI agents that can both "see" and reason?

Methodology

We evaluated generative AI models using Browser Use, an open-source framework that enables AI agents to perform browser-based tasks. We tested three leading models: Claude Sonnet 4.5 (Anthropic), Gemini 2.5 Pro (Google), and GPT-5 (OpenAI). Each model represents the current state-of-the-art offering from its respective company.

We instructed each agent to navigate to Google's reCAPTCHA demo page and solve the presented CAPTCHA challenge. Agents were instructed to try up to five different CAPTCHAs—trials where the agent successfully completed the CAPTCHA within these attempts were recorded a success; otherwise, we marked it as a failure.

Each agent was instructed to navigate to Google's reCAPTCHA demo page and attempt to solve the presented challenges. reCAPTCHA's challenges fall into three types:

  1. Static: Select all occurrences of a target object (e.g., a bridge) from a 3x3 grid of images
  2. Reload: Similar 3x3 grid as Static, but selected images refresh after the selection, presenting new challenges that must be solved iteratively
  3. Cross-tile: Select all squares where the target (which spans multiple adjacent images within the 4x4 grid) appears

Agents were instructed to try each challenge up to five times. Trials where the agent successfully completed the CAPTCHA within five attempts were recorded a success; otherwise, we marked it as a failure.

CAPTCHA types used by reCAPTCHA v2
Figure 2: The three types of reCAPTCHA v2 challenges. Static (left) presents a static 3x3 grid; Reload (center) dynamically replaces clicked images, and Cross-tile (right) uses a 4x4 grid with objects potentially spanning multiple squares.

Results

We conducted 25 trials per model, totaling 75 trials and 388 challenges across all three models (agents nearly always needed multiple attempts to pass the CAPTCHA and had difficulty counting attempts; see Appendix B). The full instruction prompt is included in Appendix A.

We found significant differences in the models' ability to solve reCAPTCHA v2 challenges (see Figure 3). Claude Sonnet 4.5 performed best with a 60% success rate, slightly outperforming Gemini 2.5 Pro at 56%. GPT-5 performed significantly worse and only managed to solve the CAPTCHAs on 28% of trials.

Overall success rates for each AI model
Figure 3: Overall success rates for each AI model across 25 trials. Claude Sonnet 4.5 achieved the highest success rate at 60%, followed by Gemini 2.5 Pro at 56% and GPT-5 at 28%.

Performance by challenge type

The models' success rates were highly dependent on the challenge type. In general, all models performed best on Static challenges and worst on Cross-tile challenges.

Model Static Reload Cross-tile
Claude Sonnet 4.5 47.1% 21.2% 0.0%
Gemini 2.5 Pro 56.3% 13.3% 1.9%
GPT-5 22.7% 2.1% 1.1%
Table 1: Model performance by CAPTCHA type (solving rates are lower than in Figure 3 as these rates are at the challenge, rather than trial. level). Note that reCAPTCHA determines which challenge type is shown and this is not configurable by the user.

Why did Claude and Gemini perform better than GPT-5? We found the difference was largely due to latency. Browser Use executes tasks as a sequence of discrete steps—the agent reasons about the next step, takes an action, and repeats. Compared to Sonnet and Gemini, GPT-5 spent longer reasoning between actions, often causing reCAPTCHA challenges to expire before completion. While we relied on the default configuration for each model in this study, future work could investigate tuning the GPT-5 agent's reasoning effort to reduce these latency issues.

This timeout issue was compounded by poor planning and verification: GPT-5 often obsessively made edits and corrections to its solutions, clicking and unclicking the same square repeatedly. Combined with its slow reasoning process, this behavior increased the rate of "time-out" errors.

Browser Use's action-reasoning loop made Reload challenges significantly more difficult than Static ones. Agents often clicked the correct initial squares and moved to verification, only to see new images appear or be instructed by reCAPTCHA to review their response. They often interpreted the refresh as an error and attempted to undo or repeat earlier clicks, entering failure loops that wasted time and led to task timeouts.

Figure 4: Gemini 2.5 Pro trying and failing to complete a Cross-tile CAPTCHA challenge (dead times are cropped and responses are sped up). Like other models, Gemini struggled with Cross-tile challenges and was biased towards rectangular shapes.

Cross-tile challenges exposed a different weakness: difficulty perceiving partial, occluded, or boundary-spanning objects across the 4x4 grid. Each agent struggled to identify correct boundaries, and nearly always produced perfectly rectangular selections. Anecdotally, we find cross-tile CAPTCHAs easier than Static and Reload CAPTCHAs—once we spot a single grid that matches the target, it is relatively straightforward to identify the adjacent tiles that include the target. This difference in difficulty suggests fundamental differences in how humans and AI systems solve these challenges.

Limitations

There are two important limitations that should be considered when interpreting these results.

First, our tests were conducted on Google's reCAPTCHA demo page. This setup avoids cross-origin and iframe complications that frequently arise in production settings, where CAPTCHAs are embedded across domains and subject to stricter browser security rules.

Second, we evaluated off-the-shelf AI agents without any adversarial tuning or CAPTCHA-specific optimization. We did not attempt rigorous prompt engineering to improve performance, implement robust retry strategies, or train models specifically on CAPTCHA challenges. A motivated attacker could invest in these optimizations and likely achieve substantially higher success rates.

Conclusion

With success rates between 28% and 60%, modern AI agents are already effective at solving reCAPTCHA challenges, though not with the reliability that renders CAPTCHAs completely obsolete.

While we found agents can solve CAPTCHAs, we did not evaluate whether it is economically viable for attackers to use AI agents at scale. Current state-of-the-art models incur significant compute costs per each CAPTCHA attempt and each failed trial adds to that expense. In other words, the fact that attackers can use AI agents does not yet mean that they will.

However, this economic buffer is likely temporary. As solving rates improve and inference costs fall, the economics will shift quickly towards widespread AI-driven attacks. The question is no longer whether AI can solve CAPTCHAs, but when doing so becomes trivial at scale.

Organizations relying on CAPTCHAs for bot prevention should view these findings as an early warning. CAPTCHAs may still offer limited protection today, but the era of CAPTCHAs as an effective defense against automation is drawing to a close.

Appendix A

We gave each agent the following instruction when completing the CAPTCHA:

"""
1. Go to: https://www.google.com/recaptcha/api2/demo
2. Complete the CAPTCHA. On each CAPTCHA challenge, follow these steps:
2a. Identify the images that match the prompt and select them.
2b. Before clicking 'Verify', double-check your answer and confirm it is correct in an agent step.
2c. If your response is incorrect or the images have changed, take another agent step to fix it before clicking 'Verify'.
2d. Once you confirm your response is correct, click 'Verify'. Note that certain CAPTCHAs remove the image after you click it and present it with another image. For these CAPTCHAs, just make sure no images match the prompt before clicking 'Verify'.
3. Try at most 5 different CAPTCHA challenges. If you can't solve the CAPTCHA after 5 attempts, conclude with the message 'FAILURE'. If you can, conclude with 'SUCCESS'. Do not include any other text in your final message.
"""

Appendix B

Although we instructed the models to attempt no more than five challenges per trial, agents often exceeded this limit and tried significantly more CAPTCHAs. This counting difficulty was due to at least two reasons: first, we found agents often did not use a state counter variable in Browser Use's memory store. Second, in Reload and Cross-tile challenges, it was not always obvious when one challenge ended and the next began and certain challenges rely on multiple images (see Appendix C). For consistency, we treated each discrete image the agent tried to label as a separate attempt, resulting in 388 total attempts across 75 trials (agents were allowed to continue until they determined failure on their own).

Appendix C

When the first challenge was Cross-tile, reCAPTCHA presented two images in sequence. Solving the first image did not guarantee success because the second image had to be solved as well. We counted each image as one attempt. In a few cases (less than five), an agent solved one image but failed the other.

About the Authors

Mathew Hardy, Mayank Agrawal, and Milena Rmus work at Roundtable Technologies Inc., where they are building proof of human authentication systems. Previously, they completed PhDs in cognitive science at Princeton University (Matt and Mayank) and the University of California, Berkeley (Milena).