OpenAI may request an ad’s landing page to validate it and understand where the ad is relevant. A page opening normally for you does not prove that an automated crawler can reach the same content.
For ad landing-page validation, allow OAI-AdsBot. OpenAI recommends allowing OAI-SearchBot too. If you use product feeds, make sure OAI-SearchBot can also reach the product image URLs. These checks apply whether your website is hosted in Pakistan or elsewhere. Read the latest OpenAI advertiser crawler guidance.
1. Test the exact URL used in the ad
Open the full destination URL, including tracking parameters. Follow every redirect and confirm the final page:
- Loads without signing in, completing a CAPTCHA or passing a browser challenge.
- Shows the advertised product or service, not a generic error or an unrelated page.
- Works on the devices and locations your tracking or redirect service targets.
- Keeps required assets and content available after the redirect.
A homepage test is not enough when the ad points to a product page, a subdomain, a campaign URL or a third-party tracking destination.
2. Check robots.txt and required assets
Ask your developer to review the robots.txt rules on the landing-page host and any host reached through a redirect. Allow OAI-AdsBot for ad validation; also allow OAI-SearchBot where appropriate, especially for product-feed images.
Use OpenAI’s current crawler documentation and published OAI-AdsBot IP ranges and OAI-SearchBot IP ranges when your security setup needs IP allowlisting. Avoid relying only on IP addresses copied from old access logs, since infrastructure can change.
3. Review the security layers beyond robots.txt
A permissive robots.txt file cannot override a block from your hosting provider, firewall, CDN, web application firewall or bot-protection service. Ask the team that manages those controls to check whether verified crawler traffic is blocked or challenged.
- CAPTCHA, JavaScript or “prove you are human” challenges.
- Login requirements or access restrictions.
- Firewall or CDN rules applied to the final destination or image host.
- Geographic restrictions that change the response.
- Rate limits that return 429 Too Many Requests.
For Cloudflare users, OpenAI’s current advertiser guidance says OAI-AdsBot is officially verified and allowlisted. Confirm the active security configuration with your administrator before changing rules.
4. Use the response code to narrow the problem
| What the request returns | What to investigate |
|---|---|
| 403 / access denied | Find the firewall, CDN or bot rule denying the request. Check the redirected host too. |
| 404 / not found | Repair the destination or redirect. Test tracking links on different devices and locations. |
| 429 / too many requests | Review rate limits and bot events. For a large upload, test a few representative ads and submit smaller batches. |
| Blank, incomplete or challenge page | Make the intended content and required assets accessible without a browser challenge. |
Where available, use the failed request’s time and request ID to locate the event in hosting, CDN or security logs. Check the response the crawler receives; a page that works in your own browser may receive different treatment.
5. Recheck the ad after the fix
Record what changed and when. Retest the exact ad URL, follow the redirects, and verify the final page and assets. Then check the affected ads in Ads Manager. If their status does not update, follow the available resubmission or review steps. Removing a crawl block does not guarantee ad approval; the ad still has to pass review.
Frequently asked questions
Which crawler should an advertiser allow first?
OAI-AdsBot is required for ad landing-page validation. OpenAI recommends allowing OAI-SearchBot as well; product-feed image URLs should also be accessible to OAI-SearchBot.
Does changing robots.txt fix every crawler block?
No. CDN, firewall, WAF, login, CAPTCHA, browser-challenge and rate-limit rules can still block a request.
Can a working homepage confirm that the ad destination is accessible?
No. Test the exact URL used in the ad, including tracking parameters, redirects and the final destination.
What should I send to my developer or hosting provider?
Share the exact ad URL, its final destination, the error or response code, failure time and timezone, and any available request ID. Include what you already changed.
Will fixing crawler access automatically approve my ad?
No. It can resolve an access problem, but the ad remains subject to review. Resubmit or request review if the status does not refresh after the fix.
Related guides: ChatGPT Ads reporting, Conversion tracking, OpenAI Pixel, and Event quality.