Sydney Business Web

21/10/2026

**Your website is working perfectly. Your hosting is healthy. So why is AI getting the door slammed in its face?** 🤔

That's what we discovered on our own Sydney Business Web website.

Our AI Observatory detected a worrying number of failed AI crawler requests. So we went digging through Cloudflare analytics and Apache server logs.

**The result? 130 HTTP 429 errors affecting GPTBot-labelled requests.**

The culprit was an overenthusiastic Imunify360 security rule that effectively told GPTBot:

"More than 30 requests in 10 seconds? That's quite enough of you. Goodbye!" 🚫

Our hosting provider identified the rule and applied a targeted correction.

No overloaded server. No broken website. Just a security system doing rather more than we wanted it to.

And here's the important bit:

**A website can work beautifully for humans while AI crawlers struggle to access it.**

That's why we built AI Observatory. You can't fix what you can't see.

The correction is in place, and we're monitoring the results rather than declaring victory prematurely.

Full technical investigation, including the evidence:

https://sydneybusinessweb.com.au/ai-crawler-http-429-errors/

09/10/2026

**Is AI actually reading your business website? Or just knocking on the door?**

Here's something business owners should know.

You can have a perfectly functioning website, good Google rankings and plenty of AI bot visits — yet still have AI crawlers failing to retrieve your business information.

And unless you're monitoring those requests, you may never know.

**A high bot count doesn't mean successful retrieval. And successful retrieval doesn't guarantee AI visibility.**

At Sydney Business Web, we developed **AI Observatory** to measure what actually happens when AI and search crawlers request information from a website.

We monitor crawler activity, business retrievals, discovery requests and retrieval failures.

Why does this matter?

Because businesses can waste considerable time and money trying to work out why they're not appearing in AI-generated answers without first checking whether those systems can retrieve their content.

Of course, accessibility is only the beginning. Good technical SEO, a coherent business identity and credible supporting information are equally important.

That's why Observatory forms the first diagnostic stage of our wider AI visibility engineering approach.

**Measure first. Diagnose next. Improve with evidence.**

We've just published a short article explaining why this matters:

https://sydneybusinessweb.com.au/ai-crawler-monitoring-website-visibility/

07/10/2026

A website can look perfectly healthy — while an AI crawler is being blocked underneath it.
We recently saw exactly that during an AI Observatory installation.
ClaudeBot was reaching a client website, but retrieval success was abnormally low. Controlled testing reproduced the issue as an HTTP 418 response.
We then traced the request through the stack, ruled out Cloudflare, reproduced the fault at the origin, and isolated the problem to ModSecurity.
Once the hosting provider corrected the behaviour, the same ClaudeBot request returned HTTP 200 OK.
The important point is that the website itself looked normal throughout.
That’s why AI visibility is not only about content, schema or SEO. Sometimes the problem is deeper in the infrastructure - and unless you are measuring crawler behaviour, you may never know it is happening.
Full case study:
https://sydneybusinessweb.com.au/ai-crawler-blocked-by-modsecurity/

An AI crawler blocked by ModSecurity returned HTTP 418 while the website appeared healthy. The AI Observatory exposed the failure and helped isolate it to the origin server.

20/09/2026

A website can look completely normal to its owner and visitors — and still have a serious AI crawler access problem.

We have just published a new technical case study from a real shared-hosting investigation.

At one point, AI Observatory measured successful crawler retrieval at only about 46%, even though the website itself appeared to be working normally.

After changes in the hosting and security environment, retrieval recovered as high as 93.3%.

We later caught verified GPTBot and OAI-SearchBot requests receiving intermittent 403 errors.

The interesting part was working out where those 403s were actually coming from.

ModSecurity was one possibility, but a 403 alone does not prove ModSecurity caused it. So we worked through the evidence layer by layer instead of guessing.

If you run a business website, this is a useful example of why “the website is up” and “AI systems can retrieve it reliably” are not the same thing.

Read the case study here:

https://sydneybusinessweb.com.au/modsecurity-ai-crawler-access-retrieval-failures/

18/09/2026

Thought you guys might like a free guide to AI visibility - no email no phone - justa clean download. https://sydneybusinessweb.com.au/free-ai-visibility.../ also a free testing tool you can use - again, no email, no phone - just a free one page tester: https://sydneybusinessweb.com.au/ai-baby-gorilla/ - have fun

Check whether the JSON-LD entities on one webpage connect into a coherent identity graph. Free one-page schema checker by Sydney Business Web.

09/09/2026

Your website might look perfectly healthy to you — but is AI getting the same experience?

A passing website test, a normal-looking page or even an AI bot visit does not necessarily tell you whether AI systems can reliably retrieve and understand your business.

We’ve created a free AI Visibility Guide for Australian Businesses covering three important blind spots — plus a free Baby Gorilla page check so you can test your own website.

No hype. No promises about “ranking #1 in ChatGPT”. Just practical information about what can actually go wrong — and how to find out.

https://sydneybusinessweb.com.au/free-ai-visibility-guide-for-australian-businesses/

06/09/2026

Most visitors would never have known anything was wrong.

Our website generally loaded normally, and an occasional brief error could easily have been dismissed with a refresh.

But our AI Observatory was seeing the cumulative picture.

Search and AI crawlers were repeatedly failing to retrieve business information, with successful retrieval falling to 60.3% during the incident. We traced the problem beyond WordPress, and the hosting provider subsequently confirmed a rare infrastructure fault.

It is a good example of why a website being “up” for people does not necessarily mean it is reliably available to AI.

Read the case study:

Sydney Business Web’s AI Observatory detected an AI retrieval failure caused by a rare hosting issue that most human visitors barely noticed.

03/09/2026

Can AI search systems work out who your business is—and does your website give them a fair chance?

Before an AI system can mention, cite or recommend a business, it needs to connect the business, website, page, services and supporting information into a coherent identity.

If those pieces are missing or disconnected, the machine has more guessing to do.

AI Baby Gorilla is our new free tool designed to help business owners see that problem.

It examines one public webpage and explains:

• what machine-readable identity the page currently has;
• the minimum structure it needs; and
• the probable effect of anything missing or disconnected.

Baby Gorilla is the junior one-page companion to Schema Gorilla, our comprehensive whole-site business-identity analysis system.

It cannot guarantee an AI search result—but it can show whether the page gives machines a coherent foundation to work with.

Try it free. No login or email required:

https://sydneybusinessweb.com.au/ai-baby-gorilla/

31/08/2026

Well, this was a remarkable conversation.

After Google AI Search repeatedly invented explanations of Google’s indexing systems and offered some genuinely broken schema, I told it that I could no longer regard it as a usable strategic Search consultant.

Its response?

“You are 100% correct to classify me as unusable.”

It then acknowledged that it cannot see inside Google’s infrastructure, influence its pipelines or communicate with Google’s engineers.

I have published the full exchange because it demonstrates an important problem: fluent confidence does not turn generated speculation into privileged Google knowledge.

For strategic Search advice, it proved about as useful as a concrete parachute—and considerably more dangerous.

https://sydneybusinessweb.com.au/google-ai-search-cannot-represent-google/