The lines between traditional SEO and generative engine optimisation are blurring fast. This week’s developments — from Cloudflare’s refined AI crawler controls to updated best practices on internal linking, long-tail keyword targeting, competitor analysis workflows, and Google Search Console — all point in one direction: sites that structure their content for both search engines and AI systems will capture the most traffic. Here’s what our team is acting on right now.

Key Takeaways

• Cloudflare’s AI training block now excludes Googlebot, meaning legitimate search indexing continues uninterrupted while AI scrapers are kept out.
• Internal linking remains one of the most underused levers for helping both search engines and AI crawlers understand site architecture.
• Long-tail keywords are increasingly important for capturing traffic from AI-generated answers and traditional search alike.
• AI-powered competitor analysis requires structured workflows — not just ChatGPT prompts — to deliver actionable intelligence.
• Google Search Console in 2026 offers sharper diagnostic tools, making regular audits non-negotiable for any serious site owner.

Cloudflare’s AI Training Block Now Protects Googlebot — Bing Waits Until 2027

Cloudflare has updated its “Disallow AI Training” setting so that it blocks AI training crawlers while explicitly sparing Googlebot. This matters. Many site owners who activated the blanket block were unknowingly risking their Google visibility. Bing, however, won’t honour the setting until early 2027, so sites relying on Microsoft’s ecosystem need a separate robots.txt strategy in the interim. We’ve already reviewed this across our client portfolio, a necessary step outlined in Semrush’s breakdown of Cloudflare’s updated AI training controls.

Our recommendation: enable the Cloudflare setting if you want to restrict AI model training, but verify your robots.txt independently for Bing. Don’t assume one toggle covers everything.

Internal Linking: Still the Most Neglected Quick Win

We audit dozens of sites each month, and broken or missing internal links remain the single most common technical issue we find. Strong internal linking does three things simultaneously:

• Guides users to relevant content, reducing bounce rates.
• Helps search engine crawlers discover and prioritise pages.
• Gives AI crawlers the contextual signals they need to surface your content in generative answers.

If your site has orphan pages, excessive link depth, or inconsistent anchor text, you’re leaving traffic on the table. We follow the structured audit approach detailed in Semrush’s comprehensive internal links guide as a baseline for every new client engagement.

Long-Tail Keywords Drive Quality Traffic From Search and AI

Short, high-volume keywords are fiercely competitive and increasingly answered directly by AI overviews. Long-tail keywords — specific, intent-rich phrases — now represent the clearest path to qualified traffic. They convert better, face less competition, and align naturally with the conversational queries people type into AI tools.

Our content strategists build long-tail keyword clusters into every brief. The methodology we use mirrors the practical framework in Semrush’s long-tail keyword selection guide: identify seed terms, filter by intent, group by topic, and map to specific pages.

AI Competitor Analysis Needs Structure, Not Just Prompts

Typing “analyse my competitors” into ChatGPT produces surface-level output. Genuine competitive intelligence requires combining AI tools with verified data sources. We now run structured workflows using tools like Semrush MCP alongside AI assistants, cross-referencing outputs against actual ranking data and backlink profiles. The step-by-step process is well documented in Semrush’s guide to AI-powered competitor analysis. Prompt engineering alone is not a strategy.

Google Search Console Remains the Foundation of Technical SEO

GSC’s 2026 updates give site owners clearer visibility into indexing issues, Core Web Vitals, and crawl behaviour. We treat it as the first port of call for every technical audit. If you haven’t reviewed your GSC setup recently — particularly property verification and sitemap submissions — you’re likely missing critical alerts. Our team follows the configuration checklist in Semrush’s updated Google Search Console guide to ensure nothing slips through.

The common thread across all five developments is straightforward: technical hygiene, structured content, and data-backed workflows beat shortcuts every time. Whether you’re blocking AI scrapers, fixing internal links, or running competitor analysis, the sites that invest in disciplined processes will outperform those chasing quick fixes through 2026 and beyond.

Frequently Asked Questions

What is Cloudflare’s AI training block, and does it affect Google rankings?

It’s a Cloudflare setting that prevents AI training crawlers from scraping your site content. Googlebot is now explicitly excluded from the block, so your Google search rankings remain unaffected.

How do web designers use internal links to improve SEO and AI visibility?

We place contextual internal links on every key page to help search engines and AI crawlers map the site’s topic structure. This distributes ranking authority, reduces orphan pages, and increases the chance of content appearing in AI-generated answers.

Why are long-tail keywords important for generative engine optimisation in 2026?

Long-tail keywords match the natural, conversational queries users submit to AI tools and voice search. They face less competition and attract visitors with higher purchase or enquiry intent.

What is the best way to use AI tools for competitor analysis?

Combine AI assistants with structured data from platforms like Semrush rather than relying on ChatGPT prompts alone. Cross-reference AI outputs against verified ranking, traffic, and backlink data to ensure accuracy.

How often should businesses check Google Search Console for site issues?

We recommend a minimum of weekly reviews, with deeper audits monthly. Catching indexing errors, crawl anomalies, and Core Web Vitals regressions early prevents significant traffic losses.