What Is Googlebot?

Transcript: What Is Googlebot?

This transcript of What Is Googlebot? comes from the video's captions, lightly cleaned. Click a time and the set jumps to that moment.

Welcome and introductions

0:00 Hi there folks. Welcome to this new podcast. We're going to do it every week. We're going to cover a wide spectrum of SEO subjects. You'll be able to watch this and listen to this on all your most popular podcast apps, but you'll also be able to watch it on YouTube on David G's Quaid's YouTube channel. What's What's the name of the show going to be? It's going to be called the David GU SEO show. And like I say, u me and David are going to cover almost everything you need to know around SEO and AI and digital marketing in 2026. So David, would you like to quickly introduce yourself to the listeners and viewers?
Yeah. Hey, I've been doing SEO for 26 plus years. A lot of it in cyber security and SAS and AI but I think I've worked in every industry and it's something I love doing and I think that the best way to learn SEO is to understand the basic mechanics and once you understand the basic mechanics become simple. So that's my mission in life. Yeah, we've been doing these we've been doing some videos every Friday for the f past few months and we just decided to form to make it more formal and also turn it into a podcast.

1:34 Hopefully you will be al along on the journey and also you recommend us to some of your friends. So what's going to be the title of this show? Well, with SEO and what is Googlebot? Got some questions here for the all knowledgeable goat of SEO, David. So let's go straight into it. So David, is Googlebot really crawling spider or is it what just a comforting metaphor hiding how AI driven ranking actually works? So that was a bit of a mouthful. So I think actually it's better to do intro. What do you think m what do you think Googlebot is and what on the highest level what is it

Googlebot's two modes

2:29 Doing?
So I think Googlebot is a it's an application that Google runs that fetches pages and it runs in two modes. First mode is called discovery where it follows a crawl list. It fetches every file in the crawl list, opens it up, interrogates it and then feeds other systems like the indexing system, the snippet builder and the spam systems. The second mode is retrieval. Sorry, that was retrieval. The second mode is this. So retrieval basically means it gets a crawl list from a priority manager, follows everything in the crawl list, retrieves those documents, breaks it up for the systems. The second mode is discovery. Discovery is where it will go out and [clears throat] it'll be given something like a sitemap or a list of other URLs that have been discovered and it'll go and look for other URLs and put them into other crawl lists. So, it's either building a crawl list or following a crawl list and

Why the spider story is a fairy tale

3:34 Fetching all the files and essentially as it opens all the files, it's building new crawl lists. I think that there's a lot of misinformation and I think the misinformation stems from three ideas and I think what I wanted to cover today is the spider idea. I think 26 years ago the spider when people were just getting computers at home for the first time when people were learning to use browsers the story of the spider came out and I think the problem with this the story of this the spider is that it's partially fictional but it's somehow become ingrained and it's somehow still prevalent within especially circles you know web engineers and web developers still talk about spiders as a process that starts from the very top of a website and goes all the way through to the end of a website. And that's just not true anymore. In fact, I don't think that was ever true. So, the if you go to and I blame Google. I think there's a lot of misinformation in for example, if you take the topic of crawling, it's actually a very complex fragmented system. It's not it's not done in one go, right? It doesn't find a website, start at the homepage, and crawl down. So if you go to for example, let's call it public facing pages like the Google how it works page.

4:52 Or you go to YouTube and you ask what is Googlebot or what is a spider? It'll tell you the story about a spider opening up your website and building a map of your site and trying to understand your website and trying to read all the pages and that's fanciful.

How Google split up crawling

5:08 Google broke up the crawling system a long time ago, most notably for caffeine when it realized that the top part of the pyramid, right, gets the most amount of traffic, right? So your most important websites like your Reddits, your YouTubes, your CNN's, your Microsofts, your Apples, your Amazons, they get all the traffic. And then the next layer down maybe gets to share 50% of the traffic. And then the next tier down makes up like 90% of the website of I'm sorry of the websites in the world. And crawling is optimized in Google not to crawl whole websites but to crawl the most frequently accessed websites. And I think when I listen to web engineers and also geo web designers talking about how AI crawlers work and crawling depth and things like that. They sort of sound like proxies to understanding authority and understanding popularity. And I think people are building a different version of what crawlers are to try to understand how they're getting indexed in Google and how they're getting indexed in AI. And I think most of those models are wrong. And again, most of the best information about crawling comes from people like Gary Illyes, John Mueller and Martin Splitt, but mostly Martin Splitt and Gary Illyes.

6:31 And most that information has been coming out since 2016 and they're in places like Off the Record YouTube channels certain shows that they do as well as LinkedIn posts and LinkedIn articles or quotes that they give mainly to Search Engine Roundtable via Barry Schwartz Rusty Brick. If you're looking for a really, really good news source, Barry Schwartz has always been my go-to guy in terms of of objectively trying to convey what Google says without interpreting it. I think a lot of SEOs interpret and give their own spin on things, and that's where it becomes problematic. In essence, I'm giving my own spin here, but what I'm trying to do is tie people to not just look at the Google documentation, which has now become very old, right? So we can see that the Google documentation for example for disavowing back links has become aged and dated. It says you should use it for disavowing back links you don't recognize whereas John Mueller's advice is you should just never use it and it's I can see I'm I'm I'm using you Jonathan. I know this is deeply technical. And this is what I love about Etsio the deep down part. So, I guess to bring it up to higher level, there's a story about spiders crawling your website and trying to understand and looking at all of the pages. And I'm trying to say that's a work of fiction. Sometimes I call it the Incy Wincy spider fairy tale.

8:01
Right. So, that leads on to the next question. What does Googlebot actually see when it visites your site compared to what we're seeing?

What Googlebot sees and how often it crawls

8:14 So goo Googlebot will see one page at a time and it'll see them out of order. So as it's crawling through a crawl list, the pages that a Googlebot is crawling are all in the same pool of importance. So if it's a very high level pool of importance, Google will be calling for example the XML feeds from CNN, the XML feeds from Amazon, from eBay, from big news sites, from big marketing sites like Cisco or Citrix or Moto. It doesn't there's no attempt to try to crawl your site as a whole. It crawling is done on the most important pages have to get crawled multiple times a day and because pages are linked to by other pages which aren't necessarily important, Google defers pages that just don't get clicks into smaller pools. So, if your website hasn't yet had clicks, all of your pages are going to be in a pool where the number of bots to pages is like 1 to 100,000, which means your pages are at best getting crawled once or twice a month, even if they have lots and lots of links to them. It's only when your pages get links from very active pages or start to get clicks that they get started to get crawled more frequently.

9:29 Included in that would be your sitemap pages. And so your sitemap pages, even though you're building them and you keep refreshing them and adding to them to search console, it doesn't mean Google's actively reading them. Fetching a page and processing page, two different parts of the same puzzle. So what does Googlebot see? If it's just a regular HTML page that server rendered, Googlebot will just read the HTML. It'll ignore most of the JavaScript unless it needs to parse some of that for like the snippet builder or for working out if links are visible or not. It takes data that it needs like what other pages it links to and it puts that into the indexing database and then it moves on to the next page. If your page is needs to be rendered like it's in JavaScript or JScript or any client side, it's all of the Googlebots are free Chromium. That's for both Gemini and Google and they will try to render the page so that they can discover the content and then pass the content to the indexing system as well as find links

Googlebot as the USPS of the internet

10:36 And that's really all they do. They're very basic systems. I like to call them the USPS of the internet, right? They they take a look at letters. They look at the address it's going to, the address it's coming from, and then they put it into a pile, and then that pile gets shifted to the next town or city. They just really are very basic. I think a lot of people think that bots look at pages and if they don't like pages, they won't index them or if they look at pages and think that they're thin. They ignore them. They don't do any processing like that. They're they're really just the couriers of of Google's internet. And Gemini uses the same crawlers. The difference is that Gemini and other LLMs will crawl pages in real time. In other words, when a page when you're doing a prompt and certain number of pages are returned in the query fan out, they will fetch those pages in near in real time or near real time and then synthesize them for the answer. And that's really all that bots do.
All right. But the thing is, David, and I do agree with you, there's a lot of misinformation about spidering and but we get back to this crux story. This crux I won't say a story but the crux of the of the problem if it's a problem is that you know people and over the last 18 months especially they're finding their pages have dropped off considerably in traffic for various reasons or they're finding a lot their pages are no longer indexed. So, if this Googlebot isn't making a quality decision about the website or about certain key pages, what is making that decision? Because people are seeing a lot of movement from where they were or in general I don't know if you would agree with this statement but they have they're seeing a lot of volatility in where their certain pages used to produce a certain amount of traffic and

Why pages lose traffic

13:00 They no longer are.
Absolutely. So within Google systems, Google pick tries to build this idea of authority. And so crawlers essentially gather up all of that authority on the internet for the indexing system to understand. So as they go to each page and they find what the page is linking to, they're going to tell the indexing service, look, Jonathan's page links to Kurt's Fan's page and it says Kurt is a great WordPress designer. And so the indexing service gets that from the crawler and helps that assess Kurt Fen site. Then the algorithms look at okay people have been searching for Kurt site or Jonathan's site and they're looking for WordPress designer or best SEO for university sites and they start looking at how how much how often do people who go to this page go back and do another search. And [clears throat] I think in the last couple of updates, Google has moved the needle much more towards looking at pogo sticking and trying to figure out are people sticking and Navboost. And then if you look at your search console and you look at the pages that lost traffic, what you're looking for is around the time that Barry Schwartz would have said there's volatility cuz he has a volatility meter and you can see it. It's also recorded in Semrush. They record Google updates. You're looking for if you look at the average position line for a keyword and a page.

14:36 Don't do it across your whole site, right? But for a page or a page and a keyword, you're looking for a very marked dip and those will usually line up with updates. And what's happened is when Google rolls out the core update and the algorithm and rewrites the tables, essentially moving it from say version 17 to version 18, they are writing down the positions of your pages based on a new analysis of user engagement. So, your website was performing before because Google was looking at X, Y, and Zed. Now, it's looking at GF, JJ, H, and K and saying, "Actually, there's a new signal here. We're not happy with this page's performance." Dropping it and seeing if it gravitates up. And if it doesn't gravitate up, you see this curved bell where you fall off. And it is correctable. And I think some of the problems are that it we're seeing it as potentially like is it AIO traffic that's killing it? You know, in other words, when we're ranking first, but we're in the AIO view, so it's people are reading our content, they're not clicking through to our site. I think that when you see those curves, be optimistic because I think it is recoverable. One of the things I think you've got to look at is
Can I interrupt slightly? How is that?

15:53 Because I think AI overviews it's been shown does dramatically reduce traffic to the if you're you got the top one space after the overview you're going to see a 30 50 60% decline in traffic. Is that not correct?
Exactly. But I think the problem is assuming that everybody has been hit by the same problem. And so by we took a couple of pages that were very high

Getting lost traffic back

16:25 Volume for clients and we did it in in a few days right this isn't this wasn't even a month now they're they're blessed with a ton of authority right you know they have like a DA52 site and what we what we realized is that there are some queries that people are coming in and let's just say like let's just imagine that this company does they print flyers Right. And people searching for free flyers, they don't do free flyers, so they were dropping off. And it's kind of like Google read that drop off as sort of a wider signal for all of their related queries. And so what we did is we went through the pages and made sure the word free wasn't in the page. So if they mention for example we are chemical free, we found another way to write chemical free because we no longer wanted to associate with free search because that was causing a bigger taper off of people bouncing. And in some cases we republished the pages and gave them a new title. And in some cases what we did is we looked at the H2s and said these are these are H2s that necessarily don't match what people are searching for. And there are searches that we could see in search console that better describe what people are looking for than what we had described. So for example, if you have a very long H2 and your H2 is why some of the best universities are embracing digital marketing like SEO, your H2 might actually resonate better if you just had why universities should embrace SEO.

18:03 And that realignment and then republishing and then reindexing with Google has seen we've seen 150,000 impressions regained in one day. So if you've got this curve and it's going down, we've literally been able to go up and rise above the previous impact in impression range. So, in other words, if your page was on page one and you were seeing 10,000 impressions a day and you move to position 11, your impressions are going to drop with it. And so, that's a factor of losing a page and not an AIO. And so, if you start to look at GSC and interrogate the data at a keyword to page level, you're going to be able to see the cause and the solution out. So if this if the problem is you drop to page two, your clicks and impressions are going to drop because nobody goes to page two, right? Because we all know the mob hides bodies on page three cuz no one ever looks on page three, right? So if we go back and look at search console and see that there are pages that are falling for impressions, maybe there's other searches that aren't covered by AIO or we have been written down because of not matching search intent as a lot of people called it or because of pogo sticking. And I think it's worthwhile looking at what did people come in for this page and what did the page talking talk about and realigning it and then republishing it and then seeing if retries it rather than assuming the traffic is gone forever. I think far too many people just give up and say well we've lost that traffic it's gone. I think in as much as 50 to 60% of cases you've lost it for another reason. I'm not saying that AIO's aren't involved, but to say that you can't ever get traffic back I have 200 domains in my GSC. None of them are beyond recovery, if that makes sense.

20:01 Well, I see where you're coming from, but I think when it comes to, you know, I can't actually see how you do recover from AI overview because
If you're qu you know, they're just reading the AI overview. Is given the answer that they learn.
What I'm saying is that not every not every search toggles AIO, but not every person doesn't click because of the AIO. The AIO takes up more space. So, if you were in position six last year, the new position six might be on the top of page two. So, if you're seeing your impressions go down with your clicks, it's recoverable. And so what I'm saying is, you know, so you made an important statement where you said, well, I don't see how it's possible. And so you can't let that thought limiting cliche stop you from trying.
Right?
And what I'm saying is the way to try is to look at did your page and the keywords that Google's showing you for. Right? So if you look at search console, you might see keywords that you hadn't thought about that you were on page three for that you may need to add to your page to make it more relevant to that traffic could now bring you replacement traffic.

21:23 So I'm not saying you can overcome AIO's. What I'm saying is you can overcome losing traffic from Google by looking at the search phrases, the

How LLMs read pages

21:31 Search intent, the relevance between the two, your percentage relevance match. Right now. So how has Google shifted to a AI based understanding, not just link foring? I think you've had some de, you know, I think with a lot the debates I've seen you have recently with leaders in the industry or people that promote AI to a certain level. They can't really explain how AI is understanding the page. And you point out that you know so maybe you could explain to who's listening and watching this how you think first of all do you think well basically how AI is understanding pages and bringing an answer to an end user.

22:44 Okay. So, first of all, I think AI in Google ranking is very limited. I know they talk about it. I know they've got BERT and SpamBrain, but these are really old tools and I I think the weight that they have in ranking is is not quite what people think. But in terms of like how AIO's and how LLM's return results, it's it's kind of a moving feast, right? So, so they're they're getting expanded results back from Google. They might get 10 30 results back in a in a query fan out. Maybe even 100. And they might they're starting to get more selective. They also are trying to find brand results when branded terms are used. And so, I think the [snorts] first thing we need to understand about how LLMs read content is they don't read it like humans, right? They convert it to a mathematical model. It's they tokenize it. So they don't see words, they see concepts and they see relationships between concepts. So a lot of people will tell you that structure is important. Structure can be important if it's getting lost, right? If if it's impossible for the LLM to understand that you're talking about your brand and a topic, is it is it related or not related? You kind of have to be affirmative in or confirmative in in the sense but I think you also have to understand that they convert it to a tokenized model.

24:18 Sometimes the best ways to find out if an LLM understands your content is just to take it and paste it back in an LLM and ask it to summarize. And if it gives you a summary that's different to what you were writing, then no LLM is going to arrive at a different conclusion. Right?
The second thing I think is that the age of single branded domains is over. I think brands are going to have to have multiple domains. I think it's incredibly dangerous and naive. And I had a great podcast with TJ Robinson and Edward Sturm last week where we I think all of us agree that EMDs, partial match domains, information sites are going to be more and more part of the successful SEO campaign. A lot of big brands, they, you know, it looks like they run on one domain, but even companies like Nike and Microsoft have more than 19 active domains. I think that for smaller companies the block and the barrier to entry in past was the cost of design and the cost of building content. But with LLM it's getting easier to build satellite sites and I think it's going to become more and more important. Again, if you come back to how many pages the LLMs are consuming versus Google, right? Google gives you 10 results, 70% of people will click on
Interrupt here slightly because this is the big this is the big contradiction that I'm seeing in the industry because yes, you are correct and I and I love TJ, you know, he's been on my other podcast.

25:55 I think he's very knowledgeable, but he's gone fully AI, hasn't he? He's he's totally AI dominated. But I see an enormous contradiction here is that you are correct. It's it's easier to produce content. It's easier maybe to produce a website. I think to produce a website that gets a result is actually got harder. But the actual mechanics of building a website has probably got easier. So that's a contradiction. Building a website's got easier. Getting a result from a website has got a lot harder. Or you could counter me and say it's always been hard, Jonathan. All right. But then like somebody like Edwards you know and many of the guests well AI slop you people won't read AI if they know it's written AI and I think to some level he's correct. You've got to produce content that stops pogo.

27:11 Yeah, you got to produce content that engages. That's different. Well, isn't that a total contradiction of actually of having multiple domains? Cuz basically you need to concentrate and customize your content and if you have multiple domains and you haven't got you know if you got enormous marketing team doesn't apply. But if you're a small to mediumsiz business, you need to focus more on quality individual more individualistic content, not what AI can produce.
I think if if you're producing a page a day, there's nothing stopping you from putting one pa putting two pages on one domain and three on another, right?
It's not a volume game, right? You don't have to produce 100 pages a month to launch a satellite domain. A satellite domain is not going to rank for the whole breadth of things that a that your branded domain is going to rank for. They're going to rank for everything related to the domain. And so you probably only need to copy 10% of the content over.

28:26 Mhm.
I producing another take on the same content I don't think dilutes diminishes the value or quality of the content. It's just publishing it on another domain. And I
Well, it's just the volume of publishing, isn't it? That's what I was
Well, if you if you look at a lot of blogs, a lot of people produce blogs and don't actually go to check to see if their blogs are even performing, right? I mean, do you know how many of the last 20 blogs, for example, you published, are all 20 of them performing?
I mean, I don't know. I don't even know for my own blog, right?
No, I don't. So you could you could easily take 40% of your blog posts from the past year, two years, and say these aren't working and copy them over to your other blog, right? Because it's probably a structural difference difficulty. So I think that the that setting up an EMD is maybe like a 50page project. It's not creating 10 a week. I think it's it's it's pretty easy and you can again you just have to leverage content you already have. It's just another take. The challenge, of course, is getting authority, but you can get authority from your own domain. You can link your own domain to an EMD. And we've already got two great questions in in from the community on the left hand side that I want to get to.

29:47 But the the whole point of EMDs is that they rank. They're easier to rank. And so, let's let's answer Josh.com's first question. And hey David, I'd love to hear your thoughts on if EMDs are viable for local SEO. There's a great question. Absolutely. I think that both the way the map pack works and the way your organic desktop ranking like outside of the map pack support ranking in the map pack EMDs work very well. And if you've already got a branded domain, for example, you've got you know, Josh joshesplumbing.com and you're based out of down in Austin, Texas, you could always get Austin plumbers.com and link to that from your branded domain and get that to co-ank with you. And I think that's a very good strategy. But yeah, EMDs work very well. You can if you if you search for SEO New York City for example about it despite the number of reviews that a lot of the big agencies have there are a lot of EMD domain.
Can you explain what EMD is?

31:01 Yeah, EMD is an means exact match domain and effectively the way Google knows that your domain matches your brand is because the word in the domain matches your branded search. Right? It's it's it's as old as it's as old as Google. And so people noticed if you if you bought if let's say you were trying to rank for something like best SEO books online, if you bought the domain name best SEO bookonline.com, it would automatically rank first. And it got so popular that EMD became an entire industry of its own within SEO. And so Google did a lot of work to devalue the almost immediate and guaranteed effect of EMDs, and they say they brought it down to null. I'd say they've brought it down by about 30% at best. Because if they do bring it down anymore, then when people search for Nike, Nike.com isn't going to show anymore. And so EMDs, ask any SEOs like Charles Floate, even Edward, you know, we're all in agreement that EMDs can work very well. And so I think they do happen if, I think

Should you block AI bots?

32:14 They do work very well in local search as well.
Well, let's go on to the next question. For AI bots, should we be splitting them into robots.txt? Stephen, it's Deon, if you want to give us more context. Unless the bot is causing any downtime or it's putting a lot of load on your servers. I wouldn't effectively most of the AI bots are you've got to decide are they fetching for training which I'm not sure if if that's actually how training works and a lot of them are fetching live or near live. So in other words, your site's being returned in a query fan out. They're coming to the page to fetch the page. If they can't get that page, they can't synthesize you into the results. Or if your if your page is timing out, I I would just leave them out of your robots.txt. For me, robots.txt is just about blocking bots. And so the only the only use case I can think of is that you would be blocking them because they're causing load in terms of whether or not they're used for training.

33:33 Most of the training happens offline, right? They're given a corpus of content. They don't go browsing the web for that corpus of content unless there's an element now that goes and browses it and brings it offline for the next version to be trained, but that's not going to be available for a while. Training to me is giving them word patterns, right? Like how do we use words in in different sequence of each other? Not do we learn about your brand and go hunting for your brand. So I don't block any or allow any. I just allow all of the bots. I was using Cloudflare for a while and I found that it blocked so many of the AI bots that I turned it off. Even though I went into the settings and I try to say allow AI bots. I don't see any benefit for marketing sites to turn off AI bots. It's not going to stop. It's not going to stop them. It's not going to stop the AIOS. It's just going to exclude you. You're not I don't give any proprietary information

Outdated crawl budget advice

34:34 Away from any clients. So, I don't see any reason to block them.
We got some more questions, but I think it's best if we leave that to the end after the podcast part of the show and then we can answer them then. So, let's just finish off. So what so what technical SEO advice is outdated because it presumes assumes a simple literal slider crawling. So what are some of the key things you think are totally outdated?
That's a great question. I think the idea that you can somehow get bots to crawl pages or that you can you can you know that you've got a crawl budget that you can manage or that if you have less pages the bot spends more time on the pages you really want to concentrate on. It just doesn't work that way. Pages are first of all cued by importance. And so if you've got a page that's very low in the importance drive, even if all of your [clears throat] pages on your site link to it, if they're not able to transfer authority and the page just doesn't get clicks, it's just not going to matter, right? You need to understand that authority is a separate concept. And yes, it does travel by links, but you have to understand the limitations that come with authority. They're in the original PageRank patent. You can you can ask any LLM about how internal working links. You also have a link click probability score, right? So Google will look at a page and say let's say Jonathan has got a link to all of his favorite podcasters, all of the his favorite people, his partners. So, he links to me, he links to Kurt, links to Edward. But let's just say you put me at the very bottom of the page, right?

36:30 You're just, you know, at the footer, and you kind of have the link, it's not underlined. It's it's it's a link, but it's not very visible. When Googlebots rendering that, it can see that link isn't visible. And so, it will reduce the likelihood of or probability of someone clicking on that link, and that will reduce the amount of authority that passes through. [clears throat] So, the idea that you can just sort of like build a website with an infinite number of internal links and that internal links help the spider understand your site, it's a proxy for what's really happening and it's a misunderstanding. And so you really can't just look at like, well, the I've got the crawlers coming to the pages. If the crawlers accessing the pages, the crawlers like the pages. It's just a very naive way of looking at things. It's not very realistic, right? But a lot of people on Reddit and LinkedIn continue to talk this way, you know, and I don't know if it's web engineers that watch the video and have just decided that's accurate and it's never going to change, but I think web engineers and SEOs really need to start reading about some of this content that's all, like I said, the problem with it is it's all over the place, right? Some of Gary Illyes's comments are in LinkedIn, some of them are in Reddit discussions, some of them are in search engine round table. They are split level, but you can go to an LLM and like Claude and say, "Look, go and find all of the cases where Gary or Martin Splitt or John Mueller are talking about how crawling works that it's not a one and done that it's done on priority of pages. So if you've got 100 pages, 10 pages might be getting crawled every hour. Other pages might have thousands of zero value links across the web.

38:15 They're going to be put in a queue that only allows them to be crawled once a month. I think another big myth is that Google wants to save money. Google wants to find every single page out there and they will crawl the same page a 100 times. Doesn't mean that they fetch it or that they process it or index it, but they want to make sure there's no stone unturned.
Yeah. But what's you know you know it's happened to a lot of my content or happened to a lot of the newer content is my domain authority reduced index crawled but not indexed people are seeing that a lot you know crawled but not indexed.
So let's talk about why that's happening. So I think you'll agree that the for almost every page there is a lot of pages right so for example if you wanted to I know WP tonic does a lot of hosting for communities and colleges schools things like that right and that's a very important area like there are lots of universities and right now universities in America are facing lower [snorts] enrollments right I believe the University of Syracuse is almost facing financial crisis. So they have to go out and market themselves better, right? So they now need to get a company like you to help them with their SEO and their content strategy.

39:40 But at the same time, you know, you can't go and rank for WordPress hosting, right? You know, it's not going to happen.
And the reason for that is because there are so many bigger people, right? There's who's up there? You got WP Engine. If you're looking at the quality ones, obviously the cheaper end, you got GoDaddy, you got a whole host of the cheaper ones, but it's it's around Kinsta, WP Engine. Yeah, there's about three or four. Yeah.
Okay. So, from Google's perspective, it's not about saving money on the index because the these Googlebots are running 24/7, right? They're they're running they're calling the same pages all the time. But if your index already has 100,000 or a million different companies vying for WordPress hosting, right, and you've got your authority is like one or two or three or 10 or 100 and Kinsta is 100 million and Godaddyy's 100 billion, there's no point in letting you into that index because you're going to be at the bottom.
Yeah. And so Google said a few years ago, we're going to lower the barrier of entry from what you need from external backlinks in order to enter our index.

40:59 And so it opened up more cornerstoneing opportunities. It opened up zero keyword difficulty index indexes that are easy to enter. The problem is that a lot of these pages that you have and other people have are in indexes where they've just started losing clicks or Google's been harsher about reading the pogo sticking data or you've also lost other backlinks and other pages that were linking to you also lost traffic including internal pages, right? Especially if your traffic looks flat or your traffic looks like it's going down by like 1%. That means your all of your pages are losing traffic. That means they're passing less authority and so it's just all winding down, right? And so you have to find a way of stemming it. You have to find new content. You have to realign your old pages. You have to find new backlinks. You have to find did you delete a page that used to have a backlink. You've got to go back and and look at the fundamental mechanics. Are you getting authority? Is it landing on a page or is it landing on a broken page? If it lands on a broken page, it's gone. So you've got to put the page back. You can put in a 301, but it's not as ideal. I would try and resurrect the page if it's got a lot of back links and just change the content, right? You can dilute authority by having too many internal links. And so if you're seeing crawled not index is if it's from new content, it's because you have a topical authority issue.

42:21 Either they're outside of your topical authority or you're not linking to them from pages with authority. And so at that point, if they're if they're actually on topic, then you're either targeting a keyword that's out of your viable topical authority or you've got to go back and fix the fun. There's no point in producing more content anymore, right? You can just stop right there. And what you need to do is like go back and undo how you got written down and and start [clears throat] building for more traffic unless you're getting a million clicks a day. If you if you were somebody getting 10,000 ticks a day, there are other keywords out there, right?
All right, Tio, I think we're we're going to wrap up the podcast part of the show, but we're going to stay on and answer some of the questions that come through. You can go to at the present moment, you can go to the WP Tonic YouTube channel, join us live around 12:00 noon Eastern time every Friday and we'll be around to answer your questions. We're probably going to be moving that at some stage, but at the present moment. But so David, what's the best way for people to find out more about you and what your thoughts and what you're up to?

43:42 So I like I'll be hanging out here with you every Friday, which I'm very excited for. There's also X. And there's LinkedIn. X is probably the best place. I know X doesn't sound like the greatest place, but the SEO community is very large in X and it's very active in X.
I think it depends on how you're using X. I've got lists and I have people
And I just don't generally go on the general stream because it's just not my it's just not my cup of tea. So,
But I think for I think you'll agree with me, the SEO community on X is very positive. It's very helpful. There's a lot of good sharing. I've got I've made some great friends from your show, from Edward's show, and from X.
I think there's a lot of people sharing. There's a person on X. I'm going to have to look up his name because I'm going to butcher. I feel embarrassed even.
Never. Sudan Mahunden who helped me so much discover to discover my knowledge panel and my creators panel he doesn't gatekeep let me read out his ex maybe you can share his ex handle for me can I can I put it in chat
Yeah sure
What I'll do is I'll send it private chat. Can I
Can I send it?

45:15 How do I chat here?
Good question. I'll private chat. It's right at the bottom near I don't show in your window. I'm seeing your yours. Oh, I think we have to sort that out. Actually, you caught me there, David. But we do that. So can we go on to the comments
Of well sorry I just wanted to say that if you if you if you come and follow Sigan Gavin Gotra Edward
[clears throat]
Jonathan Denwood you'll find a good community of ACOs to ask and people to test and trial. I think Charles Floate does an amazing job of challenging traditional thinking. There's a lot of thinking about SEO because it is reverse engineered that a lot of people have confirmation bias and I think it's really good to undo that and test the reverse a sometimes.
That's fantastic. And if you want to contact me, it's really easy. Just go to X and look up Jonathan Denwood or LinkedIn at Jonathan Denwood or we're going to wrap up the podcast part of the show, but we're going to continue and we're going to be answering some of these questions that have come through as we were doing the live recording of the show. We'll see you next Friday, folks. Bye.

46:43 So, let's have a look at this. Is the is it dangerous if EMDs share the same reviews and portfolio work as a branded site?
So, I don't think so. But I think it is I think it is I think it breaks Google service if you set up a different Google profile. And I think it's if you present the EMD as a separate corporation that might be problematic. But I don't think Google will care about another domain talking about your reviews. There are limitations on the aggregated review schema and how you can use it. And like I'm not sure you're allowed to use reviews calculated from other websites in aggregated schema on your site. But yes, in generally speaking, you can relist the same portfolio work, the same customer reviews as text on your site if they're on Google reviews or if they're on Clutch or whatever based on their own. But yeah, I don't think it's dangerous for the EMD, especially
The way I'm talking about EMDs in this case is that it's an extension of your brand. It's not trying to look like a third party and produce misinformation.

48:03 Yeah, I'm talking about these EMDs as an extension like you'd have a different store, right?
Oh, right. Are you blocking one and allowing the other? I don't totally follow that.
I think we covered that already. I think I jumped into that one first. That was to say I'm not blocking any and I don't think you should.
All right, there we go.
Thank you, Stefan.
Big fan.
I appreciate that. Thank you so much. So Stephan again so if crawling priority is cued then what's the fastest way to move small side pages up the queue?
Okay good question. So in the beginning you can obviously jump you can leap frog by using the manual inspection and index request right which a lot of people do. There's indexing services which Google's not terribly fond of and I understand it. I get it that people use them. I think it's it's okay on the stepping stone, but what you really want to do is build out. So, what you normally do with cornerstoneing and Jonathan, this is an area we're working on, right? Where we're looking at keywords for your businesses based on the lowest keyword difficulty score, right? So, WordPress hosting very difficult, right?

49:22 But WordPress hosting for students and teachers
Much lower, right? It's I think it's an 11.
So if you start building your content around that and then link them from your homepage, so your homepage is getting clicks and you link to it, that's how you get them up. So you as you build your second page and you're going to build out your next tier, you're going to link to that page from that.
Yeah. But I think and I' I'm presuming this is the skill one of the fundamental skills of a SEO expert or somebody that really knows. But I might be totally wrong here. But isn't there always a strain between topical authority and cannibalization? You know, because if it's too focused, you just you start and there's a certain group of keywords or phrases that you might be able to compete with or you know, I might be talking out my backside. You could say to me there's so many things to discuss but you know the pages are going to start repeating themselves you know but I suppose there's there's methodologies that you could reduce that but isn't there always this strain between topical authority and cannibalization that Google looks at your pages and said well you've you've covered this five times and this new page it's got a little bit of texture stuff on it. But I suppose your argument is that they're you know they can see they can't actually see that but they can see the after effect because the after effect is somebody hits a link goes to another page and it's basically just covering what the page they've just come from they're going to bounce. So Google just sees the after effects. It sees the you know like people unless they're producing enormous amount of AI this is what I fundamentally have learned from you over the months of our discussions is unless they're producing hundreds thousands of AI pages and producing really strong patterns and signals Google's technology will see that and you'll get hammered in the end you'll get mount AI basically ally you get an enormous jump in clicks and then whack you get whacked but you got to produce a ton of content right but what happens to other people is yeah they can use AI but it's not very good AI and it's not it's not producing so they Google sees the after effects of people bouncing and not engaging and not going to other pages.

52:28 That's what Google's seen. And what I've got the drift right.
Yes. Yes. And you can see it in your clickthrough rate, right? Like if you go to search console and you look at a particular page, you'll see if you if you limit the number if you limit the rank positions of the keywords, right? So you use the filter at the bottom and you limit everything from one to four, you will see that keywords at the exact same position, some keywords will have half the clickthrough rate. And that's literally measuring people searching, going on your site, going back and searching again, and clicking on another result. And that's what's the that's the old way of measuring it. A lot of people think that Google collects data from Chrome, let's not enter that debate now.
But I think where you started off earlier in terms of like building out your content calendar cannibalization and then train.
Yeah. So, let's let's let's park scaled content because scaled content is a very
Yeah, you've got to go hell for leather. And I know you're worried about it, but
Well, a lot of people will be worried.
A lot of people are worried about I I see a lot of scaled content on X and Reddit where people go, "Oh, I found this engine that does SEO on autopilot." And it's not. It's just

Cannibalization explained

53:39 You're publishing over, right? So the cannibalized content is because of a particular issue where Google's not good at dealing with two URLs in the same index. It's just terrible. It always has been. Matt Cutts called it the duplicate content issue. It's why there's a duplicate content warning in search console for no other reason. And so if you if you had a page, let's just go back to your example, best hosting for schools and teachers, and then you built a page best, let's call it hosting for teachers and students, and then another page student and teacher hosting. That those three slugs, those three document names are all going to cannibalize with each other because they're all going to fit in the same index, right? And they're going to come back in the same results. The next most common area of cannibalization, especially for local SEO, is
I'm sorry, I didn't follow that. Are you
Sign basically where the document covers exactly the same thing? So having a document called best student and teacher hosting.
Yeah. And student and teacher hosting are the same thing because the word best doesn't the best isn't is a descriptive word.

54:55 Yeah.
Okay.
I've got it there.
It's easier to understand in local SEO. Let's say you build a site called John's plumbers in Nevada. Right.
So, I just want to really recap because I haven't got your knowledge level, but I think I've got it now. What you're saying is that the page titles, the slug and the ul, it's just too similar. It's got a slight change, but the meaning's about roughly the same.
Exactly.
So, Google's just going to say, "Well, you're covering the same stuff basically."
No, no, no. Google's not Google doesn't do that at all. Google puts them both in the same That's what I'm saying. Google's not a content appreciation engine. It doesn't weigh in on content. It lets users weigh in on content.
Yes.
Yeah. If you build if you build a web page on your site called word WordPress oneword hosting and then another page called WP hosting. WP and WordPress are semantically the same keyword in Google's universe and have been forever. Those two document names will both go into the same index. They'll both go into WordPress hosting and they will both go into WP hosting and they will block each other in the results. And that's cannibalization in its purest form. It's very common in service businesses. For example, if you do landscaping NYC New York and then landscaping New York and then landscaping New York City, those three pages will cannibalize because Google can't tell the difference between New York City and New York State. And so you see it in law as well. You'll see people putting in a personal injury page and then a personal injury page for New York City and then a personal injury page for NYC. Those three will just eventually start to cannibalize each other. Or if your homepage is ranking second or third place for something and you try to build a more detailed, more focused page, you can't really shift that from your homepage to these pages.

56:53 And so if you're lucky, it just won't rank. And if you're unlucky, it will cannibalize because it'll go into rotation testing. So, but for what I'm talking about in cornerstone, if you're building pages like WordPress hosting for college, WordPress hosting for community, WordPress hosting for schools, WordPress templates, WordPress design, WordPress tech support, those have very little chance of cannibalizing each other. And you're you're okay. You're going to have to explain that one more go, but not on this show. But we're going to wrap it up now, folks. Thanks for joining us live. Like I say, we'll be back next Friday around 12 Eastern time. We do the podcast and then anybody joins us live. We'll answer your questions after we've done the podcast part of the recording. And we'll see you soon, folks. Bye.