Just launched a dynamic website? Kudos to you! It could be your best sales move yet, presenting your best face to online prospects and convincing them to buy into your story.
How will your prospects visit your website if it’s nowhere in the search results?
Understanding the process
When a Googlebot visits a web page, it first checks whether you’ve allowed crawling by reading the robots.txt file or the meta robots tag. If this file carries instructions to skip the page, Googlebot will abort the task.
The classic bot approach of making an HTTP request and fetching URL works efficiently for server-side rendered pages.
The bot would then parse the rendered HTML for links and queue the URLs discovered for crawling. Rendered HTML is used for indexing the page.
Now let’s find out how we can optimize web pages for search engines:
1. Recheck robots.txt and meta robots tag
Before introducing the website, check your robots.txt file and meta robots tag to make sure someone hasn’t blocked the bots accidently.
If you’re not sure when to use the robots.txt file and meta robots tags, here is a tip. The .txt file works best for blocking an entire section of websites, while meta tags are used for disallowing particular pages.
All major search engines like Google, Yahoo!, Bing, Baidu, and Yandex support robots meta tag values like index, noindex, follow, nofollow, etc.
You may also issue different instructions to various search engines. For this purpose, you may put in the value of the content attribute to the intended search engine such as Googlebot.
Be careful as search engine bots will simply ignore instructions that they’re unable to understand.
2. Use HTTP status codes to interact with bots
Bots use HTTP status codes to determine what has gone wrong during the crawling process.
Use a meaningful status code to convey to bots if a certain web page should not be crawled or indexed, such as a 401 code for pages requiring a login.
HTTP status codes can also inform bots if a certain page has been shifted to a new URL, instructing it to index the page accordingly.
Here’s a list of meanings of various HTTP codes for your reference:
- 401/403: Page unavailable due to permission issues
- 301/302: Page now available at a new URL
- 404/410: Page unavailable now
- 5xx:Issues on the server-side
3. Ensure your code is compatible
Your SEO professionals should be aware of the limitations of these bots regarding various features. At the least, your code should be written keeping in mind the limitations of major search engines like Google and Bing.
There is always some difference between your perception and what the crawlers are accessing and rendering with respect to your content.
To find out how Googlebot is viewing your page, you could use the URL inspection tool in Search Console.
4. Use unique titles and snippets
- Every page on your site should have a specific title.
- Make page titles descriptive and concise.
- Avoid boilerplate titles such as ‘cheap products for sale’.
- Avoid any inkling of keyword stuffing.
- Use titles to provide some additional information about your website.
The Google algorithm creates snippets directly from the page content. The search engine may show different content for different searches, depending on the user’s search.
There are two techniques to suggest content for these snippets—structured data and meta description tags.
Structured data [See above image] will help Google understand the page better. It may also use meta tags for presenting a snippet if it decides appropriate content is available there.
5. Try lazy-loading of images
Images take a toll on your bandwidth and performance. A solution to this issue comes in the form of lazy-loading which involves loading images later and that too in a search-friendly manner.
If you favor an infinite scroll experience for your viewers, implement a mechanism to support paginated loading. It enables your users to share and re-engage with your content. Use the History API to update the URL when your system facilitates the dynamic loading of content.
1. Expecting a bot to index content requiring user permission
Googlebot will ignore any content that requires users’ consent. Googlebot and its Web Rendering Service (WRS) clear local storage, HTTP cookies, and session storage data across page loads, so you cannot rely on data persistence to serve content.
Googlebot doesn’t support WebGL. If you’re using WebGL to render photo effects in the browser, deploy server-side rendering to pre-render the photo effects.
2. Struggling to ensure web components are search-friendly
You may resolve this by putting your content into light DOM (Document Object Model) whenever possible.
Light DOM resides outside the component's shadow DOM and the markup is written by a user of your component.
Shadow DOM defines the component’s internal structure, scoped CSS, and encapsulates the implementation details.
For example, when putting a new HTML id/class to use, it may cause conflict with an existing element on the page.
There may be unexpected bugs, style selectors may go awry, and CSS specificity may become a problem. Using light and shadow DOMs will help you take care of these issues.
3. Not testing websites for mobile-friendly features
Mobile-friendliness is a key feature of successful websites, yet many fail to test it before release.
Google offers a website where you can paste the URL in the provided space, click ‘Analyze,’ and wait for a few seconds to find out how mobile-friendly your site is.
You may also use mobile-friendly test tools offered by independent service providers.
As the site owner, eventually it is up to you to ensure that your web pages are SEO-friendly so that you can derive maximum mileage out of your investment.
A string of tools are available to assist you in your goal of creating a website that search engines love. That said, your SEO professional’s skills and experience play a big role as well.