Some of the content came from another CMS
The site used several systems. Some content was managed in a headless CMS: it was used to prepare and supply content, but had no public website of its own. That content appeared on the e-commerce site.
In this setup, the page's JavaScript fetched data from the CMS and inserted it into the page.
For visitors, the content loaded quickly. Caching kept the display smooth, with no visible flash or empty area before the text appeared.
Diagnosing the issue in Search Console
To investigate the poor performance, I inspected the rendered pages in Search Console. The content in question was missing. The areas where it should have appeared were empty.
What made this surprising was that the pages were indexed and already appeared for some keywords, even though their rankings were poor. A page can, however, be indexed with incomplete content. Elements Google can access, such as the title tag, can provide clues about its subject even when the text loaded from the CMS is unavailable. Appearing in search results did not mean Google had access to all the content.
The inspection details in Search Console did show resources blocked by robots.txt. I knew the main site's file well: none of its rules could explain why these resources were blocked. I needed to look at which host served them and which rules applied there.
That changed the focus of the investigation. Before rewriting the content or trying to improve its rankings, I needed to understand why Google was not retrieving it.
Google can execute JavaScript, but rendering a page also requires access to the resources that supply its content. A fast loading experience for visitors does not establish whether Google can retrieve those resources. Google explains that it also uses the HTML produced by JavaScript rendering for indexing. The content needs to be present in that rendered HTML.
The block was on the CMS
The CMS was not intended to attract organic traffic and was not monitored in Search Console. I therefore checked its domain's robots.txt directly. It blocked Google, including access to the resources supplying data to the e-commerce pages.
The CMS had no public frontend to rank in search, so blocking it might have seemed harmless from an SEO perspective. But it still played a part in displaying the site: Google needed to fetch its content to complete the pages.
A visitor's browser could make those requests. Google had to respect the crawl restrictions. Caching and fast loading could not solve that access problem.
A page can be accessible while a resource it depends on is not. The rules in robots.txt apply to the host serving each resource, not just the address of the page you want to rank. That made the block less obvious to spot.
Allowing access without indexing the technical URLs
I removed the block that prevented Google from retrieving the content. I also added noindex directives to the relevant HTTP response headers using X-Robots-Tag.
X-Robots-Tag: noindexThere were two separate goals: let Google fetch the public resources the pages needed, and keep the CMS's technical URLs out of search results. The e-commerce pages themselves had to remain indexable.
The HTTP header sends that instruction without relying on a tag in an HTML page. But Google has to be able to access the response to read it.
What I took away from it
After the fix, organic traffic to the affected pages increased significantly.
The main lesson for me was to trace how the content reaches the page. In this kind of architecture, the investigation also needs to cover where the content comes from and what allows Google to retrieve it.
When part of a site has poor search visibility for no obvious reason, comparing the expected content with Google's rendered HTML is a practical place to start. The URL Inspection tool lets you examine that rendering and the resources it loaded. In this case, that check led me to the right place: the CMS feeding the pages.