Growing a newly launched website from scratch through search engines like Google can be a challenging process. However, thanks to AI-powered search engines (such as ChatGPT, OpenAI Search, etc.), new and intriguing methods are emerging to boost a website’s visibility. I decided to approach this as a case study for my own website and created a GPT Search Index Request.
Starting Point: Analyzing Log Files
My website was still very new and was not yet receiving significant traffic from Google. I regularly analyzed my log files to monitor which search engines were visiting my site and to track the behavior of bots.
In my initial examinations, I noticed that the AI crawler bots from OpenAI (AI Crawler) — specifically GPTBot, OAI-SearchBot, and ChatGPT-User — had not visited my website at all. As clearly shown in my first image, there was no activity from OpenAI’s side.
Experimental Efforts and My First Steps
After thoroughly reviewing OpenAI’s documentation (https://platform.openai.com/docs/gptbot), I decided to take action. Even though my site was not yet blocked, I believed that explicitly granting permission for these three bots on my site would facilitate their crawling.
Therefore, I began by explicitly allowing the following user-agents in my robots.txt file:
- GPTBot
- OAI-SearchBot
- ChatGPT-User
Additionally, I whitelisted the IP addresses specified by OpenAI directly via my .htaccess file. You can access the IP list here: https://platform.openai.com/docs/gptbot.
Afterwards, I provided my website to OpenAI using ChatGPT and requested a comprehensive review through my sitemap. I also included a brief that detailed the quality of my content, the concept, and the significance of my site while it was being examined.
Results: OpenAI Bots Discovering My Site
Following these experiments, the results came in very quickly. When I rechecked the traffic in my log files (as you can see in the second image), my website was being thoroughly crawled by ChatGPT-User and OAI-SearchBot.
This increase in bot traffic was a very promising outcome.
To measure the effect more clearly, I conducted tests from different accounts and IP addresses. In these tests, when performing queries related to my site’s focus — for example, searches like “Who is Tolunay Togul?” — I observed that my site’s content was being referenced by ChatGPT and that it provided very detailed answers.
Early Branding is Possible with AI Search
This study clearly demonstrated to me that even a newly established website that does not yet receive significant traffic from Google can build its brand on AI-based search engines. Although there are not yet fixed criteria for standing out in AI search engines, these deliberate optimizations can significantly boost a site’s visibility.
Considering that a significant portion of search behavior and traffic is likely to shift to AI-based tools in the future—especially in conjunction with GEO—I strongly recommend that new websites engage in such efforts and actively monitor bot traffic through log file analysis.
