Hey @angeluisfc,
That API endpoint is used to check what the current version of the site is and is required for the plug-in to work. I’m not sure i understand what you mean by, “After checking the logs files, I’ve seen Googlebot spends most of the time on the “wp-admin” folder with those URLs.”. Do you mind explaining the problem you’re experiencing in more detail?
Hi @jordanleven
Thanks for the answer. Wouldn’t that work of the plugin causing any kind of problem on the server?
Regarding Googlebot. I’ve seen that the main urls Googlebot are crawlings are those generared by the plugin instead of crawling the main URLs of the site (html).
@angeluisfc,
Yes, since this request for the current page version can be taxing in the server, request intervals max out at one every 30 seconds per page. If your server is having difficulty keeping up with the requests for larger sites, this interval as be adjusted up to once every two minutes.
As far as the crawling goes, I believe that the Ajax and is required to be allowed by bots. You can always adjust your own robots.txt file if you’d like.
https://wordpress.stackexchange.com/questions/403753/why-does-do-robots-allow-wp-admin-admin-ajax-php-by-default
Hi @jordanleven
Then it happens as I thought. Thanks for the confirmation. Maybe I could adjust the interval, as you said. However, I think I’ll try without it.
I know about robots.txt and admin-ajax… which is why I noticed the issue with Googlebot. And that’s why I think could be worth it to mention it (that Googlebot is spending a lot of your crawling budget on the admin-ajax pages).
Anyway, thanks for your help, and for the link.
