Skip to main content

Add Website to Search Engines

This is WordPress Lite!

This page relates exclusively to WordPress Lite, a lightweight version of WordPress available between October 2024 and May 2025. It is no longer possible to create new WordPress Lite websites. We thank all WordPress Lite users and are happy to report that all feedback has been implemented and made available by default. This documentation will remain online and accessible until all WordPress Lite websites have been moved to the official WordPress Service.

Click here to return to the main WordPress documentation.

!!! info To manipulate files, such as creating/editing the "robots.txt" file, please refer to manipulate files documentation for more details on how to do it.

Make your website findable on search engines​

If you would like your website to be found through search engines, such as Google, Bing, Yahoo or others, the following steps must be configured to allow the websites to be indexed.

  1. Running the following command can be done to create or overwrite the robots.txt file with instructions to index the website but ignore the management pages:
echo "User-agent: *
Allow: /

Disallow: /trackback/
Disallow: /wp-admin/
Disallow: /wp-content/
Disallow: /wp-includes/" > /opt/app-root/src/robots.txt
  1. On the administrator panel, https://<your-website>.web.cern.ch/wp-admin, go to Settings > Reading and disable the button "Discourage search engines from indexing this site".

image-63.png

!!! info After setting correct instructions for the website to be crawled, it can still take some time for search engines to catch up. To accelerate the process, you can request specific engines to crawl the website again.

Hide your website from search engines​

If you would like your website to be hidden from search engines, create a "robots.txt" file.

  1. Create/Update the file "robots.txt" to guarantee the denial of crawling, all is required is to run the following command when acessing the website through Openshift console:
echo "User-agent: *
Disallow: /" > /opt/app-root/src/robots.txt
  1. On the administrator panel, https://<your-website>.web.cern.ch/wp-admin, go to Settings > Reading and enable the button "Discourage search engines from indexing this site".

image-63.png

With these two steps, the website should now be ignored by search engines and no longer indexed.