Make your website findable or not in Search engines

Info

To manipulate files, such as creating/editing the "robots.txt" file, please refer to manipulate files documentation for more details on how to do it.

Make your website findable on search engines

If you would like your website to be found through search engines, such as Google, Bing, Yahoo or others, the following steps must be configured to allow the websites to be indexed.

1. Running the following command can be done to create or overwrite the robots.txt file with instructions to index the website but ignore the management pages:

echo "User-agent: *
Allow: /

Disallow: /trackback/
Disallow: /wp-admin/
Disallow: /wp-content/
Disallow: /wp-includes/" > /opt/app-root/src/robots.txt

1. On the administrator panel, https://<your-website>.web.cern.ch/wp-admin, go to Settings > Reading and disable the button "Discourage search engines from indexing this site".

Info

After setting correct instructions for the website to be crawled, it can still take some time for search engines to catch up. To accelerate the process, you can request specific engines to crawl the website again.

Hide your website from search engines

If you would like your website to be hidden from search engines, create a "robots.txt" file.

1. Create/Update the file "robots.txt" to guarantee the denial of crawling, all is required is to run the following command when acessing the website through Openshift console:

echo "User-agent: *
Disallow: /" > /opt/app-root/src/robots.txt

1. On the administrator panel, https://<your-website>.web.cern.ch/wp-admin, go to Settings > Reading and enable the button "Discourage search engines from indexing this site".

With these two steps, the website should now be ignored by search engines and no longer indexed.