Introduction
This guide explains how to set robots.txt in Magento so that it uses the recommended default options. This will allow search engines to crawl the site efficiently, and will help it to show up in search results.
Requirements
- Magento Admin account with access to the Design section.
Skill Level
Basic knowledge of Magento.
Instructions
Time needed: 5 minutes.
- Log into Magento Admin.
- Go to the Design Configuration page.
In the main menu go to Content > Design > Configuration.
- Edit the Configuration.
For to the website / scope you want to edit, click Edit. This would usually be for the first Global option if you have a single website.
- Edit the Search Engine Robots details.
Click the Search Engine Robots section.
In the Default Robots drop down, select the relevant option. For most sites which are available to the public, INDEX, FOLLOW is the best option.
Click the Reset to Defaults button. This applies the settings that are recommended by Magento. This will not remove the sitemap link if you have set this separately.
Click Save Configuration.
More Info
Guide to Search Engine Robots: https://docs.magento.com/user-guide/marketing/search-engine-robots.html