Categories
Magento

How to set robots.txt in Magento

Introduction

This guide explains how to set robots.txt in Magento so that it uses the recommended default options. This will allow search engines to crawl the site efficiently, and will help it to show up in search results.

Requirements

  1. Magento Admin account with access to the Design section.

Skill Level

Basic knowledge of Magento.

Instructions

Time needed: 5 minutes.

  1. Log into Magento Admin.

  2. Go to the Design Configuration page.

    In the main menu go to Content > Design > Configuration.Magento Content and Configuration menu options - screenshot

  3. Edit the Configuration.

    For to the website / scope you want to edit, click Edit. This would usually be for the first Global option if you have a single website.Design Configuration Edit button - screenshot

  4. Edit the Search Engine Robots details.

    Click the Search Engine Robots section.
    In the Default Robots drop down, select the relevant option. For most sites which are available to the public, INDEX, FOLLOW is the best option.
    Click the Reset to Defaults button. This applies the settings that are recommended by Magento. This will not remove the sitemap link if you have set this separately.
    Click Save Configuration.Search Engine Robots options - screenshot

More Info

Guide to Search Engine Robots: https://docs.magento.com/user-guide/marketing/search-engine-robots.html