The big problem with using robots.txt to prevent being crawled is that it doesn't account for them crawling your site from exterior links. There are two basic ways to prevent this from happening.
Block All Incoming Links and Use the "Disallow" Tag in robots.txt
Blocking incoming links and using the disallow tag would make your website completely a fortified fortress on an island. Although effective, this isn't a good idea. No spiders are going to come to your site, but then neither is any traffic. The only rationale behind doing this would be if you don't want anyone other than people you've told about the site to find it.
Granted, there are markets for sites like this, but they're by and large illegal ones that the owners just don't want causal surfers to find. You really must have something you want to hide in your content to use this technique, in which case you'd be better off not having a website.
Besides that, do you have any idea how much work it would take to keep all inbound links from coming to your site? Let's just say that it would be more trouble than it's worth.
Make Your Landing Page a Login Page
Having the main landing page be a login page is the more sensible approach between the two. No spider is going to go past your landing page in this case. Again, it's shooting yourself in the foot by blocking the spiders here, unless you have some mighty fine SEO happening on this page.
The better option is to have at least one page that can be crawled. If you want traffic of any sort, this shouldn't come as a surprise. It's best if that one optimized page isn't a login page. There are two reasons for this judgment:
Login Pages Don’t Get Much Love From Search Engines
It's easier to optimize a page for the search engines if you don't have a gigantic login field sitting smack in the middle of it. With all the other stuff you could be putting on the page to make it rank well in the search engines, having a login field is just too much clutter. Instead, make your landing page into a squeeze page, and provide links to the login page.
You Have to Entice Traffic to Your Site
In order to effectively get traffic to your site, you have to entice them in. All too often, people think that ranking well on the search engines is all it takes to get traffic. While this is true, it's not the ultimate goal of building a website. You want to monetize it. That means getting them past the first page. You have to sell your service here, and then provide a login page after they've paid. It just makes sense.
Brandon Williams - I'm an experienced writer in several subjects. Over the years as an Internet content writer, I've worked with several online marketing ...
Join the Conversation