How To Do Keyword Research for SEO on a large scale



Enterprise SEO is different from a regular seo in a way that every step you follow in your seo process is scalable and is largely automated to reduce the turnaround time (time required to do seo and produce results) of a seo project.   By the time you will end up reading this post, you should be able to do competitive keyword research on a website with over 1 million web pages in less than 60 minutes or so. So here we go:


Step1: Scan your competitor’s website using Xenu link sleuth.

Xenu’s Link Sleuth is a free tool primarily used to check websites for broken links but it can also be used for many other things like scanning the title tags of your competitors and this is what we are going to do. Scan the title tags of the competitors as titles generally contain the keywords your competitor is targeting.

Speed up your crawling

Speed up your crawling

Xenu can crawl over 1 million URLs. But you would probably won’t like to wait for hours before the crawling finally stops. So if you want to speed up crawling then deselect the ‘check external links’ check box (to skip crawling of external links) and skip the images, CSS and Java Script files from being crawled by adding their folders in the box as shown above. Feel free to add as many folders, categories, or sections of competitor’s site you don’t want to be crawled by Xenu. Then click on the ‘more options’ button and uncheck all the check boxes as shown in the image above. The point here is, don’t waste your time crawling hundred and thousands of image files (unless you are esp. crawling for image optimization purpose), CSS files, Java script files and irrelevant sections of the site (like forum posts) which are not used by competitors to target keywords. All this will speed up crawling and save your precious time.


Step-2: Export the crawled data into excel and apply filters

Once you have crawled the website and have got the report then select ‘Export to TAB-separated File…’ from the Xenu’s ‘File’ menu. Open the file in MS Excel and apply filters to filter out the data by ‘text/html’ as shown in the image below:

Step-3: Export the data into a notepad

Copy and paste the title tag column from excel into a notepad file (.txt) to remove formatting. Remove all the brand terms (if any) from your title tags column data by using ‘Replace’ command (CTRL-H) in notepad. Your title tags column data now should contain a list of all the keywords your competitor is targeting. Keep a copy of this list as you will need it after some time.

Step-4: Export the data into your favorite keyword research tool

Copy and past the first 100 keywords into your favorite keyword research too like Google Adwords Keyword tool. You can export only 100 keywords at a time to Google keyword tool. If you want to remove this limitation then use Google Adwords API. Click on the check box ‘only show ideas closely related to my search terms’ before you get the keyword data from Google Adwords keyword tool.

Step-5: Download you keyword research data into excel and filter out the right data

Download the keyword research data from Google Keyword tool into excel. Remove all those keywords with less than 100 monthly search volume (or whatever you prefer). Create a new excel sheet and copy paste the original title tag list into it.


Go to the ‘original title tag list’ sheet as shown in the image above. Add a new column named ‘Global monthly search volume’ (or local monthly search volume; whatever you prefer). Apply ‘vlookup’ to get search volume date for your original keyword list. I am interested only in the keywords my competitor is targeting. That’s why i have gone through so much trouble.

Now you have the list of all the keywords your competitor is targeting through title tags and you also have the search volume data of each keyword.

Step-6: Group your keywords into relevant categories

Just getting a huge list of keywords with their search volume data won’t help you much if you can’t categorize it into relevant groups. Again categorizing hundred and thousands of keywords is a very time consuming process. If you do it manually it may take several weeks. Here wordstream keyword grouper tool comes handy. This tool is free to use and can group a list of up to 10000 keywords (along with their search volume data) into relevant categories in few seconds. To use this tool just copy and paste the list of keywords into the wordstream keyword grouper box and click on the ‘Get Profitable Keywords Group’ button. You will get groups something like this:


Click on the ’email all groups’ button and then download all the keyword groups on your hard disk. Through these groups you can now decide the content categories/topics to write on which can help you in targeting the competitor’s keywords. This post is just a glimpse of how seo tasks are managed at the enterprise level.

Related PostKeyword Research for e-commerce Websites – Tutorial


Do you know the difference between Web Analytics and Google Analytics?

99.99% of course creators themselves don’t know the difference between Web analytics, Google Analytics (GA) and Google Tag Manager (GTM).

So they are teaching GA and GTM in the name of teaching Web analytics.

They just copy each other. Monkey see, monkey do.

But Web analytics is not about GA, GTM.

It is about analyzing and interpreting data, setting up goals, strategies and KPIs.

It’s about creating strategic roadmap for your business.

Web Analytics is the core skill. Google Analytics is just a tool used to implement ‘Web Analytics’.

You can also implement ‘Web analytics’ via other tools like ‘adobe analytics’, ‘kissmetrics’ etc.

Using Google Analytics without the good understanding of ‘Web analytics’ is like driving around in a car, in a big city without understanding the traffic rules and road signs.

You are either likely to end up somewhere other than your destination or you get involved in an accident.

You learn data analysis and interpretation from Web analytics and not from Google Analytics.

The direction in which your analysis will move, will determine the direction in which your marketing campaigns and eventually your company will move to get the highest possible return on investment.

You get that direction from ‘Web analytics’ and not from ‘Google Analytics’.

You learn to set up KPIs, strategies and measurement framework for your business from ‘Web analytics’ and not from ‘Google Analytics’.

So if you are taking a course only on 'Google Analytics’, you are learning to use one of the tools of ‘Web analytics’. You are not learning the ‘Web analytics’ itself.

Since any person can learn to use Google Analytics in couple of weeks, you do no get any competitive advantage in the marketplace just by knowing GA.

You need to know lot more than GA in order to work in Web analytics and marketing field.

So what I have done, if you are interested, is I have put together a completely free training that will teach you exactly how I have been able to leverage web/digital analytics to generate floods of news sales and customers and how you can literally copy what I have done to get similar results.

Here what You'll Learn On This FREE Web Class!

1) Why digital analytics is the key to online business success

2) The number 1 reason why most marketers are not able to scale their advertising and maximize sales.

3) Why Google and Facebook ads don’t work for most businesses & how to make them work.

4) Why you won’t get any competitive advantage in the marketplace just by knowing Google Analytics.

5) The number 1 reason why conversion optimization is not working for your business.

6) How to advertise on any marketing platform for FREE with an unlimited budget.

7) How to learn and master digital analytics and conversion optimization in record time.


My best selling books on Digital Analytics and Conversion Optimization

Maths and Stats for Web Analytics and Conversion Optimization
This expert guide will teach you how to leverage the knowledge of maths and statistics in order to accurately interpret data and take actions, which can quickly improve the bottom-line of your online business.

Master the Essentials of Email Marketing Analytics
This book focuses solely on the ‘analytics’ that power your email marketing optimization program and will help you dramatically reduce your cost per acquisition and increase marketing ROI by tracking the performance of the various KPIs and metrics used for email marketing.

Attribution Modelling in Google Analytics and Beyond
Attribution modelling is the process of determining the most effective marketing channels for investment. This book has been written to help you implement attribution modelling. It will teach you how to leverage the knowledge of attribution modelling in order to allocate marketing budget and understand buying behaviour.

Attribution Modelling in Google Ads and Facebook
This book has been written to help you implement attribution modelling in Google Ads (Google AdWords) and Facebook. It will teach you, how to leverage the knowledge of attribution modelling in order to understand the customer purchasing journey and determine the most effective marketing channels for investment.

Himanshu Sharma

Digital Marketing Consultant and Founder of

Himanshu helps business owners and marketing professionals in generating more sales and ROI by fixing their website tracking issues, helping them understand their true customers' purchase journey and helping them determine the most effective marketing channels for investment.

He has over 12 years of experience in digital analytics and digital marketing.

He was nominated for the Digital Analytics Association's Awards for Excellence. The Digital Analytics Association is a world-renowned not-for-profit association that helps organisations overcome the challenges of data acquisition and application.

He is the author of four best-selling books on analytics and conversion optimization:

error: Alert: Content is protected !!