Keyword Research: The Complete Guide

Here is the first thing I tell anyone who hands me a spreadsheet sorted by search volume. Close it. You are looking at the wrong column. I have spent 20 years in search, and the most expensive mistake I see is people falling in love with big numbers next to a keyword.

Volume tells you how many people type a phrase. It tells you nothing about why they typed it, whether they will buy, or whether you can win the click. A keyword with 50,000 monthly searches and zero buying intent is a parking lot. A keyword with 90 searches and a credit card already out is a cash register. "Emergency plumber open now near me" barely registers on volume, but that person is standing in two inches of water. If you sell plumbing, you want that one.

The volume trap, and how it bankrupts content budgets

Here is how it usually goes. A team pulls 2,000 keywords, sorts by volume, and writes 40 articles for the fattest terms. Six months later they have traffic and no revenue. The traffic is real. It just never wanted what they sell.

Nobody ever got paid for traffic. They got paid for the right traffic showing up at the right moment.Shmul

Every search is a person trying to get a job done. Your job is to figure out which job, because the format, the angle, and the page you build all flow from it. Get the intent wrong and you can do everything else perfectly and still lose.

There are four classic intent types, and I want you to actually internalize them, not just nod at them. Most people can recite the list and still map keywords wrong, because they read the keyword instead of reading the human behind it.

• Informational: the searcher wants to learn. "What is keyword cannibalization." They want an answer, not a sales pitch. Sell here and you lose.
• Navigational: the searcher already wants a specific brand or page. "Ahrefs login." You almost never win these unless you are the brand.
• Commercial investigation: the searcher is comparing before buying. "Best CRM for small business," "Semrush vs Ahrefs." Money is close. Reviews and comparisons win here.
• Transactional: the searcher is ready to act. "Buy standing desk," "hire SEO consultant." This is the bottom of the funnel and the closest to revenue.

The fifth one nobody puts on the slide: local and situational

"Near me," "open now," "in Austin," "same day." These ride on top of the other four and they change everything. A local transactional query is the most valuable real estate in search. The searcher has intent, urgency, and a location. If you serve a geography, this is where you live.

How do you confirm intent when you are not sure? Open an incognito tab and search it yourself. The results page is Google telling you, in plain sight, what it believes the intent is. If page one is all comparison posts, the intent is commercial, and your how-to guide will never rank there no matter how good it is. I cover reading the SERP more in my technical SEO guide.

Keywords are not strings of text. They are receipts of a moment when a human needed something badly enough to type it into a box. The jobs-to-be-done lens, borrowed from product thinking, is the most underrated tool in keyword research. It turns a flat list into a map of human need.

The framing is simple. People do not buy products, they hire them to make progress in a specific situation. Nobody wants a quarter-inch drill. They want a quarter-inch hole. The keyword is the drill. The job is the hole. Optimize for the job and the keyword takes care of itself.

Turn keywords into job statements

For every cluster of keywords, write a one-line job statement in the searcher's own words. It forces you to stop optimizing for a robot and start solving for a person. The structure: when [situation], I want to [motivation], so I can [outcome].

1. 1Take a keyword cluster, say everything around "project management software for agencies."
2. 2Write the job: 'When my agency keeps blowing deadlines, I want a tool my whole team will actually use, so I can stop losing clients to chaos.'
3. 3Notice what that reveals: the real pain is team adoption and lost clients, not features. Now your content has an angle nobody else has.
4. 4Build the page around the job, then weave the keywords in naturally. The keyword serves the human, not the other way around.

This is also the cheat code for standing out in a crowded space. Everyone writes the same feature comparison. Almost nobody writes to the actual job. When you name the situation and the fear out loud, the searcher feels understood, and that feeling is what makes them trust you. Trust is also the core of E-E-A-T, and it starts here, at the keyword.

A keyword is a person mid-sentence. Your job is to finish the sentence the way they would have.Shmul

If you are still doing one keyword, one article, you are playing a game that ended years ago. Search engines, and now AI engines, reward depth on a topic, not scattered pages chasing single phrases. The unit of modern SEO is not the keyword. It is the cluster.

A topic cluster is one broad pillar page that covers a subject at the strategic level, surrounded by a set of cluster pages that each go deep on one sub-question, all linked together. The pillar tells search engines what you are an authority on. The clusters prove it. The internal links pass the authority around.

How to build a pillar map

1. 1Pick a pillar topic broad enough to support 15 to 40 sub-pages but narrow enough that you can genuinely own it. 'Keyword research' is a pillar. 'Marketing' is a galaxy, too broad.
2. 2Brain-dump every sub-question, angle, and adjacent term. Pull from autocomplete, People Also Ask, forums, and your sales calls.
3. 3Group those into clusters by intent and sub-topic. Each cluster becomes one page.
4. 4Draw the links: every cluster page links up to the pillar and across to its closest siblings. The pillar links down to every cluster.
5. 5Find the gaps. Empty branches on your map are content you have not written yet, which is exactly your roadmap.

Clusters also kill cannibalization. When five of your pages all target nearly the same phrase, they compete with each other and split the signal. A clean cluster map gives every keyword exactly one home. If you suspect you already have overlap, an internal link audit usually surfaces it, which I touch on in internal linking and link building.

Every keyword sits somewhere on the path from 'I have a vague problem' to 'take my money.' If you only write bottom-funnel pages, you have no audience. If you only write top-funnel pages, you have an audience that never buys. The skill is balance, and the trick is knowing which keyword belongs where.

I think in three stages, because the customer does. Awareness, where they are naming the problem. Consideration, where they are weighing options. Decision, where they are choosing a provider. Each stage has its own keyword fingerprint and its own job.

• Awareness keywords: question-shaped, problem-shaped. 'Why is my website slow,' 'what is technical SEO.' Serve these with guides and explainers. You are earning trust, not asking for the sale.
• Consideration keywords: comparison and category shaped. 'Best site speed tools,' 'managed hosting vs VPS.' Serve with comparisons, roundups, and honest reviews.
• Decision keywords: provider and action shaped. 'Hire technical SEO consultant,' 'pricing.' Serve with service pages, demos, and clear calls to action.

The bridge that most people skip

The magic is not in writing each stage. It is in linking them. Your awareness article on 'why is my website slow' should send the reader to your consideration piece on tools, which sends them to your decision page on services. That internal path is a sales funnel disguised as helpful content. Most sites build the three stages and then forget to connect them, so the visitor reads one page and leaves. For example, an awareness piece on speed should hand off to Core Web Vitals and then to a service page.

People think keyword research is about owning the fanciest tool. It is not. The tools all pull from similar data and spit out similar numbers. The difference between an amateur and a pro is not the tool, it is whether they trust the numbers blindly or read them with a skeptical eye.

Use whatever you have. Google Keyword Planner is free and gives you ranges. Ahrefs, Semrush, and Moz give you richer data and competitor views. Google Search Console is the most honest source you own, because it shows the queries you already get impressions for. Even Google autocomplete and People Also Ask are research tools most people ignore.

How to read the numbers without fooling yourself

• Search volume is an estimate, not a fact. Treat it as a rough order of magnitude. The difference between 200 and 260 is noise. The difference between 200 and 20,000 is signal.
• Keyword difficulty scores are a starting opinion, not a verdict. They mostly measure backlinks to ranking pages and ignore intent match, content quality, and freshness. A 'hard' keyword with weak, outdated top results is often winnable.
• Cost per click is your secret intent signal. High CPC means advertisers pay real money for that click, which means it converts. A high-CPC keyword with modest volume is often gold.
• Trend direction matters more than the snapshot. A keyword climbing year over year beats a flat one with double the volume.

Here is the move almost nobody makes. Before you trust any difficulty score, manually inspect the top five results. Are they authority sites or random blogs? Are they actually answering the query well, or are they thin? Is the content two years stale? A tool gives you a number. Your eyes give you the truth. The tool says 'hard,' your eyes say 'these results are weak and old,' and your eyes win.

Two of the highest-return moves in keyword research are sitting in plain sight, and most people never run them properly. The first is the competitor gap. The second is the long tail. Both reward specificity, and both let a smaller player beat a bigger one.

Finding the competitor gap

A gap analysis is dead simple in concept. You take the keywords your competitors rank for, subtract the ones you already rank for, and what is left is your opportunity list, validated by the fact that someone in your space already proved those keywords drive traffic. Most tools have a gap or content-gap feature built in. Feed it two or three real competitors and yourself.

1. 1Pick three true competitors, the sites that actually outrank you for terms you want, not just big brands in your space.
2. 2Run the keyword gap report so it shows terms where they rank and you do not.
3. 3Filter that list by intent and relevance. Ignore keywords that do not match your business, no matter how juicy the volume.
4. 4Sort what remains by a mix of intent strength and how weak the ranking competitor's page looks. That is your priority queue.

The gap also tells you what to avoid. If every competitor ranks for a term and they all have hundreds of strong backlinks, that is not a gap, that is a wall. Spend your energy where the door is open. Backlinks still matter for the hard stuff, covered in my link building guide.

The long tail is where the leverage is

Long-tail keywords are the longer, more specific phrases with lower individual volume. 'Standing desk' is a head term, brutal to rank for, vague intent. 'Best standing desk for tall people under 500 dollars' is long tail, easy to rank for, and the person typing it is ready to buy. The specificity is the value, not a weakness.

Long-tail keywords are also your beachhead. You rank for the specific stuff first, build topical authority and links, and that authority lifts your ability to compete for the fatter head terms later. You earn the right to the big keywords by dominating the small ones. This is doubly true now, because AI engines love specific, well-answered questions, which is exactly where chapter eight comes in.

Here is the part that makes a lot of old-school keyword advice obsolete. People do not just type three-word fragments into a box anymore. They ask ChatGPT a full paragraph. They speak a question to their phone. They follow up. The keyword, as a tidy little phrase, is dissolving into natural conversation. If your research stops at phrases, you are researching for a search engine that is fading.

In AI search, the unit is the question and the entity, not the keyword string. When someone asks an AI engine 'what's the best project management tool for a five-person design agency that hates complicated software,' there is no single keyword to optimize for. There is a job, a set of entities, and a need for a clear, citable answer. The engine assembles a response from sources it trusts. Your goal shifts from ranking a page to becoming a source worth quoting.

Three new things to research

• Questions, in full natural language. Not 'pm software agency' but the whole spoken or typed question, including the constraints and the frustration baked into it.
• Entities, the named things an engine recognizes. Brands, products, concepts, people, places. AI engines reason over entities and their relationships, so naming them clearly and accurately matters more than keyword density ever did.
• Prompts, the actual phrasing people use when they talk to an AI. These are longer, more conversational, and full of context. Collect them like you used to collect keywords.

This is the heart of GEO, generative engine optimization, and it changes what a content page is for. You still do keyword research, but you extend it. For each cluster you ask: what full questions would someone ask an AI about this, and what entities must my content nail to be the obvious source? I go deep on the strategy in what is GEO, and on the tactics in how to get cited in ChatGPT.

Practically, this means structured, scannable content with direct answers near the top, clear definitions, and proper markup so machines can parse you. Schema helps engines understand your entities, covered in the schema markup guide. The same intent and cluster thinking still applies, you are just now also optimizing to be cited inside AI Overviews and to surface in Perplexity.

We spent 20 years optimizing for ten blue links. Now the box answers the question itself. Keyword research did not die, it grew up into question and entity research.Shmul

Theory is useless without a process you can run on a Tuesday when you have a deadline. So here is the exact workflow I run, condensed. It folds everything in this guide into a sequence you can repeat for any site, any niche, any time.

1. 1Seed. List your services, products, and the problems you solve in plain language. Pull your own Search Console queries. This is your starting universe, grounded in your actual business.
2. 2Expand. Run those seeds through your tools, autocomplete, People Also Ask, forums, and AI prompts. Collect phrases, questions, and the entities that keep showing up.
3. 3Classify by intent. Tag every term as informational, navigational, commercial, transactional, or local. Cut anything that does not move a real customer toward you, no matter the volume.
4. 4Write job statements. For each cluster, write the one-line job in the searcher's own words, situation, motivation, outcome.
5. 5Cluster and map. Group terms into pillars and clusters. Draw the internal links. Find the empty branches, those are your roadmap.
6. 6Map to the funnel. Tag each cluster as awareness, consideration, or decision, and make sure you have coverage at every stage with links connecting them.
7. 7Validate against the SERP and the AI. Open the real results and the AI answers for your priority terms. Confirm intent, judge whether you can win, and note who you must beat.
8. 8Prioritize. Score by intent strength, winnability based on real SERP weakness, and business value. Sort. That sorted list is your publishing calendar.
9. 9Publish, link, and measure. Ship the content, wire the internal links, then watch Search Console and AI citations. Double down on what moves, prune what does not, and re-run this loop every quarter.

That is the whole thing. Notice that volume never appears as a sorting key until the very end, and even then it is a tiebreaker, not the boss. Intent leads. Jobs lead. Winnability leads. Volume rides along.

Anybody can pull a keyword list. The money is in knowing which lines to ignore, and most people never learn that part.Shmul