Search is an ad business. That’s not cynicism; it’s straight from Alphabet’s filings: Google Services generates revenue primarily from advertising across Search, YouTube, and network properties. If you don’t start from that, you’ll misunderstand why the SERP behaves the way it does—and why your organic and paid visibility keeps getting squeezed. (Alphabet Investor Relations)
This guide does two things:

1. shows exactly how to find a competitor’s organic and paid keywords in the U.S./English market using Google-native signals plus reputable third-party data, and

2. lays out the uncomfortable incentives that stack the deck (AI summaries, zero-click answers, ad placements, vertical modules) so you can counter with realistic tactics.

The uncomfortable truths shaping today’s SEO game
1) Google’s incentives favor less clicking and more containment
AI Overviews now sit atop many results with generated summaries and links, and Google has begun placing ads inside those AI modules for commercial queries. That pushes traditional organic listings even further down. Multiple independent analyses indicate that users click fewer links when AI summaries appear—Pew found link clicks dropped roughly from 15% of visits without AI summaries to 8% when summaries were present. Google’s own messaging counters that AI Overviews “include links to learn more” and can increase diversity of sites, but the click math is not your friend. (blog.google)
Add the longstanding trend toward zero-click answers (featured snippets, knowledge panels, calculators, weather, etc.). Several studies in 2024–2025 estimate a majority of searches now end without a click to the open web (methodologies vary; directionally they agree). If you build a plan that assumes an easy stream of organic clicks, you’re betting against these trends. (SparkToro)
2) Organic real estate is crowded out by paid units and Google’s own verticals
Ads can appear above the top organic results (“top ads”) or at the bottom, and placement is dynamic. That means your #1 organic result may still render beneath multiple ads (and now AI modules). Meanwhile, Google surfaces its own vertical experiences—Local Packs/Business Profiles, Flights, Hotels, Knowledge Panels—before and around traditional listings. Those modules keep users engaged inside Google and siphon attention from classic blue links. (Google Help)
3) The rules (and the goalposts) move—fast
Core updates roll several times a year; “helpfulness” signals have been folded into Google’s broader ranking systems; and spam policies and manual actions can de-list or demote content and entire sites. If your strategy relies on brittle tricks (expired domains, link schemes, doorway pages), you are gambling against written policies that empower site-wide or page-level demotion. Track Search Status to understand when volatility is algorithmic vs. site-specific. (Google for Developers)
4) Scraping SERPs is fragile and risky at scale
Google’s ToS has long banned automated queries without permission; in 2025, increased anti-scraping enforcement caused outages at rank-tracking tools. If you need programmatic SERP context, use Programmable Search / Custom Search JSON API or rely on vendor datasets—don’t hang mission-critical workflows on headless scraping. (Google Policies)
5) Regulators keep pointing at market power—and that affects the playing field
U.S. courts have ruled against Google in search and ad tech monopolization cases, and the EU’s long-running Shopping case found self-preferencing that diverted traffic to Google’s own service. You can’t litigate your way to rankings, but the outcomes explain why organic “fairness” often feels theoretical. (Department of Justice)
Conclusion: Assume rising friction to earn organic clicks, more pay-to-play surface area, and higher CPCs year-over-year in paid (with variance by vertical). WordStream’s 2025 U.S. benchmark shows higher CPC and cost-per-lead vs. 2024. Budget and plan accordingly. (WordStream)
What “competitor keywords” really means (and the hard limits)

• Organic competitor keywords = queries where a competitor’s domain/URL ranks in Google’s results. You cannot access their private Search Console data; GSC requires verified ownership and permissions. (Google Help)

• Paid competitor keywords = queries where a rival appears in your auctions. Google Ads provides Auction Insights and Keyword Planner for volumes/costs by United States / English. (Google Help)

The 5 reliable methods to find competitor keywords (U.S./English)
1) Map paid intent with Google Ads (fast and defensible)

• Auction Insights shows overlap and outranking share per campaign/ad group (Search, Shopping, PMAX). Export and pivot to see which rivals actually collide with you. (Google Help)

• Keyword Planner → “Discover new keywords” → paste a competitor URL → set Location: United States and Language: English; export ideas with CPC/top-of-page bids. This is the canonical, Google-native estimate of demand/cost. (Google Help)

Why this matters now: As AI Overviews, knowledge panels, and ads crowd above organic, paid data is the fastest sanity-check on which queries still drive converting traffic in the U.S. context. (The Verge)
2) Pull organic coverage from a reputable third-party index
Use one main source to avoid analysis paralysis; spot-check with a second when stakes are high.

• Semrush (Organic Research & Keyword Gap) for U.S. rankings and domain-to-domain comparisons. (AdInfusion)

• Ahrefs (Site Explorer → Organic Keywords) for per-country rank/URL exports (U.S.). (Department of Justice)

• SpyFu for a blended lens on SEO + PPC history. (Google Help)

Expect discrepancies—each vendor crawls and models differently.
3) Run a Keyword Gap to expose “they rank, you don’t”
In Semrush’s Gap tool: your domain + up to four competitors → Database: U.S. → export Missing (they rank, you don’t) and Weak (you rank poorly). This surfaces opportunity clusters quickly. (status.search.google.com)
4) QA the live SERP without breaking rules
If you need exact titles/URLs/snippets for a shortlist, use Programmable Search via the Custom Search JSON API. It returns JSON results from your Programmable Search Engine and keeps you within policy (no headless scraping against google.com). (Google for Developers)
5) Mine SERP features to expand topical coverage
Design pages that can earn (or at least align to) the features Google actually surfaces:

• Featured Snippets & People Also Ask → shape your headings and FAQs around the questions Google already shows. (Google for Developers)

• Add Article, FAQPage, HowTo (when appropriate) structured data; validate with Rich Results Test before publishing. (Google for Developers)

The Bouncebeam workflow (U.S./English), adjusted for today’s realities
Phase A — Paid “x-ray” (60–90 minutes)

1. Export Auction Insights for core Search campaigns; rank overlapping advertisers by impression overlap and outranking share. (Google Help)

2. Keyword Planner with competitor landing pages; lock to United States / English; export ideas with CPCs and top-of-page bids. (Google Help)

3. Shortlist ~30 commercial queries with sane CPCs for probe campaigns. Rising CPCs are normal—benchmark against 2025 norms, not 2023 nostalgia. (WordStream)

Phase B — Organic coverage (90–120 minutes)

1. Export each competitor’s top U.S. keywords (desktop and, where available, mobile) from Semrush or Ahrefs with ranking URLs. (AdInfusion)

2. Run Keyword Gap (U.S. database) → export Missing/Weak. (status.search.google.com)

3. For ambiguous terms, fetch top results via Custom Search JSON API to verify intent and page archetype; no scraping. (Google for Developers)

Phase C — Cluster and decide actions (45–60 minutes)

• Group by intent (Informational / Commercial / Transactional) and SERP archetype (guide, comparison, tool, calculator, product page).

• Label each cluster Own (build best-in-class), Leapfrog (upgrade an existing near-ranking page + internal links), or Ignore (bad economics or misaligned intent).

• Note where AI Overviews or verticals dominate; if the module absorbs clicks (e.g., Hotels/Flights/Local), bias toward paid or tool/calculator experiences that still win attention. (Google Help)

Phase D — Ship and measure

• Publish with Article + FAQPage / HowTo (only if steps are genuine) and validate with Rich Results Test. (Google for Developers)

• Track in Search Console Performance with Country = United States; watch impressions/CTR/positions over time and annotate around Core Updates from the Search Status Dashboard. (Google Help)

How the “rigged table” changes your tactics (and what actually works)

• Expect fewer organic clicks on informational terms that trigger AI Overviews or heavy SERP features; focus on commercial clusters where ads and conversion economics justify investment. Validate with small PPC probes before sinking months into content. (blog.google)

• Design for the modules that exist: Local intent? Optimize your Business Profile and local signals, not just your blog. Travel/ecom? Understand how Flights/Hotels/Shopping reframe discovery and put landing pages where users still click. (Google Help)

• Assume cost pressure in Google Ads. Benchmarks show higher CPCs and cost-per-lead YoY; negotiate your portfolio around ROI, not vanity impression share. (WordStream)

• Stay compliance-safe: If you need live SERP auditing at scale, use Google’s Programmable Search API or vendor indexes; headless scraping is brittle and violates terms. (Google for Developers)

Step-by-step: a free (or nearly free) U.S./English process you can run this week

1. From a clean browser, run core queries; list repeating SERP-competitors. Keep your tests U.S./English (results vary by location/device). (Google Help)

2. Keyword Planner → “Discover new keywords” using a competitor URL; set United States / English; export volume and cost. (Google Help)

3. One vendor pass: pull U.S. organic keywords for 3–5 competitors (Semrush/Ahrefs) and run a Keyword Gap. (AdInfusion)

4. QA 50–100 terms with Custom Search JSON API to confirm intent and page archetype. (Google for Developers)

5. Decide: Own / Leapfrog / Ignore—aligned to CPC reality and SERP module pressure. (WordStream)

6. Publish with structured data; validate; measure in Search Console (U.S. filter) and annotate core updates. (Google for Developers)

Bouncebeam demonstration (what we actually deliver)

• A merged CSV of competitor organic + paid terms (one index + Planner + compliant SERP QA), labeled by intent and recommended action. (AdInfusion)

• A one-page opportunity brief for the top clusters: SERP archetype, PAA-aligned FAQs, internal links, and whether ads/AI modules demand a paid handshake to capture demand. (Google for Developers)

• An optional PPC probe plan to validate two clusters before scaling content in a market where CPCs rise and clicks consolidate. (WordStream)

FAQ (hard-edged and practical)
Can I see a competitor’s keywords in my Google Search Console?
No. Search Console only exposes data to verified owners/users for their properties. Use third-party indexes for competitor organic sets. (Google Help)
Is it okay to scrape Google’s results to build competitor lists?
Google’s Terms of Service prohibit automated queries without permission, and anti-scraping enforcement has disrupted popular tools. Use Programmable Search or vendor datasets instead. (Google Policies)
Do AI Overviews reduce clicks to websites?
Independent studies (e.g., Pew) show users click fewer links when AI summaries appear; publishers have filed complaints/lawsuits citing traffic losses. Google disputes methodology and says AI Overviews drive diverse clicks. Reality: plan for fewer organic clicks on affected queries. (Pew Research Center)
Why do my “#1” rankings sometimes underperform?
Because top ads often render above organic, and vertical modules (Local/Flights/Hotels/Knowledge) soak attention first. Your page can be “#1” but visually below high-CTR elements. (Google Help)
Is Google’s dominance just a conspiracy theory?
U.S. courts have issued rulings against Google in search and ad tech; the EU’s Shopping case established self-preferencing findings. Regardless of appeals and remedies, plan for a landscape where gatekeeper incentives do not align with your traffic goals. (Department of Justice)
On-page execution you shouldn’t skip (U.S./English)

• Title tags

  • How to Find Competitor Keywords (U.S.): Free & Pro Methods, Tested

  • See Your Competitors’ Keywords Fast: Exact Steps + Google-Verified Workflows

• Meta description
  A pragmatic U.S. guide to competitor keywords across paid and organic. Google-native steps, vendor cross-checks, and a repeatable checklist.

• URL
  /how-to-find-competitor-keywords

• Structured data
  Apply Article to the guide, FAQPage to Q&A, and HowTo only if you truly include step sequences; validate with Rich Results Test. (Google for Developers)

• U.S./English discipline
  In Planner and vendor tools, lock Location = United States and Language = English; device/location shifts change results and demand. (Google Help)

Source list (representative, high-trust and recent)

• Alphabet financials & revenue mix: 2024 Form 10-K (filed Feb 2025). (Alphabet Investor Relations)

• AI Overviews (official): rollout and help docs. (blog.google)

• AI Overviews impact: Pew 2025; reportage/litigation around traffic effects. (Pew Research Center)

• Ads inside AI Overviews (commercial queries): The Verge (Oct 2024). (The Verge)

• Zero-click trend: SparkToro 2024; Similarweb explainer (2025). (SparkToro)

• Ad placement mechanics: Google Ads Help (top/bottom positions, Ad Rank). (Google Help)

• Local/vertical modules: Business Profile, Flights, Hotels, Knowledge Panels (Google docs). (Google Help)

• Ranking systems & updates: systems guide; spam/manual actions; status dashboard. (Google for Developers)

• Anti-scraping reality: Google ToS; SEJ on 2025 tool outages; Programmable Search API. (Google Policies)

• Antitrust context: U.S. DOJ press releases (2024–2025), EU Shopping decision summary. (Department of Justice)

• CPC/benchmark pressure: WordStream 2025 PPC benchmarks. (WordStream)

Bottom line
You won’t get a competitor’s private keyword data. You can triangulate their organic and paid queries credibly—but you must do it with your eyes open: AI summaries shrink click-through, ads crowd the fold, vertical modules keep users inside Google, scraping is brittle, and paid costs drift upward. The durable strategy is: validate with Ads, harvest real gaps with one reputable index, QA with Google’s API, and build pages aligned to the SERP that actually exists—not the one you wish existed. (Google Help)
If you want this turned into a CMS-ready draft for Wix Studio (pure text + images/tables only), say the word and I’ll adapt the formatting and provide export-ready CSV templates for your Planner and gap outputs.