Code search and indexing for AI agents: text, semantic and symbol search

AI coding agents find code in three ways: text search with ripgrep-style tools, semantic search over an embedding index, and symbol search through a language server. Claude Code and Codex rely on text search unless a language server or MCP server is added; Cursor’s agent also has semantic search. Matching the search to the question, and excluding noise, decides how much context discovery costs.

You ask the agent where subscription cancellations are handled. It greps for “cancel”, opens a dozen files, including a generated client and a deprecated v1 handler, and answers confidently with the dead code path. Half the context window is gone before it writes a line. This page is for developers who drive agents daily and for tech leads who set up the repository so every agent on the team searches it well.

What you’ll set up

• A map of how Claude Code, Codex and Cursor search a repository, and which layers you can add.
• A decision table that picks text, semantic or symbol search for a given question.
• Ignore-file and permission settings that keep generated code, fixtures and secrets out of results.
• Four copy-paste prompts for scoped discovery, feature tracing, usage audits and delegated exploration.
• A two-command check that proves the agent’s findings are real and complete without reading every file.

How do Claude Code, Codex and Cursor find code?

All three agents start from text search. What differs is what else is built in and what you have to add.

Search layer	Claude Code	Codex	Cursor
Text search (grep, file globbing)	Built in	Built in, through the shell	Built in
Semantic search over an embedding index	Add an MCP server	Add an MCP server	Built in
Symbol search (definitions, references, call hierarchy)	LSP tool, after you install a code intelligence plugin	Add an MCP server	Unverified; add an MCP server
Delegated exploration in a separate context	Built-in Explore subagent	Subagents	Subagents

Claude Code

Text search. On macOS, Linux and WSL, Claude Code leaves the Grep and Glob tools out of the default tool set. Claude searches with find and grep through the Bash tool, which run embedded versions of bfs and ugrep. On Windows, Grep and Glob are part of the default set. Elsewhere, naming either tool in --allowedTools when you start the session restores both; --tools gives you exactly the ones you list (checked against the tools reference and Claude Code 2.1.283, latest channel, on 2026-09-26).

Once enabled, two behaviours of those tools matter for noise. Grep is built on ripgrep and skips gitignored files. Glob does not respect .gitignore by default and caps results at 100 files; set CLAUDE_CODE_GLOB_NO_IGNORE=false before launch to make it skip ignored files.

Symbol search. The LSP tool jumps to definitions, finds references, lists symbols, finds implementations and traces call hierarchies. It stays inactive until you install a code intelligence plugin for your language and put the language server binary on your PATH. It also reports type errors after each edit. In cloud sessions, plugin language servers do not start, so the tool stays inactive there.

Delegated exploration. Claude delegates to the built-in Explore subagent when it needs to understand a codebase without changing it, so the search results stay out of your main conversation. Explore skips your CLAUDE.md files, which means a navigation map kept only in CLAUDE.md does not reach it.

Codex

Text search. Codex runs shell commands in its sandbox. Its bundled instructions tell the model to prefer rg for text and rg --files for file names, falling back to other tools when rg is missing (codex-rs/core/gpt_5_codex_prompt.md at rust-v0.157.1). ripgrep skips files matched by .gitignore, .ignore and .rgignore.

No built-in index. Codex 0.157.1 lists no semantic-index or language-server feature in codex features list (checked against 0.157.1). Symbol and semantic search come from MCP servers you add with codex mcp add.

Scoping. Start Codex in the right directory with -C DIR, and add a second repository with --add-dir DIR instead of copying code into the prompt. In the terminal UI, /mention attaches a file directly, and /status shows the session’s token usage.

Cursor

Three search tools. The agent runtime that Cursor ships in its @cursor/sdk 1.0.32 package contains a semantic codebase search tool, a ripgrep-based grep tool and a glob file search tool. The agent chooses between them per query. The same runtime also defines go-to-definition and symbol-search tool types; whether the agent uses them was not verifiable here.

Exclusions. The same runtime refuses agent reads of paths listed in .cursorignore and passes the list to its ripgrep file walk. The file uses .gitignore syntax.

What is not stated here. cursor.com was unreachable when this page was checked, so indexing timings, the settings screen layout and whether a path in .cursorignore also leaves the semantic index are not given as facts. Confirm them in Cursor’s own documentation before you rely on them. For team-wide exclusions, see how to lock down Cursor for a team.

Which search strategy fits the question?

Most wasted context comes from asking a text search a conceptual question, or a semantic search a structural one. Pick by the question, not by the tool.

The question you are asking	Best search	Why	Prompt cue
“Where is applyDiscount defined?”	Text search	You know the exact identifier, so one rg -w call answers it	Name the identifier
“Where do we retry failed payments?”	Semantic search, or text search on likely identifiers	The concept has no fixed name	Ask for candidate identifiers first
“Who calls applyDiscount, and through which routes?”	Symbol search	Text search misses aliased imports and matches comments	“Use the language server”
“How does checkout work end to end?”	Delegated exploration	A broad read that should not land in your main context	“Use a subagent” or Explore
“Rename applyDiscount everywhere”	Symbol search, then the type checker	References must be complete, and the compiler proves they are	“Find references, then run typecheck”

On a small repository, text search is enough, and adding an index costs more setup than it saves. The case for symbol or semantic search grows with size, with dynamic naming and with the number of near-duplicate implementations.

Set up a repository so every agent searches it well

1. Exclude the noise from every tool. Generated code, vendored dependencies, fixtures, build output and lock files dominate text-search results. Keep them out once, in the files each tool reads:

   Claude Code

   On macOS, Linux and WSL, Claude searches with grep and find through Bash by default. Whether that embedded grep honours .gitignore is not documented (checked 2.1.283), so do not count on it. Either start the session with claude --allowedTools Grep, which restores the ripgrep-backed Grep and Glob tools, or rely on a “Searching this repository” section in CLAUDE.md that tells Claude to use rg when it is installed (Claude Code does not guarantee it) and otherwise to pass exclude flags such as --exclude-dir to grep:

   ## Searching this repository
   - src/generated/ and src/__generated__/ are generated. Search src/api/ for the source schema instead.
   - fixtures/ holds large test data. Do not read it unless a test names a fixture file.
   - Search with `rg` if it is installed (it skips gitignored files); otherwise pass --exclude-dir=node_modules --exclude-dir=dist --exclude-dir=src/generated to grep.

   CLAUDE_CODE_GLOB_NO_IGNORE=false matters only once Glob is enabled, on Windows or after --tools or --allowedTools. Set it in your shell profile then, because Glob ignores .gitignore by default while Grep respects it.

   Codex

   Commit an .ignore file at the repository root. ripgrep reads it, so tracked noise drops out of Codex’s rg searches without changing what Git tracks:

   # .ignore: skipped by ripgrep, still tracked by Git
   src/generated/
   src/__generated__/
   fixtures/
   *.snap

   Add the same “Searching this repository” section to AGENTS.md so the model knows where the real source lives.

   Cursor

   Commit a .cursorignore with the same entries. The agent’s reads and its grep file walk skip matching paths:

   # .cursorignore (gitignore syntax)
   src/generated/
   src/__generated__/
   fixtures/

2. Write a short navigation map. Five to ten lines in CLAUDE.md or AGENTS.md naming where each layer lives (routes, services, data access, tests) save the agent a directory crawl on every task. Keep it to paths, not prose; documentation as context covers what else belongs there, and file organization for agents covers naming files so a search finds them.

3. Add symbol search where the codebase is large or typed. In Claude Code, install the code intelligence plugin for your language and the language server it expects:

   # terminal
   npm install -g typescript-language-server typescript
   claude plugin install typescript-lsp@claude-plugins-official

   For Codex, or for one server shared across tools, Serena provides symbol-level retrieval through language servers over MCP. Install it from upstream, not from a marketplace, because its README warns the marketplace commands are outdated:

   # terminal
   uv tool install -p 3.13 serena-agent
   claude mcp add --scope user serena -- serena start-mcp-server --context claude-code --project-from-cwd
   serena setup codex

   The code-intelligence MCP servers guide compares Serena, codebase-memory-mcp, Claude Context (semantic search that needs a vector database and an embedding provider) and Repomix.

4. Measure context before and after. Run the same discovery prompt on a fresh session and compare /context in Claude Code or /status in Codex. If the numbers barely move, the change was not worth keeping.

Prompts that keep code discovery cheap and checkable

Each prompt asks for citations in path:line:symbol form. The citations are what make the answer checkable in the next section.

Copy-paste prompt: scoped code discovery

Find where subscription cancellation is handled in this repository.

Rules:
- Start by listing 3-5 identifiers you will search for (function, route and
  event names), then search for those, not for the English word "cancel".
- Skip generated code, fixtures and anything under a v1/ or legacy/ path
  unless nothing else matches.
- Read only the functions that match, not whole files. Stop after 6 files.

Report:
1. Each relevant location as path:line:symbol (for example
   src/billing/cancel.ts:42:cancelSubscription), one per line with nothing
   else on it.
2. A three-sentence summary of the flow.
3. What you searched for that returned nothing, so I can narrow it.

Copy-paste prompt: trace a feature from route to database

Trace what happens when a user submits POST /api/checkout, from the route
handler to the database write.

For each hop (route, middleware, service, data access, query), give one
line in the exact form path:line:symbol (for example
src/api/checkout.ts:18:handleCheckout) with nothing else on it, and put
one sentence on what it does on the next line.
Use the language server for definitions and references where it is
available; otherwise use rg with -w on exact identifiers.
Read only the functions on the path. Finish with the list of tables written.

Copy-paste prompt: audit every usage of a pattern

List every place that calls an external HTTP API and handles its errors.

1. Search with rg for the HTTP client calls used here (fetch, axios,
   got, httpx: check package manifests first to see which apply).
2. For each call site, output one line:
   path:line:symbol | client | how errors are handled (throw, retry, swallow, log)
3. Group the lines by handling style and flag every "swallow".
Output the raw list first, then the grouping. Do not edit any file.

Copy-paste prompt: delegate a broad exploration

Use a subagent to map how authentication works in this repository:
entry points, session storage, token refresh, and where permissions
are checked. The subagent should return only a summary of at most
20 lines with path:line:symbol citations, not file contents.
Then tell me which three files I should read myself before changing
the session lifetime.

In Claude Code, the last prompt usually lands on Explore; because Explore skips CLAUDE.md, put any paths it must avoid into the prompt itself. In Codex and Cursor, the same wording starts a subagent.

How do you verify what the agent found without reading every file?

A confident search answer can still cite a line that does not exist, or miss half the callers. Two deterministic checks catch both, and neither needs you to read the code.

Check that every citation is real. Save the agent’s path:line:symbol lines to citations.txt. The prompts above ask for this exact format; for the audit prompt, keep only the first column (cut -d'|' -f1 audit.txt | sed 's/ *$//' > citations.txt). Then run:

# terminal: every cited line must exist and contain the cited symbol
while IFS=: read -r path line sym; do
  if sed -n "${line}p" "$path" 2>/dev/null | grep -qF -- "$sym"; then
    echo "ok   $path:$line"
  else
    echo "FAIL $path:$line ($sym)"
  fi
done < citations.txt

For languages with :: in symbol names (C++, Rust, PHP), use a different separator such as | in the prompt and in IFS.

Any FAIL means the agent paraphrased, guessed a line number or cited a file that moved. Paste the failures back and ask it to re-search those entries.

Check that the list is complete. Compare the files the agent named with an independent text search for the same identifier. Use the identifier from your own search (here the rename example, applyDiscount):

# terminal: files that mention the symbol but are missing from the agent's list
rg -l -w applyDiscount src | sort > expected.txt
cut -d: -f1 citations.txt | sed "s|^\./||" | sort -u > found.txt
comm -13 found.txt expected.txt

Each file printed is a candidate the agent skipped. Some are comments or tests; the rest are real misses. For a rename or signature change, the final proof is the type checker and the test suite: a green tsc --noEmit (or your language’s equivalent) plus passing tests shows every reference was updated, which no amount of reading does as reliably. The developer who asked for the search signs off on the citation check; the change itself goes through your normal agent PR review gates.

What can search reach that it should not?

Search is also a read path to secrets. Three gaps are worth knowing before you rely on ignore files for security:

• Claude Code read-deny rules have limits. A deny rule such as Read(./.env) covers the built-in file tools and file commands Claude Code recognises in Bash, such as cat. It does not cover a command that reads files without naming them, such as grep -r pattern . run from the directory that holds the file. For OS-level enforcement, enable the sandbox; see permissions, sandboxes and approval modes.
• .cursorignore is enforced by Cursor’s agent runtime, not by the operating system. A terminal command the agent runs is gated separately by Cursor’s run modes and sandbox.
• Hosted semantic indexes send code off the machine. An MCP server such as Claude Context sends code chunks to its embedding provider unless you run the vector database and embeddings locally. Clear that with whoever owns your data policy before you index a private repository.

When code search goes wrong

The agent cites generated or vendored code. The noise is not excluded for that tool. Add the path to the tool’s ignore file from step 1, start a fresh session, and rerun the prompt. In Claude Code on macOS, Linux or WSL, check whether a Bash grep or find call returned it; restore the Grep tool with --allowedTools Grep or add the path to the CLAUDE.md search rules. In sessions where Glob is enabled, check whether Glob returned the file; it ignores .gitignore unless CLAUDE_CODE_GLOB_NO_IGNORE=false is set.

The answer describes the wrong implementation. Repositories with v1 and v2 handlers, or a copy in a migration script, fool both text and semantic search. Name the directory that holds the live code in the prompt, and add a line to the navigation map saying which version is current.

Discovery eats the context window. The agent is reading whole files to answer a location question. Add “read only matching functions” and a file limit to the prompt, or delegate the search to a subagent so only its summary returns. Context windows and token limits covers compaction when it is already too late.

The LSP tool returns errors or is never used. The language server binary is not on PATH, the plugin for that language is not installed, or the session runs in the cloud, where plugin language servers do not start. Run the server binary by hand to confirm it starts, then restart the session.

Semantic search returns plausible but wrong matches. Embedding search ranks by similarity, not by correctness. Ask for candidate identifiers first, then confirm each with an exact text or symbol search, and run the completeness check above.

Codex falls back to slow searches. rg is not installed in the environment, so the model falls back to other tools as its instructions allow. Install ripgrep in the image or dev container that Codex runs in.

Where to go next with code search and indexing

Code-intelligence MCP servers Install Serena, codebase-memory-mcp, Claude Context or Repomix and run a symbol-level refactor.

File organization for agents Name and place files so a text search finds the right one first.

Custom subagents in Claude Code Define your own exploration subagent, including one that loads your navigation rules.

Cost per context Put a price on search-heavy sessions and decide where an index pays off.