# Code search and indexing for AI agents: text, semantic and symbol search URL: https://developertoolkit.ai/en/workflows/context/codebase-indexing/ Description: How Claude Code, Codex and Cursor find code, which search fits which question, how to cut noise, and how to check the agent's findings without reading every file. AI coding agents find code in three ways: text search with ripgrep-style tools, semantic search over an embedding index, and symbol search through a language server. Claude Code and Codex rely on text search unless a language server or MCP server is added; Cursor's agent also has semantic search. Matching the search to the question, and excluding noise, decides how much context discovery costs. You ask the agent where subscription cancellations are handled. It greps for "cancel", opens a dozen files, including a generated client and a deprecated v1 handler, and answers confidently with the dead code path. Half the context window is gone before it writes a line. This page is for developers who drive agents daily and for tech leads who set up the repository so every agent on the team searches it well. ## What you'll set up - A map of how Claude Code, Codex and Cursor search a repository, and which layers you can add. - A decision table that picks text, semantic or symbol search for a given question. - Ignore-file and permission settings that keep generated code, fixtures and secrets out of results. - Four copy-paste prompts for scoped discovery, feature tracing, usage audits and delegated exploration. - A two-command check that proves the agent's findings are real and complete without reading every file. ## How do Claude Code, Codex and Cursor find code? All three agents start from text search. What differs is what else is built in and what you have to add. | Search layer | Claude Code | Codex | Cursor | | --- | --- | --- | --- | | Text search (grep, file globbing) | Built in | Built in, through the shell | Built in | | Semantic search over an embedding index | Add an MCP server | Add an MCP server | Built in | | Symbol search (definitions, references, call hierarchy) | `LSP` tool, after you install a code intelligence plugin | Add an MCP server | Unverified; add an MCP server | | Delegated exploration in a separate context | Built-in `Explore` subagent | Subagents | Subagents | **Claude Code** **Text search.** On macOS, Linux and WSL, Claude Code leaves the `Grep` and `Glob` tools out of the default tool set. Claude searches with `find` and `grep` through the `Bash` tool, which run embedded versions of `bfs` and `ugrep`. On Windows, `Grep` and `Glob` are part of the default set. Elsewhere, naming either tool in `--allowedTools` when you start the session restores both; `--tools` gives you exactly the ones you list (checked against the tools reference and Claude Code 2.1.283, `latest` channel, on 2026-09-26). Once enabled, two behaviours of those tools matter for noise. `Grep` is built on ripgrep and skips gitignored files. `Glob` does **not** respect `.gitignore` by default and caps results at 100 files; set `CLAUDE_CODE_GLOB_NO_IGNORE=false` before launch to make it skip ignored files. **Symbol search.** The `LSP` tool jumps to definitions, finds references, lists symbols, finds implementations and traces call hierarchies. It stays inactive until you install a code intelligence plugin for your language and put the language server binary on your `PATH`. It also reports type errors after each edit. In cloud sessions, plugin language servers do not start, so the tool stays inactive there. **Delegated exploration.** Claude delegates to the built-in `Explore` subagent when it needs to understand a codebase without changing it, so the search results stay out of your main conversation. `Explore` skips your `CLAUDE.md` files, which means a navigation map kept only in `CLAUDE.md` does not reach it. **Codex** **Text search.** Codex runs shell commands in its sandbox. Its bundled instructions tell the model to prefer `rg` for text and `rg --files` for file names, falling back to other tools when `rg` is missing (`codex-rs/core/gpt_5_codex_prompt.md` at `rust-v0.157.1`). ripgrep skips files matched by `.gitignore`, `.ignore` and `.rgignore`. **No built-in index.** Codex 0.157.1 lists no semantic-index or language-server feature in `codex features list` (checked against 0.157.1). Symbol and semantic search come from MCP servers you add with `codex mcp add`. **Scoping.** Start Codex in the right directory with `-C DIR`, and add a second repository with `--add-dir DIR` instead of copying code into the prompt. In the terminal UI, `/mention` attaches a file directly, and `/status` shows the session's token usage. **Cursor** **Three search tools.** The agent runtime that Cursor ships in its `@cursor/sdk` 1.0.32 package contains a semantic codebase search tool, a ripgrep-based grep tool and a glob file search tool. The agent chooses between them per query. The same runtime also defines go-to-definition and symbol-search tool types; whether the agent uses them was not verifiable here. **Exclusions.** The same runtime refuses agent reads of paths listed in `.cursorignore` and passes the list to its ripgrep file walk. The file uses `.gitignore` syntax. **What is not stated here.** cursor.com was unreachable when this page was checked, so indexing timings, the settings screen layout and whether a path in `.cursorignore` also leaves the semantic index are not given as facts. Confirm them in Cursor's own documentation before you rely on them. For team-wide exclusions, see [how to lock down Cursor for a team](/en/cursor-ide/team/privacy-security/). ## Which search strategy fits the question? Most wasted context comes from asking a text search a conceptual question, or a semantic search a structural one. Pick by the question, not by the tool. | The question you are asking | Best search | Why | Prompt cue | | --- | --- | --- | --- | | "Where is `applyDiscount` defined?" | Text search | You know the exact identifier, so one `rg -w` call answers it | Name the identifier | | "Where do we retry failed payments?" | Semantic search, or text search on likely identifiers | The concept has no fixed name | Ask for candidate identifiers first | | "Who calls `applyDiscount`, and through which routes?" | Symbol search | Text search misses aliased imports and matches comments | "Use the language server" | | "How does checkout work end to end?" | Delegated exploration | A broad read that should not land in your main context | "Use a subagent" or `Explore` | | "Rename `applyDiscount` everywhere" | Symbol search, then the type checker | References must be complete, and the compiler proves they are | "Find references, then run typecheck" | On a small repository, text search is enough, and adding an index costs more setup than it saves. The case for symbol or semantic search grows with size, with dynamic naming and with the number of near-duplicate implementations. ## Set up a repository so every agent searches it well 1. **Exclude the noise from every tool.** Generated code, vendored dependencies, fixtures, build output and lock files dominate text-search results. Keep them out once, in the files each tool reads: **Claude Code** On macOS, Linux and WSL, Claude searches with `grep` and `find` through `Bash` by default. Whether that embedded `grep` honours `.gitignore` is not documented (checked 2.1.283), so do not count on it. Either start the session with `claude --allowedTools Grep`, which restores the ripgrep-backed `Grep` and `Glob` tools, or rely on a "Searching this repository" section in `CLAUDE.md` that tells Claude to use `rg` when it is installed (Claude Code does not guarantee it) and otherwise to pass exclude flags such as `--exclude-dir` to `grep`: ```markdown ## Searching this repository - src/generated/ and src/__generated__/ are generated. Search src/api/ for the source schema instead. - fixtures/ holds large test data. Do not read it unless a test names a fixture file. - Search with `rg` if it is installed (it skips gitignored files); otherwise pass --exclude-dir=node_modules --exclude-dir=dist --exclude-dir=src/generated to grep. ``` `CLAUDE_CODE_GLOB_NO_IGNORE=false` matters only once `Glob` is enabled, on Windows or after `--tools` or `--allowedTools`. Set it in your shell profile then, because `Glob` ignores `.gitignore` by default while `Grep` respects it. **Codex** Commit an `.ignore` file at the repository root. ripgrep reads it, so tracked noise drops out of Codex's `rg` searches without changing what Git tracks: ```text # .ignore: skipped by ripgrep, still tracked by Git src/generated/ src/__generated__/ fixtures/ *.snap ``` Add the same "Searching this repository" section to `AGENTS.md` so the model knows where the real source lives. **Cursor** Commit a `.cursorignore` with the same entries. The agent's reads and its grep file walk skip matching paths: ```text # .cursorignore (gitignore syntax) src/generated/ src/__generated__/ fixtures/ ``` 2. **Write a short navigation map.** Five to ten lines in `CLAUDE.md` or `AGENTS.md` naming where each layer lives (routes, services, data access, tests) save the agent a directory crawl on every task. Keep it to paths, not prose; [documentation as context](/en/workflows/context/documentation-as-context/) covers what else belongs there, and [file organization for agents](/en/workflows/context/file-organization/) covers naming files so a search finds them. 3. **Add symbol search where the codebase is large or typed.** In Claude Code, install the code intelligence plugin for your language and the language server it expects: ```bash # terminal npm install -g typescript-language-server typescript claude plugin install typescript-lsp@claude-plugins-official ``` For Codex, or for one server shared across tools, Serena provides symbol-level retrieval through language servers over MCP. Install it from upstream, not from a marketplace, because its README warns the marketplace commands are outdated: ```bash # terminal uv tool install -p 3.13 serena-agent claude mcp add --scope user serena -- serena start-mcp-server --context claude-code --project-from-cwd serena setup codex ``` The [code-intelligence MCP servers guide](/en/ecosystem/mcp/code-intelligence-mcp/) compares Serena, codebase-memory-mcp, Claude Context (semantic search that needs a vector database and an embedding provider) and Repomix. 4. **Measure context before and after.** Run the same discovery prompt on a fresh session and compare `/context` in Claude Code or `/status` in Codex. If the numbers barely move, the change was not worth keeping. ## Prompts that keep code discovery cheap and checkable Each prompt asks for citations in `path:line:symbol` form. The citations are what make the answer checkable in the next section. **Copy-paste prompt: scoped code discovery** ```text Find where subscription cancellation is handled in this repository. Rules: - Start by listing 3-5 identifiers you will search for (function, route and event names), then search for those, not for the English word "cancel". - Skip generated code, fixtures and anything under a v1/ or legacy/ path unless nothing else matches. - Read only the functions that match, not whole files. Stop after 6 files. Report: 1. Each relevant location as path:line:symbol (for example src/billing/cancel.ts:42:cancelSubscription), one per line with nothing else on it. 2. A three-sentence summary of the flow. 3. What you searched for that returned nothing, so I can narrow it. ``` **Copy-paste prompt: trace a feature from route to database** ```text Trace what happens when a user submits POST /api/checkout, from the route handler to the database write. For each hop (route, middleware, service, data access, query), give one line in the exact form path:line:symbol (for example src/api/checkout.ts:18:handleCheckout) with nothing else on it, and put one sentence on what it does on the next line. Use the language server for definitions and references where it is available; otherwise use rg with -w on exact identifiers. Read only the functions on the path. Finish with the list of tables written. ``` **Copy-paste prompt: audit every usage of a pattern** ```text List every place that calls an external HTTP API and handles its errors. 1. Search with rg for the HTTP client calls used here (fetch, axios, got, httpx: check package manifests first to see which apply). 2. For each call site, output one line: path:line:symbol | client | how errors are handled (throw, retry, swallow, log) 3. Group the lines by handling style and flag every "swallow". Output the raw list first, then the grouping. Do not edit any file. ``` **Copy-paste prompt: delegate a broad exploration** ```text Use a subagent to map how authentication works in this repository: entry points, session storage, token refresh, and where permissions are checked. The subagent should return only a summary of at most 20 lines with path:line:symbol citations, not file contents. Then tell me which three files I should read myself before changing the session lifetime. ``` In Claude Code, the last prompt usually lands on `Explore`; because `Explore` skips `CLAUDE.md`, put any paths it must avoid into the prompt itself. In Codex and Cursor, the same wording starts a subagent. ## How do you verify what the agent found without reading every file? A confident search answer can still cite a line that does not exist, or miss half the callers. Two deterministic checks catch both, and neither needs you to read the code. **Check that every citation is real.** Save the agent's `path:line:symbol` lines to `citations.txt`. The prompts above ask for this exact format; for the audit prompt, keep only the first column (`cut -d'|' -f1 audit.txt | sed 's/ *$//' > citations.txt`). Then run: ```bash # terminal: every cited line must exist and contain the cited symbol while IFS=: read -r path line sym; do if sed -n "${line}p" "$path" 2>/dev/null | grep -qF -- "$sym"; then echo "ok $path:$line" else echo "FAIL $path:$line ($sym)" fi done < citations.txt ``` For languages with `::` in symbol names (C++, Rust, PHP), use a different separator such as `|` in the prompt and in `IFS`. Any `FAIL` means the agent paraphrased, guessed a line number or cited a file that moved. Paste the failures back and ask it to re-search those entries. **Check that the list is complete.** Compare the files the agent named with an independent text search for the same identifier. Use the identifier from your own search (here the rename example, `applyDiscount`): ```bash # terminal: files that mention the symbol but are missing from the agent's list rg -l -w applyDiscount src | sort > expected.txt cut -d: -f1 citations.txt | sed "s|^\./||" | sort -u > found.txt comm -13 found.txt expected.txt ``` Each file printed is a candidate the agent skipped. Some are comments or tests; the rest are real misses. For a rename or signature change, the final proof is the type checker and the test suite: a green `tsc --noEmit` (or your language's equivalent) plus passing tests shows every reference was updated, which no amount of reading does as reliably. The developer who asked for the search signs off on the citation check; the change itself goes through your normal [agent PR review](/en/workflows/ship/agent-pr-review/) gates. ## What can search reach that it should not? Search is also a read path to secrets. Three gaps are worth knowing before you rely on ignore files for security: - **Claude Code read-deny rules have limits.** A deny rule such as `Read(./.env)` covers the built-in file tools and file commands Claude Code recognises in `Bash`, such as `cat`. It does not cover a command that reads files without naming them, such as `grep -r pattern .` run from the directory that holds the file. For OS-level enforcement, enable the sandbox; see [permissions, sandboxes and approval modes](/en/workflows/harness/permissions-and-sandboxing/). - **`.cursorignore` is enforced by Cursor's agent runtime**, not by the operating system. A terminal command the agent runs is gated separately by Cursor's run modes and sandbox. - **Hosted semantic indexes send code off the machine.** An MCP server such as Claude Context sends code chunks to its embedding provider unless you run the vector database and embeddings locally. Clear that with whoever owns your data policy before you index a private repository. ## When code search goes wrong **The agent cites generated or vendored code.** The noise is not excluded for that tool. Add the path to the tool's ignore file from step 1, start a fresh session, and rerun the prompt. In Claude Code on macOS, Linux or WSL, check whether a `Bash` `grep` or `find` call returned it; restore the `Grep` tool with `--allowedTools Grep` or add the path to the `CLAUDE.md` search rules. In sessions where `Glob` is enabled, check whether `Glob` returned the file; it ignores `.gitignore` unless `CLAUDE_CODE_GLOB_NO_IGNORE=false` is set. **The answer describes the wrong implementation.** Repositories with v1 and v2 handlers, or a copy in a migration script, fool both text and semantic search. Name the directory that holds the live code in the prompt, and add a line to the navigation map saying which version is current. **Discovery eats the context window.** The agent is reading whole files to answer a location question. Add "read only matching functions" and a file limit to the prompt, or delegate the search to a subagent so only its summary returns. [Context windows and token limits](/en/workflows/context/context-windows/) covers compaction when it is already too late. **The `LSP` tool returns errors or is never used.** The language server binary is not on `PATH`, the plugin for that language is not installed, or the session runs in the cloud, where plugin language servers do not start. Run the server binary by hand to confirm it starts, then restart the session. **Semantic search returns plausible but wrong matches.** Embedding search ranks by similarity, not by correctness. Ask for candidate identifiers first, then confirm each with an exact text or symbol search, and run the completeness check above. **Codex falls back to slow searches.** `rg` is not installed in the environment, so the model falls back to other tools as its instructions allow. Install ripgrep in the image or dev container that Codex runs in. ## Where to go next with code search and indexing - [Code-intelligence MCP servers](/en/ecosystem/mcp/code-intelligence-mcp/) — Install Serena, codebase-memory-mcp, Claude Context or Repomix and run a symbol-level refactor. - [File organization for agents](/en/workflows/context/file-organization/) — Name and place files so a text search finds the right one first. - [Custom subagents in Claude Code](/en/claude-code/automate/custom-subagents/) — Define your own exploration subagent, including one that loads your navigation rules. - [Cost per context](/en/workflows/context/cost-per-context/) — Put a price on search-heavy sessions and decide where an index pays off.