# The OpenAI ecosystem: A developer's guide to building agents with OpenAI # The OpenAI ecosystem: A developer's guide to building agents with OpenAI OpenAI ships something new almost every month. Models get renamed, preview features appear, and entire product lines arrive with little warning. If you're building agents, you need a clear picture of which pieces expect code, which are visual builders, and how they all fit together. This guide walks developers through OpenAI's ecosystem with a focus on building agents. We compare code-first APIs with no-code builders, explain what each product does, and where relevant, demonstrate how to integrate them with external tools using MCP. ## Code or no-code OpenAI offers two main paths for building agents: - The code-first route gives developers full control over prompts, state, and infrastructure. - The no-code route provides visual builders and managed hosting that business users can operate. You'll likely start with one and migrate to the other, or run both in parallel. Here's how they compare: | | Code-first APIs | No-code builders | |------------------|-------------------------------------------------------------|--------------------------------------------------------------| | Control level | Total control over prompts, state, infrastructure | Configuration only, OpenAI hosts | | Who maintains it | Developers | Product managers, support staff, operations, also developers | | Tool integration | Call any external service via code | Use built-in connectors or workspace-approved integrations | | When to use | Custom apps, compliance-driven workloads, advanced copilots | Fast pilots, embedded assistants, business-run automations | | Key products | Responses API, Realtime API, Agents SDK | AgentKit, ChatGPT GPT Builder, Workflows, Operator | The rest of this guide digs into each path and explains how the products within them work. ## Building agents with code This is the best place to start if you want full control over the prompts, latency, and infrastructure of your agent. The following APIs all sit behind the same pay-as-you-go account. ### Responses API: The foundation The Responses API replaces the old Chat Completions endpoint and the legacy Assistants API. It accepts multimodal input, streams outputs, handles tool calling and JSON mode, and supports the newer reasoning models (o1 and o3). Compared to the legacy Assistants API, the Responses API is stateless by default, so you decide where conversation history lives. Similarly, it improves on the old Chat Completions API by adding first-class JSON Schema validation and tool definitions. Use the Responses API when you need a single request-response cycle: send a prompt, and get a completion. The API supports function calling, so your agent can invoke tools mid-generation. You define tools using the JSON Schema, the model decides when to call them, and you execute the tool logic in your code before sending the result back. The following basic example calls the Responses API directly: ```ts const openai = new OpenAI({ apiKey: process.env.OPENAI_API_KEY }); async function main() { const response = await openai.responses.create({ model: "gpt-5", input: "What's the weather in San Francisco?", tools: [ { name: "get_weather", type: "function", function: { description: "Get current weather for a location", parameters: { type: "object", properties: { location: { type: "string" }, }, required: ["location"], }, }, }, ], }); console.log(response.output_text); } main(); ``` ### Realtime API: Voice and streaming The Realtime API handles WebRTC or WebSocket sessions with sub-second latency. You stream audio, tool results, or screen events, and the model responds in the same channel. Compared to plain Responses API calls, you trade stateless requests for a session object that handles turn-taking. Use the Realtime API for voice conversations, live transcription, or any scenario that requires continuous bidirectional communication. The API manages the session state, handles interruptions (when a user speaks over the model), and supports tool calling throughout the conversation. The diagram below shows the Realtime API architecture: - Your application establishes a WebSocket or WebRTC connection. - Audio streams in both directions, and the model can invoke tools mid-conversation. - When a tool call happens, the session pauses, your code executes the tool, and the result streams back into the conversation. This architecture enables natural voice interactions. The model can ask clarifying questions, call tools to fetch data, and respond with synthesized speech in real time without breaking the conversation flow. Check out our [MCP use case guide for Realtime agents](/mcp/using-mcp/use-cases) to learn how to build a voice assistant with the Realtime API. ### Agents SDK: Orchestration in code The Agents SDK is for developers who want to define agent logic in code rather than configuration. It works with both the Responses API and the Realtime API, allowing you to programmatically define workflows in Python (with Node.js support coming). The SDK is open source, so you can version control your agent definitions and test them locally before deployment. **How it works:** You define agents with instructions, tool catalogs, and models in code. The Agents SDK handles multi-step workflows, persistent state, and evaluation traces. Unlike the stateless Responses API, the SDK manages conversation history, tool outputs, and run metadata for you. Here's a basic example: ```python function App() { return ( ); } ``` ChatKit handles user identity automatically. Just pass a `userId`, and the assistant remembers context across sessions. The widget includes typing indicators, file uploads (PDFs, images), voice input, and conversation history. Users can scroll through past conversations or start new threads. #### ChatKit Studio ChatKit Studio is a visual interface for building and configuring ChatKit instances without code. It lets you configure models, instructions, data sources, tools, and UI themes, with a live preview panel for testing the assistant before deployment. We recommend using an OpenAI-hosted backend for ChatKit, where OpenAI handles hosting and scaling, but a self-hosted option is also available if you want to run ChatKit on your own infrastructure using the ChatKit Python SDK. ### ChatGPT MCP connectors ChatGPT supports MCP connectors across the Plus, Pro, Team, Enterprise, and Edu tiers. Register your MCP servers through the Connectors interface, and ChatGPT can call them during conversations. Unlike earlier limitations, MCP connectors now support both read and write operations, so ChatGPT can update tickets in your issue tracker, trigger workflows in your automation system, write to databases, and call internal APIs. MCP servers connect to ChatGPT via remote endpoints using Server-Sent Events (SSE) or streaming HTTP protocols. OAuth handles authentication when needed. ChatGPT displays confirmation modals before executing write or modify actions, and workspace admins control which users can register custom connectors. #### Atlas Browser OpenAI launched ChatGPT Atlas in October 2025, a Chromium-based browser with ChatGPT built into the browsing experience. Agent mode (available for Plus, Pro, and Business users) lets ChatGPT work autonomously within the browser, researching topics, automating tasks, planning events, or booking appointments while you browse. Atlas also has first-class support for ChatGPT connectors as well as for Apps SDK-based integrations. ### Widget Builder and Apps SDK: From limited tools to full dev support ChatGPT's development story started with `/search` and `/fetch` as the only tools available for deep research. Today, these limited tools have evolved into full developer support through the Apps SDK and Widget Builder. #### Apps SDK The Apps SDK is the foundation for building ChatGPT integrations. It bridges your backend services and ChatGPT, letting you define tools, handle authentication, and manage state across conversations. The SDK handles the protocol details so you focus on exposing your API's capabilities. At its core, the Apps SDK is an MCP server that runs in your infrastructure. You define tools using the standard MCP schema (with tool names, descriptions, input schemas, and handler functions). ChatGPT discovers these tools and calls them when users ask for something your app can do. The SDK adds features beyond basic MCP, including OAuth flows for user authentication, session persistence so your tools remember context across messages, and rate limiting per user or per workspace. It handles error recovery automatically. If your tool times out or returns an error, the SDK retries or prompts the user for clarification. Here's how you define a tool with the Apps SDK: ```ts async function loadKanbanBoard() { const tasks = [ { id: "task-1", title: "Design empty states", assignee: "Ada", status: "todo" }, { id: "task-2", title: "Wireframe admin panel", assignee: "Grace", status: "in-progress" }, { id: "task-3", title: "QA onboarding flow", assignee: "Lin", status: "done" } ]; return { columns: [ { id: "todo", title: "To do", tasks: tasks.filter((task) => task.status === "todo") }, { id: "in-progress", title: "In progress", tasks: tasks.filter((task) => task.status === "in-progress") }, { id: "done", title: "Done", tasks: tasks.filter((task) => task.status === "done") } ], tasksById: Object.fromEntries(tasks.map((task) => [task.id, task])), lastSyncedAt: new Date().toISOString() }; } server.registerTool( "kanban-board", { title: "Show Kanban Board", _meta: { "openai/outputTemplate": "ui://widget/kanban-board.html", "openai/toolInvocation/invoking": "Displaying the board", "openai/toolInvocation/invoked": "Displayed the board" }, inputSchema: { tasks: z.string() } }, async () => { const board = await loadKanbanBoard(); return { structuredContent: { columns: board.columns.map((column) => ({ id: column.id, title: column.title, tasks: column.tasks.slice(0, 5) // keep payload concise for the model })) }, content: [{ type: "text", text: "Here's your latest board. Drag cards in the component to update status." }], _meta: { tasksById: board.tasksById, // full task map for the component only lastSyncedAt: board.lastSyncedAt } }; } ); ``` The `context` object gives you access to the authenticated user, their workspace, and any state you've stored in previous tool calls. This lets you build stateful agents that remember preferences, cache API responses, or track multi-step workflows. The Apps SDK supports multiple authentication patterns. - **For workspace integrations:** Use OAuth to let users connect their accounts (for example, Slack, Google Calendar, or Salesforce). The SDK stores the OAuth tokens securely and refreshes them automatically. Your tool handlers receive the user's token via the context, so you make API calls on their behalf without exposing credentials to ChatGPT. - **For internal tools:** Skip OAuth and rely on ChatGPT's workspace auth. If an admin registers your app in a Team or Enterprise workspace, the SDK trusts that all users in that workspace are authorized. #### Widget Builder Built on top of the Apps SDK, Widget Builder lets you return rich UI components from your MCP tools, not just plain text. When ChatGPT calls your tool, it responds with interactive widgets, such as data tables, forms, charts, or custom visualizations. The user sees these rendered directly in the ChatGPT interface. Widgets bind to data using placeholders like `{{task.title}}` that fill in when your tool returns results. You can rapidly develop widgets using the low-code Widget Builder in ChatKit Studio. With Widgets, your MCP server does more than answer questions — it becomes interactive. A task management tool returns a widget with checkboxes to complete tasks. A CRM tool returns a customer card with click-to-call buttons. A metrics tool returns live charts that update when the user asks follow-up questions. Your MCP server defines tools with a `_meta` field that points to a widget URI. When ChatGPT calls the tool, your server returns data and the widget template. ChatGPT renders the widget using your template and the returned data. ```ts server.registerResource( "html", "ui://widget/widget.html", {}, async (req) => ({ contents: [ { uri: "ui://widget/widget.html", mimeType: "text/html", text: `
`.trim(), _meta: { "openai/widgetCSP": { connect_domains: [], resource_domains: ["https://persistent.oaistatic.com"], } }, }, ], }) ); ``` ### Workflows and Operator Workflows let you pin deterministic steps around model calls. You define stages that can branch, await human review, or trigger tools. Operator is OpenAI's research preview agent that controls a browser and handles long-running tasks. Both use the same tool definition format as the Responses API. Register your MCP tools once and make them available across these runtimes. Workflows is currently invite-only for enterprise customers; Operator is also on a waitlist and expects clear guardrails in your application. ## The supporting cast: Models, creative tools, and infrastructure These services share billing with the APIs but introduce their own review processes and constraints. We'll cover them briefly so you know where they fit. ### Models: Naming and routing OpenAI's model names shift frequently. As of early 2025, the main families are: - **GPT-5:** The flagship model for complex reasoning and multi-step tasks - **GPT-5-mini:** A lightweight variant optimized for speed and cost that is good for high-volume tasks - **GPT-5-nano:** The smallest, cheapest model in the family, which is best for fast responses for simple queries - **GPT-5-Codex (high), GPT-5-Codex (medium), GPT-5-Codex (low):** Specialized models for code generation and software development tasks, where high offers the most accuracy and low prioritizes speed All these models are available through the Responses API and ChatGPT (depending on your tier). The Codex variants power the Codex developer tools. Pricing varies by model, so we recommend routing expensive models through a dedicated MCP tool to audit usage. ### Creative endpoints: Images, audio, and video **DALL-E 3 image generation:** DALL-E 3 handles prompt-to-image generation, inpainting, and image variation. The responses return URLs or Base64 blobs. When using MCP, you wrap the call in a tool and return a resource with the download link so the client can show a thumbnail or fetch the file later. **Voice Engine and the Audio API endpoint:** With these tools, OpenAI covers both text-to-speech and speech-to-text. When these capabilities are paired with the Realtime API, latency is low enough for real-time agents. A common pattern is an MCP tool that takes text, calls the audio API, uploads the result to object storage, and returns a resource URI for the client to stream. This is the same pattern OpenAI uses in ChatGPT's mobile apps when you ask about a photo and expect a spoken answer. **Sora video:** OpenAI's text-to-video model, Sora, generates short, high-fidelity clips. Access is currently limited to creative partners and enterprise pilots. Rendering jobs take minutes, so treat them as asynchronous tasks: expose an MCP tool that submits the prompt, return a completion handle, and stream progress updates as the job advances. ### Infrastructure and retrieval **Embeddings and vector stores:** Embeddings endpoints (`text-embedding-3-large`, `text-embedding-3-small`, and domain variants) power retrieval pipelines. Vector stores add metadata filtering and chunk management. These services are regional, so double-check availability if your MCP servers run in another region. Treat long-running jobs as MCP completables: queue the work, return a completion handle, and let the client poll or subscribe. **Fine-tuning:** The Fine-tuning endpoint adapts a base model to your tone or domain, but you still call the resulting model through the Responses API. It lets you designate jobs and use fine-tuned model IDs in subsequent calls. **Batch jobs:** Batch processing lets you send tens of thousands of prompts at a lower cost. It pairs nicely with MCP. Expose a `submit_batch` tool and emit status updates as resources. **Files API:** The Files endpoint stores training data or retrieval corpora and is the only sanctioned way to share larger documents with OpenAI services. ## Connecting it all: MCP integration patterns MCP gives you one adapter surface as OpenAI's products evolve. Here are the main patterns for integrating your MCP servers with OpenAI's ecosystem. ### Direct integration: MCP server wraps OpenAI API This is the pattern we showed earlier with the Responses API. Your MCP server keeps your OpenAI API key server-side and exposes a tool that calls the API. The client (Claude, Cursor, or another MCP host) calls your tool, and you forward the request to OpenAI. This keeps secrets off the client and lets you audit all OpenAI usage in one place. Add rate limiting, cost tracking, or custom logging before forwarding calls. ### Reusable catalogs: Call from multiple runtimes The biggest MCP benefit is consistency. Once you model an internal API as an MCP server, you call it from Claude, ChatGPT, Cursor, and any other MCP-compatible runtime. Define your tool schema once, register it in ChatGPT Team workspaces and Claude projects, and both runtimes use it. When OpenAI adds a new agent product (like Workflows or Operator), you don't rebuild integrations — you just point it at your existing MCP catalog. ### Web search: Built-in tools and Deep Research **Responses API web search:** The Responses API includes a `web_search` tool that you enable per account. Pass the tool in your request, and the model decides when to use it. When triggered, the API returns responses with URL citations and source annotations. The limitation is that you don't control the search provider, citation format, or result ranking. **Deep Research:** ChatGPT's Deep Research feature (launched February 2025) is an autonomous agent that conducts multi-step web research. Give it a prompt, and it searches, analyzes, and synthesizes hundreds of sources to produce a comprehensive report. Deep Research takes 5-30 minutes and uses a version of the o4 model optimized for web browsing. It's available in the ChatGPT Plus, Team, Enterprise, and Pro tiers, with API access launched in June 2025. **Custom retrieval with MCP:** Instead of conducting generic web searches, Deep Research supports MCP tool queries that can fetch internal documentation, compliance databases, or proprietary data sources. This returns structured results (document titles, relevance scores, snippets, and metadata) that the model can reason over. Custom retrieval gives you control and auditability — you can log every query and view citations you can verify. ## Plans, pricing, and requirements OpenAI's offerings cluster into several pricing tiers. Here's a snapshot as of early 2025 to help you navigate what you need. ### API pay-as-you-go Pay-as-you-go is required for Responses API, Realtime API, Assistants API, embeddings, vector stores, fine-tuning, batch jobs, and creative endpoints. You pay per token or per request, and the price varies by model. Most regions require a verified billing method and a $5 prepayment to activate the account. Reasoning models (`o1` and `o3`) cost significantly more per token, so we recommend routing them through a dedicated MCP tool to audit usage. ### Add-on tools Adding web search capability enables `web_search` and `url_get` tools in the Responses API. Web search is still rolling out; Plus, Team, and Enterprise customers can usually flip it on through the dashboard, while some organizations need to email support to request access. Pricing is per search query (roughly $10 per 1,000 searches). To avoid this cost, build your own search MCP tool instead. ## Next steps for builders If you're a developer, start by building MCP servers that expose your own tools and data sources. Create servers that query your databases, call your internal APIs, access your file systems, or integrate with services you use (like GitHub, Stripe, and CRMs). Define each tool using the JSON Schema, handle authentication, and test locally before deploying. This gives you a reusable catalog of tools that works across Claude, Cursor, ChatGPT, and any other MCP-compatible runtime. If you're building a platform, decide whether to centralize tools using the Agents SDK or expose them directly through ChatGPT workspaces with the Apps SDK. A common approach is to use both: maintain one MCP catalog that powers both SDK-driven agents and workspace GPTs. For ChatGPT Team or Enterprise plans, register your MCP servers in the workspace so every GPT model inherits them automatically. This means you approve tools once, and they become available everywhere, including in ChatKit embeds and the desktop apps. Keep an eye on OpenAI's release notes. This ecosystem shifts fast, but MCP gives you one adapter surface as the products evolve. The more you express your systems as MCP tools, the easier it is to evaluate whatever OpenAI launches next without rebuilding integrations.