Enabling JSON lines responses

JSON Lines (JSONL) or Newline Delimited JSON (NDJSON) is a simple and efficient format for streaming structured data. Each line in the stream is a valid JSON object, making it ideal for streaming large datasets, log files, or real-time data feeds. This format is particularly useful for processing data line by line without loading the entire response into memory.

INFO

The format is known by two names and content types, which are completely interchangeable:

JSON Lines (JSONL): application/jsonl
Newline Delimited JSON (NDJSON): application/x-ndjson

Either content type can be used in the OpenAPI specification, as they are functionally identical (each line must be a valid JSON object that matches the specified schema).

Here’s an example of using an SDK to stream log data in JSONL/NDJSON format:

import { SDK } from '@speakeasy/sdk';
const sdk = new SDK();
async function streamLogs() {
  const result = await sdk.logs.fetch_logs();
  for await (const event of result) {
    // Each event is a parsed JSON object from the stream
    console.log(`[${event?.timestamp}] ${event?.message}`);
  }
}
streamLogs().catch(error => {
  console.error('Error streaming logs:', error);
});

INFO

The JSONL/NDJSON streaming feature is currently supported in TypeScript, Python, Go, and Java. Let us know if you’d like to see support for other languages.

Modeling JSONL/NDJSON in OpenAPI

To implement line-delimited JSON streaming in generated SDKs, model an API endpoint that serves a stream in the OpenAPI document. Each line in the response will be a JSON object matching the specified schema. Either application/jsonl or application/x-ndjson can be used as the content type.

Basic implementation

The example below shows an operation that streams log events:

paths:
  /logs:
    get:
      summary: Stream log events
      operationId: stream
      tags:
        - logs
      parameters:
        - name: query
          in: query
          required: true
          schema:
            type: string
      responses:
        "200":
          description: Log events stream
          content:
            # Either content type can be used:
            application/jsonl:
              schema:
                $ref: "#/components/schemas/LogEvent"
            # OR
            application/x-ndjson:
              schema:
                $ref: "#/components/schemas/LogEvent"
components:
  schemas:
    LogEvent:
      description: A log event in line-delimited JSON format
      type: object
      properties:
        timestamp:
          type: string
        message:
          type: string

Endpoints with multiple response types

For APIs that support both batch and streaming responses, use URL fragments to define separate paths for each response type:

paths:
  /analytics:
    get:
      summary: >
        Get analytics events as a batch response
      operationId: getBatch
      tags: [analytics]
      parameters:
        - name: start_date
          in: query
          required: true
          schema:
            type: string
            format: date
      responses:
        "200":
          description: Analytics events batch
          content:
            application/json:
              schema:
                type: array
                items:
                  $ref: "#/components/schemas/AnalyticsEvent"
  /analytics#stream:
    get:
      summary: >
        Stream analytics events in real-time
      operationId: stream
      tags: [analytics]
      parameters:
        - name: start_date
          in: query
          required: true
          schema:
            type: string
            format: date
      responses:
        "200":
          description: Analytics events stream
          content:
            application/x-ndjson:
              schema:
                $ref: "#/components/schemas/AnalyticsEvent"

Use the appropriate method based on the requirements:

import { SDK } from '@speakeasy/sdk';
const sdk = new SDK()
// Batch method
const batchResponse = await sdk.analytics.getBatch({
  start_date: "2024-01-01"
});
// Streaming method
const stream = await sdk.analytics.stream({
  start_date: "2024-01-01"
});
for await (const event of stream) {
  // handle stream
}