Enabling file streaming operations
Support for streaming is critical for applications that need to send or receive large amounts of data between client and server without first buffering the data into memory, potentially exhausting this system resource.
Streaming Download
If you create an endpoint with a top-level binary response body, you can treat that response as a streamable, and iterate over it without loading the entire response into memory. This is useful for large file downloads, long-running streaming responses and more.
In an OpenAPI documented this can be modeled as a binary stream. Here's an example of a get
operation with content type as application/octet-stream
.
/streamable: get: operationId: streamable responses: "200": description: OK content: application/octet-stream: schema: title: bytes type: string format: binary
For response streaming in TypeScript SDKs expose a ReadableStream
, a part of the Streams API web standard (opens in a new tab).
import fs from "node:fs";import { Writable } from "node:stream";import { SDK } from "@speakeasy/super-sdk";async function run() { const sdk = new SDK(); const result = await sdk.streamable("UR123"); const destination = Writable.toWeb( fs.createWriteStream("./report.csv") ); await result.data.pipeTo(destination);}run();
Streaming Uploads
Uploading a very large file is one use case where streaming can be useful.
Certain SDK methods will be generated that accept files as part of a multi-part request. It is possible and typically recommended to upload files as a stream rather than reading the entire contents into memory.
This avoids excessive memory consumption and potentially crashing with out-of-memory errors when working with very large files. In this example a request to upload a file is managed as a multipart/form-data
request.
/file: post: summary: Upload file operationId: upload requestBody: content: multipart/form-data: schema: $ref: '#/components/schemas/UploadFileRequest' required: true responses: '200': description: '' headers: Action-Id: required: true schema: type: string content: application/json: schema: $ref: '#/components/schemas/File'
As an example, in Node.js v20, streaming a large file to a server using an SDK is only a handful of lines. On the browser,
users would typically select files using <input type="file">
and the SDK call is identical to the sample code below.
import { openAsBlob } from "node:fs";import { SDK } from "@speakeasy/super-sdk";async function run() { const sdk = new SDK(); const fileHandle = await openAsBlob("./src/sample.txt"); const result = await sdk.upload({ file: fileHandle }); console.log(result);}run();
Depending on your JavaScript runtime, there are convenient utilities that return a handle to a file without reading the entire contents into memory:
- Node.js v20+: Since v20, Node.js comes with a native
openAsBlob
function innode:fs
(opens in a new tab). - Bun: The native
Bun.file
(opens in a new tab) function produces a file handle that can be used for streaming file uploads. - Browsers: All supported browsers return an instance to a
File
(opens in a new tab) when reading the value from an<input type="file">
element. - Node.js v18: A file stream can be created using the
fileFrom
helper fromfetch-blob/from.js
(opens in a new tab).