Azure Speech

Durable speech-to-text and text-to-speech with word timestamps, speaker diarization, subtitles, private artifacts, large files, and service-credit billing

Gencow provides Azure Speech through the generated ai.speech.* facade. Use it when you need word or phrase timestamps, speaker diarization, SRT/VTT subtitles, Azure neural voices, or speech jobs that continue while your app is asleep.

Provider credentials, media preprocessing, job execution, private artifact storage, and service-credit settlement are managed by the platform. Do not put Azure credentials in tenant app code or call the provider directly.

Setup

Add the AI component, then import the generated facade from gencow/ai.ts:

npx gencow@latest add AI

import { ai } from "./ai";

Choose a Speech Model

Model	Best for	Execution
`azure-speech-fast-transcription`	Word/phrase timing, diarization, language candidates, SRT/VTT, large audio and video	Durable async job
`gpt-4o-mini-transcribe`	Lower-cost plain transcription	Durable async job; large sources may be chunked
`gpt-4o-transcribe`	Higher-quality plain transcription	Durable async job; large sources may be chunked
`azure-speech-neural-tts`	Azure voice catalog, locale-specific neural voices, prosody controls, long TTS jobs	Sync for short text or durable async job
`gpt-4o-mini-tts`	Instruction-driven short-form TTS	Sync for short text or durable async job

Azure Speech-only options require an explicit Azure Speech model. The platform does not silently fall back to a plain transcript when timestamps, diarization, or subtitles were requested.

Upload Audio or Video

STT starts from a private app storage object. For small files, store the upload through a write procedure and pass the returned storageId to the speech API:

import { v } from "@gencow/core";
import { procedure } from "./runtime";

export const uploadAudio = procedure.mutation
    .name("speech.uploadAudio")
    .input(v.object({ file: v.file() }))
    .handler(async ({ context: ctx, input }) => {
        ctx.auth.requireAuth();

        const storageId = await ctx.storage.store(input.file);
        return { storageId };
    });

For large media, use the storage direct-upload grant flow so the source does not pass through tenant app memory. The hosted storage adapter selects the supported upload transport:

import { v } from "@gencow/core";
import { procedure } from "./runtime";

export const createAudioUpload = procedure.mutation
    .name("speech.createAudioUpload")
    .input(v.object({ filename: v.string(), contentType: v.string(), size: v.number() }))
    .handler(async ({ context: ctx, input }) => {
        ctx.auth.requireAuth();
        if (!ctx.storage.createUploadGrant) {
            throw new Error("Direct upload is unavailable");
        }

        return ctx.storage.createUploadGrant(input);
    });

export const finalizeAudioUpload = procedure.mutation
    .name("speech.finalizeAudioUpload")
    .input(v.object({ uploadId: v.string() }))
    .handler(async ({ context: ctx, input }) => {
        ctx.auth.requireAuth();
        if (!ctx.storage.finalizeUpload) {
            throw new Error("Direct upload is unavailable");
        }

        const result = await ctx.storage.finalizeUpload({
            uploadId: input.uploadId,
            visibility: "private",
        });
        return { storageId: result.assetId };
    });

The browser uploads bytes to the returned grant URL and then calls finalizeAudioUpload:

const grant = await createAudioUpload({
    filename: file.name,
    contentType: file.type || "application/octet-stream",
    size: file.size,
});

const uploadUrl = new URL(grant.url, window.location.origin);
let response: Response;

if (grant.method === "POST") {
    const form = new FormData();
    for (const [name, value] of Object.entries(grant.fields ?? {})) {
        form.append(name, value);
    }
    form.append("file", file);
    response = await fetch(uploadUrl, { method: "POST", body: form });
} else {
    response = await fetch(uploadUrl, {
        method: "PUT",
        headers: grant.headers,
        body: file,
    });
}

if (!response.ok) throw new Error(`Upload failed with HTTP ${response.status}`);

const { storageId } = await finalizeAudioUpload({ uploadId: grant.uploadId });

The effective upload limit is the lowest of the speech provider limit, your app storage file-size limit, remaining storage quota, and plan policy.

Submit an Azure Speech Transcription

ai.speech.transcribe() returns a job immediately. It does not wait for the provider to finish:

import { v } from "@gencow/core";
import { ai } from "./ai";
import { procedure } from "./runtime";

export const startTranscription = procedure.mutation
    .name("speech.startTranscription")
    .input(v.object({ storageId: v.string(), requestId: v.string() }))
    .handler(async ({ context: ctx, input }) => {
        ctx.auth.requireAuth();

        const job = await ai.speech.transcribe({
            storageId: input.storageId,
            model: "azure-speech-fast-transcription",
            language: "ko-KR",
            timestamps: "word",
            diarization: { enabled: true, maxSpeakers: 2 },
            subtitles: ["srt", "vtt"],
            profanityFilter: "masked",
            idempotencyKey: input.requestId,
        });

        return { jobId: job.id, status: job.status };
    });

Use either one known language or up to 10 candidateLanguages. Do not send both in the same request:

const job = await ai.speech.transcribe({
    storageId,
    model: "azure-speech-fast-transcription",
    candidateLanguages: ["ko-KR", "en-US", "ja-JP"],
    timestamps: "phrase",
});

STT Options

Option	Values	Notes
`language`	BCP-47 locale such as `ko-KR`	Mutually exclusive with `candidateLanguages`
`candidateLanguages`	1 to 10 BCP-47 locales	Provider chooses among the candidates
`timestamps`	`none`, `phrase`, `word`	`word` includes word-level start and duration
`diarization`	`false` or `{ enabled: true, maxSpeakers?: 2..35 }`	Returns anonymous IDs such as `speaker-0`
`subtitles`	`srt`, `vtt`	Stored as private artifacts
`profanityFilter`	`masked`, `raw`, `removed`	Defaults to `masked`
`idempotencyKey`	Non-empty string, max 256 characters	Reusing the same key prevents duplicate jobs

Diarization and subtitles need phrase timing. If timestamps is omitted, the platform promotes it to phrase; explicitly combining either feature with timestamps: "none" returns a validation error.

Async Jobs and App Sleep

Speech transcription is always exposed as a durable platform job, for both short and long inputs. Your app does not need a Gencow workflow to keep the job alive. After submission, the tenant app may finish the request or go to sleep; the media worker continues processing and stores the result durably.

Persist the jobId in your product data and poll it from a later request:

const job = await ai.speech.getTranscriptJob(jobId);

switch (job.status) {
    case "queued":
    case "reserved":
    case "running":
    case "retrying":
        return { state: "processing" as const };
    case "succeeded":
        return { state: "ready" as const, transcriptStorageId: job.transcriptStorageId };
    default:
        return { state: "failed" as const, error: job.error };
}

ai.speech.waitForTranscript() is a polling convenience for tests, scripts, or bounded requests that stay awake. Do not hold a normal HTTP request open for a large recording. Return the job ID and poll instead.

Status	Meaning
`queued`	Accepted and waiting for worker capacity
`reserved`	Service credits are reserved
`running`	Media or provider processing is active
`retrying`	A transient failure is waiting for another attempt
`succeeded`	Transcript and requested artifacts are ready
`failed_refunded`	Processing failed and reserved credits were refunded
`canceled_refunded`	Job was canceled and reserved credits were refunded
`payment_failed`	Credit reservation or final settlement could not complete

Read Transcript Results

Successful Azure Speech jobs store a private canonical transcript. Fetch it by job ID:

const result = await ai.speech.getTranscriptResult(jobId);

if (result.object === "gencow.speech.transcript") {
    console.log(result.text);

    for (const phrase of result.phrases) {
        console.log(phrase.speaker, phrase.startMs, phrase.text);
        for (const word of phrase.words ?? []) {
            console.log(word.text, word.startMs, word.durationMs);
        }
    }
}

The canonical shape includes:

type AzureSpeechTranscript = {
    object: "gencow.speech.transcript";
    version: 1;
    provider: string;
    model: "azure-speech-fast-transcription";
    durationMs: number;
    languages: string[];
    timestampLevel: "none" | "phrase" | "word";
    text: string;
    phrases: Array<{
        text: string;
        startMs: number;
        durationMs: number;
        speaker?: string;
        confidence?: number;
        words?: Array<{
            text: string;
            startMs: number;
            durationMs: number;
            confidence?: number;
        }>;
    }>;
};

Fetch requested subtitles separately:

const srt = await ai.speech.getTranscriptSubtitle(jobId, "srt");
const vtt = await ai.speech.getTranscriptSubtitle(jobId, "vtt");

The transcript, SRT, and VTT remain private app storage objects. Do not publish them unless your product explicitly intends public access.

STT Limits and Media Processing

Path	Source size	Duration	Processing
Azure Speech Fast	Less than `500,000,000` bytes	Less than 5 hours	Supported audio streams directly; video/unsupported media is extracted or transcoded
Azure Speech Fast with diarization	Less than `500,000,000` bytes	Less than 2 hours	Same path, no chunking
Azure OpenAI transcription	Up to `500 MiB`	Up to 2 hours	Sources over `25 MiB` are preprocessed and chunked

The Azure Speech size cap is exclusive: 499,999,999 bytes may be accepted, while 500,000,000 bytes is rejected before provider work. The duration cap is also exclusive for Azure Speech.

Common direct audio types include AAC, AMR, FLAC, M4A/MP4 audio, MP3, OGG, Opus, Speex, WAV, WebM audio, and WMA. Video sources are accepted through app storage and sent to the media worker for audio extraction/transcoding.

Azure Speech Fast does not split a recording into provider chunks. This preserves word timing and speaker continuity. If extraction or transcoding is needed, the media worker performs it outside the tenant app runtime.

Azure Neural Text-to-Speech

List voices for a locale instead of hardcoding an unverified voice name:

const voices = await ai.speech.listVoices({ locale: "ko-KR" });
const voice = voices.data[0];

For up to 4,096 characters, synthesize synchronously. The result is stored in private app storage:

const audio = await ai.speech.synthesize({
    input: "안녕하세요. Gencow Azure Speech 예제입니다.",
    model: "azure-speech-neural-tts",
    language: "ko-KR",
    voice: voice.name,
    format: "mp3",
    quality: "high",
    rate: 1.0,
    pitchSemitones: 0,
    volume: 100,
    includeReadGrant: true,
});

return {
    storageId: audio.storageId,
    downloadUrl: audio.downloadUrl,
    bytes: audio.bytes,
    creditsCharged: audio.creditsCharged,
};

Option	Values
`language`	Required BCP-47 locale
`voice`	Required name from `listVoices()`
`format`	`mp3`, `wav`, `opus`
`quality`	`standard`, `high`; high currently requires MP3
`rate`	`0.5` to `2.0`
`pitchSemitones`	`-12` to `12`
`volume`	`0` to `100`

includeReadGrant: true returns a short-lived download URL. The durable result is storageId; generate a new private read grant later when an old URL expires.

Long TTS Jobs

For text over 4,096 characters, store UTF-8 text privately and create a durable synthesis job. The job input supports up to 100,000 Unicode code points and 1 MiB of stored text:

const inputStorageId = await ctx.storage.storeBuffer(
    Buffer.from(longNarration, "utf8"),
    "narration.txt",
    "text/plain; charset=utf-8",
);

const job = await ai.speech.createSynthesisJob({
    inputStorageId,
    model: "azure-speech-neural-tts",
    language: "ko-KR",
    voice: voice.name,
    format: "mp3",
    quality: "high",
    idempotencyKey: narrationId,
});

return { jobId: job.id };

narrationId must be a persisted product request ID that is reused when the same synthesis request is retried.

Poll with ai.speech.getSynthesisJob(job.id). After success, job.outputStorageId identifies the private audio object:

const job = await ai.speech.getSynthesisJob(jobId);
if (job.status !== "succeeded" || !job.outputStorageId) {
    return { status: job.status };
}

const grant = await ctx.storage.createReadGrant(job.outputStorageId, {
    ttlSeconds: 300,
    disposition: "attachment",
});

return { status: job.status, storageId: job.outputStorageId, downloadUrl: grant.url };

Long TTS is also independent of tenant app sleep. The platform chunks text at safe boundaries, synthesizes each part, and stores one final private audio artifact. The maximum output size is 32 MiB.

Service-Credit Cost

Azure Speech uses Gencow service credits at the configured Azure retail-cost meter. 10,000 service credits represent USD 1.

Model	Meter	Current rate
`azure-speech-fast-transcription`	Audio duration	`1 credit / second`
`azure-speech-neural-tts`	Azure billable characters	`0.15 credit / character`

Examples:

60-minute STT = 3,600 seconds x 1 credit = 3,600 credits (USD 0.36)
1,000 Hangul TTS characters x 0.15 credit = 150 credits (USD 0.015)

Han ideographs count as two Azure billable characters; Hangul counts as one. Prosody controls such as rate, pitch, and volume add small SSML markup and can slightly increase the billable character count.

Diarization, word timestamps, SRT, and VTT currently have no separate Gencow surcharge. The platform reserves credits before provider work and captures the actual amount on success. Provider and processing failures end as failed_refunded; the reserved amount is returned. The job/result metadata is authoritative, so do not calculate the final charge only in browser code.

Rates can change when the upstream retail meter changes. Check the current model catalog and returned credit metadata for production billing displays.

Production Checklist

Keep source media, transcripts, subtitles, and synthesized audio private by default.
Use direct-upload grants for large media instead of buffering it in an app request.
Store the speech jobId in product data and make submission idempotent.
Poll durable job status; do not keep a normal HTTP request open for long media.
Treat succeeded as ready only when the result storage ID is present.
Show failed_refunded and payment_failed as different user states.
Create short-lived read grants only when a browser needs a private artifact.
Never log transcript text, private file URLs, credentials, or raw provider responses.

RAG & Memory CLI Reference