Blog

Why Temporal Exists, and the Four Services Behind Every Workflow Step

The Temporal server never runs your code: it remembers (event history in PostgreSQL) and hands out jobs (task queues). This opener of a series on self-hosting Temporal on PostgreSQL covers replay and determinism, then traces one request hop by hop through Frontend, History and Matching, counting every database transaction along the way.
24 min readPart 1 of 10
On this page
  1. The problem: forgetting where you were
  2. The event history
  3. The server never runs your code
  4. Replay, determinism, and at-least-once
  5. Four services, split by how they grow
  6. The mechanics every request shares
  7. Two kinds of task, served by long poll
  8. Every History save: an append, then one transaction
  9. Follow-up tasks: a transactional outbox
  10. Sync match, backlog, and the handshake
  11. Tracing one workflow
  12. Start
  13. Activity scheduled, started, completed
  14. Time is a row
  15. One hot workflow ID
  16. Count the trips
  17. Four patterns, and where to look
  18. What I'd tell myself before go-live

TL;DR: The Temporal server never runs your code. Your workers do. The server does two things only: it remembers (it appends every finished step, with its result, to an event history in PostgreSQL) and it hands out jobs (it puts workflow tasks and activity tasks on task queues for workers to long-poll). When a worker dies, another one replays the workflow code from the top and reuses the recorded results, which is why workflow code must be deterministic and every side effect belongs in an activity. Inside the server, four services split the work: Frontend (stateless front door), History (the only writer of workflow state, partitioned into shards), Matching (task queues), and the internal worker (Temporal's own housekeeping). Every workflow step is one History save on one shard: append events, then one transaction for the mutable state plus follow-up task rows that a background processor later delivers to Matching. A one-activity workflow costs about 7 transactions, and that arithmetic is why History plus PostgreSQL is almost always the bottleneck.


"Temporal is slow" is the sentence that starts most Temporal investigations, and it can mean very different things: the workers are out of slots, a History host is handing its shards over, PostgreSQL commits are slow, or one workflow ID is being hammered with signals. Each has a different fix, and none of them shows up clearly on a CPU graph. You can only tell them apart if you know the path a request takes: which service handles it, which rows it writes, and which background helper picks up the follow-up work.

This is the first post in Running Temporal Server in Production, a series about self-hosting Temporal Server on PostgreSQL, written from running it in production. The series goes through the machine (Frontend, History, shards, Matching, workers), how your code meets it (timeouts, retries, determinism), the PostgreSQL persistence layer and memory, metrics and capacity planning, cluster membership and scaling, and finally failure drills and a reference production architecture. It ends with a one-page cheat sheet.

This post lays the foundation: why Temporal exists, what the server actually does, and one request traced hop by hop with the database writes counted.

The problem: forgetting where you were

Take a boring job: send an invoice. (1) Render the PDF. (2) Email it to the client. (3) Mark it "sent" in the database.

The process crashes right after step 2. If it starts over, the customer gets two emails. If it gives up, the invoice is never marked sent. The root cause is that a normal program keeps "which step am I on?" in memory, and memory is gone when the process crashes, the pod is evicted, or pm2 restarts your Node app during a deploy.

Try to fix it yourself and you rebuild Temporal one piece at a time:

  • Retry the whole thing: a crash after step 2 re-runs step 2 and sends a second email.
  • Save progress in a table: the crash can land between "email sent" and "row updated". You also need each step's result (step 2 needs the PDF URL), not just "done".
  • Add timers ("remind them if unpaid in 7 days"): you can't sleep for 7 days in memory, so you add a scheduled-jobs table, a cron, and locking so two crons don't fire the same reminder.
  • Run on many machines: two copies grab the same invoice, so you add leases and owner columns.

You've now built a durable log of finished steps and their results, a scheduler for future work, a queue that hands work to machines, and locks so only one machine owns a job. That's Temporal. Every part of it exists to protect against a specific crash or race. When something later in this series looks overcomplicated, ask which crash it's protecting against. There's always one.

The event history

Temporal's version of the progress table is the event history: an append-only log of everything that happened to one workflow run. Each event has an increasing ID. The server writes it, workers read it, and nothing ever edits it. The easiest way to picture it is a friend with a notebook writing down every step you finish.

For the three-step invoice, a finished run looks like this (real event names):

IDEventMeaning
1WorkflowExecutionStartedSomeone asked for invoice 42 (with the input)
2–4WorkflowTaskScheduled / Started / CompletedA worker ran the workflow code and decided: run makePdf
5ActivityTaskScheduledmakePdf is waiting for a worker
6ActivityTaskStartedA worker ran it (written at the end, with the final attempt number)
7ActivityTaskCompletedResult = the PDF URL
8–13decide, then sendEmailSame six-event pattern
14–19decide, then markSentSame six-event pattern
20–22WorkflowTaskScheduled / Started / CompletedDecide again: nothing left, finish
23WorkflowExecutionCompletedDone

Three things are already visible:

  • Results live in the history. Event 7 carries the PDF URL. That's how sendEmail gets it after a crash.
  • The rhythm is "decide, do, decide, do, …, finish." The whole server architecture follows from it.
  • 23 events for 3 business steps. Each is a database write somewhere. Keep that number in mind.

In PostgreSQL the events are stored in batches as encoded blobs in history_node, with branch metadata in history_tree. You read them with the CLI, not SQL:

bash
# One workflow's history, event by event
temporal workflow show --workflow-id invoice-42 --namespace default
 
# Count events (replay cost grows with this number)
temporal workflow show --workflow-id invoice-42 --output json | jq '.events | length'
 
# Current summary: status, pending activities and their attempt counts
temporal workflow describe --workflow-id invoice-42

Besides the history, the server keeps a short mutable state per run: what's pending right now (activities, timers, children, signals), the next event ID, and the current workflow task. That's one executions row plus side tables, updated on every step. The history is for replay. Mutable state is what the server uses for its own quick decisions.

The server never runs your code

Here is the invoice in TypeScript:

typescript
// activities.ts: real side effects; may fail, may be retried
export async function makePdf(invoiceId: string): Promise<string> { /* render, upload, return URL */ return '...'; }
export async function sendEmail(invoiceId: string, pdfUrl: string): Promise<void> { /* call the mail API */ }
export async function markSent(invoiceId: string): Promise<void> { /* UPDATE invoices SET status='sent' */ }
 
// workflows.ts: only decides the order
import { proxyActivities } from '@temporalio/workflow';
import type * as activities from './activities';
 
const { makePdf, sendEmail, markSent } = proxyActivities<typeof activities>({
  startToCloseTimeout: '1 minute',      // how long one attempt may take
});
 
export async function sendInvoice(invoiceId: string): Promise<void> {
  const pdfUrl = await makePdf(invoiceId);   // step 1
  await sendEmail(invoiceId, pdfUrl);        // step 2
  await markSent(invoiceId);                 // step 3
}
 
// starter.ts: your app asks the server to start one run
await client.workflow.start(sendInvoice, {
  taskQueue: 'email_queue',       // the queue where tasks for this workflow are placed
  workflowId: 'invoice-42',       // business ID; unique per namespace while running
  args: ['42'],
});

The workflow decides the order of steps, activities touch the outside world, and both run inside your worker process (in my case Node apps under pm2). That leads to the idea the rest of this series is built on:

The Temporal server never runs your code. Your workers do. The server only remembers (writes the history to PostgreSQL) and hands out jobs (puts tasks on queues for workers to pick up).

The server never sees sendInvoice or sendEmail. It only sees what the worker sends back: "I decided to run sendEmail" and "sendEmail finished, here's the result". So you're operating two separate systems:

Temporal serverSDK worker (your app)
What it doesRemembers, hands out jobs, keeps timeRuns workflow and activity code
Deployed asInfrastructure (temporal-server)Application code (pm2, a k8s Deployment…)
Where it gets slowHistory + PostgreSQL, Matching, Frontend limitsToo few pollers or slots, slow activities, CPU
What "down" meansNothing moves; nothing committed is lostJobs wait in queues; the server is healthy but idle

One useful consequence: if all your workers are down, the server keeps working. Timers still fire, signals are still recorded, new workflows are still accepted. Tasks wait until a worker comes back, and then the backlog drains in a burst. Nothing is lost, only paused.

The trap is that "Temporal is slow" often means "the workers are slow", and adding server hosts does nothing when workers have no free slots. Schedule-to-start latency rising while temporal_worker_task_slots_available sits at 0 is a worker problem. The metrics post in this series covers that triage in full.

Replay, determinism, and at-least-once

When a worker dies mid-workflow, the next one doesn't jump to step 4. It runs the workflow function from the top with the whole history in hand. That's replay.

Workers never write events. Running workflow code produces commands ("schedule activity sendEmail", "start a timer", "complete"), and the server turns commands into events. During replay, each time your code would produce a command, the SDK checks the next event in the history. If ActivityTaskScheduled(makePdf) and ActivityTaskCompleted(url) are already there, the SDK doesn't send the command. It resolves the await immediately with the saved URL. The first command with no matching event is new, so it goes to the server and real work starts.

Replay: each line of workflow code is checked against the history; matched steps reuse saved results, the first unmatched step is sent to the server, and a changed decision causes a non-determinism error.
Replay: each line of workflow code is checked against the history; matched steps reuse saved results, the first unmatched step is sent to the server, and a changed decision causes a non-determinism error.

Replay runs on the worker, in memory, and is usually milliseconds for short histories. A worker that still has the run in its sticky cache skips it and only receives new events. Full replay happens after a restart, an eviction, or a handover to another worker, which is why a pm2 restart at peak makes workflow tasks slow for about a minute. Cost grows with history length, so a 50,000-event history is expensive to replay. That's one reason continue-as-new exists.

Replay only works if the code makes the same decisions every time. If the code flips a coin and it lands differently on replay, the code no longer matches the history, and you get a non-determinism error (NDE). So workflow code must be deterministic (no direct randomness, clock reads, API calls, or config that can change), and anything non-deterministic goes in an activity, whose result is recorded and read back. Waiting 7 days is fine in a workflow with the SDK's sleep(), because the server keeps the timer.

The TypeScript SDK sandboxes workflow code: Math.random() is seeded and Date.now() returns the workflow task's time, so both are safe, and network modules simply aren't available (exact behaviour depends on SDK version). On a mismatch it raises DeterminismViolationError and by default fails the workflow task, not the workflow. The server retries the task, so the workflow looks stuck rather than failed until you fix and redeploy. Versioning and patched() get their own post later in the series.

The history makes decisions exactly-once. It does not make the outside world exactly-once:

Crash momentHistory saysWhat happens nextSecond email?
Before sendEmail startedmakePdf completed, nothing for sendEmailReplay reaches sendEmail, finds no entry, schedules itNo
While sendEmail runsActivityTaskScheduled(sendEmail), no completionServer waits for the attempt's timeout, then retriesPossibly
After sendEmail completedActivityTaskCompleted(sendEmail)Replay reuses the result, schedules markSentNo

The middle row is the real edge: activities are at-least-once. If the worker dies after the mail API accepted the message but before it reported "done", the server can't know the email left, so it retries. "Temporal is exactly-once" isn't a reason to skip an idempotency key on a charge-card activity. Pass a stable key (workflow ID + activity ID) to the provider, or check "already sent?" first.

The flip side is the rule the whole server is built on: nothing is confirmed until it's written to PostgreSQL. If the server crashes before confirming, the caller gets an error and retries. Server failures cost time, never data. "Paused, not lost" will come up in every failure scenario in this series.

Two settings shape the history directly. Workflows have no default retry policy (activities do, with unlimited attempts). History length and size are capped by limit.historyCount.warn / .error (10,240 / 51,200 events) and limit.historySize.warn / .error (10 MB / 50 MB). Those defaults are version-dependent, so check common/dynamicconfig/constants.go at your release tag and your own overrides.

Four services, split by how they grow

One process could do all of this, and on a laptop it works. In production the jobs grow differently: 2,000 polling workers need front-door capacity but not more history writing, and a burst of workflow steps needs the opposite. They also differ in state. The front door forgets everything after each request, but the history writer must never have two copies writing the same workflow. Put them in one process and every restart is as delicate as the most delicate part. So Temporal splits the server by how each job grows and what it has to remember.

Temporal service topology: your client app, UI and SDK workers talk only to Frontend through the load balancer; Frontend routes to the History shard owner or the Matching partition owner; History writes events, mutable state and tasks to the default store and visibility rows to the visibility store.
Temporal service topology: your client app, UI and SDK workers talk only to Frontend through the load balancer; Frontend routes to the History shard owner or the Matching partition owner; History writes events, mutable state and tasks to the default store and visibility rows to the visibility store.

Frontend is the front door on port 7233. Clients, workers, the CLI and the UI all come in here. It handles auth, validation, namespace lookup and rate limiting, then routes the request to the History host that owns the shard for this workflow ID, or, for polls, to the Matching host that owns the task-queue partition. It's stateless.

History is the core and the only writer of workflow state. The workflow ID space is split into a fixed number of shards, and each shard has exactly one owning History host at a time. History appends events, updates mutable state, and creates the follow-up tasks that drive everything else.

Matching holds task queues like email_queue. Workers long-poll it, and it pairs tasks with waiting polls. Each queue is split into partitions, each owned by one Matching host.

Internal worker (--service=worker) runs Temporal's own system workflows: Schedules, batch operations, namespace deletion, scanners and scavengers. It's a client of Frontend, like your workers.

Around them: your client app (anything calling StartWorkflowExecution, SignalWorkflowExecution, …), the UI (temporalio/ui, a separate process and just another Frontend client), a load balancer that spreads gRPC/HTTP2 connections across Frontends only, and PostgreSQL as two logical databases: the default store (temporal: shards, executions, history, task queues) and the visibility store (temporal_visibility: executions_visibility, for List and Search).

"Worker" means two things. An SDK worker is your process, and its temporal_* metrics come from your process. The internal worker is a Temporal server role. When an activity backlog grows, adding Temporal server hosts does nothing if the SDK workers are saturated.

Frontend is easy to operate: add copies, and a restart costs a few SDK retries and reconnecting long polls. History is the delicate one, because a workflow can have only one writer at a time. That single problem is why shards exist, and the shards post covers it in detail. When a History host restarts, everything it confirmed is already in PostgreSQL. Its shards just have no owner for a few seconds, so their workflows pause. Another host claims them by bumping the shard's range_id and loads them from PostgreSQL with a cold cache, so latency spikes for a while. Some background tasks run twice, which is harmless because each re-checks state first. Frontend restart = a few retries. History restart = a short pause plus an ownership handover. Neither loses data.

The mechanics every request shares

Two kinds of task, served by long poll

The server hands out exactly two kinds of task. A workflow task means "run the workflow code forward until it blocks again". It's short, deterministic, and produces commands. An activity task means "run this side-effecting function". It can be long and produces a result or an error. Both go through Matching on separate internal queues, even when they share a name like email_queue.

Workers get tasks by long poll: they ask once and Matching parks the request for up to matching.longPollExpirationInterval (default 1 minute; the SDK's own poll timeout is a bit longer, around 70 s in the Go SDK). The payoff is zero dispatch delay. The cost is that every parked poll holds memory on Frontend (the connection) and Matching (the waiting request), and their number depends on pollers, not load:

text
waiting polls ≈ worker processes × task queues × (workflow pollers + activity pollers)
 
200 workers × 3 task queues × 10 pollers        =  6,000 waiting polls
200 workers × 3 task queues × (10 + 10 pollers) = 12,000 waiting polls

Every pm2 instance opens its own pollers for every queue it serves. Adding instances "to go faster" multiplies parked polls even when there's no extra work, so do this arithmetic before scaling workers out. Also: never alert on service_latency for Poll* operations. They're slow by design. Alert on schedule-to-start latency and backlog instead.

Every History save: an append, then one transaction

Every change to a workflow is one History save on one shard, and it always has the same two-write shape:

  1. Append events to history_node (persistence operation AppendHistoryNodes; a new run also writes history_tree).
  2. One SQL transaction that checks the shard's ownership stamp, updates mutable state, and inserts follow-up tasks (CreateWorkflowExecution / UpdateWorkflowExecution).
text
             ┌─ AppendHistoryNodes ───────────► history_node (+ history_tree on new run)
History  ────┤
(shard S)    └─ BEGIN
                  SELECT range_id FROM shards ... FOR SHARE   (fencing; shard-row updates take FOR UPDATE)
                  UPSERT executions             (mutable state blob, next_event_id, db_record_version)
                  UPSERT/DELETE *_info_maps     (pending activities/timers/children/signals)
                  UPSERT current_executions     (only on start/close/continue-as-new)
                  INSERT transfer_tasks / timer_tasks / visibility_tasks
                COMMIT

A host that has lost the shard fails the range_id check with ShardOwnershipLostError. That's how a zombie node is kept from corrupting a workflow. If the transaction fails after the append succeeded, the appended events are orphans that nothing points to, and the history scavenger on the internal worker deletes them later.

Two locks sit in front of this. A per-workflow lock serializes updates to one workflow. A per-shard semaphore, history.shardIOConcurrency (default 1), limits concurrent persistence writes per shard. The shards post covers both.

Follow-up tasks: a transactional outbox

The rows inserted in that transaction are history tasks, the to-do notes for follow-up work:

TableExamplesExecuted byAction
transfer_tasksWorkflowTask, ActivityTask, SignalExecution, StartChildExecution, CloseExecutionthe shard's transfer queue processorRPC to Matching or to another shard
timer_tasksUserTimer, ActivityTimeout, WorkflowTaskTimeout, ActivityRetryTimer, WorkflowBackoffTimer, DeleteHistoryEventthe shard's timer queue processorfire at visibility_timestamp
visibility_tasksStartExecution, UpsertExecution, CloseExecutionthe visibility queue processorwrite executions_visibility

This is a transactional outbox, and it's why Temporal doesn't lose work. If History crashes right after the commit, the task row is still in PostgreSQL, and the next shard owner reloads and executes it. Executors are idempotent: before acting, each one re-checks mutable state ("is this activity still scheduled with this ID?") and drops stale tasks. A task can execute twice, but its effect happens once. Finished rows are range-deleted.

Sync match, backlog, and the handshake

When the transfer processor calls Matching (AddWorkflowTask / AddActivityTask), either a worker poll is already parked on that partition and gets the task directly, a sync match with no DB write, or no poll is waiting and Matching inserts the task into the tasks table (CreateTasks) as backlog. Each backlog task costs an insert, a read and a delete, plus task_queues ack-level updates. When the sync-match ratio drops, it's usually not enough worker capacity, not Matching, and Matching's PostgreSQL writes go up. Worker capacity is a database-load lever.

Before handing a task over, Matching asks History "may I start this?" (RecordWorkflowTaskStarted / RecordActivityTaskStarted). Matching's copy can be stale: already started, timed out, or the workflow closed. Without this handshake, a duplicate or expired task would reach a worker and the work would run twice. The handshake also records "started" and returns the current attempt number. If History says the task is stale, Matching drops it and keeps the poll waiting.

Tracing one workflow

Start

Sequence of a workflow start: a synchronous round trip from client through Frontend to History and two PostgreSQL writes, followed by asynchronous dispatch of the first workflow task through the transfer queue, Matching and the handshake.
Sequence of a workflow start: a synchronous round trip from client through Frontend to History and two PostgreSQL writes, followed by asynchronous dispatch of the first workflow task through the transfer queue, Matching and the handshake.

  1. Client → Frontend: StartWorkflowExecution(workflowId, workflowType, taskQueue, input, requestId, …). Frontend authorizes, validates, applies rate limits (frontend.rps, frontend.namespaceRPS) and computes the shard, then looks up its owner in the History membership ring:

    text
    shardID = farm.Fingerprint32(namespaceID + "_" + workflowID) % numHistoryShards + 1
  2. Frontend → History (shard owner). History builds mutable state plus 1 WorkflowExecutionStarted and 2 WorkflowTaskScheduled, and writes AppendHistoryNodes.

  3. CreateWorkflowExecution in one transaction: an executions row; current_executions, which is where workflow-ID uniqueness is enforced (an open run means WorkflowExecutionAlreadyStarted, subject to the ID reuse/conflict policy, and a retried start with the same requestId returns the same run ID); a WorkflowTask transfer task; a StartExecution visibility task; and timeout timers only if set.

  4. Run ID returned. The workflow is now durable. If every server process restarted now, nothing would be lost.

  5. Asynchronously: the transfer processor (notified in memory right after the commit) calls AddWorkflowTask, Matching sync-matches or backlogs it, the handshake appends 3 WorkflowTaskStarted with a WorkflowTaskTimeout timer (start-to-close, default 10 s), and the worker receives events 1–3.

That's about 4 History write round trips plus a visibility write before the workflow has done anything useful.

Note what is missing from steps 1–4: History never calls Matching inside your start request. A slow Matching delays the first workflow task, not the start. If starts return in 15 ms but workers get the task 4 s later, the task waited in one of two places: History's transfer queue (task_latency_queue) or the Matching backlog (temporal_workflow_task_schedule_to_start_latency, sync-match ratio). With eager workflow start (SDK opt-in plus system.enableEagerWorkflowStart, default true on recent main and off in older releases), the first workflow task is returned in the start response and step 5 disappears.

Activity scheduled, started, completed

await sendEmail(...) doesn't call sendEmail. The worker ends its workflow task with RespondWorkflowTaskCompleted([ScheduleActivityTask]). History fills in the default activity retry policy if none was given, appends 4 WorkflowTaskCompleted and 5 ActivityTaskScheduled, and in one transaction writes an activity_info_maps row (attempt 1, timeouts, policy), an ActivityTask transfer task, and an ActivityTimeout timer. Dispatch goes through Matching on the activity queue, and at the handshake History updates activity_info_maps and writes start-to-close and heartbeat timers, but appends no event.

That's deliberate. The server source says so directly: for an activity with a retry policy (nearly all of them), "we might need to retry, so do not append started event just yet, instead update mutable state and will record started event when activity task is closed." Failed attempts don't bloat the history. (Temporal's own architecture doc simplifies this and shows the event at start. The source is authoritative.) Each heartbeat is another activity_info_maps update, with some throttling depending on version, so thousands of activities heartbeating every second means thousands of DB updates per second.

When the worker reports RespondActivityTaskCompleted, History checks the token against the current attempt. A stale one (for example the attempt already timed out) is rejected with NotFound and nothing is written. Otherwise History appends 6 ActivityTaskStarted (deferred, with the final attempt), 7 ActivityTaskCompleted, and 8 WorkflowTaskScheduled, deletes the activity_info_maps row, and writes a WorkflowTask transfer task. That task goes to the sticky queue of the worker that ran the previous workflow task. That worker still has the run cached, so it receives only events 6–9 and continues after the await.

Two details matter in production:

  • Buffered events. If a workflow task is already running when an activity completes or a signal arrives, History doesn't start a second one. Two concurrent deciders could issue conflicting commands, so there's only one decision in flight per workflow. New events are buffered (capped by history.maximumBufferedEventsBatch, default 100 on recent main) and flushed when the running task completes. It's also why CompleteWorkflowExecution is rejected with UnhandledCommand while buffered events exist.
  • Sticky misses. If the sticky task isn't picked up within the sticky schedule-to-start timeout (SDK default 5 s), History reschedules on the normal queue and the next worker replays the full history. Frequent restarts turn every workflow task into a full replay, which shows up as worker CPU and ReadHistoryBranch reads.

Time is a row

await sleep('1 hour') keeps nothing running for an hour.

Timers and retries: a user timer is a timer_tasks row fired by the shard's timer queue; an activity retry updates mutable state and writes a retry timer with no history events; a workflow retry closes the run and creates a new one behind a backoff timer.
Timers and retries: a user timer is a timer_tasks row fired by the shard's timer queue; an activity retry updates mutable state and writes a retry timer with no history events; a workflow retry closes the run and creates a new one behind a backoff timer.

History appends TimerStarted, writes a timer_info_maps row and a timer_tasks UserTimer row with visibility_timestamp = now + 1h (only for the run's earliest pending timer), and then forgets about it. The worker can evict the run or disappear entirely. The shard's timer processor reads timer_tasks in fire-time order, a little ahead. At T + 1h it re-checks mutable state, appends TimerFired, schedules a workflow task, and writes the next timer row if there is one. A cancelled timer's row isn't deleted. It fires, finds nothing pending, and does nothing.

Every kind of server-side time works this way: ActivityTimeout, WorkflowTaskTimeout, ActivityRetryTimer, WorkflowBackoffTimer, run/execution timeouts, and DeleteHistoryEvent, which deletes a closed run's rows when retention expires. That delete is done by History, not the internal worker.

"Retry" covers two different mechanisms:

  • Activity retry stays inside one run and writes no history events per failed attempt. History computes min(initialInterval × backoffCoefficient^(attempt−1), maximumInterval), bumps the attempt in activity_info_maps, and writes an ActivityRetryTimer. When it fires, the timer processor calls Matching directly. An activity that failed 50 times before succeeding shows three activity events in its history, but the database absorbed ~50 rounds of state updates, timer rows and dispatches. With the default policy (1 s initial, ×2, max 100 s, unlimited attempts), a dead downstream means real DB load with zero progress.
  • Workflow retry creates a new run, and only if a retry policy was set at start. In one transaction, run 1 closes with newExecutionRunId, run 2 is created (attempt 2, original input), current_executions moves to run 2, and a WorkflowBackoffTimer is written. Run 2 replays from event 1. Cron and continue-as-new use the same machinery, always on the same shard, because the shard key is namespace + workflow ID, not run ID.

Two design rules follow. First, a million workflows sleeping "until midnight" fire in the same second. They're spread across shards, but each shard's timer queue still has to chew through its share, so add jitter. Second, timers use wall-clock timestamps, so keep NTP healthy. The timeouts and retries post goes deeper on all of this.

One hot workflow ID

Signals follow the same pattern. Frontend routes to the target's shard owner, and History appends WorkflowExecutionSignaled (buffered if a workflow task is in flight), records the requestId in signals_requested_sets for dedup, and schedules a workflow task. A workflow signalling another workflow on a different shard goes through a transfer task and a direct History→History call, roughly 4–6 transactions across two shards for one line of code (my estimate from the trace; measure it).

The trap is load on one workflow ID. Every update to it serializes on its lock on one shard, so thousands of signals to one workflow get RESOURCE_EXHAUSTED with cause BusyWorkflow, and no number of History hosts helps. More hosts spread different workflows, never one. The fix is in workflow design: batch the signals, or split the entity across several workflow IDs.

Count the trips

Add it all up for the simplest real workflow, start → one activity → complete:

Count the trips for a one-activity workflow: seven History saves, each an append plus a transaction except the activity start, producing eleven events.
Count the trips for a one-activity workflow: seven History saves, each an append plus a transaction except the activity start, producing eleven events.

2 workflow tasks + 1 activity task = about 7 transactions and 6 history appends for 11 events, plus background work (2 visibility writes, range-deleting finished task rows, the retention delete days later). The three-step invoice is 4 workflow tasks + 3 activity tasks: about 15 transactions, 12 appends, 23 events. Adding a one-hour sleep to the one-activity version makes it about 10 transactions and 9 appends for 16 events.

These counts are illustrative and inferred from the traces: sync match everywhere, no retries, no heartbeats. Eager start and eager activities remove a few, and backlog matches add tasks writes. For estimating:

text
transactions ≈ 1 (start) + 2 per workflow task + 2 per successful activity attempt + 1 per timer fired
appends      ≈ transactions − 1 per activity start

Two activities and no timers: 1 + 3×2 + 2×2 = 11 transactions. Each is at least one PostgreSQL commit while holding the shard's write slot, so at 10 ms per commit that's ~110 ms of shard time for one small workflow (illustrative numbers).

That's why Temporal capacity is measured in state transitions per second (state_transition_count), not workflows per second. A "workflow" can be 10 transitions or 10,000. It's also why History plus PostgreSQL is almost always the bottleneck. To count the trips for real:

promql
# State transitions per second: the real unit of Temporal load
sum(rate(state_transition_count_sum[5m]))   # or the counter form on your version; grep state_transition
 
# Persistence calls per operation
sum by (operation) (rate(persistence_requests{service_name="history"}[5m]))

Four patterns, and where to look

Everything above reduces to four patterns:

  1. One workflow, one shard, one writer. Every change is a transaction on the workflow's shard, fenced by range_id. One workflow ID's throughput is bounded by serialized commits, and no number of hosts changes that.
  2. Intent first, dispatch later. History never calls Matching or another shard inside your request. It writes a task row, and a queue processor delivers it. "Accepted" and "delivered" are different moments, so watch queue lag (task_latency_queue).
  3. Matching stores only what it can't hand over. Sync match = no DB write. Backlog = insert + read + delete.
  4. Time is a row. Timers, timeouts, retries and retention all land on timer_tasks, and they create DB load later than the call that caused them.

The map I use when someone says "Temporal is slow":

SymptomMost likely hopFirst metric
Clients get RESOURCE_EXHAUSTEDFrontend rate limits, or one busy workflowservice_errors_resource_exhausted by resource_exhausted_cause
Start/Signal p99 high, DB fineHistory CPU, shard semaphore, workflow lockservice_latency{service_name="history"}, semaphore_latency{operation="ShardInfo"}, lock_latency
Start/Signal p99 high, DB commits slowPostgreSQLpersistence_latency by operation
Accepted but tasks reach workers latetransfer/timer queue lagtask_latency_queue
Schedule-to-start high, worker slots at 0SDK workerstemporal_worker_task_slots_available
Schedule-to-start high, slots freeMatching, partitions, poll limitspoll_success_sync ÷ poll_success, Matching CPU
Timers fire latetimer queue, timer_tasks range readstask_latency_queue{operation=~"Timer.*"}
promql
# Where does a start spend its time? Request vs shard semaphore vs DB (p99)
histogram_quantile(0.99, sum by (le) (rate(service_latency_bucket{service_name="history", operation="StartWorkflowExecution"}[5m])))
histogram_quantile(0.99, sum by (le) (rate(semaphore_latency_bucket{operation="ShardInfo"}[5m])))
histogram_quantile(0.99, sum by (le) (rate(persistence_latency_bucket{operation="CreateWorkflowExecution"}[5m])))
 
# Sync match ratio: closer to 1 means workers keep up
sum(rate(poll_success_sync[5m])) / sum(rate(poll_success[5m]))

Metric names and operation labels vary by server and SDK version, and some SDKs add _seconds or _total suffixes. Check what each role actually exports before you build a panel:

bash
curl -s http://<history-host>:9090/metrics \
  | grep -E '^(service_latency|persistence_latency|semaphore_latency|task_latency_queue)' \
  | cut -d'{' -f1 | sort -u
 
# Is anyone polling this queue right now?
temporal task-queue describe --task-queue email_queue

For the worker side of metrics, and why I export SDK metrics over OTLP when several workers share a host, see Temporal SDK metrics: Prometheus exporter vs OTLP.

What I'd tell myself before go-live

  • The server remembers and hands out jobs. Workers run all the code. "Temporal is slow" is a worker problem until the metrics say otherwise.
  • Nothing is confirmed before it's in PostgreSQL, so server failures cost time, not data. Plan for the pause and the recovery burst.
  • Workflow code must be deterministic. Activities are at-least-once. Every side effect goes in an activity with an idempotency key.
  • Every step is an append plus a transaction on one shard. A one-activity workflow is ~7 commits. Measure load in state transitions.
  • Intent first, dispatch later. A fast start with a late first task points at the transfer queue or the Matching backlog.
  • Worker capacity is a database lever. Missing pollers turn free sync matches into tasks inserts, reads and deletes.
  • Count your pollers before scaling workers out. Processes × queues × pollers, and Frontend and Matching pay for every one.

Next in the series: Frontend and History, covering the rate limiters at the front door, and how History's event log, mutable state and task tables make sure it never loses a task, even when it crashes mid-step.

Keep reading