Blog

Temporal Server Cheat Sheet: The Decision Trees, Knobs, Metrics and Checklists I Keep Open On Call

The whole Running Temporal Server in Production series on one page: the shelf/desk/server triage tree, the scaling tree that checks PostgreSQL before any Temporal role, the knob pairs people mix up, the formulas worth carrying, a metrics checklist with labelled starting thresholds, one-line failure drills, and the go-live checklist.
22 min readPart 10 of 10
On this page
  1. The ten lines
  2. Tree one: shelf, desk or server?
  3. Two shortcuts
  4. When nothing is high but the SLO still burns
  5. Tree two: what do we scale?
  6. Knobs people confuse
  7. The four scaling knobs
  8. The other pairs
  9. The numbers I keep redoing
  10. Metrics checklist
  11. The traps
  12. Failure drills, one line each
  13. Go-live readiness checklist
  14. Questions I can now answer
  15. Closing the series

TL;DR: When someone says "Temporal is slow", ask one question first: is the wait on the shelf (no worker picked the task up), at the desk (the activity itself is slow) or in the server (admitting, writing or dispatching it is slow)? Four numbers answer it in a fixed order: schedule-to-start, slots available, activity execution latency, server latency. When the question becomes "what do we scale?", check in a fixed order too: SDK workers, then PostgreSQL, then Frontend, History, Matching, the internal worker, and finally limits and plumbing. PostgreSQL comes before every Temporal role because scaling Temporal in front of a saturated database makes it worse. Every action has a side effect you must be able to name before you turn the knob. A latency stuck at an exact number is a timer or a limit, not load. Size everything for one node down, and remember that Temporal fails paused, not lost: the danger is the recovery burst.


This is the last post in the series, and it's the one I actually keep open during an incident. The other nine explain why Temporal behaves the way it does on PostgreSQL. This one is the compressed version: the two decision trees, the knobs that look alike and aren't, the arithmetic I keep redoing, the metrics I alert on, and the checklist I'd walk before calling a cluster production-ready.

The reason it exists is a 3 a.m. lesson. You open Grafana, there are two hundred panels, and every one of them is a little red, because when one part of Temporal is slow everything downstream waits too. A slow disk makes commits slow, which keeps the per-shard pens busy, which makes History slow, which makes dispatch slow, which raises schedule-to-start on every queue. Going down a list of metrics top to bottom doesn't work: every metric looks guilty. What works is a fixed order of questions where each answer picks the branch.

Where something here needs more than a line, I link to the post that owns it.

The ten lines

If I could only keep ten sentences from the whole series, they'd be these:

  1. The server remembers and hands out jobs. Workers run all the code. (post 1)
  2. Every step is saved to PostgreSQL before History says OK. Failures cost time, not data. (post 2)
  3. A shard is a box of workflows with exactly one owner. The membership ring picks the owner; the range_id stamp in PostgreSQL enforces it. (post 3)
  4. numHistoryShards is chosen once, forever. Adding hosts moves boxes; it never adds boxes.
  5. Shards scale many workflows. They do nothing for one hot workflow.
  6. "Temporal is slow": first ask whether the wait is on the shelf, at the desk, or inside the server. (post 7)
  7. A latency stuck at an exact number (4.9 s, 70 s) is a timer or a limit, not load.
  8. The smallest ceiling wins, and 60–70% of it, with one node down, is "full".
  9. Anything sized per host or per shard changes when a node dies. Size for N-1. (post 6)
  10. Temporal fails paused, not lost. The danger is the recovery burst. (post 9)

Tree one: shelf, desk or server?

A task can only wait in three places. A task queue is a shelf at the job counter (Matching); a worker's poller is a hand that grabs a task only when one of its slots, a desk, is free; behind the counter is the server. So:

  1. On the shelf, because no hand came for it: schedule-to-start.
  2. At the desk, because the work itself is slow: activity execution latency.
  3. In the server, because admitting, writing or dispatching it was slow: server latency.

The master triage tree: check SDK schedule-to-start, then activity execution latency, then server latency, stopping at the first one that is high, with the server branch broken into five checks in order
The master triage tree: check SDK schedule-to-start, then activity execution latency, then server latency, stopping at the first one that is high, with the server branch broken into five checks in order

1. Schedule-to-start high? (temporal_activity_schedule_to_start_latency, temporal_workflow_task_schedule_to_start_latency). The wait is on the shelf. Pollers and slots tell you who to blame:

What you seeWhat it means
High schedule-to-start, 0 pollersWorkers are down, or polling the wrong queue name (check the label trap: - shows up as _)
High, pollers present, temporal_worker_task_slots_available at 0Workers are full: add desks or workers, or the activity is slow (step 3)
Low schedule-to-start, but the job runs slowlyThe activity is slow: your code or what it calls
High on every queue at once, workers idle with free slotsThe server is slow to hand out jobs (step 4)

3. Activity execution latency high? (temporal_activity_execution_latency). The wait is at the desk: a slow API or database, a missing HTTP timeout, a provider rate-limiting you. Not Temporal. More slots only pile more load onto the slow downstream.

4. Server latency high? (service_latency by operation, e.g. StartWorkflowExecution). Look inside the server, in this order:

Look atIf high, it is…Depth
service_errors_resource_exhausted by causeA rate limiter saying no: RPS, concurrent polls, busy workflow, persistence limit, system overloadedpost 2
semaphore_latency p95The queue for a shard's IO slot (a busy shard)post 3
persistence_latency p99 by operationPostgreSQL or its diskpost 6
lock_latency p99One hot workflow waiting on its own lockpost 3
cache_usage + memory_heap + CPUGC thrashpost 6

The sentence I say out loud on a call: "First I separate shelf time from desk time from server time, then go deeper only where the time actually is."

Two shortcuts

An exact number is a timer or a limit, not load. Load produces a spread; a setting produces a flat line. Find the setting equal to the number. Three I've hit:

  • ~70 s heartbeat stalls: a 60 s heartbeat timeout plus ~10 s before the server noticed and scheduled the retry. pm2 had SIGKILLed the worker after 5 s. (post 5)
  • A flat ~4.9 s schedule-to-start on every queue: the per-host transfer-tray read limit (history.transferProcessorMaxPollHostRPS, 120 on that cluster; verify the name and default on your version) shared by 512 shards, so each shard's transfer tray was read only every 512 ÷ 120 ≈ 4.3 s, and every dispatch waited that long.
  • 5–10 s workflow-task stalls after a deploy: the sticky-queue timeout expiring after workers were SIGKILLed. (post 4)

Before listing suspects, ask which single setting would produce this number. We once saw 800 UpdateTaskQueue writes/s. The rate is roughly active partitions ÷ matching.updateAckInterval. At the 1-minute default, 800/s needs 48,000 active partitions: not plausible. At 0.5 s it needs 400, about 100 queues × 4 partitions: plausible. So the interval had been set far too low. Fixing it took the rate from 795/s to 8/s. Doing that arithmetic out loud beats any dashboard.

When nothing is high but the SLO still burns

Then it's limits and plumbing, not capacity:

  • A load balancer idle timeout shorter than a long poll (~70 s): constant poll errors.
  • Membership churn: shard_closed_count and ShardOwnershipLost outside deploys (ports, clock skew, GC pauses). (post 8)
  • History tray read limits too low for the shard count: the flat 4.9 s.
  • RESOURCE_EXHAUSTED with cause persistence limit: persistenceMaxQPS too low for N hosts.
  • One noisy namespace; DNS or TLS errors.
  • A broken graph, not a broken cluster: a worker counter read with rate() instead of sum_over_time(), or le="1" instead of le="1.0".

Fix config; do not buy hosts.

Tree two: what do we scale?

The first tree tells you where the time is. Once the time is in the server and something is genuinely saturated, the second tree tells you what to change.

The scaling decision tree: check SDK workers, PostgreSQL, Frontend, History, Matching and the internal worker in that order, each leaf with what to do, its side effect and what to verify first
The scaling decision tree: check SDK workers, PostgreSQL, Frontend, History, Matching and the internal worker in that order, each leaf with what to do, its side effect and what to verify first

"High" here means above your own measured knee, sustained for 10–15 minutes. Temporal publishes no thresholds. The text version, condensed:

text
START: an SLO is burning (latency, schedule-to-start, backlog, RESOURCE_EXHAUSTED)
│
├─0─ SDK WORKERS saturated?
│     schedule_to_start p99 up AND slots_available ≈ 0 AND server latencies normal
│     DO: more worker replicas, or more slots/pollers
│     SIDE EFFECT: more completions → more Frontend RPS, History writes, PG commits;
│                  more pollers → more parked long polls (CONCURRENT_LIMIT, server memory)
│     VERIFY: PG has headroom; worker CPU isn't the real limit; the downstream API can take it
│
├─1─ POSTGRESQL saturated?
│     PG CPU/IO wait high, commit/WAL fsync latency up, persistence_latency up on ALL ops, ALL hosts
│     DO: tune (autovacuum on queue tables, checkpoints), faster disk, bigger primary,
│         dedicated host, split visibility, cut writes per workflow
│     DON'T: add Temporal hosts, raise shardIOConcurrency, add partitions (all push MORE at PG)
│     VERIFY: not one bad query (pg_stat_statements), not vacuum lag, not connections, not UI searches
│
├─2─ FRONTEND saturated?
│     ├─ limit hit, CPU low → raise frontend.rps / namespaceRPS (policy, not capacity)
│     │                       VERIFY: downstream headroom; maybe that namespace SHOULD be limited
│     └─ CPU high on all    → add Frontend replicas; per-host limits now multiply by N
│
├─3─ HISTORY saturated?
│     ├─ CPU-bound (PG fine)          → add History hosts; ~1/N shards move, caches go cold
│     │                                 VERIFY: shards ≫ hosts, connection budget, not GC thrash
│     ├─ shard-IO-bound (CPU low, PG fine, semaphore wait high)
│     │                               → shardIOConcurrency 1 → 2 → 4, one step at a time
│     ├─ ONE HOT WORKFLOW (BusyWorkflow, lock latency on few IDs)
│     │                               → redesign: split by key, batch signals, continue-as-new
│     │                                 NO scaling helps: one workflow = one shard = one lock
│     └─ OUT OF SHARDS                → new Service with more shards + migrate namespaces
│
├─4─ MATCHING saturated?
│     ├─ one task queue hot  → raise read, then write partitions for THAT queue
│     └─ CPU high on all     → add Matching hosts (partitions ≥ hosts, or new hosts own nothing)
│
├─5─ INTERNAL WORKER lagging? (schedules fire late, scanners slow) → run a 2nd/3rd copy
│
└─6─ NOTHING saturated → limits and plumbing. Fix config; do not buy hosts.

SDK workers go first because they're the most common cause and the cheapest fix. PostgreSQL goes second because every Temporal-side scaling action sends more concurrent work at the one primary; if the database is the wall, everything else makes it worse. And of History's four "saturated" leaves, only the CPU-bound one is fixed by adding hosts.

Every pass through the tree should end in one sentence of this shape:

"X is the bottleneck because metric M shows S. Adding more Y won't help because Z. I'll change A, which increases pressure on B, so I first verify C."

If I can't fill in every slot, I gather more data before touching anything.

One version note: the resource_exhausted_cause values appear as RpsLimit / ConcurrentLimit / BusyWorkflow / PersistenceLimit in some versions' metric labels and as RPS_LIMIT / CONCURRENT_LIMIT / … in the API enum. Check what your /metrics actually emits before writing a query.

Knobs people confuse

Twelve pairs of Temporal knobs and names that get mixed up, each with a one-line difference
Twelve pairs of Temporal knobs and names that get mixed up, each with a one-line difference

The four scaling knobs

KnobWhat it addsScopeTakes effectMain side effect
More History hostsCPU and memory to run existing shardsCluster membershipOn join (shards move)Shard movement, more DB connections
history.shardIOConcurrencyConcurrent DB writes per shardPer shard, dynamic configOn shard reload/restartMore concurrent PG transactions
numHistoryShardsTotal units of ownership and parallelismWhole Service, immutableNever (new Service)Memory and background DB polling per shard
Task queue partitionsParallelism for one task queuePer queue, dynamic configLivePollers spread thinner, more forwarding

Which of these are fixed forever, read at init (restart needed) or live is its own table in post 8; the short version is that numHistoryShards, the cluster name and the persistence backend are decided once, shardIOConcurrency and cache sizes need a reload or restart, and rate limits, partitions and updateAckInterval change live.

The other pairs

PairThe difference
Pollers vs slotsPollers only fetch, in milliseconds; slots do the work. A poller reaches out only if a slot is free. Extra pollers just park more long polls on Frontend and Matching.
maxConcurrentActivityTaskExecutions vs maxConcurrentActivityTaskPollsActivity slots vs activity pollers (TypeScript SDK names); same split for workflow tasks.
frontend.namespaceRPS vs frontend.globalNamespaceRPSPer host (multiplies by N, halves when one of two hosts dies) vs cluster-wide per namespace, enforced as limit ÷ live Frontends.
frontend.namespaceCount vs frontend.namespaceRPSParked long polls per namespace per host (CONCURRENT_LIMIT) vs requests per second (RPS_LIMIT).
matching.updateAckInterval vs history.<tray>ProcessorUpdateAckIntervalMatching's backlog bookmark (in task_queues) vs History's task-tray bookmark (in shards). Same idea, two services.
Task queues vs History task traysMatching's named queues your workers poll vs History's internal transfer_tasks, timer_tasks, visibility_tasks. Both get called "queues".
lock_latency vs semaphore_latencyWaiting on one workflow's lock (hot workflow) vs waiting for a shard's IO slot (busy shard). Different fixes.
GOMEMLIMIT vs cache sizeGOMEMLIMIT is the room the garbage collector works in; the cache is what fills it. Shrink the cache; keep GOMEMLIMIT ≈ 2/3 of the container limit.
cache_usage vs cache_sizecache_usage{cache_type="mutablestate"} is how full the mutable-state cache is. cache_size reports a misleading number on some versions.
Schedule-to-Start vs Start-to-Close vs Schedule-to-Close vs HeartbeatWait in the queue (not retried) vs one attempt (retried) vs everything including retries (final) vs max silence between heartbeats (retried, fast death detection).
patched() vs Worker VersioningA version branch inside the code vs routing old runs to old workers by build ID.
Replay vs retryReplay re-runs workflow code against the history; no activity re-runs. Retry re-runs an activity attempt.
Internal worker vs SDK workerTemporal's own system workflows (schedules, batch, scanners) vs your process running your code. Retention deletes are neither: they're History timer tasks.

The numbers I keep redoing

Most of these are from our own clusters; the rules of thumb are labelled. They're there to show the shape of the arithmetic, not as limits to copy.

WhatFormula / number
Which shardfarm.Fingerprint32(namespaceID + "_" + workflowID) % numHistoryShards + 1 (namespace ID, not name; shard IDs start at 1; run ID not included)
Shard IO ceilingmax writes/s ≈ shards × shardIOConcurrency ÷ time each write holds the slot
Too few shards vs enough4 shards × 1 slot at ~10 ms ≈ 400 writes/s (slots ~97% busy, starts went from 46 ms to 1.4 s) vs 512 × 4 at ~13 ms ≈ 157,000/s
Parked long pollsworkers × task queues × pollers per worker = 200 × 3 × 10 = 6,000
Bookmark writesUpdateTaskQueue/s ≈ active partitions ÷ updateAckInterval (795/s at 0.5 s → 8/s fixed)
Tray read intervalshards ÷ transferProcessorMaxPollHostRPS = 512 ÷ 120 ≈ 4.3 s → flat ~4.9 s schedule-to-start
Queueing (rule of thumb, M/M/1)wait ≈ service time × busy ÷ (1 − busy): 50% → 1×, 80% → 4×, 90% → 9×, 97% → 32×
Planning band (rule of thumb)Peak at 60–70% of the smallest ceiling, with one node removed
N-1 target(N − 1) ÷ N × knee, minus a margin
Writes per workflowRate of persistence_requests for the create/update-execution and AddTasks operations ÷ rate of workflows started: measure yours, per workflow type
History CPU~185 ops/s per core on our hosts
Disk ceilingA P30 disk ≈ 685 ops/s; 50 wf/s × 12 writes = 600/s = 88% busy: too hot
Disk runway15% left ÷ 4% a day ≈ 3.75 days → really 1–2 days to act
Looping workflow~11 events per 5-min loop × 288 ≈ 3,200/day → ~50,000-event limit in ~15 days
Default retries1, 2, 4 … capped at 100 s, forever → ~860 attempts a day
Recovery burst12,000 timers × ~4 writes ≈ 48,000 writes ÷ 685/s ≈ 70 s of disk at 100%
History ratio~1 History process per 500 shards, Temporal's documented starting ratio
Partition rate (rule of thumb, Temporal staff)~100 tasks/s per partition

The derivations for the capacity rows live in post 7; the shard rows in post 3.

Metrics checklist

Metric names and labels change between releases, so before writing any alert I grep the real /metrics output on that version. Thresholds below are starting points I'd use on day one and replace with benchmark-derived values as soon as I have them. Server metrics are unprefixed; SDK metrics start with temporal_ (more on the SDK side in this earlier post).

MetricMeansStarting thresholdWhen crossed
service_requests, service_errors by operationRequest rate and error ratio per APIError ratio over 1% for 10 mSplit client errors (NotFound) from server errors
service_latency p99 by operationWhere the time goes, per roleOver 2× benchmark baseline for 10 mThe first role to slow down is upstream of the problem
service_errors_resource_exhausted by causeA rate limiter rejecting workSustained over 1/s for 10 mThe cause picks the branch. SDKs retry it quietly: users see slowness, not errors
persistence_latency p99 by operationDB round trip as Temporal sees itOver 100 ms on writes for 10 mPG commit latency, CPU, IO; then network and pool waits
persistence_errorsConnection refused, timeouts, failoverAny sustainedPage. PG health, failover state, max_connections
task_latency_queue p95History tasks running late (timers, transfers)Over 5 s for 15 mHistory CPU vs shard IO vs PG. Flat exact value: tray read limits
semaphore_latency{operation="ShardInfo"}, lock_latencyWaiting for a shard IO slot, or a workflow lockRising while CPU and PG are fineStep shardIOConcurrency; or a hot workflow if concentrated on few IDs
sharditem_created_count, shard_closed_countMembership churnAbove 0 outside deploysPorts, clocks, GC pauses, a flapping host
cache_usage{cache_type="mutablestate"}Mutable-state cache fullnessClimbing to the cap with heap near GOMEMLIMITCap the cache (GC thrash) or grow it (DB reads)
poll_success, poll_success_sync, poll_timeoutsDid tasks meet a waiting poller?Sync share under 80% on a normally synced queueWorkers short, or partition starvation
Backlog count / ageTasks waiting in the DBAge over your schedule-to-start SLOWorkers if slots are exhausted; partitions if one queue is hot
temporal_*_schedule_to_start_latencyTime a task waited for your workerp99 over 1 s on latency-sensitive queuesTriage step 1 / scaling branch 0
temporal_worker_task_slots_availableFree slots0 for 10 mRaise slots if CPU and downstream allow, else add workers
temporal_request_failure, temporal_long_request_failureSDK ↔ Frontend failuresSustained non-zeroLB idle timeout, TLS, Frontend health
PG: commit latency, IO wait, WAL, connections, dead tuples on queue tables, replication lag, disk %The databaseConnections over 80% of max; lag over 10 s; disk over 70%Scaling branch 1; autovacuum on queue tables; expand disk early

The traps

TrapWhat goes wrongInstead
SDK worker metrics are delta, not cumulativerate() and increase() read every drop as a counter reset and return nonsensesum_over_time() on worker counters; rate() is fine on server metrics
le labels are floatsle="1" matches nothing and you get an empty graph with no errorle="1.0"
Task queue labels are sanitised- becomes _; describing the label's spelling shows a false 0 pollersUse the real queue name with -
AveragesThey hide the fsync stalls in the tailp95/p99 from histograms

The queries for the four triage numbers and the five server checks:

promql
# 1. Schedule-to-start (SDK; if your workers export delta histograms, use sum_over_time)
histogram_quantile(0.95, sum by (le, task_queue) (rate(temporal_activity_schedule_to_start_latency_bucket[5m])))
 
# 2. Free slots per queue and worker type
min by (task_queue, worker_type) (temporal_worker_task_slots_available)
 
# 3. Time at the desk
histogram_quantile(0.95, sum by (le, activity_type) (rate(temporal_activity_execution_latency_bucket[5m])))
 
# 4. Server latency per operation
histogram_quantile(0.99, sum by (le, operation) (rate(service_latency_bucket{service_name="frontend"}[5m])))
 
# 4a. Which rate limiter
sum by (resource_exhausted_cause) (rate(service_errors_resource_exhausted[5m]))
# 4b. Shard IO slot queue
histogram_quantile(0.95, sum by (le) (rate(semaphore_latency_bucket{service_name="history"}[5m])))
# 4c. The database
histogram_quantile(0.99, sum by (le, operation) (rate(persistence_latency_bucket{service_name="history"}[5m])))
# 4d. One hot workflow
histogram_quantile(0.99, sum by (le) (rate(lock_latency_bucket{service_name="history"}[5m])))
# 4e. GC thrash: cache fullness and heap
sum by (instance) (cache_usage{cache_type="mutablestate"})
sum by (instance) (memory_heap)
bash
# Check the names on your version first
curl -s http://<history-host>:<metrics-port>/metrics | grep -iE 'schedule_to_start|slots_available|semaphore|lock_latency|cache_usage'
 
# Pollers on a queue (the real name, with '-')
temporal task-queue describe --task-queue <queue-name> --namespace <ns>
sql
-- What the database is waiting on
SELECT wait_event_type, wait_event, state, count(*)
FROM pg_stat_activity
WHERE datname = 'temporal'
GROUP BY 1, 2, 3
ORDER BY 4 DESC;

Failure drills, one line each

Anything committed to PostgreSQL is safe; anything not committed is retried by whoever sent it. The walkthroughs are in post 9.

FailureLost?Main dangerFirst move
History node diesNoSurvivor not sized for N-1Confirm shard counts add up. Don't panic-restart the survivor: it moves every shard again
DB down or failoverNo (committed data)The recovery burstFix the DB. Don't scale workers up during recovery; tray read limits act as a brake
Disk fullNo, but stuckYou can't delete your way outPrevent it: alert at 70% and 80%. Expand first, clean up after
Bad deploy (non-determinism)NoPanicked mass-terminationRoll back the worker code; stuck workflows resume on their own
Workers downNoThe backlog burst hits downstreamBring workers back gradually; cap activity concurrency
Whole cluster downStarts failThe app must handle thatThe code that starts workflows retries or buffers

Go-live readiness checklist

The checklist I'd walk before calling a self-hosted cluster production. The reasoning behind the topology items is in post 9, the shard count in post 3.

Topology

  • At least 3 Temporal hosts (N+1 headroom), all-role unless metrics justify a split
  • Internal worker on 2 or more hosts
  • SDK workers on separate machines or pods from the Temporal hosts
  • numHistoryShards chosen deliberately for a 3–5 year horizon
  • Cluster name, shard count and persistence config in Git, identical on every member

Network and membership

  • broadcastAddress is each host's private IP, never 127.0.0.1
  • All gRPC and membership ports open between every pair of hosts, both directions
  • NTP/chrony synced on every host and the database
  • Ring verified: every role lists every expected member; cluster_membership rows are fresh

Frontend and load balancer

  • One DNS name in front of all Frontends
  • gRPC health check on temporal.api.workflowservice.v1.WorkflowService
  • LB idle/read timeouts of at least 2 minutes (longer than a long poll)
  • Drain-on-shutdown configured; rolling deploy tested
  • TLS (mTLS for workers if required) and an authorizer
  • Per-host limits sized so N-1 hosts carry peak, or global limits used

PostgreSQL

  • Dedicated primary plus standby (Patroni with a 3-node DCS, or managed HA)
  • Sync vs async replication chosen deliberately; RPO written down
  • max_connections at least the measured pool total plus headroom; pg_hba per host
  • Autovacuum tuned for the queue tables; bloat monitored
  • Backups and PITR proven by an actual restore
  • PgBouncer only if verified for your exact Temporal version (I wrote up PgBouncer separately)
  • Visibility database placement decided
  • Disk alerts at 70% and 80%; growth per day tracked; free space kept for pg_repack

History memory

  • Mutable-state cache sized to the host (count or bytes); GOMEMLIMIT ≈ 2/3 of the container limit
  • Per-host, not per-shard, cache limits, checked for the N-1 case:
yaml
history.enableHostLevelEventsCache: true        # one events cache per node, not per shard
history.eventsHostLevelCacheMaxSizeBytes: 128MiB
history.cacheSizeBasedLimit: true               # limit the mutable-state cache in bytes
history.hostLevelCacheMaxSizeBytes: 256MiB

Observability

  • Prometheus scrapes every server process, every SDK worker, PG, hosts and the LB
  • Dashboards from temporalio/dashboards, fixed for your metric names
  • Worker delta counters queried with sum_over_time(); le labels written as floats
  • Alerts for Frontend availability/latency, persistence errors and latency, RESOURCE_EXHAUSTED, task queue lag, shard churn, process down, SDK schedule-to-start, PG connections and replication lag
  • Starting thresholds replaced with benchmark-derived values
  • A runbook link on every alert

Capacity and resilience

  • Benchmark (omes or benchmark-workers) run on the production topology; knee recorded
  • Production peak at or below knee × (N−1)/N
  • Writes per workflow measured per workflow type
  • Drills passed: kill each role, PG switchover and failover, network partition, slow host, rolling deploy
  • A post-failover check for wedged shards in the runbook

Application side (details in post 5)

  • Every activity has Start-to-Close (or Schedule-to-Close); long ones heartbeat
  • Permanent errors are non-retryable; a maximumAttempts or Schedule-to-Close safety net exists
  • Side-effecting activities send idempotency keys
  • Replay tests against real production histories in CI; workflow changes use patched() or Worker Versioning
  • Long-lived or busy workflows use continue-as-new (or Schedules); hot workflows split by key
  • Payloads small; blobs in object storage, keys in the history
  • Graceful worker shutdown: shutdownGraceTime below pm2's kill_timeout; signals reach child processes
  • Pollers sized to keep slots fed, not "as many as possible"

Operations

  • Upgrade path documented (schema first, one minor version at a time)
  • Retention and archival set per namespace
  • Per-namespace rate limits for noisy tenants
  • Dynamic config changes reviewed in Git, one knob at a time
  • SLOs (availability, latency, schedule-to-start) with error budgets

Questions I can now answer

When I started, I couldn't have answered most of these. Each answer is a chain, because that's how you explain it on an incident call: because X, then Y, then Z; users see this; confirm with that; fix.

A worker crashes after the email activity finished but before the next step ran. Does the customer get a second email? No. The new worker replays the workflow from the top; at sendEmail the history already has ActivityTaskCompleted, so it uses the stored result instead of calling the activity. The exception is a crash while the activity is still running: then it's retried, which is why side-effecting activities need idempotency keys.

A colleague wants to change numHistoryShards from 512 to 2048 in config and restart, "because we're slow". What's wrong? Two things. The count is stored at cluster creation; config can't change it, and if it ever re-hashed, every workflow would land in the wrong shard. And nobody has shown shards are the bottleneck; at 512 × 4 IO slots they almost certainly aren't. Walk the triage tree first.

One workflow takes 300 signals/s, lock_latency is high, semaphore_latency is fine. Will more shards help? No. One workflow ID hashes to one shard whatever the count, and all signals queue at its one lock. Split by key, batch signals, and continue-as-new; 300/s is about a million events an hour, far beyond the ~50k-event history limit.

A teammate adds an activity call between two existing ones and deploys. What happens to running workflows? Those already past that point replay into a mismatch, a non-determinism error: the workflow task fails and retries, and the workflow is stuck in Running, not failed and not lost. Roll back, redo the change behind patched(), and add a replay test against real histories. Never mass-terminate: that re-runs steps that already happened.

A History node is fine for a day after each restart, then hits 100% CPU at 0% iowait. A teammate wants a higher GOMEMLIMIT. Something fills memory slowly: the mutable-state cache. Confirm with cache_usage climbing and runtime.gcDrain on top of a CPU profile (it was 94% for us). Cap the cache (we went to 16,000 entries, 3.6 cores → 0.3); a bigger GOMEMLIMIT only gives the cache a bigger room to fill, and near the container limit it causes an iowait storm instead.

History CPU is 90%, PG CPU 30%, persistence latency flat. And the reverse: PG CPU 95%, History 30%? The first is genuinely CPU-bound History: rule out GC thrash, then add History hosts after checking PG headroom and connections. The second is the opposite case: History is idle because it's waiting on commits, and more History hosts would push more concurrent transactions at the saturated primary. Fix PostgreSQL.

Closing the series

Ten posts ago the goal was to look at a running cluster and say: "this part is the bottleneck because X; adding more Y won't help because Z; I should change A, but that puts pressure on B, so I check C first." That sentence is the whole series. The rest is knowing enough about the machine to fill in the blanks:

  1. The server only remembers and hands out jobs, and everything it remembers ends at one PostgreSQL primary. That primary is the ceiling, so it's checked before any Temporal role.
  2. Separate shelf, desk and server time first. Most "Temporal is slow" pages end at the workers or at the code the workers call.
  3. Exact numbers are settings. Find the one equal to the number before suspecting load.
  4. Name the side effect before turning the knob. Hosts move shards, pollers park connections, more concurrency means more commits at the same disk.
  5. Shards are forever and don't help one hot workflow. Choose the count once; fix hot workflows in the workflow.
  6. Size for N-1, including the per-host limits and per-shard caches that silently double on a survivor.
  7. Temporal fails paused, not lost. Roll back, don't terminate, and plan for the recovery burst.

If you're starting at the beginning, the series opens with why Temporal exists and its four services.

Find me on GitHub. Corrections and Temporal war stories welcome.

Keep reading