Temporal production readiness - Develop

This guide explains what you need to develop to use Temporal in your production environment.

Data encryption

Temporal Server stores and persists the data handled in your Workflow Execution. Encrypting this data ensures that any sensitive application data is secure when handled by the Temporal Server.

For example, if you have sensitive information passed in the following objects that are persisted in the Workflow Execution Event History, use encryption to secure it:

Inputs and outputs/results in your WorkflowWhat is a Workflow Execution?

A Temporal Workflow Execution is a durable, scalable, reliable, and reactive function execution. It is the main unit of execution of a Temporal Application.

Learn more, ActivityWhat is an Activity Execution?

An Activity Execution is the full chain of Activity Task Executions.

Learn more, and Child WorkflowWhat is a Child Workflow Execution?

A Child Workflow Execution is a Workflow Execution that is spawned from within another Workflow.

Learn more
SignalWhat is a Signal?

A Signal is an asynchronous request to a Workflow Execution.

Learn more inputs
MemoWhat is a Memo?

A Memo is a non-indexed user-supplied set of Workflow Execution metadata that is returned when you describe or list Workflow Executions.

Learn more
Headers (verify if applicable to your SDK)
QueryWhat is a Query?

A Query is a synchronous operation that is used to report the state of a Workflow Execution.

Learn more inputs and results
Results of Local ActivitiesWhat is a Local Activity?

A Local Activity is an Activity Execution that executes in the same process as the Workflow Execution that spawns it.

Learn more and Side EffectsWhat is a Side Effect?

A Side Effect is a way to execute a short, non-deterministic code snippet, such as generating a UUID, that executes the provided function once and records its result into the Workflow Execution Event History.

Learn more
Application errors and failures
Failure messages and stack traces are not encoded as codec-capable Payloads by default; you must explicitly enable encoding these common attributes on failures. For more details, see Failure ConverterWhat is a Failure Converter?

A Failure Converter converts error objects to proto Failures and back.

Learn more.

Using encryption ensures that your sensitive data exists unencrypted only on the Client and the Worker Process that is executing the Workflows and Activities, on hosts that you control.

By default, your data is serialized to a PayloadWhat is a Payload?

A Payload represents binary data such as input and output from Activities and Workflows.

Learn more by a Data ConverterWhat is a Data Converter?

A Data Converter is a Temporal SDK component that serializes and encodes data entering and exiting a Temporal Cluster.

Learn more. To encrypt your Payload, configure your custom encryption logic with a Payload CodecWhat is a Payload Codec?

A Payload Codec transforms an array of Payloads into another array of Payloads.

Learn more and set it with a custom Data ConverterWhat is a custom Data Converter?

A custom Data Converter extends the default Data Converter with custom logic for Payload conversion or Payload encryption.

Learn more.

A Payload Codec does byte-to-byte conversion to transform your Payload (for example, by implementing compression and/or encryption and decryption) and is an optional step that happens between the wire and the Payload ConverterWhat is a Payload Converter?

A Payload Converter serializes data, converting objects or values to bytes and back.

Learn more:

User code <--> Payload Converter <--> Payload Codec <--> Wire <--> Temporal Server

You can run your Payload Codec with a Codec ServerWhat is a Codec Server?

A Codec Server is an HTTP server that uses your custom Payload Codec to encode and decode your data remotely through endpoints.

Learn more and use the Codec Server endpoints in Web UI and tctl to decode your encrypted Payload locally. For details on how to set up a Codec Server, see Codec Server setupCodec Server setup

Run a Codec Server with your Payload Codec and then configure the Web UI and CLI to use the server endpoints.

Learn more.

However, if you plan to set up remote data encodingWhat is remote data encoding?

Remote data encding is using your custom Data Converter to decode (and encode) your Payloads remotely through endpoints.

Learn more for your data, ensure that you consider all security implications of running encryption remotely before implementing it.

In codec implementations, we recommend running the function (such as compression or encryption) on the entire input Payload and putting the result in the data field of a new Payload with a different encoding metadata field. Using this technique ensures that the input Payload's metadata is preserved. When the encoded Payload is sent to be decoded, you can verify the metadata field before applying the decryption. If your Payload is not encoded, we recommend passing the unencoded data to the decode function instead of failing the conversion.

Examples for implementing encryption:

Examples for implementing compression:

Go
Java
PHP
Python
TypeScript

Create a custom Payload Codec

Create a custom PayloadCodec implementation and define your encryption/compression and decryption/decompression logic in the Encode and Decode functions.

The Payload Codec converts bytes to bytes. It must be used in an instance of CodecDataConverter that wraps a Data Converter to do the PayloadWhat is a Payload?

A Payload represents binary data such as input and output from Activities and Workflows.

Learn more conversions, and applies the custom encoding and decoding in PayloadCodec to the converted Payloads.

The following example from the Data Converter sample shows how to create a custom NewCodecDataConverter that wraps an instance of a Data Converter with a custom PayloadCodec.

// Create an instance of Data Converter with your codec.
var DataConverter = converter.NewCodecDataConverter(
    converter.GetDefaultDataConverter(),
    NewPayloadCodec(),
)
//...
// Create an instance of PaylodCodec.
func NewPayloadCodec() converter.PayloadCodec {
    return &Codec{}
}

Implement your encryption/compression logic in the Encode function and the decryption/decompression logic in the Decode function in your custom PayloadCodec, as shown in the following example.

// Codec implements converter.PayloadEncoder for snappy compression.
type Codec struct{}

// Encode implements converter.PayloadCodec.Encode.
func (Codec) Encode(payloads []*commonpb.Payload) ([]*commonpb.Payload, error) {
    result := make([]*commonpb.Payload, len(payloads))
    for i, p := range payloads {
        // Marshal proto
        origBytes, err := p.Marshal()
        if err != nil {
            return payloads, err
        }
        // Compress
        b := snappy.Encode(nil, origBytes)
        result[i] = &commonpb.Payload{
            Metadata: map[string][]byte{converter.MetadataEncoding: []byte("binary/snappy")},
            Data:     b,
        }
    }

    return result, nil
}

// Decode implements converter.PayloadCodec.Decode.
func (Codec) Decode(payloads []*commonpb.Payload) ([]*commonpb.Payload, error) {
    result := make([]*commonpb.Payload, len(payloads))
    for i, p := range payloads {
        // Decode only if it's our encoding
        if string(p.Metadata[converter.MetadataEncoding]) != "binary/snappy" {
            result[i] = p
            continue
        }
        // Uncompress
        b, err := snappy.Decode(nil, p.Data)
        if err != nil {
            return payloads, err
        }
        // Unmarshal proto
        result[i] = &commonpb.Payload{}
        err = result[i].Unmarshal(b)
        if err != nil {
            return payloads, err
        }
    }

    return result, nil
}

Set Data Converter to use custom Payload Codec

Set your custom PayloadCodec with an instance of DataConverter in your Dial client options that you use to create the client.

The following example shows how to set your custom Data Converter from a package called mycodecpackage.

//...
c, err := client.Dial(client.Options{
        // Set DataConverter here to ensure that Workflow inputs and results are
        // encoded as required.
        DataConverter: mycodecpackage.DataConverter,
    })
//...

For reference, see the following samples:

Create a custom Payload Codec

Create a custom implementation of PayloadCodec and use it in CodecDataConverter to set a custom Data Converter.

The Payload Codec does byte-to-byte conversion and must be set with a Data Converter.

Define custom encryption/compression logic in your encode method and decryption/decompression logic in your decode method.

The following example from the Java encryption sample shows how to implement encryption and decryption logic on your payloads in your encode and decode methods.

class YourCustomPayloadCodec implements PayloadCodec {

  static final ByteString METADATA_ENCODING =
      ByteString.copyFrom("binary/encrypted", StandardCharsets.UTF_8);

  private static final String CIPHER = "AES/GCM/NoPadding";

  // Define constants that you can add to your encoded Payload to create a new Payload.
  static final String METADATA_ENCRYPTION_CIPHER_KEY = "encryption-cipher";

  static final ByteString METADATA_ENCRYPTION_CIPHER =
      ByteString.copyFrom(CIPHER, StandardCharsets.UTF_8);

  static final String METADATA_ENCRYPTION_KEY_ID_KEY = "encryption-key-id";

  private static final Charset UTF_8 = StandardCharsets.UTF_8;

  // See the linked sample for details on the methods called here.
  @NotNull
  @Override
  public List<Payload> encode(@NotNull List<Payload> payloads) {
    return payloads.stream().map(this::encodePayload).collect(Collectors.toList());
  }

  @NotNull
  @Override
  public List<Payload> decode(@NotNull List<Payload> payloads) {
    return payloads.stream().map(this::decodePayload).collect(Collectors.toList());
  }

  private Payload encodePayload(Payload payload) {
    String keyId = getKeyId();
    SecretKey key = getKey(keyId);

    byte[] encryptedData;
    try {
      encryptedData = encrypt(payload.toByteArray(), key); // The encrypt method contains your custom encryption logic.
    } catch (Throwable e) {
      throw new DataConverterException(e);
    }
    // Apply metadata to the encoded Payload that you can verify in your decode method before decoding.
    // See the sample for details on the metadata values set.
    return Payload.newBuilder()
        .putMetadata(EncodingKeys.METADATA_ENCODING_KEY, METADATA_ENCODING)
        .putMetadata(METADATA_ENCRYPTION_CIPHER_KEY, METADATA_ENCRYPTION_CIPHER)
        .putMetadata(METADATA_ENCRYPTION_KEY_ID_KEY, ByteString.copyFromUtf8(keyId))
        .setData(ByteString.copyFrom(encryptedData))
        .build();
  }

  private Payload decodePayload(Payload payload) {
    // Verify the incoming encoded Payload metadata before applying decryption.
    if (METADATA_ENCODING.equals(
        payload.getMetadataOrDefault(EncodingKeys.METADATA_ENCODING_KEY, null))) {
      String keyId;
      try {
        keyId = payload.getMetadataOrThrow(METADATA_ENCRYPTION_KEY_ID_KEY).toString(UTF_8);
      } catch (Exception e) {
        throw new PayloadCodecException(e);
      }
      SecretKey key = getKey(keyId);
      byte[] plainData;
      Payload decryptedPayload;

      try {
        plainData = decrypt(payload.getData().toByteArray(), key); // The decrypt method contains your custom decryption logic.
        decryptedPayload = Payload.parseFrom(plainData);
        return decryptedPayload;
      } catch (Throwable e) {
        throw new PayloadCodecException(e);
      }
    } else {
      return payload;
    }
  }

  private String getKeyId() {
    // Currently there is no context available to vary which key is used.
    // Use a fixed key for all payloads.
    // This still supports key rotation as the key ID is recorded on payloads allowing
    // decryption to use a previous key.

    return "test-key-test-key-test-key-test!";
  }

  private SecretKey getKey(String keyId) {
    // Key must be fetched from KMS or other secure storage.
    // Hard coded here only for example purposes.
    return new SecretKeySpec(keyId.getBytes(UTF_8), "AES");
  }

  //...
}

Set Data Converter to use custom Payload Codec

Use CodecDataConverter with an instance of a Data Converter and the custom PayloadCodec in the WorkflowClient options that you use in your Worker process and to start your Workflow Executions.

For example, to set a custom PayloadCodec implementation with DefaultDataConverter, use the following code:

WorkflowServiceStubs service = WorkflowServiceStubs.newLocalServiceStubs();
  // Client that can be used to start and signal Workflows
  WorkflowClient client =
      WorkflowClient.newInstance(
          service,
          WorkflowClientOptions.newBuilder()
              .setDataConverter(
                  new CodecDataConverter(
                      DefaultDataConverter.newDefaultInstance(),
                      Collections.singletonList(new YourCustomPayloadCodec()))) // Sets the custom Payload Codec created in the previous example with an instance of the default Data Converter.
              .build());

For example implementations, see the following samples:

Codec Server setup

Use a Codec Server to decode your encoded payloadsWhat is a Payload?

A Payload represents binary data such as input and output from Activities and Workflows.

Learn more and integrate it with the Temporal Web UI and CLI commands when debugging your Workflows.

A Codec Server is an HTTP or HTTPS Server that you create and host. It must be configured to use a Payload CodecWhat is a Payload Codec?

A Payload Codec transforms an array of Payloads into another array of Payloads.

Learn more with the required decode logic and encryption keys.

The Codec Server is independent of the Temporal Server and decodes your encrypted payloads through endpoints. When you set the codec endpoint in the Temporal Web UI, the Web UI uses the remote endpoint to send encoded payloads to the Codec Server and receive decoded payloads from the Codec Server. See API contract requirements. Decoded payloads are then displayed in the Workflow Execution Event History on the Web UI.

Note that when you use a Codec Server, the decoded payloads are visible only to you on the Web UI; payloads on the Temporal Server (whether on Temporal Cloud or a self-hosted Temporal Cluster) remain encrypted.

Because you create, operate, and manage access to your Codec Server in your controlled environment, ensure that you consider the following:

When you set your codec endpoint with your Web UI, expect your Codec Server to receive a large number of requests per Workflow Execution from the Web UI.
Ensure that you secure access to the decrypted data from your Codec Server.
The Temporal Web UI only displays the decoded payloads received from your Codec Server in real-time; it does not store or send the data back to the Temporal Server (whether on Cloud or self-hosted Temporal Cluster).
You might have latencies introduced in the Web UI when sending and receiving payloads to the Codec Server.

To create a Codec Server, you need the following components:

A Payload CodecWhat is a Payload Codec?

A Payload Codec transforms an array of Payloads into another array of Payloads.

Learn more with the requisite keys and logic to decode your encrypted payloads. You can use the Payload Codec that you applied with your Data Converter to encode your Payloads and configure it with your Codec Server. However, if you are writing your Codec Server in a different SDK from the one that applies the Data Converter, ensure that your logic and keys are correctly replicated.
Key management infrastructure or plan for sharing your encryption keys between the Workers and your Codec Server.
CORS configuration on the HTTP endpoints in your Codec Server for sending and receiving requests from the Temporal Web UI.

For examples on how to create your Codec Server, see following Codec Server implementation samples:

API contract specifications

When you create your Codec Server to handle requests from the Web UI, the following requirements must be met.

Endpoints

The Web UI/CLI calls the POST method with the /decode endpoint.

In your Codec Server, create a /decode path and pass the incoming payload to the decode method in your Payload Codec.

You can also add a verification step to check whether the incoming request has the required authorization to access the decode logic in your Payload Codec.

Headers

Each request from the Web UI to your Codec Server includes the following headers:

Content-Type: application/json: Ensure that your Codec Server can accommodate this MIME type.
X-Namespace: {namespace}: This is a custom HTTP Header. Ensure that the CORS configuration in your Codec Server includes this header.
[Optional] Authorization: <credentials>: Include this in your CORS configuration when enabling authorization with your Codec Server.

For details on setting up authorization, see Authorization.

Request body

The general specification for the POST request body contains payloads. By default, all field values in your payload are base64 encoded, regardless of whether they are encrypted by your custom codec implementation.

The following example shows a sample POST request body with base64 encoding.

{
  "payloads": [{
    "metadata": {
      "encoding": <base64EncodedEncodingHint>
    },
    "data": <encryptedPayloadData>
  }, ...]
}

CORS

Enable Cross-Origin Resource Sharing (CORS) requests on your Codec Server to receive HTTP requests from the Temporal Web UI.

At a minimum, enable the following responses from your Codec Server to allow requests coming from the Temporal Web UI:

Access-Control-Allow-Origin
Access-Control-Allow-Methods
Access-Control-Allow-Headers

For example, for Temporal Cloud Web UI hosted at https://cloud.temporal.io, enable the following in your Codec Server:

Access-Control-Allow-Origin: https://cloud.temporal.io
Access-Control-Allow-Methods: POST, GET, OPTIONS
Access-Control-Allow-Headers: X-Namespace, Content-Type

For details on what a sample request/response looks like from the Temporal Web UI, see Sample Request/Response. If setting authorization, include Authorization in your Access-Control-Allow-Headers. For details on setting up authorization, see Authorization.

Authorization

To enable authorization from the Web UI, your Codec Server must be an HTTPS Server.

Temporal Cloud

Temporal Cloud uses Auth0 to authenticate access. The Temporal Cloud UI provides an option to pass access tokens (JWT) to your Codec Server endpoints. Use the access tokens to validate access and then return decoded payloads from the Codec Server.

You can enable this by selecting Pass access token in your Codec Server endpoint interface where you add your endpoint. Enabling this option in the Temporal Cloud UI adds an authorization header to each request sent to the Codec Server endpoint that you set.

In your Codec Server implementation, verify the signature on this access token (in your authorization header) against the JWKS endpoint provided to you.

The token provided from Temporal Cloud UI contains the email identifier of the person requesting access to the payloads. Based on the permissions you have provided to the user in your access control systems, set conditions in your Codec Server whether to return decoded payloads or just return the original encoded payloads.

Self-hosted Temporal Cluster

On self-hosted Temporal Clusters, configure authorization in the Web UI configurationTemporal Web UI configuration reference

The Temporal Web UI Server uses a configuration file for many of the UI's settings.

Learn more in your Temporal Cluster setup.

With this enabled, you can pass access tokens to your Codec Server and validate the requests from the Web UI to the Codec Server endpoints that you set.

Note that with self-hosted Temporal Clusters, you must explicitly configure authorization specifications for the Web UI and CLI.

The following sample provides implementation examples for applying authentication on your Codec Server using the Go SDK.

Codec Server

Sample request/response

Consider the following sample request/response when creating and hosting a Codec Server with the following specifications:

Scheme: https
Host: dev.mydomain.com/codec
Path: /decode

HTTP/1.1 POST /decode
Host: https://dev.mydomain.com/codec
Content-Type: application/json
X-Namespace: myapp-dev.acctid123
Authorization: Bearer <token>

{"payloads":[{"metadata":{"encoding":"anNvbi9wcm90b2J1Zg==","messageType":"dGVtcG9yYWxfc2hvcC5vcmNoZXN0cmF0aW9ucy52MS5TdGFydFNob3BwaW5nQ2FydFJlcXVlc3Q="},"data":"eyJjYXJ0SWQiOiJleGFtcGxlLWNhcnQiLCJzaG9wcGVySWQiOiJ5b3VyLXNob3BwZXItaWQtZXhhbXBsZSIsImVtYWlsIjoieW91ci1lbWFpbEBkb21haW4uY29tIn0"}]}

200 OK
Content-Type: application/json

{
  "payloads": [{
    "metadata":{
      "encoding": "json/protobuf",
      "messageType": "temporal_shop.orchestrations.v1.StartShoppingCartRequest"
    },
    "data":{
      "cartId":"example-cart",
      "shopperId":"your-shopper-id-example",
      "email":"your-email@domain.com"
    }}]
}

Hosting your Codec Server

Your Codec Server can be hosted at an organization level or locally.

Organization-level hosting

Hosting the Codec Server for your organization simplifies both key management used for decryption and versioning the codec itself. Consider the following details for a multi-tenant approach to setting up your Codec Server:

Ingress: Your server will require ingress configuration for your users to access the server.
Authorization: You must set explicit authorization checks to validate requests to your Codec Server.

Local hosting

Locally hosting the Codec Server is simpler to get started. However, consider the following before choosing to do so:

A single URL configuration is accepted for the Cloud account. This means some agreed-upon policy on the URL must be made for everyone using the Namespaces in this account. For example, if you configure your remote codec endpoint to be http://localhost:8080/codec, every developer must host your Codec Server locally at that port.
Alternatively, you can use the local hosts file to allow each developer to choose where to host. For example, configure the remote codec endpoint as http://codec.server and allow each developer to control what it maps to locally.
Distributing encryption keys that can decrypt the payloads at your organization can be a security risk.

Set your Codec Server endpoints with Web UI and CLI

After you create your Codec Server and expose the requisite endpoints, set the endpoints in your Web UI and CLI.

CLI

After the Codec Server is started, provide the exposed endpoint to CLI using the --codec_endpoint command option.

For example, if you are running your Codec Server locally and expose port 8888 as your endpoint, run the following command to set the codec endpoint globally.

temporal env set --codec-endpoint "http://localhost:8888"

If your codec endpoint is not set globally, use the --codec-endpoint option with your CLI commands. For example, to see the decoded output of the Workflow Execution "yourWorkflow" in the Namespace "yourNamespace", run the following command.

temporal --codec-endpoint "http://localhost:8888" --namespace "yourNamespace" workflow show --workflow-id "yourWorkflow"  --run-id "<yourRunId>" --output "table"

For details, see the CLI reference.

Web UI

On Temporal Cloud and self-hosted Temporal Clusters, you can set the codec endpoints in the Web UI.

Codec Server endpoint setting

In the top-right corner on the Web UI, select Configure Codec Server. In the codec endpoint dialog, enter the URL and port number for your codec endpoint. This sets the codec endpoint on the currently selected Namespace. Refresh your Workflow Execution page to see encoded/decoded data.

In self-hosted Temporal Clusters where you set up your UI Server, you can also set the codec endpoint in the UI server configuration file. Specify the codec endpoint in the UI server configuration fileTemporal Web UI configuration reference

The Temporal Web UI Server uses a configuration file for many of the UI's settings.

Learn more as shown in the following example.

codec:
    endpoint: {{ default .Env.TEMPORAL_CODEC_ENDPOINT "{namespace}"}}

Start the UI server to use this endpoint on the Web UI for decoding data in Workflow Executions in the specified Namespace.

Data encryption​

Codec Server setup​

API contract specifications​

Endpoints​

Headers​

Request body​

CORS​

Authorization​

Sample request/response​

Hosting your Codec Server​

Organization-level hosting​

Local hosting​

Set your Codec Server endpoints with Web UI and CLI​

CLI​

Web UI​