`job.proto`

path mgmt/v1alpha1/job.proto

package mgmt.v1alpha1

Messages

`ActivityFailure`

Name	Type	Description
`message`	`string`

`ActivityOptions`

Config that contains various timeouts that are configured in the underlying temporal workflow(s) and activities

Name	Type	Description
`schedule_to_close_timeout`	optional `int64`	Total time that a workflow is willing to wait for an activity to complete, including retries. Measured in seconds
`start_to_close_timeout`	optional `int64`	Max time of a single Temporal Activity execution attempt. This timeout should be as short as the longest psosible execution of any activity (e.g. table sync). Important to know that this is per retry attempt. Defaults to the schedule to close timeout if not provided. Measured in seconds
`retry_policy`	`RetryPolicy`	Optionally define a retry policy for the activity If max attempts is not set, the activity will retry indefinitely until the start to close timeout lapses

`AwsS3DestinationConnectionOptions`

Name	Type	Description

`AwsS3SourceConnectionOptions`

Name	Type	Description
`connection_id`	`string`

`CancelJobRunRequest`

Name	Type	Description
`job_run_id`	`string`
`account_id`	`string`

`CancelJobRunResponse`

Name	Type	Description

`CreateJobDestination`

Name	Type	Description
`connection_id`	`string`
`options`	`JobDestinationOptions`

`CreateJobDestinationConnectionsRequest`

Name	Type	Description
`job_id`	`string`
`destinations`	repeated `CreateJobDestination`

`CreateJobDestinationConnectionsResponse`

Name	Type	Description
`job`	`Job`

`CreateJobRequest`

Name	Type	Description
`account_id`	`string`	The unique account identifier that this job will be associated with
`job_name`	`string`	The unique, friendly name of the job. This is unique per account
`cron_schedule`	optional `string`	Optionally provide a cron schedule. Goes into effect if the job status is set to enabled
`mappings`	repeated `JobMapping`
`source`	`JobSource`
`destinations`	repeated `CreateJobDestination`
`initiate_job_run`	`bool`	Initially trigger a run of this job regardless of its status or cron schedule
`workflow_options`	`WorkflowOptions`	Specify timeouts and other workflow options for the underlying temporal workflow
`sync_options`	`ActivityOptions`	Specify timeout and retry options for data synchronization activities Data sync activities are any piece of work that involves actually synchronizing data from a source to a destination For the data sync and generate jobs, this will be applied per table

`CreateJobResponse`

Name	Type	Description
`job`	`Job`

`CreateJobRunRequest`

Name	Type	Description
`job_id`	`string`

`CreateJobRunResponse`

Name	Type	Description

`DeleteJobDestinationConnectionRequest`

Name	Type	Description
`destination_id`	`string`

`DeleteJobDestinationConnectionResponse`

Name	Type	Description

`DeleteJobRequest`

Name	Type	Description
`id`	`string`

`DeleteJobResponse`

Name	Type	Description

`DeleteJobRunRequest`

Name	Type	Description
`job_run_id`	`string`
`account_id`	`string`

`DeleteJobRunResponse`

Name	Type	Description

`GenerateSourceOptions`

Name	Type	Description
`schemas`	repeated `GenerateSourceSchemaOption`
`fk_source_connection_id`	optional `string`

`GenerateSourceSchemaOption`

Name	Type	Description
`schema`	`string`
`tables`	repeated `GenerateSourceTableOption`

`GenerateSourceTableOption`

Name	Type	Description
`table`	`string`
`row_count`	`int64`

`GetJobNextRunsRequest`

Name	Type	Description
`job_id`	`string`

`GetJobNextRunsResponse`

Name	Type	Description
`next_runs`	`JobNextRuns`

`GetJobRecentRunsRequest`

Name	Type	Description
`job_id`	`string`

`GetJobRecentRunsResponse`

Name	Type	Description
`recent_runs`	repeated `JobRecentRun`

`GetJobRequest`

Name	Type	Description
`id`	`string`

`GetJobResponse`

Name	Type	Description
`job`	`Job`

`GetJobRunEventsRequest`

Name	Type	Description
`job_run_id`	`string`
`account_id`	`string`

`GetJobRunEventsResponse`

Name	Type	Description
`events`	repeated `JobRunEvent`
`is_run_complete`	`bool`

`GetJobRunLogsStreamRequest`

Name	Type	Description
`job_run_id`	`string`
`account_id`	`string`
`window`	`LogWindow`	The time window in which to retrieve the logs
`should_tail`	`bool`	Whether or not to tail the stream. Note: only works with k8s-pods and is not currently supported with Loki logs
`max_log_lines`	optional `int64`	Optionally provide a max log limit
`log_levels`	repeated `LogLevel`	Provide a list of log levels to filter by. If any of these are UNSPECIFIED, all log levels are returned.

`GetJobRunLogsStreamResponse`

Name	Type	Description
`log_line`	`string`
`timestamp`	optional `google.protobuf.Timestamp`

`GetJobRunRequest`

Name	Type	Description
`job_run_id`	`string`
`account_id`	`string`

`GetJobRunResponse`

Name	Type	Description
`job_run`	`JobRun`

`GetJobRunsRequest`

Name	Type	Description
`job_id`	`string`
`account_id`	`string`

`GetJobRunsResponse`

Name	Type	Description
`job_runs`	repeated `JobRun`

`GetJobStatusRequest`

Name	Type	Description
`job_id`	`string`

`GetJobStatusResponse`

Name	Type	Description
`status`	`JobStatus`

`GetJobStatusesRequest`

Name	Type	Description
`account_id`	`string`

`GetJobStatusesResponse`

Name	Type	Description
`statuses`	repeated `JobStatusRecord`

`GetJobsRequest`

Name	Type	Description
`account_id`	`string`

`GetJobsResponse`

Name	Type	Description
`jobs`	repeated `Job`

`IsJobNameAvailableRequest`

Name	Type	Description
`name`	`string`
`account_id`	`string`

`IsJobNameAvailableResponse`

Name	Type	Description
`is_available`	`bool`

`Job`

Name	Type	Description
`id`	`string`	The unique identifier of the job
`created_by_user_id`	`string`
`created_at`	`google.protobuf.Timestamp`
`updated_by_user_id`	`string`
`updated_at`	`google.protobuf.Timestamp`
`name`	`string`	The unique, friendly name of the job
`source`	`JobSource`
`destinations`	repeated `JobDestination`
`mappings`	repeated `JobMapping`
`cron_schedule`	optional `string`
`account_id`	`string`	The account identifier that a job is associated with
`sync_options`	`ActivityOptions`	Specify timeout and retry options for data synchronization activities Data sync activities are any piece of work that involves actually synchronizing data from a source to a destination For the data sync and generate jobs, this will be applied per table
`workflow_options`	`WorkflowOptions`	Specify timeouts and other workflow options for the underlying temporal workflow

`JobDestination`

Name	Type	Description
`connection_id`	`string`
`options`	`JobDestinationOptions`
`id`	`string`

`JobDestinationOptions`

Name	Type	Description
`postgres_options`	`PostgresDestinationConnectionOptions`
`aws_s3_options`	`AwsS3DestinationConnectionOptions`
`mysql_options`	`MysqlDestinationConnectionOptions`

`JobMapping`

Name	Type	Description
`schema`	`string`
`table`	`string`
`column`	`string`
`transformer`	`JobMappingTransformer`

`JobMappingTransformer`

Name	Type	Description
`source`	`TransformerSource`
`config`	`TransformerConfig`

`JobNextRuns`

Name	Type	Description
`next_run_times`	repeated `google.protobuf.Timestamp`

`JobRecentRun`

Name	Type	Description
`start_time`	`google.protobuf.Timestamp`
`job_run_id`	`string`

`JobRun`

Name	Type	Description
`id`	`string`	The id of the job run. This will currently be equivalent to the temporal workflow id
`job_id`	`string`	The unique identifier of the job id this run is associated with
`name`	`string`	The name of the job run.
`status`	`JobRunStatus`	the status of the job run
`started_at`	`google.protobuf.Timestamp`	A timestamp of when the run started
`completed_at`	optional `google.protobuf.Timestamp`	Available if the run completed or has not yet been archived by the system
`pending_activities`	repeated `PendingActivity`	Pending activities are only returned when retrieving a specific job run and will not be returned when requesting job runs in list format

`JobRunEvent`

Name	Type	Description
`id`	`int64`
`type`	`string`
`start_time`	`google.protobuf.Timestamp`
`close_time`	`google.protobuf.Timestamp`
`metadata`	`JobRunEventMetadata`
`tasks`	repeated `JobRunEventTask`

`JobRunEventMetadata`

Name	Type	Description
`sync_metadata`	`JobRunSyncMetadata`

`JobRunEventTask`

Name	Type	Description
`id`	`int64`
`type`	`string`
`event_time`	`google.protobuf.Timestamp`
`error`	`JobRunEventTaskError`

`JobRunEventTaskError`

Name	Type	Description
`message`	`string`
`retry_state`	`string`

`JobRunSyncMetadata`

Name	Type	Description
`schema`	`string`
`table`	`string`

`JobSource`

Name	Type	Description
`options`	`JobSourceOptions`

`JobSourceOptions`

Name	Type	Description
`postgres`	`PostgresSourceConnectionOptions`
`aws_s3`	`AwsS3SourceConnectionOptions`
`mysql`	`MysqlSourceConnectionOptions`
`generate`	`GenerateSourceOptions`

`JobSourceSqlSubetSchemas`

Name	Type	Description
`postgres_subset`	`PostgresSourceSchemaSubset`
`mysql_subset`	`MysqlSourceSchemaSubset`

`JobStatusRecord`

Name	Type	Description
`job_id`	`string`
`status`	`JobStatus`

`MysqlDestinationConnectionOptions`

Name	Type	Description
`truncate_table`	`MysqlTruncateTableConfig`
`init_table_schema`	`bool`
`on_conflict`	`MysqlOnConflictConfig`

`MysqlOnConflictConfig`

Name	Type	Description
`do_nothing`	`bool`

`MysqlSourceConnectionOptions`

Name	Type	Description
`halt_on_new_column_addition`	`bool`
`schemas`	repeated `MysqlSourceSchemaOption`
`connection_id`	`string`
`subset_by_foreign_key_constraints`	`bool`

`MysqlSourceSchemaOption`

Name	Type	Description
`schema`	`string`
`tables`	repeated `MysqlSourceTableOption`

`MysqlSourceSchemaSubset`

Name	Type	Description
`mysql_schemas`	repeated `MysqlSourceSchemaOption`

`MysqlSourceTableOption`

Name	Type	Description
`table`	`string`
`where_clause`	optional `string`

`MysqlTruncateTableConfig`

Name	Type	Description
`truncate_before_insert`	`bool`

`PauseJobRequest`

Name	Type	Description
`id`	`string`
`pause`	`bool`
`note`	optional `string`

`PauseJobResponse`

Name	Type	Description
`job`	`Job`

`PendingActivity`

Name	Type	Description
`status`	`ActivityStatus`
`activity_name`	`string`
`last_failure`	optional `ActivityFailure`

`PostgresDestinationConnectionOptions`

Name	Type	Description
`truncate_table`	`PostgresTruncateTableConfig`
`init_table_schema`	`bool`
`on_conflict`	`PostgresOnConflictConfig`

`PostgresOnConflictConfig`

Name	Type	Description
`do_nothing`	`bool`

`PostgresSourceConnectionOptions`

Name	Type	Description
`halt_on_new_column_addition`	`bool`
`schemas`	repeated `PostgresSourceSchemaOption`
`connection_id`	`string`
`subset_by_foreign_key_constraints`	`bool`

`PostgresSourceSchemaOption`

Name	Type	Description
`schema`	`string`
`tables`	repeated `PostgresSourceTableOption`

`PostgresSourceSchemaSubset`

Name	Type	Description
`postgres_schemas`	repeated `PostgresSourceSchemaOption`

`PostgresSourceTableOption`

Name	Type	Description
`table`	`string`
`where_clause`	optional `string`

`PostgresTruncateTableConfig`

Name	Type	Description
`truncate_before_insert`	`bool`
`cascade`	`bool`

`RetryPolicy`

Defines the retry policy for an activity

Name	Type	Description
`maximum_attempts`	optional `int32`	Maximum number of attempts. When exceeded the retries stop even if not expired yet. If not set or set to 0, it means unlimited, and rely on activity ScheduleToCloseTimeout to stop.

`SetJobSourceSqlConnectionSubsetsRequest`

Name	Type	Description
`id`	`string`
`schemas`	`JobSourceSqlSubetSchemas`
`subset_by_foreign_key_constraints`	`bool`

`SetJobSourceSqlConnectionSubsetsResponse`

Name	Type	Description
`job`	`Job`

`SetJobSyncOptionsRequest`

Name	Type	Description
`id`	`string`	The unique identifier of the job
`sync_options`	`ActivityOptions`	The sync options object. The entire object must be provided and will fully overwrite the previous result

`SetJobSyncOptionsResponse`

Name	Type	Description
`job`	`Job`

`SetJobWorkflowOptionsRequest`

Name	Type	Description
`id`	`string`	The unique identifier of the job
`worfklow_options`	`WorkflowOptions`	The workflow options object. The entire object must be provided and will fully overwrite the previous result

`SetJobWorkflowOptionsResponse`

Name	Type	Description
`job`	`Job`

`TerminateJobRunRequest`

Name	Type	Description
`job_run_id`	`string`
`account_id`	`string`

`TerminateJobRunResponse`

Name	Type	Description

`UpdateJobDestinationConnectionRequest`

Name	Type	Description
`job_id`	`string`
`connection_id`	`string`
`options`	`JobDestinationOptions`
`destination_id`	`string`

`UpdateJobDestinationConnectionResponse`

Name	Type	Description
`job`	`Job`

`UpdateJobScheduleRequest`

Name	Type	Description
`id`	`string`
`cron_schedule`	optional `string`

`UpdateJobScheduleResponse`

Name	Type	Description
`job`	`Job`

`UpdateJobSourceConnectionRequest`

Name	Type	Description
`id`	`string`
`source`	`JobSource`
`mappings`	repeated `JobMapping`

`UpdateJobSourceConnectionResponse`

Name	Type	Description
`job`	`Job`

`WorkflowOptions`

Config that contains various timeouts that are configured in the underlying temporal workflow More options will come in the future as needed

Name	Type	Description
`run_timeout`	optional `int64`	The timeout for a single workflow run. Measured in seconds

Enums

`ActivityStatus`

Name	Number	Description
`ACTIVITY_STATUS_UNSPECIFIED`	`0`
`ACTIVITY_STATUS_SCHEDULED`	`1`
`ACTIVITY_STATUS_STARTED`	`2`
`ACTIVITY_STATUS_CANCELED`	`3`
`ACTIVITY_STATUS_FAILED`	`4`

`JobRunStatus`

An enumeration of job run statuses.

Name	Number	Description
`JOB_RUN_STATUS_UNSPECIFIED`	`0`	if the job run status is unknown
`JOB_RUN_STATUS_PENDING`	`1`	the run is pending and has not started yet
`JOB_RUN_STATUS_RUNNING`	`2`	the run is currently in progress
`JOB_RUN_STATUS_COMPLETE`	`3`	the run has successfully completed
`JOB_RUN_STATUS_ERROR`	`4`	the run ended with an error
`JOB_RUN_STATUS_CANCELED`	`5`	the run was cancelled
`JOB_RUN_STATUS_TERMINATED`	`6`	the run was terminated
`JOB_RUN_STATUS_FAILED`	`7`	the run ended in failure
`JOB_RUN_STATUS_TIMED_OUT`	`8`	the run was ended pre-maturely due to timeout

`JobStatus`

Name	Number	Description
`JOB_STATUS_UNSPECIFIED`	`0`
`JOB_STATUS_ENABLED`	`1`
`JOB_STATUS_PAUSED`	`3`
`JOB_STATUS_DISABLED`	`4`

`LogLevel`

Name	Number	Description
`LOG_LEVEL_UNSPECIFIED`	`0`
`LOG_LEVEL_DEBUG`	`1`
`LOG_LEVEL_INFO`	`2`
`LOG_LEVEL_WARN`	`3`
`LOG_LEVEL_ERROR`	`4`

`LogWindow`

Name	Number	Description
`LOG_WINDOW_NO_TIME_UNSPECIFIED`	`0`
`LOG_WINDOW_FIFTEEN_MIN`	`1`
`LOG_WINDOW_ONE_HOUR`	`2`
`LOG_WINDOW_ONE_DAY`	`3`

Services

`JobService`

`GetJobs`

Method	`GetJobs`
Request	`GetJobsRequest`
Response	`GetJobsResponse`
Description

`GetJob`

Method	`GetJob`
Request	`GetJobRequest`
Response	`GetJobResponse`
Description

`CreateJob`

Method	`CreateJob`
Request	`CreateJobRequest`
Response	`CreateJobResponse`
Description

`DeleteJob`

Method	`DeleteJob`
Request	`DeleteJobRequest`
Response	`DeleteJobResponse`
Description

`IsJobNameAvailable`

Method	`IsJobNameAvailable`
Request	`IsJobNameAvailableRequest`
Response	`IsJobNameAvailableResponse`
Description

`UpdateJobSchedule`

Method	`UpdateJobSchedule`
Request	`UpdateJobScheduleRequest`
Response	`UpdateJobScheduleResponse`
Description

`UpdateJobSourceConnection`

Method	`UpdateJobSourceConnection`
Request	`UpdateJobSourceConnectionRequest`
Response	`UpdateJobSourceConnectionResponse`
Description

`SetJobSourceSqlConnectionSubsets`

Method	`SetJobSourceSqlConnectionSubsets`
Request	`SetJobSourceSqlConnectionSubsetsRequest`
Response	`SetJobSourceSqlConnectionSubsetsResponse`
Description

`UpdateJobDestinationConnection`

Method	`UpdateJobDestinationConnection`
Request	`UpdateJobDestinationConnectionRequest`
Response	`UpdateJobDestinationConnectionResponse`
Description

`DeleteJobDestinationConnection`

Method	`DeleteJobDestinationConnection`
Request	`DeleteJobDestinationConnectionRequest`
Response	`DeleteJobDestinationConnectionResponse`
Description

`CreateJobDestinationConnections`

Method	`CreateJobDestinationConnections`
Request	`CreateJobDestinationConnectionsRequest`
Response	`CreateJobDestinationConnectionsResponse`
Description

`PauseJob`

Method	`PauseJob`
Request	`PauseJobRequest`
Response	`PauseJobResponse`
Description

`GetJobRecentRuns`

Method	`GetJobRecentRuns`
Request	`GetJobRecentRunsRequest`
Response	`GetJobRecentRunsResponse`
Description	Returns a list of recently invoked job runs baseds on the Temporal cron scheduler. This will return a list of job runs that include archived runs

`GetJobNextRuns`

Method	`GetJobNextRuns`
Request	`GetJobNextRunsRequest`
Response	`GetJobNextRunsResponse`
Description	Returns a list of runs that are scheduled for execution based on the Temporal cron scheduler.

`GetJobStatus`

Method	`GetJobStatus`
Request	`GetJobStatusRequest`
Response	`GetJobStatusResponse`
Description

`GetJobStatuses`

Method	`GetJobStatuses`
Request	`GetJobStatusesRequest`
Response	`GetJobStatusesResponse`
Description

`GetJobRuns`

Method	`GetJobRuns`
Request	`GetJobRunsRequest`
Response	`GetJobRunsResponse`
Description	Returns a list of job runs by either account or job

`GetJobRunEvents`

Method	`GetJobRunEvents`
Request	`GetJobRunEventsRequest`
Response	`GetJobRunEventsResponse`
Description

`GetJobRun`

Method	`GetJobRun`
Request	`GetJobRunRequest`
Response	`GetJobRunResponse`
Description	Returns a specific job run, along with any of its pending activities

`DeleteJobRun`

Method	`DeleteJobRun`
Request	`DeleteJobRunRequest`
Response	`DeleteJobRunResponse`
Description

`CreateJobRun`

Method	`CreateJobRun`
Request	`CreateJobRunRequest`
Response	`CreateJobRunResponse`
Description

`CancelJobRun`

Method	`CancelJobRun`
Request	`CancelJobRunRequest`
Response	`CancelJobRunResponse`
Description

`TerminateJobRun`

Method	`TerminateJobRun`
Request	`TerminateJobRunRequest`
Response	`TerminateJobRunResponse`
Description

`GetJobRunLogsStream`

Method	`GetJobRunLogsStream`
Request	`GetJobRunLogsStreamRequest`
Response	`GetJobRunLogsStreamResponse` stream
Description	Returns a stream of logs from the worker nodes that pertain to a specific job run

`SetJobWorkflowOptions`

Method	`SetJobWorkflowOptions`
Request	`SetJobWorkflowOptionsRequest`
Response	`SetJobWorkflowOptionsResponse`
Description	Set any job workflow options. Must provide entire object as is it will fully override the previous configuration

`SetJobSyncOptions`

Method	`SetJobSyncOptions`
Request	`SetJobSyncOptionsRequest`
Response	`SetJobSyncOptionsResponse`
Description	Set the job sync options. Must provide entire object as it will fully override the previous configuration

Messages
Enums
Services
- JobService

job.proto

Messages​

ActivityFailure​

ActivityOptions​

AwsS3DestinationConnectionOptions​

AwsS3SourceConnectionOptions​

CancelJobRunRequest​

CancelJobRunResponse​

CreateJobDestination​

CreateJobDestinationConnectionsRequest​

CreateJobDestinationConnectionsResponse​

CreateJobRequest​

CreateJobResponse​

CreateJobRunRequest​

CreateJobRunResponse​

DeleteJobDestinationConnectionRequest​

DeleteJobDestinationConnectionResponse​

DeleteJobRequest​

DeleteJobResponse​

DeleteJobRunRequest​

DeleteJobRunResponse​

GenerateSourceOptions​

GenerateSourceSchemaOption​

GenerateSourceTableOption​

GetJobNextRunsRequest​

GetJobNextRunsResponse​

GetJobRecentRunsRequest​

GetJobRecentRunsResponse​

GetJobRequest​

GetJobResponse​

GetJobRunEventsRequest​

GetJobRunEventsResponse​

GetJobRunLogsStreamRequest​

GetJobRunLogsStreamResponse​

GetJobRunRequest​

GetJobRunResponse​

GetJobRunsRequest​

GetJobRunsResponse​

GetJobStatusRequest​

GetJobStatusResponse​

GetJobStatusesRequest​

GetJobStatusesResponse​

GetJobsRequest​

GetJobsResponse​

IsJobNameAvailableRequest​

IsJobNameAvailableResponse​

Job​

JobDestination​

JobDestinationOptions​

JobMapping​

JobMappingTransformer​

JobNextRuns​

JobRecentRun​

JobRun​

JobRunEvent​

JobRunEventMetadata​

JobRunEventTask​

JobRunEventTaskError​

JobRunSyncMetadata​

JobSource​

JobSourceOptions​

JobSourceSqlSubetSchemas​

JobStatusRecord​

MysqlDestinationConnectionOptions​

MysqlOnConflictConfig​

MysqlSourceConnectionOptions​

MysqlSourceSchemaOption​

MysqlSourceSchemaSubset​

MysqlSourceTableOption​

MysqlTruncateTableConfig​

PauseJobRequest​

PauseJobResponse​

PendingActivity​

PostgresDestinationConnectionOptions​

PostgresOnConflictConfig​

PostgresSourceConnectionOptions​

PostgresSourceSchemaOption​

PostgresSourceSchemaSubset​

PostgresSourceTableOption​

PostgresTruncateTableConfig​